A Unified View of Masked Image Modeling
Authors: Zhiliang Peng
Li Dong
Hangbo Bao
Furu Wei
Qixiang Ye
Venue: Accepted by TMLR ()

Abstract: Masked image modeling has demonstrated great potential to eliminate the label-hungry problem of training large-scale vision Transformers, achieving impressive performance on various downstream tasks. In this work, we propose a unified view of masked image modeling after revisiting existing methods. Under the unified view, we introduce a simple yet effective method, termed as MaskDistill, which reconstructs normalized semantic features from teacher models at the masked positions, conditioning on corrupted input images. Experimental results on image classification and semantic segmentation show that MaskDistill achieves comparable or superior performance than state-of-the-art methods. When using the huge vision Transformer and pretraining 300 epochs, MaskDistill obtains 88.3% fine-tuning top-1 accuracy on ImageNet-1k (224 size) and 58.8 semantic segmentation mIoU metric on ADE20k (512 size). Code is enclosed in the supplementary materials.

Published in Transactions on Machine Learning Research (03/2023)
A Unified View of Masked Image Modeling
Zhiliang Peng'
pengshaionng@@@mnllsssusssommm
University of Chinese Academy of Sciences
Li Dong Microsoft Research
lidong1 (ooonggimiicrooff..com
Hangbo Bao Microsoft Research
Hiubbao@mierrssffllccom
Furu Weil Microsoft Research
fuwei@@ierrooff..com
Qixiang Ye!
guye
University of Chinese Academy of Sciences
Reviewed on OpenReview: https: // openreviee. net/ net/forttn forum? id= dd=wmGU/hhBBe
Abstract
Masked image modeling has demonstrated great potential to eliminate the label-hungry problem of training large-scale vision Transformers, achieving impressive performance on various downstream lasks. In this work, we propose a unified view of masked image modeling after revisiting existing methods. Under the unified view, we introduce simple yet effective method, termed as MASKDISTILL, which reconstructs normalized semanntc features from teacher models at the masked positions, conditioning corrupted input images. Experimental results on image classification and semantic segmentation show that MASKDISTILL achieves comparable or superior performance than state-of-the-ars methods. obtains 88.3% fine-tuning top-1 accuracy on ImageNet-lk 1k (224 size) and 58.8% semantic segmentation mIoU metric on ADE20k (512 size). The code and pretrained models will be available at https://aka. ma/ unnimim.
When using the huge vision Transformer and pretraining 300 epochs,
1 Introduction
In recent years, Transformer architectures have shown promising results in the natural language processing field (Vaswani et al., 2017) and computer vision field (Dosovitskiy el al., 2020). Transformer, in the process of scaling up, is easy to overfit the small datasets and tends to demand more and more data. In NLP, self-supervised pretraining methods based on language modeling (Radford & Narasimhan, 2018; Devlin et al., 2019; Dong et al., 2019), have successfully addressed this problem. Motivated by masked language modeling, BEIT (Bao et al., 2022) proposes masked image modeling (MTM) to relieve the label-hungry problem of vision Transformers (ViT; Dosovitskiy et al. 2020), which shows impressive results in learning visual representatious.
MIM is conceptually simple: models accept the corrupted input image and predict the target of the masked content. Take the pioneering work BEIT (Bao et al., 2022) as example, the encoder accepts corrupted image patches as input and then predicts the corresponding discrete visual tokens from the tokenizer (Ramesh et al., 2021) at the masked positions. A After that, the main difference between previous work lies in the architecture design (He et al., 2022; Chen et al., 2022a) and reconstruction targets (He et al., 2022; Liu et al., 2022b; Wei al., 2021; 2022a; Bacvski et al., 2022).
Contribution during internship Microsoft. Research. Corresponding authors.

---

Published in Transactions on Machine Learning Research (03/2023)
In this work, provide unified view of masked image modeling, as illustrated in Equation 1 and Figure 1: teacher model, a normalization layer, a student model, MTM head, and a loss function. According to it, we conduct a systemic comparison of the recent MIM works and present it in Table 1. The most significant difference is the teacher model selection, e.g., pixel values, tokenizers, pretrained models, and the momentum updated teacher.
Under this unified view, we induce simple yet effective method, named MASKDISTILL. As shown in Figure 1, the ingredients of MASKDISTILL contain teacher model based CLIP (Radford et al., 2021), fully-connection layer MIM head, layer normalization for target feature, and the Smooth-L; loss function. Compared to existing methods in Table 1, MASKDISTILL is loyal to the most straightforward design, but shows impressive results. Compared to knowledge distillation, MASKDISTILL pays more attention to extrapolating the masked patches rather than mimicking the target features.
We conduct MIM pretraining on ImageNet-1k (Russakovsky et al., 2015) for basc-, large- and huge-size ViTs. After that, we evaluate pretraining models on downst visual tasks, image classification on ImageNet-Ik, and semantic segmentation on ADE20k (Zhou et al., 2019). With the large-size CLIP teacher, MASKDISTILL using ViT-H/14 14 can achieve 88.3% accuracy on ImageNet-Ik 1k and 58.8 mloU on ADE20K, by pretraining 300 epochs.
The contributions of this study are summarized as follows:
We provide unified view of masked image modeling: a teacher model, a normalization layer, student. model, a MIM head, and loss function.
We propose a simple yet effective method, termed as MASKDISTILL.
We conduct extensive experiments on downstream tasks including ImageNet fine-tuning and semantic sogmentation. Experimental results show that the proposod approach improves performance across various settings.
2 A Unified View of Masked Image Modeling
In this section, we provide a unified view of the masked image modeling (MIM) task: a teacher model T, a normalization layer N, a student model S, a MIM head H. and an objective function L. that measures the distance between the representation of the teacher model T and that of the student model S. The pretraining task can be unified as:
MIM = C(N (I
(1)
where Lill and mmasked denote the full (original) image and the masked image respectively. According Equation 1, we summarize the recent popular MIM works in Table 1.
1) Teacher models T. According to the semantic information of target, we split. them into two groups: low-level and high-level target. For the low-level target, VIT (Dosovitskiy et al., 2020), MAE (He et al., 2022), SimMIM (Liu et al., 2022b), ConvMAE (Gao et al., 2022), HiVIT (Zhang et al., 2022) and GreenMIM (Huang et al., 2022) utilize the original or normalized pixels as the MIM target. MaskFeat (Wei et al., 2021) introduces the feature descriptor HOG (Dalal & Triggs, 2005) as the regression target. And Ge2-AE regresses pixel and frequency from 2D-Discrete Fourier Transform in parallel. As for high-level target, BETT (Bao al., 2022), CAE (Chen et al., 2022a), SplitMask (El-Nouby al., 2021), PeCo (Dong et al., 2021) and BEIT v2 (Peng et al., 2022) predict the discrete tokens (instantiated as code in the visual tokenizer (Ramesh et al., 2021; Esser et al., 2021; Peng et al., 2022). MaskFoat (Wei et al., 2021) proposes to directly regress the pretrained model (e.g., DINO (Caron et al., 2021) and DeiT (Touvron et al., 2020)). MVP (Wei et al., 2022a) extends the pretrained model to the multimodal pretrained model CLIP (Radford et al., 2021). Moreover, following the BYOL paradigm (Grill al., 2020), data2vec (Bacvski et al., 2022), MSN (Assran et al., 2022), ConMIM (Yi et al., 2022), SIM (Tao et al., 2022) and BootMAE (Dong et al., 2022) construct the regression target from the momentum updated teacher to boost itself online.
2

---

Published in Transactions on Machine Learning Research (03/2023)
Table Systemic comparisons of masked image modeling methods from a unified view.
Methods
Teacher T
Student S
MIM Head H Normalization N
Loss Function L.
Low-level pinel / feature
VIT (Dosovitskly ct al., 2020)
Pixel
VIT
FC
N/A
MAE (He et al., 2022)
Pixel
VIT
Decoder
LayerNorm
22
SimMIM (Liu et al., 2022b)
Pixel
Swin
FC
MaskFent (Wei et al., 2021)
HOG
VII
FC
f2
-e (Liu et al., 2022a)
Pixel& Frequency
VIT
Decoders
ConvMAE (Gao et al., 2022)
Pixel
Hybrid VIT
Decoder
LayerNorm
HIVIT (Zhang al., 2022)
Pixel
HIVIT
Decoder
LayerNorm
82
CIecuMIM (Huang al., 2022)
Pixel
Swin
Decoder
LayerNorm
E2
Hegi-leuel feature
BETT (Bao et al., 2022)
dVAE
VIT
FC
CrossEntropy
CAE (Chen et. ul., 2022a)
dVAE
VIT
Decoder
CrossEntropy
SplitMask (El-Nouby et al., 2021)
dVAE
VIT
Decoder
InfoNCE&CrossBut.
PeCo (Dong al., 2021)
VQGAN
VIT
FC
CrossEntropy
BETT v2 (Peng ct al., 2022)
VQ-KD
VIT
FC
CressEntropy
MaskFeat (Wei et al., 2021)
DINO
VII
FC
(E2)
Cosine
MVP (Wei et al., 2022a)
CLIP
VIT
FC
(F2)
Cosine
MILAN (Hou et al., 2022)
CLIP
VII
Decoders
22-Norm
E
MinCo (Zhou et al., 2022)
MoCoe3
VIT
FC
InfoNCE
dala2vee (Baevski et ul. 2022)
EMA
VIT
FC
LayerNorm
Smootthef
MSN (Assran et al., 2022)
EMA
VIT
FC
/
CrossEntropy
SIM (Tao al., 2022)
EMA
VII
Decoder
BatchNorm
UniCrad loss
SdAE (Chen et al., 2022b)
EMA
VII
Decoder
LayerNorm
Cosine
ConMTM (Yl al., 2022)
EMA
VIT
FC
BatchNorm
InfoNCE
ExtreMA (Wu et al., 2022)
EMA
VII
CrossAtt
LayerNorm
Cosine
BoutMAB (Dong et al., 2022)
EMA&Pixel
VIT
Decoders
LayerNorm
f2 Smooth-6
MaskDistill (Ours)
CLIP
VIT
FC
LayerNoru
2) Student models S. MIM task is suitable for the models root in attention interaction, like ViT (Dosovitskiy et al., 2020), Swin Transformers (Liu et al., 2022b), and some variants (Gao et al., 2022; Zhang et al., 2022). Because backbone architecture not the primary focus of this study, we choose the vanilla ViT (Dosovitskiy et al., 2020) as the analytical anchor.
3) MIM Heads H. BETT (Bao et al., 2022) uses a simple fully-connection (FCC layer as the task head to generate prediction at the masked positions. MAE (He et al., 2022) introduces a decoder to decouple the masked prediction task from the encoder. In fact, the aim of the decoder in MAE is still to predict the target pixel at the masked positions. Therefore, we consider the decoder as a MIM head in Table 1. And this decoupling decoder is adopted by many recent works (Liu el al., 2022a; Gao el al., 2022; Zhang el al., 2022; Chen al., 2022a; El-Nouby et al., 2021; Tao et al., 2022: Dong et al., 2022).
4) Normalization Layers N. MAE (He et al., 2022) also introduces per-patch normalized pixels (i.e., layer normalization without affine transformation) as the target to boost local pixels contrast, resulting in better performance. Meanwhile, normalization is usually applied for avoiding feature collapse in methods based on contrastive learning (Grill et al., 2020; Chen et al., 2020). Similarly, EMA-based MIM methods (Tao et al., 2022; Baevski et al., 2022; Yi et al., 2022) adopt various normalization methods to stabilize training as well as boost performance. There no collapse issue when the teacher is pixels or frozen models by default.
f2 losses are appropriate for feature regression. When the target is discrete tokens, the cross entropy loss is the primary choice. Notably, after applying layer normalization, the variance of target fcature rises, resulting in volatile loss, whereas Smooth-e, loss is a trade-off between & and f,, performing more stable. Of course, cosine similarity loss is also an alternative choice.
5) Loss functions L. When the target is pixel or feature,
From Table 1, one can find that the main difference the teacher models: pixel, momentum-updated teachers, and pretrained models. Pixel is easy to access but struggles with low-level semantic knowledge. Momentum-updateel teachers do not need extra models datasets but tend to suffer from the collapse
3

---

Published in Transactions on Machine Learning Research (03/2023)
Teacher
Normlization
Pixel, EMA, CLIP, DINO, MAE, BEİT...
LN, BN, Identity,
LI, L2, Smooth-L.1, Cosine,
Input Image
Loss
Student
MIM Head
ViT, Swin,
FC, Decoder
Mask
Figure 1: Unified view of the masked image modeling framework. The bold text denotes the default ingredients MASKDISTILL.
issue. Pretrained models are off- the-shelf and contain more rich semantic information than pixels, but how to prepare a high-quality teacher model is an essential problem.
3 Masked Distillation
Knowledge distillation (Hinton et al., 2015) has shown to be a promising approach for compressing a large model (referred as the teacher model) into a small model (referred as the student model), which utilizes much fewer parameters and computations while attaining comparable results on downstream tasks.
Based on the unified view, offer simple yet effective method, named MASKDISTILL, to distill student model in masked image modeling fashion. However, purpose is not to compress the teacher model T into the student model S, but to boost S to outperform T. We instantiate the student model S as ViT (Dosovitskiy et al., 2020) for comparison with others.
Specially, given the input image æ E #x where (H, W) the resolution and C the number of image channels, the student S first divides æ into N non-overlapping patches {x]) and then linear projects it into patch embeddings [cm), Following that, select roughly 40% of the image patch embeddings to be masked, in block-wise strategy (Bao al., 2022). Denoting the masked position set M, use shared learnable embedding to replace the original patch embeddings ef if i E M. After that, get the masked sequence:
etM - Mi eM (1- 5(i M))
(2)
where 8(.) is the indicator function. Subsequently, we prepend a learnable class token eass and add the learnable positional embeddings, and then feed those into stacked transformer blocks. Lastly, a masked image modeling head (usually instantiate as a fully-connected layer) applied for predicting feature O e R(N+1)xD, where is the dimension of target features.
Given a pretrained teacher model T, like DINO (Caron al., 2021) and CLIP (Radford et. al., 2021), the same image is fed into T to get the target feature [ natch-to-patch. To ensure that the output resolution of S and T is the same, the input resolution for T should be adjusted. Finally, the training objective of MASKDISTILL can be formulated as:
LMAKKDSSTLL = Σ Iog(t(())))))= = [M
Σ Smooth-Li(o, LN((t)) NEM
(3)
NEM
where LN is the layer normalization without affine transformation.
4

---

Published in Transactions on Machine Learning Research (03/2023)
Table 2: Fine-tuning results on ImageNet-IK and ADE20k. 'PT Epochs' denotes the pretraining epochs.
'Rel. Pos. means relative positional embeddings.
Teacher
PT Epochs
Rel. Pos.
Classification Top-1 Ace (%)
Segmentation mloU (%)
Methods
Model
Data
Bise-size models (ViT-B/16)
BEIT (Bao et al., 2022)
DALL-E
250M
800
X
83.2
45.6
MAE (He et al., 2022)
Pixel
1600
X
83.6
48.1
CAE (Chen et al., 2022a)
DALL-E
250M
1600
X
83.9
50.2
SdAE (Chen et al., 2022b)
EMA
300
N/A
84.1
48.6
SIM (Tao et al., 2022)
EMA
1600
N N/A A
83.8
N/A
MaskFeat (Wei et al., 2021)
HOG
1600
N
84.0
N/A
PeCo (Dong et al., 2021)
VQGAN
IN-1k
300
N A
84.1
46.7
PeCo (Dong et al., 2021)
VQGAN
IN-1k
800
N/A
84.5
48,5
data2vec (Baevski et al., 2022)
EMA
800
X
84.2
N/A
CLIP (Radford et al., 2021)
Text
/
X
84.9
51.1
MVP (Wei et al., 2022a)
CLIP-B
400M
300
N/A
84.4
52.4
BErT v2 (Peng et al., 2022)
VQ-KD
400M
1600
85.5
53.1
MASKDISTILL (ours)
CLIP-B
400M
300
X
84.9
52.7
MASKDISTILL (ours)
CLIP-B
400M
300
/
85.0
53.8
MASK (ours)
CLIP-B
400M
800
V
85.5
54.3
Targe-size models (VIT-I/16))
MaskFeat (Wei et al., 2021)
HOG
1600
N/A
85.7
N/A
MAE (He et al., 2022)
Pixel
1600
X
85.9
53.6
CAE (Chen et al., 2022a)
DALL-E
250M
1600
X
86.3
54.7
data2vec (Baevski ct al., 2022)
EMA
1600
X
86.6
N/A
BErT v2 (Peng et al., 2022)
VQ-KD
400M
1600
V
87.3
56.7
MILAN (Hou et al., 2022)
CLIP-B
400M
400
X
86.7
55.3
MASKDISTILL (ours)
CLIP-B
400M
300
86.8
56.3
MASKDISTILL (ours)
CLIP-B
400M
800
87.1
56.5
Table Fine-tuning results on ImageNet- 1K and ADE20k. The teacher is CLIP ViT-L/14.
Model
PT
Rel.
Classification Top-1 Acc (%)
Segmentation mIoU (%)
Methods
Size
Epochs
Pos.
Scaling up to larger teacher, CLIP
ViT-L/14
MASKDISTILL (ours)
ViT-B/16
300
/
85.3
54.3
MASKDISTILL (ours)
ViT-L/16
300
/
87.6
57.9
MASKDISTILL (ours)
VIT-H/14
300
/
88.3
58.8
4
Experiments
We perform pretraining and then evaluate fine-tuning performance on various downstrream tasks, such as image classification and semantic segmentation. Moreover, we conduct ablation studies compare the contributions of different design choices.
4.1 Setup
For all protraining experiments, we only use the ImageNet-1k 1k dataset (Russakovsky al., 2015) contains 1.28M images. We adopt the block masking strategy to corrupt the input images for the student model, but keep the full images for the teacher, to construct the asymmetric informational bottleneck. All the teacher
5

---

Published in Transactions on Machine Learning Research (03/2023)
Table 4: Robustness evaluation on ImageNet vari- ants (Hendrycks et al., 2021b;a; Wang al., 2019). leacher model CLIP ViT-Base (Radford et al., 2021).
Table 5: MASKDISTILL us knowledge distillation. The
ImageNett Adversarial
ImageNett Rendition
ImageNet Sketch
Student Models
Mask
Pretaining Epochs
Classification Accuracy (%)
Methods
Ratios
VVT-B/16
0
300
85.3
MAE
35.9
48.3
34.5
ViT-B/16
BETT v2
54.4
61.0
45.6
40%
300
85.0 (-0.3)
MASKDISTTLL
53.3
64.4
47.3
0
800
85.2
ViT-B/16
VIT-L/16
40%
800
85.5 (+0.3)
MAE
57.1
59.9
45.3
BEIT v2
69.0
69.9
53.5
ViT-L/16
0 40%
300
85.4
MASKDISTILL
69.0
75.3
56.9
300
86.8 (+1.4)
model checkpoints are from the official publication. When utilizing CLIP ViT-L/114 as a teacher, we set the input image resolution to 196x 196 for the teacher to match the number of patches with student ViT-B/ 16 or ViT-I/16. As for the student model, we use the ViT-Base/Large equipped relative positional embeddings following BETT (Bao al., 2022; Peng al., 2022). For the pretraining selting, we mainly follow BETT (Bao et al., 2022; Peng et al., 2022): batch size 2048, learning rate 1.5e-3, AdamW optimizer with weight decay 0.05, drop path 0.1 (0.2) for ViT--asselarge), block-wise mask 40,, epochs 300/800. More details can be found in Appendix.
Evaluation. We consider the popular evaluating protocol for image classification on ImageNet-Ii 1k dataset: foneltnning top-1 accuracy. We adopt the BETT (Bao et al., 2022) fine-tuning recipe: For ViT-Base, we fine-tune it for 100 epochs with 20 epochs warm-up, and use AdamW optimizer with weight decay 0.05, learning rate 5e-4, and decays in cosine schedule, layer decay 0.65; For ViT-Large, we fine-tune it for 50 epochs with 5 epochs warm-up, layer decay 0.75. For ViT-Huge, we fine-tune it for 30 epochs with 5 epochs warm-up, layer decay 0.85. All the resolutions of input images are 224 x 224.
As for the semantic segmentation task, we evaluate the mloU metric on ADE20K dataset (Zhou et al., 2019) with UperNet (Xiao et al., 2018) framework. The input image resolution for training and evaluating are 512 x 512. Remarkably, for the ViT-H/14 in Table 3, we convert. it to ViT-H/16 for semantic segmentation task. Similarly, AdamW optimizer with weight decay of 0.05 is applied. Additionally, the training steps are 160K, and the batch size is 16. And we employ learning rate {5c-5, 80-5, 1c-4), layer decay 0.75 (0.85), drop path 0.1 (0.2) for ViT-Base (Large). More details can be found in Appendix.
4.2 Main Results
Table reports the top-1 accuracy of some self-supervised methods on ImageNet-Ik using ViT (Dosovitskiy et al., 2020) models. For ViT-base, MASKDISTHLL with 800 epochs pretraining schedule obtains 85.5% top-1 accuracy, surpasses CLIP (Radford et al,, 2021), MVP Wei et al. (2022a), data2vec (Baevski et al., 2022) and MaskFeat (Wei et al., 2021) by 0.6%, 1.1%, 1.3% and 1.5% respectively. And MASKDISTILL also achieves comparable performance with BETT v2 (Peng et al., 2022) on ImageNet-1k but outperforms BETT v2 by 1.2 mIoU on ADE20k. More comparison with BEIT v2 can be found in Section 4.4. When scaling up the student to ViT-Large, MAsKDisTILL achieves 86.8% top-1 accuracy and 56.3 mloU. Compared to the recently proposed MILAN (Hou al., 2022), MASKDISTILL outperforms it by 1% on the semantic segmentation task under the less pretraining epochs.
In Table 3, we use the CLIP ViT-Large/14 checkpoint as the teacher model and pretrain student models for 300 epochs. One can see that MASKDISTILL can get consistent improvements compared to teacher CLIP ViT-Base/16. Remarkably, MASKDISTILL can reach 88.3% accuracy on ImageNet-Il and 58.8 mloU on ADE20k by using the ViT-Huge backbone.
Robustness evaluation. Following MAE (He al., 2022) and BETT v2 (Peng et al., 2022), we test the robustness of MASK on three ImageNet validation sets, ie., AetageNet-AAvessaae (Hendrycks et al.,
6

---

Published in Transactions on Machine Learning Research (03/2023)
Table 6: Fow-shot image classification on ImageNet-Ik: We freeze the backbone and only learn the classifier
during training.
Few
Shot Numbers
Methods
FT
k=2
k=4
k=8
k=16
k=32
ViT-B/16
BEIT (Bao et al., 2022)
83.2
1.7
3.0
5.0
7.0
8.9
MAE (He et al., 2022)
83.6
11.5
21.5
31.5
39.5
46.4
CLIP (Radford et al., 2021)
84.9
35.4
45.2
55.1
61.3
65.6
BEIT v2 (Peng et al., 2022)
85.5
36.6
47.6
56.3
63.0
67.6
MASKDISTILL (ours)
85.5
37.8
48.7
56.3
62.3
66.2
2021b), ImageNet--Rendttinn (Hendrycks al., 2021a) and ImageNet-Sketch (Wang al., 2019). In Table 4, both MAE and BETT v2 pretrain 1600 epochs, while MASKDISTILL pretrains 800 epochs but achieves comparable superior performance.
4.3 Comparison with Knowledge Distillation
In Table 5, we compare MASKDISTILL with knowledge distillation, which can be considered as special case of MASKDISTILL where the mask ratio is 0 and loss is calculated on all patches. Knowledge distillation surpasses MASKDISTILL by 0.3% when the pretraining schedule is 300 epochs, but is inferior to MASkDistILA by 0.3% when the pretraining schedule 800 epochs. Remarkably, MASKDISTILL outperforms knowledge distillation by a significant gain when the student model scales up to largo-size models. The commonly used teacher model is CLIP ViT-Base, which reaches 84.9% fine-tuning accuracy in terms of image classification ImageNet-lk.
When the student is larger than the teacher, the student is casy to fully reconstruct the latent space of the teacher without information bottleneck. This is why ViT-L/16 obtains comparable performance with ViT-B/16 (85.4% os 85.3% in Table 5). But in MASKDISTILL, under the condition of the corrupted input, the student encouraged to extrapolate the masked patches, rather than mimicking features at visible patches.
4.4 Comparison with BEiT v2
In BEIT v2 (Peng et al., 2022), CLIP ViT-Base as the teacher model is responsible for distilling vector quantized visual tokenizer, which provides the supervision for the subsequent MIM phase. But compared with MASKDISTILL, the quantized mechanism in BEIT v2 omits some fino-grained details from the teacher model. And these details are beneficial to the fast convergence of MASKDISTILL, e.g., MASKDISTILL, achieves comparable image classification performance with 800 epochs pretraining while BErT v2 need to pretrain 1600 epochs, as demonstrated in Table 2. That is, MASKDISTILL can avoid the codebook collapse problem in the tokenizer training phase (Peng et al., 2022) and achieve comparable performance. Mcanwhile, such fine-grained details as supervision enhance the robustness MASKDISTULA as shown in Table
4.5 Few shot classification
In Table 6, we report the fewshot image classification results of BErT (Bao et al., 2022), MAE (He et al., 2022), CLIP (Radford et al., 2021), BEIT v2 (Peng et al., 2022) and MASKDISTILL ImageNet-1k 1k (Russakovsky et al., 2015). We randomly choose k samples from each category and use them to learn classifier, while keeping other parameters frozen during training. We use the entire validation set for evaluation. For cach method, we use their public model weights and sweep a wide range of learning rales for fair comparison. From Table 6, one can see that MASKDISTILL can consistently surpass the teacher under various settings. When k-2/4/8, MASKDISTILL, can achieve the best fewshot performance. Morcover, we also find that fewshot. performance is correlated with fine- man-tuning performance in terms of MIM, which consistent with observation in contrastive learning (Fricsson al., 2021).

---

Published in Transactions on Machine Learning Research (03/2023)
Table Comparison of the MIM time consuming and GPU memory.
Method
Supervision
Average step time (s)
GPU memory (G)
ViT-B/16 batchsize 64 on each GPU
MAE (He et al., 2022)
Normalized pixels
0.358
10.6
data2vec (Baevski et al., 2022)
EMA features
0,636
13.5
BEiT v2 (Peng et al., 2022)
VQ-KD tokens
0.605
15.5
DISKIL
CLIP-B features
0.487
12.5
VTT-L/16, batchsize 32 on each GPU
MAE (He et al., 2022)
Normalized pixels
0.551
13.0
data2vec (Baevski et al., 2022)
EMA features
1.079
21.9
BETT v2 (Peng et al., 2022)
VQ-KD tokens
0.769
21.8
MISKDISTILL
CLIP-B features
0.718
19.7
4.6 Comparison on Training Cost
In Table 7, We test all models under the same settings to compare the training cost. Average step time is calculated from (total training time) (total steps). GPU memory is measured when the training phase is stable. MAE (He et al., 2022), data2vec (Bacvski et al., 2022) and BEiT v2 (Peng et al., 2022) are evaluated by using their official codebase. Compared with using pixel values as reconstruction, other methods tend to spending more time on obtaining targets, including our MASKDISTILA. Compared with EMA--basdd data2vec and tokenizer-based BErT v2, MASKDISTILL enjoys faster training time and lower GPU memory cost.
4.7 Ablation Studies
Teacher models. We collect some popular models to act as the teacher MASKDISTILL, and pretrain a student model ViT-Base for 300 epochs in MTM fashion. The performance of the teacher and student are shown in Table 8. From #1 to #6, where teacher models are CLIP and SLIP (Mu et al., 2021) trained on the image-text pair datasets (YFCC15M, CC3M, CC12M and private 400M) in language-guided contrastive way, MISKDISTILL consistently boost the teacher model by 0~3.3% accuracy. For #7, teacher model is ResNet, which is trained in the supervised way. Despite the gap in architecture, the student still enjoys the significant. gain (83.5% us 76.2%). From #8 to #9, teacher models SimCL. (Chen al., 2020) and DINO (Caron al., 2021) only use image data. MASKDISTILL boosts them by 1.6% and 0.9% respectively.
Comparing #1, #2 and #8 in Table 8, where the same dataset and training epochs are applied to teachers, students in #1 and #8 respectively achieve 83.8% and 84.1%, but the former using the text information and the later not, implying that the language-guided supervision is not essential. Moreover, comparing # #1 and #9, both teacher and student in #9 trained on ImageNet-lk can reach comparable performance with those #1 #5, which further suggests that the extra language information not the key.
From #10 to #12, we choose the model trained by MIM itself act as the teacher model. We find that MASKDISTILL consistently outperform the corresponding teacher. However, comparing #9 with # 10, where teacher can reach the same fine-tuning accuracy, students in #8 can obtain better performance in terms of fine-tuning accuracy and segmentation mIoU than those in #9, indicating that contrastive pretrained models tend to be the better but not the only solution.
Loss functions & Normalization. We compare MSE, cosine similarity and smooth-li loss equipped with various normalization layers, then present the results in Table 9. From Table 9, see that smooth-l, loss equipped with LN can achieve better performance under the supervision of both DINO and CLIP, indicating that Normalization plays an important role in masked image modeling task.
Target layer selection. Usually, the deeper layer feature of a model is biased the special task, e.g., image-image contrastive learning in DINO and image-text contrastive learning in CLIP. But whether it.
8

---

Published in Transactions on Machine Learning Research (03/2023)
Table 8: Ablation studies on teacher models used MASKDISTILL. For {CLIP, SLIP, SimCLRJi the line-tuning accuracy and model checkpoint are from SLIP (Mu el. al., 2021). For CLIP* and DINO, we use the official model checkpoint and follow BETT (Bao et al., 2022) fine-tuning recipe to get the top-1 accuracy. The teacher models in all methods are ViT-Base model except ResNet-50 (He et al., 2016). The student. model is ViT-Base and is pretrained for 300 epochs.
Teacher Model T
Student Model S
Teacher
Data
Text
ImageNet (%)
ImageNet (%)
ADE20k (%)
#1
CLIP
YFCC15M
80.5
83.8 (+3.3)
47.4
#2
SLIPt
YFCC15M
/
82.6
84.3 (+1.7)
49.9
#3
SLIPt
YFCC15M
/
83.4
84.6 (+1.2)
50.8
#4
CLIP
CC3M
/
79.5
83.7 (+4.2)
45.7
#5
CLIP
CC12M
82.1
84.1 (+2.0)
48.3
#6
CLIP
Private 400M
/
84.9
85.0 (+0.1)
53.8
#7
ResNet
ImageNet-Il
X
76.2
83.5 (+7.3)
46.9
#8
SimCLRt
YFCC15M
X
82.5
84.1 (-1.6)
49.4
#9
DINO
ImageNet-1k
X
83.6
84.5 (-0.9)
50.4
10
MAE
ImageNet- 1k
X
83.6
84.3 (-0.7)
49.3
11
BETT
ImageNet-ll
X
83.2
83.8 (-0.6)
46.6
BEiT v2
ImageNet-1k 1k
X
84.7
85.0 (+0.3)
52.1
86
56
85
65
52
84
48
83
random mask
random mask
random mask
block mask
block mask
block mask
82
25
44
0.2
0.4
0.6
0.8
0.2
0.4
0.6
0.8
0.2
0.4
0.6
0.8
(a) Fine-tuning Accuracy
(b) Linear-probing Accuracy
(c) Segmentation mloU
Figure 2: The block-wise mask us random mask, under various mask ratios.
is beneficial for MASKDISTILL is not revealed. We conduct experiments target feature from last layer, average of last 3 layers and average of last 6 layers. As shown in Table 10, the last layer's features are better for DINO teachers while the last 6 layers' features are better for CLIP teachers. Moreover, results on the segmentation task show that the last layer features as target are superior. Therefore, we choose the last layer fcature as the default target feature for all experiments.
Masked strategy. For the masked strategy, we evaluate the block-wise (Bao et al., 2022) masked method and random masked mothod Figure 2. The block-wise masked method performs better than random mask under low mask ratios, while worse than random mask under high mask ratios. Taking the three evaluation protocols (fine-tuning ImageNet-1k, linear-probing ImageNet- and semantic segmentation ADE20k) into consideration, we choose the block-wise mask with 40% mask ratio as the final decision.
4.8 Analysis: MIM Enhances Shape Bias
We explore whether the masked image modeling methods can enhance the shape-biased ability or not. The fraction of correct decisions based on object shape is characterized as shape bias. Naseer al. (2021) present.

---

Published in Transactions on Machine Learning Research (03/2023)
Table 9: Ablation study loss functions and nor- Table 10: Ablation study of target feature selection
malization layers. All models are pretrained for 300 epochs.
in MASKDISTILL. All models are pretrained for 300
epochs.
T
L
Norm
ImageNet
ADE20k
Target
ImageNet
ADE20k
MSE
X
84.3
49.6
Tast layer
84.5
50.4
DINO
Cosine
(f2
84.5
49.6
DINO
Mean (last 3 layers)
84.4
49.7
Smooth-41
LN
84.5
50.4
Mean (last layers)
84.3
49.8
MSE
X
84.6
52.8
Last layer
85.0
53.8
CLIP
Cosine
(l2)
84.9
52.9
CLIP
Mean (last 3 layers)
85.0
53.5
Smooth-41
BN
84.9
53.1
Mean (last 6 layers)
85.1
53.4
Smooth-41
LN
85.0
53.8
Fraction of 'shape' decisions
Fraction of 'shape' decisions
0.9
0.8
0.7
0.6
0.5
0.4
0.3
0.2
0.1
0.9
0.8
0.7
0.6
0.5
0.4
0.3
0.2
0.1
0
Shape categories
Shape categories
0.1 0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
Fraction of 'texture' decisions
Fraction of 'texture' decisions
Figure 3: Shape-biased analysis under the teacher supervision of CLIP ViT-B/16 (left:) and MAE ViT-B/16 (right). Circle, triangle and star denote humans, teachers and students, respectively. Vertical lines are the corresponding average values. Masked image modeling enhances the shape bias. (Best viewed in color)
that human usually is much more shape-biased compared with supervised classification models, such as convolutional networks, and vision Transformers. We evaluate the shape bias capacily on a stylized version of ImageNet (Naseer al., 2021) by using the checkpoints fine-tuned on the original ImageNet-Ik dataset. As shown in Figure masked image modcling tends to promote the shape bias of the models. The results partially explains why MASKDISTILL generalizes better ImageNet variants shown in Table 4.
5 Related Work
Masked image modcling. Masked language modeling task root in Transformers has achieved great success in learning strong language representations in recent years (Devlin et al., 2019; Dong el al., 2019; Bao et al., 2020). Inspired by it, BErT (Bao et al., 2022) proposes a tass--the--pediic framework to recover discrete visual tokens (Ramesh al., 2021), which shows the great potential of masked image modcling for the computer vision field. After that, various target supervision has been explored under the masked image modeling framework, such original normalized pixels (He et al., 2022; Dong et al., 2021; Liu et al.,
10

---

Published in Transactions on Machine Learning Research (03/2023)
2022b; Gao et al., 2022; Liu et al., 2022a; Zhang al., 2022; Huang et al., 2022), high-level features (Wei et al., 2021; 2022a; Peng et al., 2022; Zhou et al., 2022; Hou et al., 2022), and EMA-updated models (Baevski et al., 2022; Assran et al., 2022; Tao et al., 2022: Chen et al., 2022b; Yi et al., 2022; Wu et al., 2022; Dong et al., 2022). In this work, we decouple and analyze the components the recent masked image modcling works, and then propose a simple yet effective paradigm for masked image modeling.
Contrastive learning. As simple but effective self-supervised method, contrastive learning methods have ushered in rapid progress in recent years. The main idea is to enforce similarity over augmented views of an image and push the views augmented from other images away (Dosovitskly et al., 2016; Wu et al., 2018; Hjelm et al., 2019; He et al., 2020; Chen et al., 2020), or avoid model collapse after removing negative pairs (Grill et al., 2020; Chen & He, 2020; Chen et al., 2021; Caron et al., 2021). In the multimodal field, CLIP (Radford et al., 2021) and ALIGN (Jia et al., 2021) can learn image-lnguaaee alignment reprosentation, by grouping positive image- pairs (an image and corresponding lag or caption) closer and separaling negalive image- text pairs. And SLIP (Mu et al., 2021) combines language supervision and image self-supervision to further boost the learned visual representations. In this work, we consider contrastive models as the target for masked image modeling.
Knowledge distillation. Knowledge distillation (Hinton et al., 2015) considers the output of the teacher model as the pseudo label to learn the student model. Such a strategy squeezes the potential of small models and brings impressive gains. After that, knowledge distillation is transferred to various tasks (Touvron et al., 2020; He et al., 2019; Yang et al., 2021) and domains (Jiao et al., 2020; Wang et al., 2020). Wei et al. (2022b) proposes that using the normalized feature from teacher fully distiils a same size student. However, in this work, MASKDISTILL aims to reconstruct the corresponding teacher output at masked patches rather than mimicking the teacher's feature each patch.
6 Conclusion and Limitations
We summarized the existing MIM works upon the proposed unified view: teacher models, student models, normalization layers and MIM heads. After that, we propose a simple yet effective method, termed as MASKDISTILI, which predicts the normalized semantic features from CLIP's visual encoder at masked positions based on the corrupted input image. The simple framework beats many previous works with special designs and shows impressive performance across model sizes and tasks. In the future, we would like to explore the proposed method for multimodal pretraining (Wang et al., 2022).
The proposed MASKDISTILL requires extra teacher model, similar to the tokenizer in BEIT series. Compared with the methods using pixels as targets, the teacher model in MASKDISTILL needs to spend extra time to obtain target features. Meanwhile, we point out that toggaaee-giided supervision is not essential in Subsection 4.7 on the academically accessible multi-model datasets, YFCC15M. But whether this conclusion is correct on private 400M image-text pair datasets remains an unknown question.
References
Mahmoud Assran, Mathilde Caron, Ishan Misra, Piotr Bojanowski, Florian Bordes, Pascal Vincent, Armand Joulin, Michael G. Rabbat, and Nicolas Ballas. Masked siamese networks for labcl-officient learning. ArXio, abs/2204.07141, 2022.
Alexei Baevski, Wei-Ning Hsu, Qiantong Xu, Arun Babu, Jiatao Gu, and Michael Auli. Data2vec: A general framework for self-supervised learning in speech, vision and language. arXiv preprint 2tt::220000555 2022.
Hangbo Bao, Li Dong, Furu Wei, Wenhui Wang, Nan Yang, Xiaodong Liu, Yu Wang, Jianfeng Gao, Songhao Piao, Ming Zhou, and Hsiao- Hon. UniLMv2: Psoudo-maskod language models for unified language model pre-training. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, volume 119 of Proceedings of Machine Learning Research, pp. 642 652. PMIRR 2020.
11

---

Published in Transactions on Machine Learning Research (03/2023)
Hangbo Bao, Li Dong, Songhao Piao, and Furu Wei. BEiT: BERT pre-training of image transformers. In International Conference on Learning Representations, 2022.
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. arXiv preprint 4rXix:2104,14294, 2021.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. preprint arXiv:2002 05709, 2020.
Xiaokang Chen, Mingyu Ding, Xiaodi Wang, Ying Xin, Shentong Mo, Yunhao Wang, Shumin Han, Ping Luo, Gang Zeng, and Jingdong Wang. Context autoencoder for self-supervised representation learning. arXiv preprinl ar Xrv: 03026, 2022a.
Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. preprinl orXi::2011.10560, 2020.
Xinlei Chen, Saining Xie, and Kaiming He. An empirical study of training self-supervised vision transformers. ArXiv, abs/2104.02057, 2021.
Yabo Chen, Yuchen Liu, Dongsheng Jiang, Xiaopeng Zhang, Wenrui Dai, Hongkai Xiong, and Qi Tian. Sdae: Self-distillated masked autoencoder. ArXiu, Abs/2208.00049, 2022b.
Navnoct Dalal and Bill Triggs. Histograms of oriented gradients for human detection. In 2005 IEEE computer society conference on computer vision and pattern recognition (CVPR'05), volume 1, pp. 986-893. leee, 2005.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina. Toutanova. BERT: pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 4171-4186. Association for Computational Linguistics, 2019.
Li Dong, Nan Yang, Wenhui Wang, Furu Wei, Xiaodong Liu, Yu Wang, Jianfeng Gao, Ming Zhou, and Hsiao-Wuen Hon. Unified language model pre-training for natural language understanding and generation. In Advances in Neural Information Processing Systems 32: Annual Conference on Newral Information Processing Systems 2019, NewrIPS 2019, December 8-14, 2019, Vancowver, BC, Canada, pp. 13042-13054, 2019.
Xiaoyi Dong, Jianmin Bao, Ting Zhang, Dongdong Chen, Weiming Zhang, Lu Yuan, Dong Chen, Fang Wen, and Nenghai Yu. Peco: Perceptual codebook for bert pro-training of vision transformers. arXiv preprint arXiv:2111.12711 2021.
Xiaoyi Dong, Jianmin Bao, Ting Zhang, Dongdong Chen, Weiming Zhang, Lu Yuan, Dong Chen, Fang Wen, and Nenghai Yu. Bootstrapped masked autoencoders for vision bert pretraining. ArXiu, abs/2200.007116 2022.
A. Dosovitskiy, P. Fischer, J. Springenberg, M. Riedmiller, and T. Brox. Discriminative unsupervised feature learning with exemplar convolutional neural networks. IEEE Transactions on Pattern Analysis & Machine Intelligence, 38(09):1734 1747, sep 2016. ISSN 1939-3539. doi: (0.1009/TPAM..201522
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. preprint 1xXi:22010.11929 2020.
Alaaeldin El-Nouby, Gautier Izacard, Hugo Touvron, Ivan Laptev, Hervé Jegou, and Edouard Grave. Are large-scale datascts necessary for self-supervised pre-training? arXiv preprint arXiv:2112. 10740, 2021.
Linus Ericsson, Henry Gouk, and Timothy M Hospedales. How well do self-supervised models transfer? In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognétion, pp. 5414-5423, 2021.
12

---

Published in Transactions on Machine Learning Research (03/2023)
Patrick Esser, Robin Rombach, and Bjorn Ommer. Taming transformers for high-resolution image synthesis. In CVPR, pp.
2021.
Peng Gao, Teli Ma, Hongsheng Li, Jifeng Dai, and Yu Jiao Qiao. Convmae: Masked convolution meets masked autoencoders. ArXiv, abs/ 00//22255.3822 2022.
Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre H Richemond, Flena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Daniel Guo, Mohammad Gheshlaghi Azar, Bilal Piot, Koray Kavukeuoglu, Rémi Munos, and Michal Valko. Bootstrap your own latent: new approach self-supervised learning. In NeurIPS, 2020.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, 2016.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In CVPR, 2020.
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In CVPR, 2022.
Tong Hc, Chunhua Shen, Zhi Tian, Dong Gong, Changming Sun, and Youliang Yan. Knowledge adaptation for efficient semantic segmentation. 2019 IEEE/CVE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 578 778-587, 2019.
Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, Dawn Song, Jacob Steinhardt, and Justin Gilmer. The many faces of robustness: A critical analysis out-of-distribution generalization. In IEEE ICCV, 2021a.
Dan Hendrycks, Kevin Zhao, Steven Basart, Jacob Steinhardt, and Dawn Song. Natural adversarial examples. In JEEE CVPR, 2021b.
Geoffrey Hinton, Oriol Vinyals, Jeff Dean, al. Distilling the knowledge in neural network. arXiv preprint arXiv:1508.025: 2(7), 2015.
R. Devon Hjehm, Alex Fedorov, Samuel Lavoie- Marchilonn, Karan Grewal, Phil Bachman, Adam Trischler, and Yoshua Bengio. Learning deep representations by mutual information estimation and maximization. In International Conference on Learning Representations, 2019.
Zejiang Hou, Fei Sun, Ken-Kuang Chen, Yuan Xie, and S. Y. Kung. Milan: Masked image pretraining on 00009, 2022.
language assisted reprosentation. ArXiv,
Lang Huang, Shan You, Mingkai Zheng, Fei Wang, Chen Qian, and T. Yamasaki. Green hierarchical vision transformer for masked image modeling. ArXiv, abs/2225.1335 2022.
Chao Jia, Yinfci Yang, Ye Xia, Yi-Ting Chen, Zarana Parckh, Hicu Pham, Quoe Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision-language representation learning with noisy text supervision. In International Conference on Machine Learning, pp.
PMLR, 2021.
Xiaoqi Jiao, Yichun Yin, Lifeng Shang, Xin Jiang, Xiao Chen, Linlin Li, Fang Wang, and Qun Liu. Tinybert: Distilling bert for natural language understanding. ArXin, 1tt///999..33.1... 2020.
Hao Liu, Xinghua Jiang, Xin Li, Antai Guo, Deqiang Jiang, and Bo Ren. The devil is in the frequency: Geminated gestalt autoencoder for sclf-supervised visual pre-training. ArXiu, abs/2204.08827,, 2022a.
Ze Liu, Han HU, Yutong Lin, Zhuliang Yao, Zhenda Xie, Yixuan Wei, Jía Ning, Yue Cao, Zheng Zhang, Li Dong, Furu Wei, and Baining Guo. Swin transformer v2: Scaling up capacity and resolution. In International Conference on Computer Vision and Pottern Recognition (CVPR), 2022b.
Norman Mu, Alexander Kirillov, David Wagner, and Saining Xie. Slip: Self-supervision meets language-image pre-training. arXiv preprint arXiv:2112 12. 12750, 2021.
13

---

Published in Transactions on Machine Learning Research (03/2023)
Muhammad Muzammal Nascer, Kanchana Ranasinghe, Salman H. Khan, Munawar Hayat, Fahad Shah- baz Khan, and Ming-Hsuan Yang. Intriguing properties of vision transformers. Advances in Newral Information Processing Systems, 34:23296 23308, 2021.
Zhiliang Peng, Li Dong, Hangbo Bao, Qixiang Ye, and Furu Wei. BEiT v2: Masked image modeling with vector-quanlized visual tokenizers. arXiv preprint arXim:2008.000 06366, 2022.
Alec Radford and Karthik Narasinhan. Improving language understanding by generative pro-training. 2018.
Alec Radford, Jong Wook Kim, Chris Hallacy Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. Tn ICML, pp. 8748 8763. PMLR, 2021.
A. Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and llya Sutskever. Zero-shot text-to-image generation. ArXio, abs/ abs/2102.12092, 2021.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C Berg, and Li Fei-Fei. Imagenct large scale visual recognition challenge. IJCV, 2015.
Chenxin Tao, Xizhou Zhu, Gao Huang, Yu Qiao, Xiaogang Wang, and Jifeng Dai. Siamese image modeling for self-supervised vision representation learning. arXiv preprint NNw:2206.01204 2022.
Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-cfficient image transformers & distillation through attention. preprint arXiv:2012, 12877, 2020.
Ashish Vaswani, Noam Shazoer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett (cds.), Advances in Neural Information Processing Systems 30: Annual Conference Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pp. 5998-60999 00088600,00000 2017.
Haohan Wang, Songwei Ge, Zachary Lipton, and Eric P Xing. Learning robust. global representations by penalizing local predictive power. In Advances in Neural Information Processing Systems, pp. 10506-10555 2019.
Wenhui Wang, Furu Wei, Li Dong, Hangbo Bao, Nan Yang, and Ming Zhou. MiniLM: Deep self-attention distillation for task-agnostic compression of pre-trained transformers. ArXiv, abs/2002.10957,, 2020.
Wenhui Wang, Hangbo Bao, Li Dong, Johan Bjorck, Zhiliang Peng, Qiang Liu, Kriti Aggarwal, Owais Khan Mohammed, Saksham Singhal, Subhojit Som, et al. Image foreign language: BEiT pretraining for all vision and vasion-language tasks. arXiv preprint (tx.::2208.14444 2022.
Chen Wei, Haoqi Fan, Saining Xie, Chao Yuan Wu, Alan Yuille, and Christoph Feichtenhofer. Masked feature prediction for self-supervised visual pre-training. arXiv preprint drXiv:211200913: 2021.
Longhui Wei, Lingxi Xie, Wengang Zhou, Houqiang Li, and Qi Tian. Mvp: Multimodality- visual pre-training, arXiv preprinl ar Xrx::22033.06 05175, 2022a.
Yixuan Wei, Han Hu, Zhenda Xie, Zheng Zhang, Yue Cao, Jianmin Bao, Dong Chen, and Baining Guo. Contrastive learning rivals masked image modeling in fine-tuning via feature distillation. arXie preprint arXiv:2205.44141 2022b.
Zhirong Wu, Yuanjun Xiong, Stella X Yu, and Dahua Lin. Unsupervised feature learning via non-parametric instance discrimination. In CVPR, 2018.
Zhirong Wu, Zihang Lai, Xiao Sun, and Stephen Lin. Extreme masking for learning instance and distributod visual representations. ArXio, abs/2206.04665 2022.
14

---

Published in Transactions on Machine Learning Research (03/2023)
Tete Xiao, Yingeheng Liu, Bolei Zhou, Yuning Jiang, and Jian Sun. Unifted perceptual parsing for scene understanding. In ECCV, 2018.
Jing Yang, Brais Martinez, Adrian Bulat, and Georgios Tzimiropoulos. Knowledge distillation via softmax regression representation learning. In ICLR, 2021.
Kun Yi, Yixiao Ge, Xiaotong Li, Shusheng Yang, Dian Li, Jianping Wu, Ying Shan, and Xiaohu Qie. Masked image modeling with denoising contrast. ArXiv, 00s/2225.009611 2022.
Xiaosong Zhang, Yunjie Tian, Wei Huang, Qixiang Ye, Qi Dai, Lingxi Xie, and Qi Tian. Hivit: Hierarchical vision transformer meets masked image modeling. ArXiv, abs abs/2205.14949, 2022.
Bolei Zhou, Hang Zhao, Xavier Puig, Tete Xiao, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Semantic understanding of scenes through the ADE20K dataset. Int. J. Comput. Vis., 127(3))302-321, 2019.
Qiang Zhou, Chaohui Yu, Hao Luo, Zhibin Wang, and Hao Li. Mimco: Masked image modeling pre-training with contrastive leacher. aiviv preprint or 2209 03063, 2022.
15

---

Published in Transactions on Machine Learning Research (03/2023)
A Hyperparameters for MaskDistill Pretraining
Hyperparamneters
Base Size
Large Size
Huge Size
Layers
12
32
Hidden size
768
1024
1280
FFN imner hidden size
3072
4096
5120
Attention heads
12
16
16
Layer scale
0.1
1o-5
1e-5
Patch size
16 x 16
16 16
14 x 14
Training epochs
300/800
Batch size
2048
Adam
1e-8
Adam $
(0.9, 0.999)
Peak learning rate
1.5e-3
Minimal learning rate
1e-5
Learning rate schedule
Cosine
Warmup epochs
10
Stoch. depth
0.1
0.2
0.25
Gradient clipping
3.0
Dropout
X
Weight decay
0.05
Data Augment
Imput resolution
224 224
Color jitter
0.4
Table 11: Hyperparameters for MASKDISTILL pretraining on ImageNet-1K.
B
Hyperparameters for ADE20K Semantic Segmentation Fine-tuning
Hyperparameters
VIT-B/16 ViT-L/16
Relative positional embeddings Shared relative positional embeddings
x
Peak learning rate
{0.5, 0.8, 1.0, 1.5)e-4 160K
Fine-tuning steps
Batch size
16
Adam
1e-8
Adam B
(0.9, 0.999)
Layer-wise learning rate decay
0.75
0.85
Minimal learning rate
0 Linear
Learning rate schedule
Warmup steps
1500
Dropout
x
Stoch. depth Weight decay
0.1
0.2
0.05
Input resolution
512 x 512
Table 12: Hyperparameters for fine-tuning MASKDISTILL on ADE20K.
16

---

Published in Transactions on Machine Learning Research (03/2023)
C
Hyperparameters for Image Classification Fine-tuning
Table 13: Hyperparameters for fine-tuning MASKDISTILL on ImageNet-IK..
Hyperparameters
ViT-B/16
ViT-L/16
ViT-H/14
Peak learning rate
5e-4
5e-4
2c-4
Finc-tuning epochs
100
30
Warmup epochs
20
5
Layer-wise learning rate decay
0.65
0.8
0.85
Batch size
1024
Adam
1e-8
Adam 3
(0.9, 0.999)
Minimal learning rate
1c-6
Learning rate schedule
Cosine
Stoch. depth
0.1
0.2
0.25
Repeated Aug
x
Weight decay
0.05
Label smoothing
0.1
Dropout
X
Gradient clipping
Erasing prob.
0.25
Input resolution
224
Rand Augment
9/0.5
Mixup prob.
0.8
Cutmix prob.
1.0
17


Reviews:
[{"review_id": "36Ry5eHGOc", "reviewer": "Reviewer_FqMf", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The paper concludes the existing MIM methods into a unified view and optimizing existing settings in a simple and clean framework. Especially, it is nice that Table 1 and Table 2 clearly illustrate the difference between existing methods and the proposed MaskDistill.\n- As an experiment-driven model design, extensive experiments are conducted to prove the effectiveness of the module selection of the unified MIM framework. The comparison analysis with BEiT v2 by codebook collapse is also interesting.\n\nWeaknesses:\n- This work is a good successor to past MIM methods, but it has not made further breakthrough contributions to the current MIM framework. Its technical contribution is limited to empirical influence on the research area. Readers are hard to be inspired with limited insights.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper focuses on the problem of masked image modeling. The author proposes a unified view of masked image modeling through the combination of a teacher model, a normalization layer, a student model, a MIM head, and a proper loss function. The simple yet effective method called MaskDistill is based on the unified view of MIM methods. Extensive experimental evaluations are conducted on several downstream tasks, and the results seem good under various settings.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "The writing is clear and the content is easy to follow. No additional revision is needed from my perspective.", "broader_impact_concerns": "This paper does not involve ethical issues.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "a5drcvOcmp", "reviewer": "Reviewer_yZN2", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n1. The proposed way to unify various MIM approaches seems reasonable and many of the recent approaches in a common setting based on  some of the design decisions. \n2. MaskDistill is a simple and effective way to distill information from self-supervised pre-trained teacher models to student models for self-supervised learning. \n3. The method seems to lead to consistent improvements in the settings considered. \n4. The authors provide a good analysis of the proposed approach \n\nWeaknesses/Questions:\n1. Can your approach be extended to non-ViT based teachers ? For example using ResNets which have been explored a lot in the prior SSL work ? One approach could perhaps be use of pre-pooling features as proxy to patch features. \n2.  Missing baseline : It seems that an important baseline is missing : Start with the teacher checkpoint and use BEiT/MAE/etc to do SSL on pretraining dataset without any model distillation. This will help evaluate whether distillation is needed\n3. The authors mention the extra time & compute for forward pass through the teacher. This could be of practical significance. It will be beneficial to explicitly include this information (training time, FLOP) in the analysis section. \n4. Did the authors try to include the distillation loss in addition to the usual MIM loss with separate heads? For example distillation + MAE-like pixel reconstruction \n5. It will be interesting to perform some analysis on effect of the proposed technique on transfer learning tasks like [a].\n\n[a] How Well Do Self-Supervised Models Transfer? \n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "- The paper proposes a unified view of various masked image modeling-based SSL approaches. \n- Specifically, the paper divides them into : teacher models, normalization layer, student model, the MIM head and the loss function used. \n- A simple method termed MaskDistill is proposed which trains a student model to reconstruct normalized semantic features from teacher model at masked positions. \n- The authors show that the proposed approach leads to consistent improvements on IN top-1 accuracy and downstream task of semantic segmentation. ", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please refer to Weaknesses & Questions. (2) and (3) are critical for me. (1), (4) and (5) will help strengthen the paper. \nMinor writing comments : \n  - avoid vague terms like : Page 1 \"proper loss function\"\n  - It would help to make the captions verbose. \n  - Conclusion \"susection\"", "broader_impact_concerns": "A broader impact section is not included. I do not think this is a major concern for this paper though.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "yK80TfZg0B", "reviewer": "Reviewer_bw3D", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "\n\nStrengths:\n\n- The paper is well written and easy to follow.\n- The idea seems interesting and the results are quite good.\n- There is experiments on semantic segmentation and classification task which is good practice.\n- The ablations table Table 6, 7 and 8 are interesting.\n\nWeakness:\n\n- Architecture used for the experiments: \n\nThe paper state: \"We conduct MIM pretraining on ImageNet-1k (Russakovsky et al., 2015) for base-, large- and huge-size ViTs.\" according to the code provided in appendix it's seem the architecture is a beit like architecture (for instance with relative position encoding). These are important changes. It is important to clarify that the architecture used is not ViT to avoid confusion.\n\n\n- Comparison with other approaches:\n\na) The Table 2 comparison is not very fair as the teacher models used were not pre-trained on the same data.\nIt would be interesting to merge Table 2 and  Table 6 by mentioning the data used for the teacher and providing results with Maskdistill teacher trained with different dataset to be comparable with other architecture.\n\nb) Table 2 and 4: The paper refers to ViT for all the architecture. However some architecture used relative position encoding and some other not. It is important to clarify this because the comparison is misleading. An interesting comparison would be to use ViT-B as defined in the original paper a teacher trained on ImageNet only and compare the results with MAE or PeCO.\n\nc) Currently there are no comparisons with other approches quite similar like MAE that could be considered as an ablation. Each time there some change in the architecture , change in the initialisation (according to the code the paper used a kind of Fixup `fix_init_weight` in the code) and extra training data  used for the teacher. But there is no ablation on this component.\n\nd) Table 2 The different approaches presented do not always use a teacher. It would be interesting to report the training cost of each approach to understand the different trade-offs.\n\n- Experiments to show the advantage of mask image modeling:\n\nThe paper state: \"Masked image modeling has demonstrated great potential to eliminate the label-hungry problem of training large-scale vision Transformers, achieving impressive performance on various downstream tasks\".\n \nSo the ImageNet-1k to ImageNet-1k is maybe to restrictive for self-supervised approaches. It may be interesting to consider the following settings:\n\na) Pre-training on larger dataset:\nThere is a lack of experiments with pre-training on a large dataset without labels followed only by a full finetuning on a smaller dataset. For instance a pre-training on ImageNet-1k  followed by full finetuning on ImageNet-1k.\n\nb) Pre-training on dataset with different distribution:\nFor instance a pre-training on iNaturalist-2021  followed by full finetuning on ImageNet-1k.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper proposes to revisit the mask image modeling approach by unifying the approach using an external model and the approach using reconstruction from the original image. The paper evaluates its approach on image classification and semantic segmentation tasks.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "At present it is difficult to position the proposed approach with approaches in the literature. Indeed, there are too many changes that interfere with the comparison. The adjustments suggested are in weakness \"Comparison with other approaches\".\nThe point \"Experiments to show the advantage of mask image modeling\" would strengthen the work but is not critical ", "broader_impact_concerns": "No concerns", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review:  While the novelty of the proposed techniques is not very high, reviewers find the paper interesting based on the strong experimental section and results. The paper brings in new results to an area with strong current research interest. I recommend a minor revision since two of the reviewers think the authors should incorporate an additional downstream task and adjust the manuscript further based on their comments (see recommendation from reviewer FqMf).
Decision: Accept (poster)