A Closer Look at Self-supervised Lightweight Vision Transformers
Authors: Shaoru Wang
Jin Gao
Zeming Li
Weiming Hu
Venue: Submitted to ICLR 2023 (2023)
Keywords: Self-supervised Learning
Vision Transformers
Lightweight Networks

Abstract: Self-supervised learning on large-scale Vision Transformers (ViTs) as pre-training methods has achieved promising downstream performance. Yet, how much these pre-training paradigms promote lightweight ViTs' performance is considerably less studied. In this work, we mainly develop and benchmark self-supervised pre-training methods, e.g., contrastive-learning-based MoCo-v3, masked-image-modeling-based MAE on image classification tasks, and some downstream dense prediction tasks. We surprisingly find that if proper pre-training is adopted, even vanilla lightweight ViTs show comparable performance on ImageNet to previous SOTA networks with delicate architecture design. We also point out some defects of such pre-training, \eg, failing to benefit from large-scale pre-training data and showing inferior performance on data-insufficient downstream tasks. Furthermore, we analyze and clearly show the effect of such pre-training by analyzing the properties of the layer representation and attention maps for related models. Finally, based on the above analyses, a distillation strategy during pre-training is developed, which leads to further downstream performance improvement for MAE-based pre-training.

Under review as a conference paper at ICLR 2023
A CLOSER LOOK AT SELF-SUPERVISED LIGHTWEIGHT VISION TRANSFORMERS
Anonymous authors Paper under double-blind review
ABSTRACT
Self-supervised learning on large-scale Vision Transformers (ViTs) as pre-training methods has achieved promising downstream performance. Yet, how much these pre-training paradigms promote lightweight ViTs' performance is considerably less studied. In this work, we mainly develop and benchmark self-supervised pre-training methods, e.g., contrasiie-leeamin--bae modeling-based MAE on image classification tasks, and some downstream dense prediction tasks. We surprisingly find that if proper pre-training is adopted, even vanilla lightweight ViTs show comparable performance SOTA networks with delicate architecture design. We also point out some de- fects of such pre-training, e.g., failing to benefit from large-scale pre-training data and showing inferior performance on data-insufficient downstream tasks. Further- more, we analyze and clearly show the effect of such pre-training by analyzing the properties of the layer representation and attention maps for related models. Finally, based on the above analyses, distillation strategy during pre-training is developed, which leads to further downstream performance improvement for MAE-based pre-traee uraiiraini
MoCo-v3, masked-image-
ImageNet to previous
INTRODUCTION
Self-supervised learning (SSL) has shown great progress in representation learning without heavy reliance on expensive labeled data. SSL focuses on various pretext tasks for pre-training. Among
them, several works (He et al. 2020 Chen et al. 2020 Grill et al. 2020 Caron et al., 2020 Chen
al. 2021a Caron al., 2021) based on contrastive learning (CL) have achieved comparable or even better accuracy than supervised pre-training when transferring the learned representations
to downstream tasks. Recently, another trend focuses on masked image modeling (MIM) (Bao
let 2021 He et al. 2021: Zhou al. (022), which perfectly fits Vision Transformers (VīTs) (Dosovitskiy et al. 2020) for vision tasks, and achieves improved generalization performance. Most of these works, however, involve large networks with little attention paid smaller ones. Some works (Fang et al. 2020; Abbasi Koohpayegani et al.) 2020 Choi et al. 2021) focus on contrastive seff-supervised learning small convolutional networks (ConyNets) and improve the performance by distillation. However, the pre-training of lightweight ViT's is considerably less studied.
for
Efficient neural networks are essential for modern on-device computer vision. Recent study achieving top-performing lightweight models mainly focuses on designing network architectures
(Sandler al. 2018, Howard al., 2019 Graham al. 2021 Ali et al.) 2021 Heo et al.) 2021
Touvron et al. 2021b: Mehta & Rastegari 2022) Chen et al. 2021b Pan et al., 2022), while with little attention on how to optimize the training strategies for these models. We believe the latter is also of vital importance, and the utilization of pre-training is one of the most hopeful approaches along this way, since has achieved great progress on large models. To this end, we develop and benchmark recently popular self-supervised pre- pre-training methods, e.g., CL-based MoCo-v3 (Chen) cet al., a) and MIM-based MAE (He et al. 2021), along with fully-supervised pre-training for lightweight ViTs as the baseline both ImageNet and some other classification tasks as well as some dense prediction tasks, e.g., object detection and segmentation. We surprisingly find that if proper pre-training is adopted, even vanilla lightweight VITs show comparable performance to previous SOTA nenworks with delicate design on ImageNet, which achieves 78.5% top-1 accuracy on ImageNet with vanilla ViT-Tiny (5.7M). We also observe some intriguing defects of such pre-

---

Under review as a conference paper at ICLR 2023
training, fagling benefit from large-scale pre-training data and showing inferior performance on data- inaainssffffiinnt downstream tasks.
These findings motivate to dive deep into the working mechanism of these pre-training meth- ods for lightweight ViTs. More specifically, we introduce a variety of model analysis methods to study the pattern of layer behaviors during pre-training and fine-tuning, and investigate what really matters for downstream performance. First, we find that lower layers of the pre-trained models matter more than higher ones if sufficient downstream data is provided, while higher layers matter in data-insufficient downstream tasks. Second, we observe that the pre-training alters the attention behaviors of the final recognition model little, without introducing locality inductive bias, which is, however, the commonly adopted rule for recent network architecture design (Mehta & Rastegari, 2022 Heo et al., 202T, Touvron et al.] 2021b Liu et al.) 2021). Based on the above analyses, we also develop a distillation strategy for MAE-based pre-training, which improves the pre-training of lightweight ViTs. Better downstream performance is achieved especially on intaiinsufficient classi-
fication tasks and detection tasks.
PRELIMINARIES AND EXPERIMENTAL SETUP
VITs. We use ViT-Tiny (Touvron et al. 2021a) as the base model in our study to examine its downstream performance with pre-traiming, which contains 5.7M parameters. We adopt the vanilla architecture, consisting of 12 layers with the embedding dimension of 192, except that the number of heads is increased 12 we find it can improve the model's expressive power. We this improved version by default. ViT-Tiny is chosen for study because it is an ideal experimental object, on which almost all existing pre-training methods can directly apply, and has rather naive structure, which can eliminate the influence of the model architecture on our analysis to a great extent.
Evaluation Metrics. Linear probing has been popular protocol to evaluate the quality of the pre-trained weights (He et al.) 2020) Chen et al.] 2020 Grill et al.] 2020) [aron et al.] [2020), in which only the prediction head is tuned based on the downstream training set while the pre-trained representations are kept frozen. However, prior works point out that linear evaluation does not
always correlate with utility (He et al., 2021 Newell & Deng 2020).
Tine-tuning is another evaluation protocol, in which all the layers are tuned by first initializing them with the pre-trained models. We adopt this by default. Besides, layer-wise Ir decay (Bao et al. is also taken into consideration. By default, we do the evaluation on ImageNet (Deng et al.
2021
2009)
by fine-tuning the train split and evaluating
the validation split. Several other downstream
classification datasets (Nilsback & Zisserman 2008 Parkhi et al., 2012): Maji et al.] 2013 Krause
al. 2013] Krizhevsky al. 2009 Van Horn al.. 2018) and object detection and segmentation tasks on COCO (Lin et al. 2014) are also exploited for comparison in our study.
Compared Methods. Baseline: We largely follow the recipe in DeiT Touvron et al. 202 lap except for some hyper-parameters of augmentations (see Appendix A. for our improved recipe) and fully-supervised train ViT-Tiny from scratch for 300 epochs on the training set of ImageNet- 1k. It achieves 74.5% top-1 accuracy on the validation set of ImageNet-Ik, surpassing that in the original architecture (72.2%) through modifying the number of heads to 12 from 3, and further reaches 75.8% by adopting the improved training recipe, which finally serves as our strong baseline to examine the pre- pre-training. We denote this supervised trained model by DeiT-Tiny.
MAE: MAE (He et al., (02T) is selected as a representative for MIM-based pre-training methods, which has a simple framework with low training cost. We largely follow the design of MAE except that the encoder is altered to ViT-Tiny. Several basic factors and components are adjusted to fit the smaller encoder (see Appendix A.2). By default, we do pre-training the train split of ImageNet- Ik (Deng et al., 2009) (dubbed INIK) for 400 epochs, and denote the pre- pre- trained model as MAE-Tiny. Tiny.
MoCov3: We also implement contrastive SSL pre- pre-training counterpart to achieve more thorough study. MoCo-v3 (Chen et al. 2021a) is selected for its simplicity. We use MoCov3-Tiny to denote this preitedi model with 400 epochs. Details provided in Appendix A.3
Some other methods, e.g., MIM-based SimMIM MiaeMMXXiieer et al. (2022) and CL-based DINO Caron et al. (202)) are also involved, but are moved Appendix B.5 due to the space limitation.
2

---

Under review as a conference paper at ICLR 2023
Table 1: Comparisons on pre-training methods. We report top-1 accuracy on the validation set of ImageNet- 1k (Deng et al. 2009). INIK and IN2IK indicate the training set of ImageNet- Ik and ImageNet-211 (Deng)
et al. 2009). The pre-training time is measured on 8x V100 GPU machine. ViT-Tiny is adopted for all entries.
orr."" represents the training recipe in Touvron et al. 2021a and 'impr." represents our improved recipe (see
Appendix A.1
Pre-training Data
Fine-tuning Top-1 Acc. (%)
Methods
Epochs
Time (hour)
recipe
from scratch
ori.
74,5
from scratch
impr.
75.8
Supervised
Steiner et al.
2021
IN2IK w/ labels
30
20
impr.
76.9
Supervised
Stemer et at.
2021
IN2IK w/ labels
300
200
impr.
77.8
MoCo-v3
nen et aL.
2021a
IN w/o labels
400
52
impr.
73.7
MAE
HC et al. 2021
INIK w/o labels
400
23
impr.
78.0
3 How WELL DOES PRE-TRAINING WORK ON LIGHTWEIGHT VITS?
MAE outperforms other pre-training methods on ImageNet. We develop and benchmark fully- supervised and suff-superviee pre-training methods on ImageNet, as reported in Tab. [ For all of the pre-trained models, fine-tune them for 300 epochs on INIk for fair comparisons. It can be seen that most of these supervised and self-supervised pre-training methods improve the downstream performance, whilst MAE outperforms others and consumes moderate training cost. Meanwhile the pre- pre-training of MoCo-v3 leads to performance degradation. We denote the fine-tuned model based on the pre-training of MAE-Timy as MAE-Tiny-FT.
Enhanced vanilla ViTs with pre-training are comparable to previous SOTA networks. We further compare the enhanced ViT-Tiny (5.7M) with MAE pre-training to the DeiT-Tiny (Touvron et al. 2021a baseline and other previous lightweight ConvNets and ViT derivatives in Tab. 2 We report top-1 accuracy along with the model parameter count and the throughput, which is borrowed from PyTorch Image Models (timm) (Wightman) 2019). In specific, during fine-tuning to 1000 epochs following Touvron et al. (2211)) The resulting models are on par with even outperform most previous Nenv and ViT derivatives with comparable parameters or throughput. This demonstrates the usefulness of the advanced lightweight ViT pre-training strategy which orthogonal to the network architecture design strategy in the ViT derivatives. Besides, we also compare with the methodology of pre-training lightweight Conv or ViT derivatives for a more fair comparison (Fang al. 2020 Abbasi Koohpayegani et al., 2020. Gao al., 202T] Choi et al., 2021). We find that most of them are evaluated under the linear probing protocol. We thus implemented the above methodology by ourselves, i.e., adopting the pre-training with SEED (Fang et al., 2020) on EfficientNet-BO (Tan & Le| 2019) under the fine-tuning protocol. The result, however, shows no improvement (from 77.7% to 77.2%)
extend the training epochs
The pre-training benefits little from large-scale data. Furthermore, we observe that MAE is robust to the pre- training dataset scale and class distribution in contrast MoCo-v3 as shown in Tab. R We consider two subsets of INIK containing 1% and 10% of the total examples (1% INIK and 10% INIK) balanced in terms of classes (Assran class distribution
Table 3: Effect of pre-training data.
Top-1 accuracy is reported.
Datasets
McCo-v3
MAE
INIK
73.7
78.0
1% INIK
(-0.6)
((01)
10% INIK
(-0.2)
(+0.0)
INIK-LT
73.0
(-011-
ctt al. 2021), one subset with
IN2IK
73.8 (+0.1) 78.0 (+0.0)
[Liu et al. 2019) (INIK-LT), and IN21K. This observation
is consistent with El-Nouby et al. (2021) on larger ViTs. It also reveals the limitation of these pre-training methods that they fail to benefit from large-scale pre- reeetraining data.
Downstream data scale matters. As shown in Tab. 4 we transfer the learned representations of different preitrained models to several other downstream tasks (Nilsback & Zisserman 2008 Parkhi
et al. 2012 Maji et al. 2013 Krause et al. 2013: Krizhevsky et al. 2009 Van Horn et al. 2018) to investigate their effects. In addition to using the self-supervised pre- preitedined models, i.e., MAE- Tinyyy and MoCov3-Thie based on INIK with 300-epoch pre-training (i.e., DeiT-Tiny) also involved. An interesting ob- servation is that the self-supervised pre-training approaches achieve downstream performance far
both of which are pre-trained for 400 epochs,
superviedd counterpart
3

---

Under review as a conference paper at ICLR 2023
Table 2: Comparisons with previous SOTA networks on ImageNet-Ik. We report top-1 accuracy on ImageNet- 1k validation set (Deng et al. 2009). ImageNet Real (Beyer et al. 2020) and ImageNet V2 matched frequency (Recht et al. 2019). along with throughput and parameter count. The throughput is borrowed from timm (Wightman| 2019), which is measured on single RTX 3090 GPU with batch size fixed 1024 and mixed precision. INTK and IN21K indicate the training set of ImageNet-1k and ImageNet-21k: findicates that distillation is adopted during the supervised training (or fine-tuning). indicates the original architecture of ViT- ViT-Tiny, and others use the improved architecture (number of heads is changed to 12), e.g., MAE- MAE-Tiny-FT in the table.
Methods
pre-train data
#param.
throughput (image/s)
Val Top- 1
Real Top-1
V2 Top-1
ConvNets
ResNet-18 et 2016
12M
8951
69.7
77.3
57.2
ResNet-50
at.
2016 Wightman et. 2021
25M
2696
80.4
85.7
68.7
EfficientNet-Bl
(Tan & Le
2019
5M
5369
77.7
84.0
66.3
EfficientNNttB0
Fang
2020
INIK w/o labels
5M
5369
77.2
83.5
65.9
EtficientNet-BI
Tan L
2019
8M
2953
78.8
84.6
67.5
MobileNet-v2 Sandler et al.
2018
4M
7909
72.0
80.2
60.2
MobileNNt-v3
Howwrd er 2019
5M
9113
75.2
82.2
63.4
MobileNet-v331
Beyer
2021
5M
9113
77.0
Vision Transformers Derivative
LeViT-128 Graham al. 2021
9M
13276
78.6
84,8
66.6
LeVIT-192 Graham et al. 2021
11M
11389
80.0
85.6
67.9
XCIT-T12/16* et 2021
7M
3157
78.6
84.1
67.0
PTT-Tit/ 1000 epochs [ee et al., 2021
5M
76.4
82.0
63.1
CaiT-XXS-244 Touvron al. 2021b
12M
1351
78.4
85.2
67.4
MobileVTT-S (Mehta Rastegan
6M
1900
78.3
66.9
Swin-IG (Liu et 2021 Chen
2021b
7M
77.3
LVT Yang 2021
6M
74.8
EdgeViT-XX (Pan al.
2022
7M
77.5
Mobile-Fomner 294M (Chen al.
2021b
11M
77.9
Vanilla Vision Transformers
DeiT-Tiny" Touvron al. 20
6M
4844
72.2
80.1
60.4
DeiT-Tiny" 4f 1000 epochs ( Touvron et al. 2021a
6M
4764
76.6
83.9
65.4
MAF: Min--T MAE-Tiny-FT 1000 epuchs
INIK w/ labels
6M
4020
78.0
84.3
66.2
INIK w/o labels
6M
3956
78.5
85.3
67.1
Table 4: Transfer evaluation on classification tasks and dense-prediction tasks. Self-supervised pre- training approaches generally show inferior performance to the sully-supervised counterpart. Top- accuracy is reported for classification tasks and AP is reported for object detection (det.) and instance segmentation (seg.) tasks. The description of each dataset represented (train-
Datasets
Flowers
Pets
Aircraft (TU/U100))
Cars
Cifar100
iNat18
COCO.det.) COCO
Init.
(328/1122)
(41/4137)
(8L88/196)
(50/100/100)
(488/245112)
(188850090)
supervised DeiT-Tiny
96.4
93.1
73.5
85.6
85,8
63.6
40.7
36.5
self-superviiee
MoCos3-Tiny MAE-Tiny
94.8
87.8
73.7
83.9
83.9
54.5
40.0
36.0
85.8
76.5
64.6
78.8
78.9
60.6
38.9
35.1
behind the selly-supervisedd counterpart, while the performance gap narrowed more or less as the data scale of the downstream task increases. Moreover, MAE even shows inferior results to MoCo- v3. We conjecture that it is due to their different layer behaviors during pre-training and fine-tuning, e.g., undesired representations of the higher layers in MAE- Tiny, which will be discussed in detail in the following section. We refer the reader to Appendix A.4 for more details about those tasks.
For more thorough study, we further evaluate downstream object detection and segmentation tasks on COCO (Lin et al. (0144) based on Li et al. (2021) (see Appendix A.3 for details), with dif- ferent pre- trainerie models initialization of the backbone, shown in Tab. |4 The self-supervised pre-trate tre-training also lags behind the fully-supen sully-super counterpart and MAE-Time still shows worse re-
sults than MoCov3-Tiny.
4

---

Under review as a conference paper at ICLR 2023
Deil-Tiny
MAE-Tiny
MoCov3-Tiny
Corresponding Similarity
Sim Mmnllartt to DATECTIO
1.0
Layers Det-Tim Tany
Lagers MAE-TTay
Layars MaCova-Tioy
0.8
0.6
64
DefT-Tiny
MAB-Tioy
MoCoc3-Tiny
12
5
12
0.0
Layers Dall-Tayy
Layers DelFTTyn
Layers DelT-Tiny
Depth
Layer representation similarity within and across models as heatmaps (the left three columns), with and y axes indexing the layers (the 0 index indicates the patch embedding layer), and higher values indicate higher similarity. We also plot the corresponding layer similarity the last column based on the diagonal elements of the left heatmaps.
Figure I:
REVEALING THE SECRETS OF THE PRE-TRAINING
In this section, we introduce some model analysis methods to study the pattern of layer behaviors during pre-training and fine-tuning, and investigate what matters for downstream performances.
4.1 LAYER REPRESENTATION ANALYSES
We first adopt Centered Kemel Alignment (CKA) method Cortes et al. 2012 Nguyen et al. 2020 to analyze the layer representation similarity across and within networks. Specifically, CKA com- putes the normalized similarity in terms of the Hilbert-Schmidt Independence Criterion (HSIC (Song ct al. 2012) between two feature maps or representations, which is invariant to the orthogonal trans- formation of representations and isotropic scaling (detailed in Appendix A.66p. Fourier analysis of feature maps is also involved in analyzing the behaviors of the models.
Lower layers matter more than higher ones if sufficient downstream data is provided. We visualize the layer representation (Rep.) similarity between several pre-trained models and DeiT- Tiny as heatmaps in Fig. [ The similarity within DeiT--Tiny is also presented for reference (the left column). We also plot the corresponding layer similarity in the last column based the diag- onal elements of the left heatmaps. We choose DeiT-Tiny as the reference because we consider the higher similarity between the pre-trained models and DeiT-Tiny (classification model fully- supervised trained from scratch) indicates more relevance to recognition for the self-supervised lay- ers. Although the similarity does not directly indicate whether the downstream performance is good or not, it indeed reflects the pattern of layer representation to certain extent. In Appendix [.I stronger supervised trained ViTs are introduced as reference models, and we find that these super- vised ViTs generally have similar layer representation structures.
First, We observe relatively high similarity between MAE- Tiny and DeiT-Tiny for lower layers, while low similarity for higher layers. Similar phenomenon is observed w.r.t. other ref- erence models shown in Appendix B.I. It indicates fewer semantics are extracted for MAE Tiny at a more abstract level in higher layers. Another empirical evidence is the low linear probing performance of MAE-Tiny (23.4% top-1 accuracy).
ImageNet
75
74
73
72
In contrast, MoCov3-Tiny aligns DeiT-Tiny well across almost all layers. However, the fine-tuning evaluation in Tab. shows that adopting the MAE-Tiny initialization significantly im- proves the performance while MoCov3-Tim degrades perfor- mance. Thus, we hypothesize that lower layers matter much more than higher ones for the pre-trained models. In order to verify the hypothesis, we design another experiment by only reserving several leading blocks of pre-trained models and ran- domly initializing the others, and then fine-tuning them on ImageNet (for the sake of simplicity, we
71
MAE-Tiny McCov3-Thhx
70
2
4
8
10
12
Number of Reserved Blocks
Figure 2: Lower layers of pre-trained models contribute to most gains on downstream ImageNet dataset.
5

---

Under review as a conference paper at ICLR 2023
CIFARI00 10k/166//1000
90
Sk/8k/102
80
Pets
444/337
Aoc (86)
Asc (96)
60
Aoc
30
MAE Tuy
20
MAE Tiny
Mae
McCova-THe Tin
Mo Tiny
MoCova-Tan Tay
40
20
2
8
10
12
10 12
0
2
4
6
8
10
12
Numbber of Reserved Blocko
Namber of Reserved Blocks
Number of Reserved Blocks
Figure 4: The contributions from higher layers on performance gain increase as the downstream dataset scale shrinks, which indicates higher layers matter in iata-inssuffcient downstream tasks.
only fine-tune these models on INIK for 100 epochs). Fig. shows that reserving only a certain number of leading blocks achieves a significant performance gain over randomly initializing all the blocks (i.e., totally training from scratch) for both MAE-Tiny and MoCov3-Ting: Whereas, further reserving higher layers leads to marginal gain for MAE-Tiny or even degradation for MoCov3-Tiny, which demonstrates our hypothesis.
Then examine why MoCov3-Tiny performs worse than MAE-Tiny. The Fourier analysis feature maps is carried out supplement beyond CKA-based similarity, since find that the similarity is largely dominated by low-frequency com- ponents. In Fig. 3] we plot the Alog amplitude across layers, which is the difference between the log amplitude at normal- ized low frequency (0.0%) and high frequency (I.Or) in each layer. It is also used in Park & Kim (202T) to analyze the differences between ViTs and ConvNets. We find that large amount of high-freyuency signals are reduced in the first layer of MoCov3-Tiny (i.e., patch embedding layer), which shows a great difference from other models. This behavior possibly strengthens the robustness against various image augmenta- tions, which is beneficial to the instance discrimination task, feature map at the
Deil-Tlay
MAF-Tiny MoCor8 Tiny
ALog Amplitude
-1
-1.5
-2
2.5
-3.5
8
10
12
Depth
Figure 3: MoCov3-Tiny behaves dif- ferently at the first layer, reducing a lot of tigh-freeuency signals.
but results in an over-spatially-smoodee very beginning of the network forward processing, leading to inferior downstream performance on ImageNet.
Higher layers matter in data-insufficient downstream tasks. Previous works (Touvron et al.) 202 Raghu et al., (021) demonstrate the importance of relatively large dataset scale for high- performance VIT's with large model sizes. We also observe a similar phenomenon on lightweight VITs even with the self-supervised pre-training adopted as discussed in Sec. 3 It motivates to study the key factor in downstream performance on data-insufficient tasks.
We conduct similar experiments as those in Fig. 2 on small-scale downstream datasets. The results are shown in Fig. 4 We observe consistent performance improvement as the number of reserved blocks from pre-trained models increases. And the smaller the dataset scale, the more the per- formance gain from higher layers. It demonstrates that higher layers are still valuable and matter in data-in data-insufficient downstream tasks. Furthermore, we observe comparable performance for the transfer performance of MAE-Tiny and MoCov3-Timg when only a certain number of lower layers are reserved, while MoCov3-Tiny surpasses when higher layers are further adopted. indicates the higher layers of MoCov3-Tiny work better than MAE-Tiny on data-insufficient downstream tasks, which is also consistent with our CKA-based analyses shown in Fig. [ that MoCov3-Tiny learns more semantics at abstract level relevant to recognition in higher layers (high similarity to reference recognition models higher layers) than MAE-Tiny. And also we conjecture that high- freuufryyynncy information matters less in these relatively easier tasks.
4.2 ATTENTION MAP ANALYSES
The attention maps reveal the behaviors for aggregating information in the attention mechanism, which are computed from the compatibility of queries and keys by dot-product operation. We intro-

---

Under review as a conference paper at ICLR 2023
140 Global
DeiT-Tiny
140
MAE-Tim
140
-oCov3-Tiny
100
MAE-Tiny-FT
Attn. Distonce
Locul
Deych
Depth
Depth
Depth
Broun
DeiT-Tiny
MAE-Tiny
MoCov3-Tiny
MAE-THyy-FT
Attn. Entoopy
Concentrated
Degth
Depth
Depth
Figure 5: Attention distance and entropy analyses. We visualize the averaged attention distance and entropy across all tokens different attention heads w.r.t. the layer number.
duce two metrics for further analyses on the pre-trained models, i.e., attention distance and attention entropy. The attention distance for the j-th token of h-th head is calculated as:
Di.j* Σ softmax (A
(1)
where A, R is the attention map for the h-th attention head, and Gt,; is the Euclidean distance between the spatial locations of the i-th and j-th tokens. is the number of tokens. And the attention entropy is calculated as:
En. Σ softmax( (An)()))))))))))( log(ss)tosssstmma (0 (An)ss),
(2)
Specifically, the attention distance reveals how much local vs. global information is aggregated, and a lower distance indicates that each token focuses more on neighbor tokens. The attention entropy reveals the concentration of the attention distribution, and lower entropy indicates that each token attends fewer tokens. We analyze the averaged attention distance and entropy across all the tokens in different attention heads, as shown in Fig. 5
The pre-training with MAE alters the attention behaviors of the final recognition model little. First, we compare MAB-Tiny-FT with DeiT-Tiny. The former adopts MAE-Tiny as initialization and then is tine-tuned on INIK. and the latter is supervised trained from scratch on INIK. We observe very similar attention behaviors between them. They both have diverse attention heads in lower layers, which aggregate both local and global tokens with both concentrated and broad focus, and more global and broad attention in higher layers. It indicates that the pre-training does not alter the behaviors of the ultimate recognition model much, but provides better initial state.
The pre-training with MAE improves the results not by bringing locality inductive bias. Then, we focus the attention behaviors of MAE-Tiny. It shows similar patterns to DeïT-Tiny on lower layers, but more distinct patterns on higher layers, which means its higher layers concen- trate (low entropy) on local spatial information (low distance). We hypothesize that the behaviors are related to the aim of the pixel reconstruction task in MIM (Masked Image Modeling). Con- sidering only lower layers of MAE-Tiny matter as analyzed previously, we think the pre-training improves the performance not by introducing the locality inductive bias. It indicates that the bias may not be necessary for lightweight models to achieve top performance, which is, however, the key idea of many successful lightweight network architectures (Mehta & Rastegari 2022] Touvron ett al. 2021b; Heo et al.) (022). As for the MoCov3- Tiny, we observe relatively global and broad attention with low diversity in lower layers, which may be not suitable for downstream tasks,
5 DISTILLATION IMPROVES PRE-TRAINED MODELS
In this section, we focus on developing a distillation strategy for the top-performing MAE pre- training on lightweight models, to remedy some defects of this strategy. In the previous section,
7

---

Under review as a conference paper at ICLR 2023
MAE-Base
Distilled MAE-Tiny
77.5
Layers Distilled MAE- -iny
distill lngers
Layers MAE- -AE-Bose
0.8
77
9-
70.5
6
6
wyo discill.
0.4
3
3-
76
0.2
75.5
0.0
12
9 Layers Daif-Tin
6
12
6 9
12
Distilled Index
Layers DeiT-Tiny
Figure 7: Distillation on attention maps of higher layers improves per- formance most.
Figure 6: Distillation helps to compress the good representation of the teacher (MAE-Base) to the student, thus the distilled student shows higher similarity to the supervised trained DeiT-Tiny.
we have conjectured that it is hard for MAE to learn good representation relevant to recognition in higher layers, which results in unsatisfactory performance on insufffiffff downstream tasks. A natural question is that can it gain more semantic information by scaling up the models. We further examine a large pre-trained model, MAE-Base (He al. (021), and find achieves better alignment to DeiT-Tiny. as shown in the left column of Fig. 6 It indicates that it is possible extract features relevant to recognition in higher layers for the scaled-up up encoder in MAE pre-training.
These observations motivate to compress the knowledge of large pre-trained models to tiny ones, i.e., applying knowledge distillation during the pre-training phase for lightweight ViTs. Although it is common practice to perform distillation obtain pre-trained compressed language models (Sanh et al. 2019 Jiao et al.) 2020; Wang et al.) 2020 2021 Sun et al.) [2020] Su et (02T), how apply distillation obtain better lightweight ViT pre-training under the masked image modeling framework is still unexplored. We fill this gap and propose some useful techniques.
Distillation methods. Specifically, pre-trained MAE-Base (He al. 2021) is introduced as the teacher network. We adopt the attention- distillation, which is formulated as follows based on the mean squared error (MSE) between the corresponding layers of the teacher and student:
Latta = MSE(A", M,*),
(3)
where AT E gixixx and A E R8 xtxl refer to the attention maps of the teacher and student with h and h attention heads, and I is the number of tokens. A learnable mapping matrix M E RA is introduced to align the number of heads. The teacher is also applied only on the same unmasked patches in the encoder as the student during the distillation.
Distillation on lower or higher layers? We first examine applying the above layer-wise distilla- tion on which layer contributes to the most performance gain. Though is a direct way to apply distillation on all corresponding layers of the teacher and student, it actually slows down the training speed. As shown in Fig. 7 only distilling on the attention maps of the last transformer blocks pro- mote the performance most, even surpassing those distilling on all layers or other single lower layers (for the sake of simplicity. we only fine-tune the pre-trained models on INIK for 100 epochs). is consistent with the analyses in Sec. 4 Specifically, the lower layers learn good representation them- selves during the pre- pre-training with MAE and thus distilling on these layers contributes to marginal improvement, while the higher layers rely on a good teacher to guide them to capture rich semantic features.
Distillation improves pre-trained models for downstream tasks. We further evaluate the dis- tilled pre-trained model on several downstream classification tasks Nilsback & Zisserman 2208 Parkhi et al. 20122 Maji et al., 2013 Krause et al., 2013) Krizhevsky et al.] Van Horn et al. Deng] et al. 2009) and dense prediction tasks. For simplicity, we only apply distillation on the attention maps of the last layer. The visualization results in Fig show that the good representation relevant to the recognition of the pre-trained teacher is compressed to the distilled MAE-Tiny, Espe- cially the quality of higher layers improved. It contributes to better downstream performance on both classification and dense-prediction tasks as shown in Tab. 5 especially on object detection and segmentation tasks, surpassing the supervised pre-training counterpart by large margin.The The above results also support insight in Sec. 4.1
8

---

Under review as a conference paper at ICLR 2023
Table 5: Distillation improves downstream performance on classification tasks and object detection and segmentation tasks. Top-1 accuracy is reported for classification tasks and AP is reported for object detection (det.) and instance segmentation (seg.) tasks.
Init.
Datasets
Flowers
Pets
Aircraft
Cars
Cifar100
INat18
ImageNet
COCO(det)
COCO
supervised TTny
96.4
93.1
73.5
85.6
85.8
63.6
40.7
36.5
supervied
MAE-Tiny
85.8
76.5
64.6
78.8
78.9
60.6
78.0
38.9
35.1
Distilled MAE-Tiny
95.2 o90
89.1 (11a
79.2 6160
87.5 (4.7)
85.0 (6.
63.6 638
78.4 w44
42.7 (4300
38.2 (3.0)
RELATED WORKS
Self-supervised learning (SSL) focuses different pretext tasks (Gidaris et al. 2018 Zhang
eet al. 2016 Noroozi & Favaro, 2016 Dosovitskiy et al. 2014) for pre-training without using man-
ually labeled data. Among them, contrastive learning (CL) has been popular and shows promising
results on various convolutional networks (ConvNets) (He et al. 2020 Chen et al. 2020 Grill et al.,
2020 Caron
2020) and ViTs (Chen et al.) 2021a Caron et al.] 2021). Recently, methods based
on masked image modeling (MIM) achieve the state-of-the-ae of-the-art on ViTs (He et al.) [021 Bao et al.)
2021 Zhou et al. 2022 has been demonstrated that these methods can scale up well on larger models, while their performance on lightweight ViTs is seldom investigated.
Vision Transformers (ViT')
Dosovitskiy et al. (2020) apply Transformer architecture (a stack
of attention modules Vaswani et al. (017 on image patches and show very competitive results in various visual tasks (Touvron et al. 2021a Liu al. 2021 Li et al.) 2022). The performance of ViTs has been largely improved thanks to better training recipes (Touvron et al. [2021a] Steiner al., 2021] Touvron et al., 2022). As for lightweight ViT's most works focus integrating ViTs and ConyNets (Graham et al. 2021 Heo et al., [222]] Mehta & Rastegari 2022 Chen et al. 2021b
Yan al. 2021), while few focus on how to optimize the networks.
Knowledge Distillation is a mainstream approach for model compression (unressiiiooBBiuuuu Buciluă al., 2006), in which a large teacher network is trained first and then more compact student network is optimized
to approximate the teacher (Hinton al. 2015 Romero et al. 2014 [Shen & Xing 2022). Touvron et al. (2021a) achieves better accuracy ViTs by adopting ConvNet the teacher. With regard to
the compression of the pre-trained networks, some works (Sanh et al. 2019 Jiao et al.) 2020 Wang
ct al. 2020 2021 Sun et al. 2020 Su et al. 2021) attend to distill scale-scale pre-trained language
models. In the context of computer vision, series of works (Fang et al. 2020 Abbasi Koohpayegani
eet al. 2020 Choi et al.) 2021 Shen et al., (202)) focus on transferring knowledge of large pre-trained
networks based on CL to lightweight ConvNets. There are few works focusing on improving the quality of lightweight pre-trained ViTs based on MIM by distillation thus far.
7 DISCUSSIONS
Limitations Our study is restricted to classification tasks and some dense-predictio tasks, e.g., object detection and segmentation. We leave the exploration of more tasks for further work.
Conclusions We investigate the self-supervised pre-training of lightweight ViTs, and demonstrate the usefulness of the advanced lightweight ViT pre-training strategy in improving the performance of downstream tasks. Some properties about the pre-training are revealed, e-g., these methods fail to benefit from large-scale pre-training data, and show more dependency on the downstream dataset scale. We also present some insights on what matters for the downstream performance with pre- training by analyzing the layer representation and attention map. They may indicate potential future directions in improving pre- treiirann on lightweight models, the value which has also been demon- strated as it guides the design of our proposed distillation strategy and helps to achieve much better downstream performance. We expect our research may provide useful experience and advance the study of self-supervised learning on lightweight ViTs.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Soroush Abbasi Koohpayegani, Ajinkya Tejankar, and Hamed Pirsiavash. Compress: Self- supervised learning by compressing representations. Adv. Neural Inform. Process. Syst., 33: 12980-12992, 2020.
Alaaeldin Ali, Hugo Touvron, Mathilde Caron, Piotr Bojanowski, Matthijs Douze, Armand Joulin, Ivan Laptev, Natalia Neverova, Gabriel Synnaeve, Jakob Verbeek, al. Xeit: Cross-covariance image transformers. Adv. Neural Inform. Process. Syst., 34, 2021.
Mahmoud Assran, Mathilde Caron, Ishan Misra, Piotr Bojanowski, Armand Joulin, Nicolas Ballas, and Michael Rabbat. Semi-supervised learning of visual features by non- con-paametricallyy predicting view assignments with support samples. In Int. Conf. Comput. Vis., pp. 8443-8452, 2021.
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint 160.006600 2016.
Hangbo Bao, Li Dong, and Furu Wei. Beit: Bert pre-training of image transformers. arXiv preprint 200.:22/06.0254. 2021.
Lucas Beyer, Olivier Hénaff, Alexander Kolesnikov, Xiaohua Zhai, and Aäron van den Oord. Are we done with imagenet? arXiv preprini arXiv:2006.0715 07,, 2020.
Lucas Beyer, Xiaohua Zhai, Amélie Royer, Larisa Markeeva, Rohan Anil, and Alexander Kolesnikov. Knowledge distillation: good teacher is patient and consistent. arXiv preprint arXiv:2106.05237, 2021.
Cristian Buciluă, Rich Caruana, and Alexandru Niculescu-Mizii. Model compression. In KDD, pp- 535-541, 2006.
Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Un- supervised learning of visual features by contrasting cluster assignments. In Adv. Neural Inform. Process. Syst., 2020.
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In Int. Conf. Comput. Vis., pp. 6650-9660, October 2021.
Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He. Improved baselines with momentum contrastive learning. arXiv preprint arXiv:2003.0422 2020.
Xinlei Chen, Saining Xie, and Kaiming He. An empirical study of training self-supervised vision transformers. In Ini. Conf. Comput. Vis., pp. 9640-9649, 2021a.
Yimpeng Chen, Xiyang Dai, Dongdong Chen, Mengchen Liu, Xiaoyi Dong, Lu Yuan, and Zicheng Liu. Mobile-former: Bridging mobilenet and transformer. arXiv preprint arXiv:2108.05885, 2021b.
Hee Min Choi, Hyoa Kang, and Dokwan Oh. Unsupervised representation transfer for small net- works: believe can distill on-the-fly. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wort- Vaughan (eds.), Adv. Neural Inform. Process. Syst., 2021.
net 2ott//ooruumm1dd33311111111
Corinna Cortes, Mchryar Mohri, and Afshin Rostamizadeh. Algorithms for learning kernels based centered alignment. The Journal of Machine Learning Research, 13:795-828, -288, 2012.
Ekin Dogus Cubuk, Barret Zoph. Jon Shlens, and Quoe Le. Randaugment: Practical automated data augmentation with a reduced search space. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Ady. Neural Inform. Process. Syst., volume 33, pp. 18613-18622 18624. Cur- ran Associates, Inc., 2020. URL httppss:/ppoccceennggg.oo// wttps/eerrr.....pppppeeeeee. fi 1lle/ 1ile/d85bed d85b638666eee Ffccbbb3efcccbl..c F0ccb111cc 44d0a 3bb 7b 71w.0777888002288 b00.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In IEEE Conf. Comput. Vis. Pattern Recog., pp. 248-255. Ieee, 2009.
10

---

Under review as a conference paper at ICLR 2023
Alexey Dosovitskiy, Jost Tobias Springenberg, Martin Riedmiller, and Thomas Brox. Discriminative unsupervised feature learning with convolutional neural networks. Adv. Neural Inform. Process. Syst., 27:766-774, 2014.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, al. An image worth 16x 16 words: Transformers for image recognition at scale. In Int. Conf. Learn. Represent, 2020.
Alaaeldin El-Nouby, Gautier Izacard, Hugo Touvron, Ivan Laptev, Hervé Jegou, and Edouard Grave. Are large-scale datasets necessary for self-supervised pre-training? arXiv preprint arXiv:2112.10740, 2021.
Zhiyuan Fang, Jianfeng Wang, Lijuan Wang, Lei Zhang, Yezhou Yang, and Zicheng Liu. Seed: Self-supervised distillation for visual representation. In Int. Conf. Learn. Represent., 2020.
Yuting Gao, Jia-Xin Zhuang, Ke Li, Hao Cheng, Xiaowei Guo, Feiyue Huang, Rongrong Ji, and Xing Sun. Disco: Remedy self-supervised learning on lightweight models with distilled con- trastive learning. arXiv preprint arXiv:2104.099 091244, 2021.
Spyros Gidaris, Praveer Singh, and Nikos Komodakis. Unsupervised representation learning by predicting image rotations. In Int. Conf. Learn. Represent., 2018.
Priya Goyal, Piotr Dollár, Ross Girshick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, An- drew Tulloch, Yangqing Jia, and Kaiming He. Accurate, large minibatch sgd: Training imagenet in hour. arXiv preprint
neee::166607777 2017.
Benjamin Graham, Alaaeldin El-Nouby, Hugo Touvron, Pierre Stock, Armand Joulin, Hervé Jégou, and Matthijs Douze. Levit: A vision transformer in convnet's clothing for faster inference, In Int. October 2021.
Conf. Comput. Vis., pp. 12259-12266
Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Pires, Zhaohan Guo, Mohammad Azar, al. Bootstrap your own latent: A new approach self-supervised learning. In Adv. Neural Inform. Process. Syst., 2020.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. IEEE Conf. Comput. Vis. Pattern Recog., pp. 770-778, 2016.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. IEEE Conf. Comput. Vis. Pattern Recog,, pp- 9729-97388 2020.
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. arXiv preprint arXi::2111.06377 2021.
Byeongho Heo, Sangdoo Yun, Dongyoon Han, Sanghyuk Chun, Junsuk Choe, and Seong Joon Oh. Rethinking spatial dimensions of vision transformers. In Int. Conf. Comput. Vis., pp. 11936- 11945, 2021.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in neural network. arXiv preprint arXiv:1503.0255 2015.
Andrew Howard, Mark Sandler, Grace Chu, Liang-Chich Chich Chen, Bo Chen, Mingxing Tan, Weijun Wang, Yukun Zhu, Ruoming Pang, Vijay Vasudevan, Quoc V. Le, and Hartwig Adam. Searching for mobilenetv3. In Int. Conf. Comput. Vis., October 2019.
Gao Huang, Yu Sun, Zhuang Liu, Daniel Sedra, and Kilian Q Weinberger. Deep networks with stochastic depth. In Eur. Conf. Comput. Vis., pp. 646-661. Springer, 2016.
Sergey loffe and Christian Szegedy. Batch normalization: accelerating deep network training by reducing internal covariate shift. In Int. Conf. Machine Learning., pp. 456, 2015.

---

Under review as a conference paper at ICLR 2023
Xiaoqi Jiao, Yichun Yin, Lifeng Shang. Xin Jiang, Xiao Chen, Linlin Li, Fang Wang, and Qun Liu. Tinybert: Distilling bert for natural language understanding. In Findings of Empirical Methods in Natural Language Process., pp. 4163 -174,, 2020.
Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations for fine- fine-grained categorization. In Int. Conf. Comput. Vis. Worksh. pp. 554-561, 2013.
Alex Krizhevsky et al. Learning multiple layers of features from tiny images. Technical Report, 2009.
Yanghao Li, Saining Xie, Xinlei Chen, Piotr Dollar, Kaiming He, and Ross Girshick. Benchmarking detection transfer learning with vision transformers. arXiv preprint arXiv:2111.11429, 2021.
Yanghao Li, Hanzi Mao, Ross Girshick, and Kaiming He. Exploring plain vision transformer back- bones for object detection. arXiv preprint arXhv:2203.16523 2022.
Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pictro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Eur. Conf. Comput, Vis., pp. 740-755. Springer, 2014.
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hicrarchical vision transformer using shifted windows. In Int. Conf. Comput. Vis., pp. 10012-10022, 2021.
Ziwei Liu, Zhongqi Miao, Xiaohang Zhan, Jiayun Wang. Boqing Gong, and Stella X. Yu. Large- scale tailed recognition in an open world. In IEEE Conf. Comput. Vis. Pattern Recog., June 2019.
lly Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts. arXiv preprint arXiv: 1608.038,, 2016.
Subhransu Maji, Esa Rahtu, Juho Kannala, Matthew Blaschko, and Andrea Vedaldi. Fine-grained visual classification of aircraft. arXiv preprint 130655551 2013.
Sachin Mehta and Mohammad Rastegari. Mobilevit: Leght, general-purpont and mobile- friendly vision transformer. In Int. Conf. Learn. Represent., 2022. URL https:
openreview net fortt/foru? wiwpammmdiiv--ssed d
Alejandro Newell and Jia Deng. How useful is self self- supervised pretraining for visual tasks? In IEEE Conf. Comput. Vis. Pattern Recog., pp. 7345-7355 2020.
Thao Nguyen, Maithra Raghu, and Simon Kornblith. Do wide and deep networks learn the same things? uncovering how neural network representations vary with width and depth. In Int. Conf. Learn. Represent., 2020.
Maria-Elena Nilsback and Andrew Zisserman. Automated flower classification over a large number of classes. In 2008 Sixth Indian Conference Computer Vision, Graphics & Image Processing, pp. 729-729. IEEE, 2008
Mehdi Noroozi and Paolo Favaro. Unsupervised learning of visual representations by solving jigsaw puzzles. In Eur. Conf. Comput. Vis., pp. 69 64.84. Springer, 2016.
Junting Pan, Adrian Bulat, Fuwen Tan, Xiatian Zhu, Lukasz Dudziak, Hongsheng Li, Georgios Tzimiropoulos, and Brais Martinez. Edgevits: Competing light- cnns on mobile devices with vision transformers. arXiv preprint 22Xx:22005003336 2022.
Namuk Park and Songkuk Kim. How do vision transformers work? In Int. Conf. Learn. Represent, 2021.
Omkar M Parkhi, Andrea Vedaldi, Andrew Zisserman, and C. V. Jawahar. Cats and dogs. In IEEE Conf. Comput. Vis. Pattern Recog, pp. 3498-3505, 2012. doi: 10.1 10.1119/CVPR... WWW.VVVPP..12.228099999
Maithra Raghu, Thomas Unterthiner, Simon Kornblith, Chiyuan Zhang, and Alexey Dosovitskiy. Do vision transformers see like convolutional neural networks? Adv. Neural Inform. Process. Syst, 34, 2021.
12

---

Under review as a conference paper at ICLR 2023
Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do imagenet classifiers generalize to imagenet? In International Conference on Machine Learning, pp. $389-5400.
PMLR, 2019.
Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang, Carlo Gatta, and Yoshua Bengio. Fitnets: Hints for thin deep nets. arXiv preprint arXiv:1412.6550, 2014.
Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Siang-Chieh Chen. Mo- bilenetv2: Inverted residuals and linear bottlenecks. In IEEE Conf. Compur. Vis. Pattern Recog., pp. 45200, 2018.
Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. Distilbert, distilled version of bert: smaller, faster, cheaper and lighter. arXiv preprint arXiv:1910.0110: 2019.
Zhiqiang Shen and Eric Xing. fast knowledge distillation framework for visual recognition. In Eur. Conf. Comput. Vis., pp. 673-690. Springer, 2022.
Zhiqiang Shen, Zechun Liu, Jie Qin, Lei Huang, Kwang-Ting Cheng, and Marios Savvides. S2-bnn: Bridging the gap between supe-sue real and 1-bit neural networks via guided distribution calibration. In IEEE Conf. Comput. Vis. Pattern Recog., pp. 2165-2174, 2021.
Le Song, Alex Smola, Arthur Gretton, Justin Bedo, and Karsten Borgwardt, Feature selection via dependence maximization. The Journal of Machine Learning Research, 13(5), 2012.
Andreas Steiner, Alexander Kolesnikov, Xiaohua Zhai, Ross Wightman, Jakob Uszkoreit, and Lucas Beyer. How to train your vit? data, augmentation, and regularization in vision transformers. arXiv preprint ar Xrxiv:2006. 102.00 2021.
Weiyue Su, Xuyi Chen, Shikun Feng, Jiaxiang Liu, Weixin Liu, Yu Sun, Hao Tian, Hua Wu, and Haifeng Wang. Ernie-tiny: progressive distillation framework for pretrained transformer com- pression. arXiv preprint arXiv:2106.0223 02241, 2021.
Zhiqing Sun, Hongkun Yu, Xiaodan Song. Renjie Liu, Yiming Yang, and Denny Zhou. Mobile- bert: compact aask-agnostic bert for resource-limited devices. In Association for Computational Linguistics, pp. 2158-2170, 2020.
Mingxing Tan and Quoc Le. Efficientnet: Rethinking model scaling for convolutional neural net- works. In Int. Conf. Machine Learning.. pp. 6105-6114. PMLR, 2019.
Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Herve Jegou. Training data-efficient image transformers & distillation through attention. In Int. Conf. Machine Learning.. volume 139, pp. 10347-10357, July 2021a.
Hugo Touvron, Matthieu Cord, Alexandre Sablayrolles, Gabriel Synnaeve, and Hervé Jégou. Going deeper with image transformers. In Int. Conf. Comput. Vis., pp. 32 42, 2021b.
Hugo Touvron, Matthieu Cord, and Hervé Jégou. Deit ii:: Revenge of the vit. arXiv preprint arXiv:2204.071118 2022.
Grant Van Horn, Oisin Mac Aodha, Yang Song, Yin Cui, Chen Sun, Alex Shepard, Hartwig Adam, Pietro Perona, and Serge Belongie. The inaturalist species classification and detection dataset. In IEEE Conf. Comput. Vis. Pattern Recog., June 2018.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. Adv. Neural Inform. Process. Syst., 30, 2017.
Wenhui Wang, Furu Wei, Li Dong, Hangbo Bao, Nan Yang, and Ming Zhou. Minilm: Deep self- attention distillation for task-agnostic compression of pre-trained transformers. Adv. Neural form. Process. Syst., 33:5776-5788, 2020.
Wenhui Wang. Hangbo Bao, Shaohan Huang, Li Dong, and Furu Wei. Minilmv2: Multi heati-hhaddee self- attention relation distillation for compressing pretrained transformers. In Findings of Int. Joint Conf. on Natural Language Process, pp. 2150-2151, 2021.
13

---

Under review as a conference paper at ICLR 2023
Ross
Wightman. Pytorch image models.
https://ggttaab..mm///tttttttttt
iamehh-imag-mmdeell 2019.
Ross Wightman, Hugo Touvron, and Hervé Jégou. Resnet strikes back: An improved training procedure in timm. arXiv preprint arXiv:2110.0044 2021.
Zhenda Xie, Zheng Zhang, Yue Cao, Yutong Lin, Jianmin Bao, Zhuliang Yao, Qi Dai, and Han Hu. Simmim: A simple framework for masked image modeling. In IEEE Conf. Comput. Vis. Pattern Recog., 2022.
Haotian Yan, Zhe Li, Weijian Li, Changhu Wang, Ming Wu, and Chuang Zhang, Contnet: Why not convolution and transformer at the same time? arXiv preprint arXiv:2 204i 13497, 2021.
Chenglin Yang. Yilin Wang, Jianming Zhang, He Zhang, Zijun Wei, Zhe Lin, and Alan Yuille. Lite vision transformer with enhanced self-attention. arXiv preprint arXiv:2112. 2021.
Sangdoo Yun, Dongyoon Han, Seong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo. Cutmix: Regularization strategy to train strong classifiers with localizable features. In Int. Conf. Comput, Vis., pp. 6023-6032, 2019.
Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. In Int. Conf. Learn. Represent. 2018.
Richard Zhang, Phillip Isola, and Alexei A Efros. Colorful image colorization. In Eur. Conf. Comput. Vis., pp. 649-666. Springer, 2016.
Jinghao Zhou, Chen Wei, Huiyu Wang, Wei Shen, Cihang Xie, Alan Yuille, and Tao Kong. ibot: Image bert pre-training with online tokenizer. Int. Conf. Learn. Represent, 2022.
14


Reviews:
[{"review_id": "ub_hCVK5X-6", "reviewer": "Reviewer_nNkX", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents an analysis of two self-supervised training methods for lightweight vision transformers. The methods are MAE (a masking method) and MoCoV3 (a contrastive method). It shows that although MAE appears stronger when training (w/o labels) and evaluating on ImageNet, the roles reverse on downstream tasks. It analyzes the differences of these two networks and compares them a supervised method (DeitT), by looking at layer-by-layer similarity metrics, as well as analysis of the locality vs. globality of the attention maps. Finally it proposes to use distillation from a large-scale MAE model to improve the tiny MAE. It does this with significant improvements to downstream tasks.", "summary_of_the_review": "The papers presents a lot of analysis around lightweight ViTs, and shows the efficacy of student/teacher distillation for mask-based self-supervised pre-training. The paper presents few novelties in terms of technique and method, but the quality and novelty of the analysis and strength of final distilled results are solid contributions. The topic is of broad interest, so many could benefit from this work. Balancing the lack of novelties with the potential benefits it is close to borderline for me. However, I do lean in favor of acceptance.", "strength_and_weaknesses": "Strengths:\n- This is a pretty in depth analysis of the difference between a MAE- and a MoCo-based self-supervised ViT.\n- The distillation of the MAE-Tiny from MAE-Base leads to significant improvements over the MAE-Tiny without teacher guidance. The final results compared to supervised pretraining is strong and a significant achievement.\n\nWeaknesses:\n- The paper mostly presents analysis and new applications of old techniques. It is pretty light on novelties though.\n- Not all analysis is tied back to experimental results, which means it's not clear if we can draw actionable conclusions from it. If the analyses had led to more actionable changes that resulted in improvements, the case would have been stronger.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "QgSFBaSbw6", "reviewer": "Reviewer_UM5C", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper aims to analyze how the lightweight ViTs (e.g., ViT-tiny and DeiT-tiny) perform when using self-supervised pretraining methods. By conducting a variety of experiments, the authors show that: 1) MIM-based methods, like MAE, helps more than contrastive learning based methods, like MOCOv3; 2) Low-level layers matter more than high-level layers when sufficient data for finetuning is available; 3) KD methods help improve the representative ability of high-level layers.\n\nPrevious works mostly focus on large-scale models, which often contain more than 80M parameters. In contrast, these paper aims to reveal how lightweight ViTs behave on both ImageNet and some downstream tasks, which makes this paper much more different than previous works on self-supervised methods.", "summary_of_the_review": "The intention of this paper is interesting. There are no relevant papers reporting similar conclusions. Though the novelty of this paper is not significant enough, regarding the thorough experiments that have been done, I give a score of 5 at this moment. I would like to lift the rating if the authors can more clearly explain the significance of the contributions.", "strength_and_weaknesses": "Strength\n- Analyzing the performance of lightweight ViTs in self-supervised learning manners is important in that in most cases it is not appropriate to use large-scale models but only tiny-sized ones. This paper clearly shows how lightweight ViTs behave under different settings and evaluates the pretrained models on downstream tasks.\n- Authors show that low-level layers are more important than high-level layers after pretraining. This is an important signal for the development of SSL methods in future.\n- Distilling the knowledge from a large model to lightweight ones is an interesting topic. This paper provides an effective way to do so and analyze how to do distillation helps more for the lifting the model performance.\n\nWeaknesses\n- Only two SSL methods are selected for presentation. One is MOCOv3 and the other one is MAE. It would be better if two methods are selected for explanation for each type of SSL method. The conclusion would be more convincing.\n- From Table 3, we see that the pre-training benefits little from large-scale data.  With only 10% of the ImageNet 1k data are provided, the classification performance after finetuning is already good. I suppose there are some explanations on the reasons but obviously there is not.\n- The analysis is interesting. I am looking forward to taking some message about how to design better network architectures or how to develop more advanced self-supervised learning methods.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "1IE5PzEDKj", "reviewer": "Reviewer_PEwP", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This is an empirical paper focusing on exploring the task of self-supervised learning on lightweight vision transformers (ViTs). In this paper, the authors observed several discoveries, such as: the tiny model fails to benefit from large-scale pre-training data and shows inferior performance on data-insufficient downstream tasks. They further proposed a distillation strategy during pretraining to improve the representation ability of compact ViTs. Experiments are conducted on ImageNet pre-training and multiple downstream tasks and datasets.", "summary_of_the_review": "Overall, this is an empirical paper with some trivial observations which are not well supported by the experiments and some are even not new. Also, the organization and writing can be improved significantly in this paper. Thus, I tend to reject it.", "strength_and_weaknesses": "### Strengths:\n\n   - It is interesting to explore suitable methods for lightweight vision transformers or other efficient models in the self-supervised learning manner. \n\n   - This paper provided extensive experiments on ImageNet pre-training and multiple downstream datasets and tasks, such as classification, object detection, and segmentation.\n\n\n### Weaknesses:\n\n   - Though this is an empirical paper, the novelty and originality in it are fairly limited, as well as the significance and contribution which are also not strong. The observations that tiny models fail to benefit from large-scale pre-training data, and rely more on the downstream dataset scale are a little bit straightforward and not surprising. The use of knowledge distillation for self-supervised learning on lightweight models also has been proposed for a long time, e.g., on low-bit efficient models [1] and mobile-level models [2].\n\n[1] Shen, Z., Liu, Z., Qin, J., Huang, L., Cheng, K. T., & Savvides, M. (2021). S2-bnn: Bridging the gap between self-supervised real and 1-bit neural networks via guided distribution calibration. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 2165-2174).\n\n[2] Fang, Zhiyuan, Jianfeng Wang, Lijuan Wang, Lei Zhang, Yezhou Yang, and Zicheng Liu. \"SEED: Self-supervised Distillation For Visual Representation.\" In International Conference on Learning Representations. 2021.\n\n   - Some statements in this paper are not well supported, such as “lower layers of the pre-trained models matter more than higher ones if sufficient downstream data is provided, while higher layers matter in data-insufficient downstream tasks.” I think a better and fair comparison design is crucial and also necessary for this argument. The current experiments for this part are not rigorous to prove it.\n\n   - The writing and organization of this paper can also be improved. For instance, it’s not clear to me why Table 1 is located in the early part of the paper. I did not get much information from it and do not know what the insight of this table is.\n\n   - Overall, this paper seems a little bit incremental without providing new conclusions or discoveries over previous literature.\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject