Hybrid Distillation: Connecting Masked Autoencoders with Contrastive Learners
Authors: Bowen Shi
XIAOPENG ZHANG
Yaoming Wang
Jin Li
Wenrui Dai
Junni Zou
Hongkai Xiong
Qi Tian
Venue: ICLR 2024 poster (2024)
Keywords: contrastive learning
mask image moding
feature distillation
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning

Abstract: As two prominent strategies for representation learning, Contrastive Learning (CL) and Masked Image Modeling (MIM) have witnessed significant progress. Previous studies have demonstrated the advantages of each approach in specific scenarios. CL, resembling supervised pre-training, excels at capturing longer-range global patterns and enhancing feature discrimination, while MIM is adept at introducing local and diverse attention across transformer layers. Considering the respective strengths, previous studies utilize feature distillation to inherit both discrimination and diversity. In this paper, we thoroughly examine previous feature distillation methods and observe that the increase in diversity mainly stems from asymmetric designs, which may in turn compromise the discrimination ability. To strike a balance between the two properties, we propose a simple yet effective strategy termed Hybrid Distill, which leverages both the CL and MIM teachers to jointly guide the student model. Hybrid Distill emulates the token relations of the MIM teacher at intermediate layers for diversity, while simultaneously distilling the final features of the CL teacher to enhance discrimination. A progressive redundant token masking strategy is employed to reduce the expenses associated with distillation and aid in preventing the model from converging to local optima. Experimental results demonstrate that Hybrid Distill achieves superior performance on various benchmark datasets.

Published as a conference paper at ICLR 2024
HYBRID DISTILLATION: CONNECTING MASKED AU- TOENCODERS WITH CONTRASTIVE LEARNERS
Bowen Shi!
Xiaopeng Zh Yaoming Wang'
Jin Li
Wenrui Dai
Junni Zou'
Hongkai Xiong'
Qi Tian
'Shanghai Jiao Tong University Huuawei Inc.
{s jejtushhoowen, wang-yaoming, deserve.1i, daiwenrui, zoujunni, www.gggihhhww.ccommmmm com
x kionghongkai/ess edu..edu.coo en; exphistory@gmail.c com, tian. tww..qilnniii.ee..e..
ABSTRACT
As prominent strategies for representation learning, Contrastive Learning (CL) and Masked Image Modeling (MIM) have witnessed significant progress. Previous studies have demonstrated the advantages of each approach specific scenarios. CL, resembling supervised pre-training, excels at capturing longer- range global patterns and enhancing feature discrimination, while MIM is adept at introducing local and diverse attention across transformer layers. Considering the respective strengths, previous studies utilize feature distillation to inherit both discrimination and diversity. In this paper, we thoroughly examine previous fea- ture distillation methods and observe that the increase in diversity mainly stems from asymmetric designs, which may in turn compromise the discrimination abil- ity. To strike a balance between the two properties, we propose a simple yet ef- fective strategy termed Hybrid Distill, which leverages both the CL and MIM teachers to jointly guide the student model. Hybrid Distill emulates the token relations of the MIM teacher at intermediate layers for diversity, while simulta- neously distilling the final features of the CL teacher to enhance discrimination. A progressive redundant token masking strategy is employed penses associated with distillation and aid in preventing the model from converg- ing to local optima. Experimental results demonstrate that Hybrid Distill achieves superior performance on various benchmark datasets. The code is available at httt:a/s//sg...tbb...oo/////////
reduce the ex-
INTRODUCTION
Self-supervised pre-training has recently emerged promising alternative supervised image classification (He et al. 2osoooos et al. 2020), particularly with Contrastive Learning (CL) and Masked Image Modeling (MIM). The former one, typical representatives MoCo (He al. 2020) and SimCLR (Chen et al. 2020a), learns invariant representation for positive views via dir- ferent augmentations of the same image. Furthermore. CLIP (Radford al.) (02T] extends CL in a multi-modal manner by pairing the given image with its corresponding text description. While the latter, including MAE (He et al., 2022) and SimMIM (Xie et al. 2022b), aims to reconstruct the masked image patches and has become mainstream due to its efficieccy brought by mask operations.
The different pre- pre-training paradigms of CL and MIM advance a series of studies (Xie et al.] 2022a] Park et al., 2023 Wang al., (223) that aim at understanding their intrinsic properties. These studies point out that CL behaves more similarly to supervised pre-training. i.e., provides models with longer-range global patterns focusing on object shape, particularly in the last few layers (Park cet al., 2023). which enables feature representation with better discrimination that are beneficial for recognition (Huang et al. 2022). However, shown in Fig. I it causes self-attention in the last few layers to collapse into homogeneity, with attention distances located within a small distance range. In contrast, MIM brings both local and global attention and evenly distributed representations across all layers (Xie et al., [202aa] Park et al. 2023), and this attention diversity contributes to its better generalization on downstream fine-tuning specifically for dense-level tasks (Park al. (023). Nevertheless, MIM underperforms in linear probing, mainly due to its lack of discrimination ability.
"Corresponding author. This work done when Bowen Shi interned Huawei Inc.

---

Published as a conference paper at ICLR 2024
140
120
(a) DeiT
(b) CLIP
MAE
Figure Average attention (a) DeïT, (b) CLIP and (c) MAE pre-trained models.
Previous methods (Wei et al.,
2022b; Fang et al. 2022 Liu et al. 2022 Wei et al. 2022a Peng
al. 2022 propose to utilize feature distillation to ensure both discrimination and diversity. Among them, dBOT Liu et al. 2022) replaces the reconstructing objective of MAE with the feature maps of different pre-trained teachers. It finds that feature distillation can bring diverse attention regardless of the teacher model, and after multi-stage distillation, the performance is comparable across different teachers even with the randomly initialized ones. Also observing that distillation benefits diversity, FD (Wei al. (202bb directly distills feature maps from supervised/CL, teachers to relieve the attention collapse and achieves considerable downstream performance gains. Although interesting and important, we argue that their findings are incomplete.
This paper re-examines these findings and reconsiders the importance of diversity and discrimina- tion. We reveal the following observations: (i) The increase diversity derives from the asym- metric architecture designs, rather than feature distillation itself. (Section 2.2) After removing the asymmetric attention in Wei et al., (202b)) and encoder-decoder designs in Liu al.] (2022), we observe a negligible increase (or even a decrease) in attention diversity. (ii) The asymmetric decoder de facto harm the discrinination over the encoder side, for it migrates the semantic information of the teacher model. (Section 2.3) Due to the decomposition of the encoding and de- coding functions, the student encoder tends to summarize more general information, thus gradually losing semantics obtained from teachers and yielding similar results after multi-stage distillation (Liu et al. 2022). (iii) Mask reconstruction of high-level semantics does not help improve di- versity. (Section 2.4 Reconstructing high-level information (Peng et al. 2022 Fang et al. 2022] Wei et al. 2022a) is similar to direct feature distillation and lacks the diversity found in MIM, which implies that the attention diversity of MIM mainly comes from low-level reconstruction objectives.
Based on the above observations, we argue that uni-model distillation is limited for transferring both diversity and discrimination the student side. To solve this issue, we propose a simple yet effective feature distillation method, termed as Hybrid Distill, to simultancously inherit these two properties via distilling knowledge from both the supervised/CL and MIM teachers. Hybrid Distill makes careful designs for the distilling target and location. Specifically, we find that the relational modcling ability of MIM is crucial for preserving token diversity, while the feature maps of su- pervised/CL. teachers are beneficial for discrimination. Accordingly, we set the token relations of the MIM teacher and the feature maps of the supervised/CL. teacher as the distilling objectives. The token relations cur, while the feature maps are distilled in the final layer to preserve semantics. Additionally, Hybrid Distill utilizes a progressive redundant token masking strategy reduce distilling costs and prevent falling into local optima. Experiment results show that the distilling strategy works surprisingly well even when using MAE and CLIP teachers, i.e., MAE pretrained with only 1.28M ImageNet images also boosts the large-scale (400M) pretrained CLIP teacher on different downstream tasks.
distilled in layers preceding the final layer where attention collapse tends to
In nutshell, this paper makes the following contributions:
We re-examine the findings of previous feature distilling methods and point out that their increas- ing diversity mainly arises from the use of asymmetric designs, while these designs may in turn compromise the discrimination.
We further propose Hybrid Distill framework that utilizes both supervised/CL, and MIM teach- ers to provide the student with higher-quality discrimination and diversity. Distilling targets and locations are carefully designed Hybrid Distill to fully exploit the strengths of both teachers.
We conduct property analysis demonstrate that the representations exhibit both discrimination and diversity in Hybrid Distill. Experiments on various downstream tasks, including classification, detection, and segmentation, also showease its superiority.
2

---

Published as a conference paper at ICLR 2024
Smooth L1
Smooth LI
Various Decoder
Linear Projection
Sup/CL Teacher
Sup/CL Teacher
Student
Student
(a) Feature distillation
(b) Mask feature reconstruction
The distillation configurations inspected in Sec. 2 We examine the influence of (a) dif- ferent decoders for feature distillation (no decoder, linear projection, asymmetric decoder), as well as replacing the feature distillation objective with (b) mask feature reconstruction. Blue and grey squares represent visible and mask tokens, respectively.
Figure 2:
2 MODEL EVALUATION: DIVERSITY AND DISCRIMINATION
This section re-examines the findings of previous feature distillation mask feature reconstruction works (Liu et al. 2022 Wei et al. 2022b) illustrated in Sec. [ and highlights their limitations in incorporating diversity and discrimination. The distillation settings follows et al. (2022); Wei et al. (202b),, where only features in the final layer of a ViT-B teacher (Dosovitskiy et al. 2020) utilized distillation objective. Smooth L1 loss is used during the distillation process.. Different configurations we inspected are shown in Fig. 2
2.1 PRELIMINARY
Definition of discrimination and diversity. We first introduce the definitions of diversity and discrimination used to judge the representations.
Discrimination means that the representations contain more global patterns tailored to object shapes, which is beneficial for recognizing objects and distinguishing images. A similar definition of discrimination can be found in Huang et al. (2022).
Diversity means that the model pays attention to both local and global information and can acquire representations through more diverse attention with different attention distances, particularly in the last few layers. This notion of diversity is similarly defined and discussed in Wei et al. (2022b); Xie
al. 2022a Park et al. 2023).
Evaluation strategies. We measure discrimination and diversity by per-head average attention dis- tance (Dosovitskiy al. 2020) and normalized mutual information (NMI) (Strehl & Ghosh 2002).
We also include additional linear probing evaluation in Sec. B.I of the appendix.
The average aftention distance, which is also used in Liu et al. (202)); Wei et al. (202bb ; Xie et al. 2022a): Park et (2023), calculates the distance between the query and the key tokens based on the attention weights and averages them by head in each transformer layer, providing insight of whether the attention is global or local. A low attention distance value means that queries attend to small regions near the query location, while a high value means queries have a larger receptive field.
The NMI metric, utilized in Park et al. (023), measures the attention is attending to different tokens or similar ones. A low NMI indicates attention maps are less reliant on the query tokens, implying that all queries focus similar tokens. Let p(g) N represent the distribution of query tokens, where N is the total token number. The joint distribution of query and key computed as p(q.k) <kkg))p(q). where x (kll)) is the normalized self-attention matrix. Thus, NMI can be calculated by Ilg.k) where 7 I(g, k)) is the mutual information and H(-) is the marginal entropy.
HHH())(())
2.2 THE INCREASE IN DIVERSITY DERIVES FROM THE ASYMMETRIC DESIGNS
Fig. 3 (a)-(c) measures the average attention distance after feature distillation with various decoders appended the student encoder. Note that for the asymmetric decoder, we also visualize the average attention distance of the decoder for comparison (r.f. Fig. C)), while the actual encoder layers are the first twelve layers. It can be seen that using no decoder or linear projection leads to a negligible increase (or even decrease) in attention diversity compared to teacher models (r.f. Fig. [ (a)-(b)),
3

---

Published as a conference paper at ICLR 2024
(a) decoder
(b) Linear projection
(c) Asymmetric decoder (d) Mask feature reconstruction
Figure 3: Average attention distance when using (a) decoder, (b) linear projection, and (c) asym- metric decoder after the student encoder and setting (d) mask feature reconstruction as the learning objective. The first and the second rows are distilled using DeiT and CLIP teachers, respectively.
(a) Various decoders
(b) Encoder and asymmetric decoder
(c) Mask feature reconstruction
Figure 4: The normalized mutual information (NMI). We visualize (a) the encoder behavior when using various decoders and (c) the mask feature reconstruction objective. (b) further visualizes the behavior of the introduced asymmetric decoder based on (a).
reflecting that feature distilling itself cannot bring diversity. Adding extra attention layers at the decoder can make the student encoder more diverse, but hinders its discrimination since the last encoder layer (numbered 11) no longer captures long-range patterns. Fig. Faa further compares NMI using the DeiT teacher and the results are in line with Fig. i.e., without asymmetric designs, the student collapses into homogeneity and behaves similarly in the last few layers. Conversely, the use of asymmetric decoders greatly reduces discrimination of the encoder, queries no longer pay attention to the main subjects in the last encoder layer.
Note that the above discussions focus decoders, while for FD (Wei al. 2022b), the asymmetric designs rise from adding additional learnable parameters and relative position bias at the encoder side. In Sec. B.2 of the appendix, demonstrate that for FD, the increase in diversity also arises from these designs while the diversity brought by them is limited and not always significant.
2.3 THE ASYMMETRIC DECODER HARMS THE ENCODER DISCRIMINATION
Fig. Bic) and Fig. b)) further measure the average attention distance and NMI of the asymmetric decoder. Our findings suggest that the decoder transfers the discrimination of the teacher, as its behavior similar that of the last few layers the teacher model where queries pay attention to similar tokens. Reducing the number of decoder layers does not eliminate this transfer, as further demonstrated in Sec. B33] of the appendix. Since only the student encoder is retained and applied to downstream tasks after distillation, the semantic information maintained is weakened, which explains why dBOT (Liu et al. 2022), different teachers tend to yield similarly behaving models after multi-stage distillation. Note that dBOT conducts feature distilling in mask reconstruction way, while we demonstrate in both Sec. 2.4 and the visualization in Sec. B.4 of the appendix that behaves similarly to directly distilling features.
2.4 MASK RECONSTRUCTION OF LEGEL SEMANTICS DOES NOT IMPROVE DIVERSITY
Fig. d) and Fig. [c) examine the influence of mask reconstructing high-level information. To eliminate the effect of the asymmetric decoder, feed both the masks and tokens into the encoder
4

---

Published as a conference paper at ICLR 2024
Sup/CI Teacher
MAC
Teacher
CLE
1101
192nd2aa IMAE
Relatlons
Features
Token Dropping
Distillation
Relalion
Dcpih
Features
Color Bar
Tuned
Masked
#L-1 #L-2
Student at.
F Frozen
Unmasked
More Irrelevant
Figure 5: Hybrid Distill pipeline and its effectiveness in ensuring discrimination and diversity. Note that this figure, we further average the per-head attention distance and obtain a single average attention distance for each layer reflect the per-layer overall distance.
simultaneously and only linear projection the decoder. The overall process is thus similar to SimMIM (Xie et al. (2022b), except that we use the high-level information obtained from the su- pervised/CL teacher the distilling objective. The attention distance and the NMI value prove that reconstructing high-level information brings no diversity gains towards directly distilling features, which is consistent with the finding of Xue al. (2022), i.e. reconstruction is unnecessary for MIM with semantic-rich teachers. This phenomenon also implies that the diversity of MIM mainly arises from the low-level reconstructing objective since diversity is absent in high-level reconstruction.
HYBRID DISTILLATION
From the above discussion, conclude that existing single-teacher distillation pipelines have lim- itations in providing discrimination and diversity. In this section, we propose hybrid distillation framework, termed Hybrid Distill, to ensure both discrimination and diversity.
3.1 OVERVIEW
Given supervised/CL pre-trained model Te, and MIM pre-trained model Tm Hybrid Distill simultaneously distills knowledge from these two different types of teachers, aims at combining their respective advantages enhance the model So where is its learnable parameters. ViT (Dosovisskiy et al. 2020) is adopted for all the models, and T,,, is provided by MAE (He al. (022) while Ta is provided by DeiT (Touvron al. 2021] or CLIP (Radford et al. 2021). Specifically, we employ symmetric design in Hybrid Distill, i.e., avoiding adding any types of decoder the student model maintain the discrimination. The overall Hybrid Distill framework is shown in Fig. and its overall objective is:
representations in randomly initialized student
E OD (T.(z) O (M + x)} 2~N
(D M, S o x)),
where is an element-wise product operation. . mask provided by the teacher model using the strategy described in Sec. 3.2 and M @ x denotes the unmasked patches. D(.,.) is the distance measurement, and we use smooth L1 distance our experiment. a and 33 is the hyperparameter that controls the contribution of the two teachers. Note that we do not distill the final output features Tm((x) from the MIM pre-trained model but instead use the token relations preceding the last layer, denoted as T' (z), the learning objective. Sy is similar to TV (z), which represents the token relations of the student model. Details are illustrated in Sec. 3.2
3.2 DISTILLING STRATEGIES
What to distill? Different from previous works (Wei et al. 2022b Fang et al. 2022 Xue et al.
2022) that directly distill the features of teacher models, analyze that the diversity of MIM arises from their superior token-level relationship modeling, while features from supervised/CL teachers excel image-level discrimination. Hence, apply different distilling targets to T. and T...
5

---

Published as a conference paper at ICLR 2024
Layer
Loyer 12
Layer
Layer
(0) Average head distance
(b) NMI
(c) Amention visualization
Figure 6: The (a) average attention distance, (b) NMI, and (c) attention visualization of the student model obtained from Hybrid Distill with MAE and CLIP teachers.
Specifically. taking Tm as example, we decompose Tm into T,, o 2 o ... o The where Tü the ht layer of T and composed of a multi-head self-attention (MSA) layer and MLP layer. Given I'm as the input of the fth layer, the calculation in T can be represented as:
R (mmm) Qi (mim) K (=rm)"
MSA (#t Softmax (R (////V/ V (=
(2)
Th (m))) 2th + MLL + MSA,, (s(m))
where Q K; and V denotes the linear mappings for zrt, and equals the dimension of amm- Then, for MIM teacher Tm, supervised/CL teacher Te, we set the output features (xr)) as the target.
the token relation Ry (set, as the distilling target, while for
Where to distill? As shown in Fig. [b), supervised and CL. models tend to collapse into homo- geneity in the last few layers, so Hybrid Distill chooses to distill token relations from Tm in these layers address this collapse and improve diversity. While for the last layer of S which is cru- cial for discrimination, Hybrid Distill directly distills knowledge from Te using the output features. Specifically, we distill token relations from Tm at the L- 1 and L layers and distill features from Te at the L layer of VIT. Accordingly, the learning objective (xx and T (x) in Eq. become:
Te(x) = (z). T", (x) = [R (2), (x)]
Distillation acceleration. Since tokens are redundant in image, it is promising to mask some unnecessary tokens of the student model to reduce memory and time costs. We also find that removing redundant tokens can play regulatory role, helping the model avoid local optima during the distillation process. Specifically, we use the MIM teacher T,, to guide the identification re- dundant tokens and provide the token mask. Inspired by Li al. (2023). redundant token masking strategy, which generates token masks at different layers of Tm a pro- gressive manner. Given token sequence a and the mask M provided by the previous layer, define the tokens in i @ M and are top K% similar to their average token as redundant tokens in the tt layer and generate a redundant token mask for them. The above process is denoted as T(x O M K). Next, we update M using T(x,, C M K) and M as follows:
propose progressive
[M - T( O Mi-1 K), M
ifiET if o I.
MC. m
(4)
where / is the set of layers required to update the token mask. For M all elements are set to 1. Finally, we set the mask M for the student model as M ML
3.3 PROPERTY ANALYSIS
Average attention distance. Fig. 6[a visualizes the average attention distance of the student model with CLIP and MAE as teachers, while the visualization of CLIP and MAE teachers are shown Fig. []] These visualizations demonstrate that Hybrid Distill enhances the discrimination ability of the student model, compensating for the semantic lacking of the MAE teacher. Moreover, Hy- brid Distill avoids succeeding attention collapse from the CLIP teacher and generates more diverse representations in the last few layers.
Normalized mutual information. Fig. b(b) further inspects the NMI. The results demonstrate
that the mutual information between tokens is significantly enhanced in the layers where the MAE

---

Published as a conference paper at ICLR 2024
Table I: Main results on ImageNet-IK classification, COCO detection and instance segmenta- tion, and ADE20K semantic segmentation. *: using MAB+DeiT teachers. f: using pretrained MAE+CLIP teachers. I: using ImageNet-2Ik dataset for distillation.
COCO
Method
Backbone
Distill.
IN-IK
ADE20K
Ap
Ap
DeiT Touvrro et 2021
81.8
46.9
41.5
47.0
MoCo V3 (Chen et a. 2021
83.2
45.5
40.5
47.1
DINO
Caron et al. 2021
83.3
46.8
41.5
47.2
MAE
He et al. 2022
VIT-B
83.6
48.4
42.6
48.1
CAE
Chen et al. 2022a
83.3
48.0
42.3
47.7
SdAE
[Cen et al.
2022b)
84.1
48.9
43.0
48.6
CLIP (Radford et al. (0021)
83.6
47.6
42.3
49.6
Distill-Deit
82.0
47.7
42.1
47.3
Distill-MAE
VIT-B
83.7
49.1
43.1
47.8
Hybrid Distill*
83.7
50.3
44.2
49.1
Distill-CLII
84.8
49.5
43.5
50.3
ViT-B
Hybrid Distill
85.1
50.6
44.4
51.5
MAE (He al.) 2022
85.9
54.0
47.1
53.6
ViT-L
CLIP (Radford et al. 2021)
86.1
52.7
46.2
54.2
Hybrid Distill
VIT-L
87.6
54.4
47.4
55.9
Hybrid Distill
VIT-L
88.0
54.6
47.6
56.3
token relationships are distilled. Besides, this enhancement does not compromise the discrimination obtained from CLIP, as evidenced by attention in the final layers still attending to similar tokens.
Attention visualization. Fig. ( further visualizes the attention between given query and other keys at different layers to examine behaviors. Compared to MAE, Hybrid Distill exhibits better discrimination ability, ie., the query tokens of the last layer have global attention towards the main object of the images, regardless of their location. Besides, Hybrid Distill also improves the locality of the model in the 10th layer, where attention collapse is known to occur in the CLIP teacher.
4 EXPERIMENTS
4.1 IMPLEMENTATION DETAILS
Our experiments conducted 8 V100 GPUs. The batch size, learning rate, and weight decay are set to 1024, 6e-4, and 0.05, respectively. AdamW (Loshchilov & Hutter] (017) optimizer and cosine decay (Loshchilov & Hutter 2016) schedule is used. The input size 224 For ViT-B, the distillation is based on ImageNet- IK Russakovsky et al. (2055), and the epoch is 300 for main results and 100 for ablation studies. For ViT-L, we conduct 300 epoch distillation based on ImageNet-Ie K and 40 epoch distillation based on ImageNet-21K, respectively. The hyperparameter a and are set to 1.0 and the redundant token masking set is set to [0, 2//32TT/3 following [i et (023)) The performances are tested on different downstream tasks, including ImageNet- CIFAR100 Krizhevsky et al. 2009), Cars (Krause et al. 2013), and iNaturalist19 (Van Horn et al., 2018) classification, COCO (Lin et al. 2014) object detection and instance segmentation, and ADE20K
(hoou et al. 2019) segmentation. More downstream details are included the appendix.
4.2
MAIN RESULTS
This section presents benchmark results of Hybrid Distill on different downstream. We also list results for supervised and self-supervised pre-trained models, well as 300-epoch uni-distillation baselines which use the same symmetrical structures as Hybrid Distill for comparison. As shown Tab. Hybrid Distill achieves performance gains on all downstream tasks, especially for the dense-level ones that rely more diversity. Specifically. although the performance of DeiT suboptimal, its strength can be complementary to MAE and brings considerable benefits, i.e., when using DeiT and MAE teachers. Hybrid Distill achieves 50.3 Apbox and 44.2 Apmask on COCO, well 49.1 mloU ADE20K, surpassing Distill-MAE by 1.2, 1.1, and 1.3, respectively. Similarly, Hybrid Distill achieves 50.6 Aphox and 44.4 Apmask on COCO, well as 51.5 mloU on ADE20K when using CLIP and MAE teachers, outperforming Distill-CLIP by 1.1, 0.9, and 1.2, respectively. When using the ViT-I. backbone and larger-scale ImageNet-21 21 dataset for distillation,

---

Published as a conference paper at ICLR 2024
Table 2: Classification results on CIFAR 100, Cars and INautralist19, x: using MAE+DeiT teachers.
f: using MAE+CLIP teachers. f using ImageNet-211 dataset for distillation.
Method
Backbone
CIFARI00
Cars
INtuuralist99
Mean
DeiT (oovroon al. 2021)
ViT-B
91.4
92.0
77.3
86.9
MAE (He et al. 2022)
VīT-B
89.6
89.5
75.2
84.8
Distill-Deil
VīT-B
91.2
92.5
78.3
87.3
Distill-MAE
VīT-B
90.3
93.1
79.0
87.5
Hybrid Distill*
VīT-B
91.7
94.1
80.2
88.7
DisP
ViT-B
91.6
94.3
81.6
89.2
Hybrid Distill
VIT-B
92.0
94.5
81.9
89.5
Hybrid Distill
VIT-L
94.5
95.6
85.3
91.8
Table 3: Different combinations two teacher Table 4: Different combinations of two teacher models. Tr(#): CLIP, (x)): MAE. *: using the ImageNet-100 100 pretrained weights.
models. Tx(x): DeiT, Tm (x)): MAE.
Targets (x(
AP box
A Apmas
Targets
Apbox
Apmask
47.5
41.8
Tx(x)
49.1
43.1
Tm
48.9
43.1
(m(x)
48.9
43.1
Tx(x) + (x)
46.8
41.5
T((x) + (x)
49.1
43.2
Tm + (T)
48.9
43.2
Ta(x) + +T
50.4
44.1
Te(x) (x)
50.0
43.9
Te(x) + (x))
49.5
43.5
the performance can be further boosted to 54.6 Apbox, 47.6 Apmask and 56.3 mloU on respective tasks. Hybrid Distill does not bring noticeable gains on the ImageNet-1K K dataset, which is in line with our expectations since diversity of limited help for image-level classification, especially when the distillation and downstream data distribution are the same (Xie et al., (022a)). However, the still comparable results towards Distill-CLIP reflect that Hybrid Distill increases diversity without sacrificing discrimination. Besides, we prove in Tab. that diversity can also bring benefits to image classification when distillation and downstream data distribution are different, i.e., Hybrid Distill achieves more significant gains on several small-scale classification datasets.
4.3 ABLATION STUDY
Different combinations of two teachers. We first evaluate the benefits of combining teachers for distillation. As shown in Tab. adding additional MAE attention regularization can bring notice- able improvements (2.5 on Apbox and 1 on A Apmack) compared to directly distilling from the DeiT teacher. Moreover, the additional attention regularization cannot bring benefits when only using single DeiT teacher, which suggests that the benefits come from the introduction of MAE teacher. The above conclusions are consistent when using CLIP and MAE teachers as illustrated in Tab, We also try much weaker version of MAE teacher which is only pre- pre-trained on ImageNet- 100 for 100 epochs in Tab. 4 We lower the weight of this teacher to avoid its impact on discrimination. The results are still positive, which reflects the power of the MIM pre-training in modeling diversity.
Distilling target the MIM teacher. We then examine the distilling target the MIM teacher. As shown in Tab. 5 distilling the relation R brings the best detection performance (50.0Aphox, Distilling MSA,, achieves a close performance (49.8AP))) 8APbox) since its essential is also distilling rela- tionships, while directly distilling the feature maps T brings the worst performance (49.6A Pbox). Nevertheless, all these schemes outperform the DeiT' distillation baseline, and the trends sistent when using CLIP and MAE teachers, as shown in Tab. 6 Besides, we also evaluate a basic setting that directly distills the features of both the MIM and supervised/CL teachers the last layer. The results are not satisfactory especially when the weaker DeiT teacher is used. The above results highlight the effectiveness of the designs in Hybrid Distill.
Distilling position of the MIM teacher. Tab. 7 inspect the distilling position of the MIM teacher. We first experiment with distilling MAE relations at the front, middle, and back layers. Distilling at the back layers achieves better results, i.e., 1.5 p and 2. 2.4Apbox gains towards distilling at the front and middle, respectively. The results are consistent with the fact that attention collapse tends to occur in these back layers. We then ablate the number of distilling layers and find that distilling at the two layers preceding the final layer (i.e., 10, contributes to the best results.
Token masking strategy. Tab. studies different masking strategies for the student model. Since progressive drop the redundant tokens three times, the actual tokens used in the student model
8

---

Published as a conference paper at ICLR 2024
Table 5: The distilling targets of TT (a)). Ta(x): Table 6: The distilling targets of T. (1)) Ta(æ): DeiT, Tm (z): MAE. * means distilling MAE and DeiT features at the last layer.
CLIP. (x); MAE.* means distilling MAE and CLIP features at the last layer.
Targets
Apbux
A Apmaakk
Targets
Apbox
A Apimaak
47.7
42.1
T
49.7
43.7
m
T
49.6
43.5
T
49.9
44.0
MSA
49.8
43.7
MSA
50.1
44.0
R
50.0
43.9
R 50.4
44.1
Table 7: The distilling position of Tm
Table 8: The token masking strategy.
Distilling layers
Apbox
A Ammak
Strategy
Ratio
Apbox
Apmask
1-11
48.8
43.0
No 100%
50.0
43,9
1,2,3
47.4
41.9
Random
35%
49.2
43.3
5,6,7
48.3
42.7
Direct
35%
49.6
43.7
9,10,11
49.8
43.7
Progressive 13%((0%%*
48.4
42.8
10.11
50.0
43.9
Progressive 34%(70%*)
49.9
43.8
11
49.2
43.3
Progressive 73%(990%))
49.9
43.8
are (1 8)% We observe that when dropping 30% tokens at a time, Hybrid Distill achieves very close performance (49.9A pbox and 43.8Apmmaet to the no masking results and outperforms the random masking strategy and the direct masking strategy which only generates token mask at the last layer. In addition, we notice that our token masking strategy also has a regularizing effect, which can prevent the model from falling into a locally optimal when training for longer epochs. Details about this effect are included in the appendix.
5 RELATED WORK
Model pre-training. Contrastive learning (CL) (Chen et al. 2020a; He et al. 2020, Chen et al.
2020b Grill al.) 2020) and masked image modeling (MIM) (Bao al. 2022: Xie al. 2022b He et al. 2022) dominate the recent pre-trate tre-training research. The former achieved by pulling close the features of two different augment views. While the latter, inspired by masked language modeling
(Kenton & Tooutanova 2019 Zhang et al., (209)), is realized by reconstructing the mask part of the
input. Recently multi- extensions (Radford et al. 2222 Cui et al. 2022 Li et al. 2022) of the
CL pre-training have also been proposed by utilizing the image-text pairs. These different types of pre-training frameworks are proven to have different properties (Park et al. 2023) Xie et al. (2022)), both reconstruction loss and contrastive loss to combine their respective ad advantages. Different from them, we resort to distillation and exploit off-the-shelf teachers achieve better representations.
and HudnH al. (2022) Zhou al. (2022); Diang et al. (2023) further try
Knowledge distillation. Knowledge distillation (Park et al.) 2019: Tian et al.) 2019 Romero et al.) 2014) utilizes well-trained teacher to guide the feature learning of the student model, thus transfer- ring its ability to the student. Some recent works (Wei et al. [202bb Fang et al. [0222] Wang et al.) 2021 Wei et al. 2022a; Peng et al. 2022) propose to utilize it to extend existing pretrained models paradigms. Feature distillation (FD) (Wei et al. 2022b) finds that distilling the feature map of the supervised/CL. pretrained teacher can bring diverse representation to the student and make it more friendly for downstream fine-tuning dBOT (Liu et al. 2022). MVP (Wei et al. 2022a). and BETT v2 (Peng et al.) (022) change the mask reconstruction o object of MIM to the ledgledge of the teacher model boost MIM pre-training with semantic information. MILAN (Hou et al. (022) introduces a prompting decoder design to improve the distillation paradigm. In this paper, we analyze their properties and propose new hybrid distillation framework deal with their deficiencies.
CONCLUSION
This paper proposed a hybrid distillation framework that simultaneously distills knowledge from the supervised/CI. and MIM pre-trained teacher. The framework addresses the limitations of single- teacher distillation, where diversity and discrimination can not be ensured simultaneously. Specifi- cally, Hybrid Distill carefully designs the distilling target and location, i.e., distilling relations from MIM in layers where attention collapse tends to occur and distilling features from supervised/CL in the last layer preserve discrimination. A progressive redundant token masking strategy is also proposed for reducing the distilling costs. Experiments prove that Hybrid Distill can acquire better properties and achieve promising results various downstream.

---

Published as a conference paper at ICLR 2024
Acknowledgment This work was supported in part by the National Natural Science Foundation of China under Grant 62125109, Grant 62250055, Grant 61931023, Grant 61932022, Grant 02371288, Grant 62320106003, Grant 62301299, Grant T2122024, Grant 62120106007.
REFERENCES
Hangbo Bao. Dong, Songhao Piao, and Furu Wei. BEit: BERT pre-training of image trans-
formers. In International Conference Learning Representations, 2022. URL https:
//openreview. wwt.
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. Proceedings of the International Conference on Computer Vision (ICCV), 2021.
Ting Chen, Simon Korbblihh, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. arXiv preprint arXiv:2002.05709 2020a.
Xiaokang Chen, Mingyu Ding, Xiaodi Wang, Ying Xin, Shentong Mo. Yunhao Wang, Shumin Han, Ping Luo, Gang Zeng, and Jingdong Wang. Context autoencoder for self-supervised representa- tion learning. arXiv preprint arXiv:2202.0300 2022a.
Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He. Improved baselines with momentum contrastive learning. arXiv preprint arXiv:2003.04297 2020b.
Xinlei Chen, Saining Xie, and Kaiming He. An empirical study of training self-supervised vision transformers. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pp- 9640-9649, 2021.
Yabo Chen, Yuchen Liu, Dongsheng Jiang, Xiaopeng Zhang, Wenrui Dai, Hongkai Xiong, and Qi Tian. Sdae: Self-distillated masked autoencoder. In ECCV, 2022b.
Yufeng Cui, Lichen Zhao, Feng Liang, Yangguang Li, and Jing Shao. Democratizing contrastive language-image pre-training: A clip benchmark of data, model, and supervision, 2022.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition scale. arXiv preprint arXiv:2010.119299 2020.
Yuxin Fang, Wen Wang. Binhui Xie, Quan Sun, Ledell Wu, Xinggang Wang. Tiejun Huang, Xinlong Wang, and Yue Cao. Eva: Exploring the limits of masked visual representation learning at scale. arXiv preprint ar 211.v2221..06666 2022.
dean-Bastinn Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Daniel Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent: A new approach to self-supervised learning. arXiv preprint arXiv:2006.07733 2020.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference computer vision and pattern recognition (CVPR), pp. 770-778, 2016.
Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cnn. In ICCV, pp. 2961- 2969, 2017.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 9729-9738, 2020.
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked au- toencoders are scalable vision learners. P Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 16000-16009, 2022.
10

---

Published as a conference paper at ICLR 2024
Zejiang Hou, Fei Sun, Yen- Ken-Kuang Chen, Yuan Xie, and Sun- Yuan Kung. Milan: Masked image pretraining on language assisted representation. arXiv preprint Sttv:2008.060499 2022.
Zhicheng Huang, Xiaojie Jin, Chengze Lu, Qibin Hou, Ming-Ming Cheng, Dongmei Fu, Xiaohui Shen, and Jiashi Feng. Contrastive masked autoencoders are stronger vision learners. arXiv preprint (tt.:2207.113332 2022.
Ziyu Jiang, Yinpeng Chen, Mengchen Liu, Dongdong Chen, Xiyang Dai, Lu Yuan, Zicheng Liu, and Zhangyang Wang. Layer grafted pre-training: Bridging contrastive learning and masked image modeling for label-efficient representations. arXiv preprint arXiv:2302. 14138, 2023.
Jacob Devlin Ming-We Wei Chang Kenton and Lee Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In NAACL-HLLT, pp. 4171-4186, 2019.
Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations for fine-grained categorization. In Proceedings of the IEEE international conference on computer vision work- shops, pp. 554-561, 2013.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Jin Li, Yaoming Wang, XIAOPENG ZHANG, Yabo Chen, Dongsheng Jiang, Wenrui Dai, Chenglin Li, Hongkai Xiong, and Qi Tian. Progressively compressed auto-encoder for self-supervised representation learning. In The Eleventh International Conference on Learning Representations, 2023. URL /ttps://openzevile
Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-image pre-
training for unified lingon-language understanding and generation. In ICML, 2022.
Tsung-Y Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 1209-1218, 2014.
Xingbin Liu, Jinghao Zhou, Tao Kong, Xianming Lin, and Rongrong Ji. Exploring target represen- tations for masked autoencoders. arXiv preprint arXiv:2209.08 03917,, 2022.
Ilya Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts. arXiv preprint /r0iv:/608.0390 2016.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint
arXiv:1711.05000 2017.
Namuk Park, Wonjae Kim, Byeongho Heo, Tackyung Kim, and Sangdoo Yun. What do self- supervised vision transformers learn? arXiv preprint 23X5v:2305.00729, 2023.
Wonpyo Park, Dongju Kim, Yan Lu, and Minsu Cho. Relational knowledge distillation. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3967-3976, 2019.
Zhiliang Peng, Li Dong, Hangbo Bao, Qixiang Ye, and Furu Wei. BETT v2: Masked image modeling with vector-quantized visual tokenizers. 2022.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. International Conference on Machine Learning (ICML), 2021.
Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang, Carlo Gatta, and Yoshua Bengio. Fitnets: Hints for thin deep nets. arXiv preprint arXiv:1412.6550, 2014.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang. Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. International journal of computer vision (IJCV), 115(3) 2015.
11

---

Published as a conference paper at ICLR 2024
Alexander Strehl and Joydeep Ghosh. Cluster ensembles- knowledge reuse framework for com- bining multiple partitions. Journal of machine learning research, 3(Dec):583 617, 617,2001 2002.
Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive representation distillation. arXiv
preprint arXiv:1910. 19699 2019.
Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and
Herve Jegou. Training data-efficient image transformers & distillation through attention. In In- ternational Conference on Machine Learning (ICML), volume 139, pp. 10347-10357, July 2021.
Grant Van Horn, Oisin Mac Aodha, Yang Song, Yin Cui, Chen Sun, Alex Shepard, Hartwig Adam, Pictro Perona, and Serge Belongie. The inaturalist species classification and detection dataset. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 8769-8778, 2018.
Shaoru Wang. Jin Gao, Zeming Li, Xiaoqin Zhang, and Weiming Hu. A closer look at self- supervised lightweight vision transformers, 2023,
Wenguan Wang, Tianfei Zhou, Fisher Yu, Jifeng Dai, Ender Konukoglu, and Luc Van Gool. Ex- ploring cross-image pixel contrast for semantic segmentation. arXiv preprint arXiv:2101. 2021.
Longhui Wei, Lingxi Xie, Wengang Zhou, Houqiang Li, and Qi Tian. Mvp: Multimodality-gun visual pre-training In Computer Vision-ECCO ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27 2022, Proceedings, Part XXX. pp. 337-353. Springer, 2022a.
Yixuan Wei, Han Hu, Zhenda Xie, Zheng Zhang, Yue Cao, Jianmin Bao, Dong Chen, and Baining Guo. Contrastive learning rivals masked image modeling in fine-tuning via feature distillation. Tech Report, 2022b.
Tete Xiao, Yingcheng Liu, Bolei Zhou, Yuning Jiang, and Jian Sun. Unified perceptual parsing for scene understanding. In Proceedings of the European Conference Computer Vision (ECCV), pp. 418-434, 2018.
Zhenda Xie, Zigang Geng, Jingeheng Hu, Zheng Zhang, Han Hu, and Yue Cao. Revealing the dark secrets of masked image modeling. arXiv preprint 2tti::2200.15543 2022a.
Zhenda Xie, Zheng Zhang, Yue Cao, Yutong Lin, Jianmin Bao, Zhuliang Yao, Qi Dai, and Han Hu. Simmim: A simple framework for masked image modeling. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9653-9663, 2022b.
Hongwei Xue, Peng Gao, Hongyang Li, Yu Qiao, Hao Sun, Houqiang Li, and Jiebo Luo. Stare at what you see: Masked image modeling without reconstruction. arXiv preprint urXiv:2211.0888)) 2022.
Zhengyan Zhang. Xu Han, Zhiyuan Liu, Xin Jiang, Maosong Sun, and Qun Liu. Ernie: Enhanced language representation with informative entities. In ACL, pp. 1441-1451, 2019.
Bolei Zhou, Hang Zhao, Xavier Puig, Tete Xiao, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Semantic understanding of scenes through the ade20k dataset. International Journal on Computer Vision (IJC),, 127:302-321, 2019.
Qiang Zhou, Chaohui Yu, Hao Luo, Zhibin Wang, and Hao Li. Mimeo: Masked image modeling pre-training with contrastive teacher. In Proceedings of the 30th ACM International Conference on Multimedia, pp. 4487-4495, 2022.
A MORE EXPERIMENTAL RESULTS
A.1 COMPARED WITH MORE BASELINES
Tab. compares Hybrid Distill with two other methods, iee. dBOT (Liu et al. 2022) and FD (Wei
2022b), which employ asymmetric designs in distillation. We conduct distilling for 300 epochs
12

---

Published as a conference paper at ICLR 2024
Table 9: Compared with more baselines using ViT-B as the backbone. *: using MAE+DeiT teachers. t: using MAE+CLIP teachers.
COCO
Method
ADE20K
A pbux
ApMask
Distill-DeiT
47.7
42.1
47.3
Distill-MAE
49.1
43.1
47.8
Dist
49.5
43.5
50.3
FD-DeiT (Wei et al.
2022b
47.0
41.6
47.9
FD-MAE Wel ct al.
20226
48.1
42.6
47.0
FD-CLIP Wei
2022b
49.2
43.3
50.5
dBOT-DeiT (Luu et al. (022)
47.5
41.9
47.9
dBOT-MAE (Luu et al.
2022)
49.3
43.5
48.2
Hybrid Distill
50.3
44.2
49.1
Hybrid DistilI
50.6
44.4
51.5
Table 10: Object detection and instance segmentation results with Cascade Mask-RCNN. *: using MAE+DeiT teachers. t: using MAE+CLIP teachers.
Method
Epoch
A Apbox
Al Apmask
Distill-DeiT
300
50.4
43.4
Distill-MAR
300
51.9
44.7
Distill-CLIP
300
52.4
45.0
DeiT (Lu et al.
2022)
2 x 800
52.5
dBOT-MAE Liu
2022
2 x 800
52.7
dBOT-CLIP (Liu et al.
2022
1 x 1600
53.6
Hybrid Distill*
300
53.0
45.6
Hybrid Distillt
300
53.4
45.9
based on their corresponding official codes We omit the dBOT-CLIP result since dBOT specifi- cally removes the asymmetric designs for CLIP, thus its distillation process is similar to Distill- CLIP baseline. As shown in Tab. 9 their benefits towards symmetrical distillation (Distill-X) are not always significant, and the performance is inferior to Hybrid Distill, which validates the effectiveness of our framework.
A.2 RESULTS WITH CASCADE MASK-RCNN
Tab. furthher presents the object detection and instance segmentation results of Hybrid Distill with Cascade Mask-RCNN, which allows for direct comparison with dBOT (Liu et al., (022), as they also provide 1600-epoch distillation results under this setting. As shown, 300-epoch Hybrid Distill with MAF and DeiTT teachers can achieve 53.0 Apbox, outperforming 1600-epoch dBOT-DeiT (52.5 A A and dBOT-MAE (52.7 bobo Additionally, 300-epoch Hybrid Distill with MAE and CLIP teachers achieves 53.4 Apbox, which is also very close to the 1600-epoch dBOT-CL.IP result (53.6
Ap The above results reflect that due to the better properties obtained, Hybrid Distill can obtain promising results with fewer distilling epochs.
A.3 LINEAR PROBING RESULTS
Tab. shows the linear probing results on ImageNet-IK. The subpar performance of dBOT further validates our perspective discussed in Sec. 2.3 which suggests that its increase in diversity comes the expense of sacrificing discrimination. In contrast, Hybrid Distill consistently maintains outstand- ing linear probing performance, thus confirming the effectiveness of our approach in simultaneously promoting diversity and discrimination.
4BOT Liu et al. 2022
FD Wei
al.
2022b
https://ithub..om//3 Tansformer/Featuree Der-Faatttrr--iiilatttio/// Distillation. Since FD does not provide codes for down- stream verification, we uniformly perform verification under our downstream frameworks.
13

---

Published as a conference paper at ICLR 2024
Table 11: Results for linear probing on ImageNet-IK, using MAE+DeiT teachers,
using
MAE+CLIP teachers.
Method
Linear Probing
MAE
67.8
dBOT
67.9
Hybrid Distill*
80.9
Hybrid Distill
80.1
Table 12: Hybrid Distill uses MAE and DINO teachers. Object Detection and instance segmen- tation results are reported with MCask following the setting in Tab. our main paper.
Method
Apbox
A Apiask
MAE (He et al. 2022
48.4
42.6
DINO Caron et al. 2021
46.8
41.5
Distill-DING
47.5
41.9
Distill-MAE
49.1
43.1
Hybrid Distill
49.6
43.5
A.4
HYBRID DISTILLATION WITH DINO
Tab. 12 test the results of our Hybrid Distill using the MAE (He et 2022) and DINO (Caron al.
202 teachers. Under this setting, Hybrid Distill achieves 49.6 A Apbxx and 43.5 A A.mmask Although still superior to the baselines, results with DINO are not as good as those with CLIP and DeiT. We analyze that this is because the discrimination of DINO is weaker than DeiT and CLIP, which makes its complementarity with MAE also weaker than the latter two. The visualization in Fig.7 provides evidence for this. On the one hand, we notice that the average attention distance of DINO itself lower than that of DeiT and CLIP in the final layer. On the other, the attention maintenance of the final layer after distillation weaker compared with that obtained by DeiT and CLIP.
A.5 MORE ABLATION STUDIES
The choice of hyperparmeter a and B. Tab. 1 and Tab. 14 ablate different setting of and 9, respectively. It can be concluded that combining two teachers for hybrid distillation can lead to performance gains towards only using one teacher, regardless of the value of hyperparameters a and 3. Besides, we note that ensuring the weights of the more powerful model among the two teacher models e.g., MAE for MAE+DeiT teachers and CLIP for MAE+CLIP teachers) count for better performance, and setting both a and / to I for simplicity can already yield satisfactory results.
Token masking strategy and local optima. Tab. [5 further reveals that the proposed progressive redundant token masking strategy in Hybrid Distill can prevent the student from falling into local optima. As shown, when the token mask is removed and the distillation epoch is prolonged from 100 to 300, no further performance gains are observed. A similar phenomenon has also been observed in Fang et al. (2022). We analyze that over-fitting the root cause of this problem and introducing token masks alleviate it since they can play regulatory role. The performance gains achieved by the token masks provide clear support for their effectiveness.
B FURTHER DISCUSSION ABOUT DIVERSITY AND DISCRIMINATION
B. LINEAR PROBING EVALUATION FOR JUDGING THE DISCRIMINATION
This section further supplements the linear probing results judge the discrimination of the dis- tillation configurations mentioned Sec. 2 The results in Tab. 16 are consistent with the analysis presented in Sec. 2 indicating that the asymmetric decoder negatively impacts the discrimination (Sec. 2.3 and the behavior of mask feature reconstruction is similar to that directly distilling features (Sec. 2.4 These results further corroborate analysis of discrimination.
14

---

Published as a conference paper at ICLR 2024
140
140
Afte sottttion Di00res
050
60
40
000
10
10
31
(a) DINO
(b) Hybrid Distill (DINO +MAE)
Figure 7: Average attention distance of different (a) DINO baseline and (b) Hybrid Distill with MAE and DINO as teachers.
Table 13: Ablation on the hyperparameter (with B 1) which controls the contribution of the supervised/CL teacher.
(a) T.(x): DeiT, Tm (z): MAE
0
0.1
0.3
0.5
0.7
1.0
Apbox
48.9
49.2
49.7
49.9
50.0
50.0
A A pmaak
43.1
43.5
43.8
43.9
43.9
43.9
(b) (z): CLIP, Tm (m):):: MAE.
0
0.1
0.3
0.5
0.7
1.0
Apbox box
48.9
49.3
49.6
49.9
50.1
50.4
AP Ap Apmask
43.1
43.5
43.7
43.9
43.9
44.1
B.2 ASYMMETRIC ENCODER DESIGNS
Fig.8 Estuuies the asymmetric encoder designs used in FD (Wei et al. (022b), i.e., adding additional learnable parameters and relative position bias to the attention layers the student. As shown, the asymmetric encoder (Fig. ())) can improve diversity compared to using only the symmetric encoder (Fig. b)). However, compared to the DeiT teacher (Fig, a)), does not bring noticcable be found in Wei al. (022b) (Figure 2 in their paper), and its diversity improvement is also not notable when using DeiT' and CLIP teachers. Therefore, conclude that the diversity brought by the asymmetric encoder is not always significant.
diversity gains. A similar visualization
B.3 REDUCING THE NUMBER OF THE ASYMMETRIC DECODER LAYERS
Fig. g investigates the effect of reducing the number of asymmetric decoder layers. We find that even with a reduced number of decoder layers, the discrimination in the last layer of the encoder still cannot be maintained. Therefore, we abandon this asymmetric decoder design in our Hybrid Distill avoid losing discrimination.
B.4 MASK FEATURE RECONSTRUCTION WITH ASYMMETRIC DECODER
Fig. 10 compares two variants of dBOT, i.e., with the same asymmetric decoder design in dBOT but conducting direct feature distillation and mask feature reconstruction, respectively. It can be seen that the two tasks bring significant differences, i.e., the diversity is increased and the discrimi- nation lost regardless the task. These visualizations further support our claim in Sec. 2.3 and Sec. 2,4 of our main paper.
15

---

Published as a conference paper at ICLR 2024
Table 14: Ablation on the hyperparameter 8 (with a 1) which controls the contribution of the MIM teacher.
(a) Tc(r): DeiT, Tm (a)): MAE.
B
0
0.1
0.3
0.5
0.7
1.0
AP box
Apbox
47.5
48.2
49.3
49.3
49.5
50.0
AP Amask
41.8
42.6
43.4
43.4
43.5
43.9
(b) Tz(x): CLIP, (x): MAE
B
0
0.1
0.3
0.5
0.7
1.0
AP bbx
49.1
49.9
49.8
50.1
50.2
50.4
A p piaak
43.1
43.8
43.8
43.9
44.1
44.1
Table 15: The token masking strategy for alleviating over-fitting. *; using MAE+DeiT teachers. f:
using MAE+CLIP teachers.
Method
Epoch
Masking
A.
A A.mask
Hybrid Distill"
100/300
50.0/50.0
43.9/44.0
Hybrid Distill*
100/300
49.9/50.3
43.8/44.2
Hybrid Distilli
100/300
50.4/50.3
44.1/44.1
Hybrid Distilit
100/300
50.2/50.6
43.9/44.4
Table 16: Results for linear probing on ImageNet-IK K when using DeiT teachers.
Distillation configuration
Linear probing
No decoder (Fig. (a))
81.4
Linear projection (Fig. 3 (b)
81.7
Asymmetric decoder (c)
66.2
Mask feature reconstruction (Fig. (d))
79.5
C IMPLEMENTATION DETAILS FOR DIFFERENT DOWNSTREAM TASKS
Classification. We report the fine-tuning results on ImageNet- Following dBOT (Liu et al. 2022). the learning rate is set to 3e-4 and the batch size is set to 256. We also report results
CIFAR100 (Krizhevsky et al. 2009), Cars (Krause et al., 2013). and iNaturalist19 (Van Horn et al.) For these datasets, the batch size is 768 and the learning rate is 7.5e-6.
2018
Object detection and instance segmentation. Following Chen et al. (2022a), we fine-tune the student model on COCO (Lin et al. 2014) using the Mask-RCNN (He al. 2017) framework. We train the network with the Ix schedule and the learning rate is set to 3e-4 for ViT-B and 2c-4 for ViT-L. We also provide the Ix results using the Cascade Mask-RCNN framework in the appendix, and the learning rate is set to 3e-4.
Semantic segmentation. The semantic segmentation evaluation is conducted ADE20K (Zhou ct al. 2019). Following Chen et al. (2022a b). we use ViT (Dosovitskiy et al., 2020 with UperNet (iiao et al. 2018) framework and fine-tune the model for 160k iterations. The batch size, learning rate, and weight decay are set to 16, 4e-4, and 0.05, respectively.
D DISCUSSION WITH OTHER DISTILLATION METHODS
Compared to previous distillation methods (Wei et al. 2022b Fang al. 2022 Liu et al. 2022
Wei et al. 2022a Peng et al. 2022), Hybrid Distill stands out by not being restricted to using a single teacher network. In addition to addressing the limitations of single-teacher distillation in enriching both diversity and discrimination (as discussed Sec. 2 a more direct factor is that single-teacher distillation cannot create knowledge, egg,, creating additional discrimination for
16

---

Published as a conference paper at ICLR 2024
Apdte Dianee
Symmeliic
Figure 8: Average attention distance of (a) DeiT teacher and student models with (b) symmetric encoder and (c) asymmetric encoder.
300
layer
(N) 4 lagers
Bayaers
Figure 9: Average attention distance using (a) 1, (b) 4, and (c) asymmetric decoder layers, respectively.
140
140
Artertion (160100
Aecertion
40
40-
10
11
12
Attontion
30 21
(a) Direct distilling
Mask feature reconstruction
Figure 10: Average attention distance of different dBOT variants that conduct (a) direct feature distillation and (b) mask feature reconstruction, respectively.
Table 17: One-epoch training time and memory usage. Tested 8 V100 with 128 per-GPU batch size.
Method
One-epoch training time GPU memory
MAE (He et al. 2022)
9min
17807M
dBOT (Liu al. 2022)
12min42s
18953M
Hybrid Distill
11min12
9503M
student model when using the MIM teacher. Therefore, we believe that combining and utilizing existing knowledge from various teachers is more effective and convenient. Furthermore, with the growing availability large-scale pre-trained models within the community, such as hugging face, becomes increasingly valuable to explore ways utilize these models and combine their strengths. This further enhances the practical value of our Hybrid Distill, and we hope our work would shed light new directions.
17

---

Published as a conference paper at ICLR 2024
E
LIMITATION
Hybrid Distill jointly utilizes two teacher models to guide the representation learning the student. Although exhibiting promising properties and results, the additional overhead of introducing two teachers may be limitation. Fortunately, since the teacher model does not require gradient updates, the training cost of Hybrid Distill does not increase significantly, i.e., the training time of Hybrid Distill with VTT-B backbone is around 1.2 times longer than that of using single teacher. However, when compared to dBOT, which replaces the MAE reconstruction objective with high-level features from the teacher, Hybrid Distill enjoys shorter one-epoch training time and lower memory usage as shown in Tab. T7 This can attributed to fact that Hybrid Distill does not employ the resource-intensive asymmetric decoder (with 8 transformer layers) in MAE and dBOT, which, in turn, reduces the overall computational cost. Besides, when directly utilizing the available teacher models, Hybrid Distill can achieve better performance with much fewer distilling epochs, shown in Tab. IO, Based on the above discussions, we believe that the computational cost of Hybrid Distill acceptable. Another possible limitation is that Hybrid Distill does not improve CLIP much as DeiT after introducing the MAE teacher, and we analyze that it may be caused by the gap between the pre-training capacities of CLIP and MAE teachers. We look forward better MIM models that can further facilitate our work.
18


Reviews:
[{"review_id": "FfxeyjpqR3", "reviewer": "Reviewer_mHak", "summary": "After discussing the \"diversity\" and \"discrimination\" in the features of a distilled Transformer, this paper proposed a hybrid distillation framework that simultaneously distills knowledge from two distinctively different pre-trained teacher. Specifically, HybridDistill designs the distilling target and location from a pre-trained MIM model and a supervised/CL model. A progressive redundant token masking strategy is also proposed for reducing the distilling costs.", "questions": "Discussions/addressing the questions in **bold** in Weakness will be very helpful. Thanks!", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- The paper provides many results and discussion on comparing the pre-head attention distance with different types of teacher model, which could be a helpful resource for the community.\n- The paper provides through ablation studies on each component of the proposed method.", "weaknesses": "- My biggest concerns are on the validity of the motivation to encourage both \"diversity\" and \"discrimination\" in the Transformer.\n  - About \"diversity\":\n    - This submission claims that \"diversity\" ```means that the model pays attention to both local and global information and can achieve more evenly distributed representations```. What is ```evenly distributed representations'''? Specifically, which does it mean by \"distributed representation\"?\n    - My understanding is that the \"diversity\" is quantitively measured by both average attention distance and NMI. I am not sure if \"diversity\" is a good terminology in this case, which oversimplified many necessary context. I suggest, for example, \"different per-head attention distance\", to more accurately describe the discussed property. For simplicity, I will still use \"diversity\" in my review.\n    - I am **especially** not fully convinced that “diversity” itself is something clearly worth to pursue in a pre-trained Transformer. Yes, \"diversity\" is evaluated in several previous works, but often as a side information/evidence to better understand the self-attentions. However, it is not convincing that “diversity” in the last layers is something to specially encouraging. For a starter, what *should* be *ideal* per-head attention distance in each layer already requires serious discussion. It is hard to believe that focusing on long-term/global information in the last layers is unquestionably bad. If the authors argue that \"diversity\" in last layers may help dense-level tasks, while, then why not simply make better usage of multi-layer features in these dense tasks instead of forcing last layer features to be \"diverse\"?\n  - About \"discrimination\":\n    - This is a question: what metric is used to quantitively evaluate \"discrimination\" in the experiments?\n \n- Questions on results:\n  - Why MAE's COCO results in the main Table (Table 1) are **lower** than the results reported in MAE's original paper? For example, in MAE, a ViT-B is reported to get 50.3 APbox and 44.9 APmask, but 48.4 APbox and 42.6 APmask in this submission? Please highlights the experimental setting differences to MAE and other compared baselines in the paper if this is the reason. However, if the setting difference is the reason for performance difference on COCO, then why ADE20K results are matched?\n  - If directly distilling CLIP's feature harms dense tasks for its inadequate \"diversity\" in self-attention distance, then why results reported in MILAN outperforms the proposed method on COCO detection/instance segmentation and ADE20K semantic segmentation? I think MILAN directly predictions CLIP's features. Also, MILAN is not cited and compared in the submission.\n\n[MILAN] Hou et al, MILAN: Masked Image Pretraining on Language Assisted Representation.\n\n\n-----\n\n\nPost-Rebuttal Comment: I thank the authors for the response, which partially addressed my concerns on the naming/motivation of \"diversity\" vs. discrimination, and the results on COCO compared to previous methods. I adjusted my recommendation accordingly.\n\nBut I still want to highlight: Although previous works [3-5] adopting shorter schedules seem common, I believe that fine-tuning for a long schedule to see the real performance given enough training is very important to illustrate the effectiveness of the proposed methods. Sometimes the pre-trained models only show effectiveness with shorter schedule, and the gains diminish after longer training. Longer schedule provides a lens for us to better understand the proposed method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "3mKSkwBlD4", "reviewer": "Reviewer_zDe2", "summary": "This paper proposes a distillation or learning method for pretraining visual representation. Motivated from the differences in MAE, DeiT, CLIP models, in terms of feature diversity and attention distance, this paper proposes to distill differently for the two targets, specifically learn features from DeiT/CLIP models and attention maps from MAE methods. Results show some improvements over baselines on classification, detection, segmentation.", "questions": "(see weakness for details)\n------------------------------\nAfter reading all the reviews and discussions, I feel that this paper is still around borderline, and it's ok to accept it. Empirically, it's unfortunately not very useful, or to say, it's only useful in very extreme cases (when there are MAE and DEiT models, but no dBOT model, and without considering the complexity of coding, then it can be trained a bit faster). However, if it is to be accepted, these extensive studies and observations may still be interesting to some readers, e.g. Reviewer ZCMG.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. The motivation is presented quite clearly given existing studies on average attention distance and NMI.\n2. The ablations are extensive and convincing that HybridDistill helps compared with vanilla MAE or DeiT.", "weaknesses": "1. Unfair comparison. Table 1 compares to many vanilla training methods on ImageNet-1K but Hybrid Distill exploits CLIP targets and ImageNet-21K for training. Other methods exploiting the joint objectives, better models, or larger scale data are not compared.\n2. The two metrics of average attention distance and NMI are existing metrics. They provide insights to models but are still neither necessary or sufficient condition for a good representation. Existing works such as dBOT also visualized the average attention distance differences in existing methods (DeiT, DINO, MAE, etc.) and even different distillation stages and observed similar conclusion as well. Existing joint learning work with the distillation objective is producing good representation already.\n3. HybridDistill is complicated and designed specifically to ViT architecture with global attention map. Therefore, it looks challenging to apply to other architectures such as Swin or MViT or CNNs.\n4. Marginal performance gain compared with existing solutions in fair comparison. E.g. SdAE in Table 1 achieves 84.1, iBOT achieves 84.0, etc. with ViT-B without CLIP or ImageNet-21K, but Hybrid Distill achieving 83.7 lags behind. Similarly limited gain is observed with CLIP targets, e.g. dBOT 87.8% vs. this paper 87.6% with ViT-L ImageNet-1K CLIP targets, and Cascaded detection settings and ADE20K segmentation.\n5. Runtime compared with baselines. Since Hybrid Distill requires inferencing with multiple backbones and uses less aggressive masking strategy, what is the runtime of one epoch compared with vanilla MAE?\n6. The claim of using fewer epochs isn’t quite important here considering the computation used in training all the targets (including CLIP).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "oGs667Bk0h", "reviewer": "Reviewer_ZCMG", "summary": "The paper studies the feature spaces and models from two different pretraining paradigms, contrastive learning  (CL) and masked image modelling (MIM). It reaffirms differences around discrimination of the produced features as well as the attention diversity and makes a number of interesting observations. Based on those, the paper proposes Hybrid Distill, a hybrid distillation method that tries to get the best of both CL and MIM. An MIM teacher is used at intermediate layers and a CL teacher at the output and this leads to models with both token diversity as well as features with strong discrimination.", "questions": "- What do the authors mean by \"s. For ViT-L, the distillation is based on both ImageNet-1K and ImageNet-21K\"? It is for different models or for the same model?\n\n- \"The hyperparameter α and β are set to 1.0\" - how is performance affected if the two teacher losses are not balanced like this?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* The paper studies very interesting properties and tries to understand the difference in behaviour between common learning paradigms like CL and MIM.\n\n* The study of sec 2 is sound and with interesting observations.\n\n* The method presented in Sec 3 is to my knowledge technically novel. \n\n* The experimental validation seems complete and shows some gains for the proposed approach on the common setting.", "weaknesses": "1. Sec 2 does not discuss performance, ie the \"discrimination\" aspect. It is unclear if this is because the two paradigms work equally well on downstream tasks? Probably thats not exactly the case, and generalization is in the end what we care about, ie what we use these features for. Also, wrt 2.2, studies (eg Wang et al and Sariyildiz et al below as well as the papers that introduce such projectors) have shown that the size of projector/\"asymmetric decoder\" matters wrt to downstream task behaviour.\n\n\n2. The method proposed in Sec 3 tries to get the best of both worlds by using two teachers, something that makes training harder and requires more memory.\n\n3. More of a note than a weakness:  It is now hard to compare the relevant rows in Tables 1 and 2. Tab 1 would be more easy to understand and compare if the two sections are wrt the two different backbones (ie ViT-B on top, ViT-L on the bottom). Inside each section, a separate split should be wrt what is the teacher. The models that distill clip should be clearly separated, cause they start from a much stronger teacher. Same for the ViT-L from the authors, that distills im21k.  These results are not comparable to the rest, and the im21k-distilled row seems out of place in both tables. Please make sure to separte accordingly.\n\nWang, Yizhou, et al. \"Revisiting the transferability of supervised pretraining: an mlp perspective.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.\n\nSariyildiz, Mert Bulent, et al. \"No reason for no supervision: Improved generalization in supervised models.\" ICLR 2023-International Conference on Learning Representations. 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "w87YUBZIH9", "reviewer": "Reviewer_4NJ5", "summary": "The paper proposes hybrid distillation, which uses a supervised/CL teacher and a MIM teacher for a balance between discrimination and diversity. The authors use average attention distance and NMI metric to analyze the disadvantages of previous methods with respect to the above two properties, and then design hybrid distillation according to the disadvantages. Hybrid distillation achieves strong performances on various downstream tasks (classification/detection/segmentation) using ViT-B/L models.", "questions": "Please refer to weakness.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The proposed hybrid distillation is simple and reasonable according to the discrimination and diversity.\n2. Better performances on various downstream tasks.\n3. Good writing and abundant visualization.", "weaknesses": "1. It seems that using two teachers for distillation really boosts the performances on downstream tasks. Will it take a longer pre-training time compared to only using CLIP as the teacher? I hope the authors can show the training time and the memory usage during distillation.\n2. Previous CL-based methods outperform MAE on linear probing because they own strong discrimination abilities. Since the hybrid distillation keeps this ability, can authors show the linear probing performances on ImageNet-1k?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)