Title: Hybrid Distillation: Connecting Masked Autoencoders with Contrastive Learners
Venue: NeurIPS 2023
Accepted: no
Keywords: contrastive learning

mask image moding

feature distillation
Abstract: Representation learning has been evolving from traditional supervised training to Contrastive Learning (CL) and Masked Image Modeling (MIM). Previous works have demonstrated their pros and cons in specific scenarios, i.e., CL and supervised pre-training excel at capturing longer-range global patterns and enabling better feature discrimination, while MIM can introduce more local and diverse attention across all transformer layers. In this paper, we explore how to obtain a model that combines their strengths. We start by examining previous feature distillation and mask feature reconstruction methods and identify their limitations. We find that their increasing diversity mainly derives from the asymmetric designs, but these designs may in turn compromise the discrimination ability. In order to better obtain both discrimination and diversity, we propose a simple but effective Hybrid Distillation strategy, which utilizes both the supervised/CL teacher and the MIM teacher to jointly guide the student model. Hybrid Distill imitates the token relations of the MIM teacher to alleviate attention collapse, as well as distills the feature maps of the supervised/CL teacher to enable discrimination. Furthermore, a progressive redundant token masking strategy is also utilized to reduce the distilling costs and avoid falling into local optima. Experiment results prove that Hybrid Distill can achieve superior performance on different benchmarks.

Abstract

Representation learning has been evolving from traditional supervised training to Contrastive Learning (CL) and Masked Image Modeling (MIM). Previous works have demonstrated their pros and cons in specific scenarios, i.e. , CL and supervised pre-training excel at capturing longer-range global patterns and enabling better feature discrimination, while MIM can introduce more local and diverse attention across all transformer layers. In this paper, we explore how to obtain a model that combines their strengths. We start by examining previous feature distillation and mask feature reconstruction methods and identify their limitations. We find that their increasing diversity mainly derives from the asymmetric designs, but these designs may in turn compromise the discrimination ability. In order to better obtain both discrimination and diversity, we propose a simple but effective Hybrid Distillation strategy, which utilizes both the supervised/CL teacher and the MIM teacher to jointly guide the student model. Hybrid Distill imitates the token relations of the MIM teacher to alleviate attention collapse, as well as distills the feature maps of the supervised/CL teacher to enable discrimination. Furthermore, a progressive redundant token masking strategy is also utilized to reduce the distilling costs and avoid falling into local optima. Experiment results prove that Hybrid Distill can achieve superior performance on different benchmarks.

1 Introduction

Pre-training followed by fine-tuning has been a common paradigm for computer vision tasks since 20 the advent of deep learning. In the past decade, supervised image classification [16, 10, 24] over the 21 widely used ImageNet [32] has dominated the pretraining mode. Recently, self-supervised learning 22 has emerged as a promising alternative, particularly with two approaches: Contrastive Learning (CL) 23 and Masked Image Modeling (MIM). The former one, typical representatives are MoCo [14] and 24 SimCLR [4], learns invariant representation for positive views, which are usually defined as different 25 augmentations of the same image. Furthermore, CLIP [30] extends CL to a multi-modal manner, 26 which utilizes the corresponding text description of the given image as positive pairs. While the 27 latter, including MAE [13] and SimMIM [44], aims to reconstruct the masked image patches and has 28 become mainstream due to its efficiency brought by mask operations. 29

30

The different pre-training paradigms of CL and MIM facilitate a series of studies [43, 27, 38] that

31

32

aim at understanding their respective properties. These studies point out that CL pre-training behaves more similar to supervised pre-training,

i.e.

, it provides models with longer-range global patterns

targeting object shape, particularly in the last few layers [27], and enables feature representation with 33

better discrimination . However, as shown in Fig. 1(a), CL pre-training causes self-attention in the 34

last few layers to collapse into homogeneity, with attention distances located within a very small 35

distance range. In contrast, MIM pre-training can bring more diverse attention and evenly distributed 36

representations to all layers [43, 27], and this diversity contributes to its better generalization on 37

downstream fine-tuning. Nevertheless, MIM pre-training is slower to converge and underperforms in 38 linear probing, mainly due to its lack of discrimination ability. 39

Since discrimination and diversity are both crucial for downstream adaptation, previous methods 40 [41, 11, 23, 40, 29] propose to utilize feature distillation to combine the benefits of CL and MIM. 41 Among them, dBOT [23] replaces the reconstructing objective of MAE with the feature maps of 42 different pre-trained teachers. It finds that feature distillation can bring diverse attention no matter 43 what the teacher model is, and the performance is comparable across different teachers, even with 44 the randomly initialized ones, after multi-stage distillation. Also observing that distillation can yield 45 diversity benefits, FD [41] directly distills feature maps from supervised/CL teachers to relieve the 46 attention collapse and achieves considerable downstream performance gains. Although interesting 47 and important, we argue that their findings are incomplete. 48

49

50

51

52

53

54

55

56

57

58

59

60

61

This paper re-examines these findings and reconsiders the importance of diversity and discrimination. Our study reveals the following observations: (i) The increase in diversity derives from the asymmetric architecture designs, rather than feature distillation itself. (Section 2.2) After removing the asymmetric attention in [41] and encoder-decoder designs in [23] and keeping the same teacher and student structures, we observe a negligible increase (or even a decrease) in attention diversity. (ii) The asymmetric decoder de facto harm the discrimination over the encoder side, for it migrates the semantic information of the teacher model. (Section 2.3) Due to the decomposition of the encoding and decoding functions, student encoders tend to summarize more general information, thus gradually losing the semantics obtained from teachers and yielding similar results after multi-stage distillation [23]. (iii) Mask reconstruction of high-level semantics does not help improve diversity. (Section 2.4) The phenomenon of reconstructing high-level information [29, 11, 40] is similar to direct feature distillation and lacks the diversity found in MIM, which implies that the attention diversity of MIM mainly comes from low-level reconstruction objectives.

62

63

64

65

66

67

68

69

70

71

72

73

74

75

76

77

78

79

80

81

82

83

84

85

86

87

88

Based on the above observations, we argue that a better distillation strategy is needed to help student models inherit both diversity and discrimination. To this end, we propose a simple but effective feature distillation method, termed as Hybrid Distill , to fully exploit the pre-trained model. Unlike previous works, Hybrid Distill aims to distill knowledge from both the supervised/CL and MIM teacher, allowing the student model to benefit from their respective advantages. To realize this, Hybrid Distill makes careful designs for the distilling target and location. Specifically, we find that the relational modeling ability of MIM is crucial for preserving token diversity, while the feature maps of supervised/CL teachers are beneficial for discrimination . Accordingly, we set the token relations of the MIM teacher and the feature maps of the supervised/CL teacher as the distilling objectives of Hybrid Distill. The token relations are distilled in layers preceding the final layer where attention collapse tends to occur, while the feature maps are distilled in the final layer to preserve semantics. Additionally, Hybrid Distill proposes a progressive redundant token masking strategy to reduce distilling costs and prevent falling into local optima. Experiment results show that the above distilling strategy works surprisingly well even when using MAE and CLIP teachers, i.e. , MAE pretrained with only 1.28M ImageNet images can also boost the large-scale (400M) pretrained CLIP teacher on different downstream tasks.

In a nutshell, this paper makes the following distribution:

2 Model Evaluation: Diversity and Discrimination

91

(b) DeiT distillation. From left to right are: no decoder, linear projection and asymmetric decoder.

140

140

140

120

120

120

100

100

100

AttentionDistance

08

AttentionDistance

80

Attention Distance

80

ee-

ee

00

09

60

60

福

e

40

40

40

Attention Head

56

8

1011

Attention Head

Attention Head

56

7

8

9101112

140

140

140

Attention Distance

120

100

80

。

Attention Distance

120

100

80

！

8

Attention Distance

120

100

80

eee-

U

60

40

20

8

60

40

20

8

!

福

60

40

20

8

Attention Head

5

6

8

11

Attention Head

8

10

11

Attention Head

6

8

140

140

140

120

120

120

Attention Distance

100

0g

80

40

ee

8

Attention Distance

100

60

40

80

ceoe

8

e

福

Attention Distance

100

60

40

B01

ee

00

eee

20

20

20

00

Dt

Attention Head

8

1011

12

2

Attention Head

5

6

7

8

9101112

6

8

101112

Attention Head

2.1 Preliminary

We first introduce the definitions of diversity and discrimination and the evaluation strategies we used. 92 Discrimination means that the representations contain more global patterns tailored to object shapes, 93 which is beneficial for recognizing objects and distinguishing images. Diversity is a relative concept, 94 which means that the model pays more attention to local information and can achieve more evenly 95 distributed representations, particularly in the last few layers. 96

97

98

99

100

101

102

103

104

Wemeasure these properties by average head distance [41, 10] and normalized mutual information (NMI) [33]. The former calculates the average distance between the query tokens and the key tokens based on their attention weights, providing insight into whether the attention is global or local. The latter measures whether the attention is attending to different tokens or similar ones and is calculated following [27]. Specifically, let a uniform distribution p ( q ) = 1 N represent the distribution of query tokens, where N is the total token number. The joint distribution of query and key is then computed as p ( q, k ) = π ( k | q ) p ( q ) , where π ( k | q ) is the normalized self-attention matrix. Thus, NMI can be calculated by I ( q,k ) √ H ( q ) H ( k ) where I ( · , · ) is the mutual information and H ( · ) is the marginal entropy.

105

2.2 The Increase in Diversity Derives from the Asymmetric Designs

Fig. 1 measures the average head distance after feature distillation with a consistent encoder structure 106 (vanilla Vision Transformer (ViT) [10]) for both the teacher and student models, along with various 107 decoders only for the student. It can be seen that when the encoder is kept the same, using no decoder 108 or linear projection decoder leads to a negligible increase (or even decrease) in attention diversity, 109 reflecting that feature distilling itself cannot bring benefits to diversity. Adding some extra attention 110 layers to the decoder can make the student encoder more diverse, but it hinders discrimination since 111 the last layer no longer captures long-range patterns. Fig. 2(a) further compares NMI using the DeiT 112 teacher and the results are in line with the attention visualization, i.e. , without asymmetric designs, 113

(b) Encoder and decoder

(a) Various decoders

(c) Mask feature reconstruction

0.7

DeiT

DeiT

0.7

DeiT

No decoder

Encoder

Feature distillation

0.6

Linear projection

Asymmetric decoder

0.5

Decoder

0.6

Mask Feature Reconstruction

0.5

0.4

0.5

Normalized MI

Normalized MI

Normalized MI

0.4

0.4

0.3

0.3

0.3

0.2

0.2

0.2

0.1

0.1

0.1

0.0

0

2

4

Depth

6

8

10

0.0

0

2

4

6

8

Depth

10

12

1.4

16

18

0.0

0

2

4

Depth

6

8

10

(a) Encoder and decoder

(b) Mask feature reconstruction

140

140

140

120

120

120

100

100

100

Attention Distance

Attention Distance

Attention Distance

08

80

80

09

60

60

40

40

40

20

20

0

1

2

E

4

Attention Head

5

6

B

6

10

11

17

4

AttentionHead

5

6

7

8

6

1011

12

2

4

Attention Head

6

8

9

10

11

the student collapses into homogeneity and pays attention to similar tokens in the last few layers. 114 Conversely, the use of asymmetric decoders greatly reduces discrimination. 115

116

117

118

119

The above discussions focus on varying decoders, while FD [41] introduces asymmetric designs to the encoder by adding additional learnable parameters and relative position bias to the attention layers of the student. In the appendix, we demonstrate that the increase in diversity observed in FD also arises from these designs and the diversity brought by them is not always significant.

120

121

122

123

124

125

126

127

128

129

130

131

132

133

134

135

136

137

138

139

140

2.3 The Asymmetric Decoder Harms the Encoder Discrimination

Fig. 3(a) and Fig. 2(b) further measure the average head distance and NMI of the asymmetric decoder. Our findings suggest that the decoder has transferred the discrimination of the teacher, as its behavior is similar to that of the last few layers of the teacher model where attention collapse occurs. Reducing the number of decoder layers does not eliminate this transfer, as further demonstrated in the appendix. Since only the student encoder is retained and applied to downstream tasks after distillation, the semantic information that the model maintained is weakened, which explains why in dBOT, different teachers tend to yield similarly-behaving models after multi-stage distilling. Note that dBOT conducts feature distilling in a mask reconstruction way, while we demonstrate in both Sec. 2.4 and the visualization in the appendix that it behaves similarly to directly distilling features.

2.4 Mask Reconstruction of High-Level Semantics Does not Help Improve Diversity

Fig. 3(b) and Fig. 2(c) examine the influence of mask reconstructing high-level information. To eliminate the effect of the asymmetric decoder, we feed both the masks and tokens into the encoder simultaneously and use only linear projection as the decoder. The overall process is thus similar to SimMIM [44], except that we use the high-level information obtained from the supervised/CL teacher as the distilling objective. Fig. 3(b) proves that reconstructing high-level information brings no diversity gains towards directly distilling features, which is consistent with the finding of [45], i.e. , reconstruction is unnecessary for MIM with semantic-rich teachers. This phenomenon also implies that the diversity of MIM mainly arises from the low-level reconstructing objective rather than from the reconstruction itself, since diversity is absent in high-level reconstruction.

3 Hybrid Distillation

From the above discussion, we conclude that existing distillation pipelines have limitations in 141 providing discrimination and diversity. Thus, we further propose a novel hybrid distillation framework 142

143

to ensure these important properties, and this section elaborates on its details.

Sup/CL

Teacher

Masked

Frozen

MIM

Teacher

Student

#L-1 #L-2

Features

Relations

Relations

Tuned

Student #L

Features

Token Dropping

Distillation

Color Bar

More Irrelevant

Unmasked

120

MAE

110

DeiT

CLIP

HybridDistill (MAE + DeiT)

100

HybridDistill (MAE + CLIP)

AttentionDistance

06

80

70

60

50

40

D

2

4

Depth

6

8

10

3.1 Overview 144

Given a supervised/CL pre-trained model T c , and a MIM pre-trained model T m , Hybrid Distill 145 simultaneously distills knowledge from these two different types of pre-trained teachers, aims at 146 combining their respective advantages to enhance the new representations in a randomly initialized 147 student model S θ where θ is its learnable parameters. ViT [10] is adopted for all the models in Hybrid 148 Distill, and T m is provided by MAE [13] while T c is provided by DeiT [36] or CLIP [30]. 149

Specifically, the Hybrid Distill framework is shown in Fig. 4 and its overall objective is: 150

where ⊙ is an element-wise product operation. M is a mask provided by the teacher model using 151 the strategy described in Sec. 3.2 and M ⊙ x denotes the unmasked patches. D ( · , · ) is the distance 152 measurement, and we use smooth L1 distance in our experiment. α is the hyperparameter that controls 153 the contribution of the two teacher models. Note that we do not distill the final output features T m ( x ) 154 for the MIM pre-trained model but instead use the token relations in the previous ViT layers, denote 155 as T ′ m ( x ) , as the learning objective. Details are illustrated in Sec. 3.2. 156

157

158

159

160

161

162

163

164

3.2 Distilling Strategies

What to distill? Different from previous works [41, 11, 45] that directly distill the features of teacher models, we analyze that the diversity of MIM pre-trained models arises from their superior token-level relationship modeling, while supervised/CL pre-trained models excel at image-level discrimination. Hence, we apply different distilling targets to T c and T m to fully utilize their respective advantages. Specifically, taking T m as an example, we decompose T m into T 1 m ◦ T 2 m ◦ · · · ◦ T L m , where T i m is the i th layer of T m and is composed of a multi-head self-attention (MSA) layer and an MLP layer. Given x i m as the input of the i th layer, the calculation in T i m can be represented as:

where Q i m , K i m , and V i m denotes the linear mappings for x i m and d equals to the dimension of x i m . 165 Then, for MIM pre-trained model T m , we set the token relation R i m ( x i m ) as the distilling target, while 166 for supervised/CL pretrained model T c , we set the output features T i c ( x i c ) as the target. 167

Where to distill? As shown in Fig. 1(a), supervised and CL models tend to collapse into ho168 mogeneity in the last few layers, so Hybrid Distill chooses to distill token relations from T m in 169 these layers to address this collapse and improve diversity. While for the last layer of S which is 170

MAE

Layer 10

Layer 12

Layer 10

Layer 12

(a) Average head distance

(b) NMI

(c) Attention visualization

140

0.8

120

0.7

100

0.6

Attention Distance

80

IN pazjewoN

0.5

60

0.4

0.3

40

0.2

20

2

1

4

Attention Head

了

8

91011

ZI

0.1

0.0

0

MAE

HybridDistill

CLIP

2

Depth

6

8

10

crucial for discrimination, Hybrid Distill directly distills knowledge from T c using the output features. 171 Specifically, we distill token relations from T m at the L -1 and L -2 layers and distill features from 172 T c at the L layer of ViT. Accordingly, the learning objective T c ( x ) and T ′ m ( x ) in Eq. 1 become: 173

Distillation acceleration via redundant token dropping. Suppose the input is divided into N 174 tokens, i.e. , x ∈ R N × d , Hybrid Distill can directly distill token relations and features using all the N 175 tokens. However, since some tokens in the image may be redundant, it is promising to mask these 176 tokens for the student model S to reduce memory and time costs. Furthermore, removing redundant 177 tokens can play a regulatory role, helping the model avoid local optima during the distillation process. 178

179

180

181

182

183

184

185

186

187

188

189

190

191

192

193

194

195

196

197

198

199

200

201

202

203

204

205

206

Specifically, we use the MIM pre-trained teacher T m to guide the identification of redundant tokens and provide the token mask. Inspired by [20], we propose a progressive redundant token masking strategy, which generates token masks at different layers of T m in a progressive manner. Given x i m and the mask M i -1 m provided by the previous layer, we define the tokens in x i m ⊙ M i -1 m and are top K % similar to their average token as redundant tokens in the i th layer and generate a redundant token mask for them. The above process is denoted as T ( x i m ⊙ M i -1 m , K ) . Next, we update M i m using T ( x i m ⊙ M i -1 m , K ) and M i -1 m as follows:

where I is the set of layers required to update the token mask. For M 0 m , all elements are set to 1. Finally, we set the mask M for the student model as M = M L m .

3.3 Property Analysis

Average head distance. Fig. 5(a) visualizes the average head distance of the student model with CLIP and MAE as teachers, while the visualization of CLIP and MAE teachers themselves are included in Fig. 1(a). These visualizations demonstrate that Hybrid Distill enhances the discrimination ability of the student model, compensating for the semantic lacking problem of the MAE teacher. Moreover, Hybrid Distill avoids succeeding attention collapse from the CLIP teacher and generates more diverse representations in the last few layers.

Normalized mutual information. Fig. 5(b) further inspects the NMI. The results demonstrate that the mutual information between tokens is significantly enhanced in the layers where the MAE token relationships are distilled. Besides, this enhancement does not compromise the discrimination obtained from CLIP, as evidenced by attention in the final layers still attending to similar tokens.

Attention visualization. Fig. 5(c) further visualizes the attention between a given query and other keys at different layers to examine behaviors. Compared to MAE, Hybrid Distill exhibits better discrimination ability, i.e. , the query tokens of the last layer have global attention towards the main object of the images, regardless of their location. Besides, Hybrid Distill also improves the locality of the model in the 10 th layer, where attention collapse is known to occur in the CLIP teacher.

3.4 Discussion with Other Distillation Methods

Compared to previous distillation methods [41, 11, 23, 40, 29], Hybrid Distill stands out by not being restricted to using a single teacher network. In addition to addressing the limitations of single-teacher

207

208

209

210

211

212

213

214

215

216

217

218

219

220

221

222

223

224

225

226

227

228

229

distillation in enriching diversity (as discussed in Sec. 2), a more direct factor is that single-teacher distillation cannot create new knowledge, e.g. , creating additional discrimination for the student model when using the MIM teacher. Therefore, we believe that combining and utilizing existing knowledge from various teachers is more effective and convenient. Furthermore, with the growing availability of large-scale pre-trained models within the community, it becomes increasingly valuable to explore new ways to utilize these models and combine their strengths. This further enhances the practical value of our Hybrid Distill, and we hope our work would shed light on new directions.

4 Experiments

4.1 Implementation Details

Hybrid Distill is conducted on 8 V100 GPUs and is built on the codebase of dBOT [23], so most of its settings are in line with dBOT. Specifically, the batch size, learning rate, and weight decay are set to 1024 and 6e-4, and 0.05, respectively. AdamW [26] optimizer and cosine decay [25] schedule is used. The input size is 224 2 . For ViT-B, the distillation is based on ImageNet-1K and the epoch is 300 for main results and 100 for ablation studies. For ViT-L, the distillation is based on ImageNet-21K and the epoch is 40. The hyperparameter α is set to 1 . 0 and the redundant token masking set I is set to [0 , L/ 3 , 2 L/ 3] following [20]. The performances are tested on different downstream tasks. For classification, we report results on ImageNet-1K, CIFAR100 [19], Cars [18], and iNaturalist19 [37]. For object detection and instance segmentation, we fine-tune the student model on COCO [22] using Mask-RCNN [15] following [5]. For semantic segmentation, the evaluation is conducted on ADE20K [47] using the ViT with UperNet [42] following [5, 8]. More details are included in the appendix.

4.2 Main Results

This section presents benchmark results of Hybrid Distill on different downstream. We also list results for supervised and self-supervised pre-trained models, as well as 300-epoch uni-distillation baselines

230

231

232

233

234

235

236

237

238

239

240

241

242

243

244

245

246

247

248

249

250

251

252

253

254

255

256

257

258

259

260

261

262

263

264

which use the same symmetrical structures as Hybrid Distill, for comparison. As shown in Tab. 1, Hybrid Distill achieves performance gains on all downstream tasks, especially for the dense-level ones. Specifically, although the performance of DeiT is suboptimal, its strength can be complementary to MAE and brings considerable benefits, i.e. , when using DeiT and MAE teachers, Hybrid Distill achieves 50.3 AP box and 44.2 AP mask on COCO, as well as 49.1 mIoU on ADE20K, surpassing Distill-MAE by 1.2, 1.1, and 1.3, respectively. Similarly, Hybrid Distill achieves 50.6 AP box and 44.4 AP mask on COCO, as well as 51.5 mIoU on ADE20K when using CLIP and MAE teachers, outperforming Distill-CLIP by 1.1, 0.9, and 1.2, respectively. When using the VIT-L backbone, the performance can be further boosted to 54.6 AP box , 47.6 AP mask and 56.3 mIoU on respective tasks. The improvement on ImageNet-1k is not significant, probably because the distillation is performed on the same dataset, thus increasing diversity fails to bring further gains. In Tab. 2, we further evaluate Hybrid Distill on several small-scale classification datasets and observe more significant gains.

4.3 Ablation Study

This section ablates different variants of Hybrid Distill. The results are reported on dense-level COCO detection and segmentation tasks, as diversity has a stronger influence on these dense-level tasks [27].

Different combinations of two teachers. We first evaluate the benefits of combining two teachers for distillation. As shown in Tab. 3, adding additional MAE attention regularization can bring noticeable improvements (2.5 on AP box and 2.1 on AP mask ) compared to directly distilling from the DeiT teacher. Moreover, the additional attention regularization cannot bring benefits when only using a single DeiT teacher, which suggests that the benefits come from the introduction of MAE teacher. The above conclusions are consistent when using CLIP and MAE teachers, as illustrated in Tab. 4. We also try a much weaker version of MAE teacher which is only pre-trained on ImageNet-100 for 100 epochs in Tab. 4. We lower the weight of this teacher to avoid its impact on discrimination. The results are still positive, which reflects the power of the MIM pre-training in modeling diversity.

Distilling target of the MIM teacher. We then examine the distilling target of the MIM teacher. As shown in Tab. 5, distilling the relation R i m brings the best detection performance ( 50 . 0AP box ). Distilling MSA i m achieves a close performance ( 49 . 8AP box ) since its essential is also distilling relationships, while directly distilling the feature maps T i m brings the worst performance ( 49 . 6AP box ). Nevertheless, all these schemes outperform the DeiT distillation baseline, and the trends are consistent when using CLIP and MAE teachers, as shown in Tab. 6. Besides, we also evaluate a basic setting that directly distills the features of both the MAE and DeiT teachers at the last layer. The result is far from satisfactory, which highlights the effectiveness of the designs in Hybrid Distill.

Distilling position of the MIM teacher. Tab. 7 inspect the distilling position of the MIM teacher. We first experiment with distilling MAE relations at the front, middle, and back layers. Distilling at the back layers achieves better results, i.e. , 1 . 5AP box and 2 . 4AP box gains towards distilling at the

265

266

267

268

269

270

271

272

273

274

275

276

277

278

279

280

281

282

283

284

285

286

287

288

289

290

291

292

293

294

295

296

.

front and middle, respectively. The results are consistent with the fact that attention collapse tends to occur in these back layers. We then ablate the number of distilling layers and find that distilling at the two layers preceding the final layer ( i.e. , 10,11) contributes to the best results.

Token masking strategy. Tab. 8 studies different masking strategies for the student model. Since we progressive drop the redundant tokens three times, the actual tokens used in the student model are (1 -K ) 3 % . We observe that when dropping 30% tokens at a time, Hybrid Distill achieves very close performance ( 49 . 9AP box and 43 . 8AP mask ) to the no masking results and outperforms the random masking strategy and the direct masking strategy which only generates token mask at the last layer. In addition, we notice that our token masking strategy also has a regularizing effect, which can prevent the model from falling into a locally optimal when training for longer epochs. Details about this effect are included in the appendix.

5 Related Work

Representation learning. Pre-training on large-scale datasets (e.g., ImageNet [32], JFT [34], Kinetics [3], etc.) is typically utilized for downstream initialization. Except for the common supervised pretraining [16, 10, 24], contrastive learning (CL) [4, 14, 6, 12] and masked image modeling (MIM) [1, 44, 13] dominate the recent research. The former is achieved by pulling close the features of two different augment views of the input image. While the latter, inspired by masked language modeling [17, 46] in NLP, is realized by reconstructing the mask part of the input image. Recently multi-model extensions [30, 9, 21] of the CL pre-training have also been proposed by utilizing the paired text description of the given image. These different types of pre-training frameworks are proven to have different properties [27, 43], and this paper aims to combine their respective excellent properties to boost a student model.

Knowledge distillation. Knowledge distillation [28, 35, 31] utilizes a well-trained teacher to guide the feature learning of the student model, thus transferring its ability to the student. Beyond its success in supervised learning, some recent works [41, 11, 39, 40, 29] utilize it to extend existing pretrained models or paradigms. Feature distillation (FD) [41] finds that distilling the feature map of the supervised/CL pretrained teacher can bring diverse representation to the student and make it more friendly for downstream fine-tuning. dBOT [23], MVP [40], and BEiT v2 [29] change the mask reconstruction object of MIM to the knowledge of the teacher model to boost MIM pre-training with semantic information. In this paper, we analyze their properties and propose a new hybrid distillation framework to deal with their deficiencies.

6 Conclusion

This paper proposed a hybrid distillation framework that simultaneously distills knowledge from 297 both the supervised/CL pre-trained teacher and MIM pre-trained teacher to enhance the diversity and 298 discrimination of the student. The framework addresses the limitations of single-teacher distillation, 299 where increasing diversity through the use of asymmetric designs may harm discrimination. Specifi300 cally, Hybrid Distill carefully designs the distilling target and location, i.e. , distilling relations from 301 MIM in layers where attention collapse tends to occur and distilling features from supervised/CL 302 in the last layer to preserve discrimination. A progressive redundant token masking strategy is also 303 proposed for reducing the distilling costs. Experiments prove that Hybrid Distill can acquire better 304 properties and achieve promising results on various downstream. We hope our research would shed 305 light on a new direction for applying existing large-scale pre-trained models. 306

307

308

309

310

311

312

313

314

315

316

317

318

319

320

321

322

323

324

325

326

327

328

329

330

331

332

333

334

335

336

337

338

339

340

341

342

343

344

345

346

347

348

349

350

351

352

353

354

References

Hangbo Bao, Li Dong, Songhao Piao, and Furu Wei. BEit: BERT pre-training of image transformers. In International Conference on Learning Representations , 2022.

Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In Proceedings of the International Conference on Computer Vision (ICCV) , 2021.

Joao Carreira and Andrew Zisserman. Quo vadis, action recognition? a new model and the kinetics dataset. In proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , 2017.

Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. arXiv preprint arXiv:2002.05709 , 2020.

Xiaokang Chen, Mingyu Ding, Xiaodi Wang, Ying Xin, Shentong Mo, Yunhao Wang, Shumin Han, Ping Luo, Gang Zeng, and Jingdong Wang. Context autoencoder for self-supervised representation learning. arXiv preprint arXiv:2202.03026 , 2022.

Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He. Improved baselines with momentum contrastive learning. arXiv preprint arXiv:2003.04297 , 2020.

Xinlei Chen, Saining Xie, and Kaiming He. An empirical study of training self-supervised vision transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 9640-9649, 2021.

Yabo Chen, Yuchen Liu, Dongsheng Jiang, Xiaopeng Zhang, Wenrui Dai, Hongkai Xiong, and Qi Tian. Sdae: Self-distillated masked autoencoder. In ECCV , 2022.

Yufeng Cui, Lichen Zhao, Feng Liang, Yangguang Li, and Jing Shao. Democratizing contrastive language-image pre-training: A clip benchmark of data, model, and supervision, 2022.

Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929 , 2020.

Yuxin Fang, Wen Wang, Binhui Xie, Quan Sun, Ledell Wu, Xinggang Wang, Tiejun Huang, Xinlong Wang, and Yue Cao. Eva: Exploring the limits of masked visual representation learning at scale. arXiv preprint arXiv:2211.07636 , 2022.

Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre H Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Daniel Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent: A new approach to self-supervised learning. arXiv preprint arXiv:2006.07733 , 2020.

Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 16000-16009, 2022.

Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 9729-9738, 2020.

Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cnn. In ICCV , pages 2961-2969, 2017.

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR) , pages 770-778, 2016.

Jacob Devlin Ming-Wei Chang Kenton and Lee Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In NAACL-HLT , pages 4171-4186, 2019.

Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations for fine355 grained categorization. In Proceedings of the IEEE international conference on computer vision 356 workshops , pages 554-561, 2013. 357

358

359

Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.

360

361

362

363

364

365

366

367

368

369

370

371

372

373

374

375

376

377

378

379

380

381

382

383

384

385

386

387

388

389

390

391

392

393

394

395

396

397

398

399

400

Jin Li, Yaoming Wang, XIAOPENG ZHANG, Yabo Chen, Dongsheng Jiang, Wenrui Dai, Chenglin Li, Hongkai Xiong, and Qi Tian. Progressively compressed auto-encoder for selfsupervised representation learning. In The Eleventh International Conference on Learning Representations , 2023.

Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation. In ICML , 2022.

Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Proceedings of the European Conference on Computer Vision (ECCV) , pages 1209-1218, 2014.

Xingbin Liu, Jinghao Zhou, Tao Kong, Xianming Lin, and Rongrong Ji. Exploring target representations for masked autoencoders. arXiv preprint arXiv:2209.03917 , 2022.

Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) , 2021.

Ilya Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts. arXiv preprint arXiv:1608.03983 , 2016.

Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101 , 2017.

Namuk Park, Wonjae Kim, Byeongho Heo, Taekyung Kim, and Sangdoo Yun. What do self-supervised vision transformers learn? arXiv preprint arXiv:2305.00729 , 2023.

Wonpyo Park, Dongju Kim, Yan Lu, and Minsu Cho. Relational knowledge distillation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 3967-3976, 2019.

Zhiliang Peng, Li Dong, Hangbo Bao, Qixiang Ye, and Furu Wei. BEiT v2: Masked image modeling with vector-quantized visual tokenizers. 2022.

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning (ICML) , 2021.

Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang, Carlo Gatta, and Yoshua Bengio. Fitnets: Hints for thin deep nets. arXiv preprint arXiv:1412.6550 , 2014.

Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. International journal of computer vision (IJCV) , 115(3):211-252, 2015.

Alexander Strehl and Joydeep Ghosh. Cluster ensembles-a knowledge reuse framework for combining multiple partitions. Journal of machine learning research , 3(Dec):583-617, 2002.

Chen Sun, Abhinav Shrivastava, Saurabh Singh, and Abhinav Gupta. Revisiting unreasonable effectiveness of data in deep learning era. In Proceedings of the IEEE International Conference on Computer Vision (ICCV) , 2017.

Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive representation distillation. arXiv preprint arXiv:1910.10699 , 2019.

401

402

403

404

405

406

407

408

409

410

411

412

413

414

415

416

417

418

419

420

421

422

423

424

425

426

427

428

429

430

431

432

433

434

435

Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Herve Jegou. Training data-efficient image transformers & distillation through attention. In International Conference on Machine Learning (ICML) , volume 139, pages 10347-10357, July 2021.

Grant Van Horn, Oisin Mac Aodha, Yang Song, Yin Cui, Chen Sun, Alex Shepard, Hartwig Adam, Pietro Perona, and Serge Belongie. The inaturalist species classification and detection dataset. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 8769-8778, 2018.

Shaoru Wang, Jin Gao, Zeming Li, Xiaoqin Zhang, and Weiming Hu. A closer look at selfsupervised lightweight vision transformers, 2023.

Wenguan Wang, Tianfei Zhou, Fisher Yu, Jifeng Dai, Ender Konukoglu, and Luc Van Gool. Exploring cross-image pixel contrast for semantic segmentation. arXiv preprint arXiv:2101.11939 , 2021.

Longhui Wei, Lingxi Xie, Wengang Zhou, Houqiang Li, and Qi Tian. Mvp: Multimodalityguided visual pre-training. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XXX , pages 337-353. Springer, 2022.

Yixuan Wei, Han Hu, Zhenda Xie, Zheng Zhang, Yue Cao, Jianmin Bao, Dong Chen, and Baining Guo. Contrastive learning rivals masked image modeling in fine-tuning via feature distillation. Tech Report , 2022.

Tete Xiao, Yingcheng Liu, Bolei Zhou, Yuning Jiang, and Jian Sun. Unified perceptual parsing for scene understanding. In Proceedings of the European Conference on Computer Vision (ECCV) , pages 418-434, 2018.

Zhenda Xie, Zigang Geng, Jingcheng Hu, Zheng Zhang, Han Hu, and Yue Cao. Revealing the dark secrets of masked image modeling. arXiv preprint arXiv:2205.13543 , 2022.

Zhenda Xie, Zheng Zhang, Yue Cao, Yutong Lin, Jianmin Bao, Zhuliang Yao, Qi Dai, and Han Hu. Simmim: A simple framework for masked image modeling. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 9653-9663, 2022.

Hongwei Xue, Peng Gao, Hongyang Li, Yu Qiao, Hao Sun, Houqiang Li, and Jiebo Luo. Stare at what you see: Masked image modeling without reconstruction. arXiv preprint arXiv:2211.08887 , 2022.

Zhengyan Zhang, Xu Han, Zhiyuan Liu, Xin Jiang, Maosong Sun, and Qun Liu. Ernie: Enhanced language representation with informative entities. In ACL , pages 1441-1451, 2019.

Bolei Zhou, Hang Zhao, Xavier Puig, Tete Xiao, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Semantic understanding of scenes through the ade20k dataset. International Journal on Computer Vision (IJCV) , 127:302-321, 2019.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a new hybrid distillation method for Vision Transformer. It is hybrid in the sense that two teacher models pre-trained by two different approaches, namely Contrastive Learning and Masked Image Modeling, are adopted in the distillation. In this way, the student model inherit both the diversity from MIM pre-trained teacher model and the discriminability from the CL pre-trained teacher model. Further more, a progressive masking strategy is adopted to reduce redundancy during distillation. Experiments on multiple popular classification/detection/segmentation benchmarks demonstrate the effectiveness of the proposed approach.\n\nSTRENGTHS:\nThe idea of adopting two homgeneous teacher models is interetsing. Intuitively, a good design allows the student model to learn the advantages of both teacher models.\n\nWEAKNESSES:\n1. I find many experimental details in the explorative model evaluation part (section 2) are missing, making it hard for readers to assess the soundness of those experiments. Note that these experiments are crucial for justifying the motivation of this work. My questions are listed as follows: \na) What kind of distillation approach is used here? Is it the same as the one used in Hybrid Distillation, or just naive feature distillation? Also, details like where to distill and distill objectives are missing. \nb) Why are the average head distance and normalizedmutual information good indicators of the discrimination power and diversity of the learnt representations? And how are the so called `discrimination and diversity` related to the downstream performances? So far I have only seen very vague definition of these two metrics, and I am not sure if they are solid proofs that teacher models pre-trained by different objectives do have different advantages that the student must learn. \nc) The notations need further clarification. For instance, in Figure 2(a), the caption says 'various decoder' , indicating the curves are the values of NMI of the decoder. While there is actually a curve for `no decoder`, indicating the curves are the values of NMI of the encoder. These notations are very confusing.\nd) The authors claim that `The Increase in Diversity Derives from the Asymmetric Designs`, but there are no ablation on if the symmetric design of MIM architecture would lead to the same diversity. The closet design is in section 2.4, but its objective is to reconstruct high-level features from the teacher model.\n\n2. Similarly, some implementation details about the Hybrid Distillation itself are missing. For example, I do not believe the student is solely trained by the distillation objective. Is there also a supervised training objective like cross-entropy loss? The authors should make it more clear. \n\n3. I wonder how did the authors get the results of baseline methods. For example, in table 1, CLIP ViT-B achieves an accuracy of 83.6 on IN-1K, and CLIP ViT-L achieves an accuracy of 86.1. These numbers are too good to be zero-shot results, so I have to assume they are fine-tuning results. Yet, based on [1], fine-tuning CLIP ViT-B achieves an accuracy of 85.7, and CLIP ViT-L achieves an accuracy of 88.0 on IN-1K, which have achieved better or on par performance comapred to the Hybrid Distillation (85.1 for ViT-B, 88.0 for ViT-L). In this case, the proposed Hybrid Distillation approach does not seems to show enough empirical gain.\n\n4. As already discussed in the limitation section in the supplemental material, the gain of Hybrid Distillation over Distill-CLIP is not so significant (only 0.3\\% on IN-1K with ViT-B).\n\nReferences\n[1] Dong X, Bao J, Zhang T, et al. CLIP Itself is a Strong Fine-tuner: Achieving 85.7% and 88.0% Top-1 Accuracy with ViT-B and ViT-L on ImageNet[J]. arXiv preprint arXiv:2212.06138, 2022.\n\nQUESTIONS:\nSee weakness.\n\nLIMITATIONS:\nThe authors have included discussion on the limited gain over Distill-CLIP.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-22"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work presents Hybrid Distillation, which attempts to distill from both supervised/CL and MIM frameworks. The work begins by revealing certain observations regarding the interplay between self-supervised pre-training and the concepts of diversity and discrimination. Subsequently, the authors propose the Hybrid Distillation technique that leverages token relations from the MIM teacher and feature maps from the supervised/CL teacher for knowledge distillation purposes.\n\nSTRENGTHS:\nThe findings on the relationship between diversity and architecture are interesting\n\nWEAKNESSES:\n1. The presentation quality of this work is bad due to the following reasons:\n\ni) Section 2 lacks an explanation of the experimental setup, For example, what is the design and architecture of the distillation? How are different decoders used for DeiT distillation? \n\nii) The description of the metrics, such as the average head distance and normalized mutual information, is inadequate. There is insufficient clarification regarding the existence of multiple attention distances for each layer and how they reflect diversity. Furthermore, the explanation of how NMI reflects discrimination is absent.\n\niii) The analysis of the figures lacks details, making it challenging to comprehend the meaning conveyed by the illustrated figures.\n\n2. The authors state that \"Mask Reconstruction of High-Level Semantics Does not Help Improve Diversity.\" However, previous studies [1][2][3][4] have distilled knowledge from high-level semantics and achieved high performance. Does this imply that high performance can still be attained with low diversity? If so, why should we care about diversity?\n\n3. The correlation between the observations made and the design of the proposed Hybrid Distillation method is weak. For instance, how does the discovery of the \"Asymmetric Decoder\" inspire the proposed Hybrid Distillation approach?\n\n4. The absence of a discussion and comparison of relevant works is noticeable. Numerous related works, such as [5][6][7], should have been included and compared.\n\n5. Unfair comparisons are made in this work. While the proposed approach is distilled from multiple networks, it is only compared with methods that distill knowledge from a single network. Strong baselines that employ distillation from multiple networks should also be incorporated for a fair evaluation.\n\n\n[1] IBOT : IMAGE BERT PRE-TRAINING WITH ONLINE TOKENIZER\n\n[2] DINOv2: Learning Robust Visual Features without Supervision\n\n[3] Masked Feature Prediction for Self-Supervised Visual Pre-Training \n\n[4] BEiT v2: Masked Image Modeling with Vector-Quantized Visual Tokenizers\n\n[5] Mimco: Masked image modeling pre-training with contrastive teacher.\n\n[6] Layer Grafted Pre-training: Bridging Contrastive Learning And Masked Image Modeling For Label-Efficient Representations\n\n[7] Contrastive Masked Autoencoders are Stronger Vision Learners\n\nQUESTIONS:\nSee above\n\nLIMITATIONS:\nNone", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 1, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduce a new distillation method that complimentary harmonizes two distillation method of different properties.\n\nSTRENGTHS:\n* Hybrid Distillation obtained higher accuracies than DeiT, MAE, and CLIP using them.\n* Explanation with analyses (NMI, AHD, and attention map visualization)\n* The paper is clearly written\n\nWEAKNESSES:\n* When I read the explanation in the method section, the proposed method seems very inefficient compared to methods without distillation (e.g., MAE, DINO). It would be better to compare quantitatively with throughput and total training hours.\n\n* Some values in tables are different to original values in references.\n    * The COCO detection performance of MAE in this paper and MAE paper are different. The AP^box and AP^mask of MAE are reported as 50.3% and 44.9% in MAE paper while they are reported as 48.4% and 42.6% in this paper, respectively.\n    * The transfer learning result of MAE toward Naturalist19 is also different to the value in MAE paper. MAE paper report it as 80.5% for ViT-B while this paper report it as 75.2%.\n\n* Are the explanations provided in the preliminary section the author’s own contributions or are they similar analyses  conducted and explained in other references? If they are the author’s contributions, which points can be regarded as novel?\n\n* Some points are not understandable\n    * The authors distilled the features of the supervised and CL models and MIM models to features of the different layers in student model regarding diversity and discriminatively. However, in the current design, the distillation performed on the last layer affect the front layers without detaching back-propagation of the distillation loss. Is this the intended situation?\n\nQUESTIONS:\n* How much more inefficient is Hybrid Distill compared to MAE?\n* It may be better to remove unnecessary area in the right graph in Fig. 3(a)\n\nLIMITATIONS:\n* Its computational inefficiency was my primary concern, and the authors addressed it in the limitations section.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-05"}

comments: SUMMARY:
The paper conducts sufficient experiments and theoretical analysis on diversity and discrimination. 
Meanwhile,  the authors propose a simple yet effective hybrid distillation that combines contrastive learning pre-train and MIM pre-train.
This hybrid distillation achieves significant improvement on downstream tasks.

STRENGTHS:
- The paper is well written, with sufficient experiments and analysis. 
- The accuracy improvement is significant.

WEAKNESSES:
- The paper has some minor typo errors.

QUESTIONS:
1. Figure 4 seems to be wrong, it should be NMI(normalized mutual information), not attention distance.
2. For ViT-B, is the 300-epoch distillation saturated? Would more epochs of distillation bring larger gains? 
3. For ViT-L, is there an obvious difference between using ImageNet-21K dataset and ImageNet-1K dataset? Is the performance improvement due to using ImageNet-21K dataset? Can you provide evaluation results trained on the ImageNet-1K dataset?
4. For the objective function, is it feasible to adjust the weight of the CL branch [e.g.$ \alpha D ( T_{c}(x) \odot M, S_{\theta}(M \odot x) ) $]?  Does the impact of branch CL have to be larger than branch MIM?
5. Are the classification results fair(CIFAR-100, Cars, INaturalist19 )? Can you use a unified setting to finetune the models(MAE, DeiT)?

LIMITATIONS:
- Hybrid Distill does not improve CLIP as much as DeiT after introducing the MAE teacher. The authors could spend more pages analyzing the relationship between the MIM branch and the CL branch, as well as the underlying reasons.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-19

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper explores the subject of representation learning, focusing on two aspects: discrimination and diversity. Contrastive learning (CL) exhibits superior discrimination capabilities but suffers from limited diversity. Conversely, masked image modeling (MIM) offers greater diversity but shows weaker discrimination abilities.\n\nThe paper presents three insightful observations and integrates the benefits of both approaches, optimizing them for downstream tasks. In this context, \"hybrid distillation\" refers to the process where models are distilled using both contrastive learning (CL) for discrimination enhancement, and masked image modeling (MIM) for improving diversity. To reduce the training cost, the paper also proposes a token dropping strategy.\n\nSTRENGTHS:\nThe paper has a solid experimental design to first re-examine discrimination and diversity, then propose its method and last to show its improvements on different downstream tasks. The idea is simple but effective and intuitive.\n\nWEAKNESSES:\n- Please ensure that the term \"asymmetric X\" is used consistently throughout the paper. The document refers to several variants of the term, including \"asymmetric attention,\" \"asymmetric architecture,\" \"asymmetric decoder,\" and \"asymmetric designs.\" It would be beneficial to differentiate between these concepts and clarify which is the primary focus of the paper.\n\n- On line 104, the paper introduces the notation I() and H() to represent the mutual information and entropy, respectively, but does not explain how these quantities are calculated. For clarity, consider adding a brief explanation or citation for the methods used to estimate mutual information and entropy from the data. \n\n- Similarly, on line 156 the notation S' is introduced without explanation. Please consider defining or explaining how S' is derived.\n\n- When \"feature distillation\" is mentioned on line 106, adding a reference to the specific feature distillation approach used in the experiments would help clarify this concept for readers. Providing a citation would allow readers to refer to details of the feature distillation method.\n\nQUESTIONS:\n-\n\nLIMITATIONS:\n-", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-25"}