Title: KD-Zero: Evolving Knowledge Distiller for Any Teacher-Student Pairs
Venue: NeurIPS 2023
Accepted: yes
Keywords: Knowledge distillation
Abstract: Knowledge distillation (KD) has emerged as an effective technique for compressing models that can enhance the lightweight model.  Conventional KD methods propose various designs to allow student model to imitate the teacher better.  However, these handcrafted KD designs heavily rely on expert knowledge and may be sub-optimal for various teacher-student pairs.  In this paper, we present a novel framework, KD-Zero, which utilizes evolutionary search to automatically discover promising distiller  from scratch for any teacher-student architectures.  Specifically, we first decompose the generalized distiller into knowledge transformations, distance functions, and loss weights.  Then,  we construct our distiller search space by selecting advanced operations for these three components.  With sharpness and represent gap as fitting objectives, we evolve candidate populations and generate better distillers by crossover and mutation.  To ensure efficient searching, we employ the loss-rejection protocol, search space shrinkage, and proxy settings during the search process.  In this manner, the discovered distiller can address the capacity gap and cross-architecture challenges for any teacher-student pairs in the final distillation stage.  Comprehensive experiments reveal that KD-Zero consistently outperforms other state-of-the-art methods across diverse architectures on classification, detection, and segmentation tasks.  Noticeably, we provide some practical insights in designing the distiller by analyzing the distiller discovered.  Codes are available in supplementary materials.

KD-Zero: Evolving Knowledge Distiller for Any Teacher-Student Pairs

Abstract

Knowledge distillation (KD) has emerged as an effective technique for compressing models that can enhance the lightweight model. Conventional KD methods propose various designs to allow student model to imitate the teacher better. However, these handcrafted KD designs heavily rely on expert knowledge and may be sub-optimal for various teacher-student pairs. In this paper, we present a novel framework, KD-Zero, which utilizes evolutionary search to automatically discover promising distiller from scratch for any teacher-student architectures. Specifically, we first decompose the generalized distiller into knowledge transformations, distance functions, and loss weights. Then, we construct our distiller search space by selecting advanced operations for these three components. With sharpness and represent gap as fitting objectives, we evolve candidate populations and generate better distillers by crossover and mutation. To ensure efficient searching, we employ the lossrejection protocol, search space shrinkage, and proxy settings during the search process. In this manner, the discovered distiller can address the capacity gap and cross-architecture challenges for any teacher-student pairs in the final distillation stage. Comprehensive experiments reveal that KD-Zero consistently outperforms other state-of-the-art methods across diverse architectures on classification, detection, and segmentation tasks. Noticeably, we provide some practical insights in designing the distiller by analyzing the distiller discovered. Codes are available in supplementary materials.

1 Introduction

Deep Neural Networks (DNNs) have achieved great success in tackling a variety of visual recognition tasks [31, 17, 66]. Despite the appealing performance, the prevailing DNN models usually have large numbers of parameters, leading to heavy costs of memory and computation. Conventional techniques such as Neural Architecture Search [16, 6, 26, 15] and quantizing networks to use low-bit parameters [11, 53, 49] have proven to be effective for mitigating this computational burden. Recently, Knowledge Distillation (KD) has been widely used for training compact and efficient neural networks by transferring the knowledge lied in the logits [23] or features [57] from a large, pre-trained teacher to a smaller student.

Recently, although KD has made significant progress in the handcrafted designs, there are still some limitations to its practical

Teacher

feature

logits

Input

unary op

unary op

unary op

Tr nsform

OP

unary op

Tr nsform

OP

Alignment/

Projection

Distance OP

Weight OP

BatchNorm

Mask

Exp

Pow2

ReLU

SoftMax_C

MultiScale

.....

L1

L2

KL

Hard

Cosine

Pearson

Corr

.....

Distance OP

Loss

Entropy

Focal

Similarity

Hard

Conf

T: 1,4,8,16

W:1,5,25...

...

Weight OP

Student

embed

logits

feature

embed

Transform OP

Minibatch

W-28-2

W-28-2

R20

W-16-20

L2T

NORM

W-40-2

VGG13

W-40-2

VGG13

R32

PVT

ONE

PE

3.6624

3.03.39

90R6

2.00

2.00

6.00

3.029

W-16-4

0.00

1.00

R110

W-16-4

0.00

1.00

R110

R8x4

00'0

3.00

T2T

BYOT

0.00

1.00

2.00

ReviewKD

KD

KD

DIST

DIST

KD-Zero

WSLD

KD-Zero

WSLD

W-28-4

R32x4

W-28-4

R32x4

VGG8

KD

KD-Zero

SNetV1

DML

Assistant

KD/L2

KD-Zero

R50

R50

MNetV2

SNetV2

AVE-MKD

AE-MKD

application for different scenarios from three aspects: (1) Capac- ity gap problem: Traditional KD methods do not always distill better from stronger teachers because of the large teacher-student gap [27, 46]. Larger and more accurate teacher models tend to be overconfident and fail to improve students [79]. To alleviate this, architecture-level methods use the assistant model [46] or architecture search [45], introducing additional training budgets. Other KD techniques use optimized designs or manually modified distillers to reduce the disparity. However, these techniques rely on expert knowledge and require careful tuning, which suffer from poor generality and efficiency. (2) Cross-architecture issue. Teacher-student networks with different architectural styles suffer from mismatches in position dependence, receptive fields, and feature dimensions. As shown in Figure 2, while handcrafted methods ( e.g. , KD [23], DIST [27, 46] and WSLD [79]) provide slight gains in some cases, they still are weak in most cross-architecture pairs 2 . (3) Manual tuning difficulties. Different choices of knowledge sources, distillation functions, and hyperparameters can largely affect the performance of KD. However, designing and training a proper distillation setting requires trial and error, substantial effort, and experiments. Thus, two questions are raised: (1) How to efficiently discover the optimal distillation strategies without expert knowledge? (2) How to reduce the teacher-student gap with different capabilities and architectures?

-Alan Turing

As this well-said quote goes, recently, machine learning approaches have successfully replaced human experts in algorithm design, architecture search, and drug discovery. For the first problem, inspired by Auto-Zero [55], we decide to automate the process of designing the distillation functions for the first time in the field of knowledge distillation. For the second question, we systematically review previous studies of KD design for distillation gaps. Our observations reveal that: (1) Most of the distillers can be divided into three components with various key elements: knowledge transformation, distance function, and loss weights. (2) Some essential operations, like normalize ops [3] and mask ops [71], play important roles in reducing the teacher-student gap. (3) Optimal distillation functions can be effectively integrated with additional strategies ( e.g. , feature aggregation [50], projector ensemble [9], N-to-One match [67], and cross-layer mapping [28, 13] in Figure 3).

Based on the above analysis, we present KD-zero, an automated search framework that utilizes evolutionary algorithms from scratch to efficiently discover the best distiller without manual design. Specifically, our framework is organized into three parts: search space, search algorithm, and acceleration strategy. Firstly, we establish the search space with basic transform operations, distance functions, and loss weights (see Figure 1). For example, we select operators normalized in different dimensions ( e.g. , batchnorm,norm HW,C,N ), various types of activation functions ( e.g. , exp, relu, tanh, sigmoid, pow2), multi-scale process and spatial-wise/channel-wise mask transforms and other advanced operations in the knowledge transformation. Our distance function options include smooth ℓ 1 , ℓ 1 , ℓ 2 , ℓ KL , ℓ hard , ℓ Cosine , ℓ Pearson and ℓ Correlation distance. Options in the loss weight part include various values for loss factors, temperature factors, and weight calibration strategies. In this way, our search space includes over 10,000 candidates covering the existing SOTA KD methods and designs. Then, we construct a calculation graph for these candidates and use selected features, representations, and logits from the teacher-student network as input. Based graph structure, we

initialize the candidate total groups from the search space, crossover, and mutate them according to the evaluation results. We employ loss-rejection protocol and search space shrinkage for search efficiency to filter out weak candidates. With early-stop proxy settings, we achieve at least 40 × acceleration during the distiller search. For distillation gap reduction, we take the representation gap and sharpness gap between teacher-student as the fitting objectives besides the accuracy metric of the validation set. Finally, we distill student architectures with discovered distiller, and our KD-Zero surpasses existing KD approaches by a large margin without prior knowledge (see Figure 3).

In principle, our KD-Zero differs from previous hand-designed KD methods, opening new doors to automated distillation designs. Its merits can be highlighted in three aspects: (1) Effective. KD-Zero effectively reduces the teacher-student gap by presenting a general distiller search space and adaptive evolutionary search for different teacher-student pairs. KD-Zero extends the KD formulation and allows for additional gains with extra design besides distillers. In addition, it reduces human bias and ensures that the resulting distillers are optimized for the target problem or dataset. (2) Efficient. KD-Zero increases efficiency in practice by a series of flexible, systematic, and efficient search procedures without additional laborious tuning. By contrast, other manual methods with fixed KD forms involve complex parameter tuning with additional training time and resources. (3) Insightful. KD-Zero undertakes an in-depth analysis of the existing advanced distillation designs, with the aim of exploring their potential combination to produce numerous novel distillers. KD-Zero provides guidelines for practical applications and develops a new research direction. We hope our efforts on the automated design of distillers could facilitate future research for automated KD works to some extent. In summary, our contributions are:

To alleviate architecture & capability gaps of teacher-student, we present KD-Zero, the first auto-search framework for evolving best distillers from scratch to our best knowledge.

We present a comprehensive distiller search space, including advanced operations on transformations, distance functions, and loss weights. Then, we evolve the distiller search with performance and sharpness & represent the gap as fitting objectives. In addition, We achieve significant search acceleration via loss-rejection protocol & space shrinkage, and proxy settings.

We conduct extensive experiments on classification, detection, and segmentation. KD-Zero performs state-of-the-art in multiple datasets and architectures ( e.g. , CNN and vision transformer). Specifically, ResNet-18 and MobileNet with KD-Zero achieve 72.17% and 73.02% Top-1 accuracy on ImageNet, outperforming KD by 1.51%, 2.34%, respectively.

2 Related Work

Knowledge Distillation. The idea of teacher-student learning is first proposed in pioneering explorations [1, 2], and the formal definition is defined by the original KD [23]. Subsequent efforts explore on different knowledge ( e.g. , intermediate feature representations [38, 43, 37, 33, 36], sample relationships [48, 60]) and applications [18, 14, 35]. Compared to KD for distillation gap. Previous methods propose assistant teachers, architecture search, KD designs on transformations [27], distance functions [59], and weight-tuning [42, 34] for this problem. However, such KD designs rely on expert knowledge and tuning, and their performance can fluctuate significantly across different situations. In contrast to these methods, KD-Zero develops automated searches for distillers to address these difficulties that do not require additional architecture modification and manual KD design. Compared to Meta-KDs. These works [13, 42] only focus on hyperparameter tuning and involve complex optimization challenges. In contrast, our approach searches for complementary distiller design besides hyperparameters establishing a new paradigm for KD research and application.

Automated Machine Learning. AutoML [81, 80] aims to automate Network Architecture Search (NAS) and HyperParameter Optimization (HPO), making them more accessible to non-experts. NAS chooses architecture rather than KD designs. Compared to HPOs [54, 69], they generally focus on the hyperparameters on training configurations. Recent methods search for loss formulation [39, 32]. In contrast to these methods, we present a new complex search space for distiller design in transforms, distances, loss weights, and new search objectives and accelerations according to the KD task.

Distiller search space

Input

Transform

Distance

Weight

L

E

F

Norm_c

KL

Relu

L1

L2

Batchnorm

Focal

Mask

Similarity

Pow2

Cosine

Entropy

Focal

Corre

Clae

Conf

no

Pearson

Scale

KL

Scale

Logits

No,T:1 W:5

Initial population

No

No

Logits

Space

Shrinkage

Loss-rejection

Protocol

Crossover

Mutation

Next population

Scale

KL

Scale

Feature

Pow2

Pow2

Feature

Focal,T:8 W:1

Scale

Pearson

Scale

Logits

Exp

Exp

Logits

Focal,T:1 W:5

Scale

KL

Scale

Logits

No,T:1 W:5

Batch

Batch

Logits

Scale

L1

Scale

Logits

Relu

Relu

Logits

Focal,T:1 W:5

Scale

L2

Scale

Feature

Exp

Exp

Feature

Focal,T:1 W:5

Evaluations

Validate acc & loss

Sharpness

-

gap

Represent

-

gap

T

S

cat

Distillation

Evolution

3 Methodology

In this section, we first illustrate the design of our distiller search space, the search process, the acceleration, and the fitting objectives. Then we analyze the search results and give some guidelines. Finally, we analyze the student distilled via KD-Zero and expansion for different distillation scenarios. The pipeline of our approach is shown in Figure 4.

3.1 Search Space for Distillers Discovery

Search space structure. In KD, the student student S is distilled with the fixed teacher T by minimizing:

where W f and W Cal is the loss weights factor and calibration [79], τ is the temperature factor, T is transformations, D ( · , · ) is distance function measuring the knowledge difference. f T and f S are outputs ( e.g. , features, embeddings, and logits) of the teacher-student. Following this general KD formulation, our search space consists of different types of operations (see Table 1) in transformations, distance functions, and loss weights parts. Then, we use a computation graph to represent each candidate, in which the input nodes are different types of knowledge and the intermediate nodes are primitive operations. In addition, we assign three transform options as transform-1 → transform2 → transform-3 for the transformation part to ensure effective processing of the input knowledge.

Insight of space design. Our space design enjoys multiple merits. (1) Comprehensive & flexible: KD-Zero contains key elements of most existing KD methods, including the normalize, mask,

attention, and focal-calibration KDs [3, 79, 27] for distillation gaps. In addition, we introduce basic operations in loss design to improve the flexibility and diversity of the search. (2) Extended & Innovative: We extend advanced KD design for omni-dimensional and various knowledge. For example, we expand the normalized operation for batch, channel, and spatial dimensions, which can be used for logits, embeddings, and feature knowledge. Thus, such a unified and flexible search space would provide some inspiration for KD design.

3.2 Evolution Procedure, Acceleration and Objective

Evolution process. Our EA starts with an initial population of candidate distillers evaluated for their fitness based on our distillation gap-related multi-objectives. The algorithm then iteratively evolves the population over generations using genetic operators, such as selection, crossover, and mutation, to generate better solutions. Specifically, distillers are first randomly generated to form the initial population. Each candidate solution in the population is evaluated using multiple fitness functions that measure teacher-student gaps. Based on these evaluations, we select the best-performing individuals from the population to create a new population for the next generation. Then, we apply crossover to the selected individuals to create new offspring and use mutation to the new offspring to introduce diversity into the population, which helps to explore new distillers.

Search acceleration As the search space is sparse with many unpromising distillers, we employ several strategies to accelerate: (1) Loss-rejection protocol. We filter out candidates with excessive loss values or collapsed optimization during the search. (2) Search space shrinkage. We reduce sampling probabilities for the operations frequently in loss rejection and tail candidates with search iterations. (3) Proxy settings. With diverse and informative knowledge learned from a teacher, student models offer advantages in terms of faster training speeds. Based on these properties, we employ early stop the training process once the student model performs well enough to determine the quality of the candidate distillation. Nevertheless, proxy settings can also introduce evaluation uncertainties, and we alleviate this issue by introducing multiple distillation gap metrics in the following sections.

Fitting objectives. To accurately evaluate each distiller and reduce the distillation gap, we include cross-entropy loss, sharpness-gap [20] on prediction, and CKA-gap [52] on representation between teacher-student as the multi-objectives. Specifically, we conduct a training-free evolutionary search algorithm to efficiently discover the optimal distiller α ∗ from search space A , as:

where L CE is the regular cross-entropy objective with labels Y , the sharpness metric is the logarithm of the exponential sum of logits, and Centered Kernel Alignment (CKA) metric is normalized from Hilbert-Schmidt Independence Criterion (HSIC) [19] on high-level feature.

feature

embed

logits

Knowledge inputs

31%

13%

nish

13%

channel

nelu

softmax-N

local-s4

ine1

ls-HW

I0G0-6/1

mask

ms-14

3%

3%

3%

3%

19%

COS

6%

hard

11

pear

Distance functions

13%

6%

38%

focal

sim

no

entropy

6%

Weight calibration

50

100

10

12%

6%

6%

0.01

0.1

6%

Weight values

12%

31%

16

1

Temperature values

19%

25%

44%

25%

%

6%

6%

6%

19%

13%

13%

24%

25%

3.3 Results Analysis and Practical Guidance

Figure 5 present searched distillers for different models. Based on these results, some practical guidance for KD designs can be summarized as follows:

For knowledge input, feature knowledge enjoys superiority while logits and embedded knowledge share identical status. Searched distillers with feature knowledge take more proportion, and ablation studies on vanilla feature knowledge also surpass logits and embedding knowledge.

This observation also aligns with the existing conclusion that feature-based KD outperforms other KD on detection tasks [27, 63].

For transformations, the normalized-based operations play a key role in most optimal distillers, and its vanilla performance outperforms other types of transformations in ablation studies. These observations illustrate that normalization benefits distillation, consistent with the current KD methods. Scale-based and activation-based operations are also crucial for the KD because they are often present in optimal distillers and enjoy good vanilla performance in ablation studies.

For distance functions, normalized-based distances such as ℓ Pearson and ℓ correlation enjoy better results in ablation studies and are often adopted by teacher-student pairs across architectures. This suggests that these normalized-based distances can reduce distillation gaps in complex scenarios. In addition, some simple distance functions ( e.g. , ℓ 1 , ℓ 2 and ℓ KL ) also appear in the optimal distiller. These may be attributed to these distillers employing advanced transformation operations for input knowledge.

For loss weights, adjusting the temperature values is helpful for different scenarios, and the optimal weight values are generally between 1 and 10 based on search results and ablation studies. In addition, the focal weight calibration outperforms the no-weight calibration in ablation studies, and it is often used for some optimal distillers. In summary, we should employ smaller weight values and actively explore different weight calibration and temperature values to reduce distillation gaps under different teacher-student pairs.

3.4 Distilling Student via Discovered KD-Zero functions

After search, the discovered distiller L KD is used for distilling student f S combined with crossentropy L CE via label Y in single teacher f T or multiple teacher f T i KD as:

Extended to various distillation designs & scenarios KD-Zero focuses on the distiller's search and employs simple 1 × 1-Conv for channel alignment. Recently, some KD methods have proposed other designs in feature aggregation [50], projector ensemble [9], N-to-One match [67], and crosslayer mapping [28, 13]. As shown in Table 2, KD-Zero can combine well with them by replacing their default ℓ 2 / ℓ KL losses. In addition, the distiller search of KD-Zero also benefits different distillation scenarios. Specifically, Self-KDs, online KDs, and multi-teacher KDs with KD-Zero achieve extra gains than the default settings. Also, KD-Zero can combine architecture-level methods ( e.g. , Assistant [46]) to reduce distillation gaps further.

Why can KD-Zero bridge the teacher-student gap? The answer is intuitive: In KD-Zero, the search space contains many operators for distillation gap reduction, and search objectives are directly designed to reduce the prediction and representation gap between teacher-student. For example, We approximate the sharpness gap using a Taylor second expansion [20]:

Following Hindon's assumption [23] that the logits of each training sample are approximately zero-meaned, i.e., ¯ f T , ¯ f S = 0 . So the gap can be rewritten as log ( 1 + 1 2 V ar ( f T ) ) -log ( 1 + 1 2 V ar ( f S ) ) [20]. Our options like normalized transform [3, 27], and weight calibrations [79] can effectively reduce the variance of teacher-studens' outputs and minimize the distillation gap. In addition, the student models distilled with KD-zero enjoy the merits: (1) Fast convergence and superior performance. As shown in Figure 6, KD-Zero has surpassed the best accuracies of KD

Input         Student          KD          KD-Zero       Teacher

Input         Student          KD          KD-Zero       Teacher

80

70

0

Teacher

Student

(%) 1-

60

(%) enxov 1-do1

60

50

MW

40

Baseline

Baseline

30

KD

KD-Zero

KD

KD-Zero

20

20

10

0

50

100

150

200

10

20

40

60

80

100

and baseline at the beginning of the 3rd learning-rate decay stage. (2) Smaller teacher-student gap. As shown in Figure 7, features and logits of students via KD-Zero have stronger similarities with teachers than the original KD method.

4 Experiments

In this section, we assess the efficacy of our proposed KD-Zero approach on classification, detection, and segmentation tasks. Additionally, we compare its performance with other KD methods, ensuring fair comparisons by utilizing the same training settings. We report mean results based on more than 3 repeated trials. More detailed experiment settings and results are available in the Appendix.

4.1 Experiments on CIFAR-100

Implementation . We utilize the CIFAR-100 dataset [30] in knowledge distillation. During the distiller search phase, we apply 5% early-stopping training epochs with full training data for acceleration settings. Our evolutionary algorithm with 20 population sizes performs 100 iterations for each teacher-student pair. During the distillation phase, all teacher-student networks are trained using typical training settings, with a training epoch of 240. The multi-step learning rate commences at 0.1, which decays by 0.1 at 100 and 150 epochs. Recently, knowledge distillation enabled training Vision Transformers (ViT) from scratch with CNNs as teachers. To evaluate the effectiveness of KD-Zero, we conduct the evolutionary search for ViT-based distillation strategies with the same settings as the

CNN experiment. Subsequently, we train the ViT with the optimal distiller obtained and ResNet-56 as CNN teacher. The training is conducted on 224 × 224 resolution images for 300 epochs, with an initial learning rate of 5e-4 and a weight decay 0.05 using the AdamW optimizer.

Comparison results on CNN models. Table 3 presents a comparative analysis of our KD-Zero with other state-of-the-art (SOTA) KD methods. We conduct multiple trials with randomly selected distillers in the same search space, called Rand-KD, to evaluate the efficacy of our EA search. For teacher-student pairs with the same architectural style, KD-Zero outperforms the baselines by margins ranging from 3 . 16% ∼ 4 . 90% . Compared with Rand-KD and other KDs, KD-Zero obtains consistent performance gains ( 1 . 0% ∼ 3 . 2% ). Besides strengths in the same architecture pairs, KD-Zero exhibits even stronger performance when dealing with different architectural styles, while other KD methods suffer from noticeable accuracy reductions. Specifically, KD-Zero outperforms the baseline by margins of 5 . 6% ∼ 7 . 3% and the random search results by margins of 1 . 9% ∼ 2 . 3% , demonstrating the effectiveness of our design for different structures. Compared with other SOTA KD methods, our KD-Zero achieves 1 . 2% ∼ 1 . 5% gains. These results show that KD-Zero can improve each student model with simple settings under different teacher-student pairs.

Comparison results on vision transformer. Table 3 presents the results of the vanilla and distillation models employing different distillation methods. The results indicate that KD-Zero can significantly improve the performance of vision transformers with 8 . 0% ∼ 13 . 1% margins and consistently yields superior performance than other methods. In addition, our proposed method applies to various ViT architectures, thereby validating its effectiveness. Note that most ViT students possess larger model sizes ( e.g. , DeiT-Ti with 5 million parameters) and greater capabilities than the CNN teacher ( e.g. , ResNet-56 with only 0.86 million parameters). Some ViT students outperform the CNN model in the strong regularization setting on large-scale datasets. However, when it comes to ViT distillation on small datasets, employing CNN teachers helps address the issue of ViT models struggling to train effectively from scratch. In this context, using CNN teachers in distillation is akin to auxiliary training or providing additional regularization supervision. As a result, these ViTs demonstrate their original strong representation capability after distillation and consequently outperform the CNN teacher.

4.2 Experiments on ImageNet

Implementation . We additionally conduct experiments on the ImageNet[12]. Following CIFAR-100 trials, we employ similar EA settings on a subset of ImageNet for search acceleration. Then, we utilize the discovered distiller for the training of student models ( e.g. , ResNet-18 [21] and MobileNet [25]). The training settings are the same as the other KD methods and involve training for 100 epochs using a multi-step learning rate, which commences at 0.1 and decays by 0.1 at 30, 60, and 90 epochs.

Comparison results. As shown in Table 4, our proposed KD-Zero significantly improves the accuracy of baseline models, yielding gains of 2 . 5% ∼ 2 . 9% in Top-1 accuracy for ResNet-18 and MobileNet, respectively. In addition, KD-Zero surpasses other SOTA methods with clear gains, demonstrating its superiority in large-scale datasets. These findings substantiate the effectiveness of KD-Zero in distillation optimization with considerable benefits, establishing the versatility and potency of our framework. In summary, KD-Zero facilitates substantially improved predictive accuracy of student models on ImageNet, more complex domains while preserving superior performance.

Visualizations. The comparison between the Grad-CAM++ maps generated by the student model trained with KD-Zero and other methods is presented in Figure 8. The results indicate that the Grad-CAM++ map generated by the student model trained with KD-Zero is more similar to that of the teacher model compared to the student model trained independently. In contrast, independent training of the student model leads to incorrect focus areas. These findings suggest that the KD-Zero

0.53

Ablation study on search space

T

9°0

Ablation study of search

space organization

0.6

Ablation study on search algorithm

of Candidates

0.525

0.52

T

I

T

T

Top-1 Accuracy (%) of Candidates

0.55

S°0

0.5

(%)

Top-1 A

0.495

0.515

0.505

0.485

0.49

0.51

0.5

F

0.25

0.45

0.35

0.4

0.3

上

1

0.4

0.3

0.2

0.1

Naive Evolution

Our Evolution

2

40

42

$49

568

t4r

1909 2909 901 9901 9680 +919 2900 509 8*01 9+00

0.81

0.48

feature

nomm

nO

-trans

trans-layer3

trans-layer

fixed W

fixed W & T

0

20

Iteration numbers

40

09

80

10

942

4425

9.894

50115 1854 86185 00n9 08295 0.816 0.18185 8017 80175 0018

approach is more effective than traditional knowledge distillation methods in guiding the student model to learn from the teacher model, resulting in improved model interpretability and performance.

4.3 Experiments on Object Detection & Semantic Segmentation

Object detection. We conduct experiments on the MS-COCO dataset[41]. We use the optimal distiller on ImageNet to distill knowledge from teacher detectors to students. Based on the strong baseline [5], we apply KD-Zero to two-stage detector ( e.g. , Faster R-CNN [56]) and the one-stage detector ( e.g. , RetinaNet [40]), which are widely used object detection frameworks. Following common practice [40], all models are trained with a 2 × learning schedule (24 epochs). We train all the models with SGD optimizer, where the momentum is 0.9, and the weight decay is 0.0001. As shown in Table 5, our KD-Zero improves the AP by 3.5 on RetinaNet and 3.4 on Faster R-CNN, respectively, outperforms previous state-of-the-art techniques, including [59, 70, 74], for both object detectors. The results substantiate the potential of KD-Zero for scaling knowledge transfer to broader datasets and more complex computer vision problems while preserving improved accuracy.

Semantic segmentation. We evaluate KD-Zero on Cityscapes dataset[10]. Following the previous work, we adopt PSPNet-ResNet101 [78, 51] as the teacher and PSPNet and DeepLabV3[7] models with the ResNet18 backbone as the student. During distillation, the batch size is 8, and the models are trained for 40K iterations with the SGD optimizer, where the momentum is 0.9 and the weight decay is 0.0005. The results are reported with mean Intersection-over-Union (mIoU) under the single-scale evaluation setting. As shown in Table 6, the student PSPNet and DeepLabV3 get 3.17 and 3.7 mIoU improvement by adding our KD-Zero loss. These results indicate that our method surpasses the state-of-the-art distillation method for semantic segmentation, demonstrating that searched distillers facilitate student learning.

4.4 Ablation Studies

Search space organization. We individually explore various operation types and organized settings in Figure 9. Results indicate: (1) some operations like the normalized-based transforms, normalizedbased losses, focal correction, and small-weighted values bring superior gains than others. (2) our design is optimal across different settings, and the removal of transform, weight factor, and temperature factor search leads to worse results, proving our organization's strength.

Search algorithm. We use the EA for distiller search, which is gradient-free and flexible for nonconvex optimization. As shown in Figure 10, the EA obtains faster convergence and better final search results than random search. In addition, for fitting objectives in EA, Shapeness and CKA metrics demonstrate good correlations with the final distillation accuracy and combine well with cross-entropy loss to improve evaluation quality.

Generalization of searched distillers for different Teachers. In Table 8 & 7, we present detailed results of the same searched distiller with various capacities teacher models on CIFAR-100. The results indicate that our KD-Zero can significantly improve the student model in different teacher models and surpass the KD and improved KD methods ( e.g. , DIST [27], WSLD [79]).

In addition, we provide combined trials in Table 2 and Figure 5, student training and teacher-student gap analysis in Figure 6 & 7. These ablation studies demonstrate that our KD-Zero can well alleviate distillation gaps and combine with other KD designs.

5 Conclusion

In this paper, we introduce the KD-Zero framework, an innovative framework for automatically designing distillers. We construct a flexible and unified distiller search space with advanced operators on knowledge transformation, distance function, and loss weight parts. Then, we employ evolutionary search with distillation-gap reduction as objectives and search acceleration strategies, enabling KDZero to efficiently find distillers and improve student model performance in distillation training. We employ the loss-rejection protocol, search space shrinkage, and proxy settings during the search process to enhance search efficiency. Furthermore, we present some practical guidelines based on the results analysis and propose various extensions of KD-Zero to cater to different KD designs and scenarios. Comprehensive experiments on three benchmarks demonstrate the effectiveness and universality of the KD-Zero framework for various CNNs, Vision Transformer models, object detection, and semantic segmentation. In future work, we will extend KD-Zero to more tasks and explore large language models as code encoders or searcher for KD-Zero. We hope this work will inspire future research on knowledge distillation designs.

References

Lei Jimmy Ba and Rich Caruana. Do deep nets really need to be deep? In NIPS , 2014. 3

Cristian Bucila, Rich Caruana, and Alexandru Niculescu-Mizil. Model compression. In KDD , 2006. 3

Weihan Cao, Yifan Zhang, Jianfei Gao, Anda Cheng, Ke Cheng, and Jian Cheng. Pkd: General distillation framework for object detectors via pearson correlation coefficient. In NeurIPS , 2022. 2, 5, 6

A. Chattopadhyay, Anirban Sarkar, Prantik Howlader, and V. Balasubramanian. Grad-cam++: Improved visual explanations for deep convolutional networks. In WACV , 2018. 7

Kai Chen, Jiaqi Wang, Jiangmiao Pang, Yuhang Cao, Yu Xiong, Xiaoxiao Li, Shuyang Sun, Wansen Feng, Ziwei Liu, Jiarui Xu, Zheng Zhang, Dazhi Cheng, Chenchen Zhu, Tianheng Cheng, Qijie Zhao, Buyu Li, Xin Lu, Rui Zhu, Yue Wu, Jifeng Dai, Jingdong Wang, Jianping Shi, Wanli Ouyang, Chen Change Loy, and Dahua Lin. MMDetection: Open mmlab detection toolbox and benchmark. arXiv preprint arXiv:1906.07155 , 2019. 9

Kunlong Chen, Liu Yang, Yitian Chen, Kunjin Chen, Yidan Xu, and Lujun Li. Gp-nas-ensemble: a model for the nas performance prediction. 1

Liang-Chieh Chen, George Papandreou, Florian Schroff, and Hartwig Adam. Rethinking atrous convolution for semantic image segmentation. arXiv preprint arXiv:1706.05587 , 2017. 9

Pengguang Chen, Shu Liu, Hengshuang Zhao, and Jiaya Jia. Distilling knowledge via knowledge review. In CVPR , 2021. 7, 8

Yudong Chen, Sen Wang, Jiajun Liu, Xuwei Xu, Frank de Hoog, and Zi Huang. Improved feature distillation via projector ensemble. In NeurIPS , 2022. 2, 6

Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The cityscapes dataset for semantic urban scene understanding. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 3213-3223, 2016. 9

Matthieu Courbariaux, Yoshua Bengio, and Jean-Pierre David. Binaryconnect: Training deep neural networks with binary weights during propagations. In NIPS , 2015. 1

Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. pages 248-255, 2009. 8

Xueqing Deng, Dawei Sun, Shawn Newsam, and Peng Wang. Distpro: Searching a fast knowledge distillation process via meta optimization. 2022. 2, 3, 6

Peijie Dong, Lujun Li, and Zimian Wei. Diswot: Student architecture search for distillation without training. In CVPR , 2023. 3

Peijie Dong, Xin Niu, Lujun Li, Zhiliang Tian, Xiaodong Wang, Zimian Wei, Hengyue Pan, and Dongsheng Li. Rd-nas: Enhancing one-shot supernet ranking ability via ranking distillation from zero-cost proxies. arXiv preprint arXiv:2301.09850 , 2023. 1

Peijie Dong, Xin Niu, Lujun Li, Linzhen Xie, Wenbin Zou, Tian Ye, Zimian Wei, and Hengyue Pan. Prior-guided one-shot neural architecture search. arXiv preprint arXiv:2206.13329 , 2022. 1

Peijie Dong, Xin Niu, Zhiliang Tian, Lujun Li, Xiaodong Wang, Zimian Wei, Hengyue Pan, and Dongsheng Li. Progressive meta-pooling learning for lightweight image classification model. In ICASSP , 2023. 1

Nuno C Garcia, Pietro Morerio, and Vittorio Murino. Modality distillation with multiple stream networks for action recognition. In ECCV , 2018. 3

Arthur Gretton, Olivier Bousquet, Alex Smola, and Bernhard Schölkopf. Measuring statistical dependence with hilbert-schmidt norms. In International conference on algorithmic learning theory , pages 63-77, 2005. 5

Jia Guo. Reducing the teacher-student gap via adaptive temperatures. 2021. 5, 6

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. pages 770-778, 2016. 8

Byeongho Heo, Jeesoo Kim, Sangdoo Yun, Hyojin Park, Nojun Kwak, and Jin Young Choi. A comprehensive overhaul of feature distillation. In ICCV , 2019. 8

Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531 , 2015. 1, 2, 3, 6, 7, 8

Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531 , 2015. 9

Andrew G. Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam. Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint, arXiv:1704.04861 , 2017. 8

Yiming Hu, Xingang Wang, Lujun Li, and Qingyi Gu. Improving one-shot nas with shrinkingand-expanding supernet. Pattern Recognition , 2021. 1

Tao Huang, Shan You, Fei Wang, Chen Qian, and Chang Xu. Knowledge distillation from a stronger teacher. arXiv preprint arXiv:2205.10536 , 2022. 2, 3, 5, 6, 7, 9, 10

Yunhun Jang, Hankook Lee, Sung Ju Hwang, and Jinwoo Shin. Learning what and where to transfer. In ICML , 2019. 2, 6

Adrian Bulat Georgios Tzimiropoulos Jing Yang, Brais Martinez. Knowledge distillation via softmax regression representation learning. In ICLR2021 , 2021. 7, 8

Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. Tech Report , 2009. 7

Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. In NIPS , 2012. 1

Hao Li, Tianwen Fu, Jifeng Dai, Hongsheng Li, Gao Huang, and Xizhou Zhu. Autoloss-zero: Searching loss functions from scratch for generic tasks. In CVPR , 2022. 3

Lujun Li. Self-regulated feature learning via teacher-free feature distillation. In European Conference on Computer Vision (ECCV) , 2022. 3

Lujun Li, Peijie Dong, Zimian Wei, and Ya Yang. Automated knowledge distillation via monte carlo tree search. In ICCV , 2023. 3

Lujun Li and Zhe Jin. Shadow knowledge distillation: Bridging offline and online knowledge transfer. In NeuIPS , 2022. 3

Lujun Li, Liang Shiuan-Ni, Ya Yang, and Zhe Jin. Boosting online feature transfer via separable feature fusion. In IJCNN , 2022. 3

Lujun Li, Liang Shiuan-Ni, Ya Yang, and Zhe Jin. Teacher-free distillation via regularizing intermediate representation. In IJCNN , 2022. 3

Lujun Li, Yikai Wang, Anbang Yao, Yi Qian, Xiao Zhou, and Ke He. Explicit connection distillation. 2020. 3

Zelong Li, Jianchao Ji, Yingqiang Ge, and Yongfeng Zhang. Autolossgen: Automatic loss function generation for recommender systems. In Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval , pages 1304-1315, 2022. 3

Tsung-Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollar. Focal loss for dense object detection. In ICCV , 2017. 9

Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European conference on computer vision , pages 740-755. Springer, 2014. 9

Jihao Liu, Boxiao Liu, Hongsheng Li, and Yu Liu. Meta knowledge distillation. arXiv preprint arXiv:2202.07940 , 2022. 3

Xiaolong Liu, Lujun Li, Chao Li, and Anbang Yao. Norm: Knowledge distillation via n-to-one representation matching. In ICLR , 2023. 3

Yifan Liu, Changyong Shu, Jingdong Wang, and Chunhua Shen. Structured knowledge distillation for dense prediction. IEEE transactions on pattern analysis and machine intelligence , 2020. 9

Yu Liu, Xuhui Jia, Mingxing Tan, Raviteja Vemulapalli, Yukun Zhu, Bradley Green, and Xiaogang Wang. Search to distill: Pearls are everywhere but not the eyes. In CVPR , 2020. 2

Seyed Iman Mirzadeh, Mehrdad Farajtabar, Ang Li, Nir Levine, Akihiro Matsukawa, and Hassan Ghasemzadeh. Improved knowledge distillation via teacher assistant. In AAAI , 2020. 2, 6

Yulei Niu, Long Chen, Chang Zhou, and Hanwang Zhang. Respecting transfer gap in knowledge distillation. 2022. 7

Wonpyo Park, Yan Lu, Minsu Cho, and Dongju Kim. Relational knowledge distillation. In CVPR , 2019. 3, 7

Zimian Wei Xin Niu ZHILIANG TIAN Hengyue Pan Peijie Dong, Lujun Li. Emq: Evolving training-free proxies for automated mixed precision quantization. In ICCV , 2023. 1

Chen Pengguang, Liu Shu, Zhao Hengshuang, and Jiaya Jia. Distilling knowledge via knowledge review. In CVPR , 2021. 2, 6, 8

Jie Qin, Jie Wu, Xuefeng Xiao, Lujun Li, and Xingang Wang. Activation modulation and recalibration scheme for weakly supervised semantic segmentation. In AAAI , 2022. 9

Zengyu Qiu, Xinzhu Ma, Kunlin Yang, Chunya Liu, Jun Hou, Shuai Yi, and Wanli Ouyang. Better teacher better student: Dynamic prior knowledge for knowledge distillation. arXiv preprint arXiv:2206.06067 , 2022. 5

Mohammad Rastegari, Vicente Ordonez, Joseph Redmon, and Ali Joseph. Xnor-net: Imagenet classification using binary convolutional neural networks. In ECCV , 2016. 1

Esteban Real, Chen Liang, David So, and Quoc Le. Automl-zero: Evolving machine learning algorithms from scratch. In ICML , 2020. 3

Esteban Real, Chen Liang, David R. So, and Quoc V. Le. Automl-zero: Evolving machine learning algorithms from scratch, 2020. 2

Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. arXiv preprint, arXiv:1506.01497 , 2015. 9

Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang, Carlo Gatta, and Yoshua Bengio. Fitnets: Hints for thin deep nets. In ICLR , 2015. 1, 7

Xiaojie Li Jianlong Wu Fei Wang Chen Qian Shangchen Du, Shan You and Changshui Zhang. Agree to disagree: Adaptive ensemble knowledge distillation in gradient space. In NeurIPS , 2020. 6

Changyong Shu, Yifan Liu, Jianfei Gao, Zheng Yan, and Chunhua Shen. Channel-wise knowledge distillation for dense prediction. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 5311-5320, 2021. 3, 9

Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive representation distillation. In ICLR , 2020. 3, 7, 8

Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Herve Jegou. Training data-efficient image transformers & distillation through attention. pages 10347-10357, 2021. 2

Frederick Tung and Greg Mori. Similarity-preserving knowledge distillation. In ICCV , 2019. 7

Tao Wang, Li Yuan, Xiaopeng Zhang, and Jiashi Feng. Distilling object detectors with finegrained feature imitation. In CVPR , 2019. 6

Wenhai Wang, Enze Xie, Xiang Li, Deng-Ping Fan, Kaitao Song, Ding Liang, Tong Lu, Ping Luo, and Ling Shao. Pyramid vision transformer: A versatile backbone for dense prediction without convolutions. In ICCV . 2

Yukang Wang, Wei Zhou, Tao Jiang, Xiang Bai, and Yongchao Xu. Intra-class feature variation distillation for semantic segmentation. In European Conference on Computer Vision , pages 346-362. Springer, 2020. 9

Zimian Wei, Hengyue Pan, Lujun Li Li, Menglong Lu, Xin Niu, Peijie Dong, and Dongsheng Li. Convformer: Closing the gap between cnn and vision transformers. arXiv preprint arXiv:2209.07738 , 2022. 1

Liu Xiaolong, Li Lujun, Li Chao, and Anbang Yao. Norm: Knowledge distillation via n-to-one representation matching. 2022. 2, 6

Chuanguang Yang, Helong Zhou, Zhulin An, Xue Jiang, Yongjun Xu, and Qian Zhang. Cross-image relational knowledge distillation for semantic segmentation. arXiv preprint arXiv:2204.06986 , 2022. 9

Jie Yang et al. Automatically labeling video data using multi-class active learning. In Proceedings Ninth IEEE international conference on computer vision , pages 516-523. IEEE, 2003. 3

Zhendong Yang, Zhe Li, Xiaohu Jiang, Yuan Gong, Zehuan Yuan, Danpei Zhao, and Chun Yuan. Focal and global knowledge distillation for detectors. arXiv preprint arXiv:2111.11837 , 2021. 9

Zhendong Yang, Zhe Li, Mingqi Shao, Dachuan Shi, Zehuan Yuan, and Chun Yuan. Masked generative distillation. arXiv preprint arXiv:2205.01529 , 2022. 2, 8

Li Yuan, Yunpeng Chen, Tao Wang, Weihao Yu, Yujun Shi, Zi-Hang Jiang, Francis E.H. Tay, Jiashi Feng, and Shuicheng Yan. Tokens-to-token vit: Training vision transformers from scratch on imagenet. In ICCV , 2021. 2

Sergey Zagoruyko and Nikos Komodakis. Paying more attention to attention: Improving the performance of convolutional neural networks via attention transfer. In ICLR , 2017. 7, 8

Linfeng Zhang and Kaisheng Ma. Improve object detection with feature-based knowledge distillation: Towards accurate and efficient detectors. In International Conference on Learning Representations , 2020. 9

Linfeng Zhang, Jiebo Song, Anni Gao, Jingwei Chen, Chenglong Bao, and Kaisheng Ma. Be your own teacher: Improve the performance of convolutional neural networks via self distillation. In ICCV , 2019. 6

Matthew Shunshi Zhang and Bradly Stadie. One-shot pruning of recurrent neural networks by jacobian spectrum evaluation. arXiv preprint arXiv:1912.00120 , 2019. 6

Ying Zhang, Tao Xiang, Timothy M Hospedales, and Huchuan Lu. Deep mutual learning. In CVPR , 2018. 6

Hengshuang Zhao, Jianping Shi, Xiaojuan Qi, Xiaogang Wang, and Jiaya Jia. Pyramid scene parsing network. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 2881-2890, 2017. 9

Helong Zhou, Liangchen Song, Jiajie Chen, Ye Zhou, Guoli Wang, Junsong Yuan, and Qian Zhang. Rethinking soft labels for knowledge distillation: A bias-variance tradeoff perspective. 2021. 2, 4, 5, 6, 7, 10

Chendi Zhu, Lujun Li, Yuli Wu, and Zhengxing Sun. Saswot: Real-time semantic segmentation architecture search without training. In AAAI , 2024. 3

Zimian Zimian Wei, Lujun Li Li, Peijie Dong, Zheng Hui, Anggeng Li, Menglong Lu, Hengyue Pan, and Dongsheng Li. Auto-prox: Training-free vision transformer architecture search via automatic proxy discovery. In AAAI , 2024. 3

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nKnowledge distillation has received significant attention over the last decade and numerous KD approaches have been proposed so far. This paper focuses on automatically learning the best distillation strategy/loss for a given dataset and student/teacher pair. The distiller search space is composed of various transformation ops, distance functions, loss factors, temperature factors and loss calibration functions. The best distiller in this broad search space is searched using an evolutionary search process. To make the search process efficient, strategies such as loss-rejection protocol and search space shrinkage are used. Experiments were conducted on several datasets and the discovered distillation strategy (which varies based on the dataset and teacher-student combinations) is shown to perform better than various existing distillation strategies.\n\nSTRENGTHS:\nThis paper brings AuotML into research area of knowledge distillation and shows that by automatic searching we can find better distillation strategies than many existing KD approaches. I think this is interesting and useful for the community to know.\n\nThorough experimental evaluation: Experiments were conducted on many datasets (recognition, detection, segmentation) and teacher-student network pairs.\n\nWEAKNESSES:\nSome results are confusing: In Table 4, the Auto-KD performance of transformer-based students (last three columns) is significantly better than the teacher model (5-6%). Makes me wonder if the student was finetuned from some pretrained checkpoint instead of training from scratch.\n\nLimited ablations: The proposed search process uses three criteria: test loss, sharpness-gap and CKA-gap. What is the contribution of sharpness-gap and CKA-gap? Ablation studies on this are very limited. Results with only one teacher-student combination on CIFAR100 are presented in the supplementary material (Table 3).  Based on these results, the CKA-gap and sharpness-gap criteria do not seem to be that important (the performance without these is close to the standard deviation range of the performance when they are used).\n\nQUESTIONS:\nPlease see weaknesses section.\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposed to design a search space for the distillation module and apply evolution search to find the optimal combination of KD module on the pre-defined search space. The experiments are conducted are three tasks.\n\nSTRENGTHS:\nThe motivation is good, and presentation is fair.\n\nWEAKNESSES:\nKDzero is a very straightforward method that uses the concept of neural architecture search on KD modules. The novelty is limited. As a result, I expected to see strong performance gain over baselines. Unfortunately, in the current form, the experimental results are weak. \n\n1) only two teacher-student pairs are compared on ImageNet. I suggest the author to include more advanced architecture, such as vit. \n\n2) did not compare with state-of-the-art detection-based KD method. \nThe author only compares a few methods, where most of them are not kd for det. The experiments in Table 6 are insufficient to show the \"scaling knowledge transfer\" ability of KDzero. I believe more advanced det-kd methods should be included, such as GID/LD/Defeat. Also different types of teacher-student pairs as well as detector types, such as anchor-free detectors, should be compared. \n\n3）Experiments on segmentation are only conducted on a small dataset (cityscapes). \nCityscapes is a small dataset. The author should conduct experiments on ADE20K to show the effectiveness of their method. \n\nThere are some other concerns:\n1) No correlation test. The author should provide a correlation test to verify if there is a correlation between proxies, such as cka-gap and sharpness-gap, with the final search results.  \n2）The figures are too small to see, especially figures 8 and 9.\n3) The author claim that KDzero reduced the capacity gap, yet no experiments are provided. CKA value cannot reflect the teacher-student gap. \n4) It is known that the design of search space is critical in NAS. The search space in KD-zeros seems to rely heavily on expert experience, and the overall search space is very similar to state-of-the-art KD methods.\n\nQUESTIONS:\nSee weakness.\n\nLIMITATIONS:\nThe author addressed the limitation.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-05"}

comments: SUMMARY:
This work aims to find the best distiller for different teacher-student pairs to reduce the gap caused by different capabilities and architectures. KD-Zero, the first auto-search framework for distillation was proposed, including search space design, search objectives, and search accelerate techniques. It performs state-of-the-art in multiple datasets and architectures including CNN and ViT.

STRENGTHS:
- First auto-search framework for distillation.
- Rich and solid experiments, and state-of-the-art results.
- Great presentation.

WEAKNESSES:
- Efficiency, every different teacher-student pair need to search distiller, which is quite inefficient. Is there a general distiller across different teacher-student pairs?
- Reason behind the choice of the distiller. Some analysis and practical Guidance for distiller choice are given, but the conclusion is quite shallow. What is the key design for different architecture pair? What differences in the architecture pair make the distiller the best one?
- The search space designs need a broader exploration, such as a line of research on decoupling distillation into target class and non-target class[1], or knowledge distillation with label smoothing[2][3], etc.


[1]arxiv.org/abs/2203.08679 

[2]arxiv.org/abs/2107.00181v1

[3]arxiv.org/abs/2207.12980

QUESTIONS:
The results are clear in general, and I have the following questions： 
- Are there any insights into why the distiller benefits ViT so well? 
- Is there a general distiller across different teacher-student pair?
- Table 5 in the supplement indicates CIFAR100 or Imagenet. What is the time cost of searching in imagenet?

LIMITATIONS:
yes
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-05

comments: SUMMARY:
This paper presents KD-Zero, which utilizes evolutionary search to automatically discover promising distiller from scratch for any teacher-student architectures. The discovered distiller can address the capacity gap and cross-architecture challenges for any teacher-student pairs in the final distillation stage. The authors also  conduct extensive experiments on classification, detection, and segmentation to validate KD-zero's effectiveness.

STRENGTHS:
1) The writings and organizations are good.
2) The decomposition of the generalized distiller makes sense and the authors have conducted ablations with other search algorithm choice.
3) The experiments are comprehensive including various vision tasks.

WEAKNESSES:
Though the author mentioned its efficiency as one of its advantages. I still worry about its efficiency in terms of training. 
Traditional KD forms indeed needs hyper-param tuning but one expert can choose the proper temperature or other hyper-parameters according to the class number and other conditions, thus reducing the tuning laboring. The KD-Zero involves additional searching phase and I found no searching overhead reported for this phase.

QUESTIONS:
1) Is there efficiency comparison with other KD forms?

2) In table 4, some of the 'Auto-KD' performance even surpass the teacher model by large margin. why?

LIMITATIONS:
The author mentioned some limitations in the appendix.
In addition to the training efficiency, I currently find no other limitation for application.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-07