KD-Zero: Evolving Knowledge Distiller for Any Teacher-Student Pairs
Authors: Lujun Li
Peijie Dong
Anggeng Li
Zimian Wei
Yang Ya
Venue: NeurIPS 2023 poster (2023)
Keywords: Knowledge distillation
TL;DR: We present KD-Zero, the first auto-search framework for evolving best distiller from scratch to alleviate teacher-student gaps.

Abstract: Knowledge distillation (KD) has emerged as an effective technique for compressing models that can enhance the lightweight model.  Conventional KD methods propose various designs to allow student model to imitate the teacher better.  However, these handcrafted KD designs heavily rely on expert knowledge and may be sub-optimal for various teacher-student pairs.  In this paper, we present a novel framework, KD-Zero, which utilizes evolutionary search to automatically discover promising distiller  from scratch for any teacher-student architectures.  Specifically, we first decompose the generalized distiller into knowledge transformations, distance functions, and loss weights.  Then,  we construct our distiller search space by selecting advanced operations for these three components.  With sharpness and represent gap as fitting objectives, we evolve candidate populations and generate better distillers by crossover and mutation.  To ensure efficient searching, we employ the loss-rejection protocol, search space shrinkage, and proxy settings during the search process.  In this manner, the discovered distiller can address the capacity gap and cross-architecture challenges for any teacher-student pairs in the final distillation stage.  Comprehensive experiments reveal that KD-Zero consistently outperforms other state-of-the-art methods across diverse architectures on classification, detection, and segmentation tasks.  Noticeably, we provide some practical insights in designing the distiller by analyzing the distiller discovered.  Codes are available in supplementary materials.

KD-Zero: Evolving Knowledge Distiller for Any
Teacher-Student Pairs
Lujun Lif" Peijie Dongi
Anggeng Lit
Zimian Wei Ya Yang
HKUST
HKUST(GZ)) Huawei
NUST
CityU
[lilujunai, dongpeijic98 @ eemail.com/annggengllii.. (awggeng.l...yyaag..... @outlook.com, weizimian 1 6
Abstract
Knowledge distillation (KD) has emerged as an effective technique for compressing models that enhance the lightweight model. Conventional KD methods propose various designs to allow student model to imitate the teacher better. However, these handcrafted KDD designs heavily rely expert knowledge and may be sub-optimal for various teacher-student pairs. In this paper, we present a novel framework, KD-Zero, which utilizes evolutionary search automatically discover promising distiller from scratch for any teacher-student architectures. Specifically, we first decompose the generalized distiller into knowledge transformations, distance fune- tions, and loss weights. Then, we construct our distiller search space by selecting advanced operations for these three components. With sharpness and represent gap fitting objectives, we evolve candidate populations and generate better distillers by crossover and mutation. To ensure efficient searching, we employ the loss- rejection protocol, search space shrinkage, and proxy settings during the search process. In this manner, the discovered distiller can address the capacity gap and cross-archieecture challenges for any teacher-student pairs in the final distillation stage. Comprehensive experiments reveal that KD-Zero consistently outperforms other state-of-the-art methods across diverse architectures classification, detec- tion, and segmentation tasks. Noticeably, we provide some practical insights designing the distiller by analyzing the distiller discovered. Codes available in supplementary materials.
Introduction
Deep Neural Networks (DNNs) have achieved great success in tackling a variety of visual recognition tasks [31, 17, 66]. Despite the appealing performance, the prevailing DNN models usually have large numbers of parameters, leading to heavy costs of mem- ory and computation. Conventional techniques such Neural Architecture Search [16, 6, 26, 15] and quantizing networks to use low-bit parameters [11, 53, 49] have proven be effective for mitigating this computational burden. Recently, Knowledge Distillation (KD) has been widely used for training compact and efficient neural networks by transferring the knowledge lied in the logits [23] or features [57] from a large, pre-trained teacher smaller student.
Teathe
Trandbra
Tranfier
Weldk
Bos
Cad
Recently, although KD has made significant progress in the hand- crafted designs, there are still some limitations to its practical
Figure 1: Illustration of search space in KD-Zero.
*Corresponding author, f Co-first authors, equal contribution.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

N:282
1044
U21
tnct
D50
Andeett
ATS-MBCD
Figure 2: Top-1 gain (%). of WRN-16-2 (left)
Figure 3: Top-1 gain (%). of various student with KD-Zero (left). Various KD methods combined with KD-Zero (right) for ResNet-20 on C-100.
ResNet-20 (right) distilled via various capacities teacher models CIFAR-100.
application for different scenarios from three aspects: (1) Capac-
ity gap problem: Traditional KD methods do not always distill better from stronger teachers because of the large teacher-student gap [27, 46]. Larger and more accurate teacher models tend to be overcon- fident and fail to improve students [79]. To alleviate this, architecture-lett methods use the assistant model [46] or architecture search [45], introducing additional training budgets. Other KD techniques use optimized designs manually modified distillers reduce the disparity. However, these tech- niques rely on expert knowledge and require careful tuning, which suffer from poor generality and efficiency. (2) Cross-architecture issue. Teacher-studen networks with different architectural styles suffer from mismatches in position dependence, receptive fields, and feature dimensions. As shown in Figure 2, while handcrafted methods (e.g., KD [23], DIST [27, 46) and WSLD (79]) provide slight gains in some cases, they still are weak in most cross- aross-architecture pairs 2, (3) Manual tuning difficulties. Different choices of knowledge sources, distillation funetions, and hyperparameters can largely affect the performance of KD. However, designing and training a proper distillation setting requires trial and error, substantial effort, and experiments. Thus, two questions are raised: (1) How to efficiently discover the optimal distillation strategies without expert knowledge? (2) How to reduce the teacher-student gap with different capabilities and architectures?
"We may hope that machines will eventually compete with men in all purely intellectual fields." fields. -Alan Turing
As this well-said quote goes, recently, machine learning approaches have successfully replaced human experts in algorithm design, architecture search, and drug discovery. For the first problem, inspired by Auto-Zero [55], we decide to automate the process of designing the distillation functions for the first time in the field of knowledge distillation. For the second question, we systematically review previous studies of KD design for distillation gaps. Our observations reveal that: (1) Most of the distillers can be divided into three components with various key elements: knowledge transformation, distance (2) Some essential operations, like normalize ops [3] and mask ops [7] J,, play important roles in reducing the geacher-studens gap. (3) Optimal distillation functions can be effectively integrated with additional strategies (e.g., feature aggregation [50], projector ensemble [9], N-to-One match [67], and cross-layer mapping [28, 13] in Figure 3).
function, and loss weights.
Based on the above analysis, we present KD-zero, an automated search framework that utilizes evolutionary algorithms from scratch to efficiently discover the best distiller without manual design. Specifically, our framework is organized into three parts: search space, search algorithm, and acceleration strategy. Firstly, we establish the search space with basic transform operations, distance functions, and loss weights (see Figure 1). For example, we select operators normalized in different dimensions (e.g.. batchnorm, norm (W.CN), various types of activation functions (e.g., exp, relu, wwwpw/wwwnseelwwweeee
tanh, sigmoid, pow2), multi-scale process and advanced operations the knowledge transformation. Our distance function options include smooth L,. L1, l,, LKL. hhard, &Cosines f Peareon and &Correlation distance. Options in the loss weight part include various values for loss factors, temperature factors, and weight calibration strategies. way, our search space includes over 10,000 candidates covering the existing SOTA KD methods and designs. Then, we construct a calculation graph for these candidates and use selected features, representations, and logits from the teacher-student network input. Based graph structure, we
mask transforms and other
this
'In our paper. W40-2, R32x4, R8x4, MV2, SVI, and SV2 stand for WRN-40-2 ResNet32x4, ResNet8 Mob,leNetV2, ShuffleNetVI. and Shuf Ti [61] T2T-ViT-7 [72], PVT-Ti [64].

---

initialize the candidate total groups from the search space, crossover, and mutate them according to the evaluation results. We employ loss-rejection protocol and search space shrinkage for search efficiency to filter out weak candidates. With early-stop proxy settings, we achieve at least x acceleration during the distiller search. For distillation gap reduction, we take the representation gap and sharpness gap between teacher-student as the fitting objectives besides the accuracy metric of the validation set. Finally, we distill student architectures with discovered distiller, and our KD-Zero surpasses existing KD approaches by a large margin without prior knowledge (see Figure 3).
In principle, our KD-Zero differs from previous hand-designed KD methods. opening new doors to automated distillation designs. Its merits can be highlighted in three aspects: (1) Effective. KD-Zero effectively reduces the teacher-student gap by presenting a general distiller search space and adaptive evolutionary search for different teacher-studens pairs. KD-Zero extends the KD formulation and allows for additional gains with extra design besides distillers. In addition, it reduces human bias and ensures that the resulting distillers are optimized for the target problem or dataset. (2) Efficient. KD-Zero increases efficiency in practice by a series of flexible, systematic, and efficient search procedures without additional laborious tuning. By contrast, other manual methods with fixed KD forms involve complex parameter tuning with additional training time and resources. (3) Insightful. KD-Zero undertakes an in-depth analysis of the existing advanced distillation designs, with the aim of exploring their potential combination to produce numerous novel distillers. KD-Zero provides guidelines for practical applications and develops a new research direction. We hope our efforts on the automated design of distillers could facilitate future research for automated KD works to some extent. In summary, our contributions are:
To alleviate architecture & capability gaps of teacher-student, we present KD-Zero, the first auto-search framework for evolving best distillers from scratch to our best knowledge.
We present a comprehensive distiller search space, including advanced operations on transforma- tions, distance functions, and loss weights. Then, we evolve the distiller search with performance and sharpness & represent the gap as fitting objectives. In addition, We achieve significant search acceleration via loss-rejection protocol & space shrinkage, and proxy settings.
We conduct extensive experiments on classification, detection, and segmentation. KD-Zero performs state-of-the the-art in multiple datasets and architectures (e.g., CNN and vision trans- former). Specifically, ResNet-18 18 and MobileNet with KD-Zer achieve 72.17% and 73.02% Top-1 accuracy on ImageNet, outperforming KD by 1.51%, 2.34%, respectively.
2 Related Work
Knowledge Distillation. The idea of teacher-student learning is first proposed in pioneering ex- plorations [1, 2], and the formal definition is defined by the original KD [23]. Subsequent efforts explore on different knowledge (e.g., intermediate feature representations [38, 43, 33, 36), sample relationships [48, 60]) and applications [18, 14, 35]. Compared to KD for distillation gap. Previous methods propose assistant teachers, architecture search, KD designs on transformations [27], distance functions [59], and weight-tuning [42, 34] for this problem. However, such KD designs rely on expert knowledge and tuning, and their performance can fluctuate significantly across different situa- tions. In contrast to these methods, KD-Zero develops automated searches for distillers to address these difficulties that do not require additional architecture modification and manual KD design. Compared to Meta-KDs. These works [13, 42] only focus hyperparameter tuning and involve complex optimization challenges. In contrast, our approach searches for complementary distiller design besides hyperparameters establishing new paradigm for KD research and application.
Automated Machine Learning. AutoML [81, 80] aims automate Network Architecture Search (NAS) and HyperP Hlyeeraarmmeter Optimization (HPO), making them more accessible to non-experts. NAS chooses architecture rather than KD designs. Compared HPOs [54, 69], they generally focus on the hyperparameters on training configurations. Recent methods search for loss formulation [39, 32). In contrast to these methods, we present a new complex search space for distiller design transforms, distances, loss weights, and new search objectives and accelerations according to the KD task.
3

---

Distiller search space
Initial population
Evaluations
Furarr
Festare
Logis
Validale anc & loss
Scale
Sude
Siae
Introgy
Shappnes gep
Input
Transocm
Prinsn
Distance
Weight
No.T:1 W-3
Represcnt- gap
Crossover
Mutation
Best Distiller
Loss rejection
Rack
Jalu
Erp
Exp
Protocol
Space Shrinkage
S
NoT:1N63
cat
Distillation
Next population
Evolution
Figure 4: Overview of KD-Zero. In the search phase, we first randomly sample candidate distillers initialize the population and evaluate their validation performance and sharpness & represent the gap between teacher-student. Then, we perform loss-rejection protocol & space shrinkage to remove weak individuals and crossover & mutation generate new populations within promising ones. Finally, we pick up the best-performing distiller for distillation.
Table Specific operations in KD-Zero. More details of their formulas are available the Appendix.
norm-based; batchnorm. min maz, norm.w.C.N. aof.s.ooffmeee fmaa HW.C. logso tmaa SW.C. activation based: eap, mish, leaky, relu, tanh, sigmoid, pow2, pow4, log, sgrt scale-based: scale, multi scale, scale, localei. r2.44 botch, chanmel attention&mask mask-basdd: drop, satt, natt, catt, mask, other: no, bmm, mm
Transform
Distance
no-norm loss: smooth E, &1, &2, EK I.n Enard' norm loss: ECosinns E Peorson: ECorrdatiin
Weight
calibration: entropy. Focal, simn, con no: weight values: 0.OI..... 100; 7 values:1.4.
3 Methodology
In this section, we first illustrate the design of our distiller search space, the search process, the acceleration, and the fitting objectives. Then we analyze the search results and give some guidelines. Finally, we analyze the student distilled via KD-Zero and expansion for different distillation scenarios. The pipeline of our approach is shown in Figure 4.
3.1 Search Space for Distillers Discovery
Search space structure. In KD, the student student S distilled with the fixed teacher T by minimizing:
LKD= = x W2 x Wea x D(T(s//),T((x//))
(1)
where Wy and Woat is the loss weights factor and calibration [79], 7 is the temperature factor, T transformations, ().,)) )) is distance function measuring the knowledge difference. fr and fs outputs (c.g., features, embeddings, and logits) of the seacher-studen Following this general KD formulation, our search space consists of different types of operations (see Table I) in transformations, distance functions, and loss weights parts. Then, we use a computation graph to represent each candidate, in which the input nodes different types of knowledge and the intermediate nodes are primitive operations. In addition, we assign three transform options as Sanefformn--taasfton 2 ---anssform-3 for the transformation part ensure effective processing of the input knowledge.
Insight space design. Our space design enjoys multiple merits. (1) Comprehensive & flexible: KD-Zero contains key elements of most existing KD methods, including the normalize, mask,

---

attention, and cacll-calio KDs [3. 79, for distillation gaps, In addition, we introduce basic operations in loss design to improve the flexibility and diversity the search. (2) Extended & Innovative: We extend advanced KD design for omni-dimensional and various knowledge. For example, we expand the normalized operation for batch, channel, and spatial dimensions, which can be used for logits, embeddings, and feature knowledge. Thus, such a unified and flexible search space would provide some inspiration for KD design.
3.2 Evolution Procedure, Acceleration and Objective
Evolution process. Our EA starts with an initial population of candidate distillers evaluated for their fitness based on our distillation gap-related multi-objectives. The algorithm then iteratively evolves the population over generations using genetic operators, such as selection, crossover, and mutation, to generate better solutions. Specifically, distillers are first randomly generated to form the initial population. Each candidate solution the population is evaluated using multiple fitness functions that measure teacher-student gaps. Based on these evaluations, from the population to create a new population for the next generation. Then, we apply crossover to the selected individuals to create new offssring and use mutation to the new offspring to introduce diversity into the population, which helps to explore new distillers.
select the best best-performing individuals
Search acceleration As the search space is sparse with many unpromising distillers, we employ several strategies to accelerate: (1) Loss-rejection protocol. We filter out candidates with excessive loss values or collapsed optimization during the search. (2) Search space shrinkage. We reduce sampling probabilities for the operations frequently in loss rejection and tail candidates with search iterations. (3) Proxy settings. With diverse and informative knowledge learned from a teacher, student models offer advantages in terms of faster training speeds. Based on these properties, we employ early stop the training process once the student model performs well enough to determine the quality of the candidate distillation. Nevertheless, proxy settings can also introduce evaluation uncertainties, and we alleviate this issue by introducing multiple distillation gap metries in the following sections.
Fitting objectives. To accurately evaluate each distiller and reduce the distillation gap, we include cross-entropy loss, sharpness-gap [20] on prediction, and CKA-gap [52] on representation between teacher-student as the multi-objectives. Specifically, we conduct a waining-frree evolutionary search algorithm to efficiently discover the optimal distiller a* from search space A, as:
сКА gap
Sharpness- -gap
HSIC(fs,
a" arg min (Cn Y) (log(exp(fs)) log(exp((T))) aEA
HSSCCfs: HHSCCH fr)
where LCg is the regular cross-entropy objective with labels Y, the sharpness metric is the logarithm of the exponential sum of logits, and Centered Kernel Alignment (CKA) metric is normalized from Hilbert-Schnne Schmidt Independence Criterion (HSIC) [19] on high-level feature.
Figure 5: Probability of operations within each search part, which counted from the Top-3 searched distillers for all teacher-student pairs in the CIFAR-I00 100 experiment.
3.3 Results Analysis and Practical Guidance
Figure 5 present searched distillers for different models. Based on these results, some practical guidance for KD designs can be summarized as follows:
For knowledge input, feature knowledge enjoys superiority while logits and embedded knowl- edge share identical status. Searched distillers with feature knowledge take more proportion, and ablation studies vamilla feature knowledge also surpass logits and embedding knowledge.
5

---

This observation also aligns with the existing conclusion that feature-based KD outperforms other KD on detection tasks [27, 63].
For transformations, the normalized- operations play key role in most optimal distillers, and its vanilla performance outperforms other types of transformations in ablation studies. These observations illustrate that normalization benefits distillation, consistent with the current KD methods. Scale-based and activation-base operations are also crucial for the KD because they are often present in optimal distillers and enjoy good vanilla performance in ablation studies.
For distance functions, normalized- baaee distances such as & Pearson and correlation enjoy better results ablation studies and are often adopted by teacher-student pairs across architectures. This suggests that these normalized-basse scenarios. In addition, some simple distance functions (e.g., &1, l2 and k also appear in the optimal distiller. These may be attributed these distillers employing advanced transformation operations for input knowledge.
distances can reduce distillation gaps in complex
For loss weights, adjusting the temperature values is helpful for different scenarios, and the optimal weight values generally between and 10 based search results and ablation studies. In addition, the focal weight calibration outperforms the no- no-weight calibration in ablation studies, and it is often used for some optimal distillers. In summary, we should employ smaller weight values and actively explore different weight calibration and temperature values to reduce distillation gaps under different teacher-student pairs.
3.4 Distilling Student via Discovered KD-Zero functions
After search, the discovered distiller CKD is used for distilling student fs combined with cross- entropy LCe via label Y in single teacher fr or multiple teacher fr. KD as:
Linngl LCe(fs, LxD(fs, Cxxtsss Lmultiple CCE Y) + (T.) (3)
Extended to various distillation designs & scenarios KD-Zero focuses on the distiller's search and employs simple 1 Ix 1-Conv for channel alignment. Recently, some KD methods have proposed other designs in feature aggregation [50]. projector ensemble [9]. N-to-One match [67], and cross- layer mapping [28, 13J. As shown in Table 2, KD>-Zero can combine well with them by replacing their default fzlExz losses. addition, the distiller search of KD-Zero also benefits different distillation scenarios. Specifically, Self-KDs, online KDs, and multi-teacher KDs with KD-Zero achieve extra gains than the default settings. Also, KD-Zero can combine architecture-level methods (e.g., Assistant [46]) to reduce distillation gaps further.
Table 2: Top-1 (%) accuracy of KD-Zero combined with different KD designs and scenarios for ResNet-20 and WRN-16-2 on CIFAR-100.
design
Didivent RD acrounio
Review (50)
PR [0]
NORM (67]
1.2T-w (28)
ONF (26)
RYOT [75]
DMI (27)
AVE-MKD (58)
AF-MKD (58)
Assictur (46)
6
71.39
21.30
71.35
70.89
30.77
70.37
70.92
71.24
71.36
71.06
KD Zero
72.27
72.08
72.00
22.00
31.38
70.88
12.00
72.22
72.35
71,89
76.20
76.02
75.05
75.30
74.25
7412
75.33
75.21
75.65
75.35
W-16-2
KD-Zeso
76.62
26.47
76.26
76,36
75.65
75.21
76.45
76.72
76.75
76,15
Why can KD-Zero bridge the teacher-student gap? The answer intuitive: In KD-Zero, the search space contains many operators for distillation gap reduction, and search objectives are directly designed to reduce the prediction and representation gap between teacher-student. For example, We approximate the sharpness gap using Taylor second expansion [20]:
G Ggap = log(exp(fr)) - log(exp((ss) = log 1 + fr
2
/ log 1 fs
2 13
(4)
Following Hindon's assumption [23] that the logits of each training sample are approximately zero-meaned, i.e., fr. fs
0. So the gap can be rewritten as log [ (ar(fr))
log (1 + Var(ss)) [20]. Our Our options like normalized transform [3,27], and weight calibrations [79] can effectively reduce the variance of teacher-studens' outputs and minimize the distillation gap. In addition, the student models distilled with KD-zero enjoy the merits: (1) Fast convergence and superior performance. As shown in Figure 6, KD-Zero has surpassed the best accuracies of KD

---

MM
1000m
Figure 6: Comparison of training curves (left) of ResNet-20 on CIFAR-100 and ResNet-18 on ImageNet (right).
Figure 7: Penultimate-lus layer visualization (left), logits-correlation map teacher-student (ResNet- 10020) via KD (middle) & KD KD-Zero (right).
Input
Student
KD
KD-Zero
Teucher
Input
Student
KD
KD-Zero
Teucher
Figure 8: Comparison on the Grad-CAM++ [4] visualization results between the features of the baseline student model, the student model trained with KD and our KD-Zero, and the teacher model. Results are obtained on ImageNet with R (left) and MobileNet (right).
and baseline the beginning of the 3rd learning-rate decay stage. (2) Smaller teacher-student gap. As shown in Figure 7, features and logits of students via KD-Zero have stronger similarities with teachers than the original KD) method.
Experiments
In this section, we assess the efficacy of our proposed KI>-Zero approach classification, detection, and segmentation tasks. Additionally, we compare its performance with other KD methods, ensuring fair comparisons by utilizing the same training settings. We report mean results based more than repeated trials. More detailed experiment settings and results are available in the Appendix.
Table 3: Comparison of results on CIFAR-100. Most results other methods refer to the original papers [8, 60]. Gain refers to the performance gain than baseline. We report Top-1 "mean (std)" accuracies (%) for KD-Zero over runs.
Model
Same architectuual style
Different architectural style
Teacher
W-40-2
R110
R32xx4
VGG13
VGGIA
R32x44
W-401-2
R56
R56
R56
SNetV1
T2T
Techer
75.61
74.
74.31
79.42
74
74.64
79.42
75.61
72.
72.34
72.34
Student
73.26
69:06
71.14
72.50
70.36
64.60
71.82
70.50
65.08
69.37
69.22
FilNet
74.08
73.51
SP [62]
73,83
70.4
72.69
72.94
72.68
66.30
74.56
74,52
67.36
72.26
70.48
RKD [48]
73.35
69
71.82
71.90
71.48
64.52
73.21
72.21
70.39
71.88
73.63
CRD
75.45
71.46
73.48
75.51
73.94
69.73
75.65
76.05
NA
NA
NA
SRRI [29]
KD (23)
74.92
70.67
73.08
73.33
72.98
67.37
74.45
74,83
73.25
74.15
73.68
DIST [27]
75.35
71.68
73.86
75.79
73.86
69.17
76.08
75,85
72,56
73.86
73.52
WSLD [79]
75.30
71.53
73.36
71.79
74.36
68.79
75.93
75.09
71,56
75.28
74.39
KD-Zero
76.42
72.05
74.19
77.85
75.26
70.42
77.45
77.52
78.25
78.32
77.22
Gain
3.16
2.99.0.31
3.05
5.35
4.90
j.
5.63
7.02
13.17
8.95
8.00
8.00
4.1 Experiments on CIFAR-100
Implementation. We utilize the CIFAR-100 dataset [30] knowledge distillation. During the distiller search phase, we apply 5% early-stopping training epochs with full training data for accel- eration settings. Our evolutionary algorithm with 20 population sizes performs 100 iterations for each teacher-student pair. During the distillation phase, all teacher-student networks are trained using typical training settings, with a training epoch of 240. The multi-step learning rate commences at 0. which decays by 0.1 at 100 and 150 epochs. Recently, knowledge distillation enabled training Vision Transformers (ViT) from scratch with CNNs as teachers. To evaluate the effectiveness of KD-Zero, conduct the evolutionary search for ViT-based distillation strategies with the same settings the

---

Table 4: Comparison of results ImageNet. The results of other methods quote the original papers
report [8, 60, 71]. We report Top-1 "mean (std)" accuracies (%) for KD-Zero.
Teacher
Stadent
Ace.
Teacher
Stadent
KD KD]23]
AT [:3]
OFD [22]
SRRL [29]
CRD [c0]
Review [50]
MGD [71]
KD-Zer
Top-1
73.40
69.75
70.66
70.69
70.81
71.73
71.17
71.61
71.38
72.17-0
ResNet-34
18
Top-5
91.42
89.88
Top 76.16
70.13
70.68
70.72
71.25
72.19
71.37
72.36
72.35
ResNet-50
MobileNet
Top-5
92.86
89.49
90.30
90.03
90.34
90.92
90.41
91.00
90.71
91.05-+:a0
CNN experiment. Subsequently, we train the ViT with the optimal distiller obtained and ResNet-56 as CNN teacher. The training is conducted on 224 x 224 resolution images for 300 epochs, with an initial learning rate of 5e-4 and a weight decay 0.05 using the Adam optimizer.
Comparison results on CNN models. Table 3 presents a comparative analysis of our KD-Zero with other state-of-ther -hrt (SOTA) KD methods. We conduct multiple trials with randomly selected distillers in the same search space, called Rand-KD, to evaluate the efficacy of our EA search. For teacher-student pairs with the same architectural style, KD-Zero outperforms the baselines by margins ranging from 3. 1.16% ~ 4.90%. Compared with Rand-KD and other KDs, KD-Zero obtains consistent performance gains (1.0% ~ 3.2%). Besides strengths in the same architecture pairs, KD-Zero exhibits even stronger performance when dealing with different architectural styles, while other KD methods suffer from noticeable accuracy reductions. Specifically, KD-Zero outperforms the baseline by margins of 5.6% 7.3% and the random search results by margins of 1.9% ~ 2.3%, demonstrating the effectiveness of our design for different structures. Compared with other SOTA KD methods, KD-Zere achieves 1.2% 1.5% gains. These results show that KD-Zero can improve each student model with simple settings under different teacher-student pairs.
Comparison results on vision transformer. Table 3 presents the results of the vanilla and distillation models employing different distillation methods. The results indicate that KD-Zero can significantly improve the performance of vision transformers with 8.0% 13.1% margins and consistently yields superior performance than other methods. In addition, our proposed method applies to various VIT architectures, thereby validating its effectiveness. Note that most VIT students possess larger model sizes (e.g., DeiT-Ti with million parameters) and greater capabilities than the CNN teacher (e.g., ResNet-56 with only 0.86 million parameters). Some VIT students outperform the CNN model in the strong regularization setting on large-scale datasets. However, when it comes to VIT distillation on small datasets, employing CNN teachers helps address the issue of ViT models struggling to train effectively from scratch. In this context, using CNN teachers in distillation akin to auxiliary training providing additional regularization supervision. As result, these ViTs demonstrate their original strong representation capability after distillation and consequently outperform the CNN teacher.
4.2 Experiments on ImageNet
Implementation. We additionally conduct experiments on the ImageNet[12] Following CIFAR- 100 trials, we employ similar EA settings on a subset of ImageNet for search acceleration. Then, we utilize the discovered distiller for the training of student models (e.g.. ResNet- 18 [21] and MobileNet [25]). The training settings are the same as the other KD methods and involve training for 100 epochs using multi-step learning rate, which commences at 0.1 and decays by 0.1 at 30, 60, and 90 epochs.
Comparison results. As shown Table 4, our proposed KD-Zero significantly improves the accuracy of baseline models, yielding gains of 2.5% ~ 2.9% in Top-1 accuracy for ResNet- 18 and MobileNet, respectively. In addition, KD-Zero surpasses other SOTA methods with clear gains, demonstrating its superiority in large-scale datasets. These findings substantiate the effectiveness of KD-Zero in distillation optimization with considerable benefits, establishing the versatility and potency of our framework. In summary, KD-Zero facilitates substantially improved predictive accuracy of student models on ImageNet, more complex domains while preserving superior performance.
Visualizations. The comparison between the Grad-CAM++ maps generated by the student model trained with KD-Zero and other methods presented in Figure 8. The results indicate that the Grad-CAM++ map generated by the student model trained with KD-Zero is more similar to that of the teacher model compared to the student model trained independently. In contrast, independent training of the student model leads incorrect focus areas. These findings suggest that the KD-Zen
8

---

Table The results on the COCO val dataset for teacher (T) and student (S) models.
Table Results on Cityscapes val
dataset with ImageNet Pretrain.
Method
AP APs0 APrs
APs
APM
APr.
Method
mloU (%)
Two-stage detectors
DeepLabV3-R101
78.07
T: Cascade Mask RCNN-XIOI
45.6 64.1
49.7
26.2
49.6
60.0
S: Deepl.abV3-R18
74.21
S: Faster RCNN-RR -550
38.4
59.0
42.0
21.5
42.1
50.3
SKD [44]
75.42
[24]
39.7
61.2
43.0
23.2
43.3
51.7
IFVD [65]
75.59
FKD [74]
41.5 62.2
45.1
23.5
45.0
55.3
CWD [59]
75.55
CWD [59]
41.7 62.0
45.5
23.3
45.5
55.5
CIRKD [68]
76.38
DIST [27]
40.4 61.7
43.8
23.9
44.6
52.6
KD-Zero
41.9 62.7
45.5
23.6
45.6
55.6
DIST [27]
77.10
One-stage detectors
KD KD-Zero
77.38
T: RetinaNet-XX X101
41.0 60.9
44.0
23,9
45.2
54.0
S: PSPNet-RI8
72.55
S: RetinaNet-R50
37.4 56.7
39.6
20.0
40.7
49.7
SKD [44]
73.29
KD [24]*
37.2 56.5
39.3
20.4
40.4
49.5
IFVD [65]
73.71
FKD [74]
39.6
42.1
43.3
52.5
CWD [59]
74.36
FGD [70]
40.4 59.9
43.3
23.4
44.7
54.1
CIRKD [68]
74.73
DIST [27]
39.8 59.5
42.5
22.0
43.7
53.0
KD-Zero
76.25
KD-Zero
40.9 60.4
43.5
23.2
45.2
54.8
Comparison of search algorithms (left Figure 9: Ablation study of search space (left) Figure 10:
and organization (right) of ResNet-20 on C-100. and correlation visualization (right) of ResNet-20.
approach more effective than traditional knowledge distillation methods in guiding the student model to learn from the teacher model, resulting in improved model interpretability and performance.
4.3 Experiments on Object Detection & Semantic Segmentation
Object detection. We conduct experiments on the MS-COCO [ata] We use the optimal distiller on ImageNet distill knowledge from teacher detectors students. Based on the strong baseline [5], we apply KD-Zero to two-stage detector (e.g., Faster R-CNN [56]) and the one-stage detector (e.g., RetinaNet [40]), which are widely used object detection frameworks. Following common practice [40], all models are trained with 2x learning schedule (24 epochs). We train all the models with SGD optimizer, where the momentum 0.9, and the weight decay is 0.0001. As shown in Table 5, our KD-Zero improves the AP by 3.5 on RetinaNet and 3.4 on Faster R-CNN, respectively, outperforms previous state-of-the-an techniques, including [59, 70, 74], both object detectors. The results substantiate the potential of KD-Zero for scaling knowledge transfer to broader datasets and more complex computer vision problems while preserving improved accuracy.
Semantic segmentation. We evaluate KD-Zero on Cityscapes dataset[ 10]. Following the previous work, adopt PSPNet-ResNetIU [78, 51] as the teacher and PSPNet and DecpLabV3[7] models with the ResNet18 backbone as the student. During distillation, the batch size is 8, and the models are trained for 40K iterations with the SGD optimizer, where the momentum is 0.9 and the weight decay is 0.0005. The results are reported with mean intersectionooee-Unneee evaluation setting. shown Table 6, student PSPNet and DeepLabb get 3.17 and 3.7 mloU improvement by adding our KD-Zero loss. These results indicate that our method surpasses the state-of-the-an distillation method for semantic segmentation, demonstrating that searched distillers facilitate student learning.
(mloU) under the single-scale

---

Table 7: Top-1 (%) aceuracy of ResNet20 (baseline 69.06%) with different teachers models on
CIFAR-100 100. Note that the distiller is searched with ResNet1 10 as the teacher.
RT10
VGG13
W-28-2
W-40-2
W-16-4
W-28-4
R50
R32x4
Teacher
74.31
74.64
75.45
75.61
77.51
78.60
79.34
79.42
KD
70.8910.13
70.0710.23
70.83
70.35
70.2210.18
70.59
70.1220.0s
70.37_o.11
DIST
69.8219.1
70.1010.2:
70.422n.13
20.05
70.35
69.8810.21
69.79_m.a1 20
WSLD
70.89+0.17
70.12+0.15
71.19-0.25
70.85+0.29
70.86+0.2
70.39+0.16
69.48-0.24 20.62+0.22
KD-Zero
72.05a0.11
71.21
72.46=0.05
71.76=0.16
72.37
71.55+0.11.
71.74a0.17
70.98=0.44
Table 8: Top-1 (%) accuracy of WRN-16-2 (baseline 73.26%) with different teachers models on
CIFAR- 100. Note that the distiller searched with WRN-40-2 as the teacher.
RI10
VGG13
W-28-2
W-40-2
W-I6-4
W-28-4
R50
R32x4
Teacher
74.31
74.64
75.45
75.61
77.51
78.60
79.34
79.42
KD
74.0940.28
24.21
74.20=0.a6
74.49=0.18
74.61
74.03+0.12
74.67=0.1
74.57=0.12
DIST
74.35+0.11
74.86+0.18
75.34=0.22
74.92=0.15
75.79+0.17
75,04+0.18
75.36=0.22
74.39=0.25
WSLD
74.9310.28
74.8810.1
75.44
75.302o.15
76.2910.2
75.5810
75.4210.2
74.86_0-20
KD-Zero
75.50
75.45+0.15
76.05
76.42
76.50+a.19
76.40
76.02+0.16
75.86=0.15
4.4 Ablation Studies
Search space organization. We individually explore various operation types and organized settings in Figure 9. Results indicate: (1) some operations like the normalized transforms, normalized- based losses, focal correction, and seigl-weighted values bring superior gains than others. (2) our design is optimal across different settings, and the removal of transform, weight factor, and temperature factor search leads worse results, proving our organization's 's strength.
Search algorithm. We use the EA for distiller search, which gradient-free and flexible for non- convex optimization. As shown in Figure 10, the EA obtains faster convergence and better final search results than random search. In addition, for fitting objectives in EA, Shapeness and CKA metrics demonstrate good correlations with the final distillation accuracy and combine well with cross-cntropy loss to improve evaluation quality.
Generalization of searched distillers for different Teachers. In Table 8 & 7, we present detailed results of the same searched distiller with various capacities teacher models on CIFAR- 100. The results indicate that our KD-Zero can significantly improve the student model in different teacher models and surpass the KD and improved KD methods (e.g., DIST [27], WSLD [79]).
In addition, we provide combined trials in Table and Figure 5, student training and teacher-student gap analysis in Figure 6 & 7. These ablation studies demonstrate that our KD-Zero well alleviate distillation gaps and combine with other KD designs.
5 Conclusion
In this paper, introduce the KD-Zero framework, innovative framework for automatically designing distillers. We construct a flexible and unified distiller search space with advanced operators on knowledge transformation, distance function, and loss weight parts. Then, we employ evolutionary search with distillation-gan gap reduction as objectives and search acceleration strategies, enabling KD- Zero efficiently find distillers and improve student model performance in distillation training. We employ the loss-rejection protocol, search space shrinkage, and proxy settings during the search process enhance search efficiency. Furthermore, we present some practical guidelines based on the results analysis and propose various extensions of KD-Zero to cater to different KD designs and scenarios. Comprehensive experiments three benchmarks demonstrate the effectiveness and universality of the KD-Zero framework for various CNNs, Vision Transformer models, object detection, and semantic segmentation. In future work, we will extend KD-Zero more tasks and explore large language models as code encoders or searcher for KD-Zero. We hope this work will inspire future research knowledge distillation designs.
10

---

References
[1] Lei Jimmy Ba and Rich Caruana. Do deep nets really need to be deep? In NIPS, 2014. 3
[2] Cristian Bucila, Rich Caruana, and Alexandru Niculescu-Mizil. Model compression. In KDD, 2006.
[3] Weihan Cao, Yifan Zhang, Jianfei Gao, Anda Cheng, Ke Cheng, and Jian Cheng. Pkd: General distillation framework for object detectors via pearson correlation coefficient. In NeurlPS, 2022.
2, 5,
[4] A. Chattopadhyay, Anirban Sarkar, Prantik Howlader, and V. Balasubramanian. Grad-cam++: Improved visual explanations for deep convolutional networks. In WACV, 2018. 7
[5] Kai Chen, Jiaqi Wang, Jiangmiao Pang, Yuhang Cao, Yu Xiong, Xiaoxiao Li, Shuyang Sun, Wansen Feng, Ziwei Liu, Jiarui Xu, Zheng Zhang, Dazhi Cheng, Chenchen Zhu, Tianheng Cheng, Qijie Zhao, Buyu Li, Xin Lu, Rui Zhu, Yue Wu, Jifeng Dai, Jingdong Wang, Jianping Shi, Wanli Ouyang, Chen Change Loy, and Dahua Lin. MMDetection: Open mmlab detection toolbox and benchmark. arXiv preprint 0tti:/996.001//0 2019.
[6] Kunlong Chen, Liu Yang, Yitian Chen, Kunjin Chen, Yidan Xu, and Lujun Li. Gp-nas-ensemble: a model for the nas performance prediction.
[7] Liang-Chich Chen, George Papandreou, Florian Schroff, and Hartwig Adam. Rethinking atrous convolution for semantic image segmentation. arXiv preprint vrXiv:1706.05587, 2017.
[8] Pengguang Chen, Shu Liu, Hengshuang Zhao, and Jiaya Jia. Distilling knowledge via knowledge review. In CVPR, 2021. 8
[9] Yudong Chen, Sen Wang, Jiajun Liu, Xuwei Xu, Frank de Hoog, and Zi Huang. Improved feature distillation via projector ensemble. In NewrIPS, 2022. 2,
[10] Marius Cordts, Mohamed Omran, Sehastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The cityscapes dataset for semantic urban scene understanding. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3213-3223, 2016. 9
[11] Matthieu Courbariaux, Yoshua Bengio, and Jean-Pierre David. Binaryconnect: Training deep neural networks with binary weights during propagations. In NIPS, 2015. 1
[12] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. pages 248-255 2009. 8
[13] Xueqing Deng, Dawei Sun. Shawn Newsam, and Peng Wang. Distpro: Searching a fast knowledge distillation process via meta optimization. 2022. 3,6
[14] Peijie Dong, Lujun Li, and Zimian Wei. Diswot: Student architecture search for distillation without training. In CVPR, 2023. 3
[15] Peijie Dong, Xin Niu. Lujun Li, Zhiliang Tian, Xiaodong Wang, Zimian Wei, Hengyue Pan, and Dongsheng Li. Rd-nas: Enhancing one-shot supernet ranking ability via ranking distillation from zero-cost proxies. arXiv preprint 0xXi:22301.098500 2023. 1
[16] Peijie Dong, Xin Niu, Lujun Li, Linzhen Xie, Wenbin Zou, Tian Ye, Zimian Wei, and Hengyue Pan. Prior-guides one-shot neural architecture search. arXiv preprint arXiv:2206.13329 2022.
[17] Peijie Dong, Xin Niu, Zhiliang Tian, Lujun Li. Xiaodong Wang. Zimian Wei, Hengyue Pan, and Dongsheng Li. Progressive meta-pooling learning for lightweight image classification model. In ICASSP, 2023.
[18] Nuno Garcia, Pietro Morerio, and Vittorio Murino. Modality distillation with multiple stream networks for action recognition. In ECCV, 2018. 3

---

[19] Arthur Gretton, Olivier Bousquet, Alex Smola, and Bernhard Schölkopf. Measuring statistical dependence with theory, pages 63-77, 2005. 5
norms. In International conference on algorithmic learning
[20] Jia Guo. Reducing the teacher-student gap via adaptive temperatures. 2021. 5, 6
[21] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. pages 770-778, 2016.
[22] Byeongho Heo, Jeesoo Kim, Sangdoo Yun, Hyojin Park, Nojun Kwak, and Jin Young Choi. A comprehensive overhaul of feature distillation. In ICCV, 2019. 8
[23] Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. arXiv preprint
2015. 2, 3, 6, 8
[24] Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. arXiv preprint arXiv: 2015. 9
[25] Andrew G. Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam. Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint, arXiv: 04861, 2017. 8
[26] Yiming Hu, Xingang Wang, Lujun Li, and Qingyi Gu. Improving one-shot nas with shrinking and-expanding supernet. Pattern Recognition, 2021.
[27] Tao Huang, Shan You, Fei Wang, Chen Qian, and Chang Xu. Knowledge distillation from stronger teacher. arXiv preprint arXiv:2205. 10536, 2022. 2, 3, 5, 6, 7, 9, 10
[28] Yunhun Jang, Hankook Lee, Sung Ju Hwang, and Jinwoo Shin. Learning what and where to transfer. In ICML, 2019. 2,
[29] Adrian Bulat Georgios Tzimiropoulos Jing Yang, Brais Martinez. Knowledge distillation via softmax regression representation learning. In ICLR2021, 2021. 7,
[30] Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. Tech Report, 2009.
[31] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. In NIPS, 2012.
[32] Hao Li, Tianwen Fu, Jifeng Dai, Hongsheng Li, Gao Huang, and Xizhou Zhu. Autoloss-zero: Searching loss functions from scratch for generic tasks. In CVPR, 2022. 3
[33] Lujun Li. Self-regulated feature learning via teacher-free feature distillation. In European Conference on Computer Vision (ECCV), 2022.
[34] Lujun Li, Peijie Dong, Zimian Wei, and Ya Yang. Automated knowledge distillation via monte carlo tree search. In ICCV, 2023. 3
[35] Lujun Li and Zhe Jin. Shadow knowledge distillation: Bridging offline and online knowledge transfer. In NeulPS, 2022. 3
[36] Lujun Li, Liang Shiuan-NN Ni,, Ya Yang. and Zhe Jin. Boosting online feature transfer via separable feature fusion. In IJCNN, 2022. 3
[37] Lujun Li, Liang Shiuan-Ni, Ya Yang, and Zhe Jin. Teacher-free distillation via regularizing intermediate representation. In IJCNN, 2022. 3
[38] Lujun Li, Yikai Wang, Anbang Yao, Yi Qian, Xiao Zhou, and Ke He. Explicit connection distillation. 2020.
[39] Zelong Li, Jianchao Ji, Yingqiang Ge, and Yongfeng Zhang. Autolossgen: Automatic loss function generation for recommender systems. In Proceedings of the 45th International ACM SIGIR Conference on Research and Development Information Retrieval, pages 1304-1315, 2022.
12

---

[40] Tsung- Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollar. Focal loss for dense object detection. In ICCV, 2017.
[41] Tsung-Y Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European conference on computer vision, pages 740-755. Springer, 2014.
[42] Jihao Liu, Boxiao Liu, Hongsheng Li, and Yu Liu, Meta knowledge distillation. arXiv preprint arXiv:2202.07940, 2022. 3
[43] Xiaolong Liu, Lujun Li, Chao Li, and Anbang Yao. Norm: Knowledge distillation via n-to-one representation matching. In ICLR, 2023.
[44] Yifan Liu, Changyong Shu, Jingdong Wang, and Chunhua Shen. Structured knowledge distil- lation for dense prediction. IEEE transactions pattern analysis and machine intelligence. 2020. 9
[45] Yu Liu, Xuhui Jia, Mingxing Tan, Raviteja Vemulapalli, Yukun Zhu, Bradley Green, and Xiaogang Wang. Search to distill: Pearls are everywhere but not the eyes. In CVPR, 2020. 2
[46] Seyed Iman Mirzadeh, Mehrdad Farajtabar, Ang Li, Nir Levine, Akihiro Matsukawa, and Hassan Ghasemzadeh. Improved knowledge distillation via teacher assistant. In AAAI, 2020. 2,
[47] Yulei Niu, Long Chen, Chang Zhou, and Hanwang Zhang. Respecting transfer gap in knowledge distillation. 2022. 7
[48] Wonpyo Park, Yan Lu, Minsu Cho, and Dongju Kim. Relational knowledge distillation. In CVPR, 2019. 3.7
[49] Zimian Wei Xin Niu ZHILIANG TIAN Hengyue Pan Peijie Dong. Lujun Li. Emq: Evolving training-free proxies for automated mixed precision quantization. In ICCV, 2023.
[50] Chen Pengguang, Liu Shu, Zhao Hengshuang, and Jiaya Jia. Distilling knowledge via knowledge review. In CVPR, 2021. 2, 6,
[51] Jie Qin, Jie Wu, Xuefeng Xiao, Lujun Li, and Xingang Wang. Activation modulation and recalibration scheme for weakly supervised semantic segmentation. In AAAI, 2022. 9
[52] Zengyu Qiu, Xinzhu Ma, Kunlin Yang, Chunya Liu, Jun Hou, Shuai Yi, and Wanli Ouyang. Better teacher better student: Dynamic prior knowledge for knowledge distillation. arXiv preprint arXiv:2206. 06000,,06677... 2022. 5
[53] Mohammad Rastegari, Vicente Ordonez, Joseph Redmon, and Ali Joseph. Xnor-net: Imagenet classification using binary convolutional neural networks. In ECCV, 2016.
[54] Esteban Real, Chen Liang, David So, and Quoc Le. Automl-zero: Evolving machine learning algorithms from scratch. In ICML., 2020. 3
[55] Esteban Real, Chen Liang, David R. So, and Quoc V. Le. Automl-zero: Evolving machine learning algorithms from scratch, 2020.
[56] Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. arXiv preprint, arXiv:1506.0199) 2015. 9
[57] Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang, Carlo Gatta, and Yoshua Bengio. Fitnets: Hints for thin deep nets. In ICLR, 2015. 1,7
[58] Xiaojie Li Jianlong Wu Fei Wang Chen Qian Shangchen Du Shan You and Changshui Zhang. Agree to disagree: Adaptive ensemble knowledge distillation in gradient space. NeurIPS, 2020. 6
[59] Changyong Shu, Yifan Liu, Jianfei Gao, Zheng Yan, and Chunhua Shen. Channel-wise knowledge distillation for dense prediction. In Proceedings of the IEEF/CVF International Conference Computer Vision, pages 5311-5320 2021. 3,9
13

---

[60] Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive representation distillation. In ICLR, 2020. 7
[61] Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Herve Jegou. Training data-efficient image transformers & distillation through attention. pages 10347-10357, 2021.
[62] Frederick Tung and Greg Mori. Similarity-preservinn knowledge distillation. ICCV, 2019.
[63] Tao Wang, Li Yuan, Xiaopeng Zhang, and Jiashi Feng. Distilling object detectors with fine- grained feature imitation. In CVPR, 2019.
[64] Wenhai Wang, Enze Xie, Xiang Li, Deng- Ping Fan, Kaitao Song, Ding Liang, Tong Lu, Ping Luo, and Ling Shao. Pyramid vision transformer: A versatile backbone for dense prediction without convolutions. In ICCV.
[65] Yukang Wang, Wei Zhou, Tao Jiang, Xiang Bai, and Yongchao Xu. Intra-class feature variation distillation for semantic segmentation. In European Conference Computer Vision, pages 346-362. Springer, 2020. 9
[66] Zimian Wei, Hengyue Pan, Lujun Li Li, Menglong Lu, Xin Niu, Peijie Dong, and Dongsheng Li. Convformer: Closing the gap between cnn and vision transformers. arXiv preprint arXiv:2009.07738, 2022.
[67] Liu Xiaolong, Li Lujun, Li Chao, and Anbang Yao. Norm: Knowledge distillation via n-to-one representation matching. 2022. 2,
[68] Chuanguang Yang. Helong Zhou, Zhulin An, Xue Jiang, Yongjun Xu, and Qian Zhang. Cross-image relational knowledge distillation for semantic segmentation. arXiv preprint arXiv:2204.0663 069866 2022. 9
[69] Jie Yang et al. Automatically labeling video data using multi-class active learning. In Proceed- ings Ninth IEEE international conference computer vision, pages 516-523, IEEE, 2003. 3
[70] Zhendong Yang. Zhe Li, Xiaohu Jiang, Yuan Gong, Zehuan Yuan, Danpei Zhao, and Chun Yuan. Focal and global knowledge distillation for detectors. arXiv preprint arXiv:2111.11837
2021.
[71] Zhendong Yang, Zhe Li, Mingqi Shao, Dachuan Shi, Zehuan Yuan, and Chun Yuan. Masked generative distillation. arXiv preprint urXiv:2205.0/522 2022. 2,8
[72] Li Yuan, Yunpeng Chen, Tao Wang, Weihao Yu, Yujun Shi, Zi- Zi-Hang Jiang, Francis E.H. Tay, Jiashi Feng, and Shuicheng Yan. Tokens-to-token vit: Training vision transformers from scratch on imagenet. In /CCV, 2021. 2
[73] Sergey Zagoruyko and Nikos Komodakis. Paying more attention attention: Improving the performance of convolutional neural networks via attention transfer. In ICLR, 2017. 7, 8
[74] Linfeng Zhang and Kaisheng Ma. Improve object detection with feature-based knowledge distillation: Towards accurate and efficient detectors. In International Conference on Learning Representations, 2020.
[75] Linfeng Zhang, Jiebo Song, Anni Gao, Jingwei Chen, Chenglong Bao, and Kaisheng Ma. Be your own teacher: Improve the performance of convolutional neural networks via self distillation. In ICCV, 2019. 6
[76] Matthew Shunshi Zhang and Bradly Stadie. One-shot pruning of recurrent neural networks by jacobian spectrum evaluation. arXiv preprint arXiv: 1912.00/20, 2019. 6
[77] Ying Zhang, Tao Xiang, Timothy M Hospedales, and Huchuan Lu. Deep mutual learning. In CVPR, 2018. 6
14

---

[78] Hengshuang Zhao, Jianping Shi. Xiaojuan Qi, Xiaogang Wang, and Jiaya Jia. Pyramid scene parsing network. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2881-2890, 2017. 9
[79] Helong Zhou, Liangchen Song, Jiajie Chen, Ye Zhou, Guoli Wang, Junsong Yuan, and Qian Zhang. Rethinking soft labels for knowledge distillation: A bias-variance tradeoff perspective. 2021. 2, 4, 5, 6, 7, 10
[80] Chendi Zhu, Lujun Li, Yuli Wu, and Zhengxing Sun. Saswot: Real-time semantic segmentation architecture search without training. In AAAI, 2024.
[81] Zimian Zimian Wei, Lujun Li Li, Peijic Dong, Zheng Hui, Anggeng Li, Menglong Lu, Hengyue Pan, and Dongsheng Li. Auto-prox: Training-free vision transformer architecture search via automatic proxy discovery. In AAAI, 2024. 3
15


Reviews:
[{"review_id": "FIJzm1DBbT", "reviewer": "Reviewer_AvFD", "summary": "This paper presents KD-Zero, which utilizes evolutionary search to automatically discover promising distiller from scratch for any teacher-student architectures. The discovered distiller can address the capacity gap and cross-architecture challenges for any teacher-student pairs in the final distillation stage. The authors also  conduct extensive experiments on classification, detection, and segmentation to validate KD-zero's effectiveness. ", "questions": "1) Is there efficiency comparison with other KD forms?\n\n2) In table 4, some of the 'Auto-KD' performance even surpass the teacher model by large margin. why?", "limitations": "The author mentioned some limitations in the appendix.\nIn addition to the training efficiency, I currently find no other limitation for application.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1) The writings and organizations are good.\n2) The decomposition of the generalized distiller makes sense and the authors have conducted ablations with other search algorithm choice.\n3) The experiments are comprehensive including various vision tasks.", "weaknesses": "Though the author mentioned its efficiency as one of its advantages. I still worry about its efficiency in terms of training. \nTraditional KD forms indeed needs hyper-param tuning but one expert can choose the proper temperature or other hyper-parameters according to the class number and other conditions, thus reducing the tuning laboring. The KD-Zero involves additional searching phase and I found no searching overhead reported for this phase.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer AvFD, \n\n Thanks for the valuable feedback. We have tried our best to address all concerns in the last few days. If the reviewer finds our response adequate, we would really appreciate it if the reviewer considers raising the score. Please see our responses below one by one：\n\n\n\n----\n\n**Q1:** About the  Search & Training Efficiency.   \n\n**A1:**   Our responses are as follows, and we will add these in the revision. Thanks! \n\n1. We clarify the presentation search overhead and training speed in B.4 & B.5 in the Appendix. KD-Zero requires around 11GPU-hours for search, yet searched distillers share similar training speeds with simple KD methods (e.g., KD & DIST) and are superior to complex KD methods (e.g., CRD). \n2. Almost all AutoML methods, including KD-Zero introduce additional search costs. We highlight the efficiency is compared to the regular search, and our loss-rejection protocol and search space shrinkage improve 4 x search efficiency. Moreover, we adopt 5% early-stopping proxy setting to reduce the 20x search cost. \n3. KD-Zero addresses new teacher-student pairs without expert knowledge by allowing automated distillers search strategies. We can summarise new expert knowledge from the automatically searched distillers (see our Practical Guidance in Section 3.3 and Section C.1in the Appendix). \n4. Our searched distillers are transferable and general. For example, in Figure 2 in the main paper and Tables 1&2 in the Appendix, our distillers searched in one teacher can be directly transferred to different teacher distillers under both surpass some of the hand-designed methods for distillation gaps (e.g. DIST & WSLD) without additional search process. In addition, we perform additional generalization experiments to apply the searched distillers for different teacher-student pairs in the following Table. The results show that the searched distillers indeed distill well in different teacher-student pairs. \n\n| Teacher                    | W-40-2 | R110  | R32x4 | VGG13 | W-40-2 |\n| -------------------------- | ------ | ----- | ----- | ----- | ------ |\n| Student                    | W-16-2 | R20   | R8x4  | VGG8  | SNetV1 |\n| Teacher                    | 75.61  | 74.31 | 79.42 | 74.64 | 75.61  |\n| Student                    | 73.26  | 69.06 | 72.50 | 70.36 | 70.50  |\n| KD                         | 74.92  | 70.67 | 73.33 | 72.98 | 74.83  |\n| WSLD                       | 75.30  | 71.53 | 74.79 | 74.36 | 75.09  |\n| KD-Zero                    | 76.42  | 72.45 | 77.85 | 75.26 | 77.52  |\n| KD-Zero Searched on W-16-2 | 76.42  | 71.76 | 77.65 | 74.65 | 76.85  |\n| KD-Zero Searched on R20    | 76.12  | 72.45 | 77.57 | 74.86 | 77.15  |\n| KD-Zero Searched on R8x4   | 76.24  | 71.85 | 77.85 | 74.88 | 76.76  |\n| KD-Zero Searched on VGG8   | 75.97  | 72.02 | 77.23 | 75.26 | 76.62  |\n| KD-Zero Searched on SNetV1 | 75.76  | 71.63 | 76.85 | 74.53 | 77.52  |\n\n---\n\n**Q2:** About the efficiency comparison with other KD forms.\n\n**A2:** This comparison should be discussed in search and transfer scenarios:\n\n1. For transfer scenarios: The total overheads are close to simple KD because we can directly transfer the searched distillers without additional search costs. In B.5 in the Appendix, we present the training efficiency comparison of the searched distillers and other KD methods. Our searched distillers share similar training speeds with simple KD methods (e.g. KD & DIST) superior to complex KD methods (e.g. CRD). \n2. For search scenarios: The overall efficiency includes the search part in this case. The additional search overheads (see Appendix Table 5) would be 3-5x than traditional KD training costs. However, these KD methods also need additional overheads for hands-on design and debugging, which can be more time-consuming than our well-optimized search process.\n3. We will further optimize the description of search costs and add these discussions in the revision, thanks!\n\n---\n\n**Q3:** About some results in Table 4. \n\n**A3:** Yes. ViT distilled by CNN teachers achieves superior results than teachers. The reasons are as follows: \n\n1. ViT relies on self-attention mechanisms to capture global dependencies among image patches, allowing it to effectively model long-range interactions. Such ViT architecture performs well with large-scale datasets and strong regularization. However, small datasets are not sufficient to extract hierarchical information for ViT. This leads to weak results for these ViT models vanilla trained from scratch on CIFAR-100. \n\n2. CNN teacher models are good at extracting local and hierarchical feature knowledge, which ViT models need exactly. With the auxiliary supervision from the CNN teacher, the ViT model can train well on CIFAR-100 and present strong performance. \n\n3. we provide additional results on ViT-T on large-scale ImageNet datasets in the following Table. Our KD-Zero can improve the ViT from 72.0 to 76.0, surpassing the traditional KD methods e.g., DeiT by large margins, demonstrating our method's ability to generalize.\n\n   | Teacher      | Student | Teacher | Student | KD    | DeiT | DearKD-Ti | KD-Zero |\n   | ------------ | ------- | ------- | ------- | ----- | ---- | --------- | ------- |\n   | RegNetY-16GF | ViT-T   | 82.9    | 72.2    | 72.35 | 74.5 | 74.8      | 76.0    |\n\n----\n\n**Finally,** we hope these responses address the concerns, and we appreciate the constructive feedback. We are committed to improving our manuscript and believe the insights will significantly contribute to this goal. We are glad to discuss further comments and suggestions. If the reviewer finds our response adequate, we would really appreciate it if the reviewer considers raising the score sing the score.  "}, {"review_id": "VOXiIJRDm7", "reviewer": "Reviewer_3PTD", "summary": "This work aims to find the best distiller for different teacher-student pairs to reduce the gap caused by different capabilities and architectures. KD-Zero, the first auto-search framework for distillation was proposed, including search space design, search objectives, and search accelerate techniques. It performs state-of-the-art in multiple datasets and architectures including CNN and ViT.", "questions": "The results are clear in general, and I have the following questions： \n- Are there any insights into why the distiller benefits ViT so well? \n- Is there a general distiller across different teacher-student pair?\n- Table 5 in the supplement indicates CIFAR100 or Imagenet. What is the time cost of searching in imagenet?\n\n", "limitations": "yes", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- First auto-search framework for distillation.\n- Rich and solid experiments, and state-of-the-art results.\n- Great presentation.\n", "weaknesses": "- Efficiency, every different teacher-student pair need to search distiller, which is quite inefficient. Is there a general distiller across different teacher-student pairs?\n- Reason behind the choice of the distiller. Some analysis and practical Guidance for distiller choice are given, but the conclusion is quite shallow. What is the key design for different architecture pair? What differences in the architecture pair make the distiller the best one?\n- The search space designs need a broader exploration, such as a line of research on decoupling distillation into target class and non-target class[1], or knowledge distillation with label smoothing[2][3], etc.\n\n\n[1]arxiv.org/abs/2203.08679 \n\n[2]arxiv.org/abs/2107.00181v1\n\n[3]arxiv.org/abs/2207.12980", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer 3PTD, \n\nThank you for your thoughtful comments and questions. We sincerely thank the reviewer for the positive comments on our work! We would like to address the concerns as follows:\n\n---\n\n**Q1:** About general distiller and efficiency.\n\n**A1:**  Yes. General distillers are available in our search. \n\n1. KD-Zero searched distillers can generalize well to different teacher models:  In Figure 2 and Tables 1&2 in the Appendix, we actually use the distillers searched in ResNet110 (T)->ResNet20 (S) to transfer directly to the distillation of ResNet20 by different teacher models. The searched distillers improve the student model and outperform KD, DIST, and WSLD. The same findings are verified by the WRN-16-2 experiment. \n2. KD-Zero searched distillers can generalize well to different teacher-student pairs. The results on downstream tasks demonstrate that searched distillers can work across different models and datasets (see Tables 6&7). In addition, we perform additional generalization experiments to apply the searched distillers in Table 2 with different teacher-student pairs in the following Table. The results show that the searched distillers distill well in different teacher-student pairs. \n3. For efficiency: almost AutoML methods need additional search costs. We make great efforts to achieve at least 80 times faster search than regular search by loss-rejection protocol, search space shrinkage, and early-stopping proxy setting. Meanwhile, we searched distillers that can be transferred to different teacher-student pairs without additional searching costs.\n\n| Teacher                    | W-40-2 | R110  | R32x4 | VGG13 | W-40-2 |\n| -------------------------- | ------ | ----- | ----- | ----- | ------ |\n| Student                    | W-16-2 | R20   | R8x4  | VGG8  | SNetV1 |\n| Teacher                    | 75.61  | 74.31 | 79.42 | 74.64 | 75.61  |\n| Student                    | 73.26  | 69.06 | 72.50 | 70.36 | 70.50  |\n| KD                         | 74.92  | 70.67 | 73.33 | 72.98 | 74.83  |\n| WSLD                       | 75.30  | 71.53 | 74.79 | 74.36 | 75.09  |\n| KD-Zero                    | 76.42  | 72.45 | 77.85 | 75.26 | 77.52  |\n| KD-Zero Searched on W-16-2 | 76.42  | 71.76 | 77.65 | 74.65 | 76.85  |\n| KD-Zero Searched on R20    | 76.12  | 72.45 | 77.57 | 74.86 | 77.15  |\n| KD-Zero Searched on R8x4   | 76.24  | 71.85 | 77.85 | 74.88 | 76.76  |\n| KD-Zero Searched on VGG8   | 75.97  | 72.02 | 77.23 | 75.26 | 76.62  |\n| KD-Zero Searched on SNetV1 | 75.76  | 71.63 | 76.85 | 74.53 | 77.52  |\n\n---\n\n**Q2:** About the key design and differences in the architecture pair make the best distiller.\n\n**A2:**  Based on searched distillers analyses, we present KD designs guidance summary in Sec. 3.3 and Supp. Sec. C1. Following the suggestion, we will add more analyses in the revision.\n\n1. Normalized-based transforms (e.g. normC, min-max) and losses (e.g. ℓPearson, ℓCosine ), shape-based transforms (e.g. batch, scale-r1, channel) and focal-calibration, small weight factors are key designs.  \n\n2. KD design analysis from an architectural perspective: \n\n   a. Wider teacher-student models usually opt for channel-wise ops (e,g. normC in VGG8, channel in R8x4). \n\n   b. ViT models (e,g. DeiT, T2T, PVT) tend to opt for feature knowledge, which contains rich local information. \n\n   c. Large teachers (e.g., R110, WRN-40-2) prefers focal-calibration to avoid overconfidence. \n\n   d. Some cross-architecture models (e.g., SNV1, SNV2, ViT) also often shape-based transforms to better align their knowledge. This suggests that shape-based transforms help bridge heterogeneous models. \n\n   e. On large-scale ImageNet datasets, R18 adopts ℓPearson , indicating ℓPearson’s suitability for large-scale tasks.\n\n---\n\n**Q3:** About the search space designs.\n\n**A3:** Many thanks for this really great suggestion. We agree that considering recent insightful KD designs. like decoupling distillation (e.g, DKD[1]), and KD with label smoothing (e.g., IE-KD[2] and Zipf's LS[3]) will add more depth to our KD-Zero.  During rebuttal, we explore distiller search with target class & non-target class logits as inputs and obtain promising results (see the following Table). We will incorporate these insights into our search space design and discuss them in the revision.\n\n| Teacher                                    | W-40-2 | R32x4 | VGG13 | VGG13  |\n| ------------------------------------------ | ------ | ----- | ----- | ------ |\n| Student                                    | W-16-2 | R8x4  | VGG8  | MNetV2 |\n| Baseline                                   | 73.26  | 72.50 | 70.36 | 64.60  |\n| DKD                                        | 76.24  | 76.32 | 74.68 | 69.71  |\n| KD-Zero (target class &  non-target class) | 76.69  | 77.33 | 75.19 | 70.22  |\n\n---\n\n**Q4:** About the distiller benefits ViT so well.\n\n**A4:** Analyses are as follows and we will add these in the revision. Thanks! \n\n1. All our searched distillers use CNN teacher's features as input knowledge. These local and hierarchical features can help ViT train from scratch on CIFAR-100. \n2. Our distillers employ normalized-based transforms (e.g. normN) & losses (e.g. ℓPearson, ℓCosine ), shape-based transforms (e.g. batch, scale-r1, channel) , attention-based transforms (e.g. catt, mask), which can reduce CNN-ViT feature gaps.\n\n---\n\n**Q5:** About the general distiller across different teacher-student pair.\n\n**A5:** Please see response in **A1**.\n\n---\n\n**Q6:** About the time cost of searching in ImageNet.\n\n**A6:** The search phase on ImageNet took about 12 GPU-hours . We will add these in the revision. Thanks! \n\n---\n\n**Finally,** we hope that our response could satisfy the reviewer's concerns, and we value your input and believe it will greatly help us improve our manuscript. We are glad to discuss further comments and suggestions.  "}, {"review_id": "BOixikn97d", "reviewer": "Reviewer_BLS2", "summary": "The paper proposed to design a search space for the distillation module and apply evolution search to find the optimal combination of KD module on the pre-defined search space. The experiments are conducted are three tasks. ", "questions": "See weakness.", "limitations": "The author addressed the limitation. ", "rating": 3, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "The motivation is good, and presentation is fair. ", "weaknesses": "KDzero is a very straightforward method that uses the concept of neural architecture search on KD modules. The novelty is limited. As a result, I expected to see strong performance gain over baselines. Unfortunately, in the current form, the experimental results are weak. \n\n1) only two teacher-student pairs are compared on ImageNet. I suggest the author to include more advanced architecture, such as vit. \n\n2) did not compare with state-of-the-art detection-based KD method. \nThe author only compares a few methods, where most of them are not kd for det. The experiments in Table 6 are insufficient to show the \"scaling knowledge transfer\" ability of KDzero. I believe more advanced det-kd methods should be included, such as GID/LD/Defeat. Also different types of teacher-student pairs as well as detector types, such as anchor-free detectors, should be compared. \n\n3）Experiments on segmentation are only conducted on a small dataset (cityscapes). \nCityscapes is a small dataset. The author should conduct experiments on ADE20K to show the effectiveness of their method. \n\nThere are some other concerns:\n1) No correlation test. The author should provide a correlation test to verify if there is a correlation between proxies, such as cka-gap and sharpness-gap, with the final search results.  \n2）The figures are too small to see, especially figures 8 and 9.\n3) The author claim that KDzero reduced the capacity gap, yet no experiments are provided. CKA value cannot reflect the teacher-student gap. \n4) It is known that the design of search space is critical in NAS. The search space in KD-zeros seems to rely heavily on expert experience, and the overall search space is very similar to state-of-the-art KD methods. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer BLS2: \n\nThanks sincerely for taking the time to review our work. We truly appreciate the valuable feedback and constructive comments. We have  addressed all the concerns in detail with point-to-point responses  below：\n\n------\n\n**Q1:** Compare to neural architecture search.\n\n**A1:** 1. As clarified in Related Work,  KD-Zero differs significantly from NAS methods regarding its search objectives, entities, and techniques. KD-Zero optimizes transforms,  distances, and weights to enhance distillation gains, while NAS methods focus on layer stacking to achieve better efficiency. \n\n2. Although AutoML ideas apply in different fields (e.g. NAS), our KD-Zero is the first to bring AuotML into KD by searching for distillers with significant novelty and insight  **(recognized by All other Reviewers with 2 Accept & 1 Weak  Accept ratings)**. \n\n------\n\n**Q2:** About strong performance gain.\n\n**A2:**  There are two aspects to our response:\n\n1. KD-Zero obtains consistent 2.5% ∼ 2.9% gains on ImageNet.  Considering that the ImageNet is a challenging benchmark for CNNs in the mobile setting with small FLOPs, these gains are actually significant improvements. i.g., MobileNetV2 was about 0.5% higher than that of  ShuffleNetV1 (SOTA of that time). Similar situations are present for other datasets and tasks.\n2. As demonstrated in Table 3, KD-Zero enables significant extra gains with other designs, such as multi-projector, multi-layer, multi-teacher, and multi-loss.\n\n------\n\n**Q3:** About the more advanced architecture.\n\n**A3:** Following the suggestion, we provide additional results on ViT-T in the following Table. The results show KD-Zero effectively improves ViT. We will include this in the revision. Thanks!  \n\n| Teacher      | Student | Teacher | Student | KD    | DeiT | DearKD-Ti | KD-Zero |\n| ------------ | ------- | ------- | ------- | ----- | ---- | --------- | ------- |\n| RegNetY-16GF | ViT-T   | 82.9    | 72.2    | 72.35 | 74.5 | 74.8      | 76.0    |\n\n------\n\n**Q4:** About more detection-based KDs.\n\n**A4:** Following the suggestion, we provide additional  experiments on anchor-based & anchor-free detectors in the following Table. The results show that the searched distillers outperform these detection-based KDs (e.g. GID/LD/Defeat). We will include these in the revision. Thanks!  \n\n| Method              | mAP  | APS  | APM  | APL  | Method         | mAP  | APS  | APM  | APL  |\n| ------------------- | ---- | ---- | ---- | ---- | -------------- | ---- | ---- | ---- | ---- |\n| RetinaNet-Res101(T) | 38.9 | 21.0 | 42.8 | 52.4 | FCOS-Res101(T) | 40.8 | 24.2 | 44.3 | 52.4 |\n| RetinaNet-Res50(S)  | 37.4 | 20.6 | 40.7 | 49.7 | FCOS-Res50(S)  | 38.5 | 21.9 | 42.8 | 48.6 |\n| GID                 | 39.1 | 22.8 | 43.1 | 52.3 | GID            | 42.0 | 25.6 | 45.8 | 54.2 |\n| LD                  | 39.0 | 23.1 | 43.2 | 51.1 | LD             | 40.6 | 24.3 | 44.1 | 52.3 |\n| Defeat              | 39.7 | 23.4 | 43.6 | 52.9 | FGD            | 42.1 | 27.0 | 46.0 | 54.6 |\n| KD-Zero             | 39.9 | 22.9 | 43.7 | 53.6 | KD-Zero        | 42.5 | 25.2 | 46.8 | 55.1 |\n\n------\n\n**Q5**: About large-scale Seg. Datasets.\n\n**A5:**  Following the suggestion, we perform additional experiments on the ADE20K dataset (80k 512*512) and present the mIoU(%) results in the following Table. The results indicate that KD-Zero improves segmenter by 5.09% ~4.61%. We will include these experiments in the revision. Thanks!  \n\n| Teacher              | Student            | Teacher | Student | SKD[36] | IFVD[53] | CWD[48] | MGD[58] | KD-Zero(ours) |\n| -------------------- | ------------------ | ------- | ------- | ------- | -------- | ------- | ------- | ------------- |\n| PSPNet-ResNet101-R18 | PSPNet-ResNet18    | 44.39   | 30.95   | 32.37   | 32.84    | 35.09   | 35.54   | 36.04         |\n| DeepLabV3-ResNet101  | DeepLabV3-ResNet18 | 45.00   | 35.28   | 35.91   | 37.03    | 39.14   | 39.56   | 39.89         |\n\n------\n\n**Q6**: About the correlation test and  Figures' size.\n\n**A6:**  We already make correlation test in Sec. B.2  & Table 4 in Appendix. The correlation results are: CE Loss (0.755)  Sharpness-gap (0.512 ) CKA-gap (0.682), CE Loss+Sharpness-gap+CKA- gap  (0.935), proving that Sharpness-gap & CKA-gap benefit validation. We will resize Figures 8 and 9 to be clearer in the revision.\n\n------\n\n**Q7:** About the capacity gap.\n\n**A7**: We already provide results on the capacity gap in Figure 6 and Figure 7 of Sec. 3.3, which include training curves,  T-SNE visualization, and logits-correlation. The Grad-CAM++  visualization of Figure 1 in the Appendix demonstrates that the student model distilled by KD-Zero has a smaller capability gap with the teacher. CKA, a representations gap metric, has been recently used in  KD (e.g. DPK-ICLR2023, arxiv.org/abs/2206.06067) to analyze capacity gaps.\n\n------\n\n**Q8**: About the search space.\n\n**A8:**  Our responses are:\n\n1. KD-Zero‘s search space is based on our rich expert experience in KD  design and SOTA KDs, allowing researchers to apply our framework without excessive expert knowledge or trial-and-error overheads. \n2. Our search space is organized into input knowledge OPs, transform   OPs, distance OPs, and weight OPs (calibration, temperature, and weight values), with additional advanced options in Table 1. We clearly emphasize in our ablation study that our novel and insightful search space design leads to superior results (Figure 8) and search efficiency  (Figure 9) (see Sec. 3.1 and Sec. 4.4).\n3. Our automated method does not need to avoid current hand-designing  SOTA KDs because of the different tracks &  aims. Furthermore, the distillers searched (see Table 2) differ from the existing KDs. Our framework and findings can help further the development of hand-designed KDs.\n\n------\n\n **Finally,  we genuinely hope that our explanations and efforts can improve the overall evaluation of our work.** We are glad to discuss further comments and suggestions."}, {"review_id": "QnVHgW1oq3", "reviewer": "Reviewer_WY3i", "summary": "Knowledge distillation has received significant attention over the last decade and numerous KD approaches have been proposed so far. This paper focuses on automatically learning the best distillation strategy/loss for a given dataset and student/teacher pair. The distiller search space is composed of various transformation ops, distance functions, loss factors, temperature factors and loss calibration functions. The best distiller in this broad search space is searched using an evolutionary search process. To make the search process efficient, strategies such as loss-rejection protocol and search space shrinkage are used. Experiments were conducted on several datasets and the discovered distillation strategy (which varies based on the dataset and teacher-student combinations) is shown to perform better than various existing distillation strategies.", "questions": "Please see weaknesses section.", "limitations": "N/A", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This paper brings AuotML into research area of knowledge distillation and shows that by automatic searching we can find better distillation strategies than many existing KD approaches. I think this is interesting and useful for the community to know.\n\nThorough experimental evaluation: Experiments were conducted on many datasets (recognition, detection, segmentation) and teacher-student network pairs.\n\n", "weaknesses": "Some results are confusing: In Table 4, the Auto-KD performance of transformer-based students (last three columns) is significantly better than the teacher model (5-6%). Makes me wonder if the student was finetuned from some pretrained checkpoint instead of training from scratch.\n\nLimited ablations: The proposed search process uses three criteria: test loss, sharpness-gap and CKA-gap. What is the contribution of sharpness-gap and CKA-gap? Ablation studies on this are very limited. Results with only one teacher-student combination on CIFAR100 are presented in the supplementary material (Table 3).  Based on these results, the CKA-gap and sharpness-gap criteria do not seem to be that important (the performance without these is close to the standard deviation range of the performance when they are used).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer WY3i, \n\nThank you for your constructive feedback and insightful questions. We appreciate your time and effort in reviewing our work. We would like to address your concerns as follows:\n\n----\n\n**Q1:** About some results in Table 4. \n\n**A1:** The students in our experiments are indeed trained from scratch and not fine-tuned from pre-trained checkpoints. Some  ViT distilled by CNN teachers achieves superior results. The reasons are as follows: \n\n1. ViT relies on self-attention mechanisms to capture global dependencies among image patches, allowing it to effectively model long-range interactions. Such ViT architecture performs well with large-scale datasets and strong regularization. However, small datasets are not sufficient to extract hierarchical information for ViT. This leads to weak results for these ViT models vanilla trained from scratch on CIFAR-100. \n\n2. CNN teacher models are good at extracting local and hierarchical feature knowledge, which ViT models exactly need. With the auxiliary supervision from the CNN teacher, the ViT model can train well on CIFAR-100 and present strong performance. \n\n3. we provide additional results on ViT-T on large-scale ImageNet datasets in the following Table. Our KD-Zero can improve the ViT from  72.0 to 76.0, surpassing the traditional KD methods e.g., DeiT by large margins, demonstrating our method's generality.\n\n   | Teacher      | Student | Teacher | Student | KD    | DeiT | DearKD-Ti | KD-Zero |\n   | ------------ | ------- | ------- | ------- | ----- | ---- | --------- | ------- |\n   | RegNetY-16GF | ViT-T   | 82.9    | 72.2    | 72.35 | 74.5 | 74.8      | 76.0    |\n\n**Q2:** About sharpness-gap and CKA-gap. \n\n**A2:** **1. Contributions:** The sharpness-gap and CKA-gap are important criteria for evaluating the distiller's effectiveness in our proxy settings, determining the search accuracy-efficiency trade-offs. In addition, the criteria help distillers bridge the gap between teachers and students in the search by minimizing them as objectives. The training curves,  T-SNE visualization, logits-correlation in Figures 6 & 7, and Grad-CAM++  visualization in Supp. Figure 1 illustrate that selecting distillers with these criteria results in smaller gaps between student and teacher models.\n\n**2. Alternative analyses:** \n\n\n\nFollowing the suggestion, we conduct additional experiments with different teacher-student pairs to evaluate the effect of sharpness-gap and CKA-gap on KD-Zero's performance (refer to the following Table). These results show that both sharpness-gap and CKA-gap contribute to the distillation results. Moreover, CKA-gap is more important for distillation gains, which can be attributed to its accurate capturing of the gap between feature representation and teacher-student. We will include these experiments in the revision to better understand these components' roles. Thanks!  \n\n| Teacher                      | W-40-2 | R110  | R32x4 | VGG13 | VGG13  | W-40-2 |\n| ---------------------------- | ------ | ----- | ----- | ----- | ------ | ------ |\n| Student                      | W-16-2 | R20   | R8x4  | VGG8  | MNetV2 | SNetV1 |\n| Baseline                     | 73.26  | 69.06 | 72.5  | 70.36 | 64.60   | 70.50   |\n| KD                           | 74.92  | 70.67 | 73.33 | 72.98 | 67.37  | 74.83  |\n| KD-Zero                      | 76.42  | 72.45 | 77.85 | 75.26 | 70.42  | 77.52  |\n| KD-Zero without S-gap        | 76.06  | 72.12 | 77.36 | 74.86 | 70.15  | 77.18  |\n| KD-Zero without CKA-gap      | 76.24  | 72.17 | 77.58 | 75.08 | 70.25  | 77.36  |\n| KD-Zero without S &  CKA-gap | 75.97  | 71.95 | 77.05 | 74.82 | 69.89  | 76.92  |\n\n\n\n---\n\n**Finally,** we  hope our response could address the concerns, and we thank the reviewer again for the helpful comments. We are glad to discuss further comments and suggestions.  "}]

Author rebuttal:
## The Summary of Our Responses to All Official Reviews



Dear Reviewers, Area Chairs and Program Chairs, 

We sincerely thank all four reviewers for their thorough and constructive comments. We are glad that the novelty, the presentation, the basic experiments, and the performance of our work have been well recognized by most Reviewers with **2 Accept & 1 Weak Accept pre-ratings**. More encouragingly, Reviewer AvFD, L7DA, and nBFP think our first automated kd search work may be insightful for the KD community.

In the past weeks, we carefully improved the experiments (using all computational resources we have), the clarifications, and the discussions of our work to address the concerns, the questions, and the requests of all four reviewers.



**Finally**, based on the constructive comments by all four reviewers and our responses, **we will carefully revise the manuscript of our work**. We hope our detailed responses and the revised manuscript are helpful to address the concerns, the questions, and the requests of all four reviewers.



Meta-review: This paper proposes an auto-search framework to identify the best distillation method for a given teacher-student pair. In particular, the paper explores a search space with three key components -- knowledge transformations,  distance functions, and loss weights. The paper shows promising results on various vision tasks. In addition, the paper proposed various criteria such as loss rejection protocol and search space shrinkage to reduce the overhead of the auto-search. 

Most of the reviewers recognized the value of this work as it's the first work that leverages auto-search for distillation. Various questions regarding the overhead of the auto-search, transferability of obtained distillation methods to other teacher-student pairs, better ablation, and key takeaways from the paper came up during the review process. The authors did a great job at resolving all of these concerns and provided multiple additional experiments to further strengthen their contributions. One of the reviewers raised concerns about the submission while asking for comparisons with additional baselines and experiments on larger segmentation datasets. The authors have satisfactorily addressed both of these asks from the reviewer.
Decision: Accept (poster)