Don't be picky, all students in the right family can learn from good teachers
Authors: Roy Henha Eyono
Fabio Maria Carlucci
Pedro M Esperança
Binxin Ru
Philip Torr
Venue:  (2021)
Keywords: knowledge distillation
neural architecture search
nas
automl
knowledge trasfer
model compression

Abstract: State-of-the-art results in deep learning have been improving steadily, in good part due to the use of larger models. However, widespread use is constrained by device hardware limitations, resulting in a substantial performance gap between state-of-the-art models and those that can be effectively deployed on small devices. 

While Knowledge Distillation (KD) theoretically enables small student models to emulate larger teacher models, in practice selecting a good student architecture requires considerable human expertise. Neural Architecture Search (NAS) appears as a natural solution to this problem but most approaches can be inefficient, as most of the computation is spent comparing architectures sampled from the same distribution, with negligible differences in performance. 

In this paper, we propose to instead search for a family of student architectures sharing the property of being good at learning from a given teacher. 
Our approach AutoKD, powered by Bayesian Optimization, explores a flexible graph-based search space, enabling us to automatically learn the optimal student architecture distribution and KD parameters, while being 20x more sample efficient compared to existing state-of-the-art. We evaluate our method on 3 datasets; on large images specifically, we reach the teacher performance while using 3x less memory and 10x less parameters. Finally, while AutoKD uses the traditional KD loss, it outperforms more advanced KD variants using hand-designed students.

Under review as a conference paper at ICLR 2021
DON'T BE PICKY, ALL STUDENTS IN THE RIGHT FAM- ILY CAN LEARN FROM GOOD TEACHERS
Anonymous authors Paper under double-blind review
ABSTRACT
State-of of--hh--rtt-tttt results in deep learning have been improving steadily, in good part due to the use of larger models. However, widespread use is constrained by device hardware limitations, resulting in substantial performance gap between state-of- the-art models and those that can be effectively deployed on small devices.
While Knowledge Distillation (KD) theoretically enables small student models to emulate larger teacher models, in practice selecting a good student architecture re- quires considerable human expertise. Neural Architecture Search (NAS) appears as natural solution to this problem but most approaches can inefficient, as most of the computation is spent comparing architectures sampled from the same distribution, with negligible differences in performance.
In this paper, we propose to instead search for a family of student architectures sharing the property of being good learning from given teacher. Our ap- proach AutoKD, powered by Bayesian Optimization, explores a flexible graph- based search space, enabling us automatically learn the optimal student archi- tecture distribution and KD parameters, while being 20x more sample efficient compared to existing state-of-the-arr We evaluate our method on datasets; on large images specifically, we reach the teacher performance while using 3x less memory and 10x less parameters. Finally, while AutoKD uses the traditional KD loss, it outperforms more advanced KD variants using hand- band-designed students.
INTRODUCTION
Recently-develoeed deep learning models have achieved remarkable performance in variety of tasks. However, breakthroughs leading to state-of-the-art (SOTA) results often rely on very large models: GPipe, Big Transfer and GPT-3 use 556 million, 928 million and 175 billion parameters, respectively (Huang et al., 2019 Kolesnikov et al. 2020 Brown et al. 2020).
Deploying these models user devices (e.g. smartphones) is currently impractical they require large amounts of memory and computation; and even when large devices are an option (e.g. GPU be very high (Cheng
clusters), the cost of large-scale deployment (e.g. continual inference) et al. (017). Additionally, target hardware does not always natively or efficiently support all oper- ations used by SOTA architectures. The applicability of these architectures is, therefore, severely limited, and workarounds using smaller or simplified models lead to a performance gap between the technology available at the frontier of deep learning research and that usable in industry applications.
In order to bridge this gap, Knowledge Distillation (KD) emerges as potential solution, allowing small student models to learn from, and emulate the performance of, large teacher models (Hinton et al. 2015a). The student model can be constrained in its size and type of operations used, so that will satisfy the requirements fully achieving this in practice is extremely challenging, requiring extensive human expertise. For example, while we know that the architecture of the student is important for distillation (Liu et al.) 2019b), it remains unclear how design the optimal network given some hardware constraints.
the target computational environment. Unfortunately, success-
With Neural Architecture Search (NAS) it is possible to discover an optimal student architecture. NAS automates the choice of neural network architecture for specific task and dataset, given search space of architectures and search strategy to navigate that space (Pham et al. 2018 Real
al. 2017
Liu et al. 2019a Carlucei al. 2019) Zela et al. 2018 Ru et al. 2020 One im-

---

Under review as a conference paper at ICLR 2021
portant limitation of most NAS approaches is that the search space is very restricted, with a high proportion of resources spent on evaluating very similar architectures, thus rendering the approach limited in its effectiveness (Yang et al., (020). This is because traditional NAS approaches have no tools for distinguishing between architectures that are similar and architectures that are very differ- ent; a consequence, computational resources in the model. Conversely, properly exploring a large space requires huge computational resources: for example, recent work by Liu et al. (2019b) investigating how to find the optimal student re- quires evaluating 10, 000 models. By focusing on the comparison between distributions we ensure to use computational resources only on meaningful differences, thus performing significantly more efficiently; we evaluate x less architectures than the most related work to ours (Liu et al. 2019b).
needed to compare even insignificant changes
To overcome these limitations, we propose an automated approach to knowledge distillation, in which we look for a family of good students rather than a specific model. We find that even though our method, AutoKI, does not output one specific architecture, all architectures sampled from the famill of students perform well when trained with KD. This reformulation of the NAS prob- lem provides a more expressive search space containing very diverse architectures, thus increasing the effectiveness of the search procedure in finding good student networks.
Our contributions are as follows: (A) a framework for combining KD with NAS and effectively emulate large models while using a fraction of the memory and of the parameters; (B) By search- ing for an optimal student family, rather than for specific architectures, our algorithm is up to 20x more sample efficient than alternative NAS-based KID solutions; (C) We significantly outperform advanced KD methods on benchmark of vision datasets, despite using the traditional KD loss, showcasing the efficacy of our found students.
2 RELATED WORK
Model compression has been studied since the beginning of the machine learning era, with multi- ple solutions being proposed (Choudhary et al. [200 Cheng et al.] (017). Pruning based methods allow the removal of non-essential parameters from the model, with little-to-none drop in final per- formance. The primary motive of these approaches was to reduce the storage requirement, but they can also be used to speed up the model (LeCun et al., 1990] Han et al., 2015 Li et al.] 2016a). The idea behind quantization methods is to reduce the number of bits used to represent the weights and the activations in a model; depending on the specific implementation this can lead to reduced storage, reduced memory consumption and a general speed-up of the network (Fiesler et al., 1990, Soudry et al., 2014 Rastegari et al. 2016 Zhu al.) 2016).. In low rank factorization approaches, a given weight matrix is decomposed into the product of smaller ones, for example using singular value decomposition. When applied to fully connected layers this leads to reduced storage, while when applied to convolutional filters, it leads to faster inference (Choudhary et al.) 2020).
All the above mentioned techniques can successfully reduce the complexity of given model, but are not designed to substitute specific operations. For example, specialized hardware devices might only support small subset of all the operations offered by modern deep learning frameworks. In Knowledge Distillation approaches, a large model (the teacher) distills its knowledge into smaller student architecture (Hinton et al., 2015b). This knowledge is assumed to be represented in the neural network's ' output distribution, hence in the standard KD framework, the output distribution of a student's network is optimized to match the teacher's output distribution for all the training data Yun et al. 2020 Ahn et al. 2019 Yuan et al., 2020 Tian et al., 2020 Tung & Mori, 2019).
The work of Liu al. (019b) shows that the architecture of a student network is a contributing fac- tor in its ability to learn from given teacher. The authors propose combining KD with traditional NAS pipeline, based on Reinforcement Learning, to find the optimal student. While this setup leads to good results, it does at huge computational cost, requiring over days on 200 TPUs. Simi- larly, Gu & Tresp (2020) also look for the optimal student architecture, but do so by searching for a subgraph of the original teacher; therefore, it cannot be used to substitute unsupported operations.
Orthogonal approaches. looking at how KD can improve NAS, are explored by Trofimov et al.) (2200) and [i et al. (2020). The first establishes that KD improves the correlation between differ- ent budgets in mulli-fieelty methods, while the second uses the teacher supervision to search the architecture in blockwise fashion.
2

---

Under review as a conference paper at ICLR 2021
SOTA Model (Teacher)
Hierarchical Graph-Based search space
Proposes
80 based search
Student
Student
Student
Evaluation
Best Student Family
Finds
Figure 1: AutoKD leverages multi-fidelity Bayesian Optimization, hierarchical graph-based search space coupled with an architecture generator optimization pipeline, to find the optimal student for knowledge distillation. See section foo a detailed description.
SEARCHING FOR THE OPTIMAL STUDENT NETWORK GENERATOR
The AutoKD framework (Fig. . T combines Bayesian Optimization (BO), Neural Architecture Search (NAS) and Knowledge Distillation (KD). AutoKD defines a family of random network generators G()) parameterized by hyperparameter 0, from where student networks are sampled. BO uses surrogate model to propose generator hyperparameters, while students from these generators trained with KD using state-of-the-art teacher network. The student performances are evaluated and provided feedback update the BO surrogate model. To improve our BO surrogate model, the search procedure is iterated, until the best family of student networks G( is selected. this section specify all components of AutoKD. See also Fig. T and Algorithm T for overview.
3.1 KNOWLEDGE DISTILLATION
Knowledge Distillation (KD; Hinton et al. 2015b) a method to transfer, or distill, knowledge from one model to another unually-usually from large model small one mee-suhh that the small student model learns to emulate the performance of the large leacher model. KD can be formalized minimizing objective function:
LKD
Σ (()) 3,E
(1)
where 1 is the loss function that measures the difference in performance between the teacher fr and the student fs, Fú is the ith input, Yi is the ith target. The conventional loss function used in practice linear combination of the traditional cross entropy loss LeF and the Kullback-Leibler divergence LxI. of the pre-softmax outputs for JT and fs:
= - T + al.L (a (fr(z:)/r), (fs(a))/T)
where is the softmax function (xx 1/(1 exp(-x)), and is the softmax temperature. Hinton et al. (2015b) propose "softening" the probabilities using temperature scaling with 7 2 1. The parameter represents the weight trade-off between the KI. loss and the cross entropy loss Lerr- The LRD loss is characterized by the byper-parameters: a and т; popular choices are 7 E (3,4,5)
and 0.9 (Huang & Wang 2017 Zagoruyko & Komodakis, 2016 Zhu al.) 2018). Numerous other methods (Polino et al. 2018: Huang & Wang 2017]) Tung & Mori 2019) can be formulated form of Equation (2). but this paper we use the conventional loss function l.
Traditionally in KD, both the teacher and the student network have predefined architectures. In contrast, AutoKD defines search space student network architectures and finds the optimal student by leveraging neural architecture search, as detailed below.
3.2 STUDENT SEARCH VIA GENERATOR OPTIMIZATION
Most NAS method for vision tasks employ a cell-based search space, where networks are built by stacking building blocks (cells) and the operations inside the cell searched (Pham et al.) 2018 Real et al. 2017 Liu et al.) (2019a). This results in a single architecture being output by the NAS procedure. In contrast, fiexible search spaces have recently been proposed that are based

---

Under review as a conference paper at ICLR 2021
Algorithm 1: AutoKD
Input: Network generator G, BOHB hyperparametersly, training budget bmin and bmaz), Evaluation function fKD(0,b) which assesses the validation performance of generator hyperparameterd by sampling an architecture from G()) and training with the KD loss LKD (equations 1 and for b epochs.
2: Smaz |log,, Dmin
Umez
3: for E {Smaz: Smaz
0) do
4: Sample M = #mmg-t generator hyperparameters θ {0 which maximises the
raito of kernel density estimators; & (Falkner et al., 2018 Algorithm hhrri 2)
5: Initialise b= r7 bmaa
Run Successive Halving L11 et al. 201 6b
6: while b < bmaz do
7:
L (0, 0 θ}:
8:
O top_k ( L, 1[0l/])
9:
b 17 b;
end while
11: end for
12: Obtain the best performing configuration 0 for the student network generator.
13: Sample k architectures from G(O"), train them to completion, and obtain test performance.
neural network generators (Xie al. 2019 Ru al. 2020). The generator hyperparameters define
the characteristics of the family of networks being generated.
NAGO optimizes an architecture generator instead cal graph-based space which is highly expressive yet low- -wwdimensional (Ru ct al. (2020). Specifically, the search space of NAGO comprises three levels of graphs (where the node in the higher level is lower-level graph). The top level is graph of cells (Gtop) and each cell is itself graph of middle- level modules (Gmid). Each module further corresponds to graph of bottom-level operation units (Ghottom) such as relu-conv3x3-5 triplet. NAGO adopts three random graph generators define the connectivityytopooogy of Gtop- Gmid and Gbottom respectively, and thus is able to produce a wide variety of architectures with only a few generator hyperparameters. AutoKD employs NAGO as the NAS backbone for finding the optimal student family.
single architecture and proposes hierarchi-
Our pipeline consists of two phases. In the first phase (search), multi-fidelity Bayesian opti- misation technique, BOHB (Falkner et al.) 2018), is employed to optimise the low-dimensional search space. BOHB uses partial evaluations with smaller-than-fiut budget to exclude bad configu- rations early in the search process, thus saving resources to evaluate more promising contigurations. Given the same time constraint, BOHB evaluates many more configurations than conventional BO which evaluates all configurations with full budget. As Ru et al. (2020) empirically observe that good generator hyperparameters lead tight distribution of well performing architectures (small performance standard deviation), we similarly assess the performance of particular generator hy- perparameter value with only one architecture sample. In the second phase (retrainA), AutoKI uniformly samples multiple architectures from the optimal generator found during the search phase and evaluates them with longer training budgets to obtain the best architecture performance.
Instead of the traditionally used cross-entropy loss, AutoKD uses the KD loss in qquation allow the sampled architecture distill knowledge from its teacher. The KD hyperparameters tempera- ture and loss weight are included in the search space and optimized simultancously with the architecture to ensure that the student architectures can efficiently distill knowledge both from the designated teacher and the data distribution. A full overview of the framework is shown in Fig.
4 EXPERIMENTS
The first part of this section studies how KD can improve the performance of our chosen NAS backbone (NAGO). In the second part, show how a family of students, when trained with KD (AutoKD), can emulate much larger teachers, significantly outperforming current hand-crafted ar- chitectures.
4

---

Under review as a conference paper at ICLR 2021
Experimental setup. All of experiments recognition datasets CIFARIO and CIFARIO0 (Krizhevsky) 2009), as well as M/767 for large-image scene recognition (Quattoni & Torralba 2009). We limit the number of student network parameters to 4.0M for small-image tasks and 6.0M for large-image tasks. Following Liu et al. (2019b), we (Szegedy al., (2166) teacher for the large image dataset. As that
the two, small small-image, standard object
picked Inception-Resnet- model could not be directly applied to small images, and to explore the use of machine-designed network teacher, we decided use the best DARTS (Liu al. (019a) architecture to guide the search on the CIFAR datasets. For ImageNer (Deng et al. 2009), we use a Inception-Resned V2 teacher. All experiments are rum on NVIDIA Tesla VI00 GPUs.
NAS implementation. Our approach follows the search space and BO-based search protocol proposed by NAGO (Ru et al. 2020), as such our student architectures are based on hierarchical random graphs. Likewise, we employ multi-fidelity evaluation scheme based on BOHB (Falkner al., 2018) where candidates are trained for different epochs (30, 60 and 120) and then evaluated on the validation set. In total, only 300 models trained during the search procedure: using 8 GPUs, this amounts to ~22 days of compute on the considered datasets. At the end the search, we sample 8 architectures from the best found generator, train them for 600 epochs (with KD, using the optimal temperature and loss weight found during the search), and report the average performance (top-1 test accuracy). All remaining training parameters
set following Ru al. 2020).
In AutoKD, we include the knowledge distillation hyperparameters, temperature and weight, in the search space, that they optimized alongside the architecture. The temperature ranges from to 10, while the weight ranges from O to 1. Fig. 8 (Appendix) illustrates the importance of these hyperparameters when training a randomly sampled model, lending support to their inclusion.
4.1 IMPACT OF KNOWLEDGE DISTILLATION NAS
To understand the contribution from KD, we first compare vamilla NAGO with AutoKD on CI- FARI00. Fig. shows the validation accuracy distribution differrnnt epochs: clearly, using KD leads to better performing models. Indeed this can be seen in more detail in Fig. 3 where we show the performance of the best found model vs the wall clock time for each budget. is worth men- tioning that while the KD version takes longer (as it needs to compute the lessons on the fly), it consistently outperforms vanilla NAGO by a significant margin on three datasets.
Note that accuracies in Fig. refer to the best models found during the search process, while Fig. 2 shows the histograms all models evaluated during search, which are by definition lower in aceu- racy, average. At the end of search, the model is retrained for longer (as commonly done in NAS methods), thus leading to the higher accuracies also shown in Figs.
Not only does AutoKD offer better absolute performance, but it also enables better multi-fidelity correlation, as can be seen in Fig. 4 For example, the correlation between 30 and 120 epochs improves from 0.49 0.82 by using KD, result that is consistent with the findings in Trofimov al. (2020). Note that multi-fidelity methods work under the assumption that the rankings at different budgets remains consistent guarantee that the best models progress the next stage. A high correlation between the rankings is, as such, crucial.
AutalD NAGO
0.6
60 AutolD
120 AutoKO
Norma ieed Frequency
0.30
Nornal ized Frequency
Normalinn Frequency
0.
0.20
0.3
0.4
0.2
0:10-
0.00
011
15.0
70.8
72.5
72
76
70
Val Valiiaatoon ion Accuracy
Validation Accuracy
Validation Accuracy
(a) 30 epochs
(b) 60 epochs
(c) 120 epochs
Figure 2: Top-1 accuracy distribution for AutoKD and standard NAGO at different budgets on CIFAR100 100. The histograms tallied across 5 runs. Across all budgets, AutoKD samples architec- tures with improved performances top-1 accuracy compared to NAGO.
5

---

Under review as a conference paper at ICLR 2021
Val I con Rccuracy
Val Idar: Accuracs
Val id ion Accur aay
72-
12
08
13
AutokD
30 NAGO
60 NAGO
120 MAGO
150000
Time
Clock
(a) 30 epochs
(b) 60 epochs
(c) 120 epochs
Figure 3: Top-1 accuracy of the best model found during search at a given computation time on CIFARION 100 for AutoKD (red) and NAGO (blue) across different budgets. Each method was run 8 times with the bold curve showing the average performance and the shaded region the stdev.
Rank correl lation across budgets for KD
Rank correll across budgets for NoKD
00.0
120.0
00.
20.
0:75
0 15
Pasaman 0.852141
Peramas 0.829780 80
Peverean 0.723267
Apeeman 0.499176
p 000000
p-0 00000000000
p-
p-0
320
160
315
n 160
0.25
0 25
Ponamnas 8088087
Ponnmmnn 0.647190
0 000000000000
00 0.30
00.0
p=00.0 00.000005
3030
n= 310
n 305
1.000
-1.00
Figure 4: Rank Correlations between different epoch budgets for AutoKD (KD; left) and Standard NAGO (right) computed for 5 runs of NAGO and AutoKD respectively, NAGO reports a rank correlation coefficient of 5 10-1 for epoch pair 30-120, which is 3.3 10 less than that of the KD rank correlation. These results show that the rank correlation across all budget pairs vastly improves when knowledge distillation is applied.
4.2 LARGE MODEL EMULATION
At its core, AutoKD's goal is to emulate the performance of large SOTA models with smaller stu- dents. Fig. 6 shows how the proposed method manages to reach the teacher's performance while using only 1/9th of the memory small image datasets. On MIT67, the found architecture is not only using 1/3rd of the memory, but also 1/10th of parameters. Finally, it is worth noting how Au- toKD increases student performance, such the high final accuracy cannot only be explained by the NAS procedure. Indeed, looking at Fig. 7 it is clear how KD improves both the speed of conver- gence and the final accuracy. Furthermore, as shown in Fig. the optimal family of architectures actually different when searched with KD.
MIT67, CITARI0O, CIFARIO. Table shows the comparison of AutoKD with other KD methods. Notice how learning the student architecture allows AutoKD to outperform a variety of more ad- vanced KD approaches while emplying a smaller parameter count in the student. The exception to this is CIFAR10, O0 where AutoKD outperforms other methods but with a larger number of parameters. This is because the default networks in the NAGO search space have 4M parameters, which is too large for this application. Accuracy-wise. the only method doing better on CIFAR100 Yuan et al. 20200) does so with a student with significantly more parameters (34M vs 4M). Finally, AutoKD orthogonal advanced KD approaches and could be combined with any of them for even further increases in performance.
ImageNet. The improved results smaller datasets extend to large datasets well. On ImageNet, AutoKD reaches 78.0% top-1 accuracy, outperforming both|Liu Liu et al. (2019b) using the same teacher (75.5%) and vanilla NAGO (76.8%).

---

Under review as a conference paper at ICLR 2021
overali
a comy layer
cheatter
operaiion node
Figure 5: Networks sampled from the best generator parameters found by AutoKD (left) and NAGO (right) on CIFARIO 1.. The former is organized with 8 clusters of 3 nodes, while the latter has 5 clusters of 10 nodes, showcasing how the optimal configuration depends on teacher supervision. Arrows indicate information flow between nodes. As NAGO's search space is hierarchical, it contains number of sub-graphs; the yellow (blue) nodes are the input (output) nodes of these sub- sob-graphs while the grey nodes are the operation units (conv-norn-relu).
CIFAR100
CIFAR10
M1767
77
Test Necuraey
82
*
Acouray
97.2
*
Aour a0X
76
97.0
75
96.
Larze SOTA Model
96.
Large SOTA Node AutoKD Student
Lar SOTA Model
Autell Student
96.4
73
AutokD Student
MAS
78
96.
72
20
60
100
80
160
200
Memory per sanble (MB)
Memory sample le (S)
Menory per samp (M8)
Figure 6: Accuracy vs memory per sample for the SOTA Model (the teacher), the AutoKD student and best architecture found by vanilla NAS. This plot clearly shows how AutoKD finds model superior to NASSonll, managing to reach the performance of the large teacher model while using fraction of the per sample memory. Note that the MIT67 teacher has almost 10x the number of parameters of the student (54M vs 6M).
CIFAR10
CIFAR100
MIT67
Top-1 Validation Accuracy
97
Tap-1 Validation Accuracy
76
96
95
72
70
93
74
68
92
72
AutoKD
AutoKD
64
200
400
600
200
400
600
200
400
600
Epoch
Epoch
Epoch
Figure 7: Final training curves for the top generator found by NAGO and AutoKD, for CIFARIO 10, CIFARI00 and MIT67, Each generator was sampled times and the 8 corresponding architectures trained for 600 epochs. Bold line represents the average; shaded region represents std deviation.
DISCUSSION AND CONCLUSION
Improving Knowledge Distillation by searching for the optimal student architecture is a promising idea that has recently started gain attention in the community (Liu et al. 2019b Trofimov al. 2020 Gu & Tresp 2020). In contrast with earlier KD-NAS approaches, which search for specific architectures, our method searches for family of networks sharing the same characteristies. The

---

Under review as a conference paper at ICLR 2021
Table 1: Comparison with KD state-of-the-art, AutoKD uses the standard KD loss (Hinton al, 2015b), while competing methods are using modern variants. Improvement of student accuracy parenthesis with respect to the same student without KD. "fx f and "Ix denotes the crease/decrease in parameter count by the factor f relative to AutoKD for the same dataset. The top performing student accuracy (S acc) for each dataset is specified bold. For each dataset, we sampled 8 architectures and averaged them over 5 runs. For MIT67, the VID method is transfer learning task from ImageNet to MIT67, hence the absence of teacher accuracy (T acc) statistics.
Methodf
Teacher (T)
Student (S)
S params
T acc
S acc
MIT67
SKD
ResNet-18
ResNet-18
11.5M
x 1.9
55.3
60.4 (+5.1)
VID
ResNet-34
ResNet-18
11.5M
1.9
71.9 (+0.9)
VID
ResNet-34
VGG-9
10.9M
1x1.8
72.0 (+6.0)
AutoKD (ours)
InceptionResNeev2
NAGO
6.0M
76.6
76.0 (+1.8)
CIFARI00
CRD
ResNet-32 x
ShuffleNetV2
7.4M xx1.9
79.4
75.7 (+3.8)
CRD
WRN 40-2
WRN-16-2
0.7M x5.77
75.6
75.6 (+2.4)
VID
WRN 40-2
WRN 40-2
2.2M xx1.8
74.2
76.1 (+1.8)
KD-LSR
ResNet-18
ResNet-18
11.5M xx2.9
75.9
77.4 (+1.5)
SKD
ResNet-18
ResNet-18
11.5M tx2.9
75.3
79.6 (44.3)
KD-LSR
DenseNet-121
DenseNet-121
7.0M xx18
79.0
80.3 (+1.3)
KD-LSR
ResNeX122
ResNeX129
34.2M x.8.6
81.0
82.1 (+1.1)
AutoKD (ours) DARTS
NAGO
4.0M
81.9
81.2 (42.6)
CIFAR10
VID
WRN 40-2
WRN 16-1
0.7M
Lx5.7
94.3
91.9 (+1.2)
SPKD
WRN 16-8
WRN 40-2
2.2M
xx1.8
95.8
95.5 (+0.6)
AT
WRN 16-8
WRN 40-2
2.2M
Lx1.8
95.8
95.5 (+0.6)
AutoKD (ours)
DARTS
NAGO
4.0M
97.4
97.1 (+0.8)
SKD Yun 2020). VID (Ahn et al. 2019). KD-LSR Yuaan et al. 2020
CRD
Tian et al.
2020). SPKD
Tung & Mori 2019). AT (Zagorykko & Komodakis 2016).
main benefit of this approach is sample efficiency: while traditional methods spend many compu- tational resources evaluating similar architectures (Yang et al.] 2020), AutoKD is able to avoid this pitfall: for instance, the method of Liu et al.] (2019b) requires ~ 10, 000 architecture samples, while AutoKD effectively search for the optimal student family with only 300 samples. Compared to traditional KD methods, AutoKD is capable of achieving better performance with student architec- tures that have less parameters (and/or use less memory) than hand-defined ones.
Our message "DON'T T BE PICKY" refers to the fact that the macro-structure (connectivity and capac- ity) of a network is more important than its micro-structure (the specific operations). This has been shown to be true for non- non-KD NAS (Xie et al. 2019 Ru et al. 2020) and is here experimentally con- finmed for KD-NAS as well. Changing the focus of optimization in this way releases computational resources that can be used to effectively optimize the global properties of the network. Addition- ally, the fact that family of architectures can characterized by small number of hyperparameters makes the comparison of architectures more mcaningful and interpretable. In the current implemen- tation, AutoKD finds the optimal student family, in which all sampled architectures perform well: future work should explore how to fully exploit this distribution, possibly finetuning the network distribution to obtain an ever better performing model.
To summarize, AutoKD offers strategy to efficiently emulate large, state-of- of- the-art models with fraction of the model size. Indeed, our family of searched students consistently outperforms the best hand-crafted students CIFAR 10, CIFAR 100 and MTT67.
8

---

Under review as a conference paper at ICLR 2021
REFERENCES
Sungsoo Ahn, Shell Xu Hu, Andreas Damianou, Neil D Lawrence, and Zhenwen Dai. Variational information distillation for knowledge transfer. In Proceedings of the IEEE Conference on Com- puter Vision and Pattern Recognition, pp. 9163-9171, 2019.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. arXiv:2005. 2020.
Fabio Carlucci, Pedro M. Esperança, Marco Singh, Antoine Yang, Victor Gabillon, Hang Xu, Zewei Chen, and Jun Wang. MANAS: Multi-agent neural architecture search. wvxi::19090015/1 2019.
Yu Cheng, Duo Wang, Pan Zhou, and Tao Zhang. A survey of model compression and acceleration for deep neural networks, arXiv:1710.09282, 2017.
Tejalal Choudhary, Vipul Mishra, Anurag Goswami, and Jagannathan Sarangapani. A comprehen- sive survey on model compression and acceleration. Artificial Intelligence Review, pp. 1-43, 2020.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. ImageNet: A large-scale hierarchical image database. In Computer Vision and Puttern Recognition (CVPR), pp. 248-255, 2009.
Stefan Falkner, Aaron Klein, and Frank Hutter. BOHB: Robust and efficient hyperparameter op- timization at scale. In International Conference on Machine Learning (ICML), pp. 1445, 2018.
Emile Fiesler, Amar Choudry, and H John Caulfield. Weight discretization paradigm for optical neu- ral networks. In Optical interconnections and networks, volume 1281, pp. 164-173. International Society for Optics and Photonics, 1990.
Jindong Gu and Volker Tresp. Search for better students to learn distilled knowledge. arXiv preprint arXiv:2001.11612. 2020.
Song Han, Jeff Pool, John Tran, and William Dally. Learning both weights and connections for efficient neural network. In Advances neural information processing systems, pp. 1135-1143, 2015.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. arXiv:1503.02531, 2015a.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in neural network. arXiv preprint ar 1ww.:::15030053333 2015b.
Yanping Huang, Youlong Cheng, Ankur Bapna, Orhan Firat, Dehao Chen, Mia Chen, HyoukJoong Lee, Jiquan Ngiam, Quoe V Le, Yonghui Wu, al. Gpipe: Efficient training of giant neu- ral networks using pipeline parallelism. In Advances in Neural Information Processing Systems (NeurIPS, pp. 103 112.112.000 2019.
Zehao Huang and Naiyan Wang. Like what you like: Knowledge distill via neuron selectivity transfer. arXiv preprint vrXiv:1707.011 2017.
Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Joan Puigcerver, Jessica Yung, Sylvain Gelly, and Neil Houlsby. Big Transfer (BiT): General visual representation learning. In European Conference Computer Vision (ECCV), 2020.
Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical repoor,, University of Toronto, 2009.
Yann LeCun, John S Denker, and Sara Solla. Optimal brain damage. In Advances in neural information processing systems, pp. 598-605, 1990.

---

Under review as a conference paper at ICLR 2021
Changlin Li, Jiefeng Peng. Liuchun Yuan, Guangrun Wang, Xiaodan Liang, Liang Lin, and Xi- aojun Chang. Block-wisely supervised neural architecture search with knowledge distillation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp- 1989-1998, 2020.
Hao Li, Asim Kadav, Igor Durdanovic, Hanan Samet, and Hans Peter Graf. Pruning filters for efficient convnets. 2016a.
Lisha Li, Kevin Jamieson, Giulia DeSalvo, Afshin Rostamizadeh, and Ameet Talwalkar. Hyperband: A novel
approach to hyperparameter optimization. arXiv:1603.0653 2016b.
Hanxiao Liu, Karen Simonyan, and Yiming Yang. DARTS: Differentiable architecture search. In International Conference on Learning Representations (ICLR), 2019a.
Yu Liu, Xuhui Jia, Mingxing Tan, Raviteja Vemulapalli, Yukun Zhu, Bradley Green, and Xiaogang Wang. Search to distill: Pearls are everywhere but not the eyes. arXiv preprint arXiv:1911 1911. 190074, 2019b.
Hieu Pham, Melody Guan, Barret Zoph, Quae Le, and Jeff Dean. Efficient neural architecture search via parameter sharing. In International Conference on Machine Learning (ICML), pp- 1092-4101,2015
Antonio Polino, Razvan Pascanu, and Dan Alistarh. Model compression via distillation and quanti- zation. arXiv preprint arXiv:1802.000 80088000006668. 2018.
Ariadna Quattoni and Antonio Torralba. Recognizing indoor scenes. Computer Vision and Pattern Recognition (CVPR), pp. 413-420, 2009.
Mohammad Rastegari, Vicente Ordonez, Joseph Redmon, and Ali Farhadi. Xnor-net: Imagenet classification using binary convolutional neural networks. In European conference computer vision, pp. 522-542. Springer, 2016.
Esteban Real, Sherry Moore, Andrew Selle, Saurabh Saxena, Yutaka Leon Suematsu, Jie Tan, Quoe V Le, and Alexey Kurakin. Large-scale evolution of image classifiers. In International Conference on Machine Learning (ICML), pp. 2902-2911, 2017.
Binxin Ru, Pedro M Esperanca, and Fabio M Carlucci. Neural Architecture Generator Optimization. In Neural Information Processing Systems (NeurIPS), 2020.
Daniel Soudry, Itay Hubara, and Ron Meir. Expectation backpropagation: Parameter-free train- ing of multilayer neural networks with continuous discrete weights. In Advances in neural information processing systems, pp. 963-971, 2014 2014.
Christian Szegedy, Sergey Ioffe, Vincent Vanhoucke, and Alex Alemi. Inception-vés v4, inception- resnet and the impact of residual connections on learning. arXiv preprint arXiv:1602.07261, 2016.
Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive representation distillation. In Inter- national Conference on Learning Representations, 2020.
Ilya Trofimov, Nikita Klyuchnikov, Mikhail Salnikov, Alexander Filippov, and Evgeny Bur- naev. Multi-fidelity neural architecture search with knowledge distillation. arXiv preprint arXiv:2006.088 083400060084,,,.... 2020.
Frederick Tung and Greg Mori. IEEE International Conference Computer Vision pp. 1365-1377 2019.
knowledge distillation. In Proceedings of the
Saining Xie, Alexander Kirillov, Ross Girshick, and Kaiming He. Exploring randomly wired neural networks for image recognition. arXiv:
2019.
Antoine Yang, Pedro M Esperança, and Fabio M Carlucci. NAS evaluation is frustratingly hard. In International Conference Learning Representations (ICLR), 2020.
10

---

Under review as a conference paper at ICLR 2021
Li Yuan, Francis EH Tay, Guilin Li, Tao Wang, and Jiashi Feng. Revisiting knowledge distillation via label smoothing regularization. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1903-3911, 2020.
Sukmin Yun, Jongjin Park, Kimin Lee, and Jinwoo Shin. Regularizing class-wise predictions via self-knowledge distillation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 13876-13857 2020.
Sergey Zagoruyko and Nikos Komodakis. Paying more attention attention: Improving the perfor- mance of convolutional neural networks via attention transfer. arXiv preprint arXiv: wr1vv2/6/..03928, 2016.
Arber Zela, Aaron Klein, Stefan Falkmer, and Frank Hutter. Towards automated deep learning: Efficient joint neural architecture and hyperparameter search. arXiv:1807.06900 2018.
Chenzhuo Zhu, Song Han, Huizi Mao, and William Dally. Trained ternary quantization. arXiv 2016.
preprint
Xiatian Zhu, Shaogang Gong, et al. Knowledge distillation by on-the-fly native ensemble. In Ad- vances neural information processing systems, pp. 7517-7527, 2018.
A APPENDIX
94.
40
20
15
93.
Tomperature
93.
7
92. 0
91.
91.
90.
0. 0
0.3
0.
0.
0. 9
KD
Weight
Figure 8: Student model test accuracy for various temperature (7) and loss weight (a) combinations. The model was sampled from a generator with random parameters, and trained with KD on CIFARIO using the DARTS teacher. The table suggests that there is a positive correlation between the KD loss weight and the performance of the student model. Note that the variability shown when the loss to 0 solely due the inherent stochasticity of the training process.


Reviews:
[{"review_id": "VOjy_Z33oS", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary: \n\nThis paper proposes searching for an architecture generator that outputs good student architectures for a given teacher. The authors claim that by learning the parameters of the generator instead of relying directly on the search space, it is possible to explore the search space of architectures more effectively, increasing the diversity of the architectures explored. They show that this approach combined with the standard knowledge distillation loss is able to learn good student architectures requiring substantially less samples and achieving competitive performances when comparing to other knowledge distillation algorithms.\n\nPros:\n+ The paper is clear overall. A system for combining knowledge distillation and architecture search is proposed that combines surrogate functions, multi-fidelity optimization, and neural architecture generators.\n+ The results on CIFAR10 and CIFAR100 are compelling.\n\nCons:\n\n- Somewhat limited conceptual innovation. The authors combine NAGO and BOHB to obtain a solution for architecture search for knowledge distillation. While the results are solid, there is little insight about the behavior of the method, e.g., no ablations are performed beyond the comparison with NAGO, so it is hard to assess the importance of the individual components. \n- No results on ImageNet.\n- The results are compared with models that use other teacher architectures, so it is hard to determine if the improvements are due to an improved teacher or a better knowledge distillation method. The fact that both student architecture size and student accuracy are important metrics means that there is no easy way \n- No code is included, but I assume that this is something that the authors will address for the final version.\n- The use of BOHB and the surrogate loss are insufficiently described, for example, in Algorithm 1.\n- The claim that a standard neural architecture search would produce architectures sampled from the same distributions and therefore not be efficient in exploring the space of students is insufficiently explored.\n\nComments: \n\nIn essence, this paper only corresponds to a difference in how the architectures for knowledge distillation are generated. The claim that not going through the architecture generator would lead to architectures that are too similar lacks substantiation. Additionally, the paper does not have information about comparisons with other architecture search algorithms for knowledge distillation or surrogate functions, therefore the introduction of a new framework may not be warranted under the claims of the authors. \n", "extra_scores": {}, "extra_text": {}}, {"review_id": "PIobn1-RlyN", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper applied knowledge distillation (KD) on network architecture generator optimization (NAGO) which is one of NAS. Specifically, KD has used in the procedure of searching and the parameter of KD has involved in the search space. As a result, the proposed method (AutoKD) seems to improve NAGO.\n\nPros)\n- Extensive experiments are performed\n\nCons)\n- Some figures are not clearly shown. Please refine the figures (e.g., figure 2) for clarity.\n- Applying KD into NAGO seems to be naively done. It seems that the proposed method is incremental and the contribution is limited and the differences are not highlighted. \n- The result comparison in Table 1 looks not fair:\n  - KD used in this paper used better teachers following the convention, but the competitor KD-LSR and SKD in the table are self-distillation methods, so the comparison is meaningless.\n  - On CIFAR100 dataset, CRD in the original paper used WRN-40-2 as a teacher and trained the student of WRN-16-2, which has only 0.7M parameters with an accuracy of 75.64. However, this paper reports CRD used ShuffleNetV1 which have more parameters \n  - On MIT67 dataset, VID used an ImageNet-pretrained model for transfer learning, but AutoKD used the fine-tuned teacher which is much beneficial to KD in terms of performance.\n  - On CIFAR10 dataset, the compared models (WRN 16-1 and two WRN 40-2s) have fewer parameters than that of NAGO for AutoKD. Therefore, it is hard to say that AutoKD outperforms them.\n- Experimental results are somewhat unconvincing:\n   - As weight is zero in Figure 2, the accuracies in the table should show consistent performance but are deviated w.r.t temperature. The authors should clarify this.\n   - Why the accuracies of NAGO in Figure 4 look low compared to the other results in the paper? \n- Using KD on NAS leverages additional computational cost, but it is not clearly compared quantitatively\n\n\nComments)\n- The method is incremental, and the novelty is limited. The experimental results comparing with other methods are biased to the proposed method, where the competitors' performances are not fairly compared, so it is hardly convincing the results and the effectiveness of the proposed method. \n", "extra_scores": {}, "extra_text": {}}, {"review_id": "DQNOcb4dc9Y", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "\nSummary and contributions\n  The paper takes advantage of NAGO and proposes to search for a family of student network architectures instead of a single architecture, aiming to be more sample efficient. This reformulation of the NAS problem makes it possible to search in an expressive searching space, at the same time, avoid to waste time in comparing similar architectures.\n\nStrengths\n  This KD-NAS approach further develops the benefit of NAGO and makes it convenient to search for a family of student architectures. The optimization objective of finding a family instead of a single architecture helps to speed up NAS process or more sample efficient.\n\nWeaknesses\n  There have already exist KD-NAS approaches and the main difference of this work is to search for a family. This objective mainly takes advantage of the generator in NAGO, so the contribution and novelty should be reduced accordingly.\n  The purpose of some experiments in this paper is a little confusing. From my point of view, this paper aims to utilize NAS to benefit KD, at the same time, make NAS more efficient. Maybe you should compare AutoKD with earlier KD-NAS approaches instead of NAGO to show the impact of KD on NAS.\n  The title shows the fact that the macro-structure of a network is more important than its micro-structure, which has been studied in previous work. But this fact doesn’t logically lead to idea of ‘searching for a family’. Also, I think it isn’t clearly articulated that how this family of student architectures can benefit knowledge distillation. Maybe the performance gains result from the ensemble of networks.\n  Finally, I think it doesn't make much sense to have the comparisons showed in Figure 7. Besides, the logic and results shown in the visualization of Figure 6 are not clear enough to me.\n\n\n[ Detailed comments]\n1. In Chapter 3.2, the original meaning is unclear: ‘The hyperparameters ... that it represents.’\n2. In Algorithm 1, there are many unmarked sentence endings. In addition, where are the definitions of functions f and α, and what does D refer to?\n3. In Figure 6, the specific meanings of various arrows and various colors need to be marked.\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes a new approach to knowledge distillation by searching for a family of student models instead of a specific model. The key idea is that given an optimal family of student models, any model sampled from this family is expected to perform well when trained using knowledge distillation. Overall this is an interesting idea and an important direction of research. However, the reviewers raised several concerns regarding novelty and experimental evaluation. There was a clear consensus among the reviewers that the paper is not yet ready for publication. The specific reasons for rejection include the following: (i) the proposed method is somewhat incremental, and the paper's contributions should be adjusted accordingly; (ii) the experimental results in the paper do not provide a clear/fair comparison with existing approaches, and additional baselines should be considered. The reviewers have provided detailed feedback in their reviews, and we hope that the authors can incorporate this feedback when preparing future revisions of the paper.
Decision: Reject