Neural Architecture Search via Ensemble-based Knowledge Distillation
Authors: Fanxin Li
Shixiong Zhao
Haowen Pi
Yuhao QING
Yichao Fu
Sen Wang
Heming Cui
Venue: ICLR 2022 Submitted (2022)
Keywords: NAS
Knowledge Distillation
Imagenet

Abstract: Neural Architecture Search (NAS) automatically searches for well-performed network architectures from a given search space. The One-shot NAS method improves the training efficiency by sharing weights among the possible architectures in the search space, but unfortunately suffers from insufficient parameterization of each architecture due to interferences from other architectures. Recent works attempt to alleviate the insufficient parameterization problem by knowledge distillation, which let the learning of all architectures (students) be guided by the knowledge (i.e., parameters) from a better-parameterized network (teacher), which can be either a pre-trained one (e.g., ResNet50) or some searched out networks with good accuracy performance up to now. 

However, all these methods fall short in providing a sufficiently outstanding teacher, as they either depend on a pre-trained network that does not fit the NAS task the best, or the selected fitting teachers are still undertrained and inaccurate. In this paper, we take the first step to propose an ensemble-based  knowledge distillation method for NAS, called EnNAS, which assembles an outstanding teacher by aggregating a set of architectures currently searched out with the most diversity (high diversity brings highly accurate ensembles); by doing so, EnNAS can deliver a high-quality knowledge distillation with outstanding teacher network (i.e., the ensemble network) all the time. Eventually, compared with existing works, on the real-world dataset ImageNet, EnNAS improved the top-1 accuracy of architectures searched out by 1.2% on average and 3.3% at most.

Under review as a conference paper at ICLR 2022
NEURAL ARCHITECTURE SEARCH VIA ENSEMBLE- BASED KNOWLEDGE DISTILLATION
Anonymous authors
Paper under double-blind review
ABSTRACT
Neural Architecture Search (NAS) automatically searches for well-performed net- work architectures from given search space. The One-shot NAS method im- proves the training efficiency by sharing weights among the possible architectures the search space, but unfortunately suffers from insufficient parameterization of each architecture due to interferences from other architectures. Recent works attempt to alleviate the insufficient parameterization problem by knowledge distil- lation, which let the learning of all architectures (students) be guided by the knowl- edge (i.e., parameters) from a better- caiter-prrameterized network (teacher), which can be either a pre-trained one (e.g., ResNet50) or some searched out networks with good accuracy performance up to now.
However, all these methods fall short in providing a sufficiently outstanding teacher, as they either depend on pre-trained network that does not fit the NAS task the best, or the selected fitting teachers are still undertrained and inaccurate. In this paper, we take the first step to propose an ensemble-based knowledge distil- lation method for NAS, called EnNAS, which assembles an outstanding teacher by aggregating a set of architectures currently searched out with the most diversity (high diversity brings highly accurate ensembles); by doing so, EnNAS can de- liver high-quality knowledge distillation with outstanding teacher network (i.e., the ensemble network) all the time. Eventually, compared with existing works, on the real-world dataset ImageNet, EnNAS improved the top-1 accuracy of archi- tectures searched out by 1.2% on average and 3.3% at most.
INTRODUCTION
Neural architecture search (NAS) EElsken et al. 2019) can automatically search for the optimal neu- ral architecture from given search space. Neural architectures discovered by NAS have surpassed manually designed ones in accuracy on various tasks (Zoph & Le 2016 Pham et al. 2018 Ben- der et al., 2018). Instead of parameterizing each searched architecture from scratch, one- one-shot NAS methods Liu et al., 2018: Cai et al., 2018: Guo et al., 2020b) have been wildly used recently due to their moderate computational cost. One-shot NAS achieves this by letting whole NAS search space share the parameters and form a supernet, and the parameterization of the supernet only re- quires one training process. Unfortunately, despite its efficiency. one-shot NAS is often denounced for its imprecise estimation of candidate architectures' quality (Bender et al. 2018 Li et al., 2020c Zhang et al. 2020), which is evaluated by letting each candidate subner directly inherit parameters of the supernet.
Several studies have attributed this imprecise estimation in one-shot NAS to the insufficient param- eterization of search space, caused by two major factors: (1) interference between architectures (Li
et al.] 2020c) and (2) unfair training opportunities among candidates (Chu al.) 2019). Among amendments to address this insufficient parameterization issue, Knowledge Distillation (KD) (Hin-
ton et al. 2015) is a promising technique widely used recently (Cai et al.] [019 Yu et al. 2020] Li al. 2020a). It adopts external teacher to regulate the supernet training improve the pa- rameterization efficiency resulting in higher quality of searched-out architecture compared with conventional one-shot methods (Guo et al. 2020b).
Unfortunately, although knowledge distillation is promising, how to select an high-quality teacher remains an open challenge. On known datasets and tasks, existing work typically chooses trained,

---

Under review as a conference paper at ICLR 2022
known-to-be-good be-good network the teacher. DNA et
et al. 2020a) and BigNAS Yu
2020) bring
in high accuracy guidance from a trained network to perform supervised KD. Although the accuracy of teacher is critical, this does not solely guarantee good teaching results (Cho & Hariharan 2019 Mirzadeh et al. 2000).
An unsupervised method, Cream (Peng et al. 2020). surpassed supervised ones recently. It dy- namically maintains a pool of subnets with superior accuracy as the teachers to teach the remaining ones. Cream focuses more on the consistency and complementarity between teachers and students for higher KD efficiency, which accounts for its high performance in test accuracy. But it still fails find a high-quality teacher. On the contrary, it ignores the fundamental factor, i.e., the accuracy of teachers. The average accuracy gap between teachers and students is 2. 2.1% in Cream, suggesting that teachers do not have much extra knowledge to teach. This inferior teacher accuracy greatly reduces the efficiency of KD and will constrain the upper bound of students' final performance. To conclude, no prior work achieves both high accuracy and good suitability when selecting the teachers.
In order to better meet the above two-fold requirement for high-quality teachers of KD one-shot NAS, we focus on improving the accuracy of teachers selected unsupervised methods. Recent advances in ensemble online knowledge distillation (Zhang et al., 2018 Lan et al., 2018 Guo et al.) 2020a) managed to train multiple networks simultaneously and efficiently via knowledge distillation without pre-training teacher. Following them, in this paper, we propose EnNAS, ensemble- based knowledge distillation method for NAS. Specifically, For each input batch, EnNAS samples a set of candidate subnets as students, assembles outputs from the students knowledge distillation between the assembled teacher and each student. Ensemble can address the problem of low teacher accuracy in previous unsupervised works, while maintaining other good characters inherited from them, including good complementarity between teachers and students.
teacher, and performs
A challenge how to select the set of subnets for maximum knowledge distillation effect each time, i.c., selecting a set that can be assembled into a teacher of higher accuracy to teach themselves better. Ensemble tends yield better results when diversity presents among the outputs of net- works (Kuncheva & Whitaker| (200)). Therefore, we propose diversity-based sampling strategy to select subnets generating more diverse outputs than random sampling. Precisely. EnNAS measures the feature map similarity between operators in the same layer. An operator has higher probability of being sampled when its similarity with other operators the same layer is low. By doing so, the subnets generate more diverse feature maps and produce more diverse outputs. Our experiments demonstrate that the diversity- based sampling strategy managed to assemble teachers that are 5.8% more accurate than students average.
This work makes the following contributions:
We propose combine unsupervised learning and ensemble learning into the training pro- cess of KD one-shot NAS. It manages to raise the accuracy of teachers while preserves beneticial factors of unsupervised learning, The teachers it employed meet the two-fold requirement for teachers in KD one-shot NAS better than ever.
We design a sniversity-baseer sampling strategy for EnNAS to generate high-quality ensem- bles. It guarantees that the selected student subnets generate diverse outputs. Based on the diverse outputs, the assembled teacher has higher accuracy than previous works. Teachers in EnNAS reach 5.8% higher accuracy than students average, while the best previous work (Peng et al. (0200) reaches only 2.1%. 1.. This increases the upper bound of accuracy in KD one-shot NAS.
EaNAS sets up a new SOTA in one-shot NAS on Imagenet. Extensive evaluations show that compared with existing works, EnNAS improved the accuracy of produced networks by 1.2% on average and at most 3.3% on ImageNet.
2 RELATED WORKS
2.1 ONE-SHOT NEURAL ARCHITECTURE SEARCH
Neural Architecture Search (NAS) has become the mainstream approach to automate the manual process of architecture design, which significantly reduces the substantial effort of human ex- perts (Zoph & Le) 2016, Pham et al. 2018). Existing NAS strategies can be classified into two
2

---

Under review as a conference paper at ICLR 2022
categories: sample-based NAS and weight-sharing NAS. In sample-based NAS strategies (Zoph iet al., 2018: Wang et al. 2020 Zhong et al. (220), subnets are sampled from a huge search space with different sampling strategies (c.g., randomized sampling (Bender et al.) 2018), uniform sam- pling (Dong & Yang] 2019)), and then fully trained on the target dataset from scratch. However, training each subnet from scratch is expensive. Hence, sampled-based NAS strategies only can support architecture searching within small target datasets (c.g., CIFAR-10).
On the contrary, the weight-sharing strategies (Khodak al.) 2019 Li al. 2020b) (a.k.a., one-shot strategies) can greatly reduce the training cost, supporting large datasets like ImageNet. State-of- the-art weight-sharing NAS, such as Differentiable Architecture Search (DARTS) (Liu et al. 2018) and its variants (e.g., P-DARTS (Chen et al. 2019)), encode the search space into supernet. A supernet composed of all candidate operations (e-g.. conv3x3) and maintains weights for each operation. The supernet is only trained once. All subnets sampled from the supernet can directly inherit the weights in the supernet that share common graph nodes. Thus, the qualities of subnets are directly queried on the inherited weights.
However, despite many benefits, one-shot strategies with weight sharing suffer from weight cou- pling: the operators of supernet are jointly optimized, and from micro view, each subnet's s weight is interfered by other subnets. Thus, one-shot strategies produce subnets with low valida- tion accuracy compared with their ground-trutt modeling ability, especially on large datasets (e.g., ImageNet). To reduce such coupling, SPOS (Guo et al. (220b) in each training step samples one subnet to be trained, and weights still shared between subnets.
2.2 NEURAL ARCHITECTURE SEARCH WITH KNOWLEDGE DISTILLATION
Knowledge distillation (Hinton et al. 2015) for the neural network is to transfer the knowledge from a trained teacher to smaller student. Existing works on knowledge distillation can be roughly classified into two categories. The first category is to use soft labels generated by the teacher to teach the student, which was first proposed by (Ba & Caruana 2013). Hinton et al. (Hinton et al.) 2015) redefined knowledge distillation as training a shallower network to approach the teacher's output after the softmax layer. However, when the teacher model gets deeper, learning the soft-labels alone is insufficient. To address this problem, the second category proposes to employ the internal representation of the teacher guide the training of the student (Romero et al. 2014 Zhang al.,
2017 Yim et al. 2017 Wang et al. 2018).
There are already literatures that adopt knowledge distillation to accelerate NAS. Cream (Peng et al. 2020) lets each subnet, which is sampled for each training step, not only learn from the dataset, but also learn from teacher subnet with the highest up-to-date accuracy. However, Cream still wastes learning opportunities and fails to efficiently spread the knowledge throughout the supernet. DNA (Li et al., 2020a) adopts an external teacher DNN that learns the dataset supervise the NAS process, and all other NAS candidates learn from the teacher. However, the learned knowledge highly bounded by the external teacher. In this work, EnNAS iteratively generates the teacher through the ensembling of multiple student subnets, and then the ensemble teacher guides the train- ing of those student subnets. Thus, EnNAS's strategy is not bounded by any static teacher, and the learning process is totally automatic and unsupervised.
3 METHOD
3.1 OVERVIEW
We first give an overview of our proposed method EnNAS that accelerates one-shot NAS through ensembling-based knowledge distillation. In each iteration, EnNAS selects multiple subnets from the supernet and optimizes these subnets simultaneously. The ensembling soft targets are generated by aggregates logits of all subnets. All the subnets learn from the ground-truth labels well as the soft targets. The subnets are selected by the diversity-bast sampling strategy of EnNAS, which takes the operator diversity into consideration.
3

---

Under review as a conference paper at ICLR 2022
3.2 ONLINE DISTILLATION
Suppose we have selected m subnets from the supernet for given input batch {z,y). where is the input sample and y the ground-truth label, the i th subnet outputs logit 2; and produces soft probability distribution = so softmar(z)) (z;). The logit t of the ensemble teacher is weighted sum of all the subnet logits
m2
mR-1
W;zi, subject to Σ
w w;=1
1=
where w; is the weight corresponds to logit of the i th subnet. Following the ensembling method proposed in Guo et al. (2020a), the teacher logit calculated by minimizing the cross-entropy loss of the ensemble teacher logit and the ground- truuh label y. Let {wo, w1, Wm-1) and T the problem is formalized as
Z = (
min La (w Z.y)
The problem is convex and we solve it using the ECOS solver (Domahidi et al. 2013).
After generating the ensemble teacher logit, can get the teacher soften probability distribution soʃ ttmax(t) The loss L; of the - th subnet is then a combination of the cross-entropy loss with the ground-truth label y and the knowledge distillation loss with the soft target p.
C; = LCE LKD. where LKD= = -
The knowledge distillation loss bases on the KL divergence between p and q. T and À are the hyperparameters. Each subnet uses the combined loss L to update the weights of corresponding operators the supernet.
3.3 DIVERSITY-BSSER SAMPLING
In this section, we introduce the diversity-based sampling strategy of EnNAS. This strategy lets EnNAS select subnets that generate more diverse logits, which helps to generate more accurate ensemble teacher logit. We first define the concept of operator diversity and how to calculate it, and then explain the diversity- sampling strategy based on the operator diversity.
We assume the supernet M has L layers, and each layer N (0 S < L - 1) has T candidate operators denote as (oo, "1 ,0r-"",rr11 0r-11-
The operator diversity defined as the diversity of feature maps output by two operators in the same layer regarding the same input. The input of operator is the feature maps from the operator's last layer. Formally, given all possible input feature maps U {uo, "1,- Un-13 in layer A the diversity of two operators Om and on, in the layer is defined as:
((Oms 0x) = En= cos(r(om (u4)), r(x,
n
where r(-) is a function that reshapes feature map to a vector. In general, the value of ((om o,)) lies between 0 and 1. The large ((0m, on)) value corresponds to high diversity between operators On and o,.
However, infeasible get all the possible input feature maps for layer in the supernet, and we approximate the operator diversity by sampling subset of the possible input feature maps. Specifically, we sample inputs from the dataset and feed them into the supernet. The feature maps input to a layer uniformly assigned to all operators in that layer, and the output feature maps of all operators are merged as the input feature maps to the next layer. By doing so, we get the sampled input feature maps of each layer.
Next, we describe the diversity-batt based sampling strategy of EnNAS. The main idea is to sample operators that generate more diverse feature maps to form subnets. Since sample m sub- nets for an input batch, m operators are selected from each layer in the supernet. For layer in the supernet that contains operators foo, "1, (0T-)) denote the m operators sampled as
4

---

Under review as a conference paper at ICLR 2022
{zu; .i,... J We sample the operators sequentially, and the probability of sampling op- erator is based on its diversity with other operators already sampled. Specifically, for an operator set which contains h sampled operators, the probability of sampling operator oi (0 siss 1) as the next operator defined as
A {To. ..... ....
p(o.AA) A'EA A() where g(o,, A) EoyEd ((0),
g A)
(5)
size(A)
A is set that contains all possible choices of sampling h operators. A) measures the average diversity between operator o; and the already sampled operator set A. The probability of sampling an operator increases if it generates feature maps more diverse from the operators already sampled. Besides, the probability of choosing any operator as the next operator equals 1
To sum up. the diversity-basee based sampling strategy in EnNAS has the following two properties:
The subnets sampled generate more diverse outputs than random sampling.
The probability of choosing each operator is the same, and thus all the operators are trained equally.
3.4 NAS WITH ONLINE DISTILLATION
Training. Given an input batch, we sample subnets based on the probability distribution derived above. The loss of each subnet is computed based on the loss function described Section 3.2 and backpropagation is performed using the loss. All subnets update weights of the operators they contain by gradient descent.
After training the supernet for several iterations, follow the process described in Section 3.3 to update the probability distribution of operators in the supernet correspondingly. Since the opera- tor diversity does not change drastically, updating operator diversity per epoch does not affect the diversity of subnets sampled.
Searching. After the supernet is trained, perform evolution search it get architecture with optimal accuracy on the validation dataset. We first randomly initialize the evolution search with n architectures randomly picked. We then evaluate the accuracy of each architecture the validation set. The architectures with the highest accuracy are picked in each generation. The picked architectures served as parents generate the next generation by crossover and mutation. After few iterations, the architecture with the highest accuracy is selected.
Algorithm NAS with Online Knowledge Distillation
Require: supernet A w/ initialized weights w, dataset D w/ batches 1: while A not converged do
2: select teacher Ou, based on generalization error
3: sample batch (x)(1) y(0) from dataset
4: sample m subnets (fg. ......... ....,,,,m)))
5: for (1 to
do 6: calculate logit 2; of subnet I with respect to (+)
7: end for
8: calculate ensemble logit p based on (#o, ... 2m-1) 1)
9: for to m do
10: calculate Log and LKD for t
11: update 4, weights by V,, LCF and WDL
12: end for
end while
14: Derive the architecture from the supernet
4 EVALUATION
Datasets. We fully implemented our method EnNAS and used EnNAS to automatically find the most accurate architectures for image classification tasks. Our experiments were conducted a popular
5

---

Under review as a conference paper at ICLR 2022
Table 1:
Comparison with state-of-the-art NAS methods ImageNet. EaNAS w/o DS means
EnNAS without diversity-based sampling.
Top-1
Top-5
FLOPs
Supernet Training
Space
Method
Supervised
(%)
(%)
(M)
(GPU days)
SPOS
74.7
328M
12
Cream
77.6
93.3
287M
16
sl
DNA
77.1
93.3
348M
Y
4
EnNAS w/o DS
77.5
93.5
317M
N
12
EnNAS
77.9
93.6
330M
N
12
SPOS
75.1
92.5
479M
14
Cream
77.3
93.0
422M
N
19
s2
DNA
77.6
93.1
396M
Y
5
EnNAS w/o DS
78.0
93.6
436M
N
14
EnNAS
78.4
93.8
413M
14
SPOS
77.1
93.3
482M
N
17
Cream
78.7
93.7
495M
N
22
DNA
78.2
93.2
471M
Y
7
EnNAS w/o
78.6
93.4
519M
17
EnNAS
79.1
94.2
532M
17
image classification dataset, i.e., ImageNet Deng et al. (2009). ImageNet contains over 14 million images, and specifically, we used ILSVRC2012 Russakovsky et al. (2015), a subset of ImageNet which contains 1,000 object classes and 1. 1.28M training images and 50K validation images.
Baselines. We compared EnNAS with three NAS methods that achieve state-of-the-ar performance results, including SPOS Guo et al. (2020b), DNA al. (2020a), and Cream Peng et al. (2020) one-shot approach that randomly samples a subnet and trains using
SPOS is the commonly-un single batch iteratively. DNA a pre-trained network as the teacher and distills supernet use block-wise knowledge distillation. Cream leverages knowledge distillation to acceler- NAS by maintaining teacher pool and select teacher from the pool for each subset sampled. We also add a strawman approach that integrates online- enttmmlinsemlligg to one-shot NAS through ran- domly sampling set of subnets for each input batch, namely EnNAS w/o diversity- based sampling-
the student
Search. We benchmark EnNAS on three search spaces. The first search space s/ is small search space similar state-of-the-arr works for fair comparison. The search space s/ contains set of mobile inverted bottleneck convolution (MBConv) with kernel sizes {3. 5, 7} and expand ratios {1, 3. 6). An Identity operator also added for elastic depth search. We further use two large search spaces x2 and s.3 to evaluate EnNAS in more complicated search space. We add the building blocks of ShuffleNetV2 (Ma et al. 2018) into search space s/ to construct search space s2. We also increase the layers in scarch space s2 to construct search space s3.
Settings. Our experiments were conducted on the 8 Nvidia 2080TI GPUs, and each GPU is equipped with 11 GB physical memory. We train the supernet for 120 epochs using the following settings: SGD optimizer with momentum 0.9 and weight decay 4e-5, initial learning rate 0.5 with linear annealing. The chosen architecture is retrained for 500 epochs Imagenet using RMSProp optimizer with initial learning rate 0.064 and weight decay le-5. Both EnNAS and Cream start knowledge distillation from the 20th epoch. EnNAS samples subnets for each input batch. EnNAS samples only 1 of the training dataset in each epoch for fair comparison with the other baselines.
4.1 END-TO-END EVALUATION
Metrics. We leveraged three commonly-used metrics to evaluate different NAS techniques' end-to- end performance, including Top-1/5 Accuracy, FLOPS (i.e., floating point operation number of an architecture) and Search Cost. Among these metrics, Top-1/5 Accuracy stands for the accuracy of searched architecture on the test dataset.
Results. We show the results compared with state-of-the-art NAS methods on ImageNet in Ta- ble [] We re-implemented the baseline methods (i.e., Cream, DNA, and SPOS) on search space s2 and s3 since they have not been evaluated these search spaces. EnNAS achieved age top-1 accuracy 78.4% on three search spaces, which is 1.2% higher than the average top-1 accuracy compared state-of-the-art methods. This shows the effectiveness of our knowledge
aver-

---

Under review as a conference paper at ICLR 2022
distillation method, which effectively transfers knowledge from the well-performed subnets to the porrl-performed subnets. Besides, our method takes the same GPU days as SPOS to complete the architecture search. This indicates the knowledge distillation mechanism does not incur extra overhead.
We also observe that although Cream improves the architecture accuracy compared with SPOS, incurs more training cost (i.e., 4 to days on ImageNet). This is because, for each subnet sampled be trained, Cream has calculate the logits of all teachers in the teacher pool. After that, only one logit output is used for distillation. Thus, the training efficiency of Cream degrades largely. EnNAS, on the other side, leverages an ensemble teacher train all the student subnet. All logits of the teachers contribute to the NAS process.
4.2 ANALYSIS OF SEARCH PROCESS
EnsemNAS
EnsemNAS
Teacher Accuracy
EnsemNAS w/o
EnsemNNS w/o
diversity-based sampling
Subnet Subnet Diversity
diversity-base sampling
30
Cream
6
20
5
10
20
40
60
80
100
20
40
60
80
100
Search epoch
Search epoch
Figure I: Comparison of teacher accuracy of 3 methods
Figure Diversity of subnets sampled methods
To analyze the effectiveness of our method, we compared teacher top-1 accuracy during the train- ing process EnNAS, EnNAS w/o diversity-based sampling, and Cream. We randomly sampled 100 teachers used by those methods every epoch. We get the accuracy of these teachers using the validation set of ImageNet. Figure [ shows the average accuracy sampled subnets of 3 methods. EnNAS achieves better average accuracy than Cream starting from the 20 th epoch. At the end of su- pernet training, the average teacher accuracy of EnNAS 4.5% higher than that of Cream. Besides, compared with EnNAS w/o diversity- based sampling. EnNASalso achieves higher teacher accuracy during the NAS process. This is because EnNAS selects subnets generate more diverse outputs to form better ensemble teacher. We also evaluated the accuracy gap between teacher and student subnets for the 3 methods. The teachers Cream are 2.1 % higher than the subnets as students, while the ensemble teachers of EnNAS achieve 5.8% higher accuracy than student subnets.
Both EnNAS and EnNAS w/o diversity-based sampling generate teachers with higher accuracy be- cause of the online ensebmling technique EnNAS. The teachers in EnNAS promote the conver- gence of subnets in the supernet due to the larger accuracy gap.
4.3 ABLATION STUDY
In this subsection, we evaluate the effectiveness of the diversity- sampling strategy in EnNAS. We measure the diversity of subnets sampled using both the diversity- based sampling strategy and random sampling. The diversity measured by the average Fuelidean distance between the logits of each pair of subnets. As shown in Figure 2 the subnet diversity selected by EnNAS is larger than randomly sampling (i.e., EnNAS w/o diversity-basse sampling). This demonstrates that the diversity-basee sampling strategy proposed by EnNAS successfully selects subnets that generate more diverse outputs. To show whether diversity among subnets leads to a better ensemble teacher, also evaluated the accuracy of ensemble teachers generated by those two methods, discussed in Section Evaluation shows that the ensemble teachers of EnNAS are 2.5% higher than EnNAS w/o bbseed sampling average.

---

Under review as a conference paper at ICLR 2022
5 CONCLUSION
In this work, we propose an efficient neural architecture search method (i.c., EnNAS) that leverages knowledge distillation. We observe that the previous NAS methods ignore the teacher selection problem existed in NAS, which can mistakenly use teacher that leads to low distillation efficiency. We solve this problem by generating the teacher through online ensebmling. We then incorporate the ensemble teacher into our workflow. Our evaluation results show that EnNAS can out-perform existing NAS methods with better accuracy.
REFERENCES
Lei Jimmy Ba and Rich Caruana. Do deep nets really need to be deep? arXiv:1312.6184, 2013.
arXiv preprint
Gabriel Bender, Pieter-Jan Kindermans, Barret Zoph, Vijay Vasudevan, and Quoe Le. Under- standing and simplifying one-shot architecture search. In International Conference on Machine Learning, pp. 550-559. PMLR, 2018.
Han Cai, Ligeng Zhu, and Song Han. Proxylessnas: Direct neural architecture search on target task and hardware. arXiv preprint arXiv:1812.0000 2018.
Han Cai, Chuang Gan, Tianzhe Wang, Zhekai Zhang, and Song Han. Once-for-all: Train one network and specialize it for efficient deployment. arXiv preprint urXiv:1908.09799 2019.
Xin Chen, Lingxi Xie, Jun Wu, and Qi Tian. Progressive differentiable architecture search: Bridging the depth gap between search and evaluation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 1303, 2019.
Jang Hyun Cho and Bharath Hariharan. On the efficacy of knowledge distillation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 4794-4802, 2019.
Xiangxiang Chu, Bo Zhang, Ruijun Xu, and Jixiang Li. Fairass: Rethinking evaluation faimess of weight sharing neural architecture search. arXiv preprint
2019.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hi- erarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248-255 leee, 2009.
Alexander Domahidi, Eric Chu, and Stephen Boyd. Ecos: An socp solver for embedded systems. In 2013 European Control Conference (ECC), pp. 3071-3076. IEEE, 2013.
Xuanyi Dong and Yi Yang. One-shot neural architecture search via self-evaluated template network. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 3681-3690, 2019.
Thomas Elsken, Jan Hendrik Metzen, and Frank Hutter. Neural architecture search: A survey. The Journal of Machine Learning Research, 20(1):1977-2117, 2019.
Qiushan Guo, Xinjiang Wang. Yichao Wu, Zhipeng Yu, Ding Liang, Xiaolin Hu, and Ping Luo. Online knowledge distillation via collaborative learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11020-11029, 2020a.
Zichao Guo, Xiangyu Zhang, Haoyuan Mu, Wen Heng, Zechun Liu, Yichen Wei, and Jian Sun. Single path one-shot neural architecture search with uniform sampling. In European Conference Computer Vision, pp. 544-560. Springer, 2020b.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge a neural network. arXiv preprint 1rXivv1503.02531 2015.
Mikhail Khodak, Liam Li, Maria-Florina Balcan, and Ameet Talwalkar. On weight-sharing and bilevel optimization in architecture search. 2019.
8

---

Under review as a conference paper at ICLR 2022
Ludmila I Kuncheva and Christopher J Whitaker. Measures of diversity in classifter ensembles and their relationship with the ensemble accuracy. Machine learning, 51(2):181-20) 2003.
Xu Lan, Xiatian Zhu, and Shaogang Gong. Knowledge distillation by on-the-fly native ensemble. arXiv preprint arxiv: 1edii::806.0465 006.04606, 2018.
Changlin Li, Jiefeng Peng. Liuchun Yuan, Guangrun Wang, Xiaodan Liang, Liang Lin, and Xi- aojun Chang. Block-wisely supervised neural architecture search with knowledge distillation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.
1989-1998, 2020a.
Liam Li, Mikhail Khodak, Maria-Florina Balcan, and Ameet Talwalkar. Geometry-aware gradient algorithms for neural architecture search. arXiv preprint arXiv:2004.08802. 2020b.
Xiang Li, Chen Lin, Chuming Li, Ming Sun, Wei Wu, Junjie Yan, and Wanli Ouyang. Improving one-shot nas by suppressing the posterior fading. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11836-13845 2020c.
Hanxiao Liu, Karen Simonyan, and Yiming Yang. Darts: Differentiable architecture search. arXiv preprint arXiv: 1tll:110000 806.09055. 2018.
Ningning Ma, Xiangyu Zhang. Hai-Tao Zheng, and Jian Sun. Shufflenet v2: Practical guidelines for efficient enn architecture design. In Proceedings of the European conference on computer vision (ECCV), pp. 116-131, 2018.
Seyed Iman Mirzadeh, Mehrdad Farajtabar, Ang Li, Nir Levine, Akihiro Matsukawa, and Hassan Ghasemzadeh. Improved knowledge distillation via teacher assistant. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pp. 5191-5198, 2020.
Houwen Peng, Hao Du, Hongyuan Yu, Qi Li, Jing Liao, and Jianlong Fu. Cream of the crop: Dis- tilling prioritized paths for one-shot neural architccture search. arXiv preprint arXiv:2010.15821, 2020.
Hieu Pham, Melody Guan, Barret Zoph, Quoe Le, and Jeff Dean. Efficient neural architecture search via parameters sharing. In International Conference on Machine Learning, pp- 4095-4100 PMLR, 2018.
Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang, Carlo Gatta, and Yoshua Bengio. Fitnets: Hints for thin deep nets. arXiv preprint arXiv:1412.6550) 2014.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, al. Imagenet large scale visual recognition challenge. International journal of computer vision, 115(3):211-252 2015.
Hui Wang, Hanbin Zhao, Xi Li, and Xu Tan. Progressive blockwise knowledge distillation for neural network acceleration. In IJCAL, pp. 2769-2775. 2018.
Naiyan Wang. Shiming XIANG, Chunhong Pan, et al. You only search once: Single shot neural architecture search via direct sparse optimization. IEEE Transactions Pattern Analysis and Machine Intelligence, 2020.
Junho Yim, Donggyu Joo, Jihoon Bae, and Junmo Kim. A gift from knowledge distillation: Fast optimization, network minimization and transfer learning. Proceedings the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4133-4141, 2017.
Jiahui Yu, Pengchong Jin, Hanxiao Liu, Gabriel Bender, Pieter-Jan Kindermans, Mingxing Tan, Thomas Huang, Xiaodan Song, Ruoming Pang, and Quoc Le. Bignas: Scaling up neural archi- tecture search with big single-stage models. In European Conference on Computer Vision. pp- 702-717. Springer, 2020.
Miao Zhang, Huiqi Li. Shirui Pan, Xiaojun Chang, and Steven Su. Overcoming multi-mon model forget- ting in one-shot nas with diversity maximization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7809-7818, 2020.

---

Under review as a conference paper at ICLR 2022
Ying Zhang, Tao Xiang. Timothy M Hospedales, and Huchuan Lu. Deep mutual learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4320- 4328, 2018
Zhi Zhang, Guanghan Ning, and Zhihai He. Knowledge projection for deep neural networks, arXiv preprint arXiv:1710.09505. 2017.
Zhao Zhong, Zichen Yang, Boyang Deng, Junjie Yan, Wei Wu, Jing Shao, and Cheng-Lin Liu. Blockqnn: Efficient block-wise neural network architecture generation. IEEE transactions on pattern analysis and machine intelligence, 2020.
Barret Zoph and Quoc V Le. Neural architecture search with reinforcement learning. arXiv preprint 1.055.161.01158, 2016.
Barret Zoph, Vijay Vasudevan, Jonathon Shlens, and Quoc V Lc. Learning transferable architectures for scalable image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 8697-8710, 2018.
10


Reviews:
[{"review_id": "Yn9UQ_Giq4", "reviewer": "Reviewer_72m4", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n- This paper explores and tackles an useful area of improving the teacher network in self-distillation augmented NAS.\n- They propose a concrete algorithm to create stronger teachers using ensembling of subnetworks and and demonstrate that they can achieve higher performance on Imagenet compared to similar methods in a several albeit similar search spaces.\n- They introduce a novel diversity sampling algorithm to improve the diversity of models used to build the ensemble and demonstrate its affects with ablations. They show that it it improves final model performance and allows the construction of a stronger teacher with less training.\n\nWeaknesses:\n- While they achieved significant results on Imagenet, the paper could be significantly improved by inclusion of significantly more empirical results. It would be especially useful to demonstrate that the one-shot network can still transfer to other datasets. It would also benefit from results on the NAS benchmark datasets to better characterize the ability of the algorithm in generally characterizing the architecture space. Are the results you show from only one run of the algorithms?\n- The current wording and presentation of the paper make important details of the paper and the algorithm difficult to follow. It is unclear if all of the claims are strongly supported.\n- The algorithm itself could be much better understood with significantly more ablations of the different hyperparameters like the number of subnets chosen, and the weighting between the distillation loss and the Cross entropy loss.\n\n\n\nQuestions:\nIn the abstract and results you claim, \"Eventually, compared with existing works, on\nthe real-world dataset ImageNet, EnNAS improved the top-1 accuracy of architectures searched out by 1.2% on average and 3.3% at most.\"  I'm not sure based on table 1 compared to CREAM if that is accurate? I believe CREAMS average = 77.86% EnNAS average = 78.47%\n\nWill the code be released for reproducibility?\n\n\"After training the supernet for several iterations, we follow the process described in Section 3.3 to update the probability distribution of operators in the supernet correspondingly. Since the operator diversity does not change drastically, updating operator diversity per epoch does not affect the diversity of subnets sampled.\"\n\nWhat are the details for updating the probability distribution? It does not seem to appear in the algorithm psuedocode. Is it updated per epoch? Since the distribution is dependent on the previously selected subnets, how doe that work? Does this bias the one-shot network towards a particular architecture distribution? How many images are used to determine this diversity sampling distribution?  In section 3.3, you claim that the method both biases toward diverse outputs and every operator is trained equally. Could you explain further? I believe I am misunderstanding.\n\nIs this method unsupervised? The paper seems to claim that this paper, and the CREAM paper are unsupervised, but I believe both train on the labels of Imagenet. It might be a bit confusing to overload the meaning of that term. Calling it self-distillation may be less confusing?\n\nLooking at Figure 1 and 2, it seems like the gap in performance of the algorithms and diversity of subnets with and without diversity sampling significantly drops towards the end of training? Would they converge if you continued training longer? How many runs did you do in the abaltions\n\nDoes there happen to be a particular theoretical grounding for the specific derivation of the diversity-based sampling based being linearly proportional to the cosine similarity?\n\nMinor:\nensebmling in the conclusion", "summary_of_the_paper": "This paper proposes an efficient concrete novel Neural Architecture KD one-shot NAS algorithm that uses a diversity-based sampling algorithm to assemble outstanding ensemble-based teachers for Knowledge Distillation. They demonstrate their method on Imagenet with several different search spaces and significantly improve the SOTA accuracy for KD one-shot NAS. They demonstrate in ablation studies that the diversity sampling algorithm improves both the accuracy of the teacher network used in distillation as well as the performance of the final derived architectures.", "summary_of_the_review": "This paper explores a useful area of improving the performance of the teacher network in self-distillation augmented NAS and shows that this can empirically improve NAS performance. Unfortunately, I currently recommend rejection of this paper since it does not have sufficient empirical results to demonstrate the robustness of the proposed improvement and the robustness of their method on different datasets and as a one-shot method. The paper's presentation is also unclear and some of the results may be stated incorrectly so it is unclear if all the conclusions are well justified.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "QsLQvJaJfH3", "reviewer": "Reviewer_JuuW", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "- Strength \n1. Well written. Easy to understand.\n2. The motivation is clear.\n\n\n- Weakness\n1. Inconsistent experiment results. The authors claimed the proposed method ``achieved 1.2% higher than the average top-1 accuracy compared to state-of-the-art methods''. However, I checked Table 1 and found there is only about 0.5% higher than Cream. Also, I checked the revision history and found the authors claimed a 1.71% improvement on their first version. \n\n2. The idea is kind of straightforward. Using the ensemble teacher has been widely used in previous knowledge distillation works and semi-supervised learning methods (e.g., mean-teacher, co-training). As for the diversity-based sampling strategy, it is quite trivial of the current design.\n\n3. For the diversity-based sampling strategy, the authors sample operators sequentially, which might not be a good choice. In this way, the first selected/sampled operator(s) would have a huge impact on the following sampling process, which doesn't make sense. This may greatly  affect the final performance since the sampled set would be totally different given different random initializations/seeds.\n\n4. The authors did not try other sampling strategies as baselines. For example, always choosing subnets with most different connections might be a strong baseline.\n\n5. What is \\lambda just below Eq. (3)? Is it a typo or something I missed. \n\n6. What is the impact of the hyperparameter \\tau? Is it very sensitive?\n\n\n", "summary_of_the_paper": "This paper focuses on knowledge distillation for NAS, which improves the student networks by the ensemble teacher models. To generate better ensemble teacher logits, the authors proposed to sample operators that predict diverse feature maps to form subnets. \nExperiments shows the proposed method achieves better results compared to state-of-the-art KD-based NAS methods.", "summary_of_the_review": "The solution is somehow trivial (e.g., diversity-based sampling strategy). The experiments are not sufficient. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "5YwtnKj6P__", "reviewer": "Reviewer_4GyZ", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n1. The proposed method avoids pre-selecting a sufficiently accurate teacher network to guide the training of super-net. Instead, the teacher network is dynamically constructed and will have higher accuracy as the super-net is gradually trained to its optimum.\n2. The diversity-based criterion for selecting student networks seems to be a reasonable choice for constructing a high-quality ensemble as the teacher network.\n\nWeaknesses:\n1. Comparison with some more recent weight-sharing NAS methods are missing. For instance, AlphaNet [1] also aims at improving the training of super-net by replacing the commonly used KL-divergence with alpha-divergence in knowledge distillation. It seems that they achieved a better accuracy-FLOPs trade-off than this paper (~79% @ 300 MFLOPs and ~80% @ 500 MFLOPs).\n2. For constructing the ensemble, student networks are sampled based on the diversity criterion. However, the prediction accuracy of each student network is not considered. Will this lead to a sub-optimal teacher network, if the search space is not well designed (e.g. contains many low-quality operator choices)?\n3. Section 3.2, last paragraph. What does $\\lambda$ stand for? Besides, why there is a negative sign in the knowledge distillation loss in Equation (3)?\n4. Section 3.3, Equation (4). The cosine similarity ranges from -1 (completely opposite) to 1 (identical), which means that the diversity metric defined here ranges from 0 to 2, instead of \"between 0 and 1\".\n\n[1] Dilin Wang, Chengyue Gong, Meng Li, Qiang Liu, Vikas Chandra. AlphaNet: Improved Training of Supernets with Alpha-Divergence. ICML 2021.", "summary_of_the_paper": "In this paper, authors propose to dynamically construct a teacher network from the ensemble of multiple student networks, and then use it to guide the training of super-net via knowledge distillation. The ensemble of student networks is built with the diversity criterion explicitly considered to improve the teacher network's accuracy. After the super-net is well trained, the optimal network architecture is determined by evolution search.", "summary_of_the_review": "My major concerns on this paper includes:\n1. The comparison between some recent weight-sharing NAS methods, e.g. AlphaNet, is missing.\n2. The diversity-based sampling method may impose additional restraints of the design of search space of network architectures.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: All reviewers recommend rejection, and I'm following this recommendation.
Decision: Reject