Neural Architecture Search via Ensemble-based Knowledge Distillation

1 INTRODUCTION . Neural architecture search ( NAS ) ( Elsken et al. , 2019 ) can automatically search for the optimal neural architecture from a given search space . Neural architectures discovered by NAS have surpassed manually designed ones in accuracy on various tasks ( Zoph & Le , 2016 ; Pham et al. , 2018 ; Bender et al. , 2018 ) . Instead of parameterizing each searched architecture from scratch , one-shot NAS methods ( Liu et al. , 2018 ; Cai et al. , 2018 ; Guo et al. , 2020b ) have been wildly used recently due to their moderate computational cost . One-shot NAS achieves this by letting a whole NAS search space share the parameters and form a supernet , and the parameterization of the supernet only requires one training process . Unfortunately , despite its efficiency , one-shot NAS is often denounced for its imprecise estimation of candidate architectures ’ quality ( Bender et al. , 2018 ; Li et al. , 2020c ; Zhang et al. , 2020 ) , which is evaluated by letting each candidate subnet directly inherit parameters of the supernet . Several studies have attributed this imprecise estimation in one-shot NAS to the insufficient parameterization of search space , caused by two major factors : ( 1 ) interference between architectures ( Li et al. , 2020c ) and ( 2 ) unfair training opportunities among candidates ( Chu et al. , 2019 ) . Among amendments to address this insufficient parameterization issue , Knowledge Distillation ( KD ) ( Hinton et al. , 2015 ) is a promising technique widely used recently ( Cai et al. , 2019 ; Yu et al. , 2020 ; Li et al. , 2020a ) . It adopts an external teacher to regulate the supernet training to improve the parameterization efficiency , resulting in a higher quality of searched-out architecture compared with conventional one-shot methods ( Guo et al. , 2020b ) . Unfortunately , although knowledge distillation is promising , how to select an high-quality teacher remains an open challenge . On known datasets and tasks , existing work typically chooses a trained , known-to-be-good network as the teacher . DNA ( Li et al. , 2020a ) and BigNAS ( Yu et al. , 2020 ) bring in high accuracy guidance from a trained network to perform supervised KD . Although the accuracy of a teacher is critical , this does not solely guarantee good teaching results ( Cho & Hariharan , 2019 ; Mirzadeh et al. , 2020 ) . An unsupervised method , Cream ( Peng et al. , 2020 ) , surpassed supervised ones recently . It dynamically maintains a pool of subnets with superior accuracy as the teachers to teach the remaining ones . Cream focuses more on the consistency and complementarity between teachers and students for higher KD efficiency , which accounts for its high performance in test accuracy . But it still fails to find a high-quality teacher . On the contrary , it ignores the fundamental factor , i.e. , the accuracy of teachers . The average accuracy gap between teachers and students is 2.1 % in Cream , suggesting that teachers do not have much extra knowledge to teach . This inferior teacher accuracy greatly reduces the efficiency of KD and will constrain the upper bound of students ’ final performance . To conclude , no prior work achieves both high accuracy and good suitability when selecting the teachers . In order to better meet the above two-fold requirement for high-quality teachers of KD one-shot NAS , we focus on improving the accuracy of teachers selected in unsupervised methods . Recent advances in ensemble online knowledge distillation ( Zhang et al. , 2018 ; Lan et al. , 2018 ; Guo et al. , 2020a ) managed to train multiple networks simultaneously and efficiently via knowledge distillation without pre-training a teacher . Following them , in this paper , we propose EnNAS , an ensemblebased knowledge distillation method for NAS . Specifically , For each input batch , EnNAS samples a set of candidate subnets as students , assembles outputs from the students as a teacher , and performs knowledge distillation between the assembled teacher and each student . Ensemble can address the problem of low teacher accuracy in previous unsupervised works , while maintaining other good characters inherited from them , including good complementarity between teachers and students . A challenge is how to select the set of subnets for maximum knowledge distillation effect each time , i.e. , selecting a set that can be assembled into a teacher of higher accuracy to teach themselves better . Ensemble tends to yield better results when diversity presents among the outputs of networks ( Kuncheva & Whitaker , 2003 ) . Therefore , we propose a diversity-based sampling strategy to select subnets generating more diverse outputs than random sampling . Precisely , EnNAS measures the feature map similarity between operators in the same layer . An operator has a higher probability of being sampled when its similarity with other operators in the same layer is low . By doing so , the subnets generate more diverse feature maps and produce more diverse outputs . Our experiments demonstrate that the diversity-based sampling strategy managed to assemble teachers that are 5.8 % more accurate than students on average . This work makes the following contributions : • We propose to combine unsupervised learning and ensemble learning into the training process of KD one-shot NAS . It manages to raise the accuracy of teachers while preserves beneficial factors of unsupervised learning , The teachers it employed meet the two-fold requirement for teachers in KD one-shot NAS better than ever . • We design a diversity-based sampling strategy for EnNAS to generate high-quality ensembles . It guarantees that the selected student subnets generate diverse outputs . Based on the diverse outputs , the assembled teacher has higher accuracy than previous works . Teachers in EnNAS reach 5.8 % higher accuracy than students on average , while the best previous work ( Peng et al. , 2020 ) reaches only 2.1 % . This increases the upper bound of accuracy in KD one-shot NAS . • EnNAS sets up a new SOTA in one-shot NAS on Imagenet . Extensive evaluations show that compared with existing works , EnNAS improved the accuracy of produced networks by 1.2 % on average and at most 3.3 % on ImageNet . 2 RELATED WORKS . 2.1 ONE-SHOT NEURAL ARCHITECTURE SEARCH . Neural Architecture Search ( NAS ) has become the mainstream approach to automate the manual process of architecture design , which significantly reduces the substantial effort of human experts ( Zoph & Le , 2016 ; Pham et al. , 2018 ) . Existing NAS strategies can be classified into two categories : sample-based NAS and weight-sharing NAS . In sample-based NAS strategies ( Zoph et al. , 2018 ; Wang et al. , 2020 ; Zhong et al. , 2020 ) , subnets are sampled from a huge search space with different sampling strategies ( e.g. , randomized sampling ( Bender et al. , 2018 ) , uniform sampling ( Dong & Yang , 2019 ) ) , and then fully trained on the target dataset from scratch . However , training each subnet from scratch is expensive . Hence , sampled-based NAS strategies only can support architecture searching within small target datasets ( e.g. , CIFAR-10 ) . On the contrary , the weight-sharing strategies ( Khodak et al. , 2019 ; Li et al. , 2020b ) ( a.k.a. , one-shot strategies ) can greatly reduce the training cost , supporting large datasets like ImageNet . State-ofthe-art weight-sharing NAS , such as Differentiable Architecture Search ( DARTS ) ( Liu et al. , 2018 ) and its variants ( e.g. , P-DARTS ( Chen et al. , 2019 ) ) , encode the search space into a supernet . A supernet is composed of all candidate operations ( e.g. , conv3x3 ) and maintains weights for each operation . The supernet is only trained once . All subnets sampled from the supernet can directly inherit the weights in the supernet that share common graph nodes . Thus , the qualities of subnets are directly queried on the inherited weights . However , despite many benefits , one-shot strategies with weight sharing suffer from weight coupling : the operators of a supernet are jointly optimized , and from a micro view , each subnet ’ s weight is interfered by other subnets . Thus , one-shot strategies produce subnets with low validation accuracy compared with their ground-truth modeling ability , especially on large datasets ( e.g. , ImageNet ) . To reduce such coupling , SPOS ( Guo et al. , 2020b ) in each training step samples one subnet to be trained , and weights are still shared between subnets . 2.2 NEURAL ARCHITECTURE SEARCH WITH KNOWLEDGE DISTILLATION . Knowledge distillation ( Hinton et al. , 2015 ) for the neural network is to transfer the knowledge from a trained teacher to a smaller student . Existing works on knowledge distillation can be roughly classified into two categories . The first category is to use soft labels generated by the teacher to teach the student , which was first proposed by ( Ba & Caruana , 2013 ) . Hinton et al . ( Hinton et al. , 2015 ) redefined knowledge distillation as training a shallower network to approach the teacher ’ s output after the softmax layer . However , when the teacher model gets deeper , learning the soft-labels alone is insufficient . To address this problem , the second category proposes to employ the internal representation of the teacher to guide the training of the student ( Romero et al. , 2014 ; Zhang et al. , 2017 ; Yim et al. , 2017 ; Wang et al. , 2018 ) . There are already literatures that adopt knowledge distillation to accelerate NAS . Cream ( Peng et al. , 2020 ) lets each subnet , which is sampled for each training step , not only learn from the dataset , but also learn from a teacher subnet with the highest up-to-date accuracy . However , Cream still wastes learning opportunities and fails to efficiently spread the knowledge throughout the supernet . DNA ( Li et al. , 2020a ) adopts an external teacher DNN that learns the dataset to supervise the NAS process , and all other NAS candidates learn from the teacher . However , the learned knowledge is highly bounded by the external teacher . In this work , EnNAS iteratively generates the teacher through the ensembling of multiple student subnets , and then the ensemble teacher guides the training of those student subnets . Thus , EnNAS ’ s strategy is not bounded by any static teacher , and the learning process is totally automatic and unsupervised . 3 METHOD . 3.1 OVERVIEW . We first give an overview of our proposed method EnNAS that accelerates one-shot NAS through ensembling-based knowledge distillation . In each iteration , EnNAS selects multiple subnets from the supernet and optimizes these subnets simultaneously . The ensembling soft targets are generated by aggregates logits of all subnets . All the subnets learn from the ground-truth labels as well as the soft targets . The subnets are selected by the diversity-based sampling strategy of EnNAS , which takes the operator diversity into consideration . 3.2 ONLINE DISTILLATION . Suppose we have selected m subnets from the supernet for a given input batch { x , y } , where x is the input sample and y is the ground-truth label , the i− th subnet outputs logit zi and produces soft probability distribution qi = softmax ( zi ) . The logit t of the ensemble teacher is a weighted sum of all the subnet logits t = m−1∑ i=0 wizi , subject to m−1∑ i=0 wi = 1 ( 1 ) where wi is the weight corresponds to logit of the i− th subnet . Following the ensembling method proposed in Guo et al . ( 2020a ) , the teacher logit is calculated by minimizing the cross-entropy loss of the ensemble teacher logit t and the ground-truth label y . Let w = { w0 , w1 , · · · , wm−1 } and Z = { zT0 ; zT1 ; · · · ; zTm−1 } , the problem is formalized as min w LCE ( wTZ , y ) ( 2 ) The problem is convex and we solve it using the ECOS solver ( Domahidi et al. , 2013 ) . After generating the ensemble teacher logit , we can get the teacher soften probability distribution p = softmax ( t ) . The loss Li of the i − th subnet is then a combination of the cross-entropy loss with the ground-truth label y and the knowledge distillation loss with the soft target p. Li = LCE + LKD , where LKD = −τ2KL ( p , q ) ( 3 ) The knowledge distillation loss bases on the KL divergence between p and q. τ and λ are the hyperparameters . Each subnet uses the combined loss L to update the weights of corresponding operators in the supernet .

Summary: This paper focuses on knowledge distillation for NAS, which improves the student networks by the ensemble teacher models. To generate better ensemble teacher logits, the authors proposed to sample operators that predict diverse feature maps to form subnets.  Experiments shows the proposed method achieves better results compared to state-of-the-art KD-based NAS methods.