Neural Architecture Search via Ensemble-based Knowledge Distillation

1 INTRODUCTION . Neural architecture search ( NAS ) ( Elsken et al. , 2019 ) can automatically search for the optimal neural architecture from a given search space . Neural architectures discovered by NAS have surpassed manually designed ones in accuracy on various tasks ( Zoph & Le , 2016 ; Pham et al. , 2018 ; Bender et al. , 2018 ) . Instead of parameterizing each searched architecture from scratch , one-shot NAS methods ( Liu et al. , 2018 ; Cai et al. , 2018 ; Guo et al. , 2020b ) have been wildly used recently due to their moderate computational cost . One-shot NAS achieves this by letting a whole NAS search space share the parameters and form a supernet , and the parameterization of the supernet only requires one training process . Unfortunately , despite its efficiency , one-shot NAS is often denounced for its imprecise estimation of candidate architectures ’ quality ( Bender et al. , 2018 ; Li et al. , 2020c ; Zhang et al. , 2020 ) , which is evaluated by letting each candidate subnet directly inherit parameters of the supernet . Several studies have attributed this imprecise estimation in one-shot NAS to the insufficient parameterization of search space , caused by two major factors : ( 1 ) interference between architectures ( Li et al. , 2020c ) and ( 2 ) unfair training opportunities among candidates ( Chu et al. , 2019 ) . Among amendments to address this insufficient parameterization issue , Knowledge Distillation ( KD ) ( Hinton et al. , 2015 ) is a promising technique widely used recently ( Cai et al. , 2019 ; Yu et al. , 2020 ; Li et al. , 2020a ) . It adopts an external teacher to regulate the supernet training to improve the parameterization efficiency , resulting in a higher quality of searched-out architecture compared with conventional one-shot methods ( Guo et al. , 2020b ) . Unfortunately , although knowledge distillation is promising , how to select an high-quality teacher remains an open challenge . On known datasets and tasks , existing work typically chooses a trained , known-to-be-good network as the teacher . DNA ( Li et al. , 2020a ) and BigNAS ( Yu et al. , 2020 ) bring in high accuracy guidance from a trained network to perform supervised KD . Although the accuracy of a teacher is critical , this does not solely guarantee good teaching results ( Cho & Hariharan , 2019 ; Mirzadeh et al. , 2020 ) . An unsupervised method , Cream ( Peng et al. , 2020 ) , surpassed supervised ones recently . It dynamically maintains a pool of subnets with superior accuracy as the teachers to teach the remaining ones . Cream focuses more on the consistency and complementarity between teachers and students for higher KD efficiency , which accounts for its high performance in test accuracy . But it still fails to find a high-quality teacher . On the contrary , it ignores the fundamental factor , i.e. , the accuracy of teachers . The average accuracy gap between teachers and students is 2.1 % in Cream , suggesting that teachers do not have much extra knowledge to teach . This inferior teacher accuracy greatly reduces the efficiency of KD and will constrain the upper bound of students ’ final performance . To conclude , no prior work achieves both high accuracy and good suitability when selecting the teachers . In order to better meet the above two-fold requirement for high-quality teachers of KD one-shot NAS , we focus on improving the accuracy of teachers selected in unsupervised methods . Recent advances in ensemble online knowledge distillation ( Zhang et al. , 2018 ; Lan et al. , 2018 ; Guo et al. , 2020a ) managed to train multiple networks simultaneously and efficiently via knowledge distillation without pre-training a teacher . Following them , in this paper , we propose EnNAS , an ensemblebased knowledge distillation method for NAS . Specifically , For each input batch , EnNAS samples a set of candidate subnets as students , assembles outputs from the students as a teacher , and performs knowledge distillation between the assembled teacher and each student . Ensemble can address the problem of low teacher accuracy in previous unsupervised works , while maintaining other good characters inherited from them , including good complementarity between teachers and students . A challenge is how to select the set of subnets for maximum knowledge distillation effect each time , i.e. , selecting a set that can be assembled into a teacher of higher accuracy to teach themselves better . Ensemble tends to yield better results when diversity presents among the outputs of networks ( Kuncheva & Whitaker , 2003 ) . Therefore , we propose a diversity-based sampling strategy to select subnets generating more diverse outputs than random sampling . Precisely , EnNAS measures the feature map similarity between operators in the same layer . An operator has a higher probability of being sampled when its similarity with other operators in the same layer is low . By doing so , the subnets generate more diverse feature maps and produce more diverse outputs . Our experiments demonstrate that the diversity-based sampling strategy managed to assemble teachers that are 5.8 % more accurate than students on average . This work makes the following contributions : • We propose to combine unsupervised learning and ensemble learning into the training process of KD one-shot NAS . It manages to raise the accuracy of teachers while preserves beneficial factors of unsupervised learning , The teachers it employed meet the two-fold requirement for teachers in KD one-shot NAS better than ever . • We design a diversity-based sampling strategy for EnNAS to generate high-quality ensembles . It guarantees that the selected student subnets generate diverse outputs . Based on the diverse outputs , the assembled teacher has higher accuracy than previous works . Teachers in EnNAS reach 5.8 % higher accuracy than students on average , while the best previous work ( Peng et al. , 2020 ) reaches only 2.1 % . This increases the upper bound of accuracy in KD one-shot NAS . • EnNAS sets up a new SOTA in one-shot NAS on Imagenet . Extensive evaluations show that compared with existing works , EnNAS improved the accuracy of produced networks by 1.2 % on average and at most 3.3 % on ImageNet . 2 RELATED WORKS . 2.1 ONE-SHOT NEURAL ARCHITECTURE SEARCH . Neural Architecture Search ( NAS ) has become the mainstream approach to automate the manual process of architecture design , which significantly reduces the substantial effort of human experts ( Zoph & Le , 2016 ; Pham et al. , 2018 ) . Existing NAS strategies can be classified into two categories : sample-based NAS and weight-sharing NAS . In sample-based NAS strategies ( Zoph et al. , 2018 ; Wang et al. , 2020 ; Zhong et al. , 2020 ) , subnets are sampled from a huge search space with different sampling strategies ( e.g. , randomized sampling ( Bender et al. , 2018 ) , uniform sampling ( Dong & Yang , 2019 ) ) , and then fully trained on the target dataset from scratch . However , training each subnet from scratch is expensive . Hence , sampled-based NAS strategies only can support architecture searching within small target datasets ( e.g. , CIFAR-10 ) . On the contrary , the weight-sharing strategies ( Khodak et al. , 2019 ; Li et al. , 2020b ) ( a.k.a. , one-shot strategies ) can greatly reduce the training cost , supporting large datasets like ImageNet . State-ofthe-art weight-sharing NAS , such as Differentiable Architecture Search ( DARTS ) ( Liu et al. , 2018 ) and its variants ( e.g. , P-DARTS ( Chen et al. , 2019 ) ) , encode the search space into a supernet . A supernet is composed of all candidate operations ( e.g. , conv3x3 ) and maintains weights for each operation . The supernet is only trained once . All subnets sampled from the supernet can directly inherit the weights in the supernet that share common graph nodes . Thus , the qualities of subnets are directly queried on the inherited weights . However , despite many benefits , one-shot strategies with weight sharing suffer from weight coupling : the operators of a supernet are jointly optimized , and from a micro view , each subnet ’ s weight is interfered by other subnets . Thus , one-shot strategies produce subnets with low validation accuracy compared with their ground-truth modeling ability , especially on large datasets ( e.g. , ImageNet ) . To reduce such coupling , SPOS ( Guo et al. , 2020b ) in each training step samples one subnet to be trained , and weights are still shared between subnets . 2.2 NEURAL ARCHITECTURE SEARCH WITH KNOWLEDGE DISTILLATION . Knowledge distillation ( Hinton et al. , 2015 ) for the neural network is to transfer the knowledge from a trained teacher to a smaller student . Existing works on knowledge distillation can be roughly classified into two categories . The first category is to use soft labels generated by the teacher to teach the student , which was first proposed by ( Ba & Caruana , 2013 ) . Hinton et al . ( Hinton et al. , 2015 ) redefined knowledge distillation as training a shallower network to approach the teacher ’ s output after the softmax layer . However , when the teacher model gets deeper , learning the soft-labels alone is insufficient . To address this problem , the second category proposes to employ the internal representation of the teacher to guide the training of the student ( Romero et al. , 2014 ; Zhang et al. , 2017 ; Yim et al. , 2017 ; Wang et al. , 2018 ) . There are already literatures that adopt knowledge distillation to accelerate NAS . Cream ( Peng et al. , 2020 ) lets each subnet , which is sampled for each training step , not only learn from the dataset , but also learn from a teacher subnet with the highest up-to-date accuracy . However , Cream still wastes learning opportunities and fails to efficiently spread the knowledge throughout the supernet . DNA ( Li et al. , 2020a ) adopts an external teacher DNN that learns the dataset to supervise the NAS process , and all other NAS candidates learn from the teacher . However , the learned knowledge is highly bounded by the external teacher . In this work , EnNAS iteratively generates the teacher through the ensembling of multiple student subnets , and then the ensemble teacher guides the training of those student subnets . Thus , EnNAS ’ s strategy is not bounded by any static teacher , and the learning process is totally automatic and unsupervised . 3 METHOD . 3.1 OVERVIEW . We first give an overview of our proposed method EnNAS that accelerates one-shot NAS through ensembling-based knowledge distillation . In each iteration , EnNAS selects multiple subnets from the supernet and optimizes these subnets simultaneously . The ensembling soft targets are generated by aggregates logits of all subnets . All the subnets learn from the ground-truth labels as well as the soft targets . The subnets are selected by the diversity-based sampling strategy of EnNAS , which takes the operator diversity into consideration . 3.2 ONLINE DISTILLATION . Suppose we have selected m subnets from the supernet for a given input batch { x , y } , where x is the input sample and y is the ground-truth label , the i− th subnet outputs logit zi and produces soft probability distribution qi = softmax ( zi ) . The logit t of the ensemble teacher is a weighted sum of all the subnet logits t = m−1∑ i=0 wizi , subject to m−1∑ i=0 wi = 1 ( 1 ) where wi is the weight corresponds to logit of the i− th subnet . Following the ensembling method proposed in Guo et al . ( 2020a ) , the teacher logit is calculated by minimizing the cross-entropy loss of the ensemble teacher logit t and the ground-truth label y . Let w = { w0 , w1 , · · · , wm−1 } and Z = { zT0 ; zT1 ; · · · ; zTm−1 } , the problem is formalized as min w LCE ( wTZ , y ) ( 2 ) The problem is convex and we solve it using the ECOS solver ( Domahidi et al. , 2013 ) . After generating the ensemble teacher logit , we can get the teacher soften probability distribution p = softmax ( t ) . The loss Li of the i − th subnet is then a combination of the cross-entropy loss with the ground-truth label y and the knowledge distillation loss with the soft target p. Li = LCE + LKD , where LKD = −τ2KL ( p , q ) ( 3 ) The knowledge distillation loss bases on the KL divergence between p and q. τ and λ are the hyperparameters . Each subnet uses the combined loss L to update the weights of corresponding operators in the supernet .

Summary: In this paper, authors propose to dynamically construct a teacher network from the ensemble of multiple student networks, and then use it to guide the training of super-net via knowledge distillation. The ensemble of student networks is built with the diversity criterion explicitly considered to improve the teacher network's accuracy. After the super-net is well trained, the optimal network architecture is determined by evolution search.