BETANAS: Balanced Training and selective drop for Neural Architecture Search

Automatic neural architecture search techniques are becoming increasingly important in machine learning area . Especially , weight sharing methods have shown remarkable potentials on searching good network architectures with few computational resources . However , existing weight sharing methods mainly suffer limitations on searching strategies : these methods either uniformly train all network paths to convergence which introduces conflicts between branches and wastes a large amount of computation on unpromising candidates , or selectively train branches with different frequency which leads to unfair evaluation and comparison among paths . To address these issues , we propose a novel neural architecture search method with balanced training strategy to ensure fair comparisons and a selective drop mechanism to reduce conflicts among candidate paths . The experimental results show that our proposed method can achieve a leading performance of 79.0 % on ImageNet under mobile settings , which outperforms other state-of-the-art methods in both accuracy and efficiency . 1 INTRODUCTION . The fast developing of artificial intelligence has raised the demand to design powerful neural networks . Automatic neural architecture search methods ( Zoph & Le , 2016 ; Zhong et al. , 2018 ; Pham et al. , 2018 ) have shown great effectiveness in recent years . Among them , methods based on weight sharing ( Pham et al. , 2018 ; Liu et al. , 2018 ; Cai et al. , 2018 ; Guo et al. , 2019 ) show great potentials on searching architectures with limited computational resources . These methods are divided into 2 categories : alternatively training ones ( Pham et al. , 2018 ; Liu et al. , 2018 ; Cai et al. , 2018 ) and oneshot based ones ( Brock et al. , 2017 ; Bender et al. , 2018 ) . As shown in Fig 2 , both categories construct a super-net to reduce computational complexity . Methods in the first category parameterize the structure of architectures with trainable parameters and alternatively optimize architecture parameters and network parameters . In contrast , one-shot based methods train network parameters to convergence beforehand and then select architectures with fixed parameters . Both categories achieve better performance with significant efficiency improvement than direct search . Despite of these remarkable achievements , methods in both categories are limited in their searching strategies . In alternatively training methods , network parameters in different branches are applied with different training frequency or updating strength according to searching strategies , which makes different sub-network convergent to different extent . Therefore the performance of sub-networks extracted from super-net can not reflect the actual ability of that trained independently without weight sharing . Moreover , some paths might achieve better performance at early steps while perform not well when actually trained to convergence . In alternatively training methods , these operators will get more training opportunity than other candidates at early steps due to their well performance . Sufficient training in turn makes them perform better and further obtain more training opportunities , forming the Matthew Effect . In contrast , other candidates will be always trained insufficiently and can never show their real ability . Differently , One-shot methods train paths with roughly equal frequency or strength to avoid the Matthew Effect between parameters training and architectures selection . However , training all paths to convergence costs multiple time . Besides , the operators are shared by plenty of sub-networks , making the backward gradients from different training steps heavily conflict . To address this issue , we follows the balanced training strategy to avoid the Matthew Effect , and propose a drop paths approach to reduce mutual interference among paths , as shown in Fig 1 . Experiments are conducted on ImageNet classification task . The searching process costs less computational resources than competing methods and our searched architecture achieves an outstanding accuracy of 79.0 % , which outperforms state-of-the-art methods under mobile settings . The proposed method is compared with other competing algorithms with visualized analysis , which demonstrates its the effectiveness . Moreover , we also conduct experiments to analysis the mutual interference in weight sharing and demonstrate the rationality of the gradually drop paths strategy . 2 RELATED WORK . Automatic neural architecture search techniques has attracted much attention in recent years . NASNet ( Zoph & Le , 2016 ; Zoph et al. , 2018 ) proposes a framework to search for architectures with reinforcement learning , and evaluates each of the searched architectures by training it from scratch . BlockQNN ( Zhong et al. , 2018 ; Guo et al. , 2018 ; Zhong et al. , 2018 ) expands the search space to the entire DAG and selects nets with Q-learning . Network pruning methods ( Li et al. , 2019 ; Noy et al. , 2019 ) prune redundant architectures to reduces search spaces . Considering the searching policy , most of these methods depend on reinforcement learning , evolutionary algorithms and gradient based algorithms ( Bello et al. , 2017 ; Liu et al. , 2018 ; Cai et al. , 2018 ) . The most related works to our method are the ones based on weight sharing proposed by ( Pham et al. , 2018 ) , from which two streams are derived : Alternatively training methods ( Cai et al. , 2018 ; Liu et al. , 2018 ) and one-shot methods ( Brock et al. , 2017 ; Bender et al. , 2018 ; Guo et al. , 2019 ) . Methods in the first stream alternatively train architecture parameters and network parameters . During search process , operators in the super-net are selectively trained and evaluated with a certain policy and the policy is updated dynamically according to the evaluations . Among them , ENAS ( Pham et al. , 2018 ) introduces RL to select paths . DARTS ( Liu et al. , 2018 ) improves the accuracy and efficiency of paths selection policy by considering the importance of each path as trainable parameters . ProxyLessNAS ( Cai et al. , 2018 ) proposes to directly search on target datasets with single paths and makes the latency term differentiable . Single-Path NAS ( Stamoulis et al. , 2019 ) directly shares weights via super-kernel . By contrast , one-shot based methods Guo et al . ( 2019 ) ; Brock et al . ( 2017 ) ; Bender et al . ( 2018 ) firstly train each path in the super-net with equal frequency to convergence , then all the architectures are selected from the super-net and evaluated with fixed parameters . Darts+ Liang et al . ( 2019 ) improves Darts with early stop . Progressive-NAS Chen et al . ( 2019 ) gradually increases blocks while searching . HM-NAS Yan et al . ( 2019 ) uses mask to select paths . ( Sciuto et al. , 2019 ) Our work benefits from the advantages of both categories : On one hand , the importance factors are evaluated with a gradient based approach but it has no influence on training shared parameters . On the other hand , the shared parameters are updated uniformly as those in one-shot . 3 APPROACH . ProxyLessNAS ( Cai et al. , 2018 ) and Single Path One-shot ( Guo et al. , 2019 ) proposed to train the super-net with only one path on in each step to make the performance trained with weight sharing more close to that trained alone . Both of them enhance the performance of weight sharing to a higher level . ProxyLessNAS updates architecture parameters and network parameters alternatively . Paths are selectively trained according to their performance , and paths with higher performance get more training opportunities . Single Path One-shot first proposed to balanced train all paths until convergence and then use evolution algorithms to select network structures . The equivalent functions of the choice blocks in two methods are described as mPL and mOS in Eq 1 : mPL ( x ) = o1 ( x ) with probability p1 , . . . , oN ( x ) with probability p2 . , mOS ( x ) = o1 ( x ) with probability 1/N , ... , oN ( x ) with probability 1/N . ( 1 ) Our method follows the alternatively training ones , in which architecture parameters and network parameters are optimized alternatively in each step . To give a better solution to the problems discussed above , we train each candidate path with equal frequency to avoid the `` Matthew effect '' and gradually dropped least promising paths during searching process to reduce conflicts among candidate paths . 3.1 PIPELINE . The pipeline of our method is shown in Algorithm 1 . First of all , a super-net is constructed with L choice blocks O1 , O2 , . . . , OL , as shown in Fig 1 . Each choice block Ol is composed of M candidate paths and corresponding operators ol,1 , ol,2 , . . . , ol , M . The importance factor of ol , m is denoted as αl , m and αl , m are converted to probability factor pl , m using softmax normalization . Secondly , the parameters of ol , m and their importance factors αl , m are trained alternatively in Phase 1 and Phase 2 . When training αl , m , latency term is introduced to balance accuracy and complexity . Paths with αl , m lower than thα will be dropped and no more trained . Algorithm 1 Searching Process Initialization : Denote Ol as the choice block for layer l with M candidate operators { ol,1 , ol,2 , . . . , ol , M } . αl,1 , αl,2 , . . . , αl , M are the corresponding importance factors of candidate operators and initialized with identical value . Smax denotes the max number of optimization steps . 1 : while t < Smax do 2 : Phase1 : Randomly select ol , ml ∈ Ol for block Ol with uniform probability , then fix all αl , m and train the super-net constructed with the selected o1 , m1 , o2 , m2 , . . . , oL , mL for some steps . 3 : Phase2 : Fix all the parameters in ol , m and measure their flops/latency . Then evaluate each operator ol , m with both cross-entropy loss and flops/latency loss . Update αl , m according to the losses feedback . 4 : for ol , m ∈ Ol do 5 : if αl , m < thα then Ol = Ol \ { ol , m } t = t+ 1 6 : for ol , m ∈ Ol do ml = argmaxm ( αl , m ) 7 : return o1 , m1 , o2 , m2 , . . . , oL , mL Finally , after alternatively training ol , m and αl , m for given steps , paths with the highest importance factor in each choice block are selected to compose a neural architecture as the searching result . 3.2 BALANCED TRAINING . Alternatively training methods focus computational resources on most promising candidates to reduce the interference from redundant branches . However , some operators that perform well at early phases might not perform as well when they are trained to convergence . These operators might get much more training opportunities than others due to their better performance at the beginning steps . Higher training frequency in turn maintains their dominant position in the following searching process regardless their actual ability , forming the Matthew Effect . In contrast , the operators with high performance when convergent might never get opportunities to trained sufficiently . Therefore , the accuracy of alternatively training methods might degrade due to inaccurate evaluations and comparison among candidate operators . Our method follows the alternatively optimizing strategy . Differently , we only adopt gradient to architectures optimization while randomly sample paths with uniformly probability when training network parameters to avoid the Matthew Effect . More specifically , when updating network parameters of ol , m in Phase 1 and architecture parameters in Phase 2 , the equivalent output of choice block Ol is given as Opathl in Eq 2 and O arch l in Eq 3 : Opathl ( x ) = ol , m ( x ) { with probability 1M ′ , if αl , m > thα with probability 0 , else . ( 2 ) Oarchl ( x ) = ol , m ( x ) { with probability pl , m , if αl , m > thα with probability 0 , else . ( 3 ) Where M ′ is the number of remaining operators in Ol currently , and pl , m is the softmax form of αl , m . The αl , m of dropped paths are not taken into account when calculating pl , m . The parameters in both phases are optimized with Stochastic Gradient Descent ( SGD ) . In Phase 1 , the outputs in Eq 2 only depends on network parameters , thus gradients can be calculated with the Chain Rule . In Phase 2 , the outputs not only depend on the fixed network parameters but also architecture parameters αl , m . Note that Oarchl ( x ) is not differentiable with respect to αl , m , thus we introduce the manually defined derivatives proposed by Cai et al . ( 2018 ) to deal with this issue : Eq 3 can be expressed as Oarchl ( x ) = ∑ gl , m · ol , m ( x ) , where gl,0 , gl,0 , . . . , gl , M ′ is a one-hot vector with only one element equals to 1 while others equal to 0 . Assuming ∂gl , j/∂pl , j ≈ 1 according to Cai et al . ( 2018 ) , the derivatives of Oarchl ( x ) w.r.t . αl , m are defined as : ∂Oarchl ( x ) ∂αl , m = M ′∑ j=1 ∂Oarchl ( x ) ∂gl , j ∂gl , j ∂pl , j ∂pl , j ∂αl , m ≈ M ′∑ j=1 ∂Oarchl ( x ) ∂gl , j ∂pl , j ∂αl , m = M ′∑ j=1 ∂Oarchl ( x ) ∂gl , j pj ( δmj − pm ) ( 4 ) From now on , Opathl ( x ) and O arch l ( x ) are differentiable w.r.t . network parameters and architecture parameters respectively . Both parameters can be optimized alternatively in Phase 1 and Phase 2 .

Summary: This paper introduces a better searching strategy in the context of automatic neural architecture search (NAS). Especially, they focus on improving the search strategy for previously proposed computationally effective weight sharing methods for NAS. Current search strategies for the weight sharing NAS methods either focus on uniformly training all the network paths or selectively train different network paths with different frequency, where both have their own issues like wasting resources for unpromising candidates and unfair comparison among network paths. To this end, this paper proposes a balanced training strategy with “selective drop mechanism”. Further, they validate their approach by showing leading performance on ImageNet under mobile settings.