FTSO: Effective NAS via First Topology Second Operator

1 INTRODUCTION . Since the great success of AlexNet ( Krizhevsky et al. , 2012 ) in image classification , most modern machine learning models have been developed based on deep neural networks . For neural networks , their performance is greatly determined by the architectures . Thus , in the past decade , a tremendous amount of work ( Simonyan & Zisserman , 2015 ; Szegedy et al. , 2015 ; He et al. , 2016 ) has been done to investigate proper network architecture design . However , as the network size has grown larger and larger , it has gradually become unaffordable to manually search for better network architectures via trial and error due to the expensive time and resource overhead . To ease this problem , a new technique called neural architecture search ( NAS ) was introduced , which allows computers to search for better network architectures automatically instead of relying on human experts .. Early-proposed reinforcement learning-based NAS methods ( Zoph & Le , 2017 ; Baker et al. , 2017 ; Zoph et al. , 2018 ) typically have an RNN-based controller to sample candidate network architectures from the search space . Although these algorithms can provide promising accuracy , their computation cost is usually unaffordable , e.g. , 1800 GPU-days are required for NASNet . To ease the search efficiency problem , one-shot approaches ( Pham et al. , 2018 ; Cai et al. , 2019 ; Liu et al. , 2019 ) with parameter sharing were proposed . These methods first create a huge directed acyclic graph ( DAG ) super-net , containing the whole search space . Then , the kernel weights are shared among all the sampled architectures via the super-net . This strategy makes it possible to measure the candidate architecture ’ s performance without repeatedly retraining it from scratch . However , these algorithms suffer from the super-nets ’ computational overheads . This problem is particularly severe for differentiable models ( Liu et al. , 2019 ; Xu et al. , 2020 ) . Limited by current NAS algorithms ’ inefficiency , it is rather challenging to find satisfying network architectures on large-scale datasets and high-level tasks . For instance , current speed-oriented NAS approaches generally require days to accomplish one search trial on ImageNet , e.g. , 8.3 GPU-days for ProxylessNAS ( Cai et al. , 2019 ) and 3.8 GPU-days for PC-DARTS ( Xu et al. , 2020 ) . Therefore , we argue that it is essential to propose a new well-defined search space , which is not only expressive enough to cover the most powerful architectures , but also compact enough to filter out the poor architectures . Motivated by Shu et al . ( 2020 ) , who demonstrated that randomly replacing operators in a found architecture does not harm the accuracy much , we believe that it could not only bring no reduction to the testing accuracy but also significantly benefit the search efficiency if we omit the influence of operators and cluster architectures according to the topology . Thus , in this paper , we propose to separately search for the network topology and the operators . We name this new method Effective NAS via First Topology Second Operator ( FTSO ) . In this paper , we mathematically prove that FTSO reduces the required parameters by 5.3× 107 and decreases the FLOPs per iteration by 1 × 105 . Besides , FTSO significantly promotes the accuracy compared to the baseline by greatly shrinking the search space , reducing the operators complexity in magnitude and lowering the required searching period from 50 epochs to one iteration . What ’ s more , the Matthew effect is eased . Furthermore , we empirically show that FTSO is superior in both efficiency and effectiveness , accomplishing the whole architecture search in 0.68 seconds . On ImageNet , FTSO can achieve 76.4 % testing accuracy , 1.5 % higher than the baseline , within merely 18 seconds . More importantly , when we only search for one iteration , FTSO can reach 75.64 % testing accuracy , 0.74 % higher than the baseline in just 0.68 seconds . Besides , if we allow FTSO to search for 19 minutes , 76.42 % Top1 and 93.2 % Top5 testing accuracy can be obtained . In addition , FTSO can reach 97.77 % testing accuracy , 0.27 % higher than the baseline , with 99.8 % of search time saved on CIFAR10 . Although in this paper we have only implemented FTSO within a continuous search space , we illustrate in Section 5 that FTSO can be seamlessly transferred to other NAS algorithms . 2 RELATED WORK . In general , existing NAS algorithms can be divided into three categories , namely , reinforcement learning-based , revolution-based and differentiable . Early-proposed reinforcement learning-based methods ( Zoph & Le , 2017 ; Zoph et al. , 2018 ) generally suffer from high computational cost and low-efficiency sampling . Instead of sampling a discrete architecture and then evaluating it , DARTS ( Liu et al. , 2019 ) treats the whole search space as a continuous super-net . It assigns every operator a real number weight and treats every node as the linear combination of all its transformed predecessors . To be specific , DARTS ’ s search space is a directed acyclic graph ( DAG ) containing two input nodes inherited from previous cells , four intermediate nodes and one output node . Each node denotes one latent representation and each edge denotes an operator . Every intermediate node xj is calculated from all its predecessors xi , i.e. , xj = ∑ i < j ∑ o∈O expαoi , j∑ o′∈O expα o′ i , j o ( xi ) , where O denotes the collection of all candidate operators , αoi , j denotes the weight for operator o from node i to j . This strategy allows DARTS to directly use gradients to optimize the whole super-net . After the super-net converges , DARTS only retains the operators with the largest weights . In this way , the final discrete architecture is derived . The main defect of DARTS is that it needs to maintain and do all calculations on a giant super-net , which inevitably leads to heavy computational overheads and over-fitting . Proposed to relieve the computational overhead of DARTS , DARTS-ES ( Zela et al. , 2020 ) reduces the number of searching epochs via early stopping , according to the Hessian matrix ’ s max eigenvalue . PC-DARTS ( Xu et al. , 2020 ) decreases the FLOPs per iteration by only calculating a proportion of the input channels and retaining the remainder unchanged , and normalizes the edge weights to stabilize the search . To be specific , in PC-DARTS , every intermediate node xj is computed from all its predecessors xi , i.e. , xj = ∑ i < j exp βi , j∑ i′ < j exp βi′ , j fi , j ( xi ) , where βi , j describes the input node i ’ s importance to the node j , and fi , j is the weighted sum of all the candidate operators ’ outputs between node i and j . Specifically , fi , j ( xi , Si , j ) = ∑ o∈O e αoi , j∑ o′∈O e αo ′ i , j o ( Si , j ∗xi ) + ( 1−Si , j ) ∗xi , where Si , j denotes a binary vector , in which only 1/K elements are 1 . 3 FTSO : EFFECTIVE NAS VIA FIRST TOPOLOGY SECOND OPERATOR . Existing NAS approaches generally suffer from the heavy computational overhead and the unsatisfying testing accuracy leaded by the huge search space . Such problems are especially stern in one-shot and differentiable methods because , these algorithms need to maintain and even do all the calculations directly on the search space . To ease such problems , it is of great demand to investigate the correlations among different architectures and to shrink the search space according to these prior knowledge . We notice that there is an important observation in Shu et al . ( 2020 ) that randomly substituting the operators does not observably influence the testing accuracy in a found architecture . Therefore , it would be a great inspiration that we could cluster the architectures according to their connection topologies . To be specific , suppose we have found an architecture only containing the simplest operators achieves high accuracy on the testing set , if we replace all the skip connections in this architecture with powerful operators , the converted architecture can also perform well on the testing set with high confidence . Thus , in this paper , we first propose to find the effective network topology with simple operators and then we fix the topology , and search for the most suitable operators for the given topology . In this way , the testing accuracy can still be guaranteed , while the search space is shrunk in magnitude . We name this new NAS algorithm Effective NAS via First Topology Second Operator ( FTSO ) . As shown in Figure 1 , we inherit the differentiable framework of PC-DARTS , and divide the architecture search into two phases . We name these two phases topology search and operator search , and illustrate how they work in Algorithm 1 and 2 , respectively . In the first phase , we form a supernet only containing the simplest operator , skip connection . Because the skip connection operator contains no kernel weights , we only need to optimize the architecture parameters βi , j , denoting the node i ’ s importance to the node j . In fact , as shown in Table 3 , the max pooling operator also brings satisfying results for the topology search . There are two reasons why we use skip connection . The first reason is that the skip connection operator not only requires zero parameter , but also produces the minimum computational cost . The second reason is that max pooling may lead to the loss of useful information if the network is deep . Furthermore , as the the only difference between our topology search and the vanilla DARTS is the number of candidate operators , the pruned architecture ’ s connectivity can be guaranteed . After the topology search , for every intermediate node j , we only retain its connections to its predecessors i∗ with the highest two βi , j similar to DARTS and PC-DARTS . In the second phase , we search for the operators suitable for the pruned topology with two strategies . One replaces each operator in the pruned topology with a mix-operator fi , j , where fi , j is the linear combination of all the candidate operators o with weight αoi , j , denoting the importance of the operator o between the node i and j . After that , we optimize the architecture parameters , αoi , j , βi , j and the kernel weights ωoi , j alternatively . After the super-net converges , we only retain one operator o ∗ with the highest αoi , j for every two connected nodes i and j . The other is to directly replace all the operators in the pruned topology with one single operator owning the highest model capacity , e.g. , a convolution operator . In this paper , we take the second strategy as the default configuration because it is much more efficient and avoid over-fitting . The final architecture outputted by the first strategy is only a small part of the super-net . If the sub-graph can generalize perfectly on the testing set , the super-net must over-fit . Thus , under this circumstance , the optimum super-net on training set is not the optimum one on testing set , and so is the sub-graph . PC-DARTS has mentioned this issue and reduced the size of the super-net via only computing partial input channels . While in the second strategy , since no super-net is adopted and all the simple operators in the sub-graph are replaced with powerful operators , the final architecture ’ s model capacity gets promoted . Additional empirical comparisons between these two strategies can be found in Section 4.4 . Algorithm 1 : topology search Input : a set of nodes : nk Output : the pruned architecture : Ap 1 Create an directed edge ei , j with weight βi , j between each pair of nodes ni and nj ( i < j ) ; 2 Assign each edge ei , j a skip connection operator oi , j with kernel weights wi , j ; 3 while still in the first epoch do 4 Forward-propagate following nj = ∑ i < j o ( ni ) βi , j ; 5 Update architecture β by descending ∇βLval ( w , β ) ; 6 Update weights w by descending ∇wLtrain ( w , β ) ; 7 foreach node nj ∈ Ap do 8 Tj ← the second largest βi , j ; 9 foreach node ni do 10 if βi , j < Tj then 11 Prune edge ei , j 12 Derive the pruned architecture Ap . Algorithm 2 : operator search Input : the pruned architecture produced by the topology search : Ap Output : the found architecture : Af 1 if replace with convolutions then 2 Replace all the retained operators oi , j in Ap with convolutions ; 3 else 4 Assign each node nj ← ∑ i < j ∑ o∈O expαoi , j∑ o′∈O expα o′ i , j o ( ni ) ; 5 while not converged do 6 Update architecture α by descending∇αLval ( w , α ) ; 7 Update weights w by descending∇wLtrain ( w , α ) ; 8 foreach edge ei , j ∈ Ap do 9 Assign edge ei , j the operator o′ ∈ O with the highest αo ′ i , j ; 10 Derive the found architecture Af ← Ap . In DARTS , the network topology and operators are jointly searched , which makes both the size and the computational cost of the super-net extremely high . We use n to denote the number of nodes , p to denote the number of candidate operators . Since we have two input nodes , one output node and n − 3 intermediate nodes , the super-net contains totally 12 ( n 2 − 3n ) edges . At the same time , every edge keeps p operators , thus , the total number of operators in DARTS is 12 ( n 2 − 3n ) p. By comparison , there are only 12n ( n−3 ) operations in our topology search , and 2 ( n−3 ) p operations in our operator search . This is because in the topology search , every edge contains only one operator ; and in the topology search , every intermediate node only connects to two predecessors . Since n is usually close to p , FTSO reduces the number of operations from O ( n3 ) to O ( n2 ) . In addition to the reduction on the number of operations , FTSO also dramatically decreases the internal cost of the operations because , during the topology search , all the powerful operators are replaced with the simple operators . To be specific , a vanilla convolutional operator needs k2CinHoutWoutCout FLOPs and ( k2Cin + 1 ) Cout parameters , where k is the kernel size , Cin is the input tensor ’ s channel number and Hout , Wout , and Cout are the output tensor ’ s height , width and channel number respectively . By comparison , a skip connection operator needs 0 parameters and 0 FLOPs . For simplicity , assume all the candidate operators are convolutions . Since DARTS has 1 2pn ( n−3 ) edges , it needs to compute 1 2pn ( n−3 ) convolutions and 1 2pn ( n−3 ) tensor summations . Because each tensor summation consumesHinWinCin FLOPs , DARTS requires totally 12pk 2n ( n− 3 ) CinHoutWoutCout+ 1 2pn ( n−3 ) HoutWoutCout = 1 2pn ( n−3 ) HoutWoutCout ( k 2Cin+1 ) FLOPs and 12n ( n − 3 ) p ( k 2Cin + 1 ) Cout parameters . While in FTSO , if we first search for the topology and then directly substituting the operators , only 12n ( n − 3 ) tensor summation need to be calculated . Thus , the total number of FLOPs and parameters of FTSO are 12n ( n − 3 ) HinWinCin and 1 2n ( n − 3 ) respectively . As a typical configuration , let k = 5 , Cin = Cout = 512 , n = 7 , p = 8 . Then , our algorithm requires only 1p ( k2Cin+1 ) Cout = 1.9 × 10 −8 times the parameters and 1 p ( k2Cin+1 ) = 9.8× 10−6 times the forward-propagation FLOPs per iteration compared to those of DARTS . FTSO ’ s huge reduction on the parameter numbers provides us a large amount of benefits . As mentioned above , it allows the algorithm to converge in only a few iterations and prevent over-fitting . This is because when extracting the discrete sub-graph from the super-net , many architecture parameters are set to be 0 . The introduced disturbance impacts more on the over-fitting super-nets since they prefer sharper local minimums . Furthermore , it avoids the Matthew ’ s effect . Each architecture has only one iteration to tune its kernel weights in DARTS . However , within one iteration , only the operators with few parameters can converge and thus , the simpler operators outperform the powerful ones in the super-net , then obtain larger gradients to enlarge their advantages . In this way , the found architectures tend to only contain the simplest operators and perform poorly on both training and testing set . Since FTSO only contains one operator with 0 parameter , the Matthew ’ s effect is eliminated .

Summary: The paper proposes a method for Neural Architecture Search (NAS) with two stages of search. In the first  stage, the topology of the cell is searched with only one operator (skip connection) using graph pruning through gradient descents. In the later stage, there are two ways to search the operators. In the first approach, the found topology is equipped with some operators (e.g., 3x3 convolution,  skip connection, and 3x3 dilated convolution) and then the architecture parameters are optimized. Another approach is to replace all operators with one single operator e.g. convolution. The experiments show that the searching time is reduced significantly compared to DARTS and the results on CIFAR-10 and ImageNet are very competitive.