Single Shot Neural Architecture Search Via Direct Sparse Optimization

1 INTRODUCTION . With no doubt , Deep Neural Networks ( DNN ) have been the engines for the AI renaissance in recent years . Dating back to 2012 , DNN based methods have refreshed the records for many AI applications , such as image classification ( Krizhevsky et al . ( 2012 ) ; Szegedy et al . ( 2015 ) ; He et al . ( 2016 ) ) , speech recognition ( Hinton et al . ( 2012 ) ; Graves et al . ( 2013 ) ) and Go Game ( Silver et al . ( 2016 ; 2017 ) ) . Considering its amazing representation power , DNNs have shifted the paradigm of these applications from manually designing the features and stagewise pipelines to end-to-end learning . Although DNNs have liberated researchers from such feature engineering , another tedious work has emerged – “ network engineering ” . In most cases , the neural networks need to be designed based on the specific tasks , which again leads to endless hyperparameters tuning and trails . Therefore , designing a suitable neural network architecture still requires considerable amounts of expertise and experience . To democratize the techniques , Neural Architecture Search ( NAS ) or more broadly , AutoML has been proposed . There are mainly two streams for NAS : The first one is to follow the pioneering work Zoph & Le ( 2017 ) , which proposed a reinforcement learning algorithm to train a Recurrent Neural Network ( RNN ) controller that generates coded architectures ( Zoph et al . ( 2018 ) ; Pham et al . ( 2018 ) ) . The second one is the evolutionary algorithm , which iteratively evaluates and proposes new models for evaluation ( Real et al . ( 2017 ) ; Stanley & Miikkulainen ( 2002 ) ) . Despite their impressive performance , the search processes are incredibly resource-hungry and unpractical for large datasets like ImageNet , though some acceleration methods have been proposed ( Zhong et al . ( 2018 ) ; Pham et al . ( 2018 ) ) . Very recently , DARTS ( Liu et al . ( 2018b ) ) proposed a gradient-based method in which the connections are selected by a softmax classifier . Although DARTS achieves impressive performance with great acceleration , its search space is still limited to fix-length coding and blocksharing search as in previous works . In this work , we take another view to tackle these problems . We reformulate NAS as pruning the useless connections from a large network which contains the complete network architecture hypothesis space . Thus only one single model is trained and evaluated . Since the network structure is directly optimized during training , we call our method Direct Sparse Optimization NAS ( DSONAS ) . We further demonstrate that this sparse regularized problem can be efficiently optimized by a modified accelerated proximal gradient method opposed to the inefficient reinforcement learning or revolutionary search . Notably , DSO-NAS is much simpler than the existing search methods as it unifies the neural network weight learning and architecture search into one single optimization problem . DSO-NAS does not need any controller ( Zoph & Le ( 2017 ) ; Zoph et al . ( 2018 ) ; Pham et al . ( 2018 ) ) or performance predictor ( Liu et al . ( 2018a ) ) or relaxation of the search space ( Zoph & Le ( 2017 ) ; Zoph et al . ( 2018 ) ; Pham et al . ( 2018 ) ; Liu et al . ( 2018b ) ) . As a result of the efficiency and simplicity , DSO-NAS first demonstrate that NAS can be directly applied to large datasets like ImageNet with no block structure sharing . Our experiments show that DSO-NAS can achieve 2.84 % average test error on CIFAR-10 , as well as top-1 error 25.4 % on ImageNet with FLOPs ( the number of multiply-adds ) under 600M . In summary , our contributions can be summarized as follows : • We propose a novel model pruning formulation for neural architecture search based on sparse optimization . Only one model needs to be trained during the search . • We propose a theoretically sound optimization method to solve this challenging optimization problem both effectively and efficiently . • We demonstrate the results of our proposed method are competitive or better than other NAS methods , while significantly simplifying and accelerating the search process . 2 RELATED WORKS . In this section , we briefly review two research fields that may be related to our work . 2.1 NETWORK PRUNING . Network pruning is a widely used technique for model acceleration and compression . The early works of pruning focus on removing unimportant connections ( LeCun et al . ( 1990 ) ; Hassibi & Stork ( 1993 ) ; Han et al . ( 2015 ) ; Guo et al . ( 2016 ) ) . Though connection level pruning can yield effective compression , it is hard to harvest actual computational savings because modern GPU can not utilize the irregular weights well . To tackle this issue , a significant amount of works on structure pruning have been proposed . For neuron level pruning , several works prune the neurons directly by evaluating the importance of neurons based on specific criteria ( Hu et al . ( 2016 ) ; Li et al . ( 2017 ) ; Mariet & Sra ( 2016 ) ; Liu et al . ( 2017 ) ) . More generally , Wen et al . ( 2016 ) proposed the sparse structure learning . They adopted group sparsity on multiple structures of networks , including filter shapes , channels and layers . Recently , Huang & Wang ( 2018 ) proposed a simpler way for structure pruning . They introduced scaling factors to the outputs of specific structures ( neural , groups or block ) and apply sparse regularizations on them . After training , structures with zero scaling factors can be safely removed . Compared with ( Wen et al . ( 2016 ) ) , the proposed method is more effective and stable . In this work , we extend ( Huang & Wang ( 2018 ) ) into a more general and harder case , neural architecture search . 2.2 NEURAL ARCHITECTURE SEARCH . Recently , there has been growing interest in developing methods to generate neural network architecture automatically . One heavily investigated direction is evolutionary algorithm ( Meyer-Lee et al . ; Miller et al . ( 1989 ) ; Real et al . ( 2017 ) ; Stanley & Miikkulainen ( 2002 ) ) . They designed the modifications like inserting layers , changing filter sizes or adding identity mapping as the mutations in evolution . Not surprisingly , their methods are usually computationally intensive and less practical in large scale . Another popular direction is to utilize reinforcement learning with an RNN agent to design the network architecture . The pioneering work ( Zoph & Le ( 2017 ) ) applies an RNN network as the controller to sequentially decide the type and parameters of layers . Then the controller is trained by RL with the reward designed as the accuracy of the proposed model . Although it achieves remarkable results , it needs 800 GPUs to get such results , which is not affordable for broad applications . Based on this work , several methods have been proposed to accelerate the search process by limiting the search space ( Zoph et al . ( 2018 ) ) , early stopping with performance prediction ( Zhong et al . ( 2018 ) ) , progressive search ( Liu et al . ( 2018a ) ) or weight sharing ( Pham et al . ( 2018 ) ) . Despite their success , the aforementioned methods treat the search of network architecture as a black-box optimization problem . Moreover , the search spaces of them are limited due to the fixed-length coding of architecture . More recently , Ashok et al . ( 2018 ) ; He et al . ( 2018 ) integrated reinforcement learning into model compression . For efficient search , the action space is constrained as pruning ratio in He et al . ( 2018 ) . Though they only applied this method on channel selection , it also can be seen as an architecture search approach . Similarly , Véniat & Denoyer ( 2018 ) also constrained the search space by predefining it as a super-network , which is composed of a set of layers connected together . In order to learn budget-aware network structure , they proposed a budget aware objective function and solved it by policy gradient inspired algorithm . Though the building of the search space is similar to our method , the network structure is limited to ResNet fabric and convolutional neural fabrics . Our most related work is a gradient based method DARTS ( Liu et al . ( 2018b ) ) . In DARTS , a special parameter a is applied on every connection and updated during training process . A Softmax classifier is then applied to select the connection to be used for nodes . However , the search space of DARTS is also limited : every operation can only have exact two inputs ; the number of nodes are fixed within a block . 3 PROPOSED METHOD . In this section , we will elaborate the details of our proposed method . We will start with the intuition and motivations , then followed by the design of search space and the formulation of our method . Lastly , we will describe the optimization and training details . 3.1 MOTIVATIONS . The idea of DSO-NAS follows the observation that the architecture space of neural network ( or a micro structure in it ) can be represented by a completely connected Directed Acyclic Graph ( DAG ) . Any other architecture in this space can be represented by a sub-graph of it . In other words , a specific architecture can be obtained by selecting a subset of edges and nodes in the full graph . Prior works ( Zoph & Le ( 2017 ) , Liu et al . ( 2018a ) , Liu et al . ( 2018b ) ) focus on searching the architecture of two types of blocks , convolution block and reduction block . Following the idea of micro structure searching , we adopt the complete graph to represent the search space of an individual block . Then the final network architecture can be represented by a stacking of blocks with residual connections . Fig . 1 illustrates an example DAG of a specific block , whose nodes and edges represent local computation O and information flow , respectively . where λ ( i ) ( j ) is the scaling factor applied on the information flow from node j to i . Then we apply sparse regularizations on the scaling parameters to force some of them to be zero in the search . Intuitively , if λ ( i ) ( j ) is zero , the corresponding edge can be removed safely and isolated nodes can also be pruned as no contribution is made . 3.2 SEARCH SPACE . DSO-NAS can search the structure of each building block in DNN , and then share it for all the blocks in the DNN , just as all previous works did . It can also directly search the whole network structure without block structure sharing , while still keeping a competitive searching time . In the following , we will discuss the search space of each individual block first , and then specify the search space of macro-structure . A block consists of M sequential levels which are composed of N different kinds of operations . In each block , every operation has connections with all the operations in the former levels and the input of the block . Also , the output of the block is connected with all operations in the block . Then for each connection , we scale its output by a multiplier λ , and impose a sparse regularization on it . After optimization , the final architecture is generated by pruning the connections whose corresponding λ are zero and all isolated operations . The procedure of block search is illustrated in Fig . 2 . Formally , the output of the j-th operation in the i-th layer of the b-th block h ( b , i , j ) is computed as : h ( b , i , j ) = O ( b , i , j ) ( i−1∑ m=1 N∑ n=1 λ ( i , j ) ( b , m , n ) h ( b , m , n ) + λ ( i , j ) ( b,0,0 ) O ( b−1 ) ) , ( 3 ) where O ( b , i , j ) is the transformation of the j-th operation in the i-th layer of the b-th block , λ ( i , j ) ( b , m , n ) is the scaling factor from node h ( b , m , n ) to h ( b , i , j ) , and O ( b−1 ) is the output of the ( b− 1 ) -th block . Here we denote h ( b,0,0 ) = O ( b−1 ) as the input node and h ( b , M+1,0 ) = O ( b ) as the output node of the b-th block , respectively . The operation in the m-th layer may have ( m − 1 ) N + 1 inputs . Note that the connections between operations and the output of block are also learnable . The output of the b-th block O ( b ) is obtained by applying a reduction operation ( concatenation followed by a convolution with kernel size 1×1 ) R to all the nodes that have contribution to the output : O ( b ) = R ( [ λ ( M+1,0 ) ( b,1,1 ) h ( b,1,1 ) , λ ( M+1,0 ) ( b,1,2 ) h ( b,1,2 ) , ... , λ ( M+1,0 ) ( b , m , n ) h ( b , m , n ) , ... λ ( M+1,0 ) ( b , M , N ) h ( b , M , N ) ] ) +O ( b−1 ) , m ∈ [ 1 , M ] , n ∈ [ 1 , N ] ( 4 ) where identity mapping is applied in case all operations are pruned . The structure of whole network is shown in Fig . 3 : a network consists of S stages withB convolution blocks in every stage . Reduction block is located at the end of each stage except the last stage . We try two search spaces : ( 1 ) the share search space where λ is shared among blocks . ( 2 ) the full search space where λ in different blocks are updated independently . As for reduction block , we simply use convolutions with kernel size 1×1 and 3×3 , and apply them with a stride of 2 to reduce the size of feature map and double the number of filters . The output of reduction block is the sum of these two convolutions . The task of searching blocks therefore reduces to learning λ on every edge , which can be formulated as : min W , λ 1 K K∑ i=1 L ( yi , Net ( xi , W , λ ) ) + δ‖W‖2F + γ‖λ‖1 , ( 5 ) where xi and yi are input data and label respectively , K denotes the number of training samples , W represents the weights of network . δ and γ represent the weights of regularizations , respectively .

Summary: This paper proposes a neural architecture search method based on a direct sparse optimization, where the proposed method provides a novel model pruning view to the neural architecture search problem. Specifically, the proposed method introduces scaling factors to connections between operations, and impose sparse regularizations to prune useless connections in the network. The proposed method is evaluated on CIFAR-10 and ImageNet dataset.