CNAS: Channel-Level Neural Architecture Search

1 INTRODUCTION . Nowadays , deep neural networks ( DNNs ) are used extensively and successfully in many fields and applications such as computer vision , speech recognition , machine translation , and automated vehicles . Designing DNNs often requires significant architecture engineering , a large amount of trial and error by experts . Although transfer learning is widely used to save the efforts required for designing good architectures of DNNs from scratch , it is not always possible to use . Recently , there is growing interest in automating designing good neural network architectures ( 30 ; 31 ; 20 ; 24 ; 21 ; 15 ; 3 ; 28 ; 14 ; 2 ; 22 ; 7 ; 27 ; 4 ; 29 ; 10 ) . Most of them can be categorized into reinforcement learning-based ( RL ) methods , evolutionary algorithm-based ( EV ) methods , hypernetwork-based ( HY ) methods , and gradient-based ( GR ) methods , in terms of the search algorithm . RL methods ( 30 ; 31 ; 20 ; 24 ) use a controller model that enumerates a bunch of candidate models , which are trained for a fixed number of epochs from scratch , and then , is updated using the validation accuracies of the candidate models evaluated on a validation set . To reduce the search space of candidate models , some of them ( 31 ; 20 ) assume each model is composed of multiple convolutional layers called cells having the same architecture and focuses on searching for the best cell architecture . For example , in NASNet ( 31 ) , a cell is composed of five blocks , and each block composed of two operations , which are selected among a set of various convolution and pooling operations by the controller model . To reduce the search space , NASNet also transfers the learned architecture for a small dataset ( e.g. , CIFAR-10 ) to a large dataset ( e.g. , ImageNet ) . To optimize an architecture with less amount of computation , ENAS ( 20 ) exploits parameter ( weight ) sharing , which avoids training each candidate model from scratch by sharing the weights of candidate models . It constructs a large computational graph , where each subgraph represents the architecture of a candidate model , and the controller model is trained to search for a subgraph corresponding to a good candidate model . EV methods ( 23 ; 1 ; 12 ; 17 ; 26 ; 21 ; 15 ) also have been extensively studied . AmoebaNet ( 21 ) uses the same search space with NASNet , but searches a good cell architecture based on evolutionary algorithm instead of RL controller . The population is initialized with models with random architectures , and some models are sampled from the population . The model with the highest validation fitness within the samples is selected as the parent ( i.e. , exploitation ) , and a child having a mutation in terms of operations and skip connections is constructed from the parent ( i.e. , exploration ) . Hierarchical NAS ( 15 ) uses hierarchical representation for cell architecture where smaller graph motifs are used as building blocks to form larger motifs , instead of flat representation . Unfortunately , most of RL and EV methods , except ENAS , require an enormous amount of computing power for training thousands of child models . They usually need hundreds or thousands of GPU days for architecture search , which is almost impossible for a typical machine learning practitioner . HY methods and GR methods avoid such a large cost of architecture search by sharing parameters as in ENAS ( 20 ) . HY methods ( 3 ; 28 ) bypass fully training candidate models by instead training an auxiliary model , a HyperNet ( 8 ) , to dynamically and directly generate the weights of a candidate model . SMASH ( 3 ) generates an architecture of an entire network ( i.e. , macro search ) in terms of the hyperparameters of filters ( e.g. , number , size ) with fixing the type of operation in the HyperNet space , while GHN ( 28 ) generates an architecture of a cell ( i.e. , micro search ) in terms of operations in the NAS search space . GR methods ( 22 ; 7 ; 27 ; 4 ; 29 ; 10 ) do not rely on controllers , evolutionary algorithm , and hypernetworks , but exploit gradient descent on network architectures , which can significantly improve the speed of NAS . They basically relax the search space to be continuous , so that the architecture can be optimized with respect to its validation set performance by gradient descent . Here , search space corresponds to a parent network , and a child network ( subgraph ) can be derived from the parent network by gradient descent . Most of GR methods focus on searching a good cell architecture in terms of operations and repeating the same architecture as in NASNet . After architecture search , they should usually re-train the candidate architecture snapshot from scratch using the training set due to inconsistency between the performance of derived child networks and converged parent networks . As described above , one of the major trends in NAS is exploiting the concept of parameter sharing through hypernetwork or gradient descent in order to reduce the cost ( i.e. , GPU days ) of NAS . By parameter sharing , HY and GR methods can automatically optimize an architecture that can achieve the state-of-the-art performance on CIFAR-10 and ImageNet just within a few days ( as summarized in Table 4 ) . However , there is still a challenging problem in the above architecture search methods : designing search space . In principle , the search space should be large and expressive enough to capture a diverse set of promising candidate models , and at the same time , should be small enough to train with the limited amount of resources and time ( 2 ) . Some methods ( 24 ; 15 ; 22 ) addressed that defining search space is extremely important for the performance of neural architecture search . The problem about designing search space may not be solved at once . The search space of the existing NAS methods is typically defined with a shape of the overall network and a set of operations such as identity , normal convolution , separable convolution , average pooling , and max pooling . Many of them follow the NASNet search space for the shape of the network ( i.e. , stacking cells ) and define their own set of operations . Since the number of possible types of operations for search space is limited due to the search cost , the set of operations used itself may have a large impact on the performance of architecture search . In this paper , we investigate the possibility of achieving competitive performance with the state-of-theart architecture search methods with using a fixed type of operation . To achieve such a performance , we focus on the sparsity of a model . A candidate model in the existing methods has multiple types of operations connected with each other via skip connections , and each operation takes the entire feature maps ( called channel ) of certain previous nodes or cells as input and returns its entire resulting channels as output . Thus , the candidate model can be regarded as a dense model in term of input and output channels of the operations . We propose a channel-level neural architecture search ( CNAS ) method that regards channels as vertices and a single fixed operation as edges and searches for a good architecture by gradient descent . The resulting model is sparse in terms of channels . CNAS uses the existing shape of search space ( e.g. , NASNet ) , but performs macro search . Thus , the resulting architecture has different topology at different cells . In CNAS , the final sparse architecture can be searched quickly due to its simplicity , and at the same time , can compensate for the disadvantage of using homogeneous operation due to its sparsity . For CIFAR-10 , CNAS searches for the architecture in 1.1 GPU days , which achieves 2.28 % test error with 4.6 million parameters and autoaugment . The rest of the paper is organized as follows . Section 2 explains our method CNAS . Section 3 shows the experimental results , and Section 4 summarizes the characteristics of related methods . Section 5 concludes this paper . 2 CNAS METHOD . Since we focus on investigating the possibility of architecture search relying on sparsity instead of the combination of operations in this paper , we mainly use the structure of NASNet for the shape of search space , which is composed of normal cells and reduction cells , and each cell is again composed of submodules called nodes ( blocks in NASNet ) . In Section 3 , we will show the result of CNAS using different shape of search space , in particular , the structure of DenseNet . Figure 1 shows the diagram of the search space of CNAS . 2.1 SEARCH SPACE . The CNAS method consists of the following three steps : ( 1 ) Train the one-shot ( i.e. , full-edges ) model in a fixed number of epochs to make it predictive of the validation accuracies of sparse models . ( 2 ) Search the most promising sparse model satisfying a given sparsity based on a criteria ( e.g. , Taylor ) by zeroing out less important channels . ( 3 ) Re-train the most promising model from scratch ( called CNAS-R ) or fine-tune it ( called CNAS-W ) and then evaluate the final model on the test dataset . In CNAS , a vertex in a cell or a node is a single channel , and an edge is an operation . In Figure 1 ( b ) and ( c ) , the thick edges are non-trivial operations involving convolution , where the solid thick ones are for pre-processing as in other methods , and the dotted solid thick ones are the part that can be changed by architecture search . The type of operation used in CNAS is fixed as a specific one , depthwise separable 3x3 convolution since operations are not the target of architecture search . In contrast , the number of types of operations in the existing NAS methods is at least several , and the types of operations are designed differently depending on the method . In Figure 1 ( c ) , zeroing out less important channels of red squares also removes their outgoing edges to the next layer { yi } , i.e. , does not apply the operation BN ( Conv ( ReLU ( · ) ) ) to the red squares . In terms of low-level implementation , CNAS performs partial matrix operations between { y1 , y2 } and { x3 , x4 } and between { y3 , y4 } and { x5 } . Since the red edges do not need to be calculated , the corresponding convolution kernels are also not necessary . Thus , the number of weight parameters between { xi } and { yi } is reduced by 38 in Node 1 . That is , channel-level architecture search makes the model sparse . If { x1 , x2 , x3 , x4 } all are removed , then { y1 , y2 } are also removed , and only { y3 , y4 } is added to { z1 , z2 } . In general , the input channels of nodes ( i.e. , dotted squares ) are removed differently depending on whether the cell is close to input data or close to the output layer . Thus , after architecture search , each cell in CNAS has different architecture in terms of the topology of vertices and edges .

Summary: This paper aims to search a sparse but competitive architecture with using a single fixed type of operation by proposing a channel-level neural architecture search (CNAS). Different from most previous NAS works, this paper conducts NAS process on channel-level such that different cell has different topology. CNAS provides a heuristic algorithm to calculate the saliency vector and zero out the channels iteratively until satisfying a given sparsity. This paper performs CNAS on Cifar-10 and ImageNet, and analyzes the topological properties of the final model. The results of experiment demonstrate CNAS can reach a competitive model with dense models searched by baselines.