AutoSlim: Towards One-Shot Architecture Search for Channel Numbers

1 INTRODUCTION . The channel configuration ( a.k.a .. filter numbers or channel numbers ) of a neural network plays a critical role in its affordability on resource constrained platforms , such as mobile phones , wearables and Internet of Things ( IoT ) devices . The most common constraints ( Liu et al. , 2017b ; Huang et al. , 2017 ; Wang et al. , 2017 ; Han et al. , 2015a ) , i.e. , latency , FLOPs and runtime memory footprint , are all bound to the number of channels . For example , in a single convolution or fully-connected layer , the FLOPs ( number of Multiply-Adds ) increases linearly by the output channels . The memory footprint can also be reduced ( Sandler et al. , 2018 ) by reducing the number of channels in bottleneck convolutions for most vision applications ( Sandler et al. , 2018 ; Howard et al. , 2017 ; Ma et al. , 2018 ; Zhang et al. , 2017b ) . Despite its importance , the number of channels has been chosen mostly based on heuristics . LeNet5 ( LeCun et al. , 1998 ) selected 6 channels in its first convolution layer , which is then projected to 16 channels after sub-sampling . AlexNet ( Krizhevsky et al. , 2012 ) adopted five convolutions with channels equal to 96 , 256 , 384 , 384 and 256 . A commonly used heuristic , the “ half size , double channel ” rule , was introduced in VGG nets ( Simonyan & Zisserman , 2014 ) , if not earlier . The rule is that when spatial size of feature map is halved , the number of filters is doubled . This heuristic has been more-or-less used in followup network architecture designs including ResNets ( He et al. , 2016 ; Xie et al. , 2017 ) , Inception nets ( Szegedy et al. , 2015 ; 2016 ; 2017 ) , MobileNets ( Sandler et al. , 2018 ; Howard et al. , 2017 ) and networks for many vision applications . Other heuristics have also been explored . For example , the pyramidal rule ( Han et al. , 2017 ; Zhang et al. , 2017a ) suggested to gradually increase the channels in all convolutions layer by layer , regardless of spatial size . Figure 1 visually summarizes these heuristics for setting channel numbers in a neural network . Beyond the macro-level heuristics across entire network , recent works ( Sandler et al. , 2018 ; He et al. , 2016 ; Zhang et al. , 2017a ; Tan et al. , 2018 ; Cai et al. , 2018 ) have also digged into channel configuration for micro-level building blocks ( a network building block is usually composed of several 1×1 and 3×3 convolutions ) . These micro-level heuristics have led to better speed-accuracy trade-offs . The first of its kind , bottleneck residual block , was introduced in ResNet ( He et al. , 2016 ) . It is composed of 1× 1 , 3× 3 , and 1× 1 convolutions , where the 1× 1 layers are responsible for reducing and then restoring dimensions , leaving the 3 × 3 layer a bottleneck ( 4× reduction ) . MobileNet v2 ( Sandler et al. , 2018 ) , however , argued that the bottleneck design is not efficient and proposed the inverted residual block where 1 × 1 layers are used for expanding feature first ( 6× expansion ) and then projecting back after intermediate 3 × 3 depthwise convolution . Furthermore , MNasNet ( Tan et al. , 2018 ) and ProxylessNAS nets ( Cai et al. , 2018 ) included 3× expansion version of inverted residual block into search space , and achieved even better accuracy under similar runtime latency . Apart from these human-designed heuristics , efforts on automatically optimizing channel configuration have been made explicitly or implicitly . A recent work ( Liu et al. , 2018c ) suggested that many network pruning methods ( Liu et al. , 2017b ; Li et al. , 2016 ; Luo et al. , 2017 ; He et al. , 2017 ; Huang & Wang , 2018 ; Han et al. , 2015b ) can be thought of as performing network architecture search for channel numbers . Liu et al . ( Liu et al. , 2018c ) showed that training these pruned architectures from scratch leads to similar or even better performance than fine-tuning and pruning from a large model . More recently , MNasNet ( Tan et al. , 2018 ) proposed to directly search network architectures , including filter sizes , using reinforcement learning algorithms ( Schulman et al. , 2017 ; Heess et al. , 2017 ) . Although the search is performed on the factorized hierarchical search space , massive network samples and computational cost ( Tan et al. , 2018 ) are required for an optimized network architecture . In this work , we study how to set channel numbers in a neural network to achieve better accuracy under constrained resources . To start , the first and the most brute-force approach came in mind is the exhaustive search : training all possible channel configurations of a deep neural network for full epochs ( e.g. , MobileNets ( Sandler et al. , 2018 ; Howard et al. , 2017 ) are trained for approximately 480 epochs on ImageNet ) . Then we can simply select the best performers that are qualified for efficiency constraints . However , it is undoubtedly impractical since the cost of this brute-force approach is too high . For example , we consider a 8-layer convolutional networks and a search space limited to 10 candidates of channel numbers ( e.g. , 32 , 64 , ... , 320 ) for each layer . As a result , there are totally 108 candidate network architectures . To address this challenge , we present a simple and one-shot solution AutoSlim . Our main idea lies in training a slimmable network ( Yu et al. , 2018 ) to approximate the network accuracy of different channel configurations . Yu et al . ( Yu et al. , 2018 ; Yu & Huang , 2019 ) introduced slimmable networks that can run at arbitrary width with equally or even better performance than same architecture trained individually . Although the original motivation is to provide instant and adaptive accuracyefficiency trade-offs , we find slimmable networks are especially suitable as benchmark performance estimators for several reasons : ( 1 ) Training slimmable models ( using the sandwich rule ( Yu & Huang , 2019 ) ) is much faster than the brute-force approach . ( 2 ) A trained slimmable model can execute at arbitrary width , which can be used to approximate relative performance among different channel configurations . ( 3 ) The same trained slimmable model can be applied on search of optimal channels for different resource constraints . In AutoSlim , we first train a slimmable model for a few epochs ( e.g. , 10 % to 20 % of full training epochs ) to quickly get a benchmark performance estimator . We then iteratively evaluate the trained slimmable model and greedily slim the layer with minimal accuracy drop on validation set ( for ImageNet , we randomly hold out 50K samples of training set as validation set ) . After this single pass , we can obtain the optimized channel configurations under different resource constraints ( e.g. , network FLOPs limited to 150M , 300M and 600M ) . Finally we train these optimized architectures individually or jointly ( as a single slimmable network ) for full training epochs . We experiment with various networks including MobileNet v1 , MobileNet v2 , ResNet-50 and RL-searched MNasNet on the challenging setting of 1000-class ImageNet classification . AutoSlim achieves better results ( with much lower search cost ) compared with three baselines : ( 1 ) the default channel configuration of these networks , ( 2 ) channel pruning methods on same network architectures ( Luo et al. , 2017 ; He et al. , 2017 ; Yang et al. , 2018 ) and ( 3 ) reinforcement learning based architecture search methods ( He et al. , 2018 ; Tan et al. , 2018 ) . 2 RELATED WORK . 2.1 ARCHITECTURE SEARCH FOR CHANNEL NUMBERS . In this part , we mainly discuss previous methods on automatic architecture search for channel numbers . Human-designed heuristics have been introduced in Section 1 and visually summarized in Figure 1 . Channel Pruning . Channel pruning ( a.k.a. , network slimming ) methods ( Liu et al. , 2017b ; He et al. , 2017 ; Ye et al. , 2018 ; Huang et al. , 2018 ; Lee et al. , 2018 ) aim at reducing effective channels of a large neural network to speedup its inference . Both training-based , inference-time and initializationtime pruning methods have been proposed ( Liu et al. , 2017b ; He et al. , 2017 ; Ye et al. , 2018 ; Huang et al. , 2018 ; Lee et al. , 2018 ; Frankle & Carbin , 2018 ) in the literature . Here we selectively review two methods ( Liu et al. , 2017b ; He et al. , 2017 ) . He et al . ( He et al. , 2017 ) proposed an inferencetime approach based on an iterative two-step algorithm : the LASSO based channel selection and the least square feature reconstruction . Liu et al . ( Liu et al. , 2017b ) , on the other hand , trained neural networks with a ` 1 regularization on the scaling factors in batch normalization ( BN ) ( Ioffe & Szegedy , 2015 ) . By pushing the factors towards zero , insignificant channels can be identified and removed . In a recent work ( Liu et al. , 2018c ) , Liu et al.suggested that many network pruning methods ( Liu et al. , 2017b ; Li et al. , 2016 ; Luo et al. , 2017 ; He et al. , 2017 ; Huang & Wang , 2018 ; Han et al. , 2015b ) can be thought of as performing network architecture search for channel numbers . In experiments , Liu et al . ( Liu et al. , 2018c ) showed that training these pruned architectures from scratch leads to similar or even better performance than iteratively fine-tuning and pruning a large model . Thus , Liu et al . ( Liu et al. , 2018c ) concluded that training a large , over-parameterized model is not necessary to obtain an efficient final model . In our work , we take channel pruning methods ( Luo et al. , 2017 ; He et al. , 2017 ; 2018 ) as one of baselines . Neural Architecture Search ( NAS ) . Recently there has been a growing interest in automating the neural network architecture design ( Tan et al. , 2018 ; Cai et al. , 2018 ; Elsken et al. , 2018 ; Bender et al. , 2018 ; Pham et al. , 2018 ; Zoph et al. , 2018 ; Liu et al. , 2018a ; 2017a ; 2018b ; Brock et al. , 2017 ) . Significant improvements have been achieved by these automatically searched architectures in many vision and language tasks ( Zoph et al. , 2018 ; Zoph & Le , 2016 ) . However , most neural architecture search methods ( Elsken et al. , 2018 ; Bender et al. , 2018 ; Pham et al. , 2018 ; Zoph et al. , 2018 ; Liu et al. , 2018a ; 2017a ; 2018b ; Brock et al. , 2017 ) did not include channel configuration into search space , and instead applied human-designed heuristics . More recently , the RL-based searching algorithms are also applied to prune channels ( He et al. , 2018 ) or search for filter numbers ( Tan et al. , 2018 ) directly . He et al.proposed AutoML for Model Compression ( AMC ) ( He et al. , 2018 ) which leveraged reinforcement learning ( deep deterministic policy gradient ( Lillicrap et al. , 2015 ) ) to provide the model compression policy . MNasNet ( Tan et al. , 2018 ) proposed to directly search network architectures , including filter sizes , for mobile devices . In the search , each sampled model is trained on 5 epochs using an aggressive learning rate schedule , and evaluated on a 50K validation set . In total , Tan et al.sampled about 8 , 000 models during architecture search . Further , ProxylessNAS ( Cai et al. , 2018 ) proposed to directly learn the architectures for large-scale target tasks and target hardware platforms , based on DARTS ( Liu et al. , 2018b ) . For each residual block , ProxylessNAS ( Cai et al. , 2018 ) followed the channel configuration of MNasNet ( Tan et al. , 2018 ) , while inside each block , the choices can be ×3 or ×6 version of inverted residual blocks . The memory consumption issue ( Cai et al. , 2018 ; Liu et al. , 2018b ) was addressed by binarizing the architecture parameters and forcing only one path to be active .

Summary: In this paper, the authors propose a method to perform architecture search on the number of channels in convolutional layers. The proposed method, called AutoSlim, is a one-shot approach based on previous work of Slimmable Networks [2,3]. The authors have tested the proposed methods on a variety of architectures on ImageNet dataset.