Growing Efficient Deep Networks by Structured Continuous Sparsification

1 INTRODUCTION . Deep neural networks are the dominant approach to a variety of machine learning tasks , including image classification ( Krizhevsky et al. , 2012 ; Simonyan & Zisserman , 2015 ) , object detection ( Girshick , 2015 ; Liu et al. , 2016 ) , semantic segmentation ( Long et al. , 2015 ; Chen et al. , 2017 ) and language modeling ( Zaremba et al. , 2014 ; Vaswani et al. , 2017 ; Devlin et al. , 2019 ) . Modern neural networks are overparameterized and training larger networks usually yields improved generalization accuracy . Recent work ( He et al. , 2016 ; Zagoruyko & Komodakis , 2016 ; Huang et al. , 2017 ) illustrates this trend through increasing depth and width of convolutional neural networks ( CNNs ) . Yet , training is compute-intensive , and real-world deployments are often limited by parameter and compute budgets . Neural architecture search ( NAS ) ( Zoph & Le , 2017 ; Liu et al. , 2019 ; Luo et al. , 2018 ; Pham et al. , 2018 ; Savarese & Maire , 2019 ) and model pruning ( Han et al. , 2016 ; 2015 ; Guo et al. , 2016 ) methods aim to reduce these burdens . NAS addresses an issue that further compounds training cost : the enormous space of possible network architectures . While hand-tuning architectural details , such as the connection structure of convolutional layers , can improve performance ( Iandola et al. , 2016 ; Sifre & Mallat , 2014 ; Chollet , 2017 ; Howard et al. , 2017 ; Zhang et al. , 2018 ; Huang et al. , 2018 ) , a principled way of deriving such designs remains elusive . NAS methods aim to automate exploration of possible architectures , producing an efficient design for a target task under practical resource constraints . However , during training , most NAS methods operate on a large supernet architecture , which encompasses candidate components beyond those that are eventually selected for inclusion in the resulting network ( Zoph & Le , 2017 ; Liu et al. , 2019 ; Luo et al. , 2018 ; Pham et al. , 2018 ; Savarese & Maire , 2019 ) . Consequently , NAS-based training may typically be more thorough , but more computationally expensive , than training a single hand-designed architecture . Model pruning techniques similarly focus on improving the resource efficiency of neural networks during inference , at the possible expense of increased training cost . Common strategies aim to generate a lighter version of a given network architecture by removing individual weights ( Han et al. , 2015 ; 2016 ; Molchanov et al. , 2017 ) or structured parameter sets ( Li et al. , 2017 ; He et al. , 2018 ; Luo et al. , 2017 ) . However , the majority of these methods train a full-sized model prior to pruning and , Level 1 : Channel-wise Training Epochs T=0,1,2… Level 2 : Layer-w ise … ( a ) Growing CNN Layers and Filters Training Epochs FL O Ps Our Method Pruning Architecture Search ( b ) Epoch-wise Training Cost NAS Pruning Ours FL O Ps ( c ) Total Training Cost Figure 1 : Growing Networks during Training . We define an architecture configuration space and simultaneously adapt network structure and weights . ( a ) Applying our approach to CNNs , we maintain auxiliary variables that determine how to grow and prune both filters ( i.e . channel-wise ) and layers , subject to practical resource constraints . ( b ) By starting with a small network and growing its size , we utilize fewer resources in early training epochs , compared to pruning or NAS methods . ( c ) Consequently , our method significantly reduces the total computational cost of training , while delivering trained networks of comparable or better size and accuracy . after pruning , utilize additional fine-tuning phases in order to maintain accuracy . Hubara et al . ( 2016 ) and Rastegari et al . ( 2016 ) propose the use of binary weights and activations , allowing inference to benefit from reduced storage costs and efficient computation through bit-counting operations . Yet , training still involves tracking high-precision weights alongside lower-precision approximations . We take a unified view of pruning and architecture search , regarding both as acting on a configuration space , and propose a method to dynamically grow deep networks by continuously reconfiguring their architecture during training . Our approach not only produces models with efficient inference characteristics , but also reduces the computational cost of training ; see Figure 1 . Rather than starting with a full-sized network or a supernet , we start from simple seed networks and progressively adjust ( grown and prune ) them . Specifically , we parameterize an architectural configuration space with indicator variables governing addition or removal of structural components . Figure 2 ( a ) shows an example , in the form of a two-level configuration space for CNN layers and filters . We enable learning of indicator values ( and thereby , architectural structure ) via combining a continuous relaxation with binary sampling , as illustrated in Figure 2 ( b ) . A per-component temperature parameter ensures that long-lived structures are eventually baked into the network ’ s discrete architectural configuration . While the recently proposed AutoGrow ( Wen et al. , 2020 ) also seeks to grow networks over the course of training , our technical approach differs substantially and leads to significant practical advantages . At a technical level , AutoGrow implements an architecture search procedure over a predefined modular structure , subject to hand-crafted , accuracy-driven growing and stopping policies . In contrast , we parameterize architectural configurations and utilize stochastic gradient descent to learn the auxiliary variables that specify structural components , while simultaneously training the weights within those components . Our unique technical approach yields the following advantages : • Fast Training by Growing : Training is a unified procedure , from which one can request a network structure and associated weights at any time . Unlike AutoGrow and the majority of pruning techniques , fine-tuning to optimize weights in a discovered architecture is optional . We achieve excellent results even without any fine-tuning stage . • Principled Approach via Learning by Continuation + Sampling : We formulate our approach in the spirit of learning by continuation methods , which relax a discrete optimization problem to an increasingly stiff continuous approximation . Critically , we introduce an additional sampling step to this strategy . From this combination , we gain the flexibility of exploring a supernet architecture , but the computational efficiency of only actually training a much smaller active subnetwork . • Budget-Aware Optimization Objectives : The parameters governing our architectural configuration are themselves updated via gradient decent . We have flexibility to formulate a variety of resource-sensitive losses , such as counting total FLOPs , in terms of these parameters . • Broad Applicability : Though we use progressive growth of CNNs in width and depth as a motivating example , our technique applies to virtually any neural architecture . One has flexibility in how to parameterize the architecture configuration space . We also show results with LSTMs . We demonstrate these advantages while comparing to recent NAS and pruning methods through extensive experiments on classification , semantic segmentation , and word-level language modeling . 2 RELATED WORK . Network Pruning . Pruning methods can be split into two groups : those pruning individual weights and those pruning structured components . Individual weight-based pruning methods vary on the removal criteria . For example , Han et al . ( 2015 ) propose to prune network weights with small magnitude , and subsequently quantize those remaining ( Han et al. , 2016 ) . Louizos et al . ( 2018 ) learn sparse networks by approximating ` 0-regularization with a stochastic reparameterization . However , sparse weights alone often only lead to speedups on dedicated hardware with supporting libraries . In structured methods , pruning is applied at the level of neurons , channels , or even layers . For example , L1-pruning ( Li et al. , 2017 ) removes channels based on the norm of their filters . He et al . ( 2018 ) use group sparsity to smooth the pruning process after training . MorphNet ( Gordon et al. , 2018 ) regularizes weights towards zero until they are small enough such that the corresponding output channels are marked for removal from the network . Intrinsic Structured Sparsity ( ISS ) ( Wen et al. , 2018 ) works on LSTMs ( Hochreiter & Schmidhuber , 1997 ) by collectively removing the columns and rows of the weight matrices via group LASSO . Although structured pruning methods and our algorithm share the same spirit of generating efficient models , we gain training cost savings by growing networks from small initial architectures instead of pruning full-sized ones . Neural Architecture Search . NAS methods have greatly improved the performance achieved by small network models . Pioneering NAS approaches use reinforcement learning ( Zoph et al. , 2018 ; Zoph & Le , 2017 ) and genetic algorithms ( Real et al. , 2019 ; Xie & Yuille , 2017 ) to search for transferable network blocks whose performance surpasses many manually designed ones . However , such approaches require massive computation during the search — typically thousands of GPU days . To reduce computational cost , recent efforts utilize more efficient search techniques , such as direct gradient-based optimization ( Liu et al. , 2019 ; Luo et al. , 2018 ; Pham et al. , 2018 ; Tan et al. , 2019 ; Cai et al. , 2019 ; Wortsman et al. , 2019 ) . Nevertheless , most NAS methods perform search in a supernet space which requires more computation than training typically-sized architectures . Network Growing . Network Morphism ( Wei et al. , 2016 ) searches for efficient deep networks by extending layers while preserving the parameters . Recently proposed Autogrow ( Wen et al. , 2020 ) takes an AutoML approach to growing layers . These methods either require a specially-crafted policy to stop growth ( e.g. , after a fixed number of layers ) or rely on evaluating accuracy during training , incurring significant additional computational cost . Learning by Continuation . Continuation methods are commonly used to approximate intractable optimization problems by gradually increasing the difficulty of the underlying objective , for example by adopting gradual relaxations to binary problems . Wu et al . ( 2019 ) ; Xie et al . ( 2019b ; 2020 ) use gumbel-softmax ( Jang et al. , 2017 ) to back-propagate errors during architecture search and spatial feature sparsification . Savarese et al . ( 2020 ) propose continuous sparsification to speed up pruning and ticket search ( Frankle & Carbin , 2019 ) . Despite the success of continuation methods in producing sparse networks upon the completion of training , they do not operate on sparse networks during training and instead work with a real-valued relaxation . Postponing actual elimination of near zeroed-out components prevents naive application of these methods from reducing training costs . 3 METHOD . 3.1 ARCHITECTURAL CONFIGURATION SPACE . A network topology can be seen as a directed acyclic graph consisting of an ordered sequence of nodes . Each node x ( i ) in is an input feature and each edge is a computation cell with structured hyperparameters ( e.g. , filter and layer numbers in convolutional networks ) . An architectural configuration space can be parameterized by associating a mask variable m ∈ { 0 , 1 } with each computation cell ( edge ) , which enables training-time pruning ( m = 1→ 0 ) and growing ( m = 0→ 1 ) dynamics . As a running example , we consider a two-level configuration space for CNN architectures , depicted in Figure 2 ( a ) , that enables dynamically growing networks in both width ( channel-wise ) and depth ( layer-wise ) . Alternative configuration spaces are possible ; we defer to the Appendix details on how we parameterize the design of LSTM architectures . CNN Channel Configuration Space : For a convolutional layer with lin input channels , lout output channels ( filters ) and k × k sized kernels , the i-th output feature is computed based on the i-th filter , i.e . for i ∈ { 1 , . . . , lout } : x ( i ) out = f ( xin , F ( i ) ·m ( i ) c ) , ( 1 ) wherem ( i ) c ∈ { 0 , 1 } is a binary parameter that removes the i-th output channel when set to zero and f denotes the convolutional operation . m ( i ) c is shared across a filter and broadcasts to the same shape as the filter tensor F ( i ) , enabling growing/pruning of the entire filter . As Figure 2 ( a ) ( top ) shows , we start from a slim channel configuration . We then query the indicator variables and perform state transitions : ( 1 ) When flipping an indicator variable from 0 to 1 for the first time , we grow a randomly initialized filter and concatenate it to the network . ( 2 ) If an indicator flips from 1 to 0 , we temporarily detach the corresponding filter from the computational graph ; it will be grown back to the its original position if its indicator flips back to 1 , or otherwise be permanently pruned at the end of training . ( 3 ) For other cases , the corresponding filters either survive and continue training or remain detached pending the next query to their indicators . Our method automates architecture evolution , provided we can train the indicators . CNN Layer Configuration Space : To grow network depth , we design a layer configuration space in which an initial shallow network will progressively expand into a deep trained model , as shown in Figure 2 ( a ) ( bottom ) . Similar to channel configuration space , where filters serve as basic structural units , we require a unified formulation to support the growing of popular networks with shortcut connections ( e.g. , ResNets ) and without ( e.g. , VGG-like plain nets ) . We first introduce an abstract layer class flayer as a basic structural unit , which operates on input features xin and generates output features xout . flayer can be instantiated as convolutional layers for plain nets or residual blocks for ResNets , respectively . We define the layer configuration space as : xout = g ( xin ; flayer ·m ( j ) l ) = { flayer ( xin ) , if m ( j ) l = 1 xin , if m ( j ) l = 0 , ( 2 ) where m ( j ) l ∈ { 0 , 1 } is the binary indicator for j-th layer flayer , with which we perform state transitions analogous to the channel configuration space . Layer indicators have priority over channel indicators : if m ( j ) l is set as 0 , all filters contained in the corresponding layer will be detached , regardless of the state their indicators . We do not detach layers that perform changes in resolution ( e.g. , strided convolution ) .

Summary: This paper proposes a novel NAS method that searches the model architectures by grows the networks. This searching strategy determines the channel and layer configurations by assigning a binary learnable parameter m for each channel or layer. The objective is to optimize a trade-off between the model performance on the given task and the regularization on the binary indicator m.