Growing Efficient Deep Networks by Structured Continuous Sparsification

1 INTRODUCTION . Deep neural networks are the dominant approach to a variety of machine learning tasks , including image classification ( Krizhevsky et al. , 2012 ; Simonyan & Zisserman , 2015 ) , object detection ( Girshick , 2015 ; Liu et al. , 2016 ) , semantic segmentation ( Long et al. , 2015 ; Chen et al. , 2017 ) and language modeling ( Zaremba et al. , 2014 ; Vaswani et al. , 2017 ; Devlin et al. , 2019 ) . Modern neural networks are overparameterized and training larger networks usually yields improved generalization accuracy . Recent work ( He et al. , 2016 ; Zagoruyko & Komodakis , 2016 ; Huang et al. , 2017 ) illustrates this trend through increasing depth and width of convolutional neural networks ( CNNs ) . Yet , training is compute-intensive , and real-world deployments are often limited by parameter and compute budgets . Neural architecture search ( NAS ) ( Zoph & Le , 2017 ; Liu et al. , 2019 ; Luo et al. , 2018 ; Pham et al. , 2018 ; Savarese & Maire , 2019 ) and model pruning ( Han et al. , 2016 ; 2015 ; Guo et al. , 2016 ) methods aim to reduce these burdens . NAS addresses an issue that further compounds training cost : the enormous space of possible network architectures . While hand-tuning architectural details , such as the connection structure of convolutional layers , can improve performance ( Iandola et al. , 2016 ; Sifre & Mallat , 2014 ; Chollet , 2017 ; Howard et al. , 2017 ; Zhang et al. , 2018 ; Huang et al. , 2018 ) , a principled way of deriving such designs remains elusive . NAS methods aim to automate exploration of possible architectures , producing an efficient design for a target task under practical resource constraints . However , during training , most NAS methods operate on a large supernet architecture , which encompasses candidate components beyond those that are eventually selected for inclusion in the resulting network ( Zoph & Le , 2017 ; Liu et al. , 2019 ; Luo et al. , 2018 ; Pham et al. , 2018 ; Savarese & Maire , 2019 ) . Consequently , NAS-based training may typically be more thorough , but more computationally expensive , than training a single hand-designed architecture . Model pruning techniques similarly focus on improving the resource efficiency of neural networks during inference , at the possible expense of increased training cost . Common strategies aim to generate a lighter version of a given network architecture by removing individual weights ( Han et al. , 2015 ; 2016 ; Molchanov et al. , 2017 ) or structured parameter sets ( Li et al. , 2017 ; He et al. , 2018 ; Luo et al. , 2017 ) . However , the majority of these methods train a full-sized model prior to pruning and , Level 1 : Channel-wise Training Epochs T=0,1,2… Level 2 : Layer-w ise … ( a ) Growing CNN Layers and Filters Training Epochs FL O Ps Our Method Pruning Architecture Search ( b ) Epoch-wise Training Cost NAS Pruning Ours FL O Ps ( c ) Total Training Cost Figure 1 : Growing Networks during Training . We define an architecture configuration space and simultaneously adapt network structure and weights . ( a ) Applying our approach to CNNs , we maintain auxiliary variables that determine how to grow and prune both filters ( i.e . channel-wise ) and layers , subject to practical resource constraints . ( b ) By starting with a small network and growing its size , we utilize fewer resources in early training epochs , compared to pruning or NAS methods . ( c ) Consequently , our method significantly reduces the total computational cost of training , while delivering trained networks of comparable or better size and accuracy . after pruning , utilize additional fine-tuning phases in order to maintain accuracy . Hubara et al . ( 2016 ) and Rastegari et al . ( 2016 ) propose the use of binary weights and activations , allowing inference to benefit from reduced storage costs and efficient computation through bit-counting operations . Yet , training still involves tracking high-precision weights alongside lower-precision approximations . We take a unified view of pruning and architecture search , regarding both as acting on a configuration space , and propose a method to dynamically grow deep networks by continuously reconfiguring their architecture during training . Our approach not only produces models with efficient inference characteristics , but also reduces the computational cost of training ; see Figure 1 . Rather than starting with a full-sized network or a supernet , we start from simple seed networks and progressively adjust ( grown and prune ) them . Specifically , we parameterize an architectural configuration space with indicator variables governing addition or removal of structural components . Figure 2 ( a ) shows an example , in the form of a two-level configuration space for CNN layers and filters . We enable learning of indicator values ( and thereby , architectural structure ) via combining a continuous relaxation with binary sampling , as illustrated in Figure 2 ( b ) . A per-component temperature parameter ensures that long-lived structures are eventually baked into the network ’ s discrete architectural configuration . While the recently proposed AutoGrow ( Wen et al. , 2020 ) also seeks to grow networks over the course of training , our technical approach differs substantially and leads to significant practical advantages . At a technical level , AutoGrow implements an architecture search procedure over a predefined modular structure , subject to hand-crafted , accuracy-driven growing and stopping policies . In contrast , we parameterize architectural configurations and utilize stochastic gradient descent to learn the auxiliary variables that specify structural components , while simultaneously training the weights within those components . Our unique technical approach yields the following advantages : • Fast Training by Growing : Training is a unified procedure , from which one can request a network structure and associated weights at any time . Unlike AutoGrow and the majority of pruning techniques , fine-tuning to optimize weights in a discovered architecture is optional . We achieve excellent results even without any fine-tuning stage . • Principled Approach via Learning by Continuation + Sampling : We formulate our approach in the spirit of learning by continuation methods , which relax a discrete optimization problem to an increasingly stiff continuous approximation . Critically , we introduce an additional sampling step to this strategy . From this combination , we gain the flexibility of exploring a supernet architecture , but the computational efficiency of only actually training a much smaller active subnetwork . • Budget-Aware Optimization Objectives : The parameters governing our architectural configuration are themselves updated via gradient decent . We have flexibility to formulate a variety of resource-sensitive losses , such as counting total FLOPs , in terms of these parameters . • Broad Applicability : Though we use progressive growth of CNNs in width and depth as a motivating example , our technique applies to virtually any neural architecture . One has flexibility in how to parameterize the architecture configuration space . We also show results with LSTMs . We demonstrate these advantages while comparing to recent NAS and pruning methods through extensive experiments on classification , semantic segmentation , and word-level language modeling . 2 RELATED WORK . Network Pruning . Pruning methods can be split into two groups : those pruning individual weights and those pruning structured components . Individual weight-based pruning methods vary on the removal criteria . For example , Han et al . ( 2015 ) propose to prune network weights with small magnitude , and subsequently quantize those remaining ( Han et al. , 2016 ) . Louizos et al . ( 2018 ) learn sparse networks by approximating ` 0-regularization with a stochastic reparameterization . However , sparse weights alone often only lead to speedups on dedicated hardware with supporting libraries . In structured methods , pruning is applied at the level of neurons , channels , or even layers . For example , L1-pruning ( Li et al. , 2017 ) removes channels based on the norm of their filters . He et al . ( 2018 ) use group sparsity to smooth the pruning process after training . MorphNet ( Gordon et al. , 2018 ) regularizes weights towards zero until they are small enough such that the corresponding output channels are marked for removal from the network . Intrinsic Structured Sparsity ( ISS ) ( Wen et al. , 2018 ) works on LSTMs ( Hochreiter & Schmidhuber , 1997 ) by collectively removing the columns and rows of the weight matrices via group LASSO . Although structured pruning methods and our algorithm share the same spirit of generating efficient models , we gain training cost savings by growing networks from small initial architectures instead of pruning full-sized ones . Neural Architecture Search . NAS methods have greatly improved the performance achieved by small network models . Pioneering NAS approaches use reinforcement learning ( Zoph et al. , 2018 ; Zoph & Le , 2017 ) and genetic algorithms ( Real et al. , 2019 ; Xie & Yuille , 2017 ) to search for transferable network blocks whose performance surpasses many manually designed ones . However , such approaches require massive computation during the search — typically thousands of GPU days . To reduce computational cost , recent efforts utilize more efficient search techniques , such as direct gradient-based optimization ( Liu et al. , 2019 ; Luo et al. , 2018 ; Pham et al. , 2018 ; Tan et al. , 2019 ; Cai et al. , 2019 ; Wortsman et al. , 2019 ) . Nevertheless , most NAS methods perform search in a supernet space which requires more computation than training typically-sized architectures . Network Growing . Network Morphism ( Wei et al. , 2016 ) searches for efficient deep networks by extending layers while preserving the parameters . Recently proposed Autogrow ( Wen et al. , 2020 ) takes an AutoML approach to growing layers . These methods either require a specially-crafted policy to stop growth ( e.g. , after a fixed number of layers ) or rely on evaluating accuracy during training , incurring significant additional computational cost . Learning by Continuation . Continuation methods are commonly used to approximate intractable optimization problems by gradually increasing the difficulty of the underlying objective , for example by adopting gradual relaxations to binary problems . Wu et al . ( 2019 ) ; Xie et al . ( 2019b ; 2020 ) use gumbel-softmax ( Jang et al. , 2017 ) to back-propagate errors during architecture search and spatial feature sparsification . Savarese et al . ( 2020 ) propose continuous sparsification to speed up pruning and ticket search ( Frankle & Carbin , 2019 ) . Despite the success of continuation methods in producing sparse networks upon the completion of training , they do not operate on sparse networks during training and instead work with a real-valued relaxation . Postponing actual elimination of near zeroed-out components prevents naive application of these methods from reducing training costs . 3 METHOD . 3.1 ARCHITECTURAL CONFIGURATION SPACE . A network topology can be seen as a directed acyclic graph consisting of an ordered sequence of nodes . Each node x ( i ) in is an input feature and each edge is a computation cell with structured hyperparameters ( e.g. , filter and layer numbers in convolutional networks ) . An architectural configuration space can be parameterized by associating a mask variable m ∈ { 0 , 1 } with each computation cell ( edge ) , which enables training-time pruning ( m = 1→ 0 ) and growing ( m = 0→ 1 ) dynamics . As a running example , we consider a two-level configuration space for CNN architectures , depicted in Figure 2 ( a ) , that enables dynamically growing networks in both width ( channel-wise ) and depth ( layer-wise ) . Alternative configuration spaces are possible ; we defer to the Appendix details on how we parameterize the design of LSTM architectures . CNN Channel Configuration Space : For a convolutional layer with lin input channels , lout output channels ( filters ) and k × k sized kernels , the i-th output feature is computed based on the i-th filter , i.e . for i ∈ { 1 , . . . , lout } : x ( i ) out = f ( xin , F ( i ) ·m ( i ) c ) , ( 1 ) wherem ( i ) c ∈ { 0 , 1 } is a binary parameter that removes the i-th output channel when set to zero and f denotes the convolutional operation . m ( i ) c is shared across a filter and broadcasts to the same shape as the filter tensor F ( i ) , enabling growing/pruning of the entire filter . As Figure 2 ( a ) ( top ) shows , we start from a slim channel configuration . We then query the indicator variables and perform state transitions : ( 1 ) When flipping an indicator variable from 0 to 1 for the first time , we grow a randomly initialized filter and concatenate it to the network . ( 2 ) If an indicator flips from 1 to 0 , we temporarily detach the corresponding filter from the computational graph ; it will be grown back to the its original position if its indicator flips back to 1 , or otherwise be permanently pruned at the end of training . ( 3 ) For other cases , the corresponding filters either survive and continue training or remain detached pending the next query to their indicators . Our method automates architecture evolution , provided we can train the indicators . CNN Layer Configuration Space : To grow network depth , we design a layer configuration space in which an initial shallow network will progressively expand into a deep trained model , as shown in Figure 2 ( a ) ( bottom ) . Similar to channel configuration space , where filters serve as basic structural units , we require a unified formulation to support the growing of popular networks with shortcut connections ( e.g. , ResNets ) and without ( e.g. , VGG-like plain nets ) . We first introduce an abstract layer class flayer as a basic structural unit , which operates on input features xin and generates output features xout . flayer can be instantiated as convolutional layers for plain nets or residual blocks for ResNets , respectively . We define the layer configuration space as : xout = g ( xin ; flayer ·m ( j ) l ) = { flayer ( xin ) , if m ( j ) l = 1 xin , if m ( j ) l = 0 , ( 2 ) where m ( j ) l ∈ { 0 , 1 } is the binary indicator for j-th layer flayer , with which we perform state transitions analogous to the channel configuration space . Layer indicators have priority over channel indicators : if m ( j ) l is set as 0 , all filters contained in the corresponding layer will be detached , regardless of the state their indicators . We do not detach layers that perform changes in resolution ( e.g. , strided convolution ) .

Summary: This paper proposes a new principled approach to growing deep network architectures based on continuous relaxation of discrete structure optimization combined with a sparse subnetwork sampling scheme. It starts from a simple seed architecture and dynamically grows/prunes both the layers and filters during training. Through extensive experiments, the authors show that this method produces more efficient networks while reducing the computational cost of training, still maintaining good validation accuracy, compared to other NAS or pruning/growing methods.