Boosting Network: Learn by Growing Filters and Layers via SplitLBI

Network structures are important to learning good representations of many tasks in computer vision and machine learning communities . These structures are either manually designed , or searched by Neural Architecture Search ( NAS ) in previous works , which however requires either expert-level efforts , or prohibitive computational cost . In practice , it is desirable to efficiently and simultaneously learn both the structures and parameters of a network from arbitrary classes with budgeted computational cost . We identify it as a new learning paradigm – Boosting Network , where one starts from simple models , delving into complex trained models progressively . In this paper , by virtue of an iterative sparse regularization path -- Split Linearized Bregman Iteration ( SplitLBI ) , we propose a simple yet effective boosting network method that can simultaneously grow and train a network by progressively adding both convolutional filters and layers . Extensive experiments with VGG and ResNets validate the effectiveness of our proposed algorithms . 1 INTRODUCTION . In recent years , deep convolution neural networks have made remarkable achievements in compute vision and machine learning communities in addressing many important tasks , such as image classification , and segmentation . Researchers had designed many successful Deep Neural Network ( DNN ) architectures , from just have a few convolution layers like LeNet ( LeCun et al. , 1998 ) and AlexNet ( Krizhevsky et al. , 2012 ) , to have more than 10 layers , e.g. , VGG ( Simonyan & Zisserman , 2014 ) and GoogleLeNet ( Szegedy et al. , 2015 ) , and even have hundreds and thousands of layers like ResNet ( He et al. , 2016a ) . Designing a neural network architecture requires expert-level efforts to specify the key network hyper-parameters , such as type of layers , number of filters and layers ( i.e. , network width and depth ) and so on . Since the capacity of over-parameterized networks largely depends on the number of total parameters , the number of filters and layers of networks are the key hyper-parameters that shape the expressive power of neural networks . In machine learning communities , most researchers resort to AutoML methods , e.g. , Neural Architecture Search ( NAS ) , in automating architecture engineering . Critically , NAS methods indeed surpass the manually designed architectures on many tasks , such as image classification and object detection ( Zoph & Le , 2016 ; Zoph et al. , 2018 ) . To search a good architecture , various search strategies have been employed , such as random search , Bayesian optimization , reinforcement learning , and so on . Most of them require significant amount of computational cost , which is normally orders of magnitude higher than training a network . Furthermore , some of the found architectures by NAS have much more parameters than manually designed ones on the same dataset . As the field of representation learning moves closer towards artificial intelligence , it becomes important to efficiently and simultaneously learn both the structures and parameters of a network from arbitrary classes on mobile devices or even Internet of Things ( IoT ) devices . This requires more flexible strategies in dynamically handling the network width and depth , according to the scale of dataset . To this end , this paper studies a new paradigm – Boosting network ( BoN ) , where one starts from simple models , delving into complex trained models progressively . Specifically , BoN could simultaneously grow the structures and train the parameters from a simple initialized network on the data gradually to complex ones . Formally , we demand the following properties of an algorithm qualified as BoN : • It should incorporate both architecture growth ( including filters and layers ) and parameter learning simultaneously , in which the width and depth of network can be gradually updated , and the parameters of network should be updated at the same time ; • It should provide a comparable classifier for prediction tasks , as the state-of-the-art handcrafted architectures on the same dataset ; • Its computational requirements , the total parameters of final boosted network , and memory footprint should remain bounded , ideally in the same order of magnitude as training a manually engineered architecture on the same dataset . The first two criteria express the essence of boosting network ; the third criterion identifies the key difference from NAS and other trivial or brute-force solutions , such as randomly searching . This paper proposes a method for the BoN task based on the Split Linearized Bregman Iteration ( SplitLBI ) ( Huang et al. , 2016 ; Fu et al. , 2019 ) , originally proposed by Huang et al . ( 2016 ) to learn high dimensional sparse linear models and found applications in medical image classification ( Sun et al. , 2017 ) , computer vision ( Zhao et al. , 2018 ) , and training neural networks ( Fu et al. , 2019 ) . Particularly , based on differential inclusions of inverse scale spaces ( Huang et al. , 2018 ) , SplitLBI has the merit of learning both an over-parameterized model weight set ( Over-Par set ) as the Stochastic Gradient Descent ( SGD ) , and structural sparsity model weight set ( Stru-Spa set ) in a coupled inverse scale space . Essentially , SplitLBI optimizes the Stru-Spa set as sparse approximation of the Over-Par set , by gradually selecting the important filters and weights from Over-Par set , along the training epochs . Equipped with SplitLBI , our key idea of BoN comes from progressively growing networks by checking the parameters within Over-Par and Stru-Spa set . Essentially along the training epochs , if enough parameters in Over-Par set have been selected in Stru-Spa set , it would be more advisable to increase the capacity of Over-Par Set by adding new parameters . Formally , to boosting a network , we introduce a Growing and Training Network Algorithm ( GT-Net Alg ) , consisting of two parts of growing both filters and layers , i.e. , Growing and Training filters algorithm ( GT-filters Alg ) , and Growing and Training layers algorithm ( GT-layers Alg ) . Given an initial network , the GT-filters Alg can effectively grow the filters of each layer , and train the network parameters at the same time . Furthermore , the GT-layers Alg firstly employs GT-filters Alg to compute the filter configuration for the layers of each block , and then periodically check whether to add new layer to the block along the training procedure . We conduct extensive experiments on several benchmark datasets , including MNIST , Cifar-10 , and Cifar-100 . It shows that our GT-Net Alg can achieve comparable or even better performance than the competitors , with much less computational cost , and smaller size of found network . This indicates the effectiveness of our proposed algorithms . Up to our knowledge , this is the first time that a BoN type algorithm of all the three aspects above is addressed in literature . Contributions . We summarize the contributions : ( 1 ) A novel learning paradigm - Boosting network ( BoN ) , is for the first time , studied in this paper : one starts from simple models , delving into complex trained models progressively . ( 2 ) We propose a novel GT-filters Alg , which can simultaneously effectively grows the filters of each layer , and trains the network parameters , given a simple initial network . ( 3 ) We present GT-layers Alg which grows and trains layers , by exploring the over-parameterized model weight and structural sparsity model weight set . 2 RELATED WORK . To explore a good deep learning structure , recent research focuses on employing Network Architecture Search ( NAS ) ( Elsken et al. , 2018 ; Zoph & Le , 2016 ; Zoph et al. , 2018 ) by using reinforcement learning to search the network structures , such as number of filters , filter size , layer depth , and so on . Despite promising performance achieved , the computational cost of NAS algorithms themselves are prohibitive expensive , e.g. , 800 GPUs concurrently at any time training the algorithms in Zoph & Le ( 2016 ) . Several approached improved NAS by accelerating it , including weight sharing/inheritance methods , or decreasing the searching space to a specific setting ( Elsken et al. , 2017 ; Pham et al. , 2018 ; Cai et al. , 2018c ; Bender et al. , 2018 ) . But they still require significant amount of computational cost . ( Nonparametric Network ( Philipp & Carbonell , 2017 ) uses group lasso to update the network structure . ) In contrast , Our BoN aims at growing a network and making a balance between computational cost , the model size and performance of the network . Network pruning algorithms ( Han et al. , 2015 ; Abbasi-Asl & Yu , 2017 ; Molchanov et al. , 2017 ) introduce additional computational cost in fine-tuning/updating the networks . In addition , some works study the manually designed compact and lightweight small DNNs ( e.g . ShuffleNet ( Ma et al. , 2018 ) , MobileNet ( Howard et al. , 2017 ) , and SqueezeNet ( Iandola et al. , 2017 ) ) , which may still be tailored only for some specific tasks rather than boosting a network in this work . Several recent works also consider adding layers to networks . Network Morphism ( Chen et al. , 2015 ; Wei et al. , 2016 ; 2017 ; Cai et al. , 2018a ; b ) aims at accelerating the deep networks by adding layers to a shallower net while preserving the parameters of the shallower net . One recent arxiv paper – Autogrow ( Wen et al. , 2019 ) also explores adding new layers in an automatic way . But none of these works can dynamically add filters to an existing layer as our GT-filters Alg . Additionally , these methods still requires significant computational cost and training resources . 3 METHODOLOGY . 3.1 BACKGROUND : SPLIT LINEARIZED BREGMAN ITERATIONS ( SPLITLBI ) . Our whole algorithm is built upon the SplitLBI algorithm . The basic spirit of this algorithm ( Huang et al. , 2018 ) lies in two coupled spaces : weight parameter W ( Over-Par set ) to explore over-parameterized models by gradient descent and structural sparsity parameter Γ ( Stru-Spa set ) to explore important subnetwork architectures by inverse scale space where those important parameters become nonzero faster than others . The SplitLBI algorithm can be described as following , W t+1 = W t − κα∇WL ( W t , Γt ) ( 1 ) Zt+1 = Zt − α∇ΓL ( W t , Γt ) ( 2 ) Γt+1 = κ · Prox ( Zt+1 ) ( 3 ) where Z0 = Γ0 = 0 ; and L ( W t , Γt ) = Ltask ( W t ) + 12ν ‖W t − Γt‖22 indicates the loss function at t , with task specific loss Ltask ( W t ) ( e.g. , cross-entropy loss ) . W is initialized as He et al . ( 2015 ) ; Γ is learned to approximate W here . And the Prox is the proximal mapping function with the following form : Prox ( Z ) = argminΓ 1 2 ‖Γ− Z‖22 + ‖Z‖1,2 = min ( 0 , 1− 1 ‖Z‖1,2 ) Z ( 4 ) where ‖Z‖1,2 is a group Lasso ( ` 1− ` 2 ) norm for convolutional filters or simply the Lasso ( ` 1 ) norm for weights ) . The hyper-parameters of SplitLBI are α is the learning rate ; κ and ν are controlling the sparsity of learned model . As in Huang et al . ( 2016 ) , and assume P : = ( W , Γ ) , the SplitLBI in Eq . ( 1-3 ) can be rewritten as standard Linearized Bregman Iteration , resulting the loss function , Pk+1 = arg min P { 〈P − Pk , α∇L̄ ( Pk ) 〉+BpkΨ ( P , Pk ) } , ( 5 ) where Ψ ( P ) = Ωλ ( Γ ) + 1 2κ ‖P‖22 = Ωλ ( Γ ) + 1 2κ ‖W‖22 + 1 2κ ‖Γ‖22 , ( 6 ) pk ∈ ∂Ψ ( Pk ) , and BqΨ is the Bregman divergence associated with convex function Ψ , defined by BqΨ ( P , Q ) : = Ψ ( P ) −Ψ ( Q ) − 〈q , P −Q〉 , for some q ∈ ∂Ψ ( Q ) . ( 7 ) One can see that Eq . ( 1 ) is essentially an gradient descent step over the primal parameter Wt . However in Eq . ( 2-3 ) , SplitLBI lifts the original network parameters W , to a coupled parameter set , ( W , Γ ) , where a sparse proximal gradient descent ( or Linearized Bregman Iteration , Mirror Descent ) runs over the dual parameter Γ which enforces structural sparsity on network models . Along the training path , Γ set is to learn a sparse approximation of parameter set W ; and the important filters and parameters will gradually become non-zeros in Γ .

Summary: This paper focuses on topic of searching for the optimal architecture for the deep network. Building on the split linearized bregman iteration strategy, the authors propose two practical algorithms to boost network, namely GT-filters Alg and GT-layers Alg. The proposed algorithms can simultaneously grow and train a network by progressively adding both convolutional filters and layers. The experiments conducted on VGG and ResNets display the comparable accuracies between the BoN and the standard big models, but with much more compact representations and balanced computational cost.