AutoGrow: Automatic Layer Growing in Deep Convolutional Networks

1 INTRODUCTION . Layer depth is one of the decisive factors of the success of Deep Neural Networks ( DNNs ) . For example , image classification accuracy keeps improving as the depth of network models grows ( Krizhevsky et al. , 2012 ; Simonyan & Zisserman , 2014 ; Szegedy et al. , 2015 ; He et al. , 2016 ; Huang et al. , 2017 ) . Although shallow networks can not ensure high accuracy , DNNs composed of too many layers may suffer from over-fitting and convergence difficulty in training . How to obtain the optimal depth for a DNN still remains mysterious . For instance , ResNet-152 ( He et al. , 2016 ) uses 3 , 8 , 36 and 3 residual blocks under output sizes of 56 × 56 , 28 × 28 , 14 × 14 and 7 × 7 , respectively , which don ’ t show an obvious quantitative relation . In practice , people usually reply on some heuristic trials and tests to obtain the depth of a network : they first design a DNN with a specific depth and then train and evaluate the network on a given dataset ; finally , they change the depth and repeat the procedure until the accuracy meets the requirement . Besides the high computational cost induced by the iteration process , such trial & test iterations must be repeated whenever dataset changes . In this paper , we propose AutoGrow that can automate depth discovery given a layer architecture . We will show that AutoGrow generalizes to different datasets and layer architectures . There are some previous works which add or morph layers to increase the depth in DNNs . VggNet ( Simonyan & Zisserman , 2014 ) and DropIn ( Smith et al. , 2016 ) added new layers into shallower DNNs ; Network Morphism ( Wei et al. , 2016 ; 2017 ; Chen et al. , 2015 ) morphed each layer to multiple layers to increase the depth meanwhile preserving the function of the shallower net . Table 1 summarizes differences in this work . Their goal was to overcome difficulty of training deeper DNNs or accelerate it . Our goal is to automatically find an optimal depth . Moreover , previous works applied layer growth by once or a few times at pre-defined locations to grow a pre-defined number of layers ; in contrast , ours automatically learns the number of new layers and growth locations without limiting growing times . We will summarize more related works in Section 4 . Figure 1 illustrates an example of AutoGrow . It starts from the shallowest backbone network and gradually grows sub-modules ( A sub-module can be one or more layers , e.g. , a residual block ) ; the growth stops once a stopping policy is satisfied . We studied multiple initializers of new layers and multiple growing policies , and surprisingly find that : ( 1 ) a random initializer works equally or better than complicated Network Morphism ; ( 2 ) it is more effective to grow before a shallow net converges . We hypothesize that this is because a converged shallow net is an inadequate initialization for training deeper net , while random initialization can help to escape from a bad starting point . Motivated by this , we intentionally avoid full convergence during the growing by using ( 1 ) random initialization of new layers , ( 2 ) a constant large learning rate , and ( 3 ) a short growing interval . Growing sub-nets Sub-modules Stopped sub-nets Initializer Epochs Seed Discovered Figure 1 : A simple example of AutoGrow . Previous works Ours Goal Ease training Depth automation Times Once or a few Unlimited Locations Human defined Learned Layer # Human defined Learned Table 1 : Comparison with previous works about layer growth . Our contributions are : ( 1 ) We propose AutoGrow to automate DNN layer growing and depth discovery . AutoGrow is very robust . With the same hyper-parameters , it adapts network depth to various datasets including MNIST , FashionMNIST , SVHN , CIFAR10 , CIFAR100 and ImageNet . Moreover , AutoGrow can also discover shallower DNNs when the dataset is a subset . ( 2 ) AutoGrow demonstrates high efficiency and scales up to ImageNet , because the layer growing is as fast as training a single DNN . On ImageNet , it discovers a new ResNets with better trade-off between accuracy and computation complexity . ( 3 ) We challenge the idea of Network Morphism , as random initialization works equally or better when growing layers . ( 4 ) We find that it is beneficial to rapidly grow layers before a shallower net converge , contradicting previous intuition . 2 AutoGrow – A DEPTH GROWING ALGORITHM Algorithm 1 AutoGrow Algorithm . Input : 1 A seed shallow network g ( X0 ) composed of M sub-networks F = { fi ( · ; Wi ) : i = 0 . . .M − 1 } , where each sub-network has only one sub-module ( a dimension reduction sub-module ) ; an epoch interval K to check growing and stopping policies ; the number of fine-tuning epochs N after growing . Initialization : 2 A Circular Linked List of sub-networks under growing : subNetList = f0 ( · ; W0 ) → · · · → fM−1 ( · ; WM−1 ) ←−−−−−−−−−−−−−−−−−−−−−−−−−−− ; 3 The current growing sub-network : growingSubNet = subNetList.head ( ) = f0 ( · ; W0 ) ; 4 The recent grown sub-network : grownSubNet = None ; Process : 5 # if there exist growing sub-network ( s ) 6 while subNetList.size ( ) > 0 do 7 train ( g ( X0 ) , K ) # train the whole network g ( X0 ) for K epochs 8 if meetStoppingPolicy ( ) then 9 # remove a sub-network from the growing list if its growth did not improve accuracy 10 subNetList.delete ( grownSubNet ) ; 11 end 12 if meetGrowingPolicy ( ) and subNetList.size ( ) > 0 then 13 # current growing sub-network growingSubNet== fi ( · ; Wi ) 14 Wi = Wi ∪W # stack a sub-module on top of fi ( · ; Wi ) 15 initializer ( W ) ; # initialize the new sub-moduleW 16 # record the recent grown sub-network and iterate to a next sub-network 17 grownSubNet= growingSubNet ; 18 growingSubNet= subNetList.next ( growingSubNet ) ; 19 end 20 end 21 Fine-tune the discovered network g ( X0 ) for N epochs ; Output : 22 A trained neural network g ( X0 ) with learned depth . Figure 1 gives an overview of the proposed AutoGrow . In this paper , we use network , sub-networks , sub-modules and layers to describe the architecture hierarchy . A network is composed of a cascade of sub-networks . A sub-network is composed of sub-modules , which typical share the same output size . A sub-module ( e.g . a residual block ) is an elementary growing block composed of one or a few layers . In this section , we rigorously formulate a generic version of AutoGrow which will be materialized in subsections . A deep convolutional network g ( X0 ) is a cascade of sub-networks by composing functions as g ( X0 ) = l ( fM−1 ( fM−2 ( · · ·f1 ( f0 ( X0 ) ) · · · ) ) ) , where X0 is an input image , M is the number of sub-networks , l ( · ) is a loss function , and Xi+1 = fi ( Xi ) is a sub-network that operates on an input image or a feature tensor Xi ∈ Rci×hi×wi . Here , ci is the number of channels , and hi and wi are spatial dimensions . fi ( Xi ) is a simplified notation of fi ( Xi ; Wi ) , where Wi is a set of sub-modules ’ parameters within the i-th sub-network . Thus W = { Wi : i = 0 . . .M − 1 } denotes the whole set of parameters in the DNN . To facilitate growing , the following properties are supported within a sub-network : ( 1 ) the first sub-module usually reduces the size of input feature maps , e.g. , using pooling or convolution with a stride ; and ( 2 ) all sub-modules in a sub-network maintain the same output size . As such , our framework can support popular networks , including VggNet-like plain networks ( Simonyan & Zisserman , 2014 ) , GoogLeNet ( Szegedy et al. , 2015 ) , ResNets ( He et al. , 2016 ) and DenseNets ( Huang et al. , 2017 ) . In this paper , we select ResNets and VggNet-like nets as representatives of DNNs with and without shortcuts , respectively . With above notations , Algorithm 1 rigorously describes the AutoGrow algorithm . In brief , AutoGrow starts with the shallowest net where every sub-network has only one sub-module for spatial dimension reduction . AutoGrow loops over all growing sub-networks in order . For each sub-network , AutoGrow stacks a new sub-module . When the new sub-module does not improve the accuracy , the growth in corresponding sub-network will be permanently stopped . The details of our method will be materialized in the following subsections . 2.1 SEED SHALLOW NETWORKS AND SUB-MODULES . In this paper , in all datasets except ImageNet , we explore growing depth for four types of DNNs : ( 1 ) Basic3ResNet : the same ResNet used for CIFAR10 in He et al . ( 2016 ) , which has 3 residual subnetworks with output spatial sizes of 32×32 , 16×16 and 8×8 , respectively ; ( 2 ) Basic4ResNet : a variant of ResNet used for ImageNet in He et al . ( 2016 ) built by basic residual blocks ( each of which contains two convolutions and one shortcut ) . There are 4 sub-networks with output spatial sizes of 32× 32 , 16× 16 , 8× 8 and 4× 4 , respectively ; ( 3 ) Plain3Net : a VggNet-like plain net by removing shortcuts in Basic3ResNet ; ( 4 ) Plain4Net : a VggNet-like plain net by removing shortcuts in Basic4ResNet . In AutoGrow , the architectures of seed shallow networks and sub-modules are pre-defined . In plain DNNs , a sub-module is a stack of convolution , Batch Normalization and ReLU ; in residual DNNs , a sub-module is a residual block . In AutoGrow , a sub-network is a stack of all sub-modules with the same output spatial size . Unlike He et al . ( 2016 ) which manually designed the depth , AutoGrow starts from a seed architecture in which each sub-network has only one sub-module and automatically learns the number of sub-modules . On ImageNet , we apply the same backbones in He et al . ( 2016 ) as the seed architectures . A seed architecture has only one sub-module under each output spatial size . For a ResNet using basic residual blocks or bottleneck residual blocks ( He et al. , 2016 ) , we respectively name it as Basic4ResNet or Bottleneck4ResNet . Plain4Net is also obtained by removing shortcuts in Basic4ResNet . 2.2 SUB-MODULE INITIALIZERS . Here we explain how to initialize a new sub-moduleW in initializer ( W ) mentioned in Algorithm 1 . Network Morphism changes DNN architecture meanwhile preserving the loss function via special initialization of new layers , that is , g ( X0 ; W ) = g ( X0 ; W∪W ) ∀X0 . A residual sub-module shows a nice property : when stacking a residual block and initializing the last Batch Normalization layer as zeros , the function of the shallower net is preserved but the DNN is morphed to a deeper net . Thus , Network Morphism can be easily implemented by this zero initialization ( ZeroInit ) . In this work , all layers inW are initialized using default randomization , except for a special treatment of the last Batch Normalization layer in a residual sub-module . Besides ZeroInit , we propose a new AdamInit for Network Morphism . In AdamInit , we freeze all parameters except the last Batch Normalization layer inW , and then use Adam optimizer ( Kingma & Ba , 2014 ) to optimize the last Bath Normalization for maximum 10 epochs till the training accuracy of the deeper net is as good as the shallower one . After AdamInit , all parameters are jointly optimized . We view AdamInit as a Network Morphism because the training loss is similar after AdamInit . We empirically find that AdamInit can usually find a solution in less than 3 epochs . We also study random initialization of the last Batch Normalization layer using uniform ( UniInit ) or Gaussian ( GauInit ) noises with a standard deviation 1.0 . We will show that GauInit obtains the best result , challenging the idea of Network Morphism ( Wei et al. , 2016 ; 2017 ; Chen et al. , 2015 ) .

Summary: 	This paper best fits in the literature that explores growing network depth.  The main framework here is to interleave training a shallower network and adding new layers.  This paper (their final algorithm) differs from existing methods in that they: 1) initialize the new layers using standard initialization as opposed to the commonly used zero-init in this literature, 2) grows at a fixed interval , and this interval is short (to avoid the shallower nets being  overly-trained)., 3) uses a large and constant learning rate during the growing phase.