Title: AutoGrow: AUTOMATIC LAYER GROWING IN DEEP CONVOLUTIONAL NETWORKS

Abstract: Depth is a key component of Deep Neural Networks (DNNs), however, designing depth is heuristic and requires many human efforts. We propose AutoGrow to automate depth discovery in DNNs: starting from a shallow seed architecture, AutoGrow grows new layers if the growth improves the accuracy; otherwise, stops growing and thus discovers the depth. We propose robust growing and stopping policies to generalize to different network architectures and datasets. Our experiments show that by applying the same policy to different network architectures, AutoGrow can always discover near-optimal depth on various datasets of MNIST, FashionMNIST, SVHN, CIFAR10, CIFAR100 and ImageNet. For example, in terms of accuracy-computation trade-off, AutoGrow discovers a better depth combination in ResNets than human experts. Our AutoGrow is efficient. It discovers depth within similar time of training a single DNN.

1. INTRODUCTION:

Layer depth is one of the decisive factors of the success of Deep Neural Networks (DNNs). For example, image classification accuracy keeps improving as the depth of network models grows (Krizhevsky et al., 2012; Simonyan & Zisserman, 2014; Szegedy et al., 2015; He et al., 2016; Huang et al., 2017) . Although shallow networks cannot ensure high accuracy, DNNs composed of too many layers may suffer from over-fitting and convergence difficulty in training. How to obtain the optimal depth for a DNN still remains mysterious. For instance, ResNet-152 (He et al., 2016) uses 3, 8, 36 and 3 residual blocks under output sizes of 56 × 56, 28 × 28, 14 × 14 and 7 × 7, respectively, which don't show an obvious quantitative relation. In practice, people usually reply on some heuristic trials and tests to obtain the depth of a network: they first design a DNN with a specific depth and then train and evaluate the network on a given dataset; finally, they change the depth and repeat the procedure until the accuracy meets the requirement. Besides the high computational cost induced by the iteration process, such trial & test iterations must be repeated whenever dataset changes. In this paper, we propose AutoGrow that can automate depth discovery given a layer architecture. We will show that AutoGrow generalizes to different datasets and layer architectures.

There are some previous works which add or morph layers to increase the depth in DNNs. Vg-gNet (Simonyan & Zisserman, 2014) and DropIn (Smith et al., 2016) added new layers into shallower DNNs; Network Morphism (Wei et al., 2016; 2017; Chen et al., 2015) morphed each layer to multiple layers to increase the depth meanwhile preserving the function of the shallower net. Table 1 summarizes differences in this work. Their goal was to overcome difficulty of training deeper DNNs or accelerate it. Our goal is to automatically find an optimal depth. Moreover, previous works applied layer growth by once or a few times at pre-defined locations to grow a pre-defined number of layers; in contrast, ours automatically learns the number of new layers and growth locations without limiting growing times. We will summarize more related works in Section 4. Figure 1 illustrates an example of AutoGrow. It starts from the shallowest backbone network and gradually grows sub-modules (A sub-module can be one or more layers, e.g., a residual block); the growth stops once a stopping policy is satisfied. We studied multiple initializers of new layers and multiple growing policies, and surprisingly find that: (1) a random initializer works equally or better than complicated Network Morphism; (2) it is more effective to grow before a shallow net converges. We hypothesize that this is because a converged shallow net is an inadequate initialization for training deeper net, while random initialization can help to escape from a bad starting point.

Motivated by this, we intentionally avoid full convergence during the growing by using (1) random initialization of new layers, (2) a constant large learning rate, and (3) a short growing interval. (3) We challenge the idea of Network Morphism, as random initialization works equally or better when growing layers. ( 4) We find that it is beneficial to rapidly grow layers before a shallower net converge, contradicting previous intuition.

2 AutoGrow -A DEPTH GROWING ALGORITHM Algorithm 1 AutoGrow Algorithm.

Input :

A seed shallow network g(X0) composed of M sub-networks F = {fi (•; Wi) : i = 0 . . . M -1}

, where each sub-network has only one sub-module (a dimension reduction sub-module); an epoch interval K to check growing and stopping policies; the number of fine-tuning epochs N after growing.

Initialization::

A Circular Linked List of sub-networks under growing:

subNetList = f0 (•; W0) → • • • → f M -1 (•; W M -1 ) ← --------------------------- ;

The current growing sub-network: growingSubNet = subNetList.head() = f0 (•; W0);

The recent grown sub-network: grownSubNet = None; Process :

# if there exist growing sub-network(s) while subNetList.size()>0 do train(g(X0), K) # train the whole network g(X0) for K epochs if meetStoppingPolicy() then # remove a sub-network from the growing list if its growth did not improve accuracy subNetList.delete(grownSubNet); A trained neural network g(X0) with learned depth.

Figure 1 gives an overview of the proposed AutoGrow. In this paper, we use network, sub-networks, sub-modules and layers to describe the architecture hierarchy. A network is composed of a cascade of sub-networks. A sub-network is composed of sub-modules, which typical share the same output size. A sub-module (e.g. a residual block) is an elementary growing block composed of one or a few layers. In this section, we rigorously formulate a generic version of AutoGrow which will be materialized in subsections. A deep convolutional network g(X 0 ) is a cascade of sub-networks by composing functions as g(X

0 ) = l (f M -1 (f M -2 (• • • f 1 (f 0 (X 0 )) • • • )))

, where X 0 is an input image, M is the number of sub-networks, l(•) is a loss function, and X i+1 = f i (X i ) is a sub-network that operates on an input image or a feature tensor X i ∈ R ci×hi×wi . Here, c i is the number of channels, and h i and w i are spatial dimensions.

f i (X i ) is a simplified notation of f i (X i ; W i ),

where W i is a set of sub-modules' parameters within the i-th sub-network. Thus W = {W i : i = 0 . . . M -1} denotes the whole set of parameters in the DNN. To facilitate growing, the following properties are supported within a sub-network: (1) the first sub-module usually reduces the size of input feature maps, e.g., using pooling or convolution with a stride; and (2) all sub-modules in a sub-network maintain the same output size. As such, our framework can support popular networks, including VggNet-like plain networks (Simonyan & Zisserman, 2014) , GoogLeNet (Szegedy et al., 2015) , ResNets (He et al., 2016) and DenseNets (Huang et al., 2017) . In this paper, we select ResNets and VggNet-like nets as representatives of DNNs with and without shortcuts, respectively.

With above notations, Algorithm 1 rigorously describes the AutoGrow algorithm. In brief, AutoGrow starts with the shallowest net where every sub-network has only one sub-module for spatial dimension reduction. AutoGrow loops over all growing sub-networks in order. For each sub-network, AutoGrow stacks a new sub-module. When the new sub-module does not improve the accuracy, the growth in corresponding sub-network will be permanently stopped. The details of our method will be materialized in the following subsections.

2 In AutoGrow, the architectures of seed shallow networks and sub-modules are pre-defined. In plain DNNs, a sub-module is a stack of convolution, Batch Normalization and ReLU; in residual DNNs, a sub-module is a residual block. In AutoGrow, a sub-network is a stack of all sub-modules with the same output spatial size. Unlike He et al. (2016) which manually designed the depth, AutoGrow starts from a seed architecture in which each sub-network has only one sub-module and automatically learns the number of sub-modules.

On ImageNet, we apply the same backbones in He et al. (2016) as the seed architectures. A seed architecture has only one sub-module under each output spatial size. For a ResNet using basic residual blocks or bottleneck residual blocks (He et al., 2016) , we respectively name it as Basic4ResNet or Bottleneck4ResNet. Plain4Net is also obtained by removing shortcuts in Basic4ResNet.

2.2. SUB-MODULE INITIALIZERS:

Here we explain how to initialize a new sub-module W in initializer(W) mentioned in Algorithm 1. Network Morphism changes DNN architecture meanwhile preserving the loss function via special initialization of new layers, that is, g(X 0 ; W) = g(X 0 ; W ∪ W) ∀X 0 . A residual sub-module shows a nice property: when stacking a residual block and initializing the last Batch Normalization layer as zeros, the function of the shallower net is preserved but the DNN is morphed to a deeper net. Thus, Network Morphism can be easily implemented by this zero initialization (ZeroInit).

In this work, all layers in W are initialized using default randomization, except for a special treatment of the last Batch Normalization layer in a residual sub-module. Besides ZeroInit, we propose a new AdamInit for Network Morphism. In AdamInit, we freeze all parameters except the last Batch Normalization layer in W, and then use Adam optimizer (Kingma & Ba, 2014) to optimize the last Bath Normalization for maximum 10 epochs till the training accuracy of the deeper net is as good as the shallower one. After AdamInit, all parameters are jointly optimized. We view AdamInit as a Network Morphism because the training loss is similar after AdamInit. We empirically find that AdamInit can usually find a solution in less than 3 epochs. We also study random initialization of the last Batch Normalization layer using uniform (UniInit) or Gaussian (GauInit) noises with a standard deviation 1.0. We will show that GauInit obtains the best result, challenging the idea of Network Morphism (Wei et al., 2016; 2017; Chen et al., 2015) .

2.3. GROWING AND STOPPING POLICIES:

In Algorithm 1, a growing policy refers to meetGrowingPolicy(), which returns true when the network should grow a sub-module. Two growing policies are studied here:

1. Convergent Growth: meetGrowingPolicy() returns true when the improvement of validation accuracy is less than τ in the last K epochs. That is, in Convergent Growth, AutoGrow only grows when current network has converged. This is a similar growing criterion adopted in previous works (Elsken et al., 2017; Cai et al., 2018a; b) . 2. Periodic Growth: meetGrowingPolicy() always returns true, that is, the network always grows every K epochs. Therefore, K is also the growing period. In the best practice of AutoGrow, K is small (e.g. K = 3) such that it grows before current network converges.

Our experiments will show that Periodic Growth outperforms Convergent Growth. We hypothesize that a fully converged shallower net is an inadequate initialization to train a deeper net. We will perform experiments to test this hypothesis and visualize optimization trajectory to illustrate it.

A stopping policy denotes meetStoppingPolicy() in Algorithm 1. When Convergent Growth is adopted, meetStoppingPolicy() returns true if a recent growth does not improve validation accuracy more than τ within K epochs. We use a similar stopping policy for Periodic Growth; however, as it can grow rapidly with a small period K (e.g. K = 3) before it converges, we use a larger window size J for stop. Specifically, when Periodic Growth is adopted, meetStoppingPolicy() returns true when the validation accuracy improves less than τ in the last J epochs, where J K.

Hyper-parameters τ , J and K control the operation of AutoGrow and can be easily setup and generalize well. τ denotes the significance of accuracy improvement for classification. We simply set τ = 0.05% in all experiments. J represents how many epochs to wait for an accuracy improvement before stopping the growth of a sub-network. It is more meaningful to consider stopping when the new net is trained to some extent. As such, we set J to the number of epochs T under the largest learning rate when training a baseline. K means how frequently AutoGrow checks the polices. In Convergent Growth, we simply set K = T , which is long enough to ensure convergence. In Periodic Growth, K is set to a small fraction of T to enable fast growth before convergence; more importantly, K = 3 is very robust to all networks and datasets. Therefore, all those hyper-parameters are very robust and strongly correlated to design considerations.

3. EXPERIMENTS:

In this paper, we use Basic3ResNet-2-3-2, for instance, to denote a model architecture which contains 2, 3 and 2 sub-modules in the first, second and third sub-networks, respectively. Sometimes we simplify it as 2-3-2 for convenience. AutoGrow always starts from the shallowest depth of 1-1-1 and uses the maximum validation accuracy as the metric to guide growing and stopping. All DNN baselines are trained by SGD with momentum 0.9 using staircase learning rate. The initial learning rate is 0.1 in ResNets and 0.01 in plain networks. On ImageNet, baselines are trained using batch size 256 for 90 epochs, within which learning rate is decayed by 0.1× at epoch 30 and 60. In all other smaller datasets, baselines are trained using batch size 128 for 200 epochs and learning rate is decayed by 0.1× at epoch 100 and 150.

Our early experiments followed prior wisdom by growing layers with Network Morphism (Wei et al., 2016; 2017; Chen et al., 2015; Elsken et al., 2017; Cai et al., 2018a; b) , i.e., AutoGrow with ZeroInit (or AdamInit) and Convergent Growth policy; however, it stopped early with very shallow DNNs, failing to find optimal depth. We hypothesize that a converged shallow net with Network Morphism gives a bad initialization to train a deeper neural network. Section 3.1 experimentally test that the hypothesis is valid. To tackle this issue, we intentionally avoid convergence during growing by three simple solutions, which are evaluated in Section 3.2. Finally, Section 3.3 and Section 3.4 include extensive experiments to show the effectiveness of our final AutoGrow.

3.1. SUBOPTIMUM OF NETWORK MORPHISM AND CONVERGENT GROWTH:

In this section, we study Network Morphism itself and its integration into our AutoGrow under Convergent Growth. When studying Network Morphism, we take the following steps: 1) train a shallower ResNet to converge, 2) stack residual blocks on top of each sub-network to morph to a deeper net, 3) use ZeroInit or AdamInit to initialize new layers, and 4) train the deeper net in a standard way. We compare the accuracy difference ("∆") between Network Morphism and training the deeper net from scratch. Table 2 summaries our results. Network Morphism has a lower accuracy (negative "∆") in all the cases, which validates our hypothesis that a converged shallow network with Network Morphism gives a bad initialization to train a deeper net. We visualize the optimization trajectories in Appendix A.0.1 to illustrate the hypothesis.

To further validate our hypothesis, we integrate Network Morphism as the initializer in AutoGrow with Convergent Growth policy. We refer to this version of AutoGrow as c-AutoGrow with "c-" denoting "Convergent." More specific, we take ZeroInit or AdamInit as sub-module initializer and "Convergent Growth" policy in Algorithm 1. To recap, in this setting, AutoGrow trains a shallower net till it converges, then grows a sub-module which is initialized by Network Morphism, and repeats the same process till there is no further accuracy improvement. In every interval of K training epochs (train(g(X 0 ), K) in Algorithm 1), "staircase" learning rate is used. The learning rate is reset to 0.1 at the first epoch, and decayed by 0.1× at epoch K 2 and 3K 4 . The results are shown in Table 3 by "staircase" rows, which illustrate that c-AutoGrow can grow a DNN multiple times and finally find a depth. However, there are two problems: 1) the final accuracy is lower than training the found net from scratch, as indicated by "∆", validating our hypothesis; 2) the depth learning stops too early with a relatively shallower net, while a deeper net beyond the found depth can achieve a higher accuracy as we will show in Table 6 . These problems provide a circumstantial evidence of the hypothesis that a converged shallow net with Network Morphism gives a bad initialization. Thus, AutoGrow cannot receive signals to continue growing after a limited number of growths. In Appendix A.0.1, Figure 6 (a) visualizes the trajectory of c-AutoGrow corresponding to row "2-3-6" in Table 3 .

3.2. ABLATION STUDY FOR AutoGrow DESIGN:

Based on the findings in Section 3.1, we propose three simple but effective solutions to further enhance AutoGrow and refer it as p-AutoGrow, with "p-" denoting "Periodic": (1) Use a large constant learning rate for growing, i.e., 0.1 for residual networks and 0.01 for plain networks. Stochastic gradient descent with a large learning rate intrinsically introduces noises, which help to avoid a full convergence into a bad initialization from a shallower net. Note that staircase learning rate is still used for fine-tuning after discovering the final DNN; (2) Use random initialization (UniInit or GauInit) as noises to escape from an inadequate initialization; (3) Grow rapidly before a shallower net converges by taking Periodic Growth with a small K.

p-AutoGrow is our final AutoGrow. In the rest part of this section, we perform ablation study to prove that the three solutions are effective. We start from c-AutoGrow, and incrementally add above solutions one by one and eventually obtain p-AutoGrow. In Table 3 , first, we replace the staircase learning rate with a constant learning rate, the accuracy of AutoGrow improves and therefore "∆" improves; second, we further replace Network Morphism (ZeroInit or AdamInit) with a random initializer (UniInit or GauInit) and result in a bigger gain. Overall, combining a constant learning rate with GauInit performs the best. Thus, constant learning rate and GauInit are adopted in the remaining experiments, unless we explicitly specify them. Note that, in this paper, we are more interested in automating depth discovery to find a final DNN ("found net") with a high accuracy ("accu"). Ideally, the "found net" has a minimum depth, a larger depth than which cannot further improve "accu". We will show in Figure 3 that AutoGrow discovers a depth approximately satisfying this property. The "∆" is a metric to indicate how well shallower nets initialize deeper nets; a negative "∆" indicates that weight initialization from a shallower net hurts training of a deeper net; while a positive "∆" indicates AutoGrow helps training a deeper net, which is a byproduct of this work.

Finally, we apply the last solution -Periodic Growth, and obtain our final p-AutoGrow. Our ablation study results for p-AutoGrow are summarized in Table 5 and Table 4 . Table 5 analyzes the impact of the growing period K. In general, K is a hyper-parameter to trade off speed and accuracy: a smaller K takes a longer learning time but discovers a deeper net, vice versa. Our results validate the preference of a faster growth (i.e. a smaller K). On CIFAR10/CIFAR100, the accuracy reaches plateau/peak at K = 3; further reducing K produces a deeper net while the accuracy gain is marginal/impossible. In the following, we simply select K = 3 for robustness test. More importantly, our quantitative results in Table 5 show that p-AutoGrow finds much deeper nets, overcoming the very-early stop issue in c-AutoGrow in Table 3 . That is, Periodic Growth proposed in this work is much more effective than Convergent Growth utilized in previous work.

For sanity check, we perform the ablation study of initializers for p-AutoGrow. The results are in Table 8 in Appendix A.0.3, which further validates our wisdom on selecting GauInit. The motivation of Network Morphism in previous work was to start a deeper net from a loss function that has been well optimized by a shallower net, so as not to restart the deeper net training from scratch (Wei et al., 2016; 2017; Chen et al., 2015; Elsken et al., 2017; Cai et al., 2018a; b) . In all our experiments, we find this is sure even with random initialization. Figure 2 plots the convergence curves and learning process for "42-42-42" in Table 5 . Even with GauInit, the loss and accuracy rapidly recover and no restart is observed. The convergence pattern in the "Growing" stage is similar to the "Fine-tuning" stage under the same learning rate (the initial learning rate 0.1). Similar results on ImageNet will be shown in Figure 8 . Our results challenge the necessity of Network Morphism when growing neural networks.

At last, we perform the ablation study on the initial depth of the seed network. Table 4 demonstrates that a shallowest DNN works as well as a deeper seed. This implies that AutoGrow can appropriately stop regardless of the depth of the seed network. As the focus of this work is on depth automation, we prefer starting with the shallowest seed to avoid a manual search of a seed depth. 

3.3. ADAPTABILITY OF AutoGrow:

To verify the adaptability of AutoGrow, we use an identical configuration (p-AutoGrow with K = 3) and test over 5 datasets and 4 seed architectures. Table 6 includes the results of all 20 combinations. Figure 3 compares AutoGrow with manual search which is obtained by training many DNNs with different depths from scratch. The results lead to the following conclusions and contributions: 2. For ResNets, a discovered depth (" " in Figure 3 ) falls at the location where accuracy saturates. This means AutoGrow discovers a near-optimal depth: a shallower depth will lose accuracy while a deeper one gains little. The final accuracy of AutoGrow is as good as training the discovered net from scratch as indicated by "∆" in Table 6 , indicating that initialization from shallower nets does not hurt training of deeper nets. As a byproduct, in plain networks, there are large positive "∆"s in Table 6 . It implies that baselines fail to train very deep plain networks even using Batch Normalization, but AutoGrow enables the training of these networks; In Appendix A.0.3, Table 9 shows the accuracy improvement of plain networks by tuning K, approaching the accuracy of ResNets with the same depth.

3. For robustness and generalization study purpose, we stick to K = 3 in our experiments, however, we can tune K to trade off accuracy and model size. As shown in Figure 3 , Auto-Grow discovers smaller DNNs when increasing K from 3 (" ") to 50 (" "). Interestingly, the accuracy of plain networks even increases at K = 50. This implies the possibility of discovering a better accuracy-depth trade-off by tuning K, although we stick to K = 3 for generalizability study and it generalizes well.

4. In Table 6 , AutoGrow discovers different depths under different sub-modules. The final accuracy is limited by the sub-module design, not by our AutoGrow. Given a sub-module architecture, our AutoGrow can always find a near-optimal depth. With a better sub-module architecture, such as NASNet cell (Zoph et al., 2018) , AutoGrow can improve accuracy.

Finally, our supposition is that: when the size of dataset is smaller, the optimal depth should be smaller. Under this supposition, we test the effectiveness of AutoGrow by sampling a subset of dataset and verify if AutoGrow can discover a shallower depth. In Appendix A.0.3, Table 11 summarizes the results. As expected, our experiments show that AutoGrow adapts to shallower networks when the datasets are smaller.

3.4. SCALING TO IMAGENET AND EFFICIENCY:

In ImageNet, K = 3 should generalize well, but we explore AutoGrow with K = 2 and K = 5 to obtain an accuracy-depth trade-off line for comparison with human experts. The larger K = 5 enables AutoGrow to obtain a smaller DNN to trade-off accuracy and model size (computation) and the smaller K = 2 achieves higher accuracy. The results are shown in Table 7 , which proves that AutoGrow automatically finds a good depth without any tuning. As a byproduct, the accuracy is even higher than training the found net from scratch, indicating that the Periodic Growth in AutoGrow helps training deeper nets. The comparison of AutoGrow and manual depth design (He et al., 2016) is in Figure 4 , which shows that AutoGrow achieves better trade-off between accuracy and computation (measured by floating point operations).

In Appendix A.0.3, Table 10 summarizes the breakdown of wall-clock time in AutoGrow. The growing/searching time is as efficient as (often more efficient than) fine-tuning the single discovered DNN. The scalability of AutoGrow comes from its intrinsic features that (1) it grows quickly with a short period K and stops immediately if no improvement is sensed; and (2) the network is small at the beginning of growing. 

4. RELATED WORK:

Neural Architecture Search (NAS) (Zoph & Le, 2016) and neural evolution (Miikkulainen et al., 2019; Angeline et al., 1994; Stanley & Miikkulainen, 2002; Liu et al., 2017a; Real et al., 2017) can search network architectures from a gigantic search space. In NAS, the depth of DNNs in the search space is fixed, while AutoGrow learns the depth. Some NAS methods (Bender et al., 2018; Liu et al., 2018b; Cortes et al., 2017 ) can find DNNs with different depths, however, the maximum depth is pre-defined and shallower nets are obtained by padding zero operations or selecting shallower branches, while our AutoGrow learns the depth in an open domain to find a minimum depth, beyond which no accuracy improvement can be obtained. Moreover, NAS is very computation and memory intensive. To accelerate NAS, one-shot models (Saxena & Verbeek, 2016; Pham et al., 2018; Bender et al., 2018) , DARTS (Liu et al., 2018b) and NAS with Transferable Cell (Zoph et al., 2018; Liu et al., 2018a) were proposed. The search time reduces dramatically but is still long from practical perspective. It is still very challenging to deploy these methods to larger datasets such as ImageNet.

In contrast, our AutoGrow can scale up to ImageNet thanks to its short depth learning time, which is as efficient as training a single DNN.

In addition to architecture search which requires to train lots of DNNs from scratch, there are also many studies on learning neural structures within a single training. Structure pruning and growing were proposed for different goals, such as efficient inference (Wen et al., 2016; Li et al., 2016; Lebedev & Lempitsky, 2016; He et al., 2017; Luo et al., 2017; Liu et al., 2017b; Dai et al., 2017; Huang et al., 2018; Gordon et al., 2018; Du et al., 2019 ), lifelong learning (Yoon et al., 2017) and model adaptation (Feng & Darrell, 2015; Philipp & Carbonell, 2017) . However, those works fixed the network depth and limited structure learning within the existing layers. Optimization over a DNN with fixed depth is easier as the skeleton architecture is known. AutoGrow performs in a scenario where the DNN depth is unknown hence we need to seek for the optimal depth. We hypothesize that a converged shallower net may not be an adequate initialization. Figure 5 visualizes and compares the optimization trajectories of Network Morphism and the training from scratch. In this figure, the shallower net is Basic3ResNet-3-3-3 (ResNet-20) and the deeper one is Basic3ResNet-5-5-5 (ResNet-32) in Table 2 . The initializer is ZeroInit. The visualization method is extended from Li et al. (2018) . Points on the trajectory are evenly sampled every a few epochs. To maximize the variance of trajectory, we use PCA to project from a high dimensional space to a 2D space and use the first two Principle Components (PC) to form the axes in Figure 5 . The contours of training loss function and the trajectory are visualized around the final minimum of the deeper net. When projecting a shallower net to a deeper net space, zeros are padded for the parameters not existing in the deeper net. We must note that the loss increase along the trajectory does not truly represent the situation in high dimensional space, as the trajectory is just a projection. It is possible that the loss remains decreasing in the high dimension while it appears in an opposite way in the 2D space. The sharp detour at "Morphing" in Figure 5 (a) may indicate that the shallower net plausibly converges to a point that the deeper net struggles to escape. In contrast, Figure 5(b) shows that the trajectory of the direct optimization in the deeper space smoothly converges to a better minimum.

Figure 6 (a) visualizes the trajectory of c-AutoGrow corresponding to row "2-3-6" in Table 3 . Along the trajectory, there are many trials to detour and escape an initialization from a shallower net. Figure 6 (b) visualizes the trajectory corresponding to row "2-4-3" in Table 3 , which is much smoother compared to Figure 6 (a). 10 .

Table 11 summarizes the adaptability of AutoGrow to the sizes of dataset. In each set of experiments, dataset is randomly down-sampled to 100%, 75%, 50% and 25%. For a fair comparison, K is divided by the percentage of dataset such that the number of mini-batches between growths remains The convergence curves and growing process on ImageNet for (a) Basic4ResNet-9-3-6-4 and (b) Plain4Net-6-6-6-6 in Table 10 . 

:



Figures and Tables:

end if meetGrowingPolicy() and subNetList.size()>0 then # current growing sub-network growingSubNet == fi (•; Wi) Wi = Wi ∪ W # stack a sub-module on top of fi (•; Wi) initializer(W); # initialize the new sub-module W # record the recent grown sub-network and iterate to a next sub-network grownSubNet = growingSubNet; growingSubNet = subNetList.next(growingSubNet); end end Fine-tune the discovered network g(X0) for N epochs; Output :

Figure 2: p-AutoGrow on CIFAR10 (K = 3). The seed net is Basic3ResNet-1-1-1.

Figure 3: AutoGrow vs manual search obtained by training many baselines from scratch. x -axis is the number of parameters. Dataset is CIFAR10.

Figure6(a) visualizes the trajectory of c-AutoGrow corresponding to row "2-3-6" in Table3. Along the trajectory, there are many trials to detour and escape an initialization from a shallower net. Figure6(b) visualizes the trajectory corresponding to row "2-4-3" in Table3, which is much smoother compared to Figure6(a). Figure6(c)(d) visualize the trajectories of p-AutoGrow with K = 50 and 3. The 2D projection gives limited information to reveal the advantages of p-AutoGrow

Figure 6: Optimization trajectory of AutoGrow, tested by Basic3ResNet on CIFAR10. (a) c-AutoGrow with staircase learning rate and ZeroInit during growing; (b) c-AutoGrow with constant learning rate and GauInit during growing; (c) p-AutoGrow with K = 50; and (d) p-AutoGrow with K = 3. For better illustration, the dots on the trajectory are plotted every 4, 20, 5 and 3 epochs in (a-d), respectively.

Figure 7: Loss surfaces around minima found by baselines and AutoGrow. Dataset is CIFAR10.

Figure 8:The convergence curves and growing process on ImageNet for (a) Basic4ResNet-9-3-6-4 and (b) Plain4Net-6-6-6-6 in Table10.

Comparison with previous works about layer growth. Our contributions are: (1) We propose AutoGrow to automate DNN layer growing and depth discovery. AutoGrow is very robust. With the same hyper-parameters, it adapts network depth to various datasets including MNIST, FashionMNIST, SVHN, CIFAR10, CIFAR100 and ImageNet. Moreover, AutoGrow can also discover shallower DNNs when the dataset is a subset. (2) AutoGrow demonstrates high efficiency and scales up to ImageNet, because the layer growing is as fast as training a single DNN. On ImageNet, it discovers a new ResNets with better trade-off between accuracy and computation complexity.

Table content: <table><tr><td>Growing</td></tr><tr><td>sub-nets</td></tr><tr><td>Stopped</td></tr><tr><td>sub-nets</td></tr><tr><td>Sub-modules</td></tr><tr><td>Initializer</td></tr></table>

Network Morphism tested on CIFAR10.

Table content: <table><tr><td>net backbone</td><td>shallower</td><td>deeper</td><td>initializer</td><td>accu %</td><td>∆  *</td></tr><tr><td>Basic3ResNet</td><td>3-3-3</td><td>5-5-5</td><td>ZeroInit AdamInit</td><td>92.71 92.82</td><td>-0.77 -0.66</td></tr><tr><td>Basic3ResNet</td><td>5-5-5</td><td>9-9-9</td><td>ZeroInit AdamInit</td><td>93.64 93.53</td><td>-0.27 -0.38</td></tr><tr><td colspan="3">Basic4ResNet 1-1-1-1 2-2-2-2</td><td>ZeroInit AdamInit</td><td>94.96 95.17</td><td>-0.37 -0.16</td></tr><tr><td/><td/><td/><td/><td/><td/></tr></table>

Ablation study of c-AutoGrow.

Table content: <table><tr><td>dataset</td><td>learning rate</td><td>initializer</td><td colspan="2">found net  † accu %</td><td>∆  *</td><td>dataset</td><td>learning rate</td><td>initializer</td><td colspan="2">found net  † accu %</td><td>∆  *</td></tr><tr><td/><td colspan="2">staircase ZeroInit</td><td>2-3-6</td><td>91.77</td><td>-1.06</td><td/><td colspan="2">staircase ZeroInit</td><td>4-3-4</td><td>70.04</td><td>-0.65</td></tr><tr><td/><td colspan="2">staircase AdamInit</td><td>3-4-3</td><td>92.21</td><td>-0.59</td><td/><td colspan="2">staircase AdamInit</td><td>3-3-3</td><td>69.85</td><td>-0.65</td></tr><tr><td>CIFAR10</td><td colspan="2">constant ZeroInit</td><td>2-2-4</td><td>92.23</td><td>0.16</td><td>CIFAR100</td><td colspan="2">constant ZeroInit</td><td>3-2-4</td><td>70.22</td><td>0.35</td></tr><tr><td/><td colspan="2">constant AdamInit</td><td>3-4-4</td><td>92.60</td><td>-0.41</td><td/><td colspan="2">constant AdamInit</td><td>3-3-3</td><td>70.00</td><td>-0.50</td></tr><tr><td/><td>constant</td><td>UniInit</td><td>3-4-4</td><td>92.93</td><td>-0.08</td><td/><td>constant</td><td>UniInit</td><td>4-4-3</td><td>70.39</td><td>0.36</td></tr><tr><td/><td>constant</td><td>GauInit</td><td>2-4-3</td><td>93.12</td><td>0.55</td><td/><td>constant</td><td>GauInit</td><td>3-4-3</td><td>70.66</td><td>0.91</td></tr><tr><td colspan="2">† Basic3ResNet</td><td/><td/><td/><td/><td colspan="5">*  ∆ = (accuracy of c-AutoGrow) -(accuracy of training from scratch)</td></tr></table>

p-AutoGrow with different seed architecture.

Table content: <table/>

p-AutoGrow with different growing interval K.

Table content: <table><tr><td/><td>CIFAR10</td><td/><td/><td>CIFAR100</td><td/></tr><tr><td>K</td><td>found net  †</td><td>accu %</td><td>K</td><td>found net  †</td><td>accu %</td></tr><tr><td>50</td><td>6-5-3</td><td>92.95</td><td>50</td><td>8-5-7</td><td>72.07</td></tr><tr><td>20</td><td>7-7-7</td><td>93.26</td><td colspan="2">20 8-11-10</td><td>72.93</td></tr><tr><td colspan="2">10 19-19-19</td><td>93.46</td><td colspan="2">10 18-18-18</td><td>73.64</td></tr><tr><td colspan="2">5 23-22-22</td><td>93.98</td><td colspan="2">5 23-23-23</td><td>73.70</td></tr><tr><td colspan="2">3 42-42-42</td><td>94.27</td><td colspan="2">3 54-53-53</td><td>74.72</td></tr><tr><td colspan="2">1 77-76-76</td><td>94.30</td><td colspan="2">1 68-68-68</td><td>74.51</td></tr><tr><td colspan="2">† Basic3ResNet</td><td/><td colspan="2">† Basic3ResNet</td><td/></tr></table>

The adaptability of AutoGrow to datasets

Table content: <table><tr><td>net</td><td>dataset</td><td>found net</td><td>accu %</td><td>∆  *</td><td>net</td><td>dataset</td><td>found net</td><td>accu %</td><td>∆  *</td></tr><tr><td/><td>CIFAR10</td><td>42-42-42</td><td>94.27</td><td>-0.03</td><td/><td>CIFAR10</td><td>23-22-22</td><td>90.82</td><td>6.49</td></tr><tr><td/><td>CIFAR100</td><td>54-53-53</td><td>74.72</td><td>-0.95</td><td/><td>CIFAR100</td><td>28-28-27</td><td>66.34</td><td>31.53</td></tr><tr><td>Basic3ResNet</td><td>SVHN</td><td>34-34-34</td><td>97.22</td><td>0.04</td><td>Plain3Net</td><td>SVHN</td><td>36-35-35</td><td>96.79</td><td>77.20</td></tr><tr><td/><td colspan="2">FashionMNIST 30-29-29</td><td>94.57</td><td>-0.06</td><td/><td colspan="2">FashionMNIST 17-17-17</td><td>94.49</td><td>0.56</td></tr><tr><td/><td>MNIST</td><td>33-33-33</td><td>99.64</td><td>-0.03</td><td/><td>MNIST</td><td>20-20-20</td><td>99.66</td><td>0.12</td></tr><tr><td/><td>CIFAR10</td><td colspan="2">22-22-22-22 95.49</td><td>-0.10</td><td/><td>CIFAR10</td><td colspan="2">17-17-17-17 94.20</td><td>5.72</td></tr><tr><td/><td>CIFAR100</td><td colspan="2">17-51-16-16 79.47</td><td>1.22</td><td/><td>CIFAR100</td><td colspan="2">16-15-15-15 73.91</td><td>29.34</td></tr><tr><td>Basic4ResNet</td><td>SVHN</td><td colspan="2">20-20-19-19 97.32</td><td>-0.08</td><td>Plain4Net</td><td>SVHN</td><td colspan="2">12-12-12-11 97.08</td><td>0.32</td></tr><tr><td/><td colspan="3">FashionMNIST 27-27-27-26 94.62</td><td>-0.17</td><td/><td colspan="3">FashionMNIST 13-13-13-13 94.47</td><td>0.72</td></tr><tr><td/><td>MNIST</td><td colspan="2">11-10-10-10 99.66</td><td>0.01</td><td/><td>MNIST</td><td colspan="2">13-12-12-12 99.57</td><td>0.03</td></tr><tr><td colspan="4">*  ∆ = (accuracy of AutoGrow) -(accuracy of training from scratch)</td><td/><td/><td/><td/><td/><td/></tr><tr><td>1. In</td><td/><td/><td/><td/><td/><td/><td/><td/><td/></tr></table>

AutoGrow discovers layer depth across all scenarios without any tuning, achieving the main goal of this work. Manual design needs m • n • k trials, where m and n are respectively the numbers of datasets and sub-module categories, and k is the number of trials per dataset per sub-module category;

Table content: <table/>

Scaling up to ImageNet

Table content: <table><tr><td>net</td><td colspan="2">K found net</td><td colspan="2">Top-1 Top-5  † ∆ Top-1</td></tr><tr><td>Basic4ResNet</td><td>2 5</td><td colspan="2">12-12-11-11 76.28 92.79 9-3-6-4 74.75 91.97</td><td>0.43 0.72</td></tr><tr><td>Bottleneck4ResNet</td><td>2 5</td><td>6-6-6-17 6-7-3-9</td><td>77.99 93.91 77.33 93.65</td><td>0.83 0.83</td></tr><tr><td>Plain4Net</td><td>2 5</td><td>6-6-6-6 5-5-5-4</td><td>71.22 90.08 70.54 89.76</td><td>0.70 0.93</td></tr><tr><td colspan="4">† ∆ = (Top-1 of AutoGrow) -(Top-1 of training from scratch)</td><td/></tr></table>

Tao Wei, Changhu Wang, and Chang Wen Chen. Modularized morphing of neural networks. arXiv preprint arXiv:1701.03281, 2017. Wei Wen, Chunpeng Wu, Yandan Wang, Yiran Chen, and Hai Li. Learning structured sparsity in deep neural networks. In Advances in neural information processing systems, pp. 2074-2082, 2016.

Table content: <table><tr><td>Jaehong Yoon, Eunho Yang, Jeongtae Lee, and Sung Ju Hwang. Lifelong learning with dynamically</td></tr><tr><td>expandable networks. arXiv preprint arXiv:1708.01547, 2017.</td></tr><tr><td>Barret Zoph and Quoc V Le. Neural architecture search with reinforcement learning. arXiv preprint</td></tr><tr><td>arXiv:1611.01578, 2016.</td></tr><tr><td>Barret Zoph, Vijay Vasudevan, Jonathon Shlens, and Quoc V Le. Learning transferable architectures</td></tr><tr><td>for scalable image recognition. In Proceedings of the IEEE conference on computer vision and</td></tr><tr><td>pattern recognition, pp. 8697-8710, 2018.</td></tr><tr><td>A APPENDIX</td></tr><tr><td>A.0.1 OPTIMIZATION TRAJECTORIES OF NETWORK MORPHISM</td></tr></table>

p-AutoGrow under initializers with K = 3.

Table content: <table><tr><td/><td>CIFAR10</td><td/></tr><tr><td>initializer</td><td>found net  †</td><td>accu</td></tr></table>

AutoGrow improves accuracy of plain nets.

Table content: <table><tr><td>dataset</td><td colspan="2">net layer #</td><td>method</td><td>accu %</td></tr><tr><td/><td>Plain4Net-6-6-6-6</td><td>26</td><td>baseline</td><td>93.90</td></tr><tr><td>CIFAR10</td><td>Plain4Net-6-6-6-6</td><td>26</td><td>AutoGrow K = 30</td><td>95.17</td></tr><tr><td/><td>Basic4ResNet-3-3-3-3</td><td>26</td><td>baseline</td><td>95.33</td></tr><tr><td/><td>Plain3Net-11-11-10</td><td>34</td><td>baseline</td><td>90.45</td></tr><tr><td>CIFAR10</td><td>Plain3Net-11-11-10</td><td>34</td><td>AutoGrow K = 50</td><td>93.13</td></tr><tr><td/><td>Basic3ResNet-6-6-5</td><td>36</td><td>baseline</td><td>93.60</td></tr></table>

The efficiency of AutoGrow

Table content: <table><tr><td>net GPUs</td><td>growing</td><td>fine-tuning</td></tr><tr><td colspan="3">Basic4ResNet-12-12-11-11 4 GTX 1080 Ti 56.7 hours 157.9 hours</td></tr><tr><td>Basic4ResNet-9-3-6-4 4 GTX 1080</td><td>47.9 hours</td><td>65.8 hours</td></tr><tr><td>Bottleneck4ResNet-6-6-6-17 4 TITAN V</td><td colspan="2">45.3 hours 114.0 hours</td></tr><tr><td>Bottleneck4ResNet-6-7-3-9 4 TITAN V</td><td>61.6 hours</td><td>78.6 hours</td></tr></table>

The adaptability of AutoGrow to dataset sizes

Table content: <table><tr><td colspan="3">Basic3ResNet on CIFAR10</td><td colspan="2">Plain3Net on MNIST</td><td/></tr><tr><td>dataset size</td><td>found net</td><td>accu %</td><td>dataset size</td><td>found net</td><td>accu %</td></tr><tr><td>100%</td><td>42-42-42</td><td>94.27</td><td>100%</td><td>20-20-20</td><td>99.66</td></tr><tr><td>75%</td><td>32-31-31</td><td>93.54</td><td>75%</td><td>12-12-12</td><td>99.54</td></tr><tr><td>50%</td><td>17-17-17</td><td>91.34</td><td>50%</td><td>12-11-11</td><td>99.46</td></tr><tr><td>25%</td><td>21-12-7</td><td>88.18</td><td>25%</td><td>10-9-9</td><td>99.33</td></tr><tr><td colspan="3">Basic4ResNet on CIFAR100</td><td colspan="2">Plain4Net on SVHN</td><td/></tr><tr><td>dataset size</td><td>found net</td><td>accu %</td><td>dataset size</td><td>found net</td><td>accu %</td></tr><tr><td>100%</td><td>17-51-16-16</td><td>79.47</td><td>100%</td><td>12-12-12-11</td><td>97.08</td></tr><tr><td>75%</td><td>17-17-16-16</td><td>77.26</td><td>75%</td><td>9-9-9-9</td><td>96.71</td></tr><tr><td>50%</td><td>12-12-12-11</td><td>72.91</td><td>50%</td><td>8-8-8-8</td><td>96.37</td></tr><tr><td>25%</td><td>6-6-6-6</td><td>62.53</td><td>25%</td><td>5-5-5-5</td><td>95.68</td></tr></table>

Peer review (iclr2020):
This paper*s contribution is a method for automatically growing the depth of a neural network during training. It compares several heuristics that may be used to successfully achieve this goal and identifies a set of choices that work well together on multiple datasets. The paper focuses on CNNs that conform to a popular design pattern where the network is organized into a series of sub-networks, each consisting of a series of sub-modules (sometimes called blocks) operating at the same resolution. To be precise, the proposed method aims to learn the length of each series of sub-modules. A main contribution of the paper is the demonstration that it is not necessary to train a network until convergence before adding new sub-modules as proposed in past work. Instead, it is better to grow the network after training for a short while. My current decision for this paper is a weak rejection due to the points below. However, I am open to revising my opinion if these points are addressed satisfactorily. - The growing strategy identified in the paper as a superior alternative seems to be already known and used, at least in the speech recognition community. Seide et al. (2011) called it Discriminative Pre-training, and showed that it outperforms greedy layer-wise pretraining and DBN pre-training. Zeyer et al. (2017) reported that a similar method also enables the training of very deep LSTM networks which is otherwise notoriously hard. In general, the existence of prior work with the same ideas does not preclude acceptance, but the existence of this work needs to be clearly stated early on and the additional value of the current study sufficiently clarified. - I find it strange that the final networks found by the proposed method usually have the same/similar number of sub-modules per sub-network (Tables 4,5,6) on multiple datasets. The only exceptions appear to be Basic4ResNet/CIFAR100 in Table 6 and about 50% of ImageNet results in Table 7. This regularity suggests that either A) the proposed algorithm prefers to set same number of sub-modules per sub-network due to its design, or B) datasets except ImageNet have an inherent shared property that produces this result. Since option A suggests a bias in the algorithm, this peculiarity of the results needs to be investigated or explained further. - Figure 3 constitutes the main evidence that Autogrow finds approximately optimal depths as compared to manual searching, but it is not clear how the plot for baselines is obtained. For any given parameter budget, there are multiple baseline networks possible since the sub-networks can have different number of sub-modules (see previous point). This does not appear to be accounted for in Figure 3. Further, when dealing with CNNs, it would be more useful to have computation budget on the x-axis instead of the parameter budget. This would better account for the difference between increasing depth in an earlier sub-network vs. a later one. - The reported results appear to be for single trials throughout the paper. This does not seem sufficient especially for results in Tables 2 and 3 where many differences are rather small, and so drawing conclusions from these tables would be unscientific. References: Seide, Frank, et al. *Feature engineering in context-dependent deep neural networks for conversational speech transcription.* 2011 IEEE Workshop on Automatic Speech Recognition & Understanding. IEEE, 2011. https://www.microsoft.com/en-us/research/wp-content/uploads/2016/02/FeatureEngineeringInCD-DNN-ASRU2011-pub.pdf Zeyer, Albert, et al. *A comprehensive study of deep bidirectional LSTM RNNs for acoustic modeling in speech recognition.* 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2017. https://arxiv.org/abs/1606.06871 Update after rebuttal ----------------------------- I*m sympathetic to the unfortunate situation that the authors are in, since the underlying growing strategy has already been covered by prior work. As I mentioned earlier, a sufficient rewrite of the paper can clearly state what has been done already so as not to take credit from the earlier authors. A revised version of the paper has not been uploaded; I suggest that the authors do so for the future. I agree that the focus of this paper is learning the *optimal* depth by using the growing strategy. But I am not convinced that the technique indeed finds optimal depths based on the regularity of the sub-network depths mentioned in my review. The rebuttal suggests reasons for the obtained regularity, but does not prove that these regular structures obtained are indeed optimal and not an artifact of the algorithm itself. The baselines are also using the same regular architectures, which distorts the overall picture because it is possible that a non-regular architecture provides a better trade-off. While my rating doesn*t change, I do think that the work is in an interesting direction. My final suggestions for the future are: - Investigate where non-regular architectures (unequal sub-network depths) are in the trade-off between accuracy, flops and parameters. - Investigate whether the proposed algorithm can be modified to easily find non-regular architectures if they can yield equally good performance as regular ones at similar or lower cost.