AutoGrow: Automatic Layer Growing in Deep Convolutional Networks
Authors: Wei Wen
Feng Yan
Hai Li
Venue:  (2020)
Keywords: Growing
depth
neural networks
automation

Abstract: Depth is a key component of Deep Neural Networks (DNNs), however, designing depth is heuristic and requires many human efforts. We propose AutoGrow to automate depth discovery in DNNs: starting from a shallow seed architecture, AutoGrow grows new layers if the growth improves the accuracy; otherwise, stops growing and thus discovers the depth. We propose robust growing and stopping policies to generalize to different network architectures and datasets. Our experiments show that by applying the same policy to different network architectures, AutoGrow can always discover near-optimal depth on various datasets of MNIST, FashionMNIST, SVHN, CIFAR10, CIFAR100 and ImageNet. For example, in terms of accuracy-computation trade-off, AutoGrow discovers a better depth combination in ResNets than human experts. Our AutoGrow is efficient. It discovers depth within similar time of training a single DNN.

Under review as a conference paper at ICLR 2020
AutoGrow: AUTOMATIC LAYER GROWING IN DEEP CONVOLUTIONAL NETWORKS
Anonymous authors
Paper under double-blind review
ABSTRACT
Depth is a key component of Deep Neural Networks (DNNs), however, design- ing depth heuristic and requires many human efforts. We propose AutoGrow to automate depth discovery in DNNs: starting from shallow seed architecture, AutoGrow grows new layers if the growth improves the accuracy; otherwise, stops growing and thus discovers the depth. We propose robust growing and stopping policies to generalize to different network architectures and datasets. Our exper- iments show that by applying the same policy to different network architectures, AutoGrow can always discover near-optimal depth on various datasets of MNIST, FushionMNIST. SVHN, CIFARIO 10, CIFARION 100 and ImageNet. For example, in trade-off, AuloGrow discovers a better depth com-
terms of accuracy-computation bination in ResNets than human experts. Our AuloGrow is efficient. It discovers depth within similar time of training single DNN.
INTRODUCTION
Layer depth is one of the decisive factors of the success of Deep Neural Networks (DNNs). For example, image classification accuracy keeps improving the depth of network models grows (Krizhevsky et al.
& Zisserman 2014, Szegedy et al. [2015] He et al. 2016
Huang et al.] 2017). Although shallow networks cannot ensure high accuracy, DNNs composed of too many layers may suffer from over-fitting and convergence difficulty in training. How to obtain the optimal depth for DNN still remains mysterious. For instance, ResVer- 152 (He et al., 2016) uses 3, 8, 36 and 3 residual blocks under output sizes of 56 x 56, 28 x 28, 14 x 14 and 7 x 7, respectively, which don't show an obvious quantitative relation. In practice, people usually reply on some heuristic trials and tests to obtaín the depth of a network: they first design DNN with specific depth and then train and evaluate the network on a given dataset; finally, they change the depth and repeat the procedure until the accuracy meets the requirement. Besides the high compu- tational cost induced by the iteration process, such trial & test iterations must be repeated whenever dataset changes. In this paper, we propose AutoGrow that can automate depth discovery given layer architecture. We will show that AutoGrow generalizes to different datasets and layer architectures.
There are some previous works which add morph layers increase the depth in DNNs. Vg- gNet (Simonyan & Zisserman 2014) and DropIn (Smith et al. 2016 added new layers into shal- lower DNNs; Network Morphism (Wei et al. [0166] 2017 Chen al. 2015) morphed each layer to multiple layers to increase the depth meanwhile preserving the function of the shallower net. Table summarizes differences in this work. Their goal was to overcome difficulty of training deeper DNNS or accelerate it. Our goal is to automatically find an optimal depth. Moreover, previous works ap- plied layer growth by once or few times pre-defined locations grow a pre-defined number of layers; in contrast, ours automatically learns the number of new layers and growth locations without limiting growing times. We will summarize more related works in Section
Figure illustrates example of AutoGrow. It starts from the shallowest backbone network and gradually grows sub-modules (A sub-module can be one or more layers, e,g., a residual block); the growth stops once stopping policy is satisfied. We studied multiple initializers of new layers and multiple growing policies, and surprisingly find that: (1) a random initializer works equally or better than complicated Network Morphism; (2) it is more effective to grow before shallow net converges. We hypothesize that this is because a converged shallow net is an inadequate initialization for training deeper net, while random initialization can help escape from bad starting point.

---

Under review as a conference paper at ICLR 2020
Motivated by this, intentionally avoid full convergence during the growing by using (1) random initialization of new layers, (2) a constant large learning rate, and (3) short growing interval.
Epochs
Sood
subirats
Previous works
Ours
Stopoet
Goal
Ease training
Depth automation
Times
Once or
Unlimited Learned Learned
Locations
Human defined
Sub-modddn
Layer #
Human defined
Table 1:
Comparison with previous works
about layer growth.
Figure I: A simple example AutoGrow.
Our contributions are: (1) We propose AutoGrow to automate DNN layer growing and depth discov- ery. AutoCrow is very robust. With the same datasets including MNIST, MNSh,, SVHN, CIFARIO 10, CIFAR 100 and ImageNet. Moreover, AutoGrow can also discover shallower DNNs when the dataset subset. (2) AutoGrow demon- strates high efficiency and scales up to ImageNet, because the layer growing
it adapts network depth to various
fast training single DNN. On ImageNet, it discovers new ResNets with better tradc-off between accuracy and computation complexity. (3) We challenge the idea of Network Morphism, as random initialization works equally or better when growing layers. (4) We find that it is beneficial to rapidly grow layers before shallower net converge, contradicting previous intuition.
AutoGrow- - A DEPTH GROWING ALGORITHM
Algorithm AutoGrow Algorithm.
Input
shallow network g(2gg) composed sub networks (ff W.) 0... M 1), where sub network has only one sub-module dimension reduction sab-module), epoch interval K to chock growing and stopping policies; the number of fine-tuning opochs after growing.
Initialization:
A Circular Linked List of sub-nctworks under growing: subletList fo Wo)
fM 1 (-i WM-
The current growing sub-network: growingSublee = suaNetList.heed = fo (:s Wo):
The recent grown sub-actwork: grownSublees None;
Process
if exist geowing network/s)
while Wubbettiaat.. att.t.. s1.2 ()3 do
Lzain (g(X)), K) frriin whole network g(No). (NN).fo for epochs
if meetttttopg sreettStoppiggee ingpinggPlcyy () then
# remove a sub network from the growing list if its gnowth did not improve accuracy subNetListt. delete igrosnSub3eet):
end weweeeeroowogggooeeee II and subNetList. wtw.iiit..iieee size (1 >0 then
12
if
W.) W; = W; U.W 8 stack a nub module on f: (:: 9c))
growing sub-nerwork growingSubbeet
14
15
initial (W); initiolize new suh-module W
16
# record the recent
sub-nerwork and Hterate to next sub-network grownSablet: = growingSubbee growingBubbler sublet Mis.eelsst...... (gzowingBuuNee..
17
19
end
20
end
Fine-tune the discovered network a (a)g) epochs;
Output
22
A trained ncural network o(X)) with learned depth.
Figure D gives an overview of the proposed AutoGrow. In this paper, we use network, sub sub-networks, sub-module, and layers to describe the architecture hierarchy. A network is composed of a cascade of sub-networks. sub-network composed of sub-modules, which typical share the same output size. A sub-module (e.g. residual block) is an elementary growing block composed of one or few layers. In this section, we rigorously formulate a generic version of AutoGrow which will be materialized in subsections. A deep convolutional network g is cascade of sub-networks by composing functions as g(X)) = [ffM- (fas (-.- fi (fu (Xg)) .)))), where Ng is an input im- age, M is the number of sub-networks, I(-) is loss function, and X;+1 f. (X)) is sub-network that operates on an input image or feature tensor X, E R xh, xw: Here, c; is the number of chan- nels, and h, and w; spatial dimensions. f: (X:) is simplified notation of (X; W.). where W,
2

---

Under review as a conference paper at ICLR 2020
is a set of sub-modules' parameters within the i-th sub-network. Thus W [W, :i= denotes the whole set of parameters in the DNN. To facilitate growing, the following properties are supported within a sub-network: (1) the first sub-module usually reduces the size of input feature maps, egg. using pooling or convolution with a stride; and (2) all sub-modules in a sub-network maintain the same output size. As such, our framework can support popular networks, including VggNer-like plain networks (Simonyan & Zisserman, 2014), GoogLeNer (Szegedy et al., 22015), ResNets (He al. 2016) and DenseNets (Huang al. 2017). In this paper, we select ResNets and VggNer-like nets as representatives of DNNs with and without shortcuts, respectively.
With above notations, [ starts with the shallowest net where every sub-network has only one sub-module for spatial dimen- sion reduction. AutoGrow loops over all growing sub-networks in order. For each sub-network, AutoGrow stacks a new sub-module. When the new sub-module does not improve the accuracy, the growth in corresponding sub-network will be permanently stopped. The details of our method will be materialized in the following subsections.
describes the AutoGrow algorithm. In brief, Auto
2.1 SEED SHALLOW NETWORKS AND SUB-MODULES
In this paper, in all datasets except ImageNet, we explore growing depth for four types of DNNs: (1) Basic3Re sNet: the same ResNet used for CIFARIO in He et al. (2016), which has residual sub- networks with output spatial sizes of 32 32, 16 x 16 and 8 x 8, respectively; (2) a variant of ResNer used for ImageNet in He et al. (016) built by basic residual blocks (each of which contains two convolutions and one shortcut). There are 4 sub-networks with output spatial sizes of 32 x 32, 16 x 16, 8 x 8 and 4 x 4, respectively; (3) Plain3Net: a VggNer-like plain net by removing shortcuts in Basi Ceeeeeeeeeeeeee (4) PNein VggNer-like plain net by removing shortcuts in Basic e 4ResNet.
In AutoGrow, the architectures of seed shallow networks and sub-modules are pre-defined. In plain DNNs, a sub-module is a stack of convolution, Batch Normalization and ReLU; in residual DNNs, a sub-module is a residual block. In AutoGrow, a sub-network is a stack of all sub-modules with the same output spatial size. Unlike He et al. (016)) which manually designed the depth, AutoGrow starts from a seed architecture in which each sub-network has only one sub-module and automati- cally learns the number of sub-modules.
On ImageNet, we apply the same backbones in He et al. (2016) as the seed architectures. A seed architecture has only one sub-module under each output spatial size. For a ResNer using basic residual blocks or bottleneck residual blocks (He et al.] 2016), we respectively name as or Bottleneck Nes.Ne.. Plain4Net is also obtained by removing short- cuts in Basic4ResNet.
2.2 SUB- MUB-LE INITIALIZERS
Here we explain how to initialize a new sub-module Win initializer(M)) mentioned in Algo- rithm [ Network Morphism changes DNN architecture meanwhile preserving the loss function via special initialization new layers, that is, g(X;; W) = g(Y; W WUW) YYg. A residual sub-module shows a nice property: when stacking a residual block and initializing the last Batch Normalization layer as zeros, the function of the shallower net is preserved but the DNN is morphed to a deeper net. Thus, Network Morphism can be easily implemented by this zero initialization (ZeroInit).
In this work, all layers in W are initialized using default randomization, except for a special treat- ment of the last Batch Normalization layer a residual sub-module. Besides ZeroInit. we propose a new AdamInit for Network Morphism. In AdamInit, we freeze all parameters except the last Batch Normalization layer W, and then use Adam optimizer (Kingma Bal 2014) to op- timize the last Bath Normalization for maximum 10 epochs till the training accuracy of the deeper net as good as the shallower one. After AdamInit, t, all parameters are jointly optimized. We view AdamInit ieitttt as a Network Morphism because the training loss is similar after AdamInit. We empirically find that AdamInit can usually find a solution in less than 3 epochs. We also study random initialization of the last Batch Normalization layer using uniform (UniInite (GauInit) noises with a standard deviation 1.0. We will show that GauInit obtains the best result, challenging the idea of Network Morphism (Wei et al. 2016 2017
Gaussian
Chen al. 2015).
3

---

Under review as a conference paper at ICLR 2020
2.3 GROWING AND STOPPING POLICIES
In Algorithm] |
a growing policy refers to meetGrowingFoliey (), which returns true when the network should grow sub-module. Two growing policies are studied here:
1. Convergent Growth: meetGrowl ingPoicy cy () returns true when the improvement of validation aceuracy less than т in the last K epochs. That is, Convergent Growth, AutoGrow only grows when current network has converged. This is similar growing criterion adopted in previous works (Elsken et al. 2017: Cai et al., 2018a b).
2. Periodic Growth: meetGrowi www.swwiinongyy...o
always returns true, that is, the network always grows every K epochs. Therefore, K is also the growing period. In the best practice of AutoGrow, K is small (e.g. K = such that it grows before current network converges.
Our experiments will show that Periodic Growth outperforms Convergent Growth. We hypothesize that a fully converged shallower net is an inadequate initialization to train a deeper net. We will perform experiments test this hypothesis and visualize optimization trajectory to illustrate it.
() in Algorithm T When Convergent Growth () returns true if recent growth does not improve vali- dation accuracy more than т within K epochs. We use a similar stopping policy for Periodic Growth; however, as it can grow rapidly with small period K (e.g. K = 3) before con- verges, we use a larger window size J for stop. Specifically, when Periodic Growth is adopted, meet SreetStoppingPoli
stopping policy denotes wtwt.ttttpppiggeee......
is adopted, weetstoppiiggee
() returns true when the validation accuracy improves less than the last // epochs, where J >>> K.
Hyper-parameters J and K control the operation of AutoGrow and can be easily setup and gen- eralize well. denotes the significance of accuracy improvement for classification. We simply set 0.05% in all experiments. represents how many epochs to wait for an accuracy improvement before stopping the growth of sub-network. It is more meaningful to consider stopping when the new net is trained to some extent. As such, we set to the number of epochs under the largest learning rate when training baseline. K means how frequently AutoGrow checks the polices. In Convergent Growth, we simply set K T. which is long enough to ensure convergence. In Periodic Growth, K is set to small fraction of T to enable fast growth before convergence; more impor- tantly, K = 3 is very robust to all networks and datasets. Therefore, all those nyper-parameten are very robust and strongly correlated to design considerations.
3 EXPERIMENTS
In this paper, we use Basic3ResNet-22-1
for instance, to denote a model architecture which contains 2, 3 and sub-modules in the first, second and third sub-networks, respectively. Sometimes we simplify it as 2-3-2 for convenience. AutoGrow always starts from the shallowest depth of 1-1-1 and uses the maximum validation accuracy as the metric to guide growing and stopping. All DNN baselines are trained by SGD with momentum 0.9 using staircase learning rate. The initial learning rate 0.1 in ResNets and 0.01 in plain networks. On ImageNet, baselines are trained using batch size 256 for 90 epochs, within which learning rate is decayed by 0.1 x at epoch 30 and 60. In all other smaller datasets, baselines are trained using batch size 128 for 200 epochs and learning rate is decayed by 0.1x at epoch 100 and 150.
Our early experiments followed prior wisdom by growing layers with Network Morphism (Wei et al. 2016; 2017 Chen et al.] 2015 Elsken et al. 2017 Cai et al. 2018a b). i.e., AutoGrow with ZeroInit (or AdamInit) and Convergent Growth policy; however, it stopped early with very shallow DNNs, failing to find optimal depth. We hypothesize that converged shallow net with Network Morphism gives a bad initialization to train a deeper neural network. Section 3.I experi- mentally test that the hypothesis is valid. To tackle this issue, intentionally avoid convergence during growing by three simple solutions, which are evaluated in Section [.2 Finally, Section (3.3 and Section 3.4 include extensive experiments to show the effectiveness of final AutoGrow.
4

---

Under review as a conference paper at ICLR 2020
3.1 SUBOPTIMUM OF NETWORK MORPHISM AND CONVERGENT GROWTH
In this section, we study Network Morphism itself and its integration into our AutoGrow under Convergent Growth. When studying Network Morphism, we take the following steps: 1) train a shallower ResNer to converge, 2) stack residual blocks on top of each sub-network to morph to deeper net, 3) use ZeroInit or AdamInit to initialize new layers, and 4) train the deeper net in a standard way. We compare the accuracy difference ("A") between Network Morphism and training the deeper net from scratch. Sablessummarie our results. Network Morphism has a lower accuracy (negative "A") in all the cases, which validates our hypothesis that converged shallow network with Network Morphism gives a bad initialization to train a deeper net. We visualize the optimization trajectories in Appendix A.O.I to illustrate the hypothesis.
To further validate our hypothesis, we integrate Network Morphism as the initializer in AutoGrow with Convergent Growth policy. We refer this version of AutoGrow c-AutoGrow with "c-" denoting "Convergent." More specific, we take ZeroInit t or AdamInit as sub-module initial- izer and "Convergent Growth" policy in Algorithm [ To recap, in this setting, AutoGrow trains a shallower net till it converges, then grows sub-module which is initialized by Network Morphism, and repeats the same process till there is no further accuracy improvement. In every interval of K training epochs (t rarain (g(2),, K) in Algorithm "staircase" learning rate is used. The learning rate is reset to 0.1 at the first epoch, and decayed by 0.1x at epoch 2
& and 3K The results are 4 shown in Table 3 by "staircase" rows, which illustrate that c-AutoGrow can grow a DNN multiple times and finally find depth. However, there are two problems: 1) the final accuracy is lower than training the found net from scratch, as indicated by "A", validating our hypothesis; 2) the depth learning stops too early with relatively shallower net, while deeper net beyond the found depth can achieve a higher accuracy as we will show in Table & These problems provide a circumstantial evidence of the hypothesis that converged shallow net with Network Morphism gives bad ini- tialization. Thus, AutoGrow cannot receive signals to continue growing after a limited number of growths. In Appendix A.O.1. Figure a) visualizes the trajectory of c-AutoGrow corresponding to row 2--36 6" in Table 3
3.2 ABLATION STUDY FOR AutoGrow DESIGN
Based on the findings in Section 3.1 we propose three simple but effective solutions further hance AutoGrow and refer it as p-AutoGrow, with "p-" denoting "Periodic": (1) Use a large constant learning rate for growing, i.e., 0.1 for residual networks and 0.01 for plain networks. Stochastic gra- dient descent with a large learning rate intrinsically introduces noises, which help to avoid a full convergence into a bad initialization from a shallower net. Note that staircase learning rate still used for fine-tuning after discovering the final DNN; (2) Use random initialization (Uni Init or GauTnin t.) as noises to escape from an inadequate initialization; (3) Grow rapidly before a shallower net converges by taking Periodic Growth with a small K.
p-AutoGrow our final AutoGrow. In the rest part of this section, we perform ablation study to prove that the three solutions are effective. We start from c-AutoGrow, and incrementally add above solutions one by one and eventually obtain p-AutoGrow. In Table 3. first, we replace the staircase learning rate with a constant learning rate, the accuracy of AutoGrow improves and therefore "A" improves; second, we further replace Network Morphism /Zeronnt or AdamInit) with a ran- dom initializer (UniInit or GauInit) ) and result in bigger gain. Overall, combining constant learning rate with GauInit t. performs the best. Thus, constant learning rate and GauInit are adopted in the remaining experiments, unless explicitly specify them.
Table 2: Network Morphism tested on CIFARIO.
net backbone
shallower
deeper
initializer
accu
ZeroInit
92.71
-0.77
Basi C3ReNNet
3-3-3
5-5-5
AdamTnit
92.82
-0.66
ZeroInit 93.64
-0.27
Basic3ResNett
5-5-5
9-9-9
AdamInit
93.53
-0.38
7eroTnit 94.96
-0.37
Ras i N4ResNet
1-1-1-1
2-2-2-2
AdamInit
95.17
-0.16
(accuracy of Network Morphism)
(uccuracy of training from scratch)
5

---

Under review as a conference paper at ICLR 2020
Table 3: Ablation study of c-AutoGrow.
dataset
learning Tale
initializer
found net!
%
A*
dataset
learning
initializer
found net
A'
staircane
ZeroInit
2-3-6
91.77
-1.06
Haircate
ZercInit
4-3-4
70.04
-0.65
AdonTnit
3-4-3
92:21
-0.59
staireuss
AdamInit
3-3-3
69:85
-0.65
CIFARIO
constant
ZeroInit
2-2-1
0,16
CIFAR100
constant
ZereInit
3-2-4
constant
AdanInit
3-6-0
92.60
-0.41
constant
AdanInit
3-3-3
70.00
-0.50
constant
Un T
3-2-4
92.93
-0.08
Uniinit
2-4-3
70.39
0.36
constaut
GauInit
2-4-3
93.12
0.55
constant
GauInit
3-4-3
BasisSRestett
(accuracy of csAutorow) (accuracy training from scratch)
Note that, in this paper, we are more interested in automating depth discovery to find a final DNN ("found net") with high accuracy ("accu"). Ideally, the "found net" has a minimum depth, larger depth than which cannot further improve "accu". We will show in Figure that AutoGrow discovers depth approximately satisfying this property. The "A" is a metric to indicate how well shallower nets initialize deeper nets; a negative "A" indicates that weight initialization from a shallower net hurts training of a deeper net; while positive "A" indicates AutoGrow helps training deeper net, which is a byproduct of this work.
Finally, we apply the last solution Periodic Growth, and obtain our final p-AutoGrow. Our ab- lation study results for P-AutoGrow are summarized in Table [3] and Table 4. Table [3] analyzes the impact of the growing period K. In general, K is a hyper-parameter to trade off speed and accu- racy: a smaller K takes a longer learning time but discovers a deeper net, vice versa. Our results validate the preference of a faster growth (i.e. a smaller K). On CIFARIO/IIFARI0) the accuracy reaches platcau/peak at K = 3; further reducing K produces a deeper net while the accuracy gain marginal/impossible. In the following, we simply select K 3 for robustness test. More impor- tantly, our quantitative results in T that p-AutoGrow finds much deeper nets, overcoming the very-early stop issue in c-AutoGrow in Table That is, Periodic Growth proposed in this work is much more effective than Convergent Growth utilized in previous work.
For sanity check, we perform the ablation study of initializers for p-AutoGrow. The results are in Table 8 in Appendix [A...] which further validates our wisdom on selecting Gau The moti- vation of Network Morphism in previous work was to start a deeper net from a loss function that has been well optimized by a shallower net, as not to restart the deeper net training from scratch (Wei) jet al. 2016 2017] Chen et al. 2015) Elsken et al. 2017, Cai et al. 2018ala b). In all our experiments, find this is even with random initialization. Figure 2 plots the convergence curves and learning process for "42-42-42" in Table Even with GauInit, t, the loss and accuracy rapidly recover and restart is observed. The convergence pattern in the "Growing" stage is similar to the "Fine-tuning" stage under the same learning rate (the initial learning rate 0.1). Similar results on ImageNet will be shown in Figure Our results challenge the necessity of Network Morphism when growing neural networks.
At last, we perform the ablation study on the initial depth of the seed network. Table 4 demonstrates that a shallowest DNN works as well as a deeper seed. This implies that AutoGrow can appropriately stop regardless of the depth of the seed network. As the focus of this work is on depth automation, we prefer starting with the shallowest seed to avoid a manual search of a seed depth.
Fine-luning
Growing
The namber of loyers
dataset seed net!
found net
accuracy %
90
250
80
200
CIFARIO
1-1-1
42-42-42
94.27
70
150
5-5-5
46-46-46
94.16
60
train acCH
val
100
CIFARIO
1-1-1-1 22-22-22-22
95.49
vol
5-5-5-5
23-22-22-22
95.62
Basic3ReaNett or Basic4ReaNet.
0
60
120
180
240
300
360
420
480
540
600
Epoch
Table 4: p-AutoGrow with different seed archi- tecture.
Figure 2: p-AutoGrow on CIFARIO (K = 3). The seed net is Basic3ResNet-
3.3 ADAPTABILITY OF AutoGrow
To verify the adaptability of AutoGrow, we use an identical configuration (p-AutoGrow with K = 3) and test over 5 datasets and 4 seed architectures. Tanlelesssludess the results of all 20 combinations. Figure 3 compares AutoGrow with manual search which is obtained by training many DNNs with different depths from seratch. The results lead to the following conclusions and contributions:

---

Under review as a conference paper at ICLR 2020
Table p-AutoGrow with different growing interval K.
CIFARI0
CIFAR100
X
found net"
aceu %
K
found net"
accu %
50
6-5-3
92.95
50
8-5-7
72.07
20
7-7-7
95.26
20
8-11-10
72.93
10
19-19-19
93.46
10
18-18-18
73.64
23-22-22
93.98
23-23-23
73.70
3
42-42-42
94.27
54-53-53
74.72
77-75-76
94.30
68-68-68
74.51
Basi=3ResNet
Basic3Restiet
Table 6: The adaptability of AutoGrow to datasets
dataset
found net
acen %
A²
net
dataset
found net
accm %
A*
CIFAR1O
42-42-42
94.27
-0.03
CIFARID
23-22-22
90.82
6.49
CIFAR100
54-53-53
74.72
-0.95
CIFARI00
20-28-27
66.34
31.53
3kekc3kesket
SVHN
34-34-34
91.22
0.04
Plain3Net
SVHN
36-35-35
96.79
77.20
FashionMNIST
20-29-29
91.57
-0.05
FustionMNIST
17-17-17
91.19
0.56
33-33-33
20-20-20
CIFARIO
22-22-22-22
95.49
-0.10
CIFARIO
17-17-17-17
94.20
5.72
CIFARI00
17-51-15-15
79.47
1.22
CIFARI00
16-15-15-15
73.91
29.34
BasictResNet
SVHN
20-20-19-19
97.32
-0.05
PlaintNet
SVHN
12-12-12-11
97.08
0.32
FashionMNIST
27-27-27-25
94.62
FashionMNIST
13-13-13-13
MNNSTT
11-10-10-10
99.66
0.01
MNIST
13-12-12-12
99.57
0.03
(accuracy AutoGrow) (accuracy of training from scratch)
1. In 6 AutoGrow discovers layer depth across all scenarios without any tuning, achiev- ing the main goal of this work. Manual design needs le trials, where and respectively the numbers of datasets and sub-module categories, and is the number of trials per dataset per sub-module category;
2. For ResNets, discovered depth ("0" in Figure 3 falls at the location where accuracy saturates. This means AutoGrow discovers a near-optimal depth: a shallower depth will lose accuracy while deeper one gains little. The final accuracy of AutoGrow is good as training the discovered net from scratch as indicated by "A" in Table indicating that initialization from shallower nets does hurt training of deeper nets. As byproduct, in plain networks, there are large positive "A"s in Table 6 It implies that baselines fail to train very deep plain networks even using Batch Normalization, but AutoGrow enables the training of these networks; In Appendix A.0.3) Table I shows the accuracy improvement of plain networks by tuning K, approaching the aceuracy of ResNets with the same depth.
3. For robustness and generalization study purpose, we stick to K = 3 in our experiments, however, we can tune K to trade off accuracy and model size. As shown in Figure Auto- Grow discovers smaller DNNs when increasing K from (" "") to 50 ("O""). Interestingly, the accuracy of plain networks even increases at K 50. This implies the possibility of discovering better accuracy-depth trade-off by tuning K, although we stick to =3 for generalizability study and it generalizes well.
4. In Table 6 AutoGrow discovers different depths under different sub-modules. The final accuracy is limited by the sub-module design, not by our AutoGrow. Given a sub-module architccture, our AutoGrow can always find near-optimal depth. With a better sub-module architecture, such as NASNet cell (Zoph et al.) 2018). AutoGrow can improve accuracy.
Finally, our supposition is that: when the size of dataset is smaller, the optimal depth should be smaller. Under this supposition, we test the effectiveness of AutoGroww by sampling subset of dataset and verify AutoGrow can discover a shallower depth. In Appendix [A.0.]] Table [T] sum- marizes the results. As expected, our experiments show that AutoGrow adapts to shallower networks when the datasets are smaller.
3.4 SCALING TO IMAGENET AND EFFICIENCY
In ImageNet, K = should generalize well, but we explore AutoGrow with K = 2 and K to obtain accuracy-depti trade-off line for comparison with human experts. The larger K enables AutoGrow to obtain a smaller DNN to trade-off accuracy and model size (computation) and the smaller K = 2 achieves higher accuracy. The results are shown in Table 7 which proves that AutoGrow automatically finds a good depth without any tuning. As a byproductt the accuracy is even higher than training the found net from scratch, indicating that the Periodic Growth in AutoGrow
7

---

Under review as a conference paper at ICLR 2020
Table Scaling up ImageNet
net
K
found
Top-1
Top
IA Top-1
2
12-12-11-13
76.28
92.79
0.43
Basto4ReaNet
9-3-6-4
74.75
91.97
0.72
BottlenecklReate
2
6
77.99
93,91
0.83
5
6-7-3-9
77.33
93.65
0.83
2
6-6-6-6
71.22
90.08
0.70
Plain4Net
5
5-5-5-4
70.54
89.76
0.93
A (Top-1 of AutaGhow)
(Top-1 of training from scratch)
helps training deeper nets. The comparison of AutoGrow and manual depth design (He et al. 2016) is in Figure 4 which shows that AuloGrow achieves better trade-off between accuracy and computation (measured by floating point operations).
In Appendix A.0.3 Table I0 summarizes the breakdown of wall-clock time in AutoGrow. The growing/searching time is as efficient as (often more efficient than) fine-tuning the single discovered DNN. The scalability of AutoGrow comes from its intrinsic features that (1) grows quickly with a short period K and stops immediately if no improvement is sensed; and (2) the network is small at the beginning of growing.
maring fun crrshh
AnsoGrner
AutooCome An50
79
77
Accuncy
Basic3ResNet
BasicdResNet
76
75
24
Milliors
Millions
73
Manual (basie)
Top-1
AutoGrrow (basic)
2. S 8.8.8 8.2
71
Manual (bottleneck)
AutoGrow (bottleneck)
Accrney
PlaindNet
10
15 GFlops
Figure 4: AutoGrow vs. manual design (He) et al. (016)) on ImageNet. Marker area is pro- portional to model size determined by depth. ""bottl(bblleeeeekk" refers ResNets with basic (bottleneck) residual blocks.
Figure 3: AutoGrow vs manual search obtained by training many baselines from scratch. aris is the number of parameters. Dataset is CIFAR1O.
4 RELATED WORK
Neural Architecture Search (NAS) Zoph & Le 2016) and neural evolution (Mikkulainen et al. 2019 Angeline et al.) 1994 Stanley & Miikkulanenn 2002 Liu et al. 2017a Real et al. (217)) can search network architectures from gigantic search space. In NAS, the depth of DNNs the search space is fixed, while AutoGrow learns the depth. Some NAS methods (Bender et al. 2018 Liu al.) 2018b Cortes et al.] 2017) can find DNNs with different depths, however, the maximum depth pre-defined and shallower nets are obtained by padding zero operations or selecting shallower branches, while our AutoGrow learns the depth in an open domain to find a minimum depth, beyond which accuracy improvement can be obtained. Moreover, NAS is very computation and memory intensive. To accelerate NAS, one-shot models (Saxena & Verbeek 2016 Pham et al.) [2018] Bender al. 2018). DARTS (Liu al. (0188) and NAS with Transferable Cell (Zoph et al. 2018 Liu et al. 2018a) were proposed. The search time reduces dramatically but is still long from practical perspective. It still very challenging deploy these methods larger datasets such as ImageNet. In contrast, our AutoGrow can scale up to ImageNet thanks to its short depth learning time, which is as efficient as training single DNN.
In addition to architecture search which requires to train lots of DNNs from scratch, there are also many studies on learning neural structures within a single training. Structure pruning and growing
were proposed for different goals, such efficient inference (Wen et al. 2016 Li et al. 2016
Lebedev & Lempitsky 2016 He et al. 2017 Luo et al.. 2017 Liu et al. 2017b Dai et al. 2017 Huang et al.] 20188) Gordon et al.) 2018 Du et al. 2019). lifelong learning Yoon et al.) 2017) and model adaptation (Feng& & Darrell 2015 Philipp & Carbonell 2017). However, those works fixed
the network depth and limited structure learning within the existing layers. Optimization over DNN with fixed depth is easier as the skeleton architecture is known. AutoGrow performs in a scenario where the DNN depth is unknown hence we need to seek for the optimal depth.
8

---

Under review as a conference paper at ICLR 2020
REFERENCES
Peter J Angeline, Gregory M Saunders, and Jordan B Pollack. An evolutionary algorithm that constructs recurrent neural networks. IEEE transactions on Neural Networks, 5(1):54 65, 1994.
Gabriel Bender, Pieter-Jan Kindermans, Barret Zoph, Vijay Vasudevan, and Quoe Le. Understand- ing and simplifying one-shot architecture search. International Conference on Machine Learn- ing. pp. 549-558, 2018.
Han Cai, Tianyao Chen, Weinan Zhang, Yong Yu, and Jun Wang. Efficient architecture search by network transformation. In Thirty-Second AAAI Conference on Artificial Intelligence, 2018a.
Han Cai, Jiacheng Yang, Weinan Zhang, Song Han, and Yong Yu. Path-level network transformation for efficient architecture search. arXiv preprint arXiv: 186.:/886602639, 2018b.
Tianqi Chen, Ian Goodfellow, and Jonathon Shlens. Net2net: Accelerating learning via knowledge transfer. arXiv preprint arXiv:1511.05641 2015.
Corinna Cortes, Xavier Gonzalvo, Vitaly Kuznetsov, Mehryar Mohri, and Scott Yang. Adanet:
Adaptive structural learning artificial neural networks. Proceedings of the 34th International Conference Machine Learning- Volume 70, pp. 874-883. JMLR. org, 2017.
Xiaoliang Dai, Hongxu Yin, and Niraj Jha. Nest: neural network synthesis tool based on a grow-and-prune paradigm. arXiv preprint arXiv: 1711:1111.020000 1,0207, 2017.
Xiaocong Du, Zheng Li, and Yu Cao. Cgap: Continuous growth and pruning for efficient deep 2019.
learning, arXiv preprint
Thomas Elsken, Jan-Hendrik Metzen, and Frank Hutter. Simple and efficient architecture search for cnns. In Workshop on Meta-Learning (MetaLearn 2017) at NIPS, 2017.
Jiashi Feng and Trevor Darrell. Learning the structure of deep convolutional networks. In Proceed- ings of the IEEE international conference on computer vision, pp. 2749-2757, 2015.
Ariel Gordon, Elad Eban, Ofir Nachum, Bo Chen, Hao Wu, Tien-Ju Yang. and Edward Choi. Mor- phnet: Fast simple essource-constraigg structure learning of deep networks. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 1586-1595, 2018.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference computer vision and pailern recognition, pp. 770-778, 2016.
Yihui He, Xiangyu Zhang, and Jian Sun. Channel pruning for accelerating very deep neural net- works. Proceedings of the IEEE International Conference on Computer Vision, pp. 1389-1397, 2017.
Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Weinberger. Densely connected convolutional networks. In Proceedings of the IEFE conference on computer vision and pattern recognition, pp. 4700-4708, 2017.
Gao Huang, Shichen Liu, Laurens Van der Maaten, and Kilian Q Weinberger. Condensenct: An efficient densenet using learned group convolutions. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2752-2761, 2018
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization, arXiv preprint arXiv:1412.6980, 2014.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convo- lutional neural networks. In Advances in neural information processing systems, pp. 1097-1105, 2012.
Vadim Lebedev and Victor Lempitsky. Fast convnets using group-wiss wise brain damage. Proceed- ings of the IEFE Conference Computer Vision and Pattern Recognition, pp. 2554 2564, 2016.

---

Under review as a conference paper at ICLR 2020
Hao Li, Asim Kadav, Igor Durdanovic, Hanan Samet, and Hans Peter Graf. Pruning filters for efficient convnets. arXiv preprint arXiv:1608.08710 2016.
Hao Li, Zheng Xu, Gavin Taylor, Christoph Studer, and Tom Goldstein. Visualizing the loss land- scape of neural nets. Advances Neural Information Processing Systems, pp. 6391-6401, 2018.
Chenxi Liu, Barret Zoph, Maxim Neumann, Jonathon Shlens, Wei Hua, Li-Jia Li, Li Fei-Fei, Alan Yuille, Jonathan Huang, and Kevin Murphy. Progressive neural architecture search. In Proceed- ings of the European Conference on Computer Vision (ECCV), pp. 19-34, 2018a.
Hanxiao Liu, Karen Simonyan, Oriol Vinyals, Chrisantha Fernando, and Koray Kavukcuoglu. Hier- archical representations for efficient architecture search, arXiv preprini arXiv:1111.00436, 2017,
Hanxiao Liu, Karen Simonyan, and Yiming Yang. Darts: Differentiable architecture search. arXiv
preprint 1rXiv:1806.000 09055, 2018b.
Zhuang Liu, Jianguo Li, Zhiqiang Shen, Gao Huang, Shoumeng Yan, and Changshui Zhang. Learn- ing efficient convolutional networks through network slimming. In Proceedings of the IEEE International Conference on Computer Vision, pp. 2234,,24,,225 2017b.
Jian-Hac Luo, Jianxin Wu, and Weiyao Lin. Thinet: A filter level pruning method for deep neural network compression. In Proceedings of the IEEE international conference on computer vision, pp. 5058-5066, 2017.
Risto Miikuulainnn, Jason Liang, Elliot Meyerson, Aditya Rawal, Daniel Fink, Olivier Francon, Bala Raju, Hormoz Shahrzad, Arshak Navruzyan, Nigel Duffy, et al. Evolving deep neural networks. In Artificial Intelligence in the Age of Neural Networks and Brain Computing, pp- 293-312. Elsevier, 2019.
Hieu Pham, Melody Y Guan, Barret Zoph, Quoc V and Jeff Dean. Efficient neural architecture search via parameter sharing. arXiv preprint arXiv:1802.000 2018
George Philipp and Jaime G Carbonell.
Nonparametric neural networks. arXiv preprint
1rXiv:1712.054400. 2017.
Esteban Real, Sherry Moore, Andrew Selle, Saurabh Saxena, Yutaka Leon Suematsu, Jie Tan, Quoc V Le, and Alexey Kurakin. Large-scale evolution of image classifiers. In Proceedings of the 34th International Conference on Machine Cearning-Volume 70, pp. 2902-2911. JMLR. org, 2017.
Shreyas Saxena and Jakob Verbeck. Convolutional neural fabrics. In Advances in Neural Informa- fion Processing Systems, pp. 4053-4061 2016.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv: 1556, 2014.
Lestie N Smith, Emily M Hand, and Timothy Doster. Gradual dropin of layers train very deep neural networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recog- nition, pp. 4763-4771, 2016.
Kenneth O Stanley and Risto Mikkullanenn. Evolving neural networks through augmenting topolo- gies. Evolutionary computation, 10(2)999-277, 2002.
Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Du- mitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1-9, 2015.
Guangcong Wang, Xiaohua Xie, Jianhuang Lai, and Jiaxuan Zhuo Deep growing learning. In Proceedings of the IEEE International Conference on Computer Vision, pp. 2812-2820, 2017.
Tao Wei, Changhu Wang, Yong Rui, and Chang Wen Chen. Network morphism. In International Conference Machine Learning, pp. 564-572, 2016.
10

---

Under review as a conference paper at ICLR 2020
Tao Wei, Changhu Wang. and Chang Wen Chen. Modularized morphing of neural networks. arXiv preprint Ttt::1701.03288. 2017.
Wei Wen, Chunpeng Wu, Yandan Wang, Yiran Chen, and Hai Li. Learning structured sparsity in deep neural networks. In Advances in neural information processing systems, pp. 2074-2082, 2016.
Jaehong Yoon, Eunho Yang, Jeongtae Lee, and Sung Ju Hwang. Lifelong learning with dynamically expandable networks. arXiv preprint arXiv:1708.01547 2017.
Barret Zoph and Quae V Le. Neural architecture search with reinforcement learning, arXiv preprint arXiv: /61 1.01578, 2016.
Barret Zoph, Vijay Vasudevan, Jonathon Shlens, and Quoc Le. Learning transferable architectures for scalable image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 8697-8710, 2018.
A APPENDIX
A.0.1 OPTIMIZATION TRAJECTORIES OF NETWORK MORPHISM
We hypothesize that converged shallower net may not an adequate initialization. Figure vi- sualizes and compares the optimization trajectories of Network Morphism and the training from scratch. In this figure, the shallower net is Bas Seesecieeeee-- one is Basi BasiccRessie -eettett--555--- ization method is extended from Li et al. 2018). Points on the trajectory are evenly sampled every a few epochs. To maximize the variance of trajectory, space to a 2D space and use the first two Principle Components (PC) to form the axes in Figure 5 The contours of training loss function and the trajectory are visualized around the final minimum of the deeper net. When projecting a shallower net to a deeper net space, zeros are padded for the parameters not existing in the deeper net. We must note that the loss increase along the trajectory does not truly represent the situation in high dimensional space, as the trajectory is just a projection. is possible that the loss remains decreasing in the high dimension while it appears in an opposite way in the 2D space. The sharp detour at "Morphing" in Figure may indicate that the shallower net plausibly converges to a point that the deeper net struggles escape. In contrast, Figure b)) shows that the trajectory of the direct optimization in the deeper space smoothly converges better minimum.
(RexNet-20) and the deeper (ResNer-32)) in Table The initializer is ZeroInit. The visual-
PCA project from high dimensional
Figure ((a) visualizes the trajectory of c-AutoGrow corresponding to row "2-3-6" 6" in Table Along the trajectory, there are many trials to detour and escape an initialization from a shallower net. Figure b) visualizes the trajectory corresponding to row "2-4-3" in Table which is much smoother compared to Figure ((a). Figure (c) visualize the trajectories of p-AutoGrow with K 50 and 3. The 2D projection gives limited information to reveal the advantages of p-AutoGrow
10
ResNar-32
Morphing
Learning deca
Net-3t
-35
-10
Figure 5: An optimization trajectory comparison between (a) Network Morphism and (b) training from scratch.
11

---

Under review as a conference paper at ICLR 2020
1439
stop
2:00
2Hd PC 26.8
300
200 PC 16
PCS
Figure 6: Optimization trajectory of AutoGrow, tested by Basic3ResNed CIFARIO. (a)
c-AutoGrow with staircase learning rate and ZeroInit during growing; (b) c-AutoGrow with constant learning and GauInit during growing: p-AutoGrow with K 50; and (d) p-
AutoGrow with K = 3. For better illustration, the dots on the trajectory are plotted every 4, 20, and epochs (a-d), respectively.
1esdndret-17-17--1 17-177771777
Plain3Met- 111-1110
Plain3Met-23- 22-2222
(a) Baseline 90.45%
(a) Baseline 84.33%
(c) Baseline 88.48
(b) AutoGrow 93.13%
(b) AutoGrow 90.82%
(d) AutoGrow 94.20%
Figure 7: Loss surfaces around minima found by baselines and AutoGrow. Dataset is CIFAR1O.
comparing to c-AutoGrow in Figure b)). although the trajectory of our final p-AutoGrow in Fig- ure d) plausibly more similar the one of training from scratch in Figure (().
A.0.2 VISUALIZATION OF LOSS SURFACES AROUND MINIMA
Figure visualizes loss surfaces around minima by AutoGrow and baseline. Intuitively, AutoGrow finds wider or deeper minima with less chaotic landscapes.
A.0.3 MORE EXPERIMENTAL RESULTS
Figure plots the growing and converging curves for two DNNs Table I0
Table summarizes the adaptability of AutoGrow to the sizes of dataset. In each set of experiments, dataset randomly down-sampled to 100%. 75%, 50% and 25%. For fair comparison,
divided by the percentage of dataset such that the number of mini-batches between growths remains
Table 8: p-AutoGrow under initializers with
CIFARIO
CIFAR100
initializer
found net!
accu
initializer
found net!
accu
ZeroInit
31-30-30
ZeroInit
26-25-25
73.45
AdamInit
37-37-36
93.79
AdamInit
27-27-27
73.92
UniInit 28-28-28
93.82
UniInit 41-41-41
74.31
GauInit
42-42-42
94.27
GauInit 54-53-53
74.72
Masic3ResNet
RasiceResNet
12

---

Under review as a conference paper at ICLR 2020
Table 9: AutoGrow improves accuracy of plain nets.
dataset
net
layer #
method
accu %
Plain4Net-6-6-666
26
baseline
93.90
CIFARIO
AutoGrow
PlainANet-6-6--66
26
95.17
K = 30
baseline
95.33
Plain3Met-11-1111
baseline 90.45
CIFARIO
AutoGrow K 50
Plain3Met-11-11-2
34
93.13
Basic3ResNet--666-
36
baseline
93.60
Table 10: The efficiency of AutoGrow
net
GPUs
growing
fine-tuning
4 GTX 1080 Ti
56.7 hours
157.9 hours
BasiccRResWet---33--1
4 GTX 1080
47.9 hours
65.8 hours
BottLenncck-Reeseet-------------
4 TITAN V
45.3 hours
114.0 hours
Not=LenneckRReesNett----711
4 TITAN V
61.6 hours
78.6 hours
Plain4Net-6-6-6-1
4 GTX 1080 Ti
11.7 hours
29.7 hours
Plain4Net-5-5-5--
4 GTX 1080
25.6 hours
25.3 hours
80
a
99-3-6 --
(b)
70
Growing
50
70
6 6-6-6-6
The number of layers
Fin tuning
45
The mummbn layers
25
Top-1 Accuracy %
Growing
60
40
60
Fine-tuning
50
Accuracy
50
30
y
40
40
max accu val accu
25
15
30
layers
20
30
val val layers
20
20
10
10
5
10
5
30
60
90
120
150
180 epoch #
0
30
60
90
120 epoch
Figure
The convergence curves and growing process on ImageNet for (a) Basic4ResNet- 99-3--9-------------- and (b) Plain4Net----665 6-1-----66666 in Table
8:
10
Table 11: The adaptability of AutoGrow to dataset sizes
Basic CI
Plain MNIST
dataset size
found net
sccu
dataset size
found net
accu %
100%
42-42-42
94.27
100%
20-20-20
99.66
75%
32-31-31
93.54
75%
12-12-12
99,54
50%
17-17-17
91.34
50%
12-11-11
99.46
25%
21-12-7
88.18
25%
10-9-9
99.33
Basic4ResNet on CIFAR100
Plain4Net on SVHN
datasct size
found net
accu %
datasct size
found net
accu %
100%
17-51-16-15
79.47
100%
12-12-12-11
97.08
75%
17-17-16-16
77.26
75%
9-9-9-9
96.71
50%
12-12-12-11
72.91
50%
8-8-8-8
96.37
25%
6-6-6-6
62.53
25%
5-5-5-5
95.68
13

---

Under review as a conference paper at ICLR 2020
the same. As expected, our experiments show that AutoGrow adapts to shallower networks when the sizes are smaller.
14


Reviews:
[{"review_id": "r1emxj7GKH", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Contributions:\n\tThis paper best fits in the literature that explores growing network depth.  The main framework here is to interleave training a shallower network and adding new layers.  This paper (their final algorithm) differs from existing methods in that they: 1) initialize the new layers using standard initialization as opposed to the commonly used zero-init in this literature, 2) grows at a fixed interval , and this interval is short (to avoid the shallower nets being  overly-trained)., 3) uses a large and constant learning rate during the growing phase.  \nEmpirically, they show competitive results on standard image benchmarks.  \nMore interestingly (to me), they provide interesting insights to this paradigm of ‘growing networks’.  \n\nComments/Questions:\nSection 2 of the paper describes the proposed method is good details.  \n\nSection 3 of the paper describes the experiments.  Since for now I see the contribution of this paper is mostly empirical, I will give my detailed feedback here.\n3.1 (Suboptimum of Network Morphism (NM) )\nTable 2 shows NM is worse than training from scratch, and this isn’t fixed by AdamInit.\nTable 3 shows c-AutoGrow (in between p-AutoGrow and NM) still does worse than from scratch, pinpoint the problem to converged subnetworks.\n3.2  (p-AutoGrow)\nTable 3 shows +Constant LR helps, then +RandomInit helps. \nTable 4, 5 shows +Periodic gets the best performance.  \n*Suggestion* The found net is Table 4,5  are significantly deeper than those in Table 2,3, also there are no \\Delta.  Also, although within this write-up those are the highest numbers, in the broader literature of NAS this doesn’t seem to be that good.  From a quick search, many methods in the Table 1 of [1] seems to give >96% accuracy on CIFAR10, some even close to 98%.  It might be good to at least discuss why this method is limited from achieving that.\nI do like the finding that ZeroInit is unnecessary, as reported in the rest of this subsection.  However, it is unsatisfying to me that many past works (as cited by the authors) required this ZeroInit without ever trying RandomInit.  \n*Suggestion* I would love to see a more thorough discussion on why GauInit is better than ZeroInit, not just more numbers.  For example, even just text description on why past works found ZeroInit useful, and countering some of those claims would be interesting.  A more controlled experiment rather than training 2 networks by swapping this would be interesting.  ZeroInit is used in more context than just NM.  For example, good flow models like Glow also uses such initialization, for likely a different reason, but I wonder if findings here have any implication for ZeroInit more generally.\n\n3.3 (Many datasets)\nTable 6 is a strong result.  One odd thing is how deep the found-net has to be for MNIST.  This actually suggest to me that AutoGrow does not have the ability to stop early when it can.  And in the discussion, the authors argue that by using a better sub-module like in NAS they can do better.  This raises the question why the authors did not choose to use it.  I would believe it if the proposed method has obvious reasons that it can transfer to different architecture, but for now I cannot jump to the conclusion that, say, p-AutoGrow with GauInit will necessarily work when using a different sub-module.  Perhaps, the reason past NM works didn’t use a GauInit was also due to the fact that past sub-modules didn’t work with GauInit.  \n\n3.4 (Scale to ImageNet) It’d be good to add reference results from other papers.  \n\nMinor details:\n\n\nThere are some good contents in this work, but for it to be a strong *empirical* contribution, perhaps it would be more useful to include experiments on other data modality where things are not so well tuned, and show state-of-the-art results.  For it to be a strong *analysis* paper, it should expanded, at least addressing some of the *suggestions* mentioned above. \nUnrelated to my evaluation of this work, reading this makes me think we should (and can) develop theoretical understanding to this paradigm of growing networks.\n\n\n\nReferences:\n[1] https://arxiv.org/pdf/1905.13360.pdf\n\n", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}, {"review_id": "SkeAVHaRYS", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper's contribution is a method for automatically growing the depth of a neural network during training. It compares several heuristics that may be used to successfully achieve this goal and identifies a set of choices that work well together on multiple datasets.\n\nThe paper focuses on CNNs that conform to a popular design pattern where the network is organized into a series of sub-networks, each consisting of a series of sub-modules (sometimes called blocks) operating at the same resolution. To be precise, the proposed method aims to learn the length of each series of sub-modules. A main contribution of the paper is the demonstration that it is not necessary to train a network until convergence before adding new sub-modules as proposed in past work. Instead, it is better to grow the network after training for a short while.\n\nMy current decision for this paper is a weak rejection due to the points below. However, I am open to revising my opinion if these points are addressed satisfactorily.\n\n- The growing strategy identified in the paper as a superior alternative seems to be already known and used, at least in the speech recognition community. Seide et al. (2011) called it Discriminative Pre-training, and showed that it outperforms greedy layer-wise pretraining and DBN pre-training. Zeyer et al. (2017) reported that a similar method also enables the training of very deep LSTM networks which is otherwise notoriously hard. In general, the existence of prior work with the same ideas does not preclude acceptance, but the existence of this work needs to be clearly stated early on and the additional value of the current study sufficiently clarified.\n\n- I find it strange that the final networks found by the proposed method usually have the same/similar number of sub-modules per sub-network (Tables 4,5,6) on multiple datasets. The only exceptions appear to be Basic4ResNet/CIFAR100 in Table 6 and about 50% of ImageNet results in Table 7. This regularity suggests that either A) the proposed algorithm prefers to set same number of sub-modules per sub-network due to its design, or B) datasets except ImageNet have an inherent shared property that produces this result. Since option A suggests a bias in the algorithm, this peculiarity of the results needs to be investigated or explained further.\n\n- Figure 3 constitutes the main evidence that Autogrow finds approximately optimal depths as compared to manual searching, but it is not clear how the plot for baselines is obtained. For any given parameter budget, there are multiple baseline networks possible since the sub-networks can have different number of sub-modules (see previous point). This does not appear to be accounted for in Figure 3. Further, when dealing with CNNs, it would be more useful to have computation budget on the x-axis instead of the parameter budget. This would better account for the difference between increasing depth in an earlier sub-network vs. a later one.\n\n- The reported results appear to be for single trials throughout the paper. This does not seem sufficient especially for results in Tables 2 and 3 where many differences are rather small, and so drawing conclusions from these tables would be unscientific.\n\nReferences:\n\nSeide, Frank, et al. \"Feature engineering in context-dependent deep neural networks for conversational speech transcription.\" 2011 IEEE Workshop on Automatic Speech Recognition & Understanding. IEEE, 2011. https://www.microsoft.com/en-us/research/wp-content/uploads/2016/02/FeatureEngineeringInCD-DNN-ASRU2011-pub.pdf\n\nZeyer, Albert, et al. \"A comprehensive study of deep bidirectional LSTM RNNs for acoustic modeling in speech recognition.\" 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2017. https://arxiv.org/abs/1606.06871\n\nUpdate after rebuttal\n-----------------------------\nI'm sympathetic to the unfortunate situation that the authors are in, since the underlying growing strategy has already been covered by prior work. As I mentioned earlier, a sufficient rewrite of the paper can clearly state what has been done already so as not to take credit from the earlier authors. A revised version of the paper has not been uploaded; I suggest that the authors do so for the future. \n\nI agree that the focus of this paper is learning the 'optimal' depth by using the growing strategy. But I am not convinced that the technique indeed finds optimal depths based on the regularity of the sub-network depths mentioned in my review. The rebuttal suggests reasons for the obtained regularity, but does not prove that these regular structures obtained are indeed optimal and not an artifact of the algorithm itself. The baselines are also using the same regular architectures, which distorts the overall picture because it is possible that a non-regular architecture provides a better trade-off.\n\nWhile my rating doesn't change, I do think that the work is in an interesting direction. My final suggestions for the future are:\n- Investigate where non-regular architectures (unequal sub-network depths) are in the trade-off between accuracy, flops and parameters.\n- Investigate whether the proposed algorithm can be modified to easily find non-regular architectures if they can yield equally good performance as regular ones at similar or lower cost.\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "HkgRdsvQqr", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper presents a meta-learning algorithm to automatically detemine the depth of neural network through a policy to add depth if this bring improvement on accuracy.\n\nI have conserved opinion based on the technique being used here is extremely simple, basically is an implementation of naive greedy algorithm in such a scenario, which implies the problem may not be intrinsically hard, or even useful. The paper consists of detailed narrative about how these procedure are conducted, but still, it is really hard for me to find the true merit to appreciate, and why this brings a nontrivial and usefull contribution. The tables, visualization figures also didnot imply too much about whether this is more than overfitting on previous works with hand-chosen depth. ", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a neural architecture search method based on greedily adding layers with random initializations. The reviewers all recommend rejection due to various concerns about the significance of the contribution, novelty, and experimental design. They checked the author response and maintained their ratings.

Decision: Reject