Growing Efficient Deep Networks by Structured Continuous Sparsification
Authors: Xin Yuan
Pedro Henrique Pamplona Savarese
Michael Maire
Venue: ICLR 2021 Oral (2021)
Keywords: deep learning
computer vision
network pruning
neural architecture search

Abstract: We develop an approach to growing deep network architectures over the course of training, driven by a principled combination of accuracy and sparsity objectives.  Unlike existing pruning or architecture search techniques that operate on full-sized models or supernet architectures, our method can start from a small, simple seed architecture and dynamically grow and prune both layers and filters.  By combining a continuous relaxation of discrete network structure optimization with a scheme for sampling sparse subnetworks, we produce compact, pruned networks, while also drastically reducing the computational expense of training.  For example, we achieve $49.7\%$ inference FLOPs and $47.4\%$ training FLOPs savings compared to a baseline ResNet-50 on ImageNet, while maintaining $75.2\%$ top-1 validation accuracy --- all without any dedicated fine-tuning stage.  Experiments across CIFAR, ImageNet, PASCAL VOC, and Penn Treebank, with convolutional networks for image classification and semantic segmentation, and recurrent networks for language modeling, demonstrate that we both train faster and produce more efficient networks than competing architecture pruning or search methods.

Published as a conference paper at ICLR 2021
GROWING EFFICIENT DEEP NETWORKS BY STRUCTURED CONTINUOUS SPARSIFICATION
Xin Yuan University of Chicago yuanx@uchi cag.o edu
Pedro Savarese TTI-Chicago S ravaresettic.ee
Michael Maire
University of Chicago
mma reêuch! wwe@uchicago...... cago. edu
ABSTRACT
We develop an approach to growing deep network architectures over the course of training, driven by principled combination of accuracy and sparsity objectives. Unlike existing pruning or architecture search techniques that operate on full-sized models supernet architectures, our method can start from small, simple seed architecture and dynamically grow and prune both layers and filters. By combining continuous relaxation of discrete network structure optimization with scheme for sampling sparse subnetworks, we produce compact, pruned networks, while also drastically reducing the computational expense of training. For example, we achieve 49.7% inference FLOPs and 47.4% training FLOPs savings compared to a baseline ResNet-50 ImageNet, while maintaining 75.2% top-1 accuracy - all without any dedicated fine-tuning stage. Experiments across CIFAR, ImageNet, PASCAL VOC, and Penn Treebank, with convolutional networks for image classi- fication and semantic segmentation, and recurrent networks for language modeling, demonstrate that we both train faster and produce more efficient networks than competing architecture pruning search methods.
INTRODUCTION
Deep neural networks are the dominant approach to a variety of machine learning tasks, including im- age classification (Krizhevsky al., 2012: Simonyan Zisserman, 2015), object detection (Girshick, 2015; Liu et al., 2016), semantic segmentation (Long et al., 2015; Chen et al., 2017) and language modeling (Zaremba et al., 2014; Vaswani al., 2017; Devlin et al., 2019). Modern neural networks are overparameterized and training larger networks usually yields improved generalization accuracy. Recent work (He et al., 2016; Zagoruyko & Komodakis, 2016; Huang et al., 2017) illustrates this trend through increasing depth and width of convolutional neural networks (CNNs). Yet, training is compute-intensive, intensive, and real-world deployments are often limited by parameter and compute budgets.
Neural architecture search (NAS) (Zoph & Le, 2017; Liu al., 2019; Luo al., 2018; Pham et al., 2018; Savarese Maire, 2019) and model pruning (Han et al., 2016; 2015: Guo et al., 2016) methods aim reduce these burdens. NAS addresses an issue that further compounds training cost: the enormous space of possible network architectures. While hand-tuning architectural details, such as the connection structure convolutional layers, can improve performance (landola al., 2016; Sifre & Mallat, 2014; Chollet, 2017; Howard et al., 2017: Zhang et al., 2018; Huang et al., 2018), principled way of deriving such designs remains elusive. NAS methods aim automate exploration of possible architectures, producing an efficient design for a target task under practical resource constraints. However, during training, most NAS methods operate on a large superner architecture, which encompasses candidate components beyond those that are eventually selected for inclusion in the resulting network (Zoph Le, 2017: Liu et al., 2019; Luo et al., 2018; Pham et al., 2018; Savarese & Maire, 2019). Consequently, NAS-based training may typically be more thorough, but more computationally expensive, than training a single hand-designed architecture.
Model pruning techniques similarly focus on improving the resource efficiency of neural networks during inference, at the possible expense of increased training cost. Common strategies aim to generate lighter version of given network architecture by removing individual weights (Han et al., 2015; 2016; Molchanov et al., 2017) or structured parameter sets (Li et al., 2017; He et al., 2018; Luo al., 2017). However, the majority of these methods train full-sized model prior to pruning and,

---

Published as a conference paper at ICLR 2021
Frudtng
Channal-sato
Laver meoreedc
PLOPE
Training Epochs T=0,1,2...
Training Epochs
NAS
Pruning
Ours
(a) Growing CNN Layers and Filters
(b) Epoch-wise Training Cost
Total Training Cost
Figure I: Growing Networks during Training. We define an architecture contiguration space and simultane- ously adapt network structure and weights. (a) Applying our approach to CNNs, we maintain auxiliary variables that determine how to grow and prune both filters (f.e. channel-wise) and layers, subject to practical resource constraints. (b) By starting with small network and growing its size, utilize fewer resources in carly training epochs, compared to pruning or NAS methods. (c) Consequently, our method significantly reduces total computational cost of training. while delivering trained networks of comparable or better size and accuracy.
after pruning, utilize additional fine-tuning phases in order to maintain accuracy. Hubara et al. (2016) and Rastegari et (2016) propose the use of binary weights and activations, allowing inference to benefit from reduced storage costs and efficient computation through bit-counting operations. Yet, training still involves tracking high-precision weights alongside lowerpprecsiion approximations.
We take a unified view of pruning and architecture search, regarding both as acting on configuration space, and propose a method to dynamically grow deep networks by continuously reconfiguring their architecture during training. Our approach not only produces models with efficient inference characteristics, but also reduces the computational cost of training; see Figure I. Rather than starting with full-sized network supernet, start from simple seed networks and progressively adjust (grown and prune) them. Specifically, we parameterize an architectural configuration space with indicator variables governing addition removal of structural components. Figure 2(a) shows an example, in the form of a two-level configuration space for CNN layers and filters. We enable learning of indicator values (and thereby, architectural structure) via combining a continuous relaxation with binary sampling, as illustrated in Figure 2(b). A per-component temperature parameter ensures that long-lived structures are eventually baked into the network's discrete architectural configuration.
While the recently proposed AutoGrow (Wen et al., 2020) also seeks to grow networks over the course of training. our technical approach differs substantially and leads to significant practical advantages. At technical level, AutoGrow implements an architecture search procedure over predefined modular structure, subject to hand-crafted, accuracy-driven growing and stopping policies. In contrast, we parameterize architectural configurations and utilize stochastic gradient descent to learn the auxiliary variables that specify structural components, while simultancously training the weights within those components. Our unique technical approach yields the following advantages:
Fast Training by Growing:
Training is unified procedure, from which one request network structure and associated weights at any time. Unlike AutoGrow and the majority of pruning techniques, fine-tuning to optimize weights in a discovered architecture is optional. We achieve excellent results even without any fine-tuning stage.
Principled Approach via Learning by Continuation + Sampling: We formulate our approach in the spirit of learning by continuation methods, which relax a discrete optimization problem to an increasingly stiff continuous approximation. Critically, we introduce an additional sampling step to this strategy. From this combination, we gain the flexibility of exploring a supernet architecture, but the computational efficiency of only actually training a much smaller active subnetwork.
Budget-Awe Aware Optimization Objectives: The parameters governing our architectural configu- ration are themselves updated via gradient decent. We have flexibility to formulate a variety of resource-sensitive losses, such as counting total FLOPSs, in terms of these parameters.
Broad Applicability: Though we use progressive growth of CNNs width and depth as a motivating example, our technique applies to virtually any neural architecture. One has flexibility in how to parameterize the architecture configuration space. We also show results with LSTMs.
We demonstrate these advantages while comparing to recent NAS and pruming methods through extensive experiments classification, semantic segmentation, and word-level language modeling.
2

---

Published as a conference paper at ICLR 2021
Level 1: Channel-wise Configuration Space
Binary Indicator
Channel Indicator
Channel Indicator
Channel Indicator
Kig
Discrete Configuration Space
T=0
Tw1
Level 2: Layer-wise Configuration Space Layer Indicator
Layer Indicator
Layer Indicator
mank
Trainable Variable 5
Sampling
(Bs) q
Progressive Continuous Relaxation
Xie
Bern(a(fs))
Active Layers 1
Activve
Prttah
(a) Architectural Configuration Space for CNNS
(b) Optimization with Structured Continuation
Figure 2: Technical Framework. (a) We periodically restructure a CNN by querying binary indicators that define two-level configuration space for filters and layers. (b) To make optimization feasible while growing networks, we derive these binary indicators from trainable continuous mask variables. We employ structured extension of continuous sparsification (Savarese et al., 2020), combined with sampling. Binary stochastic auxiliary variables sampled according to a (Bs), generate the discrete components active at a particular time.
2 RELATED WORK
Network Pruning. Pruning methods can be split into two groups: those pruning individual weights and those pruning structured components. Individual weight-based pruning methods vary on the removal criteria. For example, Han et al. (2015) propose to prune network weights with small magnitude, and subsequently quantize those remaining (Han et al., 2016). Louizos et al. (2018) learn sparse networks by approximating fo-regularization with stochastic reparameterization. However, sparse weights alone often only lead speedups dedicated hardware with supporting libraries.
In structured methods, pruning is applied at the level of neurons, channels, or even layers. For example, Ll-pruning (Li al., 2017) removes channels based on the norm of their filters. He et al. (2018) use group sparsity to smooth the pruning process after training. MorphNet (Gordon et al., 2018) regularizes weights towards zero until they are small enough such that the corresponding output channels are marked for removal from the network. Intrinsic Structured Sparsity (ISS) (Wen et al., 2018) works on LSTMs (Hochreiter & Schmidhuber, 1997) by collectively removing the columns and rows of the weight matrices via group LASSO. Although structured pruning methods and our algorithm share the same spirit of generating efficient models, we gain training cost savings by growing networks from small initial architectures instead of pruning full-sized ones.
Neural Architecture Search. NAS methods have greatly improved the performance achieved by small network models. Pioneering NAS approaches use reinforcement learning (Zoph et al., 2018; Zoph& & Le, 2017) and genetic algorithms (Real et al., 2019; Xie & Yuille, 2017) to search for transferable network blocks whose performance surpasses many manually designed ones. However, such approaches require massive computation during the search - typically thousands of GPU days. To reduce computational cost, recent efforts utilize more efficient search techniques, such as direct gradient-basse based optimization (Liu et al., 2019; Luo et al., 2018: Pham et al., 2018; Tan et al., 2019; Cai al., 2019; Wortsman al., 2019). Nevertheless, most NAS methods perform search in a supernet space which requires more computation than training typically-sizer architectures.
Network Growing. Network Morphism (Wei et al., 2016) searches for efficient deep networks by extending layers while preserving the parameters. Recently proposed Autogrow (Wen et al., 2020) takes an AutoML approach to growing layers. These methods either require specially-crafted policy to stop growth (e.g,, after a fixed number of layers) rely on evaluating accuracy during training, incurring significant additional computational cost.
Learning by Continuation. Continuation methods are commonly used to approximate intractable optimization problems by gradually increasing the difficulty of the underlying objective, for example by adopting gradual relaxations to binary problems. Wu et al. (2019); Xie et al. (2019b; 2020) use gumbel- gumbel-softmax (Jang et al., 2017) to back-propagaa errors during architecture search and spatial feature sparsification. Savarese et al. (2020) propose continuous sparsification to speed up pruning and ticket search (Frankle Carbin, 2019). Despite the success of continuation methods in producing sparse networks upon the completion of training, they do not operate on sparse networks
3

---

Published as a conference paper at ICLR 2021
during training and instead work with real-valued relaxation. Postponing actual elimination of near zeroed-out components prevents naive application of these methods from reducing training costs.
3 METHOD
3.1 ARCHITECTURAL CONFIGURATION SPACE
A network topology can be seen a directed acyclic graph consisting of an ordered sequence of nodes. Each node (+) is an input feature and each edge is a computation cell with structured hyperparameters (e.g., filter and layer numbers in convolutional networks). An architectural configuration space be parameterized by associating a mask variable m E {0, 1} with each computation cell (edge), which enables training-time time pruning (m + 0) and growing (m 0 + 1) dynamics.
As running example, we consider a two-level configuration space for CNN architectures, depicted in Figure 2(a), that enables dynamically growing networks in both width (channel-wise and depth (layer-wise). Alternative configuration spaces are possible; we defer to the Appendix details on how we parameterize the design of LSTM architectures.
CNN Channel Configuration Space: For a convolutional layer with 4n input channels, lowe output channels (filters) and k x k sized kernels, the i-th output feature computed based the i-th filter, i.e. for E lout]:
out f fa F() mm (1)
where m. (4) {0, 1) is binary parameter that removes the i-th output channel when set zero and denotes the convolutional operation. m is shared across filter and broadcasts the same shape as the filter tensor F()), enabling growing/pruning of the entire filter. As Figure 2(a) (top) shows, we start from a slim channel configuration. We then query the indicator variables and perform state transitions: (1) When flipping an indicator variable from to 1 for the first time, we grow a randomly initialized filter and concatenate it to the network. (2) If an indicator flips from 1 to O, we temporarily detach the corresponding filter from the computational graph; it will be grown back to the its original position if its indicator flips back to 1, or otherwise be permanently pruned at the end of training. (3) For other cases, the corresponding filters either survive and continue training or remain detached pending the next query to their indicators. Our method automates architecture evolution, provided can train the indicators.
CNN Layer Configuration Space: To grow network depth, design a layer configuration space in which an initial shallow network will progressively expand into a deep trained model, as shown in Figure 2(a) (bottom). Similar channel configuration space, where filters serve basic structural units, we require a unified formulation to support the growing of popular networks with shortcut connections (e.g., ResNets) and without (e.g.. VGG-like plain nets). We first introduce an abstract layer class flayer as a basic structural unit, which operates input features Iin and generates output features Tout- flayer can be instantiated as convolutional layers for plain nets or residual blocks for ResNets, respectively. We define the layer configuration space as:
Dout g(Dini Slayer m (1) = flayer (x)));
if m = 1 if m (5)
Zins
=
where mi (1) E 1) is the binary indicator for j-th layer ftayers with which we perform state transitions analogous to the channel configuration space. Layer indicators have priority over channel indicators: if m²(") is set as 0, all filters contained in the corresponding layer will be detached, regardless of the state their indicators. We do not detach layers that perform changes in resolution (e.g., strided convolution).
3.2 GROWING WITH STRUCTURED CONTINUOUS SPARSIFICATION
We optimize trade-off between accuracy and structured sparsity by considering the objective:
min
LE(g( w @ ma); flayer mi)) + X1 ||mallo + >2 ||mellaa
(3)
W.M.eisaayer
4

---

Published as a conference paper at ICLR 2021
where f is the operation in Eq. (1) or Eq. (9) (in Appendix A.6), while g is defined in Eq. (2). w m and flayer my are general expressions of structured sparsified filters and layers and Lg denotes a loss function (e.g., cross-entropy loss for classification). The &o terms encourage spansity, while X1,2 are trade-off parameters between LE and the 4 penalties.
Budget-aware Growing. In practice, utilizing Eq. (3) might require grid search on Å and X2 until network with desired sparsity produced. To avoid such a costly procedure, we propose budget-aware growing process, guided by target budget in terms of model parameters or FL.OPs. Instead of treating λ and A2 as con- stants, we periodically update them as:
Algorithm 1 Optimization
Input: Data X (z:;)==>> labels Y (y: )=
Output: Grown efficient model G
Initialize: G, w. u, Abuse and A
Set ts all 0 vectors associating functions.
for epoch = 1 to do
Evaluate G's s sparsity ug and calculate
Au = u - uG
A1 f base Au, A2 + X
hase Au,
(4)
Update A < Abase Au; X2
Ahuse Au
in Eq. (6) using Eq. (4)
where Au calculated as the target sparsity u minus current network sparsity uG. and Afuse, A are initial base constants. early growing stages, since the network is too sparse and Au is negative, the optimizer will drive the network to- wards state with more capacity (wider/deeper). The regularization effect gradually weakens as the network's 's sparsity approaches the budget (and Au approaches zero). This allows us to adaptively grow the network and automatically adjust its sparsity level while simultaneously training model weights. Appendix A.I provides more detailed analysis. Our experiments default to defining budget by parameter count, but also investigate alternative notions of budget.
for r = 1 to I do
Sample mini- 2;. Vi from X, Y Train G using Eq. (6) with SGD
end for
Sample indicators qu ~ Bern(a((s...))
and record the index idæ where q value is 1. Update t, [idx] = to [idx] + 1
Update ,3 using Eq. (7)
end for return G
Learning by Continuation. Another issue in optimizing Eq. (3) is that ||mello and || make the problem computationally intractable due to the combinatorial nature of binary states. To make the configuration space continuous and the optimization feasible, we borrow the concept of learning by continuation (Cao et al., 2017; Wu et al., 2019; Savarese et al., 2020; Xie et al., 2020). We reparameterize m as the binary sign of a continuous variable s: sign(s) is if and if O. We rewrite the objective in Eq. (3) as:
LE ( w O sign(s.)): flayer - sign(ss))) + A1 [|sign(s>)]] + A (iigg(ss)ls (5)
min w.e..140.ftaeer
We attack the hard and discontinuous optimization problem in Eq. (5) by starting with an easier objective which becomes harder as training proceeds. We use sequence of functions whose
limit is the sign operation: for any # O, lim (BB) = sign(s) if a is sigmoid function or
(Jang et al., 2017), where B > 0 is a temperature parameter and go. is gumbel. By periodically changing 3, (B3)) becomes harder to optimize, while the objectives converges to original discrete one.
eap(( (o) lim a (Bs) = sign(s) if a is gumbel- -ootmax E,e (o.1) curp(( (xy
Maintaining Any-time Sparsification. Although continuation methods can make the optimization feasible, they only conduct sparsification via a thresholding criterion in the inference phase. In this case, the train-time architecture is dense and not appropriate in the context of growing network. To effectively reduce computational cost of training, we maintain a sparse architecture by introducing an 0-1 sampled auxiliary variable g based on the probability value a (3s). Our final objective becomes:
min J10.
LE (f( w C (B8.) @ qe)s ftayer (B8i) 4:) + A (lo()s)) + A2 [|a(3))) (6)
where qe and qi are random variables sampled from (ern(o(Bs)) and (ern(a(Ss:)), which effec- tively maintains any-time sparsification and avoids sub-optimal thresholding, shown in Figure 2(b).
Improved Temperature Scheduler. In existing continuation methods, the initial 3 value is usually set as 8 1 and scheduler is used at the end of each training epoch to update 33 in all activation
functions a, typically following 3 Sg 7², where t is current epoch and y is a hyperparameter (> 1 when is the sigmoid function, < 1 when is gumbel softmax). Both y and control the speed
5

---

Published as a conference paper at ICLR 2021
Table Comparison with the channel pruning methods LI-Pruning (Li et al., 2017). SoftNet (He et al., 2018),
ThiNet (Luo et al., 2017), Provable (Liebenwein et al., 2020) and BAR (Lemaire et al., 2019) CIFAR-10.
Model
Method
Val Acc(%)
Params(M)
FLOPs(%)
Train-Cost Savings(X) (X)
Original
92.9 (-0.0)
14.99 (100%)
100
1.0x
L.I-Prnnig
0.12 (-1.1)
2.98 (19.9%)
19.9
2.5x
VGG -16
SoftNet
(-0.8)
5,40 (36.0%)
36.1
1.6x
ThiNet
90. 0.11 (-2.1)
5,40 (36.0%)
36,1
1.6x
Provable
0.12 0-12
0.85 (5.7%)
15.0
3.5x
Ours
92.50 0.10
0.754 + 0.005 (5.0%)
13.55 + 0.03
4.95 + 0.17
Original
91.3 0.12 (-0.0)
0.27 (100%)
100
1.0x
LI-Pruning
90.9 0.10 0.10 (-0.4)
0,15 (55.6%)
55.4
1.1X
ResNet -20
SoftNet
90.8 (-0.5)
0.14 (53.6%)
50.6
1.2x
ThiNet
8 (-2.)
0.18 (67.1%)
67.3
1.1x
Provable
90.8 0.08 (-0.5)
0.10 (37.3%)
54.5
17x
Ours
90.91 0.07 (-0.4)
0.096 0.002 (35.8%)
50.20
0.01
2.40 + 0.09 x
Original
96.2 0.10 (-0.)
36.5 (100%)
100
1.0x
WRN
LI-Pruning
95.2 :: 0.10 (-1.0)
7.6 (20.8%)
49.5
1.5x
-28
BAR(I6x V)
92.0 0.08 (-4.2)
2.3 (6.3 %)
1.5
2.6x
-10
Ours
95.32 = 0.11 (-0.9) 3.443 + 0.010 (9.3%) 28.25 + 0.04
3.12 = 0.11x
at which the temperature increases during training. Continuation methods with global temperature schedulers have been successfully applied in pruning and NAS. However, in our case, global schedule leads to unbalanced dynamics between variables with low and high sampling probabilities: increasing the temperature of those less sampled at early stages may hinder their training altogether, as towards the end of training the optimization difficulty is higher. To overcome this issue, we propose a separate, structure-war wise temperature scheduler by making a simple modification: for each mask variable, instead of using the current epoch number / to compute its temperature, we set separate counter t, which increased only when its associated indicator variable sampled as in Eq. (6). We define our structure-wii wise temperature scheduler as
G = 30 7
(7)
where i, are vectors associated with the functions. Experiments use this separate scheduler by default, but also compare the two alternatives. Algorithm summarizes our optimization procedure.
EXPERIMENTS
We evaluate our method against existing channel pruning, network growing, and neural architecture search (NAS) methods on: CIFAR-10 (Krizhevsky et al., 2014) and ImageNet (Deng et al., 2009) for image classification, PASCAL (Everingham et al., 2015) for semantic segmentation and the Penn Treebank (PTB) (Marcus et al., 1993) for language modeling. See dataset details in Appendix A.2. In tables, best results are highlighted in bold and second best are underlined.
4.1 COMPARING WITH CHANNEL PRUNING METHODS
Implementation Details. For fair comparison, we only grow filters while keeping other structured parameters of the network (number of layers/blocks) the same unpruned baseline models. Our method involves two kinds of trainable variables: model weights and mask weights. For model weights, we adopt the same hyperparameters used to train the corresponding unpruned baseline models, except for setting the dropout keep probability for language modeling to 0.65. We initialize mask weights such that single filter is activated in each layer. We train with SGD, initial learning rate of 0. weight decay of 10 and momentum 0.9. Trade-off parameter X set to 0.5 on all tasks; A2 is not used since we do not perform layer growing here. We set the sigmoid function and 7 as 1 where 7' is the total number of epochs.
VGG-16, ResNet-20, and WideRessNet228-10 on CIFAR-10. Table 1 summarizes the models produced by our method and competing channel pruning approaches. Note that training cost is calculated based on overall FLOPs during pruning and growing stages. Our method produces sparser networks with less accuracy degradation, and, consistently saves more computation during training consequence of growing from simple network. For a aggressively pruned WideResNet-255 10, we observe that BAR (Lemaire et al., 2019) might not have enough capacity to achieve negligible accuracy drop, even with knowledge distillation (Hinton al., 2015) during training. Note that

---

Published as a conference paper at ICLR 2021
Table 2: Comparison with channel pruming methods: LI-Pruning (Li et al., 2017), SoftNet (He et al., 2018) and Provable (Liebenwein et al., 2020) on ImageNet.
Model
Method
Top-1 (c)(%)
Params(M)
FLOPs(%%
Train-Cost Savings( x)
Original
76.1 (-0.0)
23.0 (100%)
100
1.0())
LI-Pruning
74.7 (-1.4)
19.6 (85.2%)
77.5
1.1())
ResiNet
SoftNct
74.6 (-1.5)
N/A
58.2
1.2(x)
-50
Provable
75.2 (-0.9)
15.2 (65.9%)
70.0
1.2())
Ours
75.2 (-0.9)
14.1 (61.2%)
1.9())
Table 3: Results comparing with AutoGrow (Wen et al., 2020) CIFAR-I0 and ImageNet.
Dataset
Methods
Variants
Found Net
Val Acc(%)
Depth
Sparse Channel
Besic
23-29-31
94.50
83
Ours
Plain3Ner
11-14-19
90.99
CIFAR-10
Besic3ResNet
42-42-42
94.27
126
X
AutoGrow
Plain3Net
23-22-22
90.82
67
x
Bottleneck
5-6-5-7
77.41
23
Ours
Plain4Net
3-4-4-5
70.79
ImageNet
RottleneckRReseet
6-7-3.9
77.33
X
AutoGrow
Plain4Net
5-5-5-4
70.54
19
X
report our method's performance as mean + standard deviation, computed over 5 runs with different random seeds. The small observed variance shows that our method performs consistently across runs. ResNet-50 and MobileNettV 1 on ImageNet. To validate effectiveness on large-scale datasets, we grow, from scratch, filters of the widely used ResNet-50 on ImageNet; we do not fine-tune. Table shows our results best those directly reported in papers of respective competing methods. Our approach achieves 19.7% inference and 47.4% training cost savings in terms of FLOPs while main- taining 75.2% top-1 accuracy, without any fine-tuning stage. Appendix A.4 shows improvements on the challenging task of growing channels of an already compact MobileNetV1. In addition, Figure shows the top-1 accuracy/FIOPs trade-offs for MobileNetV1 on ImageNet, demonstrating that our method dominates competing approaches.
Deeplab-v3-
on PASCAL VOC. Appendix A.5 provides semantic segmentation results.
We detail the extensions to recurrent cells and compare proposed method with ISS based on vanilla two-layer stacked LSTM in Appendix A.6. As shown in Table 8, method finds more compact model structure with lower training cost, while achieving similar perplexity on both validation and test sets.
3-Stacked-LSTMA on PTB:
4.2 COMPARING WITH AUTOGROW
Implementation Details. We grow both filters and layers. We follow AutoGrow's settings in exploring architectural variations that define our initial seed network. layer-wise configuration space and basic structural units Stayer: Basic3ResNet, Bottleneck4ResNer, Plain3Net, Plain4Ner. Different from the initialization of AutoGrow using full-sized filters in each layer, our channel-wise configuration space starts from single filter and expands simultancously with layers. Appendix A.7 contains a detailed comparison of seed architectures. For training model weights, we adopt the hyperparameters of the ResNet or VGG models corresponding initial seed variants. For layer-wise and channel-wiss mask variables, we initialize the weights such that only single filter in each layer and one basic unit in each stage (e.g., Basic Resic3ResNert) active. We use SGD training with initial learning rate of 0.1, weight decay of 10-6 and momentum of 0.9 on all datasets. The learning rate scheduler is the same as for the corresponding model weights. Trade-off parameters base and Xhase set 1.0 and 0.1 all datasets. For fair comparison, fine-tune final models with 40 epochs and 20 epochs on CIFAR-10 and ImageNet, respectively.
Results on CIFAR-10 and ImageNet. Table 3 compares our results to those AutoGrow. For all layer-wise growing variants across both datasets, our method consistently yields a better depth and width configuration than AutoGrow, in terms of accuracy and training/inference costs trade-off. Regarding the training time of Bottleneck4Res.er on ImageNet, AutoGroow requires 61.6 hours for the growing phase and 78.6 hours for fine-tuning on TITAN V GPUs, while our method takes 48.2 and 31.3 hours, respectively. Our method offers 43% more train-time savings than AutoGrow. We not only require fewer training epochs, but also grow from a single filter to a relatively sparse network, while AutoGrow always keeps full-sized filter sets without any reallocation.
7

---

Published as a conference paper at ICLR 2021
4.3 COMPARING WITH NAS METHODS
As fair comparison with NAS methods involving search and re-training phases, we also divide our method into growing and training phases. Specifically, we grow layers and channels from the Bot- eleneck4ResNet seed architecture directly on ImageNet by setting Xhace = 2.0, X = 0.1 and the parameter budget under 7M. Then resume training the grown architecture and compare with exist- ing NAS methods in terms of parameters, top-1 validation accuracy and V100 GPU hours required by the search or growing stages, as shown in Table 4. Note that DARTS (Liu et al., 2019) conducts search on CIFAR-10, then transfers to ImageNet instead of direct search. This is because DARTS operates on supemet by including all the candidate paths and suffers from GPU memory explosion. In terms of epsch-wise FLOPs, results shown in Figure ((c) are for training an equivalent of ResNet-20 on CIFAR-10 in comparison with DARTS and Provable channel pruning approach (Liebenwein et al., 2020). Also note that the EfficientNet-BB BO architecture, included in Table 4, is generated by grid search in the MnasNet search space, thus having the same heavy search cost. To achieve the re- ported performance, EfficientNet-Bi BO utilizes additional seeezze-and-exiiatee (SE) (Hu et al., 2018) modules, AutoAugment (Cubuk et al., 2019), as well as much longer re-training epochs on ImageNet.
Table 4: Performance comparing with NAS methods AetebaNet- (Real al., 2019), MnasNet (Tan al., 2019), EfficientNeet- Bо (Tan Le, 2019), DARTS (Liu et al., 2019) and Proxyless- Net (Cai et al., 2019) on ImageNet.
ProxylessNet still starts with an over- parameterized supernet, but applies a pruning-like search method by bi- narizing the architecture parameters and forcing only one path to be acti- vated at search-time. This enables di- rectly searching on ImageNet, achiev- ing 200x more search cost savings than MnasNet. Contrasting with ProxylessNet, our method progres- sively adds filters and layers to sim- ple seed architectures while maintain- ing sparsification, which leads to sav- ings of not only epoch-wisg wise compu- tation but also memory consumption, enabling faster, larger-batch training. As such, we further save 45% of the GPU search hours, while achieving comparable accuracy parameter trade-offs.
Method
Params
Top-1
Search/Grow Cost
AmoebaNet-A
5.1M
74.5%
76K GPU hours
MnasNet
4.4M
74.0%
40K GPU hours
FfficientNet-Bi
5.3M
77.1% (+SE)
40K GPU hours
DARTS
4.7M
73.1%
N/A
ProxylessNe((GP))
7.1M
75.1%
200 GPU hours
Ours
6.8M
74.3%
80 GPU hours
Ours
6.7M
74.8%
110 GPU hours
Ours
6.9M
75.1%
140 GPU hours
4.4 ANALYSIS
Training Cost Savings. Figure 4 illustrates our sparsification dynamics, showing epoch-wise FLOPs while growing a ResNet-20. Appendix A.8 presents additional visualizations. Even with fully parallel GPU hardware, starting with few filters and layers in the network will ultimately save wall-clock time, as larger batch training (Goyal et al., 2017) can always be employed to fill the hardware.
Figure 5 shows validation accuracy, model complexity, and layer count while growing 3 Complexity is measured as the model parameters ratio of AutoGrow's 's target model. At the end of 160 epochs, method's validation accuracy is 92.36%. which is higher than AutoGrow's is 84.65% at 360 epochs. We thus require fewer fine-tuning epochs to achieve a final 94.50% accuracy on CIFAR.
Budget-Aware Growing, In Fig- ure 6, for ResNet-20 CIFAR-IO, we compare architectures obtained by (1) uniform pruning: naive pre- defined pruning method that prunes the same percentage of channels in each layer, (2) ours: variants of our method by setting different model pa- rameter sparsities as target budgets during growing, and (3) direct design: grown architectures re-initialized with random weights and re- trained. In most budget settings, our growing method outperforms direct design and uniform pruning, demon-
Table 5: Comparison with random pruning bascline on CIFAR- 10.
Model
Method
Acc(%)
Params(M)
Random
90.01 = 0.69
0.770 0.050
VGG-16
Ours
92.50 + 0.10
0.754 0.005
Random
89.18 + 0.55
0.100 0.010
ResNet-20
Ours
90.91 + 0.07
0.096 1 0.002
Random
92.26 1 0.87
3,440 0.110
WRN-28-10
Ours
95.32 1: 0.11
3.443 1 0.010
8

---

Published as a conference paper at ICLR 2021
ImageNet Top-1 Acc (%)
Top-1 Acc vs FLOPs (%))
Epoch-wise FLOPs of Each Conv
kres tama
Retained Rallor%)
Mpst 2o0d
kra0 Goons
Ours
8y02 2an2
64-
NelAp
kyad Gooss
MorphNet
0.25 Mebllctee/1(1288
Mee cord
15e FLOPs (% of 0.25 (11))eNetV1(1288)
150
Epochs
Figure 3: Performance/FLO0 trade-off's for pruned MobileNetV1 on ImageNet.
Figure 4: Epoch-wise training FLOPPs for channel growing
ResNett-2..
200
Val Acc Param Sparsity
Complox and Accuraoy158
CIFAR-10 Val Acc (%)
20-
Layer Numbers
CIFAR-ID 10 vol
Ours Growing
Growing Model Complexity
Uniform Pruning
Growing Layer Numbers
Ours Direct Design
as 102 Growing Epochs
Param Sparsity (%pruned)
20
50
20
Figure 5: Tracking validation accuracy, com- plexity and layers for Basic3RexNet growing.
Figure 6: Pruned architectures obtained by ablated methods with different parameter sparsity.
strating higher parameter efficiency. Our method also appears to have positive effect in terms of regularization or optimization dynamics, which are lost if one attempts to directly train the final compact structure. Appendix A.9 investigates FLOPs-based budget targets.
Comparing with Random Baseline. In addition to the uniform pruning baseline in Figure 6, we also compare with a random sampling baseline further separate the contribution of our configuration space and growing method, following the criterion in (Xie et al., 2019a; Li & Talwalkar, 2019; Yu al., 2020; Radosavovic et al., 2019). Specifically, this random baseline replaces the procedure for sampling entries of in Eq. 6. Instead of using sampling probabilities derived from the learned mask parameters 8, it samples with fixed probability. As shown in Table 5, our method consistently performs much better than this random baseline. These results, as well the more sophisticated baselines in Figure 6, demonstrate the effectiveness of our growing and pruning approach.
Temperature Scheduler. We compare our structurewwis temperature control to global one in channel growing experiments on CIFAR-10 10 using VGG-16, ResNet-20, and WideResNet-28. 10. srructurewiise scheduler. To achieve similar sparsity with the global scheduler,
Table 1 results the corresponding models suffer accuracy drops of 1.4%, 0.6%, and 2.5%. With the global scheduler, optimization of mask variables stops early in training and the following epochs directly training fixed compact network. This may force the network to be stuck with a suboptimal architecture. Appendix A. investigates learning rate and temperature schedule interactions.
equivalent to
CONCLUSION
We propose a simple yet effective method to grow efficient deep networks via structured continuous sparsification, which decreases the computational cost not only of inference but also of training. The method is simple to implement and quick to execute; it automates the network structure reallocation process under practical resource budgets. Application to widely used deep networks on variety of tasks shows that our method consistently generates models with better accuracy-efficieet trade-offs than competing methods, while achieving considerable training cost savings.
This work was supported by the University of Chicago CERES Center for Unstoppable Computing and the National Science Foundation under grant CN5-1956180.
Acknowledgments.

---

Published as a conference paper at ICLR 2021
REFERENCES
Han Cai, Ligeng Zhu, and Song Han. Proxylessnas: Direct neural architecture search on target task and hardware. In ICLR, 2019.
Zhangjie Cao, Mingsheng Long, Jianmin Wang, and Philip S. Yu. Hashnet: Deep learning to hash by continuation. In ICCV, 2017.
Liang-Chiell Chen, George Papandreou, Florian Schroff, and Hartwig Adam. Rethinking atrous convolution for semantic image segmentation. orXiv:1706.0558/ 2017.
Kyunghyun Cho, Bart van Merrienboer, Çaglar Gülgehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. Learning phrase representations using RNN encoder-decoder for statistical machine translation. In EMNLP, 2014.
François Chollet. Xception: Deep learning with depthwise separable convolutions. In CVPR, 2017.
Ekin Dogus Cubuk, Barret Zoph, Dandelion Mané, Vijay Vasudevan, and Quoc V. Le. Autoaugment: Learning augmentation policies from data. In CVPR, 2019.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. ImageNet: A large-scale hierarchical image database. In CVPR, 2009.
Jacob Devlin, Ming-Was Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: pre-training of deep bidirectional transformers for language understanding. In NAACL, 2019.
Mark Everingham, S. M. Ali Eslami, Luc Van Gool, Christopher K. I. Williams, John M. Winn, and Andrew Zisserman. The PASCAL visual object classes challenge: A retrospective. IJCV, 2015.
Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. In ICLR, 2019.
Ross B. Girshick. Fast R-CNN. In ICCV, 2015.
Ariel Gordon, Elad Eban, Ofir Nachum, Bo Chen, Hao Wu, Tien-Ju Yang. and Edward Choi. MorphNet: Fast & simple resource-constrainee structure learning of deep networks. In CVPR, 2018.
Priya Goyal, Piotr Dollár, Ross B. Girshick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, Andrew Tulloch, Yangqing Jia, and Kaiming He. Accurate, large minibatch SGD: training imagenet in hour.
2017.
Sam Gross and Michael Wilber. Training and investigating residual nets. http://torch.chh 2log/2016/02/000 /rwsnetsssssseeeeeeeeeeo html, 2016.
Yiwen Guo, Anbang Yao, and Yurong Chen. Dynamic network surgery for efficient DNNs. In NeurlPS, 2016.
Song Han, Jeff Pool, John Tran, and William J. Dally. Learning both weights and connections for efficient neural networks. In NeurIPS, 2015.
Song Han, Huizi Mao, and William J. Dally. Deep compression: Compressing deep neural network with pruning, trained quantization and huffman coding. ICLR, 2016.
Bharath Hariharan, Pablo Arbelaez, Lubomir D. Bourdev, Subhransu Maji, and Jitendra Malik. Semantic contours from inverse detectors. In ICCV, 2011.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, 2016.
Yang He, Guoliang Kang. Xuanyi Dong, Yanwei Fu, and Yi Yang. Soft filter pruning for accelerating deep convolutional neural networks. IJCAI, 2018
Geoffrey E. Hinton, Oriol Vinyals, and Jeffrey Dean. Distilling the knowledge in a neural network. In NewrlPS Deep Learning and Representation Learning Workshop, 2015.
10

---

Published as a conference paper at ICLR 2021
Sepp Hochreiter and Jurgen Schmidhuber. Long short-term memory. Neural Computation, 1997.
Andrew G. Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam. Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv: 770004444441 2017.
Jie Hu, Li Shen, and Gang Sun. Squeeze-annd-exiiation networks. In CVPR, 2018.
Gao Huang, Yu Sun, Zhuang Liu, Daniel Sedra, and Kilian Q. Weinberger. Deep networks with stochastic depth. ECCV, 2016.
Gao Huang. Zhuang Liu, Laurens van der Maaten, and Kilian Q. Weinberger. Densely connected convolutional networks. In CVPR, 2017.
Gao Huang, Shichen Liu, Laurens van der Maaten, and Kilian Q. Weinberger. CondenseNet: An efficient DenseNet using learned group convolutions. In CVPR, 2018.
Itay Hubara, Matthieu Courbariaux, Daniel Soudry, Ran El- El-ia,iv, and Yoshua Bengio. Binarized neural networks. In NeurIPS, 2016.
Forrest N. landola, Matthew W. Moskewicz, Khalid Ashraf, Song Han, William J. Dally, and Kurt Keutzer. SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and <IMB model size. arXiv:1602.07366. 2016.
Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In ICML, 2015.
Eric Jang, Shixiang Gu, and Ben Poole. Categorical reparameterization with gumbel-softman In ICLR, 2017.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. ImageNet classification with deep convolu- tional neural networks. In NeurIPS, 2012.
Alex Krizhevsky, Vinod Nair, and Geoffrey Hinton. The CIFAR-10 dataset. http::/www.cr toronto. waw/:e/euuuurrreeai malzz/iifat far, htm., 2014.
Carl Lemaire, Andrew Achkar, and Pierre-Mare Jodoin. Structured pruning of neural networks with budget-aware regularization. In CVPR, 2019.
Hao Li, Asim Kadav, Igor Durdanovic, Hanan Samet, and Hans Peter Graf. Pruning filters for efficient ConvNets. In ICLR, 2017. 2017.
Liam and Ameet Talwalkar. Random search and reproducibility for neural architecture search. In UAI, 2019.
Lucas Liebenwein, Cenk Baykal, Harry Lang, Dan Feldman, and Daniela Rus. Provable filter pruming for efficient neural networks. In ICLR, 2020.
Min Lin, Qiang Chen, and Shuicheng Yan. Network in network. arXhv:1312.4400 2013.
Hanxiao Liu, Karen Simonyan, and Yiming Yang. Darts: Differentiable architecture search. In ICLR, 2019.
Wei Liu, Dragomir Anguelov, Dumitru Erhan, Christian Szegedy, Scott E. Reed, Cheng-Yans Fu, and Alexander C. Berg, SSD: single shot multibox detector. In ECCV, 2016.
Jonathan Long, Evan Shelhamer, and Trevor Darrell. Fully convolutional networks for semantic segmentation. In CVPR, 2015.
Christos Louizos, Max Welling, and Diederik P. Kingma. Learning sparse neural networks through 1o regularization. In ICLR, 2018.
Jian-Hao Luo, Jianxin Wu, and Weiyao Lin. ThiNet: A filter level pruning method for deep neural network compression. In ICCV, 2017.
11

---

Published as a conference paper at ICLR 2021
Renqian Luo, Fei Tian, Tao Qin, Enhong Chen, and Tie- Yan Liu. Neural architecture optimization. In NeurlPS, 2018.
Mitchell P. Marcus, Beatrice Santorini, and Mary Ann Marcinkiewicz. Building a large annotated corpus of english: The penn treebank. Computational Linguistics, 1993.
Dmitry Molchanov, Arsenii Ashukha, and Dmitry P. Vetrov. Variational dropout sparsifies deep neural networks. In ICML, 2017.
Hieu Pham, Melody Y. Guan, Barret Zoph, Quoe V. Le, and Jeff Dean. Efficient neural architecture search via parameter sharing. In JCML, 2018.
Ilija Radosavovic, Justin Johnson, Saining Xie, Wan-Yes Lo, and Piotr Dollár. On network design spaces for visual recognition. In ICCV, 2019.
Mohammad Rastegari, Vicente Ordonez, Joseph Redmon, and Ali Farhadi. XNOR-Net: Imagenet classification using binary convolutional neural networks. In ECCV, 2016.
Esteban Real, Alok Aggarwal, Yanping Huang, and Quoc V. Le. Regularized evolution for image classifier architecture search. AAAI, 2019.
Pedro Savarese and Michael Maire. Learning implicitly recurrent CNNs through parameter sharing. In ICLR, 2019.
Pedro Savarese, Hugo Silva, and Michael Maire. Winning the lottery with continuous sparsification. In NeurlPS, 2020.
Laurent Sifre and PS Mallat. Rigid-motion scattering for image classification. PhD thesis, Ecole Polytechnique, CMAP, 2014.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. In ICLR, 2015.
Mingxing Tan and Quoc V. Le. Efficientnet: Rethinking model scaling for convolutional neural networks. In ICML, 2019.
Mingxing Tan, Bo Chen, Ruoming Pang, Vijay Vasudevan, Mark Sandler, Andrew Howard, and Quoc Le. Mnasnet: Platform-awar neural architecture search for mobile. In CVPR, 2019.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NeurlPS, 2017.
Tao Wei, Changhu Wang, Yong Rui, and Chang Wen Chen. Network morphism. In JCML, 2016.
Wei Wen, Yuxiong He, Samyam Rajbhandari, Minjia Zhang, Wenhan Wang, Fang Liu, Bin Hu, Yiran Chen, and Hai Li. Learning intrinsic sparse structures within long short-term memory. In ICLR, 2018.
Wei Wen, Feng Yan, Yiran Chen, and Hai Li. Autogrow: Automatic layer growing in deep convolu- tional networks. In KDD, 2020.
Mitchell Wortsman, Ali Farhadi, and Mohammad Rastegari. Discovering neural wirings. In NeurIPS. 2019.
Bichen Wu, Xiaoliang Dai, Peizhao Zhang, Yanghan Wang, Fei Sun, Yiming Wu, Yuandong Tian, Peter Vajda, Yangqing Jia, and Kurt Keutzer. Fbnet: Hardware-aware efficient convnet design via differentiable neural architecture search. In CVPR, 2019.
Lingxi Xie and Alan L Yuille. Genetic cnn. In ICCV, 2017.
Saining Xie, Alexander Kirillov, Ross B. Girshick, and Kaiming Hc. Exploring randomly wired neural networks for image recognition. In ICCV, 2019a.
Sirui Xie, Hehui Zheng, Chunxiao Liu, and Liang Lin. SNAS: stochastic neural architecture search. In ICLR, 2019b.
12

---

Published as a conference paper at ICLR 2021
Zhenda Xie, Zheng Zhang. Xizhou Zhu, Gao Huang, and Stephen Lin. Spatially adaptive inference with stochastic feature sampling and interpolation. In ECCV, 2020.
Tien-Ju Yang, Andrew G. Howard, Bo Chen, Xiao Zhang, Alec Go, Mark Sandler, Vivienne Sze, and Hartwig Adam. NetAdapt: Platform-aware neural network adaptation for mobile applications. In ECCV, 2018.
Kaicheng Yu, Christian Sciuto, Martin Jaggi, Claudiu Musat, and Mathieu Salzmann. Evaluating the search phase of neural architecture search. ICLR, 2020.
Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. In BMVC, 2016.
Wojciech Zaremba, llya Sutskever, and Oriol Vinyals. Recurrent neural network regularization. 1o09.v:409.2329) 2014.
Xiangyu Zhang, Xinyu Zhou, Mengxiao Lin, and Jian Sun. ShuffleNet: An extremely efficient convolutional neural network for mobile devices. In CVPR, 2018.
Barret Zoph and Quoe V. Le. Neural architecture search with reinforcement learning. In ICLR, 2017.
Barret Zoph, Vijay Vasudevan, Jonathon Shlens, and Quoe V Le. Learning transferable architectures for scalable image recognition. In CVPR, 2018.
13

---

Published as a conference paper at ICLR 2021
A APPENDIX
A. MORE DETAILED ANALYSIS FOR BUDGET-AWARE GROWING
Conducting grid search on trade-off parameters X and A2 is prohibitively laborious and time- consuming. For example, to grow an efficient network on CIFAR-I0, one needs to repeat many times a run of 160-epochs training. and then pick the best model from all grown candidates. To avoid this tedious iterative process, instead of using constants t1 and X2, we dynamically update X and X2 in our one-shot budget-aware growing optimization.
Here we discuss about how budget-aware dynamic growing works in our method. Without loss of generality, we derive the m. 's SGD update rule for the to regularization term in Eq. 3 as:
5e
mc c me mpAluse
Au dm.
- muAiase Aum
(8)
where 77 is the learning rate and is the weight decay factor. At the beginning of growing epochs, when the architecture is very over-sparsified, Au and A/oseAu Au are negative values. Then m. update is along the opposite direction of the to regularization term's gradients, encouraging m. 's sparsification. As a result, some zero-valued m. will be activated and the model complexity is strongly increased to acquire enough capacity for successful training. Then, growing becomes gradually weaker as the network's 's sparsity approaches the budget (Au to zero). Note that if the architecture over-voraaa pee-pparaettriiee, Au and X/ree Au become positive and SGD's update rule is the same as that of lu regularization. As such, our budget-aware growing can automatically and dynamically adapt the architecture complexity not only based on the task loss LE but also on the practical budget requirements in the one-shot training process.
We also note that NAS methods usually use the validation accuracy as a target during their architecture optimization phase, which may require some prior knowledge of validation accuracy on given dataset. Our growing procedure chooses sparsity budget instead of accuracy as target because: (1) During growing, validation accuracy is influenced not only by architectures but also model weights. Directly using Aace may lead to sub-optimal architecture optimization. (2) A sparsity budget target is more practical and easier to set according to target devices for deployment.
A.2 DETAILS OF EVALUATION DATASETS
Evaluation is conducted on various tasks to demonstrate the effectiveness of our proposed method. For image classification, we use CIFAR-10 (Krizhevsky et al., 2014) and ImageNet (Deng et al., 2009): CIFAR- 10 consists of 60,000 images of 10 classes, with 6,000 images per class. The train and test sets contain 50,000 and 10,000 images respectively. ImageNet a large dataset for visual recognition which contains over 1.2M images in the training set and 50K images in the validation set covering 1,000 categories. For semantic segmentation, we use the PASCAL VOC 2012 (Everingham et al., 2015) benchmark which contains 20 foreground object classes and one background class. The original dataset contains 1,464 (train), 1,449 (val), and 1,456 (test) pixel-level labeled images for training, validation, and testing, respectively. The dataset is augmented by the extra annotations provided by (Hariharan et al., 2011), 1), resulting in 10,582 training images. For language modeling, we use the word level Penn Treebank (PTB) dataset (Marcus al., 1993) which consists of 929k training words, 73k validation words, and 82k test words, with 10,000 unique words in its vocabulary.
A.3 UNPRUNED BASELINE MODELS
For CIFAR-10, we use VGG-16 (Simonyan & Zisserman, 2015) with BatchNorm (loffe & Szegedy, 2015), ResNet-20 (He et al., 2016) and WideResNet-22-100 (Zagoruyko & Komodakis, 2016) as baselines. We adopt a standard data augmentation scheme (shiftingnnirroring) following (Lin et al., 2013; Huang et al., 2016), and normalize the input data with channel means and standard deviations. Note that we use the CIFAR version of ResNet-20', VGG-162. and WideResNet-28. ResNet-20, and WideResNNt-28-100 are trained for 160, 160, and 200 epochs, respectively, with
VGG- 16,
resnet eifar dffaar/00bbbb//asseeeeeeet..y
resnet.py
14

---

Published as a conference paper at ICLR 2021
batch size of 128 and initial learning rate of 0.1. For VGG-16 and ResNet-20, we divide learning rate by 10 at epochs 80 and 120, and set the weight decay and momentum as 10 and 0.9. For WideResseet22- 10, the learning rate is divided by 5 at epochs 60, 120, and 160; the weight decay and momentum are set to 5 x 10-4 and 0.9. For ImageNet, we train the baseline ResNet-50 and MobileNetV1 models following the respective papers. We adopt the same data augmentation scheme as in (Gross & Wilber, 2016) and report top-1 validation accuracy. For semantic segmentation, the performance is measured in terms of pixel intersection-over-umee classes (mlOU). We use Deeplab-v3-. v3-ResNet- 101 (Chen et al., 2017) as the baseline model following the training details in (Chen al., 2017). For language modeling, we use vanilla two-layer stacked baseline. The dropout keep ratio is 0.35 for the baseline model. The vocabulary size, embedding size, and hidden size of the stacked LSTMs are set as 10,000, 1,500, and 1,500, respectively, which is consistent with the settings in (Zaremba et al., 2014).
(IOU) averaged across the 21
LSTM (Zaremba et al., 2014)
A.4 MOBILENETVI CHANNEL GROWING ON IMAGENET
To further validate the effectiveness the proposed method on compact networks, we grow the filters of MobileNetV1 ImageNet and compare the performance of our method to the results reported directly in the respective papers, as shown Table In MobileNetV] experiments, following the same setting with Netadapt (Yang et al., 2018), we apply our method on both (1) small setting: growing MobileNetV (128)) with 0.5 multiplier while setting the original model's multiplier as 0,25 for comparison and (2) large setting: growing standard MobileNett (2224)) while setting the original model's 's multiplier as 0.75 for comparison. Note that MobileNetV1 is one of the most compact networks, and thus is more challenging to simplify than other larger networks. Our lower-cost growing method can still generate a sparser MobileNetVI model compared with competing methods.
Table Overview of the pruning performance of each algorithm on MobileNetV1 ImageNet.
Model
Method
Top-1 Val Acc(%)
FLOPP(%)
Train-Cost Savings(X)
Original(25%%
45.1 (+0.0)
100
1.0(x)
MobileNet V1(128)
MorphNet
46.0 (+0.9)
110
0.9(x)
Netadapt
46.3 (+1.2)
81
1.1(x)
Ours
46.0 (+0.9)
73
1.7(x)
Original(75%) Netadapt
68.8 (+0.0)
100
1.0())
MobileNet
69.1 (+0.3)
87
1.2(X)
V1(224)
Ours
69.3 (+0.5)
83
1.5(x)
A.5 BHEPLAB-V3-RESME 101 ON PASCAL VOC 2012
We also test the effectiveness of our proposed method on a semantic segmentation task by growing a Bopplab-3--ReNeet-ee
model's filter numbers from scratch directly on the PASCAL VOC 2012 dataset. We apply our method to both the ResNet- 101 backbone and ASPP module. Compared to the baseline, the final generated network reduces the FLOPs by 58.5% and the parameter count by 49.8%, while approximately maintaining mloU (76.5% to 76.4%). See Table 7.
Table 7: Results on the PASCAL VOC dataset.
Model
Method
mIOU
Params(M)
FLOPs((6
Train-Cost Savings( (x)
Deeplab
Original
76.5 (-0.0)
58.0 (100%)
100
1.0(x)
-v3-
L1-Pruning
75.1 (-1.4)
45.7 (78.8%)
62.5
1.3(x)
ResNet101
Ours
76.4 (-0.1)
29.1 (50.2%)
41.5
2.3(x)
"happs//gtthbb.comccnnii http::/hh.......ooooooooooooaooa
15

---

Published as a conference paper at ICLR 2021
A.6 EXTENSION TO RECURRENT CELLS ON PTB DATASET
We focus on LSTMs (Hochreiter & Schmidhuber, 1997) with 1, hidden neurons, a common variant of RNNS that learns long-term dependencies:
ft = o ((W, @ (em² 2 ) + (U, O (mam) 2 )
1) (
by)
4 0g (W, @ (em² ))z + (U; @ (mom) 1 ) + bi)
Q = 0y (W. @ (em²) T )) + (Uo O (m.m) T) )) + bo)
c
& = ((W, @ (em ) + (Ue o (m.m² Mi-1 + bb)
a = == 4 o é, h 0r ① (c)) s.t. me E 10, 10, 1
(9)
where og the sigmoid function, C denotes element-wise multiplication and on the hyperbolic langent function. Ha denotes the input vector the time-step f, ht denotes the current hidden state, and o denotes the long-term memory cell state. Wy. W,, Wo. W. denote the input-to-hidden weight matrices and Uy, U,, U., Ue denote the hidden-to-hidden weight matrices. me is binary indicator and shared across all the gates to control the sparsity of hidden neurons.
We compare our proposed method with ISS based vanilla two-layer stacked LSTM. As shown in Table 8, our method finds more compact model structure at lower training cost, while achieving similar perplexity on both validation and test sets. These improvements may be due to the fact that our method dynamically grows and prunes the hidden neurons from very simple status towards a better trade-off between model complexity and performance than that of ISS, which simply uses the group lasso to penalize the norms of all groups collectively for compactness.
Table 8: Results on the PTB dataset.
Method Perplexity (val,test)
Final Structure
(eight(M)
FLOPs(%)
Train-Cost Savings(x)
Original
(82.57, 78.57)
(1500, 1500)
66.0M (100%)
100
1.0(x)
ISS
(82.59, 78.65)
(373, 315)
21.8M (33.1%)
13.4
3.8(x)
Ours
(82.46. 78.68)
(310, 275)
20.6M (31.2%)
11.9
5.1(x)
A.7 VARIANTS OF INITIAL SEED ARCHITECTURE
In Table 9, we make a detailed comparison among initial seed architecture variants of ours and AutoGrow (Wen et al., 2020). For both ours and AutoGrow, "Basic" and "Bottleneck" refer to ResNets with standard basic and bottleneck residual blocks, while "PlainLayers" refers stacked convolutional, batch normalization, and ReLU layer combinations. Similar with standard ResNets, for variants of the seed architecture, we adopt three stages for CIFAR-I0 and four stages for ImageNet. PlainNets can be obtained by simply removing shortcuts from these ResNet seed variants with equal stage numbers. For each stage, we start from only one growing unit, within which initial filter numbers are also initialized at one for channel growing.
A.8 TRACK OF ANY-TIME SPARSIFICATION DURING CHANNEL GROWING
Figure 7 and Figure 8 show the dynamics of train-time growing channel ratios of ResNet-20 and VGG-16 on CIFAR-I0, respectively. To better analyze the growing patterns, we visualize the channel dynamics grouped by stages in Figure 9 for ResNet-20 and Figure 10 for VGG-16, respectively. Note that, for VGG-16, we divide it into stages based on the pooling layer positions and normalize channel ratios by 0.5 for better visualization. We that our method grows more channels of earlier layers within each stage of ResNet-20. Also, the final channel sparsity ResNet-20 is more uniform due to the residual connections.
16

---

Published as a conference paper at ICLR 2021
Table 9: A detailed comparison among seed architecture variants of our method and AutoGrow (Wen et al., 2020). In growing units term, "Basic" and "Bottleneck" refer to ResNets with standard basic and bottleneck residual blocks while "PlainLayers" refers standard convolutional layer, BN, and ReLu layer combinations in VGG-like networks without shortcuts.
Families
Variants
Methods
Channel Growing
Growing Units
Stages
Shortcuts
Ours
Besic3RexNet
Basic
AutoGrow
ResNet
Ours
Bottleneck4ResNel
Bottleneck
4
AutoGrow
Ours
Plain3Net
PlainLayers
AutoGrow
x
VGG-like
Ours
Plain4Net
PlainLayers
AutoGrow
X
Channels Dynamics
Epoch-wise Channels Dynamics Conv
100
Retained Channels Ratio
Retained Channels Ratiol 9%
layr Knoen
lyerd Lowes
Uu
layd Sooed
Koar Sonrs
20-
WOLL
boans
Sayd Soand
Epochs
Epechs
Figure 7: Epoch retained channel ratio dy- Figure 8: Epoch-wise retained channel ratio dy- namies for each layer in ResNet-20.
namics for each layer in VGG-16.
A.9 FLOPS-BASED BUDGET-AWARS GROWING
We also investigate the effectiveness of setting a FLOPPs target for budget-aware growing in Figure 11. We observe similar trends among uniform pruning, ours growing, and ours direct design: in most FLOPs budget settings, our growing method outperforms direct design and uniform pruning. We also observe that when setting extreme sparse FLOPs target (e.g., 85%), our method achieves lower accuracy than the other two variants. The reason is that our channel growing is forced to only grow architectures from 99% sparsity up to 85% FLOPs and 90% parameters sparsity, during which models cannot acquire enough capacity to be well trained.
A. INTERACTIONS BETWEEN LEARNING RATE AND TEMPERATURE SCHEDDLERS
Two factors influence the growing optimization speed in our method: rate, which are hyperparameters controlled by their respective schedulers. We first visualize the structure-wise separate temperature dynamics in Figure 12 by averaging temperatures per layer during ResNet-20 channel growing on CIFAR-I0. We that temperatures rates for channels. Usually, low learning rate and high temperature in late training epochs make the network growing optimization become very stable. In Figure 13, deliberately decay y in the temperature scheduler, mirroring the learning rate decay schedule, in order to force growing until the end. As shown Figure 14, our method is still adapting some layers even the last epoch. We find that such instability degrades performance, since some newly grown filters may not have enough time to become well trained.
temperature and learning
growing with different
'The proposed configuration space can he readily applied to the compression of GRUs (Cho et al., 2014) and vanilla RNNs.
17

---

Published as a conference paper at ICLR 2021
Epoch-wise Channels Dynamics of Each Conv in ResNet-20 Layer Stage 1
Layers
149
Epochs
Epoch-wise Channels Dynamics of Each Conv in ResNet-20 Layer Stage 2
Layers
11
300
139
140
Eoochs
Epoch-wise Channels Dynamics of Each Conv in ResNet-20 Layer Stage 3
Layers
Epochs
Figure Visualization of retained channel ratio dynamics for each stage in ResNet-20.
Layers
Epoch-wise Channels Dynamics of Each Conv in VGG-16 Layer Stage
Epochs
149
Layers
Epoch wise Channels Dynamics of Each Conv in VGG-16 16 Layer Stage
Epochs
Lavers
Epoch-wise Channels Dynamics Each Conv VGG-16 Layer Stage
180
14)
Epochs
Layers
Epoch-wise Channels Dynamics of Each Conv in VGG-16 Layer Stage
Epochs
Layers
Epach-wise Channels Dynamics
VGG-16 Layer Stage
Epochs
Figure 10: Visualization of retained channel ratio dynamics for each stage in VGG-16.
Val Acc vs FLOPs Sparsity
Average Tempurature of Each Conv
CIFAR-10 Val (%)
kyerlloand
89
$8-
33
kym21cand
Growing
85
Unifarm Pruning
Ours Direct Design
FLOPs Sparsity (%pruned)
Epochs
Figure 11: Pruned architectures obtained by ablated methods with different FLOPs in channel growing. sparsity.
Figure 12: Structure wise separate temperature dynamics
Average Tempurature of Each Conv
Epach-wise FLOPs of Each Cenv
Myec Jowi
Relained FLOPs Ratio(%)
Speld cowd
Average Tempurature
Teeil coe)
1p-13
Spetz cows
Spet1 cows
now
Up-23
9yd7 Lowel
B4600 cow
Spei11 coed
Iear13 coel
Epochs
Epochs
Figure 13: Structure-wise separate decayed tem-
Figure 14: Track of epoch-wise train-time FLOPs for channel growing in ResNet-20.
perature dynamics in channel growing.
18


Reviews:
[{"review_id": "rLzfGjbIcJb", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "**Pros**:\n\n1. This paper nicely unifies two different classes of approaches (NAS + sparsity) for determining the topology of neural networks. They are combined into a single optimization problem, with binary indicators on network components and connections.\n\n2. Experiments illustrate the behavior of the method. It is good to see that the experiments dig a big deeper than end-result accuracy. For instance, the \"budget-aware growing\" is shown well to work as described by Fig 3.\n\n**Cons**:\n\n3. No attention to random seeding.\n\nThe sparsification dynamics seem likely to change somewhat from one run to the next. The submission does not describe how random seeding was done for training. Multiple runs with different seeds are not shown, and the distribution of accuracies across runs is unknown. Attention to randomness for this kind of training process seems especially important given the variances in results in the Lottery Ticket hypothesis paper.\n\n4. No comparison to simple random baseline.\n\nA large portion of the method consists of a search method over the space of possible sparse networks, combining it with growing the network to get a NAS-like method. It has been observed, though, that in a sufficiently general space of this kind one can randomly sample connections and see high accuracies. So to identify the sources of empirical gains, it is good to consider experimental baselines, such as random sampling, that separate the contribution of the search space and the search method:\n\n  * Xie et. al. \"Exploring randomly wired neural networks for image recognition\"\n  * Li & Talkwaker \"Random search and reproducibility for neural architecture search\"\n  * Yu et. al. \"Evaluating the Search Phase of Neural Architecture Search\"\n  * Radosavovic et. al. \"On Network Design Spaces for Visual Recognition.\"\n\nNote that the submission's method also is randomly choosing connections, through a somewhat involved process that also accounts for the observed sparsity of G during training. The simplest baseline seems to be the \"uniform pruning\" described in Section 4.4. This only ablates part of the method that doesn't seem to meet the same criterion here.\n\n5. Incomplete illustration of the cost/accuracy tradeoff.\n\nThe gold standard for comparison in both sparse-neural-network papers and NAS is to consider the accuracy at a range of different model costs. See for example:\n\n  * Blalock et. al. \"What is the state of neural network pruning?\" Figs 1, 3\n  * (Yang et. al., 2018) Figs 5-9\n\nThis clearly illustrates whether a method is overall better (i.e. produces better models across the entire pareto frontier), or is only better for some ranges or on one metric. For a result like the first one in Table 5 in the submission, it is unclear which model is better: they may simply be considering different points on the same cost/accuracy curve.\n\n6. As a less important aside, \"budget-aware growing\" seems to be an ad-hoc reinvention of something similar to an Augmented Lagrangian method. Explicitly describing the differences from standard optimization techniques might be good.\n\n**Reasoning for rating**:\n\nWhile the experiments are extensive, I think they miss the key comparisons that show how useful the method and each of its components is. Given that many different innovations are included in the submission, it may be a muddle for follow-up research to sort out how good each individual one is.\n\n**Misc comments**\n\n  * Check spacing around (6) in Algorithm 1\n  * Colon instead of comma after \"trainable variables\" in §4.1\n  * \"For better analyze the growing patterns\" -> \"To better analyze the growing patterns\" on page 14\n  * Wortsman et. al. \"Discovering Neural Wirings\" is another closely related work at the intersection of NAS and pruning. (with major differences from the submission)\n\n**After rebuttal**\n\nThe authors have gone above and beyond in providing additional experimental results. All of the points raised above that deal with methodological issues are completely addressed.\n\nThe sole significant weakness that remains is the lack of the kind of ablation/component studies that would justify individual design decisions. I do not disagree with the authors that this will be difficult for this work, but I still feel they would have been helpful for researchers who will be building upon this method. ", "extra_scores": {}, "extra_text": {}}, {"review_id": "mShRrPsufgN", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a new principled approach to growing deep network architectures based on continuous relaxation of discrete structure optimization combined with a sparse subnetwork sampling scheme. It starts from a simple seed architecture and dynamically grows/prunes both the layers and filters during training. Through extensive experiments, the authors show that this method produces more efficient networks while reducing the computational cost of training, still maintaining good validation accuracy, compared to other NAS or pruning/growing methods. \n\nStrength: \n(+) The proposed idea of formulating the problem as a continuous relaxation of discrete structure optimization is interesting. It seems to be a more principled approach than previous NAS or separate pruning/growing approaches. \n(+) Extensive experimental results are provided to verify the superiority over recent other methods and also to show the performance behavior of the proposed method. The overall experimental setup is systematic and comprehensive. \nThe experiments were done on widely used deep networks on various tasks.\n\nI only have concerns about the clarity of the notation and the representation of the figures. Specific examples are as follows: \n\nIn Eq. (3), it is said that f is the operation in Eq. (1). However, I couldn’t find f in Eq. (1).\n\nIt should be clarified how many temperature parameters β are in the proposed model. Only one or as many as channels and layers? If it is only one, it does not seem reasonable that all the probabilities growing or pruning channels and layers are the same.  Equation (7) seems to imply β to be a vector, but earlier notations (e.g. in Algo 1, Equation 6, etc.) seem to present it as a scalar.\n\nOverall, there are confusing symbols, whether it’s a scalar or a vector. I recommend the channel and layer indicators are denoted as vectors. It seems that each channel and each layer has its unique indicator, respectively. Also, notations should include channel and layer index if they are different depending on channels and layers.\n\nAdditionally, all the experimental results shown in the main manuscript are on convolutional neural networks while the abstract mentions recurrent neural networks. The appendix has some, but very little has the main manuscript. If it’s an important part of this manuscript, the authors should include at least a brief summary of the results. \n\nSome figures (and the text inside) are too small while containing many details, probably because of the space limit. For example, Figure 3 has many lines that are hard to analyze and texts that are not readable. \n\nIn Table 1, what’s the meaning of the underlines? I guess the second best results, but for RestNet-20, the method with the second-best FLOPs is SoftNet, not Provable. And the explanation about the boldface and underlines should be included. \n", "extra_scores": {}, "extra_text": {}}, {"review_id": "_HS7uArSE12", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a novel NAS method that searches the model architectures by grows the networks. This searching strategy determines the channel and layer configurations by assigning a binary learnable parameter m for each channel or layer. The objective is to optimize a trade-off between the model performance on the given task and the regularization on the binary indicator m.\n\nPros:\n1. The general idea of searching the architectures by growing the networks sounds very interesting. The authors propose a novel framework to achieve their idea, and also apply some tricks to speed up and simplify the optimization (e.g. budget-aware growing and learning by continuation). \n2. The paper is well-written and easy to follow.\n3. The authors conduct a series of solid experiments to verify the effectiveness of their proposed methods. The experiments show the performance of channel pruning, the remarkable improvement on AutoGrow model, and the comparsion with other NAS methods.\n\nCons:\n1. Compared to ProxylessNet, the proposed model can reduce half of the training time but does harm to the model performance.\n\nQuestions:\n1. What's the exact meaning of \"Top-1 valiadation accuracy\"? What's the different with Top-1 accuracy? Is this metric evaluated on the valiadation set?", "extra_scores": {}, "extra_text": {}}, {"review_id": "FGN0ZqCdH9L", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\n\n  This paper proposes a NAS-type work for growing a small network to a large network by adding channels and layers gradually. The authors apply the method to both CNN and LSTM networks. \n\nStrong points:\n\n  1. This paper is well-written and shows good results.\n\n  2. The proposed algorithm is sound and effective. E.g. use less wall-time as compared to other NAS approaches.\n\n\nWeak points:\n  \n  1. It seems that the number of channels and number of layers still need to be predefined (the mask size).\n\n\nQuestions:\n\n\n  1. How does the FLOPs reduction translate to runtime saving?\n\n  2. What is the target sparsity u in the experiments?\n\n  3. When growing with layers, does the author observe any middle layer is dropped and then recovered? If so, does it happen frequently? \n\n  4. At section 4.2 and 4.3, what is the size of the channel/layer mask? I believe the author still needs to define the upper bound of the network can grow. If so, does the upper bound affect the optimization?  Or the proposed method gradually expands the mask?\n\n  5. In table 4, I think Efficient-B0 should be taken into consideration as it is a recent representative approach.\n  \nAfter rebuttal:\n\nThe authors' rebuttal addressed all my questions and I upgrade my rating.\n\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes a method to grow deep network architectures over the course of training. The work has been extremely well received and has clear novelty and solid experiment validation.
Decision: Accept (Oral)