Energy-Aware Neural Architecture Optimization with Fast Splitting Steepest Descent
Authors: Dilin Wang
Meng Li
Lemeng Wu
Vikas Chandra
Qiang Liu
Venue:  (2020)
Keywords: Neural architecture optimization
splitting steepest descent

Abstract: Designing energy-efficient networks is of critical importance for enabling state-of-the-art deep learning in mobile and edge settings where the computation and energy budgets are highly limited. Recently, Wu et al. (2019) framed the search of efficient neural architectures into a continuous splitting process: it iteratively splits existing neurons into multiple off-springs to achieve progressive loss minimization,  thus finding novel architectures by gradually growing the neural network. However, this method was not specifically tailored for designing energy-efficient networks, and is computationally expensive on large-scale benchmarks.  In this work, we substantially improve Wu et al. (2019) in two significant ways:  1) we incorporate the energy cost of splitting different neurons to better guide the splitting process, thereby discovering more energy-efficient network architectures; 2) we substantially speed up the splitting process of Wu et al. (2019), which requires expensive eigen-decomposition, by proposing a highly scalable Rayleigh-quotient stochastic gradient algorithm.  Our fast algorithm allows us to reduce the computational cost of splitting to the same level of typical back-propagation updates and enables efficient implementation on GPU. Extensive empirical results show that our method can train highly accurate and energy-efficient networks on challenging datasets such as ImageNet,  improving a variety of baselines,  including the pruning-based methods and expert-designed architectures.

Under review as a conference paper at ICLR 2020
ENERGY-AWARE NEURAL ARCHITECTURE OPTIMIZA- TION WITH FAST SPLITTING STEEPEST DESCENT
Anonymous authors Paper under double-blind review
ABSTRACT
Designing energy-efficient networks is of critical importance for enabling state- of af-the-art deep learning in mobile and edge settings where the computation and energy budgets are highly limited. Recently, Wu et al. (2019) framed the search of efficient neural architectures into continuous splitting process: it iteratively splits existing neurons into multiple off-springs to achieve progressive loss minimiza- tion, thus finding novel architectures by gradually growing the neural network. However, this method was not specifically tailored for designing energy-efficient networks, and is computationally expensive on large-scale benchmarks. In this work, substantially improve Wu et al. (2019) in two significant ways: 1) we incorporate the energy cost of splitting different neurons to better guide the split- ting process, thereby discovering more energy-efficies network architectures: 2) we substantially speed up the splitting process of Wu et al. (2019), which requires expensive ggge-decompoostion. by proposing a highly scalable Rayleigh-quotient stochastic gradient algorithm. Our fast algorithm allows us to reduce the compu- tational cost of splitting to the same level of typical back- prekpppppagation updates and enables efficient implementation on GPU. Extensive empirical results show that our method can train highly accurate and energy-efficient networks on challeng- ing datasets such as ImageNet, improving variety of baselines, including the pruning-baat based methods and expert-designed architectures.
INTRODUCTION
Deep neural networks (DNNs) have demonstrated remarkable performance in solving various chal- lenge problems such as image classification (e.g. [Simonyan & ZZissemman 2013 He et al. 2016 Huang et al. 2017), object detection (c.g. He et al. 2017a) and language understanding (e.g. Devlin et al. 2018). Although large-scale deep networks have good empirical performance, their large sizes cause slow computation and high energy cost in the inference phase. This imposes a great challenge for improving the applicability of deep networks to more real-word domains, especially on mobile and edge devices where the computation and energy budgets are highly limited. It is of urgent de- mand to develop practical approaches for automatizing the design of small, highly energy-efficient DNN architectures that are still sufficiently accurate for real-world AI systems.
Unfortunately, neural architecture optimization has been known be notoriously difficult. Com- pared with the easier task of learning the parameters of DNNs, which has been well addressed by back-propagation (Rumelhart et al. 1988); optimizing the network structures casts much more challenging discrete optimization problem, with excessively large search spaces and high evaluation cost. Furthermore, for neural architecture optimization in energy-efficient settings, extra difficulties arise due to strict constraints on resource usage.
Recently, Wu et al. (2019) investigated similar notations of gradient descent for learning network architectures and framed the architecture optimization problem into continuous optimization in an infinite-dimensionall configuration space, on which novel notions of steepest descent can be derived for incremental update of the neural architectures. In practice, the algorithm optimizes a neural net- work through a cycle of paramedic updating and splitting phase. In the parametric updating phase, the algorithm performs standard gradient descent to reach stable local minima; in the splitring phase, the algorithm expands the network by splitting a subset of exiting neurons into several off- springs in optimal way. key observation is that the previous local minima can be turned into

---

Under review as a conference paper at ICLR 2020
saddle point in the new higher-dimensional space induced by splitting that can be escaped easily; thus enabling implicitly architecture space exploration and achieving monotonic loss decrease.
However, the splitting algorithm in Wu et al. (2019) treats each neuron equally, without taking into account the different amount of energy consumption incurred by different neurons, thus finding models that may not be applicable in essource-constrainnd environments. To close the gap between DNNs design via splitting and energy efficiency optimization, we propose an energy-aware splitting procedure that improves over Wu al. (019) by explicitly incorporating energy-related metrics to guild the splitting process.
Another practical issue of Wu et al. (2019) is that it requires eigen-computation of the splitting matrices, which causes a time complexity of (ndd)) and space complexity of O(nd²) when imple- mented exactly, where neuron. This makes it difficult to implement the algorithm on GPUs for modern neural networks with thousands of neurons, mainly due to the explosion of GPU memory, thus prohibiting efficient parallel calculation on GPUs. In this work, we address this problem by proposing fast gradient- based approximation of Wu al. (2019), which reduces the time and space complexity to ((nd)) and (nd), respectively. Critically, our new fast gradient-based approximation can be efficiently implemented on GPUs, hence making it possible to split very large networks, such as these for ImageNet.
the number of neurons the network, and d the dimension of each
Our method achieves promising empirical results on challenging benchmarks. Compared with prior art pruning baselines that improve the efficiency by removing the least significant neurons (e.g. Liu et al.] 2017 Li et al. 2017 Gordon et al.] 2018), our method produces a better accuracy-flope trade-off on CIFAR-I00. On the large-scale ImageNet dataset, our method finds more flops-efficient network architectures that achieve 1.0% and 0.8% improvements in top-1 accuracy compared with
prior expert-designed MobileNet (Howard et al., 2017) and MobileNetV2 (Sandler al. 2018). respectively. The gain is even more significant on the low-flops regime.
2 SPLITTING STEEPEST DESCENT
Our work builds upon a recently proposed splitting steepest descent approach (Wu et al. 2019). which transforms the co-optimization of neural architectures and parameters into a continuous opti- mization, solved by generalized steepest descent on a functional space. To illustrate the key idea, assume the neural network structure specified by a set of size parameters m = {m...... } where each mi denotes the number of neurons in the k-th layer, or the number of certain type of neurons. Denote by Om the set of possible parameters of networks of size m, then Ox = U mm which we call the configuration space, is the space of all possible neural architectures and parameters.
In this view, learning parameters of a fixed network structure is minimizing the loss inside an in- dividual sub-region 6m In contrast, optimizing in the overall configuration space Ox admits the co-optimization of both architectures and parameters. The key observation is that the optimization in O. is in fact continuous (despite being infinite dimensional), for which (generalized) steepest descent procedures can be introduced to yield efficient and practical algorithms.
In particular, Wu et al. 2019) considered a splitting steepest descent on O which consists two phases: 1) the parametric descent inside each Om with a fixed network structure m. which reduces to the typical steepest descent on parameters, and 2) the architecture descent crossing the boundaries of difffernt sub-regions On. which, in the case of Wu al. (2019), corresponds to "growing" the network structures by splitting set of critical neurons to multiple off-springs (see Figure Dh).
From the perspective of non-convex optimization, the architecture descent across boundaries of O. can be viewed as escaping saddle points in the configuration space. As shown in Figure [p. when the parametric training inside a fixed θ, gets saturated, architecture descent allows us escape local optima by jumping into higher dimensional sub-region of larger network structure. The idea is that the local optima inside 6m can be turned into a saddle point when viewed from the higher dimensional space of larger networks (Figure Ie), which is escaped using splitting descent.
Escaping local minima via splitting requires to fix a proper notion of distance on θ.0 in order to derive steepest descent algorithm. In Wu et al. 2019), steepest descent with pe- Sc-aaseesteein
2

---

Under review as a conference paper at ICLR 2020
Structural descent
(e-g,, splittingl
wx2
w/2
Enor
SGD
saddle point
Parametric updates
Parametric updatess
minima
Training time
(a)
(b)
(c)
(a) Splitting one neuron into two off-springs. (b) Steepest descent on the overall ar- chitecture space consists of both standard gradient descent on the parameters (with fixed network structures), and updates of the network structures (via splitting).
Figure 1:
(c) The local optima in the low dimensional space is turned into saddle point in higher dimensional of the augmented networks, and hence can be escaped by our splitting strategy, yielding monotonic decrease of the loss.
distance considered, which is shown to naturally correspond to the practical procedure of split- ting neurons. Here we only introduce the intuitive idea from the practical perspective of optimally splitting neurons. The readers referred to Wu et al. (2099) for more theoretical discussion.
Consider the simplest case of splitting a single neuron. Let o08, x)) be a neuron inside a neural network that we want learn from data, where 0 is the parameter of the neuron and its input variable, Assume the loss function of / has general form of
L(0) == (o x))],
(I)
where D is the data distribution, and (-) is the map from the output of the neuron to the final loss. In this work, the word "neuron" broadly refers to repeatable modules in neural networks, such as the typical hidden neurons, filters CNNs.
Assume we have achieved stable local optimum of L(0), that is, L 0 and oo) > O, so that we can further decrease the loss by local descent the parameters. In this case, splitting steepest descent enables further descent by introducing more neurons via splitting. Specifically, we split into off-springs 0 {0,)/= and replace the neuron (0,, x) with weighted sum of the off-spring neurons th W;;0 z). where w == (w; i= is set of positive weights assigned on each of the off-springs, and satisfies 2i-1 w) = 1, wi > 0. See Figure la for an illustration. This yields augmented loss function O and w:
Σ w (0,, x) )]
C. (0, w) = BLND
It is easy to see that if we set 0, = 0 for all the off-springs, the network remains unchanged. Therefore, as we change 0, in small neighborhood of it introduces a smooth change on the loss function. Splitting steepest descent is derived by considering the optimal splitting strategies to achieve the steepest descent on loss in a small neighborhood the original parameters.
Deriving splitting steepest descent Derive the optimal splitting strategy involves deciding the number off-springs m, the values of the weights (w:} and the parameters for the off-springs {0,)- In Wu et al. (2019), this is formulated into the following optimization problem:
min 0,0,w
Cm (0, w) - L(O) S.L. 110 - 01l S Σ w. = 1, we > 0, Vie [m], m E No
MEN+ (3)
where the parameters 0 ;= [0 of the off-springs are restricted within an infinitesimal e-ball of the original parameter f, that is, 116 A|| < c, with small positive step size parameter. Note that the number of off-springs m is also optimized, yielding an infinite dimensional optimization.
is achieved by either 1 (no splitting) or m = 2 (two off-springs). The property of the optimal solution is characterized (asymp- totically) by the following key splitting matrix 5(0).
Fortunately, when is very small, the optimum of Equation (
S(0) ETnD V. (00( V.B(((0.))\ooo0(0.a)
3

---

Under review as a conference paper at ICLR 2020
which is symmetric d d matrix (and d is the dimension of A). The optimum of Equation when L(0) reaches a stable local optimum (i.e., VoL(O) O, (() > 0), is determined by S via
{
c² min {Amin (S(0)) 0} + O
min
Lm (0, w) -
(4)
m. m,0,u
where Amiu A (S(O)) denotes the minimum eigenvalue of S(Ø), and it is called the splitring index.
When Amiu (S())) > 0, the loss not be improved by any splitting strategies following (4). When Amin (S))) < 0, the maximum decrease of loss, which equals e² Amin (S()))/2 can be achieved by simple strategy of splitting the neuron into
copies with equal weights, whose parameters updated along the minimum eigen cigen-vectors Umin Umin {()) of S(O), that is,
2, 0 0 + eVmin (S(0)):
02 0 eumin (5()))
W1 W2 1/2.
(5)
In this case, splitting allows us escape the parametric local optima to enable further improvement.
Splitting deep neural networks As shown in Wu et al. 2019), the result above can be naturally extended to more general cases when we need to split multiple neurons in deep neural networks. Consider a neural network with n neurons @(lm)
{0) 0"). Assume we split a subset A
of neurons with the optimal strategy in Equation (5) following their own splitting matrices, the improvement of the overall loss equals the sum individual gains:
G(A) = =- Amin(()
BEA
where Amin(() := >:nn==nnn (S, denotes the minimum eigenvalue of the splitting matrix Sy(0") as- sociated with neuron Therefore, given budget of splitting at most given number of neurons, the optimal subset of neurons to split are the top ranked neurons with the smallest, and negative minimum eigenvalues. Overall, the splitting descent Wu al.] (2019) alternates between para- metric updates with fixed network architectures, and splitting top ranked neurons to augment the architectures, until a stopping criterion is reached.
NEURAL ARCHITECTURE OPTIMIZATION VIA ENERGY-AWARE SPLITTING
The method above allows to select the best subset of neurons to split to yield the steepest descent on the loss function. In practice, however, splitting different neurons incurs different amount of increase on the model size, computational cost, and physical energy consumption. For example, splitting neuron connecting to a large number of inputs and outputs increases the size and com- putational cost the network much more significantly than splitting the ncurons with fewer inputs and outputs. In practice, convolutional layers close to inputs often have larger feature maps which lead to a high energy cost, and layers closer to outputs have smaller feature maps and hence lower computational cost. A better splitting strategy should take the cost of different neurons into account.
To address this problem, we propose to explicitly incorporate the energy cost to better guide the splitting process. Specifically, for neural network with n neurons, propose to decide the optimal splitting set by solving the following constrained optimization:
min BeAmin (C), s.t. Σ 1=
ee S Gbudgets Be E 10, A4
(6)
Here R" a binary mask, with 3r indicates whether the 4-th neuron should be split (3, or not (B, 0), and represents the cost of splitting at the current iteration. We search for the optimal subset of neurons that yields the largest descent on the loss (in terms of the splitting index), while in- curring total energy cost no larger than a budget threshold ebudget- This optimization Equation (6) standard knapsack problem. The exact solution of knapsack problems can be very expensive due to their NP-hardness. In practice, by relaxing the integrality constrains to linear constrains such that 3r E [0, 1), Vl. The continuous relaxation could then be solved using standard linear programming tools efficiently (Dantzig) 1998).
use linear programming relaxation for fast approximation
Finally, we define the optimal splitting set := (3, > 0.9, VLJ. For each neuron in A, we split into equally weighted off-springs along their splitting gradients, following Equation 5
4

---

Under review as a conference paper at ICLR 2020
In this work, take eg be the energy cost, and estimate it by the increase of flops if we split the 8-th neuron starting from the current network structure. Note that the cost of splitting the same neuron changes when the network size changes across iterations. Therefore, of every neuron at each splitting stage, based on the architecture of the current network.
re-evaluate the cost
4 FAST SPLITTING WITH RAYLEIGH- BAYLEIGH-QUOTIET GRADIENT DESCENT
A practical issue of splitting steepest descent is the high computational cost of the eigen-computation of the splitting matrices. The time complexity of evaluating all splitting indexes is O(nd)). Here is the number of neurons and d is the number of the parameters of each neuron. Meanwhile, the space complexity is ((nd)) Although this is manageable for networks with small or moder- sizes, an immediate difficulty for modern deep networks with thousands of high-dimensional neurons (~ 1000) is that we are not able to store all splitting-marries on GPUs, which necessities slow calculation on CPUs. It desirable further speed up the calculation for very large scale problems. In this section, we propose an approach for computing the splitting indexes and gradients without explicitly expanding the splitting matrices, based on fast (stochastic) gradient descent on the Rayleigh quotient.
Mhayligh-Quotien Gradient Descent for Minimum Eigenvalues The key idea is note that minimum eigenvalues and eigenvectors of a matrix S Rdxdd can be obtained by minimizing the Rayleigh quotient (Parlett 1998),
Amin min Rs(v)=- = T. 15
Umin x arg min Rg(v),
(7)
v T
which can be solved using gradient descent or other numerical methods. Although this problem non-convex, Umin can be shown to be the unique global minimum of R(v). and all the other stationary points, corresponding to the other eigenvectons, are saddle points and can be escaped with random perturbation. Therefore, stochastic or noisy gradient descent on R(v) is expected to converge to Uminn The gradient of R(v) w.r.t. can be written as follows,
V.Rs(u) = ||l|| (Sv (Su- Rs(())) & Su Rs(v)e,
which depends on S only through the matrix-vector product Sv. A significant saving in computation be obtained by directly calculating Su at each iteration, without explicitly expanding the whole matrix. This can be achieved by the following auto-differentiation trick.
Automatic Differentiation Trick Recall that the splitting matrix of a single neuron is S(0) Egww ( ( x)] To calculate S(0)e for any vector v Rd construct the following auxiliary function,
F(n) FE (Va00000))) (0,))() NER"
with which is easy to show that S(A)r F(F Here F(n) corresponds simply adding an extra term on the top of the neuron's output and can be constructed conveniently.
In the case of deep neural networks with n neurons (n
Tē=1, we can calculate all the matrix-vector product {ge = (x ) for all the neurons jointly with a single differentiation process. More precisely, for each neuron 0f, we can add term n/ V (0 a) (denoted as auxiliary activation) on its own output (see Figure 2). Thus, obtain joint function F(71....,, nm), for which it is easy to see that V F(m,........ . mh.) = ge. Vl. Therefore, simply differen- tiating F F(m...., nn) allows to obtain all {gt} simultancously.
auxiliary activation
Stochastic Gradient Rayleigh quotient Note that still need to average over the whole dataset D to measure the Rayleigh quotient gradients {g:). this is computationally expensive in the
Figure 2: Illustration of the auto-differentiation trick with auxiliary activation.
5

---

Under review as a conference paper at ICLR 2020
Algorithm Energy-aware neural architecture optimization with fast splitting steepest descent
Starting from small base network (viewed the "seed"), we gradually grow the neural network by alternating between the following two phases until an energy constrain reached:
1. Parametric Updates: Optimize neuron weights using standard methods (e.g.. SGD) until no further improvement can be made by only updating parameters.
2. Splitting Neurons:
(a) Computing the splitting index of each neuron using gradient-based approximation (Sec- 4):
(b) Finding the optimal set of neurons to split by solving the energy-awarr allocation prob-
lem in Equation (
(c) For each neuron selected above (step 2a), split it into two equally weighted off-springs along their splitting gradients, following Equation (5)
case of big data. However, can conveniently address this by approximating {ge} with subsam- pled bini-batches B. In the case of single-neuron networks, that is,
18 000 x,) + (
Vu=V,f(n with (())
[
Assume we sweep the training data times to train the Rayleigh-Quotient Quotient to convergence (see Equation (7)). In this way, the splitting time complexity for approximating all splitting indexes and gradients would be only (Tdd (7 is often a small constant). More importantly, a significant advantage of our gradient-based approximation is that the space complexity is only O(nd). In this way, all calculation could be efficient performed on GPUs, This given us an algorithm for splitting that is almost as efficient back-propagation.
Overall Algorithm Our overall algorithm in shown in Algorithm T which improves over Wu et al. (2019) by offering much lower time and space complexity, and the flexibility of incorporating energy and other costs of different neurons. It can be implemented easily using modern deep learning frameworks such as Pytorch (Paszke et al. 2017). Our code is available with the submission.
5 EXPERIMENTS
We apply our method to split small variants of MobileNetVI (Howard et al. 2017) and MobileNetV2 (Sandler et al., 2018), on both CIFAR- 10d and ImageNet dataset. We show our method finds net- works that are more accurate and also more energy-efficied compared to expert-designed architec- tures and pruned models.
Settings of Our Algorithm In all our tests of our Algorithm I we restrict the increase of the energy cost to be smaller than budget Chudget at each splitting stage. We set Cbudget adaptively to be proportional to the total flops the current network such that the flops of the augmented network obtained by splitting cannot exceed ratio and set a 0.5 unless otherwise specified.
times of the previous one. We denote by the growth
For our fast splitting indexes approximation (see section 4 RMSprop (Tieleman & Hinton (2012) optimizer with 0.001 learning rate. We find the Rayleigh- Quotient converges fast in general: for small CIFAR-10/100 datasets, train 10 epochs (T=10); for the large-scale ImageNet set, we find a small (=2) is sufficient.
batch size [B]] to be 64 and use
5.1 TESTING IMPORTANCE OF ENERGY- SPLITTING (RESULTS ON CIFAR-10)
To study the importance of our energy-aware splitting, we compare our method (denoted as splitting (energy-award to Wu al. 2019) (denoted splitting (vanilla)), which doesn't 'tt use energy metrics to guide the splitting process. In this experiment, we apply both splitting algorithms to grow a variant
/tt::://wwwwwee. cs. http

---

Under review as a conference paper at ICLR 2020
of small version of MobileNets (Howard et al., 2017) trained on the CIFAR-10 dataset, in order to test the importance of using energy cost for splitting.
Settings We test algorithm two variants of MobileNet, each of which consists one regular 3 x 3 convolution layer, followed by k = 3 and k = 6 MobileNet blocks (Howard et al.) 2017). respectively. In both variants, the resolutions are reduced 3 times evenly and one extra MobileNet block attached with fully connected layer for classification. Note that each MobileNet block con- sists a depthwise convolutional layer and a pointwise convolutional layer. In our implementation, we only split the convolutional filters in the pointwise convolutional layers and duplicate the corre- sponding depthwise convolution filters accordingly during splitting. We start with small networks that have the same number of channels (=8) across all layers to better study the behavior of how neurons are split. We set batch size to be 256 and learning rate 0.1 for 160 epochs, with learning rate dropped 10x at 80 and 120 epochs for the two variants (A 3, 6), respectively.
Results Our results are shown in Figure which shows that our splitring (energy-aware) approach yields better trade-offss of accuracy and flops than splitting (vanilla) in both cases (k: = 3 and k 6)- We find that splitting (vanilla) does discover networks with small model size (fewer parameters, see Figure (b) and (d)), but does not yield lower energy consumption in practice. These results high- light the importance of using real energy cost for guiding the splitting process in order to optimize for the best energy- -nergy-efficiency.
(a) & =
(b) =
(c) k
(d) k 6
0.85
0.85
0.75
0.75
Test Accumey
0.75
0.75
0.65
0.65
Splitting (vanilla)
Splitting tenergy-award
6.1
6.0
llgoo(()pps)
Log (0(pparametees))
Log 10(lopp))))))
Log 10(4parametees)))
Figure 3:
Comparisons between our energy-awar splitting and standard splitting in Wu et al. 2019) on CIFAR-10. Results are shown for two variants of MobileNet, one with & = 3 (4 MobileNet blocks, 9 layers in total), another with & = 6 (7 MobileNet blocks, 15 layers in total).
5.2 RESULTS ON CIFAR-100
We compare our method with several state-of-the-art pruning baselines on the CIFAR-100 dataset. We also show our fast gradient-basee baseed splitting approximation in section 4 achieves the same accu- racy as the exactly digen-computation, while significantly reducing the overall splitting time.
Settings We again apply splitting small version of MobileNet (Howard et al. 2017) (with the same network topology) to obtain a sequence of increasingly large models. Specifically, we set the number of channels of the base model to be 2, 2, 8, 8, 16, 16, 24, 24, 24, 24, 24, 24, 32, 32 for each layer, respectively. We compare our method with simple but competitive width multiplier (Howard) et 2 2017) baseline, which prunes filters uniformly across layers (denoted Width multiplier) from the original full size MobileNet. We also experiment with three state-of-the-a structured pruning methods: Praning (Bn) (Liu et al., (017),, Pruning (LI) (Li al.) 2017) and MorphNet (Gordon et al. 2018). The implementation of all the baselines are based on Liu et al. (2019)). For all methods, we normalize the inputs using channel means and standard deviations. We use stochastic gradient descent with momentum 0.9, weight decay le-4, batch size 128. We set 0. initial learning rate for 160 epochs, with learning rate decreased by 10x epochs 80, 120, respectively. For all pruned models, we report the finetune performance with the same training settings. For Morphnet, we grid search the best sparsity hyper-parameter in the range [le-8, 5e-8, Ie-9, 5e-9, 1e-10] and report the best models found.
Results Figure 4 (a) shows the results on 100, in which our method achieves the best accuracy when targeting similar flops. To draw further comparison between the splitting and pruning

---

Under review as a conference paper at ICLR 2020
0.70
0.7
Text Accurncy
0.65
Baseline size)
Text Accuracy
0.6
Width muttiplier
0.60
Pruning (L:1)
Pruning (Bn)
0.5
MorphNet
Pruning (Bn) Splitting (ours)
0.55
Spliting (ours)
6.5
7.0
7.5
8.0
6.0
6.5
7.0
7,5
8.0
Logl0rllps)
Log Log(0rdops)
Figure 4: (a) Results on CIFAR- 100 using MobileNet Howard et al. 2017); (b) we show our energy -ware splitting approach can learn more accurate and energy-effciint (with small flops) networks than pruning meth- ods (Lau et al. 2017).
approaches, we prune the final network learned by our splitting algorithm obtain a sequence of increasingly smaller models using Pruning (Bn) (Liu et al. 2017). As shown in Figure 4 (b), it is clear that our splitting checkpoints (red circles) form a better flops-aceuracy trade-off curve than models obtained by pruned from the same model (green Pentagons). This confirms the advantage of our method in neural architecture optimization, especially on the low-flops regime.
In Figure (a-b), we examine the ac- curacy and speed of our fast gradient- based iigen-aprrxximatio. We run all methods on server with one V100 GPU and 16 CPU cores and report the wall- clock time. We can see that our fast method (red dots and bars) achieves al- most the same accuracy as the split- ting based on exact cigen-decomposition (blue dots and bars), while achieving significant gain in computational time (see Figure (b)).
Splitting lapprox) Spliting texact)
0.7
10000
Splitting time (s)
8000
Test Accuncy
4000
Splitiing (spprox) Splitting (oxact)
2000
6.0
7.0
7.5
Log (0(Alpp))
(b) dsplit
Figure 5: Comparison of testing accuracy (a) and splitting time
(b) using exact eigen-decomposition (denoted splitting (ex- act)) and our fast gradient- based agee-approximation (denoted as splitting (approx)).
5.3 RESULTS ON IMAGENET
We conduct experiments on large-scale ImageNet dataset, on which our method again shows clear advantages over existing methods. Note that splitting based on exact eigen-composition no longer feasible on ImageNet and our fast gradient-based approximation must be used.
Dataset The ImageNet dataset (Deng et al. 2009) consists of about 1.2 million training images, and 50, 000 validation images, classified into 1, 000 distinct classes. We resize the image size to 224 x 224, and adopt the standard data augment scheme (mirroring and shifting) for training images (e.g. Howard et al. 2017; Sandler et al. 2018).
Settings We choose both MobileNetV1 (Howard et al. 2017) and MobileNetV2 (Sandler al., 2018) as our base net for splitting, which are strong baselines and specifically hand- and heavily tuned to optimize accuracy under a flops-constrain on the ImageNet dataset.
For parametric updates, we follow standard training settings on the ImageNet dataset using Mo- bileNets. Specifically, we train with a batch-size 128 on 4 GPUs (total batch size 512). We stochastic gradient descent with initial learning rate 0.2 and 0.1 for MobileNetVi and Mo- bileNetV2, respectively. We apply cosine learning rate annealing scheduling and use label smooth- ing (0.1) by following (He al. 2019).
For our method, we start with relative small models (denoted by Splitting-O - (seed) by shrinking the network uniformly with width multipler 0.3, and gradually grow the network via energy-aware splitting. We use Splitting-k to represent the model we discovered at the k-th splitting stage. We report the single-center-crop validation error of different models.
8

---

Under review as a conference paper at ICLR 2020
Model
MACs (G)
Top-1 Accuracy
Top-S Accuracy
MobileNetVI (1.0x)
0.569
72.93
91.14
Splitting-4
0.561
73.96
91.49
MobileNetV1 (0.75x)
0.317
70.25
89.49
AMC (He et al. 2018
0.301
70.50
89.30
Splitting-3
0.292
71.47
89.67
MobileNetVI (0.5x)
0.150
65.20
86.34
Splitting-2
0.140
68.26
87.93
Splitting-1
0,082
64.06
85.30
Splitting-O (seed)
0.059
59.20
81.82
Table 1: Results of ImageNet classification using MobileNetV1. Splitting-k: denotes the model
discovered at the k-th splitting stage. Our method yields networks with both higher accuracy and lower number muliply-and-accumulat (MAC) operations.
Model
MACs (G)
Top-1 Accuracy
Top-5 Accuracy
MobileNetV2
0.300
72.04
90.57
Splitting-3
0.298
72.84
90.83
MobileNerV2 (0.75x)
0.209
69.80
89.60
AMC (He et al. 2018
0.210
70.85
89.91
Splitting-2
0.208
71.76
90.07
MobileNetV2 (0.5x)
0.097
65.40
86.40
Splitting-1
0.095
66.53
87.00
Splitting-O(seet
0.039
55.61
79.55
Table 2: Results on ImageNet using MobileNetV2. Splitting-k denotes the model we discovered at the k-th splitting stage. MAC denotes the number of multpiyy-nnd-accummllae operations.
MobileNetV1 Results Table we find that our method achieves about 1% top-1 accuracy
improvements in general when targeting similar flops. On low-flops regime (< 0.15G flops), our method achieves 3.06% higher top-1 accuracy compared with MobileNet (0.5X) (with width mul- tiper 0.5). Also, the model found by method is 0.97% higher than prior art pruning method (AMC (He et 2018)) when comparing with checkpoints with 0.3G flops.
MobileNetV2 Results From table we find that our splitting models yield better performance compared with prior art expert-designed architectures on all flops-regimes. Specially, out splitting- 3 reaches 72.84 top-1 accuracy; this yields an 0.8% improvement model. On the low-flops regime, our splitting-2 achieves an 1.96% top-1 accuracy improvement over MobileNetV2 (0.75x); splitting-/ is 1.1% higher than MobileNetV2 (0.5x). Our performance is also about 0.9% higher than AMC when targeting 70% flops.
its corresponding baseline
5.4 ABLATION STUDY
In our algorithm. the growth ratio a controls how many neurons we could split at each splitting stage. In this section, perform in-depth analysis of the effect of different values. We also examine the robustness of our splitting method regarding randomness during the network training and splitting (c.g. parameters initializations, data shuffle).
Impact of growth ratio To find the optimal growth ratio a, we ran multiple experiments with different growth ratio a under the same settings as section 5.2 Figure (a) shows the performance of various runs. We find that the growth ratio in the range of [0.3, 0.5) tend to perform similarly well. However, the smaller growth ratio of a 0.2 tends to give lower accuracy, this may be because with small growth ratio, the neurons in the layers close to the input may never be selected because of their higher energy cost for splitting, hence yielding sub-optimal networks.

---

Under review as a conference paper at ICLR 2020
Robustness We apply method to grow a small MobileNet (Howard et al, 2017) using different ran- dom seeds for parameters initializa- tion and data shuffle under the same setting as Figure ( with growth ratio 0.5. Figure 6 (b) shows the
Test Test Accuncy
0.6
random trial random tiral
Growth ratio (0.3)
Growth ratio (0.4)
0.4
random trial random
ratio (0.5)
6.0
6.5
7.0
7.5
8.0
6.0
6.5
7.0
7.5
8.0
test performance of different models learned. As we can see from Fig- ure [6] (b), all runs perform similarly well with small variations.
(a) Logl0/fopp))
(b) Logtofflops)
Figure 6: Comparison on the test accuracy for various ab- lation study settings.
RELATED WORK
Neural architecture search (NAS) has been found a powerful tool for automating energy-efficient architecture design. Most existing NAS methods are based on black-box optimization techniques, including reinforcement learning (e.g. Zoph & Le 2017 Zoph et al. 2018) evolutionary algorithms (e.g. Real et al. 2019 2017). However, these methods
often extremely time-consuming due to the enormous search space possible architectures and the high cost for evaluating the performance of each candidate network. More recent approaches have made the search more efficient by using weight-sharing (e.g. Pham et al., (2018) Liu et al.] 2019a Cat et al. (019),, which, however, suffers from the so-called multi-model forgetting problem (Benyahia et (209) that causes training in- stability and performance degradation during search. Overall, designing the best architectures using NAS still requires a lot of expert knowledge and trial-and-errors.
In contrast, pruning- based methods construct smaller networks from pretrained wver-parameterized neural network by gradually removing the least important neurons. Various pruning strategies have been developed based on different heuristics (e.g.. Han et al.) 2016 Li et al. 2017 Luo et al,, 2017 He al. 2017b Peng al.) 2019), including energy- pruning methods that use energy consumption related metrics to gudde the pruning process (e.g., Yang et al.] 2017 Gordon et al.)
2018 He al. 2018: Yang et al. 2019). However, a common issue of these methods to alter the standard training objective with sparsity-induced regularization which necessities sensitive hyper- parameters tuning. Furthermore, the final performance is largely limited by the initial hand-crafted network, which may not be optimal in the first place.
7 CONCLUSIONS
In this work, we present a fast energy-aware splitting steepest descent approach for resource-efficient neural architecture optimization that generalizes WW et al. (2019). Empirical results on large-scale ImageNet benchmark using MobileNetV1 and MoibileNetV demonstrate the effectiveness of our method.
REFERENCES
Yassine Benyahia, Kaicheng Yu, Kamil Bennani-Smires, Martin Jaggi, Anthony Davison, Mathieu Salzmann, and Claudiu Musat. Overcoming multi-model forgetting. In Proceedings of the 36th International Conference on Machine Learning (ICML), 2019.
Han Cai, Ligeng Zhu, and Song Han. ProxylessNAS: Direct neural architecture search on target task and hardware. In International Conference on Learning Representations (ICLR), 2019.
George Bernard Dantzig. Linear programming and extensions. Princeton university press, 1998.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: large-scale hicrarchical image database. In 2009 IEEE conference computer vision and pattern recognition (CVPR), pp. 248-255. IEEE, 2009.
Jacob Devlin, Ming- Wei Chang. Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Annual Conference of the North Amer-
10

---

Under review as a conference paper at ICLR 2020
ican Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) 2018.
Ariel Gordon, Elad Eban, Ofir Nachum, Bo Chen, Hao Wu, Tien-Ju Yang, and Edward Choi. Mor- phnet: Fast simple sssurce-constraaie structure learning of deep networks. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition (CVPR), pp. 1586-1595, 2018.
Song Han, Huizi Mao, and William J Dally. Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding. International Conference on Learning Representations (ICLR), 2016.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference computer vision and pattern recognition (CVPR), pp. 770-778, 2016.
Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cnn. In Proceedings the IEEE international conference on computer vision and pattern recognition (CVPR), pp. 2961- 2969, 2017a.
Tong He, Zhi Zhang, Hang Zhang, Zhongyue Zhang, Junyuan Xie, and Mu Li. Bag of tricks for image classification with convolutional neural networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 558 0888567,57 2019.
Yihui He, Xiangyu Zhang, and Jian Sun. Channel pruning for accelerating very deep neural net- works. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), pp- 1389-1397, 2017b.
Yihui He, Ji Lin, Zhijian Liu, Hanrui Wang, Li-Jia Li, and Song Han. Ame: Automl for model compression and acceleration on mobile devices. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 784-800, 2018
Andrew Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam. Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv: 700.7744.444.. 2017.
Gao Huang. Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. Densely connected convolutional networks. Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR), pp. 4700-4708, 2017.
Hao Li, Asim Kadav, Igor Durdanovic, Hanan Samet, and Hans Peter Graf. Pruning filters for efficient convnets. International Conference on Learning Representations (ICLR), 2017.
Hanxiao Liu, Karen Simonyan, and Yiming Yang. DARTS: Differentiable architecture search.
ternational Conference on Learning Representations (ICLR), 2019a.
Zhuang Liu, Jianguo Li, Zhiqiang Shen, Gao Huang, Shoumeng Yan, and Changshui Zhang, Learn- ing efficient convolutional networks through network slimming. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), pp. 2736-2744, 2017.
Zhuang Liu, Mingjic Sun, Tinghui Zhou, Gao Huang, and Trevor Darrell. Rethinking the value of network pruning. International Conference on Learning Representations (ICLR). 2019b.
Jaan-Hao Luo, Jianxin Wu, and Weiyao Lin. Thinet: A filter level pruning method for deep neural network compression. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), pp. 5058-5066, 2017.
Beresford N Parlett. The symmetric eigenvalue problem, volume 20, siam, 1998.
Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. 2017.
11

---

Under review as a conference paper at ICLR 2020
Hanyu Peng. Jiaxiang Wu, Shifeng Chen, and Junzhou Huang. Collaborative channel pruning for
deep networks. In International Conference on Machine Learning (ICML), pp. 5113-5122, 2019.
Hieu Pham, Melody Y Guan, Barret Zoph, Quoe Le, and Jeff Dean. Efficient neural architecture search via parameter sharing. In Proceedings of the 35th International Conference on Machine Learning (ICML), 2018.
Esteban Real, Sherry Moore, Andrew Selle, Saurabh Saxena, Yutaka Leon Suematsu, Jie Tan, Quoc Le, and Alexey Kurakin. Large-scale evolution of image classifiers. 2017.
Esteban Real, Alok Aggarwal, Yanping Huang, and Quoe V Le. Regularized evolution for image classifier architecture search. In Proceedings of the AAAI Conference on Artificial Intelligence (AAAI), volume 33, pp. 4780-4789, 2019.
David E Rumelhart, Geoffrey E Hinton, Ronald Williams, et al. Learning representations by back-propagating errors. Cognitive modeling, 5(3):1 1988.
Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chine Shig--hhihhaee Chen. Mo- bilenetv2: Inverted residuals and linear bottlenecks. In Proceedings of the IEFE Conference Computer Vision and Pattern Recognition (CVPR), pp. 4510-4520, 2018.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. In International Conference on Learning Representations (ICLR), 2015.
Tijmen Tieleman and Geoffrey Hinton. Lecture 6.5-rmsprop: Divide the gradient by running average of its recent magnitude. COURSERA: Neural networks for machine learning, 4(2):26- 31, 2012.
Lemeng Wu, Dilin Wang, and Qiang Liu. Splitting steepest descent for progressive training of neural networks. Neural Information Processing Systems (NeurIPS), 2019.
Haichuan Yang, Yuhao Zhu, and Ji Liu. Ecc: Plafform-independentt energy-constrained deep neural network compression via a bilinear regression model. The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2019.
Tien-Ju Yang, Yu-Hsin Chen, and Vivienne Sze. Designing energy-efficient convolutional neural networks using energy-awar pruning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 5687-5695, 2017.
Barret Zoph and Quoe Le. Neural architecture search with reinforcement learning. Proceedings of the 35th International Conference Machine Learning (ICML), 2017.
Barret Zoph, Vijay Vasudevan, Jonathon Shlens, and Quoc V Le. Learning transferable architectures for scalable image recognition. In Proceedings of the IEEE conference on computer vision and paltern recognition (CVPR), pp. 8697-8710, 2018
12


Reviews:
[{"review_id": "rJg4nZPUjr", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper is well written and flows very well. The idea is straightforward and easy to understand. Here are my feedbacks:\n\n--Method--\n\nMethodology-wise, the novelty is somewhat limited. The main technical contributions are: 1) formulate linear programming to reduce energy costs, and the formulation of linear programming is straightforward. 2) the claimed main contribution is an application of Rayleigh-Quotient gradient descent to approximate the eigenvalue calculations in the model proposed by Wu et al (2019). \n\nI believe computing the eigenvalue shall not be the only way to solve Eq.(3), e.g. using gradient-based method or Lagrange. My main question is whether computing the global optimum really matters? Since your method is still essentially an approximation algorithm, which may break the optimality condition here. From the experimental results, it seems that your approximation is quite close to the analytical solution. Therefore, it is unclear computing a global optimum really matters here.\n\nI tried to buy the idea of escaping the saddle point with splitting (it indeed sounds straightforward and reasonable). It will be great if the author can add some empirical experiments to verify it.\n\n--Content--\nThe entire section2 is at reviewing prior works, and I believe you should cut down the content here.\n\n--Experiments--\n1. Diversity of your tests: the author main uses MobileNet in testing their ideas. It seems that their method starts at MobileNet, then tries to improve it. I suggest authors adding other recent works, e.g. FBNet, MobileNetV3, into their tests to diversify the types of networks. \n\n2. Results variations: all figures, e.g. fig.3, fig.4, in the paper lack plotting their results variations. Since the optimization may converge to different local optimum, it is more persuasive to show their performance variations.\n\n3. The final results are not surprising: in table.1 and table.2, as far as I'm aware, the mainstream accuracy for ImageNet under the mobile setting should be 75% top-1. And the SOTA top-1 accuracy on ImageNet is around 85.5%. Table.1 and table.2 do somewhat show the effectiveness of your method, but its significance is limited especially considering the limited novelty of methodology.\n\nMinors:\nIn Fig.3 k = 6, it seems vanilla splitting is better than accuracy and parameters, except for 0.2 log higher flops. I don't think this makes a compelling case here.\n\nIn Fig.4, from flops 7 ~ 9, your results are similar to Bn especially accuracy > 0.6. When accuracy < 0.6, it is less interesting, and I believe the improvement should be huge.\n\nFig.5, could you please compare the time using MAGMA from NVIDIA? I had some experiences in implementing the LAPACK and BLAS on GPUs, and it should not be that slow.\n\nOverall, this paper has some interesting results, which shows the eigenvalue can be approximated by Rayleigh-Quotient gradient descent, and show positive improvement. However, the methodological and experimental results can definitely be strengthened. The author may consider proving that achieving the global optimum on Eq.(3) really matters, then motivate the methodology. Thank you. \n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "SkxvmpoAYB", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper is based on a prior work which proposed Splitting Steepest Descent to search better network structures via splitting existing neurons to multiple off-springs. As an improvement, the authors (1) incorporate the energy cost to better guide the splitting process and (2) reduce the time and space complexity by approximating the original computation process with Rayleigh-Quotient Gradient Descent. They conduct experiments on public image classification datasets using lightweight networks as backbones to show that their algorithm outperforms existing methods. The paper is well written and easy to follow. The experiments are comprehensive and the evaluation results shows good properties of proposed method.\n\nIn brief, this paper is an improvement to a splitting algorithm in a previous work, achieving good efficiency and enabling application on large datasets. However, the theory needs more justification and the experiments results are not sufficient to show the significance of their contribution. Therefore, this paper may not be accepted unless more experiment results are given.\n\nFor the theory & modeling, the following should be addressed.\n1.\tThe mathematical justification of the optimization on energy cost is not very sound, and the definition of optimal splitting set seems arbitrary.\n2.\tGiven that the experiments are conducted on convolutional networks, it would be more illustrative if the paper describe the process of applying the algorithm on common convolutional operators.\n3.\tThe novelty is limited by the prior work.\n\nFor the experiment, the following should be addressed.\n1.\tThe experiments are mainly conducted on MobileNet network. It would be more convincing if more experiment is done on other lightweight or normal convolutional networks.\n2.\tThis paper reduces time and space complexity of the algorithm in a previous work, but there is no running time or memory footprint statistics to support this argument.\n3.\tThe paper only lists one pruning-based method as comparison in the experiments. It would be more convincing if more pruning and splitting methods are presented.\n", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "B1evkL2EqH", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\nThis paper builds on a recently proposed algorithm (\"splitting steepest descent\", Wu et al 2019) for guiding the growth of a smaller network into a larger one in architecture search. The algorithm in Wu, et al. alternates between two steps, (i) optimization of parameters for a fixed model and (ii) modification of the architecture by identifying a subset of neurons to split into more neurons, based on the \"splitting index\" of each neuron (amounting to evaluating the smallest eigenvalue of a matrix). This work builds on that in two ways: (i) it incorporates an energy budget into the optimization procedure for choosing which subset of neurons to split, which it approximately solves by a continuous relaxation, and (ii) avoids doing exact eigendecomposition to extract the minimum eigenvalue (splitting index) but instead replaces it with a more efficient SGD on the Rayleigh quotient. \n\nEvaluation:\n--Evaluations are done on variants of MobileNet on CIFAR-100 and ImageNet (the latter would be infeasible without the approximation scheme). The proposed approach appears to get better tradeoff between accuracy and FLOPs in these cases. In practice the non-energy aware \"vanilla\" networks do tend towards models that are small in size (fewer parameters) but are not necessarily low in energy consumption.\n--There is new material here, although I find the novelty a bit limited (e.g. only an additional constraint compared to the original approach of Wu et al and addressing a clear scalability issue with the original work, i.e. eigendecomposition of a matrix, with what seem straightforward approximations, ). The empirical results in Table 1 and 2 seem solid, but I'm not familiar enough with past results in this area  to evaluate their significance. ", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I did not assess the derivations or theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper extends previous work on searching for good neural architectures by iteratively growing a network, including energy-aware metrics during the process. There was discussion about the extent of the novelty of this work and how well it was evaluated, and in the end the reviewers felt it was not quite ready for publicaiton.
Decision: Reject