Balancing training time vs. performance with Bayesian Early Pruning
Authors: Mohit Rajpal
Yehong Zhang
Bryan Kian Hsiang Low
Venue:  (2021)
Keywords: Efficient Training
Multi-Output Gaussian Process
Gaussian Process
Bayesian
Single-shot network pruning
Dynamic Sparse Reparameterization
Lottery Ticket Hypothesis

Abstract: Pruning is an approach to alleviate overparameterization of deep neural networks (DNN) by zeroing out or pruning DNN elements with little to no efficacy at a given task. In contrast to related works that do pruning before or after training, this paper presents a novel method to perform early pruning of DNN elements (e.g., neurons or convolutional filters) during the training process while preserving performance upon convergence. To achieve this, we model the future efficacy of DNN elements in a Bayesian manner conditioned upon efficacy data collected during the training and prune DNN elements which are predicted to have low efficacy after training completion. Empirical evaluations show that the proposed Bayesian early pruning improves the computational efficiency of DNN training with small sacrifices in performance. Using our approach we are able to achieve a $48.6\%$ faster training time for ResNet-$50$ on ImageNet to achieve a validation accuracy of $72.5\%$.

Under review as a conference paper at ICLR 2021
BALANCING TRAINING TIME VS. PERFORMANCE WITH BAYESIAN EARLY PRUNING
Anonymous authors Paper under double-blind review
ABSTRACT
Pruning is an approach to alleviate overparameterization of deep neural network (DNN) by zeroing out or pruning DNN elements with little to efficacy at given task. In contrast to related works that do pruning before or after training, this paper presents a novel method to perform early pruning of DNN elements (e.g., neurons or convolutional filters) during the training process while preserving performance upon convergence. To achieve this, model the future efficacy of DNN elements in a Bayesian manner conditioned upon efficacy data collected during the training and prune DNN elements which are predicted to have low efficacy after training completion. Empirical evaluations show that the proposed Bayesian early pruning improves the computational efficiency of DNN training. Using our approach we are able to achieve 48.6% faster training time for ResNet-50 on ImageNet to achieve a validation accuracy of 72.5%.
INTRODUCTION
Deep neural nelworks (DNNs) are known to be overparameterized (Allen-Zhu et al. 2019) as they usually have more learnable parameters than needed for given learning task. So, trained DNN contains many ineffectual parameters that can be safely pruned or zeroed out with little/no effect on its predictive accuracy. Pruning (LeCun et al.) 1989) an approach to alleviating overparameterization of a DNN by identifying and removing its ineffectual parameters while preserving its predictive accuracy on the validation/test dataset. Pruning typically applied to the DNN after training to speed up testing-time evaluation. For standard image classification tasks with MNIST, CIFAR-10, and ImageNet datasets, can reduce the number of learnable parameters by up to 50% or more while maintaining test accuracy (Han al. 2015 Li et al. 2017 Molchanov al. 2017).
In particular, the overparameterrizttoo of a DNN also leads to considerable training time being wasted those DNN elements (c.g.. connection weights, neurons, or convolutional filters) which are eventually ineffectual after training and can thus be safely pruned. Our work in this paper considers early pruning of such DNN elements by identifying and removing them throughout the training process instead of after training ! training process without compromising the final test accuracy (upon convergence) much.
As a result, this can significantly reduce the time incurred by the
Recent work (Section 5) in foresight pruning (Lee et al. heuristics applied initialization work well prune connection weights without significantly degrading performance. In contrast to these work, we prune throughout the training procedure, which improves performance after convergence of DNNs, albeit with somewhat longer training times.
2019 Wang et al. 2020) show that pruning
In this work, we pose early pruning challenge in the optimization is accurately modeling the future efficacy DNN elements. We achieve this through the use of multi-oulpul Gaussian process which models the belief of future efficacy conditioned upon efficacy measurements collected during training (Section (.2p. Although the posed optimization problem is NP-hard, we derive an efficient Bayesian early pruning (BEP) approximation algorithm, which appropriately balances the inherent training time vs. performance tradcoff in pruning prior to convergence (Section 3.3). Our algorithm relies measure of network element efficacy, termed saliency (LeCun et al. 1989). ). The development of saliency functions an active area of research with no clear optimal choice. To accomodate this, algorithm is agnostic, and therefore
constrained optimization problem (Section 3.1 A key
In contrast, foresight pruning WWag al. 2020) removes DNN elements prior the training process.

---

Under review as a conference paper at ICLR 2021
flexible, to changes in saliency function. We use BEP to prune neurons and convolutional filters to
achieve practical speedup during training (Section 4
Our approach also compares favorably to of-h-of-theeart works such as SNIP (Lee et al. 2019) GraSP (Wang al. 2020 and momentum
based dynamic sparse reparameterization (Dettmers & Zettlemoyer 2019).
2 PRUNING
Consider a dataset of D training examples X' = fx,,..., .D}.y f.. and neural network Ne, parameterized by a vector of M pruneable network elements (e.g. weight parameters, neurons, or convolutional filters) v, _ [=f]a=1.....Mr where e, represent the network elements after t iterations of stochastic gradient descent (SGD) for t I,..., T. Let C(X, V:W., be the loss function for the neural network pV,, Pruning aims at refining the network elements v, given some sparsity budget B and preserving the accuracy of the neural network after convergence (i.c., Nep.) which can be stated as a constrained optimization problem (Molchanov et al. 2017)
(0.1 ae C(X
((X)): s.t. ||m|lo S B
(1)
where is the Hadamard product and m is a pruning mask. Note that we abuse the Hadamard product for notation simplicity: for = I,.., .,, m x v. corresponds to pruning of if m 0. and keeping of otherwise. Pruning a network element refers to zeroing the network element or the weight parameters which compute the network element. Any weight parameters which reference the output of the pruned network element are also zeroed since the element outputs a constant 0.
The above optimization problem is difficullt due to the NP-hardness of combinatorial optimization. This leads to the approach of using saliency function s which measures efficacy of network elements at minimizing the loss function. A network element with small saliency can be pruned since it's not salient in minimizing the loss function. Consequently, pruning can be done by maximizing the saliency of the network elements given the sparsity budget B:
m Ees m" s(a; X, K,V, C C) s.t. ||mm/lo S B
(2)
where s(a; X, V.V.Nar C) measures the saliency of os minimizing C after convergence through iterations of SGD. The above optimization problem can be efficienctly solved by selecting the B most salient network elements in uy.
The construction of the saliency function has been discussed in many existing works: Some ap- proaches derived the saliency function from first-order (LeCun et al. 1989 Molchanov al. 2017)
and second-order (Hassibi & Stork 1992: Wang et al. 2020) Taylor series approximations
L. Other common saliency functions include L1 (Li et 2017) La (Wen et (016) norm of the network element weights, as well as mean activation (Polyak& & Wolf 2015). In this work, we use first-order Taylor series approximation saliency function defined for neurons and convolutional filters 33 (Molchanov et al. (2017), however our approach remains flexible to arbitrary choice of saliency function on a plug-n-play basis.
3 BAYESIAN EARLY PRUNING
3.1 PROBLEM STATEMENT
As has been mentioned before, existing pruning works based on the saliency function are typically done after the training convergence (i.c., (2) to speed up the testing-time evaluation, which waste considerable time on training these network elements which will eventually be pruned. To resolve this issue, We extend the pruning problem definition (2) along the temporal dimension, allowing network elements to be pruned during the training process consisting of T iterations of SGD.
PPopular deep learning libaries do not accelerate sparse matrix operations over dense matrix operations. Thus, pruning network connections cannot be easily capitalized upon with performance improvements. It is also unclear whether moderately sparse matrix operations (i.e., operations matrices generated by connection pruning) can be significantly accelerated on massively parallel architectures such as GPUs (see Yang et al. (018) Fig. 7). See Section 5 in Buluç & Gilbert (2008) for challenges in parallel sparse matrix multiplication. "mpllmmntatton details of this saliency function can be found in Appendix A.1
2

---

Under review as a conference paper at ICLR 2021
Let 57 @ after i iterations of SGD, 8, - [6f],=1.....v for t = ....., T, and 87:;7= - [st]e=rr.....7 be vector of saliency of all the network elements between iterations 7i and T2. Our early pruning algorithm is designed with the goal of maximizing the saliency of the unpruned elements after iteration T,, yet allowing for pruning at each iteration & given some computational budget Bi,e for t = l.....??
L) be random variable which denotes the saliency of network element of
pr(mTr-, Br,e, BB) - maxmr MT ST (3a) s.t. || S B, (3b) mT 2 MT- (3c) By 2 ((d))
P B,) maxm, ((( (m, 1 B.)] (4a)
s.t. m4 S m (4b)
where B, is the trained network sparsity budget, $1.t is vector of observed values for $1:6, mo is an M M-dimensional I's vector, and I m,-1 represents an element-wis comparison between m, and m,-1. 1: mg S m for a = I,... M. At each iteration L, the saliency 8, is observed and m, E [0, in o represents a pruning decision performed to maximize the expectectation Pt+1 conditioned upon saliency measurements $1.1 collected up to and including iteration f. This recursive structure terminates with base case pr where the saliency of the unpruned elements is maximized after T iterations of training.
In the above early pruning formulation constraints (3c) and (4b) ensure pruning is performed in practical manner whereby once network element pruned, can no longer be recovered in a later training iteration. We define a trained network sparsity budget, B, (3b). which may differ significantly from initial network size |/mollo (e.g., in the case where the network is trained on GPUs, but deployed on resource constrained edge or mobile devices). We also constrain a total computational effort budget B.,. which is reduced per training iteration by the number of unpruned network elements ||m,loo- We constrain BT 2 0 (3d) to ensure training completion within the specified computational budget. Here we assume that a more sparse pruning mask m, corresponds to lower computational effort during training iteration / due to updating fewer network elements. Finally, (Ba) maximizes the saliency with pruning mask my constrained by a sparsity budget B, (3b). Our early pruning formulation balances the saliency of network elements after convergence against the total computational effort to train such network (i.e., My 8T vs. I (emilmmmllo)) (o0) This appropriately captures the balancing act of training-timy early pruning whereby the computational effort saved by early pruning network elements while preserving the saliency of the remaining network elements after convergence.
3.2 MODELING THE SALIENCY WITH MULTI-OUTPUR GAUSSIAN PROCESS
To solve the above early pruning problem, need to model the belief p p(s1-T of the saliency for computing the predictive belief ((141:T|5 of the future saliency in (4a). At the first glance, one may consider to decompose the belief: p(S1:r) & [T=_1 and model the saliency sir & [s!] of each network element independently. Such independent models, however, ignore the co-allptation and co-evolution of the network elements which have been shown to be common occurrence DNN (Hinton et al. 2012 Srivastava et al., 2014 Wang al. 2020). Also, modeling the correlations between the saliency of different network elements explicitly is non-trive non-trivial since considerable feature enginecring is needed for representing diverse network elements such as neurons, connections, convolutional filters.
p(s):r)
To resolve such issues, we use multi-outpe Gaussian process (MOGP) to jointly model the belief p($1:T) of all saliency measurements. To be specific, assume that the saliency sf of the a-th network element at iteration is a linear mixture of Q independent latent functions {t)
Q st Σg= 1 (t)). As shown in (lvaree & Lawrence 2011). if each uy(() is an independent GP with prior zero mean and covariance kt, f"), then the resulting distribution over p(8 is multivariate Gaussian distribution with prior zero mean and covariance determined by the mixing
'In contrast to PruneTrain (Lym et al. 2019), our problem definition balances training time vs. performance under an additional constraint on the trained network size (3b). We discuss this further in Section 5
Among the various types of MOGPs (see Alvarez & Lawrence [2011 for a detailed review.). we choose this linear model such that the correlations between 8f and a can he computed analytically.
3

---

Under review as a conference paper at ICLR 2021
weights: covist, s41 --==) 7874 (g This explicit covariance between sf and sy helps to exploit the co-evolution and co-adaptation of network elements within the neural networks.
To capture the horizontal asymptote trend of si...., 8q as visualized in Appendix A.2 we turn to a kernel used for modeling decaying exponential curves known as the "exponential kernel" (Swersky 8,"4
al. 2014) and kg (t,t') (1++'+9,)00 where ag and Bg are hyperparameters of MOGP and can
be learned via maximum likelihood estimation (Álvarez& & Lawrence 2011). Then, given vector of observed saliency S1.,. the MOGP regression model can provide Gaussian predictive distribution for any future saliency Sy, Thus, the predictive mean pi - [B[s S1:e] of the saliency sp and the predictive (co)variance gao' - cov[sft, sg. [ between the saliency sp. and sy can be computed d'(1: analytically, as detailed in Appendix A.3
3.3 EARLY PRUNING ALGORITHM
Solving the above optimizing problem 3 and (4) difficult due to the interplay between (melu=....... (Be.cle= and mr 8T. Instead, we consider simplification of the above problem by only considering solutions of the form mT-1 mT
m, which yields
Pe m, Bt.c. B.) A maxm, Ep(ar(Bna) (pr Bt,c- (T - m)]|mmllo. B.)l
(5)
This approach allows us lift 33d from (3), to which add Lagrange multiplier and achieve:
(m, Br B.) e maxm; Ep(or(ima) [ār(m,, B.)] + A (Br,c -T' mlloo (6)
for f ...... T and ÔJ is defined pr without constraint (3d). Consequently, such a ÅT can be solved in a greedy manner as in 2 Afterwards. we will omit B,c as parameter of fr as it no longer constrains the solution of fr. Note that the presence of an additive penalty in a maximization problem is due to the constraint Br,- 2 0 + -BT, S O which is typically expected prior to Lagrangian reformulation. The above optimization problem remains NP-hard as B,)] is submodular in m. (see Appendix B Although greedy approximations
(p(er(au.)) (m,, Epter(a..)lérr(mm exist for submodular optimization, their running time of O m1mml 1l:) remains far too slow due to the large number of network elements in DNNs. Fortunately, it can be significantly simplified by exploiting the following lemma (its proof is in Appendix C)::
Lemma 1. Let ell) be an M-dimensional one-hot vectors with the i-th element be 1. A 1 S a, S M; m E {0, 1 s.l. m A (ela) V e( = 0. Given a vector of observed saliency 81st if 19 2 "Ti and p 2 then
Epler(Bia (Pr (m V e e Ep(ar ler (m V e( S prip 4(v/0) 0 6(v/0)
where 0 A 199 + bb 20 v A phipa Mippa and & and ó are standard normal
CDF and PDF, respectively.
Here, 'V" and 'A' represent bitwise OR and AND operations, respectively. The bitwise OR operation used to denote the inclusion of ela) or e(b) in m.. Due to the strong tail decay of & and $ Lemma indicates at most marginal possible improvement provided by opting for m. = m V (b)
Ff
as opposed to m. m e(a) given p 2 p.
Lemma 1 admits the following approach to optimize d:: starting with m, = OM, we consider the inclusion of network elements m, by the descending order of (1 1M which can be computed analytically using MOGP. A network element denoted by e(a) is included in m, if it improves the objective in (5). The algorithm terminates once the highest not-yet-included element does not improve the objective function as a consequence of the penalty term outweighing the improvement in Ep(er]s...) [Pr]. The remaining excluded elements are then pruned.
Following the algorithm sketch above, we define the utility of network element of with respect to candidate pruning mask m which measures the improvement in (p(er(B...)(rr)
"We omit 4b) as it is automatically satisfied due to our simplification.
NNote pa 2 Prpa: +o.) S 0.5 and experiences tail decay proportional to po - 1rpa
4

---

Under review as a conference paper at ICLR 2021
consequence of inclusion of e(*) in my:
Aa,, m, S1:1 B.) - Batr)(aaa) [pr(e(() V m,, B.) pr B.)]
(7)
We can now take Lagrangian approach pruning decisions during iteration & by balancing the utility of network element of against the change of the penalty (i.e., X(T - t)) in Algorithm Due the relatively expensive cost of performing early pruning, we chose early prune every L'step iterations of SGD. Typically Tstep was chosen to correspond to 10-20 epochs of training. To compute (-.) we sampled from p( (1:1) and used a greedy selection algorithm per sample in ( During implementation, we also enforced an additional hard constraint |m:/lo- 2 B, which we believe is desirable for practicality reasons. We used fixed value of Bi,- = =Immallo + B3 To) in all our experiments.
Algorithm 1 Bayesian Early Pruning
Require: N, U1. To, Tatep. T, Bi,e, Bar
DNN N. Lagrangian penalty À > Train for To iterations to create seed dataset. D Track computational effort expenditure. Early prune every Tstep iterations from To-
1: Su: f train(Ne, - TT)
2: Bi B1,e - Too dim(
3: for k- 0,.... Tato
T-I=; 4- To step do
4: PT T11: + MOGP(S
D Train and perform inference. > Sort descending. Initial pruning mask.
5: ST + Lurgorr(-Huap)
6: m. t (dim(v))
7:
for f 87 sy...., ST dvs) do
Consider each network element.
8:
if Bt,c - (T- m > 0 then
> Remaining B budget can support training vf
9:
= V ela)
else Aa,, m.; St.t B.) 2 X(T - -1) then Balance utility against change of penalty,
m, = m, ela)
12:
else
13:
break
14:
prune m.)
dim(v.) is reduced here.
15:
Bu+Toup +- Br.c - Tatepllmells lleeell
16:
S41144T,uo train (Nog Tstop)
Continue training with pruned network.
17:
return, N
4 EXPERIMENTS AND DISCUSSION
We evaluate our modeling approach as well as our BEP algorithm on the CIFAR-10, CIFAR- 100 (Krizhessky] 2009), and ImageNet (Deng et al. 2009) datasets. For CIFAR-TOCCIFAR-10 we used a benchmark Convolutional Neural Network (CNN) with convolutional lavers, and dense layer For ImageNet validated on the ResNet-50 architecture (He et al. 2016a).
Due to the cubic time complexity of MOGPs, we used a variational approximation (Hensman et al. 2015). In all of our models, used 60 variational inducing points per latent function. We used GPFlow library (Matthews et al. 2017) to build our models.
4.1 MODELING EVALUATION
A key assertion in our approach is the importance of capturing co-adaptation and co-evolution effects in network elements. To verify our MOGP approach captures these effects, we compare MOGP vs. GP belief modeling where GP assumes independence in saliency measurements across network elements (i.c., p(S1T) & " p(s
A dataset of saliency measurements of convolutional filters and neurons was constructed by instru- menting the training process of our 5-layer CNN on the CIFAR-10/CIFAR-100 dataset. Keras (Chollet 2015) was used to train this model over 150 epochs
*Code available at http:////sh.....o.ooaaaaaaaaaaaa
c1 farl 1 [faall__cnn.p 0_onn_ccnyyy
Complete experimental setup details found in Appendix G.2
5

---

Under review as a conference paper at ICLR 2021
200
206
330
400
100
220
300
4.00
130
200
380
400
200
200 300
4000
100
230
300
400
300
230
300
460
Figure 1: Visualization of qualitative differences between GP and MOGP prediction. Top: GP, Bottom: 18-MOGP. Dataset is separated into training (green) and validation (blue). Posterior belief of the saliency is visualized as predictive mean (red line), and 95% confidence interval (error bar).
Comparing log likelihood (standard error) of test data for independent GPs (GP) vs. MOGP with n latent functions (n-MOGP) on collected saliency measurements from CIFAR-100 training. Measurements are given multiple of -10 (lower is better). MOGP outperforms GP, particularly on the small dataset. Results are averaged over 20 runs. Extremely large values are due to the GP model being unable to fit the data.
Table I:
Small dataset
Medium dataset
Large dataset
Lyr
Lyr
Lyr 3
Lyr
Lyr
Lyr
Lyr
Lyr
Lyr
GP 0.75(0.06)
5.7(5.7)e4
5.6(5.6)e4
0.64(0.04)
0.70(0.04)
2.13(0.05)
3.4(3.4)e3
(U.0()
1.06(0.02)
4-MGGP
0.79(0.05)
0.98(..12)
3.13(0.10)
0.44(0.04)
0.60(0.10)
2.29(0.06)
0.12(0.01)
0.24(0.03)
1.07(0.03)
8-MOGP 0.65(0.05)
(.891
3.00(0.09)
0.38(0.04)
0.60(0.10)
2.20(0.06
0.10(0.01)
0.18
1.02(0.03)
18-MGGP 0.62
(.84(..11)
2.93/0.10)
0.36(0.03)
0.56(0.10)
2.22(0.07)
0.09(0.01)
0.18
1.01(0.03)
32-MOGP 0.65(0.05) (.85()
2.89(0.10)
0.36(0.03)
(.59
2.16(0.06)
0.09
0.18(0.01
1.00(0.033
We trained belief models with small (t = [0, 26] epochs), medium = [0, 40] epochs), and large (t = [0, 75] epochs) training dataset of saliency measurements. For GPs, separate model was trained per network element (convolutional filter, or neuron). For MOGP, all network elements in single (TyerT shared one MOGP model. We evaluated these models using log likelihood of the remainder of the saliency measurements. We present the performance of the models in Table for CIFAR-100 [T Our MOGP approach better captures the saliency of network elements than GP approach. Furthermore, using additional latent functions improves MOGP modeling with diminishing returns. We visualize the qualitative differences between GP and MOGP prediction in Figure I We observe that MOGP is able to capture the long term trend of saliency curves with significantly less data than GP.
4.2 SMALL-SCALE EXPERIMENTS
We applied the early pruning algorithm the aforementioned architecture, and training regimen. We investigated the behavior the penalty parameter, A. We observed that the penalty parameter was difficult to tune properly, either being too aggressive pruning, too passive. To rectify this issue, we used a feedback loop to determine the penalty at iteration f, ) dynamically. Dynamic penalty scaling [2] uses feedback from earlier pruning iterations to increase decrease the iteration penalty at time t: A > ((1A)** ((T I)]|m hntl/m/////// /lo Br.- - 1)]. The dynamic penalty is increased if the anticipated compute required to complete training, (T 1) |mm lo begins exceed the amount of compute budget remaining, By... In such case, higher penalty is needed to satisfy the computational budget constraint as per (6). We compare dynamic penalty scaling, and penalty without scaling in Fig. 2 using To = 20 epochs, Tstep 10 epochs for the first convolutional layer of our CNN. Going forward, we use dynamic penalty scaling in our experiments.
10In our observations, jointly modeling the belief of multiple layers' saliency measurements using MOGP yielded no measurable improvement in lieliikld
"I For CIFAR-10 see Appendix G.1 G.1 Further details can be found in Appendix D

---

Under review as a conference paper at ICLR 2021
Dynamic penalty scaling
Static penalty
+
Bs - Am
Convolutional Kemels remaining
30
Bs 19, A-1-04
30
1
Bs 19, Xm
9. 1=
25
Bs 1m
20
20
15-
40
60
80
100
120
140
20
60
100
120
140
Epochs
Epochs
Figure 2: Comparing dynamic penalty scaling vs. static on pruning 32-convolutional filter layer in a CNN. Dynamic penalty scaling encourages gradual pruning across a wide variety settings of X.
Table 2:
Performance (standard error) against percentage of neurons/tilters pruned per layer with varying À for tested algorithms.
CIFAR-10
CIFAR-100
70%
80%
90%
95%
70%
80%
90%
95%
DSR
74.1(0.2)%
65.3(0.4)%
50.8
22. 22.1(5.0)%
37.9(0.7)%
29.7(0.1)%
17.5(0.3:%
4.4(1.4)%
SNIP
75.4[4.7)%
67.7
5.88
29.4(4.91%
22.9(9.0)%
16.7
9.9
2.2
GraSP
74.6(0.61%
66.5(0.9)%
50.66
32.9(1.01%
28.4(7.0)%
22.6(5.41%
13.9(3.2)%
1.0
BEP le 2
75.9(0.3)% %
69.7(0.4))
5..8%1.0)%
18.915.41%
40.6(0.))%
32.2
19.1
7.1(1.6%%
BEP le-4
75.4(1.7)% %
70.5(3.2)%
56.7(0.9)%
36 36.1
41.3(0.3)% 9%6
32.4
19.7
8.5
BEP le 76.0(0.1)%
70.6(0.2)%
%
56.2(0.4)%
30.4(5.1)9
40.6(0.2)%
33.0(0.5)%
19.5(0.5)%
6.6(1.5)%
We compare our work with SNIP (Lee 2019). GraSP (Wang et 2020), and momentum-basee dynamic sparse remaparameterization (DSR) (Dttteers & Zettlemoyer (219)). To compare against DSR, we instantiate smaller network of the size BEP yields after training has completed as is a prune and regrow method. The SNIP and GraSP approaches are extended to neurons/flterrs by averaging the saliencies of the constituent weight parameters. We experimented with various degrees of sparsity, using BEP to prune a portion of filters/neurons of each layer We present the results in Table 2 Our approach better preserves performance at equivalent sparsity. A lower penalty yields higher performing results showing À serves well at balancing performance vs. computational budget.
investigate the robustness BEP and MOGP hyperparameters. We vary the number of MOGP variational inducing points, MOGP latent functions, and Tatep and observe the performance of BEP 1e-4 on CIFAR-10/CIFAR at 80%, 90%, and 95% sparsity. We present these results in Table 3 We observe that in general, all hyperparameters are robust to changes. Mild degradation is observed in the extremal hyperparameter settings.
4.3 SPEEDING UP RESNET TRAINING ON IMAGENET
Our chief goal in this work is to speed up training of large-scale DNNs such as ResNet (He et al. 2016a b) the ImageNet dataset. Pruning ResNet requires a careful definition of network element saliency to allow pruning of all layers. ResNet contains long sequences of residual units with matching number of input/output channels. The inputs of residual units are connected with shortcut connections (i.e., through addition) to the output of the residual unit [4 Due to shortcut connections, this structure requires that within sequence of residual units, the number of inputs/output channels of all residual units must match exactly. This requires group pruning of residual unit channels for
In our observations saliency measurements don't 't well capture network element efficacy when comparing across layers. Thus pruning whole networks using network element saliency yields poor performing networks with bottlenecks. This limitation of saliency functions well known (See Molchanov et al.) (2017) Appendix A.1 and A.2: Wang et al. (2020) Section 3 last paragraph). Development of saliency functions which overcome this shortcoming while remaining performant is a difficult open problem outside the scope of this work.
Precise details of the ResNet architecture may be found in He et al. (2116a) Section
7

---

Under review as a conference paper at ICLR 2021
Table 3: Ablation study showing performance (standard error) vs, varying early pruning hyperpa- rameters: MOGP variational inducing points (Ind. pnts.), MOGP latent functions (Lat. func.), T step- Default setting for hyperparameters are 60, 1.0x, and 10 respectively. Outside of the higest sparsity setting, 95%, all hyperparameters are robust to changes, with mild degradation observed in the extremal settings.
CIFAR-10
CIFAR-100
80%
90%
95%
80%
90%
95%
26
69.6(0.4)%
56.8(
29.7(4.9)%
19.2(0.5)% 31.7(0.6)%
8.3(0.7)
Ind. pnts.
40
55.6(0.6)% 70.9(0.))%
30.5(5.1)%
19.6(0.3)% 32.3(0.7)
6.6(1.4)%
60
55.7(0.9)% 70.5(3.2)%
36.1(1.1)%
32.4(0.3)%
%
19.7(0.8)%
8.5(0.8)%
90
70.4(0.3)%
55.1(0.7)%
35.5(1.9)%
32.6(0.4)%
18.5(0.6)%
8.7(0.3)%
0.25
70.4(0.4)%
55.6(0.8)%
35.8(0.2)%
16.3(3.8)% 32.6(0.3)%
7.4(88)%
Lat. func.
0.50x
70.0(0.2)% %
56.9(0.4)%
34.5(0.61%
18.9(0.7)% 32.1
7.0(1.5)%
1.0x
70.5(3.2)%
55.7
36.1(1.11%
19.7(0.8)% % 8.5(0.8) 32.4(0.3)%
2.0x
69.8(0.3)%
55.7(0.7)%
34.8(0.5)%
32.0(0.4)%
20.8(0.2)% 7.7(0.4)%
2
(9.2(0.5)%
54.7(0.6)%
29.4(5.01%
32.1(0.21%
20.0(0.3)%
4.3(1.5)%
Tatep
5
(..2
55.6(0.51%
31.6(5.4)%
19.4(0.4)% 32.7(0.4)%
5.2
10
(3.5(%
55.7(0.9)%
36.1
32.4(0.3)%
19.7(0.4)%
8.5
20
(..3( 56.2(0.1)% 29.8(5.0)%
32.8(0.5)% 19.6(0.4)% 6.8(1.5)%
sequence of residual units, where group pruning output channel of residual unit sequence requires pruning it from the inputs/outputs of all residual units within the sequence
We trained ResNet-50 with BEP as well as SNIP and GraSP We group pruned less aggressively as residual unit channels feed into a large number residual units, thus making aggressive pruning likely to degrade performance. We BEP iterations at t [15, 20, 25, 35, 45, 55, 75) epochs. We trained for 100 epochs on 4x Nvidia Geforce GTX 1080 Ti GPUs. More experimental details found in Appendix G.2 We present our results in Table 4 We achieve higher performance than related techniques, albeit at longer wall time. Our approach captures the training time vs. performance tradeoff present in DNNs, unlike competing approaches.
RELATED WORK
Pruning and related techniques. Initial works in DNN pruning center around saliency based pruning after training including Skeletonization (Mozer & Smolensky 1988). Optimal Brain Dam- age and followup work (Hassibi Stork 1992 LeCun et al.) 1989) as well as sensitivity based pruning (Karnin 1990). In recent years, saliency functions been adapted to pruning neurons or convolutional filters. et al. (2017) define a saliency function on convolutional filters by using the the objective function as a saliency measure. Dong et al. (2017) propose layer wise pruning of weight parameters using a Hessian based saliency measure. Several variants of pruning after training exist, Han et al. (2015) propose iterative pruning where pruning is performed in stages alternating with fine tune training. Guo et al.) (2016) suggest dynamic network surgery, where pruning is performed on-the-fly fly during evaluation time. Li et al. (2017) and He et al. propose reinforcement learning for pruning decisions. A comprehensive overview may be found in Gale et al. 2201)).
L1 norm. Molchanov et al. (2017) propose using first-order Taylor-series approximation
and
2019 Yim et al. 2017) aim trained network into a smaller network. Weight sharing Nowlan & to transfer the capabilities
Knowledge distillation (Hinton et al. 2015 Lu et al. 2017 lung & Mori
Hinton 1992 Ullrich et al. (017)) and low rank matrix factorization (Denton et al. 20144] Jaderberg
et al. 2014) aim to compress the parameterization of neural networks. Network quantization (Cour-
bariaux et al. 2015 Hubara et al. 2017 Micikevicius et al., 2018) use lower fidelity representation of network elements (c.g. 16-bit) to speed up training and evaluation. Although speedup during training is achievable through network quantization, this technique requires hardware support and only
We formally define saliency on residual unit sequences in Appendix G.3
We omit comparison to DSR due to differing underlying deep learning library which makes walltime comparisons inaccurate.
8

---

Under review as a conference paper at ICLR 2021
Table 4: BEP vs. SNIP and Grasp on ResNet-50 on ImageNet dataset. We vary percentage of residual unit sequence channels (Seq) and filters pruned (Lyr). "Train' refers to wall time during network training. 'Prune' refers to pruning modeling/inferrnce overhead. Benchmark wall time for ResNet-50 is 55h on our hardware. Benchmark performance is 75.7% for unpruned ResNet-50.
Seq 30%,
Lyr 60%
Seq 60%. Lyr 90%
Seq 30%, Lyr 98%
Top-1
Top-5
Train
Prune
Top-1
Top-5
Train
Prune
Top-1
Top-5
Train
Prune
SNIP
72.0%
90.6%
27.9h
0.7h
62.0%
83.8%
15.8h
0.7h
50.9%
74.2%
21.7h
0.7h
GraSP
72.1%
90.6%
27.1h
2.7h
61.6%
83.6%
16.5h
2.7h
52.2%
75.4%
21.7h
2.7h
BEP le
72.2%
90.6%
31.6h
2.9h
62.0%
83.8%
22.8h
1.5h
53.7%
76.8%
27.8h
2.5h
BЕP le
72.5%
91.0%
34.8h
2.2h
62.3%
84.5%
22.0h
1.8h
53.5%
76.6%
27.8h
2.6h
provides coarse granularity in trading off computational effort vs. performance. Current GPUs only extend native support to 16-bit floating point operations. Furthermore, our approach orthogonal to quantization allowing the techniques to be combined for further speedup.
Initialization time or training time pruning. FFankle & Carbin (2019) show that a randomly initialized DNN contains small subnetwork, which if trained by itself. yields equivalent performance to the original network. SNIP (Lee et al. (219)) and GraSP (Wang et al. 2020) propose pruning connection weights prior to the training process through first order and second order saliency function respectively. Sparse Evolutionary Training (Mocanu et al.) 2018) propose initializing networks with sparse topology prior to training. Narang al. (2017) consider connection weight pruning during training for recurrent neural networks using a heuristic approach.
Dynamic sparse reparameterization considers pruning and regrowing parameter weights during the training process (Bellec et al. 2018 Dettmers & Zettlemoyer 2019 Mostafa & Wang 2019). Dai et al.) (2019) propose grow and prune approach learning network architecture and connection layout. We differ from existing work as our focus is on speeding up neural network training, meanwhile other works in training time pruning aim to achieve sparse network layouts. To the best of our knowledge, except for small speedups presented in (Dettmers & Zettlemoyer (019), the above works do not demonstrate speedup during training time using popular deep learning libraries run on modern GPUs.
Prune'Train (Lym al. 2019) also proposes pruning filters during training to achieve speedup while minimizing degradation to performance with periodic pruning iterations. In contrast to our approach, FruneTrain does not allow specification of the desired network size after training. A specified network size may be useful if training for resource constrained devices such as mobile phones or edge devices. We compare with PruneTrain under the early pruning problem definition in Appendix
CONCLUSION
This paper presents novel efficient algorithm to perform pruning of DNN elements such as neurons, or convolutional layers during the training process. To achieve early pruning before the training converges while preserving the performance of the DNN upon convergence, a Bayesian model (i.e., MOGP) is used to predict the saliency of DNN elements in the future (unseen) training iterations by exploiting the exponentially decaying behavior of the saliency and the correlations between saliency of different network elements. Then, we exploit a property (Lemma] I of the objective function and propose an efficient Bayesian early pruning algorithm. Empirical evaluations on benchmark datasets show that our algorithm performs favorably to related works for pruning convolutional filters and neurons. Our approach remains flexible changes in saliency function, and appropriately balances the training time vs. performance tradeoff in training DNNs. We are able to train an early pruned ResNet-50 model achieving a 48.6% speedup (37h vs. 55h) while maintaining validation accuracy of 72.5%.
9

---

Under review as a conference paper at ICLR 2021
REFERENCES
Zeyuan Allen-Zhu, Yuanzhi Li, and Yingyu Liang. Learning and generalization in overparameterized neural networks, going beyond two layers. In Proc. NeurIPS, pp. 6155-6166, 2019.
Mauricio A. Álvarez and Neil D. Lawrence. Computationally efficient convolved multiple output Gaussian processes. JMLR, (2(1):1459-115) 2011.
Karl Johan Åström, Tore Hägglund, Chang C Hang, and Weng K Ho. Automatic tuning and adaptation for pid controllers-a survey. Control Engineering Practice, 1(4):699-714, 1993.
Guillaume Bellec, David Kappel, Wolfgang Maass, and Robert A. Legenstein. Deep rewiring: Training very sparse deep networks. In Proc. ICLR, 2018.
Richard E Bellman. Adaptive control processes: guided tour. Princeton university press, 2015.
Aydin Buluę and John R. Gilbert. Challenges and advances in parallel sparse matrix-matrix multipli- cation. In Proc. ICCP, pp. 503-510, 2008.
François Chollet, 2015, TRL https: http:///tttab...coooooooooett//s
Matthieu Courbariaux, Yoshua Bengio, and Jean-Pierre David. Binaryconnect: Training deep neural networks with binary weights during propagations. arXiv:1511 TDv:1511.00300 1.00633, 2015.
Xiaoliang Dai, Hongxu Yin, and Niraj K. Jha. Nest: A neural network synthesis tool based grow-and-prune paradigm. IEEE Trans. Computers, 68(10): 2019.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Fei-Fei Li. ImageNet: A large-scale hierarchical image database. In Proc. CVPR, pp. 248-255, 2009.
Emily Denton, Wojeiech Zaremba, Joan Bruna, Yann LeCun, and Rob Fergus. Exploiting linear structure within convolutional networks for efficient evaluation. In Proc. NeurlPS, pp. 1269-1277, 2014.
Tim Dettmers and Luke Zettlemoyer. Sparse networks from scratch: Faster training without losing performance. (tVv:1907.04844 2019.
Xin Dong, Shangyu Chen, and Sinno Jialin Pan. Learning to prune deep neural networks via layer-wise optimal brain surgeon. Proc. NewrIPS, pp. 4857-4867, 2017.
Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. In Proc. ICLR, 2019.
Trevor Gale, Erich Elsen, and Sara Hooker. The state of sparsity in deep neural networks. arXiv:1902.09574, 2019.
Yiwen Guo, Anbang Yao, and Yurong Chen. Dynamic network surgery for efficient DNNs. In Proc. NeurlPS, pp. 1379-1387,20) 2016.
Song Han, Jeff Pool, John Tran, and William Dally. Learning both weights and connections for efficient neural networks. In Proc. NeurIPS, pp. 1135-1143, 2015.
Babak Hassibi and David G. Stork. Second order derivatives for network pruning: Optimal brain surgeon. In Proc. NewrIPS, pp. 164-171, 1992.
Kaiming He. Xiangyu Zhang, Shaoging Ren, and Jian Sun. Deep residual learning for image recognition. In Proc. CVPR, pp. 770-778, 2016a.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Identity mappings in deep residual networks. In Proc. ECCV, pp. 4432 4440, 2016b.
Yihui He, Ji Lin, Zhijian Liu, Hanrui Wang. Li-Jia Li, and Song Han. AMC: Automl for model compression and acceleration on mobile devices. In Proc. ECCV, pp. 815-832.
James Hensman. Alexander Matthews, and Zoubin Ghahramani. Scalable variational Gaussian process classification. In Proc. AISTATS, pp. 351-360, 2015.
10

---

Under review as a conference paper at ICLR 2021
Geoffrey E. Hinton, Nitish Srivastava, Alex Krizhevsky. llya Sutskever, and Ruslan R. Salakhutdinov. Improving neural networks by preventing co-adaptation of feature detectors. arXiv:1207.0580, 2012.
Geoffrey E. Hinton, Oriol Vinyals, and Jeffrey Dean. Distilling the knowledge in neural network. arXiv:1503.02531 2015.
Itay Hubara, Matthieu Courbariaux, Daniel Soudry, Ran HI- Yaniv, and Yoshua Bengio. Quantized neural networks: Training neural networks with low precision weights and activations. JMLR, 18 (1):6869-6898, 2017.
Max Jaderberg, Andrea Vedaldi, and Andrew Zisserman. Speeding up convolutional neural networks with low rank expansions. In Proc. BMVC, 2014.
Ehud D. Karnin. A simple procedure for pruning back proppaaapattioon trained neural networks. IEEE Trans, Neural Networks, 1(2):239-245 1990.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Proc. ICLR, 2015.
Alex Krizhevsky. Learning multiple layers of features from tiny images. Master's thesis, Univ. Toronto, 2009.
Yann LeCun, John S. Denker, and Sara A. Solla. Optimal brain damage. In Proc. NeurlPS, pp. 598-605, 1989.
Namhoon Lee, Thalaiyasingam Ajanthan, and Philip H. S. Torr. Snip: Single-shot network pruning based connection sensitivity. In Proc. ICLR, 2019.
Hao Li, Asim Kadav, Igor Durdanovic, Hanan Samet, and Hans Peter Graf. Pruning filters for
efficient convnets. Proc. ICLR, 2017.
Liang Lu, Michelle Guo, and Steve Renals. Knowledge distillation for smoll-footprint highway networks. In Proc. ICASSP, pp. 4820 4824, 2017.
Sangkug Lym, Esha Choukse, Siavash Zangeneh, Wei Wen, Sujay Sanghavi, and Mattan Erez. Prunetrain: fast neural network training by dynamic sparse model reconfiguration. In Proc. SC, pp. 36:1-36:13, 2019.
Alexander Matthews, Mark van der Wilk, Tom Nickson, Keisuke Fujii, Alexis Boukouvalas, Pablo León-Villagrá, Zoubin Ghahramani, and James Hensman. GPflow: A Gaussian process library using tensorflow. JMLR, 18(1):1-6, 2017.
Paulius Micikevicius, Sharan Narang, Jonah Alben, Gregory Diamos, Erich Elsen, David García, Boris Ginsburg, Michael Houston, Oleksii Kuchaiev, Ganesh Venkatesh, and Hao Wu. Mixed precision training. Proc. ICLR, 2018
Decebal Constantin Mocanu, Elena Mocanu, Peter Stone, Phuong Nguyen, Madeleine Gibescu, and Antonio Liotta. Scalable training of artificial neural networks with adaptive sparse connectivity inspired by network science. Nature, 91):1-12, 2018
Pavlo Molchanov, Stephen Tyree, Tero Karras, Timo Aila, and Jan Kautz. Pruning convolutional neural networks for resource efficient inference. In Proc. ICLR, 2017.
Hesham Mostafa and Xin Wang. Parameter efficient training of deep convolutional neural networks by dynamic sparse reparameterization. In Proc. ICML, pp.
2019.
Michael Mozer and Paul Smolensky. Skeletonization: A technique for trimming the fat from a network via relevance assessment. In Proc. NewrlPS, pp. 107-115. 1988.
Saralces Nadarajah and Samuel Kotz. Exact distribution of the max/min of two gaussian random variables. Trans. VLSI, 16(2):210-212, 2008.
Sharan Narang, Greg Diamos, Shubho Sengupta, and Erich Elsen. Exploring sparsity in recurrent neural networks. In Proc. ICLR, 2017.
11

---

Under review as a conference paper at ICLR 2021
Steven J. Nowlan and Geoffrey E. Hinton. Simplifying neural networks by soft weight-sharing Neural Computation, 4(4):473-493, 1992.
Adam Polyak and Lior Wolf. Channel-level acceleration of deep face representations. IEEE Access, 3:2163-2175, 2015.
Nitish Srivastava, Geoffrey E. Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: A simple way to prevent neural networks from overfitting. JMLR, 15(1):1299-1588 19588, 2014.
Kevin Swersky, Jasper Snock, and Ryan Prescott Adams. Freeze-thaw Bayesian optimization. arXiv:1406.3896, 2014.
Frederick Tung and Greg Mori. Similarity-preserving knowledge distillation. In Proc. ICCV, pp- 1365-1374, 2019.
Karen Ultrich, Edward Meeds, and Max Welling. Soft weight-sharing for neural network compression. In Proc. ICLR, 2017.
Chaoqi Wang, Guodong Zhang, and Roger B. Grosse. Picking winning tickets before training by preserving gradient flow. In Proc. ICLR, 2020.
Wei Wen, Chunpeng Wu, Yandan Wang, Yiran Chen, and Hai Li. Learning structured sparsity in deep neural networks. In Proc. NeurlPS, pp. 2074-2082, 2016.
Carl Yang. Aydin Buluç, and John D. Owens. Design principles for sparse matrix multiplication on the GPU. In Proc. Euro-Par, pp. 672-687, 2018.
Junho Yim, Donggyu Joo, Ji-Hoon Bae, and Junmo Kim. gift from knowledge distillation: Fast optimization, network minimization and transfer learning. In Proc. CVPR, pp. 7130-7138. 2017.
12

---

Under review as a conference paper at ICLR 2021
Figure 3: Convolutional filter saliency over 150 epochs of SGD on CIFAR-10.
A MODELING DETAILS
A. SALIENCY FUNCTION
In this work, we use first order Taylor-series saliency function proposed by Molchanov et al. (2017). Our design (Section 3 remains flexible to allow usage of arbitrary saliency functions in a plug play basis. We partition a DNN of I. layers, where each layer contains Ct convolutional filters, into a sequence of convolutional filters [Zt,e]==l,... ...... Each filter 2t,c 1gCt-1XW--IXHI. + +R W,xH. be considered Wr E RCxx0:x0;, br.e are kernel weights and bias. with receptive field O, x O³, '*' represents the convolution operation, R. is the activation function, P represents the output of zę-1 & [==-terle=1,....C._ with Po corresponding to an input X X. and We, H₃ are width and height dimensions of layer l for é L....,L. Let Ne - zg.o..... oze denote partial neural network of layers (e....,'JDsesee e'sssssss The Taylor-series saliency function the convolutional filter zt. denoted as a(Le, c\) is defined
network element in er and zt,c (P--) & R(W P be,e) where
D
WexHe Σ ==1
(xa) AC(P!
yas Na5NNe p(xa) E,a.
<(L,, c)) A
(8)
We x HE
where p(xx) is the output of the partial neural network Najure with as the input and (Pteyli=i,.... W. interprets the output of the c-th filter in vectorized form. This function uses the first-order Taylor-erres approximation of approximate the change in loss if zt,e was changed to a constant function. Using the above saliency definition, pruning filter zτ,c corresponds to collectively of #141 which utilize the output 2t.c. This definition can be extended to elements (e.g. neurons) which output scalars by setting Wt He 1.
zeroing Wi,cs b,,0 well weight parameter [8 [W ft=..ee.ssselee==......
A.2 ON THE CHOICE OF THE "EXPONENTIAL KERNEL"
We justify our choice of the exponential kernel as a modeling mechanism by presenting visualizations of saliency measurements collected during training, and comparing these samples drawn from the exponential kernel k,,t. samples exhibit exponentially decaying behavior, which makes the exponential kernel a strong for modeling saliency evolution time.
[1+ shown in Figs. 3 3 Both the saliency and the function
8"
Furthermore we note that the exponential kernel was used to great effect in Swersky et al. (0014) with respect to modeling loss curves function of epochs. Loss curves also exhibit asymptotic behavior, similar to saliency measurement curves, thus providing evidence for the exponential kemnl being an apt fit for our task.
A.3 PREDICTIVE DISTRIBUTION OF THE SALIENCY
Let the prior covariance matrix be Kn [cov[s], of ma.aa=........ for any < < < T. ll Plr...r Given a vector of observed saliency $1,t, the MOGP regression model can provide a Gaus-
For brevity, we omit parameters X' ), 22
Here use ( distinguish indexing into tensor from indexing into the sequence of tensors [Wante
13

---

Under review as a conference paper at ICLR 2021
1.25
-380
-10
(a) a = 3 = 0.5
(b) a
1.0
(c) 3 = 1.5
Figure 4: Function samples drawn from the exponential kernel.
sian predictive distribution p( [$1:) = Nu Kejpa) for any future saliency By with the following posterior vector and covariance matrix: Pelia & Ki Kr. Kejia & Kea Kir K i where (er ف [cov[sf, of "f Then, the a-th element p of pelia is the predictive mean of the saliency 86. And the (a, a'I-th element of Kyn denoted as ell is the predictive (co)variance between the saliency sp and sy.
B SUBMODULARITY OF E[pT]
In (6). the problem of choosing m from {0, can be considered as selecting a subset of indexes from (1...., M) such that mf = for A, and mf 0 otherwise. Therefore, P(m) A Epter 61.)) (m, B.)] can be considered a function which will show to be submodular. To keep notation consistency, will remain using P(m) instead of representing it as function of the index subset A.
Lemma (Submodularity). Let m', m" E {0, 13 and ela) be arbitrary M M-dimensional one hot vector with I a S M. We have P(m' V ve(a)) - P(m') 2 P(m" V elal) - P(m") ) for any m S m", m' e(a) and m" A ela) =0
Proof. According to
Ep(ar(ana) (rr(m, B.)l = Ep(ar(a (aa)) max [my ST, s.t. |m llo S B,, my 2 m)
Let a(m) - arg maxma mm ST, s.t. | llu S B,, mm return the optimized mask mr given any m. Am - min(a(m) @ ST) be the minimal saliency of the network elements selected at iteration T for P(m). Then, we have
P(m c(a))= = Ep(er(ãrs)) prr(m V e()), e()),B.))
Epper(aau)) [ār(m, Ba) Am + max (sq, Am)]
The second equality is due to the fact that the network element 57 would only replace the lowest included element in my in order to maximize the objective. Then,
P(m V ela)) P(m)
Ep(or(an.)) jrr(m, B.) - Am + max(sf Am)] Ep(sv(an.) (pr (m, B3)] Ep(or(aui)) -Amm + max(s7, Am)
Ep(orlãni)) [max(s?- Am;
Given m' 2 m", we have Amy < Am" since mr - in a(m') is tighter constraint than that in a(m"). Consequently, we can get sQ Am 2 oe Am", and thus
[P(m' V ve(a)) P(m')) [P(m" V ella) P(m")]
14

---

Under review as a conference paper at ICLR 2021
C PROOF OF LEMMA 1
restate Lemma clarity.
Lemma 1. Let e! be an M-dimensional one-hot vectors with the i-th element be 1. A S a S M; m E 1) s.t. m A (ela) V e(b)) = O. Given a vector of observed saliency S if PTraa 2 Pria and prppa 2 then
Bp(erIBnn) ( Epler and (m e(a))] S 17 2(v/0)+ + +0
where 0 A CDF and PDE, respectively.
Tii + Tiia
20 T v - 17p1 uTypa and t and b are standard normal
To prove this Lemma, we prove the following first:
Lemma 3. (p (pr(m e(O))] Epperlinn) [pr(m (pr(mVelll) vell))] s E[max(s}) -2- O)].
Proof. Due to 9 we have
pl Bii) p V v(()))
pller Bid) (pr(m vel
P(m V vb))
P(m) (P(m V ela)) P(m))
(ppler(are))
m
Am, 0)] - Ep(er(á..) [max((9
Am: 0)]
Epper Bia))
m Am, 0) max Am. 0)]
(10)
(s())
max(s}
s Am st) max(0, Am sg.)]
(11)
(R.))
[max(s)
s7.00 0)]
(12)
The equality 1T is achieved by adding Am ot in each term of the two max functions in (10). The
inequality 12 be proved by considering the following cases:
Am 8 2 0, then
max
24 Am -57) mmax(0, Am s4)
max((s) /// s7. Am -s (Am s9)
= max(s
s9. - (Am -s)). 0)
< max(s) -9,0)
If Am s4 < 0, then
max(s/
,4 Am -s4) maxx(0,Am) -s7)
= -mxx(s/ ( sq, Am st)
< max(s/ - 0).0).
Next we utilize well known bound regarding the maximum of two Gaussian random variables (Nadarajah & Kotz 2008), which we restate:
Lemma 4. Let s6 be Gaussian random variables with means 4ºth .po and standard devia- tions a², o²,0", b then El/max(""ss) & 1. ( (1²==²)) + 1² p²=p"" + Oi(( u² (²="") where 0 & V2 + [a 2cov(8", 88, 5²) and Φ @ are standard normal CDF and PDF respectively.
15

---

Under review as a conference paper at ICLR 2021
Then,
Bp(er(ena) mmxx - s7, 0)]
Eplerlaa (oler mmax(s4 s4)
Ep(ar|au. [89]
/ 1st
1T 0rb
14 M
S <1éppp + p )
pr
0
T - 19711 =nrpuc @
1 (T
MT/1:
+ A
p
-1
0
S 11rp2 $
T
0
MTj
1 1st
0
The first inequaltiy follows from Lemma 4 The second inequaltiy is due & and PTl: 2 O.
8
D DYNAMIC PENALTY SCALING AS A FEEDBACK LOOP
We designed a feedback loop to automatically determine X during early pruning. A proportional feedback loop can be defined as follows 19
Kp e(t)
(13)
where K2 2 is proportional constant which modulates A according to a signed measure of error e(.) at time f. Note that A 2 À as e(t) 2 0, and the opposite occurs if e(t) S O, which allows the error to serve as feedback to determine d Implicitly, de asserts some control over e(t 1), and thus closing the feedback loop.
Traditional PID approaches to determine Kp do not work our case as may vary over several orders of magnitude. Consequently, a natural choice for K2. is À itself which preserves the same order of magnitude between Kp and A:
A = A(1 1 E(t)).
(14)
Here we make two decisions to adapt the above our task. First, as À is likely to extremely small, we use exponentiation, as opposed to multiplication. Secondly as x < in practice, we use 1 e(t) as an exponent:
X, = X^ - (1-) = ((1/A)=((1))
(15)
The above derivation is complete with our definition of e(t):
e(1) (T' (T- Br.c 1.
(16)
The above determines error by the discrepancy between the anticipated compute required to complete training (T - t) mlll vs. the remaining budget B,, with e(t) = if the two are equal. This is a natural measure of feedback for expect the two be equal if is serving well to early prune the network.
E COMPARISON WITH PRUNETRAIN
We compare with PruneTraii (mm et al.) 2019 in Table PruneTrrain uses an orthogonal technique of dynamically increasing the minibatch size achieve further wall time improvements. This prevents accurate wall time comparisons between BEP and PruneTrains To compare with Prune which
19t approach is inspired from Aström et al. (1993) for an introductory survey.
(PID) controllers (Bellman, 2015).
16

---

Under review as a conference paper at ICLR 2021
Table 5: Comparing BEP with Prune Train ResNet-50 ImageNet dataset. PruneTrain uses stronger ResNet-50 baseline with Top-1 1 76.2% performance. BEP uses a 75.7% baseline. "Train" refers to wall time during network training. 'Prune' refers to pruning modeling/inference overhead. Benchmark wall time for ResNet-50 is 55h our hardware. Comparison performed using 47% of ResNet-50 baseline inference FLOPs. Train FLOPs refers to proportion of ResNet-50 baseline training FLOPs used to train the network.
47% Inference FLOPPs
Top-1 (A) Top-5
Train Prune
Train FLOPs
Prune'Train 74.3% (-1.9%)
60%
BЕP le-1 74.1% (- 1.6%)
91.8%
35.5h
3.9h
55.2%
BEP le-4 73.9% (- 1-18%) 91.7%
36.5h
3.2h
56.0%
Table 6: Notations used elsewhere in the paper.
Notation
Definition
M
Total number of network elements in Neural Network.
T
Total iterations of SGD in training procedure.
Random variable representing the saliency measurement of network element a at time f. Sequence of random variables [s]]e=1.....M
St
Sequence of random variables [8r]am1 ...,T-
Sta
The realization of random variable 81.t-
m
Pruning mask time
B,
Trained network sparsity budget.
Brc
Computational effort budget at time t.
E[st. [1c]
oa!
cov of [
does not constrain the trained network size (b), we train ResNet-50 under equivalent inference cost as a network trained by Prune'Traie Train. To compute train/inference cost (FLOPs) for a convolutional layer, we used formula defined in Molchanov et 2017) A.I:
FLOPs & 2HW (C,, K2 + 1)Cout
(17)
where H, W, Cin are input height, width, and channels respectively, K is the convolutional kernel size, and Cout the number of output channels of the layer.
Under equivalent inference cost, BEP 1e-1 outperforms PruneTrain in Top-1 performance. We also
find that BEP 1e-1 and BEP 1e-4 consumes fewer training FLOPs when compared baseline.
It should be noted that PruneTrai does not provide mechanism to constrain the trained network size, thus it is unclear how utilize in order solve the early pruning problem 33)
F TABLE OF NOTATIONS
We list table of notations used elsewhere in the paper in Table 6
17

---

Under review as a conference paper at ICLR 2021
Table 7:
Comparing log likelihood (standard error) of test data for Independent GPs (GP) vs. MOGP with latent functions (n-MOGP) on collected saliency measurements from CIFAR-10 training.
Small dataset
Medium dataset
Large dataset
Lyr
Lyr
Lyr 3
Lyr
Lyr
Lyr
Lyt
Lyr 2
Lyr
GP 1.19(0.5)
(006()
1.07(107))5
0.96(0.04)
0.93(0.03)
2.47(0.04)
0.49(0.01)
0.48(0.01)
1.33(0.02)
4-MOGP
1.15(0.05)
0.89))
2.44/0.05)
0.80(0.03) 0.91(0.02)
2.20
0.38(0.02)
0.39(0.02)
0.25(0.02)
8-MOGP
1.09(0.04)
(.00)
2.38(0.04)
0.84(0.03)
0.78(0.03) 2.16(0.03)
0.32(0.01)
0.35(0.02)
0.20(0.02)
18 MOGP
0.97/0.04)
(.00)
2.33(0.04)
0.83(0.03)
0.76(0.03)
2.13(0.03)
0.31(0.01)
0.35(0.02)
1.20(
32-MOGP
0.96(0.06)
(.80()
2.32(0.04)
0.79(0.03) 0.74(0.03) 2.13(00.3)
(.31
0.34(0.02)
1.20(0.02)
G MORE EXPERIMENTAL RESULTS AND EXPERIMENTAL DETAILS
G.1 GP vs. MOGP LOG-LIKELIIOOD ON CIFAR-10 DATASET
Table 7 presents the results of the experiment in Section 4.1 for the CIFAR-10 dataset.
G.2 EXPERIMENTAL DETAILS
To train our CIFAR-10 and CIFAR-100 100 models we used an Adam optimizer (Kingma & Ba 2015) with an initial learning rate of 0.001. The learning rate used exponential decay of k 0.985, and batch size of 32 was used. Training was paused three times evenly spaced per epoch. During this pause, we collected saliency measurements using 40% of the training dataset. This instrumentation subset was randomly select from the training dataset at initialization, and remained constant throughout the training procedure. We performed data preprocessing of saliency evaluations into a standardized filters. For the convolutional layers we used 12 latent MOGP functions. For the dense layer we used latent MOGP functions.
[0, 10) range
20 We used (8) to measure saliency of neurons/convolution
For our ResNet-50 model we used an SGD with Momentum optimizer with an initial learning rate of 0.1. The learning rate was divided by ten at / [30, 60, 80) epochs. We collected saliency data every 5 iterations of SGD, and averaged them into buckets corresponding to 625 iterations of SGD to form our datasct. We used a minimum 4 latent functions per MOGP, however this was dynamically increased if the model couldn't 't fit the data up to maximum of 15.
We sampled 10K points from our MOGP model to estimate A(-) for CIFAR-IO/CITAR-100. For ResNet sampled 15K points. We repeated experiments 5 times for reporting accuracy CIFAR- 10/CIFAR-100.
G.3 PRUNING ON RESNET
ResNet architecture is composed of sequence of residual units: Z, 2 F(P where P is the output of the previous residual unit Z and '+' denotes elementwise addition. Internally, F is typically implemented as three stacked convolutional layers: F(P:-1) & [zt, ze, ze, (Pr-1) where že,, ΣE2, Σt, are convolutional layers. Within this setting we consider convolutional filter pruning. Although ž6, Ze, may be pruned using the procedure described earlier. Pruning z2, requires a different procedure. Due to the direct addition of P to F(P,-1). the output dimensions of Z and 21, must match exactly. Thus ResNet architecture consists of sequences of residual units of length B with matching input/output dimensions: Ç & [Ze]e=1.....B s.t. dim dim dim (Pma B). We propose group pruming of layers [ where filters are removed from all zr, in residual unit sequence in tandem. We define c((),c) ΣE s([fa, c]), where s(-) is defined for convolutional layers as in (8). To prune the channel e from Ç. we prune it from each layer in [=r3]==1..... Typically we pruned sequence channels less aggressively than convolutional filters as these channels feed into several convolutional layers.
Gennrallyy saliency evaluations are relatively small (S 0.01), which leads to poor fitting models positive liglliill Precise details of our data preprocessing is in Appendix G.4
18

---

Under review as a conference paper at ICLR 2021
G.4 DATA PREPROCESSING
We followed the same data preprocessing procedure for both our small scale and ImageNet ex- periments. To standardize the saliency measurements for training dataset 81.t in mod- eling experiments clip them between 0 and upper bound computed (ercentile(8);: 95) x 1.3. This procedure removes outliers. We used 1.3 as multiplier, as this upper bound used transform test dataset as well, which may have higher saliency evaluations.
follows: ub -
After clipping the training data, we perform a trend check for each element va by fitting a Linear Regression model the data $i.-- For $1: with an increasing trend (i.e., the linear regression model has positive slope) we perform the transformation i = ub & The reasoning behind this is that the exponential kernel strongly prefers decaying curves. After this preprocessing, we scale up the saliency measurements to (0, 10) range: $1 $1 x 10. We found that without scaling to larger values, log- likelhood of our models demonstrated extremely high positive values due small values of unscaled saliency measurements.
transform the test data our modeling experiments 1:7 with the same procedure using the same ub and per-element v regression models as computed by the training data. We measure log-likelihood after this transformation for both the test dataset our small scale experiments.
During the BEP Algorithm, the same steps followed, however we inverse the trend check transformation (gi. = ub 8f:e) on the predicted MOGP distribution sT prior to sampling for estimation of A (.).
19


Reviews:
[{"review_id": "nMLdqHEWgXG", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary\n\nThis paper introduces a method for pruning during the training process in order to filter out unimportant/redundant components of the network continuously to speed up training and perform gradual pruning over the training process. The proposed approach is novel in the sense that the vast amount of prior work on pruning has focused on either (i) pruning on network initialization (e.g., SNIP, etc.) or (ii) pruning after the network has been fully trained (e.g., Magnitude Pruning, among many others). The introduced method uses the Taylor-series based saliency criterion (of Molchanov et al., 2017) and uses a multi-output Gaussian process to predict future saliencies and to determine whether a parameter can be safely removed early on during training.\n\n\nRationale for Score\n\t\nAs far as the negatives go, I take issue with the fact that the proposed approach seems to be highly complicated -- requiring a multitude of hyper-parameters/design choices, and tuning functions/ablation studies (e.g., for lambda). The empirical results are also not very compelling -- the proposed approach requires many more training hours than compared approaches that prune on initialization (cf., SNIP or GRASP in Table 3) and attains only a modest (.3% to .5%) improvement in pruning performance as measured by test accuracy.\n\nWith that said, I recommend weak acceptance with the hope that this work inspires more research in this area and that the shortcomings will be remedied in subsequent works that build upon the techniques introduced in this paper. I believe that this work has merit in pushing the community in the direction of trying to achieve one of the overarching goals of pruning: an efficient way to simultaneously train and search for an optimized network architecture for a particular application.\n\n\nStrengths\n\n- The paper is highly relevant to the ML and optimization communities; the premise that, e.g., filters that would have been pruned anyway after training, should not be trained to save computation time (and improve pruning performance) is very intuitive and appealing.\n\n- It is commendable that the authors tackle the very difficult problem of pruning during training and try to model the interdependencies/future uncertainty in a principled way (using MOGP). To my knowledge, there is no other work that attempts to tackle this problem as rigorously this paper does.\n\n- The method is overall motivated by principled insights and there is some analysis to justify parts of the method (Lemma 1)\n\n- The authors perform evaluations on appropriate benchmarks (ResNet50 trained on ImageNet) and achieve superior pruning results (in terms of test accuracy after training/pruning) relative to those of recently-proposed, state-of-the-art approaches (SNIP and GRASP)\n\n\nWeaknesses\n\n- The proposed algorithm is not parameter-free (unlike SNIP, which is virtually parameter-free), is quite complicated (and I imagine difficult to implement), and there is little justification for certain components of the method, e.g., the dynamic scaling function (and choices of lambda), whether the simplification of m_{t-1} = … = m_{T} is mild enough. It is not clear to me how a practitioner can run the proposed algorithm in a parameter-free way without having to conduct ablation studies of their own first, especially since, as the authors note, “We observed that the penalty parameter was difficult to tune properly, either being too aggressive at pruning, or too passive” as the justification for the dynamic scaling function\n\n- Parts of the paper are too dense and notation-heavy, and this hurts readability and understanding significantly, e.g., Lemma 1, paragraph regarding the introduction of the saliency function on pg. 2.\n\n- The presented experimental results are not very compelling. For example, in Table 3, we see that BEP 1e-4 achieves a ~.4% improvement over SNIP and GRASP, at the cost of ~7-8.4 more hours of training time. This calls into question the effectiveness of the proposed approach -- which is, at the end of the day, meant to *speed up* training + pruning by removing unnecessary components of the network early on.\n\n\nClarity \n\n- The paper is reasonably well-written and organized overall. It was clear that the authors compressed some of the mathematical expressions/lemmas (e.g., statement of Lemma 1), which is somewhat understandable given the page limit, but this hurt readability and understandability.\n\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "m38SFfTc7-Y", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper introduces a new method to accelerate training by saliency-based pruning. The method predicts future saliency for neurons based on observed saliency with a multi-output Gaussian process (MOGP), then greedily prunes neurons with least saliency at fixed intervals during training. The authors provide extensive mathematical analysis to show that the algorithm produces pruning mask solutions that are close to the optimum of the formulated optimization (the reviewer is unable to verify). The experimental results showed improvements in task accuracies of trained models but with longer training times. \n\nThe reviewer believes that the proposed method is novel, as it considers historical statistics during training to provide a more accurate saliency prediction. While this is interesting, the reviewer is concerned with the practicality. The paper can be improved with answers to the following questions:\n* The mathematical analysis showed that the algorithm produces pruning mask solutions that are close to the optimum. Is it possible to quantify? The reviewer worries that under a series of approximations and heuristic-based modeling below, the solution no longer aligns with the goal of pruning optimality, and thus makes the mathematical proofs irrelevant:\n\t* 3.1 problem statement as the pruning objective,\n\t* 3.2 saliency as MOGP with exponential kernel,\n\t* 3.3 subsequent simplifications,\n\t* 4 variational approximation of MOGP.\n* What is the overhead introduced by the maximum likelihood estimation of the MOGP, and Bayesian early pruning (BEP, Algorithm 1)? Does the benefit of having a pruned model always outweigh the costs of pruning?\n* How accurate is the saliency prediction? Can this be quantified and illustrated somehow, e.g. with the predicted values on Figure 2 in the appendix? If this is not accurate that one may expect the components in the pruning procedure can be replaced with simpler variants without a detrimental impact. For instance, instead of using MOGP for prediction, one may consider static saliency. (Or is this identical to traditional approaches used by SNIP and GRASP?)\n\nAdditionally, experiments should not only have averaged results but also provide standard deviations.\n\nThe paper is in general well-written, the reviewer has minor complaints:\n* In Algorithm 1, `\\mu_{T \\mid 1:t}` is not assigned value anywhere.", "extra_scores": {}, "extra_text": {}}, {"review_id": "QQsaYZ8Icj", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents a training-time pruning method for deep neural network. The main idea is to include network elements only if they could improve the predictive mean of the saliency, where the saliency measures the efficiency of the network elements in terms of minimizing the loss function. \nThe paper presents a clear objective of the early pruning, which is to preserve the sub-network that can maximize the saliency function. This optimization problem is NP-hard, and even approximation is very expensive. The authors proves that one can simplify the approximation by ranking the network element by predictive mean of the saliency function.\n\nThe authors evaluated their method empirically and showed that it is superior than the GP modeling, and provides a trade-off between accuracy and the training time.\n\n+ This paper is well organized. The theoretical analysis is well written and provides a good review to readers who lack relevant background.\n+ The idea is interesting and Lemma 1 should be of value to researchers working on the similar problem.\n+ Reading from the review section, the problem of training-time pruning is not well studied yet, whereas this paper could be seen as an important contribution.\n- The experiment section is a bit hard to follow. Among other problem, I find it hard to understand the intuition behind the dynamic penalty scaling.\n- It is not clear to me why the dynamic sparse reparameterization methods are not listed as baseline in evaluations.\n\nOverall I think this is a good paper. The authors could improve it by addressing the two problems in the evaluation section.", "extra_scores": {}, "extra_text": {}}, {"review_id": "MtvY8vJNhjI", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes a Bayesian-based approach to early prune parameters, which are predicted to have low saliency/importance, with the goal of accelerating the training of deep neural networks. The predictor is a \"multi-output Gaussian process\" which is computation expensive.\n\nThe writing quality and clarity of this paper is OK, but I recommend the authors to include a table of mathematical notations.\n\nThe idea of using a Gaussian process based predictor to predict the importance of parameters during the training process is interesting. However, this paper only compares with some methods (e.g. SNIP and GraSP) which prune parameters before training, inspired by the lottery ticket hypothesis paper. Lots of pruning methods can prune parameters during training before the lottery ticket hypothesis paper appears. For example, PruneTrain https://arxiv.org/abs/1901.09290 can reduce end-to-end training time of ResNet-50 by 39% without losing accuracy by simply using a previous sparsity regularization method (while this paper has a significant accuracy drop to 72.5% for ResNet-50). The paper should compare with those more superior methods, regardless of the fact that in Table 3 it is unclear if BEP can outperform SNIP/GraSP or not under the same \"Time\".\n\nMoreover, the method introduces new hyperparameters. To tune the hyperparameters, the method should be run multiple times. I cannot see how it will make training a neural network faster, unless the hyperparameters are super robust to generate, which is not deeply discussed.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper considers the problem of pruning deep neural networks (DNNs) during training. The key idea is to include DNN elements only if they improve the predictive mean of the saliency (efficiency of the DNN elements in terms of minimizing the loss function). The objective of early pruning is to preserve the sub-network that can maximize saliency. This optimization problem is NP-hard, and even approximation is very expensive. The paper proves that one can simplify the approximation by ranking the network element by predictive mean of the saliency function.

The proposed approach is novel as most of the prior work on pruning has focused on either (i) pruning on network initialization  or (ii) pruning after the network has been fully trained. 

Couple of issues with the paper are:
1. Current approach is somewhat complicated with many hyper-parameters
2. Experimental results are not very compelling when compared to pruning on network initialization

Overall, my assessment is that the paper takes a new research direction and has the potential to inspire the community, and followup work may be able to overcome the above two issues in future. However, due to the remaining shortcomings, the paper is not judged ready for publication in its present form. I strongly encourage to resubmit the paper after addressing the above two concerns.
Decision: Reject