Year: 2020

Abstract:
Pruning is an approach to alleviate overparameterization of deep neural network (DNN) by zeroing out or pruning DNN elements with little to no efficacy at a given task. In contrast to related works that do pruning before or after training, this paper presents a novel method to perform early pruning of DNN elements (e.g., neurons or convolutional filters) during the training process while preserving performance upon convergence. To achieve this, we model the future efficacy of DNN elements in a Bayesian manner conditioned upon efficacy data collected during the training and prune DNN elements which are predicted to have low efficacy after training completion. Empirical evaluations show that the proposed Bayesian early pruning improves the computational efficiency of DNN training. Using our approach we are able to achieve a 48.6% faster training time for ResNet-50 on ImageNet to achieve a validation accuracy of 72.5%.

heading: 1 INTRODUCTION
text: Deep neural networks (DNNs) are known to be overparameterized (Allen-Zhu et al., 2019) as they usually have more learnable parameters than needed for a given learning task. So, a trained DNN contains many ineffectual parameters that can be safely pruned or zeroed out with little/no effect on its predictive accuracy. Pruning (LeCun et al., 1989) is an approach to alleviating overparameterization of a DNN by identifying and removing its ineffectual parameters while preserving its predictive accuracy on the validation/test dataset. Pruning is typically applied to the DNN after training to speed up testing-time evaluation. For standard image classification tasks with MNIST, CIFAR-10, and ImageNet datasets, it can reduce the number of learnable parameters by up to 50% or more while maintaining test accuracy (Han et al., 2015; Li et al., 2017; Molchanov et al., 2017).
In particular, the overparameterization of a DNN also leads to considerable training time being wasted on those DNN elements (e.g., connection weights, neurons, or convolutional filters) which are eventually ineffectual after training and can thus be safely pruned. Our work in this paper considers early pruning of such DNN elements by identifying and removing them throughout the training process instead of after training.1 As a result, this can significantly reduce the time incurred by the training process without compromising the final test accuracy (upon convergence) much.
Recent work (Section 5) in foresight pruning (Lee et al., 2019; Wang et al., 2020) show that pruning heuristics applied at initialization work well to prune connection weights without significantly degrading performance. In contrast to these work, we prune throughout the training procedure, which improves performance after convergence of DNNs, albeit with somewhat longer training times.
In this work, we pose early pruning as a constrained optimization problem (Section 3.1). A key challenge in the optimization is accurately modeling the future efficacy of DNN elements. We achieve this through the use of multi-output Gaussian process which models the belief of future efficacy conditioned upon efficacy measurements collected during training (Section 3.2). Although the posed optimization problem is NP-hard, we derive an efficient Bayesian early pruning (BEP) approximation algorithm, which appropriately balances the inherent training time vs. performance tradeoff in pruning prior to convergence (Section 3.3). Our algorithm relies on a measure of network element efficacy, termed saliency (LeCun et al., 1989). The development of saliency functions is an active area of research with no clear optimal choice. To accomodate this, our algorithm is agnostic, and therefore
1In contrast, foresight pruning (Wang et al., 2020) removes DNN elements prior to the training process.
flexible, to changes in saliency function. We use BEP to prune neurons and convolutional filters to achieve practical speedup during training (Section 4).2 Our approach also compares favorably to state-of-the-art works such as SNIP (Lee et al., 2019), GraSP (Wang et al., 2020), and momentum based dynamic sparse reparameterization (Dettmers & Zettlemoyer, 2019).

heading: 2 PRUNING
text: Consider a dataset of D training examples X = {x1, . . . ,xD},Y = {y1, . . . , yD} and a neural network Nvt parameterized by a vector of M pruneable network elements (e.g. weight parameters, neurons, or convolutional filters) vt , [vat ]a=1,...,M , where vt represent the network elements after t iterations of stochastic gradient descent (SGD) for t = 1, . . . , T . Let L(X ,Y;Nvt) be the loss function for the neural network Nvt . Pruning aims at refining the network elements vt given some sparsity budget B and preserving the accuracy of the neural network after convergence (i.e., NvT ), which can be stated as a constrained optimization problem (Molchanov et al., 2017):
minm∈{0,1}M |L(X ,Y;Nm vT )− L(X ,Y;NvT )| s.t. ||m||0 ≤ B (1)
where is the Hadamard product and m is a pruning mask. Note that we abuse the Hadamard product for notation simplicity: for a = 1, ..,M , ma × vaT corresponds to pruning vaT if ma = 0, and keeping vaT otherwise. Pruning a network element refers to zeroing the network element or the weight parameters which compute the network element. Any weight parameters which reference the output of the pruned network element are also zeroed since the element outputs a constant 0.
The above optimization problem is difficult due to the NP-hardness of combinatorial optimization. This leads to the approach of using saliency function s which measures efficacy of network elements at minimizing the loss function. A network element with small saliency can be pruned since it’s not salient in minimizing the loss function. Consequently, pruning can be done by maximizing the saliency of the network elements given the sparsity budget B:
maxm∈{0,1}M ∑M a=1m as(a;X ,Y,NvT ,L) s.t. ||m||0 ≤ B (2)
where s(a;X ,Y,NvT ,L) measures the saliency of vaT at minimizing L after convergence through T iterations of SGD. The above optimization problem can be efficienctly solved by selecting the B most salient network elements in vT .
The construction of the saliency function has been discussed in many existing works: Some approaches derived the saliency function from first-order (LeCun et al., 1989; Molchanov et al., 2017) and second-order (Hassibi & Stork, 1992; Wang et al., 2020) Taylor series approximations of L. Other common saliency functions include L1 (Li et al., 2017) or L2 (Wen et al., 2016) norm of the network element weights, as well as mean activation (Polyak & Wolf, 2015). In this work, we use a first-order Taylor series approximation saliency function defined for neurons and convolutional filters3 (Molchanov et al., 2017), however our approach remains flexible to arbitrary choice of saliency function on a plug-n-play basis.

heading: 3 BAYESIAN EARLY PRUNING

heading: 3.1 PROBLEM STATEMENT
text: As has been mentioned before, existing pruning works based on the saliency function are typically done after the training convergence (i.e., (2)) to speed up the testing-time evaluation, which waste considerable time on training these network elements which will eventually be pruned. To resolve this issue, We extend the pruning problem definition (2) along the temporal dimension, allowing network elements to be pruned during the training process consisting of T iterations of SGD.
2Popular deep learning libaries do not accelerate sparse matrix operations over dense matrix operations. Thus, pruning network connections cannot be easily capitalized upon with performance improvements. It is also unclear whether moderately sparse matrix operations (i.e., operations on matrices generated by connection pruning) can be significantly accelerated on massively parallel architectures such as GPUs (see Yang et al. (2018) Fig. 7). See Section 5 in Buluç & Gilbert (2008) for challenges in parallel sparse matrix multiplication.
3Implementation details of this saliency function can be found in Appendix A.1.
Let sat , s(a;X ,Y,Nvt ,L) be a random variable which denotes the saliency of network element vat after t iterations of SGD, st , [sat ]a=1,...,M for t = 1, . . . , T , and sτ1:τ2 , [st]t=τ1,...,τ2 be a vector of saliency of all the network elements between iterations τ1 and τ2. Our early pruning algorithm is designed with the goal of maximizing the saliency of the unpruned elements after iteration T , yet allowing for pruning at each iteration t given some computational budget Bt,c for t = 1, . . . , T :
ρT (mT−1, BT,c, Bs) , maxmT mT · sT (3a) s.t. ||mT ||0 ≤ Bs (3b) mT ≤̇mT−1 (3c) BT,c ≥ 0 (3d)
ρt(mt−1, Bt,c, Bs) , maxmtEp(st+1|s̃1:t) [ ρt+1(mt, Bt,c−||mt||0, Bs) ] (4a)
s.t. mt ≤̇mt−1 (4b)
where Bs is the trained network sparsity budget, s̃1:t is a vector of observed values for s1:t, m0 is an M -dimensional 1’s vector, andmt ≤̇mt−1 represents an element-wise comparison between mt and mt−1: mat ≤ mat−1 for a = 1, . . . ,M . At each iteration t, the saliency st is observed and mt ∈ {0, 1}M in ρt represents a pruning decision performed to maximize the expectectation of ρt+1 conditioned upon saliency measurements s1:t collected up to and including iteration t. This recursive structure terminates with base case ρT where the saliency of the unpruned elements is maximized after T iterations of training.
In the above early pruning formulation4, constraints (3c) and (4b) ensure pruning is performed in a practical manner whereby once a network element is pruned, it can no longer be recovered in a later training iteration. We define a trained network sparsity budget, Bs (3b), which may differ significantly from initial network size ||m0||0 (e.g., in the case where the network is trained on GPUs, but deployed on resource constrained edge or mobile devices). We also constrain a total computational effort budget Bt,c which is reduced per training iteration t by the number of unpruned network elements ||mt||0. We constrain BT,c ≥ 0 (3d) to ensure training completion within the specified computational budget. Here we assume that a more sparse pruning maskmt corresponds to lower computational effort during training iteration t due to updating fewer network elements. Finally, (3a) maximizes the saliency with a pruning maskmT constrained by a sparsity budget Bs (3b). Our early pruning formulation balances the saliency of network elements after convergence against the total computational effort to train such network (i.e., mT · sT vs. ∑T t=1||mt||0). This appropriately captures the balancing act of training-time early pruning whereby the computational effort is saved by early pruning network elements while preserving the saliency of the remaining network elements after convergence.

heading: 3.2 MODELING THE SALIENCY WITH MULTI-OUTPUT GAUSSIAN PROCESS
text: To solve the above early pruning problem, we need to model the belief p(s1:T ) of the saliency for computing the predictive belief p(st+1:T |s̃1:t) of the future saliency in (4a). At the first glance, one may consider to decompose the belief: p(s1:T ) , ∏M a=1 p(s a 1:T ) and model the saliency s a 1:T , [sat ]t=1,...,T of each network element independently. Such independent models, however, ignore the co-adaptation and co-evolution of the network elements which have been shown to be a common occurrence in DNN (Hinton et al., 2012; Srivastava et al., 2014; Wang et al., 2020). Also, modeling the correlations between the saliency of different network elements explicitly is non-trivial since considerable feature engineering is needed for representing diverse network elements such as neurons, connections, or convolutional filters.
To resolve such issues, we use multi-output Gaussian process (MOGP) to jointly model the belief p(s1:T ) of all saliency measurements. To be specific, we assume that the saliency sat of the a-th network element at iteration t is a linear mixture5 of Q independent latent functions {uq(t)}Qq=1: sat , ∑Q q=1 γ a q uq(t). As shown in (Álvarez & Lawrence, 2011), if each uq(t) is an independent GP with prior zero mean and covariance kq(t, t′), then the resulting distribution over p(s1:T ) is a multivariate Gaussian distribution with prior zero mean and covariance determined by the mixing
4In contrast to PruneTrain (Lym et al., 2019), our problem definition balances training time vs. performance under an additional constraint on the trained network size (3b). We discuss this further in Section 5.
5Among the various types of MOGPs (see Álvarez & Lawrence (2011) for a detailed review.), we choose this linear model such that the correlations between sat and sa ′ t′ can be computed analytically.
weights: cov[sat , s a′ t′ ] = ∑Q q=1 γ a q γ a′ q kq(t, t ′). This explicit covariance between sat and s a′
t′ helps to exploit the co-evolution and co-adaptation of network elements within the neural networks.
To capture the horizontal asymptote trend of sa1 , . . . , s a T as visualized in Appendix A.2, we turn to a kernel used for modeling decaying exponential curves known as the “exponential kernel” (Swersky et al., 2014) and set kq(t, t′) , βq αq
(t+t′+βq) αq where αq and βq are hyperparameters of MOGP and can
be learned via maximum likelihood estimation (Álvarez & Lawrence, 2011). Then, given a vector of observed saliency s̃1:t, the MOGP regression model can provide a Gaussian predictive distribution for any future saliency st′ . Thus, the predictive mean µat′|1:t , E[s a t′ | s̃1:t] of the saliency sat′ and the predictive (co)variance σaa ′
t′|1:t , cov[s a t′ , s
a′ t′ | s̃1:t] between the saliency sat′ and sa ′
t′ can be computed analytically, as detailed in Appendix A.3.

heading: 3.3 EARLY PRUNING ALGORITHM
text: Solving the above optimizing problem (3) and (4) is difficult due to the interplay between [mt′ ]t′=t,...,T , [Bt′,c]t′=t,...,T , and mT · sT . Instead, we consider a simplification of the above problem by only considering solutions of the formmT−1 = mT−2 = . . . = mt which yields6:
ρ̂t(mt−1, Bt,c, Bs) , maxmt Ep(sT |s̃1:t) [ρT (mt, Bt,c − (T − t)||mt||0, Bs)] (5) This approach allows us to lift (3d) from (3), to which we add a Lagrange multiplier and achieve:
ρ̂t(mt−1, Bt,c, Bs) , maxmt Ep(sT |s̃1:t) [ρ̂T (mt, Bs)] + λ (Bt,c − (T − t)||mt||0) (6) for t = 1, . . . , T − 1 and ρ̂T is defined as ρT without constraint (3d). Consequently, such a ρ̂T can be solved in a greedy manner as in (2). Afterwards, we will omit Bt,c as a parameter of ρ̂T as it no longer constrains the solution of ρ̂T . Note that the presence of an additive penalty in a maximization problem is due to the constraint BT,c ≥ 0 ⇔ −BT,c ≤ 0 which is typically expected prior to Lagrangian reformulation. The above optimization problem remains NP-hard as Ep(sT |s̃1:t)[ρ̂T (mt, Bs)] is submodular inmt (see Appendix B). Although greedy approximations exist for submodular optimization, their running time of O(||mt−1||20) remains far too slow due to the large number of network elements in DNNs. Fortunately, it can be significantly simplified by exploiting the following lemma (its proof is in Appendix C.):
Lemma 1. Let e(i) be an M -dimensional one-hot vectors with the i-th element be 1. ∀ 1 ≤ a, b ≤ M ; m ∈ {0, 1}M s.t.m ∧ (e(a) ∨ e(b)) = 0. Given a vector of observed saliency s̃1:t , if µaT |1:t ≥ µ b T |1:t and µ a T |1:t ≥ 0, then
Ep(sT |s̃1:t)[ρT (m ∨ e(b))]− Ep(sT |s̃1:t)[ρT (m ∨ e(a))] ≤ µbT |1:t Φ(ν/θ) + θ φ(ν/θ) where θ , √ σaaT |1:t + σ bb T |1:t − 2σ ab T |1:t , ν , µ b T |1:t − µ a T |1:t , and Φ and φ are standard normal CDF and PDF, respectively.
Here, ‘∨’ and ‘∧’ represent bitwise OR and AND operations, respectively. The bitwise OR operation is used to denote the inclusion of e(a) or e(b) in mt. Due to the strong tail decay7 of φ and Φ, Lemma 1 indicates at most marginal possible improvement provided by opting formt = m ∨ e(b) as opposed tomt = m ∨ e(a) given µaT |1:t ≥ µ b T |1:t.
Lemma 1 admits the following approach to optimize ρ̂t: starting with mt = 0M , we consider the inclusion of network elements inmt by the descending order of {µaT |1:t} M a=1 which can be computed analytically using MOGP. A network element denoted by e(a) is included in mt if it improves the objective in (5). The algorithm terminates once the highest not-yet-included element does not improve the objective function as a consequence of the penalty term outweighing the improvement in Ep(sT |s̃1:t)[ρT ]. The remaining excluded elements are then pruned.
Following the algorithm sketch above, we define the utility of network element vat with respect to candidate pruning mask mt≤̇mt−1 which measures the improvement in Ep(sT |s̃1:t)[ρT ] as a
6We omit (4b) as it is automatically satisfied due to our simplification. 7Note as µaT |1:t ≥ µbT |1:t, Φ(·) ≤ 0.5 and experiences tail decay proportional to µaT |1:t − µbT |1:t.
consequence of inclusion of e(a) inmt: ∆(a,mt, s̃1:t, Bs) , Ep(sT |s̃1:t) [ ρT (e (a) ∨mt, Bs)− ρT (mt, Bs) ] . (7) We can now take a Lagrangian approach to pruning decisions during iteration t by balancing the utility of network element vat against the change of the penalty (i.e., λ(T − t)) in Algorithm 1. Due to the relatively expensive cost of performing early pruning, we chose to early prune every Tstep iterations of SGD. Typically Tstep was chosen to correspond to 10-20 epochs of training. To compute ∆(·) we sampled from p(sT |s̃1:t) and used a greedy selection algorithm per sample as in (2). During implementation, we also enforced an additional hard constraint ||mt||0 ≥ Bs which we believe is desirable for practicality reasons. We used a fixed value of B1,c = ||m0||0T0 +Bs(T − T0) in all our experiments.
Algorithm 1 Bayesian Early Pruning Require: N , v1, T0, Tstep, T , B1,c, Bs, λ . DNN N , Lagrangian penalty λ
1: S1:T0 ← train(Nv1 , T0) . Train for T0 iterations to create seed dataset. 2: BT0,c ← B1,c − T0 dim(v1) . Track computational effort expenditure. 3: for k ← 0, . . . , T−T0Tstep ; t← T0 + kTstep do . Early prune every Tstep iterations from T0. 4: µT |1:t, σT |1:t ←MOGP (S1:t) . Train and perform inference. 5: sT ← argsort(−µT |1:t) . Sort descending. 6: mt ← 0dim(vt) . Initial pruning mask. 7: for a← s1T , . . . , s dim(vt) T do . Consider each network element. 8: if Bt,c − (T − t)||mt||0 > 0 then . Remaining Bt,c budget can support training vat . 9: mt = mt ∨ e(a) 10: else if ∆(a,mt, s̃1:t, Bs) ≥ λ(T − t) then . Balance utility against change of penalty. 11: mt = mt ∨ e(a) 12: else 13: break 14: prune(vt,mt) . dim(vt) is reduced here. 15: Bt+Tstep,c ← Bt,c − Tstep||mt||0 16: St+1:t+Tstep ← train(Nvt , Tstep) . Continue training with pruned network. 17: return N

heading: 4 EXPERIMENTS AND DISCUSSION
text: We evaluate our modeling approach as well as our BEP algorithm on the CIFAR-10, CIFAR100 (Krizhevsky, 2009), and ImageNet (Deng et al., 2009) datasets. For CIFAR-10/CIFAR-100 we used a benchmark Convolutional Neural Network (CNN) with 4 convolutional layers, and 1 dense layer.8 For ImageNet we validated on the ResNet-50 architecture (He et al., 2016a).
Due to the cubic time complexity of MOGPs, we used a variational approximation (Hensman et al., 2015). In all of our models, we used 60 variational inducing points per latent function. We used GPFlow library (Matthews et al., 2017) to build our models.

heading: 4.1 MODELING EVALUATION
text: A key assertion in our approach is the importance of capturing co-adaptation and co-evolution effects in network elements. To verify our MOGP approach captures these effects, we compare MOGP vs. GP belief modeling where GP assumes independence in saliency measurements across network elements (i.e., p(s1:T ) , ∏M a=1 p(s a 1:T )).
A dataset of saliency measurements of convolutional filters and neurons was constructed by instrumenting the training process of our 5-layer CNN on the CIFAR-10/CIFAR-100 dataset. Keras (Chollet, 2015) was used to train this model over 150 epochs.9
8Code available at https://github.com/keras-team/keras/blob/master/examples/ cifar10_cnn.py
9Complete experimental setup details found in Appendix G.2.
GP 0.75(0.06) 5.7(5.7)e4 5.6(5.6)e4 0.64(0.04) 0.70(0.04) 2.13(0.05) 3.4(3.4)e3 0.31(0.02) 1.06(0.02) 4-MOGP 0.79(0.05) 0.98(0.12) 3.13(0.10) 0.44(0.04) 0.60(0.10) 2.29(0.06) 0.12(0.01) 0.24(0.03) 1.07(0.03) 8-MOGP 0.65(0.05) 0.89(0.11) 3.00(0.09) 0.38(0.04) 0.60(0.10) 2.20(0.06) 0.10(0.01) 0.18(0.01) 1.02(0.03) 18-MOGP 0.62(0.05) 0.84(0.11) 2.93(0.10) 0.36(0.03) 0.56(0.10) 2.22(0.07) 0.09(0.01) 0.18(0.01) 1.01(0.03) 32-MOGP 0.65(0.05) 0.85(0.09) 2.89(0.10) 0.36(0.03) 0.59(0.10) 2.16(0.06) 0.09(0.02) 0.18(0.01) 1.00(0.03)
We trained belief models with small (t = [0, 26] epochs), medium (t = [0, 40] epochs), and large (t = [0, 75] epochs) training dataset of saliency measurements. For GPs, a separate model was trained per network element (convolutional filter, or neuron). For MOGP, all network elements in a single layer10 shared one MOGP model. We evaluated these models using log likelihood of the remainder of the saliency measurements. We present the performance of the models in Table 1 for CIFAR-100.11 Our MOGP approach better captures the saliency of network elements than a GP approach. Furthermore, using additional latent functions improves MOGP modeling with diminishing returns. We visualize the qualitative differences between GP and MOGP prediction in Figure 1. We observe that MOGP is able to capture the long term trend of saliency curves with significantly less data than GP.

heading: 4.2 SMALL-SCALE EXPERIMENTS
text: We applied the early pruning algorithm on the aforementioned architecture, and training regimen. We investigated the behavior of the penalty parameter, λ. We observed that the penalty parameter was difficult to tune properly, either being too aggressive at pruning, or too passive. To rectify this issue, we used a feedback loop to determine the penalty at iteration t, λt dynamically. Dynamic penalty scaling12 uses feedback from earlier pruning iterations to increase or decrease the iteration penalty at time t: λt = λ [(1/λ)∧ ((T − t)||mt||0/Bt,c − 1)]. The dynamic penalty is increased if the anticipated compute required to complete training, (T − t)||mt||0 begins to exceed the amount of compute budget remaining, Bt,c. In such case, a higher penalty is needed to satisfy the computational budget constraint as per (6). We compare dynamic penalty scaling, and penalty without scaling in Fig. 2 using T0 = 20 epochs, Tstep = 10 epochs for the first convolutional layer of our CNN. Going forward, we use dynamic penalty scaling in our experiments.
10In our observations, jointly modeling the belief of multiple layers’ saliency measurements using MOGP yielded no measurable improvement in log-likelihood.
11For CIFAR-10 see Appendix G.1. 12Further details can be found in Appendix D.
DSR 74.1(0.2)% 65.3(0.4)% 52.8(0.5)% 22.1(5.0)% 37.9(0.7)% 29.7(0.1)% 17.5(0.3)% 4.4(1.4)% SNIP 75.4(4.7)% 67.7(0.7)% 50.8(0.8)% 29.4(4.9)% 22.9(9.0)% 15.7(6.1)% 9.9(3.7)% 2.2(1.2)% GraSP 74.6(0.6)% 66.5(0.9)% 50.7(0.6)% 32.9(1.0)% 28.4(7.0)% 22.6(5.4)% 13.9(3.2)% 1.0(0.0)% BEP 1e−2 75.9(0.3)% 69.7(0.4)% 54.8(1.0)% 18.9(5.4)% 40.6(0.2)% 32.2(0.6)% 19.1(0.5)% 7.1(1.6)% BEP 1e−4 75.4(1.7)% 70.5(3.2)% 55.7(0.9)% 36.1(1.1)% 41.3(0.3)% 32.4(0.3)% 19.7(0.8)% 8.5(0.8)% BEP 1e−7 76.0(0.1)% 70.6(0.2)% 56.2(0.4)% 30.4(5.1)% 40.6(0.2)% 33.0(0.5)% 19.5(0.5)% 6.6(1.5)%
We compare our work with SNIP (Lee et al., 2019), GraSP (Wang et al., 2020), and momentum-based dynamic sparse remaparameterization (DSR) (Dettmers & Zettlemoyer, 2019). To compare against DSR, we instantiate a smaller network of the size BEP yields after training has completed as it is a prune and regrow method. The SNIP and GraSP approaches are extended to neurons/filters by averaging the saliencies of the constituent weight parameters. We experimented with various degrees of sparsity, using BEP to prune a portion of filters/neurons of each layer.13 We present the results in Table 2. Our approach better preserves performance at equivalent sparsity. A lower penalty yields higher performing results showing λ serves well at balancing performance vs. computational budget.
We investigate the robustness of BEP and MOGP hyperparameters. We vary the number of MOGP variational inducing points, MOGP latent functions, and Tstep and observe the performance of BEP 1e−4 on CIFAR-10/CIFAR-100 at 80%, 90%, and 95% sparsity. We present these results in Table 3. We observe that in general, all hyperparameters are robust to changes. Mild degradation is observed in the extremal hyperparameter settings.

heading: 4.3 SPEEDING UP RESNET TRAINING ON IMAGENET
text: Our chief goal in this work is to speed up training of large-scale DNNs such as ResNet (He et al., 2016a;b) on the ImageNet dataset. Pruning ResNet requires a careful definition of network element saliency to allow pruning of all layers. ResNet contains long sequences of residual units with matching number of input/output channels. The inputs of residual units are connected with shortcut connections (i.e., through addition) to the output of the residual unit.14 Due to shortcut connections, this structure requires that within a sequence of residual units, the number of inputs/output channels of all residual units must match exactly. This requires group pruning of residual unit channels for
13In our observations saliency measurements don’t well capture network element efficacy when comparing across layers. Thus pruning whole networks using network element saliency yields poor performing networks with bottlenecks. This limitation of saliency functions is well known (See Molchanov et al. (2017) Appendix A.1 and A.2; Wang et al. (2020) Section 3 last paragraph). Development of saliency functions which overcome this shortcoming while remaining performant is a difficult open problem outside the scope of this work.
14Precise details of the ResNet architecture may be found in He et al. (2016a) Section 3.
a sequence of residual units, where group pruning an output channel of a residual unit sequence requires pruning it from the inputs/outputs of all residual units within the sequence.15
We trained ResNet-50 with BEP as well as SNIP and GraSP.16 We group pruned less aggressively as residual unit channels feed into a large number of residual units, thus making aggressive pruning likely to degrade performance. We ran BEP iterations at t = [15, 20, 25, 35, 45, 55, 75] epochs. We trained for 100 epochs on 4× Nvidia Geforce GTX 1080 Ti GPUs. More experimental details found in Appendix G.2. We present our results in Table 4. We achieve higher performance than related techniques, albeit at longer wall time. Our approach captures the training time vs. performance tradeoff present in DNNs, unlike competing approaches.

heading: 5 RELATED WORK
text: Pruning and related techniques. Initial works in DNN pruning center around saliency based pruning after training including Skeletonization (Mozer & Smolensky, 1988), Optimal Brain Damage and followup work (Hassibi & Stork, 1992; LeCun et al., 1989) as well as sensitivity based pruning (Karnin, 1990). In recent years, saliency functions been adapted to pruning neurons or convolutional filters. Li et al. (2017) define a saliency function on convolutional filters by using the L1 norm. Molchanov et al. (2017) propose using a first-order Taylor-series approximation on the objective function as a saliency measure. Dong et al. (2017) propose layer wise pruning of weight parameters using a Hessian based saliency measure. Several variants of pruning after training exist. Han et al. (2015) propose iterative pruning where pruning is performed in stages alternating with fine tune training. Guo et al. (2016) suggest dynamic network surgery, where pruning is performed on-the-fly during evaluation time. Li et al. (2017) and He et al. propose reinforcement learning for pruning decisions. A comprehensive overview may be found in Gale et al. (2019).
Knowledge distillation (Hinton et al., 2015; Lu et al., 2017; Tung & Mori, 2019; Yim et al., 2017) aim to transfer the capabilities of a trained network into a smaller network. Weight sharing (Nowlan & Hinton, 1992; Ullrich et al., 2017) and low rank matrix factorization (Denton et al., 2014; Jaderberg et al., 2014) aim to compress the parameterization of neural networks. Network quantization (Courbariaux et al., 2015; Hubara et al., 2017; Micikevicius et al., 2018) use lower fidelity representation of network elements (e.g. 16-bit) to speed up training and evaluation. Although speedup during training is achievable through network quantization, this technique requires hardware support and only
15We formally define saliency on residual unit sequences in Appendix G.3 16We omit comparison to DSR due to differing underlying deep learning library which makes walltime
comparisons inaccurate.
SNIP 72.0% 90.6% 27.9h 0.7h 62.0% 83.8% 15.8h 0.7h 50.9% 74.2% 21.7h 0.7h GraSP 72.1% 90.6% 27.1h 2.7h 61.6% 83.6% 16.5h 2.7h 52.2% 75.4% 21.7h 2.7h BEP 1e−1 72.2% 90.6% 31.6h 2.9h 62.0% 83.8% 22.8h 1.5h 53.7% 76.8% 27.8h 2.5h BEP 1e−4 72.5% 91.0% 34.8h 2.2h 62.3% 84.5% 22.0h 1.8h 53.5% 76.6% 27.8h 2.6h
provides coarse granularity in trading off computational effort vs. performance. Current GPUs only extend native support to 16-bit floating point operations. Furthermore, our approach is orthogonal to quantization allowing the techniques to be combined for further speedup.
Initialization time or training time pruning. Frankle & Carbin (2019) show that a randomly initialized DNN contains a small subnetwork, which if trained by itself, yields equivalent performance to the original network. SNIP (Lee et al., 2019) and GraSP (Wang et al., 2020) propose pruning connection weights prior to the training process through a first order and second order saliency function respectively. Sparse Evolutionary Training (Mocanu et al., 2018) propose initializing networks with sparse topology prior to training. Narang et al. (2017) consider connection weight pruning during training for recurrent neural networks using a heuristic approach.
Dynamic sparse reparameterization considers pruning and regrowing parameter weights during the training process (Bellec et al., 2018; Dettmers & Zettlemoyer, 2019; Mostafa & Wang, 2019). Dai et al. (2019) propose a grow and prune approach to learning network architecture and connection layout. We differ from existing work as our focus is on speeding up neural network training, meanwhile other works in training time pruning aim to achieve sparse network layouts. To the best of our knowledge, except for small speedups presented in (Dettmers & Zettlemoyer, 2019), the above works do not demonstrate speedup during training time using popular deep learning libraries run on modern GPUs.
PruneTrain (Lym et al., 2019) also proposes pruning filters during training to achieve speedup while minimizing degradation to performance with periodic pruning iterations. In contrast to our approach, PruneTrain does not allow specification of the desired network size after training. A specified network size may be useful if training for resource constrained devices such as mobile phones or edge devices. We compare with PruneTrain under the early pruning problem definition in Appendix E.

heading: 6 CONCLUSION
text: This paper presents a novel efficient algorithm to perform pruning of DNN elements such as neurons, or convolutional layers during the training process. To achieve early pruning before the training converges while preserving the performance of the DNN upon convergence, a Bayesian model (i.e., MOGP) is used to predict the saliency of DNN elements in the future (unseen) training iterations by exploiting the exponentially decaying behavior of the saliency and the correlations between saliency of different network elements. Then, we exploit a property (Lemma 1) of the objective function and propose an efficient Bayesian early pruning algorithm. Empirical evaluations on benchmark datasets show that our algorithm performs favorably to related works for pruning convolutional filters and neurons. Our approach remains flexible to changes in saliency function, and appropriately balances the training time vs. performance tradeoff in training DNNs. We are able to train an early pruned ResNet-50 model achieving a 48.6% speedup (37h vs. 55h) while maintaining a validation accuracy of 72.5%.

heading: A MODELING DETAILS
text: A.1 SALIENCY FUNCTION
In this work, we use a first order Taylor-series saliency function proposed by Molchanov et al. (2017). Our design (Section 3) remains flexible to allow usage of arbitrary saliency functions in a plug-n-play basis. We partition a DNN of L layers, where each layer ` contains C` convolutional filters, into a sequence of convolutional filters [z`,c] c=1,...,C` `=1,...,L . Each filter z`,c : RC`−1×W`−1×H`−1 → RW`×H` can be considered as one network element in vT and z`,c(P`−1) , R(W`,c ∗ P`−1 + b`,c) where W`,c ∈ RC`×O`×O ′ ` , b`,c are kernel weights and bias.with receptive field O` × O′`, ‘∗’ represents the convolution operation, R is the activation function, P`−1 represents the output of z`−1 , [z`−1,c′ ]c′=1,...,C`−1 with P0 corresponding to an input xd ∈ X , and W`, H` are width and height dimensions of layer ` for ` = 1, . . . , L. Let Nz`:z`′ , z`′◦, . . . , ◦z` denote a partial neural network of layers [`, . . . , `′]1≤`≤`′≤L. The Taylor-series saliency function on the convolutional filter z`,c denoted as s([`, c]) is defined17:
s([`, c]) , 1
D D∑ d=1 ∣∣∣∣∣∣ 1W` ×H` W`×H`∑ j=1 ∂L(P(xd)` , yd;Nz`+1:zL) ∂P (xd) `,c,j P (xd) `,c,j ∣∣∣∣∣∣ . (8) where P(xd)` is the output of the partial neural network Nz1:z` with xd as the input and [Pxd`,c,j ]j=1,...,W`×H` interprets the output of the c-th filter in vectorized form. This function uses the first-order Taylor-series approximation of L to approximate the change in loss if z`,c was changed to a constant 0 function. Using the above saliency definition, pruning filter z`,c corresponds to collectively zeroing W`,c, b`,c as well as weight parameters18 [W`+1,c′,{:,:,c}]c′=1,...,C`+1 of z`+1 which utilize the output of zl,c. This definition can be extended to elements (e.g. neurons) which output scalars by setting W` = H` = 1.
A.2 ON THE CHOICE OF THE “EXPONENTIAL KERNEL”
We justify our choice of the exponential kernel as a modeling mechanism by presenting visualizations of saliency measurements collected during training, and comparing these to samples drawn from the exponential kernel kq(t, t′) , β α
(t+t′+β)α , as shown in Figs. 3-4. Both the saliency and the function samples exhibit exponentially decaying behavior, which makes the exponential kernel a strong fit for modeling saliency evolution over time.
Furthermore we note that the exponential kernel was used to great effect in Swersky et al. (2014) with respect to modeling loss curves as a function of epochs. Loss curves also exhibit asymptotic behavior, similar to saliency measurement curves, thus providing evidence for the exponential kernel being an apt fit for our task.
A.3 PREDICTIVE DISTRIBUTION OF THE SALIENCY
Let the prior covariance matrix be Kτ1:τ2 , [cov[s a t , s
a′ t′ ]] a,a′=1,...,M t,t′=τ1,...,τ2
for any 1 ≤ τ1 ≤ τ2 ≤ T . Given a vector of observed saliency s̃1:t, the MOGP regression model can provide a Gaus-
17For brevity, we omit parameters X , Y ,Nz1:zL , L. 18Here we use {} to distinguish indexing into a tensor from indexing into the sequence of tensors [W`+1,c′ ].
sian predictive distribution p(st′ |s̃1:t) = N (µt′|1:t,Kt′|1:t) for any future saliency st′ with the following posterior mean vector and covariance matrix: µt′|1:t , K[t′t]K −1 1:t s̃1:t, Kt′|1:t , Kt′:t′ −K[t′t]K−11:tK>[t′t] where K[t′t] , [cov[s a t′ , s a′ τ ]] a,a′=1,...,M τ=1,...,t . Then, the a-th element µ a t′|1:t of µt′|1:t is the predictive mean of the saliency sat′ . And the [a, a ′]-th element of K[t′t] denoted as σaa ′
t′|1:t is the predictive (co)variance between the saliency s a t′ and s
a′ t′ .
B SUBMODULARITY OF E[ρ̂T ]
In (6), the problem of choosing m from {0, 1}M can be considered as selecting a subset A of indexes from {1, . . . ,M} such that mat = 1 for a ∈ A, and mat = 0 otherwise. Therefore, P (m) , Ep(sT |s̃1:t)[ρ̂T (m, Bs)] can be considered as a set function which we will show to be submodular. To keep notation consistency, we will remain using P (m) instead of representing it as a function of the index subset A.
Lemma 2 (Submodularity). Let m′, m′′ ∈ {0, 1}M , and e(a) be arbitrary M-dimensional one hot vector with 1 ≤ a ≤M . We have P (m′ ∨ e(a))− P (m′) ≥ P (m′′ ∨ e(a))− P (m′′) for any m′ ≤̇m′′,m′ ∧ e(a) = 0, andm′′ ∧ e(a) = 0.
Proof. According to (3), Ep(sT |s̃1:t)[ρ̂T (m, Bs)] = Ep(sT |s̃1:t) [ max mT [ mT · s̃T , s.t. ||mT ||0 ≤ Bs,mT ≤̇m ]] Let α(m) , arg maxmT [ mT · s̃T , s.t. ||mT ||0 ≤ Bs,mT ≤̇m ] return the optimized mask mT given anym, Λm , min(α(m) sT ) be the minimal saliency of the network elements selected at iteration T for P (m). Then, we have
P (m ∨ e(a)) = Ep(sT |s̃1:t) [ ρ̂T (m ∨ e(a), Bs) ] = Ep(sT |s̃1:t) [ρ̂T (m, Bs)− Λm + max(s a T ,Λm)]
The second equality is due to the fact that the network element vaT would only replace the lowest included element inmT in order to maximize the objective. Then,
P (m ∨ e(a))− P (m) = Ep(sT |s̃1:t) [ρ̂T (m, Bs)− Λm + max(s a T ,Λm)]− Ep(sT |s̃1:t) [ρ̂T (m, Bs)]
= Ep(sT |s̃1:t) [−Λm + max(s a T ,Λm)] = Ep(sT |s̃1:t) [max(s a T − Λm, 0)] (9)
Givenm′ ≤̇m′′, we have Λm′ ≤ Λm′′ sincemT ≤̇m in α(m′) is a tighter constraint than that in α(m′′). Consequently, we can get sat − Λm′ ≥ sat − Λm′′ , and thus
[P (m′ ∨ e(a))− P (m′)] ≥ [P (m′′ ∨ e(a))− P (m′′)] .

heading: C PROOF OF LEMMA 1
text: We restate Lemma 1 for clarity.
Lemma 1. Let e(i) be an M -dimensional one-hot vectors with the i-th element be 1. ∀ 1 ≤ a, b ≤ M ; m ∈ {0, 1}M s.t.m ∧ (e(a) ∨ e(b)) = 0. Given a vector of observed saliency s̃1:t , if µaT |1:t ≥ µ b T |1:t and µ a T |1:t ≥ 0, then
Ep(sT |s̃1:t)[ρT (m ∨ e(b))]− Ep(sT |s̃1:t)[ρT (m ∨ e(a))] ≤ µbT |1:t Φ(ν/θ) + θ φ(ν/θ)
where θ , √ σaaT |1:t + σ bb T |1:t − 2σ ab T |1:t , ν , µ b T |1:t − µ a T |1:t , and Φ and φ are standard normal CDF and PDF, respectively.
To prove this Lemma, we prove the following first: Lemma 3. Ep(sT |s̃1:t) [ ρT (m ∨ e(b)) ] − Ep(sT |s̃1:t) [ ρT (m ∨ e(a)) ] ≤ E[max(sbT − saT , 0)].
Proof. Due to (9), we have
Ep(sT |s̃1:t) [ ρT (m ∨ e(b)) ] − Ep(sT |s̃1:t) [ ρT (m ∨ e(a)) ] = P (m ∨ e(b))− P (m)− (P (m ∨ e(a))− P (m)) = Ep(sT |s̃1:t) [ max(sbT − Λm, 0) ] − Ep(sT |s̃1:t) [max(s a T − Λm, 0)]
= Ep(sT |s̃1:t) [ max(sbT − Λm, 0)−max(saT − Λm, 0) ] (10)
= Ep(sT |s̃1:t) [ max(sbT − saT ,Λm − saT )−max(0,Λm − saT ) ] (11)
≤ Ep(sT |s̃1:t) [ max(sbT − saT , 0) ] (12)
The equality (11) is achieved by adding Λm − saT in each term of the two max functions in (10). The inequality (12) can be proved by considering the following two cases:
If Λm − saT ≥ 0, then
max(sbT − saT ,Λm − saT )−max(0,Λm − saT ) = max(sbT − saT ,Λm − saT )− (Λm − saT ) = max(sbT − saT − (Λm − saT ), 0) ≤ max(sbT − saT , 0) .
If Λm − saT < 0, then
max(sbT − saT ,Λm − saT )−max(0,Λm − saT ) = max(sbT − saT ,Λm − saT ) ≤ max(sbT − saT , 0) .
Next we utilize a well known bound regarding the maximum of two Gaussian random variables (Nadarajah & Kotz, 2008), which we restate:
Lemma 4. Let sa, sb be Gaussian random variables with means µa, µb and standard deviations σa, σb, then E[max(sa, sb)] ≤ µaΦ ( µb−µa θ ) + µbΦ ( µb−µa θ ) + θφ ( µb−µa θ ) where θ ,√
[σb]2 + [σa]2 − 2cov(sb, sa) and Φ, φ are standard normal CDF and PDF respectively.
Then,
Ep(sT |s̃1:t)[max(s b T − saT , 0)]
= Ep(sT |s̃1:t)[max(s b T , s a T )]− Ep(sT |s̃1:t)[s a T ]
≤ (µbT |1:t + µ a T |1:t)Φ (µbT |1:t − µaT |1:t θ ) + θφ (µbT |1:tµaT |1:t θ ) − µaT |1:t
= µbT |1:tΦ (µbT |1:t − µaT |1:t
θ
) + θφ (µbT |1:tµaT |1:t θ ) + µaT |1:t
( Φ ( µbT |1:t − µ a T |1:t
θ
) − 1 )
≤ µbT |1:tΦ (µbT |1:t − µaT |1:t
θ
) + θφ (µbT |1:t − µaT |1:t θ ) The first inequaltiy follows from Lemma 4. The second inequaltiy is due to Φ (µbT |1:t−µaT |1:t θ ) ≤ 1 and µaT |1:t ≥ 0.

heading: D DYNAMIC PENALTY SCALING AS A FEEDBACK LOOP
text: We designed a feedback loop to automatically determine λt during early pruning. A proportional feedback loop can be defined as follows19:
λt , λ+Kp × e(t) (13)
where Kp ≥ 0 is a proportional constant which modulates λt according to a signed measure of error e(·) at time t. Note that λt ≥ λ as e(t) ≥ 0, and the opposite occurs if e(t) ≤ 0, which allows the error to serve as feedback to determine λt. Implicitly, λt asserts some control over e(t+ 1), and thus closing the feedback loop.
Traditional PID approaches to determine Kp do not work in our case as λ may vary over several orders of magnitude. Consequently, a natural choice for Kp is λ itself which preserves the same order of magnitude between Kp and λ:
λt = λ+ λ× e(t) = λ(1 + e(t)). (14)
Here we make two decisions to adapt the above to our task. First, as λ is likely to be extremely small, we use exponentiation, as opposed to multiplication. Secondly as λ ≤ 1 in practice, we use 1− e(t) as an exponent:
λt = λ ∧ [1− e(t)] = λ [(1/λ) ∧e(t)] . (15)
The above derivation is complete with our definition of e(t):
e(t) , (T − t)||mt||0/Bt,c − 1. (16)
The above determines error by the discrepancy between the anticipated compute required to complete training (T − t)||mt||0, vs. the remaining budget Bt,c with e(t) = 0 if the two are equal. This is a natural measure of feedback for λ as we expect the two to be equal if λ is serving well to early prune the network.

heading: E COMPARISON WITH PRUNETRAIN
text: We compare with PruneTrain (Lym et al., 2019) in Table 5. PruneTrain uses an orthogonal technique of dynamically increasing the minibatch size to achieve further wall time improvements. This prevents accurate wall time comparisons between BEP and PruneTrain. To compare with PruneTrain which
19This approach is inspired from Proportional-Integral-Derivative (PID) controllers (Bellman, 2015), see Åström et al. (1993) for an introductory survey.
does not constrain the trained network size (3b), we train ResNet-50 under equivalent inference cost as a network trained by PruneTrain. To compute train/inference cost (FLOPs) for a convolutional layer, we used a formula defined in Molchanov et al. (2017) A.1:
FLOPs , 2HW (CinK2 + 1)Cout (17)
where H , W , Cin are input height, width, and channels respectively, K is the convolutional kernel size, and Cout is the number of output channels of the layer.
Under equivalent inference cost, BEP 1e−1 outperforms PruneTrain in Top-1 performance. We also find that BEP 1e−1 and BEP 1e−4 consumes fewer training FLOPs when compared to baseline. It should be noted that PruneTrain does not provide a mechanism to constrain the trained network size, thus it is unclear how to utilize it in order to solve the early pruning problem (3), (4).

heading: F TABLE OF NOTATIONS
text: We list a table of notations used elsewhere in the paper in Table 6.
GP 1.19(0.5) 1.08(0.06) 1.07(1.07)e5 0.96(0.04) 0.93(0.03) 2.47(0.04) 0.49(0.01) 0.48(0.01) 1.33(0.02) 4-MOGP 1.15(0.05) 0.89(0.06) 2.44(0.05) 0.91(0.02) 0.80(0.03) 2.20(0.03) 0.38(0.02) 0.39(0.02) 1.25(0.02) 8-MOGP 1.09(0.04) 0.86(0.05) 2.38(0.04) 0.84(0.03) 0.78(0.03) 2.16(0.03) 0.32(0.01) 0.35(0.02) 1.20(0.02) 18-MOGP 0.97(0.04) 0.80(0.05) 2.33(0.04) 0.89(0.03) 0.76(0.03) 2.13(0.03) 0.31(0.01) 0.35(0.02) 1.20(0.02) 32-MOGP 0.96(0.06) 0.81(0.06) 2.32(0.04) 0.79(0.03) 0.74(0.03) 2.13(0.03) 0.31(0.01) 0.34(0.02) 1.20(0.02)

heading: G MORE EXPERIMENTAL RESULTS AND EXPERIMENTAL DETAILS
text: G.1 GP VS. MOGP LOG-LIKELIHOOD ON CIFAR-10 DATASET
Table 7 presents the results of the experiment in Section 4.1 for the CIFAR-10 dataset.
G.2 EXPERIMENTAL DETAILS
To train our CIFAR-10 and CIFAR-100 models we used an Adam optimizer (Kingma & Ba, 2015) with an initial learning rate of 0.001. The learning rate used an exponential decay of k = 0.985, and a batch size of 32 was used. Training was paused three times evenly spaced per epoch. During this pause, we collected saliency measurements using 40% of the training dataset. This instrumentation subset was randomly select from the training dataset at initialization, and remained constant throughout the training procedure. We performed data preprocessing of saliency evaluations into a standardized [0, 10] range.20 We used (8) to measure saliency of neurons/convolutional filters. For the convolutional layers we used 12 latent MOGP functions. For the dense layer we used 4 latent MOGP functions.
For our ResNet-50 model we used an SGD with Momentum optimizer with an initial learning rate of 0.1. The learning rate was divided by ten at t = [30, 60, 80] epochs. We collected saliency data every 5 iterations of SGD, and averaged them into buckets corresponding to 625 iterations of SGD to form our dataset. We used a minimum of 4 latent functions per MOGP, however this was dynamically increased if the model couldn’t fit the data up to a maximum of 15.
We sampled 10K points from our MOGP model to estimate ∆(·) for CIFAR-10/CIFAR-100. For ResNet we sampled 15K points. We repeated experiments 5 times for reporting accuracy on CIFAR10/CIFAR-100.
G.3 PRUNING ON RESNET
ResNet architecture is composed of a sequence of residual units: Z` , F(P`−1) + P`−1, where P`−1 is the output of the previous residual unit Z`−1 and ‘+’ denotes elementwise addition. Internally, F is typically implemented as three stacked convolutional layers: F(P`−1) , [z`3 ◦ z`2 ◦ z`1 ] (P`−1) where z`1 , z`2 , z`3 are convolutional layers. Within this setting we consider convolutional filter pruning. Although z`1 , z`2 may be pruned using the procedure described earlier. Pruning z`3 requires a different procedure. Due to the direct addition of P`−1 to F(P`−1), the output dimensions of Z`−1 and z`3 must match exactly. Thus a ResNet architecture consists of sequences of residual units of length B with matching input/output dimensions: ζ , [Z`]`=1,...,B , s.t. dim(P1) = dim(P2) = . . . = dim(PB). We propose group pruning of layers [z`3 ]`=1,...,B where filters are removed from all z`3 in a residual unit sequence in tandem. We define s([ζ, c]) , ∑B `=1 s([`3, c]), where s(·) is defined for convolutional layers as in (8). To prune the channel c from ζ , we prune it from each layer in [z`3]`=1,...,B . Typically we pruned sequence channels less aggressively than convolutional filters as these channels feed into several convolutional layers.
20Generally, saliency evaluations are relatively small (≤ 0.01), which leads to poor fitting models or positive log-likelihood. Precise details of our data preprocessing is in Appendix G.4.
G.4 DATA PREPROCESSING
We followed the same data preprocessing procedure for both our small scale and ImageNet experiments. To standardize the saliency measurements for a training dataset s̃1:t in our modeling experiments we clip them between 0 and an upper bound computed as follows: ub , percentile(s̃1:t, 95) × 1.3. This procedure removes outliers. We used 1.3 as a multiplier, as this upper bound is used to transform test dataset as well, which may have higher saliency evaluations.
After clipping the training data, we perform a trend check for each element va by fitting a Linear Regression model to the data s̃a1:t. For s̃ a 1:t with an increasing trend (i.e., the linear regression model has positive slope) we perform the transformation s̃a1:t = ub− s̃a1:t. The reasoning behind this is that the exponential kernel strongly prefers decaying curves. After this preprocessing, we scale up the saliency measurements to a [0, 10] range: s̃1:t = s̃1:t × 10. We found that without scaling to larger values, log-likelihood of our models demonstrated extremely high positive values due to small values of unscaled saliency measurements.
We transform the test data in our modeling experiments s̃t+1:T with the same procedure using the same ub and per-element va regression models as computed by the training data. We measure log-likelihood after this transformation for both the test dataset in our small scale experiments.
During the BEP Algorithm, the same steps are followed, however we inverse the trend check transformation (s̃a1:t = ub− s̃a1:t) on the predicted MOGP distribution of sT prior to sampling for estimation of ∆(·).

Review summaries:
This paper introduces a method for pruning during the training process in order to filter out unimportant/redundant components of the network continuously to speed up training and perform gradual pruning over the training process. The proposed approach is novel in the sense that the vast amount of prior work on pruning has focused on either (i) pruning on network initialization (e.g., SNIP, etc.) or (ii) pruning after the network has been fully trained (e.g., Magnitude Pruning, among many others). The introduced method uses the Taylor-series based saliency criterion (of Molchanov et al., 2017) and uses a multi-output Gaussian process to predict future saliencies and to determine whether a parameter can be safely removed early on during training.

This paper introduces a new method to accelerate training by saliency-based pruning. The method predicts future saliency for neurons based on observed saliency with a multi-output Gaussian process (MOGP), then greedily prunes neurons with least saliency at fixed intervals during training. The authors provide extensive mathematical analysis to show that the algorithm produces pruning mask solutions that are close to the optimum of the formulated optimization (the reviewer is unable to verify). The experimental results showed improvements in task accuracies of trained models but with longer training times.