Optimizing Learning Rate Schedules for Iterative Pruning of Deep Neural Networks
Authors: Shiyu Liu
Rohan Ghosh
John Chong Min Tan
Mehul Motani
Venue: Accepted by TMLR ()

Abstract: The importance of learning rate (LR) schedules on network pruning has been observed in a few recent works. As an example, Frankle and Carbin (2019) highlighted that winning tickets (i.e., accuracy preserving subnetworks) can not be found without applying a LR warmup schedule. Renda, Frankle and Carbin (2020) also demonstrated that rewinding the LR to its initial state at the end of each pruning cycle can improve pruning performance. In this paper, we go one step further by first providing a theoretical justification for the surprising effect of LR schedules. Next, we propose a LR schedule for network pruning called SILO, which stands for S-shaped Improved Learning rate Optimization. The advantages of SILO over existing LR schedules are two-fold: (i) SILO has a strong theoretical motivation and dynamically adjusts the LR during pruning to improve generalization. Specifically, SILO increases the LR upper bound (max_lr) in an S-shape. This leads to an improvement of 2% - 4% in extensive experiments with various types of networks (e.g., Vision Transformers, ResNet) on popular datasets such as ImageNet, CIFAR-10/100. (ii) In addition to the strong theoretical motivation, SILO is empirically optimal in the sense of matching an Oracle, which exhaustively searches for the optimal value of max_lr via grid search. We find that SILO is able to precisely adjust the value of max_lr to be within the Oracle optimized interval, resulting in performance competitive with the Oracle with significantly lower complexity.

Published in Transactions on Machine Learning Research (08/2023)
Optimizing Learning Rate Schedules for Iterative Pruning of Deep Neural Networks
Shiyu Liu
shiyu www..nus..cc.om
Deportment of Electrical and Computer Engineering
College of Design and Engineering
National University of Singapore
Rohan Ghosh
rghosh.92@gmaiillcoo com
Department of Electrical and Computer Engineering College of Design and Engineering
National Universily of Singapore
John Chong Min Tan
johntancm Cu. wtw.mus..dd.......
Department of Electrical and Computer Engineering College of Design and Engineering
National University of Singapore
Mehul Motani
molani@nus..c www.wiww..u......... edu.sg
Deportment of Electrical and Computer Engineering College of Design and Engineering
N.1 Institute for Health
Institute of Data Science
Institute for Digital Medicine (WisDM) National Universily of Singapore
Reviewed on OpenReview: https: // openreview. net/ forum? id= nGW2Hotpq3
Abstract
The importance of learning rate (LR) schedules on network pruning has been observed in few recent works. As example, Frankle and Carbin (2019) highlighted that winning tickets (i.e., accuracy preserving subnetworks) schedule. Renda, Frankle and Carbin (2020) also demonstrated that rewinding the LR to its initial state at the end of each pruning cycle can improve pruning performance. In this paper, we go one step further by first providing a theoretical justification for the surprising effect of LR schedules. Next, we propose LRR schedule for network pruning called STL.O, which stands for S-shaped Improved Learning rate Optimization. The advantages of SILO existing LR schedules
not be found without applying LR warmup
two-fold: (i) SILO has strong theoretical motivation and dynamically adjusts the LR during pruning to improve generalization. Specifically, SILO increases the LR upper bound (max_ 1r) in an S-shape. This leads to an improvement of 2% - 4% in extensive experiments with various lypes of networks (e.g., Vision Transformers, ResNet) on popular datasets such as ImageNet, CIFAR-10/100. (ii) In addition to the strong theoretical motivation, SILO is empirically optimal in the sense matching an Oracle, which exhaustively searches for the optimal value of max ir via grid search. We find that SILO is able to precisely adjust the value of max Ir be within the Oracle optimized interval, resulting in performance competitive with the Oracle with significantly lower complexity.

---

Published in Transactions on Machine Learning Research (08/2023)
1 Introduction
Network pruning is the process of simplifying neural networks by pruming weights, filters or neurons. (LeCun et al., 1990; Han et al., 2015b). Several saate-o--thearr pruning methods (Renda et al., 2019; Frankle & Carbin, 2019) have demonstrated that a significant quantity of parameters can be removed without sacrificing accuracy. This greatly reduces the resource demand of neural networks, such as storage requirements and energy consumption (He et al., 2020; Wang et al., 2021; Good & el al, 2022; Vysogorets & Kempe, 2023).
The inspiring performance of pruming methods hinges on a key factor Learning Rate (LR). Specifically, Frankle & Carbin (2019) proposed the Lottery Ticket Hypothesis and demonstrated that the winning tickets (i.e., the pruned network that can train in isolation to full accuracy) can not be found without applying a LR warmup schedule. In a follow-up work, Renda et al. (2019) proposed LR rewinding which rewinds the LR schedule to its initial state during iterative pruning and demonstrated that it can outperform standard fine-tuning. In summary, the results both works suggest that, besides the pruning metric, LR also plays an important role in network pruning and could be another key to improving the pruning performance.
In this paper, take existing studics one step further and aim to optimize the choice of LR for iterative network pruming, We explore new perspective on adapting the LR schedule to improve the iterative pruming performance. In the following, we summarize the workflow of our study together with our contributions.
1. Motivation and Theoretical Study. In Section 3.1, we explore the optimal choice of LR during network pruming and find that the distribution of weight gradients tends to become narrower during pruming, suggesting that a larger value of LR should be used to retrain the pruned network. This finding is further verified by our thcoretical development in Section 3.2. More importantly, our theoretical results suggest that the optimal increasing trajectory of LR should follow an S-shape.
2. Proposed SILO. In Section 4, propose novel LR schedule for network pruning called SILO, which stands for S-Shaped Improved Learning rate Optimization. Motivated by our theoretical development, SILO precisely adjusts the LR. by increasing the LR. upper bound (max_ 1r) in an S-shape. We highlight that SILO is method agnostic and works well with numerous pruning methods.
3. Experiments. In Section 5.2, we compare SILO to four LR schedule benchmarks via both classical and state-of-the-an the-art (SOTA) pruning methods. We observe that SILO outperforms LR schedule benchmarks, leading to an improvement of 2% 4% in extensive experiments with various networks (e.g.. Vision Trans- former (Dosovitskiy et al., 2020), ResNet (He et al., 2016), VGG (Simonyan & Zisserman, 2014), DenseNet-40 (Huang et al., 2017) and MobileNetV2 (Sandler et al., 2018)) on large-scale datasets such as ImageNet (Deng et al., 2009) and popular datasets such as CIFAR-10, C-FAR-100 (Krizhevsky et al., 2009).
4. Comparison to Oracle. In Section 5.4, examine the optimality of SILO by comparing it to an Oracle which exhaustively searches for the optimal value of max_lr via grid search. We find that SILO is able to precisely adjust max_Ir Ir to be within the Oracle's optimized max_Ir lr interval at each pruming cycle, resulting in performance competitive with the Oracle, but with significantly lower complexity.
2 Background
In Section 2.1, we first review prior works on network pruning. Next, in Section 2.2, we highlight the important role of L.R in network pruning and position our work in the context of network pruning.
2.1 Prior Works Network Pruning
Network pruning is an established idea dating back to 1990 (LeCun et al., 1990). The motivation is that networks tend to be overparameterized and redundant weights can be removed with a negligible loss in accuracy (Arora el al., 2018; Allen-Zhu et al., 2019; Denil et al., 2013). Given a trained network, one pruning cycle consists of three steps: (i) Prune the network according to certain heuristics; (ii) Freeze pruncd parameters as zero. (iii) Retrain the pruned network recover the accuracy. Repeating the pruning cycle multiple times until the target sparsity or accuracy met is known as iterative pruning. Doing often results in better performance than one-shot pruning (ie.,, perform only one pruning cycle) (Han
2

---

Published in Transactions on Machine Learning Research (08/2023)
et al., 2015b; Frankle& & Carbin, 2019; Li et al., 2017). There two types of network pruning unstructured pruning and structured pruning- which will be discussed in detail below.
Unstructured Pruning removes individual weights according to certain heuristics, such as magnitude (Han et al., 2015b; Frankle & Carbin, 2019) or gradient (Hassibi & Stork, 1993; LeCun et al., 1990; Lee et al., 2019; Xiao et al., 2019; Theis et al., 2018). Examples are (LeCun et al., 1990), which performed pruning based on the Hessian Matrix, and (Theis et al., 2018), which used Fisher information to approximate the Hessian Matrix. Similarly, Han et al. (2015b) removed weights with the smallest magnitude, and this approach was further incorporated with the three-stage iterative pruning pipeline in (Han et al., 2015a).
Structured Pruning involves pruning weights in groups, neurons, channels or filters (Yang et al., 2019; Molchanov et al., 2017; 2019; Luo et al., 2017; Yu et. al., 2018; Tan & Motani, 2020; Wang et al., 2020b; Lin et al., 2020; Zhang& & Freris, 2023). Examples are (Hu et al., 2016), which removed neurons with high average zero output ratio, and (Li et al., 2017), which pruned ncurons with the lowest absolute summation values of incoming weights. More recently, Yu et al. (2018) proposed the neuron importance score propagation algorithm to evaluate the importance of network structures. Molchanov et al. (2019) used Taylor expansions to approximate a filter's contribution to the final loss and Wang et al. (2020a) optimized the neural network architecture, pruning policy, and quantization policy together in a joint manner.
Other Works In addition to works mentioned above, several other works also share some deeper insights in network pruning (Liu et al., 2019b; Wang et al., 2020c; Li & et al, 2022; Wang & et al, 2022). For example, Liu al. (2019a) demonstrated that training-from-sceee fcom-scratch on the right sparse architecture yields better results than pruning from pre-trained models. Similarly, Wang et al. (2020c) suggested that the fully- trained network could reduce the search space for the pruned structure. More recently, Luo & Wu (2020) addressed the issue of pruning residual connections with limited data and Ye et al. (2020) theoretically proved the existence of small subnetworks with lower loss than the unpruned network. You & et al (2022) motivated the use of the affine spline formulation of networks to analyze recent pruning techniques. Liu et al. (2022) applied the network pruning technique in graph networks and approximated the subgraph edit distance. One milestone paper (Frankle & Carbin, 2019) pointed out. that re-initialiving with the original parameters (known as weight rewinding) plays an important role in pruning and helps to further prune the network with negligible loss in accuracy. Some follow-on works (Zhou et al., 2019; Renda et al., 2019; Malach et al., 2020) investigated this phenomenon more precisely and applied this method in other fields (e.g., transfer learning (Mehta, 2019) and natural language processing (Yu et al., 2020)).
2.2 The Important Role of Learning Rate
Several recent works (Renda et al., 2019; Frankle & Carbin, 2019) have noticed the important role of LR in network pruning. For example, Frankle & Carbin (2019) demonstrated that training VGG-19 with a LR warmup schedule (i.e., increase LR to le-1 and decrease it to 1e-3) and a constant LR of 1e-2 results in comparable accuracy for the umpruned network. However, as the network is iteratively pruned, the LR warmup schedule leads to higher accuracy (sce Fig.7 in (Frankle & Carbin, 2019)). In a follow-up work, Renda et al. (2019) further investigated this phenomenon and proposed retraining technique called LR rewinding which can always outperform the standard retraining technique called fine-tuning (Han et al., 2015b). The difference is that fine-tuning trains the unpruned network with LR warmup schedule, and retrains the pruned network with a constant LR (i.., the final LR of the schedule) in subsequent pruming cycles (Liu et al., 2019b). LR rewinding rainsins the pruned network by rewinding the LR warmup schedule to its initial state, namoly that LR rewinding uses the same schedule for every pruning cycle. As an example, they demonstrated that retraining the pruned ResNet-50 using LR rewinding yields higher accuracy than fine-tuning (see Figs.1 & 2 in (Renda et al., 2019)). In (Liu & et al, 2021), the authors also suggest that when pruning happens during the training phase with a large LR, models can casily recover from pruning than using a smaller LR. Overall, The results in these works suggest that, besides the pruning metric, LR also plays an important role in network pruning and could be another key to improving network pruning.
Our work. In this paper, we explore a new perspective on adapting the LR schedule to improve the itcrative pruning performance of ReLU-based networks. The proposed LR schedule is method agnostic and can work well with numerous pruning methods. We mainly focus on iterative pruning of ReL.U-based networks for two
3

---

Published in Transactions on Machine Learning Research (08/2023)
98%
78%
(a)
Unpraited Network
(b)
Ungruned Network 8.33)
90%
70%
Probability Distribution
Probabliity Distribution
Network [i 64.0. 0.31)
88%
Pruned Network [t 13.4, 335e-3)
(2 G4B, 5.2n-1]
68%
Pruned Network [i 13.4 6.27)
80%
78%
68%
10%
5%
5%
0%
0%
-20
-15
- Weight Gradionts (x110")
10
25
38
49
60
71
115
Hidden Representation (x10
Figure 1: (a) The distribution of all weight gradients when iteratively pruning a fully connected ReLU network using global magnitude, where À is the percent of weights remaining and is the standard deviation of the distribution. (b) The corresponding distríbution of hidden representations.
reasons: (i) Tterative pruning tends to provide better pruning performance than one-shot pruning as reported in the literature (Frankle & Carbin, 2019; Renda et al., 2019). (ii) ReLU has been widely used in many classical neural networks (e.g., ResNet, VGG, DenseNet) which have achieved outstanding performance in various tasks (e.g., image classification, object detection) (He et al., 2016; Simonyan & Zisserman, 2014).
3 A New Insight on Network Pruning
In Section 3.1, we first provide a new insight in network pruning using experiments. Next, in Section 3.2, we provide a theoretical justification for our observed new insight and present some relevant. theoretical results.
3.1 Weight Gradients during Iterative Pruning
(1) Experiment Sctup. To exclude the influcnce of other factors, we start from a simple fully connected ReLU-based network with three hidden layers of 256 neurons each (results of other popular networks are summarized later). We train the network using the training dataset of CIFAR-10 via SGD (Ruder, 2016) (momentum 0.9 and weight decay of le-4) with a batch size of 128 for 500 epochs. All hyperparameters are tuned for performance via grid search (e.g., LR from 1e-4 to 1e-2). We apply the global magnitude (Han et al., 2015b) (i.e., remove weights with the smallest magnitude anywhere in the network) with a pruning rate of 0.2 (i.e., prune 20% of the remaining parameters) to iteratively prune the network for 10 pruning cycles and plot the distribution of all weight gradients when the network converges in Fig. 1(a), where X is the percent of weights remaining. In Fig. 1(a), there are 10 visible bins estimated by the Sturges' Rule (Scott, 2009) and cach bin consists of three values (i.c., the probability distribution of networks with different A). The edge values range from -0.022 to 0.027 with a bin width of 0.004.
(2) Experiment Results. In Fig. 1(a), observe that the distribution of weight gradients tends to become narrower, i.c., the standard deviation of weight gradients reduces from 8.4e-3 to 3.3e-3 when the network is iteratively pruned to x = 13.4. As an example, the unpruned network (A = 100) has more than 7% of weight gradients with values greater than 0.008 (rightmost bars) or less than -0.012 (leftmost 2 bars), while the pruned network (Λ = 13.4) has less than 1% of weight gradients falling into those regions. It suggests that the magnitude of weight gradients tends to docrease as the network is iteratively pruned.
(3) New Insight. During the backpropagation, the weight update of wi is w; w. + a ow: where is the LR and L is the loss function. Assume that a is well-tuned to ensure the weight update (i.e., a ac is sufficiently large to prevent the network from getting stuck in local optimal points (Bengio, 2012; Goodfellow et al., 2016). As shown in Fig. 1(a), the magnitude of the weight gradient (i.c., Sey tends to docrease as the network is iteratively pruned. To preserve the same weight updating size and effect as before, a gradually larger value of LR (i.e., a) should be used to retrain the pruned worw during iterative pruning.
OC
4

---

Published in Transactions on Machine Learning Research (08/2023)
(4) Result Analysis. We now provide explanation for the change in the distribution of weight gradients. First, we assume each J,w; (i.e., Xi E R is the neuron input and E R is the associated weight) is an i.i.d. random variable. Then, the variance of the neuron's pre-activation output (Ei=1 T,Ws, is the number of inputs) will be Ei= Var(z,w;). Pruning the network is equivalent to reducing the number of imputs from to n - k. This results in smaller variance of Var(a,, leading to smaller standard deviation. Hence, the distribution of the pre-activation output after pruning is narrower. Since ReLU returns its raw input if the imput is non-negative, the distribution of hidden representations (output of hidden layers) becomes narrower as well. This can be verified from Fig. 1(b), where we plot the distribution of hidden representations from the previous experiment. The key is that the weight gradient dc is proportional to the hidden representation 2; that associates with w; (i.c., Bas the distribution of hidden representations becomes narrower, leading to narrower distribution of weight gradients. As result, a larger LR. should be used to retrain the pruned network.
She;
ac x 24). As the network is iteratively pruned,
(5) More Gencralized Results. (i) Effect of Batch Normalization (BN) (Ioffe & Szegedy, 2015): BN is a popular technique to reformat the distribution of hidden representations, so as to address the issue of internal covariate shift. We note that similar performance trends can be observed after applying BN as well (see Fig. in the Appendix). (ii) Popular CNN Networks and Pruning Methods: In addition to the global magnitude used before, two unstructured pruning methods (i.e., layer magnitude, global gradient) suggested by (Blalock et al., 2020) and one structured pruning method (L1 norm pruning) (Li et al., 2017) are examined as well. Those methods are used to iteratively prune AlexNet (Krizhevsky & Hinton, 2010), ResNet-20 and VGG-19 using CIFAR-10. The results using these popular neural networks largely mirror those in Figs. ((a) & (b) as well. We refer the interested reader to Figs. 5 - in the Appendix.
3.2 Theoretical Study and Motivation
In this subsection, we theoretically investigate how network pruning can influence the value of the desired LR. The proofs of the results given here are provided in the Appendix. First, we present some definitions.
Definition 1. Average Activation Norm (EΛл): Given network with fixed weights, input X from distribution P, and a layer II (X), ... [x())} with N nodes where h;(X) represents the function at the 7th node. Then (AA(H)= = Exl 1 Σ, (.))). This quantity reflects the average strength of the layer's 's activations.
Definition 2. Average Gradient Norm (Ewa): Let W [Wi, wa] and W [w] ..., wol represent the flattened weight. vector before and after one epoch of training (via. backpropagation). Then Fwe is the average change in weight magnitude in a single training epoch (for the active unpruned weights), i.e. Ewa W E.((w; a. EwG quantifies how much the weights change after one epoch of training.
We now demonstrate the impact of network pruning on the average activation norm of hidden layers.
Theorem 1. Consider a ReLU activated neural network represented as X W1 H W2, Y, where X E Rd is the input, H (X), H,,)),.., Hx)) is of infinite width (N oo), and Y is the network output. W1 and W2 represent network parameters (weights, biases). Furthermore, let X (N(0, and W1 N(0, I), where I is the identity matrix and ax, ow are scalars. Now, let us consider an iterative pruning method, where in each iteration a fraction 0 < p S 1 of the smallest magnitude weights are pruned (layer-wise pruning). Then, after & iterations of pruning, it holds that
4EAA(H) 2 oi + doxooi (11
4 e (1-(1-p) ( V==ef"
(1)
where erf (.) is the inverse error function.
Next, based on the above result, the following theorem establishes how the average gradient norm depends on the LR of the neural network, and the pruning iteration.
Theorem 2. In the setting of Theorem 1, we consider a single epoch of weight update for the network across training dataset. S {(X1, Vi), (x), using the cross-entropy loss, where Y; E (0. 1}. Let
5

---

Published in Transactions on Machine Learning Research (08/2023)
denote the learning rate. Let denote the R.H.S of equation 1 by C(ax, ow,,, Let the final layer weights before and after one training epoch be W2 and W2 respectively. We have,
E (a
Eww (W2, W2)1 2 a'nC(oxx a'
(2)
for some constant 7, where No. (0, o represents the distribution of W2 after being initialized as the Gaussian N(0,0 and prunod for & iterations.
Remark 1. (Pruning and LR) Theorems and 2 together establish how the choice of LR influences the lower bound of average gradient norm. Theorem 1 shows that the lower bound of activation norm of the hidden layer decreases as the network is pruned, and as Theorem 2 shows, this also reduces the lower bound of average gradient norm per epoch. Thus, to counter this reduction, it is necessary to increase the learning rate a as the number of pruning cycles grows, in order to ensure that the R.H.S of equation 2 remains fixed. Remark 2. (S-shape of LR During Iterative Pruning) We fix the average gradient norm to negate the impact of weight gradients reducing while pruming (see Fig. 1). Theorem 2 implies that to maintain a fixed average gradient norm of Bw2--Na(0.0 ry (Ew, W3]] = K, we must have the learning rate a S k)) We find that this upper bound resembles an S-shape trajectory during iterative pruning (see the red line in Fig. 2).
(K/7C(ox- ow .w.P. (..P,k))
Remark 3. (Additional Results) Note that we extend Theorem I's result to the case of fully connected neural networks of arbitrary depth in Proposition 1 of the to the case of networks of arbitrary depth Corollary of the Appendix. Although Theorem assumes 2-layer neural networks of infinite width, we extend to the case of finite hidden neurons (sce Proposition 2 of A Appendix), yielding probabilistic bound of the same form as Theorem 1. Note that the results in both Theorems are empirically verified in Section B of the Appendix.
the Appendix. Similarly, we extend Theorem 2's result
4 A New Learning Rate Schedule
In Section 4.1, we shortlist four LR benchmarks for comparison. In Section 4.2, we introduce SILO and highlight the difference with existing works. In Section 4.3, we detail the algorithm for SILO.
4.1 LR Schedule Benchmarks
Learning rate is the most important hyperparameter in training neural networks (Goodfellow et al., 2016). The LR schedule is to adjust the value of LR during training by a pre-defined schedule. Three common LR schedules are summarized as follows.
1. LR Decay starts with large LR and linearly decays it by a certain factor after a pre-defined number of epochs. Several recent works (You et al., 2019; Ge et al., 2019; An et al., 2017) have demonstrated that decaying LR helps the neural network to converge better and avoids undesired oscillations in optimization.
2. LR Warmup is to increase the LR to large value over certain epochs and then decreases the LR by a certain factor. It is a popular schodule used by many practitioners for transfer learning (He et al., 2019) and network pruning (Frankle & Carbin, 2019; Frankle et al., 2020).
3. Cyclical LR (Smith, 2017) varies the LR cyclically between a pre-defined lower and upper bound. It has been widely used in many tasks (You al., 2019).
All the three LR schedules described above and constant LR will be used as benchmarks for performance comparison. We note that, in addition to LR schedules which vary LR by pre-defined schedule, adaptive LR. optimizers such AdaDelta (Zeiler, 2012) and Adam (Kingma & Ba, 2014) provide heuristic based approaches to adaptively vary the step size of weight update based on observed statistics of the past gradients. All of them are sophisticated optimization algorithms and much work (Gandikola el al., 2021; Jentzen et al., 2021) has been done to investigate their mechanisms. In this paper, the performance of all benchmarks and SILO will be evaluated using SGD with momentum = 0.9 and a weight decay of 1e-4 (same as (Renda et al., 2019; Frankle & Carbin, 2019)). The effect of those adaptive LR optimizers on SILO will be discussed in Section 6.
6

---

Published in Transactions on Machine Learning Research (08/2023)
100 Percent of Weights Remaining A
Slow Growth II
1r +
Fist Growth
Learning Rate
No Growth
Slow Growth
aax
B6x lr
Pirst Pruning Cycle
Pruning Cycle q
Subsequent Pruning Cycles
Figure 2: Illustration of STLO during pruning. The S-shape red line is motivated from Theorem 2.
4.2 SILO Learning Rate Schedule
To ensure the pruned network is properly trained during iterative pruming, we propose an S-shaped Improved Learning rate Schedule, called SILO, for iterative pruning of networks. As illustrated in Fig. 2, the main idea of the proposed SILO is to apply the LR warmup schedule at every pruning cycle, with a gradual increase of the LR upper bound (i.e., max_ __]rr in an S-shape as the network is iteratively pruned. This LR warmup schedule is meant to be flexible and can change depending on different networks and datasets.
The S-shape in SILO is inspired by Theorem 2 (see Remark 2) and will be further verified by comparing to Oracle. We divide the S-shape into four phases and provide the intuition behind each phase as follows.
1. Phase-1: No Growth, SILO does not increase max 1r until the pruning cycle q (see Fig. 2). It is because the unpruned network often contains a certain amount of weights with zero magnitude. Those parameters are likely to be pruned at the first few pruning cycles, and removing such weights has negligible effect on the distribution of weight gradients.
Phase-2: Slow Growth I, the pruning algorithm has removed most zero magnitude weights and started pruning weights with small magnitude. Pruning such weights has a small effect on distribution of weight gradients. Hence, slightly increase max ir after pruning cycle q.
3. Phase-3: Fast Growth, SILO greatly increases max_ 1r. It is because the pruning algorithm now starts removing weights with large magnitude and the distribution of weight gradients becomes much narrower. This requires a much larger LR for meaningful weight updates.
Phase-4: Slow Growth II, the network is now heavily pruned and very few parameters left in the network. By using the same pruning rate, very small portion of the weights will be pruned. This could cause marginal effect on the distribution of weight gradients. Hence, SILO slightly increases max lr.
We note that SILO is designed based on the assumption that existing pruning methods tend to prune weights with small magnitude. The key difference with existing LR schedules (e.g., LR warmup) is that SILO is adaptive and able to precisely increase the value of max 1r as the network is iteratively pruned, while existing LR schodules do not factor in the need to change max 1r during different pruning cycles.
4.3 Implementation of SILO
As for the implemenntttin of SILO, we designed a function to estimate the value of max 1r as shown below.
max 1 1+ 1 tc.
(3)
where = 1 - (1 m is the input of the function and max_1a 1r is the output of the function. When computing the value y, the parameter p is the pruning rate and m is the number of completed pruning cycles. In equation 3, the parameters B and q are used to control the shape of the curve. The larger the 8, the later the curve enters the Fast Growth phase. The parameter q determines at which pruming cycle SIT.O enters the Slow Growth I phase. When q 0, the No Growth phase will be skipped and y will be
7

---

Published in Transactions on Machine Learning Research (08/2023)
Algorithm Algorithm for the proposed S-shaped Improved Learning rate Optimization (SILO)
Input:
lower bound e, upper bound + c, pruning rate p, number of pruning cycles L, number of training epochs t, S-shape control term 8, delay term q.
for to do
if n q then
max 1 =
4: else
5:
max_ 1r =
+ 1- m)
6: for i 0 to t do
7:
(1) linearly warmup the LR to max 1r
8: (2) drop the value of LR by 10 at cortain epochs
the proportion of pruned weights at the current pruming cycle. The parameters e and determine the value range of max max_lr. As the network iteratively pruned, y increases and max _a increases from E to + accordingly. The details of the SILO algorithm are summarized in Algorithm 1.
Parameter Selection for SILO. Algorithm requires several inputs for implementation. The value of can be tuned using the validation accuracy of the unpruned network while the value of 8 can be tuned using the validation accuracy of the pruned network with targeted sparsity. The pruning rate p and pruning cycles L are chosen to meet the larget sparsity. The number of training epochs t should be large enough to guarantee the network convergence. Let q= 1 and B could be a good choice and yield promising results as we demonstrate in Section 5. Furthermore, based on our experience, the value of q and could be tuned in the range of [0, 3] and [3, 6], respectively. Lastly, we note that the sensitivity of parameters in the proposed SILO will be further examined in Section 5.3.
5 Performance Evaluation
In Sections 5.1, we first summarize the experiment setup. In Section 5.2, compare the performance of the proposed SILO to four LR schedule benchmarks. In Section 5.3, we examine the sensitivity of parameters in SILO. In Section 5.4, we present the value of max_ ir estimated by the proposed SILO at each pruning cycle and compare it to an Oracle which exhaustively searches for the optimal max lr via grid search.
5.1 Experimental Setup
We demonstrate that SILO can work well with different pruning methods across a wide range of networks and datasets. We shortlist two classical pruning methods (global weight, global gradient) suggested by (Blalock et al., 2020) and three state-of-the-at pruning method (Tterative Magnitude Pruning (TMP) (Frankle & Carbin, 2019), Layer-adaptive Magnitude-basee based Pruning (LAMP) (Lee et al., 2020)) and Lookahead Pruning (LAP) (Park et al., 2020). The details for each experiment are as follows.
Pruning ResNet-20 (He et al., 2016) on CIFAR-10 via global magnitude.
Pruning VGG-19 (Simonyan & Zisserman, 2014) on CIFAR-10 via global gradient.
3. Pruning DenseNet-40 (Huang et al., 2017) the CIFAR-100 dataset using T.AMP.
4. Pruning MobileNetV2 (Sandler et al., 2018) on the CIFAR-100 dataset using LAP.
5. Pruning ResNet-50 on ImageNet (i.e., ImageNet-1000) (Deng et al., 2009) using TMP.
6. Pruning Vision Transformer (ViT-B-16) (Dosovitskiy et al., 2020) on CIFAR-10 using IMP.
In each experiment, to demonstrate the robustness compare SILO with a fixed value of (q = 1, 3 =5) constant LR and the three shortlisted LR schedules: (i) LR decay, (ii) cyclical LR and (iii) LR warmup (described in Section 4.1). The implementation
parameters in the proposed SILO, we
details of each LR schedule summarized in Table 1.
8

---

Published in Transactions on Machine Learning Research (08/2023)
Schedule
Description (Iters: Iterations)
LR decay (a, b)
linearly decay the value of LR from over b Iters.
cyclical LR (a, b, c)
linearly vary between and b with step size of c Iters.
LR warmup (a, b, c, d,
increase to over b Iters, 10x drop at c, Iters.
STTO (6, 6, b, c, d, e)
LR warmup (max 1r, b, c, d, e), where max_ __rr increases from to e + 5 during iterative pruning (see equation 3).
Table I: Descriptions of LR schedule benchmarks and the proposed SILO.
(1) Methodology. We train the network using the training dataset via SGD with momentum = 0.9 and a weight docay of 1e-4 (same as (Renda al., 2019; Frankle Carbin, 2019)). Next, we prune the trained network with a pruning rate of 0.2 (i.e., 20% of remaining weights are pruned) in 1 pruning cycle. We repeat 25 pruning cycles in run and early-stop top-1 test accuracy (i.e., the corresponding test accuracy when early stopping criteria for validation error is met) to evaluate the performance. The results are averaged over 5 runs and the corresponding standard deviation are summarized in Tables 2 - 7, where the results of pruning ResNet-20, VGG-19, DenseNet-40, MobileNetV2,, ResNet-50 and Vision Transformer (ViT---16) are shown, respectively. Some additional details (c.g., training epochs, optimizer, batch size, etc) and results for more values of are given in Tables 16 21 in the Appendix.
(2) SOTA LR Schedules. To ensure fair comparison against prior SOTA LR schedules, we use LR schedules reported in the literature. Specifically, LR schedules (i.c., LR-warmup) from Table 2 6 are from (Trankle & Carbin, 2019), (Frankle et al., 2020), (Zhao et al., 2019), (Chin et al., 2020) and (Renda et al., 2019), respectively. The LR schedule (i.e., cosine decay) in Table 7 is from (Dosovitskiy et al., 2020).
(3) Paramcters for other LR schedules. For the othor schodules without a single "best' LR in the literalure, we lune the value of LR for each of them via grid search with a range from le-4 to le-1 using the validation accuracy. Other related parameters (e.g., step size) are also tuned the same manner. Lastly, we highlight that all LR schedules used, including SILO, are rewound to the initial state at the beginning of each pruning cycle, which is the same the LR rewinding in (Renda et al., 2019).
(4) Source Code & Devices: We use Tesla V100 devices for our experiments, and the source code is available at https://githu/: htgpthabbccm//arriin11/////.
5.2 Performance Comparison
(1) Reproducing SOTA results. By using the implementatiins reported in the literature, we have correctly reproduced SOTA results. For example, the benchmark results of LR warmup in our Tables 2 7 are comparable to Fig. 11 and Fig. 9 of (Blalock et al., 2020), Table 4 in (Liu et al., 2019b), Fig. in (Chin et al., 2020), Fig. 10 in (Frankle et al., 2020), Table 5 in (Dosovitskiy et al., 2020), respectively.
(2) SILO outperforms SOTA results. The key innovation of SILO is that the LR precisely increases as the network is pruned, by increasing max 1r in an S-shape as À decreases. This results in a much higher accuracy than all LR schedule benchmarks studied. For example, in Table 2, the top-1 test accuracy of SILO is 1.8% higher than the best performing schedule (i.e., LR- warrup) at 5.72. SILO also obtains the best performance when using larger models in Table 3 (i.e., 2.6% higher at X 5.72) and using more difficult datasets in Table 4 (i.., 4.0% higher at À = 5.72).
(3) Performance on ImageNet. In Table 6, we show the performance of SILO using IMP (i.c., the lottery ticket hypothesis pruning method) via ResNet-50 on ImageNet which contains over 1.2 million images from 1000 different classes. We observe that SILOO still outperforms the best performing LR schedule benchmark (LR- (uR-warmup) by 1.9% at 8.59. This improvement increases to 3.2% when À reduces to 5.72.
(4) Performance on SOTA networks (Vision Transformer). Several recent works (Liu al., 2021: Yuan et al., 2021) demonstrated that transformer based networks tend to provide excellent performance
9

---

Published in Transactions on Machine Learning Research (08/2023)
Original Top-1 Test Accuracy = 91.7% (A = 100)
32.8
26.2
8.59
5.72
constant LR
88.1 0.9
87.5 + 0.7
82.8 + 0.9
79.1 = 0.8
LR decay
89.8 0.4
89.0 0.7
83.9 + 0.6
79.8 + 0.7
cyclical LR
89.7 0.8
88.2 0.7
84.1 + 0.8
80.3 + 0.7
LR-warmup
90.3
89.8
85.9 +
81.2 + 1.1
SILO (Ours)
90.8 = 0.5
90.3 = 0.4
87.5 =
82.7 =
Table 2: Top-1 test accuracy = standard deviation of pruning ResNet-20 on CIFAR-10 via global magnitude.
Original Top-1 Test Accuracy 92.2% (A 100)
32.8
26.2
8.59
5.72
constant LR.
88.8 i 0.6
87.4 t 0.7
82.2
73.7 = 1.3
LR decay
89.4 + 0.4
88.6 t 0.5
83.3 +
75.4 =
cyclical LR
89.8 + 0.5
89.1 +
83.7 =
75.7 =
LR-warmup
90.2 + 0.5
89.8 + 0.8
84.5 +
76.5 + 1.0
SILO (Ours)
90.6 + 0.6
90.3 + 0.6
86.1 + 0.8
78.5 + 1.0
Table 3: Top-1 test accuracy standard deviation of pruning VGG-19 on CIFAR-10 10 using global gradient.
Original Top-1 Test Accuracy = 74.6% (A 100)
32.8
26.2
8.59
5.72
constant LR
70.3 + 0.8
68.1 + 0.7
60.8 +
59.1 + 1.2
LR. decay
71.2 + 0.8
69.0 + 0.6
62.6 + 1.2
60.3 = 1.4
cyclical LR
70.9 + 0.6
69.4 + 0.6
63.0 + 1.1
60.8 = 1.3
LR- wR-warmmup
71.5 + 0.7
69.6 0.8
63.9 + 1.0
61.2 = 0.9
SIT.O (Ours)
72.4 0.7
70.8 0.8
65.7 1.2
63.7 + 1.0
Table 4: Top-1 test accuracy + standard deviation of pruning DenseNet-40 on CIFAR-100 using LAMP.
Original Top-1 Test Accuracy = 73.7% (A = 100)
32.8
26.2
8.59
5.72
constant LR.
69.8 + 1.1
68.2 0.9
63.8 1.1
62.1 1.2
LR decay
70.9 + 1.0
69.4 0.6
65.1 + 0.8
64.0 + 111
cyclical LR
71.5 + 0.7
69.6 0.6
65.3
64.3 1.2
LR- (R-warmup
72.1 + 0.8
70.5 0.9
66.2
64.8 1.5
SILO (Ours)
72.5 +
71.0 + 0.7
68.8 + 0.8
66.8 =
Table 5: Top-1 test accuracy + standard deviation of pruning MobileNetV2 on CIFAR-100 using LAP.
in computer vision tasks (c.g., classification). We now examine the performance of SILO using Vision Trausformer (i.e., ViT-B16 with resolution of 384). We note that the ViT-B16 uses Gaussian Error Linear Units (GELU, GELU(x) xœ(x), where (xx) is the standard Gaussian cumulative distribution function) the activation function. We note that both ReLU and GELU have the unbounded output, suggesting that SILO could be helpful for pruning GELU based models as well.
We repeat the same experiment setup as above and compare the performance of SILO to other LR schedules using ViT-B16 Table 7. We observe that SILO able to outperform the standard implementation (cosine decay, ie., decay the learning rate via the cosine function) by 1.3% À 8.59 in top-1 test accuracy. This improvement increases to 1.6% when À reduces to 5.72.
10

---

Published in Transactions on Machine Learning Research (08/2023)
Original Top-1 Test Accuracy = 77.0% (A 100)
32.8
26.2
8.59
5.72
constant LR.
74.2 + 0.8
73.9 0.7
70.5 + 0.6
69.2 0.9
LR decay
75.6 0.5
75.1 0.5
72.7 0.8
70.5 0.6
cyclical LR
76.5 0.5
75.5 0.6
73.4 0.8
71.2 0.7
LR-warmup
76.6 + 0.2
75.8
73.8
71.5 =
SILO (Ours)
76.8
+ 0.4
76.1 + 0.7
75.2 = 0.8
73.8 =
Table 6: Top-1 test accuracy + standard deviation of pruning ResNet-50 on ImageNet using IMP.
Original Top-1 Test Accuracy 98.0%
100)
32.8
26.2
8.59
5.72
constant LR
96.4 + 0.5
96.0 0.7
83.0 + 0.9
80.1 = 0.8
cosine docay
97.2 + 0.2
96.5 0.6
84.1
81.6 1.1
cyclical LR
97.0 + 0.2
96.5 +
83.4
81.0 +
uR-wwrmuup
97.3 + 0.6
96.8 + 0.7
84.4 +
82.1 =
SILO (Ours)
97.7 + 0.5
97.4 + 0.6
85.5 + 0.9
83.4 + 0.8
Table 7: Top-1 test accuracy standard deviation of pruning Vision Transformer on CIFAR-10 using IMP.
Percent of Weights Remaining, À
32.8
26.2
8.59
5.72
LR-warmup (benchmark)
90.3 0.4
89.8 0.6
85.9 + 0.9
81.2 +
SILO (q 1, 5)
90.8 0.5
90.3 + 0.4
87.5 + 0.8
82.7 + 1.2
SILO (q = 2, 8 5)
90.5 = 0.6
90.1 = 0.2
86.9 = 0.4
82.4 + 1.1
SILO (q = 8 4)
90.9 + 0.4
90.2 + 0.5
87.1 + 0.9
82.2 + 1.4
STLO (q = 3 =3)
90.5 + 0.9
90.6 + 0.8
87.5 + 0.6
83.1 + 1.6
SILO (q 1, 8
90.4 0.2
89.2 0.7
85.1 + 1.2
80.8 -
SILO (q 5, 8 5)
90.2 0.3
89.8 + 0.5
85.5 + 1.0
80.5 + 1.6
Table 8: Performance (Top-1 test accuracy + standard deviation) of SILO with different values of q and B when pruming ResNet-20 CIFAR-10 via global magnitude.
Percent of Weights Remaining, X
32.8
26.2
8.59
5.72
LR-warmup (benchmarkk
90.2 + 0.5
89.8 + 0.8
84.5 + 0.9
76.5 + 1.0
SILO (q =
5)
90.6 0.6
90.3 = 0.6
86.1 1 0.8
78.3 - 1.0
SILO (q = 2,
5)
90.7 0.5
90.0 + 0.4
86.5 0.9
79.2 - 1.3
SILO (q =3, - 4)
90.3 0.2
90.4 0.3
85.8 - 0.6
78.8 - 0.9
SILO (q 1
90.8 0.8
90.1 + 0.6
85.5 + 0.9
78.2 + 1.1
STLO (q = 1, =
90.0 0.4
88.5 + 0.9
82.1 + 1.4
75.3 + 1.3
SILO (q = 5, 3 5)
90.1 0.5
89.1 + 0.6
83.2 + 1.1
75.7 + 1.5
Table 9: Performance (Top-1 test accuracy + standard deviation) of SILO with different values of g and B when pruning VGG-19 on CIFAR-10 via global gradient.
5.3 Sensitivity of Parameters in SILO
In Scction 5.2, we demonstrate the robustness of parameters in SILO by using fixed value of q 1, 3 and compare it to benchmark schedulers using different networks, datasets and pruning methods. We now repeat the setup in Tables 2 and examine the sensitivity of these two parameters in Tables 8
11

---

Published in Transactions on Machine Learning Research (08/2023)
Percent of Weights Remaining, X
100
51.3
32.9
21.1
5.72
Oracle optimized max_lr _r (x10-2)
4
4.6
9.0
9.8
10.2
Oracle optimized interval (x10-2)
[3.6,
4.2]
[4.2,
5.4]
[8.0,
9.6]
[9.2,
10.4]
[9.8, 10.6)
SILO's estimated max_ 1r (x10-2)
4
4.32
9.2
9.9
9.99
Table 10: Comparison between Oracle optimized max_ Oracle optimized interval (both obtained via grid
search) and the value of max 1r estimated by SILO when iteratively pruning VGG-19 on CIFAR-10.
Percent of Weights Remaining, X
100
51.3
41.1
32.9
21.1
Oracle optimized max_lr 1r (x10-2)
3.4
3.8
4.6
5.6
6.2
Oracle optimized interval (x 2)
[2.8, 3.6]
[3.4,
4.2]
[3.8, 5.2]
[5.4, 6.6]
[5.4, 6.8)
SILO estimated max_ 1r (x10-2)
3
3.2
4.7
6.4
6.9
Comparison between Oracle tuned max 1r, Oracle optimized max_1r interval (both obtained via grid search) and max_lr _l estimated by SILO when iteratively pruning ResNet-20 on the CIFAR-10 dataset.
Table 11:
The takeaway message from Tables 8 9 is three-fold: (1) The parameters of SILO can provide promising results (i.e., out outperfor benchmark schedulers) within the suggested value range, i.e., 0 S q S 3, 3 < B < 6 (see first rows in Tables 8 --99 9). (ii) When the value of and / fall outside the suggested range, SILO may fail to outperform benchmarks (see last rows in Tables - 9). (iii) also suggests that q and /3 could be tuned within a relatively small range and does not require as much effort as tuning the hyperparameters.
5.4 Comparing SILO to an Oracle
Our new insight suggests that, due to the change in distribution of hidden representations during iterative pruning, LR should be re-tuned at cach pruning cycle. SILO provides a method to adjust the max max_lr in an S-shape, which is backed up by a theoretical result (see Theorem 2). We now further examine the S-shape trajectory of SILO by comparing SILO's estimated max_lr to an Oracle, which uses the same LR warmup structure as SILO but exhaustively searches for the optimal value of max_lr at each pruning cycle. The Oracle's max ir at the current pruning cycle is chosen by grid search ranging from le-4 to le-1 and the best performing value (i.e., determined by validation accuracy) is used to train the network. The results of max_1r determined this way when itcratively pruning a VGG-19 on CIFAR-10 using the global magnitude are detailed in Table 10 via two metrics:
Oracle optimized max_Ir: The value of max that provides the best validation accuracy.
2. Oracle optimized interval: The range of max Ir which provides comparable performance to Oracle optimized max 1r (i.e., within 0.5% of the best validation accuracy).
SILO vs Oracle (Performance): In Table 10, the max ir estimated by SILO falls in the Oracle opti- mized max_Ir ir interval at each pruning cycle, suggesting that SILO can precisely adjust max ir to provide competitive performance with Oracle. This further verifies the S-shape trajectory of max_ 1r used in SILOO.
SILO Oracle (Complexity): The process of finding the Oracle tuned max max_lr Ir requires a significantly larger computational complexity in tuning due to the grid search. Assume that max_Ir Ir is searched from a sampling space of [0,, ..., 0z] for A pruning cycles. Hence, the complexity of the Oracle will be O(r²). On the other hand, SILO controls the variation of max 1r at each pruning cycle via four parameters: ranges of max_lr: [e, e + 6], delay term q and S-shape control term B. Similar to the Oracle, both and can be searched from a range of n values. As we have recommended before, and can be tuned in the range of [o, 3), [3, 6), respectively. As a result, SILO has a complexity of O(n?), which exponentially less complex than the Oracle's complexity, but with competitive performance. Lastly, we highlight that similar performance trends can be observed using ResNet-20 via global magnitude (see Table 11).
12

---

Published in Transactions on Machine Learning Research (08/2023)
Params: 227K; Train Steps: 63K Iters; Batch: 128; Pruning Rate: 0.2
100
32.9
21.1
5.72
2.03
constant LR
88.4
84.8
83.5
7..
67.1
LR decay
88.6
87.1
83.7
76.1
66.0
cyclical LR
88.9
86.9-t.55
84.1=
77.0+0.9
64.4
LR-warmup
89,1
87.2
84.5=0.6
75.2+1.1
65.1
STT,O
89.2
87.9
86.3
79.5+1.7
71.7+2.3
Table 12: Performance comparison (averaged lop-1 test accuracy + sid over runs) of iteratively pruning ResNet-20 on CIFAR-10 dataset using global magnitude and Adam optimizer (Kingma & Ba, 2014).
Params: 227K; Train Steps: 63K Iters; Batch: 128; Pruning Rate: 0.2
100
32.9
21.1
5.72
2.03
constant LR
$7.3
83.4
81.5
6..5
55.1
LR decay
88 88.4
84.8
77.8=0.9
67.1
58.3+1.6
cyclical LR
88.1
84.7
81.9
67.5
56.3
LR-warmup
88.9
85.1
81.7=0.4
67.3
57.1
SILO
88.7
86.1
83.1
72.5+1.3
63.5
Table 13: Performance comparison (top-1 test accuracy + std over runs) of iteratively pruning ResNet-20 CIFAR-10 dataset using global magnitude and RMSProp optimizer (Ticleman & Hinton, 2012).
6 Reflections
In summary, SITO is an adaptive LR schedule for network pruming with theoretical justification. SILO outperforms existing benchmarks by 2.1% 3.2% via classical networks and datasets. For SOTA networks (c.g., Vision Transformer) and large scale datasets (c.g., ImageNet), it leads to an improvement of 3% 5.6%. More importantly, via the S-shape trajectory, SILO can obtain comparable performance to Oracle with significantly lower complexity. We now conclude the paper by presenting some reflections.
(1) The gain of SILO. As compared to existing LR schedules, the advantage of SILO is three-fold: (i) SILO is specially designed for network pruning which gradually increases the value of LR in an S-shape as the network is gradually pruned. (ii) The S-shape trajectory is thcoretically justified and empirically examined by comparing to an Oracle. The results suggest that SILO can obtain a comparable performance to the Oracle with a significantly lower complexity. (iii) As for its performance, STLOO outperforms existing LR schedules by 2% 4%, which is demonstrated via extensive experiments.
(2) Connection to Prior Work. Our work explores the important role of LR. in network pruning and provides a new insight - as the ReLUUbased network is ileratively prumed, a larger LR should be used to retrain the pruned network. This new insight could be used to explain the surprising effect of LR schedules observed in prior works. Specifically, Frankle & Carbin (2019) highlighted that they can only find winning tickets after applying LR warmup schedule. Using insights from our analysis, we attribute this LR warmup increasing the LR a large value (c.g., Frankle & Carbin (2019) increases LR le-1 when training VGG-19) which is better for pruned networks. Similarly, Renda al. (2019) proposed LR rewinding and demonstrated that it outperforms standard fine-tuning. We attribute this to LR rewinding ensuring that relatively larger LR. (as compared to fine-tuning) is used to better train pruned networks in every pruning cycle.
(3) Performance using Adaptive LR Optimizers. In the main paper, we only evaluate the performance of SILO using SGD. We note that the weight update mechanism is different for other adaptive learning rate optimizers, which may potentially affect the performance of SILO. In Tables 13, we conduct similar performance comparison using Adam (Kingma & Ba, 2014) and RMSprop (Ticleman & Hinton, 2012), and SIT.O still outperforms all LRR schedule benchmarks studied.
13

---

Published in Transactions on Machine Learning Research (08/2023)
Percent of Weights Remaining, À
30.4
24.7
7.29
LR-Warmup
88.5 = 0.9
87.1 + 1.2
82.2 + 1.4
(ii) SILO (q = 1, 1=.33) =3) 3)
89.4 + 0.8
88.5 + 1.4
83.9 + 1.7
(ii) L.R-Warupp
87.4 + 0.7
85.2 + 0.9
80.8 + 1.6
(iv) SILO = 1, 1, 3
88.1 - 0.6
87.1 + 1.1
82.3 + 1.4
Table 14: Performance comparison between SILO and LR-warmup when pruning VGG-19 (rows (i) and (ii)
and ResNet-20 (rows (iii) and (iv)) using L1 Norm filter pruning (Li et al., 2017) on CIFAR-10 dataset.
(4) Extension of SILO to Structured Pruning. In Section 3.1, we have examined the distribution of weight gradients using structured pruning and shown that structured pruning also suffer from the issue of decrcasing weight gradients (sce Fig. 7 in the Appendix for more details). This suggests that SILO could be applicable to structured pruning as well.
In Table 14, we compare the performance of SILO to LR-warmun using L1 Norm filter pruning (Li et al., 2017) on ResNet-20 and VGG-19. We observe that SILO still outperforms the benchmark LR schedule, leading to an improvement of 2.0% (compare SILO at row (ii) to LR-warmup at row (i) when X = 7.29). We note that the performance improvement is not as significant as that of working with unstructured pruning. We suspect it is due to that the distribution of weight gradients may change differently when the network is structurely pruned. In such case, SILO may need to be re-designed for a larger performance gain.
(5) Effect of SILO on Weight Update and Generalization. We also examine the effect of STT.O on distribution of weight update (weight gradient learning rate) and the experimental results demonstrate that, with SILO, the distribution of weight update of pruned networks becomes less centralized, suggesting that SILO helps to mitigate the issue of decreasing weight gradients. As a result, the pruned network is better trained with SILO, leading to much better generalization performance. We refer the interested reader to Figs. 8 10 in the Appendix.
(6) Future Research. (i) We demonstrate the performance of SILO on ReLU-based networks (ResNet, VGG) and GELU-basod networks (Vision Transformer). We note that similar LR schodules could be used for networks with other activation functions (e.g., PReLU) and we plan to explore this in future research. (ii) Moreover, the main motivation for STL.O is that the distribution of weight. gradients tends to become narrower after pruning. An approach to automatically determine the value of max 1r from the distribution of weight gradients is definitely worth deeper thought. (iv) As mentioned above, when it comes to the case of structured pruning, the improvement of SILO is not as large that of working with unstructured pruming. This could be due to that the distribution of weight gradients may change differently when the network is structurely pruned. In such a case, SILO may need to be re-designed for a larger performance gain.
Acknowledgements
This research is supported by A*STAR, CISCO Systems (USA) Pte. Ltd and the National University of Singapore under its Cisco-NUS Accelerated Digital Economy Corporate Laboratory (Award 12100180002). Additionally, we would like to thank the members of the Kent-Ridge AI research group at the National University of Singapore for helpful feedback and interesting discussions on this work.
References
Zeyuan Allen-Zhu, Yuanzhi Li, and Zhao Song. A convergence theory for deep learning via over- parameterization. In Proceedings of the International Conference on Machine Learning (ICML), volume 97, pp. 242-252, 2019.
Wangpeng An et al. Exponential decay sine wave learning rate for fast deep neural network training. In 2017 IEFE Visual Communications and Image Processing (VCIP), pp. 1-4. TEFE, 2017.
14

---

Published in Transactions on Machine Learning Research (08/2023)
Larry C. Andrews. Special functions of mathematics for engineers, volume 49. Spic Press, 1998.
Sanjeev Arora, Nadav Cohen, and Elad Hazan. On the optimization of deep networks: Implicit acceleration by overparanelerixation.
Proceedings of the Intervational Conference on Machine Learning (ICML), volume 80, pp. 244-253, 2018.
Yoshua Bengio. Practical recommendations for gradient-based training of deep architectures. In Neural Networks: Tricks of the trade, pp. 437 478. Springer, 2012.
Davis Blalock al. What is the state of neural network pruning? In Proceedings of the Machine Learning and Systems (MLSys), 2020.
Ting- Wu Chin, Ruizhou Ding, Cha Zhang, and Diana Marculescu. Towards efficient model compression via learned global ranking. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1518 1528, 2020.
Jía Deng et al. Imagenet: A large-scale hierarchical image database. In 2009 IEEE/CVE Conference Computer Vision and Pattern Recognition (CVPR), pp. 248 255, 2009.
Misha Denil et al. Predicting parameters in deep learning. In Proceedings of the Advances in Neural Information Processing Systems (NeurIPS), pp. 2148 2156, 2013.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Un- terthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth Transformers for image recognition at scale. arXiv preprint Xr:i0:2010. 11929, 2020.
16x16 words:
Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. In Proceedings of the International Conference Learning Representations (ICLR), 2019.
Jonathan Frankle, Gintare Karolina Dziugaite, Daniel M Roy, and Michael Carbin. Stabilizing the lottery 2019.
ticket hypothesis. orXiv preprint
Jonathan Frankle et al. Linear mode connectivity and the lottery licket hypothesis. In Proceedings of the 2020.
International Conference on Machine Learning (ICML), pp.
Venkata Gandikota et al. vqsgd: Vector quantized stochastic gradient descent. In Proceedings of the Inter- national Conference on Artificial Intelligence and Statistics, pp. 2197 2205, 2021.
Itai Gat, Yossi Adi, Alex Schwing, and Tamir Hazan. On the importance of gradient norm in pac-bayesian bounds. Advances in Neural Information Processing Systems (NeurIPS), 35:16068 16081, 2022.
Rong Ge et al. The step decay schedule: A near optimal, geometrically decaying learning rate procedure for least squares. arXiv preprint 0rXiv:1904.12889 2019.
Aidan Good and et al. Recall distortion in neural network pruning and the undecayed pruning algorithm. Advances in Newral Information Processing Systems (NeurIPS),
2022.
Tan Goodfellow, Yoshua Bengio, and Aaron Courville. Deep learning. MTT press, 2016.
Song Han, Huiki Mao, and William J. Dally. Deep compression: Compressing deep neural net works with pruning, trained quantization and huffman coding. arXie preprint arXiv:1510.001 2015a.
Song Han et al. Learning both weights and connections for efficient neural network. In Proceedings of the Advances in Neural Information Processing Systems (NeurIPS), pp. 1135 1143, 2015b.
Babak Hassibi and David G. Stork. Second order derivatives for network pruning: Optimal brain surgeon. In Proceedings of the Advances in Neural Information Processing Systems (NeurIPS), pp. 164 171, 1993.
Kaiming He al. Deep residual learning for image recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 770-778, 2016.
15

---

Published in Transactions on Machine Learning Research (08/2023)
Tong He al. Bag of tricks for image classification with convolutional neural networks. In Proceedings of the IEEE/CVI Conference on Computer Vision and Patlern Recognition (CVPR), pp. 558-561 567, 2019.
Yang He et al. Learning filter pruning criteria for deep convolutional neural networks acceleration. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognation (CVPR), June 2020.
Hengyuan Hu et al. Network trimming: A data-driven neuron pruning approach towards efficient deep architectures. Xiv preprint orXio:1607.03250 2016.
Gao Huang et al. Densely connected convolutional networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pallern Recognition (CVPR), pp. 4700 4708, 2017.
Sergey Ioffc and Christian Szegedy. Batch normalization: Accelcrating deep network training by reducing internal covariate shift. In Proceedings of the International Conference on Machine Learning (ICML), volume 37, pp. 448 456, 2015.
Arnulf Jentzen et al. Strong error analysis for stochastic gradient descent optimization algorithms. IMA Journal of Numerical Analysis, 41(1):455 1924555--49--- 2021.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization, 2014.
Alex Krizhevsky and Geoff Hinton. Convolutional deep belief networks on cifar-10. Unpublished manuscript, 40(7):1-9, 2010.
Alex Krizhevsky et al. Learning multiple layers of features from tiny images. 2009.
Ya Le and Xuan Yang. Tiny imagenet visual recognition challenge. CS 231N, 2015.
Yann LeCun, John S. Denker, and Sara A. Solla. Optimal brain damage. In Proceedings of the Advances in Neural Information Processing Systems (NewrIPS), pp. 598 605, 1990.
Jaeho Lee, Sejun Park, Sangwoo Mo, Sungsoo Ahn, and Jinwoo Shin. Layer-adaptive sparsity for the magnitude-based pruning. In International Conference on Learning Representations (ICLR), 2020.
Namhoon Lee et al. SNIP: Single-shot network pruning based connection sensitivity. In Proceedings of the International Conference on Learning Representations (ICLR), 2019.
Hao Li al. Pruning filters for efficient convnets. In Proceedings of the International Conference on Learning Representations (ICLR), 2017.
Yawei Li and et al. Revisiting random channel pruning for neural network compression. Proceedings of the IEEE/CVE /CVF Conference on Computer Vision and Pattern Recognétion (CVPR), pp. 191-201, 2022.
Mingbao Lin, Rongrong Ji, Yan Wang, Yichen Zhang, Baochang Zhang, Yonghong Tian, and Ling Shao. Hrank: Filter pruning using high-rank feature map. Proceedings of the IEEE/CVF Conference on Computer Vision and Paltern Recogrilion (CVPR), pp. 1529 1538, 2020.
Hanxiao Liu, Karen Simonyan, and Yiming Yang. DARTS: Differentiable architecture search. In Proceedings of the International Conference on Learning Representations (ICLR), 2019a.
Linfeng Liu, Xu Han, Dawei Zhou, and Liping Liu. Towards accurate subgraph similarity computation via neural graph pruning. Transactions on Machine Learning Research (TMLRR), 2022. URL https: //openreview
Shiwei Liu and al. Sparse training via boosting pruning plasticity with neuroregeneration. In Advances in Neural Information Processing Systems (NewrIPS), volume 34, pp. 9908-9922, 2021.
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp- 10012-10022, 2021.
16

---

Published in Transactions on Machine Learning Research (08/2023)
Zhuang Liu et al. Rethinking the value of network pruning. In Proceedings of the International Conference on Learning Represenlations (ICLR), 2019b.
Jian-Hao Luo and Jianxin Wu. Neural network pruning with residual-connections and limited-data. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1458-1467, 2020.
Jian-Hao Luo, Jianxin Wu, and Weiyao Lin. Thinct: A filter level pruning method for deep neural network compression. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), pp- 2017.
Fran Malach et al. Proving the lottery ticket hypothesis: Praning is all you need. In Proceedings of the International Conference on Machine Learning (ICML), 6682 6691, 2020.
Rahul Mehta. Sparse transfer learning via winning lottery tickets. In Proceedings of the Advances in Neural Information Processing Systems Workshop on Learning Transferable Skills, 2019.
Pavlo Molchanov et al. Pruning convolutional neural networks for resource efficient inference. Proceedings of the International Conference on Learning Representations (ICLR), 2017.
Pavlo Molchanov et al. Importance estimation for neural network pruning, In Proceedings of the IEEE/CVF Conference Compuler Vision and Pallern Recognation (CVPR), pp. 11264-11272, 2019.
Sejun Park, Jaeho Lee, Sangwoo Mo, and Jinwoo Shin. Lookahead: A far-sighted alternative of magnitude- based pruning. 2020.
Alex Renda, Jonathan Frankle, and Michael Carbin. Comparing rewinding and fine-tuning in neural network pruning. In International Conference on Learning Representations (ICLR), 2019.
Sebastian Ruder. An overview of gradient descent optimization algorithms. arXiv preprint arXi::169900470 2016.
Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chiel Chen. Mobilenetv2: Inverted residuals and linear bottlenecks. In Proceedinggs of the IEEE Conference on Compuler Vision and Pattern Recognition (CVPR), pp. 4510-4520 2018.
David W Scott. Sturges' rule. 2009.
Wiley Interdisciplinary Reviews: Computational Statistics, 1(3):303-300 306.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint 4rXic:1409.1550 1556, 2014.
Leslie N. Smith. Cyclical learning rates for training neural networks. In 2017 1EEE Winter Conference on Applications of Computer Vision (WACV), pp. 464-472. IEEE, 2017.
Chong Min John Tan and Mehul Motani. Dropnet: Reducing neural network complexity via iterative pruning. In Proceedings of the International Conference on Machine Learning (ICML), pp. 9356-9366. PMLR, 2020.
Lucas Theis et al. Faster gaze prediction with dense networks and fisher pruning. arXiv preprint arXiv:1801.0577 2018.
Tijmen Tieleman and Geoffrey Hinton. Lecture 6.5-rmsprop: Divide the gradient by a running average of its recent magnitude. COURSERA: Neural networks for machine learning. 4(2):26-31, 2012.
Artem Vysogorets and Julia Kempe. Connectivity matters: Neural network pruning through the lens of effective sparsity. Journal of Machine Learning Research, 24(99):1-23, 2023.
Huan Wang and al. Recent advances on neural network pruning at initialization. In Proceedings of the International Joint Conference Artificial Intelligence, IJCAT, Vienna, Austria, pp. 23 29,2,, 2022.
17

---

Published in Transactions on Machine Learning Research (08/2023)
Tianzhe Wang et al. Apq: Joint search for network architecture, pruning and quantization policy. In Proceedings of the IEEE/CVF Conference on Computer Vision and Paltern Recognation (CVPR), June 2020a.
Yulong Wang et al. Dynamic network pruning with interpretable layerwise channel selection. In Proceedings of the AAAI Conference Artificial Intelligence, volume 34, pp. 0299-63000 2020b.
Yulong Wang et al. Pruning from scratch. Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pp. 12273-12280, 2020c.
Zi Wang, Chengcheng Li, and Xiangyang Wang. Convolutional neural network pruning with structural redundancy reduction. In Proceedings of the IEEE/CVF Conference on Compuler Vision and Pallern Recognition (CVPR), pp. 14913-14922 June 2021.
Xia Xiao, Zigeng Wang, and Sanguthevar Rajasekaran. Autoprune: Automatic network pruning by regu- larizing auxiliary parameters. In Proceedings of the Advances in Neural Information Processing Systems (NeurIPS), pp. 13681 13691, 2019.
He Yang ct al. Filter pruning via geometric median for deep convolutional neural networks acceleration. In Proceedings of the IEEE Conference Compuler Vision and Pallern Recogration (CVPR), pp. 4335 -344, 2019.
Mao Ye al. Good subnetworks provably exist: Pruning via greedy forward selection. In Proceedings of the International Conference on Machine Learning (ICML), pp. 10820-10830. PMLR, 2020.
Haoran You and et al. Max-affine spline insights into deep network pruning. Transactions on Machine Learning Research (TMLR), 2022. ISSN 2835-8856.
Kaichao You et al. How does learning rate decay help modern neural networks? ar preprint arXiv:1908.0188 2019.
Hao nan Yu et al. Playing the lottery with rewards and multiple languages: lottery tickets in RL and NLP. In Proceedings of the International Conference Learning Representations (ICLR), 2020.
Ruichi Yu et al. Nisp: Pruning networks using neuron importance score propagation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pottern Recognition (CVPR), pp. 9194 9194-9203, 2018.
Li Yuan, Qibin Hou, Zihang Jiang, Jiashi Feng, and Shuicheng Yan. Volo: Vision outlooker for visual recognition. arXiv preprint or Xix:22106.18 13112, 2021.
Matthew D Zeiler. Adadelta: adaptive learning rate method. arXiv preprint arXiv:1212.5701, 2012.
Yuyao Zhang and Nikolaos M Freris. Adaptive filter pruning via sensitivity feedback. IEEE Transactions on Neural Networks and Learning Systems (TNNLS), 2023.
Chenglong Zhao et al. Variational convolutional neural network pruning. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2780-2781 2019.
Hattie Zhou et al. Deconstructing lottery tickets: Zeros, signs, and the supermask. In Proceedings of the Advances in Newral Information Processing Systems (NeurIPS), 2019.
18


Reviews:
[{"review_id": "cquf165QgD", "reviewer": "Reviewer_ALQq", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength:\n\nIn this study, the paper focuses on exploring the optimal selection of the learning rate during network pruning, supported by theoretical derivations. Additionally, a novel learning rate schedule named SILO is introduced for network pruning, showcasing superior generalization compared to existing benchmarks of learning rate schedules across a wide range of network architectures.\n\n\nWeakness: \n\nFirst, I find it difficult to comprehend the necessity of the fixed weight-gradient energy ($E_{WG}$). It would be helpful if the author could provide an explanation for its inclusion, along with relevant references to support its use.\n\nSecond, the assumptions made for Theorem 1 and 2 in this paper are excessively stringent. It assumes that (1) the weights follow a Gaussian distribution, (2) that the input to each layer also follows a Gaussian distribution, and (3) the neuron width is infinitely large *simultaneously*. To derive the S-shape trajectory using Theorem 1 & 2, this paper assumes that the weights consistently adhere to a Gaussian distribution throughout training, which is not practical. Furthermore, in deep neural networks, the input to the ($\\ell$+1)-th layer can be viewed as the output of the ($\\ell$)-th layer. However, with the ReLU activation function, assuming that the input of the ($\\ell$)-th layer follows a Gaussian distribution does not guarantee that the input to the ($\\ell$)-th layer will still follow a Gaussian distribution.\n\nThird, it is crucial for this paper to establish a connection between its theoretical results and the improved generalization, which is the most significant numerical finding. Furthermore, providing a comparison with existing theoretical analyses of pruned networks would be essential in assessing the theoretical contributions of this paper.\n\nSome additional questions:\n\nQ1. Expanding the proposed methods to include structured pruning approaches would enhance computational efficiency when implementing them in real-world applications. It would be beneficial if the author could explore the extension of the methods to incorporate structured pruning techniques.\n\nQ2. It would be highly valuable if the author could offer insights into the underlying intuition behind how adjusting the learning rate can result in improved generalization, considering it from an optimization perspective.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "In this paper, the focus lies on investigating the impact of learning rate (LR) schedules on network pruning. A novel LR schedule called SILO is introduced for network pruning, which exhibits enhanced generalization compared to existing LR schedule benchmarks across diverse network architectures. ", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "See in Strengths and Weaknesses.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "jTWOYSEpMp", "reviewer": "Reviewer_zCPr", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n- The method and motivation intuitively make sense\n- Mild improvements upon currently employed schedulers\n- Results are architecture agnostic\n\nWeaknesses\n- The improvements seem to be marginal\n- SILO introduces, relative to other common schedules, additional hyper parameters for tuning\n- Some unclear claims in the proofs", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This work proposes SILO, a learning rate schedule that follows a sigmoid type of curve, for better training of sparse networks. The authors motivate such a learning rate from a specific observation; as the network gets pruned the distribution of the gradients becomes more concentrated around small values, so the learning rate needs to appropriately increase. This is shown empirically in the case of iterative magnitude pruning of ReLU networks, where both the weight gradients as wells as the magnitude of the hidden representations (post ReLU) is reduced. The authors also provide theoretical arguments on why this is the case when considering Gaussian data and a two layer neural network with an infinite width hidden layer.  The main idea is that the gradient norm can be lower bounded by a function that depends on the learning rate and the activation norm, where the latter is also lower bounded by a function that depends on the sparsity ratio of the model. Therefore, if one desires a specific gradient magnitude, the learning rate needs to be adapted in a way that depends on the sparsity ratio and this relationship manifests as a sigmoid-type of curve. In practice, the authors define a learning rate schedule as $\\delta \\text(Sigmoid(\\beta \\log \\frac{\\gamma(p, q)}{1-\\gamma(p, q)})) + \\epsilon$, where $\\gamma$ depends on the pruning ratio $p$ and $\\delta, \\beta, q, \\epsilon$ are hyperparameters of the scheduler.\n\nThe authors show that this schedule does better than several learning rate schedulers used in practice in various tasks that involve pruning diverse set of models on CIFAR 10, 100 and Imagenet.  \n", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "The work is generally interesting and presents, in hindsight, an interesting observation. SILO ends up being a simple enough learning rate schedule that does provide some improvements upon the baselines. Having said that, the improvements seem marginal, especially given the relatively large standard deviation associated with each of the numbers; more often than not, the second best learning rate schedule has results that are within standard deviation of SILO.  Besides that, the proofs of the theoretical claims in the appendix could use a bit more polish as currently they can be hard to follow and some of the claims are unclear.\n\nBased on these, the main requests I have / clarifications I would like on the experimental side are:\n- Given the current status of the results and given that SILO does introduce more hyper parameters in the schedule, it would be interesting to see how robust SILO and the rest of the learning rate schedulers are with respect to their hyperparameters. \n- Given that the main argument for SILO is that the gradient magnitudes change with pruning, it is peculiar that there are still benefits when doing Adam / RMSProp, which in general are invariant to the scale of the gradient. Perhaps one potential cause would be that the moment estimates might take a long time to adapt to the new gradient statistics that stem from pruning. In this case, it would be interesting to see what the improvements of SILO are when doing Adam / RMSProp and, e.g., the moments are “reset” on each pruning cycle. \n \nAnd on the theoretical / claim side: \n- The authors adopt non-standard definitions of “activation energy” and “weight gradient energy”, which are equivalent to squared L2 norms. I would suggest the authors to use the latter, more standard, definitions.\n- At the end of page 17 I believe it should be $\\mu_{trunc} = \\int_{-\\inf}^{\\inf} x P_{trunc}(Z_j(X) = x)dx$\n- At eq. 15 I believe that the first term should be  $\\int_{-\\inf}^{-\\beta} W^2 P(W)dW$.\n- The transition from eq. 10 to eq.11 is not clear\n- At the proof of theorem 1 the authors use $Sign(W) = 0$ for $-\\beta \\leq W \\leq \\beta$ and $Sign(W) = 1$ otherwise. As the $Sign()$ is used primarily to get the actual sign of a variable, I would propose to instead use the indicator function, i.e., something like $I[|W| > \\beta]$.\n- At the extension of theorem 1 to the arbitrary depth it is mentioned below eq 30 that $\\delta_1, \\delta_2, …, \\delta_M$ are independent of $X_1, W_1^{1k}, W_2^{k1}$, which is a peculiar statement as $\\delta_1, …, \\delta_M$ are functions of $X_1, W_1^{1k}$. Could the authors elaborate?\n- At the proof of theorem 2 before eq. 46 the authors claim that $p(\\alpha_l^k | h_i^k) = p(\\alpha_l^k)$ which is once again peculiar as, from what I understand, $\\alpha_l^k$ is the softmax probability for class $k$ which is a function of $W_2$ and the hidden layer $H^k$ (and $h_i^k$ is just one specific dimension of $H^k$).\n\nAnd some various minor things:\n- Some of the discussion points in the comparison at 5.2 do not align with the results at the table. For example, at 5.2 (3) the authors mention that SILO outperforms LR-Warmup by 1.9% but at Table 6 I see a 1.4% improvement (similarly for the 3.2% at $\\lambda=5.72$, which at Table 6 is 2.3%).\n- At (3) of page 12 it is mentioned that Tables 10-11 are in the Appendix whereas they are in the main text.\n\n", "broader_impact_concerns": "No concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "OGcw2NAzax", "reviewer": "Reviewer_d58D", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "I enjoyed reading the paper as it was clearly written and I felt no details were hidden or missed. I also enjoyed the rather simple insights that lead motivates the theoretical and empirical study. The method has a few moving bits and pieces and a couple of hyperparameters that require tuning, nevertheless, the method seems quite robust. Although it wouldn't be a method that would be on the top of my list to try in practice, due to its additional complexity, I think the study is well executed and is of interest to the TMLR community. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper suggests a new learning rate scheduler for iteratively pruning of deep ReLU networks. Motivated by the insight that the gradient magnitude, as well as the hidden representation, reduce during iterative pruning which the authors try to tackle and therefore motivate a new scheduler increasing the learning rate during the iterations. The method is well benchmarked against related works and baselines  and achieves SOTA results on various benchmarks. ", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "I think a couple of additional details and ablations would make the paper even stronger:  \n\n1. Does the method actually help increase the gradients, after scaling with a larger learning rate, and keep it constant throughout the cycles? This is just a verification that your motivation that mitigates this problem helps throughout pruning cycles makes sense. \n\n2. Regarding your theoretical results: Can you verify these bounds in practice? If you apply pruning techniques in two-layer nets, do you observe these bounds in practice? And then, does your silo scheduler mitigates this in practice? This would be of great interest to me. Otherwise, these results seem a bit overwhelming although I admire the effort in producing these results. Disclaimer: I did not study the proofs of the results and therefore did not understand the implications from them fully. Please correct me if I misunderstood something. \n\n3. Ablations: It would be nice to get an intuition of the sensitivity of the algorithms' hyperparameters. Can you provide a small study on one of your benchmarks that show what happens when moving e.g. \\epsilon or \\gamma?\n\n", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper studies the learning rate schedule for iterative neural network pruning. Motivated by empirical observation and theoretical motivation, it proposes an S-shaped learning rate schedule during pruning iterations. The approach shows performance gains across various datasets, architectures, and optimizers. 

Three reviewers provided detailed and constructive comments for this paper and the authors responded with additional experimental results and theoretical analysis. Most issues, except one remaining assumption under the theoretical results, were well addressed in the revised version. Nevertheless, the authors have provided empirical evidence in support of the theoretical motivation (e.g. Fig. 3) as well as the effectiveness of the LR schedule (e.g. Table 11). These contributions are meaningful for the community and AE suggests the Accept decision.
Decision: Accept (poster)