A Unified Framework for Soft Threshold Pruning
Authors: Yanqi Chen
Zhengyu Ma
Wei Fang
Xiawu Zheng
Zhaofei Yu
Yonghong Tian
Venue: ICLR 2023 poster (2023)
Keywords: Network Pruning
Network Compression
Spiking Neural Networks

Abstract: Soft threshold pruning is among the cutting-edge pruning methods with state-of-the-art performance. However, previous methods either perform aimless searching on the threshold scheduler or simply set the threshold trainable, lacking theoretical explanation from a unified perspective. In this work, we reformulate soft threshold pruning as an implicit optimization problem solved using the Iterative Shrinkage-Thresholding Algorithm (ISTA), a classic method from the fields of sparse recovery and compressed sensing. Under this theoretical framework, all threshold tuning strategies proposed in previous studies of soft threshold pruning are concluded as different styles of tuning $L_1$-regularization term. We further derive an optimal threshold scheduler through an in-depth study of threshold scheduling based on our framework. This scheduler keeps $L_1$-regularization coefficient stable, implying a time-invariant objective function from the perspective of optimization. In principle, the derived pruning algorithm could sparsify any mathematical model trained via SGD. We conduct extensive experiments and verify its state-of-the-art performance on both Artificial Neural Networks (ResNet-50 and MobileNet-V1) and Spiking Neural Networks (SEW ResNet-18) on ImageNet datasets. On the basis of this framework, we derive a family of pruning methods, including sparsify-during-training, early pruning, and pruning at initialization. The code is available at https://github.com/Yanqi-Chen/LATS.

Published as a conference paper at ICLR 2023
A UNIFIED FRAMEWORK FOR SOFT THRESHOLD PRUNING
Yanqi Chen 1. Zhengyu Ma Wei Fang Xiawu Zheng': Zhaofei Yu Yonghong Tian National Engineering Research Center of Visual Technology. School of Computer Science, Peking University; IInstitute for Artificial Intelligence, Peking University: 'Peng Cheng Laboratory yhtian@pku. www.eppkuuuuuu.......o edu.cn, mazhy@pel ac. en; ('Corresponding author)
ABSTRACT
Soft threshold pruning is among the cutting-edge pruning methods with state-of- ! the-art performancd" However, previous methods either perform aimless search- ing the threshold scheduler simply set the threshold trainable, lacking the- oretical explanation from a unified perspective. this work, we reformulate soft threshold pruning
implicit optimization problem solved using the Iterative Seennkage--Tresholdii Algorithm (ISTA), a classic method from the fields sparse recovery and compressed sensing. Under this theoretical framework, all threshold tuning strategies proposed in previous studies of soft threshold prun- ing concluded as different styles of tuning Li Ze-reuularization term. We fur- ther derive an optimal threshold scheduler through an in in-depth study of threshold scheduling based on our framework. This scheduler keeps L1 -eguglllarizttin coef- ficient stable, implying a time-invariant objective function from the perspective of optimization. In principle, the derived pruning algorithm could sparsify any math- ematical model trained via SGD. We conduct extensive experiments and verify its state-of-the-are performance on both Artificial Neural Networks (ResNet-50 and MobileNet-V1) and Spiking Neural Networks (SEW ResNet-18) on ImageNet datasets. On the basis of this framework, we derive a family of pruning meth- ods, including sparsify-during -taiinnn,,iiin, early pruning, and pruning at initializa- tion. The code available at httpt::////.. /topps://iii t.ub..u..b.... com/Yanq Chwn/IATT
INTRODUCTION
Pruning has been thriving area of network compression. Since the day deep neural networks stretch their tentacles to every corner of machine learning applications, the demand for shrinking the size of network parameters has never stopped growing. Fewer parameters usually imply less
computing burden on eeoouurce-oontraineed hardware such as embedded devices or neuromorphic chips. Some pioneering studies have revealed considerable redundancies in both Artificial Neural Networks (ANNs) (Han et al. 2015) 2016 Wen et al. 2016 Liu et al. 2017) and Spiking Neural
Networks (SNNs) (Qi et al. 20188: Chen et al. 2021 Yin et al. 2021 Deng et al. 2021 Kundu
al. 2021 Kim et al. 2022b).
In essence, pruning can be formulated as an optimization problem under constraint on Lo norm, the number of nonzero components in network parameters. Assuming L is the loss function of vectorized network weight w, we expect lower Lo norm ||wl|o along with lower loss L(w). Despite different formulations like hard constraints
min
|
((w)Sc
min [|w]oo
L(w);
(2)
or soft constraints (penalized)
min + + w|]
(3)
2020) is the first to achieve >50% Top-1 accuracy of ImageNet on ResNet-50 under 999 sparsity. STDS (Chen al. (002) is the first pruning algorithm achieving acceptable performance degradation (~> 3% under 88.8% sparsity) for spiking neural networks with 18+ layers.
For example, STR (Kusupati et al

---

Published as a conference paper at ICLR 2023
all these forms are without exception NP-Hard (Ntarajan 1995 Davis et al. 1997: Nguyen et al.
2019). Relaxing Lo norm to (p < p < 1) norm will not make it more tractable for it is still strongly NP-Hard (Ge et al. 201T). Nowadays, research
pruning and sparse optimization is mainly focused on the l--regularized problem, the tightest convex relaxation of Lo norm, which dates back to a series of groundbreaking studies on compressed sensing (Donoho 2006 Candes it al. (2066). These researches technically allows us to solve L Li-regularized problem as an alter- native or, sometimes even an equivalent option (Candès) 2008) confront Lo norm constraint. A pruning are still firmly rooted in solving the L1 regularized optimization problem. Be that as it may, L1 regularization is mostly employed for shrinking the magnitude of weight before the hard thresholding step, which has started to be replaced by other sorts of novel regularization (Zhuang et al. 2020).
variety of modern methods such as magnitude-
In the past few years, a new range of pruning methods based on soft threshold reparameterization of weights has been developing gradually. The term "reparameterrzattoon here refers to a specific map- ping to network weights from a latent space of hidden parameters O. The "geometry" of latent space could be designed for guiding actual weights w towards sparsity. In soft threshold prun- ing, the mapping is element-wise soft threshold function with time-variant threshold. Among these studies, two representative ones are Soft Threshold weight Reparameterization (STR) (Kusu- pati al. 2020) and State Transition of Dendritic Spines (STDS) (Chen et al., (022). They both achieve the best performance of that time. STDS further demonstrates the analogy between soft threshold mapping and a structure in biological neural systems, i.e., dendritic filopodia and ma- ture dendritic spines. However, few researchers notice that soft threshold mapping also appear as the shrinkage operator in the solution of LASSO [Tibshirani] (996) when the design matrix or- Algorithm (ISTA) (Daubechies al. 2004. Elad] 2006), which used to be popularized in sparse recovery and compressed sensing. ISTA has many variants EBiucas-Diass & Figueiredo 2007 BBeck & Teboulle, 2009b Bayram & Selesnick 2010) and has long been certified as an effective sparsification methods in all sorts of fields like deep learning (He et al. 2017 Zhang et al. 2018 Bai et al. 2020), computer vision (Beck& & Teboulle 2009a| Dong et al.
thonormal. The studies on LASSO further derives the Iterative Shrinkage-Threhhollee
| 2013), medical imageology (Lustig et al. 22077] Otazo et al. 2015) and geophysics (Herrmann & Hennenfent 2008). Despite an abecedarian analysis on the similarity between STDS and ISTA, many issues remains to be addressed, such as 1) the exact equivalence between ISTA and the growing threshold in soft threshold pruning, 2) the necessity of setting threshold trainable in STR, and 3) the way to improve existing methods without exhaustively trying different tricks for scheduling threshold.
In this work, we proposed a theoretical framework serving as a bridge between the underlying L1- regularized optimization problem and threshold scheduling. The bridge is built upon the key finding that soft threshold pruning is an implicit ISTA for nonzero weights. Specifically, we prove that the Ln coefficient in the underlying optimization problem is determined by both threshold and learning rate. In this way, any threshold tuning strategy can now be interpreted as scheme for tuning L1 penalty. We find that a time-invariant L coefficient lead to performance towering over previous pruning studies. Moreover, we bring a strategy of tuning L1 penalty called continuation strategy (Xiao & Zhang 2012), which was once all the rage in the field of sparse optimization, to the field of pruning. It derives broad categories of algorithms covering several tracks in the present taxonomy of pruming. In brief, our contributions are summarized as follows:
Theoretical cornerstone of threshold tuning strategy. To the best of our knowledge, this is the first work interpreting increasing threshold as an ever-changing regularized term. Under theoretical analysis, we present a unified framework for the local equivalence of ISTA and soft threshold pruning. It enables us to make a comprehensive study on threshold tuning using the classic method in sparse optimization.
Learning rate adapted threshold scheduler. Through our proposed framework, we reveal the strong relation between the learning rate scheduler and the threshold scheduler. Then we show that an time-invariant L1 coefficient requires the changing of threshold being proportional to the learning rate. The Learning rate Adapted Threshold Scheduler (LATS) built upon L1 coefficient achieves a state-of-the-art performance-sparsity tradeoff' on both deep ANNs and SNNs.
Sibling schedulers cover multiple tracks of pruning. We propose an early pruning al- gorithm by translating the homotopy continuation algorithm into a pruning algorithm with
2

---

Published as a conference paper at ICLR 2023
our framework. It achieves indistinguishable performance LATS conventional early pruning method. Moreover, the algorithm in the pruning-at- iete--attiiiiizaatiin setting erases some subsequent layers in ResNet and maintains the identity mapping, shrinking deep ResNet to a shallow one.
RELATED WORKS
There has been deluge of pruning algorithms emerged since the term "deep compression" in- vented. These various studies emphasize different points like granularity (structured or unstructured) and stage of pruning (at initialization, during training. post training). The difference in granularity is similar to that between LASSO and group LASSO. Empirically, unstructured pruning tends to reach higher sparsity under the same accuracy degradation. For the pruning phase, sparsify during training commonly lead to higher accuracy than early phase one, e.g., pruning at initialization. Moreover, pruning during training is cheaper than post-training pruning when the overhead of dense training considered. Some most relevant works are introduced as follows.
Sparsify during training. The terminology sparsif-duuinn-traaine (also called gruning-while learning, serggdduriinttiiiinnn training networks simultaneously including those iteratively pruned networks. Recent works in this
is mentioned in Hoefler et al. (2021), which refers to pruning and
area includes STR (Kusupati et al.] (220)) Top-KAST (Jayakumar al. 2020), CS (Continuous
Sparsification) (Savarese et 2020).. RigL (Evci et al. 2020h, Woodrisher (Singh & Alistarh, 2020), PSGD (Kim et al. (020), GraNet (Liu al. 202 Powerprop (Schwarz al., (021),
ProbMask (Zhou et al. 2021). GPO (Wang et al. 20022, OptG (Zhang et al.. (002) and STDS (Chen al. 2022). Many of these works are based the reparameterization of weights using either binary mask or element-wise nonlinear mapping. The former choose to confront Lo constraint directly while the latter are committed to adjusting the landscape of loss function around zero. Our method is based on soft threshold reparameterization, which is piecewise linear and has an intrinsic connection the ISTA with L1 regularization.
Early pruning. We refer to a variant of sparsify-during-taaile as early pruning here, which only exerts pruning network in the early stage of training. includes pruning initialization, e.g. GraSP (Wang et al. 2020). SynFlow (Tanaka et al. 2020), SBP-SR (Hayou et al. 202I), ProsPr (Alizadeh et al. 2022). and the conventional early pruning methods which stop pruning after several epochs of training You et al.) 2020 Liu et al. 2021b Rachwan et al. 2022). Most of these works are inspired by the discovery Lottery Ticket Hypothesis (LTH) (Frankle Carbin 2019) or SNIP (Lee et al. 2019), if not both. LTH suggests we can find sparse subnetwork with comparable performance to the dense network after iterative retraining, while SNIP manages to train from an initial sparse network. A wide array of criteria like connectivity sensitivity are taken for finding such sparse networks in the carly stage with promising accuracy after training.
PRELIMINARIES
Notation. We use x to denote the element- tiinent-wiieeeee absolute value of and ||>||p denotes the p- norm of æ. æ O denotes the element- wise product of x and y. If not otherwise specified, the superscript within parenthesis (+)) denotes x at i-th iteration of gradient descent. The element-wise sigmoid function is denoted by a(x); : 1/(1 e- The soft threshold mapping is also an element-wise mapping defined by Sa(x); := sign(r;) max { [ d, 0} with scalar threshold d.
3.1 SOFT THRESHOLD PRUNING
Basically, soft threshold pruning will iteratively execute following three core steps:
(i) Mapping hidden weight to actual weight w through the soft threshold mapping (t) <- ()(()) during training, where is a trainable hidden weight with the same shape as w.
iii Training hidden weight 0 through backpropagation in latent space
(iii) Growing threshold pushes the term max [16,] - d, 03 in (x()) towards zero and thereby enforces sparsity for w.
3

---

Published as a conference paper at ICLR 2023
Algorithm The general form of soft threshold pruning algorithm coupled with vanilla SGD (STR and STDS for instance).
Input: initialized network parameters w/ threshold scheduler function s()). initial threshold 2(9), final threshald P. Initial learnable
parameter Bust loss function (tw), the number of training iterations T, La penalty A.
Output: trained sparse parameters w
1: g(o) Stiz. (0)) <- as (00)
w
Initialization of weight and threshold
2: for 0. I......
do
4: gir+1) g(0) (50 aa
3: A0(*) V (=(((((())
Computing gradient with respect to hidden weight
)
Update hidden weight. (f is learning rate
5: g(tt)) gth (V.(
Threshold training
6: d(++1) -p((ltt) l0((t + 11/7)
Update threshold
7: w(++1)
(
Di Update the actual weight
8: end for
9:
return w (T)
We provide a general form of soft threshold pruning as Algorithm [ which is a prototype of both STR and STDS, GPO change the mapping in line to a convex combination of soft threshold and identity mapping. leading to a slight difference. However, GPO only obtain marginal performance improvement with respect to STR, and will soon degenerate to STR as discussed in Appendix C Therefore, the following of this paper are focused on STR and STDS. The differences between them are concluded from two aspects.
Gradient computing. With the reparameterization mapping w < (,(0))), the forward step includes mapping via 0 + w and evaluating loss with w as the actual network weight. Hence, gradient is backpropagated via path L - w - O. The learning rule is thus used for updating O instead of un. Note that Sy is non-differentiable d and has zero gradient interval (-d, STR takes advantage of the subgradient at +-d and leaves zero gradients as it is. STDS views S2 as an identity mapping during backward, and provides a convergence analysis by approximating Sa using a smooth surrogate Sa(x)= = 1 [[(a(x - d)) <(-a(z + d))]. where ((x) == log(1 + e²) denotes the softplus function.
Threshold tuning. These studies spontaneously try manipulating threshold d for different sparsity. STR assigns an independent threshold for each layer, resulting into threshold vector d. Besides, STR further parameterizes the threshold by another trainable parameter s. The mapping from s to is individually designed for CNN and RNN. Compared STR, STDS set threshold manually by introducing the threshold scheduler dt g(t/T) ( D. wherein scheduler function g [0, [0, 1] is increasing and satisfies g(0) = 0, (1) 1, T is the total training steps. The formulation based the idea that inereasing threshold from D could follow different paths. The final threshold D is the only adjustable hyperparameter for different sparsity levels when g is given.
Larger D always leads higher sparsity in practice.
The above techniques are summarized in Tab.
Table 1: Techniques used in previous studies.
Method
Reparameterization mapping
Gradient of mapping
Threshold
Note
(0,.4)
layer-wise (glohal for STR GS), trainable, and Ly regularized s. are layer-wise, trainable. and l.y regularized g is increasing. 900 0, 0,0(1)
STR Kusupati al. 2020
S2(A)
Suhgradient
d=
[a(s),
for CNN
for RNN
GPO
w k)),(()
Gradient
a(*)
Wang et al. 2022
STDS Chen 2022
Su(0)
1 (Viewed as identity)
d
a(t/T)
3.2 ITERATIVE SHRINKAGE- ALGORITHM
ISTA is initially derived from solving linear inverse problem with regularization
minweR" {I[Ax bl2 + r(z)) where A mmxn and b E RT. ISTA later extended to general objective min (F(x)= = f(x) + r(m)} with assumptions listed below
4

---

Published as a conference paper at ICLR 2023
(i) Objective function f R - R is continuous differentiable, and L-smooth, i.e., || f(x) f & | - y 22, Va, y E IR,, where Ly > 0 is the Lipschitz constant of V..
(ii) Regularization function r IR + is continuous convex and can nonsmooth.
(iii bounded from below.
Leave the regularization r(z) alone, applying vanilla SGD f can be viewed as iteratively cal- culating proximal regularization (Martinet 1970) of the linearized I at æ, which is suggested by the following fact: v (()) arg min [x)) (y x. V + 2n lly ==12) where
explained "stepsize" in optimization "learning rate" the context of deep learning. For given point x, Fzy) can be approximated by expanding f to the quadratic term in similar vein
1 fly 2=12 r(y):
av = [(x + (y Vf(a)) + 2nj
(4)
The above problem admits unique minimizer
arg min Fy; æ)) = arg min
(5)
2n
S Note that = Dj gives a upperbound of (t) which is obtained by the descent lemma f(y)
2017). It implies with proper choice of 7, we ((x (y- ()x)) + 1 lly x 12 Beck
are virtually optimizing the upperbound of F(y using minimizer Eq. 5 The general form of ISTA iteratively solves Eq. 5 as x (1+1) arg min. (æ; (()),, which is also known as proximal gradient methods (Combettes vast optimization literature like FISTA (Beck & Teboulle, 2009b) and GIST Gong et al., 2013). For sparsity, we are interested in L1 regularization term r(x) p 1. O. Since L1 norm separable, we have closed-form solution with element- wise soft threshold operation as
Wajs. 2005). The detailed convergence analysis is presented in
ar ((+1) (p (xx)))))))
Eq. 6 gives the ISTA update rule under L1 regularization.
A FRAMEWORK FOR SOFT THRESHOLD PRUNING
In this part, will formulate the growing threshold under soft threshold reparameterization as implicit ISTA. For simplicity, we assume the threshold is global across all parameters, which is consistent with the setting of STDS and STR-GS, i.e., the global threshold version of STR. Hence, we use scalar d rather than vector in Algorithm to denote the global threshold.
To begin with, investigate the update rule of nonzero components in actual weight
@(++)) - (t) - n ( V (C(w Vo(w(Ol)) d())), (1+1) < Sqoin (2
Line in Algorithm
(7)
Line in Algorithm
(8)
Assuming the sign of weight remains unchanged after an update, which happens when the gradient has the opposite sign of weight or the gradient magnitude is sufficiently small, we have the following Lemma:
Lemma (Local update rule). The update rule Eq. 7 and Eq. implies the following update of nonzero component (0,0g in actual weight
w (++) SSutt (t) (0 Vn (C(w()))))
(9)
The formal version and proof of Lemma is given Appendix A Note that the form in Eq. equivalent to the ISTA update rule with threshold equal forward finite difference d(++1) ) Recall Eq. 6, the corresponding optimization problem can be deduced as Theorem
Theorem 1. Let L(w) be the loss function depending on the network weight w. which is further reparamterized by hidden weight 0 and threshold d as un(0d) d) Sa(0). When applying vanilla
5

---

Published as a conference paper at ICLR 2023
SGD reparameterized nonzero weight, the update rule is locally equivalent solving the follow- ing problem using ISTA with penalty term
min F(w)= := L(w) d(t+) cltt)
||1/|/
(10)
17(0)
where 10 dd()) denotes the learning rate and threshold at the 1-th iteration, respectively. The mag- nitude of L1 Li-penalty the quotient of forward finite difference of threshold scheduler divided by learning rate.
Theorem serves as the glue holding the learning rate scheduler (²) ttreshhldd d and penalty ult) together. It also some extent justify the increasing threshold for a positive penalty term. Under this framework, we further explain Appendix that the original STDS uses an improper threshold scheduler. Moreover, the validity of training threshold is discussed in Appendix
FINDING OPTIMAL THRESHOLD SCHEDULER
In this part, we are devoted to evaluating some threshold schedulers based on our theoretical frame- work and literature on sparse optimization. With the framework, any past strategy of tuning L penalty can be converted to a feasible pruning algorithm today.
5.1 LEARNING RATE ADAPTED THRESHOLLD SCHEDULER
If the optimization problem in Theorem is fixed, or in other words, the L1 penalty is invariant ( = μ during learning, we can deduce unique threshold scheduler LATS. In LATS, the change on threshold d( d( must be proportional to the learning rate 17(5) during training.
Corollary (Learning rate Adapted Threshold Scheduler). For fixed Ln regularized problem min (F(w) L(w) + /l[wa|11)
(11)
where μ > 0 is the time-indeeee indeieddeeenentt L1 penalty coefficient, the threshold scheduler is governed by the learning rate scheduler ( (
For the most frequently used testbed, ResNet-50 (He et al. 2016) on ImageNet dataset (Deng et al, 2009, researchers usually use the cosine annealing learning rate scheduler (Loshchilov & Hutter 2017 with Mmin 0. Assuming the initial threshold is zero, simple algebra shows the definition of corresponding LATS with slight abuse of notation
((n = 1.7max
2n
sin
2N
+ 2
(1
(1 cos
TLT
(12)
4
sin
N
2N
where the threshold ((n.8) depends on epoch id n and batch id b. Here n 0, 1, I....,
N denotes
the current id of training epoch, and b= 1, B denotes the batch id in n-th epoch. We elaborate
the detailed derivation of Eq. 12 in Appendix
5.2 SIMPLIFIED THRESHOLD SCHEDULER
Computation such Eq. 12 is intricate for implementation. In effect, painstakingly coding LATS according to given learning rate scheduler not inevitable. To ease the computing burden, we turn replacing sum of learning rate with integration of learning rate function.
To be specific, assuming the learning rate scheduler can be expressed by r)(n.b) h(n/N), the simplified threshold scheduler is defined by
( N S h(z)dr
(13)
So h(x)dr
The simplification is loyal the idea that the value of Riemann integral could be approximated by rectangle method. Eq. can be interpreted with scheduler form in STDS ((t) g(l/T)

---

Published as a conference paper at ICLR 2023
with scheduler function g(a) Sot h(x)dx// Sg h(r)dd and final threshold d(N-1,B), The detailed derivation of Eq. 13 is given in Appendix
Now we have the Simplified LATS (S-LATS for short) for the cosine annealing learning rate sched- uler (x)) Tmax (1 (Ta)
d(0) MMlmx T
Jo 2 costz)dr
pTmmax T
So 2 cos costx)dæ 2 sin
(14)
2
T
The final threshold is D pammax 7/2, which satisfies D μr. Tuning D is thus akin changing the magnitude of penalty. In the following discussion, we employ the final threshold D instead of d to lighten the notation. The threshold schedulers in the rest this work will thus be expressed in a unified form of d(t) = g((/T)) D. We evaluate LATS and S-LATS under identical
Table 2: Comparison of LATS and S-LATS when applicd ResNet-50 on ImageNet dataset.
STDS LATS
STDS S-LATS
Final threshold D
Spursity (%)
Top-1 Acc. (%)
Spansity (%)
Top-1 Acc, (%)
0.1
79.97
76.53
79.95
76.75
0.5
95.54
73.12
95.53
73.03
1.0
97.43
69.56
97.43
69 64
5.0
99.27
53.80
99.28
$3.64
final thresholds D 0.5, and S-LATS are indistinguishable from accuracy and sparsity. Thus, we turn to simplified threshold scheduler in the following discussion.
0.5, 1.0, 5.0, which can be gleaned from Tab. 2
The results show LATS
5.3 CONTINUATION STRATEGY
Also known as "warm starting", continuation strategy is designated for accelerating convergence. Similar to anncaling of learning rate, continuation refers to gradually reducing Li penalty during learning. also explained in Hale et al. (2008) as an analogue to the homotopy algorithms common trick in abundant classic literature
in statistics. Continuation method used to concerning sparse optimization including GPSR (Figueiredo et al. (007), fixed point continuation
method (Hale et al. 2008). SpaRSA (Wright et al. 2009) and NESTA (Becker al. 2011
5.3. PGH SCHEDULER
In the series works of proximal gradient homotopy (PGH) (iiao & Zhang 2012 2013 Lin Xiao
2014]. the researchers provide proof of geometric convergence rate when inducing exponentially decaying L1 coefficient ( ,4, where 0 < 3 < 1 is constant. Considering our formulation PGH can be translated into the PGH scheduler (simplified
of soft threshold pruning Theorem [
using Eq 13), which can be written
f 1 cos 8 D,
( = (PTGH(//T)
(15)
fo [1 (1 + cos στ) ( dr
where the analytic form of gPGH shown below
BPGH(A) (a:) 2² - -1) (8) (8" -2) + log( (log(3) cos(TT)) + n (sin(T2) (Tx))
(16)
7²(3 123-11 (g)
5.3.2 LINK TO EARLY PRUNING
For the PGH scheduler, we interpolate 3 between and and get series of different PGH sched- ulers. As shown in Fig. [ the increasing in threshold slows time, which is caused by decaying L1 penalty. For 0 < 3 1, pruning is ignored when penalty is below a preset threshold, we get a family of early pruning algorithms. They will stop pruning different stages. Recall that the regular- ized term is proportional to the forward finite difference and thus can be approximated by derivative, for conventional early pruning, regard (bGu(t/T) < 0.1 the termination criterion of pruning.

---

Published as a conference paper at ICLR 2023
There are two limit cases when 3 approaches 0 1. It is obvi- ous that 3 + 1 leads to S-LATS for no decay is applied. When + O, the penalty is always this case, PGH scheduler degenerates to a magnitude-hann based prun- ing after weight initialization followed by normal training stage. This is also referred to as sparse-to-sparse training or pruning at initialization method.
1.00
except for the beginning. In
(rould)
0.75
- 1e-10
1e-05
S-LATS
0.00
0.25
0.50
0.75
1.00
Figure 1: PGH schedulers un- der different 3.
EXPERIMENTS
In this section, we test proposed threshold scheduler S-LATS on both deep ANNs and SNNs. The favorable performances against the previous studies confirmed. In all experiments we switch sparisty levels by changing D, which is equivalent to tuning L1 penalty coefficient. We also tune hyperparameter in PGH scheduler maintain different phases of early pruning algorithm. Com- pared to the dense baseline, no tuning on other training hyperparameters is needed, which minimize the effort when applying other networks.
6.1 S-LATS
S-LATS achieves state-of-the-ar performances both ANNs (ResNet-50, MobileNet-Vi (Howard et al. 2017)) and SNNs (SEW ResNet-18 18 (Fang et al.] (021)). The results on ResNet- networks and MobileNet-Vi
illustrated in Fig. 2 We add Gradual Magnitude Pruning (GMP) (Zhu & Gupta 2017) into comparison for few recent studies are conducted on MobileNet-V1. Notably,
method surges ahead of all the other baselines under <98% sparsity for pruning ResNet-50, which is shown in Tab. 3 It should be noted that the origin STDS excludes the last FC layer from pruning SNNs, while the results reported here shown by rerunning it with the last layer pruned.
We also admit it fails to achieve comparable performance with a few baselines like OptG and in
Table 3: Comparison of ResNet-50 Top-1 accuracy on ImageNet in recent studies using 100 training epochs. For some studies without strict control on sparsity, the closest sparsity is reported behind the performance. Performance of S-LATS is averaged over three trials.
Spunity
99%
STR
246
(69.9
74.31 (90.25)
70.40
67.22
51.82 (98.5)
STR-28
256
74.13 74.13 (89.54)
62.17 (97.91)
GraNel
256
76
74.5
72.3
70,5
PaisMaak
256
74.68
71.5
66.83
61.07
Opt
256
74.28
12.35
70.85
67.2
62.1
WiedFisher
256
76.73
75.25
72.16
65-47
S-LATS
256
76. 17 (79 (79.05)
755.43uu17 (89
73.20 7A020 (95.
71.49 (96.5))
69.49 (97.4))
67.25 29 (98.01)
58. 19.3 (99
S-LATS
1024
76.61am25 (79020)
75.87auus (90.15)
74.29,aza (95.01)
72,80,uus (96.53)
70.7% (97.54)
69.15 (98.00)
61.90am (98.93)
67.2
Top KAST
(Powespeop)
76.34
Top-KAST
(bwsseep-RRK)
extreme sparsity (299%), which suggests the theory might be imperfect under such conditions. An ablation study shows S-LATS outperforms the default threshold scheduler of STDS, which is shown Appendix
Top1 noc. (6)
70
SIH
70
68
S-LATS (256)
66
Opta
66
STR
S-LATS
TOO-1
GO
S-LATS
85
S-LATS (1024)
PPreeran (4008)
PreaMesk
GMP
STCS: (Sine ach-)
55
PG corespic (4006)
WoodFlaher
WiodFFlle
50
50
80
85
85
130
75
85
80
70
80
90
80
90
100
Sparsty (34)
Sparsity
Spansiy (%)
Sparsity (%)
Figure 2: Performance of several SOTA pruning strategies of ResNet-50 (Leftmost & Rightmost), MobileNet-Vi (Middle left) and SEW ResNet-18 (Middle right) on ImageNet. All trials uses the standard training setting (256 batch size) except the rightmost one, which uses an enlarged batch size marked in parenthesis. Detailed layerwise sparsity and accuracy are given Appendix
8

---

Published as a conference paper at ICLR 2023
Pruning using large batch size. Powerprop (Schwarz et. 2021). adopts batch size of 4096 for pruning of ResNet-50 and achieves fascinating performances under high sparsity. Hence, we explore the large batch size setting. Due to limited resources, only increase it to 1024 and enlarge the learning rate correspondingly. The rightmost of Fig. 2 shows indeed leads to higher performance, which outperforms all other SOTA studies. Astonishingly, even though the performance of the dense network slightly degrades, the accuracy of the sparse ones is improved overall. On the basis of the above finding, we believe applying even larger batch size, like 4096 used in Powerprop, to our method may lead to top-notch performance tradeoff.
6.2 PGH SCHEDULER
Conventional early pruning Our experiments of early pruning includes 3 = 0.1, 10-5, 10-10
The corresponding ending criteria t/T 0.743, 0.382, 0.231 are given by numerical solution. indicates pruning roughly stops at the 74th, the 38th and the 23th epoch. The network using PGH scheduler with smaller 3 converges faster to a sparse one, which is illustrated in the left of Fig. 3
Surprisingly, we find in the middle of Fig. 3 that for different 3, the datapoint of accuracy against
Laper Spuaaty
Spach
Figure 3: Overall sparsity during learning when final threshold D 0.1 (Left). Performance under different sparsity levels (Middle). Layerwise sparsity of PGH scheduler under pruning at initialization setting 3 + 0 (Right).
sparsity almost lies on the curve S-LATS. It suggests these schedulers have practically the same performances as S-LATS, but with faster convergence to sparse networks. With the help of PGH scheduler, we are able to find sparse networks earlier with negligible performance degradation.
Pruning at initialization We also try 3 O, which refers increasing the threshold to its maximum at the first iteration. Note that our method is agnostic about the structure completely pruned shown in the right of Fig.
of network. Hence, some layers
wherein three consecutive layers within a residual block tend be pruned simultaneously. owing to the skip connection in ResNet, the feature can still pass through shortcuts to the final FC layer, and thus the whole networks are still normally trained. The afore- mentioned results are collected in Tab.
Table 4: Results pruning at initialization experiments.
However,
Final threshold D
0.1
0.11
0.13
0.15
Overall sparsity (%)
87.16
90.00
93.11
95.64
Top-1 Acc. (%)
74.69
72.89
68.23
62.22
Zeroed layers
0
9
9
27
CONCLUSION & DISCUSSION
In this work, we present a framework interpreting increasing threshold as constantly changing penalty term and reveal the underlying connection between soft threshold pruning and ISTA. We also derive a couple of threshold schedulers, which achieve comparable performance to current SOTA works and cover multiple tracks of pruning. It is worth noting that our method is agnostic about the object pruning. This design endows our method with versatility while treating weight wheresoever equally, and yet becomes totally ignorant of nowadays pruning researches like spar- sity budget allocation, e.g., Erdös--éénn (Mocanu et al. 2018) and Frdós-Rény Rényi-Kernel Kernel (ERK) (Evci) et al., 2020), or commonsense in this area like "Leave at least path from input through out- put". - We believe our method can for sure further benefit from knowledge in the prosperous field of network pruning.

---

Published as a conference paper at ICLR 2023
ACKNOWLEDGMENTS
This work is supported by grants from the National Key R&D Program of China under Grant 2020AAA01035, the Key-Area Research Development Program of Guangdong Province (201180001400002, and the National Natural Science Foundation of China under contract No. 62088 102, No. 62176003, No. 62006132, No. 62027804 and No. 61825101. The computing resources of Pengcheng Cloudbrain are used in this research.
REFERENCES
Milad Alizadeh, Shyam A. Tailor, Luisa M Zintgraf, Joost van Amersfoort, Sebastian Farquhar, Nicholas Donald Lane, and Yarin Gal. Prospect pruning: Finding trainable weights at initializa- tion using meta-gradients. In International Conference on Learning Representations, 2022.
Haoli Bai, Jiaxiang Wu, Irwin King, and Michael Lyu. Few shot network compression via cross distillation. Proceedings of the AAAI Conference on Artificial Intelligence, 34(04):3203-3210, Apr. 2020.
llker Bayram and Ivan W. Selesnick. A subband adaptive iterative shrinkagg//hrresholding algorithm. IEEE Transactions on Signal Processing, 58(3): 131-1143, 2010.
Amir Beck. First- Wrst--rder Methods in Optimization, pp. 109. Society for Industrial and Applied Mathematics, 2017.
Amir Beck and Marc Teboulle. Fast gradient-based algorithms for constrained total variation image denoising and deblurring problems. IEEE Transactions on Image Processing, 18(11))2419-22)) 243-2444, 2009a.
Amir Beck and Mare Teboulle. fast iterative thrinkage-thrrssholling algorithm for linear inverse problems, SIAM Journal on Imaging Sciences, 2(1):183-202, 2009b.
Stephen Becker, Jérôme Bobin, and Emmanuel J. Candès. NESTA; A fast and accurate first-order method for sparse recovery. SIAM Journal on Imaging Sciences, 4(1):1-39, 2011.
Yoshua Bengio, Nicholas Léonard, and Aaron Courville. Estimating or propagating gradients through stochastic neurons for conditional computation. arXiv preprint 130.5333 2013.
JosÉ M. Bioucas-Dias and MÁrio A. T. Figueiredo. A new TwIST: Two-step iterative shrink- age/thresholding algorithms for image restoration. IEEE Transactions Image Processing, 16 (12):2992-3000 -004, 2007.
Emmanuel J. Candès. The restricted isometry property and its implications for compressed sensing. Comptes Rendus Mathematique, 346(9))589-592, 2008
Emmanuel J. Candès, Justin K. Romberg, and Terence Tao. Stable signal recovery from incomplete and inaccurate measurements. Communications on Pure and Applied Mathematics, 59(8): 1207 1223, 2006.
-iang-Chich Chen, George Papandreou, Iasonas Kokkinos, Kevin Murphy, and Alan L. Yuille. Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs. IEEE Transactions on Pattern Analysis and Machine Intelligence, 40(4): 834 848, 2018.
Yanqi Chen, Zhaofei Yu, Wei Fang, Tiejun Huang, and Yonghong Tian. Pruning of deep spiking neural networks through gradient rewiring. In Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, IJCAI-21, pp. 1713-1721, 2021.
Yanqi Chen, Zhaofei Yu, Wei Fang, Zhengyu Ma, Tiejun Huang, and Yonghong Tian. State transi- tion of dendritic spines improves learning of sparse spiking neural networks. In Proceedings of the 39th International Conference on Machine Learning, pp. 3701-3715, 2022.
Mark W Coffey. On some series representations of the Hurwitz zeta function. Journal of Computa- fional and Applied Mathematics, 216(1):297-305, 2008
10

---

Published as a conference paper at ICLR 2023
Patrick L. Combettes and Valérie R. Wajs. Signal recovery by proximal heward-backward splitting. Multiscale Modeling Simulation, 4(4): (144):168811200 1200, 2005.
Ingrid Daubechies, Michel Defrise, and Christine De Mol. An iterative thresholding algorithm for linear inverse problems with sparsity constraint. Communications on Pure and Applied Mathematics,
2004.
Geoff Davis, Stephane Mallat, and Marco Avellaneda. Adaptive greedy approximations, Construc- tive approximation, 13(1),57-98, 1997.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Fei-Fei. Imagenet: A large-scale hier- archical image database. In 2009 IEEE Conference Computer Vision and Pattern Recognition, pp. 248-25822 255, 2009.
Lei Deng. Yujie Wu, Yifan Hu, Ling Liang, Guoqi Li, Xing Hu, Yufei Ding, Peng Li, and Yuan Xie. Comprchensive snn compression using admm optimization and activity regularization. IEEE Transactions on Neural Networks and Learning Systems, pp. 1-15, 2021.
Weisheng Dong, Lei Zhang, Guangming Shi, and Xin Li. Nonlocally centralized sparse representa- tion for image restoration. IEEE Transactions on Image Processing, 22(4):1620-1630, 2013.
David L Donoho. Compressed sensing. IEEE Transactions on Information Theory, 52(4): 1306, 2006.
Michael Elad. Why simple shrinkage is still relevant for redundant representations? IEEE Transac- tions on Information Theory, 52(12))5559-5555 2006.
Utku Evei, Trevor Gale, Jacob Menick, Pablo Samuel Castro, and Erich Elsen. Rigging the lottery: Making all tickets winners. Proceedings of the 37th International Conference on Machine Learning, pp. 2943-2952, 2020.
Wei Fang, Zhaofei Yu, Yanqi Chen, Tiejun Huang, Timothée Masquelier, and Yonghong Tian. Deep residual learning in spiking neural networks. In Advances in Neural Information Processing Systems, pp. 21056-21069, 2021.
MÁrio A. T. Figueiredo, Robert D. Nowak, and Stephen J. Wright. Gradient projection for sparse Application compressed sensing and other inverse problems. IEEE Journal of Selected Topics in Signal Processing, 1(4):586-555 597, 2007.
reconstruction:
Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. In International Conference on Learning Representations, 2019.
Trevor Gale, Erich Elsen, and Sara Hooker. The state of sparsity in deep neural networks. arXiv preprint arXiv: /902.00090000 09574, 2019.
Dongdong Ge, Xiaoye Jiang, and Yinyu Ye. note the complexity of Lp minimization. Mathe- matical programming, 129(2):285-290) 299. 2011.
Pinghua Gong, Changshui Zhang, Zhaosong Lu, Jianhua Huang, and Jieping Ye. A general itera- tive shrinkage and thresholding algorithm for non-convex regularized optimization problems. In Proceedings of the 30th International Conference on Machine Leurning, pp. 37-45, 2013.
Elaine T. Hale, Wotao Yin, and Yin Zhang. Fixed-point continuation for m-minimmzzation: Method- ology and convergence. SIAM Journal on Optimization, 1903):1007-11000 2008
Song Han, Jeff Pool, John Tran, and William Dally. Learning both weights and connections for efficient neural network. Advances in Neural Information Processing Systems, 2015.
Song Han, Huizi Mao, and William Dally. Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding. In International Conference on Learning Representations, 2016.
Soufiane Hayou, Sean-FFanccos Ton, Amaud Doucet, and Yee whye Teh. Robust pruning at initial- ization. In International Conference Learning Representations, 2021.
11

---

Published as a conference paper at ICLR 2023
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
Yihui He, Xiangyu Zhang, and Jian Sun. Channel pruning for accelerating very deep neural net- works. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), Oct 2017.
Felix Herrmann and Gilles Hennenfent. Non-parametric seismic data recovery with curvelet frames, Geophysical Journal International, 173(1):233-448, 04 2008.
Torsten Hoefler, Dan Alistarh, Tal Ben-Nun, Nikoli Dryden, and Alexandra Peste. Sparsity in deep learning: Pruning and growth for efficient inference and training in neural networks. Journal of Machine Learning Research, 22(241):11-22 2021.
Andrew G Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam. Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv:1704.04861, 2017.
Carl Gustav Jacob Jacobi. De usu legitimo formulae summatoriae maclaurinianae. Journal für die reine und angewandte Mathematik, 11-263-272- 1834.
Siddhant Jayakumar, Razvan Pascanu, Jack Rae, Simon Osindero, and Erich Elsen. Top-kast: Top-k always sparse training. In Advances in Neural Information Processing Systems, pp. 20754-20754, 2020.
Jangho Kim, Yion Yoo, and Nojun Kwak. Position-based scaled gradient for model quantization and pruning. In Advances in Neural Information Processing Systems, pp. 20415-20426, 2020.
Youngeun Kim, Yuhang Li, Hyoungseob Park, Yeshwanth Venkatesha, and Priyadarshini Panda. Neural architecture search for spiking neural networks. In Computer Vision ECCV 2022, pp. 36-56, 2022a.
Youngeun Kim, Yuhang Li. Hyoungseob Park. Yeshwanth Venkatesha, Ruokai Yin, and Priyadarshini Panda. Exploring lottery ticket hypothesis in spiking neural networks. In Com- puter Vision - ECCV 2022, pp. 102-120, 2022b.
Donald E Knuth. Johann faulhaber and sums of powers. Matheratiiss of Computation, 61(203): 277-294, 1993.
Souvik Kundu, Gourav Datta, Massoud Pedram, and Peter A. Beerel. Spike-thrift: Towards energy- efficient deep spiking neural networks by limiting spiking activity via attention-guided compres- sion. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pp. 3953-3962, 2021.
Aditya Kusupati, Vivek Ramanujan, Raghav Somani, Mitchell Wortsman, Prateek Jain, Sham Kakade, and Ali Farhadi. Soft threshold weight reparameterization for learnable sparsity. In Proceedings of the 37th International Conference on Machine Learning, pp. 5544-5555, 2020.
Namhoon Lee, Thalaiyasingam Ajanthan, and Philip Torr. SNIP: single-shot network pruning based on connection sensitivity. In International Conference on Learning Representations, 2019.
Qihang Lin and Lin Xiao. An adaptive accelerated proximal gradient method and its homotopy continuation for sparse optimization. In Proceedings of the I International Conference on Machine Learning, pp. 73-81, 2014.
Shiwei Liu, Tianlong Chen, Xiaohan Chen, Zahra Atashgahi, Lu Yin, Huanyu Kou, Shen, Mykola Pechenizkiy, Zhangyang Wang, and Decebal Constantin Mocanu. Sparse training via boosting pruning plasticity with neuroregeneration. In Advances in Neural Information Processing Sys- tems, pp.
202
Shiwei Liu, Lu Yin, Decebal Constantin Mocanu, and Mykola Pechenizkiy. Do actually need dense over- over- parmatterrriinnoonnn in-time over paramettttttinnnnonnn in sparse training. In Proceedings of the 38th International Conference Machine Learning, pp. 6989-7000, 2021b.
12

---

Published as a conference paper at ICLR 2023
Wei Liu, Andrew Rabinovich, and Alexander Berg. Parsenet: Looking wider to better. arXiv preprint arXiv:1506.045: 2015.
Zhuang Liu, Jianguo Li, Zhiqiang Shen, Gao Huang, Shoumeng Yan, and Changshui Zhang. Learn- ing efficient convolutional networks through network slimming. Proceedings of the IEEE International Conference Computer Vision (ICCV), 2017.
Iyy Loshchilov and Frank Hutter. SGDR: Stochastic gradient descent with warm restarts, In Inter- national Conference on Learning Representations, 2017.
Michael Lustig, David Donoho, and John M. Pauly. Sparse MRI: The application of compressed sensing for rapid MR imaging. Magnetic Resonance in Medicine, 58(6): (966):1122) 182-1195, 2007.
Wolfgang Maass. Networks of spiking neurons: The third generation of neural network models. Neural Networks, 1019);165--167,, 1997.
Bernard Martinet. Régularisation d' inéquations variationnelles par approximations successives. Re- vue Francaise d' d'infrrmttquu et de Recherche operationelle, 4:154-158, 1970.
Paul A. Merolla, John V. Arthur, Rodrigo Alvarez-lcaza, Andrew S. Cassidy, Jun Sawada, Filipp Akopyan, Bryan L. Jackson, Nabil Imam, Chen Guo, Yutaka Nakamura, Bernard Brezzo, Ivan Vo, Steven K. Esser, Rathinakumar Appuswamy, Brian Taba, Arnon Amir, Myron D. Flickner, William P. Risk, Rajit Manohar, and Dharmendra S. Modha. A million spiking-neuron integrated circuit with a scalable communication network and interface. Science, 345(6197):668-673 2014.
Decebal Constantin Mocanu, Elena Mocanu, Peter Stone, Phuong H Nguyen, Madeleine Gibeseu, and Antonio Liotta. Scalable training of artificial neural networks with adaptive sparse connec- tivity inspired by network science. Nature communications, 91):1-12, 2018.
Byunggook Na, Jisoo Mok, Seongsik Park, Dongjin Lee, Hyeokjun Choe. and Sungroh Yoon. Au- toSNN: Towards energy-efficient spiking neural networks. In Proceedings of the 39th Interna- tional Conference on Machine Learning, pp. 16253- 2022.
Balas Kausik Natarajan. Sparse approximate solutions linear systems. SIAM Journal on Comput- ing. 24(2):227 233,, 1995.
Emre O. Neftei, Hesham Mostafa, and Friedemann Zenke Surrogate gradient learning in spiking neural networks: Bringing the power of gradient- optimization to spiking neural networks. IEEE Signal Processing Magazine, 36(6):51-6) 2019.
Thanh Nguyen, Charles Soussen, Jérôome Idier, and El-Hadi Djermoune. NP-hardness of EO mini- mization problems: revision and extension to the non-negative setting. In 2019 13th International conference on Sampling Theory and Applications (SampTA), pp. 1-4, 2019.
Ricardo Otazo, Emmanuel Candès, and Daniel K. Sodickson. Low-rank plus sparse matrix decom- position for accelerated dynamic MRI with separation of background and dynamic components. Magnetic Resonance in Medicine, 73(3): 1 125-1136. 2015.
Yu Qi, Jiangrong Shen, Yueming Wang. Huajin Tang, Hang Yu. Zhaohui Wu, and Gang Pan. Jointly learning network connections and link weights in spiking neural networks. In Proceedings of the -SeeenhSeventh International Joint Conference on Artificial Intelligence, LJCAI-18, pp. 1597 1603, 2018.
John Rachwan, Daniel Zügner, Bertrand Charpentier, Simon Geisler, Morgane Ayle, and Stephan Günnemann. Winning the lottery ahead of time: Efficient early network pruning. In Proceedings of the 39th International Conference on Machine Learning, pp. 18293-18309, 2022.
Pedro Savarese, Hugo Silva, and Michael Maire. Winning the lottery with continuous sparsification. In Advances in Neural Information Processing Systems, pp. 1 11380-11900, 2020.
Jonathan Schwarz, Siddhant Jayakumar, Razvan Pascanu, Peter E Latham, and Yee Teh. Power- propagation: A sparsity inducing weight reparameterisation. In Advances in Neural Information Processing Systems, pp. 28889-28903, 2021.
13

---

Published as a conference paper at ICLR 2023
Sidak Pal Singh and Dan Alistarh. Woodfisher: Efficient second-order approximation for neural network compression. In Advances in Neural Information Processing Systems, pp. 15098-18109, 2020.
Hidenori Tanaka, Daniel Kunin, Daniel Yamins, and Surya Ganguli. Pruning neural networks without any data by iteratively conserving synaptic flow. In Advances in Neural Information Processing Systems, pp. 6377-6389, 2020.
Robert Tibshirani. Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society: Series B (Methodological), 580 ):267-288, 1996.
Chaoqi Wang, Guodong Zhang, and Roger Grosse. Picking winning tickets before training by preserving gradient flow. In International Conference on Learning Representations, 2020.
Xiaodong Wang, Xianxian Zeng, Yun Zhang, Dong Li, and Weijun Yang. Learning soft threshold for sparse reparameterization using gradual projection operators. Neurocomputing, 198:381-3900 2022.
Wei Wen, Chunpeng Wu, Yandan Wang, Yiran Chen, and Hai Li. Learning structured sparsity in deep neural networks. In Advances Neural Information Processing Systems, pp. 2082-2090, 2016.
Paul Werbos. Backpropagation through time: what does and how do it. Proceedings of the IEEE, 18510):1500-156, 1990.
Stephen Wright, Robert D. Nowak, and MÁrio A. T. Figueiredo. Sparse reconstruction by sepa- rable approximation. IEEE Transactions on Signal Processing, 5777)2479-249499 2009.
Yujie Wu, Lei Deng, Guoqi Li. Jun Zhu, and Luping Shi. Spatio-temporal backpropagation for training nigh-performance spiking neural networks. Frontiers in Neuroscience, 12:331, 2018.
Lin Xiao and Tong Zhang. A proximal-gradient homotopy method for the 11-regularized least- squares problem. In Proceedings of the 29th International Conference on Machine Learning (1C pp. 839-846, 2012.
Lin Xiao and Tong Zhang. A proximal-gradient homotopy method for the sparse least-squares problem. SIAM Journal on Optimization, 2362):002-1091, 2013.
Zeke Xie, Issei Sato, and Masashi Sugiyama. A diffusion theory for deep learning dynamics: Stochastic gradient descent exponentially favors flat minima. In International Conference on Learning Representations, 2021.
Hang Yin, John Boaz Lee, Xiangnan Kong. Thomas Hartvigsen, and Sihong Xie. Energy-efficien models for high- sigh-dimensional spike train classification using sparse spiking neural networks. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery Data Mining, pp. 2017-2025, 2021.
Haoran You, Chaojian Li, Pengfei Xu, Yonggan Fu, Yue Wang, Xiaohan Chen, Richard G. Baraniuk, Zhangyang Wang, and Yingyan Lin. Drawing early-bird tickets: Toward more efficient training of deep networks. In International Conference on Learning Representations, 2020.
Dejiao Zhang, Haozhu Wang, Mario Figueiredo, and Laura Balzano. Learning share: Simultane- ous parameter tying and sparsification deep learning, In International Conference on Learning Representations, 2018
Yuxin Zhang, Mingbao Lin, Mengzhao Chen, Zihan Xu, Fei Chao, Yunhan Shen, Ke Li, Yongjian Wu, and Rongrong Ji. Optimizing gradient-dris driven criteria in network sparsity: Gradient is all you need. arXiv preprint arXiv:2201. 2022.
Xiao Zhou, Weizhong Zhang. Hang Xu, and Tong Zhang. Effective sparsification of neural networks with global sparsity constraint. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. -608, 2021.
14

---

Published as a conference paper at ICLR 2023
Michael Zhu and Suyog Gupta. To prune, not to prune: exploring the efficacy of pruning for model compression. arXiv preprint arXiv:1710.01878, 2017.
Tao Zhuang. Zhixuan Zhang, Yuheng Huang, Xiaoyi Zeng. Kai Shuang, and Xiang Li. Neuron-level structured pruning using polarization regularizer. Advances in Neural Information Processing Systems, pp. 9865-9877, 2020.
A PROOF OF THEOREMS AND LEMMAS
For clarity, we restate the theorem or lemma in the main text again here. Lemma (Local update rule). The update rule below
((++1) + g(t) - V (C(w o Vo(w(( )d(t)) w (++1) t S S ( (0(++))) imply the following update of any nonzero component w00. in actual weight
(17)
(18)
((++1) -gt+t (w 17 V. V. (c(w(()))
(19)
when [0(++1) > d(t) and the sign condition sign (0(++1)) sign are met.
Proof. For any nonzero weight w(t) + 0, g(t) = w(t)) - (t)) sign(w!!!), using Eq. 17 we have g(t+1) = (t) + (tt) sign(w!!) 17 (f(w))))
(20)
Let (1(+1)
(t()) 17 (1) V. (L(w(t))) be the target point of vanilla SGD without regularization.
Recall Eq. 20 we have
(++1) 0(++1) - d sign(w("))
sigu(0(++))))((++)) sign( )a(())
(21)
(ig (og
- d(()),
which has the same sign as ((++1), Now we have (ign(w(((+1)) sign(0((+1)) = sign(O(0) = sign(w((t) sign(w((+1)
To evaluate the updated weight, by Eq. 18 Eq.
20
Eq.
21 and the definition of soft threshold map-
ping, we derive
(++1) sign((((+1)) max ( (0) + (1)) (0) - (((u)))() d(++1).00 .0+
(ign( max{ (e+1) (0)) sigu( - ( 00}
sign((((+1)) maxf sign (1( ) (1w()11) + dd ((+1)) 0}
(22)
sign((((+1) max (1(()) 11| + (()) - d((+1) .0]
(ign( max max(/e((+1) (++1) (d(c+1) (d(c+1)- )
Sput V (C(w(())))
B ORIGINAL. THRESHOLD SCHEDULER IN STDS
In STDS, the authors propose the Sine scheduler d(+)) (1+sim(((++))))0 With the form of simplified threshold scheduler in Eq. 13 by Theorem [ the corresponding penalty
D.
15

---

Published as a conference paper at ICLR 2023
( has the form
10 d(t+1) d(e))
17(1)
D cos (p) cos( [++1)
Mmxx (1 + COs
t1 T
2D sin
sin Lx + 2T
(23)
Mmax
008
tat
T
tx sin
2 C
COs T
tm 2T
C tan(
function of training progress t/T' with constant C. Investigate the function tan /2) on interval (0, T). we have μ(x) is increasing from 0 to +00, which implies p can be sufficiently large during training. The loss is thus insignificant compared the regularization term the last stage of training and leads to performance degradation with respect to S-LATS.
C DISCUSSION ABOUT TRAINING THRESHOLD
In the main text, we propose a framework explaining the style of growth in threshold as an ever- changing optimization problem. However, only manually designed threshold schedulers. We make a discussion here and show training threshold is not as easy as STR or GPO did. We will show GPO and STR share the same discussion of training threshold since GPO will degenerate to STR in few training epochs. Moreover, we suggest not simply setting threshold trainable if one really wants to investigate optimization of threshold.
C.I La PENALTY DOMINATES EARLY TRAINING OF STR.
In the official codebase of STR, we notice the trainable sparse threshold is also together with weight decay (L2 regularization A|| (12) À is of magnitude around 10 10 The initial value Sait is usually set to negative number with large magnitude around - 100 ~ 10 for CNN trials. It is easy see the magnitude of La regularized term is around 0.01 1 in the early stage of training.
Take CNN for instance. For given loss function L and weight wy of the I-th layer, the gradient passed to threshold sı can be estimated
Ow! as
vs, ((un((s 01)) Vwn
Σ
VwL(wi) V (S))))))))
(24)
(w,0))((,,0))
(1 -1 ((t)))
Σ
VwL(wi) sign(0)
(w (uw)))(w;.cm 0:) w
wherein each term of sum has magnitude less than 1 ((w,) since sigmoid function (xx is bounded in 1). Given that the stochastic gradient noise across parameters in wi admits to Lévy distribution (Xie et al.] 2021). some negative and positive terms will balance each other, which makes wonder whether the regularization term dominates the training of threshold. Therefore, com- pare the gradient to L2 penalty by tracing the magnitude ratio of v.,C to A|st| during training of ResNet-50 on ImageNet, which is shown Fig. 4
It is evident that the La penalty plays the leading role in the early stage rather than the gradient. We even observe for several final layers, the penalty always dominates the training of update of thereby be rewritten to (t+1) s10 (1) X), where t7(1) is the learning rate the
threshold. The
16

---

Published as a conference paper at ICLR 2023
conv1 (layer 1)
1yer1122comm22 (layer 10)
layer2 2.eer2...omm conv3 (layer 21)
2
20
60
80
100
0
20
40
60
80
100
20
40
60
80
100
Epoch
Epoch
Epoch
layer 2. 2.corv1 (Jayer 32)
layer4.2.comv (layer 52)
(layer 54)
1.0
M
Epoch
Epoch
Esoch
Figure 4: Magnitude ratios of gradient to Iz-regllarization
in different layers when training
Asy
using STR at 90.23% overall sparsity. Data within an epoch are averaged using geometric mean.
I-th iteration. Regardless of gradient, at the beginning of training, STR can be viewed as special case of threshold scheduler as follows
(
a
Sinit
[ = 1,2....
(25)
1=0
By Theorem 1 the corresponding penalty can be derived by
(00) ((++)) d(t)
r)(1)
(s((+1)) =s((t)
17())
a ((()(((((11) x(())
(26)
17(0)
(1)))(()))))))
17(1)
(3()))(1
where lies between (1+1) and sl0 The existence of is shown by Lagrange's Mean Value Theorem. It rather difficult to analytically give the explicit expression of ((() since it relies on the behavior of 8, which has dynamic decay rate - (1) X.
Loosely speaking, we adopt the approximation 3 xe g()), We still cannot show the explicit form of y(+),, but now we can investigate the trend of penalty by analyzing function μ(x) -o' -x(222, which is increasing in (-oo, 7). Here = - -1.4434 is the unique negative root of e ex - -x ++=1=1 0.
In the early stage, 8 a negative number with a much larger magnitude than 7. Based on the above, is increasing in the early stage.
C.2 GPO: FALL. BACK STR SHORTLY
concludes STR is majorly influenced by Le decay on s. We will see that GPO has a similar behavior on & and GPO will degenerate STR shortly after the training begins.
In GPO. the authors introduces another trainable parameter 3 the reparameterization w = (1 ())) k0 with k 10-6[3]. GPO starts from the identity mapping 0 by initializing
17

---

Published as a conference paper at ICLR 2023
10 The gradient passed to 3t in the 1-th layer is given by
(ww
((m 0())
Vm, f(ur)),
05,
(27)
10 sign(
E
V V Vw-((w)) (0 Sa(0))
(w,0)E(w,.0)
{0.
d
Notice that 0 Sa(0) (d sign(()),
has magnitude not greater than d, which derives that
the gradient ((wi) are added with coefficient whose magnitude is below 10 10-6d.
From the codebase of GPO, confirm the weight decay B is 10-4, Since the initial 8 is 10 the L2 regularized term is of magnitude around 10 10 100 at the beginning of training. Recall that d os) < 1, it is obvious that the gradient pass to 3 has much smaller magnitude than the L2 regularization. For this reason, the gradient can be ignored for Furthermore, will shrink exponentially to almost zero and the mapping in GPO will fall back to STR within few epochs, which can be seen by w (1 (0)) k9 N Sa(O).
C.3 FOCUS ON THRESHOLD SCHEDULER INSTEAD THE FINAL THRESHOLD.
Conventional wisdom suggests that when parameter is trainable, it will be optimized automat- ically. This idea should only work for those directly determining the performance, e.g., weights in a dense network. For pruning, authors of STDS find the differences in positions of performance versus sparsity curve should be ascribed to the evolving patterns of the threshold. To verify this, we replace the threshold training mechanism STR and default scheduler in STDS by several schedulers with the same final threshold D = 10 3 (D 0.5 for STDS) shown below
Sine: d(t)= } [1 + sin( ( B)
Linear: d(() D
. Log2: d(t) = logg f + 1)D
We conduct the training ANN ResNet-50 for both methods, the results are shown in Tab. It is
Table Comparison of threshold schedulers when applied to ResNet-50 on ImageNet.
STR (D 10-3)
STDS (D 0.5)
Scheduler
Sparsity (%)
Top-1 Acc. (%)
Sparsity (%)
Top- Acc. (%)
Sine
94.
71.51
95.72
72.42
Linear
95.41
69.85
98.46
59.79
Log2
95.95
68.55
98.05
64.74
obvious that even though the final thresholds are set equally, the accuracy and overall sparsity vary with the scheduler. In fact, simply setting threshold trainable indicates one only cares about whether the final threshold is optimal, which turns out to be tangential issue in soft threshold pruning.
The above results suggest the correct manner to manipulate threshold is pursuing well-performed scheduler. Even if one studies the learning of threshold, it should be concentrated on the optimiza- tion of threshold scheduler, which may require tools in discrete-time optimal control. Due to the complex coupling between constantly changing threshold and final performance, discussion based on discrete-time optimal control is beyond the scope of this work.
D DETAILED DERIVATION FOR LATS AND S-LATS
In this part, we provide detailed derivations for LATS and S-LATS, which covers Eq. 12 and Eq.
13
18

---

Published as a conference paper at ICLR 2023
D.1 LATS FOR COSINE ANNEALING SCHEDULER
In most of the deep learning applications, the training process includes the schedule of the learning rate, which also known as learning rate scheduler. Generally, learning rate updated at the end of an epoch. Assuming there are N training epochs in total, each of which includes B training mini-batches. The learning rate scheduler is defined as 17(*,)) = h(n/N), which evaluates the learning rate at the b-th mini-batch the n-th epoch. Here n = O, I,..., - denotes the current id of the training epoch, and = 1, 2, .... B denotes the batch id in n-th epoch. We denote h: [0, 1] > R+ as the scheduler function for the learning rate. For cosine annealing scheduler with Muiin O, we have
h(x)) /max (1 + cos(TTr))
(28)
In Corollary the threshold scheduler for LATS is obtained by - + > 10, where ( the threshold after mini-batches from the beginning. Under the learning rate scheduler described in Eq. 28 the threshold ((n,)) is shown by accumulating all previous learning rates, which gives
=
n
d(mbb = (0000) + b- ((-h(n/N) + Σ h(1/N)
(29)
For cosine annealing learning rate scheduler, we have
d(n,b) (0000) + pJmax
b(1 cos
RVT
+ B
Σ
(1
+
COS
2n
2
N
N
k=0
(30)
(0,0
p.mmax
b(1
Bn
B
iT
N
N
k=
To evaluate the sum of cos(iv /N), we give the following results
n-1
in
iT
COS
COS
sin
N
sin
N
2N
2N
i
1
sin
+
π
sin
sin
N
2N
N
2N
2N
1=0
2i + 1
Σ
(2i - 1
sim
sin
sin
2N
2N
2N
1=0
(31)
2i 1
2i
sim
sin
sin 2N
2N
2N
2n
sin
+ sin
sin 2N
2N
2.N
sin 12m11 2N
sin 2N
Recall Eq. 30
we have
((.b)) ((0.0) Mmmax
b(1 + cos
72T N
Bn + B
1
+
sin
2n-
2N
2
2
2 sin EN
(32)
b
B
sin
2n-1 2N
= (0.00) jammax
(1
+
2n +
N
2N
When ((00)) 0, this gives LATS in the form of Eq.
12
19

---

Published as a conference paper at ICLR 2023
D.2 THE DETAILED MOTIVATION OF S-LATS
The implementation of LATS is rather complicated and requires meticulous coding. Worse still, for some learning rate schedulers, e.g, polynomial decay scheduler (Liu al. 2013 Chen et al. 2018)
Ymmax (1-4
(33)
where Æ > 0 a constant 0.9 in aforementioned studies), the form of LATS cannot be reduced like cosine annealing scheduler in most cases. To see this, we write the corresponding LATS
( + pM7maxx
Σ 1=0
1 8
(34)
Note that 1 - f makes up an arithmetic progression, and the threshold is the sum of their powers. Simplifying the sums of powers of arithmetic progression requires the so-called Bernoulli number (Jacobi 1834 Knuth 1993) when x: is integer. For a general x >> O, the expression of Eq. 34 includes Σk= kP, which is further based the Hurwitz zeta function (Coffey) 2008). In such case, we cannot analytically compute LATS, which forces us to do the summation in Eq. 34 It thus brings about the accumulative error. In brief, cannot expect each learning rate scheduler corresponds to an analytical and simple form of LATS.
the generalized harmonic numbers H²
To handle this, we turn to an approximation rather than precisely evaluating d(t), Returning to Eq. 29 with (0000) ignored, we split the right hand side into two terms E h(t/N) and b h(n/N).
The first term could be viewed as left Riemann sum in two steps 1) interpolate points 1/N, 2/N,..., (n 1)/N, n/N with constant spacing 1/N, the width of the rectangles 2) eval- uate the sum of rectangle areas with height h(i/N). leads to the approximation of the Riemann integral
MON B Σ h(/NN) = BN Σ M(1/N) BN P. h(x)dr.
(35)
The second term match b/ B part of a residual tiny rectangle
b M(//N)
b h(n/N) BN
B B
(36)
To sum up, Eq. 35 and Eq.
36 together make numerical approximation of integral
BN fo Co/N+b/ Bb/BB h(x)dd,, which is shown schematically in Fig. Note that we could write training progress as t/T now, where = Bn + b is the current iteration and T BN is the total training iterations.
So far, we successfully replace the summation with integration. However, the current final threshold is BN f h(z)dr, which is different from the real one d' (I,B)) To keep the final threshold d(T)
d (, we normalize the integral as follows
e N(N-L.B) L h(2)dr
(37)
So Mx)dxx
For most learning rate scheduler functions, integration is much easier than summation. S-LATS enables us to apply our pruning method on wider varieties of deep learning applications.
E PRUNING EXPERIMENTS ON MOBILENET-V VI USING S-LATS
Besides the ResNet-like structures mentioned in the main text, also conduct experiments MobileNet-V1 (Howard et al. 2017) to show the power of our proposed methods S-LATS on the lightweight network. To make fair comparison, choose those SOTA studies using the standard
20

---

Published as a conference paper at ICLR 2023
1.0
0.8
Learning Rale Scheduler h(-)
0.6
0.4
0.2
0.0
N
Training Progress
Figure 5: Explanation on numerical approximation of integral fn fN/N+b/BN hxx)dr. Cosine anneal- ing learning rate scheduler is exemplified in darker blue curve. The area of n cyan rectangles corresponds to Eq. 335 The area of the tiny yellow rectangle Eq. 36
training setting, i.e., batch size of 256 and 100 training epochs. They include STR (Kusupati et al.
2020), gradual pruning WoodFisher (Singh & Alistarh] 2020), and a modern implementation (Gale et al., (019) of Gradual Magnitude Pruning (GMP) (Zhu & Gupta [017]. Note that we do not compare S-LATS to OptG, since OptG adopts 1.8x (180) training epochs, or the comparison would be unfair.
The results are shown in Tab. 6 Apparently, our proposed method towers over previous sparsify- during-training work. We elaborate on the sparsity budgets and the corresponding final thresholds in Tab. 12 of Appendix [ The hyperparameters for training MobileNet-V are stated in Tab, of
Appendix H
Table 6: Performance comparison of MobileNet-Vi ImageNet using standard training setting (256 batch size, 100 epochs). The results of GMP are gleaned from the manuscripts of STR and OptG. The accuracy of our method is averaged over three trials.
Method
Top-1 Acc. (%)
Sparsity (%)
Dense
71.95
0
GMP
67.70
74.11
STR
68.35
75.28
STR
66.52
79.07
WoodFisher
70.09
75.28
S-LATS
68.25
81.84
GMP
61.80
89.03
STR
64.83
85.80
STR
62.10
89.01
STR
61.51
89.62
WoodFisher
63.87
89.00
S-LATS
66.73+0.08
85,87
S-LATS
65.63+0.20
88.22
S-LATS
64.93
89.08
21

---

Published as a conference paper at ICLR 2023
F ABLATION STUDY OF THRESHOLD SCHEDULER ON RESNET-50
F.1 REMOVE L2 DECAY FOR FAIR COMPARISON
To evaluate the performance gains brought by the threshold scheduler alone, the weight decay or L2 penally must be removed. To explain this, recall line Algorithm [ we know the update rule of hidden weight 0 is affected by both gradient and La penalty X| However, the analysis in Lemma Tis based on vanilla SGD without weight decay. To account for this inconsistency, let's first investigate the influence of weight decay the equivalent optimization problem.
17 has an additional penalty term In the presence of weight decay, the update rule described by Eq. as follows
((+1) <- (()) 11 V w (C(un Vo(w dd))) - 10)000000)))
(38)
Following derivation in Eq. 20 we have
@((+1) (t)) + ( sign(w(() (1) - vn Vo (f(w(()) T0 )g
(39)
Similarly, we denote ((++1) : w ny V (C((((())) to be the target point vanilla SGD
without regularization. With Eq. 39 have
((++1) = 0(++1) - (()) sign(w )n 20(1)
sign(0(+2))) (((+1))| ( sign(0(0)d - siiig((()))
(40)
sign(o((+1))) | (10(+)))))()1111)1 sign(
sign( sign (0(1))
d(() 1110(0)),
If g((+1) still satisfies the local relation that sign(w(*+1)) sign(o((+1)) sign(A())
sign(w(")) sign( (t))))),,,,, mimicking Eq. 22 have
w (++1) ( xpoces (w ( V (C(w
(41)
Apparently, the L2 penalty of 0 lies into the equivalent J, penalty term of ur in the ISTA rule, making the analysis of the corresponding threshold scheduler intractable. Accordingly, decide to remove weight decay, i.e., and set to zero in the ablation study to prevent an unpredictable threshold scheduler.
F.2 SINE SCHEDULER IN STDS VS S-LATS
After removing weight decay, we rerun the original STDS and our methods under several sparsity levels (through changing D) while keeping the other hyperparameters and the batch size of 256. As illustrated in Fig. 6 the results on ResNet-50 show our method clearly surpasses the original STDS on the ImageNet dataset. theoretical analysis is enclosed Appendix B
72
Acc. (%)
68
STDS S-LATS
66
STDS (Sine sch.)
94
95
96
97
98
Sparsity (%)
Figure 6: Performance comparison of original STDS and our method.
22

---

Published as a conference paper at ICLR 2023
G SPARSITY VS FIRING RATE IN SNNS
G.1 AN OVERVIEW OF SNNS
Spiking neural networks (SNNs) are honored as the third generation of neural network models (Maas 1997). derived from biological neural network modeling. SNNs are composed of spik- ing neurons, which release spikes in binary form, and connections between neurons. The model of spiking neurons is dynamical system described by one or more ordinary differential equations (ODE) and firing threshold. The dynamical system is also called "subthreshold dynamics" in the context of computational neuroscience. A spike is generated and passed to all postsynaptic spiking neurons when the variable representing membrane potential exceeds the firing threshold. Today, the most commonly used neuron model is the Leaky Integrate-anc and-Fire (LIF) model. Specifically, LIF has the subthreshold dynamic as follows
du(t) (u(1) Urest) Tm dt
Σ Iw,,
(42)
where u(t) is the membrane potential at time t, Urest is the resting potential, Tm is the membrane constant, / and w denote input spikes and input weights respectively. The firing behavior of LIF neurons is depicted an instantaneous jump of membrane potential shown below
lim
uat! + At) Mesas if utt) 2 Uhh.
(43)
where Ven 1f are the firing threshold and firing time respectively.
The ODE in Eq 42 can be discretized via the Euler method and transformed into an RNN-like itera- tive computing manner as follows
u [ = ul( 1] 1 (-(ult- - 1] -rrest + Σ, w:/.(ll).
Ti
(44)
# a|] = H(ul( wnh),
u[t] = s|t] fest + (1 -s stt))u[t"]
where w[["], u[(]] are the membrane potential before and after firing at timestep & respectively, (-) the Heaviside step function modeling jump behavior when a spike is triggered. However, training techniques in RNN, such as backpropagation through time (BPTT) (Werbos) 1990) cannot be di- rectly applied to SNNs for the spiking behavior described by Heaviside is non differentiable. Thanks to the surrogate gradient method proposed in Wu et al. (2018); Neftci et al. (2019). researchers can now incorporate BPTT into training of SNNs by switching to "differentiable mode" of the Heav- iside step function when computing gradient. It refers to replacing Heaviside step with differen- tiable surrogate function. Surrogate gradient resembles the straight-throut estimator (Bengio et al.) 2013) closely in both computing style and ideology.
G.2 REDUCING SNNS COST ON NEUROMORPHIC HARDWARE
SNNs are considered energy efficient when deployed on a series of dedicated hardware, also known as neuromorphic hardware event-driven hardware. On these chips, the computation is triggered only when there are incoming spikes and weights are nonzero (Merolla et al. 2014).. For this reason, there are three mainstream methods for alleviating the energy cost of a given SNN on neuromorphic chips including 1) unstructured pruning of weights, 2) reducing the number of spikes, and 3) search- ing for efficient SNN structures. The NAS studies on SNNs (Na et al., 2022 Kim et al., (022a) are not based on existing SNN structure, so we omit the discussion of NAS methods.
Many recent studies have made ample signs of progress on pruning and reducing spike counts. They confirm there is weak correlation between the number of spikes and weight sparsity (Deng et al.) 2021 Chen et al.) 2022 Kim et al. 2022b). We also evaluate the spike counts pruned SNNs. Compared to the number of spikes, more frequently used metric is average firing rate, which is obtained by averaging the number of spikes across timesteps and spiking neurons. We collect the average firing rate of each trial during inference using pruned SEW ResNet-18. 18. We further provide a plot of the average firing rate against the sparsity, which is shown in Fig. 7 The weak relationship between the number of spikes and weight sparsity is manifested in the slightly decreased average
23

---

Published as a conference paper at ICLR 2023
0.1400
0.1375
Avg. Hinng Hate
0.1350
0.1325
0.1300
0.1275
0
20
40
60
80
Sparsity (%)
Figure 7: The trend of average firing rate against sparsity.
firing rate. Despite a downward trend in the average firing rate, the relative magnitude of the decline trifling. It is consistent with previous observations and suggests pruning is inefficient means of reducing the number of spikes in SNNs.
In conclusion, pruning is an efficient way to induce weight sparsity and lower cost. However, we should not expect the suppression of firing rates as a bonus.
H TRAINING HYPERPARAMETERS
We make the detailed setting in our experiments clear in Tab. 7 Tab. 8 and Tab. g
Table ANN ResNet-50 hyperparameters.
Description
Notation
Value
# Epoch
100
Optimizer
Momentum SGD (momentum 0.875)
Overall batch size
256
1024
Max. learning rate
7max
0.256
0.512
Learning rate scheduler
Cosine annealing
Warmup epochs
5
Label smoothing
0.1
Weight decay
X
3.05e-5 (0 for ablation study in Appendix
Prune BN layers?
No
Prune first and last layers?
Yes
Table 8: ANN MobileNet-VI hyperparameters.
Description
Notation
Value
# Epoch
100
Optimizer
Momentum SGD (momentum = 0.875)
Overall batch size
256
Max. learning rate
Tmax
0.256
Learning scheduler
Cosine annealing 5
Warmup epochs
Label smoothing
0.1
Weight decay
A
3.05e-5
Prune BN layers?
No
Prune first and last layers?
Yes
24

---

Published as a conference paper at ICLR 2023
Table 9: SNN SEW ResNet-18 hyperparameters.
Description
Notation
Value
# Epoch
320
Optimizer
Momentum SGD (momentum = 0.9)
Overall batch size
2566
Max. learning rate
Tmna
0.1
Learning rate scheduler
Cosine annealing
Weight decay
X
0
Prune BN layers?
No
Prune first and last layers?
Yes
Simulation timesteps
4
SEW function
ADD
I SPARSITY BUDGETS
Table 10: Sparsity budgets of ResNet-50 using STDS + S-LATS on ImageNet (256 batch size).
Linal threshold
0.2
0.7
1.0
1.4
1.5
2.0
3,0
3,5
5.0
Acc.
76.75
75.52
13,00
72.04
71.47
67.47
67.012
64.20
60.35
55,85
56.79
53.64
50.41
Layeris)
Sparsit (9%)
Ovenil
19.05
89:57
92.99
94.60
95.12
95.53
95.13
96.58
96.94
97.43
05.01
98:11
98:48
98.59
99.02
99.13
09
99.39
59.57
67.00
69.70
70.89
78.37
83.04
85.55
layarl Oeomd
74.00
75.44
76.03
83.25
84.00
83.84
585 06
91.33
89:73
91.77
layerl 2
91.38
02.01
05.18
03.06
04.96
05:07
05.87
97.08
97.96
98.44
98.51
layert
97:19
18.13
98.12
layort Didmessmppe.
91.72
94.32
95.99
98.45
98.84
Loorl.Lcomv2
85.31
94.75
95.32
95.36
97.18
57.55
95.02
99.20
99.64
layert.1.cmm 1.om5
09.48
79.34
86.22
35.1
91.01
92.60
54.74
95.3
97.07
98.
U5 06.06
98.95
59.72
layerl 1gmm
64.66
75.52
82.27
86.07
$7.00
87.35
89.77
55.64
01.05
03.55
94.15
05.00
96.27
97.03
97.97
97.82
98.76
98.91
layerl 2g0m2
63.04
76.05
84.24
86.97
58.20
99.35
90.86
90.05
01.01
53.00
94.51
94.34
96.93
97.15
95.01
97.94
98.56
99.25
layert 3.0m3
68.90
80.36
86.72
87.87
99.11
50.09
91.77
92.69
92.75
54.42
96.64
95.80
97.84
96.26
99.04
98.71
99 19.18
99.35
baye2 Decmr
59.82
73.85
8011
82:21
8113
86.
69.43
59.1
58.64
91 19
91 51
92: 74
98.22
06.01
97.5 m
97 22:26
07 99
99.45
layer2 2.cmm
72.00
85.04
85.93
62:21
92.57
91.1 19
94.54
94.90
$4.89
95.74
03
93.01
93. 02:50
08.04
99-27
98
66.61
99. 16
71.13
82.92
85.95
39:32
90.37
81.58
82.03
95.15
90.37
94.38
96.03
96.03
96.76
50.78
95.00
97.85
98.12
98.72
layer2 Doowassddos
$1.42
90.00
92.85
94.0%
94.78
80.23
85.66
96.25
96.44
97.40
98.0
98.25
08.86
99:03
99.05
59 10
99.31
layer2 Leoov]
K3.46
00:47
91.65
50.06
90.00
95.01
96,73 75
56.10 75
91.00
98.61
98. 29
99
99:21
99.02
59:27
99.26
2
90.41
96.27
07.17
96.90
07.76
98.31
99:02
99
99:46
99.61
99.56
1eyor2.L.om3
73.53
91.63
93.93
93.80
95.03
96.00
96.79
97.37
98
29
26.66
99.04
99.25
99.04
byjor2
87.51
97
98.84
3
90.11
05.32
97.55
98.82
09.04
lajer2 2oomr
90.06
91.10
94.14
96.0
96.3
47.47
47.94
98.05
98.45
98.79
laper2 2.om2
83.83
91.28
9150
01.06
97.00
96.67
97.29
91.94
95.25
98.75
98.92
98.1
laye22 3
72.81
65.00
89/01
91.17
02.16
07.10
01.00
37
05.35
95.35
96.33
61.00
97 07:90
08.27
95.75
99.03
19
9906
layer3 1
61.26
74.70
80.15
83.80
$4.1
85.51
87.13
35.01
59.51
50.43
91.82
92.57
93.79
95.04
95.56
96.42
96.69
97.29
layer3 2200m
$2.41
91.42
94.51
95.78
56.15
96.09
95.94
97.27
97.56
97.94
98.33
98.33
99.65
98.87
99.04
99.12
99.25
99.35
3
71.21
62.15
86.3 33
88.76
80.16
80.01
91.43
02.04
02.46
01.73
04.97
05.40
08.3
97.28
07.5 50
97.05
06
09.11
layer3 C.dweaaglleee
$6.21
03.20
05.30
96.83
0606
90.29
07.77
02.08
08.15
06.53
05.63
98.96
09.24
09.39
99:64
99.56
09 60
99.60
layer3 Leans
56.76
92.91
05.17
96.46
96.48
97.00
07.25
97.80
97.95
06.37
05.67
98.82
99.16
99.38
99:41
99:35
09.50 06.56
99.63
layer3 2omv2
66.67
91.08
91
96.79
9009
97.51
97.07
96.99
56.12
98.53
91
98.99
99.3 11:11
99.51
99.51
99.05
59.67
19:12
byyrr coms
06108
91.4
10
94.53
46
86 12
96-15
91.5 81
87.03
98 46
95.00 10.04
99
09
1.0m
90.52
95.09
07.01
96.80
97.13
97.58
95.36
98.25
98.65
99.00
99.32
99.27
99.55
96.22
97.25
97.26
97.35
00.93
98.09
98.92
99.02
99.20
3
01.99
05.21
90.24
07.61
57.05
98.25
99.14
94.60
99.05
99.53
95.39
97.70
98.97
99.12
99.49
99.55
99.62
35mm
96.82
67.45
91.13
95.10
03.12
01.00
05.05
96
07 10
98.3
96
06.66
99.06
99.26
99.35
99.45
59.53
77.62
67.58
91.12
92.47
01.41
04.15
94.67
96 03
06
06.44
95.
45.55
96.
98.85
94.84
99.19
99
99.17
2
63.13
91.67
94.67
95.63
96.17
96.66
07.30
07.48
0757
08
08 61
96.71
99.1
99.42
99.53
99.63
09.75
99.72
layer3 3
76.09
87.14
91.38
92.55
53.79
$1.55
95.29
95.91
56.41
50.00
93.90
99.10
98.52
98.14
99.3
99.23
99.62
99.62
comm
73.48
85.83
89.60
91.89
92.45
03.36
91.35
94.65
58.2
95.33
96.09
ST.02
97.61
98.39
95.88
98.85
99.15
99.25
layer 3cmm2
61. 11.39
91.61
55.30
55.55
26.25
95.02
97.39
57 56
99. 00
98.99
991 10
99.35
99.53
99.50
59.61
layer3
73.98
85.35
89:25
91.80
92.53
93.13
94.12
95.16
95.16
93.98
96.99
97.31
98.06
58
99.14
99.21
99.25
99.44
64.48
77.49
82.89
85.92
$7.00
$7.53
89.26
90.33
91.34
92.09
03.05
93.85
96.79
85.53
95.25
96.65
97.19
97.40
98.07
96.17
54
96.4
45
66
96.33
06/05
09
99
99.
99.
99.44
09
59.55
99.57
92.21
95.09
05.75
94.53
05
96
96
75
97.61
97.78
98
08
94.97
96.30
96.77
07.13
07.42
07.63
07:02
05.33
98.41
98
70
09.20
92.13
09.20
layerl Lomel
96.0
97A4
96.00
58.59
99.35
layerl. 20m2
97.00
97.90
95.43
96.63
97.29
98.64
98.85
74.54
92.98
65.26
98.63
97.13
97.94
98.35
98.54
95.95
98.03
99.21
99.29
220m2
79.66
97.04
97.38
67.81
6800
94.36
08.56
98.7
98
99.01
99.25
99.29
99.35
99.49
99.00
layori 3
66.98
70.13
84.53
87.56
16.74
80.3
62.60
01.60
02.61
01.53
05
65.41
96.41
07.57
07.79
98.05
98.51
98
56.44
94.47
96.60
97.56
97.77
98.10
98.44
98
06.70
99.04
99.25
99.27
99.40
99.49
99.54
99.55
99.57
99.60
25

---

Published as a conference paper at ICLR 2023
Table 11: Sparsity budgets of ResNet--S 50 using STDS + S-LATS on ImageNet (1024 batch size).
Finul Uhreshldd I
0.1
0.2
0.23
0.25
0.3
0.4
0.475
0.5
0.6
0.73
0.8
1.0
1.13
1.5
2.0
5.0
3.5
4.0
Top-l Auc, (98)
76.11
76.15
75.97
75.88
75.58
74.00
74.61
74.00
73.68
72.54
72.73
71.67
70.68
69:30
67.16
63.81
61
60:40
Layer(s)
Sporsity (99)
Ovenil
79.00
83.81
92.34
95.95
96.53
96.78
97.30
97.54
98.00
95.58
98.79
98.95
99.04
sutel
40.67
46.00
55.02
53.64
56.36
62.73
65.07
64.45
71.75
70:07
72.
74.04
76.87
77.51
70.50
83.93
$4.56
67.38
layeri Rcomr]
52.32
58.42
65.
64.92
65.36
72.00
74.71
74.07
79.30
79.30
91.84
32.93
$5.82
K3.20
57.
90.33
91.04
59,997
layeri Deom2
75. 23.36
K2.51
85.45
85.65
87.70
89.99
91.06
91.87
92.69
91.00
94.54
58.89
95.41
95.79
95.87
97:73
58.00
59. 10
layeri Deoms
90.77
77:27
81.99 89
K3.85
06.08
$5.84
89:89
50.96
01.83
01:23
94.27
96.
95.66
06.89
97.55
97.92
59
layert Cdowesamglle
64.32
71.25
74.69
75.47
78.57
$2.09
85.73
83.44
$6.99
67.38
88.5 22.92
$9.49
90.45
92.49
93.05
94.65
94.98
93.21
layert Looor
14.98
79.30
82.61
82.99
$5.29
$6.96
87.43
$9.42
50-72
91.63
02.4 41
94.35
93.60
95.77
95.26
97.35
97.49
98.77
layerl 2
19.04
02.69
00.72
94.11
95
96:
97.5
98.41
layerl Lemm3
77.44
90.27
01.71
98:14
93.52
94.73
96: 23-
96.
03.05
layerl 2ems
79.77
87.58
$9.15
90.01
92.96
93.51
04.43
98-02
97.99
layerl
35.74
91.18
91.73
98:01
layert
42
layer2 220m22
88.13
95.92
96.96
98.52
98.50
layer2 0coms
75.94
84.57
8558
53.35
00.73
90.4
91.45
92.50
93.00
94.92
94.87
95.76
47.01
97.90
97.97
layer2 C.dwammllee
64.75
82:05
91.27
91.94
92.5
94384
03.10
05.20
05.70
06.67
06.3
07.33
97.45
97.99
95.41
98.67
08.03
98.5
layer2 Lo00 1
59.00 39.09
90.01
91.85
03.14
92.3
94.67
95.55
95.54
05.65
97.00
07.05
07.73
98.01
96.25
95.93
99.25
99
99.39
laye2 2
$7.41
89.54
93.13
93.65
92.44
95.04
05.99
96.01
96.07
97.35
07.4
98.36
96.26
96.57
99.17
99.27
99 69.65
99.65
byor2 2
61.23 11.75
67.44
86.0
88.00
$7.04
00-40
92.51
15
01.88
94.30
94:48
05.95
06.1
91
91.58 10.09
98.80
09 16
09. 10
byyr222.0 ccmm
10
65.06
86.77
88:33
80.76
01.55
91.45
92.95
03.09
99.55
94.63
95.11
96.11
06. 55
97.05
97.93
088 79
08
lager2.2.oms
79.02
57.50
88.76
39.76
3957
82.55
82.77
92.61
93.93
94.01
95.00 09.09
95.26
96.
97.34
47.50
98.37
99.17
98.96
71.18
K2.41
83.43
35.35
$700
89.28
80/05
90.3
91.40
92.80
93.58
93.93
93.80
93.97
96.96
97.95
58.82
99.57
layer2.3.om 3cmm1
71 19
81 11.09
84.35
35.60
$7.40
850
90.54
90.11
91.0 14
M1 10
94 29
90
99.18
13.13
97.81
5/.87
98
2
92.67
93.19
94.32
03.00
95.55
96.04
96-57
97.52
97.67
97.97
3
92.05
93.02
94.78
94.09
95.57
95.97
96.83
97.56
98.05
28.25
62.93
77.81
79.06
53.61
85.64
57.35
89-57
90.65
91.63
92.85
94.27
93.65
96.26
3
03.05
94.79
96.74
95.29
98.09
98.76
99.06
99.36
59.44
99.49
92.88
93.65
94.35
95.64
96.29
96.61
9700
90.33
97.94
98. 49
98.00
99.00
99.25
99.39
99.1
layer3.1 20mv2
15.96
93.44
93.99
94.22
96.1%
96.67
67.06
01.31
69.40
15
98.10 19
98.3
96.37
98.00
99.1
99 34
599
99
3
74.52
63.38
87.49
85.09
50.54
92.11
93.02
93.65
94.35
95.16
05.00
96.05
97.08
90.38
95.07
98.35
98.85
98.00
layer3 1
$4.36
91.82
93.20
93.33
93.53
95:07
95.61
96.23
96.26
50.09
97 00
97.45
98.08
98.13
95.50
99.02
99.16
99.13
$3.49
01.97
93.01
03.50
04.3
95.51
95.15
06.71
06.5
07:35
03. 50
97.67
08.11
06.21
95.75
99.15
09 20:20
00.32
3
75.10
83.78
87.87
$8.06
80.36
91.66
92.73
05.95
94.17
05.06
05.6
05.91
96.
06.30 06.89
95.07
93.41
98.71
93.53
1
$0.11
89.01
00.31
91.45
92.54
93:60
94.34
05.01
95.76
96.18
06.36
97.12
97.36
07.65
95.25
98.61
99.07
99.00
liyer3
K2.76
91:07
92.45
92.81
53:24
15
96.43
56.95
97.19
91 36
81.85
98 mm
08.5
99.29
99.23
59:37
99.15
byor331cmm
87.54
89.8
97.03
93.14
95.50
94.2
95.83
95 54
96 93
96 50
97.52
94.04
98 73
85.00
90,46
92.61
93.25
07
04.55 56
95 68
20
65.77 71
96 46
97.1
97.73
98.47
92.50
93.33
95.47
96.05
96.63
67.20
07.72
97.97
97.90
99.21
75.51
91.95
92.59
05.53
96.11
03.27
95.35
97.41
91.95
68.00
97.03
97.74
97.85
99.10
99.30
520m3
91.96
93.12
94.61
94.68
95.12
95.74
96.61
97.57
98.10
98.47
98.61
layord D
65.17
76.27
76776
8106
83.14
86.00
87.24
$7.75
50.04
00.13
90.94
01.85
92
91.47
94.53
95.65
56
19
64.07
93.20
94.53
95.35
96.59
01.09
98.20
96.55
98.51
98.68
98.7 75
98.92
98:97
99.09
99.21
99.5
49
90.45
72.08
63.90
85.07
26.04
$7.64
$9.50
90.95
91.05
92.18
93.23
93.76
94.72
95.02
95.95
95.67
97.51
97.81
98 98.10
$3.95
92.56
93.61
98.12
55.02
96.16
95.57
96.80
97.16
90.64
90.66
98.01
98.22
98.54
98.83
99.05
99.25
99.24
layer) Lomml
77.97
89:000
89:05
01
92.44
94.43
94.94
95.25
09.95
96.62
96.91
97.39
97
08.12
99.09
98.85
9901
99.1
layer& L
50.73
90.68
91.40
92.49
94003
95.76
95.24
96.56
97.11
97.62
07.88
98.30
98.47
06.53
99.02
99.32
99.38
99.48
71.47
85.15
86.25
87.73
39.75
82.46
92.96
93.54
94.38
95.09
05.65
96.33
96.71
97.38
07.58
93.12
98.25
98.56
1
09.25
83.56
80.10
3666
89.26
81.54
83.58
93.38
94.55
95.19
05.67
96.91
96.90
57.62
47.87
93.41
98
98.48
98.75
byorr 2omm2
80
91.75
94.72
14
30
97.00
70
98 46
98 47
00.85
98.0
99.27
09
70.58
80.52
82.77
86
85.80
74
90.33 50
91.41
91.00
93.85
93.07
05.32
99.19
97.41
61
96.77
97.0
95.54
98.05
96.2
99.6
99.30
99.35
99.41
99.22
99.02
99.69
26

---

Published as a conference paper at ICLR 2023
Table 12: Sparsity budgets of MobileNet-V/ V1 using STDS + S-LATS on ImageNet.
Final threshold
0.4
0.6
0.8
0.9
Top-1 Acc. (%)
68.44
66.64
65.41
65.13
Layer(s)
Sparsity (%)
Overall
81.84
85.87
88.22
89.08
model.0.0
51.39
62.27
57.06
55.90
model.1.0
46.53
62.85
54.86
56.94
1odel.1.3
60.45
76.76
67.63
68.51
model.2.0
11.28
21.70
18.75
18.23
model.2.3
52.82
61.65
61.56
68.92
model.3.0
23.96
25.43
26.04
31.86
model.3.3
53.38
59.64
64.90
67.77
model.4.0
2.69
5.64
8.94
5.38
model.4.3
62.50
68.58
72.97
75.10
model.5.0
20.70
26.26
29.34
31.38
model.5.3
68.64
74.57
78.17
79.69
model.6.0
15.32
20.49
20.40
25.00
model.6.3
79.08
84.26
87.14
87.85
model.7.0
24.80
29.28
34.44
37.83
model.7.3
81.65
86.53
89.26
90.23
model.8.0
35.72
40.89
48.55
48.59
model.8.3
81.10
85.35
88.22
89.20
model.9.0
40.06
42.73
46.59
50.85
m.del.9.3
79.38
84.02
87.30
87.97
10.0l.10.0
33.88
40.41
42.99
45.33
model.
75.01
80.66
83.84
85.23
model. 11.0
26.52
28.56
31.77
31.42
model.
71.13
77.06
80.75
82.36
model. 12.0
10.44
13.24
14.67
16.06
model.
80.64
84.83
87.28
88.15
model.13.0
43.65
46.79
48.35
49.09
13.3
82.10
85.47
87.57
88.36
fc
92.40
95.25
96.54
96.93
27

---

Published as a conference paper at ICLR 2023
Table 13: Sparsity budgets of ResNet-50 using PGH scheduler in pruning initialization setting ImageNet.
Final threshold D
0.1
0.11
0.13
0.15
Top-1 Acc. (%)
74.69
72.89
68.23
62.22
Layer(s)
Sparsity (%)
Overall
87.16
90.00
93.11
95.64
conv1
32.05
33.48
34.66
34.75
1ayer1.0.com
38.92
38.70
37.48
33.89
1ayer1.0.conv2
67.15
67.55
66.07
59.91
layer1.
1yver1.0.oonv3
59.89
60.16
57.60
49.34
Nyyrr1.0.dowssmmpl..0
58.89
61.24
64.05
60.72
layer1.1.com/
60.78
63.04
61.00
53.19
layerl.1.conv/
65.13
64.05
60.48
54,98
layerl.I.conv3
57.58
55.88
49.53
42.51
layer1.2.comg
53,80
55.80
51.01
43.32
layer1.2.conv2
58.37
59.34
55.33
51.22
layer1.2.com'.
61.24
62.34
60.16
48.72
layer2.0.convl
49.99
49.88
45.61
42.38
layer2.0.conv2
70.38
69.61
72.09
86.96
layer2.0.conv3
63.37
61.13
59.27
82.79
layor22..donnsamle...
75.97
74.72
70.28
59.11
layer2.1.convl
77.21
75.13
66.76
59.39
layer2.1.conv.
76.35
74.61
73.73
86.70
1ayer2.1.comv3
63.60
60.96
58.64
81.81
layer2.2.convl
68.74
66.52
56.07
49.60
laycr2.2.comv2
70.42
69.81
69.10
87.02
layer2.2.conv3
63.30
60.06
54.06
83.67
layer2.3.conv
63.87
59.87
44.49
43.37
1ayer2.3.conv2
70.13
68.83
67.57
86.60
1ayer2.3.comv3
66.09
62.03
49.41
82.74
layer3.0.com/
54.05
54.86
76.17
100.00
mnyer3..comm
82.47
89.52
99.25
100.00
layer3.0.comv'
65.61
82.20
98.40
100.00
aayer3.0.downsample..
78.21
71.87
54.37
53.19
layer3.1.convl
78.03
78.40
79.45
100.00
l.cov3.1.conv2
82.93
92.45
98.20
100.00
layer3.1.comv.
69.54
87.02
96.55
100.00
layer3.2.convl
74.19
72.72
89.19
100.00
layor3.2.comv2
81.22
89.29
99.56
100.00
1ayer3.2.conv3
68.24
82.87
99.03
100.00
1ayer3.3.comv
66.58
63.63
88.10
100.00
layer3.3.conv2
78.70
88.28
99.54
100.00
1ayer3.3.comv3
65.76
80.80
99.00
100.00
1ayer3.4.convl
57.57
59.99
75.83
100.00
layer3.4.conv2
79.10
88.66
98.66
100.00
layer3.4.comv3
62.33
80.67
97.62
100.00
1ayer3.5.convl
53.63
58.43
78.13
100.00
layer3.5.conv2
79.08
88.88
99.04
100,00
1ayer3.5.conv3
58.30
80.30
98.32
100.00
layer4.0.conv
90.52
100.00
100.00
100.00
layer4.0.conv2
99.80
100.00
100.00
100,00
layer4.0.comv3
99.37
100.00
100.00
100.00
aayer4...downsample..
75.46
79.81
86.40
91.34
layer4.1.conv
97.11
100.00
100.00
100.00
l.yor4.1.conv2
99,90
100.00
100.00
100.00
layer4.1.com/
99.56
100.00
100.00)
100.00
layer4.2.comv
98.61
100.00
100.00
100.00
1ayer4.2.conv2
99.96
100.00
100.00
100.00
1ayer4.2.conv3
99.74
100.00
100.00
100.00
fe
83.95
82.61
81.69
86.05
28

---

Published as a conference paper at ICLR 2023
Table 14: Sparsity budgets of SEW ResNet-18 using STDS + S-LATS on ImageNet.
Final threshold
0.5
1.0
2.0
3.0
5.0
7.0
15
20
Top-1 Acc. (%)
62.59
62.3
60.806
59.816
$7.572
55.454
53.74
50.024
47.586
Layer(s)
Sparsity (%)
Overall
60.11
71.18
79.74
83.75
88.11
89.96
92.57
94.30
95.21
conv
38.70
49.83
62,40
66.90
73.93
79.44
80.71
85.43
86,90
pyyer1.0.com.
48.98
60.42
73.58
76.98
82.13
86.19
87.39
90.02
91.90
37.88
50.73
61.17
66.69
71.79
76.28
79.39
82.53
84,40
layer1.1.comv1.
40.37
54.44
66.75
72.12
77.59
80.65
84.02
86.53
87.84
Iayer1...comv2.00
39.94
51.82
62.50
67.47
73.18
77.47
80.40
84.16
85.53
layer2.0.comv 1.0
39.78
52.67
64.46
68.41
74.30
77.91
81.07
83.65
85.84
1ayer2.0.comv2.00
46.68
59.70
74.26
79.64
83.32
85.99
88.92
90.62
fayer2.0.downsample...0
16.81
25.96
36.11
41.39
47.29
54.48
59.00
63.48
67.22
layer2.1.comv1..0
48.87
62.95
73.58
77.19
81.24
84.64
87.51
89.59
91.89
1.yer2.1.comv2.0
49.85
63.01
71.87
77.04
81.44
84.69
87.65
90.72
91.80
layer3.0.convL.
48.98
60.98
71.23
75.30
80.16
83.48
85.91
88.49
90.02
layer3.0.comv2.0)
56.54
68.36
77.14
81.23
86.21
88.38
91.20
93.05
94.38
1yver330..oomsmmppe......
27.11
37.56
48.76
54.21
61.76
66.29
70.50
75.62
79.10
layer3.1.comv1..
61.25
72.95
80,88
83.91
87.98
89.96
91.59
93.66
94,48
layer3.1.comv2.0
60.16
71.00
79.03
82.63
86.38
88.59
90.49
92.50
93.61
layer4.0.comv1..0
58.92
70.02
78.17
82.08
86.23
88.38
90.44
92.65
93.86
layer4.0.comv2.00
64.63
74.90
82.40
85.93
89.74
91.58
93.27
95.14
95.96
layer4.0)
31.91
42.72
53.25
58.97
65.57
69.29
72.34
75.55
77.45
Jnyer4.1.comv
1.0
67.33
77.95
85.81
89.57
93.42
95.61
96.99
97.89
98.38
1.conv.0.
72.57
80.14
84.08
88.71
91.79
94.04
95.35
96.01
fc
33.00
52.40
71.29
79.66
86.71
89.96
92.52
94.95
96.27
Table 15: Sparsity budgets SEW ResNet-18 using our implementation original STDS (STDS Sine scheduler) ImageNet.
Final threshold D
0.6
0.8
1.5
3.0
5.0
Top-1 1 Acc. (%)
61.114
60.218
57.458
52.966
48.436
Layer(s)
Sparsity (%)
Overall
76.06
79.91
85.91
90.62
93.19
convl
48.07
53.99
68.15
79.49
85.71
Mayer1.0.comy
61.37
69.80
81.65
88.07
91.37
fayer1.0.conv/
56.20
62.73
73.62
79.45
83.24
layerI.1.convil
56.73
64.55
79.27
84.19
87.55
layerl.1.conv2.0
58.62
65.86
73.35
79.39
84.28
layer2.0.convI..
58.34
66.05
75.50
80.44
85.06
layor2.0.comv2..0
66.05
72.17
80.13
85.38
88.55
layer2.0.downsaapl...00
28.70
36.28
49.60
58.87
64.78
layer2.1.cony1.0
68.32
74.54
82.47
87.28
89.96
layer2.1.conv2.0
68.65
73.85
80.92
86.52
89.55
layer3.0.cony1.00
68.04
73.05
80.11
86.17
88.89
layer3.0.conv2.00
74,94
79.00
85.20
89.69
92.33
layer33...oonnsamle..0
44.28
50.82
62.14
70.68
76.31
layer3.1.conv1.0
78.62
82.50
87.89
91.36
93.57
l.cor3.1
76.93
80.73
86.32
90.13
92.28
layer4.0.convl..
76.41
80.41
86.30
90.59
92.95
layer4.0.comv2.00
80.60
83.84
88.43
92.07
93.92
layar4.0.downnsampe..00
49.57
55,31
63.96
71.63
75,94
82.60
85.81
90.74
94.64
96.62
77.17
79.68
84.09
88.96
92.32
fc
46.12
56.43
77.07
90.62
95.18
29


Reviews:
[{"review_id": "quZCWTPcZ4", "reviewer": "Reviewer_GHaU", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper has reformulated soft threshold pruning as an implicit optimization problem solved using the Iterative Shrinkage-Thresholding Algorithm (ISTA), and proposed a unified theoretical framework for soft threshold pruning. Experiments have shown that the proposed method outperforms SOTA soft threshold pruning algorithms, such as STR and STDS.", "summary_of_the_review": "The considered problem is very important. The proposed framework is technically sound and achieves better performance than existing SOTA methods. It would be nice to test the method on more architectures and diverse hyper-parameter settings.", "strength_and_weaknesses": "Strength:\n1. The paper is well written and the presentation is clear.\n2. The pruning problem is an important one.\n3. The theoretical soundness is good.\n\nWeaknesses:\n1. The method is evaluated on relatively a small number of networks. Whether the performance improvement persists for other network architectures, such as recurrent neural networks?\n2. How the performance is influenced by hyper-parameters? Whether the performance improvement is robust under varying hyper-parameters?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "T1boEnLTKfq", "reviewer": "Reviewer_3EDJ", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper drew a connection between soft threshold pruning and Iterative Shrinkage-Thresholding Algorithm (ISTA). Based on this interpretation of soft threshold pruning the authors derive novel threshold schedulers that appear to perform better than alternatives in terms of sparsity-accuracy trade-off.", "summary_of_the_review": "While the authors investigated an important problem and conducted original research, the quality of presentation and empirical evaluation is significantly below that required for acceptance into ICLR.", "strength_and_weaknesses": "Strength:\n- The authors aspire to come up with a unified theoretical backbone of a variety of soft thresholding pruning algorithm. This is an important problem because the community can benefit from a shared theoretical infrastructure for analyzing threshold pruning techniques, which are often empirical in nature.\n- The authors propose a theoretically grounded threshold scheduler and demonstrate  some benefits of this threshold scheduler empirically.\n\nWeakness:\n- The paper is highly disorganized on at least two levels.\n    - **Inadequate explanation of core concepts.** The authors made little to no attempt to introduce and define essential concepts such as \"threshold scheduler\" for the audience early in the paper. As a result I have to start with Sec.3 Preliminaries and work my way back to make sense of the abstract and introduction.\n    - **Disorganized prsentation of technical details.** This disorganized style of writing manifests even when the author discusses specific technical details. For example, when the authors introduce Eq.2, the symbol \\mu appears out of nowhere. Upon further examination, I realize that this variable \\mu is implicitly defined in the L.3 of the pseudocode presented in Algorithm. 1 as the output of the learning rate scheduler function. The author must clearly define symbols before using them.\n\n        While the authors may very well have much to share with and educate their audience, the disorganized writing style is a significant obstacle to achieving this potential.\n\n- There are major red flags in experimental evaluation.\n\t- **Reproducibility and fairness of comparison**. I believe authors obtain individual data points in Fig. 2 comparing the proposed threshold scheduler with alternatives by \"tuning final threshold D\". But the author does not provide details for how this threshold is tuned. This raises questions on 1). whether the core empirical evaluation of this paper can be reproduced 2). whether comparison with alternative soft threshold pruning techniques is fair.\n\t- **Missing simple baseline.** For the left plot of Fig. 2, the authors instrumented a prior soft threshold pruning technique called STDS to use their proposed threshold scheduler and demonstrated superior performance to other soft threshold pruning techniques. However, it is unclear whether this superior performance is due to the STDS pruning algorithm or the proposed threshold scheduler. The author should report performance of STDS with the default threshold scheduler as an important baseline.\n\t- **No characterization of uncertainties.** The performance of pruning algorithms can have significant variance between different weight initialization/SGD data order. The results presented in this work lack characterization of uncertainties stemming from this variance.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "OhFUyFUtJF", "reviewer": "Reviewer_Lscv", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents a unified framework of iterative soft threshold pruning (ISTA). The framework reveals that previous studies with threshold pruning can be casted as adding regularization terms (i.e., L1-regularization) for training. The authors further connect the learning rate with pruning threshold, and propose the learning rate adapted optimal threshold scheduler. The framework also has the link to other pruning methods such as early pruning, pruning at initialization. Experimental results on various network architectures (ResNet-50, MobileNet-V1 and Spiking Neural Networks) demonstrate the effectiveness of the approach. ", "summary_of_the_review": "The paper provides a comprehensive and theoretical study of soft threshold pruning. While it is not new to tackle network pruning with ISTA or proximal mapping of $\\mathcal{L}_1$ regularizer, the theory developed in this paper further motivates the design of optimal pruning threshold scheduler. The experiments are also adequate and effective to demonstrate the proposed solutions. The authors also provide in-depth analysis on how the framework relates early pruning, spiking neural nets, and more in the appendix. One potential issue with the paper is the practicality, as unstructured pruning can hardly bring actual acceleration. Overall, the paper is quite informative and well presented. ", "strength_and_weaknesses": "Strengths:\n\n- The paper establishes nice connections between the learning rate and pruning threshold scheduler via Theorem 1. \n\n- The methodologies developed in this paper are well grounded in theory, e.g., the learning rate adapted optimal threshold scheduler for pruning, and PGH schedulers that connects to early pruning.\n\n- The authors conduct extensive studies across various network architectures (ANNs & SNNs) as well as different pruning settings. \n\nWeakness: \n\n- While I appreciate the authors' effort in introducing the optimal threshold scheduler and sibling schedulers for pruning, it is not new to formulate pruning with local ISTA, which has been studied by plenty of previous works, e.g., [1,2,3].\n\n  - [1] He Y, Zhang X, Sun J. Channel pruning for accelerating very deep neural networks. Proceedings of the IEEE international conference on computer vision. 2017: 1389-1397.\n\n  - [2] Zhang D, Wang H, Figueiredo M, et al. Learning to share: Simultaneous parameter tying and sparsification in deep learning, International Conference on Learning Representations. 2018.\n\n  - [3] Bai H, Wu J, King I, et al. Few shot network compression via cross distillation. Proceedings of the AAAI Conference on Artificial Intelligence. 2020, 34(04): 3203-3210.\n\n- Only unstructured pruning is discussed in the paper, this however, has limited acceleration in practice. I wonder whether similar results can be established for structured pruning, e.g., via $\\mathcal{L}_{2,1}$ regularization. \n\nDetailed comments: \n\n- Some parts of the paper are not self-contaiend. While it is claimed to be verified on Mobilenet-V1 and spiking neural networks, their results are in the appendix and not referenced in the main text. \n\n- Section 3.1 could have been organized better. There is some notation overlapping (e.g., $d$, $\\phi(\\cdot)$ and $g(\\cdot)$ for the pruning threshold). It would be better to have a short overview of STR and STDS first.\n\n- Provide more details on how Eq. 12 and Eq. 13 are derived.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster