Towards Energy Efficient Spiking Neural Networks: An Unstructured Pruning Framework
Authors: Xinyu Shi
Jianhao Ding
Zecheng Hao
Zhaofei Yu
Venue: ICLR 2024 spotlight (2024)
Keywords: Spiking Neural Networks
Network Pruning
Primary area: applications to neuroscience & cognitive science

Abstract: Spiking Neural Networks (SNNs)  have emerged as energy-efficient alternatives to  Artificial Neural Networks (ANNs) when deployed on neuromorphic chips.  While recent studies have demonstrated the impressive performance of deep SNNs on challenging tasks, their energy efficiency advantage has been diminished. Existing methods targeting energy consumption reduction do not fully exploit sparsity, whereas powerful pruning methods can achieve high sparsity but are not directly targeted at energy efficiency, limiting their effectiveness in energy saving. Furthermore, none of these works fully exploit the sparsity of neurons or the potential for unstructured neuron pruning in SNNs. In this paper, we propose a novel pruning framework that combines unstructured weight pruning with unstructured neuron pruning to maximize the utilization of the sparsity of neuromorphic computing, thereby enhancing energy efficiency. To the best of our knowledge, this is the first application of unstructured neuron pruning to deep SNNs. Experimental results demonstrate that  our method achieves impressive energy efficiency gains. The sparse network pruned by our method with only 0.63\% remaining connections can achieve a remarkable 91 times increase in energy efficiency compared to the original dense network, requiring only 8.5M SOPs for inference, with merely 2.19\% accuracy loss on the CIFAR-10 dataset. Our work suggests that deep and dense SNNs exhibit high redundancy in energy consumption, highlighting the potential for targeted SNN sparsification to save energy.

Published as a conference paper at ICLR 2024
TOWARDS ENERGY EFFICIENT SPIKING NEURAL NET- WORKS: AN UNSTRUCTURED PRUNING FRAMEWORK
Xinyu Shi!?, Jianhao Ding', Zecheng Hao? & Zhaofei Yu!
Institute for Artificial Intelliennce, Peking University.
2 School of Computer Science, Peking University {xyshi, d 998, haozecheng, yuzf12] edu.
ABSTRACT
Spiking Neural Networks (SNNs) have emerged as energy mnergy-efficient alternatives to Artificial Neural Networks (ANNs) when deployed neuromorphic chips. While recent studies have demonstrated the impressive performance of deep SNNs on challenging tasks, their energy efficiency advantage has been diminished. Exist- ing methods targeting energy consumption reduction do not fully exploit sparsity, whereas powerful pruning methods can achieve high sparsity but are not directly targeted at energy efficiency, limiting their effectiveness in energy saving. Further- more, none of these works fully exploit the sparsity of neurons or the potential for unstructured neuron pruning in SNNs. In this paper, we propose novel pruning framework that combines unstructured weight pruning with unstructured neuron pruning maximize the utilization of the sparsity of neuromorphic computing, thereby enhancing energy efficiency. To the best of our knowledge, this the first application of unstructured neuron pruning to deep SNNs. Experimental re- sults demonstrate that our method achieves impressive energy efficiency gains. The sparse network pruned by our method with only 0.63% remaining connec- tions can achieve a remarkable 91 times increase in energy efficiency compared to the original dense network, requiring only 8.5M SOPs for inference, with merely 2.19% accuracy loss on the CIFAR- 10 dataset. Our work suggests that deep and dense SNNs exhibit high redundancy in energy consumption, highlighting the potential for targeted SNN sparsification to save energy. Codes are available at https///ghhuub.ooomy///////oolnn Pruning
INTRODUCTION
Spiking Neural Networks (SNNs), considered as the third generation of Artificial Neural Networks (ANNs) (Maass) 1997), have attracted significant attention on low-power edge computing with lim- ited resources in recent years. In SNNs, the biologically-inspinn
neurons generate sparse and dis- crete events by firing binary spikes to communicate with postsynaptic neurons (Krestinskaya et al.) 2019). Compared with ANNs, SNNs have the energy-saving advantage due to their event- event-driven na- ture when deployed on neuromorphic hardware (Furber et al. 2014 Merolla et al.] 2014 Pei et al. 2019) and have become popular in neuromorphic computing recent years (Schuman al.
Recently, there has been a growing interest in developing deep SNN architectures capable of han- dling challenging tasks such as object recognition (Kim & Panda 2021 Zhu et al. (022). detec- tion (Kim et al. 2020), text classfication (Lv et al. 2023), and robotic control (Tang et al. 2021)
These architectures, such as Spiking VGG (Sengupta et al. 2019 Lee et al. 2020) and SpikingRes-
Net (Zheng et al. [222]] Hu et al. al, 2021 |Fng et al. 202 la) employ large-scale SNNs with numerous layers achieve superior task performance (Duan al. (022). However, this comes at the cost of
compromising the energy-saving benefits and necessitating greater computational resources.
Researchers make great efforts to enhance energy efficiency in SNNs. Since energy consumption of SNNs is closely tied to synaptic operations, existing methods aimed at improving energy efficiency primarily focus on two approaches: reducing the number of spikes or reducing the number of synap- tic connections, i.e., reducing the model size. To reduce the number of spikes, several methods focus
"Corresooniigg author

---

Published as a conference paper at ICLR 2024
on designing SNNs reasoning process to achieve spike number reduction the algorithmic level, such as using sparse coding (Han & Roy, 2020; Rueckauer & Liu 2018) or reducing the required
reasoning time steps of the model (Ding al. [021] Bu et al. [002 Chowdhury et al. 2022). Other
algorithms specifically target spike reduction as an optimization goal (Kundu et al. 2222 Na et al.)
2022). These efforts have shown promising results in reducing energy consumption. However,
should be noted that these methods do not fundamentally eliminate model redundancy.
Another technical route is reduce the size of the model so as eliminate redundancy. This ap- proach involves various methods, including pruning (Chen et al. 202T 2022). quantization (Wang
2020: Chowdhury et al. 2021; Deng et 2021). and knowledge distillation (Kushawaha et al. bitwise precision of the model parameters, while knowledge distillation methods transfer knowledge from a large model to lightweight alternative. These methods reduce the model size in a structured manner, resulting in dense lightweight models. In comparison, fine-grained pruning shows a remark- able ability to eliminate redundancy in unstructured manner and achieve high sparsity, thereby reducing synaptic connections without significant performance degradation. These researches sug- gest that pruning can significantly reduce energy consumption (Chowdhury al.. 2021).
al.
2021 Takuya et al. 2022 Tran et al. [022] Xu et al., (023). Quantization methods reduce the
While pruning is a promising energy reduction approach, previous methods have not directly op- timized energy consumption. Existing pruning works have primarily been adapted from pruming methods designed for ANNs and focused more on enabling SNNs to run on hardware with limited computational resources. Consequently, these methods have limited impact energy consump- tion since they were not specifically designed for energy reduction purposes. Furthermore, most existing fine-grained pruning methods in ANNs primarily emphasize weight pruning, while the pruning of neurons is typically performed in a structured manner. When applied to SNNs, they overlook the fact that neurons in SNNs can also be pruned in a fine-graine manner. This oversight hinders the full utilization of the sparse computing capabilities of neuromorphic hardware.
This paper presents a comprehensive pruning framework for enhancing energy efficiency in SNNs. Our approach combines unstructured weight pruning with unstructured neuron pruning, leveraging the advantages of the sparse and event-driven nature of neuromorphic computing. By reducing energy consumption while maintaining low-performance loss, method achieves a state-of-the- art balance between accuracy and energy efficiency. The main contributions are summarized as:
We develop model to analyze the energy consumption of SNNs and evaluate the effec- tiveness of weight and neuron pruning in reducing energy consumption.
We propose fine-graid fine-grained pruning framework that integrates unstructured weight and neu- ron pruning to enhance the energy efficiency of SNNs. To the best of our knowledge, this is the first application of unstructured neuron pruning to deep SNNs.
We present novel design for the energy penalty term, addressing the ill-posed problem that arises when jointly pruning neurons and weights under energy constraints.
Our method outperforms previous works in reducing energy consumption while maintain- ing comparable performance. The sparsest network pruned by our proposed method re- quires only 8.5M SOPs for inference, which is 91 times less than the original dense net- work, with merely 2. 19% accuracy loss on the CIFARIO dataset.
RELATED WORK
Pruning methods for ANNs. Pruning methods for ANNs can be broadly classified into three cat- egories (Hoefler et al. 202T): 1) Unstructured weight pruning, 2) Structured weight pruning, and 3) Structured neuron pruning. Most unstructured weight pruming methods come from two starting points:
heuristic strategies and training rwining-aware methods. Heuristic methods. such as magnitude- based weight pruning (Han et al. 2015 Zhu & Gupta, 2018 Kusupati et al. (020), are simple yet effective, and have garnered significant attention in early studies. Recent approaches mainly utilize training-aware methods, which leverage gradient-base criteria for pruning. These criteria are de- signed in diverse ways, such as modifying the behavior of forward propagation (Lin et al.) 2020) or approximating 4o regularization (Savarese et al. 2020). Different structured weight pruning methods exhibit varying levels of regularity, including stride-level, vector-level, kernel-level, and filter-level pruning (Mao et al., 2017). Previous studies (Wen et al. 2016 Anwar et al.) al, (017)) have investigated the performance and efficiency of these methods across different regularity levels. In the case of
2

---

Published as a conference paper at ICLR 2024
structured neuron pruning, the focus has primarily been the channel level. Existing approaches include utilizing trainable channel selectors (Luo & Wu (020), evaluating importance scores of neu- rons (Yu et al. 2018), applying regularization to the scaling factors of Batch Normalization (BN)
layers (Zhuang et al. 2020). In the case of unstructured neuron pruning, several works dynamically "prune" neurons, L.e,, "prune" out small activations (Kurtz et al. 2020: [Sekikawa & Uto [02T).
Pruning methods for SNNs. Recent research in SNN pruning mainly focuses on weight pruning, drawing inspiration from the well-established pruning methods developed for ANNs. Typical works include using the magnitude-basee method (Yin et al., 2021), combining classic optimization tools with the SNN training method (Deng et al. 2021), etc. In addition, recent studies have explored bio-inspired pruning algorithms that leverage the similarities between SNNs and neural systems. These algorithms primarily focus on modeling the synaptie regrowth process, employing probabil- ity (Bellec et al.) 2018) or gradient (Chen et al.] 202T 2022) as criteria for regrowth. In contrast to weight pruning, the study of neuron pruning for SNNs remains relatively limited. Wu et al. (2019)
propose method based on neuron similarity to select neurons for removal. However, this approach cannot be applied to SNNs with complex and deep structures, limiting its practicality.
Other energy-savingg methods for SNNs. Apart from pruning, there are three main energy- methods: 1) Quantization, 2) Reduction of firing rates, and 3) Knowledge distillation. Quantiza- tion methods aim to enhance efficiency by reducing the precision of weights (Deng al. 2021)
to match the reduced bit precision of neuromorphic hardware platforms (Qiao et al.) 2021: Stro-
matias et al.) 2015). Recent research on quantization methods for SNNs mainly focuses on weight
binarization (Qiao et al. 202I; Kheradpisheh et al. [2222] Wang et al. 2020). Directly reducing the firing rate of neurons helps decrease computational overhead. Previous approaches introduce
penalty term for firing rate as form of regularization (Deng et al. 2021; Neil et al.] (2016). or
employ network architecture search methods (Na et al.) 2022) to discover SNN structures with low firing rates. Knowledge distillation explores transferring knowledge from large-scale ANNs (Takuya) et al. 2021| Tran et al. 2022] Xu et al., [2023] Guo et al. 2023) or SNNs (Kushawaha et al. 2021)
to smaller-scale SNNs to compress the models and reduce energy consumption.
3 MOTIVATION
In this section, we first model the energy consumption of SNNs. Based on this model, we analyze the limits of existing pruming methods of SNNs and point out that pruming neurons and weights in SNNS contribute to balancing energy efficiency and performance.
3.1 ENERGY CONSUMPTION MODEL
The primary metric used assess the energy consumption of neuromorphic chips is the average energy required to transmit a single spike through a synapse (Furber (016)). This is due to the sig- nificant energy cost associated with synapse processing, making it erucial factor in overall energy consumption. For paardware-indpeendee the entire system corresponding to one synaptic operation (SOP) on average as known constant, and use the SOPs as the metric of energy consumption of the model. Specifically, we employ the following model to estimate the energy consumption of an SNN:
theoretical analysis, we treat the energy consumption of
E = CE #SOP = CE Escr, s:C,
(1)
where Cg denotes the energy consumption of one SOP, #SOP E; s,C, denotes the total num- ber of synaptic operations. For each presynaptic neuron i in an SNÑ, s; denotes the number of spikes fired by this neuron, and c; denotes the number of synaptic connections from this presynap- tic neuron. It's worth noting that such a linear energy model may not be suitable for all hardware architectures. We believe that it matches the architectures that are highly optimized for sparsity. A detailed discussion of the suitability can be found in the appendix.
Energy consumption model for sparse SNNs. For a sparse SNN, we restate Eq. ① as follows:
E = CE
Σ
st Σ
n
A
post nij
In this paper, we define SOP the operations performed when a spike passes through synapse.
3

---

the following unified discrete-time model to describe the Published as a conference paper at ICLR 2024
Symsptic connectione SNN4
Number of left connection after different pruning
Weight
Symptic connection
a) dense
bj weight pruning
neuron pruning
line grained pruzing
Figure 1: Diagram of synaptic connections in SNNs and number of connections left after pruning. Left: Neurons, weights, and synaptic connections in a convolutional layer with 5 x 5 input and 3 x 3 filter. Right: The effect of different kinds of pruning on the number of synaptic connections. The number on the top of each subfigure indicates the number synaptic connections left.
Here, for each synaptic connection from i-th presynaptic neuron its j-th postsynaptic neuron, (nf"e, O1g. n2 {0, 1} {0. 1} {0, 1 denotes the states presynaptic neuron, synapse, and postsynaptic neuron. 1 means alive, while 0 means pruned. A is the logical AND operation.
3.2 BENEFITS OF FINE-GRAINED SNN PRUNING
Eq. 2 illustrates that presynaptic neurons, synapses, and postsynaptic neurons all contribute to computation in SNNs. When any of these components are pruned, the corresponding connections are also pruned, eliminating the need for computations associated with those connections. Fig. provides example of simple convolution operation with a 3x3 filter and 5x5 input with zero-padding, demonstrating how fine-grained pruning significantly reduces synaptic connections.
While both neuron pruning and weight pruning valid approaches, existing SNN pruning methods focus more on weight pruning, and neuron pruning is only performed in a structured manner or on shallow fully-connected networks (Wu al. 2019). The main concern is that in shallow fully- connected networks, neuron pruning can effectively be replaced by weight pruning since pruning a neuron is equivalent eliminating all associated weights. However, this approach is not applicable to convolutional layers in SNNs. In convolutional operations, the convolution kernel weights are multiplied with each element the input channel (ignoring stride). Pruning all weights associated with a neuron would result in the entire channel being pruned. Therefore, weight pruning fails to other words, is not fine- -reingr enough.
capture the unstructured nature of neuron pruning,
Except for structural considerations, fine tneegrained unstructured neuron pruning offers a better trade- off between performance and energy efficiency. The spatial correlation among features in con- volutional layers means that neighboring elements often carry similar information. Consequently, unstructured neuron pruning, when applied at an appropriate sparsity level, does not compromise performance. This enables further reduction in energy consumption while maintaining comparable performance.
METHOD
In this section, we first introduce the spiking neuron model SNNs, and then present a general formulation of the unstructured pruning framework with energy consumption constraint. Finally, we delve into the design of the energy constraint and derive the comprehensive minimization problem.
4.1 SPIKING NEURON MODEL.
Similar to Fang al. (2021b), dynamics of spiking neurons:
w|{ = g(ul4, Σ we we)+
(3a)
oft] = H(((t] - unb)
(3b)
ult 1] oft) Mrest + o(t))u(t)
(3c)
4

---

Published as a conference paper at ICLR 2024
Here Litt] denotes the imput current from the i-th presynaptic neuron at time-step t, and w, is the corresponding synaptic weight. v/t] and u(t+ denote the membrane potential of the neuron before and after firing time-step t. H is the Heaviside step function. oft] E {0, 1] denotes whether or not the neuron fires a spike at time-step f. When the membrane potential v|(] at time step reaches the firing threshold lhth the neuron will fire spike, and the membrane potential after firing ult + 1] will reset to the resting potential Wyeat. The function g(:-)) describes the neuronal dynamics and differs between different spiking neuron models. For example, the Leaky Integrate-and-Fire (LIF)
model can be defined by Eq. 4
1 (x|11 - (u[[) Wrest) )),
s(u(4,=(()) =14]
(4)
where T denotes the membrane time constant and xr(t] denotes the input.
4.2 PRUNING OF WEIGHTS AND NEURONS
In order to describe the sparse spiking subnetwork, we introduce binary masks my and Wa to indicate the state of the neurons and weights, respectively. Given neuron, we restate Eq. (3a) with binary masks as follows:
oft = g(u(t), E(mu: (m,, I((t)),
(5)
where (M,,i, mw.+) E {0, 1 x {0, 1} are the binary masks the i-th presynaptic neuron and the corresponding synaptic weight. A mask value of indicates that the presynaptic neuron synaptic weight is pruned, while a value of indicates that it is active.
4.3 SPARSE STRUCTURE LEARNING WITH ENERGY CONSTRAINTS
Given a spiking neural network f(-) with weights w E Rd and masks of weights and neurons m E {0, 1) m, E 10,1, ddz, where d, and d, denote the number of neurons and weights of network f, respectively, we formulate the learning the sparse SNN as a loss minimization problem with energy constraints as follows:
arg min c(f(; w. m. + AE(:; f, mms m,,). M,
(6)
Here mw E {0, 13dd and m, E {0, 13 denote the masks of weights and neurons, respectively. @ denotes element-wise multiplication. L(f(; m. m., @ w. m.,) denotes the loss of the prediction
of model [(; m. @ w. m... E(s m.) ) is the energy consumption penalty term, corre- sponding to the estimated energy consumption of the network [(= w, m,) based the model presented in Eq. Q The parameter À controls the trade-off between performance and energy consumption.
To address the non-diffcrentiabiiit of the binary mask m. we re-parameterize m by introducing a new parameter a E RR approximate m. Specifically, we define m H(a), where H(-) represents the Heaviside step function. As H()) is also non- -on-iffferentialle, we propose using the scaled sigmoid function o (a;; 3) = a (Ba) = Ite- to approximate the Heaviside step function H-)). The scaled sigmoid function is smooth and approaches the Heaviside step function as 3 tends to infinity, that is, g (o:; () (ax)
By re- e--rrammterizingg the problem in this way, we can reformulate the original minimization problem of Eq. (6) as follows:
arg min lim (3a C w. (Bea.. + XB(s) an ae Ci, 8-000
To achieve an effective approximation of the binary mask m, scale factor 3 must be large enough. However, using fixed large value for 3 can result in model degradation. In our experiments, adopt gradual scheduling approach for the scale factor 3, transitioning it from an initial value
3 to a final value 3r using the exponential function ((t) = 30
5r T, where T, represents the 3o
number of training epochs. After training the weights and masks for T. epochs, we replace the scaled sigmoid function with the Heaviside step function and continue training with fixed masks. An overview of the proposed method is depicted in Fig.
5

---

Published as a conference paper at ICLR 2024
sparse Weights
sparso
binary
mask
musk
binary
mask
music
masked
kernel
HQ)
layer/ /
layer f+1
convolution
spicing nourom
clament-wwis multiply
Figure 2: Overview of our framework.
4.4 DESIGN OF ENERGY CONSUMPTION PENALTY TERM
Eq. 7 represents the general form of the minimization problem, incorporating an energy consump-
tion penalty term. However, it does not provide specific details regarding the implementation of the penalty term. this section, we will delve into the design of the penalty term. According to Eqs. ( and (7). the penalty term can be defined as follows:
E= C= Σ 8 Σ a (Ba . (BCm..t))) (Ba
(8)
where a Owwag p.iy are the re-parameterized mask values of ni Ozj. pi respectively. Note that here we replace the logical AND operation with multiplication to facilitate easier optimization.
This penalty term is directly derived from the energy consumption model of sparse SNNs by substi- tuting the continuous masks into Eq. (2). While it provides an intuitive formulation, optimizing this term can be challenging. The main difficulty arises from the multiplication of the three continuous masks, which makes the problem ill-posed. Consequently, the solution becomes highly sensitive to the initial values of hyperparameters such as the scale factor Bo and the reparameterized mask ou- Improper initialization can easily lead to degradation, i.e., one type of elements (neurons or weights) completely pruned, resulting in a trivial solution the minimization problem.
To solve this problem, we convert this composite constraint term into two independent sub- constraints by rewriting Eq. ( as follows:
E, = CCE Σ
Pre) m))
$;
Σ o (Baw,ss) (B3 = CE E Cn (Ba
(9a)
Ew = CC Σ ((aa.pp))
Σ Spg (Baam)p) o (Bampp))
= CCE Σ P
e (Ba...p)
(9b)
In Eq. (aa.. we first place the presynaptic neuron mask a (Ba in the outer summation as each spike fired by the presynaptic neuron associated with itself. For Eq. (9b). since the total number of SOPs independent of the counting method, we adopt an alternative counting approach. Specifically, we count the number of spikes transmitted through the synapses associated with each weight. In this context, for the p-th weight, a a(Baw.p) represents its mask, while 8pq, aB3o and a (Bor))) denote the number of transmitted spikes, the mask of the presynaptic neuron, and the mask of the postsynaptic neuron, respectively, for the q-th synapse associated with this weight.
To obtain independent sub-constraints, we simplify the dynamic nature connections by approx- imating the number of connections associated with each element a constant. Specifically, in Eq. (9a), we focus on thepresynaptic neuron and treat a (Bor,ss)) and ar (Ba as constant. Sim- ilarly, in Eq. (9b). we focus on synaptic weight and treat ( and o (Ban.))) as constant. Note that in Eq. (9a), specifically focus the SOPs associated with presynaptic neurons, since a neuron in SNNs typically functions as both a presynaptic and postsynaptic neuron, and accu- rately counting the SOPs associated with postsynaptic neurons proves be challenging. To further

---

Published as a conference paper at ICLR 2024
Table 1: Performance under different sparsity levels
Top-1 Acc. (%)
Acc. Loss
Avg. SOPs
Conn.
Neuron (%)
Weight
Dataset
Arch.
Ratio
(9)
(M)
(%)
(%)
(dense)
92.84
0
776.12
100
100
100
10
92.63
-0.21
38.32
20.25
2.97
21.03
34.24
CI
Conv, FC
2 10-11
-0.79
3 10-11
91.47
-1.37
11.98
64.78
0.87
10.31
23,00
x 10
90.65
-2.19
8.50
91.31
0.63
8.23
19.34
(densc)
0
647.30
10
81.9
-0.5
47.81
13.54
6.80
42.73
36.35
DVS-CIFARIO
VGGSNN
-1.4
31.86
20.32
4.46
35.13
33,02
7 10
79.0
-3.4
10,02
64,60
1.27
18.56
22.56
1 x 10
78.3
-4.1
6.75
95.90
0.77
14.11
18.66
(dense)
ImageNet
SEW ResNet18
10
10-10
61.89
-3.18
-1.29
311.70
177.99
7.52
4.29
28.60
19.31
74,86
27.74
35,26
2 10
$8.99
.4.19
116.88
11.45
14.19
51.17
23.52
simplify this term, we assume that the average firing rate remains constant. Thus, the number of transmitted spikes on each synapse s; and 8pg can be regarded constants. Based on these approxi- mations, we treat Cn = 8; E, o (BBaw,js) (Ba and 6w.p = Eg Spy P (Bar.pp)) (Bar..p) as constant, here e; and ep denote the approximate SOPs associated with i-th neuron and p-th weight, respectively. Furthermore, given that the sigmoid function and SOPs are always positive, we observe that this penalty term equivalent to the regularization.
Fn Ce e (Bax))l1, Ew CE (
(10)
where we use the vector form of en and e, to replace the scalar form of en,; and ew.p. By applying Eq. (10 to Eq, 77. the final form of the minimization problem is as follows:
arg min lim (( a (Baxw) @ w, ((3a,))) + Al (3a 1 + o 00.00.. 8-000
(11)
Note that the energy constant CF omitted here, can be absorbed into the parameter during optimization. The entire learning process is summarized in Algorithm 1 of the appendix.
5 EXPERIMENTS
In this section, we evaluate the effectiveness of our method for classification tasks on the CIFAR- 10 (Krizhevsky et al. 2009), DVS-CIFARIÓ (Li et al. (207)), and ImageNet (Deng al.) 2009) datasets. We first analyze the performance under different sparsity levels, including accuracy and energy consumption. Further, we compare our method with other state-of-the-art approaches. Fi- nally, we examine the effectiveness of unstructured neuron pruning and the combination with weight pruning by ablation study. More details experimental settings can found the appendix.
5.1 PERFORMANCE UNDER DIFFERENT SPARSITY LEVELS
We analyze the test accuracy and energy consumption under different sparsity levels and report the results in Tab. [ Here "Avg. SOPs" denotes the average SOPs required for an instance in inference. This value is used to estimate energy consumption, as discussed in Sec. 3.J Additionally, "Ratio" represents the energy saving ratio compared to the original dense network. Conn.", "Neuron", and "Weight" refer the percentage of remaining connections, neurons, and weights, respectively.
For the CIFAR-10 dataset, we adjust the parameter À from 5 x 10 12 to 5 x 10-11 to control the sparsity level. The sparsest network achieved, with À = 5 10-11 retains only 0.63% of the original connections. Remarkably, single instance now requires only 8.5M SOPs for inference, which is 91.31 times less than the original dense network, with merely 2.19% decrease test accuracy. Even for smaller values of X, our method remains effective in reducing synaptic connections and energy consumption. For instance, we achieve 20.25 times more energy efficiency than the original dense network with only 2.97% connections and 0.21% accuracy loss. These results demonstrate the ef- fectiveness of our method in discovering sparse networks with remarkably low energy consumption. Additional experimental results the CIFAR-100 dataset be found in Tab. Al the appendix.

---

Published as a conference paper at ICLR 2024
Table 2: Performance comparison between the proposed method and the state-of-the-art methods
Pruning Method
Top-1 Acc.
Acc. Loss
Avg. SOPs
Conn.
Param.
(9E)
(%)
(M)
(%)
(M)
90.19
-0.13
107.97
2.91
25.03
15.54
ADMM
7 Cons, FC
88.18
-2144
49.72
6.32
10.34
6.22
79.63
-10.69
28.65
10.96
5.35
3.11
92. 02.54
-0.30
371.05
2.09
36.72
10.43
92.50
-0.34
232.51
3.33
22.24
4.42
Grad R
6 Cony, FC
92.03
-0.81
143.69
5.40
11.85
1.86
91.37
-1.47
87.73
8.85
6.24
0.86
89.32
-3.52
41.89
18.53
2.15
0,26
92.08
-0.63
298.24
2.11
32.71
2.53
ESLSNNI
ResNet19
91.46
178.10
3.54
16.06
CIFARIO
90.90
-1.81
108.89
5.79
8.221
0.63
92.49
-0.35
147.22
5.27
11.33
1.71
92.49
-0.35
87.94
8.83
5.29
0.82
Cony,
91 06
-1.78
37.16
20.89
1.91
0.36
92.63
-0.21
38.32
20.25
2.97
12.57
This work
6 Cons, FC
91.47
-1.37
11.98
64.78
0.87
8.44
90.65
80.6
-1.8
266.
2.43
36,19
1.94
ESISNN
VGGSNN
79.2
77.5
-4.9
79.65
8.13
8.51
0.48
2.87
DVS-CIFARI0
STDS
VGGSNN
10
81.1
-1.3
76.56
8.45
10.17
0.65
81.9
-0.5
47.81
13.54
6.80
3.52
81.0
20.32
This work
VGGSNN
10
79.0
-3.4
10.02
64.60
1.27
2.18
78.3
95.90
62.84
-0.34
1082.60
1.24
77.54
7.42
2.61
STDS
SEW 18
61.30
-1.88
424.07
3.15
29.29
1.94
59.93
-3.25
273.39
4.89
21.08
ImageNet
58.06
-5.12
182.49
7.33
15.71
0.75
56.28
-6.90
134.64
9.94
12.95
0.52
61.89
-1.29
311.70
4.29
28.60
3,93
This work
SEW R8
60.00
-3.18
177.99
7.52
19.31
3.10
58.99
-4.19
116.88
11.45
14.19
2.62
implementation
The definition of "connections" in this paper is different from GradR and STDS. Here Conn, refers, to percentage of synaptic left, "connections" GradR and STDS weights left, corresponding
For the DVS-CIFAR10 dataset, our method achieves top-1 accuracy of 81.9% (merely 0.5% aceu- racy loss) and is 13.54 times more energy efficient than the original dense network when constrained to 6.80% connectivity, Moreover, we achieve 95.90 times more energy efficiency than original dense network with only 0.77% connections and 4.1% 1% accuracy loss. These results demonstrate that our method also applies to neuromorphic datasets.
Additionally, evaluated the performance of method a large-scale dataset, namely, the ImageNet dataset. The ratio of energy efficiency gains lower compared to the CIFARIO and DVS-CIFAR10 datasets. This is because ImageNet classification is more challenging. and SEW ResNet18 is not large model, resulting in less redundancy available for pruning. Nevertheless, our method still achieves 4.29 times more energy efficiency than the original SEW ResNet18 with 1.29% top-1 accuracy loss, and 11.45 times more efficiency with 4. 199 accuracy loss.
5.2 COMPARISON WITH THE STATE-OF-THE-ART
To evaluate the effectiveness of our method, we compare it with other state-of-the-art SNN algo- rithms ADMM (Deng et al. 2021), Grad (Chen et al. 2021). ESLSNN (Shen et al. 2023) and STDS (Chen et al. (022). We present the results in Tab, Since these works did not provide de the erage SOPs required for inference on the test set, we evaluate these values using our implementation. Additionally, we visualized the comparative results for the CIFAR-10 dataset in Fig.
8

---

Published as a conference paper at ICLR 2024
92-
91-
913
Accuac
91.0
pospaie ADMe
935
ESLSNN
prgesed
weigt-orly
STDS
55
10
103
12.5
15.0
20.0
22.5
250
27.5
Figure 3: Comparison with the state-of-the-art
Figure Results of ablation study
As shown in Tab. 2 and Fig. [] our method achieves comparable performance while significantly reducing energy consumption compared to the state-of-the-art methods. For example, on the CIFAR- 10 dataset, the current leading method, STDS, achieves a top-1 accuracy of 90.21 % with 1.35% connections and 26.81M SOPs. In contrast, our method achieves higher accuracy of 90.65% with only 0.63% connections and 8.5M SOPs. In Fig. 3 our data line is positioned in the upper left corner of all other methods, indicating that our proposed method achieves lower SOPs with equal accuracy.
Furthermore, we observe an interesting phenomenon. Fewer weights left does not necessarily lead to lower energy consumption. As indicated in Tab. 2 GradR achieves fewer weights than STDS (0.26M vs. 0.28M) using the same dense network structure. However, GradR requires an average of 41.89M SOPs, whereas STDS only requires 26.81M SOPs. This discrepancy arises from the fact that the model pruned by GradR has higher connectivity (2. 15% vs. 1.35%).. According to our energy consumption model in Sec. 3.1 assuming the average firing rate of each neuron remains the same, energy consumption is directly proportional to the connectivity. Therefore, this phenomenon validates our proposed energy consumption model that the SOPs is proportional the connectivity.
5.3 ABLATION STUDY
To validate the effectiveness of combining unstructured neuron pruning with weight pruning, we conduct experiments where we individually prune neurons or weights and compare their perfor- mance with proposed method on the CIFAR-10 dataset. For fair comparison, all experimental settings remain the same, except for the prunable elements and the weight factor the penalty term X. As shown in Fig. 4 our proposed method outperforms pruning only type of element. No- tably, the performance difference becomes more significant as the sparsity level increases. Detailed experimental results are listed in Tab, A2 in the appendix.
Upon further analysis of the results, we identify the limitations of pruning only one type of element. Pruning weights could achieve comparable or slightly worse performance low sparsity levels, while still resulting in high energy consumption. proves ineffective in further reducing energy consumption. For instance, as sparsity increases, the accuracy drops by 1.22%, while SOPs de- crease by a mere 3.24M. On the other hand, pruming neurons compromises performance but offers substantial reduction energy consumption with less accuracy loss. Therefore, combining neuron and weight pruning represents a trade-off between accuracy and energy consumption, enabling more effective reduction in energy consumption while maintaining acceptable performance levels.
CONCLUSION
In this paper, we propose a novel framework that combines unstructured weight and neuron pruning to enhance the energy efficiency of SNNs. We detail the design of the energy consumption constraint to overcome the ill-posed problem. Experimental results demonstrate the efficacy of our approach in significantly reducing the energy consumption of deep SNNs, achieving higher energy efficiency than previous methods while maintaining comparable performance. Our work highlight the immense potential of SNN sparsification methods that target energy-saving.

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGMENTS
This work was supported by the National Natural Science Foundation of China (62176003, 62088102) and by Beijing Nova Program (20230484362).
REFERENCES
Sajid Anwar, Kyuyeon Hwang, and Wonyong Sung. Structured pruning of deep convolutional neural networks. ACM Journal on Emerging Technologies in Computing Systems, 13(3):1-18, 2017.
Guillaume Bellec, David Kappel, Wolfgang Maass, and Robert Legenstein. Deep rewiring: Training very sparse deep networks. In Proceedings of the International Conference on Learning Repre- sentations, pp. 1-24, 2018.
Abhiroop Bhattacharjec. Ruokai Yin, Abhishek Moitra, and Priyadarshini Panda. Are snns truly energy-efficient? hardware perspective. arXiv preprint arXiv:2309.03388 2023.
Tong Bu, Wei Fang. Jianhao Ding, PengLin Dai. Zhaofei Yu, and Tiejun Huang. Optimal ANN- SNN conversion for high-accuracy and ultra-low-latency spiking neural networks. In Proceedings of the International Conference on Learning Representations, pp. 1-20, 2022.
Yanqi Chen, Zhaofei Yu, Wei Fang, Tiejun Huang, and Yonghong Tian. Pruning of deep spiking neural networks through gradient rewiring. In Proceedings of the International Joint Conference on Artificial Intelligence, pp. 1713- 1721, 2021.
Yanqi Chen, Zhaofei Yu, Wei Fang, Zhengyu Ma, Tiejun Huang, and Yonghong Tian. State transi- tion of dendritic spines improves learning sparse spiking neural networks. In Proceedings of the International Conference on Machine Learning, pp. 3701-3715, 2022.
Sayeed Shafayet Chowdhury, Isha Garg, and Kaushik Roy. Spatio-tempora pruning and quantiza- tion for low-latency spiking neural networks. In Proceedings of the International Joint Conference on Neural Networks, pp. 1-9, 2021.
Sayeed Shafayet Chowdhury, Nitin Rathi, and Kaushik Roy. Towards ultra low latency spiking neural networks for vision and sequential tasks using temporal pruning. In Proceedings of the European Conference on Computer Vision, pp. 709-726, 2022.
Mike Davies, Narayan Srinivasa, Tsung- Han Lin, Gautham Chinya, Yongqiang Cao, Sri Harsha Choday, Georgios Dimou, Prasad Joshi, Nabil Imam, Shweta Jain, et al. Loihi: A neuromorphic manycore processor with on-chip learning. IEEE Micro, 38(1):82-99, 2018.
Mike Davies, Andreas Wild, Garrick Orchard, Yulia Sandamirskaya, Gabriel A Fonseca Guerra, Prasad Joshi, Philipp Plank, and Sumedh R Risbud. Advancing neuromorphic computing with loihi: A survey of results and outlook. Proceedings of the IEEE, 109(5):911-934, 2021.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. ImageNet: A large-scale hierarchical image database. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 248-255, 2009.
Lei Deng, Yujic Wu, Yifan Hu, Ling Liang, Guoqi Li, Xing Hu, Yufei Ding, Peng Li, and Yuan Xie. Comprehensive SNN compression using ADMM optimization and activity regularization. IEEE Transactions on Neural Networks and Learning Systems, 34(6): 2021.
Shikuang Deng, Yuhang Li, Shanghang Zhang, and Shi Gu. Temporal efficient training of spiking neural network via gradient re- reiieighnnng. In Proceedings of the International Conference on Learning Representations, pp. 1-15, 2022.
Jianhao Ding, Zhaofei Yu, Yonghong Tian, and Tiejun Huang. Optimal ANN-SNN conversion for fast and accurate inference deep spiking neural networks. In Proceedings of the International Joint Conference on Artificial Intelligence, pp. 3328-2336 2021.
10

---

Published as a conference paper at ICLR 2024
Chaoteng Duan, Jianhao Ding, Shiyan Chen, Zhaofei Yu, and Tiejun Huang. Temporal effective batch normalization in spiking neural networks. In Advances in Neural Information Processing Systems, pp. 34377-34390, 2022.
Wei Fang, Zhaofei Yu, Yanqi Chen, Tiejun Huang, Timothée Masquelier, and Yonghong Tian. Deep residual learning in spiking neural networks. In Advances in Neural Information Processing Systems, pp. 21056-21069, 2021a.
Wei Fang, Zhaofei Yu, Yanqi Chen, Timothée Masquelier, Tiejun Huang, and Yonghong Tian. In- corporating learnable membrane time constant to enhance learning of spiking neural networks. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 2661-2671, 2021b.
Steve Furber. Large-scale neuromorphic computing systems. Journal of Neural Engineering, 13(5):
051001, 2016.
Steve B Furber, Francesco Galluppi, Steve Temple, and Luis Plana. The spiNNaker project. Proceedings of the IEEE, 102(5):652- (668556666,,666 2014.
Yufei Guo, Weihang Peng, Yuanpei Chen, Liwen Zhang, Xiaode Liu, Xuhui Huang, and Zhe Ma. Joint A-SNN: Joint training of artificial and spiking neural networks via self-distillation and weight factorization. Pattenn Recognition, 142:109633 2023.
Bing Han and Kaushik Roy. Deep spiking neural network: Energy efficiency through time based coding. In Proceedings of the European Conference on Computer Vision, pp. 388-404, 2020.
Song Han, Jeff Pool, John Tran, and William Dally, Learning both weights and connections for efficient neural network. In Advances in Neural Information Processing Systems, pp. 1135-1143, 2015.
Song Han, Xingyu Liu, Huizi Mao, Jing Pu, Ardavan Pedram, Mark Horowitz, and William Dally. Eie: Efficient inference engine compressed deep neural network. ACM SIGARCH Computer Architecture News, (4(3):243-2555 2016.
Torsten Hoefler, Dan Alistarh, Tal Ben-Nun, Nikoli Dryden, and Alexandra Peste. Sparsity in deep learning: Pruning and growth for efficient inference and training neural networks. The Journal of Machine Learning Research, 22(1): 121):10082-11005 2021.
Yangfan Hu, Huajin Tang, and Gang Pan. Spiking deep residual networks. IEEE Transactions on Neural Networks and Learning Systems, 34(8):5200-5205. 2021.
Saeed Reza Kheradpisheh, Maryam Mirsadeghi, and Timothée Masquelier. BS4NN:
binarized spik- ing neural networks with temporal coding and learning. Neural Processing Letters, 12(5): 1273, 2022.
Seijoon Kim, Seongsik Park, Byunggook Na, and Sungroh Yoon. Spiking- YOLO: spiking neural network for energy energy-efficient object detection. In Proceedings of the AAAI Conference on Artificial Intelligence, pp. 11270-11277, 2020.
Youngeun Kim and Priyadarshini Panda. Revisiting batch normalization for training low-latency deep spiking neural networks from scratch. Frontiers in Neuroscience, 15:773954, 2021.
Olga Krestinskaya, Alex Pappachen James, and Leon Ong Chua. Neuromemristive circuits for edge computing: A review. IEEE Transactions on Neural Networks and Learning Systems, (1):442.3, 2019.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Souvik Kundu, Gourav Datta, Massoud Pedram, and Peter A Beerel. thike-thrift: Towards energy- efficient deep spiking neural networks by limiting spiking activity via attention-guided compres- sion. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 3963-3962, 2021.
11

---

Published as a conference paper at ICLR 2024
Mark Kurtz, Justin Kopinsky, Rati Gelashvili, Alexander Matveev, John Carr, Michael Goin, William Leiserson, Sage Moore, Nir Shavit, and Dan Alistarh. Inducing and exploiting acti- vation sparsity for fast inference deep neural networks. In Proceedings of the International Conference on Machine Learning, pp- 5533-5543, 2020.
Ravi Kumar Kushawaha, Saurabh Kumar, Biplab Banerjee, and Rajbabu Velmurugan. Distilling spikes: Knowledge distillation in spiking neural networks. Proceedings of the International Conference on Pattern Recognition, pp. 1536-4544, 2021.
Aditya Kusupati, Vivek Ramanujan, Raghav Somani, Mitchell Wortsman, Prateek Jain, Sham Kakade, and Ali Farhadi. Soft threshold weight reparameterization for learnable sparsity. In Proceedings of the International Conference on Machine Learning, pp. 5544-5555, 2020.
Chankyu Lee, Syed Shakib Sarwar, Priyadarshini Panda, Gopalakrishnan Srinivasan, and Kaushik Roy. Enabling spike-based backpropagation for training deep neural network architectures. Fron- fiers Neuroscience, 14:1 19, 2020.
Hongmin Li, Hanchao Liu, Xiangyang Ji, Guoqi Li, and Luping Shi. Cifarl0-dvs: an event-stream dataset for object classification. Frontiers in Neuroscience, 11:309, 2017.
Yuhang Li, Youngeun Kim, Hyoungseob Park, Tamar Geller, and Priyadarshini Panda. Neuromor- phic data augmentation for training spiking neural networks. In Proceedings of the European Conference on Computer Vision, pp. 631-649, 2022.
Tao Lin, Sebastian U Stich, Luis Barba, Daniil Dmitriev, and Martin Jaggi. Dynamic model pruning with feedback. In Proceedings of the International Conference on Learning Representations, pp. 1-22, 2020.
Jia Luo and Jianxin Wu. AutoPruner: An end-to-end trainable filter pruning method for efficient deep model inference. Pattern Recognition, 107:107461, 2020.
Changze Lv, Jianhan Xu, and Xiaoqing Zheng. Spiking convolutional neural networks for text classification. In Proceedings of the International Conference Learning Representations, pp- 1-17, 2023.
Wolfgang Maass. Networks of spiking neurons: the third generation of neural network models. Neural networks, 10(9):155-16671, 1997.
Huizi Mao, Song Han, Jeff Pool, Wenshuo Li, Xingyu Liu, Yu Wang, and William J Dally. Ex- ploring the regularity of sparse structure in convolutional neural networks. In Proceedings the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshop, pp. 13-20, 2017.
Paul A Merolla, John V Arthur, Rodrigo Alvarez-Icaza, Andrew S Cassidy, Jun Sawada, Filipp Akopyan, Bryan Jackson, Nabil Imam, Chen Guo, Yutaka Nakamura, et al. million spiking- neuron integrated circuit with scalable communication network and interface. Science, 345 (6197):668 6611066666677,,,, 2014.
Byunggook Na, Jisoo Mok, Seongsik Park, Dongjin Lee, Hyeokjun Choe, and Sungroh Yoon. Au- toSNN: Towards energy-efficient spiking neural networks. In Proceedings of the International Conference on Machine Learning, pp. 16253-16269, 2022.
Daniel Neil, Michael Pfeiffer, and Shih-Chii Liu. Learning to be efficient: Algorithms for training low-latency, low-compute deep spiking neural networks. In Proceedings of the ACM Symposium Applied Computing, pp. 293-299222 2016.
Jing Pei, Lei Deng, Sen Song, Mingguo Zhao, Youhui Zhang, Shuang Wu, Guanrui Wang, Zhe Zou, Zhenzhi Wu, Wei He, et al. Towards artificial general intelligence with hybrid tianjic chip architecture. Nature, 572(7767):10--111 2019.
Guanchao Qiao, Ning Ning. Yue Zuo, Shaogang Hu, Qi Yu, and Yecheng Liu. Direct training of hardware-friendle weight binarized spiking neural network with surrogate gradient learning towards spatio-tempora event- event-based dynamic data recognition. Neurocomputing, 457:203-213 2021.
12

---

Published as a conference paper at ICLR 2024
Ning Qiao, Hesham Mostafa, Federico Corradi, Mare Osswald, Fabio Stefanini, Dora Sumislawska, and Giacomo Indiveri. A reconfigurable on-line learning spiking neuromorphic processor com- prising 256 neurons and 128k synapses. Frontiers in neuroscience, 9:141, 2015.
Bodo Rueckauer and Shih-Chii Liu. Conversion of analog to spiking neural networks using sparse temporal coding. In Proceedings of the IEEE International Symposium on Circuits and Systems, pp. 1-5, 2018.
Pedro Savarese, Hugo Silva, and Michael Maire. Winning the lottery with continuous sparsification. In Advances Neural Information Processing Systems, pp. 11380-11399 11390, 2020.
Catherine D Schuman, Thomas E Potok, Robert M Patton, J Douglas Birdwell, Mark E Dean, Garrett S Rose, and James S Plank. A survey of neuromorphic computing and neural networks in hardware. arXiv preprint arXiv: 705.06963, 2017.
Yusuke Sekikawa and Keisuke Uto. Learning sparsify differences of synaptic signal for efficient event processing. In Proceedings of the British Machine Vision Conference, pp. 1-17, 2021.
Abhronil Sengupta, Yuting Ye, Robert Wang, Chiao Liu, and Kaushik Roy. Going deeper in spiking neural networks: VGG and residual architectures. Frontiers in Neuroscience, 13:95, 2019.
Jiangrong Shen, Qi Xu, Jian K Liu, Yueming Wang, Gang Pan, and Huajin Tang. ESL-SNNs: An evolutionary structure learning strategy for spiking neural networks. In Proceedings of the AAAI Conference on Artificial Intelligence, pp. 86-93, 2023.
Evangelos Stromatias, Daniel Neil, Michael Pfeiffer, Francesco Galluppi, Steve B Furber, and Shih- Chii Liu. Robustness of spiking deep belief networks to noise and reduced bit precision of neuro- inspired hardware platforms. Frontiers in Neuroscience, 9:222, 2015.
Sugahara Takuya, Renyuan Zhang, and Yasuhiko Nakashima. Training low-latency spiking neural network through knowledge distillation. In Proceedings the IEEE Symposium in Low-Power and High-Speed Chips. pp. 1-3, 2021.
Guangzhi Tang, Neelesh Kumar, Raymond Yoo, and Konstantinos Michmizos. Deep reinforcement learning with population-coded spiking neural network for continuous control. In Proceedings of the Conference on Robot Learning, pp. 2016-2029, 2021.
Thi Diem Tran, Kien Trung Le, and An Luong Truong Nguyen. Training low-latency deep spiking neural networks with knowledge distillation and batch normalization through time. In Proceed- ings of the International Conference on Computational Intelligence and Networks, pp. 1-6, 2022.
Yixuan Wang, Yang Xu, Rui Yan, and Huajin Tang- Deep spiking neural networks with binary weights for object recognition. IEEE Transactions on Cognitive and Developmental Systems, 13 (3):514-523, 2020.
Ziheng Wang. SparseRT: Accelerating unstructured sparsity gpus for deep learning inference. In Proceedings of the ACM International Conference on Parallel Architectures and Compilation, pp. 31-42, 2020.
Wei Wen, Chunpeng Wu, Yandan Wang, Yiran Chen, and Hai Li. Learning structured sparsity in deep neural networks. Advances in Neural Information Processing Systems, pp. 2074-2082, 2016.
Doudou Wu, Xianghong Lin, and Pangao Du. An adaptive structure learning algorithm for multi- layer spiking neural networks. In Proceedings of the International Conference on Computational Intelligence and Security, pp. 98-102, 2019.
Qi Xu, Yaxin Li, Jiangrong Shen, Jian I. Liu, Huajin Tang, and Gang Pan. Constructing deep spiking neural networks from artificial neural networks with knowledge distillation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7886-7895, 2023.
Hang Yin, John Boaz Lee, Xiangnan Kong, Thomas Hartvigsen, and Sihong Xie. Energy-efficins models for high-ding sigh-dimensional spike train classification using sparse spiking neural networks. In Proceedings of the ACM SIGKDD Conference on Knowledge Discovery & Data Mining, pp- 2017-2025, 2021.
13

---

Published as a conference paper at ICLR 2024
Ruokai Yin, Abhishek Moitra, Abhiroop Bhattacharjee, Youngeun Kim, and Priyadarshini Panda. Sata: Sparsity-award aware training accelerator for spiking neural networks. IEEE Transactions on Computer- -ided Design of Integrated Circuits and Systems, (166):122--13888 2023.
Ruichi Yu, Ang Li, Chun-Fu Chen, Jui-Hsin Lai, Vlad Morariu, Xintong Han, Mingfei Gao, Ching- Yung Lin, and Larry S Davis. Nisp: Pruning networks using neuron importance score propagation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 9194
9203, 2018.
Hanle Zheng. Yujie Wu, Lei Deng, Yifan Hu, and Guoqi Li. Going deeper with firectly-trained larger spiking neural networks. In Proceedings of the AAAI Conference on Artificial Intelligence, pp. 11062-11070, 2021.
Michael Zhu and Suyog Gupta. To prune, or not to prune: exploring the efficacy of pruning for model compression. In Proceedings of the International Conference on Learning Representations Workshop, pp. 1-10, 2018.
Yaoyu Zhu, Zhaofei Yu, Wei Fang, Xiaodong Xie, Tiejun Huang, and Timothée Masquelier. Train- ing spiking neural networks with event-driven backpropagation. In Advances in Neural Informa- tion Processing Systems, pp. 30528-30541, 2022.
Yaoyu Zhu, Wei Fang, Xiaodong Xie, Tiejun Huang, and Zhaofei Yu. Exploring loss functions for time-based training strategy in spiking neural networks. In Advances in Neural Information Processing Systems, 2023.
Tao Zhuang. Zhixuan Zhang, Yuheng Huang, Xiaoyi Zeng, Kai Shuang, and Xiang Li. Neuron-level structured pruning using polarization regularizer. Advances in Neural Information Processing Systems, pp. 9865-9877 2020.
A APPENDIX
A.1 LEARNING PROCESS
The learning process of our proposed method, utilizing stochastic gradient descent (SGD), is sum- marized in Algorithm should be noted that this algorithm can be adapted to other optimizing strategies, such SGD with momentum or Adam, with straightforward modifications.
To enhance readability, we utilize simplified notation in Algorithm Les L., and E
representing specific terms as
Cx(x)= = L(f(z; O w, (Bax))))
(Ala)
C.(x) = [(f(;; (aw)) O w, (ax)))
(Alb)
E o (Baw))l + a (Bax))l1
(Alc)
Here, Le(-) denotes the loss network f with continuous masks. L.(-) denotes the loss of network with binary masks, z is the mini- batch input, and & corresponds to the penalty term.
We also review the meaning of some symbols: ew and e,, are constants, is the parameters of network f. Chp and an, are the re- parameterized masks of weights and neurons, respectively, a--) the sigmoid function, /3 is the scale factor of the sigmoid function, H-)) is the Heaviside step function, O is the element- wise multiplication operation.
A.2 INTRODUCTION AND PREPROCESSING OF DATASETS
CIFAR-10 The CIFAR-10 dataset (Krizhevsky et al. 2009) static image dataset. contains 60,000 samples divided into 10 categories, with 6,000 samples per category. Each category consists of 5,000 training samples and 1,000 testing samples. Each image the dataset has a resolution of 32 x 32 pixels and is in color.
We perform data normalization to ensure that the inputs have zero mean and unit variance. Addi- tionally,
the common data augmentation similar to those used in previous works (Chen al.) 2021 2022; Deng et al.) 2021 Shen et al.) 2023) Zhu et al., (023), i.c., random horizontal flipping and random cropping with a padding of 4, avoid over-fitting
14

---

Published as a conference paper at ICLR 2024
Algorithm 1: Learning Process
Input: Network J. parameters w, re-parameterized masks aw, Oln weight of penalty term X, learning rate initial scale factor 80, final scale factor frr, pruning epochs T,, fine-tuning epochs y. number of batches of training set N
Output: Sparse network f(;; H (aw)
H ((,))
// Initialization
Calculate Ew, en of each weight and neuron.
Initialize w, Qw: Ctni
Initialize 3 = 3p:
// Pruning Phase
for i = to T, do
foreach batch x in training set do
Update parameters w f- w - VV (
Update weight masks Ow t- av 17 (V a (x(() + Va E);
Update neuron masks On t
n a (z(() + V a E);
Update scale factor B B fr (/N-
9o
// Fine-tuninn tuning Phase for i 7 do
foreach batch r in training set do Update parameters ur f- - 1j WWC,(I))
DVS-CIFAR10 The DVS-CIFARIO dataset (Li et al. 2017)
neuromorphic dataset consisting
of event streams, providing rich temporal information. It consists of 10,000 samples divided into 10 categories, with 1, 1,000 samples per category. Each sample is 128. Since the DVS-CIFAR
stream with size of 128 x does not separate samples into the training set and test set, we choose the first 900 samples of each category for training and the last 100 for testing.
For each sample in the DVS-CIFARIU 10 dataset, we split the event stream into 10 equal slices, each containing the same number of events. Then, for each slice, we integrate the events into a single frame and use this frame as the input on that particular time step. Consequently, the time step we use on DVS- WVS-CIFARIO is 10.
Compared to static datasets, neuromorphic datasets typically contain more noise. As result, these datasets are more susceptible to over-fitting than static datasets, particularly for well-trained SNNs. To mitigate overfitting, we adopt the data augmentation technique proposed in (Li et al., 20022), which is also employed by ESLSNN (Shen et al. 2023). For fair comparison, we apply this data augmentation approach to STDS (Chen et al. 2022 in our implementation as well.
ImageNet The ImageNet dataset (Deng et al. 2009) is a large-scale dataset widely used com- puter vision tasks. It consists of approximately 1.2 million high-resolution images, categorized into 1,000 distinct classes, each containing roughly 1,000 images. Each class contains approximately 1,000 images, and encompasses diverse range of objects and scenes, thus reflecting real-world scenarios effectively.
We use the standard preprocessing, i.e.. data normalization, randomly crop and resize the images to 224,224, and random horizontal flipping for data augmentation.
The CIFAR- 100 dataset is an extension of the CIFAR- 10 dataset, designed to provide more challenging and diverse benchmark for image recognition algorithms. It contains 100 dif- ferent classes or categories for classification, encompassing a wide range of objects and concepts compared to the CIFAR-10 dataset's limited set of 10 classes.
CIFAR-100
For the CIFAR-100 dataset, we perform the same data preprocessing techniques as for CIFAR-10 dataset.
15

---

Published as a conference paper at ICLR 2024
A.3 EXPERIMENTAL SETUP OF OUR PROPOSED METHOD
CIFAR-10 For the CIFAR-10 dataset, we use a convolutional SNN structure with 6 convolutional layers and 2 colle-connected layers ([126633) X-xxx333M2222PPP
similar the network proposed in previous work (Fang et al. 2021b). Pruming is applied to all neurons and weights in the network, except for neurons in the fully-connected layers. The batch size and simulation step are set to 16 and 8, respectively. We use the Adam optimizer for optimizing both weights w and masks Oue Orn- We train the model for a total of 1000 epochs with an initial learning rate of rj = 0.0001. The subnetwork is produced at epoch 800, and is then fine-tuned for extra 200 epochs.
DVS-CIFARI0 For the DVS-CIFARIO dataset, we use the VGGSNN structure proposed in previ- ous work (Deng et al. 2022), which consists of 8 convolutional layers and one fully-connected layer (64C3-128C3-APP) x
We select the last 1/10 of each cat- egory as the test set and the first 9/10 as the training set. The batch size and simulation step are set to 64 and 10, respectively. We train the model for a total of 320 epochs, with 240 epochs dedicated to maining-while-runnie
and 80 epochs for fine-tuning. We use the SGD optimizer with an ini- tial learning rate of 1 0.025 and cosine decay to O. Additionally, we apply data augmentation (Li ct al. 2022) and the TET (Deng et al. 2022) loss in all experiments conducted on the DVS-CIFARIO dataset.
ImageNet For the ImageNet dataset, we use the SEW ResNet18 structure proposed by Fang et al. (202 We train the model for a total of 320 epochs, with 280 dedicated to vraining-while-rrmiiee and 40 epochs for fine-tuning. We follow the setup outlined in Fang et al. (2021a), with a batch size of 256, and a simulation step of 4. We use Adam optimizer with an initial learning rate of rj = 0.001 and cosine decay to O.
CIFAR-100 For the CIFAR-100 dataset, we use the SEW ResNet18 structure (Fang et al. 202 with modifications fit different input sizes. Similar to the other datasets, the model is trained for total of 320 epochs, with 280 epochs dedicated to pruning and 40 epochs for fine-tuning. The batch size and simulation step are set to 64 and 10, respectively. We use the SGD optimizer with an initial learning rate of 1 0.025 and cosine decay to O.
Hyperparameters The key hyperparameters in our proposed method are the weight factor of the penalty term X, the initial value of masks agg the initial and final values of the shape factor of the scaled sigmoid function 8n and 3r. We set og = 0, 8o 5, and 3r 1000 in ALL experiments, and adjust the sparsity of the network by varying the value of X.
A.4 DETAILED EXPERIMENTAL RESULTS
Tab. AI liists the experimental results under more sparsity levels, as well as the experimental results
on the CIFAR-100 100 dataset. For CIFAR10 dataset, we add results of X 1 x 10-11 and À
10-10, For the DVS-CIFAR10 dataset, we add the result of À 5 x 10 10-10 We also add experimental results on the CIFAR-100 dataset with À varies from 2 x 10 to 3 10-9. On both the CIFARIO, DVS-CIFARIN 10, and CIFAR 100 dataset, our proposed method can effectively reduce the average SOPs to less than 10M, with accuracy loss of at most 4.1%. This demonstrates that our method can effectively reduce energy consumption.
The detailed results of the ablation study are listed in Tab. A2 Comparing with the results on the CIFAR 10 dataset in Tab. AT our proposed method achieves better performance than both the pruning welght-only method and the pruning neuron-only method.
A.5 EXPERIMENTAL SETUP OF COMPARISON EXPERIMENTS
Tab. A3 provides the general experimental settings for the comparison experiments. Extra hyperpa- rameters for each method can be found in Tab. A4
Note that all the compared methods employ different approaches to control sparsity. For ADMM (Deng et al., (021). we conduct separate experiments with target sparsity levels set to 0.8, 0.9, and 0.95. For GradR (Chen et al. 2021), we set the penalty to 2 x 10-4, 5 10-4, 1 10-3"
16

---

Published as a conference paper at ICLR 2024
Table A1: Detailed experimental results
Dataset
Arch.
Top-1 Acc. (5%
Acc, Loss (%)
Avg. SOPs (M)
Ratio
Conn.
(%)
Neuron (5)
Weight
(%r
(dense)
92.84
776.
100
5 10-12
92.63
-0.21
38.32
20.25
2.97
21.03
34.24
92.28
CIFARIO
Cony, 2
2 10
92.05
00.99
16.47
47.12
1.16
12:27
26.05
11.98
10
90.65
-2.19
8.50
91.31
0.63
8.23
19.34
x 10-10
89.37
-3.47
5.03
154,30
0.38
5.73
13.65
(dense)
82.4
647.30
x
81,9
-0.5
47,81
13.54
6,80
42.73
36.35
DVS-CIFARIO
VGGSNN
3
81,0
-1.4
14.66
44.15
1.95
23.05
25.88
7
79.0
-3,4
10.02
64.60
1.27
18.56
22.56
x 10-4
78.3
6:75
95.90
0.77
14,11
18666
(dense)
74.16
250,91
100
100
100
10
10-10
71.99
-2.17
24.53
10.23
10.36
56.56
27.98
CIFARI00
SEW R
71.43
-2.73
21.16
11.86
8.67
$1.59
25,93
x 10-3
71.31
-2.85
15.54
16.15
6.10
21.96
2 10-8
70.45
-3.71
9.60
26, 14
3.60
28.74
17.07
10
69.41
6.79
(dense)
63.18
1337,87
100
100
100
ImageNet
SEW R
61.89
-1.29
311.70
4.29
28.60
74.86
35,26
10
60.00
-3.18
177.99
7.52
19.31
68.27
27.74
2 10
$8.99
-4.19
116.88
11.45
14.19
51.17
23.52
Table A2: Experimental results of ablation study
Top-1 Acc. (%)
Acc. Loss
Avg. SOPs
Conn. (%)
Weight (%)
Neuron
Method
(%)
(M)
(%)
10-10
92.10
24.90
1.35
17.41
10
91.85
-0,99
18.40
0.94
12.15
Pruning Weight Only
3 x 10-10
91.41
-1.43
15.51
0.74
9,36
100
5 x 10-10
91.04
-1.80
11.86
0.55
6.47
100
1 x 10-9
89.82
-3.02
8.62
0.30
3,85
100
10-10
91.49
-1.35
27.29
2.88
100
4.51
Pruning Neuron Only
10-10
90.71
-2.13
20.47
2.21
100
10-10
89.96
-2.88
16.54
1.83
100
Table A3: General experimental settings
Dataset
Method
Epochs
Optimizer
Learning Rate
Batch Size
Time Steps
ADMM
235
Adam
0.001
50
8
GradR
2048
Adam
0.0001
16
8
CIFARIO
ESLSNN
500
Adam
0.001
64
2
STDS
2048
Adam
0.0001
16
Ours
1000
Adam
0.0001
16
8
ESLSNN
320
SGD
0.1
64
10
DVS-CIFAR10
STDS
320
SGD
0.025!
64
10
Ours
320
SGD
0.025!
64
10
STDS
320
SGD
0.1
256
4
ImageNet
Ours
320
Adam
0.001
256
CIFAR100
Ours
320
SGD
0.025
64
Cosine decay
2 x 10-3,5 10-3, the same as the original settings. Similarly, for STDS (Chen al. 2022).
set the target threshold to 1.0, 2.0, 3.0, 4.0, and 5.0 on CIFARIO dataset and 0.1, 0.6, 0.8, 1.5, 3.0 and 5.0, ImageNet dataset, the same the original settings. For the experiments of STDS
17

---

Published as a conference paper at ICLR 2024
Table A4: Extra experimental settings
Method
Description
Value
Pre-training epochs
200
ADMM re-training epochs
20
ADMM
Hard pruning re-training epochs
15
Penalty coefficient for ADMM
5 x 10
GradR
Target sparsity
0.95
STDS
Threshold scheduler
Sine
Magnitude and SET* Momentum
Pruning rule
ESLSNN
Growth rule
Initial value of masks
Ours
Initial value of scale factor
Final value of scale factor
1000
The original setting 100, we extend the pre-training epochs to improve performance
2 For CIFARIO
For
on RVS-CIFARIO in our implementation, we set to 0.05, 0.2, and 0.5. For ESLSNN set the target sparsity to 0.8, 0.9, 0.95.
(Shen et al,
2023
A.6 VISUALIZATION
(a) sparse neurons
(b) sparse weights
Figure A1: Visualization of sparse neurons and weights
In this section, we visualize the sparse structure obtained by pruning the network using our method. Fig. (a] illustrates the sparsity pattern of the encoding layer neurons in the sparse CifarNet (the 6 Conv, FC network used on the CIFAR10 dataset; we refer to this network as CifarNet in the following) with
111 In the figure, each small square represents a 32 x 32 feature map, and the color of each pixel represents the state of neuron. Red indicates active (non-pruned) neuron, while white indicates pruned neuron. Fig. AT(b) illustrates the sparsity pattern of the weights in the first convolutional layer of the same sparse CifarNet. Each corresponds to input channel, and each column corresponds to an output channel. The x 3 little square at each intersection corresponds to convolution kerne..
18

---

Published as a conference paper at ICLR 2024
From Fig. Al(a) observe that the neurons in the encoding layer form self-organizing multi- resolution structure. Some channels are nearly dense, with only a few neurons pruned. These channels have the full resolution and retain the most information. Some channels have moderate level of sparsity, with around 50% to 25% of neurons remaining, resulting in reduced resolution. In some channels, only few neurons remain, capturing higher-level information lower resolutions. Redundant channels have been completely removed.
Understanding the sparse structure of the convolutional kernels is challenging. However, we can observe that the convolutional kernels corresponding to the completely pruned channels are also fully pruned.
Figure A2: Visualization of sparse neurons and weights in each convolution layer
Fig. A2 visualize the sparse structure of the other layers. From left to right, the first to fifth convolu- tional layers are shown in order. It can be that the difference in sparsity between channels tends to be smaller for the deeper layers. This is because, for deeper layers, the output represents higher- level feature maps that longer exhibit multi-resolution structure. Consequently, the sparsity patterns become more homogeneous across channels.
A.7 HYPERPARAMETER ANALYSIS
B. 1.-100 1000
91.5
80
8,=200 200
8o=1
5000
91.0
90.5
$9.5
$9.0
$9.0
10
30
10
40
Aug.
Ang.
(a) Different 8r with
(b) Different So with 3r 1000
Figure A3: Comparison of performance under different By and So
In the main text, set fo and By 1000 providing without further explanation. In this sec- tion, we investigate how these hyperparameters, Bn and By, affect the performance of our proposed method.
Our experimental setup is as follows. We train CifarNet on the CIFARIO dataset for a total of 150 epochs, with 100 epochs dedicated pruning and 50 epochs for fine-tuning. We vary one
19

---

Published as a conference paper at ICLR 2024
Table A5: Performance comparison between the proposed method and other pruning methods
ImageNet dataset
Top-1 Acc.
Acc, Loss
Avg. SOPs
Conn. (5)
Param. (M)
Pruning Method
Arch.
Ratio
(%)
(56
(M)
59.48
-3.74
402.75
3.33
29.25
1.94
ADMM
SEW ResNer18
55.85
-7.37
262.39
5.11
21.04
1.24
50.57
-12.65
173.10
7.76
15.68
0.75
62.84
-0.34
1082.60
1.24
77.54
7.42
61.51
-1.67
512.95
2,61
34.79
2,38
61.30
-1.88
424.07
3.15
29.29
1.94
STDS
SEW ResNet18
59.93
-3.25
273.39
21.08
1.24
58.06
-5.12
182.49
7.33
15.71
0.75
56.28
-6.90
134.64
9.94
12.95
0.52
61.89
-1.29
311.70
4.29
28.60
3,93
This work
SEW ResNet18
60.00
-3.18
177.99
7.52
19.31
3.10
58.99
-4.19
116.88
11.45
14.19
2.62
hyperparameter while keeping the other fixed. Specifically, we set to 5 x 10- 10 10-11, 2 x 10 and 10-1,, respectively, assess the performance under different sparsity levels. Other settings are inherited from the experiments in the main text and have been detailed above.
We first analyze the effect of different values of 3r performance. As shown in Fig. ((a) observe that a smaller value of 3y, such as Br = 200, yields slightly better performance under low sparsity conditions (A
10 12). However, it leads to significant degradation under high sparsity conditions (A 5 x 10 5.70M SOPs with 81.64% accuracy, not shown in the figure). This degradation occurs because a too-small BT prevents the scaled sigmoid function from adequately approximating the Heaviside step function. Consequently, there is substantial gap between pruning and fine-tuning due the replacement of the not fully converged continuous masks with binary masks. On the other hand, 3T 5000 achieves equal or slightly better performance under high sparsity conditions (A
10- 11 and À
10 10-11), but slightly impairs performance under and À = 1 10-11).
low sparsity conditions (A = 5 x 10 12
fact, 30 has less impact on performance. As shown in Fig. 3(b) we observe that 30 1, 3o 5, and 9o 25 achieve nearly identical performance when À 5 x 10-12 or x 10 10-11, Similar to the case of Br, using a small value of 3o leads to a slight degradation under high sparsity conditions. It remains unclear why there is slight degradation for 3 = and 25 when À = x 10
Next, we examine the impact 30 on performance.
In summary, the key point is avoid using excessively small values of Br for high sparsity levels. Furthermore, Su exhibits greater robustness compared to 3r, indicating that careful adjustment of 3o is unnecessary.
A.8 COMPARISON ON IMAGENET DATASET
Tab. A5 lists detailed comparison ImageNet dataset between the proposed method and other pruning methods including ADMM and STDS. Experimental results show that our method achieves comparable performance while significantly reducing energy consumption compared STDS and ADMM.
A.9 DIFFERENCE BETWEEN ANNS AND SNNS IN UNSTRUCTURED NEURON PRUNING
The reason why unstructured neuron pruning is more suitable for SNNs lies in the sparse computing manner of SNNs and neuromorphic hardware. In ANNs, neurons primarily function as stateless activation functions, such as Sigmoid or ReLU. These activation functions typically yield limited output sparsity. Consequently, ANNs heavily rely on dense matrix computations and are commonly implemented on hardware architectures optimized for dense computing, such GPUs. These chitectures are not designed to effectively exploit the unstructured sparsity of neurons. In contrast, SNNs employ sparse event-driven computing their neurons communicate with each other by fir-
20

---

Published as a conference paper at ICLR 2024
ing sparse, discrete spikes. Thus, SNNs are typically implemented neuromorphic hardware that is highly optimized for sparsity. Therefore, unstructured neuron pruning is more suitable for SNNs.
It is important to note that the above analysis is only for the common case. In fact, some work has successfully utilized sparsity in ANNs by designing novel hardware (Han et al. 2016), utilizing AVX instructions on CPUs (Kurtz et al.) 2020), or designing new algorithms for GPUs (Wang) 2020)_ We highly appreciate these efforts. However, is crucial to note that these methods of exploiting sparsity are not widely adopted yet. Therefore, the analysis presented above is generally reasonable as it applies to most cases.
A. DETAILED DISCUSSION ON HARDWARE SUITABILITY OF OUR ENERGY MODEL
In the following, we discuss in detail which architectures are suited to use SOPs as an energy con- sumption metric and which are not.
First, we would like to emphasize the significance of SOPs as an important metric in both the hard- ware and software domains. Many works neuromorphic hardware provide the energy consump- tion per SOP (or equally, energy per synaptic spike op, energy per connection. SOPs per second per Watt (SOPS/W), etc.)
important metric for assessing hardware energy efficiency. Notable examples include Loihi (Davies et al. 2018) (23.6pl per synaptic spike op), TrueNorth (Merolla et al. 2014) (46 GSOPS/W (typical), 400 GSOPS/W (max)), and TianJic (Pei al. 2019) (649 GSOPS/W (max)). As stated in (Furber 2016). "Since synaptic processing dominates the system energy cost, this (denotes energy per (synaptic) connection) is the best guide the overall energy ef- ficiency of the system." For this reason, many SNN works use SOPs as the metric for estimating their energy consumption estimation, such as ESLSNN (Shen et al. (023) compared in our manuscript. Therefore, it is reasonable to use SOPs as meaningful energy consumption metric.
While SOPs is an important metric, we recognize that it does not cover all aspects of hardware energy consumption like memory access and data movement. Therefore, for architectures where other aspects energy consumption, such as memory accesses, dominate and are not proportional to SOPs, it may not be accurate to use SOPs two types of these architectures. The first is the memory access-dominated architectures, such as SATA (Yin et al. 2023). As demonstrated in (Bhattacharjec et al., (023), the energy consumption associated with memory access in SATA is significantly higher than that of computing and signifi- cantly depends the dataflow. The second is the analog-based architectures such (Qiao al., 2015). Architectures with analog synapses typically consume much more energy to generate a spike compared to synaptic computation, thus their energy consumption may not be proportional SOPs.
the only metric of energy consumption. We list
However, we believe that there are architectures that match our energy model due to their full ex- ploitation of sparsity. By fully utilizing sparsity, the energy consumption of the whole system may be reduced to a level that is linear to the sparsity of synaptic operations, if the other aspects of en- ergy consumption are also sparsely triggered by the synaptic operations. Loihi (Davies et al. 2018) is exactly such kind of architecture. Loihi is highly optimized neuromorphic chip designed for efficient neuromorphic computing. incorporates two key features to enhance energy efficiency: local memory access and asynchronous spike communication. By colocating memory and comput- ing within the core, Loihi minimizes the need for global memory access and reduces data travel distances. Although the detailed architecture design of Loihi has not been made public, the Loihi team claims that Loihi's neuromorphic mesh and cores are built with asynchronous circuits and, at the transistor level, communicate event-driven tokens of information between logic stages, which helps utilize the sparsity (Davies et al.) 2021). While there is no direct evidence specific to SNNs, experimental results other tasks indicate that the energy consumption of Loihi is roughly pro- portional to the computing. We believe that a high degree of optimization that exploits sparsity in memory access, data communication, and computation is key point in making the overall energy consumption roughly linear with the amount of computation.
21


Reviews:
[{"review_id": "xlh582dq0G", "reviewer": "Reviewer_gonx", "summary": "The paper proposes a framework towards unstructured pruning of weights and neurons in Spiking Neural Networks for high sparsity and significant reduction in spikes. Masks are learned during training to trade off accuracy loss and energy consumption. Experiments are provided to demonstrate competitive reduction in spiking operations with higher accuracy than previous works.", "questions": "In Table 2 (comparison with state-of-the-art), the accuracy for GradR-CIFAR10 and STDS-ImageNet are obtained from the original papers, but the percentage of connections reported differs from the original papers. Please clarify these differences (i.e. if there is a difference in definition).", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The authors provide a sound approach towards learned pruning for energy at high-granularity. Experimental details including hyperparameter sensitivity and ablation are addressed and elaborated in sufficient detail. From their results, the proposed method is demonstrated to achieve a competitive accuracy-energy trade-off compared to recent works. Lastly, the paper is clearly written with enough details provided for reproducibility.", "weaknesses": "1.The central idea doesn't appear to offer a breakthrough. The hyperparameter search may not sufficiently differentiate this work. Although combining energy factors into training is a novel approach for SNNs, it has previously been utilized in other contexts, such as artificial neural networks, for various objectives. Such as in the following paper:\n\nSalehinejad, H., & Valaee, S. (2021). Edropout: Energy-based dropout and pruning of deep neural networks. IEEE Transactions on Neural Networks and Learning Systems, 33(10), 5279-5292.\n\n2. The comparison with the state-of-the-art, especially concerning the ImageNet dataset, lacks comprehensiveness. Only one previous work is considered.\n\n3. While the proxy measure of energy-efficiency using synaptic operations (SOPs) is justified from prior neuromorphic hardware works, the paper could improve its claim for energy efficiency by providing actual energy saved in implementation on neuromorphic hardware. \n\n4. In paragraph 3 of Section 5.2, the proposed energy model is claimed to be validated by energy estimates of GradR and STDS. However, as the energy measure used (SOPs) is itself defined by the energy model, this claim seems circular.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "JnH7qXyOBi", "reviewer": "Reviewer_iYS5", "summary": "This paper is dedicated to improving the energy efficiency of SNNs. It defines the energy consumption model and explores a new route to optimizes energy efficiency directly. The authors propose a fine-gained pruning framework that combines unstructured weight and neuron pruning, along with a novel energy penalty term to address the ill-posed problem of jointly pruning neurons and weights. The paper demonstrates the effectiveness of the proposed methods on various datasets and shows that they outperform existing state-of-the-art methods in reducing energy consumption while maintaining comparable performance.", "questions": "1. For Figure 1, additional details regarding the experimental settings and result analysis should be added in the appendix.\n\n2.Why there is not a element-wise multiplication for $m_n$ in Equation (6)? Why it is different from the masks of weights $m_w$?\n\n3.Before Equation (6), it appears that the order may be reversed. Specifically, $m_w$ should be the masks of weights and $m_n$ should be the masks of neurons.", "limitations": "", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.This paper presents a pioneering effort in directly optimizing the energy efficiency of SNNs, offering novel ideas for reducing energy consumption in neuromorphic computing.\n\n2.The conclusions are impressive. The authors highlight the fact that having fewer weight (parameters) does not necessarily translate to lower energy consumption, which prompts us to reevaluate the efficacy of pruning methods in SNNs. Many existing pruning works may not effectively reduce energy consumption.\n\n3.The structure is clear and easy to follow.\n\n4.The experimental results show that the proposed method can achieve the state-of-the-art balance between accuracy and energy efficiency.", "weaknesses": "1.Some portions of the paper lacks clarity. For example, Figure 1 is not sufficiently elucidated. The authors illustrate in the motivation section that Figure 1 shows how fine-grained pruning significantly reduces synaptic connections, but it is challenging to grasp. Furthermore, some variables are not clearly defined, such as $d_n$ and $d_w$ in Equation (6).\n\n2.There are some grammar errors that require improvement. \nPage 4: 'better trade-off' should be 'a better trade-off',\nPage 7: 'refers to' should be ' refer to',\nPage 9: 'outperformes' should be 'outperforms'.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "m0mbpEVRTA", "reviewer": "Reviewer_BChF", "summary": "This paper describes a means of improving the energy consumption of SNNs via irregular pruning, that includes pruning of nodes as well as weights. The reduction in number of synaptic operations is significant with only small drops in accuracy.  The method involves adjusting the loss function with several reasonable approximations.", "questions": "1. Which accelerator architectures match well to using SOP as a metric and which do not?", "limitations": "", "rating": 3, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "The reports results are strong and the paper is very well written with clear explanations of the approach taken.  The ability to target a given sparsity level is a plus.", "weaknesses": "The cost function associated with number of synaptic operations ignores the cost associated with managing the sparsity. In CNNs it is well known that it is hard to take advantage of irregular non-structured sparsity (see e.g., https://arxiv.org/pdf/1907.02124.pdf) and this is somewhat well understand in SNNs as well. Many hardware accelerators claim to try to take advantage of sparsity, but the cost associated with managing the sparsity reduces the overall energy benefits.  See https://arxiv.org/pdf/2309.03388.pdf for an example of a good paper that identifies a high-level energy model of SNNs which captures this issue by more accurately capturing the associated memory cost of managing an SNN. \n\nThe proposed approach ignores these issues and only uses SOPs as a metric of energy consumption. At the minimum, I would argue that the paper needs to appreciate that this metric is not accurate for many hardware accelerators and highlight which architectures this metric is most suited. I hypothesize, for example, SOPs is more reasonable for many core processors like Loihi (which do not focus on weight re-use) and less accurate for architectures where dataflow such as weight stationary play a key role.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "KX5SxGQNmq", "reviewer": "Reviewer_tCqk", "summary": "Low energy consumption is a main property of spiking neural networks. This paper focuses on enhancing energy efficiency of spiking neural networks. The authors first propose the energy consumption model of SNNs and then try to optimize the energy efficiency by introducing the unstructured weight and neuron pruning framework with masks. Besides, a new energy penalty term is proposed to solve the ill-posed problem. Experimental results on CIFAR10, DVS-CIFAR10 and ImageNet datasets demonstrate the proposed method can achieve the SOTA energy efficiency with comparable performance.", "questions": "Can the authors explain why unstructured neuron pruning is not (commonly) used in ANNs?  How to optimize the first term of the loss function in Eq.11? How to choose the parameter $\\beta$ for different datasets?", "limitations": "", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "The paper is the first application of unstructured neuron pruning to deep SNNs. The combination of unstructured weight pruning and unstructured neuron pruning is interesting and efficient. Besides, the authors propose some insight methods to address the ill-posed problem when jointly pruning neurons and weights under energy constraints.", "weaknesses": "1. The difference between ANNs and SNNs in pruning (especially unstructured neuron pruning) is not fully explained. It is not clear why unstructured neuron pruning is not used in ANNs. Why is it more suitable for SNNs?\n2. The authors mainly illustrate how to optimize the penalty, but there was no explanation of how to optimize the first term of the loss function in Eq.11.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (spotlight)