Activity Pruning for Efficient Spiking Neural Networks
Authors: Tong Bu
Xinyu Shi
Zhaofei Yu
Venue: NeurIPS 2025 poster ()
Keywords: spike activity pruning
efficient model
Spiking Neural Network
Primary area: neuroscience_and_cognitive_science
TL;DR: A spike activity pruning method for spiking neural network

Abstract: While sparse coding plays an important role in promoting the efficiency of biological neural systems, it has not been fully utilized by artificial models as the activation sparsity is not well suited to the current structure of deep networks. Spiking Neural Networks (SNNs), with their event-driven characteristics, offer a more natural platform for leveraging activation sparsity. In this work, we specifically target the reduction of neuronal activity, which directly leads to lower computational cost and facilitates efficient SNN deployment on Neuromorphic hardware. We begin by analyzing the limitations of existing activity regularization methods and identifying critical challenges in training sparse SNNs. To address these issues, we propose a modified neuron model, AT-LIF, coupled with a threshold adaptation technique that stabilizes training and effectively suppresses spike activity. Through extensive experiments on multiple datasets, we demonstrate that our approach achieves significant reductions in average firing rates and synaptic operations without sacrificing much accuracy. Furthermore, we show that our method complements weight-based pruning techniques and successfully trains an SNN with only 0.06 average firing rate and 2.22M parameters on ImageNet, highlighting its potential for building highly efficient and scalable SNN models. Code is available at https://github.com/putshua/Activity-Pruning-SNN.

The human brain is remarkably energy-efficient, yet capable of performing wide range of complex tasks, such as reasoning and planning. One of the key factors behind this efficiency lies in the behavior of the neurons. As general strategy of the neural representation in the neural system (Yoshida and highly energy-efficient Activity Pruning for Efficient Spiking Neural Networks
Tong Bu Institution for Artificial Intelligence School of Computer Science
Xinyu Shi Institution for Artificial Intelligence
School of Computer Science Peking University xyshi@pku. edu, cn
Peking University putong30@pku. edu.
Zhaofei Yu
Institution for Artificial Intelligence
School of Computer Science
Peking University yuzf 12@pku edu.
Abstract
While sparse coding plays an important role in promoting the efficiency of biologi- cal neural systems, has not been fully utilized by artificial models as the activation sparsity is not well suited to the current structure of deep networks. Spiking Neural Networks (SNNs), with their event-driven characteristics, offer a more natural platform for leveraging activation sparsity. In this work, we specifically target the reduction of neuronal activity, which directly leads to lower computational cost and facilitates efficient SNN deployment on Neuromorphic hardware. We begin by analyzing the limitations of existing activity regularization methods and identifying critical challenges in training sparse SNNs. To address these issues, we propose a modified neuron model, AT-LIF, coupled with threshold adaptation technique that stabilizes training and effectively suppresses spike activity. Through extensive experiments on multiple datasets, we demonstrate that our approach achieves significant reductions in average firing rates and synaptic operations without sacrificing much accuracy. Furthermore, we show that our method com- plements weight-based pruning techniques and successfully trains SNN with only 0.06 average firing rate and 2.22M parameters on ImageNet, highlighting its potential for building highly efficient and scalable SNN models. Code is available at
Introduction
Ohki 2020), the flexibility of sparse coding enables neurons to operate manner. For example, despite the large population of neurons in specific areas of the brain tasked with particular functions, only subset of the neuron populations will be engaged simultaneously. Experimental evidence shows that sparse representation of visual sensory information in the primary visual cortex enhances the selectivity and sparseness of individual neurons, thereby improving the computational efficiency of the visual system (Vinje and Gallant 2000
Corresponding author
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

SWN
Pruning Methods
Spike Activity Pruning
Regularzzation
Threshol
AT-LIF
Redurra siclvity and 50P%
Pruring Weighte
Pruning Nourons
Pruning Nothity
Figure 1: generally reduce model complexity, activity-basee pruning is particularly effective for SNNs. The Right part shows the basic idea of our method, where the adaptive neurons (AT-LIF) are applied to regularize the threshold, so as to reduce the firing rate while stabilizing the training process.
The left figure illustrates three pruning techniques for deep models. While all methods
A similar principle can be applied to artificial computational models through activation-based pruning, which seeks to moderate neuron activations to mirror the efficiency in sparse neural coding and gain efficiency in terms of time and energy consumption. However, activation pruning has not emerged as a mainstream approach for reducing network complexity compared to more impactful pruning methods that target network weights structure [Kurtz et al. 2020], since the modest unstructured sparsity induced in activation maps seldom translates into meaningful efficiency gains with artificial neural networks on modern hardware. In contemporary architectures, the primary computational workload is carried out via matrix multiplications, where lowering a few activation values does not significantly cut down computation time resource usage.
Unlike traditional architectures, activity-based pruning offers a promising approach for brain-inspired models (Li al., 2024). such as spiking neural networks (SNNs). SNNs are well-established type of neural network that emulate the behavior of biological neurons and are increasingly recognized as more efficient alternative to conventional computational models (Maass] 1997.. One important characteristic of the spiking neural networks is that it encodes information through discrete binary spikes. Therefore, the reduction of the neural activity in SNNs will directly lead to the sparsification of the output spike vector and the reduction of the firing rate, leading to potential computational efficiency. is possible to create highly sparse network in which only a few neurons will be engaged in one round of the computation with spike activity pruning, thus directly reducing the number of operations of the model. Thanks to the rapid recent advancements of neuromorphic hardware (Peil et al. [201 DeBole et al., 2019 Davies et al., 2018 Nieves and Goodman [202]] Fang et al.) 2020 Zenke and Neftei 202 Yao et al. 20024), such sparse property can be effectively leveraged by specific neuromorphic hardware to work in an efficient and energy-saving manner, making these networks
ideal for deployment in coourre-conttrained environments. Even on traditional architectures such as GPUs and deep learning accelerators, it is possible that such models can still benefit from activation sparsity to achieve memory savings and inference speedup. This approach enables the creation of models that are capable of performing complex tasks with minimal energy consumption. leading to more brain-like AI systems terms of efficiency.
In this paper, we investigate an activity pruning technique aimed at suppressing the spike output. Our approach explores the capabilities of novel neuron model in solving the learning dilemma in sparse models and stabilize the training process. We also introduce a threshold adaptation mechanism to effectively reduce neural activity. Our contributions can be summarized as:
We specifically target the reduction of neuronal spike activity in SNNs, which directly lowers computational cost and serves as a unique advantage for efficient computation. To achieve this goal, we analyze the limitations of existing activity regularization methods and identify critical challenges associated with training sparse SNN models.
We propose novel neuron model, AT-LIF, designed stabilize the training of sparse spiking networks while effectively suppressing spike activity. It incorporates a surrent-based output and adaptive threshold mechanism, which help alleviate the gradient vanishing problem and resolve optimization conflicts.
Extensive experiments demonstrate that our method significantly reduces spike activity and synaptic operations while maintaining performance comparable to the baseline methods. We further combine our proposed method with weight pruning strategies to illustrate its potential for achieving highly sparse SNNs.

---

2 Background
2.1 Related works
Network pruning is a well-established technique for artificial neural networks, with various approaches designed to reduce model complexity and improve efficiency. These pruning methods can typically be categorized into three types: unstructured weight pruning [Han et al. 2015 Zhu and Gupta 20188)
Kusupati et al.] 22220] structured weight pruning [Wen et al. |016 Mao et al., 2017| Anwar et al. 2017]. and structured neuron pruning (Yu et al. 2018 Zhuang et al., 2020]. Similar exploration
has been conducted with SNNs, where specific supervised learning algorithms are applied to prune weights or neurons [Wu et al.) 2019 Yin et al. 2021]. Most distinguishing SNN pruning methods are inspired by biological principles, aiming to replicate the brain's 's efficient learning scheme to obtain sparse connected deep SNNs [Bellec et al., [ et al., 2021 2022 Shen et al. 2023). The others draw inspiration from existing deep learning techniques and focus on jointly sparsifying the connections and neurons, introducing a complete pruning framework on deep SNNs [Deng et al.
202T Shi al. 2024). Notably, while various pruning methods have been proposed for spiking neural networks Yin et al. [202T), [Wu et al. (2019). Zhou et al. (2024), most focus on weight or neuron pruning rather than developing activity- based pruning approaches depth. The majority of the existing pruning methods rely on adding regularization terms to constrain the firing rate of
individual neurons (Deng et al. 2021 Yan et al. 2022], or regularizing the activation value in artificial neural networks before converting it into SNN [Neil et al. 2016]. In fact, direct regularization of spike activity actually conflicts with the optimization goal, leading to a trade-off in performance and sparsity, these methods are either limited to shallow networks or fail to effectively train highly sparse models [Narduzzi et al. 2022 Sakemi et al. 2023).
While pruning techniques are essential for training efficient SNNs, they cannot be separated from the underlying learning process. Specialized optimization techniques are required to ensure both efficient learning of the target function and eliminating as much redundant structure as possible. Most learning-based approaches (Bohte researches that focus on the pruning of SNNs adopts the tiriiionnttthhtiinntt (BPTT) algorithm [Wu et al. 2000 or more specifically, the RNN-like backpropagation-tee al. 2018 J. In this method, the surrogate gradient approximation is proposed to smooth the non- differentiable firing function in the neuron [Lee et al. [016 Shrestha and Orchard 2018 Fang et al., Neftei et al. 2019 Zenke and Vogels 2021 Stewart and Neftci, 2022]. Some researches also adopted the event-driven backpropagation that is able to maintain the sparsity of the gradient [Fang]
2021a
cet al.
2021b] Kim and Panda| [020 Zheng et al. [021 Lee et al. 2020 Deng et al., 2022 Mostafa
2017 Bohte et | Zhu et al. |222 Zhang et al. 2021 Another commonly used SNN training
methods are ANN-SM SNN conversion approaches [Cao et al.
2015). which map the weights of a pre-trained ANN into an rate-coding SNN [Dichl et al.]| 2015 Sengupta et al.) [209 Han et al. 22200
Deng and Gu 2021 Ding et al. 2021 Bu et 20,2 Meng et 2022]. ANN-SNN conversion
the most practical training method to train SNNs large-scale datasets since converted SNNs
always have outstanding performance [Kim et al. 2020
Hao et al. [023b]a]
2.2 Neuron model
In this paper, we use the widely adopted Letky-lntegrare-and-- model for SNNs [Zhikevich) 2004 Gerstner et al. 2014]. The discretized dynamics of membrane potential can be summarized by
x; [ = wiys, [4] + bi,
m& [t] = ru,(t - 1] + [r.
8:(1] [4] Hmm[[4] 0+),
u [t] = - (m, 0,)) -m/t),
where j and indicate the two adjacent neurons and Way and b: represent the weight and bias of the
connection from neuron j to neuron At arbitrary time-step t, the current input of neuron i is 2; []. m,[t] and u,(t] represent the value of the membrane potential before and after neuronal firing H()) is the Heaviside step function. The neuron will emit a spike, denoting by the 0-1 scaler 8; [1] when the membrane potential mi[!] reaches the firing threshold 0.. is the leaky parameter and if we explicitly set 7 to 1, the neuron will degenerate to the non-leaky model. We demonstrate the hard-reset function in Eq. 4 that reduce the neuron membrane potential to the resting potential O.
3

---

2.3 Spike activity pruning in SNNs
general method for learning sparse activated models with activation regularization involves modify- ing the loss function with a penalty term that promotes sparsity in the network's 's activations. There are various techniques to implement such regularization over neuron activation, including applying
L./1.2 regularization over the average spike rate per neuron NNeil et al., 2016 Yan et al. 2022 Deng (et al., J. or using other regularizers [Narduzzi et al. 2022]. With the constraint of minimizing
the spike activity, the general loss function of such activity-based pruning methods consists of both learning objectives for the specific task and the regularization term over the spike activity, which
M
N
Liotal =
M Σ
Clack (f(x)), y)o) + X Σ ||s.
(5)
We use Clack (term in the left) and Grey (term in the right) denotes the task-specific loss and activation regularization loss, respectively. M represents the number of training samples and N are total neuron numbers. 4g Y/ denotes the k-th data-label sample from the training dataset. f()) is the function that represents the input-output mapping of the model. The regularization term is A, Σ (æ*))l is the summation of the p-norm of the current output spike vector with the penalty coefficient parameter A. where vector s; (2*) represents the output spike pattern of neuron across all time-steps when the model receives 23 as input. Many existing activity-based pruning methods for spiking neural networks can be considered as specific instances of this general framework, as they apply similar principles to add regularization based on their activity levels. However, using only the activation regularization technique cannot achieve highly sparse model. While activation regularization can directly promote sparsity by adding constraints on parameters, this approach often leads to a trade-off between efficiency and accuracy. As the penalty term increases or the model becomes sparser, performance degradation becomes more severe, limiting its practical effectiveness.
3 Method
3.1
Learning dilemma of better performance and sparser spikes
The trade-off between the model performance and activation sparsity in artificial neural networks has been widely recognized as a challenge [Li et al. [003] Georgiadis] 2019 Kurtz et al.) [2020]. In SNNs, this issue is even more severe due to the sparser activations and inaccurate gradient estimation when using a surrogate gradient. Here we summarize two key challenges that significantly hinder the learning process in the joint optimization of task-specific loss and regularization loss. These two issues impede the effective optimization, limiting both the efficiency and performance of the model.
Conflicts optimization target The first issue that hinders the learning process is the dilemma between learning more sparse spike activations and learning non-zero representations from the input data, as the optimization objectives of Ltosk and Lreg are inherently contradictory. Let assume that in the i-th neuron, when the model receives the k-th data sample Tg from the training dataset, ÿ; is the optimal output feature representation for neuron that we attempt to learn. The regularization over the spike patterns are L2 norm. Under such assumption, the learning objective can be rewritten
M
N
Liotal
Σ
Σ
[zx() - 00112 + Ao Σ
| (2xx (2x2)1
(6)
M
M
N
(1))((=))) - 0il1 + Xa (x 0113)
M
A= 1=1
The Eq. iillustrates fundamental conflict between the target loss and the regularization term in the learning process. One the one hand, minimizing the target loss typically encourages the model to capture intricate patterns in the data and learning non-zero feature representations. The activity regularization, on the other hand, encourages sparser spike patterns and less spike output. Therefore, there is an obvious trade-off between effectively learning inner representation from input data and reducing spike activity at each layer. The reduction of the spike disrupts the representation learning at each layer since the regularization term inhibits spike firing. The trade-off between the

---

two objectives makes it challenging to simultaneously optimize for both effective representation learning and minimal spike activity.
Gradient vanishing problem with sparse activity The other issue that impedes the learning process of sparse SNNs comes from the learning process when using gradient-based learning. It has been observed that the gradient vanishing problem may significantly degrade the performance of the SNN when using Backpropagation- -aean-TThouuhTTT Time (BPTT) [Fang al., 2021a] Furthermore, this issue becomes even more severe and prevalent when activity regularization is applied, as the sparsity of spike activity will further induce gradient vanishing and prevent the model from convergence. To illustrate this issue, we first provide a theoretical analysis showing that excessive regularization of the average firing rate can cause the training process to be trapped
saddle point.
Theorem 1. We define
0} as the set of all parameler sets such that the output spike at arbitrary layers and time-steps is all zero, then the S is a set of saddle points for the optimization of the total loss Liotel. which is
{ s st({]
@Lrotal Swiy
Vi
Σ
st(t] +0
- 0,
(8)
2 rrotal Owy
= 0,
Awsy E S
(9)
S denotes the set of all parameter sets such that the output spike at an arbitrary layer and time is all zero. As described in the Eq. 9 we can prove that, in practice, the S is set of critical points for the optimization of the total loss Ezotal, since the gradient of the weight vectors with respect to the total loss Ltotall is all zero and normally it not the global minima of the total loss. Therefore, any w E S is a saddle point of the optimization problem, and the gradient will converge to zero when the spike output is zero. Detailed proof will be provided in the supplementary material.
This theorem identifies a set of saddle points in the loss function, highlighting a critical issue in the optimization process. When regularizing spike activity, the model might easily fall into the local minima with extremely sparse activity. In this case, the learning process will stop since the gradient comes to zero, and the optimization process will be trapped at the saddle point. This phenomenon emphasizes the need for carefully designed regularization strategies to avoid gradient vanishing and maintain effective learning dynamics.
3.2 Adaptive threshold LIF neuron
In the previous section, we identify two key issues that hinder the training of the sparse activated spiking neural networks from two perspectives: the conflict in optimization target and the potential gradient vanishing problem during optimization. Therefore, in this section, to alleviate the impact of the learning dilemma, we provide a feasible solution by introducing the Adaptive Threshold Neuron (AT-LIF), which includes: 1) introducing the adaptation mechanism to the neuron threshold; 2) using current-based output with 0-9 value instead of 0-1 spike output.
With the adaptation mechanism, the threshold value 0 evolves throughout training and adapt to the spike frequency at each iteration. The threshold thus becomes a learnable parameter during the training process and plays a role of both controlling the firing frequency of the neurons and the efficacy of transmitting outputs to the next layer:
og [t] = 0, (m,(tt] - 0.) = 0,st[t], 2 [t] = Wit b,.
(10)
(11)
We use o,[t] E {0, 0.} to represent the output of the AT-LIF neuron instead of the s;[t] in the common LIF neuron (Eq. D which scales up the neuron output while the average activity is low. Note that this mechanism does not imply graded output spikes, as the scaling by the threshold value does not alter the binary nature of the spike outputs. During inference, when the thresholds are fixed, the scaling can be absorbed into the subsequent layer's weights, ensuring full compatibility with all types of neuromorphic hardware. Intuitively, increasing the threshold reduces the firing frequency while preserving the average output value. Before introducing the detailed adaptive algorithm, we first demonstrate how threshold adaptation gradually reduces the neuron activity (Fig. 2 In this toy example, continually add constant input to the AT-LIF neuron and vanilla LIF neuron and compare
5

---

40
Memnbrane Vollage mi]f]
Membrarie Voltage (n(()
30
Threshold f
30
Threshoid Value
Average Output
Average Output
20
20
10
10
0
10
20
30
40
50
60
70
10
20
30
40
70
80
Time (ms)
Time (ms)
Figure 2: Left: Behavior of the vanilla LIF neuron: Right: Behavior of the AT-LIF neuron. In both figures, the black curve represents the change of the membrane potential, while the green curve represents the threshold value and the blue curve represents the average neuronal output over time.
both the output and average firing rate of these two different types of neurons. In the AT-LIF model, threshold will increase slightly after each output spike, while the threshold of the vanilla LIF will remain constant. Such adaptation of the threshold helps reduce the firing rate while preventing drop of the average output of the neuron. These results demonstrate the potential the AT-LIF model to suppress output spiking activity while preserving stable output dynamics and faithful neuronal representation of input information.
3.3 Threshold adaptation for sparse activity
In practice, we incorporate the threshold adaptation mechanism into the Backpropagation-TTrove Time (BPTT) algorithm [Wu et al. 2018) by treating the threshold at each layer as learnable parameter during training. In order to constrain the spike activity, regularize the threshold value to minimize the neuron firing rate. To constrain spike activity, we regularize the threshold values by minimizing the firing rate at each layer. Specifically, the thresholds are optimized to minimize both the task-specific loss and the activity regularization loss (arg ming Ltotat), while the synaptic weights are updated solely to minimize the task-specific target loss. Based on this formulation, derive the threshold update rule at each update iteration in Eq. 12
(iLtask 00,
do, [t]
Ap: = (1000
+ A9
sy
(12)
00:
where T' is the inference time-steps for the SNN model and 17 the learning rate during the training 8a, can be estimated by the surrogate function. To distinguish from the regularization- process. 80;
based method over average spike activity, we use Ag to represent the coefficient parameter of the threshold adaptation. In practice, consider each 0; learnable parameter that is involved in the training process, while also manually regularize its value at each iteration according to Eq. [2] We also demonstrate the forward/hackwart information flow for AT-LIF neuron in right part of the Fig 3 The full derivation and pseudo-code of the overall algorithm are provided in the supplementary material.
As for the surrogate function, we consider the triangle-shaped function as surrogate of the Heaviside function [Esser al., [0166] to estimate the derivative of the firing function of the AT-LIF neuron (Eq. 13 To prevent gradient explosion, the upper bound the surrogate function is set to while the design principle of the triangle-shaped surrogate function is strictly followed. Leveraging this surrogate, along with the proposed learning framework and adaptation rules for AT-LIF, we are able train high-performance spiking neural networks with sparser activation.
do.[1]
do:
max (1
|m
0.
0
(13)
3m
00;
0,
3.4 Stabilizing training with AT-LIF neuron
In this section, we demonstrate how the AT-LIF neuron addresses two key challenges that limit the performance of sparse SNN models. Specifically, the AT-LIF neuron resolves the inherent conflict

---

Firing Func.
regularize
Surrogate Func.
fire
charge
reset
Area=0
forward
2h W
backward
-0
0
Figure 3: Left: The surrogate function for AT-LIF neuron. The blue curve represents the firing function while the green curve represents the surrogate derivative. Right: Illustration the forward pass and backward pass of the AT-LIF neuron using the Back- Propagation-Throuee Through-Time Time algorithm.
between competing learning objectives and mitigates the gradient vanishing problem encountered when training highly sparse networks.
Solving the conffict of optimization target
As discussed in the above section, training sparse spiking neural networks typically involves jointly optimizing the task objective and the sparsity constraint as 0, w = arg mitlg Ltotals which leads to substantial conflict on optimization target. In our method, however, we introduce the AT-LIF neuron, which outputs a scaled binary signal rather than a standard binary spike.
This scaling mechanism helps maintain a steady average neuron output while simultaneously reducing the firing rate (as illustrated in Fig 2). This property allows us decouple the learning process while also maintaining stable learning process. Specifically, instead of jointly optimize the Ltotal. we separate the optimization by learning the weight parameters based on the task loss as ur arg min Ctask while simultaneously regularize the threshold to reduce spike activity as 0 arg ming Liotal-
The conflict does not necessarily arise for learning of weight parameters as the model can still effectively learn from the output representation while we simultaneously maximize the threshold to promote sparser firing events. Consequently, the inherent confiitt between the optimization objectives can, in principle, be resolved when adaptive threshold neurons are employed. As result, the network can effective learning representations while reducing spike counts during the optimization process.
Layer
Layer
Layer
Layer 4
Gradent Norm
mam
Maad
Mranh
nn
Layer
Layer 6
Layer 7
Layer 8
Gradient Norm Norm
Mill Wac m
mardy
Mnan
Epochs
Activation Regularization
AT-LIF
Figure 4: The 2-norm of the gradient value in the first 8 Convolutional layer.
Alleviating the gradient vanishing problem
On the other hand, the gradient vanishing problem will be alleviated since the gradient will no longer necessarily approach zero as the threshold value increases. One possible theoretical explanation is that the saddle point assumption (Theorem [ no longer holds true as we continually increase the threshold rather than directly constrain the spike activity. We further empirically demonstrate this phenomenon by comparing the change of the gradient value with respect to the training iterations in different activity pruning methods before and after the adoption of the AT-LIF neuron. We trained multiple SNN models using AT-LIF neuron with threshold adaptation and vanilla LIF neuron with activation regularization CIFAR-10 dataset with ResNet-20 structure and selected two models
7

---

with similar average firing rates after training for fair comparison. Here the model using AT-LIF achieved 91.06% accuracy and 0.043 average firing rate, while the model using vanilla LIF only achieved 88.31% % accuracy and larger average firing rate of 0.044. We demonstrate the comparison of the gradient values of the two models at each training iteration (Fig- 4. The green curve represents the normalized gradient 2-norm value of the weight vector in the corresponding convolution layer in the model with vanilla LIF and activation regularization, while the blue curve represents that of the model adopting AT-LIF neuron. Here the blue curve is always above the green curve and in most layers, the gradient value of the model using activation regularization much closer to zero while the gradient value in the model with AT-LIF neuron still maintains consistency across layers. This indicates that using AT-LIF can effectively mitigate the gradient vamishing problem. Moreover, this is also confirmed by the fact that the final training results of the AT-LIF model achieve better performance while achieving similar sparsity.
Experiments
4.1 General experimental setting
We provide the general hyper-parameters and the neuron parameter setting in here and the supple- mentary material for better reproducibility. Following the previous study [Chen al.) 2022 Shi et aL. 2024], we select similar architecture for evaluation, including 6 Conv, 2FC (CIFAR- 10), We also utilize the
ImageNet- 100), and VGGSNNNDVS-CIFANN
standard ResNet-20 for the CIFAR dataset [He et al. 2016).
To demonstrate the effectiveness of the pruning method, will use the following metrics in the below table, including classification accuracy (Ace.), average firing rate (Avg. FR), total parameters (Params) and total synaptic operation numbers (SOPs). The definition for the average firing rate
= Σ E'e 8; (;(x)) (t). where F& are data samples from the test set and 8; (;( spike value in each neuron at each time-step with given input æy. The synaptic operation (SOP) quantifies the total number of computational events spiking neural networks, corresponding to the number of times a spike is transmitted across synapse (Hu et al. 2018 Fang et al. 2021a] Additionally, we report the number of parameters left after applying weight-based pruning.
the
4.2 Compare with existing methods
Dataset
Arch.
Method
T
X
Ag
Acc.(%)
Avg FR.
SOPs(M)
ResNet-20
Vanilla
8
0
0
92.89
0.236
256.69
ResNet-20
AR
8
0.01
0
91.31
0,060
57.26
ResNet-20
AR
8
0.02
0
88.31
0.044
41.86
CIFAR-10
ResNet-20
AT-LIF
1c-3
92.05
0.053
61.70
ResNet-20
AT-LIF
2c-3
91.06
0.043
48.85
ResNet-20
AT-LIF
5e-3
88.31
0.033
39.33
ResNet-20
AT-LIF
8
0.01
le-3
90.70
0.025
29.23
SEW-ResNet18
Vanilla
4
0
83.44
0.138
1276.42
AR
4
0.01
0
82.94
0.083
840.90
ImageNet-100 100
SEW-ResNet18
AR
4
0.05
0
82.26
0.043
469.00
SEW-ResNet18
AT-LIF
0.01
1c-4
82,40
0.038
463.65
SEW-ResN EeW-ResNet18
AT-LIF
0.02
2e-4
81.12
0.025
328.34
VGGSNN
Vanilla
10
0
82,70
0.080
1066.40
VGGSNN
AR
10
0.005
0
81.30
0.055
673.06
DVS-CIFARIO
VGGSNN
AT-LIF
10
5e-4
82.20
0.043
626.57
VGGSNN
AT-LIF
10
0.001
Se-4
82.10
0.039
587.68
Table 1: Comparison of AT-LIF and direct activation-regularizzttoo---- methods CIFAR-10. ImageNet-100, and DVS-CIFAR datasets.
In this section, we compare our proposed model with other existing SNN pruning approaches to evaluate its performance and highlight its advantages. Here we will present two tables for fair

---

comparison. The first table compares our pruning method with existing activity regularization approaches, all of which focus only on reducing spike activity or average firing rate. The second table reports results under the setting of unconstrained structural pruning, where activity- -aaed pruning combined with connection-level or neuron-level pruning methods. Specifically, we integrate our method with the weight pruning approach STDS [Chen et al.] 2022 to demonstrate its potential to learn highly efficient SNNs.
Spike activity pruning
Since existing activity regularization methods primarily focus on penalizing spike activity, we adopt LI spike regularization as representative regularization-B based approach for comparison. Tab. presents comprehensive evaluation of our proposed AT-LIF model (AT-LIF) against two baselines: the vanilla model ( (Vanilla) and vanilla model trained with spike activity regularization (AR), across three benchmark datasets: CIFAR-10. 10, ImageNet- 100, and WVS-CIFARIO. The columns "X," " and "Xo" are the regularization coefficients for the spike activity term (Eq. 7 and the coefficient for the threshold adaptation mechanism, respectively. The column refers to the simulation timestep for SNN.
Across all tasks, the AT-LIF neuron demonstrates strong capability reduce spike activity and computational cost while maintaining high classification accuracy compared with regularization- based methods. On CIFAR-10, AT-LIF achieves a similar accuracy to the unpruned Vanilla model, while reducing the average firing rate by approximately 89.4% and lowering SOPs by nearly 88.6%. Compared activation regularization, AT-LIF achieves lower SOPs and firing rates while maintaining higher accuracy. Similar trends are observed on the ImageNet- 100 dataset, where AT-LIF maintains competitive accuracy with 1 % accuracy loss while significantly reducing SOPs and average firing rate. If sacrificing 2.3% more accuracy is acceptable, the reduction of average firing rate on the AT-LIF model will be more than 5.5 times. On the event-based DVS-CIFAR 10 dataset, AT-LIF again achieves efficient spike suppression with substantial reductions in SOPs (up to 45% fewer than Vanilla) and firing rates, while preserving accuracy. These results collectively highlight the effectiveness of AT-LIF in achieving sparse, energy-efficient SNNs without sacrificing much performance, outperforming conventional activity regularization techniques.
Dataset
Arch.
Pruning
Method
Acc.(%)
Avg FR.
Param(M)
SOPs(M)
90.71
0.072
0.12
45.48
6 Cony, FC
Conn
STDS
89.40
0.059
0.07
28.38
91.74
0,051
9.48
15.15
CIFAR-10
6 Conv, FC
Neu;Conn
Unstru.
90.76
0.049
6.93
8.49
90.72
0.024
0.19
30.11
6 Conv, FC
Act;Conn
AT-LIF
90.20
0.020
0.09
16.88
SEW-ResNet18
Conn
STDS
58.90
0.189
1.84
461.26
ImageNet
SEW-ResNet18
Neu;Conn
Unstru.
59.23
0.106
4.38
271.98
SEW-ResNet18
Act;Conn
AT-LIF
59.61
0.063
2.22
310.53
Table 2: Comparison of the composite AT-LIF method with existing SNN pruning frameworks.
Pruning both activity and weights We also integrate our method with the STDS [Chen al. 2022] to achieve the pruning of both weights and neuron activity. STDS operates by reparameterizing the weights and progressively eliminating those with small absolute values during training, leading to a sparse connectivity pattern. Our integration is straightforward: we replace the standard LIF neurons with AT-LIF neurons throughout the network, and apply threshold adaptation as described in our method. Mcanwhile, the STDS pruning algorithm is used to train the synaptic weights and induce weight sparsity. This combination allows us to simultancously achieve both activity sparsity and weight sparsity. We compare the composite approach with reproduced implementations of existing SNN pruning frameworks, including unstructured weight pruning (STDS) and unstructured weight and neuron pruning (Unstru.) (Shi et al. 2.24]. We compare the above methods in accuracy, average firing rate, parameter numbers and total SOPs. The final results are demonstrated in Tab. 2
On C00 the composite AT-LIF method achieves comparable accuracy to both STDS and Un- stru., while significantly reducing the average firing rate. Notably, AT-LIF achieves the lowest average firing rate at 0.020 and reduces SOPs to 16.88M, representing substantial gain in computational

---

efficiency. On ImageNet, AT-LIF again achieves the lowest firing rate (0.063) and favorable balance between accuracy and efficiency, with SOPs reduced compared STDS (461. 2)M) and close to those achieved by the unstructured method (271.98M) 1.99M). Overall, the results highlight the strength of AT-LIF in achieving efficient sparse SNNs with low energy cost, while maintaining performance across datasets and architectures. Compared with the Unstru. that aims to reduce SOPs and STDS that aims to reduce parameters, although our method is not optimal in every individual metric, it strikes an effective balance between spike sparsity and weight sparsity, achieving the best performance across these two key indicators. This highlights the flexibility and effectiveness of the proposed AT-LIF model in enabling both activity and weight sparsity when combined with weight pruning techniques.
4.3 Ablation study
An ablation study was conducted to evaluate the impact of specific components of the AT-LIF neuron model on the overall performance the SNNs. We systematically remove or alter the two components mentioned in Sec. 3.2 and therefore evaluate their individual contributions. The Tab. demonnssssss
CurO.
Adap.
Ae
Acc.(%)
Avg. FR
SOPs(M)
x
x
0.0
92.89
0.236
256.69
v
x
0.0
93.32
0.237
258.95
x
le-3
50.99
0,110
126.13
1e-3
92.05
0.054
61.70
Table 3: Ablation study for AT-LIF neuron on CIFAR-I0 dataset.
the model performance and efficiency under different neuron configurations. Here column "CurO." represents whether the neuron has 0-1 output 0-9 output while column "Adap." indicates whether the threshold adaptation mechanism enabled. The model without the current-based output feature fails to converge when combined with the adaptation mechanism, indicating that the resulting sparse spike output hinders effective model learning. Models that applied the adaptive threshold exhibit significant efficiency advantage over the with the vanilla LIF neuron, with notable reduction in overall computational cost. The model with AT-LIF neuron achieves the most outstanding results across all experiments spike sparsity and energy cost. Results show that both current-based based output and threshold adaptation are crucial for maintaining an optimal trade-off between minimizing excessive spike activity and preserving the model's 's ability to learn effectively.
four times more efficient compared with the baseline model in terms of
Conclusion and limitation
In this paper, we propose a novel technique to significantly reduce spike activity in SNNs while preserving their performance. By effectively suppressing spike events, our method substantially promotes the sparsity of spike outputs and decreases required synaptic operations. As the proposed method focuses reducing spike activity, it is also compatible with weight pruning or other model compression techniques. The proposed method potentially enables efficient mapping onto neuromorphic hardware Limitation While this work introduces the concept of exploiting the sparse spike activity of SNNs, still remains at the algorithmic level. The next step may involve exploring practical hardware implementations and evaluating the effectiveness of the approach in real-world applications.
modern GPU architectures.
Acknowledgement
This work is funded by National Natural Science Foundation of China (62422601, U24B20144 and (2088102), Beijing Municipal Science and Technology Program (Z241 100004224004) and Beijing Nova Program (20230484362, 20240484703), and National Key Laboratory for Multimedia Information Processing.
10

---

References
Sajid Anwar. Kyuyeon Hwang, and Wonyong Sung. Structured pruning of deep convolutional neural networks. ACM Journal on Emerging Technologies Computing Systems, 2017.
Guillaume Bellec, David Kappel. Wolfgang Maass, and Robert Legenstein. Deep rewiring: Training very sparse deep networks. International Conference on Learning Representations. 2018.
Sander M Bohte, Joost N Kok, and Johannes A La Poutré. Spikeprop: backpropagation for networks of spiking neurons, In European Symposium on Artificial Neural Networks, 2000.
Tong Bu, Wei Fang. Jianhao Ding, PengLin Dai. Zhaofei Yu, and Tiejun Huang. Optimal ANN-SNN conversion for high accuracy and ultra-low-latens spiking neural networks. In International Conference on Learning Representations, 2022.
Yonggiang Cao, Yang Chen, and Deepak Khosla. Spiking deep convolutional neural networks for energy-efficient object recognition. International Journal of Computer Vision, 2015.
Yanqi Chen, Zhaofei Yu. Wei Fang. Tiejun Huang. and Yonghong Tian, Pruning of deep spiking neural networks through gradient rewiring. In International Joint Conferences on Artificial Intelligence, 2021.
Yanqi Chen, Zhanfei Yu, Wei Fang, Zhengyu Ma, Tiejun Huang, and Yonghong Tian. State transition of dendritic spines improves learning of sparse spiking neural networks. In International Conference on Machine Learning. 2022.
Mike Davies, Narayan Srinivasa, Tsung-Han Lin, Gautham Chinya, Yongqiang Cao, Sri Harsha Choday, Georgios Dimou, Prasad Joshi, Nabil Imam, Shweta Jain, et al. Loihi: A neuromorphic manycore processor with on chip learning. IEEE Micro, 2018.
Michael V DeBole. Brian Taba, Arnon Amir. Filipp Akopyan, Alexander Andreopoulos, William P Risk, Jeff Kusnitz, Carlos Ortega Otero, Tapan K Nayak, Rathinakumar Appuswamy, al. TrueNorth: Accelerating from zero to 64 million neurons 10 years. Computer, 2019.
Lei Deng. Yujie Wu, Yifan Hu, Ling Liang, Guoqi Li, Xing Hu, Yufei Ding, Peng Li, and Yuan Xie. Compre- hensive snn compression using admm optimization and activity regularization. IEEE Transactions on Neural Networks and Learning Systems, 2021.
Shikuang Deng and Shi Gu. Optimal conversion of conventional artificial neural networks to spiking neural networks. In International Conference on Learning Representations, 2021.
Shikuang Deng, Yuhang Li, Shanghang Zhang, and Shi Gu. Temporal efficient training of spiking neural network via gradient re-weighting. In International Conference on Leurning Representations, 2022.
Peter U Diehl, Daniel Neil, Jonathan Binas, Matthew Cook, Shih-Chii Liu, and Michael Pfeiffer. Fast- classifyyng, high-accuracy spiking deep networks through weight and threshold balancing. In International Joint Conference on Neural Networks, 2015.
Jianhao Ding, Zhaofei Yu, Yonghong Tian, and Tiejun Huang. Optimal ANN-SNN conversion for fast and accurate inference in deep spiking neural networks. In International Joint Conference on Artificial Intelligence, 2021.
Steven K. Esser, Paul A. Merolla, John V. Arthur, Andrew S. Cassidy, Rathinakumar Appuswamy, Alexander Andreopoulos. David J. Berg. Jeffrey L. McKinstry. Timothy Melano, Davis R. Barch, Carmelo di Nolfo, Pallab Datta, Aron Amir, Brian Taba, Myron D. Flickner, and Dharmendra S. Modha. Convolutional networks for fast, energy-efficient ncuromorphic computing. National Academy of Sciences, 2016.
Biao Fang, Yuhao Zhang, Rui Yan, and Huajin Tang. Spike trains encoding optimization for spiking neural networks implementation in fpga. In International Conference on Advanced Computational Intelligence. 2020.
Wei Fang, Zhaofei Yu, Yanqi Chen, Tiejun Huang, Timothée Masquelier, and Yonghong Tian. Deep residual learning in spiking neural networks. Advances in Neural Information Processing Systems, 2021a.
Wei Fang, Zhaofei Yu, Yangi Chen, Timothée Masquelier, Tiejun Huang, and Yonghong Tian. Incorporating learnable membrane time constant to enhance learning of spiking neural networks. In International Conference on Computer Vision, 2021b.
Georgios Georgiadis. Accelerating convolutional neural networks via activation map compression. In Computer Vision and Pattern Recognition, 2019.
11

---

Wulfram Gerstner, Wemer M Kister, Richard Naud, and Liam Paninski. Neuronal dynumics: From single neurons networks and modely of cognition. Cambridge University Press, 2014.
Bing Han, Gopalakrishnan Srinivasan, and Kaushik Roy. RMP-SNN: Residual membrane potential neuron for enabling deeper Recognition, 2020.
and low-latency spiking neural network. Computer Vision and Pattern
Song Han, Jeff Pool, John Tran, and William Dally, Learning both weights and connections for efficient neural network. In Advances in Neural Information Processing Systems, 2015.
Zecheng Hao, Tong Bu, Jianhao Ding, Tiejun Huang, and Zhaofei Yu. Reducing ann-snn conversion erro 2023a.
through residual membrane potential. arXiv preprint
Zecheng Hao, Jianhao Ding, Tong Bu, Tiejun Huang, and Zhaofei Yu. Bridging the gap between amns and smns by calibrating offset spikes. In International Conference on Learning Representations, 2023b.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Computer Vision and Pattern Recognition, 2016.
Yangfan Hu, Huajin Tang, and Gang Pan. Spiking deep residual networks. IEEE Transactions on Neural Networks and Learning Systems, 2018.
Eugene M Izhikevich. Which model use for cortical spiking neurons? IEEE Transactions on Neural Networks. 2004.
Scijoon Kim, Scongsik Park, Byunggook Na, and Sungroh Yoon. Spiking-YOLO: energy-efficient object detection. In AAAI Conference on Artificial Intelligence, 2020.
YOLO: Spiking neural network for
Youngeun Kim and Priyadarshini Panda. Revisiting hatch normalization for training low-latency deep spiking neural networks from scratch. Frontiers in Neuroscience, 2020.
Mark Kurtz, Justin Kopinsky, Rati Gelashvili, Alexander Matveev, John Carr, Michael Goin, William Leiserson, Sage Moore, Nir Shavit, and Dan Alistarh. Inducing and exploiting activation sparsity for fast inference on deep neural networks. International Conference on Machine Learning. 2020.
Aditya Kusupati, Vivek Ramanujan, Raghav Somani, Mitchell Wortsman, Prateek Jain, Sham Kakade, and Ali Farhadi. Soft threshold weight reparamcterization for learnable sparsity. International Conference on Machine Learning, 2020.
Chankyu Lee, Syed Shakib Sarwar, Priyadarshini Panda, Gopalakrishnan Srinivasan, and Kaushik Roy. Enabling spike-based backpropagation for training deep neural network architectures. Frontiers in Neuroscience, 2020.
Jun Haeng Lee. Tobi Delbruck, and Michael Pfeiffer. Training deep spiking neural networks using backpropaga- tion. Fromtiers in Neuroscience, 2016.
Guoqi Li, Lei Deng. Huajin Tang, Gang Pan, Yonghong Tian, Kaushik Roy, and Wolfgang Maass. Brain-inspiree computing: systematic survey and future trends. Proceedings of the IEEE, 2024.
Zonglin Li, Chong You, Srinadh Bhojanapalli, Daliang Li, Ankit Singh Rawat, Sashank J Reddi, Ke Ye, Felix Chern, Felix Yu, Ruiqi Guo, et The lazy neuron phenomenon: On emergence of activation sparsity in transformers. In International Conference on Learning Representations, 2023.
Wolfgang Maass. Networks of spiking neurons: the third generation of neural network models. Neural Networks, 1997.
Huizi Mao. Song Han, Jeff Pool, Wenshuo Li. Xingyu Liu, Yu Wang, and William J Dally. Exploring the granularity of sparsity in convolutional neural networks. In Computer Vision and Pattern Recognition Workshops, 2017.
Qingyan Meng. Shen Yan, Mingqing Xiao, Yisen Wang, Zhouchen Lin, and Zhi- Luo. Training much deeper spiking neural networks with a small number time-steps. Neural Networks, 2022.
Hesham Mostafa. Supervised learning based on temporal coding in spiking neural networks. IEEE Transactions on Newral Networks and Learning Systems, 2017.
Simon Narduzzi, Siavash A Bigdeli, Shib-Chii Liu, and L Andrea Dunbar. Optimizing the consumption of spiking neural networks with activity regularization. In International Conference on Acoustics. Speech and Signal Processing, 2022.
12

---

Emre O Neftci, Hesham Mostafa, and Friedemann Zenke. Surrogate gradient learning in spiking neural networks: Bringing the power of gradient-based optimization spiking neural networks. IEEE Signal Processing Magazine, 2019.
Daniel Neil, Michael Pfeiffer, and Shih-Chii Liu. Learning to be efficient: Algorithms for training low-latency, low-compute deep spiking neural networks. In ACM Symposium on Applied Computing, 2016.
Nicolas Perez Nieves and Dan F. M. Goodman. Sparse spiking gradient descent. In Advances in Neural Information Processing Systems, 2021.
Jing Pei, Lei Deng. Sen Song, Mingguo Zhao, Youhui Zhang, Shuang Wu, Guanrui Wang. Zhe Zou, Zhenzhi Wu, Wei He, et al. Towards artificial general intelligence with hybrid tianjic chip architecture. Nature, 2019.
Yusuke Sakemi, Kakei Yamamoto, Takeo Hosomi, and Kazuyuki Aihara. Sparse-firing regularization methods for spiking neural networks with time-to- time-to-frst--piee sprst coding. Scientifie Reports, 2023.
Abhronil Sengupta, Yuting Ye, Robert Wang, Chiao Liu, and Kaushik Roy. Going deeper in spiking neural networks: VGG and residual architectures. Frontiers in Neuroscience. 2019.
Jiangrong Shen, Qi Xu, Jian K. Liu, Yueming Wang. Gang Pan, and Huajin Tang. ESL-SNNs: An evolutionary structure learning strategy for spiking neural networks. AAAI Conference on Artifciil Intelligence, 2023.
Xinyu Shi, Jianhao Ding, Zecheng Hao, and Zhaofei Yu. Towards energy efficient spiking neural networks: An
unstructured pruning framework. In International Conference on Learning Representations, 2024.
Sumit B Shrestha and Garrick Orchard. Slayer: Spike layer error reassignment in time. Advances in Neural Information Processing Systems, 2018.
Kenneth Michael Stewart and Emre Neftci. Meta-learning spiking neural networks with surrogate gradient descent. Neuromorphic Computing and Engineering, 2022.
William E Vinje and Jack L Gallant. Sparse coding and decorrelation in primary visual cortex during natural vision. Science, (8755556)) 2000.
Wei Wen, Chunpeng Wu, Yandan Wang, Yiran Chen, and Hai Li. Learning structured sparsity in deep neural networks. Advances in Neural Information Processing Systems, 2016.
Doudou Wu, Xianghong Lin, and Pangao Du. An adaptive structure learning algorithm for multi-layer spiking neural networks. In International Conference on Computational Intelligence and Security. IEEE, 2019.
Yujie Wu, Lei Deng, Guoqi Li, Jun Zhu, and Luping Shi. Spatio-temporal backpropagation for training high-performance spiking neural networks. Frontiers in Neuroxcience, 2018
Yulong Yan, Haoming Chu, Yi Jin, Yuxiang Huan, Zhuo Zou, and Lirong Zheng. Backpropagation with sparsity regularization for spiking neural network learning. Frontiers in Neuroscience, 2022.
Man Yao, Ole Richter, Guangshe Zhao Ning Qiao. Yannan Xing, Dingheng Wang, Tianxiang Hu, Wei Fang. Tugba Demirci, Michele De Marchi, et al. Spike-based dynamic computing with asynchronous sensing- computing neuromorphic chip, Nature Communications, 2024.
Hang Yin. John Boaz Lee, Xiangnan Kong, Thomas Hartvigsen, and Sihong Xie. Energy-efficient models for high- high-dimensional spike train classification using sparse spiking neural networks. In ACM SIGKDD Conference on Knowledge Discovery & Data Mining, 2021.
Takashi Yoshida and Kenichi Ohki. Natural images reliably represented by sparse and variable populations of neurons in visual cortex. Nature communications, 2020.
Ruichi Yu. Ang Li, Chun-Fu Chen, Jui-Hsin Lai, Vlad Morariu, Xintong Han, Minglei Gao, Ching-Yure Lin, and Larry S Davis. Nisp: Pruning networks using neuron importance score propagation. In Computer Vision and Pattern Recognition, 2018.
Friedemann Zenke and Emre Neftci. Brain- mainn-nnspired learning on neuromorphic substrates. Proceedings of the IEEE, 2021.
Friedemann Zenke and Tim P Vogels. The remarkable robustness of surrogate gradient learning for instilling complex function in spiking neural networks. Neural Computation, 2021.
13

---

Malu Zhang, Jiadong Wang. Jibin Wu. Ammar Belatreche, Burin Amorppaisannon, Zhixuan Zhang, Venkata Pavan Kumar Miriyala, Hong Qu, Yansong Chua, Trevor E Carlson, et al. Rectified linear postsynaptic potential function backpropagation deep spiking neural networks. IEEE Trunsactions on Neurul Networks and Learning Systems, 2021.
Hanle Zheng. Yujie Wu, Deng. Yifan Hu, and Guoqi Going deeper with directly-trainee larger spiking neural networks. In AAAI Conference on Artificial Inteiligence, 2021.
Zhaokun Zhou, Kaiwei Che, Jun Niu, Man Yao, Guoqi Li, Li Yuan, Guibo Luo, and Yuesheng Zhu, Spatial- temporal spiking feature pruning in spiking transformer. IEEE Transactions on Cognitive and Developmental Systems, 2024.
Michael Н Zhu and Suyog Gupta. To prune, or not to prune: Exploring the efficacy of pruning for model compression. In Advances in Neural Information Processing Systems, 2018.
Yaoyu Zhu, Zhaofei Yu, Wei Fang. Xiaodong Xie, Tiejun Huang, and Timothée Masquelier. Training spiking neural networks with event-driven backpropagation. In Advances in Neural Information Processing Systems. 2022.
Tao Zhuang, Zhixuan Zhang, Yuheng Huang, Xiaoyi Zeng, Kai Shuang, and Xiang Li. Neuron-level structured pruning using polarization regularizer. In Advances in Neural Information Processing Systems, 2020.
14

---

NeurIPS Paper Checklist
1. Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
Answer: [Yes]
Justification: We faithfully list the contribution the paper the introduction and also the abstract.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize other settings.
It is fine to include aspirational goals as motivation long it is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the limitation of the proposed work in the discussion section.
Guidelincs:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are violations of
these assumptions (e.g.. independence assumptions, noiseless settings, model well- speciliifccitnon asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets with few runs. In general, empirical results often depend implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures hecause fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If l'applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should that individual actions in favor of transparency play an important role developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
their best judgment and recognize
Theory assumptions and proofs
Question: For each theoretical result. does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification: For each theoretical results, the proof can be either found in the main text supplemen- tary material.
Guidelincs:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross-referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
15

---

model The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by
Formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We will make sure that the paper contain all the information to reproduce the results and we will also open-source the code.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data provided or not.
If the contribution is datasct and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, contribution novel architecture, describing the architecture fully might suffice, or the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., the case of a large language model), releasing checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require relcasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution a new model (e.g.. a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (c.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, which case authors are welcome to describe the particular way they provide for reprodueibility. In the case of closed-source source models, it may be that access to the model is limited in some way (c.g.. to registered users). but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open faithfully reproduce the main experimental results, as described supplemental material?
the data and code. with sufficient instructions
Answer: [Yes]
Justification: will provide open access to the code while the data we use are publicly available.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurTPS code and data submission guidelines lhttpa://nips.cc/pppe cc/public/ htt////ooSueeeeiiooooeoo
for more details.
While we encourage release of code and data, we understand that might not possible,
so "Na" an acceptable answer. Papers cannot rejected simply for not including code, unless this is central to the contribution (c.g., for a new open-source henchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https: : ://nis//..////////// cc/public/ ouiidess/CooeeSttmmiessssoooooo for more details.
16

---

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reprodueible, they should state which ones are omitted from script and why.
At submission time, to preserve anonymity, authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.
Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary understand results?
Answer: [Yes]
Justification:
We provide all the detail in the training and evaluating process in the supplementary
material.
Guidelines:
answer means that paper does not include experiments.
The experimental setting should be presented in the of the paper level of detail that necessary appreciate the results and make sense them.
The full details can provided either with the code, appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate informa- tion about the statistical significance of the experiments?
Answer: [No]
Justification: Due limited computational resources, we report results from single run. However, results are reproducible.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to library function, hootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error har than state that they have 96% CI, if the hypothesis of Normality of errors is verified.
For asymmetric distributions, the authors should be careful symmetric error hars would yield results that are out of range (e.g. negative error rates).
show in tables figures
error bars are reported tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
Question: For each experiment, does the paper provide sufficient information the computer resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Answer: [Yes]
Justification: The information of computer resources for each experiments can be found the supplementary material.
Guidelines:
The answer NA means that the paper does not include experiments.
17

---

The paper should indicate the type of compute workers CPU or GPU, internal cluster. or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethies httph: httpp://neurrpa... httph//rrrrppaccpppaaiee///aaeee httpppplll./.caaaauieeeeeeeeeeee
Answer: [Yes]
Justification:
This research adheres fully to the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
the authors answer No, they should explain special circumstances that require deviation from the Code of Ethics.
authors should make sure preserve anonymity (e.g.. if there is special consideration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
discuss both potential positive societal impacts and negative societal impacts of the Answer: [Yes]
Justification:
research.
Guidelines:
The answer NA means that there is no socictal impact of the work performed.
If the authors answer NA or No, they should explain why their work has socictal impact why the paper does address societal impact.
Examples of negative socictal impacts include potential malicious unintended uses (e.g.. disinformation, generating fake profiles, surveillance). fairness considerations (e.g., deploy- ment technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will foundational research not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it not needed point out that generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms could arise when technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative socictal impacts, the authors could also discuss possible mitigation strategies (c.g., gated relcase of models, providing defenses in addition to attacks, mechanisms for monitor- ing misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g. pretrained language models, image generators, scraped datasets)?
Question:
Answer: [NA]
Justification: The paper poses such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines restrictions access the model or implementing safety filters.
18

---

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but encourage authors take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification:
The creators, original owners of assets properly credited.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include a URL.
The name of the license (c.g., CC-BY 4.0) should be included for cach asset. scraped from particular source (e.g.. website), the copyright terms of service of that source should provided.
assets are released, the license, copyright information, and terms of use the package should be provided. For popular datasets, paperswi thcode. www.coom//aaac /att/tttstts has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.
For existing datasets that are re-per re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
this information not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: New assets introduced in the paper are well documented.
Guidelines:
The answer NA means that the paper does not release new assets.
Rescarchers should communicate the details of the datasct/code/model as part of their sub- missions via structured templates. This includes details about training, license, limitations,
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an amonymized URL or include anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, well as details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsoureing nor research with human subjects.
Guidelincs:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should included the main paper.
According to the NeurIPS Code of Fthics, workers involved data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals equivalent for research with human subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Question:
19

---

Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, approval (or equivalent) may be required for any human subjects research. If you ohtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applica- ble), such the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe usage of LLMs it an important. original. or non-standard component the core methods this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the rescarch, declaration is not required.
Answer: [NA]
Justification: the core method development in this research does not involve LLMs as any important, original, non-standard components.
Guidelines:
The NA that the method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer our LLM policy (https://neur http::////auriss.cc///oocrereees for what should or should not be described.
20


Reviews:
[{"review_id": "rBm2cEqmNQ", "reviewer": "Reviewer_CCTV", "summary": "Motivated by the biological principle of sparse neural coding and the need for efficient computation, the authors propose AT-LIF, an adaptive threshold leaky integrate-and-fire neuron model. AT-LIF dynamically adjusts neuron firing thresholds to suppress spike activity while maintaining output fidelity, effectively overcoming optimization conflicts and gradient vanishing issues in training sparse SNNs. The method is evaluated on CIFAR-10, DVS-CIFAR10 ImageNet across different architectures, demonstrating significant reductions in average firing rates and synaptic operations, while preserving classification accuracy in some cases. These results highlight the method's potential for building scalable and energy-efficient SNNs.", "questions": "None.", "limitations": "None.", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\nThe paper introduces a novel activity-pruning method inspired by sparse coding in the brain. This direction departs from conventional weight-based pruning and aligns well with the natural efficiency of neural systems.\nThe proposed AT-LIF neuron model is both innovative and effective. It addresses two major challenges in training sparse SNNs, the gradient vanishing and optimization conflict, through adaptive thresholding and current-based output mechanisms. The approach demonstrates compatibility with existing weight-pruning techniques. The combined use of AT-LIF with methods like STDS enables both activity and weights sparsity, significantly improving computational efficiency.\n\nWeaknesses:\nThe method still exhibits a clear trade-off between accuracy and sparsity. As shown in the experiments, increasing the threshold adaptation coefficient to induce lower spike activity leads to performance degradation.", "was_revised": true, "final_justification": "The authors have well addressed all my concerns. The proposed method can significantly reduce average firing rates and synaptic operations while preserving classification accuracy, which is promising for neuromorphic chips. I decide to increase my score to 5.", "rebuttal": "# Response to Reviewer CCTV\n> The method still exhibits a clear trade-off between accuracy and sparsity. As shown in the experiments, increasing the threshold adaptation coefficient to induce lower spike activity leads to performance degradation.\n\nWe thank the reviewer for highlighting this point. We agree that a trade-off between model accuracy and spike activity sparsity still exists, particularly when the threshold adaptation coefficient $\\lambda_t$ is increased aggressively. In fact, suppressing spike activity toward near-zero firing rates inevitably constrains the network’s expressiveness and lead to performance degradation. However, we would like to emphasize that our method provides a way to balance such trade-off more gracefully than direct activity regularization methods. \n\nWe compare the AT-LIF method with more existing spike activity pruning methods, including L1 regularization on the firing rate (L1 Reg [1]), Hoyer regularization over the output spike (Hoyer Reg [2]) and the baseline method presented in the manuscript (AR).\n\nAs shown in both Table 2 in the manuscript and the below table, AT-LIF achieves competitive accuracy under moderate sparsity constraints and demonstrates better performance than conventional activation regularization under similar average firing rates.\n\n**ResNet-20, CIFAR-10**\n|          | Accuracy | SOPs (M) | Avg. FR. |\n| -------- | -------- | -------- | -------- |\n| L1 Reg [1]    | 91.27    | 60.21    |  0.062   |\n| Hoyer Reg [2] | 88.62    | 76.79    |  0.081   |\n| AR [Baseline] | 91.31    | 57.26    |  0.060   |\n| AT-LIF [Ours] | 91.06    | 48.85    |  0.043   |\n\nNonetheless, we acknowledge that this trade-off remains a fundamental challenge, especially for extreme sparse models. We will explicitly add this discussion to the *Limitations* section of the revised manuscript.\n\n**References**\n\n[1] Deng, Lei, et al. \"Comprehensive snn compression using admm optimization and activity regularization.\" TNNLS, 2021.\n\n[2] Narduzzi, Simon, et al. \"Optimizing the consumption of spiking neural networks with activity regularization.\" ICASSP, 2022."}, {"review_id": "PyeFbIMxjT", "reviewer": "Reviewer_364k", "summary": "This paper investigates the problem of high neuronal activity in SNNs.\nThe authors identify the limitations of current activity regularization methods and propose a new neuron model, AT-LIF, alongside a threshold adaptation mechanism, to suppress spike activity during training.\nNotably, the method achieves promising results on ImageNet with a very low average firing rate and a compact model size.", "questions": "1. Has the proposed method been evaluated on real neuromorphic hardware (e.g., Loihi, Speck)?\n2. Does the introduction of threshold adaptation increase training complexity or require additional hyperparameter tuning?\n3. Can the authors discuss whether the method can be applied to other domains (e.g., NLP, speech) or large architecture like spiking Transformers?", "limitations": "Yes.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "## Strengths\n1. I think the efficiency gains are clear. The method achieves a substantial reduction in firing rate and synaptic operations.\n2. The authors propose a modified neuron model (AT-LIF) and threshold adaptation to promote sparsity in SNNs, addressing a core inefficiency in SNN deployment.\n3. I think the compatibility with pruning methods suggests broader applicability and system-level optimization potential.\n\n## Weaknesses\nNo obvious weaknesses.", "was_revised": true, "final_justification": "My concerns have been addressed.", "rebuttal": "# Response to Reviewer 364K\n> Has the proposed method been evaluated on real neuromorphic hardware (e.g., Loihi, Speck)?\n\nThanks for your suggestion. We acknowledge that our current work has not yet included an evaluation on real neuromorphic hardware platforms. As discussed in the paper, our study primarily focuses on the algorithmic development and analysis of spike activity pruning at the model level. To estimate the efficiency benefits and potential for hardware deployment, we instead report metrics such as average firing rates and synaptic operation counts (SOPs), following the standard practices in recent literature. These metrics are widely used to estimate energy and computational cost on neuromorphic systems [2,3], and we believe they provide a meaningful estimation of the performance gain achieved by our method.\n    \nNonetheless, we fully agree that empirical validation on real hardware is an important next step. As we discussed in the section *Limitations*, we plan to explore such evaluations in future work, which could offer deeper insights into the real-world applicability of our proposed approach.\n\n> Does the introduction of threshold adaptation increase training complexity or require additional hyperparameter tuning?\n\nWe thank the reviewer for raising this insightful question. Regarding training complexity, we would like to clarify that the introduction of the threshold adaptation mechanism in AT-LIF neurons does not significantly increase the computational burden. Specifically, the threshold adaptation involves a lightweight update to the threshold value at each iteration. Its computational cost is minimal compared to the standard backpropagation process, which involves updating high-dimensional weight matrices. In our implementation on single RTX 4090, we observed no noticeable overhead during training on GPUs. For example, using the ResNet-20 architecture with 32×32 input resolution on CIFAR-10, the vanilla model requires approximately 40.1 seconds per epoch, while the AT-LIF model takes about 40.5 seconds per epoch, which is a negligible increase.\n\nAs for hyperparameter tuning, we acknowledge that our method introduces an additional coefficient $\\lambda_t$, which controls the strength of the threshold regularization. In practice, we use standard strategies such as grid search or binary search over a small range. However, we would like to emphaize that unlike weight pruning, it is hard to directly control the desired sparsity of the spike activity, as different input sample triggers distinct pattern of spikes, resulting in different activation sparsity. Thus empirically setting the regularization coefficient for the penalty term is a more practical and feasible solution. Moreover, in our method, the convergence of the model is more robust to extreme values of the regularization coefficient. For example, with $\\lambda_t=1e-3$, ResNet-20 on CIFAR-10 achieves 92.05% accuracy with 61.7M SOPs. Even when $\\lambda_t$ is increased to $1e-1$, the model is still able to converge, achieving 45.52% accuracy with only 9.19M SOPs and 0.007 average firing rate. In contrast, for direct activation regularization, the model achieves 91.31% accuracy with 57.26M SOPs when $\\lambda_s = 1e-2$, but fails to converge when $\\lambda_s \\geq 0.03$. This highlights the greater stability and flexibility of our method during hyper-parameter selection.\n\nWe will incorporate this discussion into the revised manuscript for better clarity and explicitly mention this in the *Limitations* section.\n\n> Can the authors discuss whether the method can be applied to other domains (e.g., NLP, speech) or large architecture like spiking Transformers?\n\nWe thank the reviewer for this insightful question. We believe that our proposed method is broadly applicable across domains and architectures, including tasks in computer vision and natural language processing. Since the AT-LIF operates at the neuron level and does not rely on task-specific or architecture-specific assumptions, it can be easily integrated into most spiking neural networks.\n\nWe extended our method to a spiking Transformer architecture (QKFormer[1]) and conducted experiments on the CIFAR-10 dataset. The AT-LIF neuron was seamlessly incorporated into the attention blocks, and we observed that the method remained effective in reducing spike activity while maintaining competitive accuracy. As shown in the table below, our method achieved a substantial reduction in average firing rate (from 0.106 to 0.061) and synaptic operation count (SOPs reduced from 444.9M to 281.3M), with only a 0.59% drop in accuracy compared to the vanilla model. These results suggest that the AT-LIF mechanism continues to scale well to Transformer-based architectures, offering significant gains in efficiency with minimal compromise in performance.\n\n|          | Accuracy | SOPs (M) | Avg. FR. |\n| -------- | -------- | -------- | -------- |\n| Vanilla  | 89.86    | 444.9    |  0.106   |\n| AT-LIF   | 89.27    | 281.3    |  0.061   |\n\nWhile we have not yet applied the method to NLP or speech tasks due to limited rebuttal time, we believe the core idea is generalizable and plan to explore these directions in future work. We will include these results and expand the discussion in the revised manuscript.\n\n**Reference**\n\n[1] Zhou, Chenlin, et al. \"Qkformer: Hierarchical spiking transformer using qk attention.\" NeurIPS, 2024.\n\n[2] Hu, Yangfan, Huajin Tang, and Gang Pan. \"Spiking deep residual networks.\" TNNLS, 2021.\n\n[3] Shi, Xinyu, et al. \"Towards energy efficient spiking neural networks: An unstructured pruning framework.\" ICLR, 2024."}, {"review_id": "8aA8GcdjGL", "reviewer": "Reviewer_CMEG", "summary": "The paper aims to improve efficiency of SNN learning while conserving effectiveness. Proposed AT-LIF algorithm reduces spiking activity using sparse regularizer and threshold adaptation. \nThe method is experimentally validated on several datasets widely used in deep learning research.", "questions": "Eq (6) why is the output of the i-th neuron, s_i, a vector? Also, y_hat should have both i and k indices. \n\nThere 1: Why is it derivative in (8) and gradient in (9)? Are (8,9) saying the same thing? Loose index j in (8).\nZero gradient is not exactly the same as a saddle point.\nMaybe give a hint of the proof in the main text.\n\nLine 179: subscript ()_t is never used\n \nLine 196 BPTT requires memory of previous time states? Please clarify.\n\nDoesn’t it make sense to do overall evaluation first, then ablation?\n\nLine 260 what’s synaptic operation numbers?\n\nLine 271: if the threshold \\theta is constant, what’s the difference if the signal is 0-1 or 0-\\theta?\n\nLine 272: what’s synaptic output feature?\n\nLine 281: what is SOTA?\n\nTable 2: some of the best AT-LIF results are with \\lambda_s=0, i.e. no sparsity. Please comment. \nWhat is the column T?\n\nPlease give more details how your method is integrated with other methods in the experiments.", "limitations": "If the method is implemented on a SNN platform, that platform should allow for graded spikes.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Suggested algorithm is original and ingenious. The issue of vanishing gradient is duly addressed. Experiments demonstrate significant reduction of spiking activity with minimal loss of accuracy.\nThe design assumes graded spiking, which I think should be stated upfront.\nThe clarity is lacking, some terminology unclear, and overall leaves impression of hurried writing. This makes it difficult to evaluate the technical content properly.", "was_revised": true, "final_justification": "Authors response to my comments was detailed and convincing.", "rebuttal": "# Response to Reviewer CMEG\n\n> The clarity is lacking, some terminology unclear, and overall leaves impression of hurried writing.\n\nWe appreciate the reviewer’s efforts to help improve the quality of our manuscript. We will carefully revise the entire manuscript to enhance clarity.\n\n> Clarify for the graded spike in AT-LIF.\n\nWe sincerely thank the reviewer for pointing this out and apologize for the ambiguity in our original description. We would like to clarify that our method does not necessarily rely on graded spikes. While we replace the 0/1 spike output with a 0/θ value for stability and optimization purposes, the output remains binary in nature, since the neuron either fire or do not. Importantly, all neurons within the same layer share a single threshold value, which allows the scaled output to be removed during inference by absorbing the scaling factor into the synaptic weights of the subsequent layer. Therefore, the final deployed model does not require the SNN platform to support graded spikes and remains fully compatible with most event-driven neuromorphic hardware. We will make this clearer in the revised manuscript to avoid potential confusion.\n\n> Eq (6) why is the output of the i-th neuron, s_i, a vector? Also, y_hat should have both i and k indices.\n\nWe thank the reviewer for these helpful observations.\n\nIn Equation (6), $s_i(x_k)$ is treated as a vector because it represents the spike outputs of the i-th neuron across all time steps when processing the k-th input sample. Each neuron's output is a sequence of binary spike events over time. We will add description in the manuscript.\n\nWe agree that $\\hat{y}_i$ should depend on the input sample and should include both the neuron index i and the input index k. Thank you for pointing this out. We will correct the notation in the revised version to ensure clarity and consistency.\n\n> There 1: Why is it derivative in (8) and gradient in (9)? Are (8,9) saying the same thing? Loose index j in (8).\n\nThank you for pointing this out. We apologize for the confusion caused by Equations (8) and (9), and we will revise them in the manuscript to improve clarity. Specifically, the use of partial derivative notation in (8) and gradient notation in (9) was inconsistent. As $w_{ij}$ is a scaler, we will revise the Equation (9) to derivative.\nAlso index i for $s_i[t]$ in Equation (8) is incorrect and may lead to confusion. Since $w_{ij}$ denotes the weight from neuron j to neuron i, the corresponding spike output should be indexed by j, not i. We will correct this in the revised version by replacing it. \n\nWe appreciate the reviewer’s efforts for improving our manuscript and we will carefully check the equations and explanations in the manuscript to improve clarity and mathematical rigor.\n\n> Zero gradient is not exactly the same as a saddle point. Maybe give a hint of the proof in the main text.\n\nWe sincerely thank the reviewer for pointing out this critical issue. We acknowledge that our use of the term \"saddle point\" in this context was not only imprecise but incorrect. The proper terminology should be critical point, defined as any point where the gradient vanishes.\n\nWe agree that the misuse of terminology undermines the rigor of the presentation. We deeply apologize for this mistake and will correct it in the revised manuscript by replacing \"saddle point\" with \"critical point\" and including a concise explanation of the result in the main text for clarity and correctness. However, we would like to point out that our argument remains valid, that the extreme sparsity of spike activity leads to vanishing gradients and thus impedes learning with gradient-based optimization methods. \n\n> Line 179: subscript ()_t is never used\n\nThanks for your suggestion. In Equation (10), we use $\\theta_{i,t}$ to denote the temporary value of the neuron threshold during the iterative update. However, we agree that this equation is confusing and unnecessary (as also pointed out by Reviewer 7Qnw). We will remove it and replace it with a clearer narrative explanation to improve clarity.\n\n> Line 196 BPTT requires memory of previous time states? Please clarify.\n\nThanks for pointing it out. Yes, training spiking neural networks with Backpropagation Through Time (BPTT)[1] does require maintaining memory of the neuronal states across all time steps during the forward pass. This includes membrane potentials, spike outputs, and other internal variables such as the threshold values in our case. BPTT unrolls the SNN over time and computes gradients by propagating errors backward through each time step. Therefore, for accurate gradient computation, it is necessary to store all intermediate states over the full simulation window.\n\nWe will clarify this detail in the revised manuscript.\n\n> Doesn’t it make sense to do overall evaluation first, then ablation?\n\nThanks for your suggestion. In the revised version, we will restructure the experiments by first presenting the main performance results and then followed by the ablation study.\n\n> Line 260 what’s synaptic operation numbers?\n\nThanks for your suggestion. The synaptic operation number (SOP) refers to the metric that calculate the total number of computational events. Specifically, one SOP is counted whenever a spike is transmitted from a presynaptic neuron to a postsynaptic neuron through a weighted connection. This metric is widely used in SNN literature as an estimate of the computational cost and energy consumption of a model [2, 3]. We will clarify this definition in the revised manuscript for better understanding.\n\n> Line 271: if the threshold \\theta is constant, what’s the difference if the signal is 0-1 or 0-\\theta?\n\nThank you for the insightful question. As noted in our earlier response regarding the \"graded spike\" clarification, there is indeed no difference during inference when $\\theta$ is constant, since the scaling factor can be absorbed into the weights of the following layer.\n\nHowever, during training, $\\theta$ is not constant. It is treated as a learnable parameter and is dynamically adapted over time. In addition, we apply regularization on $\\theta$ to suppress spike activity. This design allows $\\theta$ to act as a learnable scaling factor, which helps to maintain stable gradient and effective representation learning, while also encouraging sparsity through threshold value growth. We will revise the manuscript to clarify this.\n\n> Line 272: what’s synaptic output feature?\n\nWe apologize that the phrase \"synaptic output feature\" is imprecise and may cause confusion. What we intended to describe is the current-based output used in the AT-LIF neuron, where the output is scaled by the threshold (0 or θ) instead of the spike output (0 or 1). We will rephrase the term \"synaptic output feature\" by \"current-based output\" to avoid ambiguity.\n\n> Line 281: what is SOTA?\n\n“SOTA” is the abbreviation for State-of-the-Art. In our manuscript, it refers to the most advanced existing methods in the literature at the time of writing. We will replace the abbreviation to ensure clarity for all readers.\n\n> Table 2: some of the best AT-LIF results are with \\lambda_s=0, i.e. no sparsity. Please comment. What is the column T?\n    \nThanks for pointing it out. To clarify:\n    \n$\\lambda_s$ is the coefficient for direct activation regularization, which penalizes spike activity in the loss function. When $\\lambda_s$ = 0, it means there is no explicit regularization on spike activity. $\\lambda_t$ on the other hand, is the coefficient for threshold adaptation regularization introduced in our AT-LIF neuron model. It indirectly controls spike sparsity by increasing the firing threshold during training. Even when $\\lambda_s$ = 0, the presence of a non-zero $\\lambda_t$ enables our model to suppress spike activity effectively.\n\nRegarding the column T, it represents the number of discrete simulation time-steps used. In the context of spiking neural networks using the discrete-time Leaky Integrate-and-Fire model, T defines how many times the neuron dynamics are updated over time in response to the given input. In table 2, a fixed T is used across models for fair comparison. \n    \nWe will revise the manuscript to include these clarifications.\n\n> Please give more details how your method is integrated with other methods in the experiments.\n\nWe thank the reviewer for this question and are happy to provide more details.\n\nIn our experiments, we combine our proposed AT-LIF method with STDS [4], which is a state-of-the-art weight pruning algorithm designed for SNNs. STDS operates by reparameterizing the weights and progressively eliminating those with small absolute values during training, leading to a sparse connectivity pattern.\n\nOur integration is straightforward: we replace the standard LIF neurons with AT-LIF neurons throughout the network, and apply threshold adaptation as described in our method. Meanwhile, the STDS pruning algorithm is used to train the synaptic weights and induce weight sparsity. This combination allows us to simultaneously achieve both activity sparsity and weight sparsity, through STDS's connection-level pruning. In practice, AT-LIF focuses on reducing spike activity, while STDS reduces spatial connections. \n\nWe will clarify this integration process in the revised manuscript to improve clarity and reproducibility. If there are any remaining questions about how the two methods are combined, please feel free to let us know.\n\n**References**\n\n[1] Wu, Yujie, et al. \"Spatio-temporal backpropagation for training high-performance spiking neural networks.\" Frontiers in Neuroscience, 2018.\n\n[2] Hu, Yangfan, Huajin Tang, and Gang Pan. \"Spiking deep residual networks.\" TNNLS, 2021.\n\n[3] Shi, Xinyu, et al. \"Towards energy efficient spiking neural networks: An unstructured pruning framework.\" ICLR, 2024.\n\n[4] Chen, Yanqi, et al. \"State transition of dendritic spines improves learning of sparse spiking neural networks.\" ICML, 2022."}, {"review_id": "KdXUYGB8rS", "reviewer": "Reviewer_7Qnw", "summary": "This paper designs AT-LIF, a novel activity pruning method for SNN. The paper first discusses the two major challenges in activity regularization pruning method and then addresses these issues by introducing (1) learnable threshold for the neuron, and (2) current-based spike output. Evaluations on three datasets demonstrate the effectiveness of proposed methods.", "questions": "The following points could be addressed to further improve the clarity and impact of the work.\n\n1.The inclusion of more SOTA pruning methods is encouraged.\n2.In Section 3.4, the authors state that the model can simultaneously learn from the output representation and promote spike sparsity, thereby resolving the optimization conflict. The authors could provide a detailed explanation on why this trade-off is achievable.\n3.The proposed AT-LIF replaces binary output with a current-based output. Could the authors clarify how this design choice how this design contributes to addressing each of the two main challenges discussed in the Sec 3.1?\n4.The choice of hyperparameter \\( \\lambda_s \\) for both the AR baseline and AT-LIF appears somewhat arbitrary and varies within and across datasets without clear explanation. Could the authors clarify how these values were selected?", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strength:\n1. Clear discussion of challenges. The paper presents a well-motivated discussion of the existing challenges in training SNN. It analyzes conflict between minimizing spike activity and preserving task-specific performance and further identifies the gradient vanishing problem. These challenges are theoretically analyzed, providing a strong motivation for the proposed AT-LIF. \n2. The proposed method is simple and easy to implement. It addresses the identified challenges by modifying only the neuron's firing threshold and output range, without introducing complex architectural changes. This lightweight design makes the method promising for practical applications.\n\n\nWeaknesses:\n1.Insufficient comparative experiments. The paper only compares the proposed AT-LIF with a basic activation regularization baseline and two weight pruning methods. This narrow comparison makes it difficult to assess the effectiveness.\n2.Unclear explanation about conflict resolution. In Section 3.4, the authors state that “the conflict between the two terms does not necessarily arise, as the model can still effectively learn from the output representation while we simultaneously maximize the threshold to promote sparser firing events” (Line225-227). However, the paper does not explain why or how the model can simultaneously achieve effective representation learning and sparse spike activity. As a result, the justification for \"resolving the optimization conflict” is not convincing.\n3.Minor clarity issues. Eq.10 and Fig.3 (Right) are not described or referenced in the main text.", "was_revised": true, "final_justification": "Authors have resolved my concerns in the rebuttal stage.", "rebuttal": "# Response to Reviewer 7Qnw\n> Insufficient comparative experiments. The inclusion of more SOTA pruning methods is encouraged. \n\nWe sincerely thank the reviewer for this constructive suggestion. In the revised manuscript, we will incorporate additional comparisons with more recent and representative pruning techniques for both activity pruning and combined activity/weight pruning.\n\nFor activity pruning, as discussed in the manuscript, most prior approaches rely on direct regularization over spike activity. To provide a more comprehensive evaluation, here we include comparisons with additional baseline techniques, including: (1) L1 regularization over average firing rate (adopted by [1] as part of its approaches), and (2) Hoyer regularization over output spike patterns [2]. These methods are evaluated using ResNet-20 on the CIFAR-10 dataset (Table below). Notably, our method achieve the superior sparsity compared to all existing results.\n\n|          | Accuracy | SOPs (M) | Avg. FR. |\n| -------- | -------- | -------- | -------- |\n| L1 Reg [1]    | 91.27    | 60.21    |  0.062   |\n| Hoyer Reg [2] | 88.62    | 76.79    |  0.081   |\n| AR [Baseline] | 91.31    | 57.26    |  0.060   |\n| AT-LIF [Ours] | 91.06    | 48.85    |  0.043   |\n\nFor combined pruning (i.e., both weights and spike activity), we also include three additional state-of-the-art baselines on CIFAR-10 for comparison: ADMM[1], GradR [3], and ESL-SNN[4].\n|          | Arch.    |   T     | Accuracy | SOPs (M) | Param (M)|\n| -------- | -------- |-------- | -------- | -------- | -------- |\n| ADMM [1]   | 7 Conv, 2 FC | 8 | 90.19    | 107.97   |  15.54   |\n| Grad-R [3] | 6 Conv, 2 FC | 8 | 91.37    | 87.73    |   0.86   |\n| ELS-SNN [4]   | ResNet-19 | 8 | 90.90    | 108.89   |   0.63   |\n| AT-LIF [Ours] | 6 Conv, 2 FC | 8 | 90.72 | 30.11    |   0.19   |\n\nWe demonstrate that AT-LIF, when integrated with weight pruning techniques, consistently achieves a favorable balance between accuracy and spike sparsity. We will incorporate all these expanded results and discussions in the revised manuscript to better support the empirical claims.\n\n> Unclear explanation about conflict resolution. The authors state that the model can simultaneously learn from the output representation and promote spike sparsity. The authors could provide a detailed explanation on why this trade-off is achievable.\n \nWe thank the reviewer for this thoughtful comment, and we are happy to provide further clarification.\n\nAs discussed in the manuscript, training sparse spiking neural networks typically involves jointly optimizing the task objective and the sparsity constraint:\n$$\\theta,w = \\arg\\min_{\\theta, w} \\mathcal L_{total} ,~~\\text{where}~ \\mathcal L_{total} = \\mathcal L_{task} + \\mathcal L_{sparsity}$$\nIn our method, however, we introduce the AT-LIF neuron, which outputs a scaled binary signal rather than a standard binary spike. This scaling mechanism helps maintain a steady average neuron output while simultaneously reducing the firing rate(as illustrated in Figure 2 of the manuscript). This property allows us to decouple the learning process. Specifically, since most deep spiking neural networks rely on rate coding, we assume that regularizing the threshold does not significantly affect the information encoded within the layers. Therefore, instead of jointly optimize the $\\mathcal L_{total}$, we seperate the optimization by learning the weight parameters based on the task loss while simotenously regularize the threshold to reduce spike activity.\n\n$$w=\\arg\\min_{w} \\mathcal L_{task}; \\quad \\theta=\\arg\\min_{\\theta} \\mathcal L_{sparsity}.$$\n\nAs a result, the network can effective learning representations while reducing spike counts during the optimization process. We will revise Section 3.4 to better highlight this mechanism and clarify how it enables a more graceful trade-off between accuracy and sparsity.\n\n> Could the authors clarify how \"current-based output\" design contributes to addressing each of the two main challenges? \n\nThank you for your insightful comments. As discussed above, the AT-LIF model enables current-based output, defined as $o_j[t] = s_j[t] \\cdot \\theta_j$.  During training, we assume that the average current-based output $\\sum_{t=1}^{T}  o_j[t]/T$ remains relatively stable despite the threshold regularization. This stability helps reduce the conflict between representation learning and sparsity penalty. \n\nMoreover, since the neuron’s output magnitude remains steady throughout training, this also helps alleviate the gradient vanishing problem commonly seen in sparse spiking neural networks. Specifically, in the AT-LIF model, the derivative with respect to the synaptic weight $w_{ij}$ can be bounded as follows\n$$\\left |\\frac{\\partial \\mathcal{L}}{\\partial w_{ij}} \\right | = \\left |\\sum_{t=1}^{T} \\frac{\\partial \\mathcal{L}}{\\partial m_{i}[t]} o_j[t] \\right | \\leqslant\n\\left (\\sum_{t=1}^{T}  o_j[t] \\right) \\left(\\max_t \\left | \\frac{\\partial \\mathcal{L}}{\\partial m_{i}[t]} \\right | \\right ).$$\nSince $\\sum_{t=1}^{T}  o_j[t]$ is assumed to remain steady during training, the upper bound of the gradient does not approach zero. This suggests that gradient vanishing is not likely to occur, thereby enabling more stable and effective optimization.\n\nFor further clarification, we now provide a more detailed analysis of how both the task loss and the sparsity loss evolve during training under different neuron output behavior (Table Below). We evaluate two ResNet-20 models with different neuron output behavior on CIFAR-10 dataset and display the change of task loss value, sparsity loss value and average gradient 2-norm in the following Table.\n\n|                  |  | Ep0  | Ep1  | Ep2 | Ep3 | Ep4 | ... | Ep10 |\n| ------------------------ |-- |--- | --- | --- | --- | --- |--- |--- |\n| w/o current-based |  Task Loss |1.77|1.47|1.37|1.43|1.56|...|2.30|\n| w/o current-based | Sparsity Loss |29.1|19.8|16.4|15.2|14.9|...|13.9|\n| w/o current-based |Gradient Norm |2.69|3.22|3.82|0.955|0.429|...|1.27e-4|\n| w/ current-based output  | Task Loss |1.79|1.51|1.32|1.21|1.12       |...|0.810|\n| w/ current-based output  | Sparsity Loss|29.3|20.1|16.5|14.5|12.9      |...|9.49|\n| w/ current-based output  | Gradient Norm |2.67|3.54|3.78|3.65|3.67 |...|12.3|\n\nFrom the results, we observe that without the current-based output, the model struggles to balance the learning objectives and suffers from a obvious gradient vanishing problem. Additionally, the ablation study presented in Section 4.2 provides more direct evidence of the importance of the current-based output.\n\nWe will further expand the discussion on this point in the revised manuscript.\n\n> Minor clarity issues. Eq.10 and Fig.3 (Right) are not described or referenced in the main text.\n    \nEquation 10 was intended to define the iterative update of the neuron threshold during training. It expresses how the threshold θ is incrementally adjusted at each training step as part of the threshold adaptation mechanism. To improve readability, we will remove Equation 10 and instead explain the threshold update more clearly through narrative description in the revised version.\n\nFigure 3 (Right) illustrates the forward and backward computational flow of the AT-LIF neuron during training using Backpropagation Through Time (BPTT). We will revise the manuscript to explicitly reference and discuss this figure in the main text to better guide the reader through the training dynamics of AT-LIF.\n\n> The choice of hyperparameter ( \\lambda_s ) for both the AR baseline and AT-LIF appears somewhat arbitrary and varies within and across datasets without clear explanation. Could the authors clarify how these values were selected?\n\nWe thank the reviewer for this valuable question. We acknowledge that our method introduces an additional coefficient $\\lambda_t$, which controls the strength of the threshold regularization. In practice, for the AR baseline, $\\lambda_s$ (the coefficient for activation regularization) was chosen based on commonly adopted values in previous works, and further adjusted via grid search to achieve a reasonable sparsity–accuracy trade-off. $\\lambda_t$ controls the strength of the threshold adaptation, can be tuned via a simple grid search or empirically set based on prior experience. \n\nHowever, we would like to emphaize that unlike weight pruning, it is hard to directly control the desired sparsity of the spike activity, as different input sample triggers distinct pattern of spikes, resulting in different activation sparsity. Thus empirically setting the regularization coefficient for the penalty term is a more practical and feasible solution.\n\nMoreover, in our method, the convergence of the model is more robust to extreme values of the regularization coefficient. For example, with $\\lambda_t=1e-3$, ResNet-20 on CIFAR-10 achieves 92.05% accuracy with 61.7M SOPs. Even when $\\lambda_t$ is increased to $1e-1$, the model is still able to converge, achieving 45.52% accuracy with only 9.19M SOPs and 0.007 average firing rate. In contrast, for direct activation regularization, the model achieves 91.31% accuracy with 57.26M SOPs when $\\lambda_s = 1e-2$, but fails to converge when $\\lambda_s \\geq 0.03$. This highlights the greater stability and flexibility of our method during hyper-parameter selection.\n\nWe appreciate the reviewer’s feedback and will clarify our hyper-parameter selection strategy and its implications more explicitly in the revised manuscript.\n\n**Reference**\n\n[1] Deng, Lei, et al. \"Comprehensive snn compression using admm optimization and activity regularization.\" TNNLS, 2021.\n\n[2] Narduzzi, Simon, et al. \"Optimizing the consumption of spiking neural networks with activity regularization.\" ICASSP, 2022.\n\n[3] Chen, Yanqi, et al. \"Pruning of Deep Spiking Neural Networks through Gradient Rewiring.\" IJCAI, 2021.\n\n[4] Shen, Jiangrong, et al. \"ESL-SNNs: An evolutionary structure learning strategy for spiking neural networks.\" AAAI, 2023."}]

Author rebuttal:

Meta-review: This paper introduces AT-LIF, an adaptive threshold leaky integrate-and-fire neuron model designed to reduce spiking activity in spiking neural networks (SNNs) while preserving task performance. The approach addresses key challenges in training sparse SNNs, including gradient vanishing and optimization conflicts, through threshold adaptation and current-based outputs. Empirical results across CIFAR-10, DVS-CIFAR10, and ImageNet show significant reductions in firing rate and synaptic operations, often with minimal accuracy loss, and demonstrate compatibility with weight pruning methods.

Reviewers agree that the contribution is technically solid, original, and relevant for energy-efficient neuromorphic computing. While some noted issues with clarity, limited comparisons in the initial submission, and the inherent trade-off between accuracy and sparsity, the authors’ rebuttal effectively addressed these concerns with additional results and clarifications.

Overall, the consensus is that this is a strong and promising contribution to efficient SNN training, with clear potential impact on scalable neuromorphic systems.
Decision: Accept (poster)