Improving the Sparse Structure Learning of Spiking Neural Networks from the View of Compression Efficiency
Authors: Jiangrong Shen
Qi Xu
Gang Pan
Badong Chen
Venue: ICLR 2025 Spotlight (2025)
Keywords: spiking neural networks
Primary area: applications to neuroscience & cognitive science

Abstract: The human brain utilizes spikes for information transmission and dynamically reorganizes its network structure to boost energy efficiency and cognitive capabilities throughout its lifespan. Drawing inspiration from this spike-based computation, Spiking Neural Networks (SNNs) have been developed to construct event-driven models that emulate this efficiency. Despite these advances, deep SNNs continue to suffer from over-parameterization during training and inference, a stark contrast to the brain’s ability to self-organize. Furthermore, existing sparse SNNs are challenged by maintaining optimal pruning levels due to a static pruning ratio, resulting in either under or over-pruning.
In this paper, we propose a novel two-stage dynamic structure learning approach for deep SNNs, aimed at maintaining effective sparse training from scratch while optimizing compression efficiency. 
The first stage evaluates the compressibility of existing sparse subnetworks within SNNs using the PQ index, which facilitates an adaptive determination of the rewiring ratio for synaptic connections based on data compression insights. In the second stage, this rewiring ratio critically informs the dynamic synaptic connection rewiring process, including both pruning and regrowth. This approach significantly improves the exploration of sparse structures training in deep SNNs, adapting sparsity dynamically from the point view of compression efficiency.
Our experiments demonstrate that this sparse training approach not only aligns with the performance of current deep SNNs models but also significantly improves the efficiency of compressing sparse SNNs. Crucially, it preserves the advantages of initiating training with sparse models and offers a promising solution for implementing Edge AI on neuromorphic hardware.

Published as a conference paper at ICLR 2025
IMPROVING THE SPARSE STRUCTURE LEARNING OF SPIKING NEURAL NETWORKS FROM THE VIEW OF COMPRESSION EFFICIENCY
Jiangrong Shen $4$6 Xu Gang Pan Badong Chen $B
+Faculty of Electronic and Information Engineering, Xi'an Jiaotong University #Institute Artificial Intelligence and Roboties, Xi'an Jiaotong University $State Key Lab of Brain-Machine Intelligence, Zhejiang University [[Choool of Computer Science, Dalian University of Technology Hybrid Augmented Intelligence, Xi' an Jiaotong University
BNational Key Lab of
ABSTRACT
The human brain utilizes spikes for information transmission and dynamically reorganizes its network structure to boost energy efficiency and cognitive capabil- ities throughout its lifespan. Drawing inspiration from this spike-based computa- tion, Spiking Neural Networks (SNNs) have been developed to construct event- driven models that emulate this efficiency. Despite these advances, deep SNNs continue to suffer from over-par peee-paramtteizztiion during training and inference, a stark contrast the brain's ability to self-organize. Furthermore, existing sparse SNNs are challenged by maintaining optimal pruning levels due to a static pruning ratio, resulting in either under or over-pruning. In this paper. we propose a novel two-stage dynamic structure learning approach for deep SNNs, aimed at main- taining effective sparse training from scratch while optimizing compression effi- ciency. The first stage evaluates the compressibility of existing sparse subnetworks within SNNs using the PQ index, which facilitates an adaptive determination of the rewiring ratio for synaptic connections based data compression insights. the second stage, this rewiring ratio critically informs the dynamic synaptic connection rewiring process, including both pruning and regrowth. This approach significantly improves the exploration of sparse structures training in deep SNNs, adapting sparsity dynamically from the point view of compression efficiency. Our experiments demonstrate that this sparse training approach not only aligns with the performance of current deep SNNs models but also significantly improves the efficiency of compressing sparse SNNs. Crucially, it preserves the advantages of initiating training with sparse models and offers promising solution for imple- menting edge Al on neuromorphic hardware.
INTRODUCTION
Spiking Neural Networks (SNNs) have garnered increasing attention due to their event-driven prop- erties, high spatiotemporal dynamics, and structural and learning plasticity that mimic biological
neural processing (Maass 1997 Subbulakshmi Radhakrishnan et al. [021] Fang et al. 2023 Hu
et al. 2023). Unlike traditional artificial neural networks that rely on continuous signal computa- tion. SNNs process information using discrete events, aligning more closely with the energy-efficient mechanisms observed in human neural activity. The post-synaptic neurons in SNNs receive spike trains from pre-synaptic neurons and emit output spikes upon crossing a firing threshold (Stanojevic al., 2024 Zhou et al. 2023 Hu et al. (021). Consequently, bio-inspired SNNs offer signifi- cant advantages in energy efficiency, making them especially suitable for neuromorphic computing in Edge AI such as event-based vision (Liu et al. 2024b 2025). where energy constraints
Accepted by ICLR 2025. The corresponding author: xuqi@dlut.edu.co.

---

Published as a conference paper at ICLR 2025
paramount (Imam & Cleland 2020 Pei al. 2019 Deng al. 202 Despite these inherent ad- vantages, the deployment of increasingly deep SNNs introduces substantial challenges, particularly over ver-parameterization during training and inference, leading to excessive computational overhead and memory usage. This misalignment with the resource-efficient requirements of edge devices calls for innovative solutions.
Current research on the sparse structure learning of deep SNNs aims to address the over- parameterization issue. These methodologies are predominantly categorized by their computational cost throughout the whole training process. The first one is the gradually structural sparsification ap- proach with the non-spars network as the initial status. For instance, the gradient reparameterization (Chen et al. [201]] (202)) approach implements the gradually efficient sparsification for deep SNNs with learnable pruning speed by redefining the weight parameters and threshold growing function. Alternatively, the second category called fully sparsification methods initiate with sparse SNNs to maintain connection sparsity throughout training, exemplified by sparse evolutionary rewiring (Shen) et al. 2023) and the lottery-ticket hypothesis (Kim et al., 2022). We advocate for the latter due to its compatibility with hardware constraints like on-chip training. Considering the intrinsic connec- tion between network sparsity and compressibility, the PQ index has been proven to satisfy the six properties of an ideal sparsity measure (Hurley & Rickard, 2009), and employed as the indicator of vector sparsity in the traditional artificial neural networks in (Diao al., 2023). However, that sparsity measure analysis ignores the unique spatial and temporal dynamics in SNNs. Therfore, most existing fully sparse training methods for SNNs employ static pruning ratios or predetermined sparsity levels, ignoring the analysis of spatiotemporal dynamics of SNNs and lacking the necessary flexibility like self-reorganizing in human brain and often leading to under or over-pruning.
the
Observing the brain's flexible organization of large-scale functional networks, which adapt through environmental interactions, offers a clue towards solving deep SNNs' over-parratttrizzatinn. Dur- ing brain development, synaptic connections undergo structural plasticity. forming new synapses and eliminating existing ones (De Vivo et al., 2017 Barnes& & Finnerty. 2010 [Bennett et al. 2018). This rewiring process forms flexible network structure and promotes synaptic sparsity contributing to the brain's low power consumption. Therefore, emulating the brain's structural synaptic plastic- ity through dynamic structure learning approach could be key to developing more flexible deep SNN models. Meanwhile, the density of synaptic connections in the brain optimizes throughout development, although precise control mechanisms remain unclear. Thus, we explore optimizing the pruning ratio from a neural network compression perspective as explored in machine learning, where data compression theory helps quantify the compressibility of sub-network during each connection updating iteration, thereby avoiding under or over-pruning (Neill (020). By combining with the biologically plausible rewiring mechanism in human brain and neural network compres- sion theory in machine learning, we attempt to give solution about the sparse training method from scratch for deep SNNs with adaptive and suitable pruning ratio setting.
In light of these above insights, this paper proposes a novel two-stage sparse structure learning method from scratch for SNNs, utilizing the PQ index to measure appropriate compressibility. This method not only maintains sparse training throughout the learning process but also mitigates the issues of under-pruning and over-pruning in sparse SNNs. Our contributions are as follows:
We introduce a pioneering two-stage dynamic structure learning framework for deep SNNs that utilizes the PQ index to gauge and dynamically adjust structure of sparse subnetworks according to compressibility. This novel approach tailors the rewiring ratio throughout the training process, providing fine-tuned, adaptive mechanism that enhances the founda- tional training dynamics of deep sparse SNNs.
Our methodology extends traditional sparse training approaches for SNNs by implement- ing a continuous, iterative learning process across two stages. In the first stage, the PQ index informs the adjustment of synaptic connection rewiring ratios. In the second stage, these ratios guide a dynamic rewiring strategy that includes both the pruning and regrowth of connections. Thus the methodology optimizes the SNNs' structural efficiency and oper- ational effectiveness far beyond conventional static pruning techniques.
Through extensive empirical testing, our method not only achieves competitive perfor- mance relative existing otate-of-the-art models but also significantly enhances the ef- ficiency of sparse training from scratch implementations for deep SNNs. This rigorous validation demonstrates our approach's 's ability to maintain essential SNNs' network func-
2

---

Published as a conference paper at ICLR 2025
The Iniial Boarse SPehle Structure
Stage Genarate rolo
Stage rewking corrections by:
Traning Shit Studure
Triinnng Sparse
SNile Stricture
Updeiing in
H
H
Hi+
Hi
Growingg
Ritel Randor indisitzaiion Phamaninng tnariing
Compuus za F.
[Ragstrizatiee
[Comprassoor]
in
Parforanca Improding Danoeyy
Pofarnunoreeee Daailty denniigg
The flowchart of the proposed two-stage sparse structure learning method for SNNs. Stage involves the typical training process and attempts to identify an appropriate rewiring ratio according to PQ index. Stage II conducts the dynamic sparse structure learning method based the rewiring ratio in stage I. The iterative learning of the above two stages is employed during the whole training process, thereby implementing the sparse training from scratch for SNNs and enhancing generalization ability of the sparse model.
Figure I:
tionality while reducing computational redundancy, thus achieving superior compression SNN architectures.
2 RELATED WORKS
SNNs have seen considerable advancements in learning algorithms that have expanded their pa- rameter capacity and diversified their topological structures, These networks often incorporate tablished ANNs architectures, including VGGII, ResNets19, and Transformers, adapting them to the spske processing paradigm (Yao et al. 2024; Hu et al., 2024). Training methodologies range from direct training with surrogate gradients to conversion techniques that transform pre- trained ANNs into SNNs. While these static-topology SNNs have demonstrated significant efficacy in various applications, such as object detection and natural language understanding. they primarily emphasize synaptic weight optimization (Gast al., 2024 Zheng al.. 2024 Ren et al. 2024). This focus tends to overlook the critical aspect of synaptic connectivity learning, frequently lead- ing to parameter inefficiencies and constrained network evolution. In contrast, SNNs designed with dynamic structures learning are engineered to concurrently optimize both synaptic connections and weights. This dual optimization affords enhanced flexibility and facilitates the development of more efficient and adaptive network topologies (Xu et al. 2024 Jiang et al., 2023 Shen et al.) (024). We categorize the current sparse structure learning methods for SNNs into two distinct groups.
Gradual Sparsification of Connection Structures for SNNs. tializes the network with a non-sparse connected structure, which is iteratively optimized through- out training, resulting in gradually sparser connection structure. 1) Weight parameter optimization methods. For instance, the gradient rewiring (Grad R) method is introduced in (Chen et al. 2021) which implements sparse structure learning through redefining network connection parameters. This method ensures that the gradient of these parameters forms an angle of less than 90º with the ac- curate gradient. During model training. synaptic pruning and regeneration are iteratively applied, achieving joint learning of synaptic connections and weights. Building on this, the nonlinear gradi- ent reparameterization function that controls pruning speed through a threshold growth function is introduced in (Chen et al.) (022). further optimizing the SNNs structure. (Shi et al.) 2023) combines unstructured weight pruning with unstructured neuron pruning to maximize the utilization of the sparsity of neuromorphic computing, thereby enhancing energy efficiency. 2) Regularization-basee methods. (Deng et al. (222aa) incorporated gradient regularization into the loss function, achieving synaptic connection pruning and weight quantization based on the Alternating Direction Method of Multipliers (ADMM). Similarly, Yin et al. combined sparse spike encoding with sparse network connections, using sparse regularization establish models for spike data transmission and network sparsification (Yin et al. 202T). There are also some studies to explore the connection pruning for spiking-based Transformer structure (Liu et al. 2024a). 3) Comeecion-rlattnniii-eteemmm
This kind of methods typically ini-
3

---

Published as a conference paper at ICLR 2025
based methods. The synaptic sampling method based on Bayesian learning is proposed in (Kappel et al. 2013), modeling dendritic spine movement characteristics to achieve synaptic connection re- construction and weight optimization. Combining unsupervised STDP rules with supervised Tem- potron training, SNNs with connection gates are developed in (Qi et al. (018). It improves the performance while reducing connections via sparse SNNs. There are also other studies to introduce the plasticity-based pruning methods for deep SNNs (Han et al. 2024b).
Fully Sparsification of Connection Structures for SNNs. A different strategy involves initializing the network with a sparse connection structure from the start and continually optimizing this sparse structure throughout training. This fully sparse training approach is particularly advantageous for hardware implementation in resource- conotrre- environments, such as on-chip training in hard- ware chips. 1) Synaptic connection-rewiring -baedd methods. These evolutionary structure learning methods are proposed for deep SNNs by drawing inspiration of rewiring mechainism in human brain (Han et al.) 2024a| Shen et al.) [023] Li et al., 2024b). This method employs synaptic growth and pruning rules to adaptively adjust the connection structure based on gradients, momentum, or am- plitude during training, maintaining a certain level of sparsity in synaptic connections and achieving methods. The architecture search could also generate sparse SNNs, such as the lottery ticket hypothesis. The Early-Time lot- tery ticket hypothesis method proposed in (Kim et al.] 2022) demonstrates that winning sparse sub- networks exist in deep SNNs, similar to traditional deep ANNs. Further, the utilization-awan LITH method, which incorporates intra-layer connection regeneration and pruning during training, ad- dresses hardware load imbalance issues caused by unstructured pruning methods (Yin et al. 2024).
effective sparse training of SNNs. 2) nottery-ticket whpothesss-baast
Despite these advancements, a gap remains in the deployment of fully adaptive and efficient SNNs architectures, particularly in resource-constrained environments such as edge computing devices. This underscores the necessity for novel methods that not only refine the sparsity and efficiency of these networks but also maintain adaptive learning capabilities throughout their lifecycle during the whole training process. The need for dynamic, flexible SNN models that mirror the human brain's ability to reorganize and optimize its neural pathways in real-time is clear.
Our research addresses this gap by proposing a two-stage dynamic sparse structure learning ap- proach for SNNs from scratch, leveraging the latest advances in neural network compression and synaptic plasticity. This method promises to significantly enhance the adaptability and efficiency of deep SNNs, positioning them viable solution for next-generation neuromorphic computing
Ma et al. (2024). particularly as future highly efficient perceptron applications (Chen et al.) 2025 Guan & Zhao 2022 Guan al. 2024 Liang al. 2024 increasingly demand more computational resources for higher performance. We believe that by integrating adaptive synaptic pruning and growth mechanisms, our approach will set new standard for sparse structure learning in SNNs, aligning closely with the natural efficiencies observed in biological neural processes.
3 METHODS
The main goal of our study is to implement the fully sparse training from scratch for SNNs with the dynamic compressibility during the training process. In detail, we first introduce the proposed two-stage sparse learning framework for SNNs. After that, the first and second stage computations for obtaining the right rewiring ratio and rewiring sparse networks are described, respectively.
3.1 THE FRAMEWORK OF THE TWO-STAGE SPARSE LEARNING METHOD
As illustrated in Fig. D and Algorithm we design the two-stage sparse training method for SNNs with appropriate rewiring ratio for each iteration during the training process. The sparse weight connections are initialized according to the guarantee that the synaptic connection for each neuron has the same connection probability. As- suming there are n and n k-1 neurons in the neighboring two layers, then the probability of weight connection mask Mr
Rényi (ER) Random Graph. The ER graph could
between two neurons in these two layers satisfies
1) <(n² + n -1) n nー
=
(1)
where e is a constant (or scaling factor) that influences the edge probability and accounts for sparsity or connectivity scaling. Then the corresponding weight value W can be initialized by the commonly
4

---

Published as a conference paper at ICLR 2025
Algorithm
The two-stage sparse training process of SNNs.
Input Data: F₁, 1, 2, N. Labels of Input Data:yi, 1, 2..,, N.
Parameters: The weight mask is M.
The weight matrix:
W. The updating iterations:
E frequency.
1: for each assigned sparse layer of the SNNs do
2: Initialize the sparse weight mask of the connected layer the Erdös- topology:
3: end for
4: Initialize trained weight parameters;
5: for each training iterations do
6: & Stage
7: Perform standard training procedure with W; = M, @ W3;
8:
Perform weights updates for W;;
9:
Compute the total number of model parameters d; M,
10:
Compute PQ Index [w; and the lower bound of the amount of remaining parameters ra Compute the rewiring ratio c;;
11:
12:
C Stage II
13:
if current training epoch % Epoch frequency == 0: then for each assigned sparse layer of SNNs do
14:
15:
Remove the fraction e of synaptic connections according to the pruning rule;
16:
Regrow the fraction G; of synaptic connections according to the growing rule; end for
18:
end if'
end for
20:
return The sparse SNNs with W.
used initialization method, such Xavier Initialization and random normal distribution Initializa- tion. Since then, we have been able to obtain the initialized sparse SNNs. After that, the initialized SNNs would trained over multiple iterations. through the iterative training of the first stage and second stage in each iteration. It is worth noting that the SNNs would remain sparse and dynamically search for the suitable rewiring ratio in the following training process.
In detail, the first stage involves the typical training process and attempts to identify an appropriate rewiring ratio based on temporarily trained weights. The rewiring ratio is calculated according to the PQ index, efficient measure of the compressibility of neural network models (Diao et al.) 2023). The PQ index helps quantify the redundancy the network, thereby informing the following rewiring strategy. In the second stage, the dynamic sparse structure learning method based on the rewiring method is adopted implement sparse training from scratch. The connections are itera- tively pruned and regrown according to the specified rewiring ratio. This iterative training approach ensures that the network continuously adapts and optimizes its structure, thereby improving perfor- mance. The rewiring method allows for dynamic adjustment, promoting the activation and growth of previously dormant connections, which contributes to SNNs' capability enhancement.
By integrating these two stages, method achieves efficient and effective sparse training for SNNs, leveraging the compressibility insights gained in the first stage to guide dynamic structural adjust- ments in the second stage. This approach not only maintains the sparsity and efficiency of the model but also enhances its generalization ability.
3.2 COMPRESS THE SPARSE SNNS BASED ON PQ INDEX
After the sparse initialization based on ER graph, the synaptic connections between neurons would become sparse randomly. Then in the following training process, the SNNs model would be trained according to the two stages sparse training method.
In the first stage, we train the sparse SNNs and compute the appropriate rewiring ratio according to PQ index (,.(W) (we simplified it as T(W) in the detailed derivation the supplementary materi-
als). Here is the derivation of the sparsity measure (W) 1 - di - [[ for spiking neural
[
5

---

Published as a conference paper at ICLR 2025
networks (SNNs), incorporating the formula update and focusing on scaling invariance, sensitivity to sparsity reduction, and cloning invariance, combined with spatiotemporal dynamics and sparsity in SNNs. In detail, the scaling invariance in SNNs corresponds to: (1) Independence of weight scal- ing: If the weight matrix W is scaled (e.g.. multiplied by a constant), its sparsity structure remains unchanged, and so should (.,g(W), (2) Independence of temporal scaling: Changes in spike magni- tudes (the activation value) should not affect the sparsity measure, ensuring the measure accurately reflects temporal dynamies. We give detailed derivation in SNNs, it ensures that 4. (W) remains unaffected when all weights are scaled proportionally (e.g., multiplying W by constant & > 0). The scaling weight magnitudes or activation value intensity does not change the network sparsity. Meanwhile, we analyze that (W") keeps sensitivity to spatial and temporal sparsity in SNNs, that is, the distribution of weights or spike activations (firing rates). When changes weight distribution with more nonzero weights, leading to reduction in p.o (W) corresponds to sparsity decreasing-
When temporal sparsity decreases (more neurons firing at the same time), the distribution becomes denser, which directly affects the ratio || [ we demonstrate that the sparsity measure I... (W) should remain invariant when the weight matrix is cloned in spatial and temporal dimensions. This ensures that cloning or repeating the matrix does not affect the sparsity measure.
leading to a decrease in Lp.9(W) (W). In addition,
In detail, the PQ index for the non-zero vector Wi E R. with any 0 < < q computed by:
T. (W.) = dt +(1 W; 1p W 114),
(2)
where || W; Ip equals to (2) wy 12)2/,, in which wy.j = 1...d is the non-zero element in W,. Then the lower bound of the retaining number of model parameter W, can be obtained by:
r; d1 + a,,-~ 10/(0-///LLLL p)[1
- fp. (W.))
(3)
where d; = MM.. Then the pruned ratio with better compressibility is computed by:
= Ld: mai-mmin((11-11 - - di ).
(4)
in which the 7 and 33 are the scaling factor and maximum rewiring ratio, respectively. In detail, the hyperparameter of y is used to scale the rewiring ratio according to PQ index. The bigger would obtain the higher rewiring ratio. We follow the settings in (Diao et al. 2023) to set 7 and 8 = 0.9 to prevent the model are over-pruned seriously. Nw, is the total number of parameters in W,. Assume
the indices set of W; with the largest weight magnitude. Then a, denotes the smallest value satisfying ΣjęM; |0yl S a, EjEm; juyl"" The big a, implies the model parameters are redundant and would result in higher rewiring ratio. Thus we set the a, to be 0.001 in the experiments to slow down the pruning speed and improve the stability of sparse model training. We give an example in Fig. 2 (a).
Since then, the right rewiring ratio for weight parameters in sparse SNNs has been figured out to improve compressibility and prevent sparse SNNs from either over-pruning
(Li
et al.
2024a Xu et al. 2024).
During the training process in each iteration, before the above spasity measurement process, need to update the weight matrix according to the learning algorithm in our model. In this paper, we adopt the iterative Leaky Integrate-and-Fire (LIF) neuron model in SNNs to enhance information integration and temporal representation (Wu et al. 2019). The membrane potential u(t) of postsy- naptic neuron is updated based on the membrane potential at and the integrated presynaptic neuron input:
(5) where 7 is the leaky factor set to 0.5, and (t) represents the spike inputs. When uut) exceeds the firing threshold of Vih, the neuron fires spike, and u(t) is set to be 0. Consequently, the neuron output and the membrane updating are given by:
u(() = ru(l - 1) + (Mr O W.)x((1,
a(t + 1) = Ouu(t 1) Vin),
(6)
ult - 1) m(t + 1)(1 a(t 1)),
(7)
To ensure that the output signal at each time step approximates the target distribution, we utilize the temporal efficient training loss function as in (Deng et al. 20215):
T LTET
Lcr(0(),,),
(8)

---

Published as a conference paper at ICLR 2025
where T denotes the time steps and Leg is the cross-entropy loss function. Then the masked weights are updated according to the gradient descent rule with surrogate gradient function.
3.3 THE CONNECTION REWIRING OF DYNAMIC SPARSE STRUCTURE LEARNING
After the above stage obtains the appropriate rewiring ratio. the connection rewiring method is fol- lowed to improve the stability and generalization ability of the sparse SNNs, instead of pruning the weights with the smallest magnitudes directly. The connection rewiring method is motivated by the synaptic rewiring mechanism in the human brain. The synaptic rewiring in the brain, covering the processes of synaptic pruning (elimination) and synaptic growth (formation), plays vital role in neural development, learning, memory, and overall cognitive function. This dynamic remodeling of synaptic connections promotes the brain's ' adaptability and efficiency in processing information. Meanwhile, the effectiveness of the rewiring operation has been demonstrated in earlier works for employ the dynamic connection rewiring process
the structure learning of SNNs. Therefore, to implement the sparse training of SNNs models from scratch, thus improving the stability and generalization ability of the sparse SNNs.
The connection rewiring method could implement the effective and fast training by the iterative training of pruning and regrowing connections, avoiding the introduction of additional parameters that could increase memory usage. The pruning rule ensures the elimination of less significant connections in SNNs, reducing computational complexity while preserving the core structure of the network. We rank the weights magnitude according to their absolute values in sparse SNNs trained the first stage, and prune the weight connections with the rewiring ratio of o the above section. However, the only operation of pruning may destroy the stable convergence of sparse SNNs and restrict the network's expressive capacity. To fully leverage the information processing capacity of the original large SNNs without any pruning, it is crucial that all connections are activated during training. Consequently, the growth rule is employed to promote the regeneration of connections that have not been activated for significant period of time. Different pruning and regrow rules adapt to the proposed two-stage sparse structure learning framework. For simplify, we adopt the comenttum-base growing rule, to prioritize the regeneration of synaptic connections according to the momentum of the parameters. This approach ensures that connections showing significant momentum, and thus potential importance, are prioritized for regrowth.
In detail, after obtained the suitable rewiring ratio in the first stage, we could then use this ratio e; to guide the training in the second stage. Assuming the standard training procedure for sparse SNNs is followed by freezing the masked weights as W; M, @ W, where the mask M; is obtained according to the pruning and regrowing principle as mentioned above. After obtaining the updated W. we begin to compute the rewiring ratio again as described in the first stage. Therefore, through the iterative process between the first and second stage, our model can improve the efficiency of sparse training SNNs from scratch with the adaptive rewiring ratio.
EXPERIMENTS
In this section, we evaluate the performance of our proposed two-stage sparse structure learning method for SNNs. We conduct experiments the CIFAR10. CIFAR100, and DVS-CIFAR10 datasets, including both ablation studies and comparative experiments. The experiments environ- ments are NVIDIA-4090 GPU computation devices based on PYTORCH framework.
4.1 ANALYSIS ON THE DYNAMIC SPARSE TRAINING DURING TRAINING PROCESS
The performances of the proposed two-stage dynamic sparse training of SNNs are validated on two different rewiring scopes, including neuron- wise rewiring and layer-wise rewiring. The neuron- wise rewiring would adopt the connection rewiring each neuron of model parameters, which would prune and regrow d p connected weight parameters for each neuron, and the rewiring ratio of each neuron is computed by the PQ index respectively. While the layer-wise one conducts weight parameter rewiring for each layer separately.
Effectiveness in the layer-wise rewiring scope. The accuracy and connection density of the pro- posed two-stage sparse training method for SNNs are illustrated in Fig. The initial connection
7

---

Published as a conference paper at ICLR 2025
Spansly meceummment by with the rewking ratio
524
92.48
0.00
762
602
92.1
868
65.74
ation
638
0.40
example:
91.8
99.4
I compressed
91.8
0.30
Spanid mescunement by tharonu with the reséiing rato
912
91.
88.8
0:
0.10
008
0.10
Iteradonn
58.
604
0.00
Daratoo
Aborray
Derody
MocuRR (6)
[N
Figure 2: An example for training process of our method (a). The performance of the proposed two-stage sparse training method for SNNs, CIFARIO (b) and CIFARIO0 (c) datasets, in the neuron-Ve wise pruming scope. The bar chart represents the accuracy achieved by the proposed two- stage sparse training method. The solid line reflects the density of synaptic connections in the SNNs model.
67.6
0.0
92.35
20.3
12.27
92:27
0.62
0.50
22.08
80
042
65.43
501448
0.40
ACQussy
91.11
0.39
01.00
Cendtty
87:77
01.6
13:38
$11.43
0.30
0.29
0.30
812
01.4
0.23
0.16
$1.05
0.20
0.20
0.15
0.13
90.0
0111
86
6.1
0.10
5
0
2
10
Iloration
Accuracy
Trend
Denety
Figure 3: The performance of the proposed two-stage sparse training method for SNNs, on CIFARIO (a) and CIFAR 100 (b) datasets, in the layer-wise pruning scope. The bar chart represents the accu- racy achieved by the proposed method. The solid line reflects the density of synaptic connections in the SNNs model. The dashed line is the trend analysis of accuracy using a two-period moving av- erage. This diagram depicts the correlation between the density of the model and the enhancements performance achieved using our two-stage sparse structure learning technique.
density is set to be 0.5, which means that there only half of the connections to be activated when initialization. Meanwhile, the connections in our model remain sparse, ranging from 0.5 to 0.11 dur- ing the whole training process. In addition, the number of iterations in the sparse training process increases, the proposed two-stage sparse training method generates a relatively suitable rewiring ra- tio in the first stage, gradually reducing the synaptic connection density in SNNs. It is notable that the proposed model achieves its peak accuracy of 92.38% and 70.3% during the fourth iteration with connection density of 30% on the CIFARIO and CIFAR 100 datasets, respectively. The peak ac- curacy is even higher than that (about 92.2% on the CIFAR10 110 dataset) of densely connected SNNs. This improvement can be attributed to the connection rewiring, which introduces more activated parameter space and enhances the performance of sparse training by exploring extensive parameters throughout the sparse training process.
Simultaneously, the overall accuracy of the sparse SNNs exhibits a fluctuating trend. In the initial iterations, our model's stage produces appropriate levels of sparsity, which decreases the density of synaptic connections in the sparse SNNs while improving accuracy. However, as the iterations continue and the model becomes more compressed, the performance starts to decline moderately due to increased sparsity. At crucial point, when the model achieves its highest level of accuracy during the fourth iteration with a connection density 30% for CIFAR1O, additional pruning results in a collapse when important parameters are eliminated, leading considerable performance decrease.
8

---

Published as a conference paper at ICLR 2025
Effectiveness in the neuron-wise rewiring scope. We also verify the performance of our proposed two-stage sparse straining method in the neuron- wise scope. As illustrated in Fig. 2 we analyze the accuracy and the corresponding density within four iterations, for these four iterations have shown the main trend change as in the situation of layer-wise scope. As shown in Fig. P the proposed model exhibits similar accuracy oscillation phenomena in the CIFAR10 10 dataset when using layer- wise sparse structure training, akin to the neuron-wise approach. The proposed model achieves an accuracy of 92.48% at the second iteration with connection density only 41%. However, in the neuron-basee based scenario on the CIFARI00 dataset, no similar oscillation phenomena are observed. This could be due the initial high sparsity, which may have led the pruning of some critical synaptic connections. Thus, the remaining connections could not be insufficiently trained, resulting decreased performance as the connection densities reduce.
1:20
92.98
1.20
Ros/oog (5)
91.43
$1.52
91.73
$1.99
9167
$1.00
46
30
91.42
0.84
60:43
0.00
0.53
92.5
90.44
0.40
0.42
0.35
0.5
0.21
0.43
0.35
0.30
0.20
022
0.1
3
90
10
Doratoo
Plamain Sipaase
GredunlySporss
Ramaan mg Bporso
(8)
Figure 4: The ablation experiments of our proposed two-stage sparse training method for SNNs. (a) The accuracy comparison between the gradually sparse training and sparse training from scratch. (b) The connection density comparison between the gradually sparse training and sparse training from scratch.
The above phenomena are consistent with the proposed model's behavior: The pruning process ini- tially removes redundant parameters. This results in a decrease in the sparsity of model parameters, leading to an improvement performance due to regularization effects. As the model compressed more, some critical parameters are pruned when the model reaches convergence, resulting in in- crease in sparsity and a slight decrease in performance. Eventually, as the model begins to collapse, all weakened parameters removed, leaving only the essential parameters needed to sustain per- formance. Consequently, the level of sparsity decreases dramatically, leading to a noticeable decline in performance.
Ablation study on the sparse training from scratch. To evaluate the effectiveness of sparse train- ing for the proposed two-stage sparse structure learning method, we conduct the ablation study by comparing the performance with gradually sparse training and sparse training from scratch. As illus- trated in Fig. [4. the performance of the sparse training from scratch (Remaining Sparse) outperforms that of the gradually sparse training from the initial fully non-sparse connections. The reason lies in that the manner of the sparse training from scratch explores similar thorough parameter space to the non-sparse model and masks some noises caused by the redundancy parameters. Besides, the proposed model for sparse training from scratch demonstrates superior network connection sparsity than the gradually sparse training model at the same number of iterations. This allows the sparse training from scratch model to more quickly identify the optimal rewiring rate and achieve better per- formance. Additionally, the sparse training from scratch model is more hardware- -fiindff than the gradually sparse training model, making it more suitable for sparse training in hardware-constrann environments especially on-chip learning.
4.2 PERFORMANCE COMPARISON TO OTHER METHODS
We compare the performance of the proposed two-stage sparse structure learning method for SNNs with other current state-of-the of-the--rttttttt SNNs: ADMM (Deng et al. 2021a), Grad (Chen et al. 2021)
ESLSNN (Shen et al. 2023) and STDS (Chen et al. 2022). UPR (Shi et al. 2023).
As shown in Tab. [ the proposed two-stage sparse structure training method achieves competi-
tive performance among the various methods while retaining the advantage of sparse training from

---

Published as a conference paper at ICLR 2025
Table I: Performance comparison of the proposed two-stage sparse structure learning approach for SNNs with other models.
Pruning
Top-1 Acc. %)
Acc.
Conn.
Param.
SOPS
Dataset
Architecture
T
Method
Loss( %
(%)
(M)
(M)
ADMM
Conv, 2 FC
8
90.19
-0.13
25.03
15.54
Grad
Conv, FC
92.54
-0.30
36.72
10.43
ESLSNN
ResNet19
2
91.09
-1.7
50
6.3
180.56
CIFARIO
STDS
Conv, 2 FC
8
92.49
-0.35
11.33
1.71
147.22
UPR
Conv, 2 FC
8
92.05
-0.79
1.16
9.56
16.47
ResNet19
92.48
+1.18
40.58
5.12
158.35
This
2
Neuron-wise
92.1
26.63
3.36
121.49
work
ResNet19
92.38
+0.11
29.72
3.7
133.26
Layer-wise
91.99
-0.28
17.91
2.26
110.65
ESLSNN
ResNet19
2
73.48
-0.99
50
6.32
186.25
CIFAR100
70.45
-3.71
3.60
9.60
UPR
ResNet18 SEW
4
69.41
-4.75
2.48
6.79
This
ResNet19
2
70.3
+1.07
29.48
3.73
140.27
work
Layer-wise
DVS- CIFARIO
ESLSNN
VGGSNN
10
78.3
-0.28
10
0.92
129.64
STDS
VGGSNN
10
79.8
-2.6
4.67
0.24
38.85
78.3
-0.5
0.77
1.81
6.75
UPR
VGGSNN
10
81.0
-1.4
4,46
2.5
31,86
This
VGGSNN
10
78.4
+0.08
30
2.76
189.02
work
Layer-wise
scratch. Notably, compared to fully non-sparse models, sparse training model can even improve performance while maintaining a certain level of sparsity through dynamic iteration and searching for appropriate rewiring ratio. For example, the CIFAR 10 dataset, the model trained using our proposed method under the neuron-wise scope improves performance by approximately 1% com- pared the fully non-sparse model while maintaining sparsity of 30% 40%. On the CIFARI dataset, the performance of SNNs model with our two-stage sparse training method is also improved 1.07% compared the non-sparse model with only 29.48% connection. is worth noting that the proposed two-stage training method proceeds sparse training from scratch and maintains sparse training during the whole training process. These results demonstrate that our proposed model helps the original non-sparse model mask redundant parameters and enhance the generalization capability of the sparse model during iterative training by continuously finding the appropriate rewiring ratio.
CONCLUSION
To summarize, this study has introduced novel two-stage dynamic structure learning method tai- lored for SNNs that effectively addresses the challenges fixed pruning ratios and the limitations of static sparse training methods prevalent in current models. In the first stage of our strategy, employ the PQ index to evaluate the compressibility of sparse subnetworks. This enables us to make informed adjustments the rewiring ratios of synaptic connections. This adaptive technique enables the model to circumvent the drawbacks of insufficient pruning or excessive pruning. In the second stage, the predetermined rewiring ratios guide the dynamic synaptic connection rewiring, incorporating both pruming and regrowth strategies. This approach not only improves the compres- sion efficiency of sparse SNNs but also boosts their performance. The iterative learning process implemented across both stages ensures continuous improvement and adaptation of the sparse net- work structure throughout the training phase. The experimental results validate that the proposed dynamic structure learning greatly improves the compression efficiency of SNNs. Additionally, it either matches or exceeds the performance benchmarks set by current models in certain circum- stances. Crucially, this strategy maintains the benefits of sparse training from the scratch, which is particularly advantageous in settings with restricted hardware resources, like neuromorphic hard- ware on Edge AI.
10

---

Published as a conference paper at ICLR 2025
6 ACKNOWLEDGEMENT
This work was supported by National Natural Science Foundation of China under Grant (No. 62306274, 62088102, 62476035, 62206037, 61925603), Open Research Program of the National Key Laboratory of PrainnMachhin Intelligence, Zhejiang University (No. BM12400012).
REFERENCES
Samuel Barnes and Gerald T Finnerty. Sensory Experience and Cortical Rewiring. The Neurosci- entist, 1998,,88-198 2010.
Sophie H Bennett, Alastair Kirby, and Gerald T Finnerty. Rewiring the Connectome: Evidence and Effects. Neuroscience Biobehavioral Reviews, 88:51-63 2018.
Kang Chen, Yajing Zheng, Tiejun Huang, and Zhaofei Yu. Rethinking high-speed image recon- struction framework with spike camera. arXiv preprint arXiv:2501.04477 2025.
Yanqi Chen, Zhaofei Yu, Wei Fang, Tiejun Huang, and Yonghong Tian. Pruning of deep spiking neural networks through gradient rewiring. arXiv preprint arXiv:2105.049: 2021.
Yanqi Chen, Zhaofei Yu, Wei Fang, Zhengyu Ma, Tiejun Huang, and Yonghong Tian. State tran- sition of dendritic spines improves learning of sparse spiking neural networks. In International Conference on Machine Learning, pp. 3701-3715. PMLR, 2022.
Luisa De Vivo, Michele Bellesi, William Marshall, Eric Bushong, Mark H Ellisman, Giulio Tononi, and Chiara Cirelli. Ultrastructrral Evidence for Synaptic Scaling Across the Wake/sleep Cycle. Science, 355(6324):507-55 2017.
Lei Deng, Yujie Wu, Yifan Hu, Ling Liang, Guoqi Li, Xing Hu, Yufei Ding, Peng Li, and Yuan Xie. Comprehensive snn compression using admm optimization and activity regularization. IEEE transactions on neural networks and learning systems, 2021a.
Shikuang Deng, Yuhang Li, Shanghang Zhang, and Shi Gu. Temporal efficient training of spiking neural network via gradient re-weighting. In International Conference on Learning Representa- tions, 2021b.
Enmao Diao, Ganghua Wang, Jiawei Zhan, Yuhong Yang, Jie Ding, and Vahid Tarokh. Pruning deep neural networks from sparsity perspective. arXiv preprint arXiv:2302.000 05601, 2023.
Wei Fang, Yanqi Chen, Jianhao Ding, Zhaofei Yu, Timothée Masquelier, Ding Chen, Liwei Huang, Huihui Zhou, Guoqi Li, and Yonghong Tian. Spikingjelly: An open-source machine learning infrastructure platform for spike-based intelligence. Science Advances, 9(40)eaad14480, 2023. doi: 10. 10.1116/siidvvadd144888 URL https: http:/wwwcinneee. wtrgg//c/abb//l...
sciadv ad11480
Richard Gast, Sara A Solla, and Ann Kennedy, Neural heterogeneity controls computations in spiking neural networks. Proceedings of the National Academy of Sciences, 121(3)2231188511,,, 2024.
Banglei Guan and Ji Zhao. Relative pose estimation for multi-camera systems from point correspon- dences with scale ratio. In Proceedings of the 30th ACM International Conference on Multimedia, pp. 5036-5044, 2022.
Banglei Guan, Zhao, and Laurent Kneip. Six-point method for multi-camera systems with reduced solution space. In European Conference on Computer Vision, pp. 109-126. Springer, 2024.
Bing Han, Feifei Zhao, Wenxuan Pan, and Yi Zeng. Adaptive sparse structure development with pruning and regeneration for spiking neural networks. Information Sciences, pp. 121481, 2024a.
Bing Han, Feifei Zhao, Yi Zeng, and Guobin Shen. Developmental plasticity-inspired adaptive pruning for deep spiking and artificial neural networks. IEÉE Transactions on Pattern Analysis and Machine Intelligence, 2024b.
11

---

Published as a conference paper at ICLR 2025
Yangfan Hu, Huajin Tang, and Gang Pan. Spiking deep residual networks. IEEE Transactions Neural Networks and Learning Systems, 34(8):5200-5200 2021.
Yangfan Hu, Qian Zheng, Xudong Jiang, and Gang Pan. Fast-snn: Fast spiking neural network by converting quantized ann. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45 (12::1544--1466, 2023.
Yifan Hu, Lei Deng, Yujie Wu, Man Yao, and Guoqi Li. Advancing spiking neural networks toward deep residual learning. IEEE Transactions on Neural Networks and Learning Systems, 2024.
Niall Hurley and Scott Rickard. Comparing measures of sparsity. IEEE Transactions on Information Theory, 55(10):4723 2009.
Nabil Imam and Thomas A Cleland. Rapid online learning and robust recall in neuromorphic olfactory circuit. Nature Machine Intelligence, 2(3):181-191, 2020.
Tingting Jiang, Qi Xu, Xuming Ran, Jiangrong Shen, Pan Lv, Qiang Zhang, and Gang Pan. Adaptive deep spiking neural network with global-local learning via balanced excitatory and inhibitory mechanism. In The Twelfth International Conference on Learning Representations, 2023.
David Kappel, Stefan Habenschuss, Robert Legenstein, and Wolfgang Maass. Network plasticity as bayesian inference. PLoS computational biology, 11(11)=e 2015.
Youngeun Kim, Yuhang Li, Hyoungseob Park, Yeshwanth Venkatesha, Ruokai Yin, and Priyadarshini Panda. Exploring lottery ticket hypothesis in spiking neural networks. In Euro- pean Conference on Computer Vision, pp. 102-120. Springer, 2022.
Yaxin Li, Xuanye Fang. Yuyuan Gao, Dongdong Zhou, Jiangrong Shen, Jian K Liu, Gang Pan, and Qi Xu. Efficient structure slimming for spiking neural networks. IEEE Transactions on Artificial Intelligence, 2024a.
Yaxin Li, Qi Xu, Jiangrong Shen, Hongming Xu, Long Chen, and Gang Pan. Towards efficient deep spiking neural networks construction with spiking activity based pruning. arXiv preprint arXiv:2406.01072, 2024b.
Shunkun Liang, Banglei Guan, Zhenbao Yu, Pengju Sun, and Yang Shang. Camera calibration using a collimator system. In European Conference on Computer Vision, pp. 374-390. Springer, 2024.
Yue Liu, Shanlin Xiao, Bo Li, and Zhiyi Yu. Sparsespikformer: A co-design framework for token and weight pruning in spiking transformer. In ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 6410-6414, IEEE, 2024a.
Zibin Liu, Banglei Guan, Yang Shang, Shunkun Liang, Zhenbao Yu, and Qifeng Yu. Optical flow- guided 6dof object pose tracking with an event camera. Proceedings of the 32nd ACM Inter- national Conference on Multimedia, pp. 6501-6509, 2024b.
Zibin Liu, Banglei Guan, Yang Shang, Qifeng Yu, and Laurent Kneip. Line-based 6-dof object pose estimation and tracking with event camera. IEEE Transactions Image Processing, 2024c.
Zibin Liu, Banglei Guan, Yang Shang, Yifei Bian, Pengju Sun, and Qifeng Yu. Stereo event-based. 6-dof pose tracking for uncooperative spacecraft. IEEE Transactions on Geoscience and Remote Sensing, 2025.
De Ma, Xiaofei Jin, Shichun Sun, Yitao Li, Xundong Wu, Youneng Hu, Fangchao Yang, Huajin Tang, Xiaolei Zhu, Peng Lin, et al. Darwin3: large-scale neuromorphic chip with a novel isa and on-chip learning. National Science Review, 11(5):nwae10) 2024.
Wolfgang Maass. Networks of spiking neurons: the third generation of neural network models. Neural networks, 10(9):665916671, 1997.
James O' Neill. An overview of neural network compression. arXiv preprint arXi::000..03669 2020.
12

---

Published as a conference paper at ICLR 2025
Jing Pei, Lei Deng, Sen Song, Mingguo Zhao, Youhui Zhang. Shuang Wu, Guanrui Wang, Zhe Zou, Zhenzhi Wu, Wei He, et al. Towards artificial general intelligence with hybrid tianjic chip architecture. Nature, 572(7767):106-111 2019.
Yu Qi, Jiangrong Shen, Yueming Wang, Huajin Tang, Hang Yu, Zhaohui Wu, Gang Pan, et al. Jointly learning network connections and link weights in spiking neural networks. In IJCAI, pp. 1597-1603, 2018.
Dayong Ren, Zhe Ma, Yuanpei Chen, Weihang Peng. Xiaode Liu, Yuhan Zhang, and Yufei Guo. Spiking pointnet: Spiking neural networks for point clouds. Advances in Neural Information Processing Systems, 36, 2024.
Jiangrong Shen, Qi Xu, Jian Liu, Yueming Wang, Gang Pan, and Huajin Tang. Esl-snns: An evolutionary structure learning strategy for spiking neural networks. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pp. 86-93, 2023.
Jiangrong Shen, Wenyao Ni, Qi Xu, and Huajin Tang. Efficient spiking neural networks with sparse selective activation for continual learning. Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 611-619, 2024.
Xinyu Shi, Jianhao Ding. Zecheng Hao, and Zhaofei Yu. Towards energy efficient spiking neu- ral networks: An unstructured pruning framework. In The Twelfth International Conference on Learning Representations, 2023.
Ana Stanojevic, Stanistaw Wozniak, Guillaume Bellec, Giovanni Cherubini, Angeliki Pantazi, and Wulfram Gerstner. High-performance deep spiking neural networks with 0.3 spikes per neuron. Nature Communications, 15(1):6793, 2024.
Shiva Subbulakshmi Radhakrishnan, Amritanand Sebastian, Aaryan Oheroi, Sarbashis Das, and Saptarshi Das. A biomimetic neural encoder for spiking neural network. Nature communications,
12(1):2143, 2021.
Yujie Wu, Lei Deng. Guoqi Li, Jun Zhu, Yuan Xie, and Luping Shi. Direct Training for Spiking Neural Networks: Faster, Larger, Better. In Proceedings of the AAAI Conference Artificial Intelligence, volume 33, pp. 1311-1318, 2019.
Qi Xu, Yaxin Li, Xuanye Fang. Jiangrong Shen, Qiang Zhang, and Gang Pan. Reversing structural pattern learning with biologically inspired knowledge distillation for spiking neural networks. In Proceedings of the 32nd ACM International Conference on Multimedia, pp. 3431-3439, 2024.
Man Yao. Jiakui Hu, Tianxiang Hu, Yifan Xu, Zhaokun Zhou, Yonghong Tian, Bo Xu, and Guoqi Li. Spike-driven transformer v2: Meta spiking neural network architecture inspiring the design of next- gext-eeneration neuromorphic chips. arXiv preprint (xXXi:2244.03660 2024.
Hang Yin, John Boaz Lee, Xiangnan Kong, Thomas Hartvigsen, and Sihong Xie. Energy-efficient models for high-dimension spike train classification using sparse spiking neural networks. In Proceedings of the 27th ACM SIGKDD conference on knowledge discovery & data mining, pp- 2017-2025, 2021.
Ruokai Yin, Youngeun Kim, Yuhang Li, Abhishek Moitra, Nitin Satpute, Anna Hambitzer, and Priyadarshini Panda. Workload-balanceer pruning for sparse spiking neural networks. IEEE Trans- actions on Emerging Topics in Computational Intelligence, 2024.
Hanle Zheng, Zhong Zheng, Rui Hu, Bo Xiao, Yujie Wu, Fangwen Yu, Xue Liu, Guoqi Li, and Lei Deng. Temporal dendritic heterogeneity incorporated with spiking neural networks for learning multi-timescale dynamics. Nature Communications, 15(1):277, 2024.
Yue Zhou, Jiawei Fu, Zirui Chen, Fuwei Zhuge, Yasai Wang, Jianmin Yan. Sijie Ma, Lin Xu, Huan- mei Yuan, Mansun Chan, et al. Computational event-driven vision sensors for in-sensor spiking neural networks. Nature Electronics, 6(1 6(11))870-88 878, 2023.
13


Reviews:
[{"review_id": "8QrOZZlI9e", "reviewer": "Reviewer_a7kG", "summary": "The authors present a two-stage dynamic pruning method tailored for deep Spiking Neural Networks (SNNs). The first stage evaluates the compressibility of existing sparse subnetworks within SNNs using the PQ index. In the second stage, synaptic connections are dynamically adjusted. The authors assert that this method can significantly reduce the network's sparsity. While the paper's motivation is commendable, there are shortcomings in both the presentation and the novelty of the work.", "questions": "1. Can the authors explain why the proposed method is specifically suited to SNNs rather than Binary Neural Networks (BNNs)? Have special adaptations been made to leverage the unique characteristics of SNNs, rather than applying ANN techniques to SNNs?\n2. I recommend that the authors include additional figures to further illustrate their method. This would greatly aid reader comprehension.", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. The authors apply adaptive pruning and sparse training methods to deep SNNs, which can further enhance the energy efficiency of these networks. This initiative is worthy of recognition.\n2. The two-stage sparse training and learning method proposed by the authors appears to be feasible based on the experimental results presented.", "weaknesses": "1. To my knowledge, adaptive pruning and sparse training are well-established in ANNs. Existing works such as RigL [1]  seem to share similar ideas with the authors' approach. Additionally, DSR [2] also adjusts the rewiring rate. The authors have not provided targeted adaptations of these strategies to accommodate the binary spiking characteristics unique to SNNs.\n2. The paper mentions the PQ index but does not provide an explanation, instead merely offering a citation. This could leave readers confused about what the PQ index represents and how it is calculated.\n3. The methodology section lacks theoretical discussion. There is no explanation of why these methods are particularly beneficial or suitable for SNNs.\n4. The size of Figure 3 is noticeably inconsistent with the other figures, which affects the overall presentation quality.\n\nReferences:\n[1] Evci, U., Gale, T., Menick, J., Castro, P. S., & Elsen, E. (2020). Rigging the Lottery: Making All Tickets Winners. International Conference on Machine Learning (ICML).\n[2] Mostafa, H., & Wang, X. (2019). Parameter Efficient Training of Deep Convolutional Neural Networks by Dynamic Sparse Reparameterization. International Conference on Machine Learning (ICML)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "fbWoZLLlTg", "reviewer": "Reviewer_MLns", "summary": "This paper addresses over-parameterization in deep spiking neural networks (SNNs) during both training and inference, aiming to achieve an optimal pruning level. The authors propose a novel two-stage dynamic structure learning approach for deep SNNs.", "questions": "What is unique about the proposed work for SNNs specifically? Additionally, given the strong performance of attention-based and Transformer-based SNNs in the field, could this method also be applied effectively to those models?", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Pruning is essential for making SNNs more practical for neuromorphic hardware, distinguishing it from artificial neural network (ANN) pruning, which is often GPU-oriented.", "weaknesses": "The paper could benefit from consistent citation formatting. For instance, \"Hu et al. (2024)\" works well at the beginning of a sentence, whereas \"(Hu et al., 2024)\" fits better at the end. Maintaining good formatting would enhance readability.\n\nThe rationale for using a PQ index in the second stage is unclear. Although PQ is a general technique, it would help to see a clearer connection to SNNs specifically and how it improves this work beyond its original intent. Not clear how PQ index is related to SNN.\n\nThe paper lacks an analysis of energy consumption, which is essential for evaluating SNNs’ efficiency, particularly in neuromorphic applications.\n\nThere’s no substantial improvement in accuracy or network connectivity, which may limit the paper’s impact.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Vbdeyhzzrt", "reviewer": "Reviewer_ZYmD", "summary": "This paper proposes a dynamic pruning framework for spiking neural networks (SNN). It divides each training iteration into two stages to update the weights and structure separately. It utilizes the PQ index to gauge and dynamically adjust structure of sparse subnetworks according to compressibility. Experimental results show that the proposed method achieves competitive performance while reducing redundancy.", "questions": "Please refer to the weaknesses.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The proposed method achieves competitive or better performance than the dense model with fewer connections and reduced redundancy.\n2. It is a novel idea to dynamically calculate the rewiring ratio.", "weaknesses": "1. The details of the training process is unclear. In Figure 2, the density decreases as the iteration increases. However, as described in Algorithm 1, the remove fraction and the regrow fraction are both $c_i$, therefore the density should keep constant. Please explain why the density decreases and describe the training process in detail.\n2. The proposed method requires multiple training iterations. Does it require longer training time as compared to existing methods? Please compare the training time with existing methods.\n\nMinor:\n\n1. Duplicate definition of $c_i$ in Algorithm 1: both labels of input data and rewiring ratio.\n2. In Equation 1: undefined symbol $n^k$, you mean $n_k$? Symbol $\\epsilon$ is also undefined.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "LVrH1KZpxY", "reviewer": "Reviewer_U3Ak", "summary": "This paper proposed an two-stage sparse structure learning framework for SNNs, which is combined with the PQ index technique and can be implemented through a continuous and iterative learning process.", "questions": "See Weakness Section.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The authors provided a detailed algorithm pseudo-code for their sparse training strategy.\n2. It is interesting to combine the sparse structure learning framework of SNNs with PQ index and rewiring techniques.\n3. The authors considered various rewiring scopes, including layer-wise and neuron-wise schemes.", "weaknesses": "1. The authors did not comprehensively report the performance of previous works in Table 1. Furthermore, it seems that their performance lags behind previous works. For example, the authors claim that UPR [1] achieves 78.3% (Acc.), 0.77% (Conn.) and 1.81M (Param.) on DVS-CIFAR10, VGGSNN. However, as mentioned in [1], the original text indicates that UPR can also achieve **81.0%** (Acc.), **4.46%** (Conn.) and **2.50M** (Param.) on DVS-CIFAR10, VGGSNN. In addition, STDS [2] can achieve **79.8%** (Acc.), **4.67%** (Conn.) and **0.24M** (Param.). \nIn comparison, this work achieves **78.4%** (Acc.), **30%**(Conn.) and **2.76M** (Param.) under the same experimental condition, which is inferior to the previous results.\n2. Compared to previous work [1], the authors lack experimental results for SOPs (Synaptic Operations) and power saving ratio.\n3. Compared to previous works [1, 2], the authors lack persuasive experimental results on large-scale datasets (e.g. ImageNet-1k).\n\n[1] Shi X, et al. Towards energy efficient spiking neural networks: An unstructured pruning framework. ICLR 2024.\n\n[2] Chen Y, et al. State transition of dendritic spines improves learning of sparse spiking neural networks. ICML 2022.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Spotlight)