Sparse Spiking Neural Network: Exploiting Heterogeneity in Timescales for Pruning Recurrent SNN
Authors: Biswadeep Chakraborty
Beomseok Kang
Harshit Kumar
Saibal Mukhopadhyay
Venue: ICLR 2024 poster (2024)
Keywords: spiking neural network
SNN
network pruning
stability
neuromorphic
leaky integrate and fire
STDP
sparsification
task-agnostic pruning
timescale optimization
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
TL;DR: A task-agnostic pruning method that exploits the diversity in timescales for heterogeneous RSNNs and gives small, stable pruned networks

Abstract: Recurrent Spiking Neural Networks (RSNNs) have emerged as a computationally efficient and brain-inspired machine learning model. The design of sparse RSNNs with fewer neurons and synapses helps reduce the computational complexity of RSNNs. Traditionally, sparse SNNs are obtained by first training a dense and complex SNN for a target task and, next, eliminating neurons with low activity (activity-based pruning) while maintaining task performance. In contrast, this paper presents a task-agnostic methodology for designing sparse RSNNs by pruning an untrained (arbitrarily initialized) large model. 
We introduce a novel Lyapunov Noise Pruning (LNP) algorithm that uses graph sparsification methods and utilizes Lyapunov exponents to design a stable sparse RSNN from an untrained RSNN. We show that the LNP can leverage diversity in neuronal timescales to design a sparse Heterogeneous RSNN (HRSNN). Further, we show that the same sparse HRSNN model can be trained for different tasks, such as image classification and time-series prediction. The experimental results show that, in spite of being task-agnostic, LNP increases computational efficiency (fewer neurons and synapses) and prediction performance of RSNNs compared to traditional activity-based pruning of trained dense models.

Published as a conference paper at ICLR 2024
SPARSE SPIKING NEURAL NETWORK: EXPLOITING HETEROGENEITY IN TIMESCALES FOR PRUNING RE-
CURRENT SNN
Biswadeep Chakraborty, Beomscok Kang, Harshit Kumar Saibal Mukhopadhyay Department of Electrical and Computer Engineering Georgia Institute of Technology
Atlanta, GA 30332, USA (b1 swsdddeep, smukhopadhyays @egatecc.ee............e...e...e
ABSTRACT
Recurrent Spiking Neural Networks (RSNNs) have emerged as computationally efficient and brain-inspired learning model. The design of sparse RSNNs with fewer neurons and synapses helps reduce the computational complexity of RSNNs. Traditionally, sparse SNNs obtained by first training dense and complex SNN for target task, and, then, pruning neurons with low activity (activity-based pruning) while maintaining task performance. In contrast, this paper presents a task-agnostic methodology for designing sparse RSNNs by pruning a large randomly initialized model. We introduce a novel Lyapunov Noise Pruning (LNP) algorithm that uses graph sparsification methods and utilizes Lyapunov exponents to design a stable sparse RSNN from a randomly initialized RSNN. We show that the LNP can leverage diversity in neuronal timescales to design sparse Heterogencous RSNN (HRSNN). Further, we show that the same sparse HRSNN model can be trained for different tasks, such image classification and temporal prediction. We experimentally show that, in spite of being task-agnostic, LNP increases computational efficiency (fewer neurons and synapses) and prediction performance of RSNNs compared to traditional activity-basee pruning of trained dense models.
INTRODUCTION
Recurrent Spiking Neural Networks (RSNNs), inspired by the human brain's 's information processing mechanism, utilize spikes for efficient learning and processing of spatio-temporal data (Maass 1997).
Recent advancements SNN research have underscored the importance leveraging heterogeneity in neuronal parameters to optimize network performance (Chakraborty & Mukhopadhyay 2023b Perez Nieves et al.) 2021 She et al. 202I). These studies have demonstrated that RSNNs with diversity in neurons' integratioo//ellxatioo dynamics, referred to the heterogeneous RSNN (HRSNN), enhance the learning ability of RSNNs and show improved performance over homogeneous spiking neural networks in tasks such as spatio-temporal classification of video activity recognition (Chakraborty &
Mukhopadhyay 2022 2023a Chakraborty et al. 2023 Padmanabhan & Urban 2010)).
Though the introduction of such heterogeneity in the neuronal parameters helps in improving the performance of the model, it also increases the complexity of the model exponentially, especially as the number of neurons increases. This makes optimizing the model hyperparameters extremely hard. Moreover, standard sparse random initializations of the network make it very unstable, as observed from their Lyapunov spectra [See Suppl. Sec. A.7] The design of sparse HRSNN models with fewer neurons and synapses helps balance computational demand and performance. Thus, getting sparse HRSNN model without sacrificing on the performance is of utmost importance. Traditionally, sparse neural networks are designed by first training dense (complex) network for target task, followed by pruning neurons/synapses to reduce computation while minimizing performance drop for that task (Blalock et al. 2020 Chowdhury et al. 2021bb Chen et al., 2018). Various task-de -ask-dependent pruning methods have been explored for forward SNNs. For example. STDP-based pruning of connections and weight quantization of SNNs have been studied for energy efffietfffcientt recognition

---

Published as a conference paper at ICLR 2024
(Rathi et al. 2018). Likewise, the lottery ticket hypothesis has been studied for complex (large) SNNs (Kim et al., 2022b). Gradient rewiring has also been explored for pruning deep SNNs (Chen et al. 2021). However, the pruned models derived from such task-driven pruning algorithms are extremely overfitted to the task it is trained on and demonstrate poor generalization performance. Suppl. Sect. [.3]]. J. Also, since most of these methods are mostly adapted from pruning techniques of deep neural networks (DNNs), they do not leverage the unique temporal dynamics and neuronal timescales inherent to heterogeneous spiking networks. Further, as these methods consider performance on target task during pruning, the complexity of the final pruned models varies from task to task and even across datasets for given task.
In this paper, we present a novel agnostti method, referred to as Lyapunov Noise Pruning (LNP), for designing sparse HRSNN. In contrast to the conventional approach of designing sparse networks by task-dependent pruning of trained dense networks, our approach starts with randomly initialized and arbitrarily initialized dense (complex) HRSNN model. We leverage the Lyapunov spectrum of an
HRSNN model and techniques from spectral graph sparsification algorithms (Spielman & Srivastaval 2011 Moore& & Chaudhuri| 2020) to prune synapses and neurons while keeping the network stable
(Spielman & Srivastaval [201 Moore & Chaudhuri 2020 Vogt et al. (020). The resulting random
sparse HRSNN can next be trained for different target tasks using supervised (backpropagation) or Seeene-DDeeddet-Plaaeee
unsupervised
STDP) methods.
Our aanostgio sparse model design helps develop universally robust and adaptable models and
eliminates the need for extensive task-specific adjustments (You et al.. 2022 Liu al.] (0226). Instead of minimizing (or constraining) performance loss for a given task, LNP optimizes the model structure and parameters while pruning preserve the stability of the sparse HRSNN. This results in sparse models that are stable and flexible, and maintain performance across multiple tasks. We further show that the same sparse HRSNN obtained by LNP can be trained for various tasks, namely, image classification and time-series prediction. For image classification on the CIFAR1O. CIFAR1 100 datasets, the sparse HRSNN from LNP shows similar performance but at a much lower (as both neurons and synapses are pruned) computation cost. Likewise, we assess the efficacy of the proposed task-agnostic pruning approach by training the sparse HRSNN models for prediction tasks. We consider (I) synthetic datasets of chaotic systems, such as Lorenz and Rossler, and (2) real-world datasets, including Google Stock Price Prediction and Wind Speed Prediction. The experimental results show that the proposed LNP enhances the pruning efficiency (defined as the ratio of performance to synaptic operations) over conventional task- desendennddnnt activity- -based pruning methods. The key contributions of this paper are as follows:
Spiking Network Design Methods. We present a methodology for de- signing sparse recurrent spiking networks by task- pruning of a randomly initialized dense model. This is in contrast prior SNN pruning approaches that task-dependent, designed for feed-forward SNNs, and adapted from DNN pruning methods thereby ignoring unique temporal dynamics of recurrent spiking networks.
Sparse Recurrent
Lyapunov Noise Pruning Algorithm. We present novel Lyapunov-based noise pruning algorithm, using spectral graph sparsification methods and the Lyapunov spectrum of HRSNN network. The proposed algorithm eliminates neurons and synapses from a randomly initialized dense HRSNN model by preserving the delocalized eigenvectors for improved stability and performance of the resulting sparse HRSNN, without training on any dataset.
Effective Utilization of Neuronal Timescale Heterogeneity. The proposed approach leverages the diversity of neuronal timescales in HRSNN to assist in pruning and enhance the performance of the sparse HRSNN.
Task-agnostic Sparse Model Design. The pruning algorithms developed in this paper is task-agnostic (unsupervised). The proposed approach do not optimize for performance on a given dataset while pruning; rather, only optimizes the topology of the graph and the neuronal time constants of the HRSNN network while ensuring stability of the network.
PRELIMINARIES AND DEFINITIONS
Models: For this paper, we use a Leaky Integrate and Fire (LIF) neuron model. The heterogeneity is introduced by assigning distinct membrane time constants, Tm, to each LIF neuron, creating a varied
2

---

Published as a conference paper at ICLR 2024
M
Conventional Pruning
Current Approach
Un trained Models
Training Supervised or Unsupervised
Trained Sparse
Heterogenecus RSNN
Clustered HRSMN
HARSNN
Lyapunov Noise-based Pruning
CHRENN Timescales
Untrained Sporse HRSWI
Proposed
Heterogenecus
(b)
Figure 1: (a) Concept of HRSNN with variable Neuronal and Synaptic Dynamics (b) Figure showing the task-agnostic pruning and training of the CHRSNNHRSNN networks using LNP comparison to the current approach
distribution of these parameters. Using the above notion of heterogeneous LIF neurons, we define two distinct HRSNN models, presented in Fig Fig]Ta)): First, the HRSNN model, predicated on the framework introduced in recent research (Chakraborty & Mukhopadhyay 2023b 2022). where the time constants of each neuron are derived by sampling from gamma distribution, engendering heterogeneity in neuronal dynamics. Again, recent ad advancements in neuroscience have also elucidated that the human brain comprises multiple regions, each exhibiting distinct temporal properties. conceptualized as the multi-region network model (Perich et al., 2020: de Oliveira Junior et al., 2022). Inspired by these, we propose the Clustered Heterogeneous RSNN (CHRSNN) model, wherein the recurrent layer comprises multiple clusters of HRSNN models, each characterized by unique distribution of time constants. Each of these neurons is then connected randomly to form small-world network architecture of the overall CHRSNN.
Algorithm Lyapunov Noise Pruning (LNP) Method
1: Step 1: Synapse Pruning using Spectral Graph Pruning
2: for each eiy in connecting ni, nl do
3: Find N N(ss))
4: Compute X for in (e,, (Algorithm 2
5: Define W using harmonic mean Ax)) for & in (ei;) (Eq
6: Use b(l) at each node
7: Compute C' of firing rates
8: Preserve each et with Pıj yielding pspars for i $
9: end for
10: Step 2: Node Pruning using Betweenness Centrality
11: for each n, in Network do
12: Compute ((ne) (Algorithm) 3)
if B(n;) threshold then
14: Remove 74
end if
16: end for
Step 3: Delocalization of the Figenvectors
18: for each Apried do
19: Add edges to preserve eigenvectors and maintain stability
20: end for
21: Step 4: Neuronal Timescale Optimization
22: for each pruned model do
23: Use Lyapunov spectrum to optimize neuronal timescales ViVi E R. using BO.
24: end for
25: return Aprnnd =0
3

---

Published as a conference paper at ICLR 2024
LNP Pruning Algorithm
Step Prune Synapses
Step Prune Neurons
Step Eigenvector Preservation
Step 4: Neuronal Timescale
Unpruned Network
Optimization
Pruned HRSNN
Decoded Output
Prediction/ Classification
Input
111
Readout
Decoding
Encoding Leyer
Layer
Training
Supervised
Supervised Unsupervised
Algorithm
Figure 2: Complete flowchart showing the steps for the LNP pruning algorithm and the training methodology to the pruned HRSNN network
Methods: 1. Lyapunov Spectrum of RNN using the algorithm from (Vogt et al. 2020) by observing the network's contraction/eexpassin over time sequences. The details discussed in Suppl. Sec. A and A
2. Spectral Graph Sparsification Methods (Feng) 2019 Liu & Yu 2023 crucial for simplifying graphs while maintaining their fundamental structural and spectral attributes. They seek ultra-sparse subgraphs that effectively approximate the original Laplacian eigenvalues and eigenvectors.
3. Betweenness Centrality: C.) In network with recurrent connections, C. gauges a node's importance in facilitating communication between different network sections. Nodes with high C. form critical information channels connecting various parts of the network.
3 METHODS
3.1 LYAPUNOV NOISE PRUNING (LNP) METHOD
The proposed research presents pruning algorithm employing spectral graph pruning and Lyapunov exponents in an unsupervised model. We calculate the Lyapunov matrix, optimizing for ratios and rates handle extreme data values and incorporate observations. After pruning, nodes with the lowest betweenness centrality are removed to improve network efficiency, and select new edges are added during the delocalization phase maintain stability and integrity. This method balances structural integrity with computational efficacy, contributing to advancements in network optimization. Algorithm shows high-level algorithm for entire process.
First, we define the Lyapunov matrix of the network. To formalize the concept the Lyapunov Matrix, let us consider a network represented by a graph G(V, E)) where V is the set of nodes and E is the set of edges. For each edge eşj connecting nodes let (2) be the set of neighbors nodes 2, 2 {t.j}. Thus, the Lyapunov exponents corresponding to these neighbors calculated using the harmonic mean of the Lyapunov exponents the neighbors of nodes i, as:
Step I: Noise-Pruning of Synapses:
represented A (N(2)). The element Lij of the Lyapunov Matrix (L) is then
L n. (A(N())
(1)
EMAA(N)))AM(M))
4

---

Published as a conference paper at ICLR 2024
where À denotes individual Lyapunov exponents from the set of all such exponents of nodes i and j's neighbors. L, encapsulates the impact neighboring nodes on each edge regarding their Lyapumov exponents, which helps us evaluate the network's stability and dynamical behavior. Through the harmonic mean, the matrix accommodates the influence of all neighbors, including those with extreme Lyapunov exponents, for balanced depiction of local dynamics around each edge. Thus, the linearized network around criticality is represented as:
Dx Lx + b(t) Ax b(t)
Here x represents the firing rate of N neurons, with 2; specifying the firing rate of neuron b(t) denotes external input, including biases, and L is the previously defined Lyapunov matrix between neurons. D is diagonal matrix indicating neurons' intrinsic leak or excitability, and is defined as - L. The intrinsic leak/excitabilityy Du. quantifies how the firing rate, Vi, of neuron i alters without external input or interaction, impacting the neural network's 's overall dynamics along with æ, L, and external inputs b(t). Positive Da suggests increased excitability and firing rate, while negative values indicate reduced neuron activity over time. We aim to create a sparse network (A*ruree) with fewer edges while maintaining dynamics similar the original network. The sparse network is thus represented as:
sparse x + b(t) such that
| (As A) 20| s c/2=" Ax:|
Væ e R.N
for some small > O. When the network in Eq. 6 is driven by independent noise at each node, we define b(t) b (t),, where b is constant input vector, & is vector of IID Gaussian white noise, and a is the noise standard deviation. Let 2 be the covariance matrix of the firing rates in response to this input. The probability pij for the synapse from neuron j to neuron i with the Lyapunov exponent by is defined as:
[li Ejj ー 2Eij)
for waj > 0 (excitatory)
Pij
(p4yl(Eii Σjj 2Eij) for Waj 0 (inhibitory)
(4)
Here, p determines the density of the pruned network. The pruning process independently preserves each edge with probability Pij. yielding A where A.pasee Aiy/Pij. with probability Pij and otherwise. For the diagonal elements, denoted as 4sparse representing leak/excitability, we either retain the original diagonal, setting A Ass or we introduce a perturbation, A, defined as the difference in total input to neuron i, and adjust the diagonal A A A. A. Specifically, A; 3 = AA,) q Ej AAyll This perturbation, A., is typically minimal with a zero mean and is modification in the excitability of neuron i due alterations in total input, aligning with the known homeostatic regulation of excitability.
interpreted biologically
Step I:: Node Pruning based Betweenness Centrality: In addressing network optimization, propose an algorithm specifically designed to prune nodes with the lowest betweenness centrality (C₂) in a given graph, thereby refining the graph its most influential components. C. quantifies the influence a node has on information flow within the network. Thus, we calculate C. for each node and prune the nodes with the least values below given threshold, ensuring the retention of nodes integral to the network's 's structural and functional integrity. The complete algorithm for the node pruning given in Algorithm B in Suppl. Sect.
Step III: Delocalizing Eigenvectors: To preserve eigenvector delocalization and enhance long-term prediction performance post-pruning, we introduce predetermined number of additional edges to counteract eigenvalue localization due to network disconnection. Let G = (V, E) and G' = (V, E') represent the original and pruned graphs, respectively, where E' c E. We introduce additional edges, E", maximize degree heterogeneity, H, defined as the variance of the degree distribution
H- (d(v) d)², subject |E' U E"1 < L, where L is a predetermined limit. This is V VeV
formalized as an optimization problem to find the optimal set of additional edges, E", enhancing eigenvector delocalization and improving the pruned network's 's predictive performance. Given graph G= = (V, with adjacency matrix A and Laplacian matrix L, we analyze the eigenvalues and eigenvectors of I. study eigenvector localization. To counteract localization due pruning, introduce a fixed number, m, of additional edges to maximize the variance of the degree distribution, Var (D), within local neighborhoods, formalized as:
max R"
Var(D) subject to
|E* = m, E'nE==0, E" E
MMUN((A ) x (v))
(5)
V.4
5

---

Published as a conference paper at ICLR 2024
LN Pruning (Ours)
Activity Pruning (Baseline)
Random
30000
12000
2:50
Neurans
20000
20000
4000
1.00
1
10000
2000
20
60
80
100
32
100
69 60
100
[b)
(6
Figure 3: Comparative Evaluation Pruning Methods Across Iterations. Figs. (a) and (b) show the evolution of the number of synapses and neurons with the iterations of the LNP and AP algorithms. Fig (c) represents how the RMSE loss changes when the pruned model after each iteration trained and tested on the Lorenz63 dataset
This approach ensures eigenvector delocalization while preserving structural integrity and specified sparsity. optimizing the model's long-term predictive performance.
The goal is to maximize the variance of the degree distribution, Var( D). by selecting the best set of additional edges E" such that: (i) The number of additional edges is m. (ii) The additional edges are not part of the original edge set E'. (iii) The additional edges are selected from the neighborhoods of the vertices.
Step IV: Neuronal Timescale Optimization In optimizing RSNN, known for their complex dy- namical nature, we employ the Lyapunov spectrum to refine neuronal timescales using Bayesian optimization. During optimization, RSNNs are inherently unstable due to their variable parameters and pruning processes, affecting training dynamics' stability and model learnability. Lyapunov exponents, outlined in (Vogt al. 2020), crucial for understanding system stability and are linked to the operational efficiency of RNNs. This paper utilizes the Lyapunov spectrum as a criterion for optimizing neuronal timescales in pruned networks, aiming to maintain stability and functionality while minimizing instability risks inherent in pruning. Further details on Bayesian optimization and finalized timescales in Suppl. Secs. D
B respectively.
EXPERIMENTS AND RESULTS
4.1 EXPERIMENTAL. SETUP
The experimental process, depicted Fig. 2 begins with a randomly initialized HRSNN and CHRSNN. Pruning algorithms are used to create sparse network. Each iteration of pruning results in a sparse network; we experiment with 100 iterations of pruning. We characterize the neuron and synaptic distributions of the "Sparse HRSNN" obtained after each pruning iteration to track the reduction of the complexity of the models with pruning.
We train the sparse HRSNN model obtained after each pruning iteration to estimate performance various tasks. Note, the pruning process does not consider the trained model or its performance during iterations. As outlined in Fig. [ the sparse HRSNNs are trained for time-series prediction and image classification tasks. For the prediction task, the network is trained using 500 timesteps of the datasets and is subsequently used to predict the following 100 timesteps. For the classification task, each input image was fed to the input of the network for Tinput = 100 ms of simulation time spike trains with firing rates proportional
in the form of pixels of the input images, followed by 100 ms of the empty signal to allow the current and activity of neurons to decrease. For both tasks, the input data converted into spike trains via entodenn.,ing forming the high- high-dimensional input to the XRSNN. The output spike trains are then processed through a decoder and readout layer for final predictions or classification results.
the intensity of the

---

Published as a conference paper at ICLR 2024
Evolution of Complexity of Sparse Models during Pruning: We plot the change in the number of neu- rons and synapses for the 100 iteration steps for AP and LNP Pruning algorithms. The results for the vari- ation of the synapses and neurons with the iterations of the pruning algorithm are plotted in Figs. 3 aa and (b), respectively. The LNP methods perform better than the activity-based pruning method and converge to a model with fewer neurons and synapses. Also, the inset diagram shows the variance of the distribu- tion as we repeat the experiment 10 times for each algorithm. We added the results for random ini- tialization for each step of the iteration of the LNP initialized 10 different networks, trained them, and showed their performance in G]c.. After each itera- tion, we randomly created network with an equal number of synapses and edges as found by the LNP method and trained and tested it on the Lorenz63 dataset to get the RMSE loss. We see that the Ran- dom initialized network shows higher variance and shows more jumps signifying the randomly initial- ized model unstable without proper finetuning. In addition, we see the performance consistently getting worse as the model size keeps decreasing, sigmifying an optimal network architecture more crucial for smaller networks than for larger networks. We see that the final distribution for both synapses and neurons of the AP-based models has a higher variance than the LNP algorithm. This also highlights the stability of the proposed LNP algorithm. In addition to this, we also plot the final distributions of the timescales observed from the two methods. The complete results are shown in Suppl. Sec. B
Res-eett parret
+ LNP --RSSW op
Gradi't e Gora, 2 (Chnn 2000
X
STOS Core. 2FC (C un (8000)
RealeelS
LIF
Auaunay (%)
USSN Comi, (boloc of a- 201-5
UNP
ESLSHN Reshvato (Sher at 2022)
88
07
Acchtecture en Sanv.
06-
ADMM Corr, 2 Jrlig es n. 2541)
Fester
Aug. SOPs (M) (og)
Figure 4: Scatter Plot showing Accuracy vs. Avg. SOPs for different pruning meth- ods on CIFAR1O. Results for CIFAR100 & Lorenz63 are given in Suppl. Sec. B.4
Datasets: We evaluate the performance of the LNP pruning methods for (1) time-series prediction on chaotic systems (i.e., Lorenz Xu et al. (2018a) and Rossler systems Xu et al. (2018b)) and two real-world datasets- Google Stock Price Prediction and wind speed prediction datasets (Samanta et al.)
(2020) and (2) image classification on CIFARIO 10 and CIFARIO0 datasets Krizhevsky et al. 2009).
Further details are given in Suppl. Sec.
Baselines: We use the activity- -based pruning (AP pruning) method Rust et al. (997)) as the
baseline, where we prune the neurons with the lowest activations in each iteration. The details of the AP pruning are given in Algorithm Iaand Suppl. Sec. A This algorithm operates iteratively, pruning the least active neurons and retraining the model in each iteration. We also compare the LNP algorithm with current task task-dependen state-of-the thrt pruning algorithms from prior works, and the results are shown in Table I In addition, we also introduce the Random initialization method, where after each iteration of the LNP algorithm, we observe the number of neurons and synapses of the LNP pruned network and then generate a random Erdos- Renyi graph with the same number of neurons and synapses. We repeat this process for each step of the iteration. The results of the Random initialization method are shown in Fig. 3(c).
Evaluation Metric First, we use the standard RMSE loss which is given as RMSE(()
D D
u (t) - (t) where D is the system dimension, o is the long term standard devi-
ation of the time series, e is arbitrary threshold, and vf is the forecast. We also use another measure to measure the performance of prediction called valid prediction time (VPT) (Vlachas et al., 2020). The VPT is the time t when the accuracy of the forecast exceeds given threshold. Thus VPT(t)= = EI(MMSE()) < é) For these experiments,we we set e arbitrarily to O. Thus, a higher VPT indicates a better prediction model.
Energy Efficiency: In assessing the energy consumption of neuromorphic chips, the central measure is the energy required for single spike to pass through a synapse, notably mmergy-intensie process Furber (0016). The overall energy consumption of a Spiking Neural Network (SNN) can be estimated by counting the synaptic operations (SOPs), analogous to floating-poin operations (FL.PPs) in
7

---

Published as a conference paper at ICLR 2024
Table Comparison of Pruning Methods((=CIFARR) pruned model trained & tested on CIFAR100)
CIARRI Sparstly
Bustine Accurary
50P
Buseiine Accurary
SOP
Model
Spanily
Spursity
Spuriity
Ratio
Dong ofal.
X731717
2022
7Cens, 2PC
89.53
-3.65
00
2.91
Belle at 2018
6Coes, ZPC
92.84
-3.53
97.96
16.59
UnV
6Cozs, ZPC
92.54
-0.00
71.59
71.34
19.45
Ris ml 28226
ResNec1
98.22
0.04
97.54
71,34
97.54
18.69
Chen al [022a]
X7mph
0Cees, 2PC
92.49
0035
58.67
5.27
2023
ResNert9
91.09
-1.7
95
2.11
73448
40.99
95
14.28
ResNet (Kandam
95.99
0015
1:25
73.32
22.18
0.28
(ours)
LNVP
HRSNW STWW
HRSNNEP
M0.00 003
92.37 0.01
1170
.081
007
0.05
94.01
95.06
35.67
19:01 010
0.4
0334
70.12
113
0.71
-2.37 0.332
22
92.00
04.21
42.57 0.0
4821 1339
CHRSNN BP
02.45
.0.74
0m
0.07
95.04
96
45.32
R037
07
0.58
10:30
73,45
0.66
4..2
-TR5 2171
93.47
97.04
50.35
0.64
Table 2: Table comparing the performance the Lorenz 63 and Google datasets. The complete results for the Rossler system and Wind prediction are given in Suppl. Sec. B
Pruning Method
Model
Training Method
Avg. SOPs (M)
Lorenzó
Google Dataset
RMSE
VPT
RMSE
VPT
HRSNN
BP
815.77
81.51
0.248 = 0.058
44.17 6.31
0.794 a 0.096
Uppruned
STDP
710.76
79.65
0.315 = 0.042
35.75
4.65
0.905 + 0.095
3.114
CHRSNN
BP
867.42
93.12
0.235 : 0.052
47.23 2 6.02
0.78220.091
45,2815.98
STDP
744.97
80.09
0,285 = 0.021
40.17
= 5.13
1.948 a 0.179
19.25
HRSNN
BP
117.52 14.37
1,245 0.554
31,08
8.23
1.457+ 0.584
28.24+6.98
STDP
92.68 + 10.11
1.718 = 0.195
21.10 7.22
1,948 + 0.179
19,25 7.59
AP Pruned
CHRSNN
BF
157.33 18.87
1.114s 0.031
33.97
7.56
1,325a 0.566
26,47:7.42
STDP
118.77 = 10.59
1.596 = 0.194
7.33 29.41
1,987 + 0.191
17.68 + 7.38
BF
22.87 2.27
0.691 0.384
33.67
33.14:3.01 0.855:0.112
STDP
18.22 2.03
0.705 + 0. 104
32.17 1 4.62
0.917 + 0,124
30.25 + 3.26
LNP Pruned
CHRSNN
BP
19.14 2.04
0.682 = 0.312
0.832+0.10 105
34.51=2.87
STDP
14,79 1.58
0.679 + 0.098
39.24 4.15
0.901 a 0,101
32.14a 3.05
traditional Artificial Neural Networks (ANNs). The energy consumption of an SNN is calculated as E =Cg x Total SOPs CE Σ s,C,, where CE represents the energy per SOP, and Total SOPs
Σ; sy0; is the sum of SOPs. Each presynaptic neuron fires 8; spikes, connecting es synapses, with every spike contributing to one SOP. Again, for sparse SNNs, the energy model is reformulated as Shi et al. 2024)):
C5 E, (s: Σ, n A Oig A ). where the set (n. Ozj. post (0,1)² indicates the state of the presynaptic neuron, the synapse, and the postsynaptic neuron. A value of 1 denotes an active state, while 0 indicates pruning. The A symbol represents the logical AND operation. Hence, we calculate the "SOP Ratio" between the unpruned and pruned networks as metric for comparison of the energy efficiency of the pruning methods, which quantifies the energy savings relative to the original, fully connected (dense) network. This ratio provides a meaningful way to gauge the efficiency improvements in sparse SNNs compared to their dense counterparts.
Readout Layer Pruning: The read-out layer is task-dependent and uses supervised training. In this paper, we do not explicitly prune the read-out network, but the readout layer implicitly pruned. The readout layer is multi-layer (two or three layers) fully connected network. The first layer size equal to the number of neurons sampled from the recurrent layer. We sample the top 10% of neurons with the greatest betweenness centrality. Thus, the number of neurons in the recurrent layer decreases, the size of the first layer of the read-out network also decreases. The second layer consists of fixed size with 20 neurons, while the third layer differs between the classification and the prediction tasks such that for classification, the number of neurons in the third layer is equal to the number of classes. On the other hand, the third layer for the prediction task is a single neuron which gives the prediction output.
4.2 RESULTS
Performance Comparison in Classification: Table shows the comparison of our model with other state-of-the- pruning algorithms in current literature. It must be noted here that these algorithms are task-dependent and use only synapse pruning, keeping the model architecture fixed. We evaluate the
8

---

Published as a conference paper at ICLR 2024
model on the CIFARIO & CIFARIO0 datasets and observe that our proposed task-agnostic pruning algorithm performs closely to the current state-of-the-are
The pruned model derived by the LNP method is task- agnostic. As such, we can train the model with either STDP or gradient-based approaches. In this section, we compare the performance of the unsupervised STDP-trained model with the supervised surrogate gradient method to train the pruned HRSNN model (Nettci et al. 2019). First, we plot the evolution of RMSE loss with pruning iterations in Fig. 3 c) when trained and evaluated the Lorenz 63 dataset. We see that our pruning method shows minimal degradation in performance compared to the baseline activity pruned method. Further on, presents the comparative performance of different pruning methods, models, and training methods on the Lorenz 63 and the Google stock prediction datasets. It is clear from the results that the I.NP-pruned models generally outperform the Unpruned and AP-pruned counterparts across different models and training methods. Specifically, LNP pruned models consistently exhibit lower SOPs, indicating enhanced computational efficiency while maintaining competitive RMSE and VPT values, which indicate the model's ' predictive accuracy and validity, respectively. This suggests that employing the LNP pruning method can significantly optimize model performance without compromising the accuracy of predictions.
Performance Comparison in Prediction:
We conducted an ablation study, where we systematically examined various com- binations of the four sequential steps involved in the LNP method. This study's findings presented graphically, as illustrated in Fig. 13 At each point (A-E), we train the model and obtain the model's accuracy and average. Synaptic Operations (SOPs) support ablation studies. The ablation study is done for the HRSNN model, which is trained using STDP and evaluated on the CIFARIO dataset. In the figure, different line styles and colors represent distinct aspects of the procedure: the blue line corresponds Steps I and of the LNP process, the orange line to Step 3, and the green line to Step 4. Solid lines depict the progression of the original LNP process (A + B C D + E). while dotted lines represent the ablation studies conducted at stages B and C. This visual representation enables clear understanding of the individual impact each step exerts on the model's 's perfor- mance and efficiency. Additionally, it provides insights into potential alternative outcomes that might have arisen from employing different per- mutations of these steps or omitting certain steps altogether. detailed description of the ablation study is given in Suppl. Sec. B.5
Ablation Studies:
LMP Pruning
Ablitonn
>0
E*
Accurscy10)
80
Bandomly Inttaa tIeV
Port Syrapse Pouring
Post Fresevvation
Post limescale Optemtatiion
20
50
100
500
Average SOPs (M)
(logarais))
Figure 5: Plot showing Ablation studies of LNP
5 CONCLUSION
This research introduced Lyapunov Noise Pruning (LNP), a novel, task aggkstin methodology for designing sparse Recurrent Spiking Neural Networks (RSNNs), emphasizing the balance between computational efficiency and optimal performance. Unlike prevailing methods, LNP starts with a random, densely initialized RSNN model, utilizing the Lyapunov spectrum and spectral graph sparsi- fication methods to prune while maintaining network stability. Experimental results demonstrated that LNP outshone conventional activity-based pruning, reducing computational complexity with fewer neurons and synapses, and maintaining superior accuracy and validity across various datasets, including synthetic and real-world ones. The task agno-tgi nature of LNP establishes universally adaptable and robust models without extensive, task-specific adjustments, preserving critical network parameters and optimizing model structures, especially crucial in environments with constrained computational resources. The flatter minima, corresponding to more stable and robust solutions, indicated enhanced stability in the learned dynamics of the model. In summary, LNP represents significant advancement in neural network design, offering more efficient, stable, and versatile models, suitable for diverse applications and setting the stage for future innovations in the field of neural networks.

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGEMENT
This work is supported by the Army Research Office and was accomplished under Grant Number W91 INF-19-1-0447. The views and conclusions contained in this document are those of the authors and should not be interpreted as representing the official policies, either expressed or implied, of the Army Research Office or the U.S. Government.
REFERENCES
Filipp Akopyan, Jun Sawada, Andrew Cassidy, Rodrigo Alvarez-Icaza, John Arthur, Paul Merolla, Nabil Imam, Yutaka Nakamura, Pallab Datta, Gi-Joon Nam, et al. Truenorth: Design and tool flow of a mw million neuron programmable neurosynaptic chip. IEEE transactions on computer-aides design of integrated circuits and systems, 34(10):1537-1557, 2015.
David G. Anderson, M. Gu, and Christopher Melgaard. An efficient algorithm for unweighted spectral graph sparsification. 2014.
Guillaume Bellec, Darjan Salaj, Anand Subramoney, Robert Legenstein, and Wolfgang Maass. Long short-term memory and learning-to-ler -o--rrrtt-r--- in networks of spiking neurons. Advances neural information processing systems, 31, 2018.
Davis Blalock, Jose Javier Gonzalez Ortiz, Jonathan Frankle, and John Guttag. What is the state of neural network pruning? arXiv preprint arXiv:2003.03033 2020.
Efe Camei, Manas Gupta, Min Wu, and Jie Lin. Qlp: Deep q-learning for pruning deep neural networks. 2022. doi: 10.109/TCST..20221119999 UR/jattpss//ooi.o///////
SCEVI.20222 3167951
Biswadeep Chakraborty and Saibal Mukhopadhyay. Characterization of generalizability of spike timing dependent plasticity trained spiking neural networks. Frontiers in Neuroscience, 15:695357, 2021.
Biswadeep Chakraborty and Saibal Mukhopadhyay. Heterogeneous recurrent spiking neural network for spatio-temporal classification. arXiv preprint urXiv:22/1.04297, 2022. doi: 10.3389/fnins. 2023.994517. URL /ttps://doi..rgg/ 3389/ fnins.iiii........ 2023. 994517
Biswadeep Chakraborty and Saibal Mukhopadhyay. Brain-inspired spiking neural network for online unsupervised time series prediction. arXiv preprint arXiv:2304.04697 2023a.
Biswadeep Chakraborty and Saibal Mukhopadhyay. Heterogeneous neuronal and synaptic dynam- ics for spike-efficient unsupervised learning: Theory and design principles. In Submitted to The Eleventh International Conference Learning Representattinns volume 17, pp. 994517. Frontiers Media SA, 2023b. URL Accepted.
d=QTRtAqoxw) wwwp:/wwwwwwweeeffoeeeeeee. http://eweeeeooooooooooooorrrrrr
Biswadeep Chakraborty, Xueyuan She, and Saibal Mukhopadhyay. A fully spiking hybrid neural network for energy-efficient object detection. IEEE Transactions on Image Processing, 30:9014- 9029, 2021.
Biswadeep Chakraborty, Uday Kamal, Xueyuan She, Saurabh Dash, and Saibal Mukhopadhyay. Brain- Prain-inspired spatiotemporal processing algorithms for efficient event-based perception. In 2023 Design, Automation & Test in Europe Conference & Exhibition (DATE), pp. 1-6. IEFE, Publisher Name, 2023.
Ruizhi Chen, Hong Ma, Shaolin Xie, Peng Guo, Pin Li, and Donglin Wang. Fast and efficient deep sparse multi-strengte spiking neural networks with dynamic pruning. In 2018 international joint conference on neural networks (ijenn), pp. 1-8. IEFE, 2018. doi: 10.1109/JCCNN.200 8.8893399 U wtt.httpp:/// /oo. orgg 0. 11 09/ JCNN 201 1001 8. 8489339
Yanqi Chen, Zhaofei Yu, Wei Fang, Tiejun Huang, and Yonghong Tian. Pruning of deep spiking neural networks through gradient rewiring. arXiv preprint arXiv:2105004900 2021. doi: 10.24963/ ijcai. 1jc2i.20 UtL.httss:/ http://....../0000/0/// http://.1..2..331err/ 20221/222332222222666
10

---

Published as a conference paper at ICLR 2024
Yanqi Chen, Zhaofei Yu, Wei Fang, Zhengyu Ma, Tiejun Huang, and Yonghong Tian. State tran- sition of dendritic spines improves learning of sparse spiking neural networks. In International Conference Machine Learning, pp. 7701-3715. PMLR, 2022a.
Yuxuan Chen, Jiyan Qiu, Zidong Han, and Chenhan Bai. Lgrass: Linear graph spectral sparsification for final task of the 3rd acm-china international parallel computing challenge. 2022b. doi: 10.48550//r2x122222..779
URL htt.:/ii...o//////////e wttp:/..........///.........//.. 48550/arxiv 2212. 07297
Sayeed Shafayet Chowdhury, Isha Garg, and K. Roy. Spatio-temporal pruning and quantization for low-latency spiking neural networks. 202 la. doi: 10.1 10.1109/UCCN552337. 2021119311111. URL
https://dol: wttp://...1111/IIeee
2021 9534111
Sayeed Shafayet Chowdhury, Isha Garg, and Kaushik Roy. Spatio-temporal pruning and quantization
for low-latency spiking neural networks. In 2021 International Joint Conference on Neural Networks (IJCN), pp. 1-9. IEEE, 2021b. doi: 0A.10099ICNN2233722220.933 httts::/dbi.orgg wttp://....1111//111 www.:1119911111 WTA11199/INNN887711
URL
Mike Davies, Narayan Srinivasa, Tsung-Han Lin, Gautham Chinya, Yongqiang Cao, Sri Harsha Choday, Georgios Dimou, Prasad Joshi, Nabil Imam, Shweta Jain, et al. Loihi: A neuromorphic manycore processor with on-chip learning. Ieee Micro, 38(1):82-99) 2018.
Laereio Oliveira Junior, Florian Stelzer, and Liang Zhao. Clustered and deep echo state networks for signal noise reduction. Machine Learning, 11 111(8)2885-2200 2022.
Lei Deng, Yujie Wu, Yifan Hu, Ling Liang, Guoqi Li, Xing Hu, Yufci Ding, Peng Li, and Yuan Xie. Comprehensive snn compression using admm optimization and activity regularization. IEEE transactions on neural networks and learning systems, 2021.
K. Doherty, David M. Rosen, and J. Leonard. Spectral measurement sparsificattin for pose-graph slam. 2022. doi: 10.1 IROS47612 2022. 9981584
URL URL/attps///oo.oor//00..11
Rainer Engelken, Fred Wolf, and Larry F Abbott. Lyapunov spectra of chaotic recurrent neural networks. Physical Review Research, 544)0043 2023.
Zhuo Feng. Grass: Graph spectral sparsification leveraging scalable spectral perturbation analysis.
IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems, 2019. doi: 10. 2020 2tt.///DDD..2...2.553333333 2968543
htRt::tts:///oo. http://oiooogr/g//// org/ 11.00.11990 1ww.11109/CAA.
Jean Feydy, Thibault Séjourné, François François-Xavier Vialard, Shun-ichi Amari, Alain Trouvé, and Gabriel
Peyré. Interpolating between optimal transport and mmd using sinkhorn divergences. In The 22nd International Conference Artificial Intelligence and Statisties, pp. 2681-2690 PMLR, 2019.
Peter Frazier. A tutorial bayesian optimization. arXiv preprint
2018.
Steve Furber. Large-scale neuromorphic computing systems. Journal of neural engineering, 13(5):
051001, 2016.
Erol Gelenbe, Zhi-Hong Mao, and Yan-Do Li. Function approximation with spiked random networks. IEEE Transactions on Neural Networks, 10(1):3-9, 1999.
Wulfram Gerstner and Werner M Kistler. Mathematical formulations of hebbian learning. Biological cybernetics, 87(5):404 415, 2002.
Wulfram Gerstner, Raphael Ritz, and J Leo Van Hemmen. Why spikes? hebbian learning and retrieval of time-resolved excitation patterns. Biological cybernetics, 69(5):503-515, 1993.
Wenjun Guo, Xianghong Lin, and Xiaofei Yang. A supervised learning algorithm for recurrent spiking neural networks based bp-stdp. In International Conference Neural Information URL
Processing, pp. 583-590, Springer, 2021. doi: 10.1007/978-3--30-9223000 //doi.
1007 0000000073370 --30---7---------------00-00000- 92307-5 68
Bing Han, Feifei Zhao, Yi Zeng, and Guobin Shen. Developmental plasticity-inspired adaptive pruning for deep spiking and artificial neural networks. 2022, doi: 10.48550/arxi..2211.222
URL (tIhttsss: / / orgg 1 axxi
11

---

Published as a conference paper at ICLR 2024
Zhiqiang He, Yaguan Qian, Yuqi Wang. Bin Wang, Xiaohui Guan, Zhaoquan Gu, Xiang Ling, Shaoning Zeng, Haijiang Wang, and Wujie Zhou. Filter pruning via feature discrimination deep neural networks. 2022. doi: 1007/978-3 031 9803-8_1
15. Uttpttpa://do..o0//////
T. Hoefler, Dan Alistarh, Tal Ben-Nun, Nikoli Dryden, and Alexandra Peste. Sparsity in deep learning: Pruning and growth for efficient inference and training in neural networks. 2021. URL
https://arzs org abs / 02. 00554
Nicolangelo lannella and Andrew D. Back. A spiking neural network architecture for nonlinear function approximation. Neural Nenworks, 14(6):933 939, 2001. ISSN 0893-6080. doi: https: w/bb:o////0000066099944400000000 URL https: www. http://wwwwcinddieeeeeeee com/
sci acc.enee/arrticlll//11//88388882
Mina A Khoci, Amirreza Yousefzadch, Arash Pourtaherian, Orlando Morcira, and Jonathan Tapson. Sparet: Sparse asynchronous neural network execution for energy efficient inference. In 2020 2nd IEEE International Conference on Artificial Intelligence Circuits and Systems (AICAS), pp- 256 260. IEEE, 2020.
Dachyun Kim, Biswadeep Chakraborty, Xueyuan She, Edward Lee, Beomseok Kang, and Saibal Mukhopadhyay. Moneta: A processing waatinng----mmmnn no--memory-basse hardware platform for the hybrid convolutional spiking neural network with online learning. Frontiers Neuroscience, 16:775457, 2022a.
Seijoon Kim, Seongsik Park, Byunggook Na, and Sungroh Yoon. Spiking-yolo: Spiking neural network for energy-efficient object detection. In Proceedings of the AAAI Conference Artificial Intelligence, volume 34, pp. 11270-11277, 2020.
Youngeun Kim, Yuhang Li, Hyoungseob Park, Yeshwanth Venkatesha, Ruokai Yin, and Priyadarshini Panda. Exploring lottery ticket hypothesis in spiking neural networks. In European Conference on Computer Vision, pp. 102-120. Springer, 2022b.
Alex Krizhevsky, Geoffrey Hinton, al. Learning multiple layers of features from tiny images. 2009.
Uday Kulkarni, Sitanshu S Hallad, A. Patil, Tanvi Bhujannavar, Satwik Kulkarni, and S. Meena. A survey on filter pruning techniques for optimization of deep neural networks. 2022. doi: 10. 1109/ RAAAAC55078..222.99877777 9987264
htLLhttps://do..rrrr 1000111A1111SS0500 2022
Jaehoon Lee, Yasaman Bahri, Roman Novak, Samuel S Schoenholz, Jeffrey Pennington, and Jascha Sohl-Dickstein. Deep neural networks as gaussian processes. arXiv preprint 171.001711.00165. 2017.
Fangxin Liu, Wenbo Zhao, Yongbiao Chen, Zongwu Wang, and Fei Dai. Dynsnn: A dynamic approach to reduce redundancy in spiking neural networks. 2022a. doi: 10.1 10.1099//4 2022.9746566. hRtpp:
2022. 9746566
Yuanxin Liu, Fandong Meng, Zheng Lin, Peng Fu, Yanan Cao, Weiping Wang, and Jie Zhou. Learning win lottery tickets in bert transfer via task-agnostic mask training. arXiv preprint arXiv:2204.11218 2022b.
Zhiqiang Liu and Wenjian Yu. pgrass-solver: A parallel iterative solver for scalable power grid analysis based on graph spectral sparsification. IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems, 2023. doi: 10.110/7TCAD2222233335555 URL https
7/ //dii.oog/1..110////C///
11w. TCAD 2023. 3237754
Zhiqiang Liu, Wenjian Yu, and Zhuo Feng. fegrass: Fast and effective graph spectral sparsification for scalable power grid analysis. 2022c. doi: 10. 20.1099CADD.2221.030000 30666474777 URL. https
7/doi oro/ioorgg100 1109/TCAD. 2021. 3060647
Wolfgang Maass. Networks of spiking neurons: the third generation of neural network models.
Neural networks, 1699 1997.
12

---

Published as a conference paper at ICLR 2024
Eli Moore and Rishidev Chaudhuri. Using noise to probe recurrent neural network structure and prune synapses. Advances in Neural Information Processing Systems, 33:14046-14055 2020.
Emre O Neftci, Hesham Mostafa, and Friedemann Zenke. Surrogate gradient learning in spiking neural networks. IEEE Signal Processing Magazine, 36:61-63, 2019.
Krishnan Padmanabhan and Nathaniel N Urban. Intrinsic biophysical diversity decorrelates neuronal firing while increasing information content. Nature neuroscience, ((10):1276-1288) 2010.
P. Panda and K. Roy. Learning to generate sequences with combination of hebbian and non-hebbian plasticity in recurrent spiking neural networks. 2017. doi: 10.3389/mmns.22077009999.. URL
https://doi. org/10 12389ffnins 2017.0017.0000 0066600693
Nicolas Perez-Nieves, Vincent CH Leung, Pier Luigi Dragotti, and Dan FM Goodman. Neural heterogeneity promotes robust learning. bioRxiv, 12(1) ):200-122 2021.
Matthew G Perich, Charlotte Arlt, Sofia Soares, Megan E Young. Clayton P Mosher, Juri Minxha, Eugene Carter, Ueli Rutishauser, Peter H Rudebeck, Christopher D Harvey, al. Inferring brain- wide interactions using data constraind recurrent neural network models. BioRxiv, pp. 2020-12, 2020.
Filip Ponulak and Andrzej Kasinski. Introduction to spiking neural networks: Information processing. learning and applications. Acta neurobiologiae experimentalis, 71 71(4):409-433, 2011.
Nitin Rathi, Priyadarshini Panda, and Kaushik Roy. Stdp-based pruning of connections and weight quantization in spiking neural networks for energy-efficient recognition. IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems, 38(4):668-61) 2018.
Alistair G Rust!, Rod Adams Stella George', and Hamid Bolouri. Activity-based pruning in developmental artificial neural networks. In Fourth European Conference on Artificial Life, pp. 224. MIT Press, 1997.
Muhammad Sabih, Ashutosh Mishra, Frank Hannig, and Jürgen Teich. Mosp: Multi-objective sensitivity pruning of deep neural networks. 2022. doi: 10. 10/19////CC333332229999997 URL
119901555//1 2022. 9969374
Sadra Sadeh and Stefan Rotter. Distribution of orientation selectivity in recurrent networks of spiking neurons with different random topologies. PloS one, 9(12);e 14237, 2014.
Yasufumi Sakai, Yusuke Eto, and Yuta Teranishi. Structured pruning for deep neural networks with adaptive pruning rate derivation based on connection sensitivity and loss function. 2022. doi: 10. 12720/jait. 13.3.295-308 URL /tLL:tttt////o..r///////// 12tp0/10...722//aa1a 1. 295-
Subhrajit Samanta, Mahardhika Pratama, and Suresh Sundaram. Bayesian neuro-fuzzy inference system for temporal dependence estimation. IEEE Transactions on Fuzzy Systems, 29(9):2479- 2490, 2020.
Xueyuan She, Saurabh Dash, Dachyun Kim, and Saibal Mukhopadhyay, A heterogeneous spiking neural network for unsupervised learning of spatiotemporal patterns. Frontiers in Neuroscience, 14:1406, 2021.
Jiangrong Shen, Qi Xu, Jian K Liu, Yueming Wang, Gang Pan, and Huajin Tang. Esl-snns: An evolu- tionary structure learning strategy for spiking neural networks. arXiv preprint arXiv:2306.03693 2023.
Xinyu Shi, Jianhao Ding, Zecheng Hao, and Zhaofei Yu. Towards energy efficient spiking neural net- works: An unstructured pruning framework. In The Twelfth International Conference on Learning Representations, 2024.
net. forum? wttp:ummmddeeeeeea00000000
Daniel A Spielman and Nikhil Srivastava. Graph sparsification by effective resistances. SIAM Journal on Computing. 1.
David Sussillo and Omni Barak. Opening the black box: low-dimensional dynamics in high- dimensional recurrent neural networks. Newral computation, 25(3):626-649, 2013.
13

---

Published as a conference paper at ICLR 2024
S. Vadera and Salem Ameen. Methods for pruning deep neural networks. 2020. doi: 10. 109/ ACCCESS.222..182666
2022. 3182659,
Werner van der Veen. Including stdp to eligibility propagation in multi-layer recurrent spiking neural networks. 2022.
Pantelis Vlachas, Jaideep Pathak, Brian R Hunt, Themistoklis Sapsis, Michelle Girvan, Edward Ott, and Petros Koumoutsakos. Backpropagation algorithms and reservoir computing in recurrent neural networks for the forecasting of complex spatiotemporal dynamics. Neural Networks, 126:
191-217, 2020.
Ryan Vogt, Maximilian Puelma Touzel, Eli Shlizerman, and Guillaume Lajoie. On Iyapunov exponents for rnns: Understanding information propagation using dynamical systems tools. arXiv preprint arXiv:2006.144 /4123, 8:818799, 2020.
Meiling Xu, Min Han, CL Philip Chen, and Tie Qiu. Recurrent broad learning systems for time series prediction. IEEE transactions on cybernetics, 50(4):1405-1417, 2018a
Meiling Xu, Min Han, Tie Qiu, and Hongfei Lin. Hybrid regularized echo state network for multivariate chaotic time series prediction. IEEE transactions on cybernetics, 2966)2235--215,, 2018b.
Ruokai Yin, Youngeun Kim, Yuhang Li, Abhishek Moitra, N. Satpute, Anna Hambitzer, and P. Panda. Workload-baland balancee pruning for sparse spiking neural networks. 2023. doi: 10.483500/rxiv.233022 06746. URL pnttts:://oi...
10. /88500/axii00 2302.00724.
Haoran You, Baopu Li, Zhanyi Sun, Xu Ouyang, and Yingyan Lin. Supertickets: Drawing task- agmostic lottery tickets from supernets via jointly architecture searching and parameter pruning. In European Conference on Computer Vision, pp. 674-690. Springer, 2022.
Wenrui Zhang and Peng Li. Temporal spike sequence learning via backpropagation for deep spiking neural networks. Advances in Neural Information Processing Systems, 33:12022 1512202-11203,, 2020.
14

---

Published as a conference paper at ICLR 2024
A SUPPLEMENTARY SECTION A
A.1 COMPUTATION OF LYAPUNOV EXPONENTS
We compute LE by adopting the well-established algorithm [57,58] and follow the implementation in [88, 55]. For particular task, each batch of input sequences is sampled from set of fixed-length sequences of the same distribution. We chose this set to be the validation set. For each input sequence in batch, a matrix Q initialized as the identity represent an orthogonal set of nearby initial states. The hidden states he are initialized as zeros.
To track the expansion and the contraction of the vectors of Q, the Jacobian of the hidden states at step t, Jr, is calculated and then applied to the vectors of Q. The Jacobian J, can be found by taking the partial derivatives of the RNN hidden states at time I, h4, with respect to the hidden states times t 1,h
Oh!
Jely Th
Beyond the hidden states, the Jacobian will depend on the input x4. This dependence allows to capture the dynamics of a network as it responds to input. The expansion factor of each vector is calculated by updating Q by computing the QR decomposition at each time step.
Q++1. R++1 (J(
If r is the expansion factor of the 7th vector at time step f- corresponding to the 7th diagonal element of R in the QR decomposition- then the 7th LE d resulting from an input signal of length T is given by
1 T log (r:)
A
T 1
The LE resulting from each input " in the batch of input sequences is calculated in parallel and then averaged. For each experiment, the LE was calculated over fixed number of time steps with different input sequences. The mean n resulting LE spectra reported as the LE spectrum. To normalize the spectra across different network sizes and, consequently the number of LE in the spectrum, we interpolate the spectrum such that retains the shape of the largest network size.
We follow the principles outlined by Engelken et al. for calculating the Lyapunov spectrum of the discrete-time firing-rate network. Conducting numerical simulations with small perturbations in Recurrent Spiking Neural Networks (RSNNs), particularly in the context of discrete spiking events, requires specific approach to capture these networks' dynamics accurately.
1. Network Initialization: Set up an RSNN with a defined architecture, synaptic weights, and initial neuronal states.
2. Creating Perturbations: Generate a slightly perturbed version of the network. This could involve minor adjustments to the initial membrane potentials other state variables of subset of neurons.
3. Simulating Network Dynamics: Run parallel simulations of the original and the perturbed RSNNs, ensuring they receive identical input stimuli. Since the networks operate discrete spike events, the state of each neuron updated based on the inputs it receives and its current potential
4. Measuring Divergence: At each time step, measure the difference between the states of the two networks. In the context of spiking neurons, this could involve comparing the spike trains of corresponding neurons in each network. This difference could be quantified using various metrics, such as spike-timing difference, spike count difference, or membrane potential differences.
5. Tracking the Evolution of the Perturbation: Observe how the initial small differences evolve. These differences might lead to significantly divergent spiking patterns in a network exhibiting chaotic dynamics.
15

---

Published as a conference paper at ICLR 2024
6. Estimating Lyapunov Exponents:
Calculate the rate which the trajectories of the original and perturbed networks diverge. This involves fitting an exponential curve to the divergence data over time. The slope of this curve gives estimate of the Lyapunov exponent. A positive exponent indicates sensitivity to initial conditions and potential chaotic dynamics. The evolution of the map is given by
b4 (b. At) - /-= (1- At Σ 2=1
Juj (t.))
In the limit At + 0, a continuous-time dynamies is recovered. For At 1, the discrete-time network is obtained. The Jacobian for the discrete-time map is
0f.
Dú (t.) 0h, 1-t,,
= (1-118 -11- + At Jejø' (hy (1.)).
The full Lyapunov spectrum is again obtained by a reorthonormmalization procedure of the Jacobians along a numerical solution of the map.
A.2 LINEARIZATION AROUND CRITICAL POINTS
Recurrent neural networks (RNNs) are useful tools for learning nonlinear relationships between time-varying inputs and outputs with complex temporal dependencies. Sussillo et al. [Sussillo & Barak (2013) have explored the hypothesis that fixed points, both stable and unstable, and the linearized dynamics around them, can reveal crucial aspects of how RNNs implement their computations. Further, they explored the utility of linearization in areas phase space that are not true fixed points but merely points of very slow movement and presented simple optimization technique that is applied to trained RNNs to find the fixed and slow points their dynamics. Lincarization around these slow regions other recent works Sadeh & Rotter (2014) have shown that, for a wide variety of connectivity patterns, linear theory based on firing rates accurately approximates the outcome of direct numerical simulations of networks of spiking neurons.
be used explore, or reverse-engineer, the behavior of the RNN. Similarly,
Building these works, we can linearize the HRSNN model around the critical points using the differential equation:
dx dt
= Dx Lx + b(t) Ax + b(t)
(6)
In this equation:
represents the vector of firing rates of neurons in the network. D is diagonal matrix indicating neurons' intrinsic leak or excitability. is the Lyapunov matrix. b(t) denotes external inputs, including biases. A is matrix defined as A D T..
The spike frequency of untrained SNNs used to approximate the firing rates (x) in the network. In untrained SNN, the firing rates can be considered raw initial responses to inputs before any learning or adaptation has occurred. This approximation is useful for constructing a linearized model it provides a baseline from which the effects of learning, pruning, and other dynamics can be analyzed. Each diagonal element D, of the matrix D quantifies how the firing rate of neuron i changes over time without external input or interaction. These elements can be determined based on the inherent properties of the neurons in the network, such as their leakiness or excitability. The Lyapunov matrix encapsulates the impact of neighboring nodes on each edge regarding their Lyapunov exponents. The elements of L are calculated using the harmonic mean of the Lyapunov exponents of the neighbors of nodes and j as detailed in your method. This matrix represents how the dynamics of one neuron affect its neighbors, influenced by the network's 's overall stability and dynamical behavior. In summary, the linearized model provided by equation 6 is a simplification that helps to understand the fundamental dynamics of the SNN. It uses the initial, untrained spike frequencies to establish a baseline for the network's 's behavior, and the matrices D and are calculated based on the intrinsic properties of the neurons and their interactions, respectively.
16

---

Published as a conference paper at ICLR 2024
We aim to create sparse network (A4pasee with fewer edges while maintaining dynamics similar to the original network. The sparse network is thus represented as:
dæ dt
Aspars b(t) such that
| (A A) x) sc Aar
V
(7)
for some small O. When the network in Eq. 6 driven by independent noise each node, define b(t) = b c where is a constant input vector, & is vector of III Gaussian white noise, and is the noise standard deviation. Let E be the covariance matrix of the firing rates in response this input. The probability Pij for the synapse from neuron j to neuron i with the Lyapunov exponent bi defined
P.j [pli + Σjj 2E()) (1iis(EEi Σjj 2Eij)
for Wij > 0 (excitatory) for waj 0 (inhibitory)
(8)
Here, p determines the density of the pruned network. The pruning process independently preserves each edge with probability Pty. yielding A where A A.purse A.y/Pij, with probability Piy and bj 0 otherwise. For the diagonal elements, denoted as A representing leak/excitability, we either retain the original diagonal, setting A Ais or we introduce perturbation, A. defined as the difference total input to neuron i, and adjust the diagonal as A = AA A. Specifically, A; Ej [A,)) ypurse Ejec [Ayil- This perturbation, A., is typically minimal with a zero mean and interpreted biologically modification in the excitability of neuron i due alterations in total input, aligning with the known homeostatic regulation of excitability.
A.3 BASELINE PRUNING METHODS
Activity pruning is a technique employed to optimize neural network models by iteratively removing the least active neurons. In this approach, outlined the Iterative Activity Pruning algorithm, the process starts with an initial Recurrent Spiking Neural Network (RSNN) model M. The algorithm operates by first evaluating each neuron's activity level, followed by pruning certain percentage (determined by the pruning rate r) of neurons that cxhibit the lowest activity. Post pruning, the model M is retrained compensate for the loss of neurons, forming an updated model M'. This cycle of pruning and retraining continues until either a maximum number of iterations T is reached, or the performance of the pruned model drops below 10% of the original, unpruned model's performance. The goal of this method is to refine the model by removing less critical neurons while maintaining or enhancing overall performance, This technique validated by comparing its efficacy classifying datasets like CIFARIO & CIFAR 100 against other state-of-the-ae pruming algorithms. The detailed algorithm is given in Algorithm 4 We also evaluate our model for classifying the CIFARIO & CIFAR 100 datasets and compare the results with current task-dependent oftte-of-the-ar pruning algorithms. The results are shown in Table T
A.4 DATASETS
A.4.1 LORENZ SYSTEM
The Lorenz system is non-linear, three-dimensional system that can be described as follows:
dr/dt σ xx) dy/dt = x(p- 2)
dz/dt zy 8z
when 10, 3 8/3, and p= 28, the system has chaotic solutions. The experimental setup, same as Xu et al. (2018a), was used in this paper, and the fourth-order Runge-Kutta method was used to generate samples. Table 3 summarizes the details of the experimental setup.
17

---

Published as a conference paper at ICLR 2024
Table 3: Details of the experimental setup for the Lorenz system.
Parameter
Value
Number of samples
20000
Initial state
[12,2.9]
Step size
0.01
Number of training samples
11250
Number of validation samples
3750
Number of test samples
5000
Table Details of the experimental setup for the Rossler system.
Parameter
Value
Number of samples
12700
Initial state
[1, 1,
Step size
0.03
Number of discarded samples
7700
Number of training samples
3000
Number of validation samples
1000
Number of test samples
1000
A.4.2 ROSSLER SYSTEM
The Rossler system is classical system, consisting of three nonlinear ordinary differential equations and can be defined by:
dx/dt = y -
dy/dl dl + az dz/dt = =(
when 0.15, 0.2, and 10, the system shows chaotic behavior. To compare the performance of MFRFNN with other methods under the same condition. the experimental setup, same as Xu et al.) (0188), was used for the Rossler system. In this setup, the fourth-order Runge-Kutte method was employed for sample generation. Some of the samples were discarded to eliminate the transient influence of the initial condition. Table 5 presents the details of the experimental setup for the Rossler system.
A.4.3 GOOGLE STOCK PRICE PREDICTION PROBLEM
Stock price prediction is non-linear and highly volatile problem. In this problem, the future value of Google stock price predicted using the current price as defined by.
j(t) f(
The dataset was obtained from Yahoo Finance during six years from 19-August-2004 to 21 -Seetember-- 2010 as in Samanta al. (2020). The training set consisted of 1529 samples, and the test set of 900 samples. To evaluate the performance of MFRENN on another real-world time series, we compared its performance with the same RFNNs and FNNs used in Box-Jenkin and wind speed prediction datasets.
A.4.4 WIND PREDICTION
The wind speed prediction problem is a non-linear, dynamic, and volatile problem in which the future value of wind speed is predicted using the current wind speed and wind direction. The dataset obtained from the lowa Department of Transport's website.1 1 The data was collected from the Washington station during one-month period (February 2011), sampled every ten minutes, and averaged hourly. There are 500 samples in the training set and 1000 samples in the test set Samanta al. 2020).. This dataset more challenging than the Box-Jenkins dataset due to the existence of
18

---

Published as a conference paper at ICLR 2024
Encoding Layer
Decoding Layer
Readout Layer
Signal
Spike Trains
XRSNN
Decoded Output Spikes
Decoded Output Signals
Fully Connected Layers
Figure Block Diagram showing the methodology using HRSNN for prediction
noise. This experiment compared the proposed method's performance with the same algorithms used in the Box-Jenkins dataset.
A.5 CALCULATION OF SOPS
evaluating the energy efficiency of neuromorphic chips, a key metric the average energy consumption for transmitting single spike across synapse, as highlighted Furber (2016) Shi et al. (2024). This metric particularly important due to the substantial energy expenditure involved in synapse processing, which significantly impacts the overall energy usage. For a theoretical analysis that is independent of specific hardware, we consider the average energy for fixed constant. The total energy consumption of Spiking
some recent works
single spike-synapse transmission Neural Network (SNN) model can be approximated by tallying the synaptic operations (SOPs) required, analogous to counting floating-point operations (FLOPs) in Artificial Neural Networks (ANNs). Our model for calculating the SNN's energy consumption is expressed as:
F = CE SOP CE Σ 8,03
Here, CF represents the energy usage per SOP, and SOP Σ, 8;G; is the cumulative count of synaptic operations. any given presynaptic neuron i, 8, signifies the spike count emitted by that neuron, while e indicates its synaptic connections. Each spike transmission from this neuron triggers synaptic operation as reaches the postsynaptic neurons, contributing to the energy expenditure.
In the context of sparse SNNs, the energy consumption model is modified follows:
E= CE Σ 8, TT Ozg n
post
(10)
In this equation, for every synaptic link from the i-th presynaptic neuron to its j-th postsynaptic neuron, the tuple (n.Th Ozg, miy post) each element of which can be either or 1, represents the states of the presynaptic neuron, the synapse, and the postsynaptic neuron, respectively, here, indicates an active state, and denotes pruned state, the symbol stands for the logical AND operation.
A.6 HRSNN MODEL
HRSNN Model Architecture: Fig. shows the overall architecture of the prediction model. Using a rate-encoding methodology, the time-series data encoded series spike trains. This high- dimensional spike train acts as the input to HRSNN. The output spike trains from HRSNN act the input to a decoder and a readout layer that finally gives the prediction results. For the classification task, we similar method. However, we do not the decoding layer for the signal but directly feed the output spike signals from HRSNN into the fully connected layer.
The read-out layer is task- dependent and uses supervised training. In this paper, do not explicitly prune the read-out network, but the readout layer is implicitly pruned. The readout layer is multi-layer (two three layers) fully connected network. The first layer size is equal the
Readout Layer:
19

---

Published as a conference paper at ICLR 2024
Table 5: Table showing the relative performance change when including the readout layer in the calculations
Neuron Spansiy Change
Synapse Spansity Change
SOP change CIFARID
Avg SOP CIFARIO0
HRSNN CHRSNN
-0.13
-6.18
-009
-0.14
-008
40.3
-0.1
-0.16
number of neurons sampled from the recurrent layer. We sample the top 10% of neurons with the greatest betweenness centrality. Thus, as the number of neurons in the recurrent layer decreases, the size of the first layer of the read-out network also decreases. The second layer consists of fixed size with 20 neurons, while the third layer differs between the classification and the prediction tasks such that for classífication, the number of neurons in the third layer is equal to the number of classes. On the other hand, the third layer for the prediction task is single neuron which gives the prediction output. The table shows the relative change in sparsity when including/cxcllddi the readout layer for calculations:
A.7 DYNAMIC CHARACTERIZATION USING LYAPUNOV SPECTRA
To show the principal dynamic characteristic of the INP-HRSNN model, we plot the full Lyapunov spectrum of the HRSNN model for three different cases the unpruned network, the pruned network, and the trained pruned network. We refer to the methodologies discussed in recent works VVgg et al. 2020); Engelken et al. (003). The Lyapunov Spectrum provides valuable additional insights into the collective dynamics of firing-rat rate networks. We plot the evolution of Lyapunov spectra with different initialization parameters. We plotted the Lyapunov spectrum, with three different probabilities of synaptic connection for the initial network (p=0.001, p=0.01, p=0.1). We plot the I Lyppnnov exponents (X₂) vs the normalized indices i/N described as follows:
d5: This axis represents the Lyapunov exponents. A positive exponent indicates chaos, meaning that two nearby trajectories in the phase space will diverge exponentially, A negative exponent suggests that trajectories converge, and zero would imply neutral stability,
i/: This axis is likely indexing the normalized Lyapunov exponents, with being the index a particular exponent and being the total number of exponents calculated. For a system with N dimensions, there are N Lyapunov exponents.
The graph shows three plots of the Lyapunov spectrum for different stages of pruning and training:
1. Unpruned:
This plot represents the Lyapunov spectrum of the neural network before any pruning has been done. The spectrum shows a range of Lyapunov exponents from positive to negative values, indicating that the network likely has both stable and chaotic behaviors. More notably, the more sparse initialized models (p = 0.1) were more unstable as the largest Lyapunov exponent is positive.
2. Pruned Pre-Training This plot shows the Lyapunov spectrum after the network has been pruned but before it has been trained again. Pruning a process in which less important connections (weights) in neural network removed, which can simplify the network and potentially lead to more efficient operation without significantly impacting performance.
3. Pruned Post-Training This plot illustrates the Lyapumov spectrum after the network has been pruned and then trained again. Retraining the network after pruning allows it to adjust the remaining weights to compensate for the removed connections, which can restore or even improve performance.
We can infer the following from the plots: Impact of Pruning: Comparing the 'Unpruned' to the 'Pruned Pre-Training' plot, it seems that pruning increases the stability of the system as the Lyapunov exponents become more negative (less chaotic). This might suggest that pruning reduces the complexity of the dynamics.
Training Effect:
The Prunndd Post- plot shows that after re- trainiiing, the exponents are less negative compared to 'Pruned Pre- Training'. which may indicate that the network is able to regain some dynamical complexity or expressive power through training.
20

---

Published as a conference paper at ICLR 2024
p=0.001
p=0.01
p=0.1
Unpruned
Pruned Pre-Training
Pruned Post- Pest-Training
0.0
0.0
-2.
22.
25
-5.0
-3.0
-3.0
-7.
-7.
-7.5
-10.0
10.0
10.00
12.5
15.0
1.0
1.0
I/N
i/N
I/N
(a)
(b)
(c)
Figure 7: Lyapunov Spectrum of the HRSNN model in three stages of pruning and training and different levels of sparsity for initialization: (a) the Lyapunov Spectrum of unpruned HRSNN model with probability connection = 0.1, 0.01, 0.001 (b) the Lyapunov Spectrum after LNP pruning
(c) the Lyapunov Spectrum after training of the pruned model
Initialization Synapse Density:
Different initialization synapse densities (arising from different p) have different effects the spectrum. High p) tends to result in more negative exponents, suggesting greater stability but potentially less capacity to model complex patterns.
Network Robustness: The fact that the Lyapunov spectrum does not dramatically change in shape across the three stages, especially in the 'Pruned Post- 'eatTTraning' stage, might imply that the network robust pruning, retaining its general dynamical behavior while likely improving efficiency.
The convergence of the spectra for different values of after pruning, both pre-and post-training, suggests that regardless the initial density of connections, the network may evolve towards a structure that is similarly efficient for the task it is being trained on. This indicates that the LNP algorithm can efficiently prune network irrespective of the initial density and also make the final model more stable.
Convergence of Spectra:
Overall, the Lyapunov Spectrum serves as useful tool for understanding how network pruning and re-training affect the dynamics of neural networks, which is important for optimizing neural networks for better performance and efficiency.
Comparison with Randomly Sparse Initialization: We can get idea of the principal dynamic characteristic LNP-HRSNN from the Lyapunov spectrum of the randomly initialized network with very low probability of connection (p=0.001). We see that such networks the largest Lyapunov exponent was positive. This unstable behavior might be because randomly generated networks lack structured connections that might otherwise guide constrain the flow of neural activity. Without these structures, the network is more likely to exhibit erratic behavior as the activity patterns less predictable and more susceptible amplification small perturbations, making them more unstable.
more unstable as
B SUPPLEMENTARY SECTION B
B.1
ADDITIONAL RESULTS
Performance Evaluation
For a more extensive performance evaluation of the LNP pruning method, we test the model on different datasets and note the RMSE loss and the VPT. The results are shown in Table From the
21

---

Published as a conference paper at ICLR 2024
Table 6: Performance of HRSNN and CHRSNN models using different pruning methods. Each model is trained on the first 200 timesteps and predicts the next 100 timesteps.
Syathelic
Real-world
Pruning Method
Model
SOP
Lorenzó3
Rosaler
Google Stock
Wind
RMSE
VPT
RMSE
VPT
RMSR
VPT
RMSR
VPT
Unprused
CHRSNN
HRSSN 710.76 79.65
0.315
0.285
6,002
$0.75
0.125
0.142
0,019
41.32
3.14
L0M8
2
28.36 3.35
Pruned
CHRSNN
118.77 10.58
1,718
1.696
0.184
0.98
0,879
0.131
0.127
37.26
0.191
3.554
2.50
2:100:
0.076
23.7
16.88
LNP Prused
HRSNN CHRSNY
2:02
14.79& 1.58 B.079 a 0.088
0.105
4..17
30.21 4.16
0.388
0.314 0.102
0.051
50:15
4..36
0.501 a 0.301
0.127
30,25
0.04
0.301
0.006
28.06 2:25
27.98
L.26
for
Praning Herios
for CHIRSNN
Boxplot of RMSE by Noise Variance
2.0
104
RMSE (log)
1.0
0.5
Detesvi
(+)
(b)
Figure 8: (a) Bar graph showing the comparative analysis how efficiency (VPT/SOPs) changes for CHRSNN and HRSNN models for the different pruning methods (b) Box plot showing the change in performance of the three trained models obtained by the three pruning methods AP, and LNP
table, we can see that models undergoing LNP Pruning generally outperform their counterparts across variety of datasets and metries. For instance, focusing on the Real-world dataset 'Wind" under the RMSE metric, the LNP Pruned models, both HRSNN and CHRSNN, exhibit the lowest error rates with 0.314 0.049 and 0.301 0.035, respectively. The underlined values across LNP Pruned models indicate a consistently superior performance, representing the lowest RMSE and VPT values among the pruned and unpruned models across all datasets. Additionally, despite having smaller FLOPS value, indicative
lighter model, LNP pruned models, particularly CHRSNN with only 0.018 FLOPS, achieve superior comparable performance against the unpruned models, demonstrating the efficacy of LNP in maintaining model accuracy while reducing computational complexity. The marked efficiency and performance enhancement accentuate the viability of LNP potent pruning strategy, rendering it optimal for environments where computational resources are constraint.
Comparing Efficiency We analyzed the performance of different pruning methods: Unpruned, Activity Pruned, and LNP Pruned, on four datasets: Lorenz63, Rossler, Google Stock, and Wind, using HRSNN and CHRSNN models, shown Suppl. Fig The bar graphs illustrate that the LNP Pruned method performs the best in terms of efficiency (delined as the ratio of VPT to FLOPS) across all datasets and models, highlighting its effectiveness in improving computational efficiency and resource use. The LNP Pruned method is particularly distinguished in CHRSNN models, where it achieves high VPT/FLOPS values in the Rossler and Lorenz63 datasets. The efficacy of this method stems from its capability to retain critical network parameters while discarding the redundant ones, thereby ensuring optimized model performance without compromising accuracy. Such optimization imperative in practical scenarios where computational resources are constrained, necessitating the development of efficient and effective models.
Comparison of Performance with Noise
Next, we test the stability of the models with varying levels of input noise. The results are plotted in Fig. offers an insightful depiction of the Prediction RMSE, represented scale, across varying input Signal-to-Noise Ratios (SNRs) for both HRSNN and CHRSNN models, subjected to different pruning methods: Unpruned, AP Pruned, and LNP Pruned. The key observation
logarithmic
22

---

Published as a conference paper at ICLR 2024
AP
HRSNN CHRSNN
Prediction RMSE (Log Scale)
HRSNN
CHRSNN
10a
Ungruned
HRSNN
CHRSNN
20
Input Signal SNR
Figure 9: Performance Analysis of Pruned and Unpruned Models with Varied Input SNR Levels
Table 7: Performance of the pruned and unpruned models with different input noise levels.
SNR (dB)
50
30
20
HRSNN
0.315=0.042
0.334+0.033
0.361=0.041
0.391
0.425+0.059
Unpruned
CHRSNN
0.285=0.011
0.304=0.038
0.338=0.042
0.354
0.362+0.048
HRSNN
1.718=0.195 1.988=0.229
2.101=0.284
2.51420.386
3.114=0.551
AP
CHRSNN
1.596=0.194
1.952+0.233
2.122=0.264
2.254=0.357
2.974=0.492
HRSNN
0.705=0.104 0.776=0.123
0.815=0.136
0.952+0.205
1.421=0.326
LNP
CHRSNN
0.679#0.098
0.714=0.111
0.754=0.129
0.825+0.201
1.397+0.294
from the figure is the universal increase in RMSE with the decrease in SNR, reflecting the inherent challenges associated with noise in input signals. Among the evaluated methods, AP Pruned models exhibit the highest RMSE across all SNR levels, indicating suboptimal performance under noise. In contrast, Unpruned models maintain the lowest RMSE, particularly at higher SNRs, showcasing their robustness to noise. The LNP Pruned method achieves balanced performance, demonstrating its capability to maintain considerable accuracy under noisy conditions while optimizing computational efficiency, hence affirming its practical applicability where a balance between accuracy and efficiency is essential. The visualization succinetly encapsulates the comparative performance dynamics, providing valuable insights into the interdependencies between noise levels, pruning methods, and model types.
Final Timescales the Pruned Network For this paper, all the experiments are initiated with 5000-neuron network. We have two different starting architectures the HRSNN model the CHRSNN model. As discussed in the text, the LNP Pruning algorithm, not only prunes the network but also optimizes the timescales. The final resultant timescale of this pruning method for given initialization of the model is shown in Fig. [0
B.2 USING THE LNP ALGORITIMM ON FEEDFORWARD NETWORKS
Modified Algorithm: In this section, extend the Lyapunov Noise Pruning Method for feedforward Networks. We note that this is not a trivial task, as the original method was engineered to exploit the recurrence present in the neuronal dynamics. Hence, for the case of feedforward neural networks,
23

---

Published as a conference paper at ICLR 2024
17.5
17.5
15.0
15.0
12.5
10.0
10.0
7.5
7.5
5.0
5.0
2.5
2.5
0.0
0.0
0
4
8
10
10
Timescale
Timescale
Lyapunov Noise Pruning (LNP)
Lyapunov Noise Pruning (LNP)
CHRSNN
HRSNN
Figure 10: Figure showing the final output timescales of the LNP pruning algorithm on the HRSNN and CHRSNN networks
solely focus on ResNet-based architectures, i.e., models with added skip connections. Thus, the four steps for the LNP-based pruning algorithm are modified as follows:
Step 1: Unstructured Pruning of Synapses: The process of synapse pruning is the same, where the synapses pruned using the noise-based pruning algorithm described earlier.
Step 2: Structured Pruning of Channels:
For feedforward SNNs like ResNets, we use channels instead of neurons for structured pruning. Let denote the k-th channel of the (-th layer in the network as oi. In each iteration, the average membrane potential of this channel the training set is computed
2 T 2 ((DD 1-1
Here, T' is the timestep of the SNN, and (t) represents the L1 norm of the membranc potential of the channel's feature map at time t. The feature map's 's membrane potential for specific neuron at time is denoted as V. with the channel's membrane potential being [V)( [v:2] hxm- A positive Vi indicates excitatory postsynaptic potential (EPSP), whereas a negative value signifies an inhibitory postsynaptic potential (IPSP).
To optimize the network, prune convolutional kernels based on the calculated channel importance scores. Channels with lower scores are deemed redundant and are more likely to be pruned. This method ensures a balanced reduction of redundant components across the network, enhancing compression and maintaining performance. We use a mask to track the structure of the network after pruning, where value of 0 indicates pruned channel and signifies a retained channel.
Step 3: Norm Preservation using Skip Connections: Norm preservation ResNets is the property that the norm of the gradient of the network with respect to the input is close to the norm of the gradient with respect to the output. This property desirable because means that the network can avoid the vanishing or exploding gradient problem, which can hamper the optimization process. Norm preservation is facilitated by the skip connections in the residual blocks, which allow the gradient to flow directly from the output to the input. Norm preservation is also enhanced as the network becomes deeper because the residual blocks act as identity mappings that preserve the norm of the gradient. After the unstructured and structured pruning in Steps and 2, we reintroduce some additional skip connections for preservation.
24

---

Published as a conference paper at ICLR 2024
Table 8: Table showing the performance of LNP on Feedforward Neural Networks
CIFARIO
CIFARI00
Baseline Accuracy
Neuron Sparsity
Synapse Sparsity
Baseline Accuracy
Neuron Sparsity
Synapse Sparsity
Model
Acuracy
Acuracy
ResNet19
92.11
2.15
73.32
-3.66=
90.48
94.32
90.48
94,32
(untrained)
0.9
0.19
0.81
0.39
ResNe119*
93.29
-0.04
74.66
-0.11
93.67
98.19
94.44
98.07
(converted) 0.74
0.01
0.65
0.02
Step 4: Neuronal Timescale Optimization: Similar to the previous case, we optimize the hyperparameters of the spiking ResNet using the Lyapunov spectrum-based Bayesian Optimization technique.
Experiments and Evaluations: for the norm preservation step, hence, for evaluating the performance of the LNP on the feedforward Neural networks, we look into the LesNet-bbsee model. We evaluate the model for CIFAR 10 and CIFARI00 datasets with two different initializations: For the first case, use the untrained ResNet19 network, then prune it and finally train it and get the model performance. For the second case, we use the pre-trained SNN, which is converted from ANN using ANN-to-SNN conversion techniques, and then use the LNP pruning on that ResNet model. The results are shown in Table We see that LNP gives extremely good results when used on the SNN, which converted from the ANN model. Hence opening up another possible application of the LNP pruning method.
Now, since the model pruning heavily relies on the skip connections
B.3 GENERALIZATION PROPERTIES
As discussed earlier, our proposed LNP pruning algorithm is task task- agnostici and does not use a dataset to train while pruning. This pre-training pruning algorithm makes the model extremely generalizable as opposed to current state-of-the-art pruning methods, which require iterative pruning and retraining. This iterative process makes these pruned models extremely overfitted to the dataset it is trained on, and would thus require retraining and repruning of the model for each dataset.
Our LNP algorithm starts with an untrained dense network, and the pruning process does not consider task performance while removing network connections (and neurons) and neuronal time scales. Consequently, the end pruned network does not overfit to any particular dataset; rather generalizable to many different datasets, and even different tasks. On the other hand, prior SNN pruning papers, SNN. During pruning, the
start with pre-trained DNN model on a given task which is converted network is simultaneously pruned and re-trained with the dataset (at each pruning step) to minimize performance loss on that dataset. This makes the model overfitted (and hence, shows good results) to that dataset. We use the following terminology for the different pruning cases:
Untrained SNN: The SNN model is not trained before pruning, and the parameters are randomly initialized. Only the final pruned network is then trained
Converted SNN: A standard DNN was trained on a dataset and then converted to an SNN with the same architecture using DNN-to-SNN conversion methods.
To verify our conjecture, we empirically study the following questions:
1. How does our pruning method perform on a converted SNN model like prior works?
Our Approach 1 (ResNet converted); We consider two cases. First, we use our pruning method on a dense Spiking ResNet, which is converted from DNN ResNet (trained on the CIFAR10 and CIFAR 100 datasets). We continue with our approach where the pruning iterations do not consider task performance. We observe that when starting from pre-trained dense model like prior works, the performance and sparsity of the network pruned with LNP are better than prior pruning methods.
Our Approach (ResNet untrained): We start with an untrained ResNet with randomly initialized weights and parameters. They use our LNP algorithm without training the model any iteration of the pruning process. The final pruned network
25

---

Published as a conference paper at ICLR 2024
Table 9: Table showing the generalization performance of the LNP algorithm compared other
baseline pruning methods
Pruning Method
Neuron Sparsity
Synapse Sparsity
CIFARIO Accuracy
CIFARIOO Accuracy
Architecture
ResNer19 (iterative praning
IMP Kim et al. (202bb)
97.54
93.18
68.95
and retraining) ResNet19
LNP
90.48
94.32
89.96 0.71
69.76 + 0.42
(Ours)
(untrained)
LNP
RexNet
(Ours)
(pre-trained)
93.67
98.19
93.25 * 0.73
74.55 * 0.63
with optimized hyperparameters (optimized using the Lyapunov spectrum) is then trained and tested on the CIFAR1O and CIFAR100 datasets
Approach by IMP Kim et al. (202bb): Iterative Magnitude Pruning (IMP) a technique for neural network compression that involves alternating between pruning and retraining steps. In this process, a small percentage of the least significant weights are pruned, and the network is then fine-tuned recover performance. This iterative cycle continues until desired level of sparsity is reached, potentially enhancing network efficiency with minimal impact on accuracy.
2. How do other task-dependent pruning methods perform when the network pruned for one task is used for a different task?
To compare this generalizability property, we took the pruned model from the works of Kim et al. (2022b), which is optimized for CIFARIO. We next train their pruned model on CIFARION sparsity, accuracy and SOPs. We compare the performance with the performance of our pruned untrained ResNet model (with random weights) and pruned converted ResNet (as described earlier). We see that the untrained ResNet generated by our LNP-hased pruning can generalize better the CIFAR100 100 dataset and show better performance. Moreover, we see that using the converted ResNet, we see better performance and efficiency of the pruned ResNet (converted) than the IMP-hased pruning model on both CIFAR 10 and CIFAR 100 datasets.
The results are shown in the Table g
B.4 SCATTER PLOTS
To get a better understanding of the performance efficiency of the final pruned models of the LNP method compared to the state-of-the-ane pruning models. We plot the accuracy vs average SOPs for the 10, and CIFAR100 classification tasks and the VPT vs. average. SOPs for the Lorenz63 prediction task. The results are shown in Fig. and [2] We observe that the LNP-based methods always show better performance with lower SOPs compared to other sfate-of-the-at pruning methods.
B.5 ABLATION STUDY
We conducted an ablation study, where we systematically examined various combinations of the four sequential steps involved in the LNP method. This study's findings are presented graphically, as illustrated in Fig. [3 At each point (A-E), we train the model and obtain the model's 's accuracy and average. Synaptic Operations (SOPs) to support ablation studies. The ablation study is done for the HRSNN model, which is trained using STDP and evaluated on the CIFAR 10 dataset.
In the figure, different line styles and colors represent distinct aspects of the procedure: the blue line corresponds to Steps and 2 of the LNP process, the orange line to Step 3, and the green line to Step 4. Solid lines depict the progression of the original LNP process (A + B + C + D + E). while dotted lines represent the ablation studies conducted at stages B and C. This visual representation enables a clear understanding of the individual impact each step exerts on the model's performance and efficiency. Additionally, it provides insights into potential alternative outcomes that might have arisen from employing different permutations of these steps or omitting certain steps altogether.
Below is detailed discussion of the figure:
26

---

Published as a conference paper at ICLR 2024
LNP Reskert8
LNP-DesNeet1
(60m 2002
Gradit o Comy,
ESLSNN Reshett (Shen al, 2023)
LNP-CHRSNN-EP
HRS
STO6 Cony, (Chvn all 20000)
CHISSNN STDP
Acsuracy (%)
LENN (Bolice ot 2014)
ESSLSNN Hesterr1
HRSNN STOP
LNP RcdNett9
85-
(Kmm si., 2002b)
LNP-CHRSWN-STDP
Archiecture
Con, S/D
INP-HRSNN-BP +
#7 Con: PC
Testlet
GradH 6 Cony, FC (Chen et 2021)
Rearte10
Comu
e Henter
*HSSH
+ LNP-HRSNN-STDP
100
2 10
2x Lo!
6
Aug. SOPs (log)
Avg. SOPs (M) (teg)
(a)
(b)
Figure 11: Scatter Plot showing the Performance vs SOPs for (a) CIFAR IO and (b) CIFAR 100
CHRSAN X
Google
Unprusedd
CHRSNN 8P
HABNN
Valid Prediction Time (VPT)
CHRSAN STOP X
x CHEESNNE
CHRSMN-R
HRSAI-ETOP
CHRSNN-BP
HRBNNLBP X
36 HUSENNA0PP
30
HAIGHN-BID
X CHRBAN STOF HRSMN- BP
CHPSNN-DP
HRSNN STOP
Avg. SOPs
Figure 12: Scatter Plot showing the Performance vs SOPs for Lorenz63 dataset
27

---

Published as a conference paper at ICLR 2024
Pruning Ablation
90
E
Accuracy (96)
A: Randomiy Initialized HRSNN
Post Synapse Pruning
65
C: Post Neuron Pruning
Post Eigenvector Preservation
E: Post Timescale Optimization
20
50
100
300
Average SOPs (M)
(log-scale)
Figure 13: Plot showing Ablation studies of LNP
1. We start with Point A, which represents the randomly initialized unpruned (dense) HRSNN network
The blue line (A B) indicates Step 1 of the LNP algorithm, where we prune the synapses using noise-based pruning method.
2. Point B is the outcome of synapse pruning the randomly initialized network. There are different directions can go from here:
Step 2 (The standard LNP step): The blue line + indicates Step of the LNP method, where prune the neurons based on the betweenness centrality. This is the standard step used in the LNP algorithm.
Step 3: The orange line B D' indicates Step (eigenvector preservation) done directly after Step (skipping Step 2 or the neuron pruning step).
This is followed by Step 4 (Timescale Optimization) to reach the ablation model E'. We see the performance E' quite high but the Avg. SOPs much higher than E (LNP solution).
Step 4: We can directly optimize the neuron timescales from B (skipping steps 2.3). reaching E". However, the performance even worse.
3. Point C is derived by using Step 2 (neuron pruning) on Point B. This reduces the SOPs significantly, but also the performance deteriorates. We can go ways from this point:
Step (The standard LNP step): We can do eigenvector preservation reach point D. Step (Timescale Optimization) directly (skipping Step 3)- This leads model with very sparse connections, but performance also takes big hit.
4. Point D derived by using Step (Eigenvector Preservation) Point C. The model perfor- mance increases a bit, but the SOPs also increase due to the addition new synapses in this step.
5. Point This is the final output of the LNP algorithm where we use Step 4 (Timescale Optimization) at point D. We see this process gives the model with the best performance and SOP.
28

---

Published as a conference paper at ICLR 2024
C SUPPLEMENTARY SECTION C
C.I RELATED WORKS
Pruning Methods for DNN Pruning methods in Deep Neural Networks (DNNs) are crucial for optimizing models, especially for deployment in resource-constrained environments. One common technique is magnitude-based pruning, where components of the network are selectively pruned based their magnitudes, often without the need for retraining the model post-pruning Hoefler ct al. (02T): Vadera & Ameen (2020). Another prevalent method is filter pruning, which focuses on pruning filters in convolutional layers to reduce the computational cost while maintaining model performance He et al. (2022): Kulkarni et al. (2022). Structured pruning is also employed, where pruning is performed in a structured manner, and the pruning rate for each layer is adaptively derived based on gradient and loss function Sakai et al. (2022). Sensitivity analysis is used to assess the effect of pruning, allowing for the identification of redundant components that can be removed with minimal impact on model accuracy Vadera & Ameen (2020). Multi-objective sensitivity pruning considers Hardware-based objectives such as latency and energy consumption along with model accuracy to optimize the pruning process (Sabih et al. (2022). Deep (-learning-basee methods like QLP intelligently determine favorable layer-wise sparsity ratios, implementing them via unstructured, magnitude-basel weight pruning Camci et al. (2022).
Pruning Methods for SNN
Pruning methods in Spiking Neural Networks (SNNs) are essential for optimizing models for deployment in resource-constrained environments. Gradient Rewiring method inspired by synaptogenesis and synapse climination in the neural system, allowing for the optimization of network structure without retraining, maintaining minimal loss of SNNs' performance on various datasets. Spatio-temporal pruning is another method that utilizes principal component analysis to perform structured spatial pruning, leading to significant model compression and latency reduction Chowdhury al. (202Ta). DynSNN proposes dynamic pruning framework that optimizes network lopology on the fly, achieving almost lossless performance for SNNs on multiple datasets Liu et al. (2022a). u-Ticket addresses the workload imbalance problem in sparse SNNs by adjusting the weight connections during Lottery Ticket Hypothesis-basee pruning, ensuring optimal hardware utilization prasticity-inspiree Adaptive Pruning (DPAP) draws inspiration from the brain's ss developmental plasticity mechanisms to dynamically optimize network structure during learning without pre-training and retraining, achieving efficient network architectures Han et al. (022)). Multi-strength SNNs employ an innovative deep structure that allows for aggressive pruning strategies, reducing computational operations significantly while maintaining accuracy.
Yin et al. (2023). Developmental
Spiking Neural Networks
Spiking neural networks (SNNs) Ponulak& & Kasinski| (201T) use bio-inspired neurons and synap- tic connections, trainable with either unsupervised localized learning rules such as spike-timing- dependent plasticity (STDP) Gerstner & Kistler (2002): Chakraborty et al. (2023) or supervised backpropagation based learning algorithms such surrogate gradient Neftei et al. (2019) SNNs are gaining popularity as a powerful low-power alternative to deep neural networks for various machine learning tasks. SNNs process information using binary spike representation, eliminating the need for multiplication operations during inference. Recent advances in neuromorphic hardware Akopyan al. (2015). Davies et al. (2018), Kim et al. (2022a) have shown that SNNs can save energy by orders of magnitude compared to artificial neural networks (ANNs), maintaining similar performance levels. Since these networks are increasingly crucial as efficient learning methods. understanding and comparing the representations learned by different supervised and unsupervised learning models become essential. Empirical results on standard SNNs also show good performance for various tasks, including spatiotemporal data classification, Lee et al. (2017): Khoei et al. (002)). scquence-to-sequenee mapping Chakraborty & Mukhopadhyay (2023a) Zhang & Li 2020), object detection Chakraborty al. (2021): 1y Kim et al. (2020), and universal function approximation Gelenbel et al. 1999); Jannella & Back (2001). Furthermore, recent research has demonstrated that introducing heterogeneity in the neuronal dynamics
et al. (222I); Chakraborty & Mukhopadhyay (2023b) 2022): (She et al. (202T) can enhance the model's 's performance to levels akin to supervised learning models.
29

---

Published as a conference paper at ICLR 2024
STDP-based learning in Recurrent Spiking Neural Network Baike-Timing-eepeee
Plasticity
(STDP) Gerstner et al. (1993) Chakraborty & Mukhopadhyay (202I) based learning in recurrent Spiking Neural Networks (SNNs) IS biologically inspired learning mechanism that relies on the precise timing of spikes for synaptic weight adjustment. STDP enables the network to learn and generate sequences and abstract hidden states from sensory inputs, making it erucial for tasks like pattern recognition and sequence generation in recurrent SNNs. For instance, a study by Guo et al. Guo et al. (02TT) proposed a supervised learning algorithm for recurrent SNNs based on BP-STDP. focusing on optimizing learning in a structured manner. Another research by van der Veen van der Veen (202) explored the incorporation of STDP-like behavior in eligibility propagation within multi- layer recurrent SNNs, demonstrating improved classification performance in certain neuron models. Chakraborty et al. Chakraborty Mukhopadhyay (2022) presented a heterogeneous recurrent SNN for spatio-temporal classification, utilizing heterogeneous STDP with varying learning dynamics for each synapse, showing comparable performance to backpropagation-train
supervised SNNs with less computation and training data. Panda et al. |aanda & Roy (2017) combined Hebbian plasticity with a non-Hebbian synaptic decay mechanism in a recurrent spiking model to learn stable contextual dependencies between temporal sequences, suppressing chaotic activity and enhancing the model's ability to generate sequences consistently.
Spectral Graph Sparsification Algorithm
Spectral graph sparsification algorithms aim to reduce the complexity of graphs while preserving their spectral properties, particularly the eigenvalues of the Laplacian matrix. One such algorithm is the unweighted spectral graph sparsification algorithm, which constructs a sparsifier with fewer edges but maintains comparable graph Laplacian matrices [Anderson et al. (2014). Another approach, feGRASS, focuses on scalable power grid analysis, utilizing elfective edge weight and spectral edge similarity to construct low-stretch spanning trees and recover spectrally critical off-tree edges,
producing spectrally similar subgraphs efficiently Liu et al. (2022c). The MAC algorithm maximizes the algebraic connectivity of sparsified measurement graphs, providing high-quality sparsification results that retain the connectivity of the graph and the quality of corresponding SL.AM solutions Doherty et al. (2022). LGRASS is a linear graph spectral sparsification algorithm designed to run in strictly linear time, optimizing bottleneck subroutines and leveraging spanning tree properties Chen et al. (0226)) These algorithms are crucial for various applications, including power grid analysis, autonomous navigation, and other domains where graph analysis is essential.
D SUPPLEMENTARY SECTION D
D.1 BAYESIAN OPTIMIZATION FOR OPTIMIZING TIMESCALES
The majority of contemporary studies involving Bayesian Optimization (BO) predominantly focus on problems of lower dimensions, given that BO tends to perform poorly when applied to high- dimensional issues (Frazier) 2018). Nevertheless, this study endeavors to leverage BO to fine-tune the neuronal and synaptic parameters within diverse RSNN model. This application of BO implies the optimization of a substantial array of hyperparameters, making the use of conventional BO algorithms challenging. To address this, we employ an innovative BO algorithm, predicated on the notion that the hyperparameters under optimization are correlated and follow a specific probability distribution, indicated by Perez et all Perez-Nieves et al. (2021). Therefore, rather than targeting individual parameters, we modify BO to approximate parameter distributions for LIF neurons and STDP dynamics. Once optimal distributions are identified, samples drawn to determine the model's hyperparameter distribution. To ascertain the data's ' probability distribution, alterations are made to BO's surrogate model and acquisition function focus on parameter distributions over individual variables, enhancing scalability across all dimensions. The loss for updates to the surrogate model is determined using the Wasserstein distance between parameter distributions.
BO employs a Gaussian process to represent the objective function's 's distribution and utilizes an acquisition function select points for evaluation. For target dataset data points z X and corresponding labels y e Y, an SNN with network structure V and neuron parameters W serves as function fv.w (x:). mapping input data x to predicted label ÿ. The optimization problem in this study is articulated as
min V.W
Σ ((..fv.wz)))
(11)
WEX
30

---

Published as a conference paper at ICLR 2024
where V represents the neuron's 'ss hyperparameter set in R. (Hyperparameter details provided in the Supplementary), and W represents the multi-variate distribution comprising the distributions of various parameters, including the membrane time constants and the scaling function constants for the STDP learning rule SRR-
Furthermore, BO requires prior distribution of the objective function f(ø) based the provided data D I: {. (f In GP-based BO, is presumed that the prior distribution of f (T adheres the multivariate Gaussian distribution, following GP with mean ÄD and covariance ΣD We, therefore, calculate ΣD., using a modified Matern kernel function, with the loss function being the Wasserstein distance between the multivariate distributions of different parameters. For higher-dimensional spaces, the Sinkhorn distance is utilized as a regularized variant of the Wasserstein distance to approximate it (Feydy et al. 2019).
DEA represents the points evaluated by the objective function. The GP will predict the mean jrDe and variance TD.... for the remaining unevaluated data Dkm The acquisition function in this study is the expected improvement (EI) the prediction fitness as:
EI (Tten) (ji - (Thes) )) ((2) + TDR. (()
(12)
where 2(-) and 6(.) represent the probability distribution function and the cumulative dis- tribution function of the prior distributions, respectively. BO will select the data 2j argmax (EI (#km) ) ; ;. E ]kn) as the next point for evaluation using the original objective function.
E SUPPLEMENTARY SECTION E
E.1 ALGORITHMS
Algorithm 2 Compute Lyapunov Exponents
1: Input: Batch of input sequences, number of time steps, n different input sequences.
2: Output: Computed Lyapunov Exponents.
3: Initialize: Choose set of sequences as the validation set.
4: for each input sequence in the batch do
5: Initialize matrix Q identity matrix.
6: Initialize hidden states he as zeros.
7: end for
8: for each time step do
9: Compute Jacobian J, as:
Oh
[Jl
Ohj
10: Update Q and Compute Q decomposition:
Qt+1, R2.1 (RE==QQRJJO.))
end for
12: for each jth vector at time step t do Compute the 4th LE A; as:
Σ log(r!)) T t-l
14: end for
Compute the average LE for each input in the batch.
16: Normalize and interpolate the LE spectrum to retain the shape of the largest network size.
17: returnLyapunov Exponents.
31

---

Published as a conference paper at ICLR 2024
Algorithm 3 Pruning Nodes with Lowest Betweenness Centrality
1: Initialize: BC(u), T
cn((v)
2: Compute: (C V using =C( Σ
Sanst
Tst
3: Identify: ( ( BC(() T
4: Prune: G'(V', EE) by removing v e P and associated edges
5: Return: (V E') =
Algorithm 4 Iterative Activity Pruning
1: Input: Recurrent Spiking Neural Network (RSNN) model M
2: Parameters: Pruning rate r, maximum number iterations
3: Output: Pruned and retrained RSNN model M'
4: Initialize iteration counter: t
5: Evaluate the initial performance of the unpruned model Pinitill
6: while & and Paurrett 0.1 Pinitial do
7: Evaluate the activity of each neuron in model M obtain activity list A
8: Sort the neurons in A in ascending order based on activity
9: Calculate the number of neurons to prune: Npruee ceil(r-numblll of neurons in M)
10:
Prune the rpan least active neurons from model M
Retrain the pruned model M obtain the new model with updated parameters M Evaluate the performance Peurent of the model M'
12:
if Peurrent < 0.1 Pintiaa then
14:
Break the loop prevent further degradation in performance
15:
end if
16: Update model: M M
17: Increment iteration counter: t t. +
18:
end while
19: return Pruned and retrained model M' =0
32


Reviews:
[{"review_id": "GkG7odSXAR", "reviewer": "Reviewer_MeNm", "summary": "This paper proposes a novel Lyapunov noise pruning algorithm to design a sparse RSNN from an untrained RSNN. The method is task-agnostic, which can be trained later for different down-streaming tasks. The experimental results shows the effectiveness of this method.", "questions": "1.\tWhy focusing on recurrent SNN? I understand Lyapunov noise pruning (LNP) is originally developed for RNNs, why do you choose to consider this method in spiking cases?\n2.\tRSNN has two hierarchical dynamics, one is the recurrency of network, the other one is the intrinsic recurrency of spiking neurons, do you distinguish these two dynamics? Or on the other hand, whether this method can be used directly in spiking neural networks?\n3.\tWhat does the word “heterogeneous” mean here? Is it just the time constant is variable compared with conventional neuron models? (though in many literatures it is already considered)", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper is well-written, with very clear illustration on the methods and implementations. \n- This paper proposes a novel task-agnostic method for designing sparse RSNN, which is quite novel compared with conventional approach of designing task-dependent pruning way.\n- Compare to conventional network pruning methods of RSNN, the proposed method shows a very good generalization ability, proving by several down-streaming tasks’ consistent performance.", "weaknesses": "- The motivation is not fully explained, see the questions below.\n- Though in the experimental parts, it is showed to be advantageous compared other pruning methods, overall, it is still hard to see how useful this method can be in real cases. Namely, one might ask, compared to a SOTA method (not only RSNN) in a specific problem (say tasks listed in the experimental parts), how much gain (in terms of performance or efficiency) does this method bring?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "MiBZE3nv6j", "reviewer": "Reviewer_udA5", "summary": "This paper presents a novel task-agnostic method called Lyapunov Noise Pruning (LNP) for designing sparse heterogeneous Recurrent Spiking Neural Networks (HRSNNs). Unlike conventional approaches that prune dense networks after training them for a specific task, LNP starts with an untrained and arbitrarily initialized dense HRSNN model. It uses the Lyapunov spectrum and spectral graph sparsification algorithms to prune synapses and neurons while maintaining network stability. The resulting untrained sparse HRSNN can then be trained for various target tasks using supervised or unsupervised methods.", "questions": "1. In the study conducted by Panda et al., it was demonstrated that their approach resulted in a higher degree of sparsity and a lower impact on accuracy when compared to the LNP method. Could you please elucidate the underlying reasons for this observed outcome?\n\n2. Could you provide a comprehensive description of the HRSNN architecture? It appears that there is a readout layer characterized by dense connections that are not subjected to pruning via the LNP technique.\n\n3. The Lyapunov spectrum is customarily analyzed in continuous dynamical systems, such as RNNs. How did the researchers address the discontinuity of spikes in RSNNs when determining the Lyapunov spectrum?\n\n4. Kindly elucidate the principal dynamic characteristic of LNP RSNNs prior to the completion of the final training process and provide an explanation as to why randomly sparse Initialized RSNNs lack this particular dynamical property. Could you also discuss dynamic property of LNP-RSNN, for example, using temporal correlation function and the Lyapunov spectrum in this context?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. In this article, authors introduced a novel method for task-agnostic pruning, achieved by pruning a Recurrent Spiking Neural Network (RSNN) prior to the training process.\n\n2. The study demonstrates that Lyapunov Noise Pruning (LNP) outperforms AP in terms of network efficiency enhancement.\n\n3. This research proposes a technique that capitalizes on the heterogeneous timescales of RSNN, offering both biological plausibility and a supportive role in the pruning process.", "weaknesses": "1. The author did not show the difference between random initialization and LNP, and didnot analyze the dynamic property of the hetergeneous timescale of LNP and how it influence the training process of RSNN.\n\n2. The author didnot carefully organize the citation and references. For example,  in Reference section, 'Yanqi Chen, Zhaofei Yu, Wei Fang, Tiejun Huang, and Yonghong Tian. Pruning of deep spiking neural networks through gradient rewiring. arXiv preprint arXiv:2105.04916, 2021a.', and 'Yanqing Chen, Zhaofei Yu, Wei Fang, Tiejun Huang, and Yonghong Tian. Pruning of deep spiking neural networks through gradient rewiring. 2021b. doi: 10.24963/ijcai.2021/236. URL https://doi.org/10.24963/ijcai.2021/236.' are same paper, but sited twice.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "QCyBoUjgsH", "reviewer": "Reviewer_YTsp", "summary": "In this study, the task-agnostic Lyapunov Noise pruning (LNP) on Heterogeneous RSNN (HRSNN) is first proposed, and designed in a graph sparsification manner. HRSNN has a similar hierarchical structure as the Liquid State Machine (LSM), which includes a subsequent readout layer. LNP includes four steps 1) random dropout in HRSNN synaptic weights, where dropout rate is determined by the Lyapunov exponent, 2) mask out neurons with the lowest betweenness centrality from HRSNN, 3) add edges to maximize degree heterogeneity within local neighborhoods, 4) a Bayesian optimization for fine-tuning hyperparameter set of neurons. Results show that LNP maintains performance on time-series prediction on two chaotic systems, and two real-world datasets when FLOPS/#parameters are reduced.", "questions": "Considering that non-recurrent SNNs are still popular among SNN communities, is it possible to generalize and experiment on purely feed-forward SNNs like previous works do?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "LNP consider the equivalence between dense and sparse network, rather than simply optimizing the performance, which makes it suitable for random graph model like HRSNN. The design of LNP makes it not only task-agnostic but also learning-rule-agnostic, endowing SNNs from distinct research backgrounds with abundant flexibility in choice. Stability in pruning is also an interesting topic considering that SNN is a dynamic system.", "weaknesses": "As an SNN pruning algorithm, there are still some vague parts influencing my rate of score:\n\n1. The calculation of sparsity is unclear to me. In most previous SNN pruning studies, the parameter/neuron sparsity includes all learnable parameters and should also include readout layers here. This vagueness could lead to bias in evaluating the power of the pruning algorithm.\n2. The results on CIFAR-10 display no error bar. Considering that the spectral graph pruning includes a random dropout, the accuracy variance should also be reported.\n3. LNP is essentially composed of four individual and loosely related parts with different goals. None of those are indispensable for one another. The authors should exhibit an ablation study on the contribution of each component.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "lOJGhD1tcq", "reviewer": "Reviewer_mdv5", "summary": "The authors significantly improved the quality of the manuscript during the revision. However, the value of the pruning strategy compared to the SOTA SNN training and quantization approach is not sufficiently significant. Based on such effort and shortcomings, I would love to increase my rating to 5. \n--------------------------------------------------------\nThis paper proposes an innovative method called LNP, which prunes an untrained recurrent spike network via the utilization of graph theory and Lyapunov spectra. The approach proposed in this paper is better suited for discovering a remarkable sub-network in liquid-state machines or similar situations. In the SNN, energy consumption is the factor that measures the network efficiency, and not only network sparsity but also spike frequency will also affect it. Due to the variation in spike frequency before and after training, it is almost impossible to predict whether the pruned substructure remains efficient after the training process.", "questions": "See the weakness please.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "The LNP enables pruning the network prior to training, and effectively enhancing the sparsity while maintaining commendable performance.", "weaknesses": "1. The authors should elucidate the motivation behind their proposed method. What significance does their method hold for the field of spiking neural networks?\n2. The network model and task employed in the experiments are too simple and do not need pruning. I am more intrigued by the adaptability of the LNP method under more intricate situations.\n3. The authors provided a detailed description of the latest pruning works in SNN in the second paragraph of the Introduction. However, their results are not referenced in Table 1. Meanwhile, the compared works in Table 1 lack reference links.\n4. The equation 3 appears abrupt. I don’t understand why the spike frequency of untrained SNN can be used to construct this linearized model. And how to calculate the matrix D and L?\n5. Activity Pruning needs to provide citation or explanation.\n6. Lorenz and Rossler are unfamiliar datasets in the field of SNN. Could you please provide the citations?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)