P-SPIKESSM: HARNESSING PROBABILISTIC SPIKING STATE SPACE MODELS FOR LONG-RANGE DEPENDENCY TASKS
Authors: Malyaban Bal
Abhronil Sengupta
Venue: ICLR 2025 Poster (2025)
Keywords: Spiking Neural Networks
Sequence Learning
Primary area: other topics in machine learning (i.e., none of the above)
TL;DR: Computationally efficient solution for long-range dependency tasks using spiking neural networks.

Abstract: Spiking neural networks (SNNs) are posited as a computationally efficient and biologically plausible alternative to conventional neural architectures, with their core computational framework primarily using the leaky integrate-and-fire (LIF) neuron model. However, the limited hidden state representation of LIF neurons, characterized by a scalar membrane potential, and sequential spike generation process, poses challenges for effectively developing scalable spiking models to address long-range dependencies in sequence learning tasks. In this study, we  develop a scalable probabilistic spiking learning framework for long-range dependency tasks leveraging the fundamentals of state space models. Unlike LIF neurons that rely on the deterministic Heaviside function for a sequential process of spike generation, we introduce a SpikeSampler layer that samples spikes stochastically based on an SSM-based neuronal model while allowing parallel computations. To address non-differentiability of the spiking operation and enable effective training, we also propose a surrogate function tailored for the stochastic nature of the SpikeSampler layer. To enhance inter-neuron communication, we introduce the SpikeMixer block, which integrates spikes from neuron populations in each layer. This is followed by a ClampFuse layer, incorporating a residual connection to capture complex dependencies, enabling scalability of the model. Our models attain state-of-the-art performance among SNN models across diverse long-range dependency tasks, encompassing the Long Range Arena benchmark, permuted sequential MNIST, and the Speech Command dataset and demonstrate sparse spiking pattern highlighting its computational efficiency.

Published as a conference paper at ICLR 2025
P-SPIKESSM: HARNESSING PROBABILISTIC SPIKING STATE SPACE MODELS FOR LONG-RANGE DEPEN-
DENCY TASKS
Malyaban Bal & Abhronil Sengupta
School of Electrical Engineering and Computer Science
The Pennsylvania State University
University Park, PA 16802
(mjb7906, sengupta/@psu
ABSTRACT
Spiking neural networks (SNNs) are posited as a computationally efficient and biologically plausible alternative to conventional neural architectures, with their core computational framework primarily using the leaky integrate-and-fire (LIF) model. However, the limited hidden state representation of LIF neurons, characterized by a scalar membrane potential, and sequential spike generation process, poses challenges for effectively developing scalable spiking models address long-range dependencies in sequence learning tasks. In this study, we de- velop a scalable probabilistic spiking learning framework for long-range depen- dency tasks leveraging the fundamentals of state space models. Unlike LIF neu- rons that rely on the deterministic Heaviside function for a sequential process of spike generation, we introduce a SpikeSampler layer that samples spikes stochas- tically based on an SSM- based neuronal model while allowing parallel computa- tions. To address non-differentiabiiiiy of the spiking operation and enable effee- tive training, we also propose a surrogate function tailored for the stochastic nature of the SpikeSampler layer. To enhance inter-neuron communication, we introduce the SpikeMixer block, which integrates spikes from neuron populations in each layer. This is followed by ClampFuse layer, incorporating a residual connection to capture complex dependencies, enabling scalability of the model. Our models attain state-of- the-hhhtttt art performance among SNN models across diverse long-rand Long-range dependency tasks, encompassing the Long Range Arena benchmark, permuted se- quential MNIST, and the Speech Command dataset and demonstrate sparse spik- ing pattern highlighting its computational efficiency. Our implementation source code is available at
INTRODUCTION
Spiking neural networks (SNNs) (Ghosh-Dastidar & Adeli 2009) have garnered attention as a bio- plausible substitute for traditional artificial neural networks (ANNs). Their appeal stems from their utilization of spike-based communication among neurons, feature that closely mimics biological processes. The inherent stateful nature of SNNs enables them to adeptly handle temporal informa- tion (Neftci et al., 2211)), further enhancing their suitability for various applications (Yamazaki et al., 2022). The sparse spike-basee based information flow characteristic of SNNs allow event-driven compu- tation and communication in neuromorphic hardware, leading to energy savings (Sengupta et al.) 2019. SNN-based models, ideal for edge computing, have undergone rigorous testing on neuro- morphic hardware platforms such as Intel Loihi 2 (Davies et al.) 2021), IBM TrueNorth (Merolla ct al. 2014), among others, showcasing orders magnitude improvements in energy efficiency.
In the progression of SNN-based architecture advancements, research has predominantly focused on employing reaky-integrater -ire (LIF) neurons (Burkitt 2006). While the dynamics modeled by LIF neurons are deemed biologically plausible, the actual operations within the brain entail ad- ditional layers complexity (Hodgkin & Huxley] 1990) and stochasticity (Harrison et al.) (005) that are not fully captured by the simplified LIF neuron model. Moreover, the sequential state up-

---

Published as a conference paper at ICLR 2025
dates and spike generation using a deterministic Heaviside function complicate the training of LIF- based SNN architectures, often requiring computationally expensive methods like backpropagation through time (BPTT) (Neftci et al. 2019). This fundamental challenge has significantly limited the
adoption of SNN models, particularly for complicated sequence learning tasks involving long-range
dependencies. Efficient algorithmic alternative conventional BPTT training (Bauer et al.) [023]
Shrestha & Orchard 2018: Lin et al.) [024] Bal & Sengupta 2022) for LIF-based spiking archi- tectures are also being proposed, but remain unexplored for long context tasks. However, in this
paper, we move beyond traditional LIF-based spike generation models to develop a computationally
efficient probabilistic SNN architecture, designed to effectively tackle long-range dependency tasks
in the spiking domain.
State Space Models (SSMs) have been recently employed to effectively model sequence learning tasks (Gu et al., 2021a Gu & Dao 2023). SSMs serve as fundamental scientific models utilized across various disciplines, notably in control theory, to articulate the behavior of dynamic systems. They offer streamlined and robust framework, facilitating comprehensive analysis and deep un- derstanding of the dynamics of complex systems as they unfold over time. In this work, we employ SSMs capture temporal dependencies within sequences of input spikes, rather than conventional real-valued data. This approach not only enables computational efficiency but also underscores the remarkable capability of SSMs in analyzing long-term temporal dependencies in spike-based data.
In this paper, we propose an SNN architecture grounded in a probabilistic state-space neuronal model, which re-iimensional hidden state of the underlying SSM as the membrane potential, providing richer rep- resentations when compared to the scalar hidden state of an LIF neuron. Dynamics of each neuron is governed by an independent set of parameters, allowing the model to flexibly learn diverse temporal dependencies across neurons, thus enhancing its processing capacity. As outlined in the methodol- ogy, instead of veal-valued inputs, we feed sequence of spikes into the P-SpikeSSM neuronal model. This enables developing a computationally efficient framework by applying convolution over the sparse spikes, instead of real-valued data. The SpikeSampler layer samples spikes from each P- SpikeSSM neuron, enabling parallel operation with minimal overhead. Moreover, to address the challenge of non-differentiabiliy inherent in the stochastic spiking function, we introduce novel surrogate in the form of E [.]. where the discrete Bernoulli random variable Se is associated with the spiking event of each neuron at time t.
Probabilistic Spiking State-Space Model:
call P-SpikeSSM. We conceptualize the
Scalable Architecture with SpikeSampler and SpikeMixer: Although individual P-SpikeSSM neurons can process
input spike sequence, addressing tasks with complex long-range depen- dencies demands a deeper, more scalable architecture capable of capturing diverse dependencies. To address this, we introduce robust architecture (Fig. [T] and training framework. We encode the real- val-vvlued input sequence, associated with a sequence learning task, into N distinct spike trains, which are fed a layer consisting of N corresponding neurons (see Section 3.2). Each neuron gener- ates spikes stochastically based on its individual input spike train, while the collective activity of the neuron population allows for effectively capturing a diverse range of dependencies across the differ- ent input spike sequences. The output spikes from each neuron population in a layer are processed through a SpikeMixer layer, facilitating inter-neuron communication. Next, a FuseClamp layer performs further aggregation and computes the probability necessary for generating the subsequent spike sequences, which are then passed to the next layer of P-SpikeSSM based neuronal units. Fur- thermore, because the model communicates and uses sparse spike trains for computation, it achieves substantial computational efficiency by significantly reducing the number of floating-pol point multipli- cation and accumulation (MAC) operations across all layers and using simpler accumulative (ACC) operations instead. Furthermore, current state-of-the-art transformer-baser spiking architectures, like (Zhou et al. 2022 Bal & Sengupta 2024 Bal et al. 2024), process input sequences in parallel but require additional time steps to simulate network dynamics. In contrast, our proposed model eliminates this overhead, substantially improving computational efficiency.
Application to Range Dependency Tasks: We evaluate the performance of our proposed spiking architecture on various datasets within the Long Range Arena (LRA) benchmark, along with sequence-basee based datasets such as permuted sequential MNIST (psMNIST) and the raw inputs of the SC10 subset of the Speech Command dataset. Our model outperforms traditional non-spiking transformer-basee architectures and, to the best of our knowledge, establishes a new benchmark for spiking models in the domain of long-range arena tasks.
2

---

Published as a conference paper at ICLR 2025
Input
Encoder
Input Spikes
P SplkkeBMMM Encoder Layer 1
nearors
P-Splka
SSM
P P-SpikeSSM Encoder Layer
Shaps
SpikeSSmpleer Layer
(b)
SpikeMizing
E0.10
P P.SpkkeBMM Encoder Layer
ClampFuse Layer
0.00
Decoder
2.20
SpikeSampler Layer
0.00
Output
0
20
Time Steps
40
60
90
100
(a)
(c)
Figure 1: (a) High-level overview of the P-SpikeSSM-basee spiking architecture for LRA tasks. (b) Graph depieting the sparsity of spiking events generated by single P-SpikeSSM neuron over input sequence length (for ListOps dataset). (c) Graph showing the layer-wise active neuron ratio (i.e., the proportion of neurons generating spikes within layer per time step) against operating time steps, for randomly sampled input from ListOps dataset. The layer-wise spiking behavior illustrates the model-wide sparsity in spiking activity contributing computational efficiency.
RELATED WORKS
The realm of sequence modeling is primarily dominated by transformer-based architectures. Eff- cient implementations like LinFormer (Wang et al.) (2020). Performer (Choromanski et al.) 2020). among others, have demonstrated scalability to long sequence lengths. Meanwhile, non-spiking
chitectures based on SSMs, such as S4 and Mamba (Gu & Dao 2023 Gu et al. 202 2020a), have
also shown the capability handle lengthy sequences. Sequence learning in SNN-based architec- tures have primarily been applied to vision-basse based datasets (Zhou et al., 2022; Fang et al. 2024) and NI.P datasets (Bal Sengupta 2024 Zhu al. 2023), typically with constrained sequence lengths. However, within the domain of SNNs, frameworks based on Legendre Memory Units (LMUs) (Liu
ct al. 2024 Voelker et al. 2020 has ventured into exploring long-range dependency tasks.
Previous efforts integrating SSMs within spiking models (Stan & Rhodes 2023 Du al. 2024)
have primarily focused on passing the SSM output through layer LIF neurons to generate spikes. Using non-linear LIF neurons negates the parallel training efficiency of SSMs, as LIF neurons pro- cess information sequentially, introducing bottleneck (see Section 3.1.4). [Stan & Rhodes (2023) seeks enhance efficiency by linearizing LIF neurons. However, because the inputs to their SSM model remain real-valued, leading to additional floating-point MAC operations, this approach fails to leverage the energy-saving potential of SNNs. Moreover, the work lacks analysis of computa- tional efficiency and energy benefits, particularly concerning neuron firing activity, leaving a critical aspect of model efficiency unaddressed. Furthermore, from a sequence processing standpoint, the inherent ability and use of SSMs to capture temporal dependencies renders the addition of LIF neu- rons superfluous, introducing unnecessary computational overhead without providing any functional improvements beyond enabling spike generation.
METHODOLOGY
In this section, we first delve into the dynamics of the proposed Probabilistic Spiking State Space Models SSM).. We then delve into the specifics our proposed spiking architecture, high- lighting the SpikeSampler, SpikeMixer, and FuseClamp layers. Additionally, we offer insights on scaling the P-SpikeSSM- b-bssed spiking model for tackling complex long-range dependency tasks and develop computationally efficient parallel training framework.
3

---

Published as a conference paper at ICLR 2025
3.1 P-SPIKESSM FORMULATION
We formulate the neuronal model as a time invariant system which takes in sequence of input spikes given as (t) E {0, at time Much like the membrane potential upholds the state of the LIF neuron, we anchor our approach in SSMs (Gu & Dao 2023] Gu et al. [2021a), crafting an n-dimensional hidden state (h(t)) time Expanding the dimensionality of the hidden state en- ables our neuronal model to achieve more comprehensive state encoding of the underlying input sequence, surpassing the limitations imposed by the scalar hidden state LIF models. The event of spike generation at time t is associated with a Bernoulli random variable S, corresponding each neuron. The probability of spiking, i.e., (t()) at time t, is modeled as a function of the output of the neural model. The continuous time neuronal dynamics expressed as,
h(t) Ah(t) (t) Pt(t) = (CCh(t) + (t) (2) = clamp(az +
(I)
0 if <
where, h(t)
db and clamply) y if and b parameters. Setting de
y
and b 0 allows using the output of the SSM directly as the probability of spiking event without further scaling or translation. A is a matrix controlling the evolution of the hidden state over time without any input spikes.
represents the influence of the input spikes (t)) C describes the mapping of the hidden state vector h(tt) to the observed outputs, i.c., (t). D is the feedforward parameter, representing any direct influence of the inputs spikes (t) on the observed output proba- bility ps (t). For the purpose of simpler formulation, following previous works on SSMs (Gu & Dao 2023]. we will consider = O, since the term Dr, can be viewed as a simple skip-connection. Furthermore, is a function that clamps the output between [0, 1J. since probability [t E [0, 1]. We utilize [t[t] to sample spikes from the underlying neuron, as discussed in Section 3.1.3 The aforementioned formulation of our SSM-based neuronal model is presented in continuous-tine setting. However, since our primary focus is on sequence modeling tasks in domains such as NLP and vision tasks, now proceed formulate the dynamics of our neuronal model in discrete time.
3.1.1 P-SPIKESSM DISCRETE TIME DYNAMICS
In order discretize our system, we sample a sequence of spikes of length L, given by X, [I, r, [2], .... x. [l) from the original continuous signal given by (t),,, with step size A such as [1 = F. (iA). The P-SpikeSSM neuronal dynamics are subsequently discretized using bilinear
transformations Tustin 1947). whereby we approximate the parameters A, B, C as A, B, C which is given as,
A= = (1 (12/2
B = - (T-
A)-LAB
C=C
where, I is the Identity matrix. The transition dynamics of the discretized system at time step t is,
h[t] = Ah(t 1] + Br,[t] ps aCh((]
(3)
where, /[[]] is the hidden state of the neuron, [=[(] is the probability of the event S, This allows us to write the transition dynamics of the system recurrence in discrete time. The sparse spiking dy- namics of our proposed neuronal model, characterized by the spike probability ps [t]. is illustrated in Fig. [p. The spiking activity manifests in temporally localized patterns, mirroring the firing patterns observed in biological neurons (Hubel & Wiesel, 1962), with periods of non-activity interspersed between bursts. Now, instead of using recurrent representation, investigate how the evolution of
4

---

Published as a conference paper at ICLR 2025
state dynamics be represented by convolution operation (with spikes input signal), thus quiring only accumulation based computationally efficient operation. Moreover, using convolution instead of recurrence-base approach enables to parallelize the framework.
3.1.2 REPRESENTING DYNAMICS AS CONVOLUTION OVER SPIKES
There are two problems with Eqn. g concerning the training of scalable spiking architecture. Firstly, training it in its recurrent form necessitates employing a BPTT approach (Gu et al. 2021b), which is impractical for longer sequence lengths due its time and memory overhead. Secondly, as the hidden state h[[ at time can be a vector of floating points rather than spikes, the transition operations involved would not take complete advantage of energy/power efficient neuromorphic hardware. To achieve a fully parallelizable training procedure and leverage SNN-based operational efficiency during inference, we investigate an alternatve formulation of Eqn. 3 as convolution operation (Gu et al. [2021b). Since the proposed neuronal model is a time invariant system, con- sidering the initial hidden state i.e., [0] be a O-vector, the recurrent relationship can be unrolled as,
M/] = A Bx,
[1] + Br, (r,'2 .... ABx,(t - 1] Br.li] [i] =
>(A"
Bx,(i))
(4)
j=1
Thus, generalizing it to the entire sequence of length L we get,
H =
k= (B,AB. ..... A A'-1B)
(5)
where, represents the non-cireular convolution operation. H represents the sequence of hidden states (h/1], //2]...., hL) of length L. Æ is convolutional kernel of length L as defined above (see Appendix A for further explanation). The output the neuronal model, ie.. probability of spiking of the neuron at time is given as,
pald] =[(K X)):))
(6)
where, kernel K =K CCB,CAR CA 1--1 B): (K * X,): E KyHsp
the 7th term of the non-circular convolution, where K; P. (p= (P [L] the sequence probability of spikes from a neuron over the operating time steps. Thus, we can compute the output sequence parallely by doing convolution of the input sequence of spikes with the weights of kernel K. Additionally, each element of the sequence Ps can be computed product of a vector of real values (elements of precomputed K)) and vector of spikes (subsequence of X,). Sparse spikes further enables skipping unnecessary computations on zero elements within the input signal. Specialized neuromorphic hardware accelerators (Ivanov et al., 2022 Davies et al.) 2021) can leveraged to perform this process, thus avoiding floating point MAC operations during inference.
dot
3.1.3 SPIKESAMPLES LAYER
The spiking event of specific P-SpikeSSM neuron time & is modeled by Bernoulli random variable S. The spike generation process utilizes the output of the neuron, [s] (Eqn. 3 6). as the probability of spiking at time demonstrated below:
S - {0r
(1
if P.(0)
otherwise,
z~ 1(0,
This seemingly simple sampling process allows the proposed framework to operate in parallel with- out incurring significant computational overhead. This SpikeSampling process can be parallelized across both the sequence dimension and the population of neurons N. develop SpikeSampler
5

---

Published as a conference paper at ICLR 2025
layer (Fig. I.), facilitating the scaling of this methodology for more complex long-range depen- dency tasks. Additionally, the on-chip deployment of models that use random number sampling from uniform distribution, on neuromorphic hardware such as Loihi-2, has been previously plored (Pierro et al. 2024). This enables the implementation of our SpikeSampler layer directly on-chip.
Surrogate Gradient: The stochastic nature of the SpikeSampler layer introduces challenges during training, as P-SpikeSSM-based spiking architectures face the problem of non-differeniaabiity of spikes. To address this and enhance stability of the learning process, during the backward phase of backpropagation, we use a surrogate operation (S,) for the stochastic spiking operation time as,
S. -[s = P -) -
+1 P1 =) P.[0]
3.1.4 WHY CHOOSE STOCHASTIC SPIKE GENERATION OVER USING LIF NEURONS?
The continuous-time internal dynamics of a basic LIF neuron is outlined below,
Detemminisdi Process
Stochastio Procoss
o[1)
Puff]
S(11
Fenedio
dampling
du
Tm
dt = (u(() Urest) + R. I(t)
(9)
(E2)
sj
PaPPI
S2
902]
where, time t, u(t) E R is the membrane potential which can be considered as the hidden state of the sys- tem; [(t) is the input current scaled by constant R: fm is the time constant associated with the decay in mem- brane potential over time; Wrest is the resting membrane potential. LIF neurons thus sequentially updates its state (u) and uses deterministic Heaviside function for spike generation.
o[L]
SUU
Pull
[SL
60M
Output Spikea
Spikk
Ouiquet
Output
Nauron
Curput
Sumpler
Spiius
Figure 2: Computational flow of the LIF-based SSM model compared to the SpikeSampler-driven P-SpikeSSM neu- ronal model. Here, L is the sequence length, u[t] represents the membrane potential of LIF neuron and s[t] denotes the spike output (either or 0) at time Unlike the LIF-based approach, which is constrained by a sequential bottle- neck, our probabilistic approach sup- ports parallel processing.
Sequential Bottleneck Issues for LIF Neurons: The above sequential process of state update and spike gen- eration causes a bottleneck during the parallel training of the underlying SSM based framework. This results in increase in both training and inference time (see Sec- tion 4.1.2) compared our method. Prior work (Stan & Rhodes 2023) attempts to linearize LIF neurons for par- allel operation with SSMs, but offers limited analysis on the impact of this parallelization on model performance.
In contrast, as shown in our results, our approach not only surpasses the accuracy achieved by previ- ous methods across multiple datasets, but does so by being computationally simpler than the former. Additionally, the previous study has not provided evidence for any contributions of LIF neurons to model performance improvement, beyond their use in spike generation.
Parallel Execution in Our Model: The SpikeSampler layer compliments the parallel computa- tional advantages of the P-SpikeSSM neuron (Eqn. ( resulting in parallel and efficient spike generation process without the additional overhead of an LIF neuron (Fig. 2]. During the back- ward phase, the surrogate gradient (Eqn. 8 is utilized for effective and efficient model training. To efficiently compute the kernel K, we capitalize on prior theoretical structural findings regard- ing non-spiking SSMs (Gu al. 2021a). By exploiting the decomposition of matrix A into comprising a normal and low-rank matrix, we achieve efficient computation of K in O(L) time complexity, More details regarding efficient computation of K is added in Appendix B
3.2 SCALING P-SPIKESSMS TO DEEPER SNN ARCHITECTURES
To expand the sequence learning capabilities of the P-SPikeSSM neuronal model and facilitate its scalability to deeper architectures, we introduce the P-SpikeSSM neuronal layer. This layer com- prises of N P-SpikeSSM neurons. The sparse spiking activity of layer at time can be characterized by analyzing the active neuron ratio, denoted as anr; [t]. This ratio is determined by the number of

---

Published as a conference paper at ICLR 2025
spikes occurring that layer time t, and is defined as: anr. Σك sts(t where N; is the total Nr
number of neurons in layer and sty[[] represents the spike generated by neuron j in that layer, at time L. The layer-wise sparse spiking activity of the P-SpikeSSM neurons is illustrated in Fig. [ The high-level architecture of the proposed spiking architecture features K stacked P-SpikeSSM Encoder layers, each of which encapsulates the layers shown in Fig. D The effect of the number of neurons on model performance is shown in Fig. 3 This demonstrates that as the population of neurons within single layer increases, the spikes generated by them more effectively capture the temporal dependencies in the input sequence.
Since each P-SpikeSSM neuron indepen- dently processes input sequence tokens, a neuron mixer layer in the form of fully-connected block is introduced. This fa- cilitates the aggregation of spikes from previous layer of P- SpikeSSM neurons and flow of information among neuronal layers, ensuring efficient processing of diverse temporal de- pendencies learned by various neurons. The output of the SpikeMixer layer is given as.
MixkeMixer Block:
Accuracy vs. P-SpikeSSM Neuron Count
100
80
Accurity (96)
90
40
20
Smiz [4] = gelu(Ia[4) Wm)
(10)
2.26 20
Blog
where, Lo(t] E {0, 13 are the N spikes from the previous SpikeSampler layer time and Wm. E NNN is a linear weight. We use gelu function as a non-linearity. The lin- layer in this module avoids floating- Moating-point MAC operations since the input to the FC block consists of spikes. However, due the gelu() activation, there is element-wise floating- point multiplication of (n²) complexity (Hendrycks & Gim- pel 2016). which is still lower than the (nx)) MAC operation in foottnng-poi matrix multiplications.
Figure 3: Results obtained from the test set of the PS-MNIST dataset. This experiment utilizes two P-SpikeSSM neuronal layers, with each layer containing N neu- rons, represented on the x-axis. The accuracy achieved is displayed on the y-axis.
FuseClamp Block:
The FuseClamp block combines the input spikes (xx,) to the encoder layer with the SpikeMixer output via a residual connection shown in Fig. Th. This is followed by batch normalization (BN), after which the output is clamped between (0, 1] using σ. The FuseClamp
block is succeeded by SpikeSampler layer that utilizes its output as probabilities to generate spikes, which are then propagated to the subsequent layer. The operation defined as:
Pfc, [t] = or (BN(ffmz(1) + x,(()))
Input Encoding and Output Decoding: We typically employ a linear layer as the input encoder, with weight Wa RIxN followed by an optional batch normalization layer and SpikeSampler layer. This process facilitates the generation of input spike sequences for all N neurons in the sub- sequent layer. This choice is informed by the prevalent structure of our datasets, wherein input data sequences commonly formatted as [24x1 For the long-range dependency based classification tasks, the output of the final P-SpikeSSM encoder layer, passed through a pooling- boased sequence decoder to generate the model prediction.
4 EXPERIMENTATION
In this section, we showcase the efficacy of our proposed P-SpikeSSM neuronal model-based SNN architecture by eval- uating their performance across various long-range dependency based datasets. We also conduct a preliminary energy analysis to highlight the computational efficacy of our proposed framework. The experiments were RTX A5000 GPUs (8) each with 24GB memory.
Model
SNN
Acc.
S4 (Gu et al. (021a)
No
98.7
LSTM (Gu 2020b
No
95.1
HSLMU Voelker et 2020
Yes
96.8
LMU (Voclker et al.
(2019)
No
97.2
DSD-SNN (Han et al. 2023
Yes
97.3
( Transformer Vaswani al.
2017
No
97.9
Spiking LMUForeer (Lu el al.
2024
Yes"
97.9
P-SpikeSSM (Our Model)
Yes"
98.4
Nvidia
Table 1: Results comparing the accuracy of our model to other methods on test set of PsMNIST dataset.
7

---

Published as a conference paper at ICLR 2025
Model
SNN
ListOps
Text
Retrieval
Image
Pathfinder
S4 (Original) GGu et al. 202
No
58.35
76.02
87.09
87.26
86.05
S4 (Improved) [Gu et a.
202Ta
No
59.60
86.82
90.90
88.65
94.20
Transformer Vaswami ct
201
No
36.37
64,27
57.46
42.44
71.40
Sparse Transformen Iay et Linformer (Wang et al. 2020
2020
No
17.07
63.58
59.59
44.24
71.71
No
35.70
53,94
52.27
38.56
76.34
Linear Translormer (lay
2020
No
16.13
65,90
53,09
42.34
75.30
FLASH-quad (Hua
2022
No
42.20
64.10
83.00
48.30
63.28
Spiking LMUFormer (Liu
2024
Yes"
37.30
65.80
79.76
55.65
72.68
Transnormer T2 (Qin ct al. 2022)
No
41.60
72.20
83.82
49.60
76.80
BinaryS4D (Stan & Khodes
2023)
Partial T*
54.80
82.50
85,30
82.00
82.60
P-SpikeSSM (Our Model)
Yes"
58.20
81.20
88.53
82,40
84.80
Results comparing the accuracy of our model against some spiking and non-spiking architectures on test sets of LRA benchmark tasks (*Model uses gelu activation but no floating point matrix multiplications. **Model uses gelu act. as well as floating point matrix multiplications).
Table 2:
4.1 RESULTS
We evaluate model (Tables 1 2 & 3 multiple long-range dependency tasks across datasets
such as PsMNIST (Le et al. 2015). Speech Command (SC10) (Warden 2018) and the long-range
arena (LRA) benchmark (Tay al. 2020). The dataset details are in Appendix C
Permuted Sequential MNIST: A simple model comprising two P-SpikeSSM Encoder layers, each with P-SpikeSSM neuronal layer consisting of 400 neurons, achieves state-of-the-art results among spiking architectures. It performs comparably to the current best non-spiking model and outperforms
non-spiking transformer-base architectures, as detailed in Table T
Speech Command: Our SNN model, featuring four P-SpikeSSM Encoder layers, each with 256 neurons, sur- passes the performance of many con- temporary non-spiking architectures on Speech Command 10 subset (SC10) as demonstrated in Table 3 Furthermore, on the 35-set Speech Command dataset, it achieves 96.23% outperforming previ- ous spiking basclines (Liu et al. 2024 Bittar & Garner 2022)
Model
SNN
Acc.
$4 (Gu et al.
2021a
No
98.3
Transtormer Vaswani et al.
2017
No
NRDE (Gu
2021a
No
16.5
Performer Choromanski
2020
30.8
CKConv(Gu et al. 2021a)
No
71.7
P-SpikeSSM (Our Model)
Yes"
95.6
Table 3: Results comparing the accuracy obtained by our
model other non-spiking architectures on set of SC10 dataset.
Long Range Arena Benchmark:
Transformer-basee non-spiking models, demonstrated in Table struggle with suboptimal per- formance on long-contex: tasks in LRA benchmark. This primarily due to the overhead incurred during the computation of attention scores, which becomes more pronounced with longer sequence lengths. In our analysis, we also compare our method against the LMU-based spiking model, Spik- FUgLee, as well as the Bin model (Stan Rhodes 2023). Although BinaryS4D is not a fully spiking model (requires floating point MAC based matrix multiplications), it incorporates a layer of LIF neurons to generate spikes from an underlying state-space model (SSM).
4.1.1
Initializing the state matrix A with HiPPO matrices (Gu et al. 2020a) leads optimal performance and rapid con- vergence. Across majority of tasks, utilizing HiPPO-IegS (further discussed in Appendix B) consistently yields the highest accuracy. Hyper-parameters employed for training model can be found in Table 4 with additional details in Appendix D
Hyper- parper-prramttess
Range
Optimal
K: Encoder Layers
(2-6)
4
Neurons per Layer
(64-400)
256
n: Hidden State Dim.
(4-64)
16
Ir: Learning Rate
(le-4 - le-1)
0.005
Batch Size
(8-256)
32
Epochs
20-200
Table 4: Hyper-parameee of our SNN models. Optimal values for ListOps dataset also shown.
8

---

Published as a conference paper at ICLR 2025
4.1.2 ABLATION STUDIES
We perform experi- ments to analyze the effect various compo- nents introduced in this work, specifically the surrogate (used during training) for the Spike- Sampler layer, the SpikeMixer, and use of nor- malization in ClampFuse layer. On more chal- lenging datasets, such ListOps, the model fails to achieve better-than-random accuracy when trained without the surrogate. For the LRA Text task, although training without the surrogate is feasible, it results in significantly reduced performance, as shown in Table 5 We also ran experiments to understand the effects of SpikeMixer layer and emphasize the performance benefits of incorporating the SpikeMixer layer, which facilitates inter-neuron communication and enhances the ability of the model to capture complex temporal dependencies. Since normalization (used in the ClampFuse layer) is typically treated as non-local operation that poses challenges for implementation on neuromorphic hardware, we conducted an experiment to assess the impact of removing normalization, and achieve promising result with minimal accuracy drop. We conducted an experiment replacing the gelu layer in the SpikeMixer block with hardware friendly relu acti- vation function (Timcheck al. 2023 and observed minimal performance degradation.
Effeet of Components:
Our Model
Accuracy
w/o Surrogate (St)
70,90
w/o SpikeMixer
68.90
w/o Normalization
77.80
w/ Fixed Param.
80.20
w/ Learnable Param.
81.20
w/ relu Activation
80.40
Table 5: Results showing the effect of different components of our proposed SNN architecture on test accuracy when trained on LRA Text dataset.
Parameterized a: The function a is defined with parameters a and b (Eqn. T. These parame- ters can either be treated as typer-parameters or learned during training. We conducted additional experiments on the LRA text dataset to analyze the impact of learning σ. Our results (Table 55) indi- cate that allowing a to be learnable improves model performance, though it introduces an additional computational cost of ( element-wise multiplications per layer (with N neurons), compared to (i.e. using output of SSM directly for sampling probability).
using fixed values of a = and
Comparing our Stochastic Model to LIF-based Implementation: We implemented a version of our model based on LIF neurons, where the output of the underlying SSM (operating over spike sequences) is passed to an LIF neuron for spike generation. We use similar experimental setup as discussed in Section 4.1 for the PsMNIST dataset. This approach achieves an accuracy of 97.7% on the PS-MNIST dataset, compared to 98.4% by our stochastic approach. Furthermore, demonstrates significantly reduced training and inference times. Specifically, training for one epoch takes approximately 1.5 minutes, compared to 6.1 minutes for the LIF-based approach the ps- MNIST dataset. For inference, our model takes only 9 seconds on the test set, whereas the LIF-based approach requires 17 seconds. This disparity is largely due the sequential processing bottleneck inherent in the LIF-based approach, which also necessitates Backpropagation Through Time (BPTT) (Neftci al.) 2019) during training. In contrast, our parallel lll framework allows for training single pass using standard backpropagation.
model
4.2 ANALYSIS OF ENERGY EFFICIENCY
We perform a preliminary analysis comparing the energy efficiency of non-spiking S4 model to our spiking model during parallel inference 45nm CMOS technology (Han al. 2015). For 32-bit floating points, ACC operations (cost :9pJ) consume 5. x less energy than MAC operations (cost 4.6p.) (Han et al. 2015). Assuming input sequence length of L. with N neurons per layer across K layers, the dominant computational cost per layer for the non-spiking S4 model (Gu et al. 2021a) NT2 and L floating point MAC operations, representing the cost of computation (underlying SSM is operated parallely and convolutional kernel is precomputed and cached) in a single $4 layer and following linear layer with hidden dimension N. For simplicity, we estimate the energy costs using standard convolution instead of FFT, implementing FFT on neuromorphic chip-which primarily relies on spike- based accumulative operations- is significantly more complex. Although a complete energy calculation includes non-liness layers like gelu(), their contribution is less (O(LN) operations) compared to the energy cost of the parallel SSM and linear layers. Furthermore, we also explore model variant replacing gelu with relu (Table 5

---

Published as a conference paper at ICLR 2025
Building our previous analysis, the primary computational cost per S-SpikeSSM Encoder Layer in our spiking model is given by L. N and IFR. LN2 soating-poin accumulation (ACC) operations, contributed primarily by the parallel operation of all the neurons of the P-SpikeSSM layer and SpikeMixer layer respectively. Here, Rmn represents the firing rate of the input layer to the P-SpikeSSM neuronal layer, while TF R. denotes the firing rate of spikes sampled from the P-SpikeSSM neuronal layer, i.c. input to the SpikeMixer. .V here denotes the number of neurons in P-SpikeSSM encoder layer. As illustrated in Fig. the majority of neurons in the layer remain dormant during the input sequence, leading to sparse communication.
To illustrate with specific example, let us consider the ListOps dataset. An iso- parametric wate-of-the-an non-spiking S4
Neuran Count
200
Density (KDE)
model achieves an accuracy of 58.35 (im-
Neuron Count
proved version: 59.60) (Gu et al. 2021a).
150
while our P-SpikeSSM achieves 58. 20. For ListOps dataset, we use L = 2K and hidden dimension of S4, i.e., N 256. The energy consumption of the non- spiking model is EANN- 4x (2Kx2Kx 256 + 2K x 256 x 256)* (4.6pJ).
4
100
3
0.0
0.2
0.4
0.6
0.8
1.0
Our P-SpikeSSM based architecture uses 4 encoder layers, each with number of neurons N 256. The values for IF are [0.08, 0.19, 0.16..166.. 0.17], and the corresponding values for IFR, are [0.03, 0.12, 0.06, 0.07] (corresponding to the four layers). Now, the total compu- tational cost of our SNN considering N neurons in each encoder layer is ESNN
Mean Spiking Activity over Sequence Length
Figure 4: Results obtained after passing randomly sam- pled inputs from ListOps dataset through model. Figure consists of histogram representing the count of neurons associated with mean probability of spiking (averaged over sequence length L) and Kernel Density Estimation (KDE) plot of the data using an exponen- tial kemell. Thus, over the entire sequence, majority of neurons (R% 90%) have close to probability of spik- ing, signifying sparse spiking pattern.
(IF x 2K x 2K x 256 + IFR., x 2K 256 X 256) (0.9p.J).
Thus, for ListOps dataset, our SNN model is 36x more energy efficient (FANN/ ESNN) than the non-spiking model.
For this analysis, we concentrate exclusively on the cost of arithmetic operations, and did not con- sider memory I/O transaction costs. Although this methodology does not include architectural de- tails in the energy analysis, it still highlights the computational efficacy of our approach. By leverag- ing the prevalence of inactive neurons and sparse spiking patterns of active neurons, we can achieve significant improvements in energy and power efficiency on neuromorphic platforms.
5 CONCLUSIONS
We propose a computationally efficient probabilistic spiking framework for addressing long-term dependency sequence learning tasks. Instead of using LIF neurons, our model uses the output of P-SpikeSSM neuronal model as the probability for generating spikes using the proposed SpikeSam- pler layer. To tackle the non-differeniiability of this stochastic spiking mechanism, we introduce a surrogate gradient approach, enabling efficient training. To ensure scalability, our architecture features SpikeMixer and ClampFuse layers, enabling effective sequence processing through sim- plified accumulation-based operations. We evaluate our models on classification tasks involving long-range dependencies, such as the LRA benchmark, PS-MNIST, and the SC10 dataset. Our mod- els consistently outperform transformer-based non-spiking counterparts, achieving state-of-the-ant performance among SNN models, while also demonstrating exceptional computational efficiency due to the inherent sparsity of spiking events. To further harness these energy efficiency benefits, future work could explore deploying the model on edge devices and neuromorphic hardware, such as Intel Loihi 2.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
This material is based upon work supported in part by the U.S. National Science Foundation under award No. CCSS #2333881, CAREER #2337646, CCF #1955815, and EFRI BRAID #2318101.
REFERENCES
Malyaban Bal and Abhronil Sengupta. Sequence learning using equilibrium propagation. arXiv preprint arXiv:2209.0963 2022.
Malyaban Bal and Abhronil Sengupta. Spikingbert: Distilling bert to train spiking language models using implicit differentiation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 10998-110066 2024.
Malyaban Bal, Yi Jiang, and Abhronil Sengupta. Exploring extreme quantization in spiking lan- guage models. arXiv preprint arXiv:2405. 025443, 2024.
Felix C Bauer, Gregor Lenz, Saeid Haghighatshoar, and Sadique Sheik. Exodus: Stable and efficient training of spiking neural networks. Frontiers in Neuroscience, 17:1 110444, 2023.
Alexandre Bittar and Philip Garner. A surrogate gradient spiking baseline for speech command recognition. Frontiers in Neuroscience, 16:865897, 2022.
Anthony N Burkitt. A review of the integrate-and-fing neuron model: homogeneous synaptic input. Biological cybernetics, 95:1-19, 2006.
Krzysztof Choromanski, Valerii Likhosherstov, David Dohan, Xingyou Song, Andreea Gane, Tamas Sarlos, Peter Hawkins, Jared Davis, Afroz Mohiuddin, Lukasz Kaiser, et al. Rethinking attention with performers. arXiv preprint arXiv:2009.1477 2020.
Mike Davies, Andreas Wild, Garrick Orchard, Yulia Sandamirskaya, Gabriel A Fonseca Guerra, Prasad Joshi, Philipp Plank, and Sumedh R Risbud. Advancing neuromorphic computing with loihi: survey of results and outlook. Proceedings of the IEEE, 109(5):911-934, 2021.
Yu Du, Xu Liu, and Yansong Chua. Spiking structured state space model for monaural speech enhancement. In ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 766-770, IEEE, 2024.
Wei Fang, Zhaofei Yu, Zhaokun Zhou, Ding Chen, Yanqi Chen, Zhengyu Ma, Timothée Masquelier, and Yonghong Tian. Parallel spiking neurons with high efficiency and ability to learn long-term dependencies. Advances Neural Information Processing Systems, 36, 2024.
Samanwoy
and Hojjat Adeli. Spiking neural networks. International journal of neural systems, (904))2955338. 2009.
Albert Gu and Tri Dao. Mamba: Linear-tima sequence modeling with selective state spaces. arXiv preprint arX1v:23/2.00052,, 2023.
Albert Gu, Tri Dao, Stefano Ermon, Atri Rudra, and Christopher Ré. Hippo: Recurrent memory with optimal polynomial projections. Advances in neural information processing systems, 33: 1484-1487, 2020a.
Albert Gu, Caglar Gulcehre, Thomas Paine, Matt Hoffman, and Razvan Pascanu. Improving the gat- ing mechanism of recurrent neural networks. In International Conference on Machine Learning. pp. 3809. PMLR, 2020b.
Albert Gu, Karan Goel, and Christopher Ré. Efficiently modeling long sequences with structured state spaces. arXiv preprint arXiv:2111.00300 2021a.
Albert Gu, Isys Johnson, Karan Goel, Khaled Saab, Tri Dao, Atri Rudra, and Christopher Ré. Com- bining recurrent, convolutional, and continuous-tinn models with linear state space layers. Ad- vances in neural information processing systems, 34:572-585, 2021b.

---

Published as a conference paper at ICLR 2025
Bing Han, Feifei Zhao, Yi Zeng, Wenxuan Pan, and Guobin Shen. Enhancing efficient contin- ual learning with dynamic structure development of spiking neural networks. arXiv preprint arXiv:2308.04749, 2023.
Song Han, Jeff Pool, John Tran, and William J. Dally. Learning both weights and connections for efficient neural networks, 2015.
LM Harrison, David, and KJ Friston. Stochastic models of neuronal dynamics. Philosophical Transactions of the Royal Society B: Biological Sciences, (60((457):1075-119)) 2005.
Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (gelus).
arXiv preprint
arXiv:1606.0844 2016.
AL Hodgkin and AF Huxley. A quantitative description of membrane current and its application to conduction and excitation in nerve. Bulletin of mathematical biology, 52:25-71, 71, 1990.
Weizhe Hua, Zihang Dai, Hanxiao Liu, and Quoc Le. Transformer quality in linear time. In Inter- national conference on machine learning, pp. 9099-9117. PMLR, 2022.
David H Hubel and Torsten N Wiesel. Receptive fields, binocular interaction and functional archi- tecture the cat's visual cortex. The Journal of physiology. 160(1):106, 1962.
Dmitry Ivanov, Aleksandr Chezhegov, and Denis Larionov. Neuromorphic artificial intelligence
systems. Frontiers in Neuroscience, 16:959626, 2022.
Patrick Kidger, James Morrill, James Foster, and Terry Lyons. Neural controlled differential equa- tions for irregular time series. Advances in Neural Information Processing Systems, 33:6696- 6707, 2020.
Quoe V Le, Navdeep Jaitly, and Geoffrey E Hinton. A simple way to initialize recurrent networks of rectified linear units. arXiv preprint arXiv:1504.00944 2015.
Jiaqi Lin, Malyaban Bal, and Abhronil Sengupta. Scaling smns trained using equilibrium propagation to convolutional architectures. arXiv preprint arXiv:2405.02555 2024.
Zeyu Liu, Gourav Datta, Anni Li, and Peter Anthony Becrel. Lmuformer: Low complexity yet powerful spiking model with legendre memory units. arXiv preprint arXiv:2402.0443 040000000482..... 2024.
Andrew Maas, Raymond E Daly, Peter Pham, Dan Huang, Andrew Y Ng, and Christopher Potts. Learning word vectors for sentiment analysis. In Proceedings of the 49th annual meeting of the association for computational linguistics: Human language technologies, pp. 142-150, 2011.
Paul A Merolla, John V Arthur, Rodrigo Alvarez-lcaza, Andrew S Cassidy, Jun Sawada, Filipp Akopyan, Bryan L Jackson, Nabil Imam, Chen Guo, Yutaka Nakamura, et al. A million spiking- neuron integrated circuit with scalable communication network and interface. Science, 345 (6197):668-66 673, 2014.
Emre O Neftci, Hesham Mostafa, and Friedemann Zenke. Surrggate gradient learning in spiking neural networks:
Bringing the power of gradient- optimization to spiking neural networks. IEEE Signal Processing Magazine, 36(6):51-63 2019.
Alessandro Pierro, Philipp Stratmann, Gabriel Andres Fonseca Guerra, Sumedh Risbud, Timothy Shea, Ashish Rao Mangalore, and Andreas Wild. Solving qubo the loihi 2 neuromorphic processor. arXiv preprint arXiv:2408.03077 2024.
Zhen Qin, Xiaodong Han, Weixuan Sun, Dongxu Li, Lingpeng Kong. Nick Barnes, and Yiran Zhong. The devil in linear transformer. arXiv preprint arXiv:22 10340, 2022.
Abhronil Sengupta, Yuting Yc, Robert Wang, Chiao Liu, and Kaushik Roy. Going deeper in spiking neural networks: Vgg and residual architectures. Frontiers in neuroscience, 13:95, 2019.
Sumit B Shrestha and Garrick Orchard. Slayer: Spike layer error reassignment in time, Advances in neural information processing systems, 31, 2018.
12

---

Published as a conference paper at ICLR 2025
Matei loan Stan and Oliver Rhodes. Learning long sequences in spiking neural networks. arXiv preprint arXiv:2401.00955, 2023.
Yi Tay, Mostafa Dehghani, Samira Abnar, Yikang Shen, Dara Bahri, Philip Pham, Jinfeng Rao, Liu Yang. Sebastian Ruder, and Donald Metzler. Long range arena: A benchmark for efficient transformers. arXiv preprint Ttt::2011.04000 2020.
Jonathan Timcheck, Sumit Bam Shrestha, Daniel Ben Dayan Rubin, Adam Kupryjanow, Garrick Orchard, Lukasz Pindor, Timothy Shea, and Mike Davies. The intel neuromorphic dns challenge. Neuromorphic Computing and Engineering, 3(3):034005, 2023.
Arnold Tustin. A method of analysing the behaviour of linear systems in terms of time series. Jour- nal of the Institution of Electrical Engineers- IIA: Automatic Regulators and Servo Mecha- nisms, 94(1):130-144 1947.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need, 2017.
Aaron Voelker, Ivana Kajić, and Chris Eliasmith. Legendre memory units: Continuous-time repre- sentation recurrent neural networks. Advances in neural information processing systems, 32, 2019.
Aaron R Voelker, Daniel Rasmussen, and Chris Eliasmith. A spike in performance: Training hybrid- spiking neural networks with quantized activation functions. arXiv preprint arXiv:2002.03553, 2020.
Sinong Wang, Belinda Z Li, Madian Khabsa, Han Fang, and Hao Ma. Linformer: Self-attention with linear complexity. arXiv preprint arXiv:2006.04700 04768, 2020.
Pete Warden. Speech commands: A dataset for limited-vocabulary speech recognition. arXiv preprint 1xXvv:1804.03209 2018.
Kashu Yamazaki, Viet-Khoa Vo-Ho, Darshan Bulsara, and Ngan Le. Spiking neural networks and their applications: A review. Brain Sciences, 12(7):863, 2022.
Zhaokun Zhou, Yuesheng Zhu, Chao He, Yaowei Wang, Shuicheng Yan, Yonghong Tian, and Li Yuan. Spikformer: When spiking neural network meets transformer. arXiv preprint WwXiv:2209.15423 2022.
Rui-Jie Zhu, Qihang Zhao, and Jason Eshraghian. Spikegpt: Generative pre-trained language model with spiking neural networks. arXiv preprint arXiv:2302.139.00 2023.
13

---

Published as a conference paper at ICLR 2025
A DERIVING CONVOLUTIONAL REPRESENTATION OF P-SPIKESSM NEURONAL DYNAMICS
At time step t, the discretized neuronal model exhibits transition dynamics described as follows:
h = Ah(t 1] + Bx, Pa [t] = o
(SI)
where, kst]] is the hidden state of the neuron, p.[t] [ is the probability of the spiking event S,. A, B, C are the discretized parameters of the time-invariant system. Considering time step 0, M(0 0, we get,
h[1] Bx, [1]
h|2] = [ + Bx
(S2)
Unrolling like this to time step we get,
A|j] A
Bx.[1]] A Bx, +... ABx,Ii 1] Bx.ld
Σ (A' "Br,
(S3)
Thus, the convolutional kernel K, whose length is given by the length of the input sequence L, is defined as,
k= AB
(S4)
Now H. ie., sequence of hidden states can be computed as a non-circular convolution given as, H K X,, where X, is the input sequence of spikes.
Thus, [d[ (((K xx)))) oE K5 j-ji-j+tt -+ 1),, where K -CB,CCB,....CAAA CA)-BB).
B HIPPO-LEGS MATRIX
HiPPO (high-order polynomial projection operators) (Gu et al. 2020a) is versatile framework that enables the analysis of various families of measures. Utilizing this operator as either a closed-form ordinary differential equation (ODE) or linear recurrence, polynomial approximation as the input function unfolds over time. HiPPO-legS can generalize to different time scales. HiPPO enables the hidden state effectively memorize the historical pattern of input spikes (in our paper). The elements of the HiPPO-legS (Scaled Legendre) matrix E Raxx given below,
efficiently update the optimal
V2m 1v2k 2k +1, m m+1, 00
if m k if m
Amk
(S5)
if m
B. COMPUTING KERNEL K
The efficient computation of K has been proposed in literature (Gu et al. 2021a). thus speeding up the parallel training of SSM based neuronal architectures. We briefly go over the overview on how it achieved. The primary concern in computing K the repeated multiplication of the state matrix to create the individual terms K; Thus to compute K, the time complexity for simple approach of chained multiplication is O L). where n is the hidden state dimension and is the sequence length. Now the idea is that. if we had the state matrix to be diagonal matrix. then theoretically we could compute K efficiently using Vandermonde product. Thus, the goal is to diagonalize the matrix A. Now, the ideal scenario is if A is normal matrix, i.e., it is diagonalizable with unitary
14

---

Published as a conference paper at ICLR 2025
Table S1: Hyper-parameters used for obtaining the best result on individual datasets used for evalu-
ating P-SpikeSSM-based SNN models.
PsMNIST
SC10
ListOps
Text
Retrieval
Image
Pathfinder
K: #Encoder Layers
2
4
4
4
6
.: Neurons per Layer
400
256
256
256
200
512
256
Hidden State Dim.
64
32
16
16
32
32
64
Ir: Learning Rate
0.01
0.002
0.005
0.0005
0.003
0.005
0.0005
Batch Size
64
32
32
Epochs
200
200
200
200
200
200
200
matrix (UAU-) is a diagonal matrix, where U square matrix such that U U- A initialized to HiPPO matrices which are not normal matrices. However, HiPPO can be decomposed into a diagonal matrix and low-rank matrix. Following this, we can leverage previous theoretical results (Gu et al. (221a) on reducing the underlying SSM to the computation of Cauchy kernels and calculate K, linear order of time complexity w.r.t the sequence length
C DATASET DETAILS
To heighten the complexity of the classification task, permuted se- quential MNIST (psMNIST) (Le et al. 2015) reconfigures the presentation of images compared to the original MNIST dataset. While MNIST has each 28 x 28 grayscale image as a unified entity, PsMNIST arranges the pixels in sequence and in permuted order. Thus, tackling this task de- mands more sophisticated models capable of effectively retaining and synthesizing information over time.
Permuted Sequential MNIST:
Speech Command Dataset (SC10): We use the 10-class subset of Speech Command dataset (War-
den 2018) following previous works (Kidger et al., 2020: Gu et al. 2021a) and evaluate model
on the raw unprocessed signals of length 16000.
To demonstrate the long-range dependency analysis capability of our spiking architecture, we leverage the Long Range Arena (LRA) benchmark (Tay et al.) 2020). spanning various classification tasks from textual image domains. Following are the five tasks utilized for evaluation.
Long range Arena Benchmark:
ListOps: In this task. our focus lies in modeling hierarchically structured data within long-context framework. The sequence length for this task is upto 2K.
Text: In this task, we process the IMDB dataset (Maas al. 201T) of movie reviews and perform the task of sentiment analysis in a byte-level. This is done to ensure a long sequence length of 4K.
Retrieval: In this task, assess the model's capacity encode and retain compressed rep- resentations essential for matching and retrieval purposes. The input consists byte-level sequences (of length 4K) from two documents, and the goal analyze their similarity, Image: In this task, perform image classification task based on sequence of pixels the original image. The dataset is CIFAR- 10 and the sequence length is 1K.
Pathfinder: In this task, we treat 32 32 image as sequence of pixels of length 1 1K. Our objective is to make a binary decision regarding whether two points, depicted as circles, are linked by path composed of dashes.
D ADDITIONAL EXPERIMENTAL RESULTS
we list the optimal set of hyper-paramette used for each the tasks. We have used cosine annealing learning rate scheduler with warmup phase. As mentioned in the main text, the state matrix A is initialized to the HiPPO-legS matrix as shown in Eqn. SS The step size for discretization (A) is restricted between [0.001, 0.1]. The memory footprint ranges from s 6GB for PsMNIST to 23GB for SC10. The dataset splits are aligned with prior literature (Tay et al. 2020
Table ST
15

---

Published as a conference paper at ICLR 2025
Additional Experiments: We also tested our model on the sequential CIFAR- 10 dataset, achieving an accuracy 85.6%. The pappmerrrameterss used are similar to the one used for LRA Image dataset, shown in Table ST If pass real valued probabilities instead of spikes, achieve improved accuracy (87.2%). However, this approach sacrifices the energy efficiency advantages provided by the spiking sparsity in original model. Our model processes the input
sequence of pixels, yet it achieves performance comparable to other state-of-the-ar models, such as PSN Fang et al. (2024) (PSN: 88.45% and Sliding PSN: 86.70%) which processes the image as a sequence of columns.
Memory Footprint: Based on previous analysis (Tay et al. 2020) on LRA tasks, the Transformer models in Table are configured with layers, hidden dimension of 256 and 4 attention heads, thrt-art per- formance in the spiking domain and comprehensively outperforms non-spiking transformer based architectures as shown in 2 Furthermore, considering identical parameters (A, B) for neurons in the same layer, the average parameter count of our models on LRA ListOps dataset around 280K, representing a reduction of & 2.1x compared to the parameter count of the transformers used for comparison.
resulting in approximately 600K parameters in total. Our models establish
16


Reviews:
[{"review_id": "IsxaZVqf5E", "reviewer": "Reviewer_wNo7", "summary": "The article proposes a framework for learning in long-range sequence tasks. It introduces a series of tactics to address the associated challenges. Firstly, the authors employ a stochastic sampler based on a State-Space Model (SSM) neuronal model in conjunction with a differentiable function. Additionally, the paper provides comparison results to demonstrate the effectiveness of the proposed approach.", "questions": "n/a", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "The paper explores the differences between two neuron models: the conventional Leaky Integrate-and-Fire (LIF) model and a modified version that employs sampling-based techniques. Additionally, the authors present comprehensive comparison results to highlight the effectiveness of their method.", "weaknesses": "The authors do not provide strong evidence that the sampling-based technique outperforms the conventional LIF neuronal model. The absence of an input/output step may also hinder the flow of internal dynamics information. I suggest that they compare their architecture using both approaches to clearly demonstrate the benefits of their method. While it is evident that their approach facilitates the parallelization of the algorithm, this does not necessarily imply that the solution can be obtained faster, as more steps might be required to achieve the appropriate spiking rate.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ZFwC4MIcCW", "reviewer": "Reviewer_Fa79", "summary": "This work introduces P-SpikeSSM, a stochastic spiking neural network (SNN) that replaces the conventional scalar LIF neuron membrane potential with an SSM linear dynamical system. The multi-dimensional of the SSM increases the expressivity of each neuron. The SSM inside each neuron is read out through a linear map, the output of which is then clamped between 0 and 1 to be interpreted as a probability. This probability is used to generate spikes stochastically during each timestep according to a Bernoulli distribution. \n\nThe inspiration for using a multidimensional SSM instead of a scalar membrane potential in the spiking neurons is the increased temporal-dynamic expressivity afforded by SSMs. The neural SSMs are initialized using HiPPO matrices, inspired by prior SSM work. \n\nThe reasoning for introducing a probabilistic spiking model is that it affords convenient parallelizability. The authors use the probability as a surrogate operation to propagate gradients through the probabilistic spiking layers.\n\nThe authors show through experiments that the P-SpikeSSM performs better or competitively with transformer and other SSM baselines, and the authors perform ablation studies, showing all components of their model discussed in the paper contribute to task performance.", "questions": "I asked the most salient questions above in the “Weaknesses” section. The questions that follow are more minor.\n\nOn line 57, the authors state that long-range dependency tasks are largely unexplored in the spiking domain, but then the authors go on to cite works on SNNs with SSMs in paragraph starting on line 139. What do the authors mean by ‘largely unexplored’? I suppose ‘largely unexplored’ is a subjective assessment, however. I might suggest more consistent language.\n\nI would suggest clarifying that the “Analysis of Energy Efficiency” section is treating parallel inference specifically.\n\nIs there anything that can be said about the hyperparameter selection process used in this work, to help justify that the ablation study experiment results are not an artifact of certain hyperparameter choices?\n\nWhile probabilistic spiking affords convenient parallelization, is there anything that can be said about the drawbacks of or parallelizable alternatives to probabilistic spiking?\n\nThank you for this fascinating work.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Significance.\nReplacing the membrane potential of a spiking neuron with an SSM is a well-motivated modification, and the encouraging results presented in this work suggest this modification could be broadly valuable in efficient neural network research.\n\nOriginality.\nThis is the first work that I’ve seen to replace the membrane potential in and SNN with an SSM. \n\nQuality.\nThe work includes ablation studies to demonstrate the necessity of each component of the model.\n\nClarity.\nThe work builds up the model in a step-by-step manner with sufficient details so that the reader can readily understand the motivation for each step.", "weaknesses": "On line 141 and the caption for Figure 2, has it truly been proven to be the case that LIF neurons cannot be parallelized? What is the distinction between a non-linear LIF neuron and a linear LIF neuron? (To me, all LIF neurons are non-linear because they have reset terms.) Do efficient algorithms exist for LIF neurons? E.g., see “EXODUS: Stable and Efficient Training of Spiking Neural Networks” by Bauer et al., or the work SLAYER that they reference?\n\nOn line 77, the authors claim the surrogate operation Expection[S_t] is novel. I believe such a surrogate operation has already been introduced. E.g., see “Automatic Differentiation of Programs with Discrete Randomness” by Arya et al. NeurIPS 2022.\n\nIn Table 1, it is unclear to me the model size/computation budget of these various models. I could imagine model size/compute budget is a key reason why one model would outperform another, so I would like to clearly understand how this table presents a fair comparison. E.g., could you show me how the P-SpikeSSM model is iso-parameter-count?\n\nOn line 102, the authors imply that P-SpikeSSM is a “fully” spiking model. While “fully” spiking is not defined in the literature, one might argue that P-SpikeSSM is not fully spiking because each spiking neuron contains a non-spiking SSM within it. I might suggest dropping the “fully” adjective. On line 415, “fully spiking” is implied to be defined as “not requiring floating point MAC operations,” but I am now confused regarding how P-SpikeSSM is fully spiking even though there is an SSM inside every neuron. \n\nI struggle to understand the “Analysis of Energy Efficiency” section. In particular I do not see how the computational cost of the n SSM state variables inside the N P-SpikeSSM neurons in each layer are accounted for. To me, it seems a factor of n is missing.\n\nI noted in line 810 “Memory Footprint” that the SSM parameters in each neuron in a layer can be shared. Are the parameters of the SSMs shared in your experiments?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "1Rf9Fcoq3c", "reviewer": "Reviewer_2Qpj", "summary": "To exploit the energy-saving potential of spiking neural networks (SNNs) and to overcome the bottleneck of SNNs in long-range dependency tasks, this work proposes a scalable probabilistic spiking learning framework based on state space models (SSMs). Unlike the other model that simply concatenates SSMs with the leaky integrate-and-fire (LIF), this work incorporates SSMs into the membrane potential update of spiking neurons, and innovatively proposes P-SpikeSSM in a bid to address the performance issues of LIF and overcome its inability of parallel computations. To build a scalable architecture, this work designs several modules, including SpikeSampler, SpikeMixer, and FuseClamp. With the contribution of these modules, P-SpikeSSM outperforms other SNN models in various long-range dependency tasks. In addition, P-SpikeSSM shows a large improvement compared to its ANN counterpart in energy efficiency analysis.", "questions": "1. The benefits of stochasticity? Would there be better or more consistent performance if probabilistic spike sampling is replaced by spike firing at a fixed or learnable threshold?\n2. How can the network be trained with the removal of the surrogate gradient (Table 5)?", "limitations": "", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The motivation for this work is clear and sound, and the presentation of the methodology is very detailed and easy to follow.\n2. P-SpikeSSM integrates the recently high-profile SSM into spiking neural networks from a novel perspective, addressing the problem that the commonly used LIF cannot handle parallel computations. It may provide new insights for future research on the combination of SSM and SNN.\n3. By incorporating P-SpikeSSM with some well-designed modules, the proposed model achieves better performance than other spiking networks and some non-spiking networks.", "weaknesses": "Major points:\n1. P-SpikeSSM is an interesting exploration, but the elements that play a key role in it seem to be just an application of what SSM proposes, such as temporal convolution to enable parallel computation.\n2. The network cannot be considered a fully spiking neural network, and is best called a partial SNN because of the use of gelu in SpikeMixer Block (Eq. 10). In addition, the introduction of gelu greatly increases the difficulty of deploying this network on neuromorphic hardware.\n3. Considering that the network is a partial SNN, it performs only marginally better than BinaryS4D, which is also a partial SNN, on some of the LRA benchmark tasks, and even worse on others (Table 2). Given that P-SpikeSSM and BinaryS4D are both based on SSM, it's debatable whether their advantages over other transformer-based networks mostly stem from SSM.\n4. For the experiment on the Speech Command dataset (Table 3), Spiking LMUFormer achieved 96.12% accuracy on the full 35-category dataset, whereas this work is only validated on a subset of 10 categories and is not compared to some recent advanced baselines [1, 2].\n\nMinor point:\n1. There is a comparison of energy consumption with the ANN counterpart, but no comparison of the number of parameters with other baselines, which is an important metric influencing performance.\n\n[1] Zeyu Liu, Gourav Datta, Anni Li, and Peter Anthony Beerel. Lmuformer: Low complexity yet powerful spiking model with legendre memory units. ICLR. 2024.\n\n[2] Alexandre Bittar and Philip N Garner. A surrogate gradient spiking baseline for speech command recognition. Frontiers in Neuroscience. 2022.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "hQyesRRBys", "reviewer": "Reviewer_EjY8", "summary": "The authors present a new model that combines state space model (SSM) layers with stochastic (Bernouilli) spiking layers. They test it on various long sequence benchmarks.", "questions": "* I think the authors could bypass the SpikeSampler layer and send the real-valued probabilities directly to the next layer. Do you confirm? Have you tried? Of course, the computational advantages of spikes would be lost, but I expect an increase in accuracy, and it would be interesting to quantify it. \n\n* Would it be possible to encourage even sparser activity via some additional term in the loss function?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Very good accuracy, outperforming all other spiking proposals and even most non-spiking ones (all except S4).", "weaknesses": "* IMHO, the main weakness is that this network is not fully spiking, and it's not clear if it could be implemented on existing neuromorphic chips (e.g., Intel Loihi). The temporal convolution of the SSM (eq  6) could probably be implemented by delays. But the gelu in eq 10, and the non-local normalization (in ClampFused layers) are problematic. I suggest the authors discuss these issues and possible solutions. Also, I think the authors should cast their network as \"Partial SNN\" in Table 1-3\n\n* IMHO psMNIST is not challenging enough (SOTA is nearly 100%, which may mask differences between approaches); I would recommend trying as well on sequential CIFAR10/100 (much more challenging).\n\n* The authors seem to ignore the sliding PSN neuron (http://arxiv.org/abs/2304.12760), which has many similarities with their model. Both use spikes. Both use temporal convolutions instead of stateful units and for this reason, both avoid BPTT and are parallelizable. A comparison with the PSN (both qualitative and quantitative) would be useful.\n\nMinor points:\n\n* \"A is a parameter controlling the evolution\" -> \"A is a matrix controlling the evolution\"\n\n* Fig 1 b: the y dimension is useless here. I would recommend plotting the curve y = p(t) instead of a heat map.\n\n* L184: \"since probability p[t] in [0, 1].\" I think in continuous time, p is a pdf, so it could be >1\n\n--\n\nPOST REBUTTAL:\n\nMy main concerns have been addressed. I raised my score to 8.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)