Parallel Spiking Neurons with High Efficiency and Ability to Learn Long-term Dependencies
Authors: Wei Fang
Zhaofei Yu
Zhaokun Zhou
Ding Chen
Yanqi Chen
Zhengyu Ma
Timothée Masquelier
Yonghong Tian
Venue: NeurIPS 2023 poster (2023)
Keywords: Spiking Neural Network
SNN
deep learning
spiking neuron
neuromorphic computing

Abstract: Vanilla spiking neurons in Spiking Neural Networks (SNNs) use charge-fire-reset neuronal dynamics, which can only be simulated serially and can hardly learn long-time dependencies. We find that when removing reset, the neuronal dynamics can be reformulated in a non-iterative form and parallelized. By rewriting neuronal dynamics without reset to a general formulation, we propose the Parallel Spiking Neuron (PSN), which generates hidden states that are independent of their predecessors, resulting in parallelizable neuronal dynamics and extremely high simulation speed. The weights of inputs in the PSN are fully connected, which maximizes the utilization of temporal information. To avoid the use of future inputs for step-by-step inference, the weights of the PSN can be masked, resulting in the masked PSN. By sharing weights across time-steps based on the masked PSN, the sliding PSN is proposed to handle sequences of varying lengths. We evaluate the PSN family on simulation speed and temporal/static data classification, and the results show the overwhelming advantage of the PSN family in efficiency and accuracy. To the best of our knowledge, this is the first study about parallelizing spiking neurons and can be a cornerstone for the spiking deep learning research. Our codes are available at https://github.com/fangwei123456/Parallel-Spiking-Neuron.

Parallel Spiking Neurons with High Efficiency and Ability to Learn Long-term Dependencies
Wei Fangl.?, 12, Zhaofei Yu', Zhaokun Zhou Ding Chen 5.2, Yangi Chen 1.2, Zhengyu Ma Timothée Masquelier", Yonghong Tian 12233
School of Computer Science, Peking University, China
Peng Cheng Laboratory, China
SScooo of Electronic and Computer Engineering. Shenzhen Graduate School, Peking University, China
SScoool of Artificial Intelligence, Peking University, China
Department of Computer Science and Engineering. Shanghai Jiao Tong University, China
"Centre de Recherche Cerveau et Cognition (CERCO), UMR5549 CNRS Univ. Toulouse 3, France
Abstract
Vanilla spiking neurons in Spiking Neural Networks (SNNs) use charge-fire-res fire-reset neuronal dynamics, which can only be simulated serially and can hardly learn long-time dependencies. We find that when removing reset, the neuronal dynamics can be reformulated in a non-iterative form and parallelized. By rewriting neuronal dynamics without reset to general formulation, Neuron (PSN), which generates hidden states that are independent of their predeces- sors, resulting in parallelizable neuronal dynamics and extremely high simulation speed. The weights of inputs the PSN are fully connected, which maximizes the utilization of temporal information. To avoid the use of future inputs for step-by- step inference, the weights of the PSN can masked, resulting in the masked PSN. By sharing weights across time-steps based the masked PSN, the sliding PSN is proposed to handle sequences of varying lengths. We evaluate the PSN family on simulation speed and temporal/static data classification, and the results show the overwhelming advantage of the PSN family efficiency and accuracy. To the best of our knowledge, this is the first study about parallelizing spiking neurons and can be cornerstone for the spiking deep learning research. Our codes are available at http:////hhhhoeooo//////eeee//// Spw.araille--isii-----eeeee-ee Neuron
propose the Parallel Spiking
Introduction
Spiking Neural Networks (SNNs) are the next generation 1 of Artificial Neural Networks (ANNs) using lower abstraction of the biological neural system. The spiking neurons the key components of SNNs, which process input currents with complex neuronal dynamics and fire spikes as outputs when their membrane potential reaches the threshold. The SNNs use discrete spikes communicate between layers, which enable the event-driven computational paradigm and have extremely high power efficiency neuromorphic chips such as True North (2, Loihi [3] and Tianjic 4 Due their high biological plausibility, SNNs have been regarded by neuroscientists as useful tools for analyzing, simulating, and learning the biological system [5] 6). With the introduction of deep learning methods
7 9 10 IT] [2). the performance of SNNs on real-world tasks has been greatly improved and the applications of SNNs are expanded [13 14 As the bridge between neuroscience and computational
science, SNNs have attracted more and more research interest in recent years.
"Corresponding author
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

charge
fire
reset
Vit- T
Hir
x[{]
MIF+
(a) The vhriillllapiinn neuron
(b) The parallel spiking neuron
The computational graphs of the vanilla spiking neuron and the parallel spiking neuron. Figure (a) is cited from [T5]. The dotted lines in Figure (b) are weights that can be masked for step-by-step computation. X(t). S(t] are the input current and the output spike, and H V are hidden states at time-step t.
Figure 1:
The binary characteristic of spikes causes lot of information losses, which is the main factor leading the lower accuracy of SNNs than ANNs. Among efforts bridge the performance gap. the improvement of spiking neurons is universal approach and has been widely explored 20 21 However, previous works are limited the existing serial charge-fire fire- reset computing paradigm shown in Fig aa), which suffers from slow simulation speed and can hardly learn long-term dependencies.
15 16 17
18
19
In this paper, we propose the Parallel Spiking Neuron (PSN) and its variants, the masked PSN as well as the sliding PSN (SPSN). Fig [[b) shows the computational graph of the PSN, which uses non con-predecessor dependency method to generate hidden states, leading to parallelizable neuronal dynamics and extremely high simulation speed. Since the connections from inputs H [t] are direct, the PSN has a better learning ability for long-term dependencies than other vanilla spiking neurons. To generate [() immediately when X arrives, connections from [1) i t 1 can be masked, as the dotted line shown in Fig (b), and then the neuron becomes the masked PSN. The parameters of the masked PSN can also be set as iimeiinvariant, which derives the sliding PSN and is more flexible for input sequences with variable lengths. The main contributions of this paper are as follows:
1) We analyze the influence of removing reset from the widely used Charge-fire-rest neuronal dynamics, showing that the spiking neuron can be parallelized without reset.
2) By rewriting the neuronal dynamics without reset to general formula, obtain the PSN with fully parallclizable neuronal dynamics. For step-by-step serial forward and variable length sequences processing, the masked PSN and the sliding PSN are also derived.
3) We compare the simulation speed of the PSN family and vanilla spiking neurons, showing an overwhelming performance advantage of the parallel neuronal dynamics. We evaluate the PSN family on sequential, static, and neuromorphic data classification tasks and achieve higher accuracy than previous spiking neurons.
2 Related Work
2.1 Deep Learning for Spiking Neural Networks
The ANN-to-SNN conversion (ANN2SNN) 22 23 24 25 26) and the surrogate training 27 are the two main deep learning methods for SNNs. The ANN2SNN method uses firing rates in SNNs to approximate the activations in ANNs by converting weights from ANNs SNNs with extra normalizations to reduce errors. It achieves close performance to that of ANNs in the ImageNet dataset (29), but requires many time- steps estimate accurate firing rates. The surrogate training method re-defines the gradient of the Heaviside function used in the spike generating process by smooth and differentiable surrogate function. With surrogate gradients, the SNNs can be trained by backpropagation through time (BPTT) and gradient descent. The surrogate training method is not based on rate coding as ANN2SNN, resulting fewer time-steps. However, the use BPTT during training requires more training time and memory usage than ANN2SNN.
28

---

2.2 Improvement of Spiking Neurons
Improving the spiking neurons inspired by biological mechanisms techniques from ANNs is practical method. The Parametric Leaky antegrated-and-Fei (PLIF) spiking neuron [T3] incorporates learnable membrane time constants and is able to combine the learning of both synapse weights and neuronal dynamics. The gated LIF (GLIF) spiking neuron [16] adds learnable gates to fuse different integration, decay, and reset methods, which enlarges the representation space and increases the heterogeneity as well as adaptivity of spiking neurons. The k-based Leaky Integrate-ande (KLIF) neuron [88] adds a learnable scaling factor adjust the surrogate gradient and applies ReLU activation to avoid the negative membrane potential. The Multi-Level Firing (MLF) unit [19] contains LIF neurons with thresholds in different levels, which relieves the gradient vanishing problem and improves expression ability. Ponghiran et al. [20] improve the spiking neurons by adding input gates, forget gates, recurrent connections, and multi-bit outputs, which reduce the mismatch between actual and surrogate gradients.
2.3 Improvement of Training Methods and Network Structure
Beyond the spiking neurons, modifying training techniques and network structure are also effective methods to promote the performance of SNNs.
Normalization The threshold-dependen batch normalization (TDBN) 30 normalizes features in both temporal and spatial dimensions and eliminates the variance introduced by thresholds, which is a more suitable normalization method than the batch normalization [3] for SNNs. Batch Normalization Through Time (BNTT) [32] decouples BN along time-steps and uses time-wise parameters, which estimates the distribution of temporal-vare variint inputs more precisely. Temporal Effective Batch Normalization (TEBN) (33 adds learnable affine transformation on each time-step and has better ability to capture temporal distributions.
Training Techniques Deep Continuous Local Learning (DECOLLE) uses local losses to avoid BPTT and implements online learning for SNNs. Online training through time (OTTT) [35] approximates iterative gradients with temporal dependencies by eligibility traces updating with time-steps, which also achieves online training and only requires constant training memory agnostic to time-steps. Sparse spiking gradient descent (36] only uses neurons whose membrane potentials are inside a given range for backpropagation, which achieves speedup and memory reduction in the backward. The attention mechanism on membrane potentials [37] eliminates redundant information, decreases firing rates, and reduces overfitting. Dspike (38) uses the finite difference to estimate gradients and achieves higher accuracy than plain surrogate gradient methods. The temporal efficient training (TET) (39 calculates loss at each time-step and averages them rather than using averaged outputs to calculate loss, which improves the temporal scalability of SNNs.
Network Structure Spike-Element
ResNet 44 solves the gradient vanishing/exploding
problems of the plain Spiking ResNet caused by sigmoid-like surrogate functions, which successfully trained the first deep SNN with more than 150 layers. Spikformer [41] modifies the softmax-based self-attention in Transformer (42 to the spike-based formulation, which is appropriate for SNNs and achieves state-of-ther -tthhartttttt accuracy on the ImageNet dataset.
2.4 Acceleration for Sequence Processing
Massively parallel computing with graphics processing units (GPUs) [43 is one of the key factors that drives deep learning research. However, the serial computing characteristic of the recurrent structure such as SNNs and Recurrent Neural Networks (RNNs) can not fully exploit the parallel computing power of GPUs. To accelerate the sequence processing, convolution- eevoluttonn-bsed based methods 44 45) discard the recurrent structure and employ the convolutional layers, which are fully parallelized during training on GPUs The gated impulse linear recurrent network [46) is another solution that uses the Parallel Prefix Sum (Scan) algorithm 47 to parallelize linear recurrences. The Transformer [42 replaces recurrences with self-attention and positional encoding, which achieves faster training speed than recurrent and convolutional encoder-decoder structures.
3

---

3 Methods
In this section, we introduce the motivation of removing neuronal reset, the idea of generalizing spiking neurons without reset as the PSN, and the derivation of the masked PSN and the sliding PSN. Note that we use regular letters such as X to represent scalars, and bold letters such X to represent tensors.
3.1 Vanilla Spiking Neurons W/WO Reset
Spiking neurons in SNNs have rich neuronal dynamics, which endow SNNs with temporal information processing ability. In most cases, the behaviors of spiking neurons can be described by three discrete- time equations
[13
Н[ = f(V[t - [V[t-11,xXtht (()),
(1)
S[4 6(H[1) - Vin),
VIO [H] (LS((t) + Vreaet Sitt, hard reset
(3)
[t] Ven- S[t], soft reset
where [() is the input current, [[]] is the membrane potential after charging but before firing,
V the membrane potential after firing, and S(t] is the output spike at time-step t. Vsh Eq 2 the threshold potential, and Vreset in (3) 3)) is the reset potential. ((x) is the Heaviside step function and (z)) for all x 2 0, otherwise (xx) O. Eq. is the neuronal charging equation, and is specific to different spiking neurons. After charging. 7 will be compared to VTA and determine whether to fire spikes, which described by Eq. After firing, the membrane potential will be reset, (3) shows. Note that there are two kinds of resets, which are the hard reset and the soft reset. If the neuron fires, the hard reset will set [tt] to Vreeet, while the soft reset will decrease v(t) by Vin- The hard reset is widely used in surrogate training for better performance observed in experiments [48), while the soft reset is preferred in ANN2SNN for lower conversion errors. When simulating SNNs, the iterative process following Eqs.qT- over time-steps is employed, which has a time complexity of (T), where T is the number of time-steps.
For frequently-used spiking neurons, B is linear and can be reformulated to a non-iterative equation we can ignore Eq. (3) in some cases, e.g., in the subthreshold regime H[] < Vva for all t during period. Then V [1] at all I. and we will only use H[{] in this case. More specifically, we take the Integrate-and-Fire (IF) neuron and the Leaky Integrate-and-Fiig (L.IF) neuron as examples. For simplicity, suppose [[-1] 0 for both types of neurons, and the resting potential for the LIF
neuron is 0. The neuronal charge equation, or Eq. [.. for the IF neuron
H H 1] + X(t).
(4)
And 4q.44) (4) can be easily reformulated as
2 Σ
H
x
(5)
2=
The neuronal charge equation for the LIF neuron is
d| = (1 _ ) H1 1|
X[I..
(6)
Tm
where Tm is the membrane time constant. Eq. 66) can also be reformulated
H
- Tm
Σ
1 - -X]]. Tm
(7)
With the non-iterative equations, H[] at time-steps, or H (I),,, H[T -1]} can be calculated in parallel. For each H() obtained by the cumulative sum operation, the time complexity can be as low as (log(()) & (log(T)) when using the Parallel Prefix Sum (Scan) algorithm. When H is given, the element-wist operation Eq. (2) can also be applied on the whole H and the

---

time complexity is (1) in devices that support parallel computing such as the CUDA devices. Thus, the whole time complexity for the neuronal charging and firing reduces (log(T)) when we can ignore the neuronal resetring.
So, how can we ignore neuronal resetring all time-steps? A possible method is setting Vo. +00 to make (3) become an identity function. However, the neuron can only output in this case, which is meaningless. Another method is to remove the neuronal resetting directly from the neuronal dynamics, which means only using Eq. T and Eq 2
This crude and simple method may raise further concerns that the absence of neuronal resetting will cause the neuron to fire uninterruptedly. Note that the firing rate is determined by inputs, charging, thresholds, and resetting. Thus, the effect of only removing reset on firing rates is not deterministic. In addition, the experiment results in the next section will show that the uninterrupted firing issue will not occur.
3.2 Parallel Spiking Neuron
After removing neuronal resetting, the charging equation of the spiking neuron can be formulated into non-iterative equation, and the solution for H[] becomes a cumulative sum operation. It can be found that 5 and Eq. are two specific cases a linear combination of X(i]. More specifically, we formulate general linear combination as
T-1 Σ
H
WA, X[d].
(8)
where Wi,; is the weight between X and H(.. Then, Wi, = O( - i) for the IF neuron without reset, and We,s = 1 (1 - 1 y O(t - i) for the LIF neuron without reset. Based on the above analysis, we propose the Parallel Spiking Neuron (PSN), whose neuronal dynamics are as follows:
Tm
H = WX.
WERT E TxN
(9)
S = - - B), SE
(10)
where X is the input sequence, W is the learnable weight matrix, H is the hidden state sequence, B is the learnable threshold, and the binary output spike sequence. N is the number of batch size, and 7 is the number of time-steps. Note that W is a T' x 'I' matrix, indicating that [ can integrate the information from all time-steps directly, rather than that of the last one or two time-steps. Thus, the PSN is T -order neuron. No iterative equation is used, and the computation of PSN is fully parallel. The time complexity of the PSN is determined by (q. which is a matrix-matrin multiplication. The simulation of PSN is much faster than that of vanilla spiking neurons because the matrix-matrix multiplication is highly optimized in linear algebra libraries such as Intel MKI. and cuBLAS. BPTT is still used during training, but also in a parallel, rather than serial formulation.
3.3 k-Order Masked Parallel Spiking Neuron
The high-order characteristic of the PSN is also double-edged sword. The output sequence S can only be generated when all X[t] have arrived, indicating that each spiking neuron layer will increase latency T to the whole SNN. Such a latency problem is also reported on some time-to-first-ppe SNNs 49 50 5. Note that although the latency unavoidable, the throughput can still approach the plain SNN with the multi-stage pipeline 49]. To solve the latency problem of the PSN, we add mask Mt to multiply W geement-wieet before generating H and propose the k-order masked PSN, whose neuronal charging equation is
H (W. M2),,
W WERTYT Mr TTXT X TERTNN
(11)
where M2 is defined
11, j (0, otherwise
M
(12)
With the mask Mg. H[] depends only on the latest & inputs (X[ - k * 1)...,,, X(1) (we
assume that [ = 0 for all i < 0) and S(t) can be computed and sent to the next layer once X[4 is
received.
5

---

When training the k-order masked PSN, the progressive masking method is employed, which uses an all-ones matrix the mask first, and gradually replaces with Mg. More specifically, we use
M& X M + X)
(13)
as the mask, and À increases from 0 to 1 during training. In the early stages of training, (A) = 1. indicating that the future information can be exploited to provide appropriate primary parameters and help the network converge.
3.4 k-Orrdr Sliding Parallel Spiking Neuron
The parameters of the PSN and the masked PSN are time-wise, which require extra operations for processing sequences with variable lengths. To solve this issue, we modify the parameters to be shared across time-steps and propose the k-order sliding PSN, whose neuronal dynamics are
1-1
H[]] = Σ W. Xlt - +++ +1],
(14)
1=0
S[t] = (
(15)
where W = [Wo, Wi..., Wx-1] E R* is the learnable weight, X[j] 0 for all < O, and Vna is the learnable threshold. Similar the masked PSN, H[1] of the sliding PSN depends only on the latest inputs as well. More specifically, the weight slides over inputs and generates hidden states, which a standard ID convolution operation. Additionally, can also be implemented by matrix-matrix multiplication. When the input sequence X E RT'xN arrives and its length T' is known, the matrix A E gT'x can be generated as
A = WR-1-1+h: i+1-k ksjsi
(16)
0, otherwise
then H = AX. According to the results of our experiments, using matrixxmatrix multiplication is faster than using convolution.
3.5 Summary of the PSN Family
As a summary, Fig shows the comparison of the parameter number Paraam and the generation of hidden states of the PSN family. Remarkably, using the PSN in deep SNNs will add negligible number of parameters because T is small in directly trained SNNs. For example, using the PSN with T = 4 will add 340 and 200 parameters in spiking ResNet-18 18 and VGG-11, which only increase 0.00291% and 0.00 0.00015% parameters of the original SNNs. Additionally, the PSN family has only one hidden state H, making them consume less memory than the vanilla spiking neurons with at least two hidden states H, V during training, which are analyzed and verified by experiments in the supplementary materials.
4 Experiments
In this section, we report the experiment results of evaluating the PSN family in aspects of simula- tion performance and temporal/static data classifying accuracy. All experiments are based on the SpikingJelly (22 framework. The details of the training are provided in the supplementary materials.
4.1 Simulation Speed Benchmark
We evaluated the simulation speed benchmark of the PSN and the LIF neuron, which is widely used in deep SNNs and acts as the baseline of the vanilla spiking neurons. Note that the implementations and simulation speed of the masked PSN and the sliding PSN are almost identical to the PSN. Thus, we take the PSN as the benchmark for the PSN family. The benchmark is running in two modes, which are inference and training. In inference, we use the PyTorch just-in-time (JIT) compiling to fuse Eq. - of the LIF neuron across all time-steps into single function to accelerate by avoiding the calling overhead too many tiny CUDA kernels. However, PyTorch JIT does not support modifying the backward for a fused forward function, so the surrogate method used in the backward of Eq. (2)

---

Weight Matrix
Input Hidden State
Weight Matrix
Input
Hidden State
X
[[0]
7
W
W
Wi,
*
X[3]
[[]]]
[[1]
X12]
12.0
22
22,2
22.3
*
X[2]
W[2]
param 7 TT
X[3]
Masked PSN T= 4,
110 W11
X
[1]
[1]
112
1/.1
22.2
3C
[2]
2[2]
(2T + +1-1111-- -
param
K(3)
Sliding PSN k 2
[0)
[100)
Wo
X(1]
V(1)
We
W
X
Bparam
It
[3]
Figure 2: Comparison of the PSN family the parameter number Pparam and the generation of hidden states.
23
bocnting lios mit
tinso-ttepo
(a) Forward (inference)
(b) Forward and backward (training)
Figure 3: The executing time ratio of the LIF neuron and the PSN for one forward in inference and one forward and backward in training.
breaks the fusion of all three equations in training. Thus, have three smaller JIT functions at each time-step for the LIF neuron in training. For the PSN, there is no difference between the implementations for inference and training, which both involve matrix-matris multiplication and an element- wise operation. We tested the neuron numbers N 2 212 216 22 and the time-step numbers T 2, 4, 8, 16, 32, 64, which typical options for deep SNNs. Denote the executing time of two neurons as (PSN and (T. respectively, then the ratio fr.r is shown in Fig It can be found that in most cases, the simulation of the PSN is much faster than that of the LIF neuron.
tpex
4.2 Learning Long-term Dependencies
To verify the learning ability for long-term dependencies of different spiking neurons, we evaluated their performance on classifying sequential CIFAR10 and CIFAR100 100. In these tasks, the image will be sent to the SNN column by column, which similar right. The sequential image classification task is widely used to evaluate the learning ability of network for long-term dependencies. The network structure is modified from [5] by replacing 2D convolutional/max pooling layers with ID convolutional and average pooling layers, and removing
how humans read from left to

---

Neuron
PSN
masked PSN
sliding PSN
GL
KLIF
PLIF
LIF
LIF
Datasct
Sequential CIFARIÓ
88.45
85.81
86.70
83.66
83.26
83.49
81.50
79.50
Sequential CIFART00
62.21
60.69
62.11
58.92
57.37
57.55
55.45
53.33
Table 1: The accuracy (%) of different spiking neurons on sequential CIFAR.
0.00
0.70
0.45
000
200
1.00
spodh
spoch
PSX
cunkee PAN
KLLIF
0.35
0.25
0.25
0.25
Scang th2
0.15
0.15
0.5
0.05
spiking zroron layee Index
<piking DraEo lapee Iodex
(a) Sequential CIFARIO
(b) Sequential CIFARION 100
Figure 4: The accuracy curves and firing rates of different neurons for classifying sequential CIFAR.
the voting layers. Due the compared neurons include PSN, 32-order masked PSN, 32-order sliding PSN, GLIF with channel- wise parameters, KLIF, PLIF, and the LIF neuron w/wo reset. We use optimal reset options determined by ablation experiments for different neurons, which are using soft reset and detach reset [33 for the PLIF and LIF neurons, and using soft reset the KLIF neuron. The GLIF neuron uses learnable gates to control all options, which do not need to be set manually. For the masked PSN, min(1 1, 8 epoch/ / (epochs 1)). where epoch denotes the current training epoch, and epochs denotes the total number of epochs.
input, T is 32, which is the width of the image. The
Thows the accuracy of all neurons on sequential CIFAR. can found that the rank accuracy is PSN > sliding PSN > masked PSN > GLIF > PLIF > KLIF > LIF > LIF wo reset, which is as expected. The PSN exploits information from all time-steps and has the highest learning ability, but it may be criticized for unfair comparison because it can read all columns at once. Therefore, accuracy of the masked PSN and the sliding PSN is more convincing because they can still run in step-by-step mode. Due to the introduction of high-order information, they work better than any other traditional spiking neurons. Meanwhile, the GLIF ncuron is the best among traditional spiking neurons, indicating that fusing different kinds of charging, decaying, and resetting increases learning ability. The LIF neuron without reset has lower accuracy than the LIF neuron, indicating that the direct removal of reset drops the learning ability of the vanilla spiking neurons. Thus, using high-order information as the PSN family is necessary to fix this decline.
To make clear comparisons, we plotted the accuracy curves during training and firing rates of each spiking neuron layer of trained SNNs with different spiking neurons, as shown in Fig 4 These accuracy curves are the moving averages of 8 epochs, and the light-colored curves show the original accuracy. The curves of the PSN family are almost always above other curves, which clearly show their superior performance convergence speed and final accuracy. Note that there sharp drop

---

Dataset
Method
Spiking Network
Time-steps
Accuracy (%)
Dspike 38
Modified ResNet-18
6
94,25
ResNet-19
6
94.50
TDBN 30
ResNet-19
6
93.16
TEBN 33
ResNet-19
6
94.71
CIFARIO
PLIF
PLIF Net
8
93.50
KI.F 18
Modified PIJFF Net
10
92.52
GL
ResNet-19
95.03
PSN
Modified PLJF Net
95.32
ResNet-34
68.19
Dspike 38
VGG-16
71.24
TET 339
SEW ResNet-34
68.00
TDBN
ResNet-34 with double channels
67.05
ImageNet
TEBN 33
SEW ResNet-34
68.28
GLIF 16
ResNet-34
67.52
SEW ResNet-18
4
63.18
SEW ResNet 40
SEW ResNet-34
67.04
SEW ResNet-18
4
67.63
PSN
SEW ResNet-34
4
70.54
Dspike 38
ResNet-18
10
75.40
TET 39
VGG
10
83.17
TDBN 30
ResNet-19
10
67.80
TEBN 33
VGG
10
84,90
CIFARIO-DVS
P5
PIJFF Net
20
74.80
KI,FF 18
Modified PLIF Net
15
70.90
GLIF 16
Wide 7B Net
16
78. 10
SEW ResNet
40
Wide 7B Net
16
74.40
slidigg PSN (K 2)
VGG
4, 8, 10
82.30, 85.30, 85.90
Table 2: Comparison of the PSN family and other methods.
in the curves the masked PSN around epoch 32, which is caused by the mask becoming completely binary. The firing rates indicate that uninterrupted firing does not occur with the removal of neuronal resetting. In general, the PSN and the masked PSN are more easily to be activated and show higher firing rates than others. Meanwhile, the firing rates of the sliding PSN and the LIF neuron without reset are slightly higher than the vanilla spiking neurons. Thus, we conclude that removal of reset does increase firing rates, but the increment degree depends the neuron's 's structure. On the other hand, considering the huge improvement in simulation efficiency and long-term dependencies learning ability, the slightly higher firing rate is acceptable.
5 shows the accuracy-order curve on the sequential CIFAR 100 dataset for the masked PSN and sliding PSN, with the highest accuracy marked by a red *. It can be seen that when the order increases from to 4, the accuracy improves quickly. When we continue to increase the order, the improvement is marginal and the accuracy even drops.
0.62
0.60
0.58
accurayy
The highest accuracy is obtained by a large, but not the largest order, which is 20 and 31 for the masked PSN and sliding PSN, respectively. But in most cases, high accuracy can be guaranteed with & T' Remarkably, the curve of sliding PSN is almost always above the curve of masked PSN, and excels that of the PSN when & 21 and k 31, indicating that the invariant parameters have better generalization ability to process temporal information.
0.56
PSN
0.54
masked PSN
SPSN
0.52
GLIF
0.50
LIF
0
10
20
30
order
Figure 5: The accuracy-order curve on the sequential CIFAR100.
4.3 Static and Neuromorphic Data Classification
Static and neuromorphic data classification tasks are also common benchmarks for SNNs. We evaluated the PSN family on the static CIFAR10, ImageNet datasets, and the neuromorphic CIFAR1O- DVS (54] dataset. The number of time-steps for static datasets small, and the latency not the main issue. Thus, we use the PSN in CIFAR10 and ImageNet. According to the results on sequential CIFAR, the sliding PSN uses fewer parameters but gets higher accuracy, and we will use for CIFARIO-DVS. The results are listed in Tab

---

CIFARI0 We modified the network structure of [1],, or the PLIF Net, by using average pooling and removing voting layers. Our PSN uses the shortest time-steps T 4 and achieves the highest accuracy of 95.32%.
ImageNet We used the SEW ResNet 40 for classifying the ImageNet dataset. We verified the performance of the SPN on SEW ResNet-18/34 and got stable 3+% accuracy improvement over the original SEW ResNet with the IF neuron. Our method achieves 70.54% accuracy and is only second to the Dspike method using one more time-step and VGG-16, which has a number of parameters 6.3 times as many as ours.
CIFARIO-DVS We used the VGG structure from 39]. Considering the fact that the DVS data contains temporal features and T is larger than the static datasets, we use the 2-order sliding PSN. We achieve 82.3 % accuracy with T = 4, which the first work to get 80+% accuracy in such few time-steps. When increasing time-steps T = 8, we achieve 85.30% accuracy, which is 2+ % higher than that of the previous SOTA method 33 and we use two fewer time-steps. With = 10, the accuracy is further improved 85.90%.
5 Conclusion
In this paper, we remove reset from neuronal dynamics vanilla spiking neurons and propose the PSN family, including the PSN, the masked PSN, and the sliding PSN. The PSN family can be simulated in parallel, which greatly accelerates the training of deep SNNs. Weights of inputs in the PSN family be fully connected, masked/shared with custom order. Experimental results of simulation speed and temporal/static data classification verify the efficiency and accuracy of the PSN family when compared with previous spiking neurons. Our work may be a milestone well cornerstone for modern spiking neurons.
Acknowledgments and Disclosure of Funding
This work is supported by grants from the National Natural Science Foundation of China (62027804, 61825101, 62088 102, and 62176003), the major key project the Peng Cheng Laboratory (PCL2021A13), and the Agence Nationale de la Recherche under Grant ANR-20-CE23-000--00 DeepSee.
References
[1] Wolfgang Maass. Networks of spiking neurons: the third generation of neural network models. Neural Networks, 10(9):1659-167) 1997.
[2] Paul A Merolla. John Arthur, Rodrigo Alvarez-Icaza, Andrew S Cassidy, Jun Sawada, Filipp Akopyan, Bryan L Jackson, Nabil Imam, Chen Guo, Yutaka Nakamura, et al. A million spiking-neuron integrated circuit with scalable communication network and interface. Science, 345(6197)6668-66 2014.
[3] Mike Davies, Narayan Srinivasa, Tsung- Han Lin, Gautham Chinya, Yongqiang Cao, Sri Harsha Choday, Georgios Dimou, Prasad Joshi, Nabil Imam, Shweta Jain, Yuyun Liao, Chit-Kwan Lin, Andrew Lines, Ruokun Liu, Deepak Mathaikutty, Steven McCoy, Arnab Paul, Jonathan Tse, Guruguhanathan Venkataramanan, Yi-Hsin Weng, Andreas Wild, Yoonseok Yang, and Hong Wang. Loihi: A neuromorphic manycore processor with on-chip learning. IEEE Micro, 38(1):82-95 99, 2018.
[4] Jing Pei, Lei Deng, Sen Song, Mingguo Zhao, Youhui Zhang, Shuang Wu, Guanrui Wang, Zhe Zou, Zhenzhi Wu, Wei He, et al. Towards artificial general intelligence with hybrid tianjic chip 2019.
architecture. Nature, 572(7767):106-)
[5] Chris Eliasmith, Terrence C. Stewart, Xuan Choo, Trevor Bekolay, Travis DeWolf, Yichuan Tang, and Daniel Rasmussen. A large-scale model of the functioning brain. Science, (:866111:1222--11
2012.
[6] Marcel Stimberg, Romain Brette, and Dan FM Goodman. Brian 2, an intuitive and efficient neural simulator. eLife, 8:e47314, 2019.
10

---

[7] Emre O Neftci, Hesham Mostafa, and Friedemann Zenke. Surrogate gradient learning in spiking neural networks: Bringing the power of gradient based optimization to spiking neural networks. IEEE Signal Processing Magazine, 36(6):51-6) 2019.
[8] Amirhossein Tavanaei, Masoud Ghodrati, Saeed Reza Kheradpisheh, Timothée Masquelier, and Anthony Maida, Deep learning in spiking neural networks. Neural Networks, 111:47-63, 2019.
[9] Yufei Guo, Xuhui Huang, and Zhe Ma. Direct learning-basse -ased deep spiking neural networks: a review. Frontiers in Neuroscience, 17:1 17:1209795, 2023.
[10] Yongqiang Cao, Yang Chen, and Deepak Khosla. Spiking deep convolutional neural networks for energy-efficient object recognition. International Journal of Computer Vision, 113(1):54 66, 2015.
[11] Bodo Rueckauer, Iulia-Alexandra Lungu, Yuhuang Hu, Michael Pfeiffer, and Shih-Chii Liu. Conversion of continuous-valle vallued deep networks efficient event-driven networks for image classification. Frontiers in Neuroscience, 1 1:682,, 2017.
[12] Saeed Reza Kheradpisheh and Timothée Masquelier. Temporal backpropagation for spik- ing neural networks with one spike per neuron. International Journal of Neural Systems, 30(06):2050027, 2020.
[13] Kaushik Roy, Akhilesh Jaiswal, and Priyadarshini Panda. Towards spike-based machine intelligence with neuromorphic computing. Nature, 575(77844:607-617, 2019.
[14] Rui Yuan, Qingxi Duan, Pek Jun Tiw, Ge Li, Zhuojian Xiao, Zhaokun Jing, Ke Yang. Chang Liu, Chen Ge. Ru Huang, et al. A calibratable sensory neuron based on epitaxial vo2 for -pike-based neuromorphic multisensory system. Nature Communications, 13(1):1-12, 2022.
[15] Wei Fang, Zhaofei Yu, Yanqi Chen, Timothée Masquelier, Tiejun Huang, and Yonghong Tian. Incorporating learnable membrane time constant enhance learning of spiking neural networks. In Proceedings of the IEEF/CVF International Conference on Computer Vision (ICCV), pages 2661-2671, 2021.
[16] Xingting Yao. Fanrong Li, Zitao Mo, and Jian Cheng. GLIF: A unified gated leaky integrate- and-fire neuron for spiking neural networks. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems (NeurIPS), 2022.
[17] Ilyass Hammouamri, Timothée Masquelier, and Dennis George Wilson. Mitigating catastrophic forgetting in spiking neural networks through threshold modulation. Transactions on Machine Learning Research, 2022.
[18] Chunming Jiang and Yilei Zhang, Klif: An optimized spiking neuron unit for tuning surrogate gradient slope and membrane potential, 2023.
[19] Lang Feng, Qianhui Liu, Huajin Tang, De Ma. and Gang Pan. Multi-level firing with spiking ds-resnet: Enabling better and deeper directly-traine spiking neural networks. In Lud De Raedt, editor, International Joint Conferences on Artificial Intelligence Organization (ICCAI), pages 2471-2477, 7 2022. Main Track.
[20] Wachirawit Ponghiran and Kaushik Roy. Spiking neural networks with improved inherent recurrence dynamics for sequential learning. In Proceedings the AAAI Conference on Artificial Intelligence (AAA/), volume 36, pages 8001-8008, 2022.
[21] Yufei Guo, Xinyi Tong, Yuanpei Chen, Liwen Zhang, Xiaode Liu, Zhe Ma, and Xuhui Huang. Reedis-snn: Rectifying membrane potential distribution for directly training spiking neural networks. In Proceedings of the IEEE//CV Conference on Computer Vision and Pattern Recognition (CVPR), pages 326-335, June 2022.
[22] Bing Han, Gopalakrishnan Srinivasan, and Kaushik Roy. Rmp-snn: Residual membrane potential neuron for enabling deeper high-accuracy and low-latency spiking neural network. In Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 13558- 13567, 2020.

---

[23] Jianhao Ding, Zhaofei Yu, Yonghong Tian, and Tiejun Huang. Optimal ann-snn conversion for fast and accurate inference in deep spiking neural networks. In Zhi-Hua Zhou, editor, Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, IJCAI-21, pages 2328-2336. International Joint Conferences on Artificial Intelligence Organization, 2021. Main Track.
[24] Tong Bu, Wei Fang. Jianhao Ding, PengLin Dai, Zhaofei Yu, and Tiejun Huang. Optimal ann- snn conversion for high-aceuracy and ultra-low-latent -lweencccy spiking neural networks. In International Conference on Learning Representations (ICIR), 2021.
[25] Shikuang Deng and Shi Gu. Optimal conversion of conventional artificial neural networks to spiking neural networks. In International Conference on Learning Representations (ICLR), 2021.
[26] Zecheng Hao, Tong Bu, Jianhao Ding, Tiejun Huang, and Zhaofei Yu. Reducing ann-snn conversion error through residual membrane potential. Proceedings of the AAAI Conference on Artificial Intelligence, 37(1):11-2) Jun. 2023.
[27] Yujie Wu, Lei Deng, Guoqi Li, Jun Zhu, and Luping Shi. Spatio-temporal backpropagation for training high-per perfoppeforrancc spiking neural networks. Frontiers in Neuroscience, 12, 2018.
[28] Sumit Bam Shrestha and Garrick Orchard. Slayer: Spike layer error reassigmment in time. In Advances in Neural Information Processing Systems (NeurIPS), pages 1419-1428, 2018.
[29] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma. Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. International Journal of Computer Vision, 115(3):211 1122211112,, 2015.
[30] Hanle Zheng, Yujie Wu, Lei Deng, Yifan Hu, and Guoqi Li. Going deeper with directly-trained larger spiking neural networks. In Proceedings of the AAAI Conference on Artificial Intelligence (AAAI), volume 35, pages 1 11062-11070. 2021.
[31] Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International Conference on Machine Learning (ICML), pages 448-456. PMLR, 2015.
[32] Youngeun Kim and Priyadarshini Panda. Revisiting batch normalization for training low-latency deep spiking neural networks from scratch. Frontiers in Neuroscience, 15, 2021.
[33] Chaoteng Duan, Jianhao Ding, Shiyan Chen, Zhaofei Yu, and Tiejun Huang. Temporal effective batch normalization in spiking neural networks. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems (NeurIPS), 2022.
[34] Jacques Kaiser, Hesham Mostafa, and Emre Neftci. Synaptic plasticity dynamics for deep continuous local learning (decolle). Frontiers in Neuroscience, 14, 2020.
[35] Mingqing Xiao, Qingyan Meng, Zongpeng Zhang, Di He, and Zhouchen Lin. Online training through time for spiking neural networks. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems (NeurIPS). 2022.
[36] Nicolas Perez-Nieves and Dan F. M. Goodman. Sparse spiking gradient descent. A. Beygelz- imer, Y. Dauphin, P. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems (NeurIPS), 2021.
[37] Man Yao, Guangshe Zhao, Hengyu Zhang, Yifan Hu, Lei Deng, Yonghong Tian, Bo Xu, and Guoqi Li. Attention spiking neural networks. IEEE Transactions on Pattern Analysis and Machine Intelligence, pages 1-18, 2023.
[38] Yuhang Li, Yufei Guo, Shanghang Zhang, Shikuang Deng, Yongqing Hai, and Shi Gu. Differ- entiable spike: Rethinking dradient-descent for training spiking neural networks. In A. Beygelz- imer, Y. Dauphin, P. Liang, and Wortman Vaughan, editors, Advances Neural Information Processing Systems (NeurIPS), 2021.
12

---

[39] Shikuang Deng, Yuhang Li, Shanghang Zhang, and Shi Gu. Temporal efficient training of spiking neural network via gradient re-weighting. In International Conference on Learning Representations (ICIRR), 2022.
[40] Wei Fang, Zhaofei Yu, Yanqi Chen, Tiejun Huang, Timothée Masquelier, and Yonghong Tian. Deep residual learning in spiking neural networks. Advances in Neural Information Processing Systems (NeurIPS), 34, 2021.
[41] Zhaokun Zhou, Yuesheng Zhu, Chao He, Yaowei Wang, Shuicheng YAN, Yonghong Tian, and Li Yuan. Spikformer: When spiking neural network meets transformer. In International Conference Learning Representations (ICLR), 2023.
[42] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez. Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in Neural Information Processing Systems (NeurIPS), 30, 2017.
[43] Rajat Raina, Anand Madhavan, and Andrew Y Ng. Large-scale deep unsupervised learning using graphics processors. In International Conference on Machine Learning (ICML), pages 873-880, 2009.
[44] Nal Kalchbrenner, Lasse Espeholt, Karen Simonyan, Aaron van den Oord, Alex Graves, and Ko- ray Kavukeuoglu. Neural machine translation in linear time. arXiv preprint arXiv: 16X0: 10099, 2016.
[45] Jonas Gehring, Michael Auli, David Grangier, Denis Yarats, and Yann N Dauphin. Convolutional sequence to sequence learning. In International Conference on Machine Learning (ICML), pages
PMLR, 2017.
[46] Eric Martin and Chris Cundy. Parallelizing linear recurrent neural nets over sequence length. In International Conference on Learning Representations (ICLR), 2018.
[47] Mark Harris, Shubhabrata Sengupta, and John D Owens. Parallel prefix sum (scan) with cuda. GPU gems, 3(39):851-876 2007.
[48]
Eimantas Ledinauskas, Julius Ruseckas, Alfonsas Jursénas, and Giedrius Buračas. Training Deep Spiking Neural Networks. arXiv preprint arXiv:2006.0444 2020.
[49] Seongsik Park, Seijoon Kim, Byunggook Na, and Sungroh Yoon. T2fsnn: deep spiking neural networks with time-to-firt--pikke coding. In 2020 57th ACM/IEEE Design Automation Conference, pages 1-6. IEEE, 2020.
[50] Dongwoo Lew, Kyungchul Lee, and Jongsun Park. A time-to-fistt-siike coding and conversion aware training for energy -efrgy-efficient deep spiking neural network processor design. In Proceedings of the 59th ACM/IEEE Design Automation Conference, pages 265-270. 2022.
[51] Lina Bonilla, Jacques Gautrais, Simon Thorpe, and Timothée Masquelier. Analyzing time-to- first-spike coding schemes. Frontiers Neuroscience, 16, 2022.
[52] Wei Fang, Yanqi Chen, Jianhao Ding, Zhaofei Yu, Timothée Masquelier, Ding Chen, Liwei Huang, Huihui Zhou, Guoqi Li, and Yonghong Tian. Spikingjelly: An open-source source machine learning infrastructure platform for spike-based intelligence. Science Advances, 940) 1480, 2023.
[53] Friedemann Zenke and Tim P Vogels. The remarkable robustness of surrogate gradient learning for instilling complex function in spiking neural networks. BioRxiv, 2020.
[54] Hongmin Li, Hanchao Liu, Xiangyang Ji, Guoqi Li, and Luping Shi. Cifar10-dve An event- stream dataset for object classification. Frontiers Neuroscience, 11, 2017.
[55] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In Proceedings of the IEEE international conference computer vision, pages 1026-1034, 2015.
13

---

[56] Zhuang Liu, Hanzi Mao, Chao- Yuan Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie. A convnet for the 2020s. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 11976-1 1986, 2022.
[57] Ilya Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts. arXiv preprint arXiv: 03983, 2016.
[58] Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. arXiv preprint arXiv: 2017.
[59] Sangdoo Yun, Dongyoon Han, Seong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo. Cutmix: Regularization strategy to train strong classifiers with localizable features. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages
6023 6032, 2019.
[60] Samuel G Müller and Frank Hutter. Trivialaugment: Tuning-free yet state-of-the-art data augmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 774-782, 2021.
[61] Zhun Zhong, Liang Zheng, Guoliang Kang, Shaozi Li, and Yi Yang. Random crasing data aug- mentation. Proceedings of the AAAI conference on artificial intelligence (AAAI), volume 34, pages 13001-13008. 2020.
[62] Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna. Re- thinking the inception architecture for computer vision. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition (CVPR), pages 2818-2826. 2016
[63] Qingyan Meng, Mingqing Xiao, Shen Yan, Yisen Wang, Zhouchen Lin, and Zhi-Qual thi-Quan Luo. Training high-performance low-latency spiking neural networks by differentiation on spike representation. In Proceedings of the IEEF/CVE Conference on Computer Vision and Puttern 2022.
Recognition (CVPR), pages
[64] Nitin Rathi and Kaushik Roy. Diet-snn: A low lot-latenyy spiking neural network with direct input encoding and leakage and threshold optimization. IEEE Transactions on Neural Networks and Learning Systems, 2021.
[65] Nitin Rathi, Gopalakrishnan Srinivasan, Priyadarshini Panda, and Kaushik Roy. Enabling deep spiking neural networks with hybrid conversion and spike timing dependent backpropagation. In International Conference on Learning Representations (ICLR), 2020.
14


Reviews:
[{"review_id": "OCDujQ2X4s", "reviewer": "Reviewer_sjmV", "summary": "This paper removes the reset mechanism from the dynamics of conventional LIF/IF neurons, and proposes to reformulate the neuronal dynamics using matrix multiplication instead of the iterative updating for the membrane potential. This matrix multiplication then can be simulated in parallel to accelerate the training of deep SNNs.\nThis paper makes strong assumption that hidden states are independent of their predecessors, and proposes the Parallel Spiking Neuron (PSN) and its variants, masked PSN, sliding PSN. \nThe whole framework is built on unfolding the computing graph over the latency $T$ and then uses a fully connected layer $H = W X$ to replace the neuron dynamics.\n", "questions": "As above in Weakness. \n\nThe whole PSN framework is built on the condition $u(t) < V_{th}$, how to get spikes if this is the precondition for PSN? \n\nThe lack of dependency between successive time-steps is a concern for me. The whole framework is built on unfolding the computing graph over the latency $T$ and then uses a fully connected layer $H = W X$ to replace the neuron dynamics.  It doesn't make sense. Can you give more detailed explanation on this?", "limitations": "\n1. The whole PSN framework is built on the condition $u(t) < V_{th}$, not clear how to get spikes if this is the precondition for PSN.\n\n2. Using future information: when calculating $H[t] = \\sum_{i=1}^T W_{t,i} X[i]$, in this PSN the future information is used. For the sliding PSN and masked PSN, the authors tried to avoid using future information. So that in order avoiding using the future information, it's better to use masked PSN and sliding PSN, but in the experiments, it's not clear which PSN version is used. \n\n3. This paper makes strong assumption that hidden states are independent of their predecessors, the Parallel Spiking Neuron (PSN) is proposed based on this.  How to describe the neuron dynamics along the time-dimension if we remove this hidden state dependency?\n", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Parallelization: Parallelizing in spiking neurons is an interesting topic in SNNs. By removing the reset mechanism, the neuronal dynamics can be reformulated in a non-iterative form. The proposed Parallel Spiking Neuron (PSN) framework allows for parallelized neuronal dynamics, enabling efficient computations across multiple processing units or threads.\n\n2. Utilization of Temporal Information: The PSN utilizes fully connected weights for inputs, maximizing the utilization of temporal information and potentially enhancing the model's ability to capture temporal patterns.\n\n3. High Simulation Speed: The PSN framework, with its independent hidden states and parallelizable dynamics, achieves extremely high simulation speed, which can be advantageous for real-time applications and large-scale simulations.\n", "weaknesses": "1. Lack of Reset Mechanism: The removal of the reset mechanism in the PSN may limit its ability to handle certain types of dynamics or tasks that rely on precise timing and reset behavior. The authors did not provide reasonable explanation of why neuronal resetting can be ignored, what should be done to compensate for reset removal. \n\n2. Large number of trainable parameters introduced by the new Weights: The use of matrix multiplication in the PSN introduces additional trainable parameters in the new weights, the performance improvement could benefit from using more trainable parameters, but not the new PSN model. With the same number of parameters for both PSN and LIF, will the performance still perform good using the new PSN compared to LIF?\n\n3. Using future information: when calculating $H[t] = \\sum_{i=1}^T W_{t,i} X[i]$, in this PSN the future information is used. For the sliding PSN and masked PSN, the authors tried to avoid using future information. So that in order avoiding using the future information, it's better to use masked PSN and sliding PSN, but in the experiments, it's not clear which PSN version is used. \n\n4. This paper makes strong assumption that hidden states are independent of their predecessors, the Parallel Spiking Neuron (PSN) is proposed based on this.  How to describe the neuron dynamics along the time-dimension if we remove this hidden state dependency?\n\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your constructive comments and questions, which are also helpful for other reviewers. Please refer to \"To All Reviewers\" for responses to **Weaknesses 1: Lack of Reset Mechanism**. Responses to other comments are as follows.\n\n## **Weaknesses 2**\n\nAlthough the PSN family uses extra trainable parameters, the increasing memory cost is negligible. To explain this point clearly, we list the number of parameters of different layers, which are shown in Table R1. Note that the parameter number of masked PSN is  $T^{2} + T$ in training, and parameters are masked progressively. After training, the parameter number reduces to $\\frac{(2T+1-k) \\cdot k}{2} + T$ in inference. \n\n| Layer                 | Description                                                  | Params                                   |\n| --------------------- | ------------------------------------------------------------ | ---------------------------------------- |\n| convolutional layer   | $C_{in}$ input channels, $C_{out}$ output channels, $k_{h} \\cdot k_{w}$ weight shape | $C_{in} \\cdot C_{out} \\cdot k_{h} k_{w}$ |\n| fully connected layer | $F_{in}$ input features, $F_{out}$ output features           | $F_{in} \\cdot F_{out}$                   |\n| PSN                   | $T$ time-steps                                               | $T^{2} + T$                              |\n| Masked PSN            | $T$ time-steps, $k$ orders                                   | $T^{2} + T$                              |\n| Sliding PSN           | $T$ time-steps, $k$ orders                                   | $k+1$                                    |\n\n**Table R1. The number of parameters of different layers.**\n\nIn lines 207-209, we have shown that using the PSN in deep SNNs will cause a negligible increase in parameters. Now let us take the SNNs in Table 1 as a new example. Denote the number of the SNN using the LIF neuron $P(LIF)$ as the baseline, we report the ratios of the number of parameters $\\frac{P(neu)}{P(LIF)}$ of SNNs in Table 1. The results are shown in Table R2. Although $T=32$ is large in these SNNs, the results in Table 1 show that the using of the PSN family will increase no more than 5% extra parameters.\n\n| Dataset\\Neuron          | PSN      | Masked PSN | SPSN     | GLIF     | KLIF     | PLIF     | LIF  | LIF wo reset |\n| ----------------------- | -------- | ---------- | -------- | -------- | -------- | -------- | ---- | ------------ |\n| **Sequential CIFAR10**  | 1.014398 | 1.014398   | 1.00045  | 1.077785 | 1.000014 | 1.000014 | 1    | 1            |\n| **Sequential CIFAR100** | 1.013777 | 1.013777   | 1.000431 | 1.074431 | 1.000013 | 1.000013 | 1    | 1            |\n\n**Table R2. The ratios of the number of parameters $\\frac{P(neu)}{P(LIF)}$ of SNNs in Table 1.**\n\n> With the same number of parameters for both PSN and LIF, will the performance still perform good using the new PSN compared to LIF?\n\nWe can easily build SNNs with fewer parameters. Denote the SNN using the LIF neuron for classifying the sequential CIFAR100 in Section 4.2 as the baseline. We reduce the number of the output/input channels of the first/second convolutional layer by 1, causing a slight reduction of parameters. We train these SNNs with the same hyper-parameters and training options as Section 4.2, and the results are shown in Table R3. The results indicate that the PSN family still has higher accuracy even with fewer parameters than the SNN using the LIF neuron.\n\n|                 | LIF (baseline) | PSN    | Masked PSN | Sliding PSN |\n| --------------- | -------------- | ------ | ---------- | ----------- |\n| **Params**      | 536548         | 520415 | 520415     | 513254      |\n| **Accuracy(%)** | 55.45          | 62.74  | 60.23      | 61.75       |\n\n**Table R3. Comparison of sequential CIFAR100 between LIF and PSN family with fewer parameters.**\n\n## **Weaknesses 3**\n\n> but in the experiments, it's not clear which PSN version is used.\n\nYou can refer to Table 2 and Section 4.3 for more details. In a word, we use the PSN for CIFAR10 and ImageNet, and the sliding PSN with order $k=2$ for CIFAR10-DVS. \n\n## **Weaknesses 4**\n\n> How to describe the neuron dynamics along the time-dimension if we remove this hidden state dependency?\n\nYou can refer to Section 3.1 for more details. Here we summarize that the iterative neuronal dynamics with removing neuronal reset can be reformulated to non-iterative equations, as Eqs. (5) and (7) show. \n\n## Questions\n\n> The whole PSN framework is built on the condition $u(t)<V_{th}$, not clear how to get spikes if this is the precondition for PSN.\n\nWe regret the unclear expression, which may mislead you. In line 151, we assume $H[t] < V_{th}$ as a method to ignore neuronal resetting. In lines 152-158, we claim that this method is meaningless, and a better method to ignore neuronal resetting is removing resetting from neuronal dynamics directly. In the PSN family, $H[t]$ is not restricted to be lower than the threshold $V_{th}$. \n\n> The lack of dependency between successive time-steps is a concern for me.\n\nDenote the initial value of $H[t]$ as $H[-1]$. In the vanilla spiking neuron, the neuronal dynamics is a typical Markov chain with the transfer function $g$ determined by the neuronal dynamics, and \n$$\nH[t] = g(H[t-1], X[t]) = g(g(H[t-2], X[t-1]), X[t]) = ...=g(g(g(H[-1], X[0]), X[1])..., X[t-1]).\n$$\nThus, $H[t]$ is actually determined by $X[0], X[1], ..., X[t]$. In PSN, masked PSN and sliding PSN, $H[t]$ is determined by $X[0], X[1], ... ,X[T-1]$, $X[0], X[1], ..., X[t]$ and $X[t-k+1], X[t-k+2], ..., X[t]$.\nIn conclusion, we can find that $H[t]$ in both vanilla spiking neurons and the PSN family is determined by input at specific time-steps. The main difference is that the dependency in vanilla spiking is indirect, which is implemented by a Markov chain. While in the PSN family, the dependency is direct, which is the weighted sum."}, {"review_id": "5KSFSkFFzw", "reviewer": "Reviewer_P1JU", "summary": "This paper proposes the Parallel Spiking Neuron (PSN), which generates hidden states that are independent of their predecessors, resulting in parallelizable neuronal dynamics and extremely high simulation speed. The weights of inputs in the PSN are fully connected, which maximizes the utilization of temporal information. The authors evaluate  the PSN family on simulation speed and temporal/static data classification, and  the results show the overwhelming advantage of the PSN family in efficiency and  accuracy. ", "questions": "1. Please explain the advantage of the parallel spiking neurons as mentioned \"high simulation speed \" in abstract.\n2. How about the whole training and test time of the proposed SNN compared with other methods?\n3. Is there any limitation once applied the parallel spiking neurons into the neuromorphic hardware？How it could map to the hardware?\n4.What is the network architecture used in Table2, such as the \"VGG\"? Please describe the detailed network architecture in the Table 2.", "limitations": "1. Since the authors claim that the main goal of the proposed parallel spiking neuron model is to improve the simulation speed, the current experiments do not reflect that advantage over other methods, in which only focusing on the accuracy and the firing rates analysis.\n2. The network architecture of the spiking networks in Table 2 is not clear.\n\nThe above limitations have been explained clearly in the following response.", "rating": 6, "confidence": 5, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "1. The motivation considering the parallel spiking neurons for high simulation speed is interesting and important for the futural application of spiking neural networks.\n2. The experiments on the large datasets, such as ImageNet, improves the techinical soundness of the spiking neural networks. ", "weaknesses": "1. Since the main goal of the proposed parallel spiking neuron model is to improve the simulation speed, the current experiments do not reflect that advantage over other methods, in which only focusing on the accuracy and the firing rates analysis.\n2. The network architecture of the spiking networks in Table 2 is not clear.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your comprehensive comments. Please refer to \"To All Reviewers\" for responses to **Question 3**. Other point-to-point responses are as follows.\n\n## **Weaknesses 1 and Questions 1, 2**\n\n> Please explain the advantage of the parallel spiking neurons as mentioned \"high simulation speed \" in abstract.\n\nYou can refer to Section 4.1 and Figure 3 for experimental results about the training and inference speed.\n\n> How about the whole training and test time of the proposed SNN compared with other methods?\n\nComparable methods include PLIF, KLIF, and GLIF. These neurons are more complex than the LIF neuron with additional operations. Thus, we add experiments about speed comparison between the PSN and the LIF neuron, and if the PSN is faster than the LIF neuron, then it is also faster than other more complex neurons.\n\nWe compare the training and inference speed (ms/batch) of the PSN and the LIF neuron with the SEW ResNet ResNet-18/34 and $T=4, 8$, which is the typical option for training deep SNNs. We use the LIF neuron from SpikingJelly, which provides the fastest implementation. The experiments are performed on an Ubuntu 18.04 server with an Intel Xeon Silver 4210R CPU, an NVIDIA A100-SXM-80GB GPU, and 256 GB of memory. \n\n|           | SEW ResNet-18 (T=4) |           | SEW ResNet-18 (T=8) |           | SEW ResNet-34 (T=4) |           | SEW ResNet-34 (T=8) |           |\n| --------- | ------------------- | --------- | ------------------- | --------- | ------------------- | --------- | ------------------- | --------- |\n|           | Train               | Inference | Train               | Inference | Train               | Inference | Train               | Inference |\n| PSN       | 45.94               | 10.91     | 47.99               | 19.17     | 60.63               | 16.74     | 71.52               | 30.87     |\n| LIF       | 60.40               | 10.55     | 98.10               | 18.82     | 94.09               | 17.52     | 149.86              | 30.42     |\n| LIF(cupy) | 49.71               | 12.08     | 53.81               | 18.74     | 77.49               | 17.36     | 82.38               | 30.19     |\n\n**Table R4. Comparison of speed (ms/batch) between the SNN using PSN and the LIF neuron.**\n\nThe results are shown in Table R4. `LIF` is the LIF neuron implemented by PyTorch, and`LIF(cupy)` is the LIF neuron implemented by CuPy. The results show that the SNN using PSN is much faster than using the LIF neuron in training, and the advantage increases when the scale of the network is larger or $T$ increases. In inference, there is not much difference among speeds, which may be caused by that the speed of spiking neuron layers is not the bottleneck.\n\nCompared with `LIF(cupy)`, the speed advantage of PSN is not so significant with `LIF`. Note that `LIF(cupy)` fuses all operations concerning all time-steps into one single CUDA kernel, which minimizes the calling overhead of CUDA kernels including memory access time and kernel launch time. The PSN is implemented by PyTorch, and its operations include a matrix-matrix multiplication ($WX$), an element-wise subtraction ($H - B$), and a Heaviside function ($\\Theta(H - B)$) in forward or a surrogate function ($\\sigma (H - B)$) composed of many math operations in backward, which has higher calling overhead than `LIF(cupy)` using a single CUDA kernel. The speed of PSN can be improved further if its neuronal dynamics apart from the matrix-matrix multiplication are fused into one single CUDA kernel. However, considering the fact that PSN implemented by PyTorch is fast enough, we have not implemented the CuPy backend for PSN.\n\n## Weaknesses 2 and Question 4\n\n> What is the network architecture used in Table2, such as the \"VGG\"? Please describe the detailed network architecture in the Table 2.\n\nSorry for the unclear expression. \n\nThen the detailed network structure in Table 2 is:\n\n- Modified PLIF Net: `{{c256k3s1-BN-PSN}\\*2-APk2s2}\\*2-Flatten-FC4096-PSN-FC10`\n- SEW ResNet: the standard SEW ResNet [1] using PSN as spiking neuron layers\n- VGG: `{c64k3s1-BN-SPSN}-{c128k3s1-BN-SPSN}-APk2s2-{c256k3s1-BN-SPSN}*2-APk2s2-{{c512k3s1-BN-SPSN}*2-APk2s2}*2-Flatten-DP-FC10`, and `SPSN` is the sliding PSN with $k=2$\n\n```\n[1] Fang, Wei, et al. \"Deep residual learning in spiking neural networks.\" Advances in Neural Information Processing Systems 34 (2021): 21056-21069.\n```\n"}, {"review_id": "3DYUJUPnhv", "reviewer": "Reviewer_Xbwd", "summary": "The paper presents an approach to improve the efficiency and accuracy of Spiking Neural Networks by using a dependency method to generate hidden states, resulting in parallelizable neuronal dynamics and a significant increase in simulation speed. ", "questions": "1. Since the proposed models use parallel input of X[t-1], X[t], X[t+1], I am wondering how memory-intensive the models will be for parallel processing, and especially how it changes for the different datasets. Basically, it would be interesting to see a comparison of FLOPS/MACs for the standard vanilla spiking neurons and the proposed parallel spiking neurons\n\n2. The VGG shows better performance for ImageNet and CIFAR-DVS. The authors show the performance of the VGG model is currently state of the art using vanilla neurons for the ImageNet dataset. It would be great if the authors could report the results of the VGG model with parallel neurons for the ImageNet dataset. I understand the tight time constraint and hope this won't be very difficult, especially because the authors have already shown results for VGG on the CIFAR-10 DVS\n\n3. I am also curious how stable are these parallel spiking neurons compared to the vanilla neurons since the reset is removed. In the presence of input/weight noise, will it still perform as well? \n\n4. When the authors evaluate the long-term dependencies, it would be interesting to see results on non-image datasets in Supplementary. Mainly because there is a high degree of correlation among the pixels for such image datasets, and I was just wondering how well it would work for other non-image-based datasets. \n\n", "limitations": "It would be great if the authors could include what are the key limitations of this work and what are the trade-offs with the current vanilla spiking neurons.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "The authors analyze the impact of removing the reset function from standard charge-fire-reset neuronal dynamics, proving that the parallelization of the spiking neuron can be achieved without it. They also present a general formula by rewriting the neuronal dynamics without a reset and introduce the PSN, a spiking neuron with entirely parallelizable neuronal dynamics. The paper also assesses the PS family's performance on sequential, static, and neuromorphic data classification tasks, showing that they attain higher accuracy than traditional spiking neurons.", "weaknesses": "Please see Questions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your positive comments. Please refer to \"To All Reviewers\" for our discussions about the trade-off with vanilla spiking neurons. Our responses to your constructive questions are as follows.\n\n## **Question 1**\n\nThanks for your variable question. We have summarized the number of memory reading/writing and FLOPS of spiking neurons in Table R5. \n\n|             | Memory Readings                   | Memory Writings | FLOPs              |\n| ----------- | --------------------------------- | --------------- | ------------------ |\n| LIF/PLIF    | $5T$                              | $3T$            | $9T$               |\n| KLIF        | $6T$                              | $3T$            | $10T$              |\n| GLIF        | $11T$                             | $2T$            | $20T$              |\n| PSN         | $T^{2} + 2T$                      | $T$             | $2T^{2}$           |\n| Masked PSN  | $\\frac{(2T+1-k) \\cdot k}{2} + 2T$ | $T$             | $(2T+1-k) \\cdot k$ |\n| Sliding PSN | $k+1+T$                           | $T$             | $(2T+1-k) \\cdot k$ |\n\n**Table R5. Counting of the memory reading/writing and FLOPs of different neurons.**\n\n## **Question 2**\n\nConsidering the fact that the tuning of hyper-parameters also requires many comparative experiments, we are not able to finish the training of VGG-16 during the rebuttal period.\n\n## **Question 3**\n\nThanks for your suggestion. We add experiments about input/weight noise on the sequential CIFAR100 classification task. Note that all noise is added during both training and inference.\n\n#### Noised Input\n\nWe add a Gaussian noise with mean 0 and variance $\\sigma^2$ on inputs $X$. The results are shown in Table R6. It can be found that at each noise level, the PSN family is still better than the LIF neuron.\n\n| Neuron\\\\$\\sigma$ | 0 (no noise) | 0.1   | 0.2   | 0.3   |\n| ---------------- | ------------ | ----- | ----- | ----- |\n| LIF              | 55.45        | 54.01 | 51.53 | 49.17 |\n| PSN              | 62.21        | 61.31 | 59.21 | 56.88 |\n| Masked PSN       | 60.69        | 57.63 | 56.17 | 56.76 |\n| Sliding PSN      | 62.11        | 59.72 | 56.74 | 55.17 |\n\n**Table R6: The accuracy on sequential CIFAR100 with noised inputs**.\n\n#### Noised Weights\n\nWe add a Gaussian noise with mean 0 and variance $\\sigma^2$ on weights $W$ and bias $B$ (if have) of all convolutional, batch normalization, and fully connected layers.\n\n| Neuron\\\\$\\sigma$ | 0 (no noise) | 0.001 | 0.005 | 0.0075 | 0.01  |\n| ---------------- | ------------ | ----- | ----- | ------ | ----- |\n| LIF              | 55.45        | 52.26 | 31.79 | 21.48  | 14.98 |\n| PSN              | 62.21        | 57.56 | 20.56 | 12.49  | 4.67  |\n| Masked PSN       | 60.69        | 53.58 | 23.16 | 11.79  | 4.58  |\n| Sliding PSN      | 62.11        | 58.05 | 38.81 | 29.33  | 21.27 |\n\n**Table R7: The accuracy on sequential CIFAR100 with noised weights**.\n\nThe results are shown in Table R7. When noise becomes large, the accuracy of both the LIF neuron and the PSN family drops quickly. The accuracy rank in the large noise environment is `Sliding PSN > LIF > PSN > Masked PSN`. It is worth noting that both the sliding PSN and the LIF neuron use shared weights across time-steps, while the PSN and the sliding PSN use temporal-wise weights. The former might have better robustness for noised network weights.\n\n## **Question 4**\n\nThanks for your advice. We add two kinds of experiments beyond image classification.\n\n### Reinforcement Learning and Control\n\nWe evaluate the performance of SNNs with different spiking neurons on the reinforcement learning and control tasks. We evaluate four typical tasks from OpenAI Gym with non-image inputs, which include Ant-v3, HalfCheetah-v3, Hopper-v3, and Walker2d-v3. We use similar encoding and decoding methods from [1]. The results are reported in Table R8. It can be found that the performance of the PSN and the sliding PSN is much higher than other neurons.\n\n|                                | Ant-v3 |         | HalfCheetah-v3 |         | Hopper-v3 |         | Walker2d-v3 |         | Average Performance Ratios |\n| ------------------------------ | ------ | ------- | -------------- | ------- | --------- | ------- | ----------- | ------- | -------------------------- |\n| DAN [2]                        | 5472   | 100.00% | 10471          | 100.00% | 3520      | 100.00% | 4999        | 100.00% | 100.00%                    |\n| PopSAN (current-based LIF [1]) | 4848   | 88.60%  | 10523          | 100.50% | 517       | 14.69%  | 4199        | 84.00%  | 71.94%                     |\n| PopSAN (LIF)                   | 4991   | 91.21%  | 8500           | 81.18%  | 2613      | 74.23%  | 3751        | 75.04%  | 80.41%                     |\n| PopSAN (PSN)                   | 5210   | 95.21%  | 9622           | 91.89%  | 3255      | 92.47%  | 4248        | 84.98%  | 91.14%                     |\n| PopSAN (SlidingPSN)            | 5362   | 97.99%  | 9849           | 94.06%  | 3406      | 96.76%  | 4603        | 92.08%  | 95.22%                     |\n\n**Table R8: The performance comparison of spiking neurons on reinforcement learning and control tasks.**\n\n### Speech Recognition\n\nWe add experiments about speech recognition. We use the same network structure and data processing methods from [3]. We achieve 95.56% accuracy, which is higher than the LIF neuron with 94.5% in [3].\n\n```\n[1] Tang, Guangzhi, et al. \"Deep reinforcement learning with population-coded spiking neural network for continuous control.\" Conference on Robot Learning. PMLR, 2021.\n[2] Fujimoto S, Hoof H, Meger D. Addressing function approximation error in actor-critic methods[C]//International conference on machine learning. PMLR, 2018: 1587-1596.\n[3] Pellegrini, Thomas, Romain Zimmer, and Timothée Masquelier. \"Low-activity supervised convolutional spiking neural networks applied to speech commands recognition.\" 2021 IEEE Spoken Language Technology Workshop (SLT). IEEE, 2021.\n```\n\n"}, {"review_id": "s8klDjaTRZ", "reviewer": "Reviewer_G2d4", "summary": "This paper introduces the parallel spiking neuron (PSN) and several variants. The primary benefit of PSN over the existing spiking neurons is the parallel implementation on digital hardware, which brings dozens of times of acceleration on GPU. The accuracy also demonstrates the effectiveness of this paper. ", "questions": "Can authors show the ablation study of PSN and vanilla SNN on static image datasets?\n", "limitations": "Listed above. ", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The idea of parallelized neuron implementation is interesting and can be efficiently processed on GPU.\n\n2. The experiments show good performances. ", "weaknesses": "The authors pinpoint a disadvantage of current SNN, that is, low running speed on GPUs. However, it is known that GPU is not the most ideal device for deploying SNN, rather, it should be the neuromorphic hardware. PSN focuses on the optimization of SNN on GPU, however, GPU cannot utilize the binary spikes to lower the energy. So, even if PSN can accelerate the inference on GPU, compared to ANN on GPU, there is no advantage of efficiency still compared to ANNs. I was wondering whether the optimization on GPU is really useful cause people can always use ANNs on GPU.\n\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your encouraging comments about the acceleration and accuracy of the PSN family. Our responses to the weakness are as follows.\n\n> I was wondering whether the optimization on GPU is really useful cause people can always use ANNs on GPU.\n\nA typical workflow to use SNNs is:\n\n1. Train SNNs to get high task performance\n2. Prune and quantize the weights of SNNs during or after training\n3. Deploy SNNs to neuromorphic chips\n\nFor the moment, many researches aim at steps 1 and 2. Although GPU is not the target device for SNNs to deploy, the massively parallel computing ability makes it become the most widely used devices to train SNNs. We believe that the introduction of PSN decreases the training cost and benefits the SNN community.\n\nMeanwhile, although the PSN family has high efficiency in GPUs, it is also compatible with neuromorphic chips. Please refer to our response to the question \"How it could map to the hardware?\" from reviewer P1JU for more details.\n\n> Can authors show the ablation study of PSN and vanilla SNN on static image datasets?\n\nThanks for your suggestion. We add ablation experiments on CIFAR10. We use the same network from the SNN to classify CIFAR10 in Table 2. Due to the limited rebuttal period, we only use 128 channels for the SNN and train 128 epochs. The results are shown in Table R9. Note that $T=4$ and we also train the masked PSN and the sliding PSN with $k=0,1,2,3$.\n\n| Neuron          | IF    | LIF   | PLIF  | KLIF  | GLIF  | PSN   |\n| --------------- | ----- | ----- | ----- | ----- | ----- | ----- |\n| **Accuracy(%)** | 92.24 | 91.86 | 92.24 | 92.47 | 91.90 | 92.95 |\n\n| Neuron\\Order | 1     | 2     | 3     | 4     |\n| ------------ | ----- | ----- | ----- | ----- |\n| Masked PSN   | 92.59 | 92.44 | 92.75 | 91.94 |\n| Sliding PSN  | 90.81 | 91.79 | 92.13 | 92.46 |\n\n**Table R9.  Ablation experiments using different spiking neurons on CIFAR10.**\n\nAs [1] suggests, the spiking neurons without leaks may have better accuracy in static datasets. And we also add the experiment using the IF neuron. Table R10 shows that the accuracy rank on CIFAR10 is `PSN > Masked PSN (k=3) > KLIF > Sliding PSN (k=4) > PLIF = IF > GLIF > LIF`. The rank indicates that the PSN family also performs better than most of vanilla spiking neurons in static datasets.\n\n```\n[1] Fang, Wei, et al. \"Deep residual learning in spiking neural networks.\" Advances in Neural Information Processing Systems 34 (2021): 21056-21069.\n```\n\n"}]

Author rebuttal:
Thanks for all reviewers' valuable comments. We are encouraged that reviewers find the idea of parallelizing spiking neurons interesting and commend the fast simulation speed of the PSN family. Meanwhile, most reviewers are concerned about the hardware implementation of the PSN family and the trade-off with vanilla spiking neurons. Our responses to these questions are as follows.

## Hardware Compatibility

To the best of our knowledge, the behavior of most event-driven neuromorphic chips is closer to the step-by-step forward propagation (send input $X[t]$ to the SNN and get output $Y[t]$) that each event is processed and routed between cores. Thus, the masked PSN and the sliding PSN can be implemented in these chips because they support the step-by-step forward propagation. The PSN requires input at all time-steps and can not work in these chips. While some heterogeneous chips [1, 2] support the layer-by-layer forward propagation (send inputs $X[0], X[1], ..., X[T-1]$ to the SNN and get outputs $Y[0], Y[1], ..., Y[T-1]$), and the PSN has the potential to work in this type of neuromorphic chips.
When deploying the PSN family on hardware, memory consumption should also be considered. Please refer to Figure 2 in the main text for details about the memory consumption of parameters. Additionally, the masked PSN and the sliding PSN require a stack with length $k$ to store $X[t-k+1], X[t-k+2], ..., X[t]$ if the hardware uses a step-by-step forward propagation.

```
[1] Kim, Sangyeob, et al. "C-DNN: A 24.5-85.8 TOPS/W complementary-deep-neural-network processor with heterogeneous CNN/SNN core architecture and forward-gradient-based sparsity generation." 2023 IEEE International Solid-State Circuits Conference (ISSCC). IEEE, 2023.
[2] Chang, Muya, et al. "A 73.53 TOPS/W 14.74 TOPS heterogeneous RRAM In-memory and SRAM near-memory SoC for hybrid frame and event-based target tracking." 2023 IEEE International Solid-State Circuits Conference (ISSCC). IEEE, 2023.
```

## Trade-off

#### **Disadvantages**

The PSN family lacks the neuronal reset. The neuronal reset is important in neuronal dynamics, as Table 1 has shown that the LIF neuron without reset has lower performance on temporal tasks. We summarize the effect of neuronal reset as follows:

1. Avoid too-high firing rates
2. Clear (hard reset) or reduce (soft reset) the influence of previous inputs after firing a spike
3. Introduce a nonlinearity during generating hidden states

For effect 1, we have shown in Figure 4 and lines 265-269 that the firing rate of the PSN family is higher than vanilla spiking neurons, but the increment degree is minor. And the firing rate is still far away from 1.0, which does not damage accuracy. Meanwhile, the learnable weights and thresholds of the PSN family can also adjust the firing rates directly.

Effect 2 works similarly to the gate mechanism in LSTMs. For the moment, the PSN family does not involve such a dynamic mechanism. The PSN simply uses all inputs without filtering, while the masked PSN and sliding PSN uses the latest $k$ inputs.

Effect 3 is caused by involving $S[t]$ in neuronal reset, which is generated by the nonlinear Heaviside function. While the generation of $H[t]$ in the PSN family is fully linear.



Although the PSN family lacks effects 2 and 3, it still works better in some temporal/static data classification tasks, as we have shown in this paper. 
The PSN family is the prototype of parallelizable spiking neurons. Based on it, effects 2 and 3 can be implemented by introducing a nonlinear gate mechanism. Here let us provide the formulation of the gated PSN. For example, when adding the input gate $\mathbf{I}$ and the forgetting gate $\mathbf{G}$ as 
$$
\mathbf{I} = \sigma(\mathbf{W_{I}}\mathbf{X} + \mathbf{B_{I}}), ~~~~~~~~~~~~~~~\mathbf{W_{I}} \in \mathbb{R}^{T \times T}, \mathbf{X} \in \mathbb{R}^{T \times N}, \mathbf{B_{I}} \in \mathbb{R}^{T}
$$

$$
	\mathbf{G} = \sigma(\mathbf{W_{G}}\mathbf{X} + \mathbf{B_{G}}), ~~~~~~~~~~~~~~~\mathbf{W_{G}} \in \mathbb{R}^{T \times T}, \mathbf{X} \in \mathbb{R}^{T \times N}, \mathbf{B_{G}} \in \mathbb{R}^{T}
$$

where $\sigma$ is the sigmoid function or the Heaviside function with surrogate gradients. Then the hidden states are generated by
$$
H[t] = G[t] \cdot H[t-1] + (1 - G[t])\cdot I[t] \cdot X[t].
$$
Although $H[t]$ is calculated by an iterative equation, it can still be parallelized by Parallel Prefix Sum (Scan) algorithm as 
$$
p[i] = \prod_{j=0}^{i}G[j], 
$$
$$
	c[i][j] = \prod_{l=i+1}^{j},
$$
$$
G[l] = \begin{cases}
			\frac{p[i]}{p[j]}, j \geq i \\\\
			0, \mathrm{otherwise}
		\end{cases},\\
$$

$$
	H[t] = \sum_{i=0}^{t}(1 - G[t])\cdot I[t] \cdot X[t] \cdot c[i][t].
$$

With CUDA devices to solve $H[t]$ at all $t$, the time complexity is $\mathcal{O}(\mathrm{log}(T))$.

#### **Advantages**

The advantages of the PSN family have been discussed in the paper in detail. We conclude them as:

1. Highly parallelizable neuronal dynamics and fast simulation speed 
2. Higher or equal task performance to the vanilla spiking neurons
3. Easy to learn long-term dependency because the connection between any $X[i]$ to $H[j]$ is direct

## Network Structure Symbols

To answer some questions, we need to show the detailed network structure. We use the following symbols to represent the network structure in our responses:

- `c2k3s4`: the convolutional layer with output channels `2`, kernel size `3`, and stride `4`
- `BN`: the batch normalize layer
- `APk2s2`: the average pooling layer with kernel size `2` and stride `2`
- `Flatten`: the flatten layer
- `DP`: the dropout layer
- `FC10` as the fully connected layer with `10` output features
- `{}*2`: `2` repeat structures, e.g., `{FC10}*2` is `{FC10-FC10}`

Meta-review: The reviewers were in agreement on the valuable contributions of this paper, and I commend the authors on both the paper and the detailed rebuttal, which was critical in winning over all reviewers.  I'm pleased to report that it has been accepted to NeurIPS.  Congratulations!  Please revise the manuscript according to the reviewer comments and discussion points.
Decision: Accept (poster)