Robust and accelerated single-spike spiking neural network training with applicability to challenging temporal tasks
Authors: Luke Taylor
Andrew J King
Nicol Spencer Harper
Venue: Submitted to ICLR 2023 (2023)
Keywords: Spiking neural networks
single-spike
accelerated training
dead neuron problem

Abstract: Spiking neural networks (SNNs), particularly the single-spike variant in which neurons spike at most once, are considerably more energy efficient than standard artificial neural networks (ANNs). However, single-spike SSNs are difficult to train due to their dynamic and non-differentiable nature, where current solutions are either slow or suffer from training instabilities. These networks have also been critiqued for their limited computational applicability such as being unsuitable for time-series datasets. We propose a new model for training single-spike SNNs which mitigates the aforementioned training issues and obtains competitive results across various image and neuromorphic datasets, with up to a $13.98\times$ training speedup and up to an $81\%$ reduction in spikes compared to the multi-spike SNN. Notably, our model performs on par with multi-spike SNNs in challenging tasks involving neuromorphic time-series datasets, demonstrating a broader computational role for single-spike SNNs than previously believed.

Under review as conference paper at ICLR 2023
ROBUST AND ACCELERATED SINGLE-SPIKE SPIKING NEU- RAL NETWORK TRAINING WITH APPLICABILITY TO CHAL- LENGING TEMPORAL TASKS
Anonymous authors Paper under double-blis lobile-blinddae review
ABSTRACT
Spiking neural networks (SNNs), particularly the single-spike variant in which neurons spike at most once, are considerably more energy efficient than stan- dard artificial neural networks (ANNs). However, single-spike SSNs are diffi- cult to train due their dynamic and non-differentiable nature, where current solutions are either slow or suffer from training instabilities. These networks have also been critiqued for their limited computational applicability such as being unsuitable for time-series datasets. We propose a new model for train- ing single-spike SNNs which mitigates the aforementioned training issues and obtains competitive results across various image and neuromorphic datasets, with up to a x training speedup and up to an 81% reduction in spikes com- pared to the multi-spike SNN. Notably, our model performs on par with multi- spike SNNs in challenging tasks involving neuromorphic time-series datasets, demonstrating broader computational role for single-spike SNNs than previ- ously believed.
INTRODUCTION
Artificial neural networks (ANNs) have achieved impressive feats over recent years, obtaining human-level performance on visual and auditory tasks (Hinton et al., 2012; He al., 2016), nat- ural language processing (Brown et al., 2020) and challenging games (Mnih et al., 2015; Silver et al., 2017; Vinyals et al., 2019). However, as the difficulty and complexity of the tasks increase, so has the size of the networks required to solve them, demanding a substantial and unsustain- able amount of energy (Strubell et al., 2019; Schwartz al., 2020). Inspired by the extreme energy efficiency of the brain (Sokoloff, 1960), spiking neural networks (SNNs) emulated on neuromor- phic computers attempt to solve this dilemma, requiring significantly less energy than ANNs (Wunderlich et al., 2019). These networks on visual (Fang et al., 2021; Zhou & Li, 2021), auditory (Yin et al., 2020; Yao et al., 2021) and rein- forcement learning problems (Patel et al., 2019; Tang al., 2020; Bellec et al., 2020).
of growing interest, obtaining noteworthy results
particular class of SNNs in which individual neurons respond with at most one spike aims to further amplify the energy and scaling advantages of standard SNNs and ANNs. Inspired by the sparse spike processing shown to exist at least for certain stimuli in the auditory and visual systems (Heil, 2004; Gollisch Meister, 2008), and forming class of universal function approx- imator (Comsa et al., 2020), these networks obtain extreme energy efficiency due to their single- spike nature (Oh al., 2021; Liang al., 2021). Although providing promising path toward building very large and energy energy-efficient networks, we are yet to understand how to properly train these SNNs. The success of the backprop training algorithm in ANNs does not naturally trans- fer to single- and multi-spike SNNs due to their non mon-differentiable activation function. Current attempts at training are either slow (as time is sequentially simulated) or suffer from training instabilities (e.g. the dead neuron problem) and idiosyncrasies (e.g. requiring particular regu- larisation) (Eshraghian et al., 2021). Additionally, it has been argued that single-spike networks have limited applicability and are not suited for temporal problems, recently pointed out by Eshraghian et al. (2021): "[...] it enforces stringent priors upon the network (e.g,, each neuron must fire only once) that incompatible with dynamically changing input data" and Zenke

---

Under review conference paper at ICLR 2023
al. (2021): "[...] only using single spikes in each neuron has its limits and is less suitable for processing temporal stimuli, such as electroencephalogram (EEG) signals, speech, or videos".
In this work we address these shortcomings by proposing networks, for which the main contributions are summarised as follows.
model for training single-spin
Our model for training single-spike SNNs eschews all sequential dependence on time and exclusively relies on GPU parallelisable non-sequential operations. We experimen- tally validate this to obtain faster training times over sequentially trained control mod- els on synthetic benchmarks (up to 16.77 speedup) and real datasets (up to x speedup).
2.
We obtain competitive accuracies on various image and neuromorphic datasets with treme spike sparsity (up to 81% fewer spikes than standard multi-spike SNNs), with our model being insensitive to the dead neuron problem and requiring careful network regularisation. In other single-spike training methods, but not in our model, the dead neuron problem tends to halt learning due to reduced network activity.
3. We showcase our model's applicability in deeper and convolutional networks, and through the inclusion of trainable membrane time constants manage to solve difficult temporal problems otherwise thought to be unsolvable by single-spike networks.
BACKGROUND AND RELATED WORK
b
(ms)
Figure 1: Spiking neuron dynamics. a. Left: multi-spike neuron emitting and receiving (per presynaptic terminal) multiple spikes. Right: Input and output activity of the neuron (bottom panel: Input raster, middle panel: Input current and top panel: Membrane potential V. Dotted line represents the firing threshold and a dot above denotes spike). b. Left: A single-spike neuron emitting and receiving (per presynaptic terminal) at most one spike per stimulus. Right: Input and output activity of the neuron).
2.1 SINGLE-SPIKE MODEL
spiking neural network (SNN) consists of artificial neurons which output binary signals known as spikes (Figure la). Assume feedforward network architecture of L fully connected layers, where each layer I consists of y(l) neurons that are fully connected to the next layer I+ via synaptic weights w((+1) R xx!" Neuron in layer emits spike s() [r] (0, 1} time if its membrane potential 17(0 [t] R reaches firing threshold Vik-
1, if v [t]> Vth
/ [4] [ty=
[x) ( otherwise
(1)
Membrane potentials evolve according to the leaky integrate and fire (LIF) model
dv (t) = = vV() + Vrest + (1)
(2)
dt
where E is the membrane time constant and R IR is the input resistance (Gerstner et al., 2014). Without loss of generality the LIF model is normalised (v (r) e [0, 1] by Vrest = Vn = 1, = see Appendix) and discretised using the forward Euler method (see Appendix), from
Note, refer continuous time and to refer discrete time.
2

---

Under review as conference paper at ICLR 2023
which the membrane potential can be computed at every discrete simulation time step r (l...., T3 for T E N using the difference equation below.
NU--
v [t+ [1 [1]
(b
[1+ + Σ 1 j=1
s [r]
(3)
Input current 1.00 1r
Spike reset
The membrane potential is charged from the current induced by the incoming presynaptic spikes [r] ( and from the constant bias current source b." Over time, this potential dissi- pates, and the degree of dissipation is captured by 0 f = = (for simulation time-step size år R). The neuron's membrane potential is resting state Vrest in the absence of any input current and emits a spike s/ [r] = if the potential rises above firing threshold V (after which it is reduced back close to resting state).
To enforce the single- spike constraint, we keep track if a neuron has spiked prior to time us- ing the variable d [t] max(s 11, (t 1)), which is zero before the first spike and one thereafter (d!" [r = 0] 0). We then redefine the output spikes as S(D(r) [r] - (1-a(") [x]) - s/9tt), [x], thus ensuring that no more than a single spike is emitted during simulation (Figure lb).
2.2 SINGLE- SPIKE TRAINING TECHNIQUES
The main problem with training single- and multi-spike SNNs is the non-differentiable nature of their activation function. This precludes the direct use of the backprop algorithm (Rumel- hart et al., 1986), which has underpinned the successful training ANNs. Various SNN training solutions have been proposed, which we group into three categories.
Shadow training Instead of directly training SNN, an already trained ANN is mapped to SNN. This approach has actively been explored the multi-spike setting (O' O'Connor et al., 2013; Esser et al., 2015; Rueckauer et al., 2016; 2017), with recent work extending this to single-spike networks (Stöckl & Maass, 2019; Park et al., 2020). Although these approaches permit the training of large networks, they come with various shortcomings. Some shortcomings are method spe- cific, such as Stöckl Maass (2019) who outline how a single ANN unit can be represented as a network of spiking units. However, this leads to undesirable blowup of network parameters in their conversion process (which is avoided by our approach). Other shortcomings are more gen- eral, such the lack of support for training neural parameters besides synaptic weights (which our approach permits) or inference accuracy being lost in the conversion process, where mapped SNNs perform worse than the original ANNs (which we avoid).
Training using the spike times An approach used to directly train SNNs using backprop in- volves passing gradients through the time of spiking, which sidesteps the aforementioned non- differentiability issue (Bohte et al., 2002; Mostafa, 2017; Comsa al., 2020; Kheradpisheh & Masquelier, 2020; Zhang et al., 2021; Zhou & I., 2021; Zhou et al., 2021). Although commonly used for training ainll-spike SNNs, this approach suffers from various shortcomings, such as 1. the dead neuron problem, where a lack of spiking activity halts the learning process (which overcome), 2. being usually constrained to integrate and fire (IF) neurons (where support both the IF and LIF model), 3. having performance dependent on the computationally costly processing of presynaptic spikes using postsynaptic potential (PSP) kernels (which show not to be necessary) and 4. requiring careful network regularisation (which we avoid).
Training using the membrane potentials Another approach to directly training SNNs using backprop is by replacing the undefined gradient of the non-differentiable spike function with a surrogate gradient (Esser et al., 2016; Hunsberger & Eliasmith, 2015; Zenke & Ganguli, 2018; Lee et al., 2016), which permits the flow of gradient through every membrane potential in time (Bellec et al., 2018; Shrestha & Orchard, 2018; Neftci et al., 2019). This method has been shown to circumvent the dead neuron problem and permit the training of other neural parameters besides synaptic connectivity (such as membrane time constants) that have been shown to improve net- work performance (Perez-Nieves et al., 2021). However, these results have not been replicated in
3

---

Under review conference paper at ICLR 2023
the fingl-spiike setting (which do). A shortcoming of this method is its slow training speed, as the network needs to sequentially be simulated at every point in time (which we overcome).
3 PROPOSED TRAINING SPEEDUP ALGORITHM
TT [1]
S 2
3
(4)
02 [1]
(M 2
[3]
(4)
0
S (1
m 2
S 19 33
S ( [4]
31
[1]
/ [2]
( [3]
1/ [4]
01
[1][1]
[ [2]
[1) [3]
/ [4]
S 1 [1]
S [2]
S P9! 33
[4]
Time (ms)
Figure 2: Illustration of our model. The computational graph of our model for 4 time steps. Input spikes ( induce currents 11), which charge the membrane potential without reset Viu These no-reset membrane potentials are mapped to erroneous output spikes ((D), which then transformed to a latent representation (d)) encoding an ordering of spikes and finally mapped to the correct output spikes ( (same coloured edges denote output from same source). Exam- ple activity of our model throughout the different stacks of processing.
We propose new model for training SNNs in which individual neurons spike at most once. Our solution overcomes the slow training speeds of prior training algorithms by eschewing all se- quential dependence and recasting the standard single-spike model to exclusively rely non- sequential operations. Although our model performs more calculations than the standard single- spike model, all these calculations (see Appendix). Our model is comprised of three main steps which are readily implementable in modern auto differentiation frameworks (Abadi et al., 2016; Paszke et al., 2017; Bradbury al., 2018). For illustration purposes, we provide a diagram of the model's computational graph (Fig- ure 2a) and an example of how input spikes are transformed throughout the model's different layers of processing (Figure 2b).
highly parallelisable and thus substantially faster to train
1. Convert presynaptic spikes to input current As in the standard model, we map the time series of presynaptic spikes si 2 to a time series of input currents 140 which is achieved using tensor multiplication.
NOI-D
[n== = Σ with s(t [[1 [r]
(4)
Calculate membrane potentials without reset contrast to the standard model, we calcu- late modified membrane potentials (i() from the input current 1 by excluding the reset mecha-
nism. By dropping the reset term -S [r] in Equation 3 and unrolling this altered equation (see Appendix), we obtain a convolutional form allowing us to calculate these no-reset membrane potentials (il)) without any sequential operations (where B = -
" BT.
p:t [t] = B'v."0 v.!0 [0] (1- B)
Σ
k 1,00 (T)
k=1
(5)
v(' (1- @ [t]
Map no-reset membrane potentials to output spikes We map the time series of reset membrane potentials 4i to outpur spikes si) (which contains at most one spike). obtain
Hold face variables demotes arrays opposed scalar values.
4

---

Under review conference paper at ICLR 2023
time series of erroneous output spikes 5 by passing no-reset membrane potentials vi through the spike function f (Equation 1)
(6)
Due the removal of the spike reset mechanism, only the first spike occurrence in 5 follows the dynamics set out by the LIF model (Equation 3) and thus all spikes succeeding the first spike occurrence are removed (compliant with the single spike assumption). We achieve this by con- structing correct output spikes s with 5 [r] for r E (1,2...., T} except 5 [r] for the smallest satisfying (/([(r) (if such ((0 exists). straightforward solution would be to it- erate over all elements in 5iD) and set all spikes succeeding the first to zero, but such sequential calculation is the very problem we set out to remediate. We propose vectorised solution this problem which is comprised of two steps:
1. Map the erroneous output spikes 5 to a latent representation = "() where ev- ery element therein encodes an ordering of the spikes. This achieved by passing the erroneous output spikes / through proposed function & (Proposition 1), which maps all elements besides the first spike occurrence value other than one (z, [t] 1 for
all except for the smallest satisfying 5/ [x] = if such exists).
2. Obtain the correct output spikes / g(z,") by passing the latent representation zi through function g, which uses the encoded spike ordering produce the correct out- puts spikes s by mapping every value besides one to zero.
1,
if 0 ; |+]
g(z
g(2(")(0== [[
(7)
0, otherwise
Proposition 1. Function (1 =1=== Ei= 5 [k(( ++ 1) acting on S 10, 1) contains at most
one element equal one (S [(]] (() for the smallest satisfying 5 [r] (if such exists).
Proof. Firstly, if ( for all e [1 then ((5 [[]] 0 for all E [1, T (follows from substitu-
tion). Secondly, if 5(0 [4] = for smallest f E [1, 7] then ((5( = (follows from substitution)
and there can exist no f2 f1 such that (SS (1) [t2] = as
(CS (0S [r+
50h (t
Σ 8 [k)((]((++ 1)) 5/0
Σ 5 [k](t k+1)+ Σ [e+1]
[+]
11
[[]] + Σ
500 [k]
k=1
Thus (155))1e2) (pSSS))tt (t for all > 4 as
H0 S
[k]
=10
EXPERIMENTS AND RESULTS
We investigate model's speedup advantages and performance real datasets in comparison to prior work. All models were implemented using PyLorch (Paszke et al., 2017) with benchmarks and training conducted cluster of NVIDIA Tesla A100 GPUs.
5

---

Under review conference paper at ICLR 2023
Training duration (n 100)
Training speedup (b 64)
speedup
1eo1 Dureltee (sec)
900
Standard
8.00x
FastsNN
700
600
Backward speedup
4,00x
Duna
-1
Hidden
2.00x
400
1.00x
200
0.43x
6
logo Simulation steps
logz Simulation steps t
Training speedup
Training speedup (t=2")
Forward speedup (n 1000)
Fwd. vs bkwd. pass (n 1000)
c
(t= b 128)
Hidden units n
Batch
10.0
200
32
Speedup
400
8
64
5.0
1
2
0.0
4
6 S Sho0s
3
1011
3 4 8 1011
Layers
logg
Simulation steps
log2 Simulation steps
Hidden units
Figure 3: Training speedup of our model over the standard model. a. Total training speedup as function of the number of hidden neurons and simulation steps (left), alongside the responding forward and backward pass speedups (right). b. Training durations of both models for fixed hidden neurons и 100 and variable batch size c. Training speedup over different number of layers for fixed steps & = 2 and batch size b = 128. d. Training speedup over large number of hidden neurons n for fixed time steps t 2" and variable batch size e. Forward pass speedup for fixed time steps = 2 and variable batch size b. f. Forward vs the backward pass speedup of our model for fixed time steps = 2 and variable batch size b-f use a sample average with the mean and s.d. plotted.
4.1 SPEEDUP BENCHMARKS
We evaluate the speedup advantages of our model over the standard single-spike model trained using surrogate gradients, by simulating the forward and backward passes for different numbers of hidden units, layers, simulation steps and batch sizes on synthetic spike dataset (see Ap- pendix).
Robust speedup for different numbers of hidden units and simulation steps We observe a considerable training speedup across range of hidden units and simulation steps in single layer (Figure 3a). We obtain an optimal speedup of 16.77 for n= = 100 units and z= time steps, where our model takes 4.34 0.9ms compared to the 7..82 it takes the standard model to complete a training pass (Figure 3b). Our model still obtains a reasonable speedup of 3.440 for largest benchmarked n 1000 units and 2 time steps (albeit the forward pass speedup being slower).* These speedups are even more pronounced when the membrane time constants are fixed (obtaining maximum speedup 17.42x) or when using smaller batch sizes (with batch sizes b = 32 and b 64 obtaining a maximum speedup of 35.05x and 25.16x, respectively; See Appendix).
Applicability deeper networks We find our model obtain substantial training speedups when using multiple layers (Figure 3c) and layers containing thousands of neurons (Figure 3d). The training speedups remain similar across an increasing numbers of layers for different num- ber of hidden units (Figure 3c). Furthermore, we obtain a speedup of 3x when using a large number of neurons (ranging between 2 2-103 20 neurons) in single layer (Figure 3d), Inter-
(1) We permit gradients to flow through the points where g(z [+] 0. This is due the convolutional algorithm chosen by cudnn (Chetlur et al., 2014).

---

Under review as conference paper at ICLR 2023
estingly, these speedups remain approximately the same across the different number of neurons, even when the batch size is changed.
Speedup advantages and room for improvement Previous attempts at accelerating SNN train- ing either speed up the backward pass (Pere--Nevee & Goodman, 2021) or remove it completely (Bellec et al., 2020). These methods however still sequentially compute the forward pass, which our model is able to accelerate (Figure 3e). Furthermore, observe the backward pass to slow down relative to the forward pass for increasing time steps (Figure 3f). Further training speedup may therefore be achieved using sparse gradient descent, as auto differentiation frameworks are not optimised for the sparse nature of SNNs (Perez-Nies) & Goodman, 2021).
4.2 PERFORMANCE ON REAL DATASETS
We investigate the applicability of our model to classify real data from different domains and of varying complexity (Table 1). These include the Yin-Yang dataset (Kriener et al., 2022) in which the goal is to classify spatial coordinates belonging to different groups, and the MNIST (LeCun, 1998) and Mushion-MNiS, (F-MNIST) (Xiao et al., 2017) image datasets, where the objective is to classify images of handwritten digits and fashion items. All these analog datasets were converted into spike representation using the time-to-fisst-yyy performance on two neuromorphic datasets, being the vision N-MNIST (Orchard et al., 2015) and the more difficult auditory SHD dataset (Cramer et al., 2020). The N-MNIST dataset is the MNIST dataset mapped onto a spike code using neuromorphic vision sensor and the SHD dataset com- prises spoken digit waveforms converted into spikes using a model of auditory bushy neurons in the cochlear nucleus.
encoding (see Appendix). We also test
d
Detaset
Spike reduction (%)
80
Yi-Tang
Accuracy ciff (%)
2.0
12.5
Activity la. u.) (a.
MNIST
Speedup x
1.5
10.0
1
20
2.5
0.0
0.0
fimsbng MNIST
F-MNIST
HANNST
SHD
Vinstnn
HAIST
FMMIST H-MNIST
SHD
Vin-iang
RNIST
F-MNIST
MMIST
SHD
Time (a.
Figure 4: Analysis of our models performance on real datasets. Difference in accuracy be- tween the standard multi-spike and our model. b. Training speedup of our model vs the standard single-spike model. Reduction in spikes of our single-spike model the standard multi-spike model (a-c use a 3 sample average with the mean and s.d. plotted). d. Training robustness of our model to solve different datasets when starting with zero network activity, which is fatal to other single-spike training methods. Top panel: Normalised training loss over time. Bottom panel: Normalised network activity over time, where the red cross denotes the absence of any spikes.
Obtaining competitive results across different image and neuromorphic datasets The results of our model across all datasets are comparable or superior to prior reported results using single- spike SNNs. We reach accuracy of 299980099 02%,00 97.91 % and 89.05% using single hidden layer net- work on the Yin-Yang, MNIST and F-MNIST datasets respectively, where best performing prior work reported an accuracy of 95.90% 98.50% and 88. 18.1% respectively. Furthermore, our single- spike model nearly obtains the same accuracies to those obtained in the standard multi-spike SNN on these datasets (Yin-Yang and MNIST < 0.3% difference; F-MNIST 2% difference; Fig- ure 4a).
Againn, this is due to the convolutional algorithm chosen by cudnn. Results reported by Kheradpisheh al. (2022).
7

---

Under review as conference paper at ICLR 2023
Table I: Performance comparison to existing literature (* denotes self self-implementationn, denotes data augmentation and f denotes trainable time constants).
Dataset
Model
Spike code
Architecture
Neuron model
Accuracy (%5
Epoch time (s)
Yin Yin-Yang
Göltz (2023)"
single
120-10
LIP (slpha-PSP)
95.9
our model
single
120-10
98 02 0.19
1.41 0.022
Nefici 92019)"
single
LIP
0.37
19.77 0.118
Zhang et (2021)?
single
(ReL-PSP)
MNIST
(2020)
single
(olpha-PSP)
model"
single
10000-10
LIpe
97.91
14.05
Nefici (2018))1
single
1000-10
LIF
97.87 0.00
65,05
Zhang et al. (2021)
single
16C P2-3PP.----52222 ---0000--P-000000 128-000-----1100
IF (RoI-PSP)
98.4
Zhou et al. (2021)
single
32C2 16C3-10
IF 99.33
Mirsadeghi al. (2021)
single
P00S-P2-1000-10
IF (PL-PSS)
98.2
oue model"
single
S0C3--2---C3-P2-1000
LIF
99.30
23:41
1201
single
3205-P2A-4CC-PP-10000
TTR
20.3541
32.34
Zhung et al. (20213)
single
1000-10
(Ret-PSP)
FMNIST
Kheradplshehh Masqueller (20220)
single
1000-10
our model'
single
LIF
16.16
Nefici 12015)**
single
1000-10
LIp"
+
80.89 0.10
Zhang al, (20213)
single
16CS- 12--30C-P2--00--22
(ReL-PSP)
90.1
Mirsadeghi al. (20211)
stngle
20C5 P2-.-PP----C--C-0 222---00------000000002-222 12-0000-12-1001100 1000-10
IF (PL-PSP)
82,8
our model'
single
LIF"
90.57 0.28
24.11 0.10
Nofici ol. 82018)*
single
1750
90.76.00.3
33.60
our model
single
300-10
LIF
95.91 +
36.29 0.54
N-MNIST
LIP
96.34 0.20
61.93
Nefhei al. (2019)"
single
LIB
97.47:
191.90
SHDD
Cramer es al. (2020)
et al. (2019)"
muiti
LIF
70.81 4:2006
48.
45,80
(2020)
multi
recurtent LIF
Perez- Nieves al. (2021)
multi
120-20
recurrent LIP
82.7
our model
single
300-20
LJF
4..50
0.68
LIF
70.32 0.30
11.27 #021
Nefici et al. (2019)*
single
300-20
LIP"
68.91
$0.12
Single-spike neurons solve challenging temporal problems using neural heterogeneity It has been noted that single-spike SNNs are well suited for static datasets (such as spike encoded im- ages) and less suited for processing temporally complex stimuli (such as audio or video) due to the single-spike constraint (Zenke et al., 2021; Eshraghian et al., 2021). Prior single-spike SNN training techniques have attempted to optimise network connectivity without learning other neural parameters, such as membrane time- time-constants, which have shown to improve perfor- mance in multi-spike SNNs (Perez-Nieves et al., 2021). We explored the effect of learning the membrane time-constants in our single-spike model. We obtained an accuracy of 44.50% using network trained with fixed time constants the temporally-comple auditory SHD dataset. However, by including learnable time constants we were able to obtain a much higher accuracy of 70.32% which is similar to the performance obtained by standard SNN with trainable time constants 70.81% 81% or recurrent connections 71.40%.
Drastic speedup in training We obtain over four-fold training speedup across all datasets over the standard single-spike SNN, with maximum speedup of x3.98x on the Yin-Yang dataset (Fig- ure 4b). We observe similar training speedups over the multi-spike SNN (see Appendix). Differ- ences in speedups are due to the different temporal lengths and input dimensions of the datasets, as well the different network architectures employed (see section 4.1).
Increased spike sparsity Our single-spike SNN is able to solve various datasets with a large reduction in spikes compared to standard multi-spike SNN (Figure 4c), with over a 44% and up to a 81% % reduction in spikes. This corroborates the value of obtaining more energy-efficient computations using single- rather than multi-spike neuromorphic systems (Liang al., 2021; Oh
Note that - unlike the neuromorphic datasets the training speedup for the image datasets is dependent on the selected number of simulation time steps for transforming image into the temporal domain (see Appendix for chosen values).
8

---

Under review as conference paper at ICLR 2023
et al., 2021; Zhou et al., 2021), energy consumption scales approximately proportional to the number of emitted spikes (Panda et al., 2020).
Training deeper convolutional architectures We evaluate our model in deeper convolutional architectures, which to date remains largely unexplored in single-spike SNNs (Mirsadeghi et al., 2022). We trained multi-layer layer convolutional network on the MNIST and F- E-MNIST datasets, ob- taining accuracies (MNIST: 99% and F-MNIST: 90.57%) similar to best performing prior work (MNIST: 99.4% and F-MNIST: 92.8%), whilst being faster to train comparison to the control (MNIST-speedup ~ x and F-MNIST-speedup 1.39x).
Robust learning and bypassing the dead neuron problem A limitation of current single-spike SNN training methods is the dead neuron problem, referring to the hinderance in learning when neurons do not spike, as the learning signal is dependent on the occurrence thereof (Eshraghian et al., 2021). Our model is able to overcome this problem as we use surrogate gradients for train- ing, in which the learning signal is instead passed through the membrane potentials. We exper- imentally verified this by showing how networks instantiated with zero starting activity (fatal to other single spike training methods) still manage to solve different datasets (Figure 4d).
5 DISCUSSION
SNNs emulated on neuromorphic hardware are a promising avenue towards addressing the en- ergy and scaling constraints of ANNs (Wunderlich et al., 2019). Single-spike SNNs further amplify these energy improvements through extreme spike sparsity, as energy consumption scales ap- proximately proportionally to the number of emitted spikes (Panda et al., 2020). To date, SNN training remains challenging due to the non- cin-differentiable nature of the spike function, pro- hibiting the direct use of the backprop training algorithm which underpins the success of ANSs. Various extensions of backprop for SNNs have been proposed, but fall short in particular as- pects. Gradients can be passed through the timing of spikes (Bohte et al., 2002; Mostafa, 2017; Kheradpisheh & Masquelier, 2020), yet this method suffers from the dead neuron problem, quires careful regularisation or imposes compurationally-exenni ternatively, gradients can be passed through the membrane potentials using surrogate gradi- ents (Shrestha & Orchard, 2018; Neftci et al., 2019), and although this method improves upon the problems of passing gradients through the spike times, it is painfully slow.
modelling constraints. Al-
In this work, we address these problems by proposing a new general model (e.g. neurons can be IF or LIF) for training single-spike SNNs, without imposing any modelling (e.g. requiring PSP ker- nels) or training constraints (e.g. requiring careful regularisation) and support training of neural parameters other than synaptic connectivity (e.g. membrane time constants). We mathemati- cally show how training can be sped up by replacing the slow sequential operations with faster convolutional ones. We experimentally validate this speedup across various numbers of units, time steps, layers and batch sizes, obtaining up to a 16.77x speedup. We show that our model can be trained across different network architectures (e.g. feedforward, hierarchical and convolu- tional) and obtain competitive results different image and neuromorphic datasets. Our results compare well against multi-spike SNNs (< 2% accuracy difference on all datasets) and obtain up to an 81% reduction in spike counts. Furthermore, our method circumvents the dead neuron problem and, for the first time, we show how single-spike SNNs can solve temporally-comples datasets on par with multi-spike SNNs by including trainable membrane time constants. Our findings therefore challenge the dogma that single-spike SNNs are only suited to non-temporal problems (Eshraghian et al., 2021; Zenke al., 2021).
We obtain training speedups on all datasets, however, find that the backward pass slows down relative to the forward pass for longer timespans. Future work could mitigate this bottleneck and accelerate training using sparse gradient descent, which has shown to accelerate the backward pass in standard SNNs by taking advantage of spike sparsity (Perez-Nieves & Goodman, 2021), Currently, our single-spike model performs slightly worse compared to its multi-spike counter- part, where better performance could be achieved by extending our model to the multi- setting and permitting recurrent connectivity. Finally, it remains an open question how the in- clusion of trainable membrane time constants in our model boost performance, requiring further theoretical analysis.
9

---

Under review as conference paper at ICLR 2023
6 REPRODUCIBILITY STATEMENT
The theoretical construction and derivations of our model are outlined in section 3 and provide accompanying derivations in the Appendix. All code is publicly available at wetet//wwbrrm///////////o//// Block under the BSD 3-Clause Licence. This includes instruc- tions on installation, data processing and running experiments reproduce all results and fig- ures portrayed in the paper. Training details are also provided in the Appendix.
REFERENCES
Martín Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Geoffrey Irving, Michael Isard, et al. (TensorFlow]; A system for (Large-Scale] machine learning. In 12th USENIX symposium on operating systems design and implementation (OSDI 16). pp. 265-283, 2016
Guillaume Bellec, Darjan Salaj, Anand Subramoney, Robert Legenstein, and Wolfgang Maass. Long short-term memory and learning-to-len to-learn in networks of spiking neurons. Advances in neural information processing systems, 31, 2018.
Guillaume Bellec, Franz Scherr, Anand Subramoney, Elias Hajek, Darjan Salaj, Robert Legenstein, and Wolfgang Maass. solution to the learning dilemma for recurrent networks of spiking neurons. Nature communications 11(1):1-15, 2020.
Sander M Bohte, Joost N Kok, and Han La Poutre. Error-backpropagation temporally encoded networks of spiking neurons. Neurocomputing, 48(1-4):17-7,,22 2002.
James Bradbury, Roy Frostig, Peter Hawkins, Matthew James Johnson, Chris Leary, Dougal Maclaurin, George Necula, Adam Paszke, Jake VanderPlas, Skye Wanderman-Me Milne, and Qiao Zhang. JAX: composable transformations of Python+NumPy programs, 2018. URL http:
//github. com/ google/ jalje./......
Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. arXiv preprint arXiv:2005. 14165, 2020.
Sharan Chetlur, Cliff Woolley, Philippe Vandermersch, Jonathan Cohen, John Tran, Bryan Catan- zaro, and Evan Shelhamer. cudnn: Efficient primitives for deep learning. arXiv preprint arXiv:1410.0759, 2014.
Julia M Comsa, Krzysztof Potempa, Luca Versari, Thomas Fischbacher, Andrea Gesmundo, and Jyrki Alakuijala. Temporal coding in spiking neural networks with alpha synaptic function. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 8529-8888 8533. IEEE, 2020.
Benjamin Cramer, Yannik Stradmann, Johannes Schemmel, and Friedemann Zenke. The heidel- berg spiking data sets for the systematic evaluation of spiking neural networks. IEEE Transac- tions Neural Networks and Learning Systems, 2020.
Jason K Eshraghian, Max Ward, Emre Neftci, Xinxin Wang, Gregor Lenz, Girish Dwivedi, Mo- hammed Bennamoun, Doo Seok Jeong, and Wei D Lu. Training spiking neural networks using lessons from deep learning. arXiv preprint arXiv:2109. 12894, 2021.
Steve K Esser, Rathinakumar Appuswamy, Paul Merolla, John Arthur, and Dharmendra energy-efficient
Modha. Backpropagation for energy efficient neuromorphic computing, Advances in neural information processing systems, 28, 2015.
Steven Esser, Paul A. Merolla, John V. Arthur, Andrew Cassidy, Rathinakumar Appuswamy, Alexander Andreopoulos, David J. Berg, Jeffrey L. McKinstry, Timothy Melano, Davis R. Barch, Carmelo Nolfo, Pallab Datta, Arnon Amir, Brian Taba, Myron D. Flickner, and Dharmen- dra S. Modha. Convolutional networks for fast, energy-effiiient neuromorphic computing. Proceedings of the National Academy of Sciences, 113(4)):1144111144 2016. doi: 10.1073/ pnas. 1eea1160881111 13.
10

---

Under review conference paper at ICLR 2023
Wei Fang, Zhaofei Yu, Yanqi Chen, Tiejun Huang, Timothée Masquelier, and Yonghong Tian. Deep residual learning in spiking neural networks. Advances Neural Information Process- ing Systems, 34:21056-21068 2021.
Wulfram Gerstner, Werner M Kistler, Richard Naud, and Liam Paninski. Neuronal dynamics: From single neurons networks and models of cognition. Cambridge University Press, 2014.
Tim Gollisch and Markus Meister. Rapid neural coding in the retina with relative spike latencies. science, 319(5866): 108-1111, 2008.
Julian Göltz, Laura Kriener, Andreas Baumbach, Sebastian Billaudelle, Oliver Breitwieser, Ben- jamin Cramer, Dominik Dold, Akos Ferenc Kungl, Walter Senn, Johannes Schemmel, et al. Fast and energy efficient neuromorphic deep learning with first-spike times. Nature machine in- telligence, 3(9):823-835, 2021.
energy-efficient
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 770-778, 2016.
Peter Heil, First latency of auditory neurons revisited. Current opinion neurobiology, 14 (4):461-467, 2004.
Geoffrey Hinton, Deng, Dong Yu, George E. Dahl, Abdel-rahman Mohamed, Navdeep Jaitly, An- drew Senior, Vincent Vanhoucke, Patrick Nguyen, Tara N Sainath, et al. Deep neural networks for acoustic modeling in speech recognition: The shared views of four research groups. IEEE Signal processing magazine, 29(6):82 97, 2012.
Eric Hunsberger. Spiking deep neural networks: Engineered and biological approaches to object recognition. 2018.
Eric Hunsberger and Chris Eliasmith. Spiking deep networks with lif neurons. arXiv preprint arXiv:1510.08829, 2015.
Saeed Reza Kheradpisheh and Timothée Masquelier. Temporal backpropagation for spiking neural networks with one spike per neuron. International Journal of Neural Systems, 30(06); 2050027, 2020.
Saeed Reza Kheradpisheh, Maryam Mirsadeghi, and Timothée Masquelier. Bs4nn: Binarized spiking neural networks with temporal coding and learning. Neural Processing Letters, 54(2): 1255-1273, 2022.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 2014.
Laura Kriener, Julian Göltz, and Mihai A Petrovici. The yan- dataset. In Neuro-Inspired Com- putational Elements Conference, pp. 107-111, 2022.
Yann LeCun. The mnist database of handwritten digits. http://yann. lecun. comtexdb/mnistl, 1998.
Jun Haeng Lee, Tobi Delbruck, and Michael Pfeiffer. Training deep spiking neural networks using backpropagation. Frontiers in neuroscience, 10:508, 2016.
Mingxuan Liang, Jilin Zhang, and Hong Chen. A 1.13 classifiiiiatio spiking neural network accelerator with a single-spike neuron model and sparse weights. In 2021 IEEE International Symposium on Circuits and Systems (ISCAS), pp. 1-5. IEEE, 2021.
Maryam Mirsadeghi, Majid Shalchian, Saeed Reza Kheradpisheh, and Timothée Masquelier. Spike time displacement based enror backpropagation in convolutional spiking neural net- works. arXiv preprint Xii:22108. 13621, 2021.
Maryam Mirsadeghi, Majid Shalchian, and Saeed Reza Kheradpisheh. Ds4nn: Direct training of deep spiking neural networks with single bpike-based temporal coding. 2022.
11

---

Under review as conference paper at ICLR 2023
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-lewe -evell con- trol through deep reinforcement learning. nature, 518(7540):529-53) 2015.
Hesham Mostafa. Supervised learning based on temporal coding in spiking neural networks. IEEE transactions on neural networks and learning systems, 29(7):327-33255, 2017.
Emre Neftci, Hesham Mostafa, and Friedemann Zenke. Surrogate gradient learning spiking neural networks: Bringing the power of gradient-based optimization to spiking neural net- works. IEEE Signal Processing Magazine, 36(6):51-63, 2019.
Peter O'Connor, Daniel Neil, Shih-Chii Liu, Tobi Delbruck, and Michael Pfeiffer. Real-time classi- fication and sensor fusion with spiking deep belief network. Prontiers in neuroscience, 7:1 78178, 2013.
Seongbin Oh, Soochang Lee, Sung Yun Woo, Dongseok Kwon, Jiseong Im, Joon Hwang, Jong- Ho Bae, Byung- Gook Park, and Jong-Ho Ho Lee. Spiking neural networks with time-to snte-to-fie to-first-spike coding using tft-type synaptic device model. IEEE Access, 9.78098-78107, 2021.
Garrick Orchard, Ajinkya Jayawant, Gregory K Cohen, and Nitish Thakor. Converting static image datasets to spiking neuromorphic datasets using saccades. Frontiers in neuroscience, 9:437, 2015.
Priyadarshini Panda, Sai Aparna Aketi, Aketi, and Kaushik Roy. Toward scalable, efficient, and accurate deep spiking neural networks with backward residual connections, stochastic softmax, and hybridization. Frontiers Neuroscience, 14:653, 2020.
Seongsik Park, Seijoon Kim, Byunggook Na, and Sungroh Yoon. T2fsnn: Deep spiking neural net- works with firrt---iirtt-siie coding. In 2020 57th ACMIIEEE Design Automation Conference (DAC), pp. 1-6. IEEE, 2020.
Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. 2017.
Devdhar Patel, Hananel Hazan, Daniel Saunders, Hava T Siegelmann, and Robert Kozma. Im- proved robustness of reinforcement learning policies upon conversion to spiking neuronal network platforms applied atari breakout game. Neural Networks, 120:108-115, 2019.
Nicolas Perez-Nieves and Dan Goodman. Sparse spiking gradient descent. Advances in Neural Information Processing Systems, 34:1 198--18888, 2021.
Nicolas Perez-Niever Vincent CH Leung, Pier Luigi Dragotti, and Dan FM Goodman. Neural heterogeneity promotes robust learning. Nature communications, 12(1):1-9,200 2021.
Bodo Rueckauer, Allia-Alexandra Lungu, Yuhuang Hu, and Michael Pfeiffer. Theory and tools for the conversion analog to spiking convolutional neural networks. arXiv preprint arXiv:1612.04052, 2016.
Bodo Rueckauer, lulia- Alexandra Lungu, Yuhuang Hu, Michael Pfeiffer, and Shih-Chii Liu. Con- version continuous valued deep networks to efficient event-driven networks for image clas- sification. Frontiers in neuroscience, 11:682, 2017.
David E Rumelhart, Geoffrey E Hinton, and Ronald Williams. Learning representations by back- propagating errors. nature, (236689))5333-536 1986.
Roy Schwartz, Jesse Dodge, Noah A Smith, and Oren Etzioni. Green ai. Communications of the ACM, (3(12):54-63, 2020.
Sumit B Shrestha and Garrick Orchard. Slayer: Spike layer error reassignment in time. Advances in neural information processing systems, 31, 2018.
David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, et al. Mastering the game of go without human knowledge. nature, 55047666) 2017.
12

---

Under review conference paper at ICLR 2023
Louis Sokoloff. The metabolism of the central nervous system in vivo. Handbook of Physiology, section, I, NNeurophysiology, 3:1843-64 64, 1960.
Christoph Stöckl and Wolfgang Maass. Recognizing images with at most one spike per neuron. arXiv preprint arXiv:2001.01662, 2019.
Emma Strubell, Ananya Ganesh, and Andrew McCallum. Energy and policy considerations for deep learning in nlp. arXiv preprint arXiv: 022433, 2019.
Guangzhi Tang, Neelesh Kumar, Raymond Yoo, and Konstantinos P Michmizos. Deep reinforce- ment learning with nopulation-codes spiking neural network for continuous control. arXiv preprint arXiv:2010.0963: 2020.
Oriol Vinyals, Igor Babuschkin, Wojciech M Czarnecki, Michaël Mathieu, Andrew Dudzik, Juny- oung Chung, David H Choi, Richard Powell, Timo Ewalds, Petko Georgiev, al. Grandmaster level in starcraft ii using multi agent reinforcement learning. Nature, 5757778):)3755-3 2019.
Timo Wunderlich, Akos Kungl, Eric Müller, Andreas Hartel, Yannik Stradmann, Syed Ahmed Aamir, Andreas Grübl, Arthur Heimbrecht, Korbinian Schreiber, David Stöckel, et al. Demon- strating advantages of neuromorphic computation: a pilotssudyy Frontiers in neuroscience, 13: 260, 2019.
Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnist: novel image dataset for bench- marking machine learning algorithms. arXiv preprint | 2017.
Man Yao, Huanhuan Gao, Guangshe Zhao, Dingheng Wang. Yihan Lin, Zhaoxu Yang, and Guoqi
Li. Temporal-wiss attention spiking neural networks for event streams classification. In Pro- ceedings of the IEEE/CVF International Conference on Computer Vision, pp. 10221-10230, 2021.
Bojian Yin, Federico Corradi, and Sander M Bohté. Effective and efficient computation with multiple-timescale spiking recurrent neural networks. In International Conference on Neu- romorphic Systems 2020, pp. 1-8, 2020.
Friedemann Zenke and Surya Ganguli. Superspike: Supervised learning in multilayer spiking neural networks. Neural computation, 30(6):1514-1511 2018.
Friedemann Zenke and Tim P Vogels. The remarkable robustness of surrogate gradient learning for instilling complex function in spiking neural networks. Neural Computation, 33(4);899- 925, 2021.
Friedemann Zenke, Sander M Bohté, Claudia Clopath, Iulia M Comşa, Julian Göltz, Wolfgang Maass, Timothée Masquelier, Richard Naud, Emre Neftci, Mihai A Petrovici, et al. Visualizing joint future of neuroscience and neuromorphic engineering. Neuron, 109(4):571-575, 2021.
Malu Zhang, Jiadong Wang, Jibin Wu, Ammar Belatreche, Burin Amornpaisannon, Zhixuan Zhang, Venkata Pavan Kumar Miriyala, Hong Qu, Yansong Chua, Trevor E Carlson, et al. Rec- tified linear postsynaptic potential function for backpropagation in deep spiking neural net- works. JEEE Transactions on Neural Networks and Learning Systems, 33(5)119471198,, 2021.
Shibo Zhou and Xiaohua Li. Spiking neural networks with dingle-spike temporal-codee neurons for network intrusion detection. In 2020 25th International Conference on Pattern Recognition (ICPR), pp. 8148 8155. IEEE, 2021.
Shibo Zhou, Xiaohua Li, Ying Chen, Sanjeev T Chandrasekaran, and Arindam Sanyal. Temporal- coded deep spiking neural network with easy training and robust performance. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pp. 11143-11151, 2021.
13

---

Under review conference paper at ICLR 2023
A APPENDIX
A.1 SPIKING NEURAL NETWORK DERIVATIONS
A.1.1 NORMALISING THE LEAKY INTEGRATE AND FIRE MODEL
Proposition 2. Any leaky integrate and fire (LIF) model dV(c) - ress RI(D) (with mem- dt brane potential V, resting potential Vrest, firing threshold Vth. resistance R, input current and membrane time constant T) can be normalised to LIF model of the form
dV (t) + I(t)
dt (such that 0 = (()) = 1, with firing threshold Vih = 1, resting potential Vrest = 0 and a resistance equal to one).
Proof. This mapping from any LIF model to the normalised LIF model is achieved using the fol- lowing transformation (taken from Hunsberger (2018)).
V(t) Vrest
V(t)
(9)
Vth Vrest
Rearranging this expression with respect to V(t) V(D(Vnn Vrest) + Vrest and substituting this into the LIF model we obtain
dt dV(1) V(t) + Vrest RI(()
dv(t) = (V(OVVEn - Vrest) + Vrest) + Vrest + RI(()
(Vth - Vrest)
dt
(10)
dv(r)
R I(t)
=V((++ Vhh Vrest
dt
Input current 7(1)
This new LIF form has resting potential Vrest 0 and firing threshold V, =1 (obtained by sub- stituting V(t)= Vrest and V(r) V₁x in Equation 9 respectively). Thus, without loss of generality, any LIF model can be mapped to normalised form using linear transformation Equation 9. O
A.1.2 DISCRETISING THE LEAKY INTEGRATE AND IRE MODEL
Proposition 3. The normalised continuous time leaky integrate and fire model dV(() dt
+
I(r) (with membrane potential V, input current and membrane time constantr) can numerically be approximated by discrete time difference equation V(t+ 1] [V[[]] (1 [B)[[t+ 1}, where f e (for simulation time resolution At).
Proof. We proceed using the forward Euler method. Let ((1) = / be constant with respect to time, for which the ordinary differential equation becomes separable.
div(A) dt +1
1 V(0-I dt In(V(t))) = - + In(k)
dV
(11)
V(t) = kexp(- -)
14

---

Under review conference paper at ICLR 2023
For initial solution ((4) time to we derive == (V( Then for constant and initial solution V(t) we obtain solution.
V(t) (V(t)) exxp( [-10 10
I
(12)
-- 10
((()) (1-exp(-- 1- -
= exp(
To obtain the discretised update equation, define simulation update time step Ar fo- decay factor & = exp(- assign continuous time points to discretised time steps - to and 1+1 to At and assume the input current be approximately constant and equal to [[r 1]
between discretised update steps / to 1+
Vi++ V1=+1)=-V10+0--1111++
(13)
A.1.3 UNROLLING THE LEAKY INTEGRATE AND MODEL WITHOUT THE
Proposition 4. Equation V(rl ('V (1 B) Σ B I(i) is equivalent to difference equation
Vir BV (1/(t) for 1.
Proof. We proceed to proof equivalence by induction. For / = we obtain
V[1] = V10 (1 f) Σ 11i
1=1
(14)
B v10j + [1)
Hence the relation holds true for the base case Assume the relation holds true for k 1, then for = k+ 1 we derive
Vík + BVIKI + +1-p)/(k+1
= =([++v10++11 (1- Σ
BK
[[il]] + +1--)/[k+]]
1=1
(15)
= (1 B) Σ Bik+ (1-
v10)+ p)
Σ B
This implies equivalence for f = lf k holds true. By the principle of induction, equivalence established given that both the base case and inductive step hold true.
A.2 ADDITIONAL MODEL THEORY: WHY IS OUR MODEL FASTER?
To address the question why our model is faster than the standard single-spike model, we analyse their respective computational complexities. Consider a single neuron with N presynaptic neu- rons simulated for T time steps. Our model has a computational complexity of O(NT²) and the computational complexity of the standard model is O(NT). However, the sequential complexity of our model is constant time O(1) (as our model eschews all sequential dependence) and the se- quential complexity of the standard model is linear O(T). Our model performs more calculations than the standard single-spike model, yet is able to obtain faster training speeds, as unlike the standard single-spike model all these calculations highly parallelisable.
15

---

Under review conference paper at ICLR 2023
A.3
ADDITIONAL DATASET DETAILS
A.3.1 SYNTHETIC SPIKE DATASET FOR THE SPEED BENCHMARKS
We generated binary spike tensors of shape Bx Nx (B being the batch size, N the number of input neurons and 7' the number of simulation steps). For every batch dimension b firing rate ты mmin mmax) was uniformly sampled (with Umin OHz and Umax 200Hz), from which random binary spike matrix of shape N T' was constructed, such that every input neuron in this matrix had an expected firing rate Hz.
A.3.2 TIME-TON TO - IRSS-SSRE SPIKE ENCODING
We encoded all analog non-spiking input data into spike raster using the time-to-first-ppe coding method (Kheradpisheh & Masquelier, 2020). Here, every scalar value 1; E {0, mmxx within an input tensor is converted into a spike train with single spike, where the time of spike (0, 7) is determined by the following equation
mmx
1
(16)
Imx
A.4 TRAINING DETAILS AND IYPERPARAMETERS
A.4.1 READOUT NEURONS
The output layer L of every trained network contained the same number of neurons as the num- ber of classes contained within the dataset being trained on. As suggested by Zenke & Vogels (2021), every neuron had firing threshold set to infinity (i.e. the spiking and reset mechanism was removed) from which the output Ob,c of readout neuron to input sample was either taken to be the maximum membrane potential over time 0b,c max, V. [t] or the summated mem- brane potential over time Ob.c E, VL. [r] (see table 2).
A.4.2 BETA CLIPPING
As the beta (h (a transformation of the membrane time constant) of every neuron was opti- mised, we had enforce correct neuron dynamics by clipping the values into the range [0, 1]. Note that TTTh = implies no memory i.e. a binary neuron, < g < implies decaying memory ie. LIF neuron and p implies full memory i.e. an IF neuron.
1, if 8 >
B0Th =
(17)
0, if p <0
A.4.3 WEIGHT INITIALISATION
The network weights in a layer were sampled from a uniform distribution U(-v U(-VN-I, VN-1), ex- cept for the Yin- Yang dataset for which the weights were sampled from U(-V2N-T V2N- VV-N-)-T.V2N-1, For the feedforward layers was set to the number of afferent connections to the layer and for the convolutional layers N = 22 for kernel shape k k. The bias terms were initialised to 0 in all net- works. All neurons in the hidden layers were initialised with a membrane time constant 7 10ms and r 20ms for the readout neurons.
A.4.4 SUPERVISED TRAINING LOSS
All networks were trained to minimise a cross-entropy loss
1
L
yb, loglobbb))
(18)
B
1=
c=1
16

---

Under review conference paper at ICLR 2023
with B and C being the number of batch samples and dataset classes respectively, and Vb.c 10, ll and Pb.c being the one hot target vector and network prediction probabilities respec-
tively. The prediction probabilities Pb.c were obtained by passing the readout neuron outputs Obe through the softmax function.
exp Ob.c
Pb.c
(19)
Σ exp 0b.k
A.4.5 SURROGATE GRADIENT
The backprop algorithm requires all nodes within the computational graph of optimisation to be differentiable. This requirements is however violated in a SNN due to the nonfdifferentiable heavy stepwise spike function f. To permit the use of backprop, df derivate 2V of the function f with surrogate gradient dv (Zenke & Ganguli, 2018), which has been shown to work well in practice (Zenke & Vogels, 2021). Here hyperparameter Paur (which we set to 10 in all experiments) defines the slope of the gradient.
replaced the undefined
d'
f (V)
(Bsurl VI + 1)
(20)
dv
A.4.6 TRAINING PROCEDURE
All models were trained using the Adam optimiser (with default parameters) (Kingma & Ba, 2014). Training started with an initial learning rate, which was decayed by factor of 10 every time the number of epochs reached a new milestone, after which the best performing model (that achieved lowest training loss) was loaded and training continued.
A.4.7 TRAINING HYPERPARAMETERS
Table 2: Dataset and corresponding training parameters.
Yin- Yin-Yang
MNIST
conv MNIST
F-MNIST
conv F-MNIST
N-MNIST
SHD
(train/test)
20k/10k
60k/10k
60k/10k
60k/10k
60k/10k
60k/10k
8156/2264
Input neurons
784
28 28
784
28 x 28
1156
700
Dutaset classes
Epochs
200
140
140
140
140
200
200
Learning rate
0.001
0.001
0.001
0.001
0.0002
Batch size B
128
128
128
64
128
128
Simulation steps T
100
100
100
300
500
resolution (ms)
Milestones
(50, 100)
(15,90, 120)
(30, 60, 90)
(15,90, 120)
(30,60,900 60)90)
(30, 60, 90)
(30, 60,90)
Output function
sum
sum
A.5 EXTENDED SPEEDUP RESULTS
17

---

Under review conference paper at ICLR 2023
Training speedup
Forword speedup
Training speedup
Forward speedup
900
000
800
8.00x
Hidden
speedup
8.00x
speedup
4.00x
400
4.00x
400
2,00
log, Simulation steps
log, Simulation stops
Figure Total training speedup using smaller batch sizes as a function of the number of hidden neurons n and simulation steps r (left), alongside the corresponding forward and backward pass speedups (right). a. Speedups using batch size b 32. b. Speedups using batch size b 64.
Training duration 1000
Training speedup 128)
Forward upoodup
b
1000
19.50x
105 Durae Auration (sec)
Model
FastSNN
Hidden units
300
8.00
Standard
200
Backward speedup
4.00x
-1
2.00x
300
1.00x
100
0.4.X
10
12
4 5
10:11
log: Simulation steps
log, Simulation steps
Training speedup
Training speedup (t=
Forward speedup (n 1000)
Fwd. bkwd. (n 1000)
2t= and b 128)
d
12.5
ng
200
32
10.0
400
64
Speedup
7.5
10
5.0
+1 Ahd & w.t.-&ootooooooo...o pig
Layers
log, Simulatior steps
logo Simulation steps
Hidden units
Figure 6: Training speedup of our model over the standard model (using fixed membrane time constants). Total training speedup as function of the number of hidden neurons n and sim- ulation steps (left), alongside the corresponding forward and backward pass speedups (right). b. Training durations of both models for fixed hidden neurons n = 100 and variable batch size b. Training speedup over different number of layers for fixed time steps t= 2 and batch size b = 128. d. Training speedup over large number of hidden neurons n for fixed time steps = 2 and variable batch size b. Forward pass speedup for fixed time steps t 27 and variable batch size b. Forward vs the backward pass speedup of our model for fixed time steps f = 2 and variable batch size b. b-f use 10 sample average with the mean and s.d. plotted.
18

---

Under review as conference paper at ICLR 2023
8
Speedup x
6
Yin-ang
MNIST
F-MNIST
N-MNIST
SHD
Figure 7: Training speedup of our model vs the standard multi-spike model across different datasets (using a 3 sample average with the mean and s.d. plotted)
19


Reviews:
[{"review_id": "xqZQytH5ayV", "reviewer": "Reviewer_5huF", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presented an interesting idea for accelerating the training procedure of SNNs. The proposed SNN is considered to reduce consumption by enabling SNNs to maintain at most one firing spike. \n\nThe organization of this paper is clear, and citations are also appropriate. The experiments conducted on various datasets verify the effectiveness of the proposed method.\n\nMajor Issuses:\n1. As claimed in this paper, the single-spike model can speed up the training procedure of SNNs due to a decreasing number of firing spikes. Thus, it is natural to verify the computational complexity of the single-spike and multi-spike models, instead of running time. In other words, the running time is not sufficient, and the computational complexity is very likely to be the same order of magnitude. \n2. As shown in this paper, when the authors reduce the number of firing spikes, the single-spike neural network not only has a faster training procedure, but also performs other comparative models, even avoiding the problem of \"dead neurons\". According to Occam's Razor, what is the price of superior performance? It is significant to be discussed in this work.", "summary_of_the_review": "Overall, I believe this is an interesting paper that focuses on an energy saving way of SNNs. However, I still find that there is something has been hidden by the authors, which urgently need to be analyzed and discussed in depth. Therefore, I tend to accept this paper if the authors fixed these issues in the next phase.", "strength_and_weaknesses": "mentioned above.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "rY4cX4UVo9", "reviewer": "Reviewer_cHb7", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper unrolls the update formula of the LIF model along the time axis and uses the vectorized variables for parallel acceleration. It proposes a simple method to determine when the first spikes are fired. In this manner, both the goal of speeding up training and lowering the spike counts are accomplished.", "summary_of_the_review": "Overall, this paper proposes an interesting idea to speed up the training SNN  through parallel acceleration. Based on the description, I think the acceleration is real but the deficit of representation power of large models remain concerned as well as its advantage of the quantized model. ", "strength_and_weaknesses": "Strength：\n1. This material is generally well-presented and easy to read.\n2. It’s a good idea to speed up the SNN training using parallel acceleration.\n\nWeakness:\n1. When comparing accuracy, acceleration, and spike reduction, specific corresponding baseline models must be provided.\n2. The simulation time steps in the experiments are usually higher than the required value. E.g., MNIST datasets may only need 8 simulation time steps, the same as conv MNIST. When the simulation time steps are small, the training acceleration of this method in the paper is not so significant as the authors claim.\n3. The article's novelty and contribution are modest, and it offers few enthralling or motivating insights.\n\nQuestions:\n1. Which part of the paper's algorithm is robust enough to declare in the title?\n2. In Table 1, most of the experimental results are worse than the existing methods, especially on more complex convolutional networks. So, I doubt whether the authors' method can be implemented into mainstream deeper convolutional to take full advantage of the training efficiency.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "i1mLmsbYBKI", "reviewer": "Reviewer_1hwt", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors proposed a new modal for training single-spike SNNs, which offers a 13.98x training speedup compared to a multi-spike counterpart. The speedup is achieved by avoiding all sequential dependence on time and exclusively relies on GPU parallelizable non-sequential operations. The effectiveness of the proposed method is reflected by five different datasets, performing on par with multi-spiking counterparts. ", "summary_of_the_review": "To me, the valuable part of SNNs is their bio-plausibility, especially in the temporal domain. Single-spike itself does not make sense to me, and the authors did not show the quantized experimental results on energy saving. Thus, the motivation for this work is unclear to me. In addition, the current neuromorphic hardware can barely mimic the simplest biological neuron. The fundamental research in the SNN domain should not be limited or favor current neuromorphic hardware. Instead, fundamental research should lead the development of neuromorphic hardware. \n\nAt the moment, I would not recommend this work. But I would like to see the authors' response.", "strength_and_weaknesses": "Strength:\n+1. The submission is well written. \n\nWeaknesses:\n-1. Single spike SNN itself does not make sense to me, which is against the bio-plausibility of SNN. Essentially, the single-spike scheme dramatically impacts the effectiveness of SNN's temporal information. The authors also pointed it out in the section \"Single-spike neurons solve challenging temporal problems using neural heterogeneity\" and remediated the problem by learning time constants. But, the accuracy is much worse than Perez-Nieves'. \n\n-2. The only motivation for using single-spike SNN is to enhance energy efficiency. But, the authors did not show any quantized experimental results to support the motivation. How much energy could it save on neuromorphic hardware?\n\n-3. For LIF, we normally set the reset potential to 0. Is \"calculate membrane potentials without reset\" still valuable if that is the case? \n\n-4. The motivation for training speedup is unclear to me. Yes, the 17x looks awesome. But how long does it take to train a multi-spike SNN model for the tested tasks? \n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "Xd1Wo3uaGpF", "reviewer": "Reviewer_HU9D", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors use spiking neural networks for image classification. They use time-to-first-spike (TTFS) coding. After a neuron has fired, any eventual subsequent spike is blocked. Training is done with a surrogate gradient.", "summary_of_the_review": "There is a new method for training TTFS-SNNs, but it's unclear what the advantages are w.r.t. existing methods.\n\n", "strength_and_weaknesses": "STRENGTHS:\n\nTTFS is appealing because it consumes few spikes, therefore little energy\n\nWEAKNESSES:\n\nThe authors limit themselves to extremely simple toy datasets (not even CIFAR!), and even on these datasets, the method is not competitive w.r.t. other proposals (see Table 1).\nZhou et al AAAI 2021 should be included in Table 1. They have decent results on CIFAR, and even on ImageNet, using TTFS!\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject