Accelerating spiking neural network training using the $d$-block model
Authors: Luke Taylor
Andrew J King
Nicol Spencer Harper
Venue: Submitted to ICLR 2023 (2023)
Keywords: spiking neural networks
accelerated training
stochastic refractory period
stochastic recurrent conductance latency

Abstract: There is a growing interest in using spiking neural networks (SNNs) to study the brain \textit{in silico} and in emulating them on neuromorphic computers due to their lower energy consumption compared to artificial neural networks (ANNs). Significant progress has been made in directly training SNNs to perform on a par with ANNs in terms of accuracy. However, these methods are slow due to their sequential nature and require careful network regularisation to avoid overfitting. We propose a new SNN model, the $d$-block model, with stochastic absolute refractory periods and recurrent conductance latencies, which reduces the number of sequential computations using fast vectorised operations. Our model obtains accelerated training speeds and state-of-the-art performance across various neuromorphic datasets without the need for any regularisation and using fewer spikes compared to standard SNNs.

Under review as conference paper at ICLR 2023
ACCELERATING SPIKING NEURAL NETWORK TRAINING USING THE d-BLOCK MODEL
Anonymous authors Paper under double-blind review
ABSTRACT
There a growing interest in using spiking neural networks (SNNs) to study the brain in silico and in emulating them on neuromorphic computers due their lower energy consumption compared to artificial neural networks (ANNs). Sig- nificant progress has been made in directly training SNNs to perform on a par with ANNs in terms of accuracy. However, these methods are slow due to their sequential nature and require careful network regularisation avoid overfit- ting. We propose new SNN model, the d-block model, with stochastic ab- solute refractory periods and recurrent conductance latencies, which reduces the number of sequential computations using fast vectorised operations. Our model obtains accelerated training speeds and state-of-the-an performance across various neuromorphic datasets without the need for any regularisation and using fewer spikes compared to standard SNNs.
INTRODUCTION
Artificial neural Networks (ANNs) are ubiquitous in achieving state-of-the-an art performance across various domains, such image recognition (He et al., 2016), natural language process- ing (NLP) (Brown et al., 2020) and computer games (Silver et al., 2017; Vinyals et al., 2019). These networks have also proven useful for studying the brain due their architectural similarities (Richards et al., 2019) and have further advanced our understanding of the computational pro- cesses underlying the visual and auditory system (Harper et al., 2016; Singer et al., 2018; Cadena et al., 2019; Francl & McDermott, 2022; Yamins & DiCarlo, 2016). However, ANNs have been criti- cised for their substantial energy demands resulting from their continued exponential growth in size (Strubell et al., 2019; Schwartz et al., 2020), as exemplified by the GPT language models scal- ing from 110 million to 1.5 billion to 175 billion parameters to deliver wer-improving advances across various NLP tasks (Radford et al., 2018; 2019; Brown et al., 2020). Furthermore, the appli- cability of ANNs to neuroscience is confined by their activation function, as the brain employs spikes rather than continuous cuntinuous-valued outputs used by ANN units.
Spiking neural networks (SNNs) are a type of binary neural network (Figure la), which over- come these challenges they consume drastically less energy than ANNs when deployed neuromorphic hardware (Wunderlich et al., 2019) and their biological realism makes them a favourable model for studying the brain in silico (Vogels et al., 2011; Denève & Machens, 2016). However, SNN training remains a challenging problem due to the non-differentiable binary ac- tivation function employed by the spiking neurons. This has historically resulted in solutions that impose constraints on the neurons, such as rate codes (O'Conna et al., 2013; Esser et al., 2015; Rueckauer et al., 2016), or only allowing neurons to spike at most once (Bohte al., 2002; Mostafa, 2017; 7; Comsa et al., 2020).
recent proposal known as surrogate gradient training can overcome these limitations and has been shown to improve training on challenging datasets using models of increasing biological realism (Eshraghian et al., 2021). Surrogate gradient training replaces the undefined derivate of the neuron's activation function with a surrogate function and uses backpropagation through time (BPPT) for training, since SNNs are a particular form recurrent neural network (RNN) (Neftci et al., 2019). SNNs thus experience many shortcomings associated with RNNs, such as their notably slow training times resulting from their sequential nature (Kuchaiev & Ginsburg, 2017; Vaswani et al., 2017). Furthermore, SNNs require multiple regularisation terms to avoid

---

Under review as conference paper at ICLR 2023
a
b
5
2-
Time (ms)
Figure I: a. Left: The standard leaky integrate and fire (LIF) model. Right: Input and output activity of the neuron (bottom panel: Input raster, lower middle panel: Input current 1, upper middle panel: Membrane potential with the dotted line representing the firing threshold, and top panel: Output spikes by vertical dotted red lines). Right: Input and output activity of the neuron (same as the LIF model but with a change in the upper middle panel, where the d-block computes membrane potentials without reset V).
S). b. Left: Our d'-lock model (here d 3 blocks with edges denoted
overfitting and to obtain sparse spiking activity (Zenke & Vogels, 2021; Perez- Nieves Goodman, 2021), where the latter is important in obtaining energy-efficient computations neuromorphic hardware (Panda et al., 2020).
In this work, we address these shortcomings by proposing new SNN model with stochastic ab- solute refractory periods and recurrent conductance latencies. The absolute refractory period is the brief period after spike during which neuron cannot spike again; it is feature of the biology not typically incorporated into standard SNNs. In recent study, Taylor et al. (2022) pro- posed model which refer neurons spike at most once. We extend this model to being multi-spike and recurrent, by recur- rently connecting and concatenating these blocks across time, and hence refer our model as the d--locc model, as each neuron can spike at most d times when employing d blocks (Figure 1b). Our contributions are summarised as follows.
block for accelerated training of SNNS in which individual
We propose a new SNN model which uses fewer sequential operations compared to con- ventional SNNs and is henceforth more parallelisable on GPUs. We experimentally val- idate our model to obtain accelerated training speeds on synthetic benchmarks (up to 34x speedup) and various neuromorphic datasets (achieving up 9x speedup).
2. Our model theoretically consumes less energy than conventional SNNs when em- ployed neuromorphic hardware, as it emits fewer spikes than standard SNNs dur- ing inference across multiple neuromorphic datasets. Furthermore, does not include any training regularisation, thus avoiding the need for time-consuming and energy- demanding hyperparameter searches to find suitable regularisation values.
3. Notably we obtain late-of-theeart results on challenging neuromorphic datasets with an 86.20% and 68.16% accuracy on the Spiking Heidelberg Dataset (SHD) and Spiking Speech Commands (SSC) dataset respectively, raising performance by and 8% over previously published results using standard SNNs.
BACKGROUND AND RELATED WORK
2.1 STANDARD SPIKING MODEL
A spiking neural network (SNN) a type of recurrent neural network in which neurons output binary signals known as spikes. A neuron i in layer (consisting of R neurons) emits spike (z)) s(O [x) =fv;"") (t)) (0, at time if its membrane potential v.!0 [r] reaches firing threshold Vih-
1, if v [x] Vn 0. otherwise
f(v
2

---

Under review as conference paper at ICLR 2023
The leaky integrate and fire (LIF) model describes the evolution of the membrane potential (1) for resting potential Vrest R, membrane time constant Te R and input resistance R R
(Gerstner et al., 2014).
av dt - -- Vress R1 (t)
(2)
Without loss of generality the model is normalised (V,40() (r) [0,1] by Vrest =0, V/h discretised (Taylor et al., 2022), such that for simulation time steps I E (1,..., T}:
1,R 1) and
NU-D
Mi
v 1= 14 B) (b + Σ
[r+ 11 wilh sl- [
Σ F=1
(1
of
s/D [d]
(3)
Input current 1.1h [e((++1]
Spike reset
The membrane potential is charged by the constant bias current source 6 and the input 5(t-D(xx N and output spikes 5 [r- 1] E N through feedforward w R xN and recurrent connectivity Wree (1) E R ( x respectively. At every time step the membrane po- = (for simulation time resolution Ar E R). The membrane potential is at rest Vrest 0 in the absence of any input current and spike [t] = outputted the potential rises above the firing threshold Vnn = (after which the potential is reduced back close to resting state).
tential dissipates by a factor 0 = β = exp(
2.2 TRAINING TECHNIQUES
To date, SNN training remains challenging. The success of backpropagation (Rumelhart al., 1986) in ANNs does not naturally translate to SNNs due to their non- differentiable nature. Vari- ous methods have been proposed circumvent this issue, however, these either fail to properly utilise the temporal dynamics of neurons or result in slow training times.
Shadow training: ANN to SNN conversion Instead of training a SNN, shadow training converts an already trained ANN to a SNN such that the firing rates of the SNN neurons approximate the activations of the ANN units (O' (O'Connor et al., 2013; Esser et al., 2015; Rueckauer et al., 2016; 2017). Although this method permits deep SNNs to perform well on challenging large-scale datasets like Imagenet (Deng et al., 2009), it endures various shortcomings. Firstly, it requires long simulation durations to obtain reasonable prediction accuracies, which have been argued to largely diminish the energy efficiencies of SNNs emulated on neuromorphic hardware (David- son & Furber, 2021). Secondly, converted SNNs perform worse than ANNs due to the conversion process. Thirdly, although this performance gap can be reduced by coupling ANN and SNN train- ing (Wu et al., 2021a;b, Kheradpisheh et al., 2021), the modelling applicability to neuroscience is limited due to the imposed rate-code (whereas the brain might instead process stimuli using temporal code (Guyonneau et al., 2004; Cariani, 1999)).
Direct SNN training Although the standard use of backpropagation is prohibited, alternative approaches have been proposed to estimate the gradients of network weights. Perturbation learning randomly perturbs network weights to approximate gradients, yet this method requires many trials average noise and scales poorly with growing network architectures (Williams, 1992; Seung, 2003). Latency learning calculates the gradients at the time of spiking as, unlike spikes, time is continuous (Bohte et al., 2002; Mostafa, 2017; Comsa al., 2020; Kheradpisheh Masquelier, 2020). However, this method enforces the modelling constraint that neurons spike at most once and affected by the dead neuron problem, where lack of spike activity is detri- mental to learning. Surrogate gradient learning circumvents these issues by replacing the unde- fined derivative the spike function with a surrogate function (Esser et al., 2016; Hunsberger Eliasmith, 2015; Zenke & Ganguli, 2018; Lee et al., 2016). This method allows networks to learn temporal dynamics since it passes gradients through time (Bellec et al., 2018; Shrestha & Orchard, 2018; Neftci et al., 2019), and has thus become the status quo for directly training SNNs. How- ever, learning is very slow as the network is sequentially simulated at every point in time (which overcome by processing multiple time steps once).
3

---

Under review conference paper at ICLR 2023
a
b
s [(10
st00
5000
500
[p]
(
block-l
block-2
Block-
(1.3)pp Pal
mr (.(2)) p3
(1.33) S
( p3
(1.3)pp
(1.3)
[2)
p x
p-01-) 2|
p-
s10
st0)
[00gg]
8 510pp
g200
5t00
[00
5/10
s s:00pr
s js]
5 (g]
Figure 2: Computational graphs. Standard model. The membrane potentials [[[r] are recur- rently dependent on the prior values [t-] and charged by input spikes S [r] and output spikes S [t] through feedforward and recurrent weights (denoted by dotted red lines) respec- tively. Output spikes are generated by passing the membrane potentials (( through the spike activation function. b. Our d-block model. Input spikes are processed by d (here d equal length blocks, where every block is a single-spike SNN (where we have adopted the fast Taylor et al. (2022) model). Within layer and block n, input spikes (((-)(r) charge the membrane po- tentials without reset V/t.ml [r], which are mapped to spike outputs 5 [r] using a fast grouping of operations m( [r] (see section 3.1). Recurrent connections (dotted red lines) transport output spikes from one block to another.
Accelerated surrogate gradient training Some work has addressed the slow training times of SNNs using surrogate gradients. Perez-Nieves Goodman (2021) managed obtain faster train- ing speeds by developing sparse backprop algorithm, which only passes gradients through the membrane potentials that are close to the firing threshold. Although they manage acceler- ate the backward pass up to 70x, these speedups depend on the development of custom CUDA kernels, which only support simple feedforward architectures (whereas we accelerate training of SNNs with recurrent connectivity and trainable neural time constants). Other works have man- aged to speed up training by removing the backward pass and performing all learning online (Bellec et al., 2020; Murray, 2019). These approaches, however, result in inferior performance accuracies on various datasets in comparison to standard surrogate gradient descent. Lastly, a core limitation of all of these speedup approaches is the prerequisite of sequentially simulating the network at every point in time (whereas we lessen this bottleneck).
FASTER TRAINING WITH THE d- d-BLOCK MODEL
We propose anew and fast algorithmic implementation for SNN called dl bloblock- with stochas- tic absolute refractory periods and stochastic recurrent conductance latencies. We develop our model by extending the accelerated single-spike SNN by Taylor et al. (2022) from being single- spike and feedforward, to being multi-spike and recurrent.
3.1 THE SINGLE-SPIKE BLOCK MODEL
In the standard SNN, the input spikes 5 [t] charge the membrane potentials (0) and out- put spikes S [() are emitted at every point in time (Figure 2a). This sequential processing is however slow. In a recent study Taylor et al. (2022) proposed a faster SNN model with the as- sumption of single-spike outputs and feedforward connectivity which avoids this sequential computation by instead computing all outpur spikes [(") simultaneously across time. This model, which we refer to block (orange box in Figure 2b), quickly transforms input spikes to output spikes using three steps:
1. Time series of presynaptic spikes 5(1-1) are mapped to time series of input currents
((() using tensor multiplication 100 [x] = Nil-D wilh sil [[]].
Hold face variables demotes arrays opposed scalar values.
4

---

Under review as conference paper at ICLR 2023
2. Instead of computing membrane potentials (r) (as done in the standard model), the block calculates membrane potentials without reset [r) (by excluding the reset term
in Equation 3), which are calculated using fast convolution 1 [t] pi [0] + (1
(1 [+] (where 3 "'"""".
3. Correct output spikes / are obtained using the following operations (which we group m in Figure 2). (1) Erroneous output spikes 5/0 are obtained by passing the no-reset membrane potentials V.D through the spike function f. (2) These erroneous output spikes are transformed into latent encoding x - [k][t 1), where every el- ement therein encodes an ordering of the spikes. (3) Correct output spikes are obtained
through transformation s g(z,"), where
I, if 11
g g(2())l [(
(4)
0. otherwise
3.2 EXTENDING THE SINGLE-SPIKE BLOCK TO MULTI-SPIKE BLOCKS
The block model only emits single spike over the simulation period T. To extend the model to use multiple spikes, we divide the simulation period T into d chunks of equal length T/d and process each chunk by block using the same neural parameters (i.e. weight sharing the synaptic connectivity and membrane time constants; Figure 2b). If the input data is not divisible by d, we split the data into d chunks of length [T/d] and final chunk of length T d LT/d]. In this multi block extension, which we refer to as the d d-block model, neurons can spike up to d times over the simulation period Every neuron is governed by stochastic absolute refractory period as a neuron can spike at most once within a block of length T/d and will therefore need to wait between and T time steps before it can spike again the next block. From hereon forth we index variables of layer in block with superscript (1, n) (e.g. the no-reset membrane potential value at time of neuron i in layer and block is denoted as ((l.a) (t)).
3.3 INCLUDING RECURRENT CONNECTIONS
The block model does not include any recurrent connectivity. Now that we have extended the model to be multi-spike, we can include recurrent connectivity by connecting the output spikes of block-n as input to the next block-(n + 1) (see red lines in Figure 2). We achieve this by cre- ating a binary vector of spike states (i.e. spike or no spike) for each neuron within block and feeding these as input to the same neurons in the next block. Specifically, the output spikes g(l.n) vx x in layer and block n are flattened along time max, 5(1.n) [x] e R
(to denote the neurons have spiked within the block's duration) before transporting them to the next block n+1 1 via the recurrent connectivity Wree (f) 2 Different from the standard SNN, these connections have a variable conduction latency equivalent in duration to the refractory periods of the afferent neurons in layer and block-n. n. The starting (no-reset) membrane potential of neuron in layer / and block- (n + 1) is then computed from the flattened output spikes of block-1 n and the first input spikes to block-(n 1).
NU-D
N0 W ee
D(l [1] (b00
Σ
w!lh s
s((-I,
[1)
max max ((l.)
(5)
Feedforward current
Recurrent current
3.4 TRAINING THE D-BLOCK MODEL
The d-block model is trained using backpropogation with surrogate gradients (Neftci et al., 2019), where we replace the undefined derivative of the spike function (Equation 1) with a fast sigmoid
function dV dv
(Bsur VI 1) (Zenke Ganguli, 2018), which has been shown work well
For simplicity we have omitted the batch dimension, which is included in our implementation and used in our experiments.
5

---

Under review conference paper at ICLR 2023
a (Feedforward) Training speedup (n 100 and 128) (Feedforward) Training speedup tt= 512 and 128)
b
25.0
16.00x
25.0
20.0
20.0
4,00x
2,00x
15.0
104
1,00x
10.0
0.40X
100
200
d
(Recurrent) Training speedup 100 and 128)
(Recurrent) Training speedup (t= 512 and 128)
25.0
16.00x
25.0
blocks
20.0
8.00x
20.0
4.00x
2.00x
106
1.00x
10.0
0.49x
8.455
100
logz Simulation steps
Hidden units /
Figure 3: Training speedup of our d-block model over the standard SNN for feedforward and recurrent networks. a. Feedforward network training speedup as a function of the number of blocks d and simulation steps r (for fixed hidden neurons n = 100 and batch size b = 128). b. Feedforward network training speedup
function of the number of blocks d and hidden neu- rons n (for fixed simulation steps r = 512 and batch size = 128). c. Same as a. but for recurrent networks. d. Same b. but for recurrent networks.
in practice (Zenke & Vogels, 2021) (here hyperparameter Psur defines the slope of the gradient). Exact training details for generating predictions, the loss function, optimisation procedures and hyperparameters can be found in the Appendix.
3.5 THEORETICAL SPEEDUP AND PERFORMANCE ADVANTAGES
Our model is more parallelisable than the standard SNN model. The computational complex- ity O(NT) of single neuron within the standard model is smaller than that of our model O(dN[T/d]²}' for presynaptic neurons, simulation time steps and number of blocks. However, the sequential complexity of our model is O(d) in comparison to the O(T) complexity of the standard model. Thus, choosing d smaller than and using GPUs designed to execute many operations in parallel leads theoretical faster training speeds within model.
We hypothesise the design our model to be more robust to overfitting as depending on the chosen number of blocks d our model emits less spikes, thus acting form implicit regular- isation on the spike code. In addition, the stochastic absolute refractory periods and stochastic recurrent conductance latencies act form of noise regularisation. Lastly, our model obtains more salient flow of gradient through time due to the recurrent conductance latencies acting as form skip connection (Srivastava al., 2015; He al., 2016), but through time.
EXPERIMENTS AND RESULTS
We evaluate the training speedup and performance of model on real datasets in comparison to prior work. We used PyTorch (Paszke et al., 2017) for all implementations with benchmarks and training conducted cluster of NVIDIA Tesla V100 GPUs.
4.1 SPEEDUP BENCHMARKS
We benchmarked the time required to complete a single surrogate gradient training step in our d- block model and the standard SNN for different number of blocks d, hidden units n, simulation steps t and batch sizes b on synthetic spike dataset (see Appendix).
The computational complexity ([dN[Tid]) of our model comes from simulating d blocks of length [T/d], where kernel length [7'd] is convolved each of the presynaptic spike trains.

---

Under review as conference paper at ICLR 2023
Robust speedup for different number of simulation steps Considering only feedforward SNNs (Figure 3a-b), we find our d-block model to train faster for a growing number of simu- lation steps, with maximum training speedup 34 for d 15 blocks and t 2048 simulation steps in a single hidden layer model of 100 feedforward neurons (Figure 3a). These speedups robust over different numbers of blocks (e.g- obtain speedups of 21 x, 21x and 18x using d= 10, d = 20 and d = 30 blocks respectively over / = 1024 simulation steps), yet start to decline when the number of blocks approaches the number of simulation steps (e.g. we obtain slower "speedup" of 0.4x when using d = 30 blocks and t = 32 simulation steps). Speedups are further raised when using smaller batch sizes (with batch sizes b = 32 and 64 obtaining a maximum speedup of x and 37x x respectively) or fixed membrane time constants (obtaining maximum speedup of 44x; See Appendix).
Robust speedups using different number of hidden units Also for feedforward SNNs, we find our d-block model to maintain robust training speedups over larger number of hidden units (e.g. obtaining 13x speedup when using 200, 400
600 hidden units for d 10 blocks and f = 512 simulation steps; Figure 3b). Again, these speedups are more pronounced when using smaller batch sizes (obtaining 15x speedup when using 200, 400 or 600 hidden units for d = 10 blocks with batch size = 32) or fixed membrane time constants (obtaining a 16x speedup when using 200, n 400 or n 600 hidden units for d 10 blocks; see Appendix).
Faster training speeds using recurrent connectivity Our recurrently connected model obtains faster training over the standard recurrent SNN. These speedups are slightly greater than the substantial speedups achieved when both models only employ feedforward connectivity, Again, in the recurrent case the training speeds increase with the number of simulation steps (with maximum speedup of 35x for d = 15 blocks and t = 2048 simulation steps; Figure 3c) and are robust over larger numbers of hidden units (with an 18x, 17x and 17x speedup when using n = 200, n = 400 and n = 600 hidden units respectively for d 10 blocks and t = 512 simulation steps; Figure 3d). As in the feedforward model, training speedups are further amplified in the recurrent model when using smaller batch sizes (with batch sizes b = 32 and b == 64 obtaining maximum speedup of 41 and x9x respectively) or fixed membrane time constants (obtaining maximum speedup of 44x; See Appendix).
4.2 PERFORMANCE ON REAL DATASETS
We tested the applicability of our model on different neuromorphic datasets of increasing diffi- cuity. We confined ourselves to neuromorphic datasets rather than image datasets as these datasets are specifically crafted for SNNs, and best utilise their temporal and neural dynamics (Lee. leak and spike timing). The simplest dataset is the N-MNIST dataset (Orchard et al., 2015), in which the MNIST image dataset (LeCun, 1998) is converted into spikes using neuromorphic vision sensor. More challenging are the Spiking Heidelberg Dataset (SHD) and Spiking Speech Commands (SSC) datasets, in which spoken digits and commands are converted into spikes us- ing model of auditory bushy cells in the cochlear nucleus (Cramer et al., 2020). These challeng- ing temporal datasets provide a good base to evaluate our model as we are able to study the effect of using multiple blocks and recurrent connectivity, whereas other datasets like images are less suited as they are readily solved using single-spike and non-recurrent SNNs (Zhou et al., 2021).
fhate-of-the-art results on neuromorphic datasets We obtain competitive accuracies across the different neuromorphic datasets, reaching an accuracy of 98.04% 86.20% and 68.16% on the N-MNIST, SHD and SSC dataset respectively (Table 1). Notably, we improve performance on the challenging SHD and SSC dataset by 3% and ~ 8% respectively over prior published results using standard recurrently connected LIF networks.
Improving performance using recurrence, more blocks and additional layers We trained multiple models (comprising single hidden layer of 128 neurons) across the different datasets to
4Our d'blocc model is identical to the standard SNN model when the number of blocks d is equal to the number of simulation steps yet our model performs more computational work per time step and hence training slows down d tends t in the limit.
7

---

Under review conference paper at ICLR 2023
a
N-MNIST
SHD
SSC
77.5
Faise
97
75.0
Tue
Accuracy
72.5
95
70.0
94
67.5
30
65.0
30
40
30
40
20
30
40
Blocks d
Blocks d
Blocks d
b
c
Layers
Datapet N-MNIST
se
100
10
100
True
SHD
Accuracy (%6
Accuracy (%)
Training speedup
SSC
Spike reduction
60
60
60
20
20
20
Blocks
Blocks
Figure 4: Analysis of our d-block model on challenging neuromorphic datasets. We use a single recurrently connected hidden layer network of 128 neurons and report results for three repeat runs of the model for which the mean and s.d. are plotted. a. Accuracy as a function of the number of blocks d using feedforward and recurrent connectivity. b. Accuracy with the spike reset being attached or detached from the computational graph during training. c. Accuracy as function of increasing number of hidden layers. d. Training speedup of our model vs the standard SNN as function of the number of blocks d. e. Reduction in spikes during inference of our model vs the standard SNN function of blocks d.
Table 1: Performance comparison to existing literature (' denotes data augmentation, & denotes trainable time constants and we report our results in bold as an average of three repeat runs).
Dataset
Model
Architecture
Neuron model
Accuracy (%)
our model
400-10 (d = 30)
recurrent LIF
98.04 0.6
N-MNIST
Shrestha & Orchard (2018)
500-500-10
feedforward LIF
98.89 + 0.06
Lee et al. (2016)
800-10
feedforward LIF
98.66
Wu et al. (2018)
400-400-10
feedforward LIF
98.78
model
256-256-20 (d 30)
recurrent LIF
86.20 0.33
SHI)
Cramer (2020)*
1024-20
recurrent LIF
83.2 1.30
Perez-Nieves et al. (2021)
128-20
recurrent LIF
82.7 - 0.80
Eshraghian & Lu (2022)
3000-20
recurrent LIF
82.27 0.27
Zenke & Vogels (2021)
256-20
recurrent LIF
82.0
Cramer et al. (2022)
186-20
recurrent LIF
76.2 1.3
Cramer et al. (2020)
128-20
recurrent LIF
71.4 1.90
our model
256-256-256-35 (d = 30)
recurrent LIF#
68.16 0.28
SSC
Perez-Nieves et al. (2021)
128-20
recurrent LIE
60.1 + 0.7
Cramer et al. (2020)
128-20
recurrent LIF
50.9 1.1
8

---

Under review as conference paper at ICLR 2023
investigate the effect of including recurrence and increasing the number of blocks. We find that including recurrent connections in our model improves prediction accuracies across all datasets, especially on the more challenging SHD and SSC datasets (Figure 4a). Increasing the number of blocks also improves performance (Figure 4a). However, performance starts to drop for a large number of blocks (d 40 across all datasets), likely due to overfitting as do not include any regularisation. We also investigate the effect of detaching the flow of gradients through the recur- rent connections, doing so has been shown improve training in standard recurrent SNNs (Zenke & Vogels, 2021). We observe no degradation in performance when detaching the spike reset terms from the computational graphs during training and find that it improves the perfor- mance on the SSC dataset (Figure 4b). Lastly, we investigate the effect of using additional hidden layers and find that additional layers improve accuracies on the SHD and SCC datasets, but de- grade performance on the N-MNIST dataset (Figure 4c).
Accelerated training We find that our model trains faster than the standard SNN, with our model obtaining maximum training speedup of ~ 5x, ~ 7x, and ~ 9x on the N-MNIST, SHD, and SSC datasets, respectively, for d 10 blocks (Figure 4d). The difference in speedups is due to the different temporal lengths and input and output dimensions of the datasets.
Increased spike sparsity during inference Our model uses fewer spikes than standard SNNs during inference, with a 50% spike count reduction across all datasets when using d 30 blocks and a 80% reduction when using d = (Figur 4e). Our model thus theoretically requires less energy than standard SNNs if emulated on neuromorphic hardware, as the energy consump- tion scales approximately proportionally with the number of emitted spikes (Panda et al., 2020). However, we find that sparsity starts to decline for a growing number of blocks. Although we obtain favourable sparsity for the SHD (59% spike count reduction) and SSC (50% spike count reduction) datasets, we find our model to obtain worse sparsity on the N-MNIST dataset (41% spike count increase) when using 40 blocks. This suggests that sparsity advantages are de- pendent on the number of blocks and the dataset.
5 DISCUSSION
Surrogate gradient descent has been a substantial algorithmic development for directly training SNNs (Eshraghian et al., 2021). However, given the relation of SNNs to RNNs (Neftci et al., 2019), SNNs have been slow to train due to their sequential nature, precluding efficient parallelisation on GPUs (Kuchaiev & Ginsburg, 2017; Vaswani et al., 2017). In a recent study Taylor et al. (2022) proposed model for accelerating the training of songle-spike SNNs. However, due to the single- spike constraint, this model less applicable to challenging neuromorphic datasets (compared to recurrently connected multi-spike SNNs). In this work we address these shortcomings by ex- tending their model to be multi-spike and recurrent, and experimentally validate this new model to achieve faster training speeds on both synthetic benchmarks (up to 34x speedup) and on ious neuromorphic datasets (up to ~ 9x speedup) compared to the standard multi-spike SNN. Furthermore, our model achieves state-of-the romorphic datasets, with an accuracy of 86.20% and 68.16%, respectively, raising performance by 3% and 8% over prior published results using standard LIF networks. We are also able to do so without any training regularisation, whereas other works employ various regularisation constraints avoid overfitting and to obtain an efficient spike code (Zenke & Vogels, 2021; Perez- Nieves & Goodman, 2021). Even so, we find our model to emit fewer spikes during inference in comparison to standard SNNs, thus theoretically lowering energy requirements relative to the standard SNN when emulating our model on neuromorphic hardware (Panda et al., 2020).
results on the challenging SHD and SSC neu-
Our work can be improved various avenues. Faster training speeds could be obtained by em- ploying a sparse backprop implementation for our model, which has shown to be effective in standard SNNs (Perez-Nieves & Goodman, 2021). Here, backprop is modified to perform less computation by taking advantage of the sparse spiking nature of SNNs. The accuracy of our model could be improved by using different surrogate gradients (Zenke & Vogels, 2021), includ- ing additional modelling mechanisms such temporal attention (Yao et al., 2021) adaptive firing thresholds (Yin et al., 2021), or by using blocks of varying lengths (where we have limited ourselves to blocks of the same length).
9

---

Under review conference paper at ICLR 2023
6 REPRODUCIBILITY STATEMENT
We outline the theoretical construction of our model in section 3. An implementation of our model and instructions replicate the experiments and results of this paper can be found at https:// http://gttubbcoom htterttornootettsooooooo///////s webstorms/ DBlock, where we publish our code under the BSD 3- 3-Clause Li- cence. Lastly, training details are provided the Appendix.
REFERENCES
Guillaume Bellec, Darjan Salaj, Anand Subramoney, Robert Legenstein, and Wolfgang Maass. Long short-term memory and peeminngtt--lear in networks of spiking neurons. Advances in neural information processing systems, 31, 2018.
Guillaume Bellec, Scherr, Anand Subramoney, Elias Hajek, Darjan Salaj, Robert Legenstein, and Wolfgang Maass. solution to the learning dilemma for recurrent networks of spiking neurons. Nature communications, 11(1):1-15, 2020.
Sander M Bohte, Joost N Kok, and Han La Poutre. Firor-backkroopggatin temporally encoded networks of spiking neurons. Neurocomputing, 48(1-4):17-37,22 2002.
Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, al. Language models are few-shot learners. arXiv preprint arXiv:2005. 2020.
Santiago Cadena, George Denfield, Edgar Walker, Leon A Gatys, Andreas S Tolias, Matthias Bethge, and Alexander S Ecker. Deep convolutional models improve predictions of macaque vl responses to natural images. PLoS computational biology, 15())ee1006897, 2019.
Peter Cariani. Temporal coding of periodicity pitch in the auditory system: an overview. Neural plasticity, 6(4):147-172, 1999.
Iulia M Comsa, Krzysztof Potempa, Luca Versari, Thomas Fischbacher, Andrea Gesmundo, and Jyrki Alakuijala. Temporal coding spiking neural networks with alpha synaptic function. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 8529 8533. IEEE, 2020.
Benjamin Cramer, Yannik Stradmann, Johannes Schemmel, and Friedemann Zenke. The heidel- berg spiking data sets for the systematic evaluation of spiking neural networks. IEEE Transac- tions on Neural Networks and Learning Systems, 2020.
Benjamin Cramer, Sebastian Billaudelle, Simeon Kanya, Aron Leibfried, Andreas Grübl, Vitali Karasenko, Christian Pehle, Korbinian Schreiber, Yannik Stradmann, Johannes Weis, et al. Sur- rogate gradients for analog neuromorphic computing. Proceedings of the National Academy of Sciences, 119(4))e2100114119 2022.
Simon Davidson and Steve B Furber. Comparison of artificial and spiking neural networks on digital hardware. Frontiers in Neuroscience, 15:345, 2021.
Sophie Denève and Christian K Machens. Efficient codes and balanced networks. Nature neuro- science, 19(3):375-385 3827,,3-22222 2016.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hier- archical image database. In 2009 IEEE conference computer vision and pattern recognition, pp. 248-255. leee, 2009.
Jason K Eshraghian and Wei D Lu. The fine line between dead neurons and sparsity in binarized spiking neural networks. arXiv preprint arXiv:2201.1191 2022.
Jason K Eshraghian, Max Ward, Emre Neftci, Xinxin Wang. Gregor Lenz, Girish Dwivedi, Mo- hammed Bennamoun, Doo Seok Jeong, and Wei D Lu. Training spiking neural networks using lessons from deep learning. arXiv preprint arXiv:2109, 12894, 2021.
10

---

Under review as conference paper at ICLR 2023
Steve K Esser, Rathinakumar Appuswamy, Paul Merolla, John V Arthur, and Dharmendra S Modha. Backpropagation for energy-efficien neuromorphic computing. Advances in neural information processing systems, 28, 2015.
Steven Esser, Paul Merolla, John V. Arthur, Andrew Cassidy, Rathinakumar Appuswamy, Alexander Andreopoulos, David J. Berg, Jeffrey L. McKinstry, Timothy Melano, Davis R. Barch, Carmelo di Nolfo, Pallab Datta, Arnon Amir, Brian Taba, Myron D. Flickner, and Dharmen- dra S. Modha. Convolutional networks for fast, energy-efficient neuromorphic computing. Proceedings of the National Academy of Sciences, 113(4)):1441111144, 2016. doi: 10.1073/ p 1asa80064851111 13.
Andrew Francl and Josh H McDermott. Deep neural network models of sound localization reveal how perception is adapted to real-world environments. Nature Human Behaviour, 6(1):111~ 133, 2022.
Wulfram Gerstner, Werner M Kistler, Richard Naud, and Liam Paninski. Neuronal dynamics: From single neurons to networks and models of cognition. Cambridge University Press, 2014.
Rudy Guyonneau, Rufin VanRullen, and Simon J Thorpe. Temporal codes and sparse represen- tations: key to understanding rapid processing in the visual system. Journal of Physiology- Paris, 98(4- (1(4-6))487-499 2004.
Nicol S Harper, Oliver Schoppe, Ben DB Willmore, Zhanfeng Cui, Jan WH Schnupp, and Andrew J King. Network receptive field modeling reveals extensive integration and multi-feature selec- tivity in auditory cortical neurons. PLoS computational biology, 12(1 ((l1))ee001111 2016.
Kaiming He, Xiangyu Xiangyu Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEFE conference on computer vision and pattern recognition, pp- 770-778, 2016.
Eric Hunsberger and Chris Eliasmith. Spiking deep networks with lif neurons. arXiv preprint arXiv:1510.08829, 2015.
Saeed Reza Kheradpisheh and Timothée Masquelier. Temporal backpropagation for spiking neural networks with one spike per neuron. International Journal of Neural Systems, 30(06): 2050027,
Saeed Reza Kheradpisheh, Maryam Mirsadeghi, and Timothée Masquelier. Spiking neural net- works trained via proxy. arXiv preprint arXiv:2109. 13208, 2021.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6982 2014.
Oleksii Kuchaiev and Boris Ginsburg. Factorization tricks for lstm networks. arXiv preprint arXiv:1703. 10722, 2017.
Yann LeCun. The mnist database of handwritten digits. http://yann. lecun. comtexdblmnistl, 1998.
Jun Haeng Lee, Tobi Delbruck, and Michael Pfeiffer. Training deep spiking neural networks using backpropagation. Frontiers in neuroscience, 10:508, 2016.
Hesham Mostafa.
Supervised learning based on temporal coding in spiking neural networks. IEEE transactions on neural networks and learning systems, 29(7):322-33255, 2017.
James M Murray. Local online learning in recurrent networks with random feedback. Elife, 8: e43299, 2019.
Emre O Neftci, Hesham Mostafa, and Friedemann Zenke. Surrogate gradient learning in spiking neural networks: Bringing the power of gradient-baser based optimization to spiking neural net- works. IEEE Signal Processing Magazine, 36(6):51-63, 2019.
Peter O'Connor, Daniel Neil, Shih-Chii Liu, Tobi Delbruck, and Michael Pfeiffer. Real-time classi- fication and sensor fusion with a spiking deep belief network. Frontiers in neuroscience, 7:178, 2013.
11

---

Under review as conference paper at ICLR 2023
Garrick Orchard, Ajinkya Jayawant, Gregory K Cohen, and Nitish Thakor. Converting static image datasets to spiking neuromorphic datasets using saccades. Frontiers neuroscience, 9:437, 2015.
Priyadarshini Panda, Sai Aparna Aketi, and Kaushik Roy. Toward scalable, efficient, and accurate deep spiking neural networks with backward residual connections, stochastic softmax, and hybridization. Frontiers in Neuroscience, 14:653, 2020.
Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. 2017.
Nicolas Perez-Nieves and Dan Goodman. Sparse spiking gradient descent. Advances in Neural Information Processing Systems, 34:11795-11806, 2021.
Nicolas Perez-Nieves, Vincent CH Leung, Pier Luigi Dragotti, and Dan FM Goodman. Neural heterogeneity promotes robust learning. Nature communications, 12(1):1-9, 2021.
Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language un- derstanding by generative pre- pre-training. 2018.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, llya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 188)99 2019.
Blake A Richards, Timothy P Lillicrap, Philippe Beaudoin, Yoshua Bengio, Rafal Bogacz, Amelia Christensen, Claudia Clopath, Rui Ponte Costa, Archy de Berker, Surya Ganguli, al. A deep learning framework for neuroscience. Nature neuroscience, 22(11)1761-1770,22 2019.
Bodo Rueckauer, Iulia-Alexandra Lungu, Yuhuang Hu, and Michael Pfeiffer. Theory and tools for the conversion of analog to spiking convolutional neural networks. arXiv preprint arXiv:1612.04052, 2016.
Bodo Rueckauer, Iulia-Alee Alexandra Lungu, Yuhuang Hu, Michael Pfeiffer, and Shih-Chin Liu. Con- version of continuous-valued deep networks to efficient event-driven networks for image clas- sification. Frontiers in neuroscience, 1 11:682, 2017.
David E Rumelhart, Geoffrey E Hinton, and Ronald Williams. Learning represeetttesstnn by back- propagating errors. nature, 3238888)) 536, 1986.
Roy Schwartz, Jesse Dodge Noah A Smith, and Oren Etzioni. Green ai. Communications of the ACM, 63(12):54-63, 2020.
H Sebastian Seung. Learning in spiking neural networks by reinforcement of stochastic synaptic transmission. Neuron, 40(6):1063-1000 1073, 2003.
Sumit B Shrestha and Garrick Orchard. Slayer: Spike layer error reassignment in time. Advances in neural information processing systems, 31, 2018.
David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, et al. Mastering the game of go without human knowledge. nature, 550(7676):354-333 359, 2017.
Yosef Singer, Yayoi Teramoto, Ben DB Willmore, Jan WH Schnupp, Andrew King, and Nicol S Harper. Sensory cortex is optimized for prediction of future input. elife, 7:e31557, 2018.
Rupesh Kumar Srivastava, Klaus Greff, and Jürgen Schmidhuber. Highway networks. arXiv preprint 1rXiv:1505.00388 2015.
Emma Strubell, Ananya Ganesh, and Andrew McCallum. Energy and policy considerations for deep learning in nlp. arXiv preprint arXiv:1906.02244 2019.
Luke Taylor, Andrew King, and Nicol Harper. Accelerating spiking neural network training, arXiv preprint arXiv:2205. 15286, 2022.
12

---

Under review as conference paper at ICLR 2023
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural informa- tion processing systems, 30, 2017.
Oriol Vinyals, Igor Babuschkin, Wojciech M Czarnecki, Michaël Mathieu, Andrew Dudzik, Juny- oung Chung, David H Choi, Richard Powell, Timo Ewalds, Petko Georgiev, et al. Grandmaster level in starcraft ii using multi-agent reinforcement learning. Nature, 575(7782)3500-344 2019.
Tim P Vogels, Henning Sprekeler, Friedemann Zenke, Claudia Clopath, and Wulfram Gerstner. Inhibitory plasticity balances excitation and inhibition sensory pathways and memory net- works. Science, 154(6062):1566-1155 1573, 2011.
Ronald J Williams. Simple statistical gradient-following algorithms for connectionist reinforce- ment learning. Machine learning, 8(3):229-25 1992.
Jibin Wu, Yansong Chua, Malu Zhang, Guoqi Li, Haizhou Li, and Kay Chen Tan. A tandem learn- ing rule for effective training and rapid inference of deep spiking neural networks. JEEE Trans- actions Neural Networks and Learning Systems, 2021a.
Jibin Wu, Chenglin Xu, Xiao Han, Daquan Zhou, Malu Zhang. Haizhou I,, and Kay Chen Tan. Progressive tandem learning for pattern recognition with deep spiking neural networks. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2021b.
Yujie Wu, Lei Deng, Guoqi Li, Jun Zhu, and Luping Shi. Spatio temporal backpropagation for training -ereghreefomccceeeeee spiking neural networks. Frontiers in neuroscience, 12:331, 2018.
Timo Wunderlich, Akos F Kungl, Eric Müller, Andreas Hartel, Yannik Stradmann, Syed Ahmed Aamir, Andreas Grübl, Arthur Heimbrecht, Korbinian Schreiber, David Stöckel, et al. Demon- strating advantages of neuromorphic computation: study. Frontiers in neuroscience, 13: 260, 2019.
Daniel LK Yamins and James DiCarlo. Using goal-driven deep learning models to understand sensory cortex. Nature neuroscience, 19(3):356-366 365, 2016.
Man Yao, Huanhuan Gao, Guangshe Zhao, Dingheng Wang, Yihan Lin, Zhaoxu Yang, and Guoqi Li. Temporal-wise attention spiking neural networks for event streams classification. Pro- ceedings of the IEEE/CVF International Conference on Computer Vision, pp. 10221-10230, 2021.
Bojian Yin, Federico Corradi, and Sander M Bohté. Accurate and efficient domaidomann classi- fication with adaptive spiking recurrent neural networks. Nature Machine Intelligence, 3(10): 905-913, 2021.
Friedemann Zenke and Surya Ganguli. Superspike: Supervised learning in multilayer spiking neural networks. Neural computation, 3016):15141141, 2018.
Friedemann Zenke and Tim P Vogels. The remarkable robustness of surrogate gradient learning for instilling complex function in spiking neural networks. Neural Computation, 33(4))899- 925, 2021.
Shibo Zhou, Xiaohua Li, Ying Chen, Sanjeev T Chandrasekaran, and Arindam Sanyal. Temporal- coded deep spiking neural network with easy training and robust performance. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pp. 11143-1115) 2021.
A APPENDIX
A.1 SYNTHETIC SPIKE DATASET FOR THE SPEED BENCHMARKS
As outlined in (Taylor et al., 2022), we constructed binary input spikes of shape B x T (B being the batch size, N the number of input neurons and T the number of simulation steps), such that every batch dimension b had a firing rate rы U umax) uniformly sampled (with Mmin OHz and Umax 200Hz). For every batch dimension generated a random binary spike matrix of shape x T, such that every input neuron the matrix had an expected firing rate of To Hz.
13

---

Under review conference paper at ICLR 2023
A.2
TRAINING DETAILS AND HYPERPARAMETERS
A.2.1 READOUT NEURONS
Every network had an output layer of readout neurons (containing the same number of neurons as the number classes within the dataset trained on), where we removed the spike and reset mechanism (as done in Zenke & Vogels (2021)). The output of readout neuron in response to input sample c was taken to be the summated membrane potential over time Ob Σ, VL [t]. b.
A.2.2 BETA CLIPPING
To enforce correct neuron dynamics we clipped the values of p into the range [0, 1].
5 = 0. if 1 <0
1, if
; >
(6)
A.2.3 WEIGHT INITIALISATION
All network connectivity weights were sampled from uniform distribution U U(-VN-I,VA where N is the number afferent connections. All biases were initialised as All neurons in the hidden layers were initialised with membrane time constant r 10ms and = 20ms for the readout neurons.
A.2.4 SUPERVISED TRAINING LOSS
We trained all networks to minimise a cross-entee enos--y loss (with and C being the number of batch samples and dataset classes respectively)
1
B Σ b=10=1
L
Σ
yb, log(
Variable yb. 10 the one hot target vector and Ph.c are the network prediction probabilities, which were obtained by passing the readout neuron outputs Ob,c through the softmax function.
exp Ob.c
Ph.c
(8)
Σ
k=1 exp 0b,k
A.2.5 SURHOGATE GRADIENT
We used the fast sigmoid function as our surrogate gradient (Zenke & Ganguli, 2018), which has been shown to work well in practice (Zenke & Vogels, 2021). Here hyperparameter Paur (which we set to in experiments) defines the slope of the gradient.
f (V) dV
(Bsurl VI 1)
A.2.6 TRAINING PROCEDURE
We used the Adam optimiser (with default parameters) (Kingma & Ba, 2014) for all training start- ing with an initial learning rate of 0.001, which was decayed by factor of 10 every time the num- ber of epochs reached a new milestone. model was saved if it managed to lower the training error at the end of each epoch.
A.2.7 TRAINING HYPERPARAMETERS AND EXTENDED SPEEDUP RESULTS
14

---

Under review conference paper at ICLR 2023
Table 2: Dataset and corresponding training parameters for best performing models.
N-MNIST
SHD
SSC
Dataset (rain/tesi)
60k/ T0k
8156/2264
75466/20362
Input newrons
1156
700
700
Dataset classes Epochs
10
20
35
50
60
80
Learning rate
0.001
0.001
0.001
Batch size B Simulation steps
128
128
128
300
500
500
Time resolution Ar (ms)
2
Milestones
(30)
(30)
(30,60)
(Feedforward) Training speedup 100 and b 32)
(Recurrnt) Training speedup (n - 100 and b 32)
20.0
40.30%
30.0
41.48x
25.0
15 18.00x
25.0
8.00x
4.00%
2.00x
2.00
1.00x
30.0
1.00x
0.64x
5.0
0.45%
[f eeedforward) Training speedup (n 100 and ò 64)
(Recurrent) Training speedup (n= 100 and 64)
37.40x
16. 16.00
16
8.00x
4.00x
4.00x
15.0
2.00
10.0
1.00x
10.0
1.00x
5.0
0.45
5,0
0.47x
(Feedforward) Training speedup (n 100 and b= 128)
(Recureent) Training speedup (n 100 and 128)
30.0
43.98x
30.0
43.74x
25.0
25.0
8.00x
28.0
8.00x
4,00
4.00M
2.00x
log: Simulation steps
log, Simulation steps
Figure 5: Training speedup of our d-block model over the standard SNN for feedforward and recurrent networks as a function of the number of blocks d and simulation steps (for fixed hidden neurons = 100). a.-b. Batch size 32. c.-d. Batch size 64. e.-f. Batch using fixed membrane time constants. Left column: Feedforward networks and right column: Recurrent networks.
but
15

---

Under review conference paper at ICLR 2023
[Feedrorward) Training speedup (t 512 and b 32)
b
(Recurrent) Training speedup (t 512 and b 32)
30.0
22.10x
30.0
26.92x
25.0
25.0
blocks
blocks
16.0
18.0
5.0
6.68x
5.0
7.13%
(Feedforward) Training speedup It
64)
(Recurrent) Training speedup (t 512 and b 64)
30.0
16.52x
35.0
20.02x
25.0
25.0
blocks
20.0
16.0
7.05
5.0
7.09x
(Feedforward) Training speedup
(Recurrent) Training speedup
20.20x
25.0
blocks d
20.0
blocks
20.0
154
15.0
10.34x
400
702
300
920
1202
Hidden units
Hidden units
Figure 6: Training speedup of our d-block model over the standard SNN for feedforward and recurrent networks as a function of the number of blocks d and hidden neurons nffor fixed sim- ulation steps 512). a.-b. Batch size 32. c.-d. Batch size 64. e.-f Batch size 128 but using fixed membrane time constants. Left column: Feedforward networks and right column: Recurrent networks.
16


Reviews:
[{"review_id": "oqokzezaJg6", "reviewer": "Reviewer_ddei", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a new SNN model, named d-block model. By adding stochastic absolute refractory periods and recurrent conductance latencies, a d-block SNN can reduce the number of sequential computations using fast vectorized operations. ", "summary_of_the_review": "The paper proposes an efficient SNN training acceleration method. Novelty and experiments are limited.", "strength_and_weaknesses": "Strength: \nThe proposed model achieves fewer sequential operations and lower energy consumption compared to conventional SNNs. They obtained SOTA accuracy scores on SHD (86.2%) and SSC (68.16%) are impressive.\n\nWeakness:\n1. The speedup experiments were conducted on small models. It is concerned that whether this method could work on larger and deeper SNNs, like ResNet.\n2. There is limited novelty considering the reference (Taylor et al, 2022.) for accelerating training single-spike SNNs. It seems this article just makes incremental contributions.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "hyXr9q86gC", "reviewer": "Reviewer_jrgn", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work propose a new SNN model named the d-block model, with stochastic absolute refractory periods and recurrent conductance latencies, which reduces the number of sequential computations using fast vectorised operations. Input spikes are processed by d equal length blocks, where every block is a single-spike SNN. This model obtains accelerated training speeds and state-of-the-art performance on the N-MNIST, SHD, and SSC datasets without the need for any regularization and using fewer spikes compared to standard SNNs. Besides, this model theoretically consumes less energy than conventional SNNs when employed on neuromorphic hardware.", "summary_of_the_review": "I think this work is less innovative and the illustration of model isn’t clear. ", "strength_and_weaknesses": "The authors didn’t give clear illustration and provide many detailed information to make readers to understand d-block model. The author makes references about single-spike SNN proposed by Taylor et al. (2022). But, they didn’t compare this work with single-spike SNN.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "cm8j9T9SIi8", "reviewer": "Reviewer_D9Qe", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents a new architecture for spiking neural networks (SNNs), called d-block model. This architecture is built upon a previously proposed work (Taylor et al. 2022). The idea is to construct multiple blocks where within each there is dependency, allowing more parallelism during training. It has been shown that the new model outperform prior works both in terms of accuracy and also traiing time.\n", "summary_of_the_review": "While the contribution of this paper is rather limited since it has been obtained by small modification to the prior work (Taylor et al. 2022), the training speedup is significant. The main issue that I have its comparison results in Table 1 where the networks architectures are not the same. I also expect to see the training time per epoch for a fair comparison.", "strength_and_weaknesses": "Strengths:\n-- The main strengths of the paper is its significant training time reduction.\n-- The proposed model outperforms existing works in terms of performance.\n-- The paper is well-written and east to understand.\n\nWeaknesses:\n-- The new model was built on top of an existing method by separating its architecture into multiple blocks. There is no mathematical foundation for the new model explaining why it should achieve a better results while its speedup during training is evident.\n-- The architectures used for comparison are not the same which makes the comparison unfair and inconclusive. For example, it is expected to have better accuracy performance when a deeper model is used for SHD and SSC datasets in Table I. \n-- Table I should also include training time per epoch for a fair comparison when using the same architecture.\n-- I also expect to see some experimental results on more challenging datasets such as CIFAR10,100 and ImageNet.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "UZxSwNhGFA", "reviewer": "Reviewer_Yn85", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors propose a method to control the number of output spikes (d-block model) and accelerate SNN calculations by avoiding the recursive calculation of membrane potential. The acceleration of SNN calculations was verified on various event datasets like N-MNIST, SHD, and SSC. As well, the model achieves high classification accuracy on the event datasets.", "summary_of_the_review": "Overall, the significance of this work is limited given the marginal progress from the previous paper Taylor et al., (2022). ", "strength_and_weaknesses": "Strength:\n\nThe idea to avoid the recursive update of membrane potential is of technical importance given the time and computational complexities of the recursive model which scale with the number of timesteps used. The authors successfully identified the feasibility of the proposed method on various event datasets.\n\nWeaknesses:\n\n1. The poor paper organization significantly undermines the readability. Also, the full explanation of the d-block model illustrated in 2b is not given. I do not still understand what $m$ and $\\tilde{V}$ and their superscripts stand for. \n\n2. Regarding the previous paper (Taylor et al. (2022)), I can find only very marginal difference of the present work from the previous work(Taylor et al. (2022)). The only difference is such that d (rather than one) spikes are used. \n\n3. The previous works compared with the present work in Table 1 are out of dated. I recommend the authors to address SOTA results.\n\n4. On Page 6, the authors address the github page for the code, which is against the double blind policy.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "SWaG1_9Dzq", "reviewer": "Reviewer_b7YL", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work extends the 1-block model in Taylor et al. (2022) to the d-block model. Compared with the LIF model, the d-block model achieves accelerated computing on GPU by using fewer sequential operations.", "summary_of_the_review": "If the authors want me to raise the score, they should convince me about the novelty and the implementability of the model.", "strength_and_weaknesses": "Strength:\n\nThe proposed model indeed enables accelerated computing on GPU, and achieves sota results on some benchmarks.\n\nWeaknesses:\n\n1. This work is quite a naive extension of Taylor et al. (2022). If the authors of this work and Taylor et al. (2022) are the same, I highly recommend the authors to combine the two works into one.\n\n2. The authors need to describe the training method explicitly in the main content. The proposed model can be treated as a modified LIF model, which is irrelevant to the training methods. From appendix A.3.5, which is not detailed, I guess that the existing surrogate gradient method is adopted. The accelerated training on GPU is due to the model's parallel computing nature, not due to some novel training method. The authors should make it clear.\n\n3. Can the d-block model be implemented on neuromorphic hardware in an event-driven manner? First, the 1-block model is equivalent to the single-spike LIF model, so I do not worry about it. But for the d-block model, the spikes from the first 3 time steps are used in the 4th time step. Is it implementable? will it be implementable on future neuromorphic hardware?\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject