Addressing the speed-accuracy simulation trade-off for adaptive spiking neurons
Authors: Luke Taylor
Andrew J King
Nicol Spencer Harper
Venue: NeurIPS 2023 poster (2023)
Keywords: spiking neural network
surrogate gradient descent
adaptive leaky integrate and fire neuron
speed-accuracy trade-off
electrophysiological recordings
TL;DR: We address the speed-accuracy trade-off in simulating recurrent ALIF SNNs, achieving up to a 50x training speedup and enabling quick and accurate fitting of real neural recordings on sub-millisecond timescales.

Abstract: The adaptive leaky integrate-and-fire (ALIF) model is fundamental within computational neuroscience and has been instrumental in studying our brains $\textit{in silico}$. Due to the sequential nature of simulating these neural models, a commonly faced issue is the speed-accuracy trade-off: either accurately simulate a neuron using a small discretisation time-step (DT), which is slow, or more quickly simulate a neuron using a larger DT and incur a loss in simulation accuracy. Here we provide a solution to this dilemma, by algorithmically reinterpreting the ALIF model, reducing the sequential simulation complexity and permitting a more efficient parallelisation on GPUs. We computationally validate our implementation to obtain over a $50\times$ training speedup using small DTs on synthetic benchmarks. We also obtained a comparable performance to the standard ALIF implementation on different supervised classification tasks - yet in a fraction of the training time. Lastly, we showcase how our model makes it possible to quickly and accurately fit real electrophysiological recordings of cortical neurons, where very fine sub-millisecond DTs are crucial for capturing exact spike timing.

Addressing the speed-accuracy simulation trade-off for adaptive spiking neurons
Luke Taylor
Department of Physiology, Anatomy and Genetics
University Oxford
Oxford, United Kingdom
luke. taylor tfaeeerrfordd.o ox ac.uk uk
Andrew King
Department of Physiology, Anatomy and Genetics University of Oxford
Oxford, United Kingdom
andrew. king@dpag ox. ac uk
Nicol S Harper
Department of Physiology, Anatomy and Genetics University of Oxford Oxford, United Kingdom
nicol. harperrrppdd.ppgg........ ox ac. uk
Abstract
The adaptive leaky integrate-and-fire (ALIF) model is fundamental within computa- tional neuroscience and has been instrumental in studying our brains in silico. Due the sequential nature of simulating these neural models, a commonly faced issue is the speed-accuracy trade-off: either accurately simulate neuron using a small discretisation time-step (DT), which is slow, or more quickly simulate neuron us- ing larger DT and incur loss in simulation accuracy. Here we provide a solution this dilemma. by algorithmically reinterpreting the ALIF model, reducing the sequential simulation complexity and permitting more efficient parallelisation on GPUs. We computationally validate our implementation obtain over 50x training speedup using small DT's on synthetic benchmarks. We also obtained a comparable performance the standard ALIF implementation on different su- pervised classification tasks yet in fraction of the training time. Lastly, we showcase how our model makes possible to quickly and accurately fit real elec- trophysiologica recordings of cortical neurons, where very fine sub-millisecond DTs are crucial for capturing exact spike timing.
Introduction
The surge of progress in artificial neural networks (ANNs) over the last decade has advanced our understanding of the potential computational principles underlying the processing of sensory information [1-2) Although these networks architecturally bear a resemblance the brain (10), they tend to omit a key physiological constraint: the spike. With their increased biological realism, spiking neural networks (SNNs) have shown great promise in bridging the gap between experimental data and computational models. SNNs can be fitted to real neural data [11 or used to simulate the
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

brain, offering new level of understanding of the complex workings of the nervous system [16 19 They also have engineering applications in energy-efficient machine learning (20]
An established class of spiking models in computational neuroscience is the leaky integrate-and- (LIF) neuron, with origins dating back to 1907 [21]. Just like in real neurons, input current (resulting from presynaptic input) charges the membrane potential of the model neurons, which then output binary signals (i.e. spikes) as form of communication (Figure T The adaptive leaky integrate- and-fire (ALIF) model [22] is a modern extension of the LIF. It more closely mimics the biology. capturing a key property of neurons, which their adaptive firing threshold (i.e. spikes become less frequent in response to a steady input current (23)). ALIF neurons have been shown to accurately fit
real neural recordings [T] 24 25 and outperform the simpler LIF neurons on various machine
learning benchmarks 26 29
Despite these modelling advances, major shortcoming of LIF and ALIF neurons is their slow inference and training times. Unlike real neurons, modelling neuron dynamics involves sequential computation over discretised time. This leads a problematic trade-off between speed and accuracy when simulating SNNs. A small DT enables accurate modelling of dynamics, but is slow to stimulate and train on computer systems such as GPUs. A large DT obtains less accurate dynamics, but at the benefit of being faster to simulate and train [3] (Figure Tb). This raises the important question of capturing the best of both worlds: is there a way to accelerate the inference and training of spiking LIF and ALIF neurons without sacrificing simulation accuracy?
In this work, address the speed-accuracy trade-off when simulating and training LIF and ALIF SNNs, and present a solution that is both fast and accurate. We take advantage of a fundamental property of neurons that is sometimes short period of time following spike initiation, during which neuron cannot fire again. As result, neuron can spike at most once within such a period (Figure [t)). We leverage this observation to develop a novel algorithmic reformulation of the ALIF model which reduces the sequential simulation complexity of the standard ALIF algorithm. Specifically, we outline how ALIF recurrent networks can be simulated with a constant sequential complexity O(1) over the ARP simulation length Tk. and how this approach can be extended to longer simulation lengths to obtain identical dynamics to the standard ALIF network algorithm. Faster simulation and training are theoretically obtained for growing length Tn. by either employing physiologically plausible ARPs of ~ 2ms and decreasing the DT to very fine value ~ ms (for realistic neural modelling), or setting the DT to a coarser value and adopting larger non-physiological ARPs (for machine learning tasks). Our main contributions are the following:
modelled the absolute refractory period (ARP). This is
We develop novel algorithmic reformulation of the ALIF model with O(T/TB) rather than O(T)- sequential complexity, for simulation length T and ARP length Tn
We find that our model achieves substantial inference (up to 40x) and training speedup (up 53x) over the standard ALIF SNN for increasing ARP time steps, and find this hold over different numbers of simulation lengths, batch sizes, number of neurons and layers.
We demonstrate the feasibility of our model to be trained using surrogate gradient descent, with our accelerated ALIF SNN achieving comparable accuracies to the standard ALIF SNN temporal spiking classification datasets yet in fraction of the training time.
Finally, we showcase how ALIF implementation makes it possible to quickly fit real electrophysiological recordings of cortical neurons, where very fine mub-miliseecond dis- cretisation steps important for accurately capturing exact spike timing.
'To avoid ambiguity, simulation length I' and ARP length TR are number of time steps (dimensionless) and not unit time.

---

b
TR
TR
TR
Expected
Firing rate (Hz)
Internai state
accurate 200
inaccurate
&
& fast
D S
slow
Error
100
ARP
ARP
10-1
10"
10
2
6
DT (ms)
Time (ms)
Figure 1: Problem overview. a. Schematic of ALIF neuron: input current charges membrane potential and outputs spikes S if firing threshold is reached (with the neuron's 's internal state evolving over time). b. An example of the simulation trade-off problem when simulating a single ALIF neuron with fixed synaptic weights receiving Poisson spike input. The simulation error and the speed grow for increasing discretisation time (DT), e. Observation for our solution: neuron emits at most a single spike during a simulation span T'p & equal in length to the neuron's 's absolute refractory period (ARP).
Background and related work
Standard ALIF model We introduce the recurrent SNN of ALIF neurons with fixed ARP and latency of recurrent transmission, defined by the following set of equations 266 31.
14 [1] (V, 00 [4] = 1 w/' [1) (Output spike)
(1)
1 [t] v." 1] + (1 2(0) 1r 1 [t)) (1 st" 11)
(Membrane potential)
(2)
NO
NO
1e (t.) + Σ j=1
w
12 - 1
w (1) sf DD)
(Input current)
(3)
Feedforward current
Recurrent
c(O) 2 TR otherwise
14 O
(Absolute refractory period)
(4)
0/ [1] + (4)
[t + 5 (Adaptation)
(5)
( [t] = P (1)
At time t, neuron within layer (consisting of ( neurons) receives input current 14"1 and outputs binary value st0) [1] E {1,0) (ie. spike or no spike) if neuron's 's membrane potential (i() [t] reaches The evolution of the membrane potential described by the normalised discretised LIF equation (Equation 2). in which the membrane potential dissipates by a learnable factor < g() at every time step and resets zero if spike occurred at the
firing threshold 0 (Equation
previous time step. The input current is comprised of constant bias source (1) and from incoming spikes reflecting feedforward W()) E R x x())) and recurrent connectivity W/ees (1) E RVO x (Equation
Here. we assume the recurrent transmission latencies be of fixed length and equal in length to the ARP, that is TR D. With a simple modification however, our methods can work for longer D, or different D on each connection, so long as 2 Tn. The ARP enforced by only allowing input current to enter the neuron if the number of time steps c, following the last spike equals or exceeds the ARP length (Equation 4p. Lastly, adaptation is implemented by raising the firing threshold 0!0h [t]
Here notation 1 conditiio denotes the indicator function, which is equal to one if the condition is true and zero otherwise.
3

---

following each spike ( - 1] (Equation 5p. which decays exponentially to baseline 000 = in the (1)
absence of any spikes (using learnt decay factor S P. - and adaptation scalar d
The ARP in biological neurons is typically about ~ 1 2ms B2 34]
Our method takes advantage of the fact that the monosynaptic connection latency between neurons in local circuits is typically also often around ~ 1 2ms [55] LIF and ALIF neuronal models are typically run with a DT of about 0. ms ms in the computational neuroscience literature and ~ Ims machine learning literature [30]. Furthermore, the firing rate of neurons is often less than the reciprocal of the ARP, suggesting that a higher TR than the ARP may still provide a reasonable approximation of neural behaviour for some purposes.
SNN training The main problem with training SNNs is the non non-differentiable nature of their activation function in Equation (whose derivative is undefined at v/ 0 [t] and zero oth- erwise). This precludes the direct use of the backprop algorithm [66 which has underpinned the successful training of ANNs. A popular solution is to replace the undefined spike derivative with a well-behaved function, referred to as a surrogate gradient [37.40] and training the network with backprop-through-timg [41]. This method also supports the training of neural parameters other than synaptic connectivity, such as membrane time constants 442 28 and adaptive firing thresholds 26--28) both of which we include in our work, as they have been shown to improve performance (and increase biological realism). It is worth mentioning that other SNN training methods exist such as mapping trained ANNs to SNNs by transforming ANN neuron activations to SNN neuron firing rates [43/46]. These methods are, however, of less interest to computational neuroscientists as they discard all temporal spike precision.
Related work Recent work has provided new theoretical insights for increasing the simulation accuracy when employing larger DT > Ims (30]. We are not aware of any work that accelerates the simulation and training times on GPUs when employing a smaller DT I Ims (although see the NEST simulation library for simulating SNNs CPUs 49). There are, however, different methods for speeding up the backward pass (i.e. how gradients are computed within the SNN), which consequentially speeds up training times. Rather than being viewed as competing approaches, these methods could further augment the speed of our solution. In sparse gradient descent, gradients only passed through neurons whose membrane potential is close firing threshold, which can significantly accelerate the backward pass when neurons are mostly silent (50). Inspired by work on training non-spiking networks [51]52) other methods completely bypass the backward pass and adjust weights online (53 27 29]. Another approach is to propagate gradient information through spikes [54-60], which similar the idea of sparse gradient descent- is fast when neurons are mostly silent. This method, however, enforces neurons to spike at most once and can suffer from training instabilities (although see 61
3 Theoretical results
We outline novel reformulation of the ALIF model, which theoretically reduces the sequential simulation complexity from O(T') to O(T/TR) (for simulation length T and ARP length Tg). Using the observation that a neuron can spike at most once over simulation length TR (equal to the ARP; Figure 1c), we propose simulating network dynamics sequentially in blocks of length TR- as opposed to simulating every time step individually Frigurehh - as we show that these blocks can be simulated with constant sequential complexity O(1). Our reformulated ALIF model is mathematically the same as the standard ALIF model, but substantially faster to simulate and train. All the proofs for the propositions be found in the Supplementary material.
3.1 Block: simulating single-spike ALIF dynamics with a constant sequential complexity
A SNN exhibits a sequential dependence due to the spike reset mechanism, where a neuron's 's membrane potential V2[4] can be reset based on its previous output S - 1]. Consequently, the simulation of a SNN necessitates sequential approach. This restriction can, however, be alleviated we assume a neuron to spike at most once over a particular simulation length (such as the ARP).
The following steps grouped together into module referred to Block (Figure 2b) b) compute

---

a
b
Output spikes
S
S
S13]
S(1]
S12)
513)
S14
S
S
15
6
[
12(2]
Initial conditions
2
3
44
6
Pu
11
512]
S
2
3
814
S|5
5
Block
Block
Vo
2
[33
7
6
Input current
[
[[3
Figure 2: Solution overview. Our proposed solution: instead of simulating network dynamics one time step after another (top), we sequentially simulate blocks of time equal in length to the neuron ARP (bottom), in which neuron can spike most once. b. A schematic of Block: our proposed solution for emulating ALIF dynamics with a constant sequential complexity O(1) over a short duration which a neuron spikes at most once.
ALIF dynamics (assuming at most one spike) without any sequential operations.
[ (T, [:) (No-reset membrane potential)
| = [(V](4)
(Faulty output spikes)
(7)
</ = ((.)
(Latent timing of spikes)
(8)
S = 1 [x[4]] =1 (Correct output spikes)
Calculate membrane potentials without reset The first step the Block (Equation converts
input current [4] membrane potentials Vi[4] without spike reset (i.e. excluding the reset mechanism
in Equation 2). This transformation is achieved using a convolution (Proposition any sequential operations.
T]. thus avoiding
Proposition 1. Membrane potentials without spike reset are computed as convolution Vi[1]
(I, [1] between input current I[[ and kernel B:[[] (1 3.)33 with the initial membrane potential encoded as I,[0]
I-
Faulty output spikes No-reset membrane potentials V.[(] are mapped to erroneous output spikes 8,(4 = ((V (Equation 7) using spike function f (Fquation 1p. This output can contain more than spike, but only the first spike complies with the standard model dynamies, due the omission of the reset mechanism and ARP constraint. Thus, to ensure that the Block only emits a single spike, all spikes succeeding the first spike occurrence removed using the following steps.
3. Latent timing of spikes Erroneous spike output 5,(4] is mapped to latent representation < ((S) (Equation 8) encoding the timing of spikes. Function o(-) (taking vector 5; as input; Proposition 2 constructed map all erroneous spikes S1((), besides the first spike occurrence, to value other than one (i.e. [ + for all except for the smallest t satisfying S. 1 if such exists).
Proposition 2. Function ((S)(tt = Ei (t ( 1+ acting on S, E {0, 1)TT contains at most one element equal to one (($.)(t) = 1 for the smallest t satisfying S.(t] (if such t exists).
4. Correct output spikes Lastly, the correct spike output S,(t] 1 21 is obtained setting
every value in 21]t], besides the value (i.e. the first spike), to (Equation 9
5

---

3.2 Blocks: simulating ALIF SNN dynamics with O(T/TR) sequential complexity
The standard ALIF SNN model can be reformulated using chaining of Blocks, which reduces the sequential simulation complexity (as each Block is simulated in O(1)). For a given ARP of length TR. we observed that neuron spikes at most once over simulation length TR (Figure T Thus, simulation length 7 can be simulated using N T Blocks, each of length Tn- Next, we outline how to simulate ALIF dynamics across Blocks to emulate the dynamics of the standard ALIF SNN. We introduce new notation index Block S n S N using subscript n (e.g. input current to neuron i simulated in Block n is expressed as Ii, [t] with time steps indexed between [1, TR] (as opposed to [1, T within a Block.
Input current and the ARP of Block The input current in the standard model (Equation 3 and 4 modified for the Block model (Proposition B) The feedforward and recurrent current to Block n n-+ derived from the presynaptic and postsynaptic spikes from Block + 1 and Block respectively,
In addition, the ARP is enforced by applying a mask derived from the latent timing of spikes 24 (Equation (8
Proposition 3. The input current 1,n41(t) [t] of neuron i simulated in Block n + 1 (of length Ta) is defined as follows, and enforces an absolute refractory period of length TR and a monosynaptic transmission latency of D = TR.
N
N Wiy Symll]) 1l 22,, 2 max Si
1,,++144 = (b.- - Σ W So5th11211++
Feedforward current
Recurrent
ARP mask
Evolving membrane potentials between Blocks Two cases are distinguished to correctly emulate the evolution of the membrane potentials between Blocks (Proposition 4): 1) if neuron i did not spike Block n (i.e. max, [ = 0), then its initial membrane potential Vi [0] in Block v + is set to its final membrane potential Vi, [Tk] in Block n. Otherwise 2), the initial membrane potential is set to zero to emulate a spike reset (and no state needs to be transferred between Blocks as the neuron is in refractory state).
Proposition 4. The initial membrane potential Vi,n+ [0] of neuron i simulated in Block n + 1 (of length TR) is equal to the last membrane potential in Block n. if no spike occurred and zero otherwise.
Vi. [T]] -10
if max; St.n = otherwise
Vin+1 = =10
Evolving adaptive firing thresholds between Blocks The adaptive firing threshold 0 of neuron in Block
derived from the initial adaptive parameter a [0] (Proposition (5) Two cases are distinguished for deriving this parameter: 1) if the neuron did not spike during the previous Block n, this parameter is set to last value, (t...[nn],
the prior Block; otherwise 2), the effect
of the spike needs to be taken into account, for which the initial adaptive parameter is expressed as pf" (a, p,-1) )). Here, m is the number of time steps remaining in Block n following the spike and a, the adaptive parameter value at the time of spiking.
The adaptive firing threshold 11,44(t] [t] of neuron simulated in Block + 1 (of length TR) is constructed from the initial adaptive parameter di,a41(0) [0), which is equal to its last value in the previous Block if no spike occurred, and otherwise equal an expression which accounts for the effect of the spike on the adaptive threshold.
Proposition 5.
+ [t] = + dep: as.p
(aim [TR]
if max; (i,nt]] otherwise
ai,n+1
[0]
p (a, Di
TR
Tr
m
Zi,n [k] > 1
Σ
0z [4]
With appropriate padding to the simulation length if it not divisible by Th.

---

3.3 Theoretical speedup for simulating ALIF SNNs using Blocks
Method
Computational Complexity
Sequential Operations
Standard
( To
O(T)
Blocks
N
O(T/TR)
Table I: Computational and sequential complexity of simulating layer of Nou neurons with Nin input neurons for T time steps using the standard method and our method (with N Blocks each of length TR).
Simulating an ALIF SNN using our Blocks, rather than the conventional approach, requires fewer sequential operations (Table D Although the computational complexity of our Blocks approach is larger than the standard upproach, the number sequential operations less. If we assume that the sequential steps in both methods are executed in an equal amount of time (as all the non-sequential operations can be run parallel on a GPU), we obtain a theoretical speedup equal to the length of the ARP T TTn-
N
4
Experiments
We evaluated the training speedup of our accelerated ALIF SNN relative the standard ALIF SNN and explored the capacity of our model to be fitted using different spiking classification datasets and real electrophysiological recordings of cortical neurons. Implementation details can be found in the com wetn///ebstormm//Been Blocks
Supplementary material and the code at httpt: /tttps://githbb http:/gtttt....eeeaeeeeeeeeeeeee
4.1 Training speedup scales with an increasing ARP
a
b
T= 512
T= 1024
T= 2048
T= 1024
Batch
50
Neurons
50
32
50
50-
128
64
40
Speedup x
40-
128
40
401
30
20-
20
20-
20-
10
10-
3
ARP (steps)
ARP (steps)
ARP (steps)
Layer
Figure 3: Training speedup of our model. a. Training speedup our accelerated ALIF model
compared to the standard ALIF model for different simulation lengths T, ARP time steps and batch sizes. b. Training speedup over different number of layers and hidden units (with an ARP of Ta = 40 time steps, T 1024 time steps and batch size 64; bars plot the mean and standard error over ten runs). Assuming DT= 1ms, then 10 time steps = Ims.
To determine how much faster our model is, we benchmarked the required training duration (forward and backward pass) of our accelerated ALIF SNN the standard ALIF SNN for varying ARP and simulation lengths using a synthetic spike dataset, with the task of minimizing the number of final-layer output spikes (see Supplementary material). We found the training speedup of our model to increase for a growing ARP (Figure 34). This speedup was more pronounced for longer simulation durations (53x speedup for = 2048) than shorter simulation durations (36x speedup for 7 512). These speedups were also present when just running the forward pass (i.e. inference using the network; see Supplementary material). Furthermore, we found the speedup of our model to be robust over varying numbers of neurons and layers (Figure Bp). Lastly, we also found our method
7

---

to perform the forward pass more than an order of magnitude faster than other publicly available
SNN implementations 63 64 (see Supplementary material).
4.2
Accelerated training on spiking classification tasks
b
N-MNIST
SHD
N-MNIST
SHD
Standard
100
100
Avg. epoch duration (8)
250
Accuracy (%)
1500
200
60
80
150
1000
40
40
100
500
50-
0-
20
20 60
30
50
60
100
ARP (ms)
ARP (ms)
ARP (ms)
ARP (ms)
Figure 4: Performance of our model on spiking datasets. a. Classification accuracy of our model and the standard ALIF SNN on the N-MNIST and SHD datasets over different (non-biological) ARPs (N-MNIST Ims=1 time step; SHD 2ms=1 time step). b. Training durations of our model and the standard ALIF SNN on the N-MNIST and SHD datasets. Horizontal gray lines plot the standard model's performance (using no ARP) and bars plot the mean and standard error over three runs.
To establish whether model learn using backprop with surrogate gradients, and perform par with the standard model, we trained our accelerated ALIF SNN and the standard ALIF SNN on the Neuromophic-MNISS (N-MNIST) (using DT Ims) 665 and Spiking Heidelberg Digits (SHD) (using DT= 2ms) [66] spiking classification datasets (both commonly used for benchmarking SNN performance 44 29 50). The task of the N-MNIST dataset is to classify spike representations
of handwritten digits, and the task of the SHD dataset is to classify spike representations of spoken digits. In all experiments we employed identical model architectures consisting of two hidden layers (of 256 neurons each) and an additional integrator readout layer, with predictions taken from the readout neurons with maximal summated membrane potential over time (as commonly done sarrogate-gradieen function 29 to overcome the non-differentiable discontinuity of the spike function (although we found that other aurrogate-gradies functions also work, see Supplementary material). Furthermore (as suggested by (68) for conventional SNNs), we only permitted surrogate gradients flow through the non-recurrent connectivity (which significantly improved performance; see Supplementary material).
[66] 68
29
42 50); see Supplementary material). We adopted the multi- multi-Gaussian
We trained our model with different bioloiolog ARPs on each dataset and contrasted the accuracy and training duration to the standard model (with no ARP). We found favourable trade-off between classification accuracy and training time for our model. Compared to the standard ALIF SNN model, our model achieved very similar, albeit slightly lower, accuracy on the N-MNIST (96.97% for ARP=10ms vs standard 97.71%) and SHD dataset (86.07% for ARP=20ms vs standard 87 87.48%), with the accuracy declining only for the largest ARPs tested (Figure F However, our model only required a fraction of the training time, with an average training epoch of 181s and 20s for the N-MNIST and SHD datasets, respectively, when using the largest tested ARP, compared to the respective standard model training times of 2034s and 268s (Figure 4b).
4.3 Quickly fitting real neural recordings on sub-millisecond timescales
We explored the ability of our model fit in vitro clectrophysiological recordings from 146 inhibitory and excitatory neurons in mouse primary visual cortex (V1) (provided by the Allen Institute (0)). In these recordings, a variable input current was repeatedly injected at various amplitudes into each real neuron, and the resulting alteration in the membrane potential was recorded (Figure 5p) For each neuron, we used half of the recordings for fitting and the other half for testing, and report all the qualitative and quantitative results on the held-out test dataset. Fitting was achieved by minimising

---

a
b
1m= ARP=2ms
DT=0.1ms ARP=8ms
DT=4ms ARP=4ms
Our model
Recording
Injection
noise
noise 2
noise
noise
noise
noise
c
e
0.8
EDV o. (06)
1.00
Aug. nearonn time
100
0.78
0.6
0.6
0.6
0.50
50-
0.4
0.25
25
0.2
1.0
Blocks Standlad
Blooks
Standard
0.0
4ms)
.05
.1
0.5
ARP (ms)
(ms)
Figure 5: Fitting cortical electrophysiological recordings. An illustration of a cortical neuron mouse V1 being recorded whilst stimulated with noisy current injection. For held-out data not used for fitting, an example current injection (bottom) and recorded membrane potential (middle) with corresponding fitted model predictions (top). c. Comparison of neuron fit accuracy our model for DT= ms (y-axis) against DT= 4ms (X-axis). Explained temporal variance (ETV) measures the goodness-of-fit (one perfect fit and zero chance-level fit).. Comparison of the fit accuracy (left) and duration (right) of our model and the standard model (both using DT= 0.lms and ARP= 2ms). e. Our model's s fit accuracy for increasing ARP (with DT= 0. 1ms). f. Our model's 's fit accuracy for increasing DT (with ARP=max(2, DT)ms), d. to f. plots the median and standard error over neurons, except that d. (right) plots the mean fit time.
the van Rossum distance between the model and real spike trains 71.. To quantitatively compare the fits of our model using different DTs and ARPs, employed the explained temporal variance (ETV) measure (used on the Allen Institute website; see Supplementary material). This metric quantifies how well the temporal dynamics of the neurons are captured by the model and takes into account trial-to-trial variability due to intrinsic neural noise. has a value one for a perfect fit and a value of zero if the model predicts at chance.
We found that our accelerated ALIF model captured the spike timing of the neurons. Pertinent to the speed-accuracy trade-off, this fit strongly depended on the chosen DT, and less the chosen ARP. Qualitatively, using DT= 0. ms and an ARP- 2ms, we found our model captured the spike timings of the neurons for current injections of varying amplitude. This still seemed to be the case when we used a larger ARP= Sms, but the model was worse at capturing spike timings when we used a larger DT= 4ms (with ARP= 4ms; Figure 5b; see Supplementary material for zoomed-in neural traces).
Quantitatively comparing the neuron fits one by one, we found that nearly all of the neuron fits were better when using DT= 0. 0.1ms (ETV= 0.80) than DT= 4ms (ETV= 0.66; Figure 5:: using an ARP= 4ms). We examined how accurate and fast our fits were compared to the standard model using an ARP- 2ms and a DT= 0s.. Both models achieved a similar ETV
0.8, yet our model only required 15.5s on average per neuron fit time compared to the 108.4s of the standard model (Figure 5d).. We further investigated whether a larger ARP could still reasonably the data for DT= ms (to benefit from faster fit). Consistent with qualitative observations,
found less marked reduction in fit accuracy when using larger ARPs (Figure 55) compared to the drop in performance when using larger DTs (Figure 5 using ARP of max (2, ms).s).
5 Discussion
ALIF neurons are popular model for studying the brain in silico. Training these neurons is, however, slow due to their sequential nature [4 (22. We overcome this by algorithmically reinterpreting the ALIF model. We found that method permits faster simulation of SNNs, which will likely

---

play an important role in future research into neural dynamics through simulation [7] 16]. We also confirmed the validity of this approach for modelling neural data. Firstly of interest to computational neuroscientists we fitted a multilayered network of neurons using two common spike classification datasets. We found that our model achieved a similar classification accuracy to that of the standard model, even if we increased the ARP to large non- ponyysssiollgical values, which drastically reduced the training duration required for both datasets. Secondly of interest to computational and experimental neuroscientists
explored the applicability of our method to fit real electrophysiologcal recordings of neurons in mouse VI using sub-millisecond DTs.
We found that our method accurately captured the spike timing of real neurons. fitting their activity patterns in fraction of the time required by the standard model (although we note that other, more pikiing-odelss recent spiking- models, might improve fit accuracies (73)). This is particularly important as datasets become larger with advances in recording techniques, requiring faster fitting solutions to characterise computational properties of neurons. As an example of the potential insights provided by our model, we found the fitted V1 neurons to have heterogenous membrane time constant distribution, suggesting that VI processes visual information on multiple timescales 42
(see Supplementary
material).
Our work will likely also be of interest to neuromorphic engineers and researchers, developing energy- efficient hardware emulate SNNs 20 These systems like real neurons run continuous time and thus require training on extremely fine DTss off-chip [74] 775. Our method could help to accelerate off-chip training times. Furthermore, the ARP hyperparameter in our model can limit high spike rates and thus reduce energy consumption on neuromorphic systems, as this consumption scales approximately proportionally with the number of emitted spikes [76].
A limitation of our method is that the training speedup scales sublinearly as opposed to linearly with an increasing ARP simulation length (see Section and 4.I]. This is likely due to GPU overheads and employed cudnn routines (77). which further improvements to our code implementation could define the ARP hyperparameter, whose
overcome. An additional limitation is the requirement value influences the training speed and test accuracy in our method and relates to biological realism. However, we found beneficial trade-off by using large non-biological ARPs on the artificial spiking classification dataset and small physiological ARPs for the neural fits (although we found larger values to also perform reasonably well) the model achieved comparable accuracy to the standard ALIF model in both cases, while also having greatly increased speed of training and simulation. In particular, the capacity of our model to accurately and quickly fit the data from real neurons is crucial in terms of the biological applicability of this approach.
Acknowledgments and Disclosure of Funding
We thank Rob Pratt and anonymous reviewers for helpful discussions; and Lorenzo Mazzaschi for feedback the manuscript. Luke Taylor was supported by the Clarendon Fund. Andrew King and Nicol Harper were supported by the Wellcome Trust (WT108369772201577) Figure 3 was created with BioRender.com.
References
[1] Nicol S Harper, Oliver Schoppe, Ben DB Willmore, Zhanfeng Cui, Jan WH Schnupp, and Andrew J King. Network receptive field modeling reveals extensive integration and multi-feature selectivity in auditory cortical neurons. PLoS Computational Biology, 12(1 1)=e1005113, 2016.
[2] Yosef Singer, Luke Taylor, Ben DB Willmore, Andrew J King, and Nicol S Harper. Hierarchical temporal prediction captures motion processing along the visual pathway. eLife, 12:e52599, 2023.
[3] Santiago A Cadena, George H Denfield, Edgar Y Walker, Leon A Gatys, Andreas S Tolias, Matthias Bethge, and Alexander S Ecker. Deep convolutional models improve predictions of macaque vI responses to natural images. PLoS Computational Biology, (5(4):100688) 2019.
[4] Andrew Francl and Josh H McDermott. Deep neural network models of sound localization reveal how perception is adapted to real-world environments. Nature Human Behaviour, 6(1): 111-133, 2022.
10

---

[5] Daniel LK Yamins and James DiCarlo. Using goal-driven deep learning models to understand sensory cortex. Nature Neuroscience, 19(3):35-3665, 2016.
[6] Shahab Bakhtiari, Patrick Mineault, Timothy Lillicrap, Christopher Pack, and Blake Richards. The functional specialization of visual cortex emerges from training parallel pathways with self-supervised predictive learning. Advances in Neural Information Processing Systems, 34: 25164-25178, 2021.
[7] Patrick Mineault, Shahab Bakhtiari, Blake Richards, and Christopher Pack. Your head is there to move you around: Goal-driven models the primate dorsal pathway. Advances in Neural Information Processing Systems, 34:28757-287771 2021.
[8] Samuel Ocko, Jack Lindsey, Surya Ganguli, and Stephane Deny. The emergence of multiple retinal cell types through efficient coding of natural movies. Advances in Neural Information Processing Systems, 31, 2018
[9] Colin Conwell, David Mayo, Andrei Barbu, Michael Buice, George Alvarez, and Boris Katz. Neural regression, representational similarity, model zoology & neural taskonomy at scale in rodent visual cortex. Advances Neural Information Processing Systems, 34:5590-5607, 2021.
[10] Blake A Richards, Timothy P Lillicrap. Philippe Beaudoin, Yoshua Bengio, Rafal Bogacz, Amelia Christensen, Claudia Clopath, Rui Ponte Costa, Archy de Berker, Surya Ganguli, et al. deep learning framework for neuroscience. Nature Neuroscience, 22(11):171111770, 2019.
[11] Renaud Jolivet, Felix Schürmann, Thomas Berger, Richard Naud, Wulfram Gerstner, and Arnd Roth. The quantitative single-neuron modeling competition. Biological Cybernetics, 99:
417-426, 2008
[12] Ryota Kobayashi, Yasuhiro Tsubo, and Shigeru Shinomoto. Made-to-order spiking neuron model equipped with multi-timescall adaptive threshold. Frontiers in Computational Neuro- science, page 9, 2009.
[13] Cyrille Rossant, Dan FM Goodman, Bertrand Fontaine, Jonathan Platkiewicz, Anna K Magnus- son, and Romain Brette. Fitting neuron models spike trains. Frontiers in Neuroscience, 5:9, 2011.
[14] Skander Mensi, Richard Naud, Christian Pozzorini. Michael Avermann, Carl CH Petersen, and Wulfram Gerstner. Parameter extraction and classification of three cortical neuron types reveals two distinct adaptation mechanisms. Journal of Newrophysiology, 17766):155-17777 2012.
[15] Christian Pozzorini, Richard Naud, Skander Mensi, and Wulfram Gerstner. Temporal whitening by power-law adaptation in neocortical neurons. Nature Neuroscience, 16(7):942-94) 948, 2013.
[16] Sophie Denève and Christian Machens. Efficient codes and balanced networks. Nature Neuroscienc,, 19(3):375-382 2016.
[17] Tim P Vogels, Henning Sprekeler, Friedemann Zenke, Claudia Clopath, and Wulfram Gerst- ner. Inhibitory plasticity balances excitation and inhibition in sensory pathways and memory networks. Science, 13446062):1569-1355 2011.
[18] Basile Confavreux, Friedemann Zenke, Everton Agnes, Timothy Lillicrap, and Tim Vogels. A meta-learning approach to (re) discover plasticity rules that carve desired function into neural network. Advances in Neural Information Processing Systems, 33:16398-16408, 2020.
[19] Lukas Braun and Tim Vogels. Online learning of neural computations from sparse temporal feedback. Advances in Neural Information Processing Systems, 34:16437-16450, 2021.
[20] Timo Wunderlich, Akos F Kungl, Eric Müller, Andreas Hartel, Yannik Stradmann. Syed Ahmed Aamir, Andreas Grübl, Arthur Heimbrecht, Korbinian Schreiber, David Stöckel, et al. Demon- strating advantages of neuromorphic computation: pilot study. Frontiers in Neuroscience, 13: 260, 2019.
[21] Louis Lapicque. Recherches quantitatives sur l'excitation electrique des nerfs traitee comme une polarization. Journal de physiologie et de pathologie générale, 9:620-635 1907.
11

---

[22] Wulfram Gerstner, Werner M Kistler, Richard Naud, and Liam Paninski. Neuronal dynamics: From single neurons to networks and models of cognition. Cambridge University Press, 2014.
[23] Eric R Kandel, James H Schwartz, Thomas M Jessell, Steven Siegelbaum, A James Hudspeth, Sarah Mack, et al. Principles of neural science, Fourth edition. Elsevier, 2000.
[24] Marie Levakova, Lubomir Kostal, Christelle Monsempès, Philippe Lucas, and Ryota Kobayashi. Adaptive integrate-and-Fii model reproduces the dynamics of olfactory receptor neuron re- sponses in moth. Journal of the Royal Society Interface, 16(157):20190246, 2019.
[25] Yuan Zeng, Terrence C Stewart, Zubayer Ibne Ferdous, Yevgeny Berdichevsky, and Xiaochen Guo. Temporal learning with biologically fitted snn models. In International Conference on Neuromorphic Systems 2021, pages 1-8, 2021.
[26] Guillaume Bellec, Darjan Salaj, Anand Subramoney, Robert Legenstein, and Wolfgang Maass. Long short-term memory and courningtto-laane in networks of spiking neurons. Advances in neural information processing systems, 31, 2018.
[27] Guillaume Bellec, Franz Scherr, Anand Subramoney, Elias Hajek, Darjan Salaj, Robert Leg- enstein, and Wolfgang Maass. A solution the learning dilemma for recurrent networks of spiking neurons. Nature Communications, 11(1):1-15, 2020.
[28] Bojian Yin, Federico Corradi, and Sander M Bohté. Effective and efficient computation with caaltiple-timescall spiking recurrent neural networks. In International Conference on Neuromorphic Systems 2020, pages 1-8, 2020.
[29] Bojian Yin, Federico Corradi, and Sander M Bohté. Accurate online training of dynamical spiking neural networks through forward propagation through time, Nature Machine Intelligence, pages 1-10, 2023.
[30] Nicolas Perez-Nieves and Dan FM Goodman. Spiking network initialisation and firing rate collapse. arXiv:2305.08887, 2023.
[31] Bojian Yin, Federico Corradi, and Sander M Bohté. Accurate and efficient time-domain classification with adaptive spiking recurrent neural networks. Nature Machine Intelligence, (10):905-913, 2021.
[32] P Andersen, H Silfvenius, SH Sundberg, On Sveen, H Wigstro, et al. Functional characteristics of unmyelinated fibres in the hippocampal cortex. Brain research, 144(1):11-18, 1978.
[33] Michael Avissar, John H Wittig, James C Saunders, and Thomas D Parsons. Refractoriness enhances temporal coding by auditory nerve fibers. Journal of Neuroscience, 33698),768117669 2013.
[34] Edmund T Rolls. Absolute refractory period of neurons involved in mfb self-stimulation. Physiology & Behavior, 7(3):111-315, 1971.
[35] Jean-Sébastien Jouhanneau, Jens Kremkow, Anja Dorm, and James FA Poulet. In vivo monosynaptic excitatory transmission between layer cortical pyramidal neurons. Cell reports, (5(10):208821000 2015.
[36] David E Rumelhart, Geoffrey E Hinton, and Ronald Williams. Learning representations by back-propagating errors. Nature, 323(6888) 1986.
[37] Steven K Esser, Paul A Merolla, John V Arthur, Andrew S Cassidy, Rathinakumar Appuswamy, Alexander Andreopoulos, David J Berg, Jeffrey L McKinstry, Timothy Melano, Davis R Barch, al. Convolutional networks for fast, energy-efficient neuromorphic computing. Proceedings of the National Academy of Sciences, 113(41)114441-1144 2016.
[38] Eric Hunsberger and Chris Eliasmith. 151.v:1510.08882, 2015.
Spiking deep networks with lif neurons.
[39] Friedemann Zenke and Surya Ganguli. Superspike: Supervised learning in multilayer spiking neural networks. Neural Computation, 1046)1111- 2018.
12

---

[40] Jun Haeng Lee, Tobi Delbruck, and Michael Pfeiffer. Training deep spiking neural networks using backpropagation. Frontiers Neuroscience, 10:508, 2016.
[41] Emre O Neftci, Hesham Mostafa, and Friedemann Zenke. Surrogate gradient learning in spiking neural networks: Bringing the power of gradient-based optimization spiking neural networks. IEEE Signal Processing Magazine, 36(6):51-63) 2019.
[42] Nicolas Perez-Nieves, Vincent CH Leung, Pier Luigi Dragotti, and Dan FM Goodman. Neural heterogeneity promotes robust learning. Nature communications, 12(1):1-9, 2021.
[43] Peter O'Connor, Daniel Neil, Shih- Chii Liu,, Tobi Delbruck, and Michael Pfeiffer. Real-time classification and sensor fusion with spiking deep belief network. Frontiers in Neuroscience, 7:178, 2013.
[44] Steve K Esser, Rathinakumar Appuswamy, Paul Merolla, John Arthur, and Dharmendra Modha. Backpropagation for meeryy-efficient neuromorphic computing. Advances in neural information processing systems, 28, 2015.
[45] Bodo Rueckauer, Iulia-Alexandra Lungu, Yuhuang Hu, and Michael Pfeiffer. Theory and tools for the conversion of analog to spiking convolutional neural networks. arXiv:/612.04052, 2016.
[46] Bodo Rueckauer, Conversion of continuous-valuee deep networks efficient event-driven networks for image classification. Frontiers in Neuroscience, 1 11:682, 2017.
Lungu, Yuhuang Hu, Michael Pfeiffer, and Shih-Chii Liu.
[47] Marc-Oliver Gewaltig and Markus Diesmann. Nest (neural simulation tool). Scholarpedia, 2 (4):1430,2) 2007.
[48] Abigail Morrison, Carsten Mehring, Theo Geisel, AD Aertsen, and Markus Diesmann. Advanc- ing the boundaries of high-connectivity network simulation with distributed computing. Neural computation, 177))1177-1881,, 2005.
[49] Abigail Morrison, Sirko Straube, Hans Ekkehard Plesser, and Markus Diesmann. Exact subthreshold integration with continuous spike times in discrete-time neural network simulations. Neural Computation, 19(1):47-79, 2007.
[50] Nicolas Perez-Nieves and Dan Goodman. Sparse spiking gradient descent. Advances Neural Information Processing Systems, 34:1 1195-1198188. 2021.
[51] Ronald J Williams and David Zipser. learning algorithm for continually running fully recurrent neural networks. Neural Computation, 1(2):270-280, 1989.
[52] Anil Kag and Venkatesh Saligrama. Training recurrent neural networks via forward propagation through time. In International Conference on Machine Learning, pages 5189-5200. PMLR, 2021.
[53] James M Murray. Local online learning in recurrent networks with random feedback. Elife, c43299, 2019.
[54] Sander M Bohte, Joost N Kok, and Han La Poutre. Error-backpropagation in temporally encoded networks of spiking neurons. Neurocomputing. 48( 17-37, 2002.
200..
[55] Hesham Mostafa. Supervised learning based temporal coding in spiking neural networks.
IEEE transactions on neural networks and learning systems, 29(7):327-3325, 2017.
[56] Iulia M Comsa, Krzysztof Potempa. Luca Versari, Thomas Fischbacher, Andrea Gesmundo, and Jyrki Alakuijala. Temporal coding in spiking neural networks with alpha synaptic function. In ICASSP 2020-2022 IEEE International Conference Acoustics. Speech and Signal Processing (ICASSP), pages 8529 8533. IEEF, 2020.
[57] Saeed Reza Kheradpisheh and Timothée Masquelier. Temporal backpropagation for spiking neural networks with one spike per neuron. International Journal of Neural Systems, 30(06): 2050027, 2020.
13

---

[58] Malu Zhang. Jiadong Wang, Jibin Wu, Ammar Belatreche, Burin Amompaisannon, Zhixuan Zhang, Venkata Pavan Kumar Miriyala, Hong Qu, Yansong Chua, Trevor E Carlson, al. Rectified linear postsynaptic potential function for backpropagation in deep spiking neural networks. IEEE Transactions on Neural Networks and Learning Systems, 336))1947--1958, 2021.
(59] Shibo Zhou and Xiaohua Li. Spiking neural networks with single-spike temporal-coded neurons for network intrusion detection. In 2020 25th International Conference Pattern Recognition (ICPR), pages 8148-814 -155, IEEE, 2021.
[60] Shibo Zhou, Xiaohua Li, Ying Chen, Sanjeev Chandrasekaran, and Arindam Sanyal. Temporal- coded deep spiking neural network with easy training and robust performance. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 11 143-11151, 2021.
[61] Julian Göltz, Laura Kriener, Andreas Baumbach, Sebastian Billaudelle, Oliver Breitwieser, Benjamin Cramer, Dominik Dold, Akos Ferenc Kungl, Walter Senn, Johannes Schemmel, et al. Fast and energy-efficiee neuromorphic deep learning with first-spike times. Nature Machine Intelligence, 3()):823-835, 2021.
[62] Yaoyu Zhu. Zhaofei Yu, Wei Fang, Xiaodong Xie, Tiejun Huang, and Timothée Masquelier. Training spiking neural networks with event-driven backpropagation. In 36th Conference on Neural Information Processing Systems (NeurIPS 2022), 2022.
[63] Christian Pehle and Jens Egholm Pedersen. Norse A deep learning library for spiking neural networks, January 2021. URL / http//000.221//odd./d/ 4422025 Documenta- tion: //sps///
[64] Wei Fang, Yanqi Chen, Jianhao Ding, Zhaofei Yu, Timothée Masquelier, Ding Chen, Liwei Huang, Huihui Zhou, Guoqi Li, and Yonghong Tian. Spikingjelly; An open-source machine learning infrastructure platform for spike-based intelligence. Science Advances, 9(40) 1480, 2023.
[65] Garrick Orchard, Ajinkya Jayawant, Gregory K Cohen, and Nitish Thakor. Converting static image datasets to spiking neuromorphic datasets using saccades. Frontiers in Neuroscience, 437, 2015.
[66] Benjamin Cramer, Yannik Stradmann, Johannes Schemmel, and Friedemann Zenke. The heidelberg spiking data sets for the systematic evaluation of spiking neural networks. IEEE Transactions on Neural Networks and Learning Systems, 2020.
[67] Sumit B Shrestha and Garrick Orchard. Slayer: Spike layer error reassignment in time, Advances in Neural Information Processing Systems, 31, 2018.
[68] Friedemann Zenke and Tim P Vogels. The remarkable robustness of surrogate gradient learning for instilling complex function in spiking neural networks. Neural Computation, 33(4):899-922 2021.
[69] Ed S Lein, Michael J Hawrylyez, Nancy Ao, Mikael Ayres, Amy Bensinger, Amy Bernard, Andrew F Boe, Mark S Boguski, Kevin S Brockway, Emi J Byrnes, et al. Genome-wide atlas of gene expression in the adult mouse brain. Nature, 445((124)) 176, 2007.
[70] Michael J Hawrylyez, Ed S Lein, Angela L Guillozet-Bongaart, Elaine H Shen, Lydia Ng, Jeremy A Miller, Louie N Van De Lagemaat, Kimberly Smith, Amanda Ebbert, Zackery Riley, al. An anatomically comprehensive atlas of the adult human brain transcriptome. Nature, 489(7416):391-399 2012.
[71] Mark CW van Rossum. A novel spike distance. Neural Computation, 13(4):751-763 2001.
[72] Jason K Eshraghian, Max Ward, Emre O Neftci, Xinxin Wang, Gregor Lenz, Girish Dwivedi, Mohammed Bennamoun, Doo Seok Jeong, and Wei Lu. Training spiking neural networks using lessons from deep learning. Proceedings of the IEEE, 2023.
14

---

[73] Christian Pozzorini, Skander Mensi, Olivier Hagens, Richard Naud, Christof Koch, and Wulfram Gerstner. Automated tighthhroughput characterization of single neurons by means of simplified spiking models. PLoS Computational Biology, 106)))1004275 2015.
[74] Jan Stuijt, Manolis Sifalakis, Amirreaa Yousefzadeh, and Federico Corradi. pbrain: An event-driven and fully synthesizable architecture for spiking neural networks. Frontiers in Neuroscience, page 538, 2021.
[75] Yuming He, Federico Corradi, Chengyao Shi, Ming Ding, Martijn Timmermans, Jan Stuijt, Pieter Harpe, Ilja Ocket, and Yao-Hong Liu. A 28.2 pc neuromorphic sensing system featuring snn-based near-sensor computation and event-driven body-channel communication for insertable cardiac monitoring. In 202/ IEEE Asian Solid-State Circuits Conference (A-SSCC), pages 1-3. IEFE, 2021.
[76] Priyadarshini Panda, Sai Aparn Aketi, and Kaushik Roy. Toward scalable, efficient, and accurate deep spiking neural networks with backward residual connections, stochastic softmax, and hybridization. Frontiers in Neuroscience, 14:653, 2020.
[77] Sharan Chetlur, Cliff Woolley, Philippe Vandermersch, Jonathan Cohen, John Tran, Bryan Catanzaro, and Evan Shelhamer. cudnn: Efficient primitives for deep learning, arXiv:1410.0771 2014.
15


Reviews:
[{"review_id": "NDobQbeT1h", "reviewer": "Reviewer_gA8o", "summary": "Spiking neural networks can be computationally expensive to implement because of their history dependence.  However, spikes can't occur at any time whatsoever; instead, after spiking, there is a refractory period during which they cannot spike again. While this period is ~1-2ms, to attain precise dynamical inferences the typical time step is ~0.1ms.  The authors leverage the former fact, creating a block design, each block length equaling 1 ARP, to substantially reduce the number of required sequential simulation steps while maintaining generally high performance on both ML and neural-fitting tasks.", "questions": " ", "limitations": "Y", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "This appears to be an important and very simple pragmatic algorithm that can benefit the field by greatly reducing SNN simulation times.  The general presentation is clear. ", "weaknesses": "Minor:\n\nFig 5: \n\nb) A timescale bar is needed, as is the ARP used.  Given that there is very little performance change with DT=0.1ms, ARP=8ms (e), it would be helpful to visualize this here as well\n\nd-f) Bar graphs that do not go to zero and have no broken axis can lead to confusion.  One or the other should be done.  I also would advocate for reversing the ordering: f, e, d.  f and e are the most compelling graphs, and f (left) provides an excellent overview comparison to begin with.  d, while relevant, is also not surprising and could be better left as a final, expected finding.  \n\nWording:\n\nLines 266 and 270, \"computational neuroscientists\" and \"biologists.\"  I understand where the authors are coming from; however, both are of interest to computational neuroscientists (who might do theory, computational studies, and/or data analysis), while \"biologists\" is overly broad.  I would suggest simply indicating that both are of interest to computational and experimental neuroscientists, or to change \"biologists\" to \"computational and experimental neuroscientists\".\n\nLine 290:\n\nI'm not sure this is a fair characterization of your impressive findings in Fig 5e.  The larger ARPs had little performance effect in that particular figure, while larger DTs had a large effect.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank you for your positive comments about the strengths of our paper and suggested edits to improve our work.\n\n> Fig 5b): A timescale bar is needed, as is the ARP used. Given that there is very little performance change with DT=0.1ms, ARP=8ms (e), it would be helpful to visualize this here as well\n\nWe have added a timescale bar, the ARP used, and included your requested visualization (see updated Figure 5b).\n\n> Fig 5d-f: Bar graphs that do not go to zero and have no broken axis can lead to confusion. One or the other should be done. I also would advocate for reversing the ordering: f, e, d. f and e are the most compelling graphs, and f (left) provides an excellent overview comparison to begin with. d, while relevant, is also not surprising and could be better left as a final, expected finding.\n\nWe have modified the bar graphs to go to zero and have rearranged the graphs as suggested (and have updated the text accordingly). See updated Figure 5.\n\n> [Wording:] Lines 266 and 270, \"computational neuroscientists\" and \"biologists.\" I understand where the authors are coming from; however, both are of interest to computational neuroscientists (who might do theory, computational studies, and/or data analysis), while \"biologists\" is overly broad. I would suggest simply indicating that both are of interest to computational and experimental neuroscientists, or to change \"biologists\" to \"computational and experimental neuroscientists\".\n\nWe agree and have changed “biologists” to “computational and experimental neuroscientists”.\n\n> [Wording:] Line 290: I'm not sure this is a fair characterization of your impressive findings in Fig 5e. The larger ARPs had little performance effect in that particular figure, while larger DTs had a large effect.\n\nThank you. To still highlight the applicability of our method (in the context of its limitations), we have reworded line 290 to:\n“However, we found a beneficial trade-off, by using large non-biological ARPs on the artificial spiking classification datasets and smaller physiological ARPs for the neural fits (although we found larger values to also perform reasonably well)”\n"}, {"review_id": "rhZMpLYyUj", "reviewer": "Reviewer_x3Zh", "summary": "The authors implement an adaptive leaky integrate-and-fire model in a parallelized way on GPUs by incorporating a fixed refractory period that enables a window of time (‘block’) to be processed in one go. This accelerates the inference and training of spiking neuron models, while retaining accuracy, as shown by the authors.", "questions": "See ‘Weaknesses’.", "limitations": "The limitations are adequately discussed.", "rating": 7, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "The authors provide an impressive array of results for speedups for training and inference of spiking neural networks (SNNs) and real neural recordings. The accuracy does not seem to decrease although a refractory period is introduced. This may have great implications for future inference of SNNs. The paper on the whole is written clearly.", "weaknesses": "It is unclear why the accuracy does not go down in the empirical results presented. If the input signal is not being integrated in the case of the refractory period, then that part of the signal is forever lost. This should naturally lead to a decrease in accuracy by the network. Please elaborate why this is not happening. Is this something specific to the tasks that you are performing, and how is the accuracy impacted in general? Simple simulations with a couple of neurons may help here, while varying the ARP. This may also give some intuition as to how to pick this number to minimize loss of accuracy while still obtaining speedups.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are grateful for your positive comments about the implications of our work.\n\n> It is unclear why the accuracy does not go down in the empirical results presented. \n\nThe accuracy does go down slightly (not quite visible in original Figure 4a, but definitely visible in original Figure 5e.\n\n> If the input signal is not being integrated in the case of the refractory period, then that part of the signal is forever lost. This should naturally lead to a decrease in accuracy by the network. Please elaborate why this is not happening. \n\nThis is happening. If the restriction on the number of spikes becomes too harsh (e.g. due to a large ARP), the performance of the network will decline. This is the reason why multi-spike SNNs tend to outperform single-spike SNNs (in which neurons can spike at most once; Eshraghian et al., 2021). However, during training, our method is still able to learn from the entire input signal (as our implementation permits gradients to flow at every time step in the network, even when the neurons are in a refractory state). Thus, our method can learn which parts of the signal are most important (and which less so) and hence the refractory period only causes a small reduction in accuracy unless it is very large.\n\n> Is this something specific to the tasks that you are performing, and how is the accuracy impacted in general? \n\nWe found we could obtain good accuracies on the machine learning benchmarks using even large ARPs (tens of milliseconds; original Figure 4a), however we had to use smaller ARPs for the neural-fitting task (less than ten milliseconds - which is expected, as you cannot fit arbitrary spike trains with a restricted number of spikes; original Figure 5e). In general, as the ARP tends to infinity, the accuracy of the network would tend to the performance of a single-spike SNN.\n\n> Simple simulations with a couple of neurons may help here, while varying the ARP. This may also give some intuition as to how to pick this number to minimize loss of accuracy while still obtaining speedups.\n\nWe have run additional experiments increasing the ARP on the machine learning benchmarks and the neural-fits, and - confirming your hunch - showcase more of a performance drop in accuracy (see new Figure 4 and 5). We hope this additional information will be helpful to users in balancing the accuracy/speedup tradeoff at large ARPs. \n"}, {"review_id": "QGaIzN66iW", "reviewer": "Reviewer_9z9h", "summary": "The submitted manuscript proposed a method to speedup simulations of adaptive leaky integrate-and-fire (ALIF) neurons on a GPU by using the fact that during the absolutely refractory period ARP, neurons don't exchange any information and thus \"blocks\" on dt/ARP can be parallelized on GPU. This approach is then used to both train ALIF neurons on typical machine learning tasks and to fit individual ALIF neuron parameters to experimentally measured membrane potentials/currents/spike trains from the Allen Institute, with demonstrate a speedup achieved by the algorithm.\n\nOverall, the submission is well-written and the numerical experiments seem to be executed with due diligence. Unfortunately, no code is provided, which makes the claims and simulations not reproducible.\nThe core idea to parallelize the time of the ARP is not new and has been used by other simulators (e.g. the NEST simulator for MPI parallelization as described here: https://nest-simulator.readthedocs.io/en/v2.20.0/guides/running_simulations.html). However, I have not seen it explicitly used for GPU parallelization and training.\nWhile the novelty of the submission seems to be minor, I would still think that it can be a useful contribution to the field once the code is made available and the implementation is benchmarked against existing implementations. Once the code is made available and benchmarks are added, I am willing to improve my score and recommend the paper for publication at NeurIPS.\n\nUPDATE POSTREBUTTAL: \nThe code is now made available.  While the originality of the contribution is somewhat limited, the preliminary benchmarks suggest that it might be a useful contribtion to the spiking network community.", "questions": "* How does the speedup depend on the number of parallel threads and on dt?\n* Put the value of ARF in the caption of Figure 4a. Was it 100 steps, so 10 ms?? How biologically realistic is an ART of 10ms?\n* * Figure 4.3 time axis labels missing! Can you put a zoomed-in version in the supplement with finer time-axis so the artifacts introduced by large ART are easier to see?\n* Figure 5c: does ETV stand for? Please write in the caption, it is only defined in the supplement.\n* figure 5d: What is ETV for Dt=0.05ms? Maybe you have even better performance (and better speedup according to the reasoning above) for smaller dt? ;)\n* Why in 5f is blocks better than standard? shouldn't they be the same?\n* 5e: What happens for shorter ARP?", "limitations": "* No code is provided, therefore the numerical results are unfortunately not reproducible. Also, the code will make the submission substantially more useful for the community.  If code is provided during the review, I will reconsider my score.\n* ALIF is not state-of-the-art (see above).\n* using ARP for parallelization speedup is not strictly new (see above)", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "* The manuscript is written clearly.\n* The manuscript is relevant for both the 'spiking machine learning' community (with the ML tasks) and the neuroscience community with the neuron-fitting examples.\n* The implementation might be helpful for the community, as this trick seems not to be widely used yet.\n* figure 4.2 is great", "weaknesses": "* The core idea seems not to be new: As described above, the idea to parallelize the time of the ARP was used by other simulators (e.g. the NEST simulator for MPI parallelization as described here: https://nest-simulator.readthedocs.io/en/v2.20.0/guides/running_simulations.html). It would be crucial to cite this previous work and/or describe what is new/different here.\n* The NEST parallelization seems to scale close to linear, thus the presented manuscript might even be not state of the art.\n* For the fitting of neuron models, ALIF is not state of the art (see for example Pozzorini et al. for a more recent benchmark of fitting different neuron models 2015 https://doi.org/10.1371/journal.pcbi.1004275). Currently, the manuscript claims \"ALIF neurons have been shown to accurately fit real neural recordings.\" in line 32, and cites a paper from 2008 that is not state-of-the-art. Please reference a more recent paper and mention the caveat that the ALIF is not state-of-the art for fitting neuronal dynamics.\n* The notation O(T) compared to (T/T_R) slightly confusing. It seems to compare quantities with different units (assuming T and T_R have units of time). Does the speedup not also depend on DT? After reading more carefully, I guess what you actually want to claim is that the computational costs scale proportional to T and that during blocks of size T_R/DT no communication between threads has to occur which can lead to a theoretical maximal speedup of min(N_threads,T_R/DT) where is the number of parallel threads of the GPU or CPU.\n\nMinor issues:\nfix latex of N^in", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> Once the code is made available, I am willing to improve my score and recommend the paper for publication at NeurIPS.\n\nThank you! We have uploaded all the code, including instructions to replicate all the experimental findings for the machine learning and neural-fitting benchmarks. In addition, we have included a getting-started notebook to help newcomers with the model. \n\nAddressing your specific concerns and questions:\n\n> The core idea seems not to be new: As described above, the idea to parallelize the time of the ARP was used by other simulators (e.g. the NEST simulator for MPI parallelization [...]). It would be crucial to cite this previous work and/or describe what is new/different here.\n\nThank you for making us aware of this work. We have now cited the NEST library in the related work section of our manuscript (Section 2). A key apparent difference is that our method supports simulation and training by backprop, whereas the NEST simulator seems to just support simulation but not training by backprop. Their speedup technique also seems bound to CPUs, whereas our method supports faster training on GPUs. Also, while NEST does transmit spikes between units at extended intervals, it does not appear to use a block update method for the membrane potential like we do, but rather to update the state of the neurons at every simulation time step.\n\n> The NEST parallelization seems to scale close to linear, thus the presented manuscript might even be not state of the art.\n\nHow scaling of NEST with parallelization (Morrison et al., 2005) relates to the scaling of our model with block length is not immediately apparent. However, our method scales linearly with the block length when simulating (but not training) ALIF SNNs for moderate batch sizes (see Figure 1 in the Supplementary). Regardless, our method is state of the art with respect to substantially speeding up the training of SNNs. \n\n> For the fitting of neuron models, ALIF is not state of the art (see for example Pozzorini et al. for a more recent benchmark of fitting different neuron models 2015 [...]). Currently, the manuscript claims \"ALIF neurons have been shown to accurately fit real neural recordings.\" in line 32, and cites a paper from 2008 that is not state-of-the-art. Please reference a more recent paper and mention the caveat that the ALIF is not state-of-the art for fitting neuronal dynamics.\n\nLIF and ALIF models are very commonly used models of biological neurons. We have included more recent publications showcasing ALIF neuron fits to real neural recordings (in the introduction), and have mentioned the caveat that there are more recent spiking-models available that can be fit to neural responses (in the Discussion), citing Pozzorini et al., 2015. We note that Pozzorini et al., 2015 also does not compare their model with the ALIF model, but with the GLM model.\n\n> The notation O(T) compared to (T/T_R) slightly confusing. It seems to compare quantities with different units (assuming T and T_R have units of time). Does the speedup not also depend on DT? \n\nThe speedup does depend on DT (smaller DT leads to faster speedup). T and T_R are in time steps, not in milliseconds. We have edited the manuscript in places where this is ambiguous and have added a footnote on page 1 to make this explicit.\n\n> After reading more carefully, I guess what you actually want to claim is that the computational costs scale proportional to T and that during blocks of size T_R/DT no communication between threads has to occur which can lead to a theoretical maximal speedup of min(N_threads,T_R/DT) where is the number of parallel threads of the GPU or CPU.\n\nThis is correct (assuming all the computation of a block can run on a single thread).\n\n> How does the speedup depend on the number of parallel threads and on dt?\n\nAddressed in the last two answers (although we can expand more if you like).\n\n> Put the value of ARF in the caption of Figure 4a. Was it 100 steps, so 10 ms?? \n\nWe have now defined the correspondence between simulation step and DT in the Figure caption (N-MNIST 1ms=1 sim. step; SHD 2ms=1 sim. step). See new Figure 4.\n\n> How biologically realistic is an ART of 10ms?\n\nNot very, neurons usually have an ARP of 1 - 2ms (see line 86), as we use in Figure 5 for modelling real neurons.\n\n> Figure 4.3 time axis labels missing! Can you put a zoomed-in version in the supplement with finer time-axis so the artifacts introduced by large ART are easier to see?\n\nAs requested, we have now added a time axis (see new Figure 5b) and added a zoomed-in version in the supplement.\n\n> Figure 5c: does ETV stand for? Please write in the caption, it is only defined in the supplement.\n\nDone. ETV stands for explained temporal variance (also defined on line 242).\n\n> figure 5d: What is ETV for Dt=0.05ms? Maybe you have even better performance (and better speedup according to the reasoning above) for smaller dt? ;)\n\nWe have run this additional experiment. We did not see an improvement in performance (see new Figure 5f). However, as expected, there was a drastic change in speedup between our method and the standard method, from x6.9 (using DT=0.1ms) to x12.9 (using DT=0.05ms).\n\n> Why in 5f is blocks better than standard? shouldn't they be the same?\n\nIn response to a request by reviewer #1, this is now Figure 5d. Exchanging learned weights between either model would result in the same performance. However, slight differences can arise during training, due to different random weight initialization, and potentially due to differences in the computational graphs between the methods (resulting in different gradients).\n\n> 5e: What happens for shorter ARP?\n\nWe have run the additional experiment and found a shorter ARP to slightly increase performance (see new Figure 5e).\n\n> Minor issues: fix latex of N^in\n\nFixed (assuming this was in the caption of Table 1).\n\n"}, {"review_id": "FTIa0TebQP", "reviewer": "Reviewer_Fo7F", "summary": "This paper focus on addressing the speed-accuracy tradeoff in adaptive spiking neurons. They utilize the existence of the absolute refractory period (ARP), and the fact that a neuron can spike at most once within this period, which could reduce the complexity to O(1), and develop an algorithmic reformulation of the adaptive ALIF model. Their model is simulated in blocks of time, instead of simulating network dynamics step by step. Their model achieves ~40x speed up in inference and ~53x speed up in training without the sacrificing the accuracy. And the model is also tested on real electrophysiological recording. ", "questions": "1. How does this method could be generalized to standard spiking neurons models (leaky integrate-and-fire model without adaptive firing threshold, Hodgkin–Huxley model, etc.)?\n2. How does the speed compared to continuous neural dynamical models such as neural ODE?\n3. What's the additional overhead time and memory cost to use this algorithm?\n ", "limitations": "1. No potential societal impact of the work.\n2. The ARP hyperparameters is not addressed, an automatically tuning algorithm might be potentially helpful.", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. Motivation:\nThe paper is well-motivated, and addressing the speed-accuracy tradeoff in the biological simulation is an important question. And using the fact that neuron can spike at most once within the absolute refractory period (ARP) to speed up the computation is novel. \n2. Evaluation:\nTwo levels of evaluations were properly designed, one is to train SNN on spiking classification tasks (N-MNIST, SHD datasets), and the other one is to fit real neural recordings. \n3. Result:\nTheir method is mathematically proved that significantly improve the speed without sacrificing the accuracy, and it's also evaluated by multiple experiments at the same time.", "weaknesses": "1. Generalization: One major weakness is that the proposed algorithm using the sparse firing in ARP to speed up the computation is quite limited to one particular type pf SNN, and hard to be generalized to other variants of neural network simulation.\n2. Method: The method's performance depends on the value of hyper parameters ARP and DT, and their values are manually tuned for different experiments, which is not practical for real applications, where the ARP is typically unknown and varied across neurons. The method will be more efficient if these hyperparameters could be automatically optimized and learned.\n2. Evaluation: No additional baseline was evaluated and compared under the same experiments. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> Motivation: The paper is well-motivated, and addressing the speed-accuracy tradeoff in the biological simulation is an important question. And using the fact that neuron can spike at most once within the absolute refractory period (ARP) to speed up the computation is novel.\n\nThank you for acknowledging the importance of our work and the novelty of our solution.\n\n> Generalization: One major weakness is that the proposed algorithm using the sparse firing in ARP to speed up the computation is quite limited to one particular type pf SNN, and hard to be generalized to other variants of neural network simulation.\n\nWe respectfully disagree. Extending our work to other variants of neural network simulation is indeed an exciting avenue for future research, but this goes beyond the scope of the present study. We have managed to accelerate one of the most commonly used spiking models: the ALIF (and as a consequence the LIF) model. These models have been instrumental to computational neuroscientists and neuromorphic engineers, and will thus be of great interest to many researchers. We also show that the sparseness - as a result of the ARP - is not necessarily a limiting factor, with our method  achieving good performance on both the ML benchmarks (original Figure 4) and the neural fits (original Figure 5). We think this represents a significant advance (as noted in the other reviews).\n\n> Method: The method's performance depends on the value of hyper parameters ARP and DT, and their values are manually tuned for different experiments, which is not practical for real applications, where the ARP is typically unknown and varied across neurons. The method will be more efficient if these hyperparameters could be automatically optimized and learned.\n\nThe DT hyper-parameter has to be defined irrespective of the method used, and not knowing its optimal values or having an automated optimization method is not a shortcoming of our work alone, but applies more generally to all SNN methods. Researchers currently use known and tried values, like 0.1ms and 1ms DT for neural fits and machine learning benchmarks respectively (Perez-Nieves and Goodman, 2023). The ARP is not commonly used in SNN literature, but it is known to be approximately 1-2ms in real neurons (see line 86 for references). We also experimentally showed that extending the ARP to large values results in negligible degradation in performance accuracy on the ML benchmarks (original Figure 4) and in the neural fits (original Figure 5), thus demonstrating the robustness of our method to this hyper-parameter. We agree that learning the ARP hyper-parameter would be impactful to SNN research as a whole.\n\n> Evaluation: No additional baseline was evaluated and compared under the same experiments.\n\nCould you please be more specific, and we can try to address any additional baselines you have in mind? We have evaluated our method on well-known machine learning benchmarks and have explored its applicability to fitting real neural recordings, which we would argue to be a respectable amount of experimental validation (especially in the context of the novel theoretical contributions of our work).\n\n> How does this method could be generalized to standard spiking neurons models (leaky integrate-and-fire model without adaptive firing threshold, Hodgkin–Huxley model, etc.)?\n\nThe ALIF is a generalisation of the LIF and thus all our work also applies to the LIF (just set $d_i^{(l)}=0$ in Equation 5 to see why). The Hodgkin–Huxley (HH) model is more complex than the LIF and ALIF models (with the neuron dynamics defined by a set of ordinary differential equations). Future work might be able to apply aspects of our work to accelerate the HH model; however this is beyond the scope of our submission. \n\n> How does the speed compared to continuous neural dynamical models such as neural ODE?\n\nThis comparison is not appropriate, as neural ODE models - as far as we are aware of - are non-spiking, and the premise of our work is to accelerate spiking neural networks.\n\n> What's the additional overhead time and memory cost to use this algorithm?\n\nOur method introduces more computation (requiring more memory), but reduces sequential processing for increasing ARP (requiring less memory). We found our method to actually require less memory on the neural fitting task (using an ARP=2ms and a DT=0.05ms) run on a NVIDIA GeForce RTX 3090  (our method: 1579MB vs standard method: 1699MB). We acknowledge that memory requirements will differ for different GPUs and different block lengths.\n\n> The ARP hyperparameters is not addressed, an automatically tuning algorithm might be potentially helpful.\n\nThis concern has been addressed above. As you suggest, a tuning algorithm might potentially be helpful, but it is known that the ARP is approximately 1-2ms in real neurons (see line 86 for references) and we show here that increasing this to non-biological values has little impact on the performance of our method when tested against either ML benchmarks or real neural recordings. We do not think that the absence of an automatic tuning algorithm is a major limitation or detracts from our novel contribution of describing a new accelerated training method, vetted on ML and neural-fitting tasks, which should be of interest to many SNN researchers.\n"}]

Author rebuttal:
Dear reviewers,

Thank you for taking the time to review our paper.  As requested, we have run additional experiments, and have appropriately updated Figures 4 and 5, which we have uploaded in the attached single-page PDF. Furthermore, we have included a zoomed-in Supplementary plot of Figure 5b of the neural traces. Lastly - as requested by reviewer 9z9h - we have made all the code available (which we have shared with the AC as set out in the NeurIPS guidelines).

Meta-review: The authors propose a method to significantly increase the simulation time for spiking neural networks (SNNs) based on the adaptive leaky-integrate-and-fire (ALIF) neuron model on GPUs. To do so, they utilize the refractory period of neurons in which no spike of the neuron can appear. The method is evaluated on standard SNN machine learning benchmarks and biologically motivated simulations.

All reviewers agree that the manuscript is well-written and that the manuscript presents an interesting contribution to the SNN simulation community. The achieved simulation and training speedups are impressive (up to 50x) with little influence on network performance.
The initial submission did not compare to other SoTA simulations. This was performed in the rebuttal phase, with convincing results. 

One weakness of the manuscript is that the originality is somewhat limited as the general idea has been implemented previously for SNN simulation. One advance is that the proposed approach is used also for training and on GPUs.

The evaluations somewhat and the manuscript is close-to-borderline. Nevertheless, the very good results suggest that the work can be accepted for NeurIPS.
Decision: Accept (poster)