Title: Addressing the speed-accuracy simulation trade-off for adaptive spiking neurons
Venue: NeurIPS 2023
Accepted: yes
Keywords: spiking neural network

surrogate gradient descent

adaptive leaky integrate and fire neuron

speed-accuracy trade-off

electrophysiological recordings
Abstract: The adaptive leaky integrate-and-fire (ALIF) model is fundamental within computational neuroscience and has been instrumental in studying our brains $\textit{in silico}$. Due to the sequential nature of simulating these neural models, a commonly faced issue is the speed-accuracy trade-off: either accurately simulate a neuron using a small discretisation time-step (DT), which is slow, or more quickly simulate a neuron using a larger DT and incur a loss in simulation accuracy. Here we provide a solution to this dilemma, by algorithmically reinterpreting the ALIF model, reducing the sequential simulation complexity and permitting a more efficient parallelisation on GPUs. We computationally validate our implementation to obtain over a $50\times$ training speedup using small DTs on synthetic benchmarks. We also obtained a comparable performance to the standard ALIF implementation on different supervised classification tasks - yet in a fraction of the training time. Lastly, we showcase how our model makes it possible to quickly and accurately fit real electrophysiological recordings of cortical neurons, where very fine sub-millisecond DTs are crucial for capturing exact spike timing.

Addressing the speed-accuracy simulation trade-off for adaptive spiking neurons

Abstract

The adaptive leaky integrate-and-fire (ALIF) model is fundamental within computational neuroscience and has been instrumental in studying our brains in silico . Due to the sequential nature of simulating these neural models, a commonly faced issue is the speed-accuracy trade-off: either accurately simulate a neuron using a small discretisation time-step (DT), which is slow, or more quickly simulate a neuron using a larger DT and incur a loss in simulation accuracy. Here we provide a solution to this dilemma, by algorithmically reinterpreting the ALIF model, reducing the sequential simulation complexity and permitting a more efficient parallelisation on GPUs. We computationally validate our implementation to obtain over a 50 × training speedup using small DTs on synthetic benchmarks. We also obtained a comparable performance to the standard ALIF implementation on different supervised classification tasks - yet in a fraction of the training time. Lastly, we showcase how our model makes it possible to quickly and accurately fit real electrophysiological recordings of cortical neurons, where very fine sub-millisecond DTs are crucial for capturing exact spike timing.

1 Introduction

The surge of progress in artificial neural networks (ANNs) over the last decade has advanced our understanding of the potential computational principles underlying the processing of sensory information [1-9]. Although these networks architecturally bear a resemblance to the brain [10], they tend to omit a key physiological constraint: the spike. With their increased biological realism, spiking neural networks (SNNs) have shown great promise in bridging the gap between experimental data and computational models. SNNs can be fitted to real neural data [11-15], or used to simulate the

brain, offering a new level of understanding of the complex workings of the nervous system [16-19]. They also have engineering applications in energy-efficient machine learning [20].

An established class of spiking models in computational neuroscience is the leaky integrate-and-fire (LIF) neuron, with origins dating back to 1907 [21]. Just like in real neurons, input current (resulting from presynaptic input) charges the membrane potential of the model neurons, which then output binary signals ( i.e. spikes) as a form of communication (Figure 1a). The adaptive leaky integrateand-fire (ALIF) model [22] is a modern extension of the LIF. It more closely mimics the biology, capturing a key property of neurons, which is their adaptive firing threshold ( i.e. spikes become less frequent in response to a steady input current [23]). ALIF neurons have been shown to accurately fit real neural recordings [11, 22, 24, 25] and to outperform the simpler LIF neurons on various machine learning benchmarks [26-29].

Despite these modelling advances, a major shortcoming of LIF and ALIF neurons is their slow inference and training times. Unlike real neurons, modelling neuron dynamics involves sequential computation over discretised time. This leads to a problematic trade-off between speed and accuracy when simulating SNNs. A small DT enables accurate modelling of dynamics, but is slow to stimulate and train on computer systems such as GPUs. A large DT obtains less accurate dynamics, but at the benefit of being faster to simulate and train [30] (Figure 1b). This raises the important question of capturing the best of both worlds: is there a way to accelerate the inference and training of spiking LIF and ALIF neurons without sacrificing simulation accuracy?

In this work, we address the speed-accuracy trade-off when simulating and training LIF and ALIF SNNs, and present a solution that is both fast and accurate. We take advantage of a fundamental property of neurons that is sometimes not modelled - the absolute refractory period (ARP). This is a short period of time following spike initiation, during which a neuron cannot fire again. As a result, a neuron can spike at most once within such a period (Figure 1c). We leverage this observation to develop a novel algorithmic reformulation of the ALIF model which reduces the sequential simulation complexity of the standard ALIF algorithm. Specifically, we outline how ALIF recurrent networks can be simulated with a constant sequential complexity O (1) over the ARP simulation length T R , and how this approach can be extended to longer simulation lengths to obtain identical dynamics to the standard ALIF network algorithm. Faster simulation and training are theoretically obtained for growing length T R , by either employing physiologically plausible ARPs of ∼ 2 ms and decreasing the DT to a very fine value ∼ 0 . 1 ms (for realistic neural modelling), or setting the DT to a coarser value and adopting larger non-physiological ARPs (for machine learning tasks). Our main contributions are the following:

We develop a novel algorithmic reformulation of the ALIF model with an O ( T/T R ) - rather than O ( T ) - sequential complexity, for simulation length T and ARP length T R . 1

We find that our model achieves substantial inference (up to 40 × ) and training speedup (up to 53 × ) over the standard ALIF SNN for increasing ARP time steps, and find this to hold over different numbers of simulation lengths, batch sizes, number of neurons and layers.

We demonstrate the feasibility of our model to be trained using surrogate gradient descent, with our accelerated ALIF SNN achieving comparable accuracies to the standard ALIF SNN on temporal spiking classification datasets - yet in a fraction of the training time.

Finally, we showcase how our ALIF implementation makes it possible to quickly fit real electrophysiological recordings of cortical neurons, where very fine sub-millisecond discretisation steps are important for accurately capturing exact spike timing.

10

- 1

10

0

10

1

DT (ms)

100

200

Firing rate (Hz)

accurate

&

slow

inaccurate

&

fast

Error

Expected

V

S

I

0

2

4

6

Time (ms)

V

ARP

ARP

T

R

T

R

T

R

a

b

c

Internal state

2 Background and related work

Standard ALIF model We introduce the recurrent SNN of ALIF neurons with fixed ARP and latency of recurrent transmission, defined by the following set of equations [26, 31].

At time t , neuron i within layer l (consisting of N ( l ) neurons) receives input current I ( l ) i [ t ] and outputs a binary value S ( l ) i [ t ] ∈ { 1 , 0 } ( i.e. spike or no spike) if a neuron's membrane potential V ( l ) i [ t ] reaches firing threshold θ ( l ) i (Equation 1). 2 The evolution of the membrane potential is described by the normalised discretised LIF equation (Equation 2), in which the membrane potential dissipates by a learnable factor 0 ≤ β ( l ) i ≤ 1 at every time step and resets to zero if a spike occurred at the previous time step. The input current is comprised of a constant bias source b ( l ) i and from incoming spikes reflecting feedforward W ( l ) ∈ R N ( l ) × N ( l -1) and recurrent connectivity W rec ( l ) ∈ R N ( l ) × N ( l ) (Equation 3).

Here, we assume the recurrent transmission latencies D to be of fixed length and equal in length to the ARP, that is T R = D . With a simple modification however, our methods can work for longer D , or different D on each connection, so long as D ≥ T R . The ARP is enforced by only allowing input current to enter the neuron if the number of time steps C ( l ) i following the last spike equals or exceeds the ARP length (Equation 4). Lastly, adaptation is implemented by raising the firing threshold θ ( l ) i [ t ]

following each spike S ( l ) i [ t -1] (Equation 5), which decays exponentially to baseline θ ( l ) i = 1 in the absence of any spikes (using learnt decay factor 0 ≤ p ( l ) i ≤ 1 and adaptation scalar d ( l ) i ).

The ARP in biological neurons is typically about ∼ 1 -2 ms [32-34]. Our method takes advantage of the fact that the monosynaptic connection latency between neurons in local circuits is typically also often around ∼ 1 -2 ms [35]. LIF and ALIF neuronal models are typically run with a DT of about ∼ 0 . 1 ms in the computational neuroscience literature and ∼ 1 ms in machine learning literature [30]. Furthermore, the firing rate of neurons is often less than the reciprocal of the ARP, suggesting that a higher T R than the ARP may still provide a reasonable approximation of neural behaviour for some purposes.

SNN training The main problem with training SNNs is the non-differentiable nature of their activation function in Equation 1 (whose derivative is undefined at V ( l ) i [ t ] = θ ( l ) i [ t ] and zero otherwise). This precludes the direct use of the backprop algorithm [36], which has underpinned the successful training of ANNs. A popular solution is to replace the undefined spike derivative with a well-behaved function, referred to as a surrogate gradient [37-40], and training the network with backprop-through-time [41]. This method also supports the training of neural parameters other than synaptic connectivity, such as membrane time constants [42, 28] and adaptive firing thresholds [26-28], both of which we include in our work, as they have been shown to improve performance (and increase biological realism). It is worth mentioning that other SNN training methods exist such as mapping trained ANNs to SNNs by transforming ANN neuron activations to SNN neuron firing rates [43-46]. These methods are, however, of less interest to computational neuroscientists as they discard all temporal spike precision.

Related work Recent work has provided new theoretical insights for increasing the simulation accuracy when employing a larger DT ≥ 1 ms [30]. We are not aware of any work that accelerates the simulation and training times on GPUs when employing a smaller DT ≤ 1 ms (although see the NEST simulation library for simulating SNNs on CPUs [47-49]). There are, however, different methods for speeding up the backward pass ( i.e. how gradients are computed within the SNN), which consequentially speeds up training times. Rather than being viewed as competing approaches, these methods could further augment the speed of our solution. In sparse gradient descent, gradients are only passed through neurons whose membrane potential is close to firing threshold, which can significantly accelerate the backward pass when neurons are mostly silent [50]. Inspired by work on training non-spiking networks [51, 52], other methods completely bypass the backward pass and adjust weights online [53, 27, 29]. Another approach is to propagate gradient information through spikes [54-60], which - similar to the idea of sparse gradient descent - is fast when neurons are mostly silent. This method, however, enforces neurons to spike at most once and can suffer from training instabilities (although see [61, 62]).

3 Theoretical results

We outline a novel reformulation of the ALIF model, which theoretically reduces the sequential simulation complexity from O ( T ) to O ( T/T R ) (for simulation length T and ARP length T R ). Using the observation that a neuron can spike at most once over simulation length T R (equal to the ARP; Figure 1c), we propose simulating network dynamics sequentially in blocks of length T R - as opposed to simulating every time step individually (Figure 2a) - as we show that these blocks can be simulated with a constant sequential complexity O (1) . Our reformulated ALIF model is mathematically the same as the standard ALIF model, but substantially faster to simulate and train. All the proofs for the propositions can be found in the Supplementary material.

3.1 Block: simulating single-spike ALIF dynamics with a constant sequential complexity

A SNN exhibits a sequential dependence due to the spike reset mechanism, where a neuron's membrane potential V i [ t ] can be reset based on its previous output S i [ t -1] . Consequently, the simulation of a SNN necessitates a sequential approach. This restriction can, however, be alleviated if we assume a neuron to spike at most once over a particular simulation length (such as the ARP). The following steps - grouped together into a module referred to as a Block (Figure 2b) - compute

Initial conditions

Output spikes

Input current

b

a

Block

Block

ALIF dynamics (assuming at most one spike) without any sequential operations.

Calculate membrane potentials without reset The first step in the Block (Equation 6) converts input current I i [ t ] to membrane potentials ˜ V i [ t ] without spike reset ( i.e. excluding the reset mechanism in Equation 2). This transformation is achieved using a convolution (Proposition 1), thus avoiding any sequential operations.

Proposition 1. Membrane potentials without spike reset are computed as a convolution ˜ V i [ t ] = ( I i ∗ ˜ β i ) [ t ] between input current I i [ t ] and kernel ˜ β i [ t ] = (1 -β i ) β t i with the initial membrane potential encoded as I i [0] = V i [0] 1 -β i .

Faulty output spikes No-reset membrane potentials ˜ V i [ t ] are mapped to erroneous output spikes ˜ S i [ t ] = f ( ˜ V i [ t ]) (Equation 7) using spike function f (Equation 1). This output can contain more than one spike, but only the first spike complies with the standard model dynamics, due to the omission of the reset mechanism and ARP constraint. Thus, to ensure that the Block only emits a single spike, all spikes succeeding the first spike occurrence are removed using the following steps.

glyph[negationslash]

Latent timing of spikes Erroneous spike output ˜ S i [ t ] is mapped to a latent representation z i [ t ] = φ ( ˜ S i )[ t ] (Equation 8), encoding the timing of spikes. Function φ ( · ) (taking vector ˜ S i as input; Proposition 2) is constructed to map all erroneous spikes ˜ S i [ t ] , besides the first spike occurrence, to a value other than one ( i.e. z i [ t ] = 1 for all t except for the smallest t satisfying ˜ S i [ t ] = 1 if such t exists).

Proposition 2. Function φ ( ˜ S i )[ t ] = ∑ t k =1 ˜ S i [ k ]( t -k +1) acting on ˜ S i ∈ { 0 , 1 } T contains at most one element equal to one φ ( ˜ S i )[ t ] = 1 for the smallest t satisfying ˜ S i [ t ] = 1 (if such t exists).

Correct output spikes Lastly, the correct spike output S i [ t ] = 1 z i [ t ] = 1 is obtained by setting every value in z i [ t ] , besides the value one ( i.e. the first spike), to zero (Equation 9).

3.2 Blocks: simulating ALIF SNN dynamics with a O ( T/T R ) sequential complexity

The standard ALIF SNN model can be reformulated using a chaining of Blocks, which reduces the sequential simulation complexity (as each Block is simulated in O (1) ). For a given ARP of length T R , we observed that a neuron spikes at most once over simulation length T R (Figure 1c). Thus, a simulation length T can be simulated using N = T T R Blocks, each of length T R . 3 Next, we outline how to simulate ALIF dynamics across Blocks to emulate the dynamics of the standard ALIF SNN. We introduce new notation to index Block 1 ≤ n ≤ N using subscript n ( e.g. input current to neuron i simulated in Block n is expressed as I i,n [ t ] ) with time steps indexed between [1 , T R ] (as opposed to [1 , T ] ) within a Block.

Input current and the ARP of a Block The input current in the standard model (Equation 3 and 4) is modified for the Block model (Proposition 3). The feedforward and recurrent current to Block n +1 are derived from the presynaptic and postsynaptic spikes from Block n +1 and Block n respectively. In addition, the ARP is enforced by applying a mask derived from the latent timing of spikes z i,n [ t ] (Equation 8).

Proposition 3. The input current I i,n +1 [ t ] of neuron i simulated in Block n +1 (of length T R ) is defined as follows, and enforces an absolute refractory period of length T R and a monosynaptic transmission latency of D = T R .

Evolving membrane potentials between Blocks Two cases are distinguished to correctly emulate the evolution of the membrane potentials between Blocks (Proposition 4): 1) if neuron i did not spike in Block n ( i.e. max t S i,n [ t ] = 0 ), then its initial membrane potential V i,n +1 [0] in Block n +1 is set to its final membrane potential V i,n [ T R ] in Block n . Otherwise 2), the initial membrane potential is set to zero to emulate a spike reset (and no state needs to be transferred between Blocks as the neuron is in a refractory state).

Proposition 4. The initial membrane potential V i,n +1 [0] of neuron i simulated in Block n +1 (of length T R ) is equal to the last membrane potential in Block n if no spike occurred and zero otherwise.

Evolving adaptive firing thresholds between Blocks The adaptive firing threshold θ i,n +1 [ t ] of neuron i in Block n +1 is derived from the initial adaptive parameter a i,n +1 [0] (Proposition 5). Two cases are distinguished for deriving this parameter: 1) if the neuron did not spike during the previous Block n , this parameter is set to its last value, a i,n [ T R ] , in the prior Block; otherwise 2), the effect of the spike needs to be taken into account, for which the initial adaptive parameter is expressed as p i m ( a s + p -1 i ) . Here, m is the number of time steps remaining in Block n following the spike and a s is the adaptive parameter value at the time of spiking.

Proposition 5. The adaptive firing threshold θ i,n +1 [ t ] of neuron i simulated in Block n +1 (of length T R ) is constructed from the initial adaptive parameter a i,n +1 [0] , which is equal to its last value in the previous Block if no spike occurred, and otherwise equal to an expression which accounts for the effect of the spike on the adaptive threshold.

3.3 Theoretical speedup for simulating ALIF SNNs using Blocks

Simulating an ALIF SNN using our Blocks, rather than the conventional approach, requires fewer sequential operations (Table 1). Although the computational complexity of our Blocks approach is larger than the standard approach, the number of sequential operations is less. If we assume that the sequential steps in both methods are executed in an equal amount of time (as all the non-sequential operations can be run in parallel on a GPU), we obtain a theoretical speedup equal to the length of the ARP T N = T R .

4 Experiments

We evaluated the training speedup of our accelerated ALIF SNN relative to the standard ALIF SNN and explored the capacity of our model to be fitted using different spiking classification datasets and real electrophysiological recordings of cortical neurons. Implementation details can be found in the Supplementary material and the code at https://github.com/webstorms/Blocks .

4.1 Training speedup scales with an increasing ARP

50

100

ARP (steps)

0

10

20

30

40

50

Speedup ×

T

= 512

Batch

32

64

128

50

100

ARP (steps)

0

10

20

30

40

50

T

= 1024

50

100

ARP (steps)

0

10

20

30

40

50

T

= 2048

1

2

3

4

5

Layer

0

10

20

30

40

50

T

= 1024

Neurons

128

256

512

a

b

To determine how much faster our model is, we benchmarked the required training duration (forward and backward pass) of our accelerated ALIF SNN to the standard ALIF SNN for varying ARP and simulation lengths using a synthetic spike dataset, with the task of minimizing the number of final-layer output spikes (see Supplementary material). We found the training speedup of our model to increase for a growing ARP (Figure 3a). This speedup was more pronounced for longer simulation durations ( 53 × speedup for T = 2048 ) than shorter simulation durations ( 36 × speedup for T = 512 ). These speedups were also present when just running the forward pass ( i.e. inference using the network; see Supplementary material). Furthermore, we found the speedup of our model to be robust over varying numbers of neurons and layers (Figure 3b). Lastly, we also found our method

to perform the forward pass more than an order of magnitude faster than other publicly available SNN implementations [63, 64] (see Supplementary material).

4.2 Accelerated training on spiking classification tasks

10

20

30

40

50

ARP (ms)

0

20

40

60

80

100

Accuracy (%)

Standard

20

40

60

80 100

ARP (ms)

0

20

40

60

80

100

10

20

30

40

50

ARP (ms)

0

500

1000

1500

2000

Avg. epoch duration (s)

20

40

60

80 100

ARP (ms)

0

50

100

150

200

250

SHD

N-MNIST

SHD

N-MNIST

a

b

To establish whether our model can learn using backprop with surrogate gradients, and perform on a par with the standard model, we trained our accelerated ALIF SNN and the standard ALIF SNN on the Neuromophic-MNIST (N-MNIST) (using DT = 1 ms) [65] and Spiking Heidelberg Digits (SHD) (using DT = 2 ms) [66] spiking classification datasets (both commonly used for benchmarking SNN performance [40, 67, 29, 42, 50]). The task of the N-MNIST dataset is to classify spike representations of handwritten digits, and the task of the SHD dataset is to classify spike representations of spoken digits. In all experiments we employed identical model architectures consisting of two hidden layers (of 256 neurons each) and an additional integrator readout layer, with predictions taken from the readout neurons with maximal summated membrane potential over time (as commonly done [66, 68, 29, 42, 50]; see Supplementary material). We adopted the multi-Gaussian surrogate-gradient function [29] to overcome the non-differentiable discontinuity of the spike function (although we found that other surrogate-gradient functions also work, see Supplementary material). Furthermore (as suggested by [68] for conventional SNNs), we only permitted surrogate gradients to flow through the non-recurrent connectivity (which significantly improved performance; see Supplementary material).

We trained our model with different non-biological ARPs on each dataset and contrasted the accuracy and training duration to the standard model (with no ARP). We found a favourable trade-off between classification accuracy and training time for our model. Compared to the standard ALIF SNN model, our model achieved a very similar, albeit slightly lower, accuracy on the N-MNIST ( 96 . 97% for ARP= 10 ms vs standard 97 . 71% ) and SHD dataset ( 86 . 07% for ARP= 20 ms vs standard 87 . 48% ), with the accuracy declining only for the largest ARPs tested (Figure 4a). However, our model only required a fraction of the training time, with an average training epoch of 181 s and 20 s for the N-MNIST and SHD datasets, respectively, when using the largest tested ARP, compared to the respective standard model training times of 2034 s and 268 s (Figure 4b).

4.3 Quickly fitting real neural recordings on sub-millisecond timescales

We explored the ability of our model to fit in vitro electrophysiological recordings from 146 inhibitory and excitatory neurons in mouse primary visual cortex (V1) (provided by the Allen Institute [69, 70]). In these recordings, a variable input current was repeatedly injected at various amplitudes into each real neuron, and the resulting alteration in the membrane potential was recorded (Figure 5a). For each neuron, we used half of the recordings for fitting and the other half for testing, and report all the qualitative and quantitative results on the held-out test dataset. Fitting was achieved by minimising

a

b

DT=0.1ms ARP=2ms

Injection

Recording

Our model

c

e

f

noise 1

noise 2

DT=0.1ms ARP=8ms

DT=4ms ARP=4ms

100ms

0.0

0.5

1.0

ETV (DT

ms)

= 4

0.00

0.25

0.50

0.75

1.00

ETV (DT

ms)

= 0.1

.05 .1 0.5

1

2

4

8

DT (ms)

0.0

0.2

0.4

0.6

0.8

1

2

4

6

8

16

ARP (ms)

0.0

0.2

0.4

0.6

0.8

ETV

d

noise 1

noise 2

noise 1

noise 2

Blocks

Standard

0.0

0.2

0.4

0.6

0.8

ETV

Blocks

Standard

0

25

50

75

100

Avg. neuron fit time (s)

the van Rossum distance between the model and real spike trains [71]. To quantitatively compare the fits of our model using different DTs and ARPs, we employed the explained temporal variance (ETV) measure (used on the Allen Institute website; see Supplementary material). This metric quantifies how well the temporal dynamics of the neurons are captured by the model and takes into account trial-to-trial variability due to intrinsic neural noise. It has a value of one for a perfect fit and a value of zero if the model predicts at chance.

We found that our accelerated ALIF model captured the spike timing of the neurons. Pertinent to the speed-accuracy trade-off, this fit strongly depended on the chosen DT, and less so on the chosen ARP. Qualitatively, using a DT = 0 . 1 ms and an ARP = 2 ms, we found our model captured the spike timings of the neurons for current injections of varying amplitude. This still seemed to be the case when we used a larger ARP = 8 ms, but the model was worse at capturing spike timings when we used a larger DT = 4 ms (with ARP = 4 ms; Figure 5b; see Supplementary material for zoomed-in neural traces).

Quantitatively comparing the neuron fits one by one, we found that nearly all of the neuron fits were better when using a DT = 0 . 1 ms (ETV = 0 . 80 ) than a DT = 4 ms (ETV = 0 . 66 ; Figure 5c; using an ARP = 4 ms). We examined how accurate and fast our fits were compared to the standard model using an ARP = 2 ms and a DT = 0 . 1 ms. Both models achieved a similar ETV of ∼ 0 . 8 , yet our model only required 15 . 5 s on average per neuron fit time compared to the 108 . 4 s of the standard model (Figure 5d). We further investigated whether a larger ARP could still reasonably fit the data for DT = 0 . 1 ms (to benefit from a faster fit). Consistent with our qualitative observations, we found a less marked reduction in fit accuracy when using larger ARPs (Figure 5e) compared to the drop in performance when using larger DTs (Figure 5f; using an ARP of max (2 , DT ) ms).

5 Discussion

ALIF neurons are a popular model for studying the brain in silico . Training these neurons is, however, slow due to their sequential nature [41, 72]. We overcome this by algorithmically reinterpreting the ALIF model. We found that our method permits faster simulation of SNNs, which will likely

play an important role in future research into neural dynamics through simulation [17, 16]. We also confirmed the validity of this approach for modelling neural data. Firstly - of interest to computational neuroscientists - we fitted a multilayered network of neurons using two common spike classification datasets. We found that our model achieved a similar classification accuracy to that of the standard model, even if we increased the ARP to large non-physiological values, which drastically reduced the training duration required for both datasets. Secondly - of interest to computational and experimental neuroscientists - we explored the applicability of our method to fit real electrophysiological recordings of neurons in mouse V1 using sub-millisecond DTs.

We found that our method accurately captured the spike timing of real neurons, fitting their activity patterns in a fraction of the time required by the standard model (although we note that other, more recent spiking-models, might improve fit accuracies [73]). This is particularly important as datasets become larger with advances in recording techniques, requiring faster fitting solutions to characterise computational properties of neurons. As an example of the potential insights provided by our model, we found the fitted V1 neurons to have a heterogenous membrane time constant distribution, suggesting that V1 processes visual information on multiple timescales [42] (see Supplementary material).

Our work will likely also be of interest to neuromorphic engineers and researchers, developing energyefficient hardware to emulate SNNs [20]. These systems - like real neurons - run in continuous time and thus require training on extremely fine DTs off-chip [74, 75]. Our method could help to accelerate off-chip training times. Furthermore, the ARP hyperparameter in our model can limit high spike rates and thus reduce energy consumption on neuromorphic systems, as this consumption scales approximately proportionally with the number of emitted spikes [76].

A limitation of our method is that the training speedup scales sublinearly - as opposed to linearly with an increasing ARP simulation length (see Section 3 and 4.1). This is likely due to GPU overheads and employed cudnn routines [77], which further improvements to our code implementation could overcome. An additional limitation is the requirement to define the ARP hyperparameter, whose value influences the training speed and test accuracy in our method and relates to biological realism. However, we found a beneficial trade-off - by using large non-biological ARPs on the artificial spiking classification dataset and small physiological ARPs for the neural fits (although we found larger values to also perform reasonably well) the model achieved comparable accuracy to the standard ALIF model in both cases, while also having greatly increased speed of training and simulation. In particular, the capacity of our model to accurately and quickly fit the data from real neurons is crucial in terms of the biological applicability of this approach.

Acknowledgments and Disclosure of Funding

We thank Rob Pratt and anonymous reviewers for helpful discussions; and Lorenzo Mazzaschi for feedback on the manuscript. Luke Taylor was supported by the Clarendon Fund. Andrew King and Nicol Harper were supported by the Wellcome Trust (WT108369/Z/2015/Z). Figure 5 was created with BioRender.com.

References

Nicol S Harper, Oliver Schoppe, Ben DB Willmore, Zhanfeng Cui, Jan WH Schnupp, and Andrew J King. Network receptive field modeling reveals extensive integration and multi-feature selectivity in auditory cortical neurons. PLoS Computational Biology , 12(11):e1005113, 2016.

Yosef Singer, Luke Taylor, Ben DB Willmore, Andrew J King, and Nicol S Harper. Hierarchical temporal prediction captures motion processing along the visual pathway. eLife , 12:e52599, 2023.

Santiago A Cadena, George H Denfield, Edgar Y Walker, Leon A Gatys, Andreas S Tolias, Matthias Bethge, and Alexander S Ecker. Deep convolutional models improve predictions of macaque v1 responses to natural images. PLoS Computational Biology , 15(4):e1006897, 2019.

Andrew Francl and Josh H McDermott. Deep neural network models of sound localization reveal how perception is adapted to real-world environments. Nature Human Behaviour , 6(1): 111-133, 2022.

Daniel LK Yamins and James J DiCarlo. Using goal-driven deep learning models to understand sensory cortex. Nature Neuroscience , 19(3):356-365, 2016.

Shahab Bakhtiari, Patrick Mineault, Timothy Lillicrap, Christopher Pack, and Blake Richards. The functional specialization of visual cortex emerges from training parallel pathways with self-supervised predictive learning. Advances in Neural Information Processing Systems , 34: 25164-25178, 2021.

Patrick Mineault, Shahab Bakhtiari, Blake Richards, and Christopher Pack. Your head is there to move you around: Goal-driven models of the primate dorsal pathway. Advances in Neural Information Processing Systems , 34:28757-28771, 2021.

Samuel Ocko, Jack Lindsey, Surya Ganguli, and Stephane Deny. The emergence of multiple retinal cell types through efficient coding of natural movies. Advances in Neural Information Processing Systems , 31, 2018.

Colin Conwell, David Mayo, Andrei Barbu, Michael Buice, George Alvarez, and Boris Katz. Neural regression, representational similarity, model zoology & neural taskonomy at scale in rodent visual cortex. Advances in Neural Information Processing Systems , 34:5590-5607, 2021.

Blake A Richards, Timothy P Lillicrap, Philippe Beaudoin, Yoshua Bengio, Rafal Bogacz, Amelia Christensen, Claudia Clopath, Rui Ponte Costa, Archy de Berker, Surya Ganguli, et al. A deep learning framework for neuroscience. Nature Neuroscience , 22(11):1761-1770, 2019.

Renaud Jolivet, Felix Schürmann, Thomas K Berger, Richard Naud, Wulfram Gerstner, and Arnd Roth. The quantitative single-neuron modeling competition. Biological Cybernetics , 99: 417-426, 2008.

Ryota Kobayashi, Yasuhiro Tsubo, and Shigeru Shinomoto. Made-to-order spiking neuron model equipped with a multi-timescale adaptive threshold. Frontiers in Computational Neuroscience , page 9, 2009.

Cyrille Rossant, Dan FM Goodman, Bertrand Fontaine, Jonathan Platkiewicz, Anna K Magnusson, and Romain Brette. Fitting neuron models to spike trains. Frontiers in Neuroscience , 5:9, 2011.

Skander Mensi, Richard Naud, Christian Pozzorini, Michael Avermann, Carl CH Petersen, and Wulfram Gerstner. Parameter extraction and classification of three cortical neuron types reveals two distinct adaptation mechanisms. Journal of Neurophysiology , 107(6):1756-1775, 2012.

Christian Pozzorini, Richard Naud, Skander Mensi, and Wulfram Gerstner. Temporal whitening by power-law adaptation in neocortical neurons. Nature Neuroscience , 16(7):942-948, 2013.

Sophie Denève and Christian K Machens. Efficient codes and balanced networks. Nature Neuroscience , 19(3):375-382, 2016.

Tim P Vogels, Henning Sprekeler, Friedemann Zenke, Claudia Clopath, and Wulfram Gerstner. Inhibitory plasticity balances excitation and inhibition in sensory pathways and memory networks. Science , 334(6062):1569-1573, 2011.

Basile Confavreux, Friedemann Zenke, Everton Agnes, Timothy Lillicrap, and Tim Vogels. A meta-learning approach to (re) discover plasticity rules that carve a desired function into a neural network. Advances in Neural Information Processing Systems , 33:16398-16408, 2020.

Lukas Braun and Tim Vogels. Online learning of neural computations from sparse temporal feedback. Advances in Neural Information Processing Systems , 34:16437-16450, 2021.

Timo Wunderlich, Akos F Kungl, Eric Müller, Andreas Hartel, Yannik Stradmann, Syed Ahmed Aamir, Andreas Grübl, Arthur Heimbrecht, Korbinian Schreiber, David Stöckel, et al. Demonstrating advantages of neuromorphic computation: a pilot study. Frontiers in Neuroscience , 13: 260, 2019.

Louis Lapicque. Recherches quantitatives sur l'excitation electrique des nerfs traitee comme une polarization. Journal de physiologie et de pathologie générale , 9:620-635, 1907.

Wulfram Gerstner, Werner M Kistler, Richard Naud, and Liam Paninski. Neuronal dynamics: From single neurons to networks and models of cognition . Cambridge University Press, 2014.

Eric R Kandel, James H Schwartz, Thomas M Jessell, Steven Siegelbaum, A James Hudspeth, Sarah Mack, et al. Principles of neural science, Fourth edition . Elsevier, 2000.

Marie Levakova, Lubomir Kostal, Christelle Monsempès, Philippe Lucas, and Ryota Kobayashi. Adaptive integrate-and-fire model reproduces the dynamics of olfactory receptor neuron responses in a moth. Journal of the Royal Society Interface , 16(157):20190246, 2019.

Yuan Zeng, Terrence C Stewart, Zubayer Ibne Ferdous, Yevgeny Berdichevsky, and Xiaochen Guo. Temporal learning with biologically fitted snn models. In International Conference on Neuromorphic Systems 2021 , pages 1-8, 2021.

Guillaume Bellec, Darjan Salaj, Anand Subramoney, Robert Legenstein, and Wolfgang Maass. Long short-term memory and learning-to-learn in networks of spiking neurons. Advances in neural information processing systems , 31, 2018.

Guillaume Bellec, Franz Scherr, Anand Subramoney, Elias Hajek, Darjan Salaj, Robert Legenstein, and Wolfgang Maass. A solution to the learning dilemma for recurrent networks of spiking neurons. Nature Communications , 11(1):1-15, 2020.

Bojian Yin, Federico Corradi, and Sander M Bohté. Effective and efficient computation with multiple-timescale spiking recurrent neural networks. In International Conference on Neuromorphic Systems 2020 , pages 1-8, 2020.

Bojian Yin, Federico Corradi, and Sander M Bohté. Accurate online training of dynamical spiking neural networks through forward propagation through time. Nature Machine Intelligence , pages 1-10, 2023.

Nicolas Perez-Nieves and Dan FM Goodman. Spiking network initialisation and firing rate collapse. arXiv:2305.08879 , 2023.

Bojian Yin, Federico Corradi, and Sander M Bohté. Accurate and efficient time-domain classification with adaptive spiking recurrent neural networks. Nature Machine Intelligence , 3 (10):905-913, 2021.

P Andersen, H Silfvenius, SH Sundberg, On Sveen, H Wigstro, et al. Functional characteristics of unmyelinated fibres in the hippocampal cortex. Brain research , 144(1):11-18, 1978.

Michael Avissar, John H Wittig, James C Saunders, and Thomas D Parsons. Refractoriness enhances temporal coding by auditory nerve fibers. Journal of Neuroscience , 33(18):7681-7690, 2013.

Edmund T Rolls. Absolute refractory period of neurons involved in mfb self-stimulation. Physiology & Behavior , 7(3):311-315, 1971.

Jean-Sébastien Jouhanneau, Jens Kremkow, Anja L Dorrn, and James FA Poulet. In vivo monosynaptic excitatory transmission between layer 2 cortical pyramidal neurons. Cell reports , 13(10):2098-2106, 2015.

David E Rumelhart, Geoffrey E Hinton, and Ronald J Williams. Learning representations by back-propagating errors. Nature , 323(6088):533-536, 1986.

Steven K Esser, Paul A Merolla, John V Arthur, Andrew S Cassidy, Rathinakumar Appuswamy, Alexander Andreopoulos, David J Berg, Jeffrey L McKinstry, Timothy Melano, Davis R Barch, et al. Convolutional networks for fast, energy-efficient neuromorphic computing. Proceedings of the National Academy of Sciences , 113(41):11441-11446, 2016.

Eric Hunsberger and Chris Eliasmith. Spiking deep networks with lif neurons. arXiv:1510.08829 , 2015.

Friedemann Zenke and Surya Ganguli. Superspike: Supervised learning in multilayer spiking neural networks. Neural Computation , 30(6):1514-1541, 2018.

Jun Haeng Lee, Tobi Delbruck, and Michael Pfeiffer. Training deep spiking neural networks using backpropagation. Frontiers in Neuroscience , 10:508, 2016.

Emre O Neftci, Hesham Mostafa, and Friedemann Zenke. Surrogate gradient learning in spiking neural networks: Bringing the power of gradient-based optimization to spiking neural networks. IEEE Signal Processing Magazine , 36(6):51-63, 2019.

Nicolas Perez-Nieves, Vincent CH Leung, Pier Luigi Dragotti, and Dan FM Goodman. Neural heterogeneity promotes robust learning. Nature communications , 12(1):1-9, 2021.

Peter O'Connor, Daniel Neil, Shih-Chii Liu, Tobi Delbruck, and Michael Pfeiffer. Real-time classification and sensor fusion with a spiking deep belief network. Frontiers in Neuroscience , 7:178, 2013.

Steve K Esser, Rathinakumar Appuswamy, Paul Merolla, John V Arthur, and Dharmendra S Modha. Backpropagation for energy-efficient neuromorphic computing. Advances in neural information processing systems , 28, 2015.

Bodo Rueckauer, Iulia-Alexandra Lungu, Yuhuang Hu, and Michael Pfeiffer. Theory and tools for the conversion of analog to spiking convolutional neural networks. arXiv:1612.04052 , 2016.

Bodo Rueckauer, Iulia-Alexandra Lungu, Yuhuang Hu, Michael Pfeiffer, and Shih-Chii Liu. Conversion of continuous-valued deep networks to efficient event-driven networks for image classification. Frontiers in Neuroscience , 11:682, 2017.

Marc-Oliver Gewaltig and Markus Diesmann. Nest (neural simulation tool). Scholarpedia , 2 (4):1430, 2007.

Abigail Morrison, Carsten Mehring, Theo Geisel, AD Aertsen, and Markus Diesmann. Advancing the boundaries of high-connectivity network simulation with distributed computing. Neural computation , 17(8):1776-1801, 2005.

Abigail Morrison, Sirko Straube, Hans Ekkehard Plesser, and Markus Diesmann. Exact subthreshold integration with continuous spike times in discrete-time neural network simulations. Neural Computation , 19(1):47-79, 2007.

Nicolas Perez-Nieves and Dan Goodman. Sparse spiking gradient descent. Advances in Neural Information Processing Systems , 34:11795-11808, 2021.

Ronald J Williams and David Zipser. A learning algorithm for continually running fully recurrent neural networks. Neural Computation , 1(2):270-280, 1989.

Anil Kag and Venkatesh Saligrama. Training recurrent neural networks via forward propagation through time. In International Conference on Machine Learning , pages 5189-5200. PMLR, 2021.

James M Murray. Local online learning in recurrent networks with random feedback. Elife , 8: e43299, 2019.

Sander M Bohte, Joost N Kok, and Han La Poutre. Error-backpropagation in temporally encoded networks of spiking neurons. Neurocomputing , 48(1-4):17-37, 2002.

Hesham Mostafa. Supervised learning based on temporal coding in spiking neural networks. IEEE transactions on neural networks and learning systems , 29(7):3227-3235, 2017.

Iulia M Comsa, Krzysztof Potempa, Luca Versari, Thomas Fischbacher, Andrea Gesmundo, and Jyrki Alakuijala. Temporal coding in spiking neural networks with alpha synaptic function. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , pages 8529-8533. IEEE, 2020.

Saeed Reza Kheradpisheh and Timothée Masquelier. Temporal backpropagation for spiking neural networks with one spike per neuron. International Journal of Neural Systems , 30(06): 2050027, 2020.

Malu Zhang, Jiadong Wang, Jibin Wu, Ammar Belatreche, Burin Amornpaisannon, Zhixuan Zhang, Venkata Pavan Kumar Miriyala, Hong Qu, Yansong Chua, Trevor E Carlson, et al. Rectified linear postsynaptic potential function for backpropagation in deep spiking neural networks. IEEE Transactions on Neural Networks and Learning Systems , 33(5):1947-1958, 2021.

Shibo Zhou and Xiaohua Li. Spiking neural networks with single-spike temporal-coded neurons for network intrusion detection. In 2020 25th International Conference on Pattern Recognition (ICPR) , pages 8148-8155. IEEE, 2021.

Shibo Zhou, Xiaohua Li, Ying Chen, Sanjeev T Chandrasekaran, and Arindam Sanyal. Temporalcoded deep spiking neural network with easy training and robust performance. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 35, pages 11143-11151, 2021.

Julian Göltz, Laura Kriener, Andreas Baumbach, Sebastian Billaudelle, Oliver Breitwieser, Benjamin Cramer, Dominik Dold, Akos Ferenc Kungl, Walter Senn, Johannes Schemmel, et al. Fast and energy-efficient neuromorphic deep learning with first-spike times. Nature Machine Intelligence , 3(9):823-835, 2021.

Yaoyu Zhu, Zhaofei Yu, Wei Fang, Xiaodong Xie, Tiejun Huang, and Timothée Masquelier. Training spiking neural networks with event-driven backpropagation. In 36th Conference on Neural Information Processing Systems (NeurIPS 2022) , 2022.

Christian Pehle and Jens Egholm Pedersen. Norse - A deep learning library for spiking neural networks, January 2021. URL https://doi.org/10.5281/zenodo.4422025 . Documentation: https://norse.ai/docs/.

Wei Fang, Yanqi Chen, Jianhao Ding, Zhaofei Yu, Timothée Masquelier, Ding Chen, Liwei Huang, Huihui Zhou, Guoqi Li, and Yonghong Tian. Spikingjelly: An open-source machine learning infrastructure platform for spike-based intelligence. Science Advances , 9(40):eadi1480, 2023.

Garrick Orchard, Ajinkya Jayawant, Gregory K Cohen, and Nitish Thakor. Converting static image datasets to spiking neuromorphic datasets using saccades. Frontiers in Neuroscience , 9: 437, 2015.

Benjamin Cramer, Yannik Stradmann, Johannes Schemmel, and Friedemann Zenke. The heidelberg spiking data sets for the systematic evaluation of spiking neural networks. IEEE Transactions on Neural Networks and Learning Systems , 2020.

Sumit B Shrestha and Garrick Orchard. Slayer: Spike layer error reassignment in time. Advances in Neural Information Processing Systems , 31, 2018.

Friedemann Zenke and Tim P Vogels. The remarkable robustness of surrogate gradient learning for instilling complex function in spiking neural networks. Neural Computation , 33(4):899-925, 2021.

Ed S Lein, Michael J Hawrylycz, Nancy Ao, Mikael Ayres, Amy Bensinger, Amy Bernard, Andrew F Boe, Mark S Boguski, Kevin S Brockway, Emi J Byrnes, et al. Genome-wide atlas of gene expression in the adult mouse brain. Nature , 445(7124):168-176, 2007.

Michael J Hawrylycz, Ed S Lein, Angela L Guillozet-Bongaarts, Elaine H Shen, Lydia Ng, Jeremy A Miller, Louie N Van De Lagemaat, Kimberly A Smith, Amanda Ebbert, Zackery L Riley, et al. An anatomically comprehensive atlas of the adult human brain transcriptome. Nature , 489(7416):391-399, 2012.

Mark CW van Rossum. A novel spike distance. Neural Computation , 13(4):751-763, 2001.

Jason K Eshraghian, Max Ward, Emre O Neftci, Xinxin Wang, Gregor Lenz, Girish Dwivedi, Mohammed Bennamoun, Doo Seok Jeong, and Wei D Lu. Training spiking neural networks using lessons from deep learning. Proceedings of the IEEE , 2023.

Christian Pozzorini, Skander Mensi, Olivier Hagens, Richard Naud, Christof Koch, and Wulfram Gerstner. Automated high-throughput characterization of single neurons by means of simplified spiking models. PLoS Computational Biology , 11(6):e1004275, 2015.

Jan Stuijt, Manolis Sifalakis, Amirreza Yousefzadeh, and Federico Corradi. µ brain: An event-driven and fully synthesizable architecture for spiking neural networks. Frontiers in Neuroscience , page 538, 2021.

Yuming He, Federico Corradi, Chengyao Shi, Ming Ding, Martijn Timmermans, Jan Stuijt, Pieter Harpe, Ilja Ocket, and Yao-Hong Liu. A 28.2 µ c neuromorphic sensing system featuring snn-based near-sensor computation and event-driven body-channel communication for insertable cardiac monitoring. In 2021 IEEE Asian Solid-State Circuits Conference (A-SSCC) , pages 1-3. IEEE, 2021.

Priyadarshini Panda, Sai Aparna Aketi, and Kaushik Roy. Toward scalable, efficient, and accurate deep spiking neural networks with backward residual connections, stochastic softmax, and hybridization. Frontiers in Neuroscience , 14:653, 2020.

Sharan Chetlur, Cliff Woolley, Philippe Vandermersch, Jonathan Cohen, John Tran, Bryan Catanzaro, and Evan Shelhamer. cudnn: Efficient primitives for deep learning. arXiv:1410.0759 , 2014.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper focus on addressing the speed-accuracy tradeoff in adaptive spiking neurons. They utilize the existence of the absolute refractory period (ARP), and the fact that a neuron can spike at most once within this period, which could reduce the complexity to O(1), and develop an algorithmic reformulation of the adaptive ALIF model. Their model is simulated in blocks of time, instead of simulating network dynamics step by step. Their model achieves ~40x speed up in inference and ~53x speed up in training without the sacrificing the accuracy. And the model is also tested on real electrophysiological recording.\n\nSTRENGTHS:\n1. Motivation:\nThe paper is well-motivated, and addressing the speed-accuracy tradeoff in the biological simulation is an important question. And using the fact that neuron can spike at most once within the absolute refractory period (ARP) to speed up the computation is novel. \n2. Evaluation:\nTwo levels of evaluations were properly designed, one is to train SNN on spiking classification tasks (N-MNIST, SHD datasets), and the other one is to fit real neural recordings. \n3. Result:\nTheir method is mathematically proved that significantly improve the speed without sacrificing the accuracy, and it's also evaluated by multiple experiments at the same time.\n\nWEAKNESSES:\n1. Generalization: One major weakness is that the proposed algorithm using the sparse firing in ARP to speed up the computation is quite limited to one particular type pf SNN, and hard to be generalized to other variants of neural network simulation.\n2. Method: The method's performance depends on the value of hyper parameters ARP and DT, and their values are manually tuned for different experiments, which is not practical for real applications, where the ARP is typically unknown and varied across neurons. The method will be more efficient if these hyperparameters could be automatically optimized and learned.\n2. Evaluation: No additional baseline was evaluated and compared under the same experiments.\n\nQUESTIONS:\n1. How does this method could be generalized to standard spiking neurons models (leaky integrate-and-fire model without adaptive firing threshold, Hodgkin–Huxley model, etc.)?\n2. How does the speed compared to continuous neural dynamical models such as neural ODE?\n3. What's the additional overhead time and memory cost to use this algorithm?\n\nLIMITATIONS:\n1. No potential societal impact of the work.\n2. The ARP hyperparameters is not addressed, an automatically tuning algorithm might be potentially helpful.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe submitted manuscript proposed a method to speedup simulations of adaptive leaky integrate-and-fire (ALIF) neurons on a GPU by using the fact that during the absolutely refractory period ARP, neurons don't exchange any information and thus \"blocks\" on dt/ARP can be parallelized on GPU. This approach is then used to both train ALIF neurons on typical machine learning tasks and to fit individual ALIF neuron parameters to experimentally measured membrane potentials/currents/spike trains from the Allen Institute, with demonstrate a speedup achieved by the algorithm.\n\nOverall, the submission is well-written and the numerical experiments seem to be executed with due diligence. Unfortunately, no code is provided, which makes the claims and simulations not reproducible.\nThe core idea to parallelize the time of the ARP is not new and has been used by other simulators (e.g. the NEST simulator for MPI parallelization as described here: https://nest-simulator.readthedocs.io/en/v2.20.0/guides/running_simulations.html). However, I have not seen it explicitly used for GPU parallelization and training.\nWhile the novelty of the submission seems to be minor, I would still think that it can be a useful contribution to the field once the code is made available and the implementation is benchmarked against existing implementations. Once the code is made available and benchmarks are added, I am willing to improve my score and recommend the paper for publication at NeurIPS.\n\nUPDATE POSTREBUTTAL: \nThe code is now made available.  While the originality of the contribution is somewhat limited, the preliminary benchmarks suggest that it might be a useful contribtion to the spiking network community.\n\nSTRENGTHS:\n* The manuscript is written clearly.\n* The manuscript is relevant for both the 'spiking machine learning' community (with the ML tasks) and the neuroscience community with the neuron-fitting examples.\n* The implementation might be helpful for the community, as this trick seems not to be widely used yet.\n* figure 4.2 is great\n\nWEAKNESSES:\n* The core idea seems not to be new: As described above, the idea to parallelize the time of the ARP was used by other simulators (e.g. the NEST simulator for MPI parallelization as described here: https://nest-simulator.readthedocs.io/en/v2.20.0/guides/running_simulations.html). It would be crucial to cite this previous work and/or describe what is new/different here.\n* The NEST parallelization seems to scale close to linear, thus the presented manuscript might even be not state of the art.\n* For the fitting of neuron models, ALIF is not state of the art (see for example Pozzorini et al. for a more recent benchmark of fitting different neuron models 2015 https://doi.org/10.1371/journal.pcbi.1004275). Currently, the manuscript claims \"ALIF neurons have been shown to accurately fit real neural recordings.\" in line 32, and cites a paper from 2008 that is not state-of-the-art. Please reference a more recent paper and mention the caveat that the ALIF is not state-of-the art for fitting neuronal dynamics.\n* The notation O(T) compared to (T/T_R) slightly confusing. It seems to compare quantities with different units (assuming T and T_R have units of time). Does the speedup not also depend on DT? After reading more carefully, I guess what you actually want to claim is that the computational costs scale proportional to T and that during blocks of size T_R/DT no communication between threads has to occur which can lead to a theoretical maximal speedup of min(N_threads,T_R/DT) where is the number of parallel threads of the GPU or CPU.\n\nMinor issues:\nfix latex of N^in\n\nQUESTIONS:\n* How does the speedup depend on the number of parallel threads and on dt?\n* Put the value of ARF in the caption of Figure 4a. Was it 100 steps, so 10 ms?? How biologically realistic is an ART of 10ms?\n* * Figure 4.3 time axis labels missing! Can you put a zoomed-in version in the supplement with finer time-axis so the artifacts introduced by large ART are easier to see?\n* Figure 5c: does ETV stand for? Please write in the caption, it is only defined in the supplement.\n* figure 5d: What is ETV for Dt=0.05ms? Maybe you have even better performance (and better speedup according to the reasoning above) for smaller dt? ;)\n* Why in 5f is blocks better than standard? shouldn't they be the same?\n* 5e: What happens for shorter ARP?\n\nLIMITATIONS:\n* No code is provided, therefore the numerical results are unfortunately not reproducible. Also, the code will make the submission substantially more useful for the community.  If code is provided during the review, I will reconsider my score.\n* ALIF is not state-of-the-art (see above).\n* using ARP for parallelization speedup is not strictly new (see above)", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}

comments: SUMMARY:
The authors implement an adaptive leaky integrate-and-fire model in a parallelized way on GPUs by incorporating a fixed refractory period that enables a window of time (‘block’) to be processed in one go. This accelerates the inference and training of spiking neuron models, while retaining accuracy, as shown by the authors.

STRENGTHS:
The authors provide an impressive array of results for speedups for training and inference of spiking neural networks (SNNs) and real neural recordings. The accuracy does not seem to decrease although a refractory period is introduced. This may have great implications for future inference of SNNs. The paper on the whole is written clearly.

WEAKNESSES:
It is unclear why the accuracy does not go down in the empirical results presented. If the input signal is not being integrated in the case of the refractory period, then that part of the signal is forever lost. This should naturally lead to a decrease in accuracy by the network. Please elaborate why this is not happening. Is this something specific to the tasks that you are performing, and how is the accuracy impacted in general? Simple simulations with a couple of neurons may help here, while varying the ARP. This may also give some intuition as to how to pick this number to minimize loss of accuracy while still obtaining speedups.

QUESTIONS:
See ‘Weaknesses’.

LIMITATIONS:
The limitations are adequately discussed.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2023-07-12

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nSpiking neural networks can be computationally expensive to implement because of their history dependence.  However, spikes can't occur at any time whatsoever; instead, after spiking, there is a refractory period during which they cannot spike again. While this period is ~1-2ms, to attain precise dynamical inferences the typical time step is ~0.1ms.  The authors leverage the former fact, creating a block design, each block length equaling 1 ARP, to substantially reduce the number of required sequential simulation steps while maintaining generally high performance on both ML and neural-fitting tasks.\n\nSTRENGTHS:\nThis appears to be an important and very simple pragmatic algorithm that can benefit the field by greatly reducing SNN simulation times.  The general presentation is clear.\n\nWEAKNESSES:\nMinor:\n\nFig 5: \n\nb) A timescale bar is needed, as is the ARP used.  Given that there is very little performance change with DT=0.1ms, ARP=8ms (e), it would be helpful to visualize this here as well\n\nd-f) Bar graphs that do not go to zero and have no broken axis can lead to confusion.  One or the other should be done.  I also would advocate for reversing the ordering: f, e, d.  f and e are the most compelling graphs, and f (left) provides an excellent overview comparison to begin with.  d, while relevant, is also not surprising and could be better left as a final, expected finding.  \n\nWording:\n\nLines 266 and 270, \"computational neuroscientists\" and \"biologists.\"  I understand where the authors are coming from; however, both are of interest to computational neuroscientists (who might do theory, computational studies, and/or data analysis), while \"biologists\" is overly broad.  I would suggest simply indicating that both are of interest to computational and experimental neuroscientists, or to change \"biologists\" to \"computational and experimental neuroscientists\".\n\nLine 290:\n\nI'm not sure this is a fair characterization of your impressive findings in Fig 5e.  The larger ARPs had little performance effect in that particular figure, while larger DTs had a large effect.\n\nLIMITATIONS:\nY", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-28"}