SlowGuess/NeurIPS_2023_OCR huggingface dataset:
To determine how much faster our model is, we benchmarked the required training duration (forward and backward pass) of our accelerated ALIF SNN to the standard ALIF SNN for varying ARP and simulation lengths using a synthetic spike dataset, with the task of minimizing the number of final-layer output spikes (see Supplementary material). We found the training speedup of our model to increase for a growing ARP (Figure 3a). This speedup was more pronounced for longer simulation durations ( $53 \times$  speedup for  $T = 2048$ ) than shorter simulation durations ( $36 \times$  speedup for  $T = 512$ ). These speedups were also present when just running the forward pass (i.e. inference using the network; see Supplementary material). Furthermore, we found the speedup of our model to be robust over varying numbers of neurons and layers (Figure 3b). Lastly, we also found our method
to perform the forward pass more than an order of magnitude faster than other publicly available SNN implementations [63, 64] (see Supplementary material).
4.2 Accelerated training on spiking classification tasks
b
Figure 4: Performance of our model on spiking datasets. a. Classification accuracy of our model and the standard ALIF SNN on the N-MNIST and SHD datasets over different (non-biological) ARPs (N-MNIST  $1\mathrm{ms} = 1$  time step; SHD  $2\mathrm{ms} = 1$  time step). b. Training durations of our model and the standard ALIF SNN on the N-MNIST and SHD datasets. Horizontal gray lines plot the standard model's performance (using no ARP) and bars plot the mean and standard error over three runs.
To establish whether our model can learn using backprop with surrogate gradients, and perform on a par with the standard model, we trained our accelerated ALIF SNN and the standard ALIF SNN on the Neuromorphic-MNIST (N-MNIST) (using  $\mathrm{DT} = 1\mathrm{ms}$ ) [65] and Spiking Heidelberg Digits (SHD) (using  $\mathrm{DT} = 2\mathrm{ms}$ ) [66] spiking classification datasets (both commonly used for benchmarking SNN performance [40, 67, 29, 42, 50]). The task of the N-MNIST dataset is to classify spike representations of handwritten digits, and the task of the SHD dataset is to classify spike representations of spoken digits. In all experiments we employed identical model architectures consisting of two hidden layers (of 256 neurons each) and an additional integrator readout layer, with predictions taken from the readout neurons with maximal summed membrane potential over time (as commonly done [66, 68, 29, 42, 50]; see Supplementary material). We adopted the multi-Gaussian surrogate-gradient function [29] to overcome the non-differentiable discontinuity of the spike function (although we found that other surrogate-gradient functions also work, see Supplementary material). Furthermore (as suggested by [68] for conventional SNNs), we only permitted surrogate gradients to flow through the non-recurrent connectivity (which significantly improved performance; see Supplementary material).
We trained our model with different non-biological ARPs on each dataset and contrasted the accuracy and training duration to the standard model (with no ARP). We found a favourable trade-off between classification accuracy and training time for our model. Compared to the standard ALIF SNN model, our model achieved a very similar, albeit slightly lower, accuracy on the N-MNIST  $(96.97\%)$  for ARP  $= 10\mathrm{ms}$  vs standard  $97.71\%$  ) and SHD dataset  $(86.07\%)$  for ARP  $= 20\mathrm{ms}$  vs standard  $87.48\%$  ), with the accuracy declining only for the largest ARPs tested (Figure 4a). However, our model only required a fraction of the training time, with an average training epoch of 181s and 20s for the N-MNIST and SHD datasets, respectively, when using the largest tested ARP, compared to the respective standard model training times of 2034s and 268s (Figure 4b).
4.3 Quickly fitting real neural recordings on sub-millisecond timescales
We explored the ability of our model to fit in vitro electrophysiological recordings from 146 inhibitory and excitatory neurons in mouse primary visual cortex (V1) (provided by the Allen Institute [69, 70]). In these recordings, a variable input current was repeatedly injected at various amplitudes into each real neuron, and the resulting alteration in the membrane potential was recorded (Figure 5a). For each neuron, we used half of the recordings for fitting and the other half for testing, and report all the qualitative and quantitative results on the held-out test dataset. Fitting was achieved by minimising
b
roomsnoise 1
DT=0.1ms ARP=2ms
noise 2
noise 2
d
Figure 5: Fitting cortical electrophysiological recordings. a. An illustration of a cortical neuron in mouse V1 being recorded whilst stimulated with a noisy current injection. b. For held-out data not used for fitting, an example current injection (bottom) and recorded membrane potential (middle) with corresponding fitted model predictions (top). c. Comparison of neuron fit accuracy of our model for  $\mathrm{DT} = 0.1\mathrm{ms}$  (y-axis) against  $\mathrm{DT} = 4\mathrm{ms}$  (x-axis). Explained temporal variance (ETV) measures the goodness-of-fit (one is a perfect fit and zero is a chance-level fit). d. Comparison of the fit accuracy (left) and duration (right) of our model and the standard model (both using  $\mathrm{DT} = 0.1\mathrm{ms}$  and  $\mathrm{ARP} = 2\mathrm{ms}$ ). e. Our model's fit accuracy for increasing ARP (with  $\mathrm{DT} = 0.1\mathrm{ms}$ ). f. Our model's fit accuracy for increasing DT (with  $\mathrm{ARP} = \max(2, \mathrm{DT})\mathrm{ms}$ ). g. To f. plots the median and standard error over neurons, except that d. (right) plots the mean fit time.
the van Rossum distance between the model and real spike trains [71]. To quantitatively compare the fits of our model using different DTs and ARPs, we employed the explained temporal variance (ETV) measure (used on the Allen Institute website; see Supplementary material). This metric quantifies how well the temporal dynamics of the neurons are captured by the model and takes into account trial-to-trial variability due to intrinsic neural noise. It has a value of one for a perfect fit and a value of zero if the model predicts at chance.
We found that our accelerated ALIF model captured the spike timing of the neurons. Pertinent to the speed-accuracy trade-off, this fit strongly depended on the chosen DT, and less so on the chosen ARP. Qualitatively, using a  $\mathrm{DT} = 0.1\mathrm{ms}$  and an  $\mathrm{ARP} = 2\mathrm{ms}$ , we found our model captured the spike timings of the neurons for current injections of varying amplitude. This still seemed to be the case when we used a larger  $\mathrm{ARP} = 8\mathrm{ms}$ , but the model was worse at capturing spike timings when we used a larger  $\mathrm{DT} = 4\mathrm{ms}$  (with  $\mathrm{ARP} = 4\mathrm{ms}$ ; Figure 5b; see Supplementary material for zoomed-in neural traces).
Quantitatively comparing the neuron fits one by one, we found that nearly all of the neuron fits were better when using a  $\mathrm{DT} = 0.1\mathrm{ms}$  (ETV  $= 0.80$  ) than a  $\mathrm{DT} = 4\mathrm{ms}$  (ETV  $= 0.66$  ; Figure 5c; using an ARP  $= 4\mathrm{ms}$ ). We examined how accurate and fast our fits were compared to the standard model using an ARP  $= 2\mathrm{ms}$  and a  $\mathrm{DT} = 0.1\mathrm{ms}$ . Both models achieved a similar ETV of  $\sim 0.8$ , yet our model only required 15.5s on average per neuron fit time compared to the 108.4s of the standard model (Figure 5d). We further investigated whether a larger ARP could still reasonably fit the data for  $\mathrm{DT} = 0.1\mathrm{ms}$  (to benefit from a faster fit). Consistent with our qualitative observations, we found a less marked reduction in fit accuracy when using larger ARPs (Figure 5e) compared to the drop in performance when using larger DTs (Figure 5f; using an ARP of  $\max(2, \mathrm{DT})\mathrm{ms}$ ).
ALIF neurons are a popular model for studying the brain in silico. Training these neurons is, however, slow due to their sequential nature [41, 72]. We overcome this by algorithmically reinterpreting the ALIF model. We found that our method permits faster simulation of SNNs, which will likely
play an important role in future research into neural dynamics through simulation [17, 16]. We also confirmed the validity of this approach for modelling neural data. Firstly - of interest to computational neuroscientists - we fitted a multilayered network of neurons using two common spike classification datasets. We found that our model achieved a similar classification accuracy to that of the standard model, even if we increased the ARP to large non-physiological values, which drastically reduced the training duration required for both datasets. Secondly - of interest to computational and experimental neuroscientists - we explored the applicability of our method to fit real electrophysiological recordings of neurons in mouse V1 using sub-millisecond DTs.
We found that our method accurately captured the spike timing of real neurons, fitting their activity patterns in a fraction of the time required by the standard model (although we note that other, more recent spiking-models, might improve fit accuracies [73]). This is particularly important as datasets become larger with advances in recording techniques, requiring faster fitting solutions to characterise computational properties of neurons. As an example of the potential insights provided by our model, we found the fitted V1 neurons to have a heterogenous membrane time constant distribution, suggesting that V1 processes visual information on multiple timescales [42] (see Supplementary material).
Our work will likely also be of interest to neuromorphic engineers and researchers, developing energy-efficient hardware to emulate SNNs [20]. These systems - like real neurons - run in continuous time and thus require training on extremely fine DTs off-chip [74, 75]. Our method could help to accelerate off-chip training times. Furthermore, the ARP hyperparameter in our model can limit high spike rates and thus reduce energy consumption on neuromorphic systems, as this consumption scales approximately proportionally with the number of emitted spikes [76].
A limitation of our method is that the training speedup scales sublinearly - as opposed to linearly - with an increasing ARP simulation length (see Section 3 and 4.1). This is likely due to GPU overheads and employed cudnn routines [77], which further improvements to our code implementation could overcome. An additional limitation is the requirement to define the ARP hyperparameter, whose value influences the training speed and test accuracy in our method and relates to biological realism. However, we found a beneficial trade-off - by using large non-biological ARPs on the artificial spiking classification dataset and small physiological ARPs for the neural fits (although we found larger values to also perform reasonably well) the model achieved comparable accuracy to the standard ALIF model in both cases, while also having greatly increased speed of training and simulation. In particular, the capacity of our model to accurately and quickly fit the data from real neurons is crucial in terms of the biological applicability of this approach.
We thank Rob Pratt and anonymous reviewers for helpful discussions; and Lorenzo Mazzaschi for feedback on the manuscript. Luke Taylor was supported by the Clarendon Fund. Andrew King and Nicol Harper were supported by the Wellcome Trust (WT108369/Z/2015/Z). Figure 5 was created with BioRender.com.
[1] Nicol S Harper, Oliver Schoppe, Ben DB Willmore, Zhanfeng Cui, Jan WH Schnupp, and Andrew J King. Network receptive field modeling reveals extensive integration and multi-feature selectivity in auditory cortical neurons. PLoS Computational Biology, 12(11):e1005113, 2016.
[2] Yosef Singer, Luke Taylor, Ben DB Willmore, Andrew J King, and Nicol S Harper. Hierarchical temporal prediction captures motion processing along the visual pathway. eLife, 12:e52599, 2023.
[3] Santiago A Cadena, George H Denfield, Edgar Y Walker, Leon A Gatys, Andreas S Tolias, Matthias Bethge, and Alexander S Ecker. Deep convolutional models improve predictions of macaque v1 responses to natural images. PLoS Computational Biology, 15(4):e1006897, 2019.
[4] Andrew Francl and Josh H McDermott. Deep neural network models of sound localization reveal how perception is adapted to real-world environments. Nature Human Behaviour, 6(1): 111-133, 2022.
[73] Christian Pozzorini, Skander Mensi, Olivier Hagens, Richard Naud, Christof Koch, and Wulfram Gerstner. Automated high-throughput characterization of single neurons by means of simplified spiking models. PLoS Computational Biology, 11(6):e1004275, 2015.
[74] Jan Stuijt, Manolis Sifalakis, Amirreza Yousefzadeh, and Federico Corradi.  $\mu$ brain: An event-driven and fully synthesizable architecture for spiking neural networks. Frontiers in Neuroscience, page 538, 2021.
[75] Yuming He, Federico Corradi, Chengyao Shi, Ming Ding, Martijn Timmermans, Jan Stuijt, Pieter Harpe, Ilja Ocket, and Yao-Hong Liu. A  $28.2\mu c$  neuromorphic sensing system featuring snn-based near-sensor computation and event-driven body-channel communication for insertable cardiac monitoring. In 2021 IEEE Asian Solid-State Circuits Conference (A-SSCC), pages 1-3. IEEE, 2021.
[76] Priyadarshini Panda, Sai Aparna Aketi, and Kaushik Roy. Toward scalable, efficient, and accurate deep spiking neural networks with backward residual connections, stochastic softmax, and hybridization. Frontiers in Neuroscience, 14:653, 2020.
[77] Sharan Chetlur, Cliff Woolley, Philippe Vandermersch, Jonathan Cohen, John Tran, Bryan Catanzaro, and Evan Shelhamer. codnn: Efficient primitives for deep learning. arXiv:1410.0759, 2014.