SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Zuckerman Mind Brain Behavior Institute, Columbia University, New York, USA  
Max Planck Institute for Dynamics and Self-Organization, Göttingen, Germany  
Göttingen Campus Institute for Dynamics of Biological Networks, Göttingen, Germany  
re2365@columbia.edu
Spiking Neural Networks (SNNs) are biologically-inspired models that are capable of processing information in streams of action potentials. However, simulating and training SNNs is computationally expensive due to the need to solve large systems of coupled differential equations. In this paper, we introduce SparseProp, a novel event-based algorithm for simulating and training sparse SNNs. Our algorithm reduces the computational cost of both the forward and backward pass operations from O(N) to  $O(\log (N))$  per network spike, thereby enabling numerically exact simulations of large spiking networks and their efficient training using backpropagation through time. By leveraging the sparsity of the network, SparseProp eliminates the need to iterate through all neurons at each spike, employing efficient state updates instead. We demonstrate the efficacy of SparseProp across several classical integrate-and-fire neuron models, including a simulation of a sparse SNN with one million LIF neurons. This results in a speed-up exceeding four orders of magnitude relative to previous event-based implementations. Our work provides an efficient and exact solution for training large-scale spiking neural networks and opens up new possibilities for building more sophisticated brain-inspired models.
The cortex processes information via streams of action potentials - commonly referred to as spikes - that propagate within and between layers of recurrent neural networks. Spiking neural networks (SNNs) provide a more biologically plausible description of neuronal activity, capturing in contrast to rate networks membrane potentials and temporal sequences of spikes. Moreover, SNNs promise solutions to the high energy consumption and  $\mathrm{CO}_{2}$  emissions of deep networks [1, 2], as well as the ability to transmit information through precise spike timing [3, 4, 5].
Unlike commonly used iterative ODE solvers that necessitate time discretization, event-based simulations resolve neural network dynamics precisely between spike events. This approach ensures machine-level precision in simulations, thereby alleviating the necessity for verification of result robustness with respect to time step size  $\Delta t$  and mitigating numerical issues, for instance in the vicinity of synchronous regimes [6]. However, a major drawback of event-based simulations is the computational cost of iterating through all neurons at every network spike time.
To tackle this challenge, we introduce SparseProp, an novel event-based algorithm designed for simulating and training sparse SNNs. Our algorithm reduces the computational cost of both the forward and backward pass from  $O(N)$  to  $O(\log (N))$  per network spike. This efficiency enables numerically exact simulations of large spiking networks and their efficient training using backpropagation through
time. By exploiting network sparsity and utilizing a change of variable to represent neuron states as times to the next spikes on a binary heap, SparseProp avoids iterating through all neurons at every spike and employs efficient state updates.
We demonstrate the utility of SparseProp by applying it to three popular spiking neuron models. While the current implementation is for univariate neuron models, we discuss a potential extension to more detailed multivariate models later.
- Introducing SparseProp, a novel and efficient algorithm for numerically exact event-based simulations of recurrent SNNs (section 2 and appendix A for minimal example code).
- Conducting a numerical and analytical scaling analysis that compares the computational cost of our proposed algorithm to conventional event-based spiking network simulations (section 3, Fig 3 and table 1).
- Providing concrete implementations of the algorithm for recurrent networks of leaky integrate-and-fire neurons and quadratic integrate-and-fire neurons (section 4 and here)
- Extending the algorithm to neuron models that lack an analytical solution for the next spike time using Chebyshev polynomials (section 5 and appendix B).
- Extending the algorithm to heterogeneous spiking networks (section 6).
In summary, our proposed SparseProp algorithm offers a promising approach for simulating and training SNNs with both machine precision and practical applicability. It enables the simulation and training of large-scale SNNs with significantly reduced computational costs, paving the way for the advancement of brain-inspired models.
2 Algorithm for Efficient Event-Based Simulations in Recurrent Networks
We consider the dynamics of a spiking recurrent neural network of  $N$  neurons that is described by a system of coupled differential equations [7, 8, 9, 10, 11, 12, 13, 14]:
$$
\tau_ {\mathrm {m}} \frac {\mathrm {d} V _ {i}}{\mathrm {d} t} = F (V _ {i}) + I _ {i} ^ {\text {e x t}} + \sum_ {j, s} J _ {i j} h \left(t - t _ {j} ^ {(s)}\right). \tag {1}
$$
Here, the rate of change of the membrane potential  $V_{i}$  depends on its internal dynamics  $F(V_{i})$ , an external input  $I_{i}^{\mathrm{ext}}$  and the recurrent input  $\sum_{j,s} J_{ij} h(t - t_{j}^{(s)})$ .  $\tau_{\mathrm{m}}$  is the membrane time constant.
Thus, when presynaptic neuron  $j$  spikes at time  $t_j^{(s)}$  the  $i^{\mathrm{th}}$  postsynaptic neuron is driven by some temporal kernel  $h(\tau)$  and coupling strength  $J_{ij}$ .
For event-based simulations, the dynamics of Eq 1 are evolved from network spike to network spike, usually based on an analytical solution of the membrane potential  $V_{i}(t)$  as a function of time  $t$ , instead of using iterative ODE solvers like the Runge-Kutta methods. A key aspect of this method for spiking networks is that the spike times are not confined to the time steps of a numerical solver, allowing them to be obtained with machine precision. To simulate the entire network dynamics this way involves four simple steps:
1. Find the next spiking neuron  $j^{*}$  and its spike time  $t_{j^{*}}$ .
2. Evolve the state  $V_{i}$  of every neuron to the next network spike time  $t_{j^{*}}$ .
3. Update the neurons  $i$  postsynaptic to  $j^{*}$  using the weight  $J_{ij}$ .
4. Reset the state of the spiking neuron  $V_{j^{*}} = V_{\mathrm{reset}}$ .
Such event-based simulation schemes are usually used for neuron models that have an analytical solution of the individual neurons' dynamics  $V_{i}(t)$  [10, 14, 15, 16, 17, 18]. The precision of the solution of event-based simulation is then limited only by the machine precision of the computer. In contrast, solving spiking network dynamics with an iterative ODE solver requires asserting that results are not dependent on the integration step size  $\Delta t$  of the integration scheme. The total accumulated error usually scale with  $\mathcal{O}((\Delta t)^p)$ , where  $p$  is the order of the ODE solver. However, both iterating
Figure 1: Change of reference frame allows efficient evolution of network state. A We map all membrane potentials  $V_{i}(t)$  to phases  $\phi_i(t)$  that evolve linearly between spikes with phase velocity  $\omega$  [10, 11, 15]. B At spike times  $t_s$ , the phase transition curve (PTC)  $g(\phi(t_s))$  has to be evaluated that tells neuron  $i$  how its phase  $\phi_i$  changes when it receives an input spike. C In the phase representation, phases  $\phi_i(t)$  evolve linearly in time between spikes, but the amount of change when the neuron receives an input spike depends on its phase. In this example at input spike time  $t_1$ , the phase change is much larger than at input spike time  $t_2$ , despite having same size in the voltage representation. D In conventional event-based simulations, the phase of all  $N$  neurons is shifted at every network spike time by  $\Delta \phi_i = \omega \Delta t$ , where  $\Delta t = (t_{s+1} - t_s)$  is the time to the next network spike. Instead, in SparseProp just threshold and reset are changed at every spike.
through all neurons to find the next spiking neuron (step 1) and evolving all neurons to the next spike time (step 2) conventionally require  $\mathcal{O}(N)$  calculations per network spike. In the following, we suggest an algorithm that for sparse networks only requires  $\mathcal{O}(\log(N))$  calculations per network spike by using a change of reference frame and an efficient data structure. In the next paragraphs, we will describe these two features in more detail.
2.1 Change of Reference Frame
For clarity of presentation, we describe our algorithm first for a network of neurons that can be mapped to pulse-coupled phase oscillators like the leaky integrate-and-fire neuron and the quadratic integrate-and-fire neuron (Fig 1 A, C), and discuss later a more general implementation. For pulse-coupled oscillators, the network dynamics can be written as [10, 11, 14, 15, 16, 17, 18, 19]:
$$
f \left(\phi_ {i} \left(t _ {s}\right)\right) = \left\{ \begin{array}{l l} g \left(\phi_ {i} \left(t _ {s}\right) + \omega \left(t _ {s + 1} - t _ {s}\right)\right) & \text {f o r} i \in \operatorname {p o s t} \left(j ^ {*}\right) \\ \phi_ {i} \left(t _ {s}\right) + \omega \left(t _ {s + 1} - t _ {s}\right) & \text {e l s e} \end{array} \right. \tag {2}
$$
where  $\omega$  is the constant phase velocity and  $g(\phi)$  is the phase transition curve (PTC) evaluated for the neurons that are postsynaptic to the spiking neuron  $j^{*}$  just before the next network spike. In this phase representation, in the absence of recurrent input pulses, each phase evolves linearly in time with constant phase velocity  $\omega$  from reset phase  $\phi^{\mathrm{reset}}$  to threshold phase  $\phi^{\mathrm{th}}$  (Fig 1 C). Thus, finding the next spiking neuron in the network  $j^{*}$  amounts for homogeneous networks where all neurons have identical phase velocity to taking the maximum over all phases  $\phi_{j^{*}}(t_{s}) = \max (\phi_{i}(t_{s}))$ . The next network spike time  $t_{s + 1}$  is given by  $t_{j^{*}} = \frac{\phi_{\mathrm{th}} - \phi_{j^{*}}}{\omega}$ .  $g(\phi)$  quantifies the phase change of a neuron upon receiving an input spike as a function of its phase (Fig 1 B). In this conventional form of the event-based algorithm [10, 10, 11, 14, 15, 16, 17, 18, 19, 20, 21], all  $N$  phases must be linearly evolved to the state immediately before the next network spike time  $t_{s + 1}$ :  $\phi_i(t_{s + 1}) = \phi_i(t_s) + \omega (t_{s + 1} - t_s)$ .
This operation, with a computational cost of  $\mathcal{O}(N)$ , can be avoided in our proposed algorithm. Instead, we introduce a modification where just the threshold and reset phase are changed by a global phase offset  $\Delta_{\phi}$ :
$$
\Delta_ {\phi} ^ {(s + 1)} = \Delta_ {\phi} ^ {(s)} + \omega \left(t _ {s + 1} - t _ {s}\right), \tag {3}
$$
Figure 2: A suitable data structure allows efficient search for next spiking neuron for numerically exact simulation of large spiking networks. A By using a priority queue, the next spiking neuron can be found without iteration through all neurons of the network at every spike time. One implementation of a priority queue is a binary heap, as the example shown. In a max-heap, each child node has a value less or equal to its parent node. If an element is deleted, removed, or added, the heap property is restored by swapping parents and children systematically. Finding the node highest priority has a numerical complexity of  $\mathcal{O}(1)$  and changing the value of any element has an amortized time complexity of  $\mathcal{O}(\log(N))$ . B Array implementation of binary heap shown in A. As per spike all postsynaptic neurons and the spiking neuron have to be updated, this requires on average  $K + 1$  operation, thus an amortized time complexity  $\mathcal{O}\left(K \cdot \log(N)\right)$  [23].
starting with  $\Delta_{\phi} = 0$  at  $t = 0$ . Thus  $\phi_{s + 1}^{\mathrm{th}} = \phi^{\mathrm{th}} - \Delta_{\phi}^{(s + 1)}$ . Similarly,  $\phi_{s + 1}^{\mathrm{reset}} = \phi^{\mathrm{reset}} - \Delta_{\phi}^{(s + 1)}$ . See Fig 1D for an illustration. In inhibition-dominated networks, the phases  $\phi(t_s)$  will thus become increasingly negative. For long simulations, all phases and the global phase should therefore be reset if the global phase exceeds a certain threshold  $\Delta_{\phi}^{\mathrm{th}}$ , to avoid numerical errors resulting from subtractive cancellation due to floating-point arithmetic [22]. We discuss this in more detail in appendix C. In summary, this change of variables reduces the computational cost of propagating the network state from  $\mathcal{O}(N)$  to  $\mathcal{O}(1)$ . Note, that we assumed here identical phase velocity  $\omega$  for all neurons, but we will relax this assumption later in section 6.
2.2 Efficient Data Structure
We next address the computational bottleneck encountered in very large sparse networks when finding the next spiking neuron, which has a numerical time complexity of  $\mathcal{O}(N)$ . To overcome this limitation, we introduce an efficient implementation using a priority queue. The priority queue is a data structure that facilitates returning elements with the highest priority and supports insertion and priority updates [24]. We implement the priority queue as a heap-ordered binary tree, where each node's value is greater than its two children's values. This structure enables returning elements with the highest priority in constant time, with a time complexity of  $\mathcal{O}(1)$ , and inserting elements or updating priorities in logarithmic time, with a time complexity of  $\mathcal{O}(\log(N))$  [23]. Thus, only  $\mathcal{O}(K\log(N))$  operations have to be performed per network spike, as for  $K$  postsynaptic neurons and for one spiking neuron, a new phase needs to be updated in the priority queue.
Algorithm 1 SparseProp: Efficient Event-Based Simulation of Sparse Spiking Network
1: initialize  $\phi (t_0),\Delta_\phi = 0$    
2: heapify  $\phi (t_0)$    
3: warm-up of network  $\phi (t_0)$    
4: for  $s = 1\rightarrow t$  do   
5: get phase of next spiking neuron:  $j,\phi_{j} = \mathrm{peak}(\phi_{i}(t_{s}))$    
6: calculate phase increment:  $d\phi = \phi^{\mathrm{th}} - \phi_{j}(t_{s}) + \Delta_{\phi}$    
7: update global phase shift:  $\Delta_{\phi} + = d\phi$    
8: evaluate phase transition curve:  $\phi_{i^{*}}^{+}(t_{s}) = g\bigl (\phi_{i^{*}}^{-}(t_{s}) + \Delta_{\phi}\bigr)$    
9: reset spiking neuron:  $\phi_j(t_{s + 1}) = \phi^{\mathrm{re}} - \Delta_{\phi}$    
10: if  $\Delta_{\phi} > \Delta_{\phi}^{\mathrm{th}}$  then   
11:  $\phi + = \Delta_{\phi}$    
12:  $\Delta_{\phi} = 0$    
13: end if   
14: end for
where  $\mathrm{peek}(\phi_i(t_s))$  retrieves the highest-priority element without extracting it from the heap. In a min-heap—utilized for the phase representation here—the function fetches the smallest element, representing the next spike time. Conversely, in a max-heap—employed for heterogeneous networks—it obtains the largest element, signifying the maximum phase. The operation generally executes in constant time,  $O(1)$ , given that the target element resides at the heap's root. Nonetheless, this function leaves the heap structure unaltered.
An example implementation of SparseProp in Julia [25] is available here. There are two core innovations in our suggested algorithm: First, we perform a change of variables into a co-moving reference frame, which avoids iterating through all neurons in step 2 (Fig 1). Second, for finding the next spiking neuron in the network, we put the network state in a priority heap (Fig 2). A binary heap has efficient operations for returning the elements with the lowest (highest) key and insertion of new elements or updating of keys [24]. We will next analyze the computational cost of SparseProp empirically and theoretically.
3 Computational Cost Scaling
Overall, the total amortized cost per network spike of SparseProp scales with  $\mathcal{O}\left(K\log (N)\right)$ . For sparse networks, where  $K$  is fixed and only  $N$  is growing, the computational complexity of SparseProp per network spike thus only scales with  $\mathcal{O}\left(\log (N)\right)$ .
As the number of network spikes for a given simulation time grows linearly with network size, the overall computational cost for a given simulation time of SparseProp scales  $\mathcal{O}\left(KN\log (N)\right)$ , which for sparse networks is far more efficient compared to a conventional implementation (see Fig. 3). Specifically, the linearthmic scaling with network size  $N$  of our approach, grants remarkable efficiency gains compared to the quadratic scaling of conventional event-based simulations, particularly for sparse networks. For example, simulating a network of  $10^{6}$  neurons and  $K = 100$  synapses for 100s takes less then one CPU hour with SparseProp, but would take more than 3 CPU years with the conventional algorithm<sup>1</sup>. We provide a detailed comparison of the computational cost in table 1.
4 Example Implementation for Integrate-And-Fire Neuron Models
4.1 Pseudophase Representation of Leaky Integrate-And-Fire Neurons
For a network of pulse-coupled leaky integrate-and-fire neurons, Eq. 1 reads in dimensionless notation
$$
\tau_ {\mathrm {m}} \frac {\mathrm {d} V _ {i}}{\mathrm {d} t} = - V _ {i} + I ^ {\text {e x t}} + \tau_ {\mathrm {m}} \sum_ {j, s} J _ {i j} \delta \left(t - t _ {j} ^ {(s)}\right) \tag {4}
$$
<sup>1</sup>These numbers are based on the benchmark on a Xeon® CPU E5-4620 v2 @ 2.60 GHz and 512 GB RAM
conventional algorithm	SparseProp
Find next spiking neuron	min(φth - φi/ω)	peek(φi)
Evolve neurons	φi += ω dt	Δ+ = Δφ
Update postsynaptic neurons	K operations	K operations + K key updates O(log(N))
Reset spiking neuron	one array operation	O(log(N))
Memory cost	O(N)	O(N)
total amortized costs per network spike	O(N+K)	O(K log(N))
total amortized costs for fixed simulation time	O(N2)	O(K N log(N))
Table 1: Comparison of computational cost for event-based simulation and training of recurrent spiking networks for different algorithms  $N$  denotes number of neurons,  $K$  is the average number of synapses per neuron. For large sparse networks  $\left( {K \ll  N}\right)$  and a fixed number of synapses per neuron  $K$  ,the dominant term grows quadratic for the conventional algorithm [10, 11, 11, 26, 27, 28] and lineararithmic  $\left( {\mathcal{O}\left( {\log \left( N\right) }\right) }\right)$  for the SparseProp algorithm.
Figure 3: Benchmark of SparseProp vs. conventional algorithm. The computational cost per network spike scales linear with network size  $N$  in a conventional event-based implementation. In the large sparse network limit, the computational bottleneck is to find the next spiking neuron in the network and to propagate all neurons phases to the next network spike. Both operations can be implemented efficiently. First, instead of shifting all neurons phases, the threshold and reset values can be shifted. Second, by using a binary heap as a data structure for the phases, finding the next phase and keeping the heap-ordering has computational complexity of  $\mathcal{O}(K\log (N))$ . A: CPU time per network spike as a function of network size  $N$  for an inhibitory network of leaky integrate-and-fire neurons with fixed number of synapses per neuron  $K$  [9]. B: CPU time per simulation time shows a quadratic scaling for conventional algorithm, but only linearthmic scaling for the novel SparseProp algorithm, Benchmark was performed on an Intel® Xeon® CPU E5-4620 v2 @ 2.60 GHz and 512 GB RAM. Parameters: mean firing rate  $\bar{\nu} = 1\mathrm{Hz}$ ,  $J_0 = 1$ ,  $\tau_{\mathrm{m}} = 10\mathrm{ms}$ ,  $K = 100$ .
If a membrane potential  $V_{i}$  reaches threshold  $V_{\mathrm{th}}$ , it is reset to  $V_{\mathrm{re}}$ . Without loss of generality, we set  $V_{\mathrm{th}} = 0$  and  $V_{\mathrm{re}} = -1$ . Between two network spikes, the solution is given by:
$$
V _ {i} \left(t _ {s + 1}\right) = I ^ {\text {e x t}} - \left(I ^ {\text {e x t}} - V _ {i} \left(t _ {s}\right)\right) \exp \left(- \frac {t _ {s + 1} - t _ {s}}{\tau_ {\mathrm {m}}}\right) \tag {5}
$$
In this pseudophase representation (slightly different from [11, 28]), the phases  $\phi_i \in (-\infty, 0]$  describe the neuron states relative to the unperturbed interspike interval.
To obtain the unperturbed interspike interval  $T^{\mathrm{free}}$ , we have to solve Eq. 5 between reset and threshold in the absence of synaptic input.
$$
\begin{array}{l} T ^ {\text {f r e e}} = - \tau_ {\mathrm {m}} \ln \left(\frac {V _ {\text {t h}} - I ^ {\text {e x t}}}{V _ {\text {r e}} - I ^ {\text {e x t}}}\right) (6) \\ = \tau_ {\mathrm {m}} \ln \left(1 + \frac {1}{I ^ {\mathrm {e x t}}}\right). (7) \\ \end{array}
$$
Its inverse is the phase velocity  $\omega = 1 / T^{\mathrm{free}}$ . The phase  $\phi_{i}$  is thus given by  $\phi_{i} = -\omega \ln \left(\frac{I^{\mathrm{ext}}}{I^{\mathrm{ext}} - V_{i}}\right)$ . The reverse transformation is  $V_{i} = I^{\mathrm{ext}}\left(1 - \exp \left[-\frac{\phi_{i}}{\omega\tau_{\mathrm{m}}}\right]\right)$ . Therefore, the phase transition curve is
$$
g \left(\phi_ {i ^ {*}} \left(t _ {s + 1} ^ {-}\right)\right) = - \omega \ln \left(\exp \left(- \phi_ {i ^ {*}} \left(t _ {s + 1} ^ {-}\right) / \omega\right) + c\right), \tag {8}
$$
where  $c$  is the effective coupling strength  $c = \frac{J}{I^{\mathrm{ext}}}$  and  $J$  is the synaptic coupling strength taken here to be identical for connected neurons. Usually, as discussed later in the appendix E in the section on balanced networks,  $J$  is scaled with  $J = \frac{J_0}{\sqrt{K}}$ , where  $K$  is the number of synapses per neuron. An example implementation of a LIF network with SparseProp in Julia [25] is available here.
4.2 Phase Representation of Quadratic Integrate-And-Fire Neurons
The quadratic integrate-and-fire (QIF) neuron has, in contrast to the LIF neuron, a dynamic spike generation mechanism and still has an analytical solution  $V_{i}(t)$  between network spikes.
For a network of QIF neurons, Eq. 1 reads in dimensionless voltage representation:
$$
\tau_ {\mathrm {m}} \frac {\mathrm {d} V _ {i}}{\mathrm {d} t} = V _ {i} ^ {2} + I ^ {\text {e x t}} + \tau_ {\mathrm {m}} \sum_ {j, s} J _ {i j} \delta \left(t - t _ {j} ^ {(s)}\right) \tag {9}
$$
The quadratic integrate-and-fire model can be mapped via a change of variables  $V = \tan(\theta/2)$  to the theta model with a phase variable  $\theta \in (-\pi, \pi]$  [29, 30, 31]. The dynamical equation between incoming spikes is the topological normal form for the saddle-node on a limit cycle bifurcation (SNIC) and allows a closed-form solution of the next network spike thanks to the exact derivation of the phase response curve [32]. Therefore, the quadratic integrate-and-fire neuron is commonly used to analyze networks of spiking neurons [10, 33, 34, 35, 36]. When  $I_i^{\mathrm{ext}} > 0 \forall i$ , the right-hand side of the dynamics is strictly positive and all neurons would spike periodically in the absence of incoming postsynaptic potentials. In this case, we can choose another particularly tractable phase representation, called phi-representation with  $V_i(t) = \sqrt{I^{\mathrm{ext}}} \tan(\phi_i(t)/2)$ , where the neuron has a constant phase velocity [10]. This transformation directly yields the phase transition curve
$$
g \left(\phi_ {i}\right) = 2 \arctan \left(\tan \frac {\phi_ {i}}{2} + c\right), \tag {10}
$$
where  $c$  is the effective coupling strength  $c = \frac{J}{\sqrt{I^{\mathrm{ext}}}}$ . The phase velocity is given by  $\omega = 2\sqrt{I^{\mathrm{ext}}}$ . We considered homogeneous networks here, where all neurons have identical external input and identical membrane time constant, and will later consider the more general case of heterogeneous networks in section 6. An example implementation of a QIF network with SparseProp in Julia [25] is available here.
5 Event-Based Simulation via Chebyshev Polynomials