SlowGuess/NeurIPS_2023_OCR huggingface dataset:
The solution provides us with two crucial pieces of information. Firstly, for each time point  $t$ , we can now associate a corresponding voltage  $V(t)$ . Secondly, we also acquire the unperturbed interspike interval  $T$ , which determines the phase velocity  $\omega$ . Using the numerically obtained solution  $V(t)$ , we create a lookup table that maps time to voltage by employing the DataInterpolations.jl package. By taking the inverse, we obtain a mapping from a voltage to the next spike time  $t(V)$ , which we also store as a separate lookup table. The two lookup tables combined give us a numerical phase transition curve  $g(\phi)$ . We confirmed for the leaky integrate-and-fire neuron and the quadratic integrate-and-fire neuron that this numerical procedure indeed yields the correct phase response curve  $d$ . Subsequently, we incorporate the numerical phase response curve  $d$  into the event-based implementation SparseProp implementation and also confirmed the spike times remained identical when comparing with LIF networks with the analytical phase transition curve and identical network realization and identical initial conditions. We used an inhibitory LIF network for this sanity check, because inhibitory pulse-coupled networks of leaky integrate-and-fire neuron are stable with respect to infinitesimal perturbations [11, 21, 39, 40, 41]. This feature in combination with instability with respect to finite-size perturbations renders inhibitory LIF networks as an ideal test-based for numerical precision, because event-based simulations should result exactly in the same spike sequence. This is not the case for networks of QIF neurons, which are typically chaotic [10] (see also [75]).
Algorithm 3 SparseProp: Event-Based Simulation Based on Lookup Tables
1: Set up the ordinary differential equation Eq. 12  
2: Numerically integrate the ODE from  $V_{\mathrm{re}}$  to  $V_{\mathrm{th}}$  to obtain  $V(t)$ .  
3: Generate a lookup table based on the obtained solution  $V(t)$ .  
4: Generate an inverse lookup table based on the solution  $t(V)$ .  
5: Combine the lookup table and inverse lookup table to obtain the phase transition curve  $g(\phi)$ .  
6: Heapify  $\phi(t_0)$   
7: Perform the warm-up of network  $\phi(t_0)$   
8: for  $s = 1 \rightarrow t$  do  
9: Get index and phase of next spiking neuron:  $j, \phi_j = \text{peek}(\phi_i(t_s))$   
10: Calculate phase increment:  $d\phi = \phi^{\mathrm{th}} + \Delta - \phi_j(t_s)$   
11: Update global phase shift:  $\Delta += d\phi$   
12: Evaluate phase transition curve:  $\phi_{i*}^{+}(t_s) = g(\phi_{i*}^{-}(t_s) + \Delta)$   
13: Reset spiking neuron:  $\phi_j(t_{s+1}) = \phi^{\mathrm{re}} - \Delta$   
14: if  $\Delta > \Delta^{\mathrm{th}}$  then  
15:  $\phi += \Delta$   
16:  $\Delta = 0$   
17: end if  
18: end for
An example implementation of an EIF network using SparseProp in Julia, in the balanced state, is available here.
As an alternative to generating two lookup tables to numerically define the phase transition curve  $g(\phi)$  using DataInterpolations.j1, we propose the use of Chebyshev Polynomials for a direct estimation of  $d$ . We provide an example implementation using ApproxFun.j1 here.
C Numerical precision of SparseProp
Since SparseProp employs event-based simulation with analytical expressions for calculating the next spike times, the algorithm operates with a precision limited only by machine epsilon. Numerically, over many spike events, the global phase offset  $\Delta \phi$  incrementally increases. For sufficiently large values of  $\Delta \phi$ , catastrophic cancellation can arise. For instance, when  $\Delta \phi = 10^6$  and this is subtracted from a very small  $d\phi$ . Under these conditions, the issue is that the adjacent floating-point number greater than  $10^6$  in double-precision arithmetic differs by approximately  $2^{-30} \approx 1.16 \times 10^{-10}$ . One can mitigate this numerical error by resetting both the global phase shift and the phases of individual neurons upon exceeding a specified threshold. This counteracts the subtractive cancellation errors inherent to floating-point calculations. Nonetheless, such resets are seldom required and occur at intervals on the order of  $cN$  steps. The coefficient  $c$  is a substantial prefactor contingent on the tolerable floating-point error; for an error of  $2^{-30}$ ,  $c$  would be  $10^6$ . This can be verified in Julia with the command log2(nextfloat(1.0e6)-1.0e6).
D Online Sparse Erdős-Rényi Connectivity Matrix Generation
In order to benchmark the simulation of large networks using the SparseProp algorithm in Fig 3, we implemented online generation of the sparse Erdős–Rényi connectivity matrix. This approach allows us to generate the set of postsynaptic neurons deterministically online, without the need to store the entire weight matrix. Storing the entire adjacency matrix for networks with  $10^{9}$  neurons would be impractical due to its prohibitively large size, even when using the compressed sparse column format. By using this online generation method, we only require memory to store the state of all  $N$  neurons. For a network of  $10^{9}$  neurons, this corresponds to approximately 64bit/double float  $\times 10^{9}$  neurons  $\times$  8bytes/bit = 7.4 GB of RAM. Our implementation draws inspiration from previous works [76, 77]. It is important to note that the example code provided in appendix A has a fixed indegree, similar to the approach in [9]. Here, we present an example of online sparse adjacency matrix generation with truly Erdős–Rényi connectivity, where the indegree follows a binomial distribution.
Code example:
This code for Julia v1.9 implements a sparse Erdős-Rényi connectivity matrix for efficiently simulating a spiking network.
using Random, DataStructures, RandomNumbers. Xorshifts, StatsBase
# before simulation:  
N, K, j, seedNet = 10^6, 10^2, 2, 1  
rng = Xoroshiro128Star(seedNet) # init. random number generator (Xorshift)  
npostAll = Array{UInt16,1} (undef, N)  
for n = 1:N  
    npostAll[n] = UInt16(randbinom(rng, N, K / N))  
end  
# before simulation when neuron j spikes:  
npost = npostAll[j]  
# spiking neuron index is seed ofrng  
Random.seed!(rng, j)  
# get postsynaptic neuron index  
postIdxes = sample(rng, 1:N-1, npost, replace = false)  
# avoid autapses  
for np = 1:npost  
    if postIdxes[np] >= j  
        postIdxes[np] += 1  
end  
end  
# before simulation, the following function was used  
function randbinom(rng, n::Integer, p::Real)  
log_q = log(1.0 - p)  
x = 0  
sum = 0.0  
while true  
    sum += log(rand(rng)) / (n - x)  
    sum < log_q && break  
    x += 1  
end  
return x
E Spiking Networks in the Balanced State
In order to provide a comprehensive understanding and ensure the accessibility of our work, we now introduce the concept of the balanced state, which motivates the scaling of the synaptic coupling strength with  $1 / \sqrt{K}$ . Neural activity in cortical tissue has typically asynchronous and irregular pattern of action potentials [78, 79], despite the fact that individual neurons can respond reliably [80, 81, 82, 83]. This is commonly explained by a balance of excitatory and inhibitory synaptic currents [84, 85], which cancels large mean synaptic inputs on average. A dynamically self-organized balance can be achieved without the fine-tuning of synaptic coupling strength in heterogeneous networks, if the connectivity is inhibition-dominated and the couplings are strong such that a small active fraction of incoming synapses can trigger an action potential [86]. The statistical properties of this state are described by a mean-field theory, which is largely insensitive to the specific neuron model [87].
In our study, we investigated large sparse networks consisting of  $N$  neurons arranged on a directed Erdős-Rényi random graph with a mean degree of  $K$ . All neurons  $i = 1,\dots ,N$  received constant positive external currents  $I^{\mathrm{ext}}$  and non-delayed  $\delta$ -pulses from the presynaptic neurons  $j\in \operatorname {pre}(i)$
The values of the external currents were chosen using a bisection method to achieve a target average network firing rate  $\bar{\nu}$ .
F Setup of the Balanced Networks
The coupling strengths in our networks were scaled with  $1 / \sqrt{K}$ , such that the magnitudes of the input current fluctuations had the same order of magnitude in all studied networks. Assuming that inputs from different presynaptic neurons exhibit weak correlations, which is justified in balanced network [13, 88], the collective input spike train received by neuron  $i$  can be effectively modeled as a Poisson process with a rate denoted as  $\Omega_{i} = \sum_{j\in \mathrm{pre}(i)}\nu_{j}\approx K\bar{\nu}\equiv \Omega$ . Here,  $\bar{\nu}$  represents the average firing rate of the network, and  $K$  denotes the average number of presynaptic neurons. Assuming the compound input spike train follows a Poisson process, the autocorrelation function of the input current can be expressed as follows:
$$
\begin{array}{l} C (\tau) = \langle \delta I (t) \delta I (t + \tau) \rangle_ {t} (14) \\ \approx \left(\frac {J _ {0}}{\sqrt {K}}\right) ^ {2} \Omega \int \delta (t - s) \delta (t + \tau - s) d s (15) \\ = \frac {J _ {0} ^ {2}}{K} \Omega \delta (\tau) (16) \\ \approx J _ {0} ^ {2} \bar {\nu} \delta (\tau) (17) \\ \end{array}
$$
where  $\delta$  denotes the Dirac delta function. In the diffusion approximation, characterized by high Poisson rate and weak coupling, fluctuations in the input currents can be described as Gaussian white noise with a magnitude given by
$$
\sigma^ {2} = J _ {0} ^ {2} \bar {\nu}. \tag {18}
$$
Note that due to the scaling of the coupling strengths  $J = -\frac{J_0}{\sqrt{K}}$  with the square root of the number of synapses  $K$  the magnitude of the fluctuations  $\sigma^2$  is independent of the number of synapses. Therefore, the input fluctuations do neither vanish nor diverge in the thermodynamic limit and the balanced state in sparse networks emerges robustly [86, 87].
The existence of a fixed point of the population firing rate in the balanced state for large  $K$  follows from the equation of the network-averaged mean current:
$$
\bar {I} \approx \sqrt {K} \left(I _ {0} - J _ {0} \bar {\nu}\right). \tag {19}
$$
In the limit of large  $K$ , self-consistency requires the balance between excitation and inhibition, specifically  $I_0 = J_0\bar{\nu}$ . If  $\lim_{K\to \infty}(I_0 - J_0\bar{\nu}) > 0$ , the mean current  $\bar{I}$  would diverge to  $\infty$ , resulting in neurons firing at their maximum rate. The resulting strong inhibition would break the inequality, leading to a contradiction. On the other hand, if  $\lim_{K\to \infty}(I_0 - J_0\bar{\nu}) < 0$ , the mean current  $\bar{I}$  would diverge to  $-\infty$ , causing the neurons to be silent. Again, the lack of inhibition breaks the inequality. The self-consistency condition in the large  $K$  limit can only be satisfied when
$$
I _ {0} - J _ {0} \bar {\nu} = \mathcal {O} \left(\frac {1}{\sqrt {K}}\right),
$$
which ensures that the excitatory external drive and the mean recurrent inhibitory current cancel each other. Note that since  $I_0 - J_0\bar{\nu} = \mathcal{O}(1 / \sqrt{K})$  the network mean current has a finite value in the large  $K$ -limit. The average population firing rate, expressed in units of the membrane time constant  $\tau_{\mathrm{m}}^{-1}$ , can be approximated as
$$
\bar {\nu} = \frac {I _ {0}}{J _ {0}} + \mathcal {O} \left(\frac {1}{\sqrt {K}}\right). \tag {20}
$$
This approximation generally becomes exact for large  $K$ . For excitatory-inhibitory mixed network, an analogous self-consistency argument can be made which results in a set of inequalities that must be fulfilled to achieve a balanced state [87].
G Code Availability
An example implementation of networks of LIF, QIF and EIF neurons using SparseProp in Julia, in the balanced state, is available at https://github.com/RainerEngelken/SparseProp.