SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} \left| u _ {i} ^ {(l)} [ t ] - u _ {t h} \right| > \left| z _ {k} \right| \delta , \quad \text {f o r} k = 1, \dots , m, \\ \operatorname {o r}, \left| u _ {i} ^ {(l)} [ t ] - u _ {t h} \right| > t \delta , \text {w h e r e} t = \max  _ {k} \left\{\left| z _ {1} \right|, \dots , \left| z _ {m} \right| \right\} \\ \end{array}
$$
Assume  $z_{k} \sim \lambda$ , where  $\lambda(t)$  denotes the PDF of the sampling distribution, with the corresponding CDF denoted by  $F_{z_k}$ . The PDF,  $\tilde{\lambda}$ , of the random variable  $|z_k|$  is given by
$$
\tilde {\lambda} (x) = \left\{ \begin{array}{l l} 0, & \text {i f} x <   0 \\ 2 \lambda (x), & \text {o t h e r w i s e .} \end{array} \right. \tag {17}
$$
The corresponding CDF is obtained by integrating the previous expression,
$$
F _ {\left| z _ {k} \right|} (x) = \left\{ \begin{array}{l l} 0, & \text {i f} x <   0 \\ 2 \left(F _ {z _ {k}} (x) - F _ {z _ {k}} (0)\right), & \text {o t h e r w i s e .} \end{array} \right. \tag {18}
$$
Further note that,
$$
F _ {t} (x) = P (t <   x) = \prod_ {k = 1} ^ {m} P \left(\left| z _ {k} \right| <   x\right) = F _ {\left| z _ {k} \right|} ^ {m} (x) \tag {19}
$$
If we denote the PDF of the random variable  $t$  as  $\hat{\lambda}$ , we obtain
$$
\hat {\lambda} (x) = m F _ {| z _ {k} |} ^ {m - 1} (x) \tilde {\lambda} (x). \tag {20}
$$
Finally, the expected back-propagation threshold takes the form
$$
\tilde {B} _ {t h} = \delta \mathbb {E} [ t ] = \delta \int_ {0} ^ {\infty} t \hat {\lambda} (t) d t. \tag {21}
$$
In the cases of distributions used in experimental sections, the previous expression simplifies. Table 1 gives the numerical values for some particular  $m$ . To obtain an expected back-propagation threshold, we would like to evaluate:
$$
\tilde {B} _ {t h} = \delta \mathbb {E} [ t ] = \delta \int_ {0} ^ {\infty} t \hat {\lambda} (t) d t = \delta m \int_ {0} ^ {\infty} t F _ {| z |} ^ {m - 1} (t) \tilde {\lambda} (t) d t
$$
For the standard normal distribution,  $\lambda = \mathrm{Normal}(0,1)$  we have,  $F_{|z|}(t) = \mathrm{erf}\left(\frac{t}{\sqrt{2}}\right)$  giving,
$$
\tilde {B} _ {t h} = \frac {2 \delta m}{\sqrt {2 \pi}} \int_ {0} ^ {\infty} t \operatorname {e r f} ^ {m - 1} \left(\frac {t}{\sqrt {2}}\right) \exp \left(- \frac {t ^ {2}}{2}\right) d t. \tag {22}
$$
For uniform continuous,  $\lambda = \mathrm{Unif}([-\sqrt{3},\sqrt{3}])$ , we have,  $F_{|z|}(t) = \frac{t}{\sqrt{3}}$  giving,
$$
\tilde {B} _ {t h} = \frac {\delta m}{\sqrt {3}} \int_ {0} ^ {\sqrt {3}} t \left(\frac {t}{\sqrt {3}}\right) ^ {m - 1} d t = \delta \sqrt {3} \frac {m}{m + 1}. \tag {23}
$$
For Laplace distribution,  $\lambda = \mathrm{Laplace}(0,\frac{1}{\sqrt{2}})$ , we have,  $F_{|z|}(t) = 1 - \exp (-\sqrt{2} t)$
$$
\tilde {B} _ {t h} = \delta m \sqrt {2} \int_ {0} ^ {\infty} t \left(1 - \exp (- \sqrt {2} t)\right) ^ {m - 1} \exp (- \sqrt {2} t) d t. \tag {24}
$$
Appendix B Additional Results
B.1 Computational Speedup in SPARSEGRAD
The SPARSEGrad framework derives speedup by performing the back-propagation in a layerwise fashion. To summarize their finding, let us start with eqn. 1.
$$
u _ {i} ^ {(l)} [ t ] = \beta u _ {i} ^ {(l)} [ t - 1 ] + \sum_ {j} w _ {i j} x _ {j} ^ {(l - 1)} [ t ] - x _ {i} ^ {(l)} [ t - 1 ] u _ {t h}
$$
which, after unfolding the recurrence and using the fact that  $u_{i}^{(l)}[0] = 0$ , can be restated as:
Figure 4: We plot training loss, backward speedup (Back.), and overall speedup (Over.) of SPARSEGRAD and LOCALZO after each gradient update performed on NMNIST data, as summarized in Table 5. The columns represent plots for three distributions Normal(0,1), Laplace(1,  $\frac{1}{\sqrt{2}}$ ), and Unif([-√3,√3]) respectively, with  $\delta = 0.05$  and  $m = 1$ . The LOCALZO algorithm converges faster than the SPARSEGRAD method and provides comparable backward and overall speedup.
$$
u _ {i} ^ {(l)} [ t ] = \sum_ {j} \sum_ {k = 0} ^ {t - 1} \beta^ {t - k - 1} x _ {j} ^ {(l - 1)} [ k ] w _ {i j} - \sum_ {k = 0} ^ {t - 1} \beta^ {t - k - 1} x _ {i} ^ {(l)} [ k ] u _ {t h} \tag {25}
$$
where, the input trace,  $\sum_{k=0}^{t-1} \beta^{t-k-1} x_j^{(l-1)}[k]$  can be computed in a layer-wise fashion from the forward propagation. Note that, by ignoring the reset mechanism in the gradient the computation, the gradient of the loss can be written as:
$$
\begin{array}{l} \frac {\partial l}{\partial w _ {i j}} = \sum_ {t} \frac {\partial l [ t ]}{\partial x [ t ]} \frac {\partial x [ t ]}{\partial u [ t ]} \frac {\partial u [ t ]}{\partial w _ {i j}} \\ = \sum_ {t} \frac {\partial l [ t ]}{\partial x [ t ]} \frac {\partial x [ t ]}{\partial u [ t ]} \sum_ {k = 0} ^ {t - 1} \beta^ {t - k - 1} x _ {j} ^ {(l - 1)} [ k ] \tag {26} \\ \end{array}
$$
where, the term  $\frac{\partial l[t]}{\partial x[t]}$  is coming from the next layer, the term  $\frac{\partial x[t]}{\partial u[t]}$  is the given by 5, so it can be computed in the forward propagation along with the input trace. Thus, whenever, many neurons are inactive, i.e.,  $\frac{\partial x[t]}{\partial u[t]} = 0$  the SPARSEGRAD frame-work can reduce the computation burden of the back-propagation.
B.2 Further comparison with SPARSEGRAD
In the section 4.4, we derived surrogates corresponding to distributions and distributions corresponding to popular surrogates. We implement LOCALZO, with  $\delta = 0.05$  for different distribution
Table 5: Performance comparison on NMNIST for  $\mathrm{m} = 1$
METHOD	TRAIN	TEST	BACK.	OVER.
z ~ NORMAL(0,1), δ = 0.05, m = 1
SPARSEGRAD	93.26 ± 0.31	91.86± 0.29	99.57	3.38
LOCALZO	94.38 ± 0.12	93.29± 0.08	92.27	3.34
z ~ LAPLACE(1,1/√2), δ = 0.05, m = 1
SPARSEGRAD	93.97 ± 0.43	92.65± 0.52	88.2	3.19
LOCALZO	94.25 ± 0.17	93.05± 0.09	83.7	3.07
z ~ UNIF([-√3,√3]), δ = 0.05, m = 1
SPARSEGRAD	93.34 ± 0.44	91.85± 0.35	83.2	3.26
LOCALZO	94.24 ± 0.46	93.05± 0.37	84.8	3.43
SIGMOID, δ = 0.05, k ≈ 30.63, m = 1
SPARSEGRAD	92.96± 0.26	91.04± 0.32	87.45	3.00
LOCALZO	93.98± 0.08	92.97± 0.05	83.54	3.02
FASTSIGMOID, δ = 0.05, k = 100, m = 1
SPARSEGRAD	93.24± 0.23	92.16± 0.20	84.87	3.18
LOCALZO	93.44± 0.13	92.52± 0.09	73.23	3.11
Table 6: Performance comparison on NMNIST for  $\mathrm{m} = 5$
METHOD	TRAIN	TEST	BACK.	OVER.
z ~ NORMAL(0,1), δ = 0.05, m = 5
SPARSEGRAD	95.02 ± 0.29	93.39± 0.25	80.0	3.40
LOCALZO	95.20 ± 0.22	93.69± 0.17	77.7	3.22
z ~ LAPLACE(1, 1/√2), δ = 0.05, m = 5
SPARSEGRAD	94.73 ± 0.29	93.13± 0.23	72.9	3.15
LOCALZO	95.07 ± 0.03	93.63± 0.05	69.4	2.80
z ~ UNIF([-√3,√3]), δ = 0.05, m = 5
SPARSEGRAD	94.82 ± 0.27	93.38± 0.17	76.4	3.14
LOCALZO	94.95 ± 0.35	93.47± 0.23	73.5	2.91
such as Normal(0,1), Laplace  $(1,\frac{1}{\sqrt{2}})$ , and Unif  $([-\sqrt{3},\sqrt{3}])$ . They all have a unit variance to ensure  $\delta$  is comparable across the distributions. The corresponding back-propagation thresholds for SPARSEGRAD are derived in Table 1.
For the Sigmoid surrogate, we take the temperature parameter  $k = a / \delta \approx 30.63$  so that  $c = \frac{\delta^2k^2}{a^2} = 1$  for the corresponding PDF derived in eqn.14. We supply SPARSEGRAD method the corresponding back-propagation threshold,  $\tilde{B}_{th} = 0.766\delta$ , as obtained in eqn. 15.
For the Fast Sigmoid surrogate, we choose  $k = 100$  following [31] so that  $c = \frac{2}{k}$ . To compute the expected back-propagation threshold, we consider finite support  $[-10, 10]$  used in the inverse transform sampling of  $z$  and evaluate  $\tilde{B}_{th} = 0.0461$ . Table 5 reports accuracies obtained by LOCALZO and SPARSEGRAD across various sampling distributions, with  $m = 1$ . LOCALZO offers better accuracies compared to SPARSEGRAD across all the distributions, with a slight compromise in speed-up in most cases. For uniform distribution, LOCALZO offers even better speed-up than SPARSEGRAD.
Table 6 reports the details of the comparison over the N-MNIST dataset for sampling  $z$  from various distributions with  $m = 5$ . LOCALZO method obtains better test accuracies. The SPARSEGRAD method is supplied with the respective surrogate and back-propagation threshold as computed in
Table 7: Hyper-parameter settings for general comparison
CIFAR-10/100	ImageNet-100	DVS-CIFAR-10	DVS-Gesture	N-Caltech/NCARS
Number epochs	300	300	300	200	200
Mini batch size	64	64, 72	64	64	16
T	6,4,2	4	10	10	10
LIF: β	0.5	1	0.5	0.5	0.5
LIF: u0	0	0	0	0	0
LIF: uth	1	1	1	1	1
LOCALZO: δ	0.5	0.5	0.5	0.5	0.5
LOCALZO: m	5	5, 20	1	5	5
LOCALZO: λ	N(0,1)	N(0,1)	N(0,1)	N(0,1)	N(0,1)
λTET	0.05	0.001	0.0001	0.05	0.05
Learning Rate	0.001	0.1	0.001	0.001	0.001
Optimizer: Adam with betas: (0.9; 0.999), Rate Scheduler: cosine annealing
Table 8: Hyper-parameter settings for comparison in SPARSEGRAD framework
FMNIST	SHD	N-MNIST
Number of Input Neurons	784	1156	700
Number of Hidden	200	200	200
Number of classes	10	10	20
Number of epochs	20	20	20
Mini batch size	256	256	256
T	100	300	500
Δt	1ms	1ms	2ms
τeff	20ms	N/A	N/A
θ	0.2	N/A	N/A
u0	0	0	0
uth	1	1	1
LOCALZO: δ	0.05	0.05	0.05
Optimiser	Adam	Adam	Adam
Learning Rate	0.0002	0.0002	0.001
Betas	(0.9; 0.999)	(0.9; 0.999)	(0.9; 0.999)
Tab.1. The LOCALZO method achieves better accuracies than SPARSEGRAD for all the distributions, though the speed-up is reduced due to higher sampling cost at  $m = 5$ .
In table 8, we further provide the hyper-parameters for the comparison in the SPARSEGRAD framework, which are replicated from their work. The FMNIST data uses latency encoding, where each input pixel  $x$ , is converted to a single spike, the spike timing is determined by:
$$
T (x) = \left\{\tau_ {\mathrm {e f f}} \log \frac {x}{x - \theta} \right.
$$
Table 7 gives the hyper-parameter setting for comparison in general framework reported in Tab. 2.
B.3 Dropout effect in LOCALZO
A fixed neuron has an active gradient for  $m = 1$ , only if  $|u_i[t] - u_{th}| < |z|\delta$  as described in eqn5. Observe that  $u_i[t]$  is specific to an input data point and time-step. The same neuron can be inactive for another data-point, or at a different time-step. Moreover, a neuron can have a non-zero gradient with respect to a single data point, even if the spikes are zero. Fixing a neuron and time-step, we plot the distribution of membrane potential (minus membrane threshold), spikes, and non-zero gradients over different data points across batches, where we implement LOCALZO on MNIST data with  $\delta = 0.5$ . The plot captures the sparsity of spikes and sparsity of neuron gradients for LOCALZO.
Figure 5: Fixing a neuron, we plot the distribution of membrane potential, spikes, and gradient activity over different data points across batches, where we implement LOCALZO on MNIST data with  $\delta = 0.5$ . The plot captures the sparsity of spikes and sparsity of neuron gradients for LOCALZO at different time-steps.