SlowGuess/NeurIPS_2023_OCR huggingface dataset:
bined LOCALZO with contemporary state-of-the-art methods in direct training of SNNs, such as tDBN [45] and TET [13]. We opted for these two techniques due to their high performance and different natures (the former is a batch normalization technique, while the latter introduces auxiliary loss in training). We refer to the combined methods as LocalZO+tDBN and LocalZO+TET, respectively. Following the recent results, we choose ResNet-19[45] architecture for CIFAR datasets, SEW-ResNet-34[17] for ImageNet-100, and VGGSNN[13] architecture for the neuromorphic datasets. Table2 summarizes these results, where LOCALZO is implemented with  $m = 5$ ,  $\delta = 0.5$ , except for DVS-CIFAR-10 where we have used  $m = 1$ . Table 7 reports the detailed training hyper-parameters for each dataset.
Results on Static Datasets: The static datasets CIFAR-10, CIFAR-100, and ImageNet-100, have the number of classes mentioned in the dataset name, while each class respectively have (5000, 1000), (500, 100), and (1300, 50) train and test images. We use constant encoding to supply the images to the SNN network, with standard latencies as mentioned in Table 2. We train separate models for different latencies and report the test results respectively. The results of CIFAR datasets are reported with cutout augmentation following TET[13], while for ImageNet-100, we use standard data augmentation (random resized crop, random horizontal flip), without and with the ImageNet Policy[9]. We note that in conjunction with LOCALZO, both tDBN and TET improve their performance significantly for all the latencies. For example, in CIFAR-10, LOCALZO improves TET between  $0.9 - 1\%$  for different latencies, while for CIFAR-100, it improves TET by  $2.5 - 3.5\%$ . The ImageNet-100 results reported in Table 2 were further enhanced to  $83.33\%$  with  $m = 20$  and batch size 72.
Results on Neuromorphic Datasets: Events of the neuromorphic datasets are collected into event frames of dimension  $(2\times H\times W)$  where H and W stand for the height and width of the frame and are resized to  $(2\times 48\times 48)$ . The temporal events are collected into a fixed number (10) of frames (a.k.a. bins), treated as the effective temporal dimension for the SNN. The experiments are reported without and with standard data augmentation. For DVS-CIFAR-10, the TET result is re-computed to avoid an obscure frame preparation step, which is replaced by an open-source routine. We obtain results superior to the state-of-the-art for DVS-CIFAR-10[13] and DVS-Gesture[17]. For N-Caltech-101 and NCARS, the improvements are  $1.2\%$  and  $2.4\%$ , respectively, compared to the state-of-the-art[18, 35].
Table 2: Comparison with the existing methods show that LOCALZO improves the accuracy of existing direct training algorithms. For the existing methods, we compare the performance with the results reported in respective literatures. For the rows with two accuracies reported, the second one is for training with additional augmentation.
Dataset	Methods	Architecture	Simulation Length	Accuracy
CIFAR10	Hybrid training[34]	ResNet-20	250	92.22
Diet-SNN[33]	ResNet-20	10	92.54
STBP[38]	CIFARNet	12	89.83
STBP NeuNorm[39]	CIFARNet	12	90.53
TSSL-BP[44]	CIFARNet	5	91.41
tDBN[45]	ResNet-19	6	93.16
4	92.92
2	92.34
LOCALZO +tDBN	ResNet-19	6	95.07
4	94.89
2	94.65
TET[13]	ResNet-19	6	94.50
4	94.44
2	94.16
LOCALZO +TET	ResNet-19	6	95.56
4	95.3
2	95.03
CIFAR100	Hybrid training[34]	VGG-11	125	67.87
Diet-SNN[33]	ResNet-20	5	64.07
tDBN[45]	ResNet-19	6	71.12
4	70.86
2	69.41
LOCALZO +tDBN	ResNet-19	6	73.74
4	74.13
2	72.78
TET[13]	ResNet-19	6	74.72
4	74.47
2	72.87
LOCALZO +TET	ResNet-19	6	77.25
4	76.89
2	76.36
ImageNet-100	EfficientLIF-Net[21]	ResNet-19	5	79.44
LOCALZO +TET	SEW-Resnet34	4	78.58, 81.56‡
DVS-CIFAR10	tdBN[45]	ResNet-19	10	67.8
Streaming Rollout [23]	DenseNet	10	66.8
Conv3D[40]	LIAF-Net	10	71.70
LIAF[40]	LIAF-Net	10	70.40
TET[13]	VGGSNN	10	74.89*, 81.45*
LOCALZO +tDBN	VGGSNN	10	72.6, 79.37
LOCALZO +TET	VGGSNN	10	75.62, 81.87
N-Caltech-101	AEGNN[35]	GNN	-	66.8
EST[18]	ResNet-34†	9	81.7
LOCALZO +tDBN	VGGSNN	10	74.65, 79.05
LOCALZO +TET	VGGSNN	10	79.86, 82.99
N-CARS	AEGNN[35]	GNN	-	94.5
EST[18]	ResNet-34†	9	92.5
LOCALZO +tDBN	VGGSNN	10	95.96, 95.68
LOCALZO +TET	VGGSNN	10	96.78, 96.96
DVS-Gesture	SEW[17]	SEW-Resnet	16	97.92
LOCALZO +TET	VGGSNN	10	98.04, 98.43
* our implementation, †pre-trained with ImageNet, ‡ 83.33 % with  $m = {20}$
5.2 Performance on Energy Efficient Implementation
In the energy-efficient implementation of the back-propagation [31], the optimization of the network weights takes place in a layer-wise fashion through the unrolling of recurrence of equation (1) w.r.t time. As the active neurons of each layer for every time step are inferred from the forward pass, gradients of only active neurons are required to be saved for the backward pass, hence saving the computation requirement of the backward pass. One may refer to [31] for further details of this implementation framework. To compare, we supply SPARSEGRAD method the surrogate approximated by LOCALZO, as per section 4.4. The SPARSEGRAD algorithm also requires a backpropagation threshold parameter,  $B_{th}$ , to control the number of active neurons participating in the back-propagation. We supply it the expected back-propagation threshold  $\tilde{B}_{th}$  of LOCALZO as obtained in sections 4.5. We follow the same experimental setting as in [31] for a fair comparison. We use a fully connected LIF neural network with two hidden layers of 200 neurons each and input and output layers. We train every model for 20 epochs and report the average training and test accuracies computed over five trials. We compute the speedup of SPARSEGRAD and LOCALZO, with respect to the full surrogate without truncation, that uses standard back-propagation. The backward speedup (Back.) captures the number of times the backward pass of a gradient update is faster, while the overall speedup (Over.) considers the total time for the forward and the backward pass and then computes the speedup. The speedup reported is averaged over all the gradient updates and the experimental trials.
We compare the performance of the algorithms on three datasets: 1) Neuromorphic-MNIST (NMNIST) [30], which consists of static images of handwritten digits (between 0 and 9) converted to temporal spiking data using visual neuromorphic sensors; 2) Spiking Heidelberg Digits (SHD) [8], a neuromorphic audio dataset consisting of spoken digits (between 0 and 9) in English and German language, totalling 20 classes. To challenge the generalizability of the learning task,  $81\%$  of test inputs of this dataset are new voice samples, which are not present in the training data; 3) Fashion-MNIST (FMNIST) [41] dataset is converted using temporal encoding to convert static gray-scale images based on the principle that each input neuron spikes only once, and a higher intensity spike results in an earlier spike.
Table 3 provides a comparison of the algorithms, using surrogates corresponding to the Normal and Sigmoid, with  $\delta = 0.05$  and  $m = 1$ . For the normal distribution, we supply SPARSEGRAD algorithm the backpropagation threshold  $\tilde{B}_{th}$  obtained in Table 1. In the section 4.4, we derived distributions corresponding to the Sigmoid surrogate. We use inverse transform sampling (see A.2.3), and take the temperature parameter  $k = a / \delta \approx 30.63$  so that  $c = \frac{\delta^2 k^2}{a^2} = 1$  and supply SPARSEGRAD method the corresponding back-propagation threshold,  $\tilde{B}_{th} = 0.766\delta$ . The LocalZO method offers better test accuracies than SPARSEGRAD, with a slight compromise in speedup due to the sampling of random variable  $z$ . The difference between training and test accuracies for the SHD dataset can be attributed to the unseen voice samples in the test data[8].
Figure 2 shows the training loss, overall speedup, and percentage of active
Table 3: Performance on NMNIST, SHD and FMNIST
Method	Train	Test	Back.	Over.
NMNIST	z ~ Normal(0,1), δ = 0.05, m = 1
SPARSEGRAD	93.26 ± 0.31	91.86± 0.29	99.57	3.38
LOCALZO	94.38 ± 0.12	93.29± 0.08	92.27	3.34
Sigmoid, δ = 0.05, k ≈ 30.63, m = 1
SPARSEGRAD	92.96± 0.26	91.04± 0.32	87.45	3.00
LOCALZO	93.98± 0.08	92.97± 0.05	83.54	3.02
SHD	z ~ Normal(0,1), δ = 0.05, m = 1
SPARSEGRAD	92.03± 0.79	74.73± 0.73	143.7	4.83
LOCALZO	91.77± 0.27	76.55± 0.93	142.8	4.75
Sigmoid, δ = 0.05, k ≈ 30.63, m = 1
SPARSEGRAD	92.19± 0.41	75.80± 0.97	140.8	4.46
LOCALZO	91.96± 0.11	76.97± 0.40	133.6	4.36
FMNIST	z ~ Normal(0,1), δ = 0.05, m = 1
SPARSEGRAD	81.91± 0.10	80.28± 0.11	15.74	1.97
LOCALZO	83.83± 0.07	81.79± 0.06	15.49	1.88
Sigmoid, δ = 0.05, k ≈ 30.63, m = 1
SPARSEGRAD	81.60± 0.11	80.02± 0.08	12.12	1.65
LOCALZO	83.39± 0.10	81.76± 0.10	12.50	1.57
neurons after each gradient step for the Sigmoid surrogate. The sparseness of active neurons (under  $0.6\%$ ) explains the reduced computational requirement that translates to the speedup.
We further implement LOCALZO with  $\delta = 0.5, z \sim$  Normal(0,1) to train a CNN architecture (Input- 16C5-BN-LIF-MP2-32C5-BN-LIF-MP2-800FC-10) and compare it with the corresponding surrogate gradient algorithm. Figure 3 shows the corresponding sparsity of the methods by plotting the number of zero elements at a neuronal level. The plot suggests that during the training, LOCALZO exhibits higher sparsity gradients than the surrogate method.
Ablation study: Table4 further shows the test accuracy of the LOCALZO method and overall speed-up for a wide range of values of  $m$  with  $z \sim \mathrm{Normal}(0,1)$ . Like Table 3, the experiments are repeated five times and mean test accuracy is reported along with standard deviation (Std.).
In general, by increasing  $m$ , the method approximates the surrogate better, still offers the regularizing effect and potentially improves the generalization, but also requires more computation. Larger  $m$  leads to more non-zero gradients at the neuronal level in the backward pass, reducing overall speed-up. On the other hand, smaller  $m$  introduces higher randomness (less "controlled"), still yielding regularization, which helps obtain better generalization, as well as potential speed-up. In conclusion,  $m$  should be treated as a hyper-parameter, its value depending on the training setting itself. In our experiments, we chose  $m = 1$  or 5 for most of the experiments, as a proof of concept, but also because it offers a nice balance between the speed-up and performance.
Figure 3: Comparison of gradient sparsity in CNN architecture
Table 4: Trade-off of accuracy vs. speedup with hyper-parameter  $m$
m	1	3	5	7	10	20	100
NMNIST
Accuracy	93.29	93.61	93.69	93.66	93.76	93.67	93.81
Std.	0.08	0.15	0.17	0.13	0.14	0.08	0.14
Over.	3.33	3.28	3.22	3.16	3.06	2.82	1.59
SHD
Accuracy	76.55	76.55	76.50	75.49	75.51	74.96	76.71
Std.	0.93	0.65	0.90	0.66	0.81	0.68	0.49
Over.	4.75	4.62	4.47	4.39	4.25	3.89	2.24
FMNIST
Accuracy	81.79	83.40	83.64	83.70	83.85	83.75	83.87
Std.	0.06	0.06	0.12	0.04	0.11	0.11	0.05
Over.	1.89	1.85	1.78	1.75	1.70	1.56	0.88
We propose a new direct training algorithm for SNNs that establishes a formal connection between the standard surrogate methods and the zeroth order method applied locally to the neurons. The method introduces systematic randomness in the training that helps in better generalization. The method simultaneously lends itself to efficient back-propagation. We experimentally demonstrate the efficiency of the proposed method in terms of speed-up obtained in training under specialized implementations and its top generalization performance when combined with other training methods, ameliorating their respective strengths.
This work is part of the research project "ENERGY-BASED PROBING FOR SPIKING NEURAL NETWORKS" performed at Mohamed bin Zayed University of Artificial Intelligence (MBZUAI), in collaboration with Technology Innovation Institute (TII) (Contract No. TII/ARRC/2073/2021).
[34] Nitin Rathi, Gopalakrishnan Srinivasan, Priyadarshini Panda, and Kaushik Roy. Enabling deep spiking neural networks with hybrid conversion and spike timing dependent backpropagation. arXiv preprint arXiv:2005.01807, 2020.
[35] Simon Schaefer, Daniel Gehrig, and Davide Scaramuzza. Aegnn: Asynchronous event-based graph neural networks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 12371-12381, 2022.
[36] Sumit B Shrestha and Garrick Orchard. Slayer: Spike layer error reassignment in time. Advances in neural information processing systems, 31, 2018.
[37] Amos Sironi, Manuele Brambilla, Nicolas Bourdis, Xavier Lagorce, and Ryad Benosman. Hats: Histograms of averaged time surfaces for robust event-based object classification. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1731-1740, 2018.
[38] Yujie Wu, Lei Deng, Guoqi Li, Jun Zhu, and Luping Shi. Spatio-temporal backpropagation for training high-performance spiking neural networks. Frontiers in neuroscience, 12:331, 2018.
[39] Yujie Wu, Lei Deng, Guoqi Li, Jun Zhu, Yuan Xie, and Luping Shi. Direct training for spiking neural networks: Faster, larger, better. In Proceedings of the AAAI conference on artificial intelligence, volume 33, pages 1311-1318, 2019.
[40] Zhenzhi Wu, Hehui Zhang, Yihan Lin, Guoqi Li, Meng Wang, and Ye Tang. Liaf-net: Leaky integrate and analog fire network for lightweight and efficient spatiotemporal information processing. IEEE Transactions on Neural Networks and Learning Systems, 33(11):6249-6262, 2021.
[41] Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms. arXiv preprint arXiv:1708.07747, 2017.
[42] Yulong Yan, Haoming Chu, Yi Jin, Yuxiang Huan, Zhuo Zou, and Lirong Zheng. Backpropagation with sparsity regularization for spiking neural network learning. Frontiers in Neuroscience, 16, 2022.
[43] Friedemann Zenke and Surya Ganguli. Superspike: Supervised learning in multilayer spiking neural networks. Neural computation, 30(6):1514-1541, 2018.
[44] Wenrui Zhang and Peng Li. Temporal spike sequence learning via backpropagation for deep spiking neural networks. Advances in Neural Information Processing Systems, 33:12022-12033, 2020.
[45] Hanle Zheng, Yujie Wu, Lei Deng, Yifan Hu, and Guoqi Li. Going deeper with directly-trained larger spiking neural networks. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 11062-11070, 2021.
Appendix: Direct Training of SNN using Local Zeroth Order Method
Appendix A Proofs of theoretical results
Proof. Based on our remark above, the only thing left to prove is that the integral  $\int_{-\infty}^{\infty}\mathbb{E}_{z\sim \lambda}[G^{2}(u;z,\delta)]du$  is finite. To this end, we have (by using equation (8))
$$
\begin{array}{l} \int_ {- \infty} ^ {\infty} \mathbb {E} _ {z \sim \lambda} [ G ^ {2} (u; z, \delta) ] d u = \int_ {- \infty} ^ {\infty} \frac {1}{\delta} \int_ {\frac {| u |}{\delta}} ^ {\infty} z ^ {\alpha} \lambda (z) d z d u = \frac {2}{\delta} \int_ {0} ^ {\infty} \int_ {\frac {| u |}{\delta}} ^ {\infty} z ^ {\alpha} \lambda (z) d z d u \\ = \frac {2}{\delta} \int_ {0} ^ {\infty} \int_ {0} ^ {| z | \delta} | z | ^ {\alpha} \lambda (z) d u d z = 2 \int_ {0} ^ {\infty} z ^ {\alpha + 1} \lambda (z) d z, \\ \end{array}
$$
which proves the lemma, as by assumptions the resulting integral is finite.
$$
\frac {d}{d u} \mathbb {E} _ {z \sim \tilde {\lambda}} [ c h (u + \delta z) ] = c \frac {d}{d u} \int_ {- \infty} ^ {\infty} h (u + \delta z) \tilde {\lambda} (z) d z = c \frac {d}{d u} \int_ {- \frac {u}{\delta}} ^ {\infty} \tilde {\lambda} (z) d z = \frac {c}{\delta} \tilde {\lambda} (- \frac {u}{\delta}) = \frac {c}{\delta} \tilde {\lambda} (\frac {u}{\delta}),
$$
which coincides with (8).
For our following result, note that a surrogate function is differentiable almost everywhere, which follows from the Lebesgue theorem on the differentiability of monotone functions. So, taking derivatives here is understood in an "almost everywhere" sense.
Proof. Let us assume that  $u \geq 0$  (the other case is similar). Then,
$$
\mathbb {E} _ {z \sim \lambda} [ c G ^ {2} (u; z, \delta) ] = \frac {c}{\delta} \int_ {\frac {u}{\delta}} ^ {\infty} z ^ {\alpha} \lambda (z) d z = - \frac {1}{\delta} \int_ {\frac {u}{\delta}} ^ {\infty} z ^ {\alpha} \frac {\delta^ {2}}{z ^ {\alpha}} g ^ {\prime} (z \delta) d z
$$
which after change of variables  $u = \delta z$  becomes  $g(u)$  and finishes our proof.
A.1 Obtaining full-surrogates on Expectation
We demonstrate performance of LOCALZO over different distributions of  $z$ , such as standard Normal, Uniform  $([\sqrt{3},\sqrt{3}])$  and Laplace  $(0,\frac{1}{\sqrt{2}})$ , for  $m \in \{1,5\}$  and  $\delta = 0.05$ . The distributions are of unit variance so that parameter  $\delta$  is comparable across the methods. We supply SPARSEGRAD algorithm the back-propagation threshold  $\tilde{B}_{th}$  obtained in Table 1. Table 5 shows the performance of the methods on the N-MNIST dataset in terms of accuracy and speedup. The LOCALZO method obtains better train and test accuracies for all cases with a slight compromise in the speedup, except for the uniform distribution where it offers better speedup for  $m = 1$  compared to the SPARSEGRAD method.
A.1.1 From standard Gaussian
Recall that the standard normal distribution  $N(0,1)$  has PDF of the form  $\frac{1}{\sqrt{2\pi}}\exp \left(-\frac{z^2}{2}\right)$ . Consequently, it is straightforward to obtain
$$
\mathbb {E} _ {z \sim \lambda} [ G ^ {2} (u; z, \delta) ] = \frac {1}{\sqrt {2 \pi}} \int_ {- \infty} ^ {\infty} \frac {| z |}{2 \delta} \exp \left(- \frac {z ^ {2}}{2}\right) d z = \frac {1}{\delta \sqrt {2 \pi}} \exp \left(- \frac {u ^ {2}}{2 \delta^ {2}}\right). \tag {11}
$$
A.1.2 From Uniform Continuous
Consider the PDF of a continuous uniform distribution:
$$
f (z; a, b) = \left\{ \begin{array}{l l} \frac {1}{b - a} & \text {f o r} z \in [ a, b ] \\ 0 & \text {o t h e r w i s e}, \end{array} \right.
$$
where  $a < b$  are some real numbers. For the distribution to be even and the resulting scaling constant of the surrogate to be 1 (which translates to  $\mathbb{E}[z] = 0$  and  $\mathbb{E}[z^2] = 1$ , respectively) we set,  $a = -\sqrt{3}$ ,  $b = \sqrt{3}$ . Then,
$$
\begin{array}{l} \mathbb {E} _ {z \sim \lambda} [ G ^ {2} (u; z, \delta) ] = \int_ {- \infty} ^ {\infty} \frac {| z |}{2 \delta} f (z) d z \\ = \frac {1}{2 \sqrt {3}} \left[ \int_ {- \sqrt {3}} ^ {- \frac {| u |}{\delta}} \frac {| z |}{2 \delta} d z + \int_ {\frac {| u |}{\delta}} ^ {\sqrt {3}} \frac {| z |}{2 \delta} d z \right] = \frac {1}{4 \sqrt {3} \delta} z ^ {2} \Bigg | _ {\frac {| u |}{\delta}} ^ {\sqrt {3}} \\ = \left\{ \begin{array}{l l} \frac {1}{4 \sqrt {3} \delta} \left(3 - \frac {u ^ {2}}{\delta^ {2}}\right) & \text {i f} \frac {| u |}{\delta} <   \sqrt {3}, \\ 0 & \text {o t h e r w i s e .} \end{array} \right. \tag {12} \\ \end{array}
$$
A.1.3 From Laplacian Distribution
The PDF of Laplace distribution is given by:
$$
f (z; \mu , b) = \frac {1}{2 b} \exp \left(- \frac {| z - \mu |}{b}\right)
$$
with mean  $\mu$  and variance  $2b^{2}$ . Setting,  $b = \frac{1}{\sqrt{2}}$  and  $\mu = 0$  and using (5) we obtain,
$$
\begin{array}{l} \mathbb {E} _ {z \sim \lambda} [ G ^ {2} (u; z, \delta) ] = \frac {2}{\sqrt {2}} \int_ {\frac {| u |}{\delta}} ^ {\infty} \frac {| z |}{2 \delta} \exp (- \sqrt {2} | z |) d z = \frac {1}{\delta \sqrt {2}} \int_ {\frac {| u |}{\delta}} ^ {\infty} z \exp (- \sqrt {2} z) d z \\ = - \frac {1}{\delta \sqrt {2}} \left(\frac {z}{\sqrt {2}} + \frac {1}{2}\right) \exp (- \sqrt {2} z) \Bigg | _ {\frac {| u |}{\delta}} ^ {\infty} = \frac {1}{2 \delta} \left(\frac {| u |}{\delta} + \frac {1}{\sqrt {2}}\right) \exp (- \sqrt {2} \frac {| u |}{\delta}). \tag {13} \\ \end{array}
$$
A.2 Simulating a specific Surrogate
A.2.1 Sigmoid
Consider the Sigmoid surrogate function, where the Heaviside is approximated by the differentiable Sigmoid function [43]. The corresponding surrogate gradient is given by,
$$
g ^ {\prime} (u) = - \frac {k ^ {2} \exp (- k u) (1 - \exp (- k u))}{(1 + \exp (- k u)) ^ {3}}
$$
Observe that  $g(u)$  satisfies our definition of a surrogate  $(g(u))$  being even, non-decreasing on  $(-\infty, 0)$  and  $\int_{-\infty}^{\infty} g(u) du = 1 < \infty$ . Thus, according to Theorem 3, we have
$$
c = - 2 \delta^ {2} \int_ {0} ^ {\infty} \frac {g ^ {\prime} (t \delta)}{t} d t = 2 \delta^ {2} k ^ {2} \int_ {0} ^ {\infty} \frac {\exp (- k \delta t) (1 - \exp (- k \delta t))}{t (1 + \exp (- k \delta t)) ^ {3}} d t = \frac {\delta^ {2} k ^ {2}}{a ^ {2}},
$$
where,  $a \coloneqq \sqrt{\frac{1}{0.4262}}$ . The corresponding PDF is given by
$$
\lambda (z) = - \frac {\delta^ {2}}{c} \frac {g ^ {\prime} (\delta t)}{z} = a ^ {2} \frac {\exp (- k \delta z) (1 - \exp (- k \delta z))}{z (1 + \exp (- k \delta z)) ^ {3}} \tag {14}
$$
Note that the temperature parameter  $k$  comes from the surrogate to be simulated, while  $\delta$  is used by LOCALZO. We compute the expected back-propagation threshold of SPARSEGRAD for  $m = 1$  as,  $\tilde{B}_{th} = \delta \mathbb{E}_{z\sim \lambda}[|z|]$ , with,
$$
\mathbb {E} _ {z \sim \lambda} [ | z | ] = 2 a ^ {2} \int_ {0} ^ {\infty} z \frac {\exp (- a z) (1 - \exp (- a z))}{z (1 + \exp (- a z)) ^ {3}} d z = \frac {a}{2} = 0. 7 6 5 9. \tag {15}
$$
A.2.2 Fast Sigmoid
Consider also the Fast Sigmoid surrogate gradient [43, 31] that avoids computing the exponential function in Sigmoid to obtain the gradient:
$$
\frac {d x}{d u} = \frac {1}{(1 + k | u |) ^ {2}} =: g (u).
$$
We choose  $\alpha = -1$  (note that  $\alpha = 1$  does not work in this case) and apply theorem 3 so that,
$$
\begin{array}{l} c = - 2 \delta^ {2} \int_ {0} ^ {\infty} \frac {1}{z ^ {\alpha}} g ^ {\prime} (z \delta) d z = 4 \delta^ {2} \int_ {0} ^ {\infty} \frac {1}{z ^ {\alpha}} \frac {k \operatorname {s i g n} (z \delta)}{(1 + k | z \delta |) ^ {3}} d z \\ = 4 \delta^ {2} k \int_ {0} ^ {\infty} \frac {z}{(1 + k \delta z) ^ {3}} d z = \frac {4}{k} \int_ {0} ^ {\infty} \frac {t}{(1 + t) ^ {3}} d t = \frac {2}{k}. \\ \end{array}
$$
The PDF is then given by
$$
\lambda (z) = - \frac {1}{c} \frac {\delta^ {2}}{z ^ {\alpha}} g ^ {\prime} (z \delta) = k ^ {2} \delta^ {2} \frac {z \operatorname {s i g n} (z \delta)}{(1 + k | z \delta |) ^ {3}}. \tag {16}
$$
To compute the expected back-propagation threshold, we note,
$$
\tilde {B} _ {t h} = \delta \mathbb {E} _ {z \sim \lambda} [ | z | ] = 2 \delta^ {3} k ^ {2} \int_ {0} ^ {\infty} \frac {z ^ {2} \mathrm {s i g n} (z \delta)}{(1 + k | z \delta |) ^ {3}} d z = 2 \delta^ {3} k ^ {2} \int_ {0} ^ {\infty} \frac {z ^ {2}}{(1 + k z \delta) ^ {3}} d z = \frac {2}{k} \int_ {0} ^ {\infty} \frac {x ^ {2}}{(1 + x) ^ {3}} d x
$$
The above integral does not converge. However, if we consider finite support [-a, a], we may compute,  $\frac{2}{k}\int_{0}^{a}\frac{x^2}{(1 + x)^3} dx$
A.2.3 Inverse Transform Sampling
To simulate a given surrogate in LOCALZO, one needs to sample from the corresponding distribution described by the PDF  $\lambda$ . Given a sample  $r \sim \mathrm{Unif}([0,1])$  and the inverse CDF  $\Lambda^{-1}$  of the distribution, the inverse sampling technique returns,  $\Lambda^{-1}(r)$ , as a sample from the distribution. Suppose the inverse CDF is not computable analytically from the PDF (or not implementable practically). In that case, we may choose a finite support over which the PDF is evaluated at a sufficiently dense set of points and compute the discretized CDF using the Riemann sum. The inverse discretized CDF is then computed empirically and stored as a list for a finite number of points (spaced regularly) between [0, 1]. Sampling from the uniform distribution then amounts to randomly choosing the indices of the list and picking the corresponding inverse CDF values.
A.3 Expected Back-propagation Thresholds
In what follows,  $m$  is the number of samples used in (6), while  $k$  is the index of a particular sample. To compute the expected back-propagation threshold, we observe that a neuron is inactive in LOCALZO back-propagation if,