SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Bhaskar Mukhoty $^{1,*}$  Velibor Bojkovic $^{1,*}$  William de Vazelhes $^{1}$  Xiaohan Zhao $^{3}$
Giulia De Masi $^{2,6}$
Huan Xiong1,5, $\dagger$
Bin  $\mathbf{G}\mathbf{u}^{1,4,\dagger}$
<sup>1</sup> Mohamed bin Zayed University of Artificial Intelligence, UAE
$^{2}$  ARRC, Technology Innovation Institute, UAE
$^{2}$  Nanjing University of Information Science and Technology, China
$^{4}$  School of Artificial Intelligence, Jilin University, China
<sup>5</sup> Harbin Institute of Technology, China
$^{6}$  BioRobotics Institute, Sant'Anna School of Advanced Studies, Pisa, Italy
Spiking neural networks are becoming increasingly popular for their low energy requirement in real-world tasks with accuracy comparable to traditional ANNs. SNN training algorithms face the loss of gradient information and non-differentiability due to the Heaviside function in minimizing the model loss over model parameters. To circumvent this problem, the surrogate method employs a differentiable approximation of the Heaviside function in the backward pass, while the forward pass continues to use the Heaviside as the spiking function. We propose to use the zeroth-order technique at the local or neuron level in training SNNs, motivated by its regularizing and potential energy-efficient effects and establish a theoretical connection between it and the existing surrogate methods. We perform experimental validation of the technique on standard static datasets (CIFAR-10, CIFAR-100, ImageNet-100) and neuromorphic datasets (DVS-CIFAR-10, DVS-Gesture, N-Caltech-101, NCARS) and obtain results that offer improvement over the state-of-the-art results. The proposed method also lends itself to efficient implementations of the backpropagation method, which could provide 3-4 times overall speedup in training time. The code is available at https://github.com/BhaskarMukhoty/LocalZ0.
Biological neural networks are known to be significantly more energy efficient than their artificial avatars - the artificial neural networks (ANN). Unlike ANNs, biological neurons use spike trains to communicate and process information asynchronously. [27] To closely emulate biological neurons, spiking neural networks (SNN) use binary activation to send information to the neighbouring neurons when the membrane potential exceeds membrane threshold. The event-driven binary activation simplifies the accumulation of input potential and reduces the computation burden when the spikes are sparse. Specialized neuromorphic hardware [11] is designed to carry out such event-driven and sparse computations in an energy-efficient way [32, 20].
*First co-author
$^{\dagger}$  Correspondence to Huan Xiong (huan.xiong.math@gmail.com), and Bin Gu (jsgubin@gmail.com).
There are broadly three categories of training SNNs: ANN-to-SNN conversion, unsupervised and supervised. The first one is based on the principle that parameters for SNN can be inferred from the corresponding ANN architecture [6, 15, 5]. Although training SNNs through this method achieves performance comparable to ANNs, it suffers from the long latency needed in SNNs to emulate the corresponding ANN or from retraining of ANNs required to achieve near lossless conversion [10]. The unsupervised training is biologically inspired and uses local learning to adjust the SNN parameters [14]. Although it is the most energy-efficient one among the three as it is implementable on neuromorphic chips [11], it still lags in its performance compared to ANN-to-SNN conversion and supervised training.
Finally, supervised training is a method of direct training of SNNs by using back-propagation (through time). As such, it faces two main challenges. The first is due to the nature of SNNs, or more precisely, due to the Heaviside activation of neurons (applied to the difference between the membrane potential and threshold). As the derivative of the Heaviside function is zero, except at zero where it is not defined, back-propagation does not convey any information for the SNN to learn [16]. One of the most popular ways to circumvent this drawback is to use surrogate methods, where a derivative of a surrogate function is used in the backward pass during training. Due to their simplicity, surrogate methods have been widely used and have seen tremendous success in various supervised learning tasks [36, 28]. However, large and complex network architectures, the time-recursive nature of SNNs, and the fact that the training is oblivious of the sparsity of spikes in SNNs make surrogate methods quite time and energy-consuming.
Regarding regularization or energy efficiency during direct training of SNNs, only a few methods have been proposed addressing these topics together or separately, most of which deal with the forward propagation in SNNs. For example, [1] uses stochastic neurons to increase energy efficiency during inference. More recently, [42] uses regularization during the training to increase the sparsity of spikes, reducing the computational burden and energy consumption. Further, [7] performs the forward pass on a neuromorphic chip, while the backward pass is performed on a standard GPU. Although these methods improve the SNN models' performance, they do not significantly reduce the computational burden or provide the potential to do so. On the other hand, [31] introduces a threshold for surrogate gradients (or suggests using only a surrogate with bounded support). However, introducing gradient thresholds has the drawback of limiting the full potential of surrogates during training.
This paper proposes a direct training method for SNNs based on the zeroth order technique. We apply it locally, at the neuronal level - hence dubbed Local Zeroth Order (LOCALZO) - with twofold benefits: regularization, which comes as a side-effect of the introduced randomness that is naturally associated with this technique, as well as a threshold for gradient backpropagation in the style of [31] which translates to potential energy-efficient training when properly implemented.
We summarize the main contributions of the paper as follows:
- We introduce zeroth order techniques in SNN training at a local level. We provide extensive theoretical properties of the method, relating it to the surrogate gradients via the internal distributions used in LOCALZO.
- We experimentally demonstrate the main properties of LOCALZO: its superior performance compared to baselines when it comes to generalizations, its ability to simulate arbitrary surrogates as well as its property to speed up the training process, which translates to energy-efficient training.
2.1 Spiking neuron dynamics
An SNN consists of Leaky Integrate and Fire neurons (LIF) governed by differential equations in continuous time [19]. They are generally approximated by discrete dynamics given in the form of
recurrence equations,
$$
u _ {i} ^ {(l)} [ t ] = \beta u _ {i} ^ {(l)} [ t - 1 ] + \sum_ {j} w _ {i j} x _ {j} ^ {(l - 1)} [ t ] - x _ {i} ^ {(l)} [ t - 1 ] u _ {t h},
$$
$$
x _ {i} ^ {(l)} [ t ] = h \left(u _ {i} ^ {(l)} [ t ] - u _ {t h}\right) = \left\{ \begin{array}{l l} 1 & \text {i f} u _ {i} ^ {(l)} [ t ] > u _ {t h} \\ 0 & \text {o t h e r w i s e ,} \end{array} \right. \tag {1}
$$
where  $u_{i}^{(l)}[t]$  denote the membrane potential of  $i$ -th neuron in the layer  $l$  at time-step (discrete)  $t$ , which recurrently depends upon its previous potential (with scaling factor  $\beta < 1$ ) and spikes  $x_{j}^{(l-1)}[t]$  received from the neurons of previous layers weighted by  $w_{ij}$ . The neuron generates binary spike  $x_{i}^{(l)}[t]$  whenever the membrane potential exceeds threshold  $u_{th}$ , represented by the Heaviside function  $h$ , followed by a reset effect on the membrane potential.
To implement the back-propagation of training loss through the network, one must obtain a derivative of the spike function, which poses a significant challenge in its original form represented as:
$$
\frac {d x _ {i} [ t ]}{d u} = \left\{ \begin{array}{l l} \infty & \text {i f} u _ {i} ^ {(l)} [ t ] = u _ {t h} \\ 0 & \text {o t h e r w i s e .} \end{array} \right. \tag {2}
$$
where we denote  $u \coloneqq u_i^{(l)}[t] - u_{th}$ . To avoid the entire gradient becoming zero, known as the dead neuron problem, the surrogate gradient method (referred to as SURROGATE) redefines the derivative using a surrogate:
$$
\frac {d x _ {i} [ t ]}{d u} := g (u) \tag {3}
$$
Here, the function  $g(u)$  can be, for example, the derivative of the Sigmoid function (see section 4.4), but in general, one takes a scaled probability density function as a surrogate (see Section 4.2 for more details).
Classically, the purpose of dropout in ANNs is to prevent a complex and powerful network from over-fitting the training data, which consequently implies better generalization properties [3]. In the forward pass, one usually assigns to each neuron in a targeted layer a probability of being "switched-off" during both forward and backward passes, and this probability does not change during the training. Moreover, the "activity" of the neuron, however we may define it, does not affect whether the neuron will be switched on or off.
Our motivation comes along these lines: how to introduce a dropout-like regularizing effect in the training of SNNs, but keeping in mind the temporal dimension of the data, as well as the neuron activity at that particular moment (heuristically, a more active neuron would be kept "on" with a high probability (randomness of the dropout) while a less active one would be "switched off", again with high probability, in a sense to be made precise shortly). Generally speaking, our idea consists of the following two steps: 1) For each spiking neuron of our SNN network, measure how active the neuron is in the forward pass at each time step  $t$ . Here, we define the activity based on how far the current membrane potential of the neuron  $u[t]$  is from the firing threshold  $u_{th}$  (this idea comes from [31]). However, unlike in [31] where the sole distance is the determining factor, we introduce the effect of randomness via a fixed PDF, say  $\lambda$ , sample  $z$  from it and say the neuron is active at time  $t$  if  $|u[t] - u_{th}| < c|z|$ , where  $c$  is some upfront fixed constant. 2) In the backward pass at the time  $t$ , if the neuron is dubbed active, we will apply some surrogate function  $g(u[t] - u_{th})$ ; otherwise, we will take the surrogate to be 0 (hence, switching off the propagation of gradients through the neuron in the latter case).
Having said this, we ask ourselves the final question: can we have a systematic way of choosing functions  $\lambda$  and  $g$  so that the expected surrogate we use (with respect to  $\lambda$ ) equals the one we chose upfront? A simple yet elegant solution that satisfies all of the above comes with zeroth order methods.
Zeroth order technique is a popular gradient-free method [26], well studied in neural networks literature. To briefly introduce it, we consider a function  $f: \mathbb{R}^d \to \mathbb{R}$ , that we intend to minimize
using gradient descent, for which the gradient may not be available or even undefined. The zeroth-order method estimates the gradients using function outputs: given a scalar  $\delta > 0$ , the 2-point ZO is defined as
$$
G ^ {2} (\mathbf {w}; \mathbf {z}, \delta) = \phi (d) \frac {f (\mathbf {w} + \delta \mathbf {z}) - f (\mathbf {w} - \delta \mathbf {z})}{2 \delta} \mathbf {z} \tag {4}
$$
where,  $\mathbf{z} \sim \lambda$  is a random direction with  $\mathbb{E}_{z \sim \lambda}[\| \mathbf{z} \|^2] = 1$  and  $\phi(d)$  is a dimension-dependent factor, with  $d$  being the dimension. However, to approximate the full gradient of  $f$  up to a constant squared error, we need an average of  $O(d)$  samples of  $G^2$ , which becomes computationally challenging when  $d$  is large, such as the number of learnable parameters of the neural network. Though well studied in the literature, properties of 2-point ZO are known only for the continuous functions [29, 4]. In the present context, we will apply it locally to the Heaviside function that produces the outputs of spiking neurons, and we justify this by providing the necessary theoretical background for doing so.
3 The LOCALZO algorithm
Applying ZO on a global scale is challenging due to the large dimensionality of neural networks[25]. Since the non-differentiability of SNN is introduced by the Heaviside function at the neuronal level, we apply the 2-point ZO method on  $h: \mathbb{R} \to \{0,1\}$  itself,
$$
G ^ {2} (u; z, \delta) = \frac {h (u + z \delta) - h (u - z \delta)}{2 \delta} z = \left\{ \begin{array}{l l} 0, & | u | > | z | \delta \\ \frac {| z |}{2 \delta}, & | u | <   | z | \delta \end{array} \right. \tag {5}
$$
where  $u = u_{i}^{(l)}[t] - u_{th}$  and  $z$  is sampled from some distribution  $\lambda$ . We may average the 2-point ZO gradient over a few samples  $z_{k}$  so that the LOCALZO derivative of the spike function is defined as:
$$
\frac {d x _ {i} [ t ]}{d t} := \frac {1}{m} \sum_ {k = 1} ^ {m} G ^ {2} (u; z _ {k}, \delta) \tag {6}
$$
where, the number of samples,  $m$ , is a hyper-parameter to the LOCALZO method. We implement this at the neuronal level of the back-propagation routine, where the forward pass uses the Heaviside function, and the backward pass uses equation (6). Note that the gradient  $\frac{dx_i[t]}{dt}$  being non-zero naturally determines the active neurons of the backward pass (as was discussed in Section 2.2), which can be inferred from the forward pass through the neuron. Algorithm 1 gives an abstract representation of the process at a neuronal level, which hints that the backward call is redundant when the neuron has a zero gradient.
4 Theoretical Properties of LOCALZO
4.1 General ZO function
For the theoretical results around LOCALZO, we consider a more general function than what was suggested by eqn. 5, in the form
$$
G ^ {2} (u; z, \delta) = \left\{ \begin{array}{l} 0, | u | > | z | \delta \\ \frac {| z | ^ {\alpha}}{2 \delta}, | u | \leq | z | \delta , \end{array} \right. \tag {7}
$$
where the new constant  $\alpha$  is an integer different from 0, while  $\delta$  is a positive real number (so, for example, setting  $\alpha = 1$  in (7), we obtain (5)).
The integer  $\alpha$  is somewhat a normalizing constant, which allows obtaining different surrogates as the expectation of function  $G^{2}(u;z,\delta)$  when  $z$  is sampled from suitable distributions.
Algorithm 1 LOCALZO
Require:  $u\coloneqq u_i^{(l)}[t] - u_{th}$  ,dist.  $\lambda$  const.  $\delta ,m$  sample  $z_{1},z_{2},\dots z_{m}\sim \lambda$
grad  $\leftarrow \frac{1}{m}\sum_{k = 1}^{m}\mathbb{I}(|u| <   \delta |z_k|)\frac{|z_k|}{2\delta}$  ifgrad  $\neq 0$  then
SaveForBackward(grad) end if return  $\mathbb{I}(u > 0)$
Backward {Invoked if grad is non-zero}
Require: gradient from chain rule: grad_input return grad_input \*grad
In practice, taking  $\alpha = \pm 1$  will suffice to account for most of the surrogates found in the literature. The role of  $\delta$  is somewhat different, as it controls the "shape" of the surrogate (narrowing it and stretching around zero). The role of each constant will be more evident from what follows (see section 4.4).
4.2 Surrogate functions
Definition 4.1. We say that a function  $g: \mathbb{R} \to \mathbb{R}_{\geq 0}$  is a surrogate function (gradient surrogate) if it is even, non-decreasing on the interval  $(-\infty, 0)$  and  $c := \int_{-\infty}^{\infty} g(z) dz < \infty$ .
Note that the integral  $\int_{-\infty}^{\infty} g(z) dz$  is convergent (as  $g(z)$  is non-negative), but possibly can be  $\infty$  and the last condition means that the function  $\frac{1}{c} g(t)$  is a probability density function. The first two conditions, that is, requirements for the function to be even and non-decreasing, are not essential but rather practical and consistent with examples from SNN literature.
Note that the function  $G: \mathbb{R} \to [0,1]$ , defined as  $G(t) := \frac{1}{c}\int_{-\infty}^{t}g(z)dz$  is the corresponding cumulative distribution function (for PDF  $\frac{1}{c} g(t)$ ). Moreover, it is not difficult to see that its graph is "symmetric" around point  $(0,\frac{1}{2})$  (or in more precise terms,  $G(t) = 1 - G(-t)$ ), hence  $G(t)$  can be seen as an approximation of Heaviside function  $h(t)$ . Then, its derivative  $\frac{d}{dt} G(t) = \frac{1}{c} g(t)$  can serve as an approximation of the "derivative" of  $h(t)$ , or in other words, as its surrogate, which somewhat justifies the terminology.
Finally, one may note that "true" surrogates would correspond to those functions  $g$  for which  $c = 1$ . However, the reason we allow  $c$  to be different from 1 is again practical and simplifies the derivation of the results that follow. We note once again that allowing general  $c$  is in consistency with examples used in the literature.
4.3 Surrogates and ZO
To be in line with classic results around the ZO method and gradient approximation of functions, we pose ourselves two fundamental questions: What sort of functions in variable  $u$  can be obtained as the expectation of  $G^2(u;z,\delta)$  when  $z$  is sampled from a suitable distribution  $\lambda$ , and, given some function  $g(u)$ , can we find a distribution  $\lambda$  such that we obtain  $g(u)$  in the expectation when  $z$  is sampled from  $\lambda$ ?
Two theorems that follow answer these questions and are the core of this section. The main player in both of the questions is the expected value of  $G^2(u;z,\delta)$ , so we start by analyzing it more precisely. Let  $\lambda$  be a distribution,  $\lambda(t)$  its PDF for which we assume that it is even and that  $\int_0^\infty z^\alpha \lambda(z) dz < \infty$ . Then, we may write
$$
\mathbb {E} _ {z \sim \lambda} [ G ^ {2} (u; z, \delta) ] = \int_ {- \infty} ^ {\infty} G ^ {2} (u; z, \delta) \lambda (z) d z = \int_ {| u | \leq | z | \delta} \frac {| z | ^ {\alpha}}{2 \delta} \lambda (z) d z = \frac {1}{\delta} \int_ {\frac {| u |}{\delta}} ^ {\infty} z ^ {\alpha} \lambda (z) d z. \tag {8}
$$
It becomes apparent from eqn. (8) that  $\mathbb{E}_{z\sim \lambda}[G^2 (u;z,\delta)]$  has some properties of surrogate functions (it is even and non-decreasing on  $\mathbb{R}_{< 0}$ ). The proofs of the following results are detailed in the appendix A.
Lemma 1. Assume further that  $\int_0^\infty z^{\alpha +1}\lambda (z)dz < \infty$ . Then,  $\mathbb{E}_{z\sim \lambda}[G^2 (u;z,\delta)]$  is a surrogate function.
Theorem 2. Let  $\lambda$  be a distribution and  $\lambda(t)$  its corresponding PDF. Assume that integrals  $\int_0^\infty t^\alpha \lambda(t) dt$  and  $\int_0^\infty t^{\alpha+1} \lambda(t) dt$  exist and are finite. Let further  $\tilde{\lambda}$  be the distribution with corresponding PDF function
$$
\tilde {\lambda} (z) = \frac {1}{c} \int_ {| z |} ^ {\infty} t ^ {\alpha} \lambda (t) d t,
$$
where  $c$  is the scaling constant (such that  $\int_{-\infty}^{\infty}\tilde{\lambda} (z)dz = 1$ ). Then,
$$
\mathbb {E} _ {z \sim \lambda} [ G ^ {2} (u; z, \delta) ] = \frac {d}{d u} \mathbb {E} _ {z \sim \bar {\lambda}} [ c h (u + \delta z) ].
$$
For our next result, which answers the second question we asked at the beginning of this section, note that a surrogate function is differentiable almost everywhere, which follows from the Lebesgue theorem on the differentiability of monotone functions. So, taking derivatives here is understood in an "almost everywhere" sense.
Figure 1: The figure shows the expected surrogates derived in section 4.4 as  $z$  is sampled from Normal(0,1), Unif([\sqrt{3},\sqrt{3}]) and Laplace(0, $\frac{1}{\sqrt{2}}$ ) respectively. Each figure shows the surrogates corresponding to  $\delta \rightarrow 0$ ,  $\delta = 0.5$  and 1. The surrogates are supplied to SPARSEGRAD methods for a fair comparison with LOCALZO as the latter uses respective distributions to sample  $z$ .
Theorem 3. Let  $g(u)$  be a surrogate function. Suppose further that  $c = -2\delta^2\int_0^\infty \frac{1}{z^\alpha} g'(z\delta)dz < \infty$  and put  $\lambda (z) = -\frac{\delta^2}{cz^\alpha} g'(z\delta)$  (so that  $\lambda (z)$  is a PDF). Then,
$$
c \mathbb {E} _ {z \sim \lambda} [ G ^ {2} (u; z, \delta) ] = \mathbb {E} _ {z \sim \lambda} [ c G ^ {2} (u; z, \delta) ] = g (u).
$$
4.4 Application of Theorem 2 and 3
Next, we spell out the results of Theorem 2 applied to some standard distributions, with  $\alpha = 1$ . For clarity, all the distributions' parameters are chosen so that the scaling constant of the resulting surrogate is 1. One may consult Figure 1 for the visual representation of the results, while the details are provided in the appendix A.1. Recall that the standard normal distribution  $N(0,1)$  has PDF of the form  $\frac{1}{\sqrt{2\pi}}\exp \left(-\frac{z^2}{2}\right)$ . Consequently, it is straightforward to obtain
$$
\mathbb {E} _ {z \sim \lambda} [ G ^ {2} (u; z, \delta) ] = \frac {1}{\sqrt {2 \pi}} \int_ {- \infty} ^ {\infty} \frac {| z |}{2 \delta} \exp \left(- \frac {z ^ {2}}{2}\right) d z = \frac {1}{\delta \sqrt {2 \pi}} \exp \left(- \frac {u ^ {2}}{2 \delta^ {2}}\right). \tag {9}
$$
In appendix, A.1, we further derive surrogates when  $z$  is sampled from Uniform and Laplace distribution.
We recall that Theorem 3 provides a way to derive distributions for arbitrary surrogate functions (that satisfy the conditions of the theorem). Consider the Sigmoid surrogate function, where the differentiable Sigmoid function approximates the Heaviside [43]. The corresponding surrogate gradient is given by,
$$
\frac {d x}{d u} = \frac {d}{d u} \frac {1}{1 + \exp (- k u)} = \frac {k \exp (- k u)}{(1 + \exp (- k u)) ^ {2}} =: g (u)
$$
Observe that  $g(u)$  satisfies our definition of a surrogate  $(g(u)$  being even, non-decreasing on  $(-\infty, 0)$  and  $\int_{-\infty}^{\infty} g(u) du = 1 < \infty$ ). Thus, according to Theorem 3, we have  $c = -2\delta^2 \int_0^\infty \frac{g'(t\delta)}{t} dt = \frac{\delta^2 k^2}{a^2}$  where,  $a := \sqrt{\frac{1}{0.4262}}$ . The corresponding PDF is given by
$$
\lambda (z) = - \frac {\delta^ {2}}{c} \frac {g ^ {\prime} (\delta t)}{z} = a ^ {2} \frac {\exp (- k \delta z) (1 - \exp (- k \delta z))}{z (1 + \exp (- k \delta z)) ^ {3}} \tag {10}
$$
Observe that the temperature parameter  $k$  comes from the surrogate to be simulated, while  $\delta$  is used by LOCALZO. Appendix A.2 provides calculations and distribution corresponding to the popular Fast-sigmoid surrogate, followed by a description of the inverse sampling method that can be used to simulate sampling for arbitrary distributions using the uniform distribution.
4.5 Expected back-propagation threshold for LOCALZO
To study the energy efficiency of the LOCALZO method when training SNNs, we compute the expected threshold  $\tilde{B}_{th}$  for the activity of the neurons, i.e. the expectation of the quantity  $|z|\delta$  when  $z$  is sampled from a distribution  $\lambda$ . It is used in the experimental section when comparing our method with the alternative energy-efficient method [31]. The expected threshold values are presented in Table 1 ( $m$  denotes the number of samples used in (6)), while the details of the derivations can be found in A.3.
Figure 2: We plot training loss, overall speedup, and percentage of active neurons for the Sigmoid surrogate, as reported in Table 3. The LOCALZO algorithm converges faster than the SPARSEGRAD method while having a similar overall speedup. The percentage of active neurons being less than  $0.6\%$  explains the reduced computational requirement, which translates to backward speedup.
5.1 General Performance of LOCALZO
First, we evaluate the generalization performance of LOCALZO as a substitute for the surrogate method on standard static image datasets such as CIFAR-10, CIFAR-100[22], ImageNet-100[12] and neuromorphic datasets such as DVS-CIFAR-10[24], DVS-Gesture[2], N-Caltech-101[30], N-CARS[37]. More specifically, by substituting surrogate gradients with gradients computed with our method, we com
Table 1: The expected back-propagation thresholds
z ~ λ	Bth/δ
λ	F|zk| (x)	m = 1	m = 5
Normal(0,1)	erf(x/√2)	0.798	1.569
Unif(√3,√3)	x/√3	0.866	1.443
Laplace(0,1/√2)	1 - e-√2x	0.707	1.615