Direct Training of SNN using Local Zeroth Order Method
Authors: Bhaskar Mukhoty
Velibor Bojkovic
William de Vazelhes
Xiaohan Zhao
Giulia De Masi
Huan Xiong
Bin Gu
Venue: NeurIPS 2023 poster (2023)
Keywords: Spiking Neural Network
Zeroth Order
Surrogate Gradient

Abstract: Spiking neural networks are becoming increasingly popular for their low energy requirement in real-world tasks with accuracy comparable to traditional ANNs. SNN training algorithms face the loss of gradient information and non-differentiability due to the Heaviside function in minimizing the model loss over model parameters. To circumvent this problem, the surrogate method employs a differentiable approximation of the Heaviside function in the backward pass, while the forward pass continues to use the Heaviside as the spiking function. We propose to use the zeroth-order technique at the local or neuron level in training SNNs, motivated by its regularizing and potential energy-efficient effects and establish a theoretical connection between it and the existing surrogate methods. We perform experimental validation of the technique on standard static datasets (CIFAR-10, CIFAR-100, ImageNet-100) and neuromorphic datasets (DVS-CIFAR-10, DVS-Gesture, N-Caltech-101, NCARS) and obtain results that offer improvement over the state-of-the-art results. The proposed method also lends itself to efficient implementations of the back-propagation method, which could provide 3-4 times overall speedup in training time. The code is available at \url{https://github.com/BhaskarMukhoty/LocalZO}.

Direct Training of SNN using Local Zeroth Order Method
Bhaskar Mukhoty'
Velibor Bojkovićl William de Vazelhes'
Xiaohan Zhao 3
Giulia De Masi2.6
Huan Xiong
Bin Gu
Mohamed bin Zayed University of Artificial Intelligence, UAE
2 ARRC, Technology Innovation Institute, UAE
Nanjing University Information Science and Technology, China School of Artificial Intelligence, Jilin University, China Harbin Institute of Technology, China
BioRobotics Institute, Sant' Anna School of Advanced Studies, Pisa, Italy
Abstract
Spiking neural networks are becoming increasingly popular for their low energy quirement in real-world tasks with accuracy comparable to traditional ANNs. SNN training algorithms face the loss of gradient information and non-differentiability due Heaviside function in minimizing the model loss over model parameters. To circumvent this problem, the surrogate method employs differentiable approx- imation of the Hcaviside function in the backward pass, while the forward pass continues to use the Heaviside as the spiking function. We propose to use the zeroth- order technique at the local or neuron level in training SNNs, motivated by its regu- effects and establish theoretical connection
larizing and potential energy-efficce between it and the existing surrogate methods. We perform experimental validation of the technique on standard static datasets (CIFAR-10, CIFAR-I00 ImageNet- 100) and neuromorphic datasets (DVS- DOYS-CIFAR-10 10, DVS-Gesture N-Caltech- NCARS) and obtain results that offer improvement The proposed method also lends itself to efficient implementations of the back- propagation method, which could provide 3-4 times overall speedup training time.
the state-of-the-art results.
The code is available at httphhtt:///ee h/gptub.//itub....... thub. www///aaakkahhhcctyoooaaaaaoaoa
Introduction
Biological neural networks are known be significantly more energy efficient than their artificial avatars the artificial neural networks (ANN). Unlike ANNs, biological neurons use spike trains to communicate and process information asynchronously. (27) To closely emulate biological neurons, spiking neural networks (SNN) use binary activation to send information to the neighbouring neurons when the membrane potential exceeds membrane threshold. The event-driven binary activation simplifies the accumulation of input potential and reduces the computation burden when the spikes are sparse. Specialized neuromorphic hardware [1]] is designed to carry out such event- dent-driven and
sparse computations in energy- effegyeeficceet way 32 20
First co-author
Corresponneecce to Huan Xiong waan..ioon..matt math @ gom)) and Bin (jsgubin @ gmail.comm)
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

There are broadly three categories of training SNNs: ANN-to-SNN conversion, unsupervised and supervised. The first one is based on the principle that parameters for SNN can be inferred from the corresponding ANN architecture 66 15 S
Although training SNNs through this method achieves performance comparable to ANNs, suffers from the long latency needed in SNNs to emulate the corresponding ANN or from retraining of ANNs required to achieve near lossless conversion [0]. The unsupervised training is biologically inspired and uses local learning to adjust the SNN parameters [14]. Although it is the most energy-efficient one among the three as it is implementable on neuromorphic chips [T] still lags in its performance compared to ANN-tO-SNE conversion and supervised training.
Finally, supervised training a method of direct training of SNNs by using back- proppppoagttiion (through time). As such, it faces two main challenges. The first is due to the nature of SNNs, or more precisely, due to the Heaviside activation of neurons (applied to the difference between the membrane potential and threshold). As the derivative of the Heaviside function is zero, except at zero where it is not defined, back-propagation does not convey any information for the SNN to learn [16]. One of the most popular ways to circumvent this drawback is to use surrogate methods, where derivative of surrogate function is used the backward pass during training. Due to their simplicity, surrogate methods have been widely used and have seen tremendous success in various supervised learning tasks [36, 28). However, large and complex network architectures, the recursvve nature of SNNs, and the fact that the training is oblivious of the sparsity of spikes in SNNs make surrogate methods quite time and mooggy-consuming..
Regarding regularization or energy efficiency during direct training of SNNs, only a few methods have been proposed addressing these topics together or separately, most of which deal with the forward propagation in SNNs. For example, [1] uses stochastic neurons to increase energy efficiency during inference. More recently, 42] uses regularization during the training to increase the sparsity of spikes, reducing the computational burden and energy consumption. Further, 7 performs the forward pass on a neuromorphic chip, while the backward pass is performed on a standard GPU, Although these methods improve the SNN models" performance, they do not significantly reduce the computational burden or provide the potential to do so. On the other hand, [31] introduces a threshold for surrogate gradients (or suggests using only surrogate with bounded support). However, introducing gradient thresholds has the drawback of limiting the full potential of surrogates during training.
This paper proposes a direct training method for SNNs based on the zeroth order technique. We apply it locally, at the neuronal level hence dubbed Local Zeroth Order (LOCALZO) with twofold benefits: regularization, which comes as a side-effect of the introduced randomness that is naturally associated with this technique, as well as threshold for gradient backpropagation in the style of [3] which translates to potential energy-efficient training when properly implemented.
We summarize the main contributions of the paper as follows:
We introduce zeroth order techniques in SNN training at a local level. We provide extensive theoretical properties of the method, relating it to the surrogate gradients via the internal distributions used in LOCALZO.
We experimentally demonstrate the main properties of LoCALZO: its superior performance compared to baselines when comes to generalizations, its ability to simulate arbitrary surrogates as well as its property to speed up the training process, which translates to energy-efficient training.
2 Background
2.1 Spiking neuron dynamics
SNN consists of Leaky Integrate and Fire neurons (LIF) governed by differential equations in continuous time [19]. They are generally approximated by discrete dynamics given in the form of

---

recurrence equations,
10 t|] = Bu," It- - 1| 1|+ - wizz 11 [t]]
( 20 ( - 1 Nuss
1 [1] h(u, [ uen) [I if [t] > Ueh
otherwise,
where ii [t] denote the membrane potential of i-th neuron in the layer at time-step (discrete) t, which recurrently depends upon its previous potential (with scaling factor 3 < 1) and spikes z = [t] received from the neurons of previous layers weighted by wiy- The neuron generates binary spike
s; [t] whenever the membrane potential exceeds threshold 4yy, represented by the Heaviside function h, followed by a reset effect on the membrane potential.
To implement the propaprroagttion of training loss through the network, one must obtain derivative of the spike function, which poses a significant challenge in its original form represented as:
dr du
uf [ uch
00
otherwise.
where we denote u u= un neuron problem, the surrogate gradient method (referred to as SURROGATE) redefines the derivative using a surrogate:
[t] Weh- To avoid the entire gradient becoming zero, known as the dead
dx; [4]
g(u)
du
Here, the function g(u) can be, for example, the derivative of the Sigmoid function (see section 4.4 but in general, one takes scaled probability density function as surrogate (see Section 4.2 for more details).
2.2 Motivation
Classically, the purpose of dropout in ANNs is prevent complex and powerful network from over-fitting the training data, which consequently implies better generalization properties [3. In the forward pass, one usually assigns each neuron in targeted layer probability of being "switched-off" during both forward and backward passes, and this probability does not change during the training. Moreover, the "activity" of the neuron, however we may define it, does not affect whether the neuron will be switched on or off.
Our motivation comes along these lines: how to introduce dropout-like regularizing effeect in the training of SNNs, but keeping in mind the temporal dimension of the data, as well as the neuron activity at that particular moment (heuristicallyy a more active neuron would be kept "on" with high probability (randomness of the dropout) while a less active one would be "switched off", again with high probability, in sense be made precise shortly). Generally speaking, our idea consists of the following two steps: 1) For each spiking neuron of our SNN network, measure how active the neuron is in the forward pass at each time step t. Here, define the activity based on how far the current membrane potential of the neuron u[[] is from the firing threshold Meh. (this idea comes from (1.. However, unlike in (31 where the sole distance is the determining factor, introduce the effect of randomness via a fixed PDF, say X, sample = from and say the neuron is active at time /. |u[t] MrA| c|2/, where is some upfront fixed constant. 2) In the backward pass the time the neuron is dubbed active, we will apply some surrogate function g - uea); otherwise, we will take the surrogate to be 0 (hence, switching off the propagation of gradients through the neuron in the latter case).
Having said this, ask ourselves the final question: functions À and g so that the expected surrogate we use (with respect to A) equals the one we chose upfront? A simple yet elegant solution that satisfies all of the above comes with zeroth order methods.
have systematic way of choosing
Zeroth order technique
well studied in neural networks literature. To briefly introduce it, consider a function f R R. that we intend to minimize
popular gradient-free method 26
3

---

using gradient descent, for which the gradient may not be available or even undefined. The zeroth- order method estimates the gradients using function outputs: given scalar 8 > 0, the 2-point ZO is defined as
G2 (w: f(W + oz) f(w Öz)
(4)
28
where, 2 a random direction with Ez- |||z| Ez~>|||>| and ((d) is a dimension dependent factor, with d being the dimension. However, to approximate the full gradient of I uu to constant squared error, we need an average of (d)) samples of G², which becomes computationally challenging when d is large, such the number of learnable parameters of the neural network. Though well studied in the literature, properties of 2-point ZO are known only for the continuous functions [29] 4]. In the present context, we will apply it locally to the Heaviside function that produces the outputs of spiking neurons, and we justify this by providing the necessary theoretical background for doing so.
The LOCALZO algorithm
Applying ZO on a global scale is challenging due to the large dimensionality of neural networks 25 Since the non-differentiabiliyy of SNN introduced by the Heaviside function at the neuronal level, we apply the 2-point 70 method on h IR + {0, 1} itself,
G² (1; h(u + z6) 1uu zō)
0,
|u]
>
0 1
(5)
26
|u|
25
where u == u,, [t] - Ven and is sampled from some distribution A. We may average the 2-point ZO gradient over a few samples 2か that the LOCALZZ derivative of the spike function is defined as:
dx. [1]
Σ
dt
where, the number of samples, m, is a hyper hwper--parannn parameter to the LOCALZO method. We implement this at the neuronal level of the pack-propagation routine, where the forward pass uses the Heaviside function, and the backward pass uses equation 66. Note that the gradient dz,lt] being non-zero dt
naturally determines the active neurons of the backward pass (as was discussed in Section 2.2). which can inferred from the forward pass through the neuron. Algorithm [ gives an abstract representation of the process at neuronal level, which hints that the backward call is redundant when the neuron has a zero gradient.
Theoretical Properties of LOCALZO
4.1 General ZO function
For the theoretical results around LOCALZO, we consider more general function than what was suggested by eqn. 5 in the form
Algorithm 1 LOCALZO
Forward
Require: u: u,
(0) [t] Meh. dist. X, const. 8,
0,
[ul > |=|0
sample 21, B2, ... 2m
(u,,,,, 2)
grad + 1 -En,
-k=1 [[[x[] I 6)eul) lzal
[u |=|6,
28
grad + 0 then
where the new constant o is an integer different from while à a positive real number (so, for example, setting
end if
in 7 obtain (5)).
return I(u >0 0)
Backward {Invoked if grad is non-zero) Require: gradient from chain rule: gnadu return grad-input * grad
The integer somewhat normalizing con- stant, which allows obtaining different surro- gates as the expectation of function G² (²(u; 2, when 2 is sampled from suitable distributions. In practice, taking a = +1 will suffice to account for most of the surrogates found the literature. The role of 5 is somewhat different, as it controls the "shape" of the surrogate (narrowing it and stretching around zero). The role of each constant will more evident from what follows (see
section 4.4

---

4.2 Surrogate functions
Definition 4.1. We say that function g : R2z surrogate function (gradient surrogate) if even, non-decreasing on the interval (-oo, 0) and e == -=oo g())d2 < oc.
Note that the integral S_ g(2)dz is convergent (as g(z) is non-negative), but possibly can be co and the last condition means that the function (() is probability density function. The first two conditions, that is, requirements for the function to even and non-decreasing, are not essential but rather practical and consistent with examples from SNN literature.
Note that the function G RR + [0, IJ. defined G(() := 5 s(=)dz is the corresponding cumulative distribution function (for PDF (t)). Moreover, it is not difficult see that its graph is "symmetric" around point (0, ()) (or in more precise terms, G(t)) = - G(-t)), hence G(t) can be seen as an approximation of Heaviside function h(t). Then, its derivative d
de G(t) g(t) can serve as an approximation of the "derivative" of h(t), or in other words, as its surrogate, which somewhat justifies the terminology.
Finally, one may that "true" surrogates would correspond to those functions g for which 1. However, the reason we allow e to be different from is again practical and simplifies the derivation of the results that follow. We note once again that allowing general is in consistency with examples used in the literature.
4.3 Surrogates and ZO
To be in line with classic results around the ZO method and gradient approximation of functions, we pose ourselves two fundamental questions: What sort of functions in variable can be obtained as the expectation of (2; z, 5) when is sampled from a suitable distribution X, and, given some function g(u), can we find a distribution such that we obtain g(u) in the expectation when is sampled from X?
Two theorems that follow answer these questions and are the core of this section. The main player in both of the questions is the expected value of (² 2, 8), so we start by analyzing it more precisely. Let be distribution, A(t) its PDF for which we assume that is even and that fo =a X(z)dz < co. Then, may write
/ C (1)), =
1 28
1
E (u;2,,,
M(=)dz
(8)
8 that 1 (G2((u (u: 2, 5)] has some properties of surrogate functions (it even and non-decreasing on R2o)). The proofs of the following results are detailed in the appendix A
It becomes apparent from eqn.
Lemma Assume further that Jo 1 (2)d2z < co. Then, E.n I (w; 2, 5)] is a surrogate function.
Theorem 2. Let X be distribution and X(1) its corresponding PDF. Assume that integrals Jo X(t)dt and Jo X(Edd exist and are funite. Let further X be the distribution with cor- responding PDF function
A(=) - - / (')(()dt,
[&|
where is the scaling constant (such that 1_ X(2)dz 1). Then,
BG (w; 2, = du E HEnxilch(u + 52)].
For our next result, which answers the second question we asked at beginning of this section, note that a surrogate function is differentiable almost everywhere, which follows from the Lebesgue theorem on the differentiabillty of monotone functions. So, taking derivatives here is understood in an "almost everywhere" sense.
5

---

2-NN0.13
0.2
Vib
12 mm Vie
Vie
Figure The figure shows the expected surrogates derived section 4.4 as z is sampled from Normal 1), Unif([v3, v3) and Laplace(O, vz) respectively. Each figure shows the surrogates corresponding to + 0, ā = 0.5 and 1. The surrogates are supplied to SPARSEGRAD methods for fair comparison with LOCALZZO the latter respective distributions to sample 2.
Theorem 3. Let gu)) be surrogate function. Suppose further that -22 So o (=(2=dd < oo and pul X(=) = az (z6) (so that A(z) is a PDF). Then,
PG2 (w; 2, 6)] E2- (u; =g(u).
4.4 Application of Theorem 2 and
Next, spell out the results of Theorem 2 applied to some standard distributions, with 1. For clarity, all the distributions' parameters are chosen so that the scaling constant of the resulting surrogate is 1. One may consult F for the visual representation of the results, while the details are provided in the appendix A.1 Recall that the standard normal distribution N(0,,) 1) has PDF of the form
V2x exp(
). Consequently, it is straightforward to obtain
1
|=|
=2
1
u
E (2; v 2m 25
exp
)dz
exp(
2
8V2T
25
In appendix,
A.I
we further derive surrogates when 2 is sampled from Uniform and Laplace
distribution.
We recall that Theorem provides a way to derive distributions for arbitrary surrogate functions ( that satisfy the conditions of the theorem). Consider the Sigmoid surrogate function, where the differentiable Sigmoid function approximates the Heaviside 43). The corresponding surrogate gradient is given by.
dr d
1s cxpp ku)
(1+ (x+eepp(-ee ku))= =: g(u)
du du 1 exp(-ku) Observe that g(u) satisfies our definition of surrogate (g(u) being even, non nen-decggg on oo (-00, 0) and 1 g(u)du = < oo). Thus, according to Theorem we have = 228 Jo 5 s'(t6) dt 8²t²2
where, a = V 0.1262 The corresponding PDF given by
((2) = g'(tt a exp( (kzz exp(-koz)
(10)
211 + exp(-koz))
Observe that the temperature parameter & comes from the surrogate be simulated, while d is used
by LOCALZO. Appendix A.2 provides calculations and distribution corresponding to the popular Fast-sigmoid surrogate, followed by description of the inverse sampling method that be used to
simulate sampling for arbitrary distributions using the uniform distribution.
4.5 Expected back-propagation threshold for LOCALZO
To study the energy efficiency of the LOCAL.ZO method when training SNNs, we compute the expected threshold Bt, for the activity of the neurons, i.e. the expectation of the quantity |2|6 when = is sampled from distribution X. It is used with the alternative energy-efficient method (31. The expected threshold values are presented in Table T (m denotes the number of samples used in (6)), while the details of the derivations can be found in A.3
experimental section when comparing our method

---

10
Scrved Hotee Lear, Aurerre
Sccrea Hotee Liw
5 1006
2068
3300
4093
1200
2008 33000
40.0
1004
2000
3300
4003
Figure 2: We plot training loss, overall speedup, and percentage of active neurons for the Sigmoid surrogate, as reported in Table The LOCALZO algorithm converges faster than the SPARSEGRAD method while having similar overall speedup. The percentage of active neurons being less than 0.6% explains the reduced computational requirement, which translates backward speedup.
Experiments
5.1 General Performance of LOCALZO
First, we evaluate the generalization perfor- mance of LOCALZO as substitute for the surrogate method on standard static image datasets such as CIFAR-IO, CIFAR-100(22), ImageNet-100f12 and neuromorphic datasets such as DVS-CIFAR-10224 DVS- WWS--esture(2]. N-Caltech-101 (30, N-CARS(37) More specif- ically, by substituting surrogate gradients with gradients computed with our method, we com- bined LOCALZO with contemporary state-of-the-art methods in direct training of SNNs. such as IDBN (45) and TET [33. We opted for these two techniques due to their high performance and differ- ent natures (the former
Table I: The expected back bock-propagation thresholds
2 A
Bun/8
Flay
Normal 1)
er VI
0.798
1.569
Unif[[v3, v3)
0.866
1.443
V
Laplace(0, ) 1-
0.707
1.615
batch normalization technique, while the latter introduces auxiliary loss in training). We refer to the combined methods as LocalZO+DBB and LocalZO+TET, respectively. Following the recent results, we choose ResNet-199 19 architecture for CIFAR datasets, SEW-ResNet- 34[17] for ImageNet- 100, and VGGSNN[13] architecture for the neuromorphic datasets. Table summarizes these results, where LOCALZO implemented with m = 5, 6 = 0.5, except for DVS- CIFAR-10 10 where we have used m 1. Table 7 reports the detailed training hyper-parameters for each dataset.
Results on Static Datasets: The static datasets CIFAR-10, CIFAR-100 100, and ImageNet- 100, have the number of classes mentioned in the dataset name, while each class respectively have (5000, 1000), (500, 100), and (1300, 50) train and test images. We use constant encoding to supply the images to the SNN network, with standard latencies as mentioned in Table 2 We train separate models for different latencies and report the test results respectively. The results of CIFAR datasets are reported with cutout augmentation following [T], while for ImageNet- 100, we use standard data augmentation (random resized crop, random horizontal flip), without and with the ImageNet Policy []] We note that conjunction with LOCALZO, both tDBN and TET improve their performance significantly for all the latencies. For example, in CIFAR- 10, LOCAL 700 improves TET between 0.9 1% for different latencics, while for CIFAR-100, it improves TET by 2.5 - 3.5%. The ImageNet- 100 results reported in Table 2 were further enhanced to 83.33% with m 20 and batch size 72.
Results on Neuromorphic Datasets: Events of the neuromorphic datasets are collected into event frames of dimension (2 H x W) where H and W stand for the height and width of the frame and are resized (2 48 48). The temporal events are collected into fixed number (10) of frames (a.k.a. bins), treated as the effective temporal dimension for the SNN. The experiments are reported without and with standard data augmentation. For DVS-CIFAR- 10, the TET result is re-computed to avoid an obscure frame preparation step, which is replaced by open-source routine. We obtain results superior to the state-of-the-art for DVS-CIFAR-10 and DVS- (VS-Gesture[1]] For N-Caltech-100 101 and NCARS, the improvements are 1.2% and 2.4%, respectively, compared to the state-of-the-arti [18] 35
7

---

Table 2: Comparison with the existing methods show that LOCALZO improves the accuracy of existing direct training algorithms. For the existing methods, we compare the performance with the results reported in respective literatures. For the rows with two accuracies reported, the second one for training with additional augmentation.
Dataset
Methods
Architecture
Simulation Length
Accuracy
Hybrid training Diet-SNN
ResNet-20
250
92.22
ResNet-20
10
92.54
STBP 38
CIFARNet
12
89.83
STBP NeuNorm
CIFARNet
12
90.53
TSSL-BI 4P 4M
CIFARNet
91.41
93.16
IDN 45
ResNet-19
4
92.92
92.34
CIFARIO
95.07
LOCALZO +DDBN
ResNet-19
94.89
94.65
94.50
TET
ResNet-19
94.44
94.16
95.56
LOCALZO +TET
ResNet-19
95.3
95.03
Hybrid training 34
VGG-11
125
67.87
Diet-SNN
ResNet-20
64.07
71.12
DDN
ResNet-19
70.86
69.41
73.74
CIFARI00
LOCALZO +DBBN
ResNet-19
74.13
72.78
74.72
TET 3
ResNet-19
74.47
72.87
77.25
LOCALZZO +TET
ResNet-19
76.89
76.36
ImageNet-100
EeficientL.FF-Ne(22 LOCALZO +TET
ResNet-19
79.44
SEW-Resner34
78.58, 81.56
tdBN[45
ResNet-19
10
67.8
Streaming Rollout
23
DenseNet
10
66.8
CD0
LIAF-Net
71.70
DVS-CIFARIO
LIAF
LIAF-Net
70.40
TET
VGGSNN
10
74.89", 81.45*
LOCAL.ZO +IDBN
VGGSNN
10
72.6, 79.37
LOCALZO +TET
VGGSNN
10
75.62, 81.87
AEGNN
GNN
66.8
N-Caltech-100 101
ESTISS
ResNet-34
81.7
ZOC +DBNN
VGGSNN
10
74.65, 79.05
LOCALZO +TET
VGGSNN
10
79.86, 82.99
AEGNN 35
GNN
94.5
N-CARS
EST
ResNet-34
92.5
LOCALZO +DBBN
VGGSNN
95 ,6, 95.68 65.68
LOCAL +TET
VGGSNN
10
96.78, 96.96
DVS-Gesture
SEW I7
SEW-Resnet VGGSNN
16
97.92
LOCALZO +TET
10
98 98.04, 98.43
our implementation, 'pre- treettraine with ImageNet, 83.33 with m 20

---

5.2 Performance on Energy Efficient Implementation
In the energy-efficient implementation of the back-propagation [1 the optimization of the network weights takes place in a layer-wise fashion through the unrolling of recurrence of equation (I) w.r.t time. As the active neurons of cach layer for every time step are inferred from the forward pass, gradients of only active neurons are required be saved for the backward pass, hence saving the computation requirement of the backward pass. One may refer to [3] for further details of this implementation framework. To compare, we supply SPARSEGRAD method the surrogate approximated by LOCALZO, as per section 4.4 The SPARSEGRAD algorithm also requires a back- propagation threshold parameter, Brs, control the number of active neurons participating in the back- poceppropagation. We supply the expected back propppproaattion threshold Be,, of LoCAL.ZO as obtained in sections 4.5 We follow the same experimental setting as in [31 for a fair comparison. We use a fully connected LIF neural network with two hidden layers of 200 neurons each and input and output layers. We train every model for 20 epochs and report the average training and test accuracies computed over five trials. We compute the speedup of SPARSEGRAD and LOCAL with respect to the full surrogate without truncation, that uses standard back proppapppaa..nnn... The backward speedup (Back.) captures the number of times the backward pass of a gradient update is faster, while the overall speedup (Over.) considers the total time for the forward and the backward pass and then computes the speedup. The speedup reported is averaged over all the gradient updates and the experimental trials.
We compare the performance of the algorithms on three datasets: 1) Neuromorphic-MNNS.
(NMNIST) [30) which consists of static images of handwritten digits (between 0 and 9) converted to temporal spiking data using visual neuromorphic sensors; 2) Spiking Heidel- berg Digits (SHD) [8) a neuromorphic audio dataset consisting of spoken digits (between and 9) in English and German language, totalling 20 classes. To challenge the gener- alizability of the learning task, 81% of test inputs of this dataset are new voice samples, which are not present in the training data; 3) Fashion-MNIST (FMNIST) [41] dataset is con- verted using temporal encoding convert static gray-scale images based the principle that each input neuron spikes only once, and a higher intensity spike results in an earlier spike.
Table 3 provides a comparison of the algorithms, using surrogates cor-
Table 3: Performance on NMNIST, SHD and FMNIST
responding to the Normal and Sig- moid, with 8 = 0.05 and m 1. For the normal distribution, we sup- ply SPARSEGRAD algorithm the back- propagation threshold Bid obtained in [ In the section 4.4 we de- rived distributions corresponding to the Sigmoid surrogate. We use in- verse transform sampling (see A.2.3). and take the temperature parameter
Method
Train
Test
Back.
Over.
NMNIST
Normal 1), 0.05,
SPARSEGRAD
93.26 + 0.31
91.86+ 0.29
99.57
3.38
LOCALZZ
94.38 + 0.12
93.29+ 0.08
92.27
3.34
Sigmoid, 8 0.05, l x= 30.63,
SPARSEGRAD LOCALZO
92.96 0.26
91.041 0.32
87.45
3.00
93.981 0.08
92.97-1 0.05
83.54
3.02
k = a/8 ~ 30.63 so that c at 1 and supply SPARSEGRAD method the corresponding back-propagation threshold, B., 0.7666. The Lo- CALZO method offers better test ac- curacies than SPARSEGRAD, with slight compromise in speedup due to the sampling of random variable The difference between training and test accuracies for the SHD dataset can be attributed to the unseen voice samples in the test (a
8ºk²
SHD
2 Normal 0 1), 5 = 0.05, m
SPARSEGRAD LOCALZZ
92.03 0.79
74.73 0.73
143.7
4.83
91.77 0.27
76.55 0.93
142.8
4.75
Sigmoid, = 0.05, R 30.63, m =
1
SPARSEGRAD LOCALZO
92.19 0.41
75.80 0.97
140.8
4.46
91.961 0.11
76.97. 0.40
133.6
4.36
FMNIST
: Normal (0, 1). 0.05, m
SPARSEGRAD LOCALZO
81.91 0.10
80.2811 0.11
15.74
1.97
83.83 0.07
81.794 0.06
15.49
1.88
Sigmoid, 8
0.05, Rs 30.63,
SPARSEGRAD LOCALZO
81.60 0.11
80.024: 0.08
12.12
1.65
Figure 2 shows the training loss, over- all speedup, and percentage of active
83.39 0.10
81.76 0.10
12.50
1.57

---

neurons after each gradient step for the Sigmoid surrogate. The sparseness of active neurons (under 0.6%) explains the reduced computational requirement that translates the speedup.
We further implement LOCALZZO with = 0.5, 2 ~ (0, 1) to train CNN architecture (Input- 16C5- BN--LI-MP22-00FFCC.
Locatzo
and com-
pare with the corresponding surrogate gradient algorithm. Figure 3 shows the corresponding sparsity of the methods by plotting the number of zero elements at neuronal level.
0.10
The plot suggests that during the training, LOCALZO ex- hibits higher sparsity gradients than the surrogate method.
0.55
Ablation study: Table4 further shows the test accuracy of the LOCAL method and overall speed-up for wide range of values of m with 2 Normal(O, 1). Like Table 3 the experiments are repeated five times and mean test accuracy is reported along with standard deviation (Std.), In general, by increasing m, the method approximates the surrogate better, still offers the regularizing effect and potentially improves the generalization, but also requires more computation. Larger m leads to more non-zero gradients the neuronal level the backward pass, reducing overall speed- up. On the other hand, smaller m introduces higher randomness (less "controlled"), still yielding regularization, which helps obtain better generalization, m should be treated as nyper-parameter its value depending on the training setting itself. In our experiments, we chose m because it offers a nice balance between the speed-up and performance.
250
400
550
Numbe: of Gradinne spáittss
Figure 3: Comparison of gradient spar- sity in CNN architecture
well as potential speed-up. In conclusion,
5 for most of the experiments, as a proof of concept, but also
Table 4: Trade-off of accuracy vs. speedup with slyeer-pramenn m
5
10
20
100
NMNIST
Accuracy
93.29
93.61
93.69
93.66
93.76
93.67
93.81
Std. 0.08
0.15
0.17
0.13
0.14
0.08
0.14
Over.
3.33
3.28
3.22
3.16
3.06
2.82
1.59
SHD
Accuracy
76.55
76.55
76.50
75.49
75.51
74.96
76.71
Std.
0.93
0.65
0.90
0.66
0.81
0.68
0.49
Over.
4.75
4.62
4.47
4.39
4,25
3.89
2.24
FMNIST
Accuracy
81.79
83.40
83.64
83.70
83.85
83.75
83.87
Std. 0.06
0.06
0.12
0.04
0.11
0.11
0.05
Over. 1.89
1.85
1.78
1.75
1.70
1.56
6 Discussions
We propose a new direct training algorithm for SNNs that establishes a formal connection between the standard surrogate methods and the zeroth order method applied locally to the neurons. The method introduces systematic randomness the training that helps in better generalization. The method simultancously lends itself to efficient back- poockpropagation. We experimentally demonstrate the efficiency of the proposed method in terms of speed-up obtained in training under specialized implementations and its top generalization performance when combined with other training methods, ameliorating their respective strengths.
Acknowledgement
This work is part of the research project "ENERGY-BASED PROBING FOR SPIKING NEURAL NETWORKS" performed at Mohamed bin Zayed University of Artificial Intelligence (MBZUAD, in collaboration with Technology Innovation Institute (TII) (Contract No. TIIARRC/2073/20211..
10

---

References
[1] Mohammed Alawad, Hong-Jun Yoon, and Georgia Tourassi. Energy efficient stochastic-based deep spiking neural networks for sparse datasets. In 2017 IEEE International Conference on Big Data (Big Data), pages 31 1-3188 2017.
[2] Arnon Amir, Brian Taba, David Berg, Timothy Melano, Jeffrey McKinstry, Carmelo Di Nolfo, Tapan Nayak, Alexander Andreopoulos, Guillaume Garreau, Marcela Mendoza, et al. A low power, fully event-based gesture recognition system. Proceedings of the IEEE conference on computer vision and pattern recognition, pages 7243-7252, 2017.
[3] Pierre Baldi and Peter J Sadowski. Understanding dropout. Advances neural information processing systems, 26, 2013.
[4] Albert $ Berahas, Liyun Cao, Krzysztof Choromanski, and Katya Scheinberg. A theoretical and empirical comparison of gradient approximations in derivative-free optimization. Foundations of Computational Mathematics, 22(2):507-560, 2022.
[5] Tong Bu, Wei Fang. Jianhao Ding, PengLin Dai, Zhaofei Yu, and Tiejun Huang. Optimal ann- snn conversion for high-accuracy and ultra-low-latee low-latency spiking neural networks. In International Conference Learning Representations, 2021.
[6] Yongqiang Cao, Yang Chen, and Deepak Khosla. Spiking deep convolutional neural networks for energy-effiiient object recognition. International Journal of Computer Vision, 113(1):54 66, 2015.
[7] Benjamin Cramer, Sehastian Billaudelle, Simeon Kanya, Aron Leibfried, Andreas Grübl, Vitali Karasenko, Christian Pehle, Korbinian Schreiber, Yannik Stradmann, Johannes Weis, et al. Surrogate gradients for analog neuromorphic computing. Proceedings of the National Academy of Sciences, (194)):2109194119 2022.
[8] Benjamin Cramer, Yannik Stradmann, Johannes Schemmel, and Friedemann Zenke. The heidelberg spiking data sets for the systematic evaluation of spiking neural networks. IEEE Transactions on Neural Networks and Learning Systems, 2020.
[9] Ekin D Cubuk, Barret Zoph, Dandelion Mane, Vijay Vasudevan, and Quoe V Le. Autoaugment: Learning augmentation policies from data. arXiv preprint arXiv: 1805.009000 2018.
[10] Simon Davidson and Steve B. Furber. Comparison of artificial and spiking neural networks on digital hardware. Frontiers in Neuroscience, 15, 2021.
[11] Mike Davies, Narayan Srinivasa, Tsung-Han Lin, Gautham Chinya, Yongqiang Cao, Sri Harsha Choday, Georgios Dimou, Prasad Joshi, Nabil Imam, Shweta Jain, et al. Loihi: A neuromorphic manycore processor with on-chip learning. leee Micro, 38(1):82 99, 2018.
[12] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large- scale hierarchical image database. In 2009 IEEE conference on computer vision and patlern recognition, pages 248-255. leee, 2009.
[13] Shikuang Deng, Yuhang Li, Shanghang Zhang, and Shi Gu. Temporal efficient training of spiking neural network via gradient re-weighting. arXiv preprini arXiv: 220.:/199.1 1946, 2022.
[14] Peter Diehl and Matthew Cook. Unsupervised learning of digit recognition using spike- timing-dependee plasticity. Frontiers in computational neuroscience, 9:99, 2015.
[15] Peter U Diehl, Daniel Neil, Jonathan Binas, Matthew Cook, Shih- Chii Liu, and Michael Pfeiffer. Fast- classilass,,ying, aich-accuracy spiking deep networks through weight and threshold balancing. In 2015 International Joint Conference on Neural Networks (IJCNN, pages 1-8. ieee, 2015.
[16] Jason K Eshraghian, Max Ward, Emre Neftei, Xinxin Wang, Gregor Lenz, Girish Dwivedi, Mohammed Bennamoun, Doo Seok Jeong, and Wei Lu. Training spiking neural networks using lessons from deep learning, arXiv preprint arXiv: 12894, 2021.
11

---

[17] Wei Fang, Zhaofei Yu, Yanqi Chen, Tiejun Huang, Timothée Masquelier, and Yonghong Tian. Deep residual learning in spiking neural networks, Advances in Neural Information Processing Systems, 34:21056-21066 2021.
[18] Daniel Gehrig, Antonio Loquercio, Konstantinos G Derpanis, and Davide Scaramuzza. End- to-end learning of representations for asynchronous event-based data. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 5633-5643, 2019.
[19] Wulfram Gerstner, Werner M Kistler, Richard Naud, and Liam Paninski. Neuronal dynamics: From single neurons to networks and models of cognition. Cambridge University Press, 2014.
[20] Seijoon Kim, Seongsik Park, Byunggook Na, and Sungroh Yoon. Spiking-yolo: spiking neural network for energy energy-efficient efffcieent object detection. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pages 1270-112777 2020.
[21] Youngeun Kim, Yuhang Li, Abhishek Moitra, Ruokai Yin, and Priyadarshini Panda. Sharing leaky-integrate-and fire neurons for memory-efficient spiking neural networks. arXiv preprint 2xXi::3305.18360 2023.
[22] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
[23] Alexander Kugele, Thomas Pfeil, Michael Pfeiffer, and Elisabetta Chicea. Efficient processing of spatio-temporal data streams with spiking neural networks. Frontiers in Neuroscience, 14:439, 2020.
[24] Hongmin Li, Hanchao Liu, Xiangyang Ji, Guoqi Li, and Luping Shi. Cifarl O-dvs: an event- stream dataset for object classification. Frontiers in neuroscience, 11:309, 2017.
(25) Yuhang Li, Yufei Guo, Shanghang Zhang, Shikuang Deng, Yongqing Hai, and Shi Gu. Differen- tiable spike: Rethinking gradient-descent for training spiking neural networks. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems, volume 34, pages 2021.
Curran Associates, Inc.,
[26] Sijia Liu, Pin-Yu Yu Chen, Bhavya Kailkhura, Gaoyuan Zhang, Alfred O Hero III, and Pramod K Varshney. A primer on zeroth-orde optmizatiio in signal processing and machine learning: Principals, recent advances, and applications. IEEE Signal Processing Magazine, 37(5):43-54, 2020.
[27] Zachary F Mainen and Terrence J Sejnowski. Reliability of spike timing in neocortical neurons. Science, 268 1995.
[28] Emre O Neftci, Hesham Mostafa, and Friedemann Zenke. Surrogate gradient learning in spiking neural networks:
Bringing the power of gradient-based optimization to spiking neural networks. IEEE Signal Processing Magazine, 36(6):51-63, 2019.
[29] Yurii Nesterov and Vladimir Spokoiny. Random gradient-free free minimization of convex functions. Foundations of Computational Mathematics, 17(2):527-566 2017.
[30] Garrick Orchard, Ajinkya Jayawant, Gregory K Cohen, and Nitish Thakor. Converting static image datasets to spiking neuromorphic datasets using saccades. Frontiers in neuroxcience, 9:437, 2015.
[31] Nicolas Perez-Nieves and Dan Goodman. Sparse spiking gradient descent. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems, volume 34, pages 11795-11888 1888. Curran Associates, Inc., 2021.
[32] Michael Pfeiffer and Thomas Pfeil. Deep learning with spiking neurons: opportunities and challenges. Frontiers in neuroscience, page 774, 2018
[33] Nitin Rathi and Kaushik Roy. Diet-snn: Direct input encoding with leakage and threshold optimization in deep spiking neural networks. arXiv preprint arXiv:2008.00 03658, 2020.
12

---

[34] Nitin Rathi, Gopalakrishnan Srinivasan, Priyadarshini Panda, and Kaushik Roy. Enabling deep spiking neural networks with hybrid conversion and spike timing dependent backpropagation. arXiv preprini arXiv:2005.01887,, 2020.
[35] Simon Schaefer, Daniel Gehrig. and Davide Scaramuzza. Aegnn: Asynchronous event-based graph neural networks. In Proceedings of the IEEE/CVE conference on computer vision and pattem recognition, pages 12371- 12371-12381, 2022.
[36] Sumit B Shrestha and Garrick Orchard. Slayer: Spike layer error reassignment in time. Advances in neural information processing systems, 31, 2018.
[37] Amos Sironi, Manuele Brambilla, Nicolas Bourdis, Xavier Lagorce, and Ryad Benosman. Hats: Histograms of averaged time surfaces for robust event-based object classification. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1731- 1740, 2018
[38] Yujie Wu, Lei Deng, Guoqi Li, Jun Zhu, and Luping Shi. Spatio-temporal backpropagation for training nigh-performance spiking neural networks. Frontiers in neuroscience, 12:331, 2018.
[39] Yujie Wu, Lei Deng, Guoqi Li, Jun Zhu, Yuan Xie, and Luping Shi. Direct training for spiking neural networks: Faster, larger, better. In Proceedings of the AAAI conference on artificial intelligence, volume 33, pages 1311-1318, 2019.
[40] Zhenzhi Wu, Hehui Zhang, Yihan Lin, Guoqi Li, Meng Wang, and Ye Tang. Liaf-net: Leaky integrate and analog fire network for lightweight and efficient spatiotemporal information processing. IEEE Transactions on Neural Networks and Learning Systems, 33(11):624--622, 2021.
[41] Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms. arXiv preprint arXiv: 1708.07747 2017.
[42] Yulong Yan, Haoming Chu, Yi Jin, Yuxiang Huan, Zhuo Zou, and Lirong Zheng. Backpropaga- tion with sparsity regularization for spiking neural network learning. Frontiers in Neuroscience, 16, 2022.
[43] Friedemann Zenke and Surya Ganguli. Superspike: Supervised learning in multilayer spiking neural networks. Neural computation, 3066):51141151) 2018.
[44] Wenrui Zhang and Peng Li. Temporal spike sequence learning via backpropagation for deep spiking neural networks. Advances in Neural Information Processing Systems, 33: 1202--1233,, 2020.
[45] Hanle Zheng, Yujie Wu, Lei Deng, Yifan Hu, and Guoqi Li. Going deeper with directly-trained larger spiking neural networks. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 11022-11070, 2021.
13

---

Appendix: Direct Training of SNN using Local Zeroth Order Method
Appendix A Proofs of theoretical results
Lemma Assume further that Jo zat ((z)dz < co. Then, Ez=s(G?(u; 6)] is a surrogate function.
Proof. Based on our remark above, the only thing left to prove that the integral I- Bzna ( 2, dd is finite. To this end, we have (by using equation
N
2
E > (G 8)]du
((z)dzduu
A())dzuu
lnl
|=|6 1a | ((=)dudz
2
2 {(z)dz,
5
which proves the lemma, by assumptions the resulting integral is finite.
Theorem 2. Let X be a distribution and M(t) its corresponding PDF Assume that integrals So ta M(tdd and Jo go+1 X()dt exist and are finite. Let further À be the distribution with cor- responding PDF function
A(2) = C
ta (a((t)tt,
|2|
where is the scaling constant (such that 1 {(2)dz 1). Then,
d
BE (u, du
E E.iilch(u + +8]]]
Proof. We have
0c - -B + 8z)] = du [ h(u ((u+)2){(2ddz===
du l }(z)dz =
d
C i
u
c
A
u
du
8
which coincides with 8
For our following result, note that surrogate function differentiable almost everywhere, which follows from the Lebesgue theorem the differentiability of monotone functions. So, taking derivatives here understood in an "almost everywhere" sense.
Theorem 3. Let g(u) be surrogate function. Suppose further that c = -28 So g' < co
5² and put A(2) = aza 0'(2)) (so that A(2) is a PDF). Then,
CE (G (G:(;; (w; 2, 8)] = Es~x Gs~ (w; (1, g(u).
Proof. Let assume that (the other case is similar). Then,
Ex~AloG 2, { z" X(z)dz = 1
52
za g (z0)dz
which after change of variables 8 becomes g(u) and finishes our proof.
A.1 Obtaining full-surrogates on Expectation
We demonstrate performance of LOCAL.ZO over different distributions of z, such as standard Normal, Uniform([V3 v3) and Laplace(O, ) for m E (1,5) and 5 = 0.05. The distributions are of unit variance so that parameter á is comparable across the methods. We supply SPARSEGRAD algorithm the back- propppprpagttoon threshold Bi, obtained in Table [ Table 5 shows the performance of the methods on the N-MNIST dataset terms of accuracy and speedup. The LOCALZO method obtains better train and test accuracies for all cases with slight compromise in the speedup, except for the uniform distribution where offers better speedup for m compared to the SPARSEGRAD method.
14

---

A.1.1
From standard Gaussian
Recall that the standard normal distribution 1) has PDF of the form quently, it is straightforward to obtain
1
exp(- ).
Conse-
2T
1
22|
2²
1
u²
E2~a |G (u; 2, (u;
exp(
)dz
exp(
(1I)
V 2m
28
2
8v
25
A.1.2 From Uniform Continuous
Consider the PDF of continuous uniform distribution:
f(z rーa for b]
otherwise,
where < are some real numbers. For the distribution to be and the resulting scaling constant of the surrogate to be 1 (which translates to [[a] = 0 and E[2?] 1, respectively) set, -v3, b v3. Then,
|2|
E |G (u; 2,]
f())dz
=
28"
-00
1
|2|
v33
|2|
de
+
def
2/3
-V3
28
الاl 28
4v 35
Iu]
(3
if [ul
< V3.
1535
8z
(12)
otherwise.
A.1.3 From Laplacian Distribution
The PDF of Laplace distribution is given by:
(z; 2b exp( 2
with mean a and variance 26 Setting, b and μ and using 55) we obtain,
V2
2
0o |a|
1
(u; (u;
exp( exp(-v21) (2p(-22//2/)d2
exp( - (1-222222
V
25
8v2 اباا
1
1 exp(
1
|ul
1
exp( - / |u]
22)
+
(13)
8/2
V2
2
25
22
8
A.2
Simulating a specific Surrogate
A.2.1
Sigmoid
Consider the Sigmoid surrogate function, where the Heaviside is approximated by the differentiable Sigmoid function (3). The corresponding surrogate gradient is given by.
dx d 1 du du 1 + expl -ku)
1: expp(- =: g(()
(1 + exp(-ku))2
and,
(((u) = 1-2 expp(kku)( exp(-ku))
expe k
Observe that g(u) satisfies our definition of surrogate (g(u) being even, non-decreasing on (-o0, 0) and f g(u)d( co). Thus, according to Theorem we have
g (tt) b dt 282
exp(-kōt)(( exp( -kp(-)
k2
c 2
1(1 exeepppp60000 dt a²
15

---

V
where, a a==
0.42662 The corresponding PDF given by
122) = 62 g'tt = a² exp( (1 exp(-koz))
(14)
z(1 + exp(-kåz))
Note that the temperature parameter comes from the surrogate to be simulated, while 5 is used by LOCALZO. We compute the expected back-propagation threshold of SPARSEGRAD for m as, Bih 6 dE==\] with,
E = 2n² /0= exp(-az)(1 - exp(-az)) a =0.7659.
z(1 axp((azz)s dz
(15)
2
A.2.2 Fast Sigmoid
Consider also the Fast Sigmoid surrogate gradient 43 31 that avoids computing the exponential function in Sigmoid to obtain the gradient:
dæ
+ k/ul)² =: g(u).
du
We choose a (note that does not work this case) and apply theorem 3 so that,
1 l sign( 26)
25
za g (zd)dz 482
Jo za (1 k| ) dz
4
/ (1+ k'
dt 2
48
dz
(1+ kdz)3
The PDF is then given by
1 X(2)
z g (z0) = 222 2 zsign(20)
(16)
k|20))3
To compute the expected back pook-propagation threshold, we note,
2² sign(20)
dz 22 /
(1 + k28)3 dz / x²
Ben = [| = 28 222
J.
(1 [1+ dz
(1 + k 20)))
J
The above integral does not converge. However, if we consider finite support [-a, a), may compute, (142)rdz de
A.2.3 Inverse Transform Sampling
To simulate given surrogate in LOCALZO, one needs to sample from the corresponding distribution described by the PDF X. Given sample Unif([0, 1)) and the inverse CDF A of the distribution, the inverse sampling technique returns, Ar1(r), as a sample from the distribution. Suppose the inverse CDF is not computable analytically from the PDF (or not implementable practically). that case, we may choose a finite support over which the PDF is evaluated at a sufficiently dense set of points and compute the discretized CDF using the Riemann sum. The inverse discretized CDF is then computed empirically and stored as list for finite number of points (spaced regularly) between [o, 1]. Sampling from the uniform distribution then amounts to randomly choosing the indices of the list and picking the corresponding inverse CDF values.
A.3 Expected Back- paopppropagttion Thresholds
In what follows, m is the number of samples used in (6). while & the index of particular sample. To compute the expected pro-ppropagttion threshold, we observe that neuron is inactive in LOCALZO back-propagation if,
(1) [t] weal > |= 15, for l: 1,..., m, m
or, [1; (0 Heal > t6, wheret = max |211, ... 12mm
16

---

Assume 24 ~ X, where X(t) denotes the PDF of the sampling distribution, with the corresponding CDF denoted by Fe.. The PDF, X, of the random variable | is given by
0, (2)(),,
ir<0
A(x)
(17)
otherwise.
The corresponding CDF obtained by integrating the previous expression,
FIzc(())= [0. (2 (x) (0), otherwise.
i <0 0
(18)
Further note that,
Fxxx= P(L <
P( <x) = TEx (x)
(19)
If we denote the PDF of the random variable as 2, we obtain
X(x: mF (z){())))()>))
(20)
Finally, the expected back-propagatio threshold takes the form
BI = [T =5 [ (a(I)dl.
(21)
In the cases of distributions used in experimental sections, the previous expression simplifies. Table T
gives the numerical values for some particular m. To obtain expected back- pro-ppropggttion threshold, we would like to evaluate:
Bu = 1E = 8 S. :i(t)dt 5m { (F)()(
For the standard normal distribution, Normal(O, 1) we have, (t) erf( (2) giving.
20m
Ler( m- t
t² / )dl.
Brn V2T
exp((
(22)
For uniform continuous, À = (Inif([v33) v3), we have, P(2(() = /3 giving,
5m
v
t( m- dt 5v3 m
Bi
(23)
V3 J
m 1
For Laplace distribution, À Laplace (0, 1 have, (1) exp(-v21)
Ba = dmv2 /--0 1(1 (11-ex((VV2)))m expe=xpppp222pp
(24)
Appendix B Additional Results
B.1
Computational Speedup in SPARSEGRAD
The SPARSEGRAD frame-work derives speedup by performing the back-propagation in layerwise fashion. To summarize their finding, let us start with eqn. II
t) = Bu (1)
1] + Σ way (1)
x; [t 1]24 1/4
which, after unfolding the recurrence and using the fact that ut [0] 0. can be restated as:
17

---

Leatt
10
1.0
1.5
3.0
1006
2068
3900
4693
1206
2048
3300
4603
1004
2000
3300
4003
SpeneSred
287
Numberal prid Lipearr
Humber of grodint uppatr
Humber of Prgreddrr updiaee
Corme
SpemeCrrd
575
5.5
2.75
432
2.0
1008
2303
3000
4000
1.00
5300
4000
Mumberar sostiinc
Figure 4: We plot training loss, backward speedup (Back.), and overall speedup (Over.) of SPARSEG- RAD and LOCALZO after each gradient update performed on NMNIST data, as summarized in Table The columns represent plots for three distributions Normal 1), Laplace(1, w/2): and
Unif( 3, v3) respectively, with 0.05 and The LOCAL.ZO algorithm converges faster than the SPARSEGRAD method and provides comparable backward and overall speedup.
u 1
Σ
8 k- a1 11- [k)ee I
Σ g gt-k-1 k=
( x1 [kjuch
(25)
k-1 (1-1) kk] can be computed in layer-wise fashion from the forward propagation. Note that, by ignoring the reset mechanism in the gradient the computation, the gradient of the loss can be written
where, the input trace, Σ k= gr-k-1
0l Owij
[i Om[(]] Ou[t]
Σ
Ox(t) Ou[t] Owi
Oi dzid
g 2 (1
(26)
dxlt du[f]
azit) is the given by so it can be where, the term azld is coming from the next layer, the term Jall
a[d
computed in the forward propagation along with the input trace. Thus, whenever, many neurons
inactive, i.c., Bul the SPARSEGRAD work reduce the computation burden of the
8an't
back-propagation.
B.2 Further comparison with SPARSEGRAD
In the section 4.4 we derived surrogates corresponding to distributions and distributions corre- sponding popular surrogates. We implement LOCALZO, with ò 0.05 for different distribution
18

---

Table 5: Performance comparison NMNIST for m=l
METHOD
TRAIN
TEST
BACK.
OVER.
NORMAL(O, 1), 8 0.05,
SPARSEGRAD
93.26 +: 0.31
91.861 0.29
99.57 92.27
3.38
LOCALZO
94.38 0.12
93.29 0.08
3.34
LAPLACE(L, )
6 0.05,
SPARSEGRAD
93.97
0.43
92.65+ 0.52
88.2
3.19
LOCALZO
94.25 + 0.17
93.05 0.09
83.7
3.07
UNIF v3). 8 0.05, m
1
SPARSEGRAD
93.34 + 0.44
91.85+ 0.35
83.2
3.26
LOCALZO
94.24 + 0.46
93.05 0.37
84.8
3.43
SIGMOID, 5 = 0.05, & x 30.63, m
SPARSEGRAD
92.96.1 0.26
91.041 0.32
87.45
3.00
LOCALZO
93.98 0.08
92.97 0.05
83.54
3.02
FASTSIGMOID,
0.05, k 100, m
SPARSEGRAD
93.244 0.23
92.161 0.20
84.87
3.18
LOCALZO
93.44 0.13
92.52+ 0.09
73.23
3.11
Table Performance comparison on NMNIST for m=5
METHOD
TRAIN
TEST
BACK.
OVER.
~ NORMAL 1), 0.05, m
5
SPARSEGRAD
95.02 0.29
93.391 0.25
80.0
3.40
LOCALZO
95.20 0.22
93.691 0.17
77.7
3.22
2 (1 ).
0.05, m
5
SPARSEGRAD
94.73 + 0.29
93.13+ 0.23
72.9
3.15
LOCALZO
95.07 0.03
93.631 0.05
69.4
2.80
UNIF([-V3, v3).
0.05, m
SPARSEGRAD
94.82 0.27
93.38 0.17
76.4
3.14
LOCALZO
94.95 0.35
93.47 0.23
73.5
2.91
such Normal(O, 1), Laplace(1, ) and Unif([-V3, v3).. They all have unit variance sure is comparable across the distributions. The corresponding back poo-pprppggttio thresholds for SPARSEGRAD are derived in Table
For the Sigmoid surrogate, take the temperature parameter à: a/ō 30.63 that 8 for the corresponding PDF derived eqn 14 We supply SPARSEGRAD method the corresponding pack-propagation threshold, Bu, = 0.7665, as obtained in eqn. 15
For the Fast Sigmoid surrogate, we choose & = 100 following 31 so that E. To compute the expected propppropngttion threshold, we consider finite support 10, 10) used in the inverse transform sampling of and evaluate Bia = 0.0461. Table 5 reports accuracies obtained by LOCALZO and SPARSEGRAD across various sampling distributions, with m 1. LOCALZO offers better accuracies compared to SPARSEGRAD across all the distributions, with a slight compromise in speed-up in most cases. For uniform distribution, LOCALZO offers better speed-up than SPARSEGRAD.
Table 6 reports the details of the comparison over the N-MNIST dataset for sampling from various distributions with m = LOCAL.Z0 method obtains better test accuracies. The SPARSEGRAD method is supplied with the respective surrogate and propaproonnntioo threshold computed in
19

---

Number of classes Table 7: Hyper-parameter settings for general comparison
CIFAR-10/100
ImageNet-100
DVS-CIFAR-10
DVS-Gesture 200
N-Caltech/NCARS
Number epochs
300
300
300
200
Mini batch size
64
64, 72
6,4,2
4
10
10
10
LIF: &
0.5
1
0.5
0.5
0.5
LIF:
0
0
0
0
0
LIF: Muh
LOCALZO:
0.5
0.5
0,5
0.5
0,5
LOCAL.ZO: m
5
5,
5
LOCAL.Z0: X
N(0,1)
N(0.1)
N(O.1)
N(0.1)
N(0,1)
ATET
0.05
0.001
0.0001
0.05
0.05
Learning Rate
0.001
0.1
0.001
0.001
0,001
Optimizer: Adam with betas: (0.9; 0.999), Rate Scheduler: cosine annealing
Table 8: Hyper-parameter settings for comparison in SPARSEGRAD framework
FMNIST
SHD
N-MNIST
Number of Input Neurons
784
1156
700
Number of Hidden
200
200
200
10
10
20
Number epochs
20
20
20
Mini batch size
256
256
T
100
300
500
A
lms
lms
2ms
Tar
20ms
N/A
N/A
0
0.2
N/A
N/A
tish
1
1
LOCALZO:
0.05
0.05
0.05
Optimiser
Adam
Adam
Adam
Learning Rate
0.0002
0.0002
0.001 (0.9; 0.999)
Betas
(0.9; 0.999)
(0.9; 0.999)
T The LOCALZZO method achieves better accuracies than SPARSEGRAD for all the distributions, though the speed-up is reduced due to higher sampling cost at = 5.
we further provide the hyper-parameterd for the comparison the SPARSEGRAD frame- work, which are replicated from their work. The FMNIST data uses latency encoding, where each input pixel v, converted to a single spike, the spike timing is determined by:
table g
T(x) Telf log r-O
Table T gives the
setting for comparison in general framework reported in Tab.
B.3
Dropout effect in LOCALZO
fixed neuron has an active gradient for m = 1, only if [u][[] - ueal < |=|ō as described eqr Observe that ui[4] is specific to input data point and time-step. The same neuron can be inactive for another data-point, or at different time-step. Moreover, a neuron can have a non-zero gradient with respect single data point, even if the spikes zero. Fixing neuron and time-step. we plot the distribution of membrane potential (minus membrane threshold), spikes, and non-zero gradients over different data points across batches, where we implement LOCALZO on MNIST data with d 0.5. The plot captures the sparsity spikes and sparsity of neuron gradients for LOCALZZO.
20

---

384
<110 -00 -uo viss woo -nt
-10
-15 2o
everty
everty
Splke,
Spike, t-d
=110m
x01-1
X20
X012-
x00-0
X01-1
000
Count
Inactive
sclive
Iracive
Iractive
Figure 5: Fixing neuron, plot the distribution of membrane potential, spikes, and gradient activity over different data points across batches, where we implement LOCALZO on MNIST data with 0.5. The plot captures the sparsity of spikes and sparsity of neuron gradients for LOCALZO at different time-steps.
21


Reviews:
[{"review_id": "XGq218MKUs", "reviewer": "Reviewer_cVHq", "summary": "This paper presented a direct SNN training algorithm that alleviates the loss of the gradient information and improves the performance of the SNN on multiple datasets, including both static image datasets and dynamic vision datasets.", "questions": "Please refer to the Weaknesses section.", "limitations": "The major limitation of the paper is the insufficient experimental results.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Rigorous theoretical and empirical analysis to justify the necessity of the Zeroth Order technique. \n\n- Comprehensive experimental results with improved performance against the previous SoTA method.", "weaknesses": "- The proposed method is only verified on small-scale datasets such as CIFAR-10/100 and DVS-CIFAR-10. Since the proposed method shows improved performance on simple vision tasks, it is necessary to further verify the performance on large-scale datasets such as ImageNet-1K or ImageNet-100. \n\n- I understand that static CIFAR datasets are standard vision tasks in most of the recent direct SNN training methods. However, one of the major advantages of SNN is the ability to process the spatial temporal visual information which widely exists in the captures of the event-based sensor/camera. Solely verifying the proposed method with the DVS-CIFAR10 dataset is insufficient, it is very useful for the research community if the author can provide the performance with more DVS datasets (e.g., IBM gesture, NCARS, N-CalTech101), which was reported by [1]. \n\n[1] AEGNN: Asynchronous Event-based Graph Neural Networks, CVPR'22. \n\n- I wonder if the proposed method is applicable to the SpikeFormer [2]? \n\n[2] Spikformer: When Spiking Neural Network Meets Transformer, ICLR'23", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**W1:** The proposed method is only verified on small-scale datasets such as CIFAR-10/100 and DVS-CIFAR-10. Since the proposed method shows improved performance on simple vision tasks, it is necessary to further verify the performance on large-scale datasets such as ImageNet-1K or ImageNet-100.\n\n**A1:** We perform experiments of ImageNet-100 dataset using an integrate and fire (IF) SEW-Resnet34 model [7] with ZerO initialization [5]. We compare LocalZO (m=5, $\\delta$=0.5) with the corresponding Gaussian Surrogate, both implemented using TET loss, trained for 200 epochs. The experiments are performed with standard augmentation (RandomResizedCrop(224),\nRandomHorizontalFlip), **with** and **without** ImageNet Policy [8]. The obtained top-1 accuracy is comparable to the SOTA results [6], and exceeds it with the presence of ImageNet Policy. We do not observe any significant loss of accuracy when compared to the surrogate method.\n\n\n| Datasets  | Surrogate |\tLocalZO | Surrogate+Aug |  LocalZO+Aug |\n| :------------- | :-------------: | :---------: | :------------------: | :------------------: |\n| Imagenet-100 |  78.38 |\t**78.58** | **81.58** | **81.56** |\n\n\n**W2:** I understand that static CIFAR datasets are standard vision tasks in most of the recent direct SNN training methods. However, one of the major advantages of SNN is the ability to process the spatial temporal visual information which widely exists in the captures of the event-based sensor/camera. Solely verifying the proposed method with the DVS-CIFAR10 dataset is insufficient, it is very useful for the research community if the author can provide the performance with more DVS datasets (e.g., IBM gesture, NCARS, N-CalTech101), which was reported by [1].\n\n**A2:**  We consider datasets suggested by the reviewer and compare results of LocalZO vs. Surrogate, where LocalZO uses the standard Gaussian distribution to sample z (m=5, $\\delta$=0.5), and the Surrogate method uses the corresponding Gaussian surrogate as obtained in eqn. (9). We perform the experiment with TET loss vs. plain cross-entropy loss accompanied with tDBN, following the experimental settings in the paper.\n\nThe events of the neuromorphic datasets are collected into event frames of dimension (2 $\\times$ H $\\times$ W), and the number of frames (a.k.a. number of bins) are considered as the temporal dimension for SNN. We set the number of frames to 10, and they are resized to dimension ($2 \\times 48 \\times 48$) for all the neuromorphic datasets. It is the same pre-processing step we followed to obtain DVS-CIFAR-10 results. \n\nWe perform the experiments under two settings, **with** and **without** data augmentation, using the VGGSNN architecture reported in the paper. Under data-augmentaion we use the standard augmentation technique of RandomCrop(48, padding=4) and RandomHorizontalFlip. We train the models from scratch for 200 epochs (batch-size 64 for DVS-Gesture, and 16 for N-Caltech and NCARS), while the other training hyper-parameters remains the same as previously reported in the paper .\n\nFor DVS-Gesture dataset (a.k.a. IBM-Gesture) we obtain the top-1 accuracy 98.43\\% which is comparable to the state-of-the-art accuracies (98\\%) reported for this dataset [1]. In N-Caltech-100 dataset we obtain top-1 accuracy of 82.99\\%, which is higher than the reported SOTA(81.7\\%) that uses transfer learning on SEW-ResNet-34 model pre-trained with Imagenet [2]. For NCARS dataset we obtain top-1 accuracy 96.96\\% which is again higher than SOTA of 94.5\\% [1]\n\nThe comparison with the corresponding Gaussian surrogate shows that the randomness introduced by LocalZO frequently helps the training to obtain better generalization performance. The better generalization by LocalZO holds irrespective to the randomness introduced by data-augmentation, and irrespective to the accompanying tDBN or TET method.\n\nWe thank the reviewer for suggesting DVS datasets, we shall include these results in the future version of the paper.\n\n\n| Datasets | Loss | Surrogate |\tLocalZO | Surrogate+Aug |  LocalZO+Aug |\n| :----------- |:------: | :-------------: | :---------: | :------------------: | :------------------: |\n|DVS-Gesture | TET  | 94.9     | **98.04**| 96.09 | **98.43** |\n|                      | tDBN | 87.89   | 95.31     | 92.97 | 91.41 |\n| N-Caltech-101 | TET | 67.24 | **79.86** | 76.04 | **82.99** |\n|                         | tDBN |  68.4 | 74.65      | 75.58 | 79.05 |\n| N-CARS |TET |  95.42 | **96.78**\t | 95.09 | **96.96**|\n|                |tDBN | 94.06\t| 95.96  | 94.83 | 95.68 |\n\n**W3:**  I wonder if the proposed method is applicable to the SpikeFormer [2] ?\n\n**A3:** Spikeformer uses derivative of the sigmoid function as the surrogate gradient for the Heaviside function (see Appendix C1 of [2]). On the other side, LocalZO method is able to simulate the derivative of the sigmoid function (see Appendix A.2.1 of our paper), and as such it is applicable to the Spikeformer. Although it would be a very interesting application of our method that we are happy to test in the future, it goes beyond of the present scope of our paper.\n\n[1] AEGNN: Asynchronous Event-based Graph Neural Networks, CVPR'22.\n\n[2] Spikformer: When Spiking Neural Network Meets Transformer, ICLR'23\n\n[3] \"Sequence approximation using feedforward spiking neural network for spatiotemporal learning: Theory and optimization methods.\" ICLR'22.\n\n[4] \"End-to-end learning of representations for asynchronous event-based data.\" ICCV'19.\n\n[5] “ZerO Initialization: Initializing Neural Networks with only Zeros and Ones” , TMLR'22\n\n[6] \"Sharing Leaky-Integrate-and-Fire Neurons for Memory-Efficient Spiking Neural Networks.\" arXiv:2305.18360 (2023).\n\n[7] \"Deep residual learning in spiking neural networks.\" Neurips (2021)\n\n[8] “AutoAugment: Learning Augmentation Policies from Data” arXiv:1805.09501 (2019)"}, {"review_id": "EdlLaXLKgr", "reviewer": "Reviewer_DTqo", "summary": "This paper proposes a new direct training algorithm for SNN, combining the standard surrogate methods and zeroth order method together. The algorithm applies the 2-point zeroth order method on the Heaviside function to generate a surrogate gradient, which is more efficient. The author applied his method to various dataset such as CIFAR-10 and CIFAR-100 and outperforms the SOTA methods.", "questions": "How to choose the distribution function of z for different tasks?", "limitations": "No. The author should discuss the limitation of the scalability of the forward gradient method in large network.", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1.\tDue to the efficiency of zeroth order method, this algorithm is computational friendly, reducing the computational burden significantly. The gradients are only back propagated through active neurons.\n2.\tAccording to theorem 3, the algorithm is able to simulate arbitrary surrogate, which means this algorithm has great expression capability.", "weaknesses": "1.This paper has limited novelty. It seems simply to be a combination the forward gradient method [1] and sparse gradient together [2]. \n\n2.Worry about the scalability of the proposed method, as forward gradient method is not widely applicable to large networks [1].\n\n3.The experiments only show marginal improvements.\n\n\n\n\n[1] SCALING FORWARD GRADIENT WITH LOCAL LOSSES. ICLR 2023\n[2] Sparse Spiking Gradient Descent. NeurIPS 2021.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**W1:** This paper has limited novelty. It seems simply to be a combination the forward gradient method [1] and sparse gradient together [2].\n\n**A1:** We try to elaborate on the motivation and technical implementation of our work, hoping to be more clear why our method is not simply a combination of two methods you mentioned.\n\n1) Loosely speaking, forward gradient method is used to estimate the updates of the weights of the network by observing the change in the outputs of the network with respect to the perturbation of the weights. In a variant of this general principle, the authors in [1] use somewhat local version, where the gradients of the weights are estimated by the perturbation of the activations of the subsequent postsynaptic neurons and learning is performed in combination with local losses.\n\n    Contrary to these principles, zeroth-order in our method stems from a different motivation and is focused solely on the spiking neuron's activation function (i.e. the Heaviside function) and its gradient. In general, one uses a fixed surrogate gradient, i.e. a function which will serve in the backward pass in place of the non-existent (in the classical sense) derivative of the Heaviside function. For us, the surrogate function changes from time step to time step, and from neuron to neuron, depending on the ``activity'' of the neuron itself (please refer to the motivation section of our paper for more details and interpretation of the activity of the neuron). Also, in our work the training is done fully end to end using a loss on the output. \n    \n2) Sparse gradient method [2] proposes to use the surrogate gradients with compact support, demonstrating its advantages in the energy efficiency, as in general there will be abundance of zero gradients that will not be passed in the backward pass. However, once the surrogate function is chosen, it is fixed throughout the training, and although comparable to using surrogate gradients which have non-compact support, it is slightly lagging behind when it comes to performance.\n \n      The surrogate gradients that are present in our method have compact support at each time step, potentially offering the benefits of sparsegrad method. However, we do not fix upfront this surrogate function, but rather it changes from time step to time step, in a somewhat random but controllable way: Random, because it depends on a random sample from a distribution and controllable because the distribution is fixed.\n\n\n     The method we propose is more than sum of its basic parts. Motivated by the effect of randomness-based regularizer such as dropout, our goal was to introduce a direct training method which will have regularizing effects due to some introduced randomness, which will offer the benefits of the sparsegrad method, but at the same time having the ability to simulate both non-compactly and compactly supported surrogate functions, hence keeping the best from both worlds (performance and potential energy efficiency). \n\n3) Next, we provide both theoretical and practical framework for our method, further establishing its soundness and validity. Our Theorems 2. and 3. provide close connection between distributions that are intrinsic to our method and surrogate functions that are obtained in the expectations showing that LocalZO can be used as a substitute for any surrogate function that is used in SNN literature. You may refer to Section 4.4 for the applications of these results for some of the more commonly used surrogates. \n\n**W2:** Worry about the scalability of the proposed method, as forward gradient method is not widely applicable to large networks [1].\n\n**A2:** As we only apply forward gradient locally, at neuronal level to provide the derivative of the Heaviside function, there is no problem in applying the method to deeper or larger networks. For example, we trained on Imagenet-100 and confirmed that LocalZO is comparable in test accuracy to the standard surrogate. Please refer to our answer to reviewer cVHq for full details on this experiment (such details will also be added in the final version of our paper).\n    \n**W3:**  The experiments only show marginal improvements.\n\n**A3:** We tested LocalZO method, as suggested by one of the referees on several other neuromorphic datasets. The method demonstrates its advantage in generalization performance compared to the standard surrogate training. Please refer to our reply to the reviewer cVHq, where we present newly obtained results and the experimental setting. \n\n**Q:** How to choose the distribution function of z for different tasks?\n\n**A:** This is an interesting and, if we may say so, a difficult question. To the best of our knowledge, there is no systematic study of performances of surrogate functions on different tasks, and in our experience in the literature one can find all sort of (reasonable) functions yielding trained models that perform well on various task and datasets (and in fact, different surrogate functions performing well on the same tasks). On the other side, the reviewer may take a look at the paper by Y. Li et al. \"Differentiable Spike: Rethinking Gradient-Descent for Training Spiking Neural Networks\" where a somewhat related question has been addressed.\n\n[1] SCALING FORWARD GRADIENT WITH LOCAL LOSSES. ICLR'23\n\n[2] Sparse Spiking Gradient Descent. NeurIPS'21."}, {"review_id": "SAocoCcVjf", "reviewer": "Reviewer_8Gkq", "summary": "The authors propose a new method for training spiking neural networks\n(SNNs). To estimate the gradient of the step function for spike\ngeneration, they propose to directly estimate the gradient by local\nsampling around the point of derivation and averaging the linearly\ncalculated slopes, a method that is known in other fields as\nzeroth-order estimation of the gradient of an arbitrary (potentially\nnon continuous) function. They show that, if the distribution used for\nsampling is chosen appropriately, the approximated gradient approaches\nin expectation the common surrogate gradient approach, where an\narbitrary surrogate function is used for smoothening the gradient of\nthe step function.  They further show how to choose the sampling\ndistribution for a couple of given surrogate functions. Finally, they\nshow that their approach outperforms previous methods on common\nbenchmark tasks in terms of accuracy. In particular, gradient\ncalculation is sparser (since gradients are deemed zero for larger\nsampled values) in particular if only few values are sampled, and thus\nmore efficiently to compute (unless one truncates the surrogate, which\nperforms similarly in this regard).\n", "questions": "The general approach is motivated by dropout, where activity is\nrandomly changed in the forward pass. However, the authors actually\n\"apply\" this principle only for the backward pass (as demanded by\nthe zeroth-order approach). Since $m=1$ for most cases in the\nexperiments, I wonder how results would change if spikes in the\nforward pass would be also randomly generated (and not only the\n(pseudo) backward spikes), in other words, when the pseudo-backward\nspikes where actually also the forward spikes. This would be similar\nto neuron with random spike generation, which actually is biologically\nvery realistic. The randomness in the forward pass would effectively\nsmooth over the Heaviside function as well. However, maybe in practice\nthe activity profile of the SNN would be impacted too much. That might\nbe an interesting discussion point.\n", "limitations": "It would have been interesting to discuss / compare the methods with SNNs that use some form of random spike generation.  ", "rating": 7, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "The paper makes a very interesting connection of the surrogates used\nto train SNNs to a sampling based zeroth order method, and shows that\nand how (in theoretical depth) samples can be generated to approximate\nany surrogate used in literature.\n\nWhile the method does not significantly changes the state-of-the-art,\nsince the SparseGrad method (that truncates the surrogates to have a\nsparser backward pass) seem to perform very similar in accuracy\nas well as compute efficiency, it nevertheless provides a very\ninteresting theoretical underpinning of how sparseness can be\nintroduced into the backward pass.\n\nThe paper is also very well written, clearly motivated and\nprovides clear mathematical proofs of their claims.\n", "weaknesses": "While the theoretical connection is interesting it could be argued\nthat the existing SparseGrad method performs similarly in practice, so that there is not much of an improvement of the state-of-the-art. However, still, the accuracy seems to be slightly improved due to the\nintroduced randomness that apparently leads to slightly better\ngeneralization.\n\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": " **W:** While the theoretical connection is interesting it could be argued that the existing SparseGrad method performs similarly in practice, so that there is not much of an improvement of the state-of-the-art. However, still, the accuracy seems to be slightly improved due to the introduced randomness that apparently leads to slightly better generalization.\n\n**A:** As suggested by one of the reviewers, we further tested our method on several neuromorphic datasets that we did not consider previously. We hope that these new results emphasize even further the advantage of LocalZO method compared to surrogate gradient. Please refer to our reply to the reviewer cVHq for the results and experimental setting.\n    \n**Q:** The general approach is motivated by dropout, where activity is randomly changed in the forward pass. However, the authors actually \"apply\" this principle only for the backward pass (as demanded by the zeroth-order approach). Since for most cases in the experiments, I wonder how results would change if spikes in the forward pass would be also randomly generated (and not only the (pseudo) backward spikes), in other words, when the pseudo-backward spikes where actually also the forward spikes. This would be similar to neuron with random spike generation, which actually is biologically very realistic. The randomness in the forward pass would effectively smooth over the Heaviside function as well. However, maybe in practice the activity profile of the SNN would be impacted too much. That might be an interesting discussion point.\n\n**A:** This is a rather curious point and one that we considered and plan to consider. As you pointed out, a neuron which is a random spike generator is biologically plausible concept, but also, on the ANN side of the story, there are results which show that ReLU or Leaky ReLU with slopes sampled from some distribution during training, while during inference one takes the expected slope, show improved performance over their respective rigid versions. \n\nTo introduce randomness in the forward pass along the above lines, one can simply, at each time step, randomly sample threshold, i.e. $u_{th}\\sim \\lambda$, where $\\lambda$ is some distribution ideally with finite support contained in positive real numbers.  Already here, there is a choice for what is the output of the neuron. For example, first natural choice is $H(u[t]-u_{th})$ ($H$ being the Heaviside function), while in the backward pass one uses the any surrogate applied to the input $|u[t]-u_{th}|$. First tests of this choice (VGG16, CIFAR10 dataset (no data augmentation), 60 epochs) show that the performance is comparable to that of the plain setting, but we noticed that the train and test accuracy are kept close to each other (while in the plain setting the network tends to overfit early on), so one may suspect that even in this simple setting the model is able to generalize well whatever it learned (of course, this is a rather simple experiment and one should not take any results or conclusions as definitive). \nA more interesting, in our modest opinion, is the situation where the neuron outputs $u_{th}*H(u[t]-u_{th})$. As you put it, the randomness in the forward pass would effectively smooth over the Heaviside function as well, seems to be satisfied in this situation. Moreover, all sorts of new phenomena arise. One can study the probability of a neuron (having fixed membrane potential) to fire, the expected output of a neuron, distribution of outputs. At the same time, in the backward pass one faces some interesting choices for the surrogate gradients. If we follow the settings of the previous experiment and use some fixed surrogate, it seems that the network is learning slightly worse than in the other setting (with similar generalization property). But, it also seems that there should be a natural choice for the surrogate, which, intuitively speaking, should depend on the two dimensional distribution of ``neuron membrane potential - neuron output'' (the situation should however be compared with the setting of probabilistic spiking neural networks, c.f. https://arxiv.org/pdf/1910.01059.pdf). \n\nIn conclusion, we firmly agree with your suggestion of introducing the randomness in the forward pass, and moreover to introduce it in such a way that the backward gradient becomes ``naturally apparent'' rather than becoming a choice. The potential elegance of this situation requires a detailed understanding and extensive experiments, that we hope to pursuit in the future."}, {"review_id": "694vstrMOK", "reviewer": "Reviewer_Rm6P", "summary": "This paper proposes a Local Zeroth Order method, which can fit arbitrary surrogate functions by sampling a group of variables from a certain distribution. Experiments have verified the superiority of the proposed scheme.", "questions": "1. I note that the firing threshold used by the authors in the code seems to be a fixed value of 1, so has the expected threshold proposed by the authors in Section 4.5 been used in actual experiments?", "limitations": "I think that the proposed ZO function may require more extensive experiments, especially on large-scale datatsets. Overall, I think this paper have met the acceptance standard. ", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "1. The authors‘ idea about fitting arbitrary surrogate functions by sampling is very novel.\n2. Theoretical analysis about ZO function is persuasive and profound.\n3. Relevant experiments, especially on ResNet-19, can show the obvious advantages of ZO function.", "weaknesses": "1. If time permits, I suggest that the authors can supplement their experiments on large-scale datasets (e.g. ImageNet).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**W:** If time permits, I suggest that the authors can supplement their experiments on large-scale datasets (e.g. ImageNet).\n\n**A:** Due to the time constraint, we were not able to run our experiments on the full ImageNet dataset, but we could run them on the Imagenet-100 dataset, which has the same dimensionality as Imagenet. On such dataset, we confirmed that LocalZO is comparable in test accuracy to the standard surrogate. Please refer to our answer to reviewer cVHq for full details on this experiment (such details will also be added in the final version of our paper). \n\n**Q:** I note that the firing threshold used by the authors in the code seems to be a fixed value of 1, so has the expected threshold proposed by the authors in Section 4.5 been used in actual experiments?\n\n**A:** Actually, throughout the paper we ambiguously use the same term ``threshold'' for two different values. One is the firing threshold ($u_{th}$) for the membrane potential in order for a neuron to spike (which in the code is set to 1), the other has been the threshold for the backpropagation ($\\tilde{B}_{th}$) of gradients, to which the results of Section 4.5 pertain. These thresholds were used in the experiments where we compare our method with SparseGrad method. "}]

Author rebuttal:

Meta-review: A zeroth-order method for training Spiking Neural Network is proposed. Three reviewers voted that the paper be accepted, but one of them DTqo recommends a reject. While all the reviewers appreciated the method, some had concerns about experiments that were resolved in the rebuttal. DTqo had concerns about novelty and lack of adaptivity. However, novelty is subjective and the method improves the result. Authors responded to the concern about adaptivity, but the reviewer didn't respond even after intervention from AC. I personally don't think its a reason enough to reject the paper. Therefore, I recommend accepting this paper.
Decision: Accept (poster)