Surrogate Gradient Design for LIF networks
Authors: Luca Celotti
Jean Rouat
Venue: Submitted to ICLR 2023 (2023)
Keywords: Surrogate Gradients
Spiking Networks
Neuromorphic Computing
Glorot Initialization

Abstract: Spiking Neuromorphic Computing uses binary activity to improve Artificial Intelligence energy efficiency. However, the non-smoothness of binary activity requires approximate gradients, known as Surrogate Gradients (SG), to close the performance gap with Deep Learning. Several SG have been proposed in the literature, but it remains unclear how to determine the best SG for a given task and network. Good performance can be achieved with most SG shapes, after an extensive search of hyper-parameters that can be costly. Thus, we aim at experimentally and theoretically define the best SG across different stress tests, to reduce future need of grid search. Here we first show that the derivative of the fast sigmoid outperforms other SG across tasks and networks, for a wide range of learning rates. Secondly, we focus on the Leaky Integrate and Fire (LIF) spiking neural model, and we show that a SG with low dampening, high sharpness, and low tail fatness, systematically leads to higher accuracy. Thirdly, we observe that the Orthogonal initialization leads the LIF to higher accuracy with most SG. Fourthly, we note that high initial firing rates, combined with a sparsity encouraging loss term, can lead to better generalization, depending on the SG shape. Finally, we provide a theoretical solution, inspired by Glorot and He initializations, to reduce the need of extensive grid-search, to find an SG and initialization that experimentally result in improved accuracy.

SURROGATE GRADIENT DESIGN FOR LIF NETWORKS
A Anonymous authors Paper under double-blind review
A ASSRRACT
Spiking Neuromorphic Computing uses binary activity to improve Artificial Intelligence energy efficiency. However, the non-smoothness of binary ac- tivity requires approximate gradients, known as Surrogate Gradients (SG), to close the performance gap with Deep Learning. Several SG have been proposed in the literature, but it remains unclear how to determinc the best SG for given task and network. Good performance can be achieved with most SG shapes, after a costly search of twper-parameters. Thus, we aim at experimentally and theoretically define the best SG across different stress tests, to reduce future need of grid search. Here we first show that complex tasks and network need more careful choice of SG, and that overall the derivative of the fast sigmoid outperforms other SG across tasks and networks, for wide range of learning rates. Secondly, we focus on the Leaky Integrate and Fire (LIF) spiking neural model, and we note that high initial firing rates, combined with a sparsity encouraging loss term, can lead to better generalization, depending on the SG shape. Finally, we provide a theoretical solution, inspired by Glorot and He initializations, to find a SG and initialization that experimentally result in improved accuracy. We show how can be used reduce the need of extensive grid-search of dampening, sharpness and tail- tail-fattes fatness.
1 INTRODUCTION
Spiking Neuromorphic Computing uses binary and sparse signals construct learning al- gorithms with higher energy efficiency (Henderson et al., 2020; Blouw et al., 2019; Davies et al., 2021; Lapique, 1907; Izhikevich, 2003). However, binary signal means that the true derivative is zero essentially always, and training with gradient descent will be at best very poor. Research has shown that designing an approximale gradient, referred to as Surrogate Gradient (SG) (Esser et al., 2016; Zenke and Ganguli, 2018; Bellec et al., 2018), significantly improves training success. However, that entails an additional hyper- parameter to choose: the SG to use. Additionally, the best SG depend on the neural architecture chosen, the task, on the learning rate, on the initialization, and so on, making it difficult to know priori which to pick. Thus, finding the best SG for particular setting, requires a time consuming grid search, and reducing that search time is desirable.
To meet that need, we stress test a wide variety of SG, focusing on one specific neuron model, the Leaky Integrate and Fire (LIF) (Lapique, 1907; Gersiner el al., 2014), and provide a mathenatical solution based on gradient stability methods (Glorot and Bengio, 2010; He et al., 2015), to design the best SG for LIF. In contrast, it is standard to pick one SG for all the experiments (Bohte, 2011; Hubara al., 2016a; Bellee et al., 2018; Zenke and Ganguli, 2018; Zenke and Vogels, 2021; Yin et al., 2021), possibly exploring the effect of changing width factor (sharpness) (Zenke and Ganguli, 2018) or a height factor (dampening) (Bellec et al., 2018). Only in the past five years, the possibility of choosing considered (Neftci et al., 2019; Zenke and Vogels, 2021). Morcover, even if many SG can achieve good performance (Zenke and Vogels, 2021), some shapes have more chances to fail training or achieve lower accuracy. seems therefore valuable to have a complete picture of when and where each SG works, and which ones better left, behind.
optimal SG has been

---

For example, more complex neural models and tasks, we measure increase in sensitivity to the choice of SG with complexity, and observe some to degrade more gracefully, which stresses the need to pick the right SG in each setting. We then focus on arguably the simplest spiking neural model, the LIF, and confirm that the initialization scheme has different. impact on each SG. Finally, to be able to propose our theoretical solution, we need to justify the use of high firing rates. Fortunately, we observe that low initial sparsity, can help generalization with high final sparsity. We use this observation setting the network on high firing rate at the beginning of training, is not in contrast with a low firing rate at the end of training. Taking this finding into consideration, and in the spirit of Glorol and He initializations (Glorot and Bengio, 2010; He et al., 2015), we propose four conditions that keep the representations and gradients stable with time. We show that these conditions provide syper-parameters that result in improved performance without additional prrrmeerr grid-search. When we observe closely the fine details of the SG shape, such as (1) its dampening, (2) its sharpness, and (3) how fast it decays to zero, i.c tail-fatness, we see that the theorctically justified choice tends to be close to the best experimental choice.
better justify, that
Our contribution therefore
We show how task and network complexity, lead learning to be more sensitive to the choice SG;
We observe that the derivative of the fast-sigmoid outperforms other SG across
tasks and networks;
High initial firing rate can promote gencralization with low final firing rate; We provide a theoretical method for SG choice based on bounding representations
that improves experimental performance.
Our method predicts dampening, sharpness and (ail-fatness, that lead to high ac- curacy experimentally on the LIF network;
PRELIMINARIES
2.1 INITIALIZATION SCHEMES
Our theoretical method for SG choice is based on techniques from the weights initializa- tion literature, that we use in an unorthodox way, to design SG. The initial values of the network parameters have a strong impact on training speed (Hanin and Rolnick, 2018) and peak performance (Glorot and Bengio, 2010: He et al., 2015). The theory often fo- cuses on fully-connected ford-forward networks (FFN), given their mathematical tractability (Roberts et al., 2022). FFNs are defined as y! b,+ (1o the network output at depth L, (r.) an activation and b E RT Wi E [Ruxn-- are the layer biases and weights, where n; is layer I size. Typically, biases sampled as zero and weights such that Mean[W]] = and Var [ar[Wi] = qj The general recommendation a 1/c x ny to avoid exploding variance of representations (Glorot and Bengio, 2010; He et al., 2015). (Glo- rot and Bengio, 2010) finds [W] = 2/(m)111 + n)) optimal for linear networks (a y), known as Glorot initialization, while (He et al., 2015) finds Var[W]] 2/m-11 for ReLU networks (a(y) (0, v)), known as Нe initialization. Instead, (Saxe et al., 2014) finds column orthogonal Wi optimal for linear networks, known as Orthogonal initialization. Usually Wi elements are drawn from uniform or a normal distribution. We propose the BiGanma distribution, such that Why Gamma(w; a, 3)/2 + Gamma(-w; a, 9)/2, Fig. 1. The BiGamma keeps the optimal variance and orthogonality without sampling weros.
where yo is the data, yı is
On the contrary, theoretical justification for recurrent networks initialization has been pro- posed for the LSTM (Mchdipour Ghazi et al., 2019), and other non spiking recurrent net- works (Hochreiter et al., 2001; Arjovsky et al., 2016; Pascanu et al., 2013). However, on spiking recurrent networks, initialization theory is missing, since arguments such as the Echo State Network (Jacger et al., 2007) do not apply to non convex activations, or ac- tivations without slope regime. In practice, (Zenke and Vogels, 2021) samples [ar[W;] = 1/3n-11 Uniform, while (Bellec et al., 2018) a V Var(W;) = 1/n-11 Normal distri- bution, for similar spiking models.
2

---

(b)
(c)
Giorot Uniform
rectangular
85
Giorot Normal
triangular
Glorot BiGamma
grase
gradint
exponential
He Uniform
gaussian a sigmeid
2.36
He Narmal
Sunne Sollii
fast sigmoid
PseudoSpike
114
ploo
BiGomma
112
-00
-1.1
Centered voltage
Figure 1: Surrogate Gradient shapes and initialization distributions. Panel (a) shows the SG investigated in this work, and (b) the tail dependence of the g-PssudoSpikk SG for q E [1.01, 16.85]. The SG considered are symmetrical around U2 = yı only plot half the curve (centered voltage i2 0). Panel (c) shows the weight sampling distributions used, Gaussian (dashed), Uniform (dottod) and BiGamma (solid), with He, orange. and Glorot initialization, green, for weight shape of (no, ni) (200, 300).
= 0, so
2.2 NEURAL MODELS AND NOTATION
Arguably the simplest spiking recurrent network model is the LIF (Tapique, 1907; Gerstner et al., 2014; Woćniak et al., 2020). It is defined Y, Oedecoyyr-11 2z-1) & where & -1 + Win + b, and Yr is the neuron membrane voltage, using (Glorot and Bengio, 2010; He et al., 2015) notation. We define z; = ( = H 2) H(v) as the spiking activity, where is the spiking threshold, D2 = y & the centered voltage, and (v:) Heaviside function with SG. The term (1 xt-1) represents a hard reset, that takes the voltage to zero after firing. The input 24 can represent the data, or a layer below. It common to write Oddeaay = 7m where dt is the computation time, Tin the membrane time constant, and to multiply the other terms by biologically meaningful constants, that compress for cleanliness. Each neuron can have ils own speed ('denay, intrinsic current b and 0. In this work, all the parameters in the LIF definition are learnable.
We denote vectors as matrices as A, and their elements as The matrix W E [R connects neurons in the same layer, with zero diagonal, and Win E (R"X81-1 connects the layer with the layer below, the data if = O. We curved brackets A(-) for functions, and square brackets A(-] for functionals that depend on a probability distribution. We use interchangeably
Mean[z]. &- Max(z]) and ă Min[z]. We use A for any parameter. In a stack of layers, we add an index / to each parameter and variable. Since the equation only depends on the previous timestep and layer, the probability distribution is Markov chain in time and depth. Therefore the statistics we discuss are computed element-wise with respect to the distribution p(y,,i(t,)) () = p(y: 1t,11, Wree, Win, by, Adecoy, 0z]t, I).
Additionally, we want to understand how the neuron complexity affects SG training. When LIF is upgraded with a dynamical threshold to maintain longer memories, we have the Adaptive LIF (ALIF) (Gerstner et al., 2014: Bellec et al., 2018). Thus, we compare LIF to ALIF, and we propose the spiking LSTM (sLSTM), App. G, defined by changing the LSTM (Hochreiter and Schmidhuber, 1997) activations for neuromorphic counterparts.
2.3 SURROGATE GRADIENTS
As seen in the previous section, a spike produced when the voltage surpasses the threshold, which mathematically can be described through a Heaviside function, (v), that is zero for O and one for v 2 O. We use the tilde to remind that SG is used for training, defined as (v) (B-vv, where 3 is the sharpness, 7 the dampening, is the shape of choice and the product. Therefore "y controls the maximal amplitude of the SG, and 3 controls the width. A high sharpness, mostly passes the gradient for v close to zero, while low sharpness also passes the gradient for a wider range of voltages. Therefore, the gradient can pass when the neuron has not fired. Unless explicitly stated, dampening and sharpness are set to one.
3

---

The SG shapes f we investigated are (1) rectangular (Hubara et al., 2016b), (2) triangular (Esser et al., 2016; Bellec et al., 2018), (3) exponential (Shrestha and Orchard, 2018), (4) gaussian (Yin et al., 2020), (5) the derivative of a sigmoid (Zenke and Vogels, 2021), and (6) the derivative of a fast-sigmoid, also SuperSpike (Zenke and Ganguli, 2018). To make the comparison between different SG fair, is chosen to have maximal value of 1 and an under the curve of 1. We also propose generalization of the derivative of the fast-sigmoid, that we call g-PssudoSpikke SG. Its tail fatness controlled by a lyper-parameter q and we use it to study tail dependence in section 3.5. More in Fig. 1 and App. D.
2.4 DATASETS
More details on the datasets can be found in App. A.
Spike Latency MNIST (sI-MNIST): the MNIST digits (LeCun et al., 1998) pixels (10 classes) are rescaled between zero and one, presented as flat vector, and each vector value x is transformed into spike timing using the transformation T(x) = Teff log( ===x) for >> 2 and T(x) = xo otherwise, with 0 = 0.2, Teff 50ms (Zenke and Vogels, 2021). The network input sequence of 50ms, 784 channels (28 x 28), with one spike per row.
Spiking Heidelberg Digits (SHD): is based the Heidelberg Digits (HD) audio dataset (Cramer et al., 2020) which comprises 20 classes of spoken digits, from zero to nine, in English and German, spoken by 12 individuals. These audio signals are encoded into spikes through an artificial model of the inner ear and parts of the ascending auditory pathway.
PennTreeBBnk (PTB): is language modelling task. The Penn TreeBank dataset (Marcus et al., 1993), is a large corpus of American English texts. We perform next time-step prediction at the word level. The vocabulary consists of 10K words, which we consider as 10K classes. The one hot encoding of words can be seen as a spiking reproscntation, even is the standard representation in the non neuromorphic literature.
2.5 TRAINING DETAILS
Our networks comprise two recurrent layers. The output of each feeds the following, and the last one foeds linear readout. Our LIF network has 128 neurons per layer the sl-MNIST task, 256 on SHD, and one layer of 1700 and another of 300 on PTB, as in (Wožniak et al., 2020). On the SHD task, the ALIF has 256 neurons and the sLSTM 85, to keep comparable number of 350K parameters. We train on the crossentropy loss. The optimizer had strong effect, where Stochastic Gradient Descent (Robbins and Monro, 1951; Kiefer and Wolfowitz, 1952) was often not able to learn, and AdaM (Kingma and Ba, 2015) performed worse than AdaBelief (Zhuang al., 2020). AdaBelief hyper-parameters are set to default, as in (Radford et al., 2018; Zenke and Vogels, 2021). The remaining typer-parameters are reported in App. A. Unless explicitly stated, we use Glorot Uniform initialization. Each experiment is run 4 times and report mean and standard deviation. Experiments are run in single Tesla V100 NVIDIA GPUs. We call our metric the mode accuracy: the network predicts the target at every timestep, and the chosen class is the one that fired the most for the longest.
3 RESULTS
3.1 SENSITIVITY TO COMPLENITY
3.1.1 METHODOLOGY
In order to portray the difficulty of choosing the right SG for the right task and network, we investigate the SG training sensitivity to task and network complexity. We estimate the task complexity by the number of classes. Thus, if CT(+) measures task complexity, (sI-MNTTT) < Cr(SHD) < (PTB.. We quantify neural complexity as in (Yin et al., 2021), and Tab. 1, by the number of operations performed per layer. In essence, if (-)() measures model complexity, then Cm (M(FF) < (ALIF))) < (sLSSM) To have comparable losses across tasks and networks, we normalize their validation values between
4

---

Validation Perplexity
a)
SI-MNIST
b)
SHD
c)
PTB
g)
LIF network
10.
20
1200
Sensitivity
10
5
2-
10-5
10
10-2
10-2
10-3
10-f 10-3 10-2
100
10
10
10
10-2
0.0 SHMNIST
SHD
PTB
d)
e)
f)
h)
Task
LIF
ALIF
SLSTM
20
20
20
SHD task
Sensitivity
10
10
10
0.1
0.0
10
10
10
10
10
10
AUF sLSTM
Learning rate
Neural Model
rectangular
triangular
exponential
gaussian
a sigmoid
fast sigmoid
Figure 2: The derivative of the fast-sigmoid outperforms other SG across tasks and networks. Grid search over SG shapes, learning rates, tasks and networks. Perplexity is loss, so, the lower the better. We report lowest validation perplexity after converged training. Panels a-f) show perplexity (y-axis), against learning rate (x-axis). In a-c) we fix the LIF network and change task, while in d-f) we fix the SHD task and change network. Panels g-h) show SG sensitivity (y-axis) against task and neural model (x-axis). Plots b) and d) are repeated for clarity. a-f) Different SG roughly agree the best learning rate. The best loss is achieved by the ô fast-sigmoid, also the most resilient to changes in the learning rate, shown in (Zenke and Vogels, 2021). g-h) The more complex the task the network, the more variance in the performance we see across SG choices and learning rates.
O and 1. For that, remove the lowest loss achieved by network in task for any seed and learning rate, and divide by the distance between the highest and lowest loss. We call the result the post-training normalized loss. We call sensitivity the standard deviation of the post-training normalized perplexity across SG, for each learning rate. We report mean and standard deviation across learning rates.
3.1.2 EXPERIMENTS
We see in Fig. 2, that task and network complexity have measurable effect on the sensitivity of training to the SG choice. We run a grid search over learning rates and SG shapes. The sensitivity to the task is shown in the upper panels, for the LIF network. We see that different SG agree the optimal learning rate. We also see that the Ô fast-sigmoid performs well for a wider range of lcarning rates. The rectangular SG is competitive on some tasks, but fails to learn with most learning rates PTB. Then focus network sensitivity, fixing the SHD task, lower panels. The triangular SG performs similarly to the exponential on the LIF network, while it underperforms the best SG on both the LIF and the sLSTM, but not on the ALIF. All this manifests a strong sensitivity
ALIF, and fails sLSTM. The exponential SG matches
the SG choice. Surprisingly, the sLSTM lags behind the LIF and ALIF, with a comparable number of parameters. The gating mechanism devised to keep the LSTM representations from exploding exponentially, are not relevant anymore for Hcaviside that cannot explode exponentially, and might have become computational burden. Fig. 2, g-h), confirm that there is a correlation between task and network complexity, and SG sensitivity.
3.2 ORTHOGONAL INITIALIZATION LEADS TO HIGHER ACCURACY
In contrast, SG training seems less sensitive to the weights initialization scheme. Fig. App. C, shows the initialization scheme effect on SG training, for the LIF network on the SHD task. We compare the Glorot (2/V ( = A4-1 + ny, (Glorot and Bengio, 2010)) with the He (2/Var(W) mt-1, (He et al., 2015)) and the Orthogonal initialization (Wraa
5

---

and Win as column orthogonal matrices, (Saxe et al., 2014)). We use three sampling distri- butions: Uniform, Gaussian and BiGamma, Fig. 1. Orthogonal Uniform is not considered since after orthogonalizatio, the distribution was not Uniform anymore. We use the same initialization for Wre and Win- The best SG with Glorot Uniform is the Ø fast-sigmoid, while with Orthogonal Normal is the exponential. He gives the best outliers, and Orthogonal Normal gives the best mean accuracy. The BiGamma reduces the result variance. Overall best mean is achieved by the derivative of the fist-s and by the exponential SG, suggesting similar behavior across initializations.
3.3 HIGH INITIAL FIRING RATE CAN PROMOTE GENERALIZATION WITH LOW PINAL FIRING RATE
3.3.1 METHODOLOGY: SPARSITY AND BINARITY ROLES IN GENERALIZATION
In order to propose our theoretical method for SG choice, we want to make sure that high initial firing rates are not pernitious neither for learning nor for final sparsity. This is 80, because in the neuromorphic literature training success is judged by (1) training performance and (2) activity sparsity, We show in Sec. 3.3.2, that low initial sparsity can improve generalization in synergy with sparsity encouraging loss term (SELT). However, the energy gains of spiking networks also come from their binary activity. A matrix-vcctor multiplication, with Rm"X" matrix, has an energy cost of mMEMAC for real vector, and of Eac for a binary vector, where p is the Bernouilli probability of the binary vector, and in our case the neuron firing rate, and EAC, AM are the energies of accumulate and a multiply-accummllte operation (Yin et al., 2021; Hunger, 2005). We quantify the sparsity of a binary vector as p. Since MAC are more costly than AC, 31 times on a 45mm complementary metal axide semiconductor (Yin et al., 2021; Horowitz, 2014), we have energy savings with any P, c.g.; when all neurons fire (p = 1) and when they fire half of the time steps (p = 1/2). This gain does not depend on the simulation speed, since it compares a spiking and an analogue computation, at the same computation speed.
We measure the Pearson correlation of initial and final firing rale pt, Ps with test loss after training, in two settings, with and without a SELT. The SELT is a mean squared error between target firing rate p: 0.01 and the layer firing rate. To achieve different ps, pre-train by on the dataset of interest, holding the other parameters untrained, using only the SELT without the classification loss. The coefficient to multiply the loss term is chosen to make all losses comparable only when the task is learned, to let the network focus first on the task and then the sparsity. We therefore chose the multiplicative factor the minimal training loss achieved without SELT, since the SELT takes values between zero and one. We switch on the SELT gradually during training. The switch starts as zero, and moves linearly to one between 1/5 and 3/5 of training. We focus on the 0 fast-sigmoid and the SHD task in the main text, but we show different SG and tasks in App. H.
3.3.2 EXPERIMENTS
We can see in Fig. 3 that with and without a SELT, higher ps correlates with performance. Correlations are bold when p-value < 0.05. Notice that SELT achieved worse final train loss (not shown). However, the high combined with SELT resulted in better test loss, thus, better generalization. However, this is not consistent across SG shapes, Fig. 7, but is consistent across tasks, Fig. 8 App. H. In fact, the triangular SG prefers low p; and the exponential SG does not show a clear trend. Incidentally, the lower layer always reaches higher sparsity, across seeds (Fig. 3), SG shapes (Fig. 7) and tasks (Fig. 8).
3.4 OUR THEORETICAL METHOD FOR SG CHOICE IMPROVES EXPERIMENTAL PERFORMANCE
Keeping in mind that we can exploit low initial sparsity regularization mechanism, propose a method for SG design in spiking recurrent networks inspired by FFN initializations (Glorot and Bengio, 2010; He et al., 2015), that keeps stable gradients with time. We propose four conditions, as four hypothesis to test, that result in a SG that depends on the network and the task. We present. the mathematical equivalent. in each subsection

---

Figure High initial firing rate can pro- mote generalization with low final fir- ing ratc. We use SHD task and the d fast- sigmoid SG. Bold correlation means p-value S 0.05. On the two left panels, learning starts from different P without a SELT, while on the two right panels a target sparsity is encour- aged. In both cases, the initial firing rate cor- relates with final performance, and a low ps is achieved successfully using SELT. Notice as well that the combination of high initial fir- ing rate and sparsity encouragement resulted in better test accuracy than on the two panels on the left, suggesting that both factors acted synergistically as regularization mechanism.
à fast sigmoid on SHD
no Sparsity Encouraging Loss Term
Sparsity Encouraging Loss Term
[n --0.42
00.34
-0.38
n -0114
-.-4..4
m--00.43
-0.38
Test Perplexiyy
1:5--
455
4.0-
0.4
3.50
sio firing rate
Initial fingn rate
Final
Initial firing rate
firing rate
layer
layer 2
I Each neuron has to fire half of the timesteps. IT Recurrent and input variances should match.
III Gradients must have equal maxima across time.
IV Gradicnts must have equal variance across time.
3.4.1 RECURRENT MATRIX MEAN SETS THE FIRING RATE (I)
As previously seen, high p; can generalize better with lower pf- Moreover, notice that
SG curves reach their highest when the neuron fires, Fig. Thus, if the voltage stays close to firing, the gradient is stronger, which is always so if Median(o] = and Varfu] []] = 0. However, Varlu] O turns off all higher moments, thus, we only assume Median[e] = 0 as the mathematical equivalent of our desiderata. When (I) is applied to LIF network, see App. E.1, the mean of the recurrent weight matrix fixes pi, further assuming Win 0, b 0, the approximation Mean [v] R Median(u)) and constant over time, we find
1 - Ostoony)
Wrec 111
(L)
The assumption Mean[u] Median(v), can be justified by noticing that if is sampled from a unimodal distribution with the first two moments defined, then [Mean[o] - M Median s V0.6V ar(p) [e] is true (Basu and DasGupta, 1997). Experimentally, we observe always uni- modal distributions, that verify [Mean[y] Median[e]] /e Varlul, with e 10-1 for the SHD task, c= 3 x 10-2 for sl-MNIST, and c 10-3 for PTB, with and without (I).
3.4.2 RECURRENT MATRIX VARIANCE CAN MAKE RECURRENT AND INPUT CONTRIBUTION TO VOLTAGE COMPARABLE (IT)
Also pertaining the forward pass, we want the neuron to be as sensitive to the network history as it is to new input at initialization, when the structure of the task is unknown. We Var[WrecTtt1]
describe it mathematically
In LIF network, it sets the
variance of the recurrent matrix that makes both contributions equal, see App. E.2, further assuming Ex = 1/2, win = O, and computing V Var[z]] and 2, on the train set, we obtain
R1-1 Var[wa.] Var[w.ed] 2(Varlz)) =Z)
2 Trec
1 2
(II)
nu
3.4.3 DAMPENING AND SHARPNESS SET GRADIENT MAXIMUM AND VARIANCE (III, IV)
Instead, to control the backward pass, we want stable gradients with time. We describe mathematically (III) as Max a = Maxl and (IV) as Var ye] ye] = Varl -t-1 Vt-
On LIF network, they set the dampening and the second moment. of the SG that keep the maximum and variance of the gradient stable with time, see App. E.3, E.4. Sharpness and
tail- tail-fatness are linked to the SG second moment, see App. F.5. Assuming o' and 00 Ut-1 as
7

---

0.80
Figure 4: Our theoretical method for SG choice improves experimental performance. Simulations the SHD task for LIF network. We propose 4 conditions lo select SG for a LIF networks. (I) requires spiking half the timesteps, (II) encourages compara- ble presynaptic and input contribution to voltage, while (IIT) and (IV) encourage gradient maxima and variance constant with time. The baseline has no conditions applied, in gray, with an exponential SG and Glorot Uniform initialization. Lower and upper panels show validation and test accuracies. (IT) has the highest impact its own, but all conditions combined achieve the best result, showing that a theory of SG design can reduce the need of extensive hyper- peper-parranete search.
Test Accuracy
0.75
0.70
0.65
Validation Accuracy
0.95
0.90
0.85
0.80
IV
no conditions
III
11
1+11
1+11
1411
+Il
+I1
Conditions
independent, and zero mean gradients at initialization, we find
1
(1
Oddecay)
(III)
(m 1) 1)wree
(IV)
(nu 1)w2
3.4.4 EXPERIMENTS
Fig. shows training results with our conditions for the LIF network on the SHD task, with exponential SG, against the unconditioned baseline. (II) improves accuracy the most when applied on own, but the best performance is achieved with all conditions together. When all conditions are applied, LIF network achieves 92.7 + 1.5 validation and 75.8 3.1 test accuracy, compared to 87.3 3 + 1.4 validation and 69.0 + 5.8 test accuracy without conditions.
3.5 THE CONDITIONS PREDICT BEST EMPIRICAL DAMPENING, SHARPNESS AND TAIL-FATNESS ON THE LIF NETWORK
We compare experimentally the performance of range of values of dampening, sharpness and tail-fatness and we assess how they compare to the theoretical prediction. Fig. shows the accuracy of the LIF network on the sI-MNIST task. Each SG has its tail decay: inverse quadratic for the 0 sigmis,, no tail for the triangular and rectangular, and exponential decays for the rest. Low dampening and high sharpness are preferred by all SG. Interestingly, the accuracy of the fast-signoid degrades less with suboptimal 7, B. The vertical dashed lines are predicted by our theoretical method, condition (IIT) for the dampening and (IV) for the sharpness of an exponential SG. We observe that they find 7, B with high experimental accuracies. This supports the claim that to reduce syper-parameter search of dampening and sharpness is possible. We use our q- PseuuoSppik SG to study the dependence with the tail- fatnoss, panel (c) Fig. 5. All tail-fatness values perform reasonably well, with maximum at q = 1.56, smaller than the 2 the à fast-sigmoid. Interestingly our theorctical solution gives q 1.898 + 0.002, surprisingly close to the experimental optimum.
DISCUSSION AND CONCLUSIONS
Surrogate Gradients have reduced the gap between Spiking Neuromorphic Computing and Deep Learning, with the consequent energy efficiency gains. Different SG can achieve similar performance, at the expense of potentially costly hyper-parameter search. Our goal was to
8

---

(a)
(b)
Validation Accuracy
Validation Accuracy
Validation Accuracy
0.91
0.7
0.7
0.5
0.90
0.3
0.3
0,89
0.1-
0.3
1.0
1'
0.3
1.0
1
10'
10
Dampening
Sharpness
q faimess
rectangular
triangular
exponential
gaussian
sigmaid
sigmoid
PPPedd
Figure 5: Low dampening, high sharpness and low tail-fatness, lead to higher accuracy on the LIF network. Analysis done on the LIF network over the sI-MNIST
task. Panel (a) shows performance for different values of dampening, sharpness set to 1, and viceversa on (b). Dampenings higher than worsen performance while the pattern is the opposite for sharpness. Dashed vertical lines are our theoretical prediction for the exponential SG, (III) for the dampening (7 = 0.20 + 0.02) and (IV) for the sharpness (8 = 1.02 0.117, which agree with the experiments. Panel (c) shows tail-fatness sensitivity of the q- P-PeeddSppike SG, for 3 = 1. The theoretical prediction gives close to optimal
q = 1.898 0.002, where the best experimentally was g = 1.56.
reduce the need of such search in the future, with experimental and theoretical insights. We saw that best SG across networks and tasks was the derivative the fast-sigmoid, also known SuperSpike, and the SG sensitivity increase with task and network complexity. Incidentally, we reached spiking state-of-the-art on the PTB task with the triangular SG. Best average over 12 seeds had 122.8 + 10.7 validation and 114.2 = 9.2 test perplexity, and best seed had 117.2 validation and 109.5 test perplexity. Previous spiking SOTA on PTB was 137.7 Lest perplexity (Woániak al., 2020). Then, we saw the Orthoyonal Normal as the best ializaliao across SG, on the LIF network and the SHD task, and our BiGamma weight distribution reduced final variance. We saw that for some SG, a high initial firing rate, combined with sparsity encouraging loss term, can improve generalization.
The literature on optimal SG is growing in activity (Neftci al., 2019; Zenke and Vogels, 2021; Yin et al., 2021). However, thcoretical framework was needed. We provide principled method for initialization and SG design, in the form of four conditions. All of them apply to any architecture. We derived the implications experimentally improved training. The dampening, sharpness and tail-fatness predicted were among the best empirically, Fig. 5. We saw preferrence for low dampening, high sharpness and low tail- tail-fatness, making the ideal SG close to a delta, with heavy tails. Also, passing gradients for voltages far from zero, could allow the network learn from outliers. Therefore, our method can help reduce the costly hyper-paramcter grid search. Conditions (II), (III) and (IV) are not restricted to spiking neurons. (I) was introduced to find the SG maxima, but all deep learning activations, have their regime of interest around zero. So, (I) also universal. Even if not. stated, (Glorot and Bengio, 2010; He et al., 2015) the form M Mean(nz] = 0, to determine that Mean[W,] = Mean = 0 at initialization Our theoretical solution applies to convolutional in and Wrec. Take Var [Wr[W..] = ny V Var(wn)) where is the number of presynaptic neurons for each postsynaptic neuron. In our case ne was equal to n--1- For rD convolutions, ne k k'ng, where & the kernel size, ny the number of filters and typically the convolution dimension is 1,2. Our conditions also apply to different reset methods, App. F. Notice that our conditions could fix 7, 3, and q, which turns them into tools for SG design. This leads to new theoretical understanding of the roles played by dampening and sharpness. The dampening keeps the maximal gradients stable through time, while the sharpness keeps the gradient variances stable through time. In summary, this work is in response to the call made by (Zenke and Vogels, 2021) for a theory of SG choice, and it is first step in that direction.
the LIF, and we saw
(I) in
9

---

REFERENCES
M. Arjovsky, A. Shah, and Y. Bengio. Unitary evolution recurrent neural networks. In International conference on machine learning, pages 1120 1128. PMLR, 2016.
Basu and A. DasGupta. The mean, median, and mode of unimodal distributions: a characterization. Theory of Probability 8 Its Applications, (2):22210 22222,22,,,2 1997.
G. Bellec, Salaj, Subramoney, Legenstein, and W. Maass. Long short-term memory and learning-to- to-learn in networks of spiking neurons. In NeurIPS, 2018.
P. Blouw, X. Choo, E. Hunsberger, and C. Eliasmith. Benchmarking keyword spotting efficiency neuromorphic hardware. In Proceedings of the 7th Annual Neuro-inspired Computational Flements Workshop, pages 1 1-8, 2019.
S. M. Bohte. Error-backpropaggaiion in networks of fractionally predictive spiking neurons. In ICANN, pages 60 68. Springer Berlin Heidelberg, 2011. ISBN 378-3-642-21735-7
B. Cramer, Y. Stradmann, J. Schemmel, and F. Zenke. The heidelberg spiking data sets for the systematic evaluation of spiking neural nelworks. IEEE Transaclions Neural Networks and Learning Systems, 2020.
M. Davies, A. Wild, G. Orchard, Y. Sandamirskaya, G. A. F. Guerra, P. Joshi, P. Plank, and S. R. Risbud. Advancing neuromorphic computing with loihi: survey of results and outlook. Proceedings of the IEEE, 109(5):911-9344 2021. doi: 10.1109/PPRO.2001. 3067593.
S. K. Esser, P. A. Merolla, J. V. Arthur, A. S. Cassidy, R. Appuswamy, A. Andreopoulos, D. Berg, J. L. McKinstry, T. Melano, D. R. Barch, et al. Convolutional networks for fast, energy-efficient neuromorphic computing. Proceedings of the national academy of sciences, 133411::1144111144 2016.
W. Gerstner, W. M. Kistler, R. Naud, and L.. Paninski. Newronal dynamics: From single neurons to networks and models of cognition. Cambridge University Press, 2014.
X. Glorot and Y. Bengio. Understanding the difficulty of training deep feedforward neural networks. In AISTATS, pages 249-256, JMLR Workshop and Conference Proceedings, 2010.
B. Hanin and D. Rolnick. How to start training: The effect of initialization and architecture. In NewrIPS, 2018.
K. He, X. Zhang, S. Ren, and J. Sun. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In ICCV, pages 1026-1033 1034, 2015.
P. Henderson, J. Hu, J. Romoff, E. Brunskill, D. Jurafsky, and J. Pineau. Towards the systematic reporting of the energy and carbon footprints of machine learning. JMLR, 21 (248):1-43, 2020.
S. Hochreiter and Schmidhuber. Long short-term memory. Neural computation, 9(8): 1735-178 1997.
S. Hochreiter, Y. Bengio, P. Frasconi, J. Schmidhuber, et al. Gradient flow in recurrent nets: the difficulty of learning long-ten term dependencies, 2001.
M. Horowitz. 1.1 computing's energy problem (and what we can do about it). In 2014 IEEE International Solid-Slale Circuils Conference Digest of Technical Papers (ISSCC), pages 10 14. IEEE, 2014.
Hubara, M. Courbariaux, D. Soudry, R. El-Yaniv, and Y. Bengio. Bi- narized neural networks In NewrIPS, volume 29. Curran Associates. Inc., 2016a. URL https: /proee///roce wwwprsseurrs... /c/paperr/216/fiile/ dasoffesrarreaaaarrtteeeertoo---
pdf.
10

---

I. Hubara, M. Courbariaux, D. Soudry, R. El-Yaniv, and Y. Bengio. Binarized neural networks. NeurIPS, 29, 2016b.
R. Hunger. Floating point operations in matrix-vector calculus, volume 2019. Munich Uni- versity of Technology, Inst. for Circuit Theory and Signal ..... 2005.
E. M. Tzhikevich. Simple model of spiking neurons. IEEE Transactions on neural networks, 14(0):1569-1555 2003.
P. Izmailov, D. Podoprikhin, T. Garipov, D. Vetrov, and A. Wilson. Averaging weights leads to wider optima and better generalization. 34th Conference Intelligence 2018, UAI 2018, 2018.
Uncertainty in Artificial
H. Jaeger, M. Lukoševičius, D. Popovici, and U. Siewert. Optimization and applications of echo state networks with leaky-integrator neurons. Neural networks, 20(3):335-33) 13223355,2,,1 2007.
J. Kiefer and J. Wolfowitz. Stochastic estimation of the maximum of regression function. The Annals of Mathematical Statistics, pages 462-466, 1952.
D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. In Y. Bengio and Y. LeCun, editors, 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track: Proceedings, 2015.
L. Lapique. Recherches quantilatives sur l'excitation electrique des nerfs traitee comme une polarization. Journal of Physiology and Pathololgy, 9:620 635, 1907.
Y. LeCun, L. Bottou, Bengio, and Haffner. Gradient-based learning applied to docu- ment recognition. Proceedings of the IEEE, 86(11):2278 2324, 1998.
1. Loshchilov and F. Hutter. Decoupled weight decay regularization. In JCLR, 2019.
M. P. Marcus, B. Santorini, and M. A. Marcinkiewicz. Building a large annotated corpus of English: The Penn Treebank. Computational Linguistics, 19(2):113-3 330, 1993.
M. Mchdipour Ghazi, M. Niclsen, A. Pai, M. Modat, M. J. Cardoso, S. Ourselin, and
L. Sørensen. On the initialization of long short-term memory networks. In ICONIP, pages 275 286. Springer, 2019.
E. O. Neftci, H. Mostafa, and F. Zenke. Surrogate gradient lcarning in spiking noural networks: Bringing the power of gradient-based optimization to spiking neural networks. IEEE Signal Processing Magazine, 36(6):51 63, 2019. doi: 10.1109/MSP.10 2019..931199
R. Pascanu, T. Mikolov, and Y. Bengio. On the difficulty of training recurrent neural networks. In International conference on machine learning, pages 1310 1318. PMLR, 2013.
A. Radford, K. Narasimhan, T. Salimans, and I. Sutskever. Improving language under- standing by generative pre-training. 2018.
H. Robbins and S. Monro. A stochastic approximation method. The annals of mathematical slatistics, pages 400-407, 1951.
D. A. Roberts, S. Yaida, and B. Hanin. The Principles of Deep Learning Theory: An Effective Theory Approach to Understanding Neurul Networks. Cambridge University Press, 2022.
A. M. Saxe, J. L. McClelland, and S. Ganguli. Exact solutions to the nonlinear dynamics of learning in deep linear neural networks. In ICLR, 2014.
S. Shrestha and G. Orchard. Slayer: Spike layer error reassignment time. In NewrIPS, 2018.
S. Wožniak, A. Pantazi, T. Bohnstingl, and E. Eleftherion. Deep learning incorporating biologically inspired neural dynamics and in m-memorr computing. Nature Machine Intel- ligence, 2(6):325 336, 2020.
11

---

B. Yin, F. Corradi, and S. M. Bohté. Effective and efficient computation with multiple- timescale spiking recurrent neural networks. In ICONS, New York, NY, USA, 2020. Association for Computing Machinery. ISBN 9781450388511. doi: 10.1145/3407197. 3407225. URL
B. Yin, F. Corradi, and S. M. Bohté. Accurate and efficient time-domain classification with adaptive spiking recurrent neural networks. Nature Machine Intelligence, 3(10):905-915 2021.
F. Zenke and S. Ganguli. Superspike: Supervised learning in multilayer spiking neural nelworks. Neural compulation, 30(6):1514 1541, 2018.
F. Zenke and T. P. Vogels. The remarkable robustness of surrogate gradient learning for instilling complex function in spiking neural networks. Neural Computation, 33(4):899- 925, 2021.
J. Zhuang, T. Tang, Y. Ding, S. C. Tatikonda, N. Dvornek, X. Papademetris, and J. Duncan. Adabelicf optimizer: Adapting stepsizes by the belief in observed gradients. NewrIPS, 33, 2020.
12


Reviews:
[{"review_id": "RgS2qyHUx4", "reviewer": "Reviewer_HAej", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This article aims to experimentally and theoretically define the best SG across different stress tests to reduce the future need for grid search and proposes techniques to improve the performance of SNN training from multiple perspectives, such as the dampening of the SG and the normalization method.", "summary_of_the_review": "Overall, it adds an interesting paragraph to the training of SNN with LIF but the performance is not convincing. ", "strength_and_weaknesses": "Strengths:\n1.\tThe authors provide empirical results on how to select SGs and initial methods to enhance the final performance.\n2.\tThe authors designed a new kind of SG that is suitable for all the datasets they used.\nWeaknesses:\n1.\tLack of comparison with other existing work.\n2.\tThis paper only provides experiments on very simple datasets and network structures. More complex cases need to be included.\n3.\tThe logic of this paper is confusing, e.g. many important figures are placed in the appendix, making it difficult for the readers to read.\n4.\tTheir main findings are obtained through some experiments with grid search while lacking logic proof. So the article's novelty and contribution are modest.\n\nQuestions:\n1. In Sec 3.1.2, what is the complete model structure for testing different SGs?\n2. 1.The datasets used in the experiments seem to be relatively small datasets. How does the experiment perform on larger datasets? I think the baseline of this work should be compared with the existing work.\n3. 1.The SG used doesn't guarantee $\\int_{-\\infty }^{+\\infty }f(x) = 1$. Maybe that's why the other SGs don't perform well?\n4. 1.In sec 3.4.1.Is there a missing equation before \"where p is the firing rate\"?\n5. 1.Fig 5. If the baseline is replaced with other initialization methods, will the results change?\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "DLzVSWiSIp", "reviewer": "Reviewer_WUfr", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work finds it unclear to determine which surrogate gradient to apply in different tasks or networks in SNN. Seeking to solve this problem, the authors have done some experiments with different surrogate gradients across tasks and networks, and come to the conclusion that the derivative of fast sigmoid outperforms other chosen surrogate gradients. Besides that, the authors have also done research on how different characteristics of surrogate gradients and initialization affect surrogate gradient learning and found that low dampening, high sharpness, low tail-fatness, orthogonal initialization, and high initial firing rate could help improve the network’s performance. Lastly, the authors provide a theoretical solution based on bounding representations to find a surrogate gradient and initialization that could improve accuracy.", "summary_of_the_review": "The discussion on how to choose SG in SNN is meaningful and missed before.\n\n", "strength_and_weaknesses": "Strength: 1. Training details are explicit, including network architecture and hyper-parameters. 2. Conclusion from the experiments is helpful in choosing surrogate gradient.\n\nWeaknesses: It doesn’t seem reasonable to make the neuron as sensitive to the network history as to new input in section 3.5.2, since the network history is an accumulation of knowlegde learnt from past so it contains more knowledge than new input.\n\nMinor typos:\n\nSec2, line 3 \"y_L the network output\" -> \"y_L is the network output\" or \"y_L means the network output\"", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "Ch8ZJbhixN", "reviewer": "Reviewer_fzV9", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "As claimed by this work, the authors present a framework for designing the best surrogate gradients of SNNs with LIF neurons. \n\nUnfortunately, this paper is hard to follow due to the unclear organization and non-standard presentation. The claimed contributions can be roughly divided into two terms, i.e., summarizing the observed results from experiments and presenting the theoretical understanding of SG design. I cannot demonstrate the correctness of the observed results since there lacks apposite definitions or measures of many terminologies, such as low dampening, high sharpness, and low tail-fatness.\n\nMore importantly, this paper provides a theoretical method for SG choice to improve the experimental performance. What are the assumptions and conclusions of this theoretical investigation?\n\nBesides, I would appreciate to have a clear clarification on the experimental details before showing the experimental results, such as shown in Subsection 3.5.5. ", "summary_of_the_review": "Overall, I believe that the presence of this paper is poor, and thus, it is difficult to verify the quality of its work, which would move the paper away from the accepted standard.", "strength_and_weaknesses": "Weaknesses:\nThe weaknesses are detailed above, including:\n1. This paper is hard to follow due to the unclear organization and non-standard presentation.\n2. Low quality. There lacks apposite definitions or measures of many terminologies, such as low dampening, high sharpness, and low tail-fatness.\n3. The assumptions and conclusions of the theoretical investigation is far from clear.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "Bqt-oj-WQN", "reviewer": "Reviewer_xUW9", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper considers different spiking neuron models and explores hyperparameters on multiple tasks in order to facilitate efficient training via the surrogate gradient method. The study focuses primarily on the shape of the surrogate gradient function. They find that low dampening, high sharpness and low tail-flatness generally lead to better performance. More precisely, the fast-sigmoid derivative appears as the best choice of surrogate gradient function. Secondly, they perform a study of different weight initialisations, which reveals that “He” gives the best outliers, “orthogonal normal” the best mean accuracy and “bigamma” the lowest variance. Thirdly, they demonstrate that using high initial firing rates combined with a sparsity encouraging loss term induces better generalisation. Finally, they investigate various conditions to ensure stable gradients and show that these can also lead to better performances.", "summary_of_the_review": "The authors test a variety of techniques, albeit somewhat inconsistently.  As a review the paper could be useful, but overall it comes across as rather rushed and incomplete.", "strength_and_weaknesses": "Strengths\n\nThe paper aims to facilitate the training of spiking neural networks via surrogate gradients, which remains more challenging and less documented compared to more popular, non-spiking artificial networks.\nSNNs are applied to rather diversified tasks involving image recognition, spoken digit recognition and language modelling.\nSome findings can lead to helpful, practical solutions for SNN training.\n\nWeaknesses\n\nThe overall narrative of the paper needs substantial improvements. A fair number of (sub)sections do not lead smoothly into the next ones. The number of components tested in the study is rather ambitious, which can be okay, but the different findings often feel disconnected, which makes the general direction hard to follow. \nFor instance, the authors define a spiking version of the LSTM, but the results with it are only shown to be inferior to those obtained with simpler, more commonly used and more biologically plausible LIF and ALIF models. Even if interesting, the inclusion of this spiking LSTM does not seem justified and feels out of the way in terms of the general objectives of the paper.\nThe use of the different models and data sets does not feel coherent. Some hypotheses are tested on all tasks and models, whereas others for instance only use the LIF on a single task.\nThe paper feels more like a report than an actual article.\n\nRemarks\n\nThe SHD data set only seems to have training and validation splits in its default version. It would be worth explaining what test split you are using.\nThe spiking LSTM, which is said to be defined in Appendix G, is missing (maybe appendixes are simply not included for this review process, in which case, this remark can be discarded).", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}]

Author rebuttal:

Decision: Reject