Differentiable Spike: Rethinking Gradient-Descent for Training Spiking Neural Networks
Authors: Yuhang Li
Yufei Guo
Shanghang Zhang
Shikuang Deng
Yongqing Hai
Shi Gu
Venue: NeurIPS 2021 Poster (2021)
Keywords: Spiking Neural Networks
Surrogate Gradients

Abstract: Spiking Neural Networks (SNNs) have emerged as a biology-inspired method mimicking the spiking nature of brain neurons. This bio-mimicry derives SNNs' energy efficiency of inference on neuromorphic hardware. However, it also causes an intrinsic disadvantage in training high-performing SNNs from scratch since the discrete spike prohibits the gradient calculation. To overcome this issue, the surrogate gradient (SG) approach has been proposed as a continuous relaxation. Yet the heuristic choice of SG leaves it vacant how the SG benefits the SNN training. In this work, we first theoretically study the gradient descent problem in SNN training and introduce finite difference gradient to quantitatively analyze the training behavior of SNN. Based on the introduced finite difference gradient, we propose a new family of Differentiable Spike (Dspike) functions that can adaptively evolve during training to find the optimal shape and smoothness for gradient estimation. Extensive experiments over several popular network structures show that training SNN with Dspike consistently outperforms the state-of-the-art training methods. For example, on the CIFAR10-DVS classification task, we can train a spiking ResNet-18 and achieve 75.4% top-1 accuracy with 10 time steps. 

Differentiable Spike: Rethinking Gradient-Descent for Training Spiking Neural Networks
Yuhang Li Yufei Guo Shanghang Zhang', Shikuang Deng', Yongging Hai Shi Gu Ibd
University of Electronic Science and Technology of China. PPeking University, 3 University of California Berkeley liyuhang699dgmail.. com, yfguoêpku edu.cn, gusQuestc. cn
Abstract
Spiking neural networks (SNNs) have emerged a biology-inspired method mimicking the spiking nature of brain neurons. This biomimicry derives SNNs' energy efficiency of inference neuromorphic hardware. However, it also causes an intrinsic disadvantage in training high-performing SNNs from scratch since the discrete spike prohibits the gradient calculation. To overcome this issue, the surrogate gradient (SG) approach has been proposed as a continuous relaxation. Yet the heuristic choice of SG leaves vacant how the SG benefits the SNN training. In this work, we first theoretically study the gradient descent problem in SNN training and introduce finite difference gradient to quantitatively analyze the training behavior of SNN. Based on the introduced finite difference gradient, we propose new family of Differentiable Spike (Dspike) functions that can adaptively evolve during training to find the optimal shape and smoothness for gradient estimation. Extensive experiments over several popular network structures show that training SNN with Dspike consistently outperforms the state-of-the-ard training methods. For example, on the CIFAR 10-DVS classification task, we can train spiking ResNet- 18 and achieve 75.4% top-1 accuracy with 10 time steps.
1 Introduction
Recently, spiking neural networks (SNNs) have received increasing attention due to their biology- inspired neural behavior and efficient computation. SNNs deal with binary spike information and therefore enjoy the advantage of multiplication-frr
inference. Neuromorphic hardware such as TrueNorth [1] and Loihi [2] demonstrates that SNNs can save energy by orders of magnitude. Hybrid architecture like Tianjic [3] suggests its potential power for general intelligence when integrated with traditional artificial infrastructure. However, the bio-mimicry also causes an intrinsic disadvantage in training performing SNNs from scratch due the discrete spike. As a result, obtaining high-performing SNN has long been a critical problem limiting SNN's deployment in practice.
To be more specific, the appealing advantage of the multiplication-fee computation in SNN is not recognized in its training [4]. Modern ML frameworks such as Pytorch [5], JAX [6], and Tensorflow [7] do not provide efficient and general instructions high-level functions accelerate the convolution with 0/1 spike activation. Moreover, the binary activation in SNN produces all- or-nothing gradients [8] that are incompatible with typical gradient-based optimization methods for efficient neural network training. To circumvent this problem, many works adopt surrogate gradients [8] (typically the derivative of a soft-relaxed function) to replace the spike non-linear gradient. Although the heuristic choice of SG is beneficial to enable the gradient descent in SNN training, the theoretical soundness of SG has not been justified so far. What's more, the gradient descent in SNN training may follow different updating rules from ANN. In fact, since the true gradient
'Equal Contributions. Cooressponding author. Primarily supported by NST-61876032.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

Pre-spike
Post-spike
Leaky Integrate-Fire
1010
110
Loss Function
Weight Matrix
Dspike
Finite Difference
E0011
Forward
Backward
Shape
Figure 1: Overall workflow of the proposed algorithm. We use a general, flexible Dspike function and control its shape with finite difference method to facilitate the surrogate gradient descent in training SNN.
value of spike activation does exist, the adoption of SG raises new question of how to replace the oo-gradient adaptively through the training process for a good updating strategy of parameters.
In this work, we study this specialized problem of adaptive gradient estimation in SNN training. We first revisit the gradient descent algorithm with the finite difference method to accurately depict the loss landscape of adopting surrogate gradient for the non-differentiabll spike firing function. Then we further propose the Differentiable Spike (Dspike) function, which can adaptively change its shape and capture the direction of finite difference gradients. With such adaptive surrogate gradients, establish an efficient algorithm training SNN with large-scale models. The overall workflow of our algorithm is visualized in Fig. 1.
Our contributions can be summarized as follows:
We identify the problem of adaptively estimating gradient in SNN training and propose novel analysis framework for SNN's surrogate gradient, through which we are able not only explain why it works well but also indicate how we can improve its adaptivity to data through training.
We also introduce the novel Dspike function built from the hyperbolic tangent function. The Dspike function can learn its shape accordingly by matching it with the finite difference gradient.
Extensive experiments on both static and dynamic datasets show that our method is highly effective and efficient. For example, model can achieve 75.4% top-1 accuracy the CIFAR-DVS. Our SNN also only consumes 7.8% energy of the corresponding ANN on CIFAR-10.
2 Related Work
ANN-SNN Conversion. Generally, the algorithms to obtain a high-performance can be divided into two routes: (1) ANN-SNN conversion and (2) direct training SNN from scratch. Conversion-biae based methods [9, 10] utilize the knowledge from ANN and convert the ReLU activation to spike activation mechanism. This type of method can produce an SNN in short time. For example, Rueckauer al. [11] propose to traverse the training dataset to find the maximum activation. Then, the SNN can be converted by directly replacing several modules. In [12], maximum activation abandoned since it may out-lie the central range of activation. The authors choose to find the percentile numbers for threshold voltage to improve the robustness. TSC [13] adopts the soft-reset (also known as reset by subtraction) mechanism and no leakage for conversion, which proves to be effective in conversion. Deng Gu [14] decompose the conversion error to each layer, then propose to reduce the error by shifting the bias term. [15] first propose the calibration to reduce the conversion accuracy. ANN-SNN conversion is fast and accurate. However, achieving near lossless conversion requires a considerable amount of time steps (>200) to accumulate the spikes, which may significantly add up the latency.
conversion method, direct training of SNNs can lower the time step and may further reduce energy consumption. Direct training leverages gradient-descent algorithms. And many works have been developed to tackle the non-differentiabbiity of spike activation. This non non-differentiablee activation function is similar to the quantization binarization
Direct Training SNN. In contrast

---

neural network [16, 17, 18] or adder neural network [19]. [20] achieve high performance on CIFAR- 10 and other neuromorphic datasets. Some works use probabilistic models to approximate the gradient [21], which is also developed in binary stochastic model [22]. Another common approach is to use rate-encoding networks for gradients calculation [23,24] They quantize the spike rate as the
magnitude of information transmitted. (Wu et al. 2019 [25] improves the leaky integrate-and-fin
(LIF) model to an iterative LIF model and develops STBP learning algorithm. Zheng et al. [26]
further propose the tdBN algorithm to balance the gradient norm and smooth the loss function in SNN training, enabling large-scale model optimization.
Preliminary
3.1 Leaky Integrate-and-Fire Model
We use bold italic letters for vector representations. For example, and y denote the input and target output variables. Bold capital letters like W denote the matrices (or tensors as clear from the text). Constants are denoted by small upright letters, e.g., a. With bracketed superscript and subscript,
can denote the time dimension and the element indices, respectively. For example, u. means the i-th membrane potential time step We adopt the well-known Leaky Integrate-and-Filt (LIF) neuron model for spiking neural network (SNN). Given a membrane potential (((), the pre-synaptic potential is updated as
(++ 1).re). ( + c(+++),, where (t++)) (t+1).
(1)
Here, 7 is constant leaky factor within (0, 1) and e(++1) is the pre-synaptic input at time step t 1. The input can be calculated from the dot-product between the weights and the spike from the previous layer. Let Vta be the firing threshold, the membrane potential will fire spike when exceeds the threshold, and then hard-reset to O, given by
y(thi) = {
if u (t+1)) > Ven
u (++1) r (t+1).pe (1 (1-y" ((+))
(2)
otherwise
After firing, the spike output (t+1) will propagate the next layer and become the input (++11) of the next layer, (note that we omit the layer index for simplicity). Ve, is set to 0.5 in this work.
Softmax in SNN. In the general classification model, the final network output will be leveraged to compute the softmax and then to find the winner class. However, the network output is highly variant per input sample, and unlike ReLU activation, the output could be positive or negative. For this reason, we choose to only integrate the network output, and do not fire them across time, as did in recent practices [26,2] 27].
1 Ynst
1
Σ (w
(3)
T
Cet
T
Then, we can compute the cross-entropy loss based on the ture label and Sonemax(yne)).
3.2 Time-dependent Batch Normalization
Batch Normalization (BN) [28] used reduce internal covariate shift and smooth the loss land- scape [29] during the optimization of an ANN. It enables the stable training of ANN even with more than 100 layers on the large-scale dataset, e.g. ResNet-152 [30]. However, BN layers are not designed to normalize spatial-temporal data at first. And simply transplanting BN in SNN training will lead to undesired results. To solve this problem, Zheng et al. [26] propose time-dependent Batch Normalization (tdBN), which normalizes the data in both spatial and temporal paradigm. tdBN will
gather the pre-synaptic input along the time dimension, denoted by C = [c(1) (2) (T) and then normalize by
C = aVi (C - [[C
# normalize
(4)
+C)
Ö - +
# affine transformation
(5)
3

---

where E' and VAR compute the mean and variance in channel dimension. and [ are learnable parameters. Note that in tdBN the normalized tensor will multiply aVi.. which can prevent over-fire and under-fire by choosing a wisely. After tdBN, the pre-synaptic input in each time step will be fed into Eq. (1). tdBN stabilizes the training of SNN and achieves good results. We also adopt this training technique in our framework. See [26] for more details of tdBN.
4 Methodology
4.1 Non-Differentiable Spike
One of the most notorious problems in SNN training is the non-differentiabiiiy of the firing function Eq. (2). The firing function could be viewed as variant of sign function, whose derivative is identified as the Dirac delta function. This derivative is O almost everywhere except for the threshold that grows to infinity. Seemingly, directly using the Dirac delta function is not an ideal choice for gradient descent- based learning. To demonstrate this problem, let L denote the task loss objective, we can compute the gradient of weights parameters using the spatial-temporal backpropagation (STBP):
0L OW =
ØL Oyll) (t.ppe dc drli 0ul dcl!) dw
Σ
(6)
In above equation, au is the gradient of firing function and is 0 almost everywhere, as aforemen- tioned. As a consequence, the gradient descent (W-- f- W W1w either freezes weights or updates the weights to infinity.
(y
To mitigate this problem, prior work seeks alternative gradient (also known surrogate gradient) to circumvent this problem. A popular surrogate gradient may refer to the rectangular function proposed in [4], given by
(1) (t..pe sign (1x(0)pp
Ven <
2 ");
a
where a is a hyper-parameter and usually set to 1. When V/A = 0.5, = 1, the gradient is simplified to if 0 I u (tr < 1 otherwise O. This is known as the Straight-Through Estimator in Bengio er al. [22] and is broadly applied in binary networks [31]. Another family of surrogate gradient may refer to triangular gradients [32], given by
0y
(t).pr
-1)
(r. y max 0,
Vea
This surrogate gradient is computed based on the distance between the potential and threshold. Despite various surrogate gradients achieve acceptable accuracy, there lacks a theoretical justification for its reason and effectiveness.
4.2 Rethinking Gradient- Dessent in Spiking Neural Networks
Gradient-descent (GD) algorithm is a first-order iterative optimization method. Theoretically, for each iteration, GD aims to find the steepest direction for the descent that minimizes the task loss:
min L(W AW) 2 min L(W) + AW OL Isse
OW
Note that the right-hand side of the minimization is approximated by the first-order Taylor expansion. Given that L(W) is constant at the current iteration, therefore only have to find the right A W that minimizes the second term. Since the 2nd term is a linear in AW, can be made as negative as we like by taking W large. Therefore the minimum is AW = - WTT
at. TwL (where VwL sw) )
Note that the above condition is only sufficient for E -> to correctly approximate the loss with Ww
Taylor expansion. However, - 0 means AW -6 TVV T. + but in practice GD updates the weights by small step (determined by learning rate), which, indicates a mismatch between the practical implementation and the theoretical assumption. In other words, gradient only inspects

---

(a) ANN
(b) SNN (Rectangular a=
1.000
Method Hyper-Param
Acc.
Cosine: Simitarity
0.599
Cosine Similaritt
2.7
86
SG rect. = 1
94.87
SG rect. a = 0.5
94.19
0.99
FDG = 0.5
94.22
94-
FDG E =022
95.14
0.99
43-
FDG
E 0.11
95,36
2 0.00
1.0000
FDG == 0.01
94.82
Epoch
Epoch
FDG
E 0.001
91.85
Figure 2 & Table Left: The cosine similarity between the gradients used for update and the FDG. In (a) we show ANN gradients and in (b) we show SNN rectangular surrogate gradients. Right: Training SNN with finite difference gradient, given different E.
the change rale of loss around an infinitely small neighborhood, however, we are interested in the change rate around a tiny (bul not infinitely small) area. In the case of SNN, the gradient will always be OF, yet for a tiny neighborhood the loss function will change, leading non-zero gradients. To penetrate the non ce--ifferentiabiiity of the firing function, we adopt technique called finite difference. estimates the gradient by evaluating the loss objective near the coordinate:
L. L - [Lw + Ee1) - L[w1 =e1) L(w, + te2) L(w - -EEx)
(10)
28
28
where w is the flattened vector of W and e; refers to standard basis vector with 1 at i-th index. e is some small constant, e.g. 0.001. We call V. L as finite difference gradient (FDG). By the definition of gradient, we know FDG becomes the true gradient when e + O. FDG can be a useful tool in GD-based learning, however, evaluating finite difference for each element in the weight vector is tedious, which greatly slows down the training process. Nevertheless, FDG still offers choice to evaluate the gradient mismatch problem quantitatively.
In order to illustrate this, we construct toy problem: a simple 2-layer CNN on the MNIST dataset. The first layer is 3x3 convolutional layer (output channel number is 6) with ReLU activation (or LIF spike activation) and average pooling for nown-sampling. Then, we directly add fully connected layer to predict the classification results. We train the toy model on MNIST for 5 epochs, with batch size 128 and learning rate 0.01 decayed to O. In effort to compute the FDG of the first layer, we have to perform 2 x x 6 x x 3- = 108 times forward pass 3, contrast, one has to perform 22 million times forward passes a ResNet-18 18 [30] to compute the FDG of all parameters.
Experiment I: FDG in ANN. We first train an ANN on MNIST and compare the disparity between real gradients and FDG. We use cosine similarity (i.c. cos sisssim(x, x22 = T=_||a|<=a[[>> to determine the angle between these two gradient vectors. For each epoch, we compute the mean cosine similarity in the first 100 training iterations. The results are demonstrated in Fig. 2(a). The FDG and real gradients consistently have extremely similar direction (> 0.99 cos sim) with varying from 0.1 to 0.001. In conclusion, we show that (1) FDG is a valid signal and (2) the real gradients can effectively find the change rate within small neighborhood in ANN optimization.
2,
Experiment I: FDG in SNN. Then, we compare the cosine similarity between the surrogate gradients in SNN and the FDG. In Fig. 2(b), we can tell that surrogate gradients share similar gradient direction with FDG when e = 0.1 (i.e. more than 0.8 cosine similarity). However, decreasing : will lead to lower cosine similarity. In the case of = 0.001, the angle between FDG and the surrogate gradients is nearly 90 degrees, (for comparison, high-dimensional space, two arbitrary random Gaussian vectors very likely to be orthogonal). This observation concludes that surrogate gradient can predict the change rate of loss in a relatively larger neighborhood, which justifies the soundness of surrogate gradients. However, SG fails to predict the FDG in small step size e. To this end, we need to figure out whether a large step size is better or a small step size is better.
Experiment III: Training SNN with FDG. In SNN optimization, a natural question arises: Whar e is optimal for gradient descent? First, the e has to be small enough to compute the non-linearity
2 AAssmmngg the potential is not equivalent to Vin- AAcorddnn to Eq. (10), for each element evaluate the loss twice, the first layer has 54 weight elements.
5

---

of the loss surface, meanwhile, it has to be large enough to ignore the noise from the rough step function. For this reason, we explore the optimal by applying FDG in SNN optimization. The results are demonstrated in the Table 1. Note that all methods are trained on MNIST for only 10 epochs here. We show that the rectangular surrogate gradient with a = 1 reaches 94.9% accuracy. For FDG optimization, we evaluate & ranging from 0.001 to 0.5, and we find 0.1 is the best choice for optimizing SNN. The results confirm our assumption that e has to be neither too big nor too small. It also tells using the instant change rate (very small s) will lead to undesired performance degradation.
4.3 Differentiable Spike
In this section, we introduce the Dspike function, a novel surrogate gradient estimator, which can cover large range of choices for SG. We choose the hyperbolic tangent function tanh(x) as our basic building function for gradient estimation:
e--e
a {o-
if > 1 if < < i < 0
-
Dspike(z)
tanh(b(x c)) + d
(1I)
When z < 0 or > 1. we choose to clip as did in other practice. The vanilla tanh function maps (-00, +00) to (-1, 1). thus we have four parameters to control its shape to alter the function mapping: from [-1, 1] to [1, We first use c Vas 0.5 to set the symmetry center to (0.5, 0.5). Parameter b(b > 0) is called the temperature factor, it controls the shape of the Dspike function. Parameter a and d can then be determined by setting (spike(O) O, Dspike(1) 1. As result, we
can rewrite the function as:
Dspike( b) = tanh (b(bb (6(x 0.5)) + tanh(t/2)
if S x 1.
(12)
2 2(tann(*/2))
Dspike function has the capability to adjust the shape of the function, leading different surrogate gradient estimations. To demonstrate that, we have the following lemma:
Lemma 4.1. By changing the temperature b, the shape of Dspike function can be altered. Specifically, we have lim Dspike(x) - x and lim Dspike(x) - sign(r 0.5).
Proof. We first prove the case b- + 04. In this case, applying Taylor expansion to the tanh we have:
2(2n) (2(2m) 1) B2n (hx))
tanh(bar) R ba
(13)
3
(2n)!
where Bn is the n-th Bernoulli number. When b + 0 we can safely ignore higher-order terms and only keep br. Thus, we can rewrite Eq. (12) to Dspike(x, b) = bx/b x. Then, we prove the case b + +oo. Since lim tanh(r) + 1, we have:
petanb(+Foc)
if > 0.5
2stanh
lim Dspiko(r, b) + 400000
tanh( (+00)
0.5
if = 0.5
(14)
2etanh
if < 0.5
2etanh
With this temperature factor, our algorithm can simulate a wide range of surrogate gradients with different smoothness. We visualize the transition Fig. from both forward perspective and back- ward perspective. As aforementioned, surrogate gradient can mimic the FDG in a relatively large neighborhood. Our intuition is that the optimal surrogate gradient can be found by choosing tempera- ture wisely. Thus task becomes to optimize temperature and network parameters simultancously during training. As did in former toy experiments (Sec. 4.2), we verify the cosine similarity between our Dspike gradients and the FDG at the beginning of each training epoch and update the temperature to better match the FDG. This algorithm is detailed in Algo. 1.
As abovementioned, evaluating FDG could be time-consuming due to the two times evaluation of the final loss for each weight element. Therefore, we choose to only compute the FDG in the first layer for two reasons. First, the first layer has much fewer weights. In ResNet-18 (for CIFARIO 10 dataset), the first layer only has 3 x x 3 x 64 = 576 elements. Second, the gradient propagation may accumulate the error in the former layers, which effectively measures the gradient mismatch.

---

Forward
Depiike Backward
b-1
2.5
b-4
0.8
b-10
dy/du
1.3
y(t)
0.4
1.0
0.0
0.8
0.0
0.2
0.1
0.6
0.8
1.0
0.0
0.2
0.1
0.6
6.8
1.0
(t), pro
0 (t), pro
Figure 3: The forward plot (left) as well as the backward plot (right) of Dspike function. By changing
temperature parameter we can manipulate the gradient estimation in SNN training.
Algorithm 1: Training SNN with Dspike gradient
Input: SNN to be trained; Training dataset, total training epoch E, total training iteration in one epoch I, FDG step size e, temperature update step size Al, initialized temperature b".
for all 1, 2, .... E-th epoch do
Collect input data and labels, computing the FDG (V of the first layer using e;
for all b in {b--1, ba-1 1--1_.. + Ab, b--1 Ab} do
[ Compute Dsipke surrogate gradient Vo,w and cos cos_sim e L., V.,w
Find the optimal temperature with the highest cosine similarity and update it to b%; for all i = 2, ......... Litteration do
Get training data and labels, compute Dspike surrogate gradient using optimal b;
Descend loss function and update weights;
return trained SNN.
4.4 Time Inheritance Training
We also introduce a novel training pipeline that can effectively reduce the time for training multiple SNNs. Unlike ANN-SNN conversion, direct training SNN is performed with fixed time step T. As a result, we must learn a new SNN from scratch if we decide to use another different T. We propose Time Inheritance Training (TTT) pipeline ameliorate this problem. First, we train SNN to converge with high 7, e.g. 20. Then, we use this well-trained model to initialize an SNN with lower T, say T 10. Consequently, the initialized model has exactly the same output the former model in the first 10 time steps, enabling better initialization and faster convergence. In other words, TIT progressively reduces the time step finetuning from higher time step SNN first. This idea resembles the curriculum learning proposed in Bengio et al. [33], which suggests learning the easy task first. Armed with TIT, we can reduce the training time of subsequent models by 75%, while sharing the same task performances.
5 Experiments
In this section, we report experimental results for our Dspike algorithm CIFAR [34] and ImageNet dataset [35]- For static image dataset, we encode the images to spike using the first layer in the SNN, as adopted in recent works [27]. For the neuromorphic image dataset, the images are already converted to 0/1 spike format. We also replace the max-pooling layer with the vverage-pooling layer in the network architecture. The detailed implementation is described in each subsection.
5.1 Network Architecture Choice
For CIFAR series experiments. We mainly adopt the ResNet-family architectures [30]. Unlike traditional ResNet structure [30], we use modificd version with stem layers and avg-downsample
7

---

Table 2: Training Spiking Neural Networks CIFARIO and COF
Dataset
Method
Type
Architecture
Time Step
Accuracy
SpikeNorm [10]
ANN-SNN conversion
VGG-16
2500
91.55
Hybrid-Train [40]
Hybrid
VGG-16
200
92.02
Spike-based BP [41]
SNN training
ResNet-11
100
90.95
STBP [4]
SNN training
5Conv, 2FC
12
90.53
TSSL-BP [42]
SNN training
5Conv, 2FC
91.41
CIFARIO
Diet-SNN [27]
SNN training
VGG-16
92.70 5
93.16
STBP-dBNN [26]
SNN training
ResNet-19
4
92.92
92.34 2
6
94.25 0.007
Dspike
SNN training
ResNet-18
4
93.66110.00
93.13--0.07 2
BinarySNN [43]
BNN-SNI conversion
VGG-15
62
63.20
Dict-SNN [27]
SNN training
VGG-16
69.67 5
CIFARI00
6
74.24110.10
Dspike
SNN training
ResNet-18
10.14
7..12:00.12 2
Rollout [44] CIFARIO-DVS STBP-ddNN [26]
Streaming roll out ANN
DenseNet
10
66.8
SNN training
ResNet-19
10
67.8
Dspike
SNN training
ResNet-18
10
7..4
block, as introduced in [36]. Our ResNet-18 architectures contains 4 stages with 2 basic residual block in each stage. Our initial channel is set to 64 and doubled in each stage. In comparison with [26], they use ResNet-19 19 with stages, with blocks in the first 2 stages and blocks in the last stage. However, they choose 128 channel and they has larger receptive field than our model. In general, their ResNet- 19 has ~ times more operations than ResNet-18. For ImageNet, VGG-16 as used in [27] and the same ResNet-34 as used in [26].
the same
5.2 CIFAR
The CIFARIO (100) dataset contains 50K training and 10K test images with 32x32 pixels. We use AutoAugment [37] and Cutout [38] for data augmentation. We use ResNet-18 18 architecture for running experiments. We train the corresponding ANN first and use it to initialize the first SNN with (T = 6), then we use Time Inheritance Training to gradually reduce the time step to 2. For FDG computation, we use the best practice ε = 0.1 in the toy experiments. The temperature is initialized to 1, and we set the update step size to Ab 0.2. We adopt SGD optimizer with 0.9 momentum. In the first round of TIT, we train the model for 300 epochs with a learning rate of 0.01 cosine decayed [39] to O. In the rest rounds of TIT. we only train the model for 50 epochs with a learning rate of 0.004. Weight decay (L.2 regularization) is set to 0.0001. We run the model with 4 GTX 1080Tis. For each run, we report the mean accuracy as well as the standard deviation with trials.
We summarize the overall results in Table 2. On CIFARIO, our work can outperform state-of-the-as by 1.09% accuracy when T = 6. It is also worth noting that the ResNet-19 used in [26] has 2x channel numbers and a larger feature-map, translating to 10x computation cost. Nevertheless, our ResNet- 18 with Dspike still achieves higher accuracy. Notably, our ResNet-18 (7 = 2, 4) is trained with TIT for only 50 epochs, yet still reaches a new state-of-the-arr. We also conduct an ablation study: training a 2 time step spiking ResNet- without TIT for 300 epochs. The accuracy is only 93, 13%, which is subpar with the TIT method. We also verify Dspike on CIFARI00, with the same training pipeline here and we find our model can achieve much better performances than Dict-SNN [27].
To further demonstrate the effectiveness of our method, we apply the Dspike function on the event- stream dataset CIFAR 10-DVS [45]. CIFAR10-DY; contains 10k 128x 128 images converted from CIFARIO 10. We split the dataset into 9k training images and 1k test images following the prior practice [26]. For data pro-psocs and augmentation, we first resize the training image frames to 48x448 48, then apply random horizontal flip. Additionally, we apply random roll within pixels. The test images are directly resized to 48 x48.. Training ayper-parametert are kept the same with
8

---

Method
Model
Acc.
#Add.
#Mult.
Energy
ANN [26]
Res-19
N/A
2285M
2285M
12.6.
ANN (ours)
Res-18
95.46
187M
187M
1.03J
SNN (27](T=5)
Res-20 92.70
142M
8.80M
168m
SNN [T6](T-2)
Res-19 92.34
360M
6.8M
355mJ
SNN (ours. T=2) Res-18 93.13
71.2M
3.52M
80.3m.J
Figure 4 & Table 3: Left: Activation sparsity for each LIF layer. The x-axis denotes the layer index while the y-axis denotes the activation sparsity. Right: The computation operations count and the energy consumption comparison between ANN and SNN for each run, we select SNN from existing literature.
Table 4: Training Spiking Neural Networks ImageNet.
Method
Type
Architecture
Time Step Accuracy
SpikeNorm [10]
ANN-SNN conversion
ResNet-34
2500
65.47
Hybrid-Train [40]
Hybrid
ResNet-34
250
61.48
Hybrid-Trane Train [41]
Hybrid
VGG-16
250
65.19
STBP-tdBN [4]
SNN training
ResNet-34
6
63.72
Diet-SNN [27]
SNN training
VGG-16
5
69.00
Dspike
SNN training
ResNet-34
6
68.19
Dspike
SNN training
VGG-16
5
71.24
the CIFAR10 experiments. Our proposed method can achieve 75.4% averaged top-1 accuracy
CIFARIO-DVS, with 7.6% absolute accuracy improvement compared with the existing state of the art.
5.3 ImageNet
In this section verify algorithm the ImageNet dataset. ImageNet dataset has more than 1250k training images and 50k test images. We use the standard pre-processing and augmentation for training data [30]. The test data is directly centered cropped to 224x 224. We train ResNet-34 and VGG-16 on ImageNet using the same architecture in the CIFAR experiments. Each network is trained for 100 epochs with SGD optimizer with 0.9 momentum. The learning rate is set 0.04 and weight decay is set to 0.00004. We train each model on 32 GTX 1080Tis with batch size of 10 per GPU. Following [26], avoid the biased statistics, we synchronize the batch mean and variance across each device. Results are demonstrated in Table 4, Dspike training obtain 4.5% improvement compared STBP-tdBN [26] and 2.2% improvement compared Diet-SNN [27].
5.4 Sparsity, Efficiency, and Energy Cost
In this section, we measure the sparsity and efficient computation well as the energy consumption of SNN. For ANN, the dot-product performs cultiply-accummlatt (MAC) operation, which is composed of addition and multiplication. In the case of SNN, its event-based operation and binary activation (except for the first rate-encoding layer) inference. Only one addition is needed if spike activated. In Fig. 4 we visualize the mean sparsity of activation of our spiking ResNet-18 18 on the whole CIFARIO test set. We can see that each LIF layer has no more than 32% sparsity rate, which greatly saves energy. We also measure the operation number in Table 3. The addition count is calculated by s T A, where is the mean sparsity. T is the time step and A is the addition number in ANN. For multiplication in SNN, we set the FLOPs of the first layer and scale it by T. Our spiking ResNet- 18 with 2 time-step has the lowest operation need for the inference. Then, we measure the energy consumption. Following [27], the energy is measured in 45nm CMOS technology. The MAC operation in ANN costs 4.6pJ energy and the accumulation in SNN costs 0.9pJ energy. Based on this, we compute the energy cost and put in the table. Our network only costs 8 for single forward, and consumes 12.8 x lower energy compared with ANN.
enable multiplication-Frit

---

5.5
Ablation Study
Our experiments use the same for dataset and network architectures. this section, we use different value for ε and compare their results. The re- sults is included in Table Generally, we find
Table 5: Accuracy comparison given different
low temperature will decrease the performance, as we explained Sec. 4.2, the desired opti- mization algorithm should ensure each update will decrease the loss function. Thus, the instan- tancous derivative in SNN cannot measure such curvature information since the instantaneous derivative is always O. Therefore, we find a large epsilon can help improve the performance. Indeed, the optimal requires tuning, but our empirical results show that 0.1 can have good results.
Dataset
0.01
5 0.05
E
0.1
E 0.2
CIFARIO CIFAR100
91.78
93.11
94.07 74.31
93.84
70.97
72.19
74.57
5.6 Visualization
In this section, we provide further experimental visualization. In Fig. 5, we plot the temperature curve for CIFARIO-DVS experiments. The spiking ResNet-18 was trained for 300 epochs with cosine learning rate decay. We initialize to 1.0 and use 26=0.2 to optimize both temperature for gradient calculation and the weights parameters for network learning. For the initial 25 epochs, the temperature remains less-than-l. We think this is primarily due to the complex loss landscape at the starting stage of the training. During the 20 80 epoch, the temperature continues to be increasing. In this stage of learning the FDG desires to lear sharp, narrow gradient with low bias but high variance. We also find the accuracy curve becomes more unstable in this stage of the training, as consequence of the sharp gradient. After 80 epochs, the temperature becomes decreasing. Although the decreasing process not aggressive as the former increasing stage, the trend is still obvious. From 80-250, the temperature decreases from 4.0 to 1.6 with some fluctuations. Then, at the end of the training stage, the temperature again becomes increasing.
The intuition behind this tuning mechanism may remain unexplainable right now, however, we think it is the intrinsic nature of the gradient estimation in SNN. Through FDG, we are able optimize the gradient calculation along with the weights learning.
Temperature
3
Mm
2
Mr
M
www
50
100
150 Epoch
200
250
300
Figure The evolutution temperature during optimization.
6
Discussion
In this work, we inspect the training process of SNN when using the surrogate gradient. The proposed finite difference gradient and Dspike function can be assembled with other optimization methods as fundamental module of gradient estimation for training SNN. This suggests that the performance of SNN can be further increased when more sophisticated training approaches are integrated with the currently proposed methods in the future. Although the improvement on model performance is very significant, be that as it may, our work is still an extension to the conventional SG. How to efficiently find the best FDG gradient to directly deal with the non-linearity of spiking function still remains problem. addition, we only consider the LIF neurons for the analyses here, the efficiency of the proposed approaches other types of spiking neurons still needs examination in practice.
10

---

References
[1] Filipp Akopyan, Jun Sawada, Andrew Cassidy, Rodrigo Alvarez-Icaza, John Arthur, Paul Merolla, Nabil Imam, Yutaka Nakamura, Pallab Datta, Gi-Joon Nam, al. Truenorth: Design and tool flow of 65 mw I million neuron programmable neurosynaptic chip. IEEE transactions on computer- aidduter-aiied design of integrated circuits and systems, 34(10):1537-11557 2015.
[2] Mike Davies, Narayan Srinivasa, Tsung-Han Lin, Gautham Chinya, Yongqiang Cao, Sri Harsha Choday, Georgios Dimou, Prasad Joshi, Nabil Imam, Shweta Jain, et al. Loihi: A neuromorphic manycore processor with on-chip learning. IEEE Micro, 38(1):82-99, 2018.
[3] Jing Pei, Lei Deng, Sen Song, Mingguo Zhao, Youhui Zhang, Shuang Wu, Guanrui Wang. Zhe Zou, Zhenzhi Wu, Wei He, et al. Towards artificial general intelligence with hybrid tianjic chip architecture. Nature, 572(7767):
2019.
[4] Yujie Wu, Lei Deng, Guoqi Li, Jun Zhu, Yuan Xie, and Luping Shi. Direct training for spiking neural networks: Faster, larger, better. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 1311-1318, 2019.
[5] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-peeformance deep learning library. arXiv preprint arXiv: 203 2019.
[6] James Bradbury, Roy Frostig, Peter Hawkins, Matthew James Johnson, Chris Leary, Dougal Maclaurin, George Necula, Adam Paszke, Jake VanderPlas, Skye Wanderman-Milhm and Qiao Zhang. JAX: composable transformations of Python+NumPy programs, 2018.
[7] Martín Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Geoffrey Irving, Michael Isard, et al. Tensorflow: A system for large-scale machine learning. In 12th [USENIX) symposium on operating systems design and implementation ((OSDI) 16), pages 665-283,77 2016.
[8] Emre O Neftci, Hesham Mostafa, and Friedemann Zenke. Surrogate gradient learning in spiking neural networks: Bringing the power of gradient-based optimization to spiking neural networks IFFE Signal Processing Magazine, 36(6):51 1-33, 2019.
[9] Bing Han, Gopalakrishnan Srinivasan, and Kaushik Roy. Rmp-snn: Residual membrane potential neuron for enabling deeper high-accuracy and low-latency spiking neural network. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages (3558-13566) 2020.
[10] Abhronil Sengupta, Yuting Ye, Robert Wang, Chiao Liu, and Kaushik Roy. Going deeper in spiking neural networks: Vgg and residual architectures. Frontiers Neuroscience, 13:95, 2019.
[11] Bodo Rueckauer, Iulia- Alexan-a Lungu, Yuhuang Hu, and Michael Pfeiffer. Theory and tools for the conversion of analog to spiking convolutional neural networks. arXiv preprint 1612:::61..04022 2016.
[12] Bodo Rueckauer, Iulia- Alexandra Lungu, Yuhuang Hu, Michael Pfeiffer, and Shih-Chii Liu. Conversion of continuous-valuse deep networks efficient event-driven networks for image classification. Frontiers in Neuroscience, 1 11:682, 2017.
[13] Bing Han and Kaushik Roy. Deep spiking neural network: Energy efficiency through time based coding. In Proc. IEEE Eur. Conf. Comput. (ECCV), pages 388-404, 2020.
[14] Shikuang Deng and Shi Gu. Optimal conversion of conventional artificial neural networks to spiking neural networks. arXiv preprint WrXiv:2103.0047: 2021.
[15] Yuhang Li, Shikuang Deng, Xin Dong, Ruihao Gong, and Shi Gu. A free lunch from ann: To- wards efficient, accurate spiking neural networks calibration. arXiv preprint a7X1:2106.00698,, 2021.
[16] Yuhang Li, Xin Dong, and Wei Wang. Additive powers-of-two quantization: An efficient non-uniform discretization for neural networks. arXiv preprint arXiv:1909.13144. 2019.
[17] Yuhang Li, Ruihao Gong, Xu Tan, Yang Yang, Peng Hu, Qi Zhang, Fengwei Yu, Wei Wang, and Shi Gu. Brecq: Pushing the limit of post-training quantization by block reconstruction. arXiv preprint 21X.v:2102.05426 2021.

---

[18] Yuhang Li, Mingzhu Shen, Jian Ma, Yan Ren, Mingxin Zhao, Qi Zhang, Ruihao Gong, Fengwei Yu, and Junjie Yan. MQBench: Towards reproducible and deployable model quantization benchmark. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 1), 2021.
[19] Hanting Chen, Yunhe Wang, Chunjing Xu, Boxin Shi, Chao Xu, Qi Tian, and Chang Xu. Addernet: Do we really need multiplications in deep learning? In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1468-1477, 2020.
[20] Arnon Amir, Brian Taba, David Berg, Timothy Melano, Jeffrey McKinstry, Carmelo Di Nolfo, Tapan Nayak, Alexander Andreopoulos, Guillaume Garreau, Marcela Mendoza, et al. A low power, fully event-based gesture recognition system. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 7243-7255 2017.
[21]
Johanni Brea, Walter Senn, and Jean-Pascer Pfister. Matching recall and storage in sequence learning with spiking neural networks. Journal of neuroscience, 33(23)99565 9575, 2013.
[22] Yoshua Bengio, Nicholas Léonard, and Aaron Courville. Estimating or propagating gradients through stochastic neurons for conditional computation. arXiv preprini arXiv: 1308.3432, 2013.
[23] Jun Haeng Lee, Tobi Delbruck, and Michael Pfeiffer. Training deep spiking neural networks using backpropagation. Frontiers in Neuroscience, 10:508, 2016.
[24] Emre O Neftci, Charles Augustine, Somnath Paul, and Georgios Detorakis. Event-driven random back-propagation: Enabling neuromorphic deep learning machines. Frontiers in Neuroscience, 11:324, 2017.
[25] Yujie Wu, Lei Deng, Guoqi Li, Jun Zhu, and Luping Shi. Spatio-temporal backpropagation for training high-performan
spiking neural networks. Frontiers in Neuroscience, 12:331, 2018.
[26] Hanle Zheng, Yujie Wu, Lei Deng, Yifan Hu, and Guoqi Li. Going deeper with directly-trained larger spiking neural networks. arXiv preprint arXiv:2011.053 1,05280, 2020.
[27] Nitin Rathi and Kaushik Roy. Diet-snn: Direct input encoding with leakage and threshold optimization in deep spiking neural networks. arXiv preprint arXiv:2008. 03658, 2020.
[28] Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In ICML, 2015.
[29] Shibani Santurkar, Dimitris Tsipras, Andrew Ilyas, and Aleksander Madry. How does batch normalization help optimization? arXiv preprint arXiv: / 1604, 2018.
[30] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, 2016.
[31] Matthieu Courbariaux, Yoshua Bengio, and Jean Pierre David. Binaryconnect: Training deep neural networks with binary weights during propagations. In NeurIPS, 2015.
[32] Guillaume Bellec, Darjan Salaj, Anand Subramoney, Robert I Legenstein, and Wolfgang Maass. Long short-term memory and learning-to- in networks of spiking neurons. arXiv preprint 103. 2018.
[33] Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston. Curriculum learning. In Proceedings of the 26th annual international conference on machine learning, pages 41-48 2009.
[34] Alex Krizhevsky, Vinod Nair, and Geoffrey Hinton. Cifar- (canadian institute for advanced research).
[35] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In CVPR, 2009.
[36] Tong He, Zhi Zhang, Hang Zhang, Zhongyue Zhang, Junyuan Xie, and Mu Li. Bag of tricks for image classification with convolutional neural networks. In Proceedings of IEEE/CVF Conference Computer Vision and Patlern Recognition, pages 558-567, 2019.
[37] Ekin D Cubuk, Barret Zoph, Dandelion Mane, Vijay Vasudevan, and Quoe V Le. Autoaugment: Learning augmentation policies from data. arXiv preprint arXiv: 1805.0951, 2018.
[38] Terrance DeVrses and Graham W Taylor. Improved regularization of convolutional neural networks with cutout. arXiv preprint 1708 2017.
12

---

[39] Ilya Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts. arXiv preprint arXiv: 03983, 2016.
[40] Nitin Rathi, Gopalakrishnan Srinivasan, Priyadarshini Panda, and Kaushik Roy. Enabling deep spiking neural networks with hybrid conversion and spike timing dependent backpropagation. arXiv preprint arXiv:2005. 000007 2020.
[41] Chankyu Lee, Syed Shakib Sarwar, Priyadarshini Panda, Gopalakrishnan Srinivasan, and Kaushik Roy. Enabling baske backpropagation for training deep neural network architec- tures. Frontiers Neuroscience, 14, 2020.
[42] Wenrui Zhang and Peng Li. Temporal spike sequence learning via backpropagation for deep spiking neural networks. arXiv preprint arXiv:2002.10065 2020.
[43] Sen Lu and Abhronil Sengupta. Exploring the connection between binary and spiking neural networks. Frontiers in Neuroscience, 14:535, 2020.
[44] Alexander Kugele, Thomas Pfeil, Michael Pfeiffer, and Elisabetta Chicca Efficient processing of spatio-temporal data streams with spiking neural networks. Frontiers in Neuroscience, 14:439, 2020.
[45] Hongmin Li, Hanchao Liu, Xiangyang Ji, Guoqi Li, and Luping Shi. Cifarl0-dvs. an event- stream dataset for object classification. Frontiers in Neuroscience, 11:309, 2017.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] Please see our discussion in
Sec. 6
(c) Did you discuss any potential negative societal impacts of your work? [No]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes]
(b) Did you include complete proofs of all theoretical results? [Yes] The proof is available in Sec. 4.3.
3. If you experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as a URL)? [Yes] Our code will be available in the supplemental.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] All implementations are described in the experiments section.
(c) Did you report error bars (e.g., with respect to the random seed after running exper- iments multiple times)? [Yes] We report the mean as well as the standard deviation accuracy in CIFAR experiments. For time consideration, we did not run them for ImageNet experiments, the results may be updated in the future.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] The computation resources description is provided in the experiment section.
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes] We use CI-
FAR10/100 [34], ImageNet [35], and CIFAR 10-DVS [45].
(b) Did you mention the license of the assets? [No] We adopt public datasets.
(c) Did you include any new assets either in the supplemental material or as URL? [No]
(d) Did you discuss whether and how consent was obtained from people whose data you're
using/curating? [No]
13

---

(c) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [No]
5. If you used crowdsoureing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
14


Reviews:
[{"review_id": "aeCUuv6J0oy", "reviewer": "Reviewer_RWyX", "summary": "Summary.\n\nThis paper attempts to facilitate the surrogate gradient descent in training SNNs. The proposed FDG quantitatively analyzes the gradients in the training procedure. And then, the authors propose a Dspike function, which is claimed as a proper searching function for the optimal surrogate gradients.  Numerical experiments demonstrate the superiority of the proposed models to several contenders in CIFAR and ImageNet data sets.\n\nContributions.\n\nI have to admit that it's hard to understand and follow the objective and soundness of this paper. At first glance at the Tables in Section 5, the proposed methods achieve a commendable result compared to other SNN models. However, I can't entirely agree with the author's claimed contributions, especially the first and second ones.\n\nAs claimed that \"through which (FDG) we are able to not only explain why it (surrogate gradient) works well but also indicate how we can improve its (gradient) adaptivity to data through training\", so what are the answers of why and how? In detail,\n\n[1] The derivation of FGD can be traced back to the directional derivative and difference in mathematics. It is reasonable to use Eq. (10) to approximate the instantaneous derivative at a certain point in ANNs, which have continuous and differentiable activations. The high-similarity curves in experiment $\\mathrm{I}$ verify my conjectures. It's a good point.\n\n[2] However, the finite difference technique is not suitable for estimating the derivative of holder-continuous and non-differentiable functions. There may be a large gap between the instantaneous and average derivatives. Thus, the similarity between FDG and surrogate gradients in SNNs cannot support the \"soundness of surrogate gradients\" in Line 175 due to the imprecision of FDG.\n\n[3] Further, how to conclude that \"This observation concludes that surrogate gradient can predict the change rate of loss in a relatively larger neighborhood\" between Lines 175-176? Doesn't dissimilarity mean invalidity? I hope the authors can clarify this point.\n\n[4] My conjecture in [2] can be verified by Figure 2. At the initial state that epoch = 0 or 1, parameters $\\mathbf{W}$ may be far away from the optimal solution. In this case, FDG is not similar to the surrogate gradient if employing small $\\epsilon$. As the setting in the experiment that \"weight decay (L2 regularization) is set to 0.0001\", the parameters may converge to the optimal local minima as weights decay to 0.0001. Thus, FDG still works, although $\\epsilon$ is small. So it may be a practical trick but not holds as the authors theoretically claimed, right?\n\nAbout the claim that \"optimal surrogate gradient can be found by choosing temperature wisely\" in Lines 210-211. In other words, the Dspike function is effective since optimal surrogate gradient can be found by choosing temperature wisely. How to evaluate this point? I would have appreciated more clarity on the experimental results not as omitted by \"As did in former toy experiments (Sec. 4.2), we verify the cosine similarity between our Dspike gradients and the FDG at the beginning of each training epoch and update the  temperature to better match the FDG.\"\n\nIn summary, I find the interests of this work and the proposed model may be effective in practice. However, this conclusion should be testified either theoretically or experimentally. Of course, this paper still needs improvement in these two aspects. So the current version is improper for acceptance.", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Clarity and Quality.\n\n[1] The formalization and theoretical derivation of this paper are clear.\n\n[2] The conclusion and proof of Lemma 4.1 are valid and correct. Unfortunately, this lemma only describes the properties of Dspike function. Can the function set of Dspike functions with learnable parameters contain the surrogate gradients obtained by the common-used approaches, such as Eqs. (7) or (8)?\n\n[3] What's the computational complexity of FDG? It is better to show the number of paras, computational overhead, and storage costs.\n\n[4] As shown in Table 2, adding TIT will reduce the performance of Dspike. TIT is not the contribution of this work; why do this here?\n\n[5] The experimental results in Table 2 are not promising. Taking the results in CIFAR-100 as an example, both architecture and Dspike functions are different. So how is the improved accuracy attributed to the power of the Dspike function with FDG? In my mind, ablation experiments are desired. Besides, the comparative models are so scant.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "As mentioned above.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3 hours"}, "extra_text": {}}, {"review_id": "X-QKp0sjMPY", "reviewer": "Reviewer_uSGb", "summary": "This paper studies the differentiable training problem of spiking neural network (SNN), which is a promising alternative to the widely used ANN. The authors first identify the problem of the current surrogate gradient of SNN and propose a new surrogate gradient estimator. I really like the author's research method of thinking about problems from experiments, but the result of the experiment is just a tuning experiment of some parameters, and I did not see the author's insight into the existing SNN training problems. Furthermore, the Spike method proposed by the author lacks clear motivation. The proposed Dspike method is just a differentiable piecewise function, which does not have much methodological contribution. The author exaggerated the experimental settings and did not present the experimental results of neuromorphic images. In summary, I think this article does not meet the standards of the NeurIPS conference.\n\nThe author’s reply addressed most of my concerns. I agree that AdderNet is another line of work with different purpose. I can see that the authors are very thoughtful in the model design, however it is not well-described in the main paper. I have raised my rating by 1.", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Main Comments:\n1.\tAdderNet [a] is also a multiplication-free neural network architecture. What is the advantage and disadvantages of Spiking Neural Network?\n2.\tHow did Time-dependent Batch Normalization (TdBN) relevant to this paper? What is the motivation for using time-dependent batch normalization? The authors argued that TdBN is designed to normalize spatial-temporal data (Line 103-104), Are there any experimental results that argument?\n3.\tWhat is the motivation of the Time Inheritance Training (TIT) proposed in Section 4.4? \n4.\tThe authors argue that they conduct experiments on both static and dynamic datasets (Line 51), however, the authors only reported results on static datasets (CIFAR and ImageNet Sec 5.1 and Sec 5.2), and I didn’t find any results related to dynamic dataset (including Supplementary materials ). Where are the results on the dynamic dataset? In Line 235, the authors mentioned a “neuromorphic image dataset”, is that the dynamic dataset? Where are the experimental results on that dataset?\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes, the authors addressed the limitations and potential improvements.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}, {"review_id": "AdKMnX6Tcw", "reviewer": "Reviewer_uEo9", "summary": "This paper proposes a new method called Differentiable Spike (Dspike) to compute the surrogate gradient for backpropagation of SNNs. In addition, an online optimization method is proposed to optimize the hyperparameter of Dspike. ", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength: I appreciate that this paper demonstrates results on more complicated datasets like CIFAR 100 and ImageNet.\n\nWeakness: The proposed method is still in the frame of surrogate gradient.\n\nPlease see the comments below for details.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "1. The main issue of this paper is novelty. Overall, this main idea of this paper is a surrogate function with the dynamic hyperparameter tuning method to enable wide range of smoothness. However, it is still in the framework of surrogate gradient method and I cannot view it as a significant improvement over previous works. This work reports a good results on several challenging datasets like CIFAR10-DVS, CIFAR100 and ImageNet. I appreciate the authors' efforts in the experiments and I think the performances reported in this paper are better than state-of-the-art results. However, in my opinion, this is more about engineering works for running experiments and hyperparameters tuning.\n\n2. For the FDG computation of the experiments, the authors set epsilon to 0.1 which is empirically coming from the toy experiments on MNIST. However, I think this parameter should depend on datasets and cannot find a universal optimal choice. Can the authors comment on why the epsilon is fixed in the experiments?\n\n3. In table 2, I hope the authors can compare the proposed method with existing works using the same network size or similar number of tunable parameters. Thus, we can understand the improvement brought by the proposed method.\n\n4. For section 5.3, I hope the author can analyze the complexity of the proposed method compared to the existing methods. For example, compared with the STBP method, how much improvement the proposed method can achieve and how much additional cost do the dynamical temperature tuning and time inheritance training method require? ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4 hrs"}, "extra_text": {}}, {"review_id": "8v5reaUNjy2", "reviewer": "Reviewer_Q9Mk", "summary": "The authors outline a principled approach--finite difference gradients (FDG)--for choosing a surrogate gradient for spiking neural networks. FDG is demonstrated to approximate gradient descent in classic rate-based ANNs very well, approximate classic rectangular surrogate gradient function descent in SNNs for medium neighborhoods and thus theoretically justify the method, and be useful in finding an optimal neighborhood in which to approximate a gradient for SNNs.  This last step is then used to optimize a continuously varying surrogate spike gradient estimator (Dspike), modeled as a tanh approximation with the gain as a free parameter. Using the chosen Dspike function (which depends on the time step), the authors then demonstrate the appreciable accuracy and energy efficiency improvements of the resultant SNN.", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The authors' approach is original (to the reviewer's knowledge), intuitive, and of significant interest. The approach and results are well motivated, well written, and understandable, and appear to represent an appreciable advance in the field of SNNs.  However, some of the comparisons that have been made leave some room for questions as to the specific improvements that Dspike provides. Indeed, some of the improvements may be greater than suggested. Addressing this would be helpful for readers to better assess the impact of the authors' work.\n\nIn particular, Table 2 shows that Dspike achieves greater accuracy than all other surveyed methods.  However, the comparisons vary in important categories, in particular, architecture. Dspike may well do better than suggested compared to STBT-tdBN for both CIFAR 10 and CIFAR 10-DVS, since the former method utilized ResNet-18 and the latter ResNet-19.  Meanwhile, the comparisons to BinarySNN and Diet-SNN are less clear since they use VGG-15 and VGG-16, respectively. It would be helpful if, in these comparisons, the comparisons involved only a change of method and not architecture as well. Note that Table 4 does indeed allow such comparisons, and the resultant improvements are potentially somewhat more modest.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes", "review_text": "", "extra_scores": {"ethical_concerns": "No", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors present a method to train spiking neural networks (SNNs) using an adaptive surrogate gradient method.
Training of SNNs is difficult due to the well-known problem of non-existing derivative of the spiking neuron output. This is usually tackled using a surrogate gradient method. However, the functional form of the surrogate gradient is quite arbitrary.

The authors first analyze the training behavior of SNNs. They propose to use a parameterized function for the surrogate gradient (sg) and estimate its best actual shape based on a comparison with a finite-difference gradient. Then they use this sg for parameter updates. They test their method on challenging data sets such as CIFAR100 and ImageNet and show excellent performance.

The paper is very well written and clear. The adaptive surrogate gradient is a novel contribution that has potential to impact future SNN training methods. The experimental results are compelling.

Weaknesses and potential improvements: 
The functional form of the adaptive surrogate gradient lacks motivation.
Reviewers have identified some unclear or over-stated statements. The authors should refine these statements accordingly. 


Decision: Accept (Poster)