Efficient Surrogate Gradients for Training Spiking Neural Networks
Authors: Hao Lin
Shikuang Deng
Shi Gu
Venue: Submitted to ICLR 2023 (2023)
Keywords: surragate gradient
spiking neural network
low extra overhead

Abstract: Spiking Neural Network (SNN) is widely regarded as one of the next-generation neural network infrastructures, yet it suffers from an inherent non-differentiable problem that makes the traditional backpropagation (BP) method infeasible. Surrogate gradients (SG), which are an approximation to the shape of the Dirac's $\delta$-function, can help alleviate this issue to some extent. To our knowledge, the majority of research, however, keep a fixed surrogate gradient for all layers, ignorant of the fact that there exists a trade-off between the approximation to the delta function and the effective domain of gradients under the given dataset, hence limiting the efficiency of surrogate gradients and impairing the overall model performance. To guide the shape optimization in applying surrogate gradients for training SNN, we propose an indicator $\chi$, which represents the proportion of parameters with non-zero gradients in backpropagation. Further we present a novel $\chi$-based training pipeline that adaptively makes trade-offs between the surrogate gradients' shapes and its effective domain, followed by a series of ablation experiments for verification. Our algorithm achieves 69.09\% accuracy on the ImageNet dataset using SEW-ResNet34 - a 2.05\% absolute improvement from baseline. Moreover, our method only requires extremely low external cost and can be simply integrated into the existing training procedure.

Under review as a conference paper at ICLR 2023
EFFICIENT SURROGATE GRADIENTS FOR TRAINING SPIKING NEURAL NETWORKS
Anonymous authors Paper under double-blind review
ABSTRACT
Spiking Neural Network (SNN) is widely regarded one of the next-generation neural network infrastructures, yet it suffers from an inherent non-differentiable problem that makes the traditional backpropagation (BP) method infeasible. Sur- rogate gradients (SG), which are approximation to the shape of the Dirac's o-function, can help alleviate this issue to some extent. To our knowledge, the majority of research, however, keep a fixed surrogate gradient for all layers, ig- norant of the fact that there exists a trade-off between the approximation to the delta function and the effective domain of gradients under the given dataset, hence limiting the efficiency of surrogate gradients and impairing the overall model perfor- mance. To guide the shape optimization in applying surrogate gradients for training SNN, we propose an indicator X. which represents the proportion of parameters with non-zero gradients in backpropagation. Further we present novel x x-based training pipeline that adaptively makes trade-offs between the surrogate gradients' shapes and its effective domain, followed by series of ablation experiments for verification. Our algorithm achieves 69.09% accuracy on the ImageNet dataset using EEW-ResNer34- a 2.05% absolute improvement from baseline. Moreover, our method only requires extremely low external cost and can be simply integrated into the existing training procedure.
INTRODUCTION
Spike Neural Networks (SNN) have gained increasing attention in recent years due to their biological rationale and potential energy efficiency as compared to the common real-value based Artificial Neural Networks (ANN). SNN communicates across layers by the addition of spiking signals. On the one hand, this spiking mechanism turns multiplicative operations to additive operations, increasing the inference procedure's 's efficiency. On the other hand, introduces an intrinsic issue of differentiability. which makes training SNNs more challenging. At present, the method for obtaining practical SNNs can be roughly divided into three categories: converting a pretrained ANN to SNN (Sengupta et al., 2019 Deng & Gu 2020 Li al.
2021), }. training with biological heuristics methods (Hao et al. 2020 Shrestha et al. 2017: Lee et al. 20188), and training with Bi-like methods (Wu et al., 2018| al. 2020 Li al. (021b]] Yang al. (2021) The converting method may not promote increased inference efficiency in practice since it requires a lengthy simulation period (high inference latency) to catch up to the accuracy of the source ANN (Sengupta al. 2019 Rueckauer al.) 2017). Although the biological heuristics technique requires just local information to change network parameters, is confined small datasets due its limitation in representing global information (Wu et al.) 2018 Shrestha et al. 2017). Compared to these two approaches, direct training with BP-like method is capable of handling complex models with a very short simulation duration attain
202 Bu et al.
adequate model performance (Zheng et al. 20200] Fang et al. 2021 Li et al. 2021b).
With the help of surrogate gradient, the SNN can directly trained through the BPTT algorithm on an ANN-based platform (Wu et al., 2018). Nonetheless, there is non-negligible performance disparity between directly trained SNN and ANN, particularly on large and complicated datasets (Deng et al.) 2020 Jin et al.) 2018). This is because training SNN with surrogate gradient can only obtain approximate gradients, and the final performance is highly affected by the surrogate gradient shape. A more suitable surrogate gradient shape usually results in better performing SNN (Neftci et al.) 2019). However, an appropriate surrogate gradient must strike a compromise between the approximation shape and the effective domain of gradients. So just altering the shape of the

---

Under review as a conference paper at ICLR 2023
surrogate gradient to be more similar the 5-function may result in the training failing due to gradient disappearance, as the gradients of most membrane potentials are extremely small. Additionally, the optimal surrogate gradient shapes for various layers may different and may change throughout the training process (Li et al., 202Tb). As result, using a fixed initial surrogate gradient shape (adequate effective domain) during the whole training phase will always have substantial gradient error, which affects the final training result.
The purpose of this work is to optimize the SNN training pipeline by adaptively altering the shape of surrogate gradient in order to control the effective domain for the surrogate gradients. We suggest an index X to denote the proportion of membrane potential with non-zero gradients in backpropagation and present technique to control the proportion of non-zero gradients (CPNG) in the network. The CPNG technique modifies the shape of surrogate gradients during network training, progressively approaching the 5-function while maintaining the index steady within an effective range to ensure training stability. Finally, each layer succeeds finding a surrogate gradient shape that makes a better balance between the approximation error to the d-function with the size of effective domain than the fixed-shape surrogate gradients. It's worth mentioning that our strategy only incurs minor additional costs during the training phase and has no effect the inference phase. We verify the compatibility of CPNG to the existing mainstream SNN infrastructures such as VGG (Simonyan Zisserman) 2014), ResNet (He al. (2066), and Sew-ResNet (Fang al., [02TT. In all reported comparative experiments, training with CPNG gives more accurate models than training with vanilla surrogate gradients.
Our main contributions can be summarized as follows:
We identify and investigate the impact of the shape of surrogate gradients on SNN training. Our finding characterizes special representative power for SNN that can be utilized improve its performance.
We propose a statistical indicator x for the domain efficiency of surrogate gradients and a x-based training method CPNG that adjusts the shape of surrogate gradients through the training process, driving the surrogate gradients close to the theoretical ò-function with ensured trainability on sufficiently large domains.
Our CPNG method improves classification accuracy on both static image datasets including CIFAR10, CIFAR 100 and ImageNet, as well as event-based image datasets such as CIFAR 10-DV. We achieve an accuracy of 69. 69.09% in the experiment that trains ImageNet on Sew-ResNer34.
2 RELATED WORK
There are two primary branches of training a high-performing deep spiking neural network, converting a pretrained artificial neural network to corresponding spiking neural network, and directly training a spiking neural network through BP-like method.
ANN-SNN Conversion ANN-SNN conversion takes advantage of the high performance of ANN
and converts the source ANN to the target SNN through normalizioniiiiiiin (Diehl al. 2015
2016) or threshold balancing (Sengupta et al. 2019). However, SNN forming this method requires a huge simulation length catch up with the souree ANN's ' performance. Numerous strategies have been proposed to shorten the simulation time, including robust threshold (Rueckauer et al. (016).
SPIKE-NORM (Sengupta et al., 2019), and RMP (Han et al.) 2020). A work (Deng & Gu 2020)
examines the conversion error theoretically, decomposes it layer by layer, and offers threshold ReLU
and shift bias procedures to decrease the error. Based on it, Li et al. (Li et al.) 2021a) divide the conversion error into clip error and floor error and design adaptive threshold, bias correction, potential correction, and weight calibration to dramatically decrease the required simulation length. A recent work (Bu et al. 2021) further proposes unevenness error, trains ANN with novel activation function and reduce simulation length.
BP-like Method HMM2-BP (Jin et al. 2018) enables SNN to adjust the spike sequence rather than just the spike certain moment. TSSL-BP (Zhang & Li 2020) decomposes the backpropagation error into inter and intra interactions, calculating the derivatives only at the spiking moment. NA algorithm (Yang et al. 2021), which calculates the gradient of the non-differentiable part through
2

---

Under review as a conference paper at ICLR 2023
finite difference. The surrogate gradient based BP algorithm uses differentiable functions instead of the Dirac's d-function to mitigate the non-differentiation problem in the process of SNN training. Different from ANNs, SNNs naturally have time attribute. Therefore, the existing studies consider both temporal and spatial information in BP procedure (Wu et al. 2018). Normalization methods crucial in SNN training, which help speed up the network's convergence and prevent gradient disappearance or explosion. For this purpose, SNN-friendly normalization algorithms such as NeuNorm (Wu al. 2019) and threshold-dependent batch normalization (tdBN) (Zheng al. 2020) have been developed. The majority of studies have employed fixed-shape surrogate gradients, and some work has preliminarily explored the performance of surrogate gradients with different shapes (Wu et al.) 2018 Nefici et al. 2019: Bellec et al.] 2018). A recent work looked into the shape-changing surrogate gradient (Li et al. a Ib), proposed Dspike as surrogate gradient, and used the finite difference guide the change of Dspike's shape, significantly improving the performance of SNN. In order to train a very deeper SNN, Gew-ResNet (Fang et al. (022I) structure is proposed, which enables SNN training even a 152-layer network.
PRELIMINARY
Through out the paper, we use bold letters to denote matrices and vectors, superscripts to identify specific layers, subscripts to denote specific neurons, and indexes to identify specific moments.
Leaky Integrate-and-Fing Model. We use the Leaky Integrate-and-Fire (LIF) module for spiking neurons, Formally, given the pre-synaptic input (denoted by c 1)) of the 7th neuron in the 1th layer at time stept + 1, we can model the iterative process in LIF as
N(-1
1 + 1| Σ
w (1-1)
4 + Tu [1[ 8,
-0 (t) + 41 [t +1++,, 1],
It H(u: + -v).
(2)
Here, (1 is the number of neurons in the (1 1)th layer, 8) 0 is the output spike of the jth neuron in the (1- 1)th layer at time t 1, w 1 is the weight between jth neuron in - 1)th layer and 7thh neuron in 1hh layer, u. ( [t] is the membrane potential of the jth neuron in the 1th layer time i, r is the membrane potential attenuation factor, H--) is the step function, and v is the activation threshold. When the membrane potential of neuron exceeds the activation threshold, spike is released and the membrane potential of the current neuron is set to zero
Surrogate Gradient Function. There are various surrogate gradient shapes adopted by previous work (Wu al., 2018 Neftei et al. 2019). In this work, used triangle-like function, rectangular- like function and arctan-like function to verify the effectiveness of CPNG. These functions are described below:
Stiangte (x) = B(1 - B|z v1) if x- vl < 1/8
8(1-
(3)
otherwise
Frectangular (I) = B if Ja - v| < 1/(33)
Sarctan (x) 1 (T v))
(4)
otherwise
where 3 represents the maximum gradient value of current surrogate gradient function. Notably, the surrogate gradient satisfy 1t [(x) 1, which is also the property of the 5-function.
Loss Function. In experiments, we use cross-entropy-loss (Lor) as the loss function. The formula given as
T out
w(L) s(C-1) Loss (oe label),
(5)
T t=1
3

---

Under review as a conference paper at ICLR 2023
where T represents the simulation time, L represents the last layer of the SNN, w represents the weight matrix of the 1 layer, and S ((-11) represents the output spike vector of the (T. 1)th layer. Consistent with (Wu et al. 2019 He et al. 2016), our final output layer is a voting layer devoid of any LIF model.
METHOD
4.1 SHAPE PARAMETERS AND EFFECTIVE DOMAIN INDICATOR
Shape Parameters. Intuitively, increasing the shape parameter 3 of surrogate gradients would drive it closer to the 8-function (Fig. riihht).. One might expect to adopt very high /3 to obtain SNN for a good performance. We first examine whether this intuitive approach possible. We trained VGG16-structuree SNN (replace ReLU with LIF, and use average pooling.) CIFAR 100 using triangle-like surrogate gradient with 3 set from 0.25 to 2 respectively. As shown Fig. Ieftt, the test accuracy increases when 33 varies from 0.25 to 1.0 but remains at 1.0% when ,3 is set to 1.5 and 2.0, indicating that properly increasing 3 may benefit the training but arbitrarily increasing will drive the training collapse. According to another viewpoint (Zenke & Vogels) [02T Herranz-Celotti & Rouat 2022). when the beta is greater than 1, the gradient explosion on the deep SNN will occur, resulting in training failure; thus, some of the most recent surrogate gradients fix the maximum value of surrogate gradients at (Suctake et al.) 2022] Zenke & Ganguli| 2018). But the narrow effective interval of the surrogate gradients the primary culprit in our experiments. We detected the maximum gradient absolute values for SNN training at the first 100 batches when 3 = ((10000) and / 1.5(1.000), respectively. These results illustrate that when /3 is 1.5, the gradient explosion does not always occur in the network.
beta-0.5
beta=1.0
bet2-2.0
Tes: accuraty
diric
beta
1.5
b.5
beta-1.0
beta-115
12
50
100
150
200
250
300
-1.0
-05
1.0
2.5
3.0
Epoch
Membrone
Figure I: Leftt: Test accuracy of different 3 when threshold is 1.0. Right: 8-function and triangle-like surrogate gradient when the threshold is 1.0.
In fact, these results unveil that efficient training of SNNs requires not only the approximation to the 8- 5-function but also the insurance for the surrogate gradients work. Thus it is necessary employ dynamic shape-chame changinhanging strategy rather than using a fixed-shape surrogate gradient. This issue is also covered by (Li et al. 2021b) as well and was owed to the lack of adaption the dataset variation.
Effective Domain Indicator. As aforementioned, if we unrestrictedly increase the 3 in order to get the surrogate gradient closer the ò-function, the training curve will remain flat without any update. This is caused by the fact that the majority of the membrane potential remains outside the effective region. On the other hand, if 3 too small, it will also lead suboptimal training outcome due to the presence of a substantial gradient error (Fig. [eeft)) between the adopted surrogate gradients and the d-function. Thus a proper surrogate gradients should maintain an optimal balance between the domain effectiveness and the å ó-function approximation.
To quantitatively guide the choice of 3, we need statistical indicator to denote the percentage of membrane potentials that fall into the domain of surrogate gradients. As illustrated in Fig. 1 the distribution of membrane potentials on each layer takes the normal shape. Thus, for the simplicity of calculation, we regard the membrane potential distribution of all neurons within the same layer Gaussian one. By calculating the mean μ and the standard deviation o of the membrane potential before this layer releases spikes, we can obtain the proportion of the neuron with non-zero gradient
4

---

Under review as a conference paper at ICLR 2023
during certain iteration (the area between the red lines in Fig. 2 rrihht)). For given 3, the effective gradient domain triangle-like surrogate gradient is 1/3, 1/5]. we can obtain the definite integral of the current normal distribution in this effective gradient domain, which is the Effective Domain Indicator X-
1/B
1
2a 2a2 dx.
X
(6)
1-11/3
2
For each layer, we record the membrane potential of all neurons in every time step (a tensor shaped like batchsize- sese--y---ttteee this indicator X. baanesttp-b--cchneel------ Sahannels-by-H-by by) and calculate the mean and variance. Based on then effectively determine what extent that there are enough membrane potentials located within the effective range of surrogate gradients to make the training progress.
tune the while ensuring
Vth. 1/beta
1.0
Narro Nert
5-2
5.00
0:55
+ 18ol
Monsiooe
Figure 2: Left: Membrane distribution of each layer in experiment training ResNet19 on CIFARIO. Right: When the threshold 1.0 and 3 1.0, the proportion of neurons with non-zero gradient.
4.2 CPNG METHOD
In this section, will cover how combiningly optimize 3 and maximize the effectiveness of surrogate gradients. To train the network successfully, there must be sufficient membrane potential values in the effective domain of the surrogate gradient, i.e., X must be large enough. The most extreme case is 8 + O, which gives + 1. Obviously, this is not an optimal solution as it introduces substantial error for the gradients. In order determine the choice
for given X. we need to investigate the gradient form dCeg see how the 3 affects the error back-propagation.
ac ast as
When = 1, we can directively get
1,2, , according to Eqn.
When = 0, 1, L 2, we can first derive
-ru [ and
0u. 7(1 - (()(t)
as!
Ou! [
from Eqn. and further have
aCen (0 0ech1e
OCCE [
t= T
Du
(7)
acos 8g
8Cog
T(1 - -(() t= 1,2, "= T-
8
Sn [+1]
act)
OLCE [s(1
Usch act
t=T T
(8)
OCCE Ju0
(-7u'(0)
t= 1,2,
as
by Eqn.
Here, u represents the membrane potentials vector of the 1th layer, ag9 [
ac is the
gradient directly obtained from previous layer. The complete dCez needs consider the dependence
(xt
between the current moment spike and the membrane potential at the next moment (Eqn. The d5 is 8Ccg can be obtained iteratively from the gradients from its succeeding layer, and 2n 8g bu(0]t]
term 5m [[+1
given by surrogate gradients. We conclude that each item of the 8CCE contains
while
0u(
(g
effective interval of surrogate gradient (x) determines
which in turn affects the calculation g

---

Under review as a conference paper at ICLR 2023
Algorithm 1 Control the Proportion of Non-zero Gradient
Input: SNN model with L layer, current iterator epoch
Xlimit, and vector Xrecorder' store each layer's ' smallest X Output: Each layer's s surrogate gradient parameter | if e = 0 then
for = 1, 2. do
calculate current x by 6qn layer-l and store at Xrecarder [1] end for
else
for 1 = 1, L. do
calculate current Xeur by Eqn
layer-l
if Xreconder [2] Xtimii then
Xrecorder [1] Xlimit
else if Xaur <Xresonderl] then
Xiecerder [1] = Xeur
end if
Xmin Xreconder
if Xmin F Xeur then
use Xmin to update using binary search method.
end if end for end if
return 33 for each layer
of dCCE and ach [
However, the surrogate gradient only an approximation of d-function, and an overly relaxed surrogate harmful to the network's final performance. If we reasonably restrict the effective interval of the surrogate gradients, it is possible to drive the SNN to select those more essential membrane potentials for backpropagation.
awe))
We also need to ensure that the new X does not make the network difficult to train, for this, CPNG sets the target X of each layer to the smallest that has occurred in the current layer during the training iteration, rather than an artificial goal. If the network can be trained when the smallest x appears, then the network should still be trained after we adjust the 3 and maintain the smallest When using CPNG, we expect that the network parameters are appropriate, that is, the network has traversed the whole dataset to prevent the misleading of network parameters by data randomness. For example, if we use CPNG once per batch, the network parameters are mostly affected by the first few batches in the early stages of network training, and the statistical indicator x obtained by using such network parameters will have a lot of randomnesses.
CPNG computes the smallest of each layer during the iteration process as Xmin and records it. If the X value a certain layer rises after an epoch, CPNG adjusts the x value the current layer to Xmin by increasing the B, otherwise, keep the current /3 fixed and update Xmin- Since the Xmin of each layer of neurons may be different, different layers may have different surrogate gradient shapes. In addition, we set safe lower bound Xlimit- When x falls below the lower bound, the may decrease to force x back above Xlimit in order to guarantee that sufficient membrane potential values are covered in the effective domain of surrogate gradient for successful SNN training. The CPNG algorithm is detailed in Algo
4.3 THE COST OF CPNG METHOD
The extra cost of CPNG occurs in two steps: (1) collecting the mean and variance of the membrane potential before releasing the spikes of each layer; (2) altering the / using the indicator x. In our experiment, we only use the mean and variance of the last batch to calculate the indicator X. which makes the cost of the first step in the same order of magnitude as the batch normalization (loffe & Szegedy 2013) operation. For the latter step, we provide a binary search method that solves the problem very fast, and further optimization algorithms can further improve the solution speed. The above analysis is the cost of using CPNG once, the overall cost takes into account the frequency of using CPNG. In experiments, we employ CPNG just once every epoch, which is quite economical

---

Under review as a conference paper at ICLR 2023
Table I: The impact of using time and batch size Vgg16+CIFAR100 experiment.
256(w/o CPNG)
256(B)
256(E)
512(w/o CPNG)
512(B)
512(E)
Acc
69.08% 705.59m
71.05%
71.54%
68.44%
69.59%
70.20%
Time
799.41m
716.67m
377.76m
424.49m
381.79m
B: Use CPNG per batch.
E: Use CPNG per epoch (last batch)
when compared to the network training time. Quantitatively, in the VGGI6+CIFARIO0 experiment, it takes average of 1.7GFLOPs obtain the output corresponding to input without CPNG, and the first step of CPNG will only add 5.59 x 10-3 additional GFLOPs, Using CPNG once per epoch takes average of 3.3 seconds of overhead (1.57% of total training time).
EXPERIMENT
To verify the effectiveness of the CPNG method, provide groups of comparative experiments (Sec. on both static and neuromorphic datasets. We also compare CPNG with existing works in Sec. and show the final of SNN after training with CPNG in Sec. 5.4
5.2
5.3
5.1
IMPLEMENTATION DETAILS
All the SNN architectures include the tdBN layer (Zheng et al. 2020) with the average-pooling layer, and compared to their ANN versions, we replace the activation function ReLU with LIF. Our experiment settings, such optimizer, learning rate, are detailed in Appendix. [A.2] Except for applying the CPNG method at the end of each epoch, all other conditions, such as learning rate, batch size, etc., consistent. The data preprocessing for each dataset included in the experiments are as follows:
CIFAR and ImageNet. We use standard processing methods for these datasets, see appendix. A.1 for more details.
CIFARIO-DVS. CIFARIO-DVS is event-based image datasets. Following the previous work LL ct al. 20216), we re-sample 10 simulation length, divide the dataset into 9k training images and 1k test images, reduce the spatial resolution 48x48, and apply data augmentation techniques.
Table 2: Ximii Experiments on BesNet18+CIFALE
Xlimit
0.05
0.1
0.2
0.3
0.4
0.5
0.6
Accuracy 76.6%
76.6%
76.37%
76.8%
76.0%
76.1%
74.1%
5.2 PERFORMANCE IMPROVEMENT WITH CPNG
Although x is employed as a statistical indicator, table] demonstrates that increasing the batch size does not always result in better performance of CPNG. This is because there is no strict positive correlation between neural network performance and batch size. We don't need increase batch size excessively to increase the sample size of X. Besides, as we mentioned in 4.2 using CPNG once per batch does not guarantee performance improvement, may bring more overhead and worse performance. Then we compared the impact of different Xlimit CPNG, shows that we may don't 't need to focus too much on Xlimit, 0.05 0.2 might be suitable. our experiments, we set =0.2 uniformly.
Additionally, we tested the applicability CPNG to various surrogate gradient functions. For surro- gate gradients with non-zero gradient everywhere, such arctan, we directly use [V/ 1 1/88, v 1/3] as the integration interval to calculate the indicator X, then use Algo to solve new 3. Considering the case of using triangle-like surrogate gradient, we control the percentage of membrane potentials with non-zere gradients, which can also be explained as: under the current surrogate gradient, find a representative interval related B, and make the calculated x this interval equal to the Xtarget- In the
7

---

Under review as a conference paper at ICLR 2023
Table 3: Examine CPNG various surrogate gradients.
Dataset
Method
Architecture
Time Step
Accuracy
Triangular
ResNet18
10
75.6%
Triangula++CPNG
ResNet18
76.37=1.02
Rectangular
ResNer18
10
74.7%
Rectangular+CPNG
ResNer18
75.3%
CIFARIO-DVS
Are'Tan
ResNet18
10
67.2%
CrCT
ResNet18
10
67.5%
Triangular+TET
ResNet18
10
79.9%
TTianP
ResNet18
10
82.3% %
Table 4: Result of training spiking neural network.
Dataset
Method
Architecture
Time Step
Accuracy
STBP-1dBN Zheng al. 2020
ResNet19
93.16%
ANN-1o-SNN La 2011
ResNer20
32 64 128
94.78% 95.30% 95.42%
CIFARIO
ANN-to-SNN Bü
2021
ResNer20
8 16 32 6
89.55% 91.62% 92.24%
CPNG
ResNet19
94.02_00.055
Diet-SNN (Rathi & Roy
2020
VGG16
S
69.67%
CIFARI00
ANN-to-Sin -NNN al
OZ
VGG16
32 64 128
73.55%
76.645 64% 77.40% 73.96%
ANN-to-SNN Bu al
VGG16
8 16 32
76.24% 77.01% %
CPNG
VGG16
5
71.32-0.20%
Dspike (L al
2021b 2022
ResNet18
10
75.40%
TET Deng et al.
VGGI
10
83.17%
CIFARIO-DVS
CPN
ResNer18
76.37%-++..2
CPNG TET
VGG11
83.83=0.00%%
ANN-to-SNN Li et 2021a
ResNet34
32 64 128
64.54% 71.12% 73.45%
ANN-to-SNY -NN-o--NNN/A Bu al.
2021
ResNer34
16 32
59.35% 69.37% 72.35%
Sew-ResNet (Fang el 2021
Sew- Sew-Res ResNeeeN
67. 67.04%
ImageNet
TET Deng
BakkingSSewRResNer33 Sew-ReaNet34
64.79%
TET Deng 2022
68. 68.00%
CPNG
Sew- New-ResNet34
4
69.09%
case of using surrogate gradients with infinite non-zero gradient interval, the representative interval if we need a smaller interval [V F1, rı]
for calculating the indicator is [ [ 1/ 1/3],
with the current mean and standard deviation, just the new 8 to 1//1- The experimental results in table3]alss also demonstrate the effectiveness our approach (More results are shown in Appendix (A.6).
Finally, verify the compatibility of CPNG with existing direct training methods TET (Deng et al. 2022 (table3) we got a 2.4% improvement on the ResNetl S+CIFAR-DVS experiment.
5.3
COMPARISON TO EXISTING WORKS
In this section, the experimental results we report all use triangle-like surrogate gradient. On some datasets, the current SOTA conversion method performs better than direct training, but they need lengthy simulation time steps, especially on the ImageNet dataset. In addition, the conversion-based method cannot be applied to neuromorphic datasets such CIFARIO-DVS, Hence, it's also necessary to design a surrogate gradient search method to optimize the direct training.
VGG16 CIFAR100. We train the SNN for 300 epochs on TITAN, with the batch size of 256 and the time step of 5. As shown in Table 6 CPNG manages to improve 2.24% accuracy. To show that CPNG effectively controls the rise of X. report the x during training in Fig. The result of ResNec20+CIFAR 100 is also shown in Table
ResNet19 CIFAR10. We use 19 (Zheng et al. 2020 in accordance with tdBN layers, the batch size of 256 and the time step of 6. Then we train the SNN for 200 epochs on TITAN, CPNG achieves increase in test accuracy to 94.02%, while its counterpart is 93.26% (Table cp.
ResNet18 CIFAR1O-DVS. We train the model for 200 epochs on 2 TITAN Xp cards, with the time step of 10 and the batch size of 72. The CPNG obtains higher test accuracy of 76.37% while
8

---

Under review as a conference paper at ICLR 2023
Syerl
Dyrtt
Syer1
0.7
835G2
83223
Indicaror
0.4
layer]
layers
Myert
02
6.2
layard
lager5
1apert0
Uyer15
01 50
100
15a
252
25n
300
5.
99
100
116
200
250
350
Epoth
Epoch
Figure 3: In the experiment of VGG16, the proportion of non-zero gradient membrane potentials of neurons different layers without CPNG (left) and with CPNG (right).
30
05
0.5
00
0.0
Figure 4: Left: Final 3 of each layer VGG16 experiment, the initial 3 is 1.0. Right: Final B of each layer in Sew-ResNer34 experiment, the initial / is 1.0..
consuming significantly less overhead than Dspike (Li et al. 2021b). In addition, the combination of CPNG and TET can achieve an average test accuracy of 83.83%, which new SOTA.
Sew-ResNet34 + ImageNet. We use the same structure and membrane decay rate, etc. as Sew- ResNet (Fang et al. (202T) We train the model for 160 epochs GTX 3090 cards with time step of 4 and the batch size of 544. Using only CPNG can achieve an accuracy of 69.04%, surpassing the current direct training SOTA.
5.4 SURROGATE GRADIENT SHAPES OF DIFFERENT LAYERS
We show the final B3 of some experiments in Fig 4 and all experimental results are presented in the appendix A..3 Various layers' 8 are different, which demonstrates that various layers match distinct surrogate gradient shapes as result of their varying membrane potential distributions.
CPNG eventually increases the 3 of most layers. Compare to CPNG, randomly increasing the B can make the network difficult train (Fig. [eftt).. Even with 3 set to 1.5, which most of the neuron layers shown in Fig. 41eft) can approach or reach, the network is still difficult to train. This demonstrates that safe increase 3 using CPNG, while unsafe to increase 3 arbitrarily.
CONCLUSION
This work proposes new perspective for directing the shape change the surrogate gradient, we propose statistical indicator that guides the shape change of the surrogate gradient, and propose the CPNG method for modifying the shape of the surrogate gradient during training while guaranteeing the proportion of membrane potential with non-zero gradients. It's possible that the failure to produce satisfactory results when pulling surrogate gradient directly to f-function due to a failure to meet the premise that the network can be trained normally. In other words, there may exists trade-off between the approximation to the d-function and the effective domain gradients under the given dataset, and CPNG helps approach the equilibrium point.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Guillaume Bellec, Darjan Salaj, Anand Subramoncy, Robert Legenstein, and Wolfgang Maass. Long short-term memory and learning-to-learn in networks of spiking neurons. In S. Bengio, H. Wallach, Н. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Gar- nett (eds.), Advances in Neural Information Processing Systems, volume 31. Curran Asso- ciates, Inc., 2018. URL htttps://proceeeeii http://wceeaapo..oeepeooo///////
neurips cctpppaper///01///lllll E2C384l551122a572257756655333333 Papeee..--pprrpppppp
Tong Bu, Wei Fang, Jianhao Ding, PengLin Dai, Zhaofei Yu, and Tiejun Huang. Optimal ann-snn conversion for high- accuracc and satra-low-latenne spiking neural networks. In International Conference on Learning Representations, 2021.
Lei Deng, Yujie Wu, Xing Hu, Ling Liang, Yufei Ding, Guoqi Li, Guangshe Zhao, Peng Li, and Yuan Xie. Rethinking the performance comparison between snns and anns. Neural Networks, 121: 294-307, 2020.
Shikuang Deng and Shi Gu. Optimal conversion of conventional artificial neural networks to spiking neural networks. In International Conference Learning Representations, 2020.
Shikuang Deng, Yuhang Li, Shanghang Zhang, and Shi Gu. Temporal efficient training of spiking neural network via gradient re- re-weighting. arXiv preprint arXiv:2202.1 1946,, 2022.
Peter U. Diehl, Daniel Neil, Jonathan Binas, Matthew Cook, and Shih Chii Liu. Fast-classifying, high-accuracy spiking deep networks through weight and threshold balancing. In Neural Networks (JJCN), 2015 International Joint Conference on, 2015.
Peter U Dichl, Guido Zarrella, Andrew Cassidy, Bruno U Pedroni, and Emre Neftci. Conversion of artificial recurrent neural networks to spiking neural networks for low-power neuromorphic hardware. In 2016 IEEE International Conference on Rebooting Computing (ICRC), pp. 1-8.
IEEE, 2016.
Wei Fang, Zhaofei Yu, Yanqi Chen, Tiejun Huang, Timothée Masquelier, and Yonghong Tian. Deep residual learning in spiking neural networks. arXiv preprint arXiv:2102.04159 2021.
Bing Han, Gopalakrishnan Srinivasan, and Kaushik Roy. Rmp-snn: Residual membrane potential neuron for enabling deeper high-accuracy and low-latency spiking neural network. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 13558-13567, 2020.
Yunzhe Hao, Xuhui Huang, Meng Dong, and Bo Xu. A biologically plausible supervised learning method for spiking neural networks using the symmetric stdp rule. Neural Networks, 121:387-395, 2020.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Identity mappings in deep residual networks. In European conference computer vision, pp. 630-645. Springer, 2016.
Luca Herranz-Celotti and Jean Rouat. Surrogate gradients design. arXiv preprint 27X1v22202.00282. 2022.
Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International conference on machine learning, pp. 448-456. PMLR, 2015.
Yingyezhe Jin, Wenrui Zhang, and Peng Li. Hybrid macro/micro level backpropagation for training deep spiking neural networks. arXiv preprint arXiv: 07860078666..... 2018.
Chankyu Lee, Priyadarshini Panda, Gopalakrishnan Srinivasan, and Kaushik Roy. Training deep spiking convolutional neural networks with stdp-based unsupervised pre-training followed by supervised fine-tuning. Frontiers in neuroscience, 12:435, 2018.
Hao Li, Zheng Xu, Gavin Taylor, Christoph Studer, and Tom Goldstein. Visualizing the loss landscape of neural nets. arXiv preprint arXiv: 1712.099 2017.
10

---

Under review as a conference paper at ICLR 2023
Yuhang Li, Shikuang Deng, Xin Dong, Ruihao Gong, and Shi Gu. A free lunch from ann: Towards efficient, accurate spiking neural networks calibration. arXiv preprint arXiv:2106.0640 2021a.
Yuhang Li, Yufei Guo, Shanghang Zhang, Shikuang Deng. Yongqing Hai, and Shi Gu. Differentiable spike: Rethinking gradient-descent for training spiking neural networks. Advances in Neural Information Processing Systems, 34, 2021b.
Emre O Neftci, Hesham Mostafa, and Friedemann Zenke. Surrogate gradient learning in spiking neural networks: Bringing the power of gradient-based optimization to spiking neural networks. IEEE Signal P Processing Magazine, 366)):51-6665111111 2019.
Nitin Rathi and Kaushik Roy. Diet-snn: Direct input encoding with leakage and threshold optimization in deep spiking neural networks. arXiv preprint (rXx::2008.03658, 2020.
Bodo Rueckauer, Iulia-Alexandra Lungu, Yuhuang Hu, and Michael Pfeiffer. Theory and tools for the conversion of analog spiking convolutional neural networks. arXiv preprint arXiv:1612.00 1612.04052 2016.
Bodo Rueckauer, Auxia-Ale Lungu, Yuhuang Hu, Michael Pfeiffer, and Shih-Chii Liu. Conver- sion of continuous-value deep networks to efficient event-driven networks for image classification.
Frontiers in neuroscience, 1:682, 2017.
Abhronil Sengupta, Yuting Ye, Robert Wang, Chiao Liu, and Kaushik Roy. Going deeper in spiking neural networks: Vgg and residual architectures. Frontiers in neuroscience, 13:95, 2019.
Amar Shrestha, Khadeer Ahmed, Yanzhi Wang, and Qinru Qiu. Stable spike-timing dependent plasticity rule for multilayer unsupervised and supervised learning. In 2017 international joint conference on neural networks (IJCNN), pp. 1999-2006. IEEE, 2017.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv: 1409. 1556, 2014.
Daniel Smilkov, Nikhil Thorat, Been Kim, Fernanda Viégas, and Martin Wattenberg. Smoothgrad: removing noise by adding noise. arXiv preprint 130.0080,, 2017.
Kazuma Suetake, Shin-ichi Ikegawa, Ryuji Saiin, and Yoshihide Sawada. $ Time step reduction of spiking surrogate gradients for training energy efficient single-step neural networks. arXiv preprint arXiv:2201.1087) 2022.
Yujie Wu, Lei Deng, Guoqi Li, Jun Zhu, and Luping Shi. Spatio-temporal backpropagation for training high- neghpperformanc spiking neural networks. Frontiers in neuroscience, 12:331, 2018.
Yujie Wu, Lei Deng, Guoqi Li, Jun Zhu, Yuan Xie, and Luping Shi. Direct training for spiking neural networks: Faster, larger, better. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pp. 1311-1318 2019.
Yukun Yang, Wenrui Zhang, and Peng Li. Backpropagated neighborhood aggregation for accurate training of spiking neural networks. In International Conference on Machine Learning, pp. 11852-11862. PMLR, 2021.
Friedemann Zenke and Surya Ganguli. Superspike: Supervised learning in multilayer spiking neural networks. Neural computation, 301))151 2018.
Friedemann Zenke and Tim P Vogels. The remarkable robustness of surrogate gradient learning for instilling complex function in spiking neural networks. Neural computation, 33(4):899-99) 925, 2021.
Wenrui Zhang and Peng Li. Temporal spike sequence learning via backpropagation for deep spiking neural networks. arXiv preprint arXiv:2002. 10085, 2020.
Hanle Zheng, Yujie Wu, Lei Deng, Yifan Hu, and Guoqi Li. Going deeper with sirectlyttriine larger spiking neural networks. arXiv preprint arXiv:2011.05280 2020.
11

---

Under review as a conference paper at ICLR 2023
A APPENDIX
A.1 DATA PROCESSING
CTFAR. For the training set, we randomly crop the image to (32,32) and apply a random horizontal flip. Then all the images will be normalized to standard normal distribution.
ImageNet. For the training set, we randomly crop the image (224.224) and apply random horizontal flip. Then for the test set, we resize the image to (256,256) and center crop the image to (224,224). Finally, all the images are normalized to a standard normal distribution.
A.2 MORE EXPERIMENT DETAILS
experimental details. We 1/20 of the training epochs for warm-up, linearly increase learning rate from 0.1/r to ir, then use cosine decay to reduce learning rate to O in the remaining epochs. In the last batch, the mean and variance of the membrane potential are additionally stored in the forward process, and after the backward process, the stored mean and variance are used to obtain new 3 according to Algorithm 1. Hyperparameters are shown in the table 5
In this section, provide
Table Experiment Setting
Experiment
CIFARI00
CIFARIO
CIFAR-DVS
ImageNet
CPNG + TET
learning
0.1
0.1
0.01
0.01
0.001
weight decay
le-4
le-4
4e-5
4e-5
4e-5
momentum
0.9
0.9
0.9
0.9
optimizer
sgd
sgd
sgd
sgd False
adam
warm-up
True
True
False
False
A.3 EXPERIMENT RESULT
In this section, we display the change of indicator (Fig. 5 Fig. 6). the final ,3 and the test accuracy of each experiment. Our codes can be found in the supplemental. As shown by Fig. 7 Fig. 8 Fig. / and Fig. 10, except for the first layer of the Sew-ResNer34+ layers have obtained a steeper surrogate gradient (a larger 99, which illustrates that surrogate gradient has further optimization space in the SNN training process. However,
experiment, almost all neuron
mentioned in Sec. 4.1 choosing a surrogate gradient closer to the 8 function at beginning will make training the network difficult, therefore we'll need tools (such CPNG) assist in finding better surrogate gradient experiment, CPNG finds a smaller / for the first
during training. In the Sow-ResNer34+ImageNe layer to ensure the proportion of membrane potential with non-zero gradients. This suggests that when backpropagating with surrogate gradient in deeper networks, the initial few layers may only have a small proportion of their parameters updated, and we should account for this more.
0.T
1yye2
9yc744
lyyer'
87113
kyass
Rpar.
19576
Indreato
0.5
05
0.4
layet
18yo17
bytr2
Sydt2
0.2
02
box77
18pe133
Sper2
50
75
169
125
150
175
700
05
25
20
75
1.0
125
750
200
Epoch
Lpoch
Figure 5: In the experiment of ResNet19, the proportion non-zero gradient membrane potentials of neurons in different layers without CPNG (left) and with CPNG (right).
12

---

Under review as a conference paper at ICLR 2023
Ryer
kyer
med1
Ryyr7
Ryend
0.7
9y60
80011
1yo15
3760
leyert
Uyeert
Hyer12
1yart
0.4
6A
Syer7
62-
9ps02
#p551
61
255
Figure 6: The proportion of non-zero gradient membrane potentials of neurons in different layers with CPNG in the experiment of ResNet18 (left) and Sew-ResNet34 (right).
70
3.0
Test accuracy
50
Final beta
2.0
30
CPNG, round1
1.0
10
round2
CPNG round3
50
100
Epoch
1.50
200
250
300
0.0
13
Figure 7: Left: Test accuracy of triplicate VGGI6+CIFAR1OG 100 experiments using CPNG. Right: Final of each layer in 10FAR100 experiment, the initial / is 1.0
A.4 USING BETA OBTAINED FROM CPNG TO TRAIN FROM SCRATCH
We fix each layer's / the same as it in Figure 4 (left), the final accuracy is 71.02%, which is 0.3% lower than using CPNG directly and 1.94% higher than the baseline. We keep = during first 1/10 epochs for warm-up, otherwise the accuracy will stay at 1%. This demonstrates that the optimal SG shape discovered by CPNG for all layers is valid, but the result under the case that directly changes the 3 to the optimal value is not excellent as the case using the CPNG method due to non-stationary transitions.
A.5 MEMBRANE POTENTIAL DISTRIBUTION
In this section, show the membrane potential distribution of each layer of neurons in all experi- ments( Fig. [I Fig. 12 Fig. 13). The layer after BN is more consistent with the normal distribution than the layer after FC. But for the convenience of calculation, approximate the membrane potential distribution all neurons to the normal distribution.
A.6 ADJUST OTHER SURROGATE GRADIENTS
The results report are based triangle- like surrogate gradient, surrogate gradient with finite non-zero gradient interval, but this does not mean that our method can only be used this case. All results shown in Tab. 6
13

---

Under review as a conference paper at ICLR 2023
90
80
2.5
Test acruracy
70
Final beta
2.0
80
1.5
CPNG round1
CPNG round2
CPNG round3
25
50
Epoch
100
125
150
175
200
0.0
10
14
16
Figure 8: Left: Test accuracy of triplicate ResNer19CCIFAR10 experiments using CPNG. Right: Final B of each layer in CIFARIO experiment, the initial 3 1.0
2.00
1.75
Test accuracy
1.50
50
125
80-
1.00
0.75
30-
CPNG_rov Idd
CPNG, round2
CPNG round3
0.25
25
125
Epoch
10 14 151617
Figure 9: Left: Test accuracy of triplicate ResNet18+CIFFRRIODY experiments using CPNG. Right: Final B of each layer in CIFAR I0-DVS experiment, the initial /3 is 1.0
A.7 SALIENCY MAP
The gradient noise generated by surrogate gradient may affect the network's attention location, resulting in a more blurred saliency map. And there's no doubt that a more appropriate surrogate gradient will produce fewer gradient noises. Thus, the saliency map's clarity may be utilized to determine the fitness of surrogate gradient. We used SmoothGrad (Smilkov et al.) 2017), which reduces the effect of visually noise. The results of SETARI0O+GGER
experiments are shown in
Fig. [4 We discovered that CPNG can assist the model to identify clearer contour information. For example, in the boys category, the saliency map obtained by using CPNG more accurately positioned the person's 's face, whereas traditional surrogate gradient also pays much attention to the surrounding background; in the elephants category, the use CPNG can clearly see the trunk as well as the elephant's 's outline, whereas traditional surrogate gradient can only obtain blurred borders.
A.8 LOSS LANDSCAPE
order illustrate the trainability of the model after using CPNG, we use the loss landscape to show the change of loss after changing the model parameters. If the loss landscape does not reveal more non-convex regions than the traditional surrogate gradient, it means that using CPNG will not make convergence more difficult. If the same weight offset is applied for various networks when displaying the loss landscape, the networks with bigger weight will exhibit more stationarity. However, due to the existence of batch normalization, the weight scaling of the network has effect the inference results, so the sharpness of loss landscape of different networks may only be due to the weight scaling. To explain the model performance between the traditional surrogate gradient and CPNG,
14

---

Under review as a conference paper at ICLR 2023
70
2.5
Tect accuracy
50
Final beta
30
CPNG
10
0
20
50
80
80 Epoch
100
120
140
160
0.0
Figure 10: Left: Test accuracy of Bew-Ressee344Imageeeet experiments using CPNG. Right: Final 3 of each layer in ImageNet experiment, the initial /3 is 1.0
Figure 11: Membrane distribution of each layer in experiment training VGG16 on CIFAR100
the loss landscape demonstration with filter- wise normalization ( et al. 2017) that mitigates the effect of weight scaling and correlates the model's generalization ability to the flatness of the loss landscape. As shown in Fig. [5] in the experiment of ResNer19+CIFAR 10, utilizing CPNG can obtain a smoother minimum and a wider locally convex region, implying that CPNG has no negative effect on network's convergence difficulty.
15

---

Under review as a conference paper at ICLR 2023
Figure 12: Membrane distribution of each layer in experiment training ResNet19 on CIFARIO
Figure 13: Membrane distribution of each layer in experiment training ResNet18 on CIFAR-DVS
16

---

Under review as a conference paper at ICLR 2023
Table 6: Examine CPNG on various surrogate gradients.
Dataset
Method
Architecture
Time Step
Accuracy
Triangular
ResNet19
93.26%
CCNNG
ResNec11
94.02:+0.05%
Rectangular
ResNet19
90.5%
CIFARIO
Rectangular-CPNG
ResNet19
93.07=+0.00 0.4%
Arc'Tan
ResNet19
93 23.24%
ArcTan+CPNG
ResNet19
93.95%
Triangular
VGG16
69.08%
CrNNG
VGG16
71.32:10.20%
Rectangular
VGG16
65. 35.54%
CIFARTO0
Rectangular+CPNG
VGG16
67.74-11.025
Arc'Tan
VGG16
68.15%
CrcTan+CPNG
VGGI6
00,4990.220%
Triangular+CPNG
ResNer20
76.09%
ArcTan
ResNet18
10
67.2%
AreTan+CPNG
ResNet18
10
67.5%
Triangular+TET
ResNet18
10
79.9%
CIFARIO-DVS
TEinnPGG
ResNet18
10
82.3%
Triangular
ResNet18
10
75.6%
CPNGG
ResNer18
10
76.37 1. 1.02
train keyboard sunflower elephant boy
willowTrec
pine Tree
rose
cockroach
sea
wolf
telephone
Figure 14: Saliency map. Three photos group, the top the original image, the middle the salilency map given by the model trained by traditional surrogate gradient, and the bottom is the saliency map given by the model trained with CPNG.
17

---

Under review as a conference paper at ICLR 2023
133
-12
-12
-139
traditional SG
CPNG
Figure 15: Loss landscape of traditional surrogate gradient and CPNG.
18


Reviews:
[{"review_id": "RBzFAgJ1CFH", "reviewer": "Reviewer_eFt1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper is about changes that need to be made to error-backpropagation in order to train a leaky-integrate and fire spiking neuron network. A technique that is commonly used is to \"smoothen\" the spike so that gradients do not collapse onto the two values of 0 or infinity. This paper suggests an adaptive smoother. They define a parametrized class of continuous functions (triangles with compact support in their case) that limit to the delta function. They then adapt the support parameter (the width of the triangle) based on the layer and the input. Specifically, (based on samples), they determine what the range of values the membrane potential takes and then use that to set the width parameter to have non-zero finite gradients for a fixed proportion of neurons.", "summary_of_the_review": "The introduction of \"adaptation\" to the surrogate gradient is not novel enough to warrant publication, particularly since there have been so many ways people have tried to get around the gradient issue for spiking neuron networks.", "strength_and_weaknesses": "Strength: The idea introduced is intuitive, experiments show improvements.\nWeekness: The idea is fairly obvious. Implementation is rather laborious and there are no formal/mathematical evaluations of why this intuitive solution ahould give better performance. The authors just suggest a technique and then evaluate it on a set of datasets.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "Bz1b7PuTXa", "reviewer": "Reviewer_z3Wm", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a method to improve the training of spiking neural networks (SNNs) via the surrogate gradient method. Their technique called CPNG controls the proportion of non-zero gradients during network backpropagation. Surrogate gradient functions are only non-zero when the membrane potential is within a certain region about the threshold value. The width of that region can be defined using a single parameter, which has so far remained fixed during training. In order to allow this parameter to vary optimally during training, the authors compute the proportion of neurons that have their membrane potential within that region of non-zero gradients, and adapt the region accordingly. If the proportion is high enough, the width decreases, and inversely, if the proportion is too small, the width increases, so that the overall activity of the network neither explodes nor vanishes. Their method shows performance improvements on all four tasks (CIFAR10, CIFAR100, CIFAR10-DVS and ImageNet) compared to the fixed width surrogate gradient.\n\n", "summary_of_the_review": "The authors have the right approach; the method makes sense and is helpful.  However, the evaluation could be improved.", "strength_and_weaknesses": "Strengths\n\nThe paper contributes to improving the surrogate gradient training of SNNs, which should be practical and profitable for the related field of research. The method is relatively simple and well explained to be reproduced.\nThe computational costs of the method are properly indicated and remain limited.\n\nWeaknesses\n\nWhen comparing with the baseline (i.e. without CPNG), it may not be clear that the value of the fixed width is optimised. For a fairer comparison, a hyperparameter search could be done, so that CPNG is compared to the best version of the fixed surrogate gradient function.\nIn relation to that, in table 3 and 6, confidence intervals are only given for CPNG entries. It would be coherent to also include such intervals for the non-CPNG baseline.\nIt appears that the choice of value for the effective domain indicator is only based on a single ad-hoc experiment with two values considered, namely 0.05 and 0.2. Here It would also be more meaningful to make a more advanced hyperparameter search. \nThe grammar could be improved.\n\nRemarks\n\nGiven that membrane potentials evolve over time, it is not exactly clear to me how the effective domain indicator is calculated. I can guess that the mean “mu” is computed over all time steps and all neurons, but it may be clearer to indicate it more explicitly inside the text.\nIt could be interesting to relate the effective domain indicator to the resulting firing rates. Does increasing the former have an impact on the latter?\nThe method is also only tested using a relatively small number of time steps (5-10). It would be good either to increase the number of steps on the same tasks, or to directly apply it to tasks that involve longer sequences such as speech for instance.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "8FCBBWSJi7_", "reviewer": "Reviewer_yRYU", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "Spiking Neural Networks (SNN) are biologically more plausible, capable of representing time as well as more energy-efficient versions of more common artificial neural networks. The main difference is that SNNs communicate across layers using additive spikes. However besides the aforementioned advantages, this makes it really hard to train such networks. Three alternatives exist to do it. One is to convert pre-trained ANN to SNN, use heuristics, and lastly use a type of backpropagation method to train these networks. Surrogate gradients help to train SNNs. However, even using surrogate gradients SNNs train less efficiently. Also, the resulting performance of SNNs is usually worse than the one of ANNs.This is because surrogate gradients can only approximate real gradients, this procedure is very susceptible to the problem of vanishing gradients and is overall less stable than in ANNs. The main idea behind this paper is to stabilize the process of training SNNs by keeping track of the proportion of non-zero gradients during training. Experimental results presented in the paper seem to suggest that the proposed strategy produces superior results to published alternatives.", "summary_of_the_review": "Overall, the paper leaves a very good impression from the novelty and reproducibility points of view. Some additional work should be done to improve the clarity of the paper, especially for people who lack the SNN background. Most importantly, additional work is expected to explain the inconsistencies pointed out in the results when it comes to comparison to the existing methods. At the moment it seems as if the authors avoid displaying information that might make their method look inferior. In goodwill, this reviewer leaves enough space for the authors to prove the reviewer wrong, therefore there is a good chance that my recommendation will change as soon as evidence to the contrary is presented. ", "strength_and_weaknesses": "This paper seems to have a substantial technical depth, overall, for the non-expert, the method proposed in this work seems to be very reasonable and novel. If not for the worrying weakness, presented below, this reviewer would argue for this paper to be accepted.\n\nAccording to this reviewer, the main weakness of this paper is in the results, more precisely in the section Comparison to Existing Work. Going through Table 4 the reviewer could not help to notice some inconsistencies. For example, when discussing the results on CIFAR100 with VGG16 the authors wrote that \"CPNG manages to improve 2.24% accuracy\", while it remained unclear for the reviewer from Table 4, where this 2.24% is coming from. Diet-SNN with an accuracy of 69.67% is 1.65% below the performance of CPNG (71.32%). Also perhaps more surprisingly the other two methods presented in Table 4, under the same CIFAR100 category show much better accuracies, ranging from 73.55% (ANN-to-SNN with time step 32 by Li et al.) all the way to 77.40% (ANN-to-SNN with time step 128 by Li et al.). Another important observation is that the results by Dspike (Li et al., 2021b) are not presented in this category at all, despite the fact that the Dspike performs relatively well on CIFAR100, with superior to CPNG performance of 74.24%. The same or similar situation is with other datasets leaving only CIFAR10-DVS, Dspike is left out of the comparison with usually superior performance. All these create an impression that the authors were trying to manipulate the readers by hiding information that presents their method in an unfavorable light. This reviewer is happy to be proven wrong and pointed at a different reason for these inconsistencies e.g. because Dspike had used a different part of the validation or test set hence making the results incomparable. Nevertheless, even if these other reasons exist for not presenting the results from Dspike, this reviewer strongly believes that the readers should have been surely presented with these reasons in advance. Showing less favorable performance compared to some alternatives should not encourage the authors to hide these alternatives, but rather admit that some other methods seem to have an edge. \n\nGenerally, a paper does require a lot of relevant background, hence might be very hard to read for non-experts. Terms such as delta-function, membrane potential neurons, pre-synaptic input, leaky integrate and fire module, and others are used without being defined nor explained, raising the bar for researchers outside of the spiking neural networks field. Having said this, formulas proposed in the paper to illustrate some key concepts are thoroughly defined and explained, making it possible to follow the narrative albeit missing some details. Another formatting issue that might become apparent for researchers outside of the field who are not as well familiar with terminology is the use of acronyms that were either not defined or defined. I.e. one time defined acronym somewhere in the abstract and later used arbitrarily in the text, is going to be confusing for the unprepared reader e.g. \"SG\", \"tdBN\", etc.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "07WY6HHGmhx", "reviewer": "Reviewer_Sqv2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "A specific surrogate gradient is described, the idea is to regulate the sharpness of the triangular surrogate gradient at each iteration so that the proportion of non-zero gradients is controlled at each layer. This simple method can be implemented efficiently and leads to a ~2% improvement on ImageNet and CIFAR 100.", "summary_of_the_review": "Overall the method is simple and might be an important step towards binary ANN and SNN, my hope is that a method of this type can make it perform as-well as real-valued Relu networks. If the claim is true it is one step in that direction. This could have a tremendous impact but a few clarifications are necessary to make sure that the performance gain is coming from the method as described in the paper.\n\n-- Review update, November 6th -- \nI was eager to try so I implemented a simple version of the algorithm in this paper. I tried this approach on a Binary MLP on MNIST with 2 hidden layers (no time steps). On this simple setup the performance was lower than with the regular triangular derivative with constant width. My implementation was certainly not identical, so there might be a way to make it work. I think this paper would have been stronger if it described a simple reproducible setup of this kind.", "strength_and_weaknesses": "The method is simple and it can be implemented easily with auto-diff which is a strong plus. The benefit seems to be substantial although more details are required to make sure the benefit comes from the adaptive scaling keeping a stable proportion of zeros and not a naive scaling of beta. So I would find the results much stronger if the performance difference between triangular and triangular+CNPG is investigated further to understand better how it works, I suggest a couple of clarifications and simple experiments which would make the paper better in my opinion:\n\n1) What is the value of beta for the Triangular control on the performance table? Traditionally the performance of the baseline would be optimized by trying different values of beta, for instance in [02, 0.5, 0.8, 1, 1.5, 2., 3.], was it done in the performance table? If not it would make sense to try, or to report at least how beta was chosen and add the performance obtained with the average beta value for Figure 4 (beta=1.5 looks like a good control for VGG16 for instance).\n\n2) Given that the method is rather ad-hoc it is rather important to demonstrate what aspects of the method matters the most. For instance, I wonder whether restarting a training with fixed layer specific betas as shown in Figure 4 would also explained the performance benefit. Crucially, even if this control is good, it does not mean that CNPG is useless because it finds those beta values automatically, but at least it would provide some insights on whether the adaptive aspect of the method is important or whether what matters is to find the right betas.\n\nA strong weakness is that this paper seems embedded into a sub-community of SNN research and ignores many relevant work outside of that: \n\n3) All of what is applied here for surrogate gradient is equally applicable to binary network where the relu activation function is simply replaced by a Heaviside function. A connection to this literature and a fair account of it's progress would be relevant, see [1] for the first publication on back-prop with quantized activation and [2] for the first paper using the triangular derivative as far as I know. On top of citing these two papers, I would find the paper much stronger if a binary ANN model like Esser et al. (basically equivalent to SNN with T=1) would be added to show that the method can also be applied in the broader quantized network community. Along those line, providing the performance of the model with relu and all identical parameters would be a good insight to see how far is the model compared to classical ANNs.\n\n4) There is already an open debate inside the SNN community about the benefit of triangular derivatives and how their sharpness was identified to be a critical parameter, see methods from [3] for instance where the height of beta is referred to as the dampening factor. It would be fair to acknowledge that.\n\n[1] Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation\nYoshua Bengio, Nicholas Léonard, Aaron Courville\n\n[2] Backpropagation for Energy-Efficient Neuromorphic Computing\nSteve K. Esser, Rathinakumar Appuswamy, Paul Merolla, John V. Arthur, Dharmendra S. Modha\n\n[3] Long short-term memory and learning-to-learn in networks of spiking neurons\nGuillaume Bellec, Darjan Salaj, Anand Subramoney, Robert Legenstein, Wolfgang Maass", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject