Take A Shortcut Back: Mitigating the Gradient Vanishing for Training Spiking Neural Networks
Authors: Yufei Guo
Yuanpei Chen
Zecheng Hao
Weihang Peng
Zhou Jie
Yuhan Zhang
Xiaode Liu
Zhe Ma
Venue: NeurIPS 2024 poster (2024)
Keywords: Spiking Neural Network
Training SNNs
Surrogate gradient
Primary area: neuroscience_and_cognitive_science

Abstract: The Spiking Neural Network (SNN) is a biologically inspired neural network infrastructure that has recently garnered significant attention. It utilizes binary spike activations to transmit information, thereby replacing multiplications with additions and resulting in high energy efficiency. However, training an SNN directly poses a challenge due to the undefined gradient of the firing spike process. Although prior works have employed various surrogate gradient training methods that use an alternative function to replace the firing process during back-propagation, these approaches ignore an intrinsic problem: gradient vanishing. To address this issue, we propose a shortcut back-propagation method in the paper, which advocates for transmitting the gradient directly from the loss to the shallow layers. This enables us to present the gradient to the shallow layers directly, thereby significantly mitigating the gradient vanishing problem. Additionally, this method does not introduce any burden during the inference phase.
To strike a balance between final accuracy and ease of training, we also propose an evolutionary training framework and implement it by inducing a balance coefficient that dynamically changes with the training epoch, which further improves the network's performance. Extensive experiments conducted over static and dynamic datasets using several popular network structures reveal that our method consistently outperforms state-of-the-art methods.

Take A Shortcut Back: Mitigating the Gradient Vanishing for Training Spiking Neural Networks
Yufei Guo; Yuanpei Chen', Zecheng Hao, Weihang Peng, Zhou Jie, Yuhan Zhang, Xiaode Liu, Zhe Ma²
Intelligent Science & Technology Academy of CASIC, China School of Computer Science, Peking University, China
yfguoêpku. edu. cn, rop4770163. com, haozecheng@pku. edu. mazhe, thu@163.... com
Abstract
The Spiking Neural Network (SNN) is biologically inspired neural network infrastructure that has recently garnrred significant attention. It utilizes binary spike activations to transmit information, thereby replacing multiplications with additions and resulting in high energy efficiency. However, training an SNN directly poses challenge due the undefined gradient of the firing spike process. Although prior works have employed various surrogate gradient training methods that use an alternative function replace the firing process during back-propagation, these approaches ignore an intrinsic problem: gradient vanishing. To address this issue, we propose a shortcut back- pack-propagation method in the paper, which advocates for transmitting the gradient directly from the loss to the shallow layers. This enables us to present the gradient the shallow layers directly, thereby significantly mitigating the gradient vanishing problem. Additionally, this method does not introduce any burden during the inference phase. To strike a balance between final accuracy and ease of training, we also propose an evolutionary training framework and implement it by inducing balance coefficient that dynamically changes with the training epoch, which further improves the network's performance. Extensive experiments conducted over popular datasets using several popular network structures reveal that our method consistently outperforms state-of-the-arr methods.
Introduction
The Spiking Neural Network (SNN) has become a popular neural network due to its efficiency and has been widely used in various fields such as object recognition Li et al. (2021a): Xiao al.) (202T), object detection Kim et al. (2019); Qu et al. (2023), and pose tracking Zou et al. 2023). The SNN operates by using binary spike signals to transmit information. When the membrane potential exceeds the threshold, the spiking neuron fires spike represented by 1; otherwise, there is no spike represented by O. This unique information processing paradigm energy-efficte multiplications of weights and activations with simple additions. Additionally, this information processing paradigm can be implemented on neuromorphic hardware Ma et al. (2017); Akooyan et al. (2015); Davies et al. (2018): PPi et al. (2019); Guo et al. (2023a), where the computational unit activates only when a spike occurs. This feature saves energy since the computational unit remains silent when there is spike. Studies have shown that an SNN can save orders magnitude more energy than its Artificial Neural Network (ANN) counterpart Akopyan et al. (2015); Davies et al. (2018).
since it replaces
an efficient event-driven-baal
computation manner
Equal Contributions. Corresponding author.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Staage
Stago 2
Stage 3
Lass
Porward Pass
Training Phase
Stage 1
Stage
Stage.3 3
Stagy 4
LIF
00000100
Inference Phase
Figure 1: The overall workflow of the proposed method. We add multiple shortcut branches from the intermediate layers to the output thus the gradient from the output could present to the shallow layers directly.
Although the SNN is energy-efficient, it is challenging to train it directly because the gradient of the firing spike process is not well-defined. This means that impossible to use gradient- maadienttbbsed optimization methods to train an SNN directly. To overcome this problem, researchers have proposed
various surrogate gradient training (SG) methods (Courbariaux
2016) Esser al. 2016 Bellec
et al. 2018 Rathi & Roy 2020) Wu et al., 2019; Neftei et al. 2019). These methods use an alternative function to replace the firing process during back pack-propagation. For example, (Bohte 2011). (Zenke
& Ganguli 2018), (Guo et al. 2022a), and (Cheng et al. [020 Guo et al. 2024] Zhang et al. 2024].
researchers used the truncated quadratic function, the si sigmoid function, the tanh-like function, and the rectangular function as surrogates, respectively. However, SG methods have an intrinsic problem: gradient vanishing. All surrogate functions are bounded, and their gradients would be close to in most intervals. As result, the gradient of the SNN would quickly decrease from output to input, causing the weights of the shallow layers of the SNN to freeze during optimization. In subsectionn 4.1 will theoretically and experimentally demonstrate the gradient vanishing problem.
To address this problem, we propose shorteut back-propagation method in this paper, which involves transmitting the gradient from the loss to the shallow layers directly. To achieve this, we add multiple shorteut branches from intermediate layers to the output in the network. This allows information from the shallow layers to reach the output and final loss directly. Consequently, the gradient from the output can present in the shallow layers, and their weights can be updated adequately, resulting in improved accuracy. Importantly, these shortcut branches can be removed without introducing any burden during the inference phase. Our proposed training framework essentially joint optimization problem the weighted sum of the loss functions associated with these shortcut branches. However, we give more weight to the main branch net, the earlier layer weights may not be updated sufficiently. Conversely, if we give more attention to the side shortcut branches, the accuracy cannot reach a high level since it directly relates to the main branch outputs rather than the side branch outputs. To balance this conflict, During early training, we pay more attention to the side branch net, allowing sufficient weight updates of the shallow layers. Towards the end of training, we increase the weight given the main branch net, which further improves final accuracy. We accomplish this by inducing dynamically changing balanced coefficient that adjusts with each training epoch. To illustrate our method's workflow, please Our paper provides several key contributions, which can be summarized as follows:
introduce evolutionary training framework.
refer to Figure
Firstly, we have identified that the gradient vanishing problem is significant issue for SNNs. We have supported this conclusion with theoretical justifications and in-depth experimental analysis. To mitigate this problem, we have proposed the shortcut back- propagation approach, which is a simple yet effective method. Importantly, it will not introduce any additional burden during the inference phase.

---

Secondly, we have proposed an evolutionary training framework that balances the weights of these branches with gradual strategy. This approach ensures that earlier layer weights can be adequately updated while also improving overall accuracy.
Lastly, we have evaluated the effectiveness and efficiency of our proposed methods on both static (CIFAR- 10, CIFAR-100, ImageNet) and spiking (CIFARIO-DVS) datasets with widely used backbones. Our results demonstrate that the SNN trained with our proposed method is highly effective, achieving a top-1 accuracy of 77.79% on CIFAR-100 using ResNet19 with only 2 timesteps. This represents significant improvement of about 3.3% compared with the current state-of-the-art SNN models even with more timesteps.
2 Related Work
2.1 Learning of Spiking Neural Networks
Unsupervised learning Dichl & Cook (2035; Hao et al. (2020), converting ANN to SNN
(ANN2SNN) Sengupta et al. (2019): Hao et al. 2023a b). and supervised learning Li et al. (0021); Guo et al. (202c)) are three commonly y used learning paradigms for SNNs. In unsupervised learning, the gppike-timing-deeee
plasticity (STDP) approach Lobov et al. (2020) is utilized to update the SNN model, which is considered more biologically plausible. However, due to the lack of explicit task guidance, this method is typically limited small-scale networks and datasets. The ANN-SNN
conversion method Han & Roy (2020); Li et al. (2021a) :Bu et al.] (20TT) Ho & Chang 2021): Bu
al. 2022); Hao et al. (2023a); Lan et al. (2023) involves training ANN first and then converting it into homogeneous SNN by transferring the trained weights and replacing the ReLU neuron with a
temporal spiking neuron. However, this method is not suitable for neuromorphic datasets as the ReLU neuron in the ANN cannot capture the rich temporal dynamics required for sequential information. Supervised learning Fang et al. (2021a); Wu al. (2018), on the other hand, adopts an alternative function during pack-propagation to replace the firing process, enabling direct training of the SNN as an ANN. This approach leverages the success of gradient- based optimization and can achieve good performance with only a few time steps, even on large-scale datasets. Moreover, supervised learning can handle temporal data effectively, making it an increasingly popular choice in SNN research. Our work also falls within this domain.
2.2 Relieving Training Difficulties for Supervised Learning of SNNs
As mentioned earlier, the surrogate gradient (SG) approach is commonly employed to address the non-differnntiability of SNNs. Various SG functions have been utilized, including the truncated quadratic function (Bohte| (01T). the sigmoid function (Zenke & Ganguli 2018), the tanh-like function (Guo et al. 2022a), and the rectangular function (Cheng et al.j 2020). While the SG method is generally effective, it can also introduce certain issues. Firstly, there is gradient mismatch between the true gradient and the surrogate gradient, resulting in slow convergence and reduced accuracy. To tackle this problem, IM-Loss (Guo et al. 2022a) proposed a dynamic manual SG method that adapts with each epoch, ensuring sufficient weight updates and accurate gradients simultaneously. In contrast to this manual design, the Differentiable Spike method (Li et al., 2021b) and the differentiable SG search method determine the optimal gradient estimation using finite difference and NAS techniques, respectively. Secondly, due to the firing function being bounded, all these SG functions are also bounded. As a result, the gradient approaches or reaches close to zero in most intervals, exacerbating the vanishing gradient problem in SNNs. To mitigate this issue, SEW-ResNet (Fang al. 202 a) suggested using the ResNet with activation before addition form, while MS-ResNet (Hu et al. 2021 advocated for the ResNet with pre-activation form. Additionally, normalization techniques have been employed to address the vanishing/cxplosiie batch normalization (tdBN) (Zheng al. 202T) normalized the data along both the channel and temporal dimensions. Other techniques such as Temporal Batch Normalization Through Time (BNTT) (Kim & Panda (02T). postsynaptic potential normalization (PSP-BN) (Ikegawa et al., (2022). and temporal effective batch normalization (TEBN) (Duan et al., 2022) recognized the significant variation in spike distributions across different timesteps, and thus regulated spike flows by applying separate timestep batch normalization. MPBN (Guo et al., 2023b) introduced an additional batch normalization step after the membrane potential updating function to reestablish data flow. Similarly, regularization loss has been utilized to alleviate gradient ixplossio/vaanshhig problems. In RecDis-
gradient problems. For example, Threshold-dependent
3

---

SNN (Guo et al. 2022c), membrane potential regularization loss was proposed to control spike flow within an appropriate range. In Spiking PointNet (Ren et al. 2023). trained-less but learning- more paradigm was proposed. This method can be seen as using a small network in the training to mitigate the training difficulty problem.
However, all these methods still need to present the gradient from the output layer to the first layer step by step, thus the gradient vanishing problem cannot be solved completely. In this paper, we propose a shortcut back-propagation method. Different from the above methods, we present the gradient from the output layer these shallow layers directly, thus the gradient vanishing problem can be solved totally.
3 Preliminary
The spiking neuron serves as the fundamental and specialized computing unit in SNNs, drawing inspiration from the human brain, In our paper, we employ the widely used spiking Leaky-Integrate- and-Fire (LIF) neuron model. This model accurately captures the behavior of biological neurons by considering the interaction between the membrane potential and input current. To show the spiking neuron in detail, we introduce the notation first. Throughout the paper, we denote the vectors in bold italic letters. For instance, we use the x and o to represent the input and target output variables. We denote the matrices or tensors by bold capital letters (e.g., W is for weights). We denote the constants by small upright or downright letters. For example, u means the i-th membrane potential at time step t. Then, the LIF neuron can be described as follows:
u (+++)..pre (f) + c(++1), where (+++)) (x.t-)
(1)
where r a constant within (0. 1), which controls the leakage of membrane potential. When 7 1, the neuron will degenerate to the Integrate-and-Fire (IF) neuron model. In the paper, we set r as 0.5. u (t+- is the pre-synaptic input at time step t + 1, which is charged by the input current (t++1) Note that we omit the layer index for simplicity. The input current is computed by the dot-product between the weights, W of the current layer and the spike output, (x(++1 from the previous layer. Once the membrane potential, u (fr exceeds the firing threshold Vih spike will be fired from the LIF neuron, given by
(t+1)_ [1 if u (t+ Vin
10 otherwise
(2)
n (1+1) u (1+)).pr (1 ((+1)).
After firing, the spike output of (t+1) will propagate to the next layer and become the input (+++ 1) of the next layer. In the paper, set Vn, as 1.
There notorious problem SNN training the firing function undifferentiable. To demonstrate this problem, we formulate the gradient by the chain rule, given
aL OW
aL do(t) (1.
(
(t+))p) Oul (t)pp)p (t+1)))
aL
(1.
OW
Since the firing function Equation is similar to the sign function. The Fu is 0 almost everywhere except for the threshold. Therefore, Thereforr,, the updates for weights would either be infinity if we use the actual gradient of the firing function.
(o
4 Methodology
4.1 The Gradient Vanishing Problem for SNNs
As aforementioned, the non-differentiabiiity of SNNs poses challenges when training them directly. To address this issue, researchers have proposed the use of surrogate gradients. In this approach, the firing function remains unchanged during the forward pass, but surrogate function employed during the backward pass. The surrogate gradient is then computed based on this surrogate function.

---

0..00300.002-000 0.000 0.001 0.002 0.003
1.003-0.022-010 0.000 0.001 0.002 0.003
(a)
(b)
Figure The gradient distributions the first layer for Spiking ResNet34 on CIFAR- 100. (a) and (b)
show the distributions for the vanilla model and the one with the shortcut back-propagation method.
There are three commonly used surrogate gradients:
(0(
max 0. - 10,000
- 11) 9
Butt).s
Vol
(
(o(())
sign
(1
(12e (t).pr Via
<
2
(4)
(g(
k<1 tanh (9)pre
Vih)))
(t(")pp
Each of these surrogate gradients includes hyperparameter that controls the sharpness and width of the gradient. However, it is evident that these gradients, their approximations, often become close to zero over a significant portion their intervals. Consequently, this poses a considerable challenge in terms of severe gradient vanishing. While residual blocks have proven effective in mitigating gradient vanishing problems traditional neural networks, their performance not optimal when applied to SNNs. To demonstrate this, we express the skip connection using the following formulation:
g(f(x)) +))
(5)
where f(-) is convolutional layers and g(-) is the activation function. The standard ResNet network composed of multiple skip connection blocks cascaded together. In ANN, ReLU is used for g(-), since ReLU is unbounded for the positive part, the gradient can be passed the input of the block directly. However, in the case LIF neurons in SNNs, the gradient will be reduced through the surrogate gradient. Thus, the standard skip connections still suffer the gradient vanishing problem in SNNs. To visually illustrate this problem, we show the gradient distributions of the first layer for Spiking ResNet34 with timesteps on the CIFAR-101 100 in the Figure 2 a). It can be seen that these gradients are close to 0 in most intervals, meaning the gradient vanishing problem is very significant for these shallow layers.
4.2 The Shortcut Back-propagation Method
Both theoretical analysis and experiments reveal the severity of the gradient vanishing problem in shallow layers of SNNs. In this paper, we propose a shortcut back-propngation method to address this issue. Specifically, the network is divided into several blocks, and we add multiple shortcut branches directly from these blocks to the output, as shown in in/Figure These blocks are then followed by global average pooling layer and a fully connected layer, resulting in the final output:
Oäinl Σ b())),
(6)
where the br(x)) represents the output of the 1-th block. While the original final output can be expressed as
Oinal by (x), where by(x) = (are)=f((bbaa f(bb
(7)
In the above equation, the n is the total number of the blocks and fi(-) is the network the 1-th block. To demonstrate how method alleviates the gradient vanishing problem, let's consider the gradient of the weight in the first layer as an example. For the original case, can be expressed as
ØL
aT.
Obn (x)
1 (b)()))
(8)
aw,
(x)) Ob,, (x)
(b)(x) OW1
5

---

Algorithm Training and inference procedure of SNN with our method.
Training
Input: An SNN to be trained; Initial balance coefficient X; training dataset; total training iteration:
Terain-
Output: The well-trained SNN.
1: for all = 1,2,...... Atrain iteration do
2: Get mini-batch training data, (in and class label, y());
3: Feed the Tin(1 (i) into the SNN and calculate every block output by (zmn and the final main
net output b,, (i));
4: Update X;
5: Calculate the final output by Equation 10
6: Compute classification loss LCF. LCF Orimal (i), y v()):;
7: Calculate the gradient w.r.t. W by [Equatio 9
al where learning rate. 8: Update W: (W W naw)
9: end for
Inference
Input: The trained SNN; test dataset; total test iteration: Lestt
Output: The result.
1: for all i 2, ......... Lest iteration do
2: Get mini-batch test data, 2mn (i) and class label, yri) in test dataset;
3: Feed the Zia (i) into the trained SNN;
4: Calculate the final main net output bn (1));
5: Calculate the final output, (ina by (Zin (i))
6: Compare the final output Ofimal (i) and y compute the classification result.
7: end for
(()) Since the (b) is always reduced through the surrogate gradient, the aw, becomes very small, resulting insufficient weight updates. However, for our method, it can expressed as
AL
aL aW1
θL (br())
(9)
(b) OW1
In our method, the gradient directly fed into the first block and subsequently to W1. This completely solves the gradient vanishing problem. To further illustrate this advantage. visualize the gradient distribution of the first layer for Spiking ResNet34 on CIFAR-10 100 in /Fiiuure b). It can be observed that the distribution is relatively flat, indicating that the gradient vanishing problem has been effectively addressed these shallow layers. Moreover, these shortcut branches can be removed during inference, incurring additional cost.
4.3
The Evolutionary Training Framework
While the shortcut back- pook-propngation method effectively addresses the gradient vanishing problem, introduces potential conflict. Each shortcut branch contributes to the final output, but if we focus too much on the shallow layer outputs, the overall accuracy may suffer. This because the shortcut branches removed during the inference phase, and the final accuracy is primarily influenced by the main branch. On the other hand, we prioritize the main branch output, the final loss may not capture enough information from the shallow layers. Consequently, the weights of these shallow layers may not be updated adequately.
To address this issue, we propose an evolutionary training framework. During the early stages of training, we prioritize the former side branch net, allowing for sufficient weight updates in the shallow layers. As training progresses, we gradually shift our focus to the main branch net until all attention on the main net at the end training. To achieve this, we introduce a balance coefficient, denoted by X(i) and adopt strategy of decreasing to adjust it as follows,
Ounal bn (x: A(1) Σ (xx,, where A(1) X(1 i
(10)

---

Table 1: Ablation study for the shortcut back propagattonnn method.
Architecture
Method
Time-step
Accuracy
Vanilla Training
2
71.42%
Shortcut Back-propagation
2
73.68 %
Evolutionary Training
74.02 %
ResNet18
Vanilla Training
72.22%
Shortcut Back-propagation
4
74.78 %
Evolutionary Training
4
74.83 %
Vanilla Training
2
69. 69.82%%
Shortcut Back-propagation
2
74.06 %
Evolutionary Training
74.17% %
ResNet34
Vanilla Training
4
69.98%
Shortcut Back- prop-proopgttion
4
75.67 %
Evolutionary Training
4.
75.81 %
In the equation mentioned above, represents the total number of training iterations, denotes the current training iteration, and À is constant. In our work, we set to a value of 0.25, The training and inference of SNN detailed in Algo.
Experiment
We conduct extensive experiments CIFAR-10(100) Krizhevsky et al. (2010), ImageNet [Dnng et al.) (2009). and DIFARIO-DVSIL The CIFAR- 10R (100) dataset comprises 50k training images and 10k test images, divided into 10(100) classes, each with 32 x 32 pixels. CIFARIO-DVS is a converted dataset derived from CIFAR-10. 10. It consists of 10k images, with images per class, 10 classes. ImageNet dataset, containing over 1, 1,250k training images and 50k test images. For these static datasets (CIFAR- 10, CIFAR-100, and ImageNet), we applied data normalization to ensure that they have mean and variance. Additionally, to prevent overfitting, we performed random horizontal flipping and cropping on all these datasets. For a fair comparison, AutoAugment Cubuk et al. (2018) was also used for data augmentation following these work Guo et al. (2022b); Li et al. (202 lb) CIFAR-10(1O0) Regarding the CIFAR I0-DVS dataset, we partitioned into 9k training images and IK test images, as described in Wu al. 2019). The training image frames were resized 48 48 as in [Zheng et al. (2021). Random horizontal flipping and random roll within range of 5 pixels were also applied during training. For the test images, simply resized them to 48 x 48 without any additional processing, following the approach of Li et al. Li (0011)) We run the model with V100.
al. 2017 to demonstrate the superior performance of our method.
significantly larger
5.1 Ablation Study
To validate the effectiveness the proposed shortcut back-propagation method, we initially conducted dataset using ResNet18 and ResNet34 with different timesteps. The results are detailed in Table For ResNet18, the accuracy achieved through vanilla training is 71.42% and 72.22% under and 4 timesteps, respectively, which aligns with existing works. Upon applying our shortcut back pack-propagation method, the accuracy improved to 73.68% and 74.78%, marking notable 2.5% enhancement. Furthermore, with the evolutionary training method. the performance of ResNet18 saw an additional improvement, reaching 74.02% and 74.83%, respectively, Under vanilla training, ResNet34 achieved accuracies of 69.82% and 69 69.98% with and timesteps, respectively. These results are actually worse than those obtained with ResNet18. This suggests that the deeper model does not exhibit better performance due to the significant gradient vanishing problem in SNNs. However, by utilizing our shortcut back peccpprppggttion method, the accuracy significantly improves to 74.06% and 75.67%, representing a remarkable 5.0% enhancement. Notably, these results surpass the performance of ResNetl 8 as well. This clearly demonstrates the effectiveness of our proposed method. Furthermore, when incorporating evolutionary training method, we observe further improvements in performance.
several ablation experiments the CIFAR- 100
7

---

Table 2: Comparison with SoTA methods CIFAR-10(100).
Dataset
Method
Type
Architecture
Timestep
Accuracy
TL Wu al.
a02
Tandem Learning
CIFARNet
89.04%
PTL Wu al. 202Tb
Tandem Learning
VGG11
16
91.24%
PLJF Fang et al. 2021b
FLIFFann
SNN training
PLIFNet
93.50%
DSR Meng et al.
2022
SNN training
ResNer18
95.40%
KDSNNNXU
2025
SNN training
ResNet18
93.41%
RocDis-SNN Guo ct al.
2022c
SNN training
ResNet19
93.64%
91.78%
Diet-SNN Rathi & Roy
2020
SNN training
ResNer20
92.54%
93.13%
Dspike Li et al. 2021b
SNN training
ResNer20
93.66%
92.34%
CIFAR-10
STBP-ddNN Zheng et al.
2021
SNN training
ResNet19
92.92%
94.16%
TET Deng et al. 2002
SNN training
ResNet19
94.44%
ResNet19
95.31%
Real Spike Guo et al.
2022d
SNN training
ResNet20
91.89%
93.89 % 10.11
ResNer18
94.30 % +.0.9
Shortcut Back-propagation
SNN training
94,47 % 1.009
ResNet19
95.19% +5.19% +00.0
93.92% +0.08
ResNet18
94.46 % #0.11
Evolutionary Training
SNN training
100113
ResNet19
95.36 % 1.0.10
T2FSNN Park al.
2020
ANN2SNN
VGG16
680
68 80%
Real Spike Guo et a.
2022d
SNN training
ResNet20
66.60%
Yang et 2022
Tandem Learning
ResNet20
31
76.08%
Diet-SNN Ruthi & Roy
2020
SNN training
ResNet20
64.07%
RecDis-SNNy Guo
2022c
SNN training
ResNer19
74.10%
71.68%
Dspike T.i 20
SNN training
ResNer20
73.35%
CIFAR-100 100
TET Deng 2022
SNN training
ResNet19
72.87%
74.47%
73.68 %e 100.100
ResNet18
74.78 % 1.0008
Shortcut propappoongn
SNN training
75.75 % 10.10
ResNet19
77.56 % 1:0.13
74.02% 10.09
ResNet18
74.83% +00.11
Evolutionary Training
SNN training
75.82% 1..5%%FF.. 10
ResNet19
77.79 % t.0.0.08
5.2 Comparison with SoTA Methods
In this section, we conducted a comparative experiment for the shortcut pack-propagttion method and the evolutionary training framework, taking into consideration several state-of-the-a of-the-art works. To ensure a fair comparison, we present the top-1 1 accuracy results based on 3 independent trials. We first evaluated our method on CIFAR-10 and CIFAR-I00 datasets. The AdamW optimizer was employed with a learning rate of 0.01 which is cosine decay to and a weight decay of 0.02, Throughout the training process, all models were trained using batch size of 128 for total of 300 epochs. For the CIFAR- 10 dataset, we chose SpikeNorm (Sengupta 2020). TSSL-BP Zhang & Li 2020 TL Wu et al. (2021a), la), (202 DSR Meng et al. 2022). KDSNN Xu et al. (2023), al. (202b)) ddBP-dBB Zheng
The results are summarized in Table Rathi al.
al. (2019), Hybrid-
PTL Wu et al. (2021b) PLIF Fang et al.
(021), TET Deng et al. (022). RecDis SEDDis-SNNN SNN Guo et al. (202cc), and Real Spike Guo et al. (202d)) as our comparison. Previous works utilizing ResNet18, ResNet19, and ResNet20 as backbones achieved the highest accuracies of 95,40% 95.31% and 93.66% with 20, 2, and 4 timesteps respectively. While our method based on 18 and ResNet19 could reach 93.92% and 95. 95.36% with 4 and 2 timesteps, respectively. Note that, ResNet1 8 is smaller than ResNer20. On the CIFAR- 100 dataset, our method can also achieve better accuracy than other prior sfate-of-the-art works with fewer timesteps. For instance, our ResNet19 model with only 2 timesteps outperforms the current best method, TET and
Diet-SNN Rathi & Roy (020). Dspike

---

Table 3: Comparison with SoTA methods ImageNet.
Method
Type
Architecture
Timestep
Accuracy
TET Deng el al. (0222
SNN training
ResNet34
64.79%
RecDis-SNNN Gto
2022c
SNN traiming
ResNet34
6
67.33%
GLIF Yao al. 2022
SNN training
ResNet34
4
67.52%
DSR
Meng
4028
SNN training
ResNet18
50
67.74%
MS-ResNer Hu et al
2023
SNN training
ResNet18
6
63. 60%
MPBN Guo al. 2023b
SNN training
ResNet18
4
63.14%
ResNet34
64.71% %
ResNeu18
63.68%
Real Spike Guo
Guo et al. 2022d
SNN training
ResNe134
67.69%
ResNet18
63.18%
SEW ResNet Fang et 2021a
SNN training
ResNet34
67.04%
ResNet18
64.47 % +0221
Shorfcut pack-propagation
SNN training
ResNet34
4
67.90% =00 17
ResNet18
60.12
Evolutionary Training
SNN training
ResNe134
68.14% 10.15
Table 4: Comparison with SoTA methods on CIFARIO-DVS.
Method
Type
Architecture
Timestep
Accuracy
STBP-ddNN Zheng et al.
2021
SNN training
ResNet19
10
67.80%
RecDis-SN SNN Guo el al.
20220
SNN training
ResNet19
10
72.42%
Real Spike Guo
(2022d
SNN training
ResNet19
10
72.85%
Dspike
2021b
SNN training
ResNet18
10
75.40%
Shorteut prop-proonnniion
SNN training
ResNer18
10
$2.00%:0..1
Evolutionary Training
SNN training
ResNet18
10
83.30 % 10.10
ReeDis-SNN even with timesteps by about 3.3%. These experimental results clearly demonstrate
the efficiency and effectiveness of our method.
We proceeded to conduct experiments on the ImageNet dataset, which more complex dataset than CIFAR. The learning rate is adjusted to 4e here. The comparative results are presented in Table 3 Notably, there have been several state-of-the-art (SoTA) baselines proposed for this dataset recently. such as RecDis-SNe Guo et al. (022c). GLIF Yao et al. (2022). DSR Meng et al. (2022). MPBN Guo al. (2023, Real Spike Guo al. (2022d), and SEW ResNet Fang et al. a It is important to note that Real Spike and SEW ResNet deviate from the typical ResNet backbone as they generate integer outputs in the intermediate layers, making them more energy-intensive compared to methods with standard backbones. In contrast, our approach adopts the standard R8sNet18 and ResNet34 architectures, yet it still outperforms Real Spike and SEW ResNet. Specifically, our method achieves an accuracy of 12% and 18% using ResNet 8 and ResNet34, respectively, surpassing Real Spike by 1.44% and 0.45%, respectively. This improvement is noteworthy and demonstrates the effectiveness of our method in handling large-scale datasets.
al. 2023b),
In addition to the aforementioned experiments, we conducted tests on the highly popular neuromorphic dataset, CIFAR10 IV-D... We use the same the foundational architecture, which notably smaller compared to ResNet19, our approach achieved remarkable accuracies of 82.00% and 83.30%, respectively. These results demonstrate a substantial improvement over previous methodologies.
setting as CIFAR. Employing ResNet 8 as
6 Conclusion
In the paper, we proved that the Spiking Neural Network suffers severe gradient vanishing with theoretical justifications and in-depth experimental analysis. To mitigate the problem, we proposed a shortcut propaaarpagtiion method. This enables us present the gradient the shallow layers directly, thereby significantly mitigating the gradient vanishing problem. Additionally, this method does not introduce any burden during inference phase. we also presented an evolutionary training framework by inducing balance coefficient that dynamically changes with the training epoch, which could further improve the accuracy. We conducted various experiments to verify the effectiveness of method.

---

Acknowledgment
This work is supported by grants from the National Natural Science Foundation of China under contracts No.12202412 and No.12202413
References
Akopyan, F., Sawada, J., Cassidy, A., Alvarez-lcaza, R., Arthur, J., Merolla, P., Imam, N., Nakamura, Y., Datta, Nam, G.-J,, et Truenorth: Design and tool flow of 65 mw 1 million neuron programmable neurosynaptic chip. IEEE transactions on computer-aided design of integrated circuits and systems, 14(10):1537-1557 2015.
Bellec, G., Salaj, D., Subramoney, A., Legenstein, R., and Maass, W. Long short-term memory and learning-to-lerd to-larr in networks of spiking neurons. Advances in neural information processing systems, 31, 2018.
Bohte, M. Broorbbackpropggation in networks of fractionally predictive spiking neurons. In International Conference Artificial Neural Networks, pp. 60-68. Springer, 2011.
Bu, T., Fang, W., Ding, J., Dai, P., Yu, Z., and Huang, T. Optimal ann-smn conversion for high- accuracy and altra-low-latency spiking neural networks. International Conference on Learning Representations, 2021.
Bu, T., Ding, J., Yu, Z., and Huang, T. Optimized potential initialization for low-latency spiking neural networks. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pp. 11-20, 2022.
Cheng, X., Hao, Xu, J., and Xu, B. Lisnn: Improving spiking neural networks with lateral interactions for robust object recognition. In ICCAI, pp. 1519-1525, 2020.
Courbariaux, M., Hubara, I., Soudry, D., Yaniv,,,, R., and Bengio, Y. Binarized neural networks: Training deep neural networks with weights and activations constrained to+ or- 1. arXiv preprint 1IXx::1602.02830. 2016.
Cubuk, D.D Zoph, B., Mane, D., Vasudevan, V., and Le, Q. V. Autoaugment: Learning augmentation policies from data. arXiv preprint 000500000, 2018.
D.,
Davies, M., Srinivasa, N., Lin, T.-H., Chinya, G., Cao, Y., Choday, S. H., Dimou, G., Joshi, P., Imam, N., Jain, S., al. Loihi: A neuromorphic manycore processor with on-chip learning. leee Micro, 38(1):82-99, 2018.
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., and Fei-Fei, L. Imagenet: A lcage-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248 255. Icee, 2009.
Deng, S., Li, Y., Zhang, S., and Gu, S. Temporal efficient training of spiking neural network via gradient re-weighting. arXiv preprint arXiv:2202.1 1946, 2022.
Dichl, P. U. and Cook, M. Unsupervised learning of digit recognition using plasticity. Frontiers in computational neuroscience, 9:99, 2015.
Duan, C., Ding, J., Chen, S., Yu, 7.,, and Huang, T. Temporal effective batch normalization in spiking neural networks. In Oh, A. H., Agarwal, A., Belgrave, D., and Cho, K. (eds.), Advances in Neural Information Processing Systems, 2022. URL httppp:://opemeevee.eeeee//ooeoe LigyyQiJqqz
Esser, S. K., Merolla, P. A., Arthur, J. V., Cassidy, A. S., Appuswamy, R., Andreopoulos. A., Berg, D. J.. McKinstry, J. L., Melano, T., Barch, D. R., et al. From the cover: Convolutional networks for fast, effrgy-efficient neuromorphic computing. Proceedings of the National Academy of Sciences of the United States of America, 113(41): 1441, 2016.
Fang, W., Yu, 7.., Chen, Y., Huang, T., Masquelier, T., and Tian, Deep residual learning in spiking neural networks. Advances in Neural Information Processing Systems, 34:21056-211 2006,, 202
10

---

Fang. W. Yu, Z., Chen, Y., Masquelier, T., Huang, T., and Tian, Y. Incorporating learnable membrane time constant enhance learning of spiking neural networks. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 2661-2671 2021b.
Guo, Y. Chen, Y., Zhang, L., Liu, X., Wang. Y., Huang, X., and Ma, Z. IM-loss: Information
maximization loss for spiking neural networks. In Oh, A. H., Agarwal, A., Belgrave, D., and Cho, K. (eds.), Advances in Neural Information Processing Systems, 2022a. URL https:// openrevies net wetts/formmmdddd aorumiidd--aae 1aw4mmmdddw=wvv____ J=34v_84m2v
Guo, Y., Chen, Y. Zhang, L., Wang, Y., Liu, X., Tong, X., Ou, Y., Huang, X., and Ma. Z. Reducing information loss for spiking neural networks. In Avidan, S., Brostow, G., Cissé, M., Farinella, G. M., and Hassner, T. (eds.), Computer Vision ECCV 2022. pp. 36 52, Cham, 2022b. Springer Nature Switzerland. ISBN
Guo, Y., Tong, X., Chen, Y., Zhang, L.. Liu, X., Ma, Z., and Huang, X. Recdis-snn: Rectifying membrane potential distribution for directly training spiking neural networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 326-334 335, June 2022c.
Guo, Y. Zhang, L.. Chen, Y., Tong, X., Liu, X., Wang. Y., Huang, X., and Ma. Z. Real spike: Learning real-valued spikes for spiking neural networks. In Computer Vision-ECCC 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XII. pp. 52-68. Springer, 2022d.
Guo, Y., Huang, X., and Ma, 7. Direct learning-based deep spiking neural networks: a review. Frontiers in Neuroscience, 17:1209795, 2023a.
Guo, Y., Zhang, Y., Chen, Y., Peng, W., Liu, X., Zhang, L., Huang. X., and Ma, Z. Membrane potential batch normalization for spiking neural networks. In Proceedings of the IEEE/CVF International Conference Computer Vision (ICCV), pp. 19420-19430, October 2023b.
Guo, Y., Chen, Y., Liu, X., Peng, W., Zhang, Y., Huang, X., and Ma, Z. Ternary spike: Learning ternary spikes for spiking neural networks. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 12244- 12252,, 2024.
Han, B. and Roy, K. Deep spiking neural network: Energy efficiency through time based coding, In European Conference on Computer Vision, pp. 388-404 Springer, 2020.
Hao, Y., Huang, X., Dong, M., and Xu, B. A biologically plausible supervised learning method for spiking neural networks using the symmetric stdp rule. Neural Networks, 121:387-395, 2020.
Hao, Z., Bu, T., Ding, J., Huang. T., and Yu, Z. Reducing ann-snn conversion error through residual membrane potential. arXiv preprint ar Xiv:::2302.0 02091, 2023a.
Hao, Z., Ding, J., Bu, T., Huang, T., and Yu, Bridging the gap between anns and snns by calibrating offset spikes. arXiv preprint arXiv:2302. 0000022.0666 10685, 2023b.
Ho, N.-D. and Chang, I.-J Tcl: an ann-to-snn conversion with trainable clipping layers. In 2021 58th ACM/IEEE Design Automation Conference (DAC), pp. 793-798. IEEE, 2021.
Hu, Y. Wu, Y., Deng, L., and Li, G. Advancing residual learning towards powerful deep spiking neural networks. arXiv preprint arXiv:2112.08954 2021.
Hu, Y., Deng, L.,, Wu, Y., Yao, M., and Li, G. Advancing spiking neural networks towards deep residual learning, 2023.
Ikegawa, S.-i.,
Saiin, R., Sawada, Y., and Natori, N. Rethinking the role of normalization and residual blocks for spiking neural networks. Sensors, 22(8), 2022. ISSN 1424-8220. doi: 10.3390/s220828766 URL httph:ttts/stt h/wp::://wwww.. http:i..omm1422 8220/2220222222 22/0//226
Kim, S., Park, S., Na. B., and Yoon, S. Spiking-yolo: Spiking neural network for energy-efficient object detection, 2019.
Kim, Y. and Panda, Revisiting batch normalization for training low-latency deep spiking neural networks from scratch. Frontiers in neuroscience, pp. 1638, 2021.
11

---

Krizhevsky, A.. Nair, V., and Hinton, G. Cifar- 10 (canadian institute for advanced research), URL http://ww. cs. toronto. edu/kriz/cfaar. html, 5(4):1, 2010.
Lan, Y., Zhang, Y., Ma, X., Qu, Y., and Fu, Efficient converted spiking neural network for 3d and 2d classification. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 9211-9220, 2023.
Li, H., Liu, H., Ji, X., Li, G., and Shi, Cifar10-dvs: an event-stream dataset for object classification. Frontiers in neuroscience, 11:309, 2017.
Li, Y., Deng, S., Dong, X., Gong, R., and Gu, S. A free lunch from ann: Towards efficient, accurate spiking neural networks calibration. In International Conference on Machine Learning, pp- 2316-6325, PMLR, 2021a.
Li, Y., Guo, Y., Zhang, S., Deng, S., Hai, Y., and Gu. S. Differentiable spike: Rethinking gradient- descent for training spiking neural networks. Advances in Neural Information Processing Systems, 34:23426-23433 2021b.
Lobov, S. A., Mikhaylov, A. N., and Kazantsev, V. B. Spatial properties of stdp in self-learning spiking neural network enable controlling a mobile robot. Frontiers in Neuroscience, 14:-, 2020.
Ma, D., Shen, J., Gu, Z., Zhang, M., Zhu, X., Xu, X., Xu, Q. Shen, Y., and Pan, G. Darwin: A neuromorphic hardware co- co-processor based on spiking neural networks. Journal of Systems Architecture, 77:43-51, 51, 2017.
Meng, Q., Xiao, M., Yan, S., Wang, Y., Lin, Z., and Luo, Z.-Q. Training figh-performance low- latency spiking neural networks by differentiation spike representation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 12444-12453, 2022.
Neftei, E. O., Mostafa, H., and Zenke, F. Surrogate gradient learning in spiking neural networks: Bringing the power of gradient-based optimization to spiking neural networks. IEEE Signal Processing Magazine, 36(6):51 63, 2019.
Park, S., Kim, S., Na, B., and Yoon, T2fsnn: Deep spiking neural networks with time-to-first-spikk coding. In 2020 57th ACM/IEEE Design Automation Conference (DAC), pp. 1-6. IEEE, 2020.
Pei, J., Deng, L.., Song, S., Zhao, M., Zhang. Y., Wu, S., Wang, G., Zou, Z., Wu, Z., He, W., et al. Towards artificial general intelligence with hybrid tianjic chip architecture. Nature, 572(7767): 106-111, 2019.
Qu, Gao, 7., Zhang, T., Lu, Y., Tang. H., and Qiao, H. Spiking neural network for ultra-low-latency and accuraceurate object detection, 2023.
Rathi, N. and Roy, K. Diet-snn: Direct input encoding with leakage and threshold optimization in deep spiking neural networks. arXiv preprini arXiv:2008. 036588, 2020.
Rathi, N., Srinivasan, G., Panda, P., and Roy. K. Enabling deep spiking neural networks with hybrid conversion and spike timing dependent backpropagation. arXiv preprint arXiv:2005.0180) 2020.
Ren, D., Ma, 7., Chen, Y., Peng, W., Liu, X., Zhang, Y., and Guo, Y. Spiking pointnet: Spiking neural networks for point clouds. arXiv preprint arXiv:2310.062.:3 2023.
Sengupta, A., Ye, Y., Wang, R., Liu, C., and Roy. K. Going deeper in spiking neural networks: Vgg and residual architectures. Frontiers neuroscience, 13:95, 2019.
Wu, J., Chua, Y., Zhang, M., Li, G., Li, H., and Tan, K. C. A tandem learning rule for effective training and rapid inference of deep spiking neural networks. IEEE Transactions Neural Networks and Learning Systems, 202
Wu, J., Xu, C., Han, X., Zhou, D., Zhang, M., Li, H., and Tan, C. Progressive tandem learning for pattern recognition with deep spiking neural networks. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(1 1):72447840, 2021b.
Wu, Y., Deng, I.,, Li, G., Zhu, J., and Shi, Spatio-temporal backpropagation for training high- performance spiking neural networks. Frontiers in neuroscience, 12:331, 2018.
12

---

Wu, Y., Deng, L., Li, G., Zhu, J., Xie, Y., and Shi, L. Direct training for spiking neural networks: Faster, larger, better. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pp. 1311-1318, 2019.
Xiao, M., Meng, Q Zhang. Z., Wang, Y., and Lin, Z. Training feedback spiking neural networks by implicit differentiation on the equilibrium state. Advances in Neural Information Processing Systems, 34:145 16- 14528, 2021.
Xu, Q., Li, Y., Shen, J., Liu, J. K., Tang, H., and Pan, G. Constructing deep spiking neural networks from artificial neural networks with knowledge distillation. arXiv preprint vrXiv:2304.05627, 2023.
Yang, Q. Wu, J., Zhang, M., Chua, Y., Wang. X., and Li, H. Training spiking neural networks with local tandem learning. arXiv preprint arXiv:22/0.04532, 2022.
Yao, X., Li, R., Mo, Z., and Cheng, J. Glif: A unified gated leaky integrate-and-fire neuron for spiking neural networks. arXiv preprint arXiv:2210. 13768, 2022.
Zenke, and Ganguli, S. Superspike: Supervised learning in multilayer spiking neural networks. Neural computation, 1046)11514-1541 2018.
Zhang, W. and Li, Temporal spike sequence learning via backpropagation for deep spiking neural networks. Advances in Neural Information Processing Systems, 33: 12022-12003, 2020.
Zhang, Y., Liu, X., Chen, Y., Peng, W., Guo, Y., Huang, X., and Ma, Z. Enhancing representation of spiking neural networks via similarity-sensitive contrastive learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 16926-16934, 2024.
Zheng, H., Wu, Y., Deng, L., Hu, Y., and Li, G. Going deeper with directly-trainet -rained larger spiking neural networks. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pp. 11062-11070, 2021.
Zou, S., Mu, Y., Zuo, X., Wang. S., and Cheng, L. Event-based human pose tracking by spiking spatiotemporal transformer, 2023.
13

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes].
Justification: We clearly state the claims made and the contributions made in both the abstract and introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [NA].
Justification:
We find no limitation which we feel must be specifically highlighted here.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes].
14

---

Justification:
We provide the full set of assumptions and complete proofs in the Section 4.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes].
Justification:
We provide the detail experiment settings in the Section 5.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
15

---

Answer: [Yes].
Justification: We provide open access to the data and code with sufficient instructions in the supplemental material.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cm ec/
for more details.
While encourage the release of code and data, understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https for more details.
//nips. //nips.cc/pil cc/public/guu httpggidddeee/eebbbiiinneeee
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes].
Justification: All implementations described in the experiments section.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes].
Justification:
We report the mean as well as the standard deviation accuracy in experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
16

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes].
Justification:
The computation resources description is provided the experiment section.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question:
Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes].
Justification:
The research conducted with the NeurlPS Code of Ethics
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [No].
Justification: There is no societal impact of the work performed.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no socictal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
17

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA].
Justification:
The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes].
Justification: The original paper for datasets used are all cited.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
18

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA].
Justification:
We adopt public datasets.
Guidelines:
The answer NA means that the paper does not release new assets,
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA].
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA].
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
19


Reviews:
[{"review_id": "Zzay3MPqxE", "reviewer": "Reviewer_runr", "summary": "This paper proposes shortcut connections between layers to mitigate the gradient vanishing problem in SNNs. Additionally, the authors present a way to phase out the shortcut connections over training so that inference can be done without these additional connections. The experiments show that this method improves training performance in several image classification tasks.", "questions": "1.Does the proposed method lead to an increase in the calculation of gradient backpropagation? How much is the increased training time.", "limitations": "None.", "rating": 6, "confidence": 5, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "1.The idea is small, but interesting and effective enough.\n\n2.The performance improvement over the existing SNN methods is noticeable.\n\n3.The paper is well-written.", "weaknesses": "1.The proposed method will increase the training time.\n\n2.In the experimental section, some newer methods should be compared with this method.\n\n3.Figure 2 lacks horizontal and vertical coordinates, and the readability and comprehensibility of the picture need to be improved.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your efforts in reviewing our paper and your recognition of our interesting ideas, notable results, and good writing. The response to your questions is given piece by piece as follows.\n\nW**1**:  The proposed method will increase the training time.\n\n**A1**: Thanks for this question. The additional training time is trivial. Here we add the training time cost comparison of Vanilla SNN and our Shortcut Back-propagation SNN for 1 epoch training with batchsize as 128 on CIFAR10 using ResNet20 with a single V100. Only about 10% of training cost is induced.\n\n| Method | Timestep | Time |\n| --- | --- | --- |\n| Vanilla SNN | 2 | 22.17s |\n| Shortcut Back-propagation | 2 | 24.49s(10.46%) |\n| Vanilla SNN | 4 | 43.68s |\n| Shortcut Back-propagation | 4 | 47.67s(9.13%) |\n\n---\n\n**Q2**: In the experimental section, some newer methods should be compared with this method.\n\n**A2**: Thanks for the advice. We will add more recent methods like these in the below papers in the revised version.\n\n[1]Yao M, Zhao G, Zhang H, et al. Attention spiking neural networks[J]. IEEE transactions on pattern analysis and machine intelligence, 2023.\n\n[2] Qiu X, Zhu R J, Chou Y, et al. Gated attention coding for training high-performance and efficient spiking neural networks[C]. Proceedings of the AAAI Conference on Artificial Intelligence. 2024, 38(1): 601-610.\n\n---\n\n**Q3**: Figure 2 lacks horizontal and vertical coordinates, and the readability and comprehensibility of the picture need to be improved.\n\n**A3**: Thanks for the advice. We have improved the picture as you suggested. Please see the general response.\n\n---\n\n**W1**: Does the proposed method lead to an increase in the calculation of gradient backpropagation? Howmuch is the increased training time.\n\n**A1**: Thanks for this question. The additional training time is trivial. Here we add the training time cost comparison of Vanilla SNN and our Shortcut Back-propagation SNN for 1 epoch training with batchsize as 128 on CIFAR10 using ResNet20 with a single V100. Only about 10% of training cost is induced.\n\n| Method | Timestep | Time |\n| --- | --- | --- |\n| Vanilla SNN | 2 | 22.17s |\n| Shortcut Back-propagation | 2 | 24.49s(10.46%) |\n| Vanilla SNN | 4 | 43.68s |\n| Shortcut Back-propagation | 4 | 47.67s(9.13%) |", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "EbK7UH74sT", "reviewer": "Reviewer_6ctb", "summary": "This paper proposes a simple method to mitigate the gradient vanishing problem in the training of SNNs. This method introduces some early classification heads (including a pooling layer and a fully connected layer) to the SNN. Because the gradients from the early classification heads pass fewer surrogate gradients, this method aids the SNN in addressing the gradient vanishing problem. The authors also suggest an evolutionary training framework that changes the loss function to gradually adjust how important early classification head outputs are during the training phase. The proposed methods are only alive in the training phase and will not affect the inference phase of SNN.", "questions": "1. How is the network divided into multiple blocks? Are there any additional rules for the insertion position and number of early classification heads?\n2. The results of using short-BP to train ResNet 18 in Table 1 and Table 2 are quite different. There may be a transcription error here.", "limitations": "NA", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This proposed method partially alleviates the gradient vanishing problem in the training of SNN with surrogate gradients. Furthermore, the method has demonstrated excellent performance across multiple datasets. The Short-BP method can be easily integrated into the SNN training process without introducing excessive computational overhead. Furthermore, the evolutionary training framework effectively mitigates the short-BP problem, which may make the network pay more attention to early classification heads than the final SNN output. The writing in this paper is clear and concise.", "weaknesses": "1. In this paper, the author only demonstrates a change in gradient distribution in the first layer. Presenting the changes in the men and variance of the absolute gradients for each layer would provide a more direct proof of their argument.\n2. The author should provide a more detailed mathematical proof to explain why the use of surrogate gradients in deep SNN would lead to gradient vanishing, as well as why direct use of residual learning will not address the problem.\n3. The author has not demonstrated their method on much deeper network architectures where the gradient vanishing problem is more severe.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your efforts in reviewing our paper and your recognition of our simple method, notable results, and good writing. The response to your questions is given piece by piece as follows.\n\nW**1**:  In this paper, the author only demonstrates a change in gradient distribution in the first layer. Presenting the changes in the men and variance of the absolute gradients for each layer would provide a more direct proof of their argument.\n\n**A1**: Thanks for this advice. Here we add the changes in the men and variance of the absolute gradients for the first 10 layers in vanilla ResNet18 our ResNet18 respectively on the CIFAR-100. It can be seen that our ResNet18 could handle the gradient vanishing problem well.\n\n| Layer | Vanilla  | Ours |\n| --- | --- | --- |\n| 10 | 0.2094/0.0361 | 0.3846/0.0739 |\n| 9 | 0.1460/0.0158 | 0.2489/0.0432 |\n| 8 | 0.1136/0.0083 | 0.2001/0.0243 |\n| 7 | 0.0957/0.0056 | 0.1699/0.0173 |\n| 6 | 0.0753/0.0034 | 0.1344/0.0106 |\n| 5 | 0.0638/0.0023 | 0.1101/0.0070 |\n| 4 | 0.0513/0.0014 | 0.0889/0.0045 |\n| 3 | 0.0349/0.0007 | 0.0611/0.0021 |\n| 2 | 0.0197/0.0002 | 0.0341/0.0006 |\n| 1 | 0.0178/0.0001 | 0.0310/0.0005 |\n\n---\n\nW2:  The author should provide a more detailed mathematical proof to explain why the use of surrogate gradients in deep SNN would lead to gradient vanishing, as well as why direct use of residual learning will not address the problem.\n\n**A2**: Sorry for the confusion. **The** **residual structure cannot solve the gradient vanishing problem of SNNs completely.**  For standard skip connections, it can be expressed as $o=g(f(x)+x)$, where f(x) is convolutional layers and g() is the activation function. The standard ResNet network is composed of multiple skip connection blocks cascaded together. In ANN, ReLU is used for g(), since ReLU is unbounded for the positive part, the gradient can be passed to the input of the block directly. However, in the case of LIF neurons in SNNs, the gradient will be reduced through the surrogate gradient. Thus, the skip connections in standard ANNs still suffer the gradient vanishing problem in SNNs, as Figure 2 in the paper also visually illustrates this problem. However, our shortcut back method can transfer the gradient from the output to the input of the block directly in SNNs.\n\n---\n\nW3:  The author has not demonstrated their method on much deeper network architectures where the gradient vanishing problem is more severe.\n\n**A3**: Thanks for this advice.  We have added more experiments for deeper network architectures. It can be seen that our method also works well with these architectures.\n\n| Model | Accuracy |\n| --- | --- |\n| ResNet18 | 72.22% |\n| Our ResNet18 | 74.78% |\n| ResNet34 | 69.98% |\n| Our ResNet34 | 75.67% |\n| ResNet50 | 65.22% |\n| Our ResNet50 | 76.81% |\n| ResNet101 | 53.71% |\n| Our ResNet101 | 76.88% |\n| ResNet152 | 42.53% |\n| Our ResNet152 | 75.49% |\n\n**Q1**:  How is the network divided into multiple blocks? Are there any additional rules for the insertion position and number of early classification heads? \n\n**W1**: Thanks for the question. These modern architectures are usually partitioned into different stages corresponding to a downsample of the input feature map and an increase of the channel numbers. We add the branchs to the exit of these stages.\n\n---\n\n**Q2**:  The results of using short-BP to train ResNet 18 in Table 1 and Table 2 are quite different. There may be a transcription error here.\n\n**W2**: Thanks for the advice. We confused the results of the Evolutionary Training and the results of the Shortcut Back-propagation. We will correct it in the revised version. Thank you again.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ofpAsTv15H", "reviewer": "Reviewer_Jb1s", "summary": "The paper trains SNNs using surrogate gradient learning. In order to mitigate the gradient vanishing problem, the paper proposed the Shortcut Back-propagation method and utilizes an evolutionary algorithm framework to balance the training of shallow and deep layers. The effectiveness of the proposed method is demonstrated through many experiments.", "questions": "1)\tWhy are the bolded values not always the best values?", "limitations": "I find no limitation about the paper.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1)\tThe shortcut backpropagation method and the evolutionary training method are novel. \n2)\tThis paper can well handle the gradient vanishing problem.\n3)\tThe paper is well-written.\n4)\tThe paper shows the effectiveness of the proposed methods through many experiments.", "weaknesses": "1)\tThe author should add more mathematical proof to demonstrate that the mentioned residual structure in SNN is not very effective? The introduction of shortcut branches might add complexity to the network architecture, which could affect the interpretability of the model.\n2)\tSome recent SOTA works should be compared with too.  The authors can also compare with paper [1][2] which obtains really good results by MS-ResNet-18 backbone with 1 or 6 timesteps on large imageNet datasets.\n\n\n[1]Yao M, Zhao G, Zhang H, et al. Attention spiking neural networks[J]. IEEE transactions on pattern analysis and machine intelligence, 2023.\n\n[2] Qiu X, Zhu R J, Chou Y, et al. Gated attention coding for training high-performance and efficient spiking neural networks[C]. Proceedings of the AAAI Conference on Artificial Intelligence. 2024, 38(1): 601-610.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your efforts in reviewing our paper and your recognition of our novel method, notable results, and good writing. The response to your questions is given piece by piece as follows.\n\nW**1**:  The author should add more mathematical proof to demonstrate that the mentioned residual structure in SNN is not very effective? The introduction of shortcut branches might add complexity to the network architecture, which could affect the interpretability of the model.\n\n**A1**: Thanks for this advice. **The** **residual structure cannot solve the gradient vanishing problem of SNNs completely.**  For standard skip connections, it can be expressed as $o=g(f(x)+x)$, where f(x) is convolutional layers and g() is the activation function. The standard ResNet network is composed of multiple skip connection blocks cascaded together. In ANN, ReLU is used for g(), since ReLU is unbounded for the positive part, the gradient can be passed to the input of the block directly. However, in the case of LIF neurons in SNNs, the gradient will be reduced through the surrogate gradient. Thus, the skip connections in standard ANNs still suffer the gradient vanishing problem in SNNs, as Figure 2 in the paper also visually illustrates this problem. However, our shortcut back method can transfer the gradient from the output to the input of the block directly in SNNs.\n\nFurthermore, the proposed methods are only alive in the training phase and will not affect the inference phase of SNN.\n\n---\n\nW2:  Some recent SOTA works should be compared with too. The authors can also compare with paper [1][2] which obtains really good results by MS-ResNet-18 backbone with 1 or 6 timesteps on large imageNet datasets.\n\n**A2**: Thanks for this advice.  We will add comparisons of these recent methods in the revised version as your suggestion. Thanks.\n\n---\n\nQ1:  Why are the bolded values not always the best values?\n\n**A1**: Sorry for the confusion.  With the same model and same timesteps, our method reaches the best.  We will further clarify this in the revised version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
Thanks for your efforts in reviewing our paper and your recognition of our interesting ideas, notable results, and good writing. Here we provide the revised picture in the PDF.

Meta-review: This paper introduces a new method to address the vanishing gradient problem for training spiking neural networks. The method's performance is demonstrated with many experiments. All reviewers agree on acceptance.
Decision: Accept (poster)