Take A Shortcut Back: Mitigating the Gradient Vanishing for Training Spiking Neural Networks

Authors: yufei  guo
yuanpei  chen
zecheng  hao
weihang  peng
zhou  jie
yuhan  zhang
xiaode  liu
zhe  ma
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-13 04:41:54

Abstract: The Spiking Neural Network (SNN) is a biologically inspired neural network infrastructure that has recently garnered significant attention. It utilizes binary spike activations to transmit information, thereby replacing multiplications with additions and resulting in high energy efficiency. However, training an SNN directly poses a challenge due to the undefined gradient of the firing spike process. Although prior works have employed various surrogate gradient training methods that use an alternative function to replace the firing process during back-propagation, these approaches ignore an intrinsic problem: gradient vanishing. To address this issue, we propose a shortcut back-propagation method in the paper, which advocates for transmitting the gradient directly from the loss to the shallow layers. This enables us to present the gradient to the shallow layers directly, thereby significantly mitigating the gradient vanishing problem. Additionally, this method does not introduce any burden during the inference phase.
To strike a balance between final accuracy and ease of training, we also propose an evolutionary training framework and implement it by inducing a balance coefficient that dynamically changes with the training epoch, which further improves the network's performance. Extensive experiments conducted over static and dynamic datasets using several popular network structures reveal that our method consistently outperforms state-of-the-art methods.

Main paper: 
 introduction: 
 The Spiking Neural Network (SNN) has become a popular neural network due to its efficiency and has been widely used in various fields such as object recognition Li et al. (2021a) ; Xiao et al. (2021) , object detection Kim et al. (2019) ; Qu et al. (2023) , and pose tracking Zou et al. (2023) . The SNN operates by using binary spike signals to transmit information. When the membrane potential exceeds the threshold, the spiking neuron fires a spike represented by 1; otherwise, there is no spike represented by 0. This unique information processing paradigm is energy-efficient since it replaces multiplications of weights and activations with simple additions. Additionally, this information processing paradigm can be implemented in an efficient event-driven-based computation manner on neuromorphic hardware Ma et al. (2017) ; Akopyan et al. (2015) ; Davies et al. (2018) ; Pei et al. (2019) ; Guo et al. (2023a) , where the computational unit activates only when a spike occurs. This feature saves energy since the computational unit remains silent when there is no spike. Studies have shown that an SNN can save orders of magnitude more energy than its Artificial Neural Network (ANN) counterpart Akopyan et al. (2015) ; Davies et al. (2018) . Figure 1 : The overall workflow of the proposed method. We add multiple shortcut branches from the intermediate layers to the output thus the gradient from the output could be present to the shallow layers directly.Although the SNN is energy-efficient, it is challenging to train it directly because the gradient of the firing spike process is not well-defined. This means that it is impossible to use gradient-based optimization methods to train an SNN directly. To overcome this problem, researchers have proposed various surrogate gradient training (SG) methods (Courbariaux et al., 2016; Esser et al., 2016; Bellec et al., 2018; Rathi & Roy, 2020; Wu et al., 2019; Neftci et al., 2019) . These methods use an alternative function to replace the firing process during back-propagation. For example, in (Bohte, 2011) , (Zenke & Ganguli, 2018) , (Guo et al., 2022a) , and (Cheng et al., 2020; Guo et al., 2024; Zhang et al., 2024) , researchers used the truncated quadratic function, the sigmoid function, the tanh-like function, and the rectangular function as surrogates, respectively. However, SG methods have an intrinsic problem: gradient vanishing. All surrogate functions are bounded, and their gradients would be close to 0 in most intervals. As a result, the gradient of the SNN would quickly decrease from output to input, causing the weights of the shallow layers of the SNN to freeze during optimization. In subsection 4.1, we will theoretically and experimentally demonstrate the gradient vanishing problem.To address this problem, we propose a shortcut back-propagation method in this paper, which involves transmitting the gradient from the loss to the shallow layers directly. To achieve this, we add multiple shortcut branches from intermediate layers to the output in the network. This allows information from the shallow layers to reach the output and final loss directly. Consequently, the gradient from the output can be present in the shallow layers, and their weights can be updated adequately, resulting in improved accuracy. Importantly, these shortcut branches can be removed without introducing any burden during the inference phase. Our proposed training framework is essentially a joint optimization problem on the weighted sum of the loss functions associated with these shortcut branches. However, if we give more weight to the main branch net, the earlier layer weights may not be updated sufficiently. Conversely, if we give more attention to the side shortcut branches, the accuracy cannot reach a high level since it directly relates to the main branch outputs rather than the side branch outputs. To balance this conflict, we introduce an evolutionary training framework.During early training, we pay more attention to the side branch net, allowing sufficient weight updates of the shallow layers. Towards the end of training, we increase the weight given to the main branch net, which further improves final accuracy. We accomplish this by inducing a dynamically changing balanced coefficient that adjusts with each training epoch. To illustrate our method's workflow, please refer to Figure 1 . Our paper provides several key contributions, which can be summarized as follows:• Firstly, we have identified that the gradient vanishing problem is a significant issue for SNNs. We have supported this conclusion with theoretical justifications and in-depth experimental analysis. To mitigate this problem, we have proposed the shortcut backpropagation approach, which is a simple yet effective method. Importantly, it will not introduce any additional burden during the inference phase.• Secondly, we have proposed an evolutionary training framework that balances the weights of these branches with a gradual strategy. This approach ensures that earlier layer weights can be adequately updated while also improving overall accuracy.• Lastly, we have evaluated the effectiveness and efficiency of our proposed methods on both static (CIFAR-10, CIFAR-100, ImageNet) and spiking (CIFAR10-DVS) datasets with widely used backbones. Our results demonstrate that the SNN trained with our proposed method is highly effective, achieving a top-1 accuracy of 77.79% on CIFAR-100 using ResNet19 with only 2 timesteps. This represents a significant improvement of about 3.3% compared with the current state-of-the-art SNN models even with more timesteps.2 Related Work Wu et al. (2018) , on the other hand, adopts an alternative function during back-propagation to replace the firing process, enabling direct training of the SNN as an ANN. This approach leverages the success of gradient-based optimization and can achieve good performance with only a few time steps, even on large-scale datasets. Moreover, supervised learning can handle temporal data effectively, making it an increasingly popular choice in SNN research. Our work also falls within this domain. 
relieving training difficulties for supervised learning of snns: 
 As mentioned earlier, the surrogate gradient (SG) approach is commonly employed to address the non-differentiability of SNNs. Various SG functions have been utilized, including the truncated quadratic function (Bohte, 2011) , the sigmoid function (Zenke & Ganguli, 2018) , the tanh-like function (Guo et al., 2022a) , and the rectangular function (Cheng et al., 2020) . While the SG method is generally effective, it can also introduce certain issues. Firstly, there is a gradient mismatch between the true gradient and the surrogate gradient, resulting in slow convergence and reduced accuracy. To tackle this problem, IM-Loss (Guo et al., 2022a) proposed a dynamic manual SG method that adapts with each epoch, ensuring sufficient weight updates and accurate gradients simultaneously. In contrast to this manual design, the Differentiable Spike method (Li et al., 2021b) and the differentiable SG search method determine the optimal gradient estimation using finite difference and NAS techniques, respectively. Secondly, due to the firing function being bounded, all these SG functions are also bounded. As a result, the gradient approaches or reaches close to zero in most intervals, exacerbating the vanishing gradient problem in SNNs. To mitigate this issue, SEW-ResNet (Fang et al., 2021a) suggested using the ResNet with activation before addition form, while MS-ResNet (Hu et al., 2021) advocated for the ResNet with pre-activation form. Additionally, normalization techniques have been employed to address the vanishing/explosion gradient problems. For example, Threshold-dependent batch normalization (tdBN) (Zheng et al., 2021) normalized the data along both the channel and temporal dimensions. Other techniques such as Temporal Batch Normalization Through Time (BNTT) (Kim & Panda, 2021) , postsynaptic potential normalization (PSP-BN) (Ikegawa et al., 2022) , and temporal effective batch normalization (TEBN) (Duan et al., 2022) recognized the significant variation in spike distributions across different timesteps, and thus regulated spike flows by applying separate timestep batch normalization. MPBN (Guo et al., 2023b) introduced an additional batch normalization step after the membrane potential updating function to reestablish data flow. Similarly, regularization loss has been utilized to alleviate gradient explosion/vanishing problems. In RecDis-SNN (Guo et al., 2022c) , a membrane potential regularization loss was proposed to control spike flow within an appropriate range. In Spiking PointNet (Ren et al., 2023) , a trained-less but learning-more paradigm was proposed. This method can be seen as using a small network in the training to mitigate the training difficulty problem.However, all these methods still need to present the gradient from the output layer to the first layer step by step, thus the gradient vanishing problem cannot be solved completely. In this paper, we propose a shortcut back-propagation method. Different from the above methods, we present the gradient from the output layer to these shallow layers directly, thus the gradient vanishing problem can be solved totally. 
preliminary: 
 The spiking neuron serves as the fundamental and specialized computing unit in SNNs, drawing inspiration from the human brain. In our paper, we employ the widely used spiking Leaky-Integrateand-Fire (LIF) neuron model. This model accurately captures the behavior of biological neurons by considering the interaction between the membrane potential and input current. To show the spiking neuron in detail, we introduce the notation first. Throughout the paper, we denote the vectors in bold italic letters. For instance, we use the x and o to represent the input and target output variables. We denote the matrices or tensors by bold capital letters (e.g., W is for weights). We denote the constants by small upright or downright letters. For example, ui means the i-th membrane potential at time step t. Then, the LIF neuron can be described as follows:EQUATIONwhere τ is a constant within (0, 1), which controls the leakage of membrane potential. When τ is 1, the neuron will degenerate to the Integrate-and-Fire (IF) neuron model. In the paper, we set τ as 0.5. u (t+1),pre is the pre-synaptic input at time step t + 1, which is charged by the input current c (t+1) . Note that we omit the layer index for simplicity. The input current is computed by the dot-product between the weights, W of the current layer and the spike output, x (t+1) from the previous layer.Once the membrane potential, u (t+1),pre exceeds the firing threshold V th , a spike will be fired from the LIF neuron, given byo (t+1) = 1 if u (t+1),pre > V th 0 otherwise , u (t+1) = u (t+1),pre • (1 -o (t+1) ).(2)After firing, the spike output o (t+1) will propagate to the next layer and become the input x (t+1) of the next layer. In the paper, we set V th as 1.There is a notorious problem in SNN training the firing function is undifferentiable. To demonstrate this problem, we formulate the gradient by the chain rule, given asEQUATIONSince the firing function Equation 2 is similar to the sign function. The ∂o (t) ∂u (t),pre is 0 almost everywhere except for the threshold. Therefore, the updates for weights would either be 0 or infinity if we use the actual gradient of the firing function.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof? Answer: [Yes] .• It should be clear whether the error bar is the standard deviation or the standard error of the mean. • It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified. • For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates). • If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text. 
the gradient vanishing problem for snns: 
 As aforementioned, the non-differentiability of SNNs poses challenges when training them directly. To address this issue, researchers have proposed the use of surrogate gradients. In this approach, the firing function remains unchanged during the forward pass, but a surrogate function is employed during the backward pass. The surrogate gradient is then computed based on this surrogate function. 0.003 0.002 0.001 0.000 0.001 0.002 0.003 There are three commonly used surrogate gradients:       ∂o (t) ∂u (t),pre = γ max 0, 1 -u (t),pre V th -1 , ∂o (t) ∂u (t),pre = 1 a sign u (t),pre -V th < a 2 , ∂o (t) ∂u (t),pre = k(1 -tanh(u (t),pre -V th )) 2 .(4)Each of these surrogate gradients includes a hyperparameter that controls the sharpness and width of the gradient. However, it is evident that these gradients, or their approximations, often become close to zero over a significant portion of their intervals. Consequently, this poses a considerable challenge in terms of severe gradient vanishing. While residual blocks have proven effective in mitigating gradient vanishing problems in traditional neural networks, their performance is not optimal when applied to SNNs. To demonstrate this, we express the skip connection using the following formulation:EQUATIONwhere f (•) is convolutional layers and g(•) is the activation function. The standard ResNet network is composed of multiple skip connection blocks cascaded together. In ANN, ReLU is used for g(•), since ReLU is unbounded for the positive part, the gradient can be passed to the input of the block directly. However, in the case of LIF neurons in SNNs, the gradient will be reduced through the surrogate gradient. Thus, the standard skip connections still suffer the gradient vanishing problem in SNNs. To visually illustrate this problem, we show the gradient distributions of the first layer for Spiking ResNet34 with 4 timesteps on the CIFAR-100 in the Figure 2 (a). It can be seen that these gradients are close to 0 in most intervals, meaning the gradient vanishing problem is very significant for these shallow layers.Both theoretical analysis and experiments reveal the severity of the gradient vanishing problem in shallow layers of SNNs. In this paper, we propose a shortcut back-propagation method to address this issue. Specifically, the network is divided into several blocks, and we add multiple shortcut branches directly from these blocks to the output, as shown in Figure 1 . These blocks are then followed by a global average pooling layer and a fully connected layer, resulting in the final output:EQUATIONwhere the b l (x) represents the output of the l-th block. While the original final output can be expressed asEQUATIONIn the above equation, the n is the total number of the blocks and f l (•) is the network of the l-th block. To demonstrate how our method alleviates the gradient vanishing problem, let's consider the gradient of the weight in the first layer as an example. For the original case, it can be expressed as Compare the final output o final (i) and y(i) to compute the classification result. 7: end for Since the ∂b l+1 (x) ∂b l (x) is always reduced through the surrogate gradient, the ∂L ∂W1 becomes very small, resulting in insufficient weight updates. However, for our method, it can be expressed asEQUATIONEQUATIONIn our method, the gradient is directly fed into the first block and subsequently to W 1 . This completely solves the gradient vanishing problem. To further illustrate this advantage, we visualize the gradient distribution of the first layer for Spiking ResNet34 on CIFAR-100 in Figure 2 (b). It can be observed that the distribution is relatively flat, indicating that the gradient vanishing problem has been effectively addressed in these shallow layers. Moreover, these shortcut branches can be removed during inference, incurring no additional cost.While the shortcut back-propagation method effectively addresses the gradient vanishing problem, it introduces a potential conflict. Each shortcut branch contributes to the final output, but if we focus too much on the shallow layer outputs, the overall accuracy may suffer. This is because the shortcut branches are removed during the inference phase, and the final accuracy is primarily influenced by the main branch. On the other hand, if we prioritize the main branch output, the final loss may not capture enough information from the shallow layers. Consequently, the weights of these shallow layers may not be updated adequately.To address this issue, we propose an evolutionary training framework. During the early stages of training, we prioritize the former side branch net, allowing for sufficient weight updates in the shallow layers. As training progresses, we gradually shift our focus to the main branch net until all attention is on the main net at the end of training. To achieve this, we introduce a balance coefficient, denoted by λ(i) and adopt a strategy of decreasing to adjust it as follows, (2021) . Random horizontal flipping and random roll within a range of 5 pixels were also applied during training. For the test images, we simply resized them to 48 × 48 without any additional processing, following the approach of Li et al. Li et al. (2021b) . We run the model with 8 V100.EQUATION 
ablation study: 
 To validate the effectiveness of the proposed shortcut back-propagation method, we initially conducted several ablation experiments on the CIFAR-100 dataset using ResNet18 and ResNet34 with different timesteps. The results are detailed in Table 1 . For ResNet18, the accuracy achieved through vanilla training is 71.42% and 72.22% under 2 and 4 timesteps, respectively, which aligns with existing works. Upon applying our shortcut back-propagation method, the accuracy improved to 73.68% and 74.78%, marking a notable 2.5% enhancement. Furthermore, with the evolutionary training method, the performance of ResNet18 saw an additional improvement, reaching 74.02% and 74.83%, respectively. Under vanilla training, ResNet34 achieved accuracies of 69.82% and 69.98% with 2 and 4 timesteps, respectively. These results are actually worse than those obtained with ResNet18. This suggests that the deeper model does not exhibit better performance due to the significant gradient vanishing problem in SNNs. However, by utilizing our shortcut back-propagation method, the accuracy significantly improves to 74.06% and 75.67%, representing a remarkable 5.0% enhancement. Notably, these results surpass the performance of ResNet18 as well. This clearly demonstrates the effectiveness of our proposed method. Furthermore, when incorporating the evolutionary training method, we observe further improvements in performance. In this section, we conducted a comparative experiment for the shortcut back-propagation method and the evolutionary training framework, taking into consideration several state-of-the-art works. To ensure a fair comparison, we present the top-1 accuracy results based on 3 independent trials. We first evaluated our method on CIFAR-10 and CIFAR-100 datasets. The AdamW optimizer was employed with a learning rate of 0.01 which is cosine decay to 0 and a weight decay of 0.02. Throughout the training process, all models were trained using a batch size of 128 for a total of 300 epochs.The results are summarized in Table 2 . For the CIFAR-10 dataset, we chose SpikeNorm Sengupta et al. (2019 ), Hybrid-Train Rathi et al. (2020) , TSSL-BP Zhang & Li (2020) , TL Wu et al. (2021a) , PTL Wu et al. (2021b) , PLIF Fang et al. (2021b) , DSR Meng et al. (2022) , KDSNN Xu et al. (2023) RecDis-SNN even with 4 timesteps by about 3.3%. These experimental results clearly demonstrate the efficiency and effectiveness of our method.We proceeded to conduct experiments on the ImageNet dataset, which is a more complex dataset than CIFAR. The learning rate is adjusted to 4e -3 here. The comparative results are presented in Table 3 . et al. (2021a) . It is important to note that Real Spike and SEW ResNet deviate from the typical ResNet backbone as they generate integer outputs in the intermediate layers, making them more energy-intensive compared to methods with standard backbones. In contrast, our approach adopts the standard ResNet18 and ResNet34 architectures, yet it still outperforms Real Spike and SEW ResNet. Specifically, our method achieves an accuracy of 65.12% and 68.14% using ResNet18 and ResNet34, respectively, surpassing Real Spike by 1.44% and 0.45%, respectively. This improvement is noteworthy and demonstrates the effectiveness of our method in handling large-scale datasets.In addition to the aforementioned experiments, we conducted tests on the highly popular neuromorphic dataset, CIFAR10-DVS. We use the same hyper-parameter setting as CIFAR. Employing ResNet18 as the foundational architecture, which is notably smaller compared to ResNet19, our approach achieved remarkable accuracies of 82.00% and 83.30%, respectively. These results demonstrate a substantial improvement over previous methodologies. 
conclusion: 
 In the paper, we proved that the Spiking Neural Network suffers severe gradient vanishing with theoretical justifications and in-depth experimental analysis. To mitigate the problem, we proposed a shortcut back-propagation method. This enables us to present the gradient to the shallow layers directly, thereby significantly mitigating the gradient vanishing problem. Additionally, this method does not introduce any burden during the inference phase. we also presented an evolutionary training framework by inducing a balance coefficient that dynamically changes with the training epoch, which could further improve the accuracy. We conducted various experiments to verify the effectiveness of our method.NeurIPS Paper Checklist Justification: We find no limitation which we feel must be specifically highlighted here. Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
experiments compute resources: 
 Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?Answer: [Yes] .Justification: The computation resources description is provided in the experiment section.Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage. • The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute. • The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper). 
code of ethics: 
 Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines?Answer: [Yes] .Justification: The research conducted with the NeurIPS Code of Ethics Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed? 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] .Justification: The paper poses no such risks.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] .Justification: The original paper for datasets we used are all cited.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. 
 
 Appendix: Justification: There is no societal impact of the work performed.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.• The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 


Decision: This paper introduces a new method to address the vanishing gradient problem for training spiking neural networks. The method's performance is demonstrated with many experiments. All reviewers agree on acceptance.

Reviews: review: paper_summary: paper_summary: This paper proposes shortcut connections between layers to mitigate the gradient vanishing problem in SNNs. Additionally, the authors present a way to phase out the shortcut connections over training so that inference can be done without these additional connections. The experiments show that this method improves training performance in several image classification tasks.
strength_weakness: strength_weakness: 1.The proposed method will increase the training time.

2.In the experimental section, some newer methods should be compared with this method.

3.Figure 2 lacks horizontal and vertical coordinates, and the readability and comprehensibility of the picture need to be improved.
questions: questions: 1.Does the proposed method lead to an increase in the calculation of gradient backpropagation? How much is the increased training time.
limitations: limitations: None.
paper_summary: paper_summary: This paper proposes a simple method to mitigate the gradient vanishing problem in the training of SNNs. This method introduces some early classification heads (including a pooling layer and a fully connected layer) to the SNN. Because the gradients from the early classification heads pass fewer surrogate gradients, this method aids the SNN in addressing the gradient vanishing problem. The authors also suggest an evolutionary training framework that changes the loss function to gradually adjust how important early classification head outputs are during the training phase. The proposed methods are only alive in the training phase and will not affect the inference phase of SNN.
strength_weakness: strength_weakness: 1. In this paper, the author only demonstrates a change in gradient distribution in the first layer. Presenting the changes in the men and variance of the absolute gradients for each layer would provide a more direct proof of their argument.
2. The author should provide a more detailed mathematical proof to explain why the use of surrogate gradients in deep SNN would lead to gradient vanishing, as well as why direct use of residual learning will not address the problem.
3. The author has not demonstrated their method on much deeper network architectures where the gradient vanishing problem is more severe.
questions: questions: 1. How is the network divided into multiple blocks? Are there any additional rules for the insertion position and number of early classification heads?
2. The results of using short-BP to train ResNet 18 in Table 1 and Table 2 are quite different. There may be a transcription error here.
limitations: limitations: NA
paper_summary: paper_summary: The paper trains SNNs using surrogate gradient learning. In order to mitigate the gradient vanishing problem, the paper proposed the Shortcut Back-propagation method and utilizes an evolutionary algorithm framework to balance the training of shallow and deep layers. The effectiveness of the proposed method is demonstrated through many experiments.
strength_weakness: strength_weakness: 1)	The author should add more mathematical proof to demonstrate that the mentioned residual structure in SNN is not very effective? The introduction of shortcut branches might add complexity to the network architecture, which could affect the interpretability of the model.
2)	Some recent SOTA works should be compared with too.  The authors can also compare with paper [1][2] which obtains really good results by MS-ResNet-18 backbone with 1 or 6 timesteps on large imageNet datasets.


[1]Yao M, Zhao G, Zhang H, et al. Attention spiking neural networks[J]. IEEE transactions on pattern analysis and machine intelligence, 2023.

[2] Qiu X, Zhu R J, Chou Y, et al. Gated attention coding for training high-performance and efficient spiking neural networks[C]. Proceedings of the AAAI Conference on Artificial Intelligence. 2024, 38(1): 601-610.
questions: questions: 1)	Why are the bolded values not always the best values?
limitations: limitations: I find no limitation about the paper.

Comments: title: thanks
thanks
comment: Thanks very much for your reply and recognition. We are happy to see that your concerns have been addressed.
Thanks very much for your reply and recognition. We are happy to see that your concerns have been addressed.
Thank you for your response. All of my concerns have been addressed, and I'd like to increase my score.
Thank you for your reply. I think this is a nice bit of discussion and could be added to the manuscript.
In light of the additional discussion, I'd like to raise my score to a 6. This is an interesting piece of work and would be a nice addition to NeurIPS.