EnOF-SNN: Training Accurate Spiking Neural Networks via Enhancing the Output Feature

Authors: yufei  guo
weihang  peng
xiaode  liu
yuanpei  chen
yuhan  zhang
xin  tong
zhou  jie
zhe  ma
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-14 11:57:09

Abstract: Spiking neural networks (SNNs) have gained more and more interest as one of the energy-efficient alternatives of conventional artificial neural networks (ANNs). They exchange 0/1 spikes for processing information, thus most of the multiplications in networks can be replaced by additions. However, binary spike feature maps will limit the expressiveness of the SNN and result in unsatisfactory performance compared with ANNs. 
It is shown that a rich output feature representation, i.e., the feature vector before classifier) is beneficial to training an accurate model in ANNs for classification. 
We wonder if it also does for SNNs and how to improve the feature representation of the SNN.
To this end, we materialize this idea in two special designed methods for SNNs.
First, inspired by some ANN-SNN methods that directly copy-paste the weight parameters from trained ANN with light modification to homogeneous SNN can obtain a well-performed SNN, we use rich information of the weight parameters from the trained ANN counterpart to guide the feature representation learning of the SNN. 
In particular, we present the SNN's and ANN's feature representation from the same input to ANN's classifier to product SNN's and ANN's outputs respectively and then align the feature with the KL-divergence loss as in knowledge distillation methods, called L_ AF loss.
It can be seen as a novel and effective knowledge distillation method specially designed for the SNN that comes from both the knowledge distillation and ANN-SNN methods. Various ablation study shows that the L_AF loss is more powerful than the vanilla knowledge distillation method.
Second, we replace the last Leaky Integrate-and-Fire (LIF) activation layer as the ReLU activation layer to generate the output feature, thus a more powerful SNN with full-precision feature representation can be achieved but with only a little extra computation.
Experimental results show that our method consistently outperforms the current state-of-the-art algorithms on both popular non-spiking static and neuromorphic datasets. We provide an extremely simple but effective way to train high-accuracy spiking neural networks.

Main paper: 
 related work: 
 In view of the success of deep CNNs, the early development of deep SNNs benefits from the experience of CNNs. Using the trained ANN to guide the generation of high-precision SNN is one of the mainstream research at present. There are three main routes to realizing this idea. (2023a,b) . This method trains an accurate ANN with ReLU activation first and then uses the ANN network parameters to construct a special SNN network that enjoys the same structure as the ANN and adopts the average output through multiple timesteps as the final result. The converted SNN usually performs worse than the original ANN, since there are more or fewer conversion errors between the ANN and SNN. To bridge this gap as much as possible, many effective methods have been proposed, such as long inference time Girshick (2015) , threshold rescaling Sengupta et al. (2018) , soft reset Han et al. (2020) , threshold shift Li et al. (2021a) , and the quantization clip-floor-shift activation function Ding et al. (2021) . However, the conversion method is restricted to rate-coding and the Integrate-and-Fire (IF) model, and needs long timesteps to obtain a satisfied accuracy.Second, using the well-trained ANN model's parameters to initial the nascent SNN has been another fundamental line of research. For example, in Rathi et al. (2020) and Rathi & Roy (2020) , a trained ANN is converted to an SNN first and then the weights of the converted SNN are fine-tuned further with the training method. Initialed with the trained ANN, the SNN shows a faster convergence than that from random initialization. Though this method usually needs fewer timesteps than the conversion method, it does not show obvious advantages for deep models.Third, there are also some works that apply the knowledge distillation technique in the SNN field that to use ANN to distill the SNN Kushawaha et al. (2021) ; Takuya et al. (2021) ; Xu et al. (2023b,a) ; Guo et al. (2023c) ; Zhang et al. (2024) . However, as has been comprehensively and systematically studied in Tian et al. (2020) , most of the knowledge distillation methods perform poorly without well-designed hyper-parameters. These knowledge distillation methods for SNNs have also verified this consensus. They all adopt complex multi-stage distillation manner and only report the results on small-scale datasets. Providing a simple but effective knowledge distillation for the SNN is difficult.Despite the above methods providing us with a lot of valuable insights, we still think there are some other improvements worth further consideration. First, these prior methods merely use the parameters of the trained ANN to guide the designing of the parameters of the SNN or adopt a complex framework to transfer knowledge from ANN to SNN individually, designing a simple and effective knowledge distillation method specifically for SNNs combining these useful experience of SNN knowledge distillation and ANN-SNN methods simultaneously are ignored. Second, these SNN methods all adopt similar architectures as ANNs that only modify the ANN activation layers as spike neuron layers simply but without considering the advantages and disadvantages of the SNNs comprehensively. In the paper, aiming at improving the output feature representation of the SNN, we will take care of these two problems well. 
preliminary and methodology: 
 This section first introduces the proposed method of aligning the SNN's output feature with the ANN's output feature. Then the Leaky Integrate-and-Fire (LIF) neuron model, why its output feature's expression ability is limited, and how to replace the LIF activation layer with the ReLU activation layer will be introduced in detail. Finally, some key details for training the SNN will be given.To improve the output feature representation ability, we advocate using the well-trained ANN to guide and improve the learning of the output feature representation of the SNN. A natural idea is to align the output feature of the SNN to that of its ANN counterpart directly like knowledge distillation methods, given by arg minEQUATIONWhere W s , h a , and h s represent the parameters of the SNN, the output feature of the ANN, and the output feature of the SNN.However, we found that this natural idea does not always work, which has also been validated by Tian et al. (2020) . The work has investigated most of the famous knowledge distillation methods and found that they perform poorly without well-designed hyper-parameters.The reason why direct distillation performs worse in SNNs comes from that: 1) discrete output feature of the SNN makes its representation space much different from that of the full precision ANN, and directly aligning their output features simply will ignore the difference of their representation spaces and induce too much constraint in the SNN optimization, which may lead to a configuration far from the global optimal; 2) this design also treats each dimension in the feature space independently, and ignores the high-level interdependencies of the feature representation; and 3) it does not take into account the classification task at all, which conflicts with the final optimization goal.To consider the above problems comprehensively, we present a simple but effective method, that uses the ANN's classifier to guide the learning of the SNN's output feature. This is inspired by some ANN-SNN methods that directly copy-paste the weight parameters from trained ANN to homogeneous SNN can obtain a well-performed SNN. Hence, we guess that utilizing a homogeneous ANN's some elements to guide the learning of SNN may be useful. In specific, we fed the SNN's output feature to the ANN's pre-trained classifier to generate the output, P s . At the same time, we obtain the ANN's output, P a from the same input image as the SNN. Then we utilize the KL-divergence loss to align the student output to the teacher output as follows:EQUATIONIn this situation, since P s and P a come from the same ANN's pre-trained classifier, if h s is same as h a , SoftMax(P s ) will become same as the SoftMax(P a ), then the L AF will become zero and the smallest. At the same time, there are still other chances that can make L AF small too. That is, L AF will not lose the consideration that h s = h a , but will also take care of other alignment situations of h s and h a . Furthermore, Eq. 2 also takes the classification task into the consideration. Hence, Eq. 2 can be seen as a more general alignment strategy than Eq. 1 designed specifically for SNNs. Finally, taking classification loss together, the total loss can be written asEQUATIONwhere L CE is the cross-entropy loss, and λ is a coefficient to balance the classification loss and the alignment loss.As abovementioned, we argue that the last spike neuron layer will limit the performance of the SNN. To better depict this, we give the specific form of a spiking neuron first. Here, we adopt the well-known Leaky Integrate-and-Fire (LIF) neuron model for an SNN. The LIF neuron adjusts the membrane potential based on the input and its membrane potential at the previous moment as follows,EQUATIONwhere U(t, pre) is the pre-membrane potential at t-th timestep, U(t) is the membrane potential at t-th timestep, τ decay is the membrane time constant to describe the membrane potential decaying which is set as 0.25 in the paper, W is the weight, and X(t) is the binary map comes from the previous layer at t-th timestep. Since the X(t) is a binary tensor, WX(t) can be realized by additions, which is more energy-efficient than multiplications.Next, if U(t) exceeds a threshold, the LIF neuron will fire a spike and then hard-reset to 0, given byEQUATIONEQUATIONwhere V th is a given firing threshold and O(t) is the output of the LIF neuron.Let set O(t, last) as the binary output of the last firing activation layer. It will be presented to the average-pooling layer and then full-connected layer, given byEQUATIONwhere y(t) is the output of the network at t-th timestep and W f is the weight matrix of the fullconnected layer. obviously, the output feature, Avgpool(O(t, last)) will be a discrete vector with a limited number of values. Hence, the representation ability of the output feature will be constrained compared to a full precision vector. We advocate for using the ReLU activation layer to replace the last LIF activation layer. Then the output of the last activation layer will beEQUATIONThus we will obtain a full precision output feature representation with a stronger expression ability.With the rich and powerful output feature, the SNN can converge to a highly accurate model easily but with a trivial cost.We train the SNN with our method using the spatial-temporal backpropagation (STBP) algorithm Wu et al. (2019b) , which treats the spiking neuron as a self-recurrent neural network thus enabling an error backpropagation mechanism for SNNs that follows the same principles as in conventional deep networks. As for the non-differentiable firing activity of the spiking neuron, we choose the STE surrogate gradients to solve it as doing in other surrogate gradient (SG) methods Rathi & Roy (2020); Guo et al. (2022b) . Mathematically, it is defined as:EQUATIONThen, the SNN model can be trained end-to-end. As described in Zheng et al. (2020) ; Guo et al. (2022b Guo et al. ( , 2023b,d),d) , with the SNN going deep, the distribution of membrane potential will shift accumulatively and fall into an inappropriate range, which will cause accuracy to decrease. To deal with this problem, we adopt the threshold-dependent Batch Normalization(tdBN) technique Zheng et al. (2020) . The training algorithm of our method for one epoch is detailed in the Algo. 1.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof? Answer: [Yes] .• It should be clear whether the error bar is the standard deviation or the standard error of the mean. • It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified. • For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates). • If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text. 
ablation study: 
 To understand how our method works in practice, a series of ablative studies for L AF and RepAct with spiking ResNet20 architecture along with different timesteps were conducted on the CIFAR-10 dataset. In the experiment, we find that when λ = 0.1, L AF can lead to a relatively better result and we adopt this setting. The top-1 accuracy of these models with L AF and RepAct alone and their combinations are shown in Tab. 1. It's can be seen that both L AF and RepAct can greatly improve the models' performances alone. furthermore, the performances can be still improved by combining these two methods. Take the first ablation experiment (i.e. timestep = 1) for example, the standard training of SNN is 90.40%. If we choose to use L AF loss and RepAct alone, the performance would boost to 92.28% and 92.50%, which are huge improvements (more than 2.0%). Using the joint L AF and RepAct loss, we get a 2.26% accuracy improvement. Moreover, we also show the output feature representation ability of these SNN models with 1 timestep using t-SNE method van der Maaten & Hinton (2008) in Fig. 2 . It can be observed that L AF and RepAct are both able to learn more discriminative features, which also correlates with quantitative accuracy gains. 
experiments: 
 In this section, we conducted extensive experiments using widely-used spiking ResNet20 Rathi & Roy ( 2020 2009), and CIFAR10-DVS Li (2017) . The firing threshold V th and the membrane potential decaying τ decay were set as 0.5 and 0.25 respectively. For static image datasets, we fed the images into the SNN model directly and used the first layer to encode the images to spikes, as in recent works Zheng et al. (2020) ; Rathi & Roy (2020) . For the neuromorphic image dataset, we used the 0/1 spike format directly. We report the top-1 accuracy results with the mean accuracy and standard deviation of 3 trials. CIFAR-10. The CIFAR-10 dataset consists of 60K 32 × 32 images in 10 classes with 50K training images and 10K test images respectively. The data normalization, random horizontal flipping, cropping, AutoAugment Cubuk et al. (2019), and Cutout DeVries & Taylor (2017) were used for data augmentation as in Li et al. (2021b) ; Guo et al. (2022b) . We utilized the SGD optimizer to train our models for 400 epochs with the 0.9 momentum and a learning rate of 0.1 cosine decayed to 0. We found that λ = 0.1 can lead to a relatively better result on the CIFAR dataset and we adopt this setting in the paper. From Tab. 3, it can be seen that our models achieve better performances than other SoTA methods for three commonly used networks in this field and can greatly reduce the timesteps, corresponding to inference time. Our VGG16 model with only 2 timesteps even outperforms the Diet-SNN Rathi & Roy (2020) with 10 timesteps by 1.31% accuracy. With only 2 timesteps, our method based on Resnet19 can also outperform the RecDis- SNN Zhang & Li (2020) , the TET Deng et al. (2022) , and the STBP-tdBN Zheng et al. ( 2020) with 6 timesteps by 0.64%, 1.69%, and 3.03% accuracy, respectively. The same superiority is also shown with the ResNet20 backbone. These comparison results clearly show the efficiency and effectiveness of our method.CIFAR-100. We also verified our method on CIFAR-100. The CIFAR-100 dataset is similiar with the CIFAR-100 but in 100 classes. With the same training pipeline and the architectures on CIFAR-10 here, we found that our method can also achieves better accuracy than other prior works with fewer timesteps. Especially, the ResNet19 trained with our method can achieve 82.43% top-1 accuracy with timesteps of only 2, which outperforms the current methods, TET Deng et al. (2022) and TEBN Duan et al. (2022) by 7.71% and 6.02% but with 6 timesteps.ImageNet. The ImageNet dataset consists of more than 1,250K training and 50K test images in 1000 classes with 224 × 224 respectively. We used standard data normalization, random horizontal flipping, and cropping for data augmentation. The SGD optimizer with the 0.9 momentum and a learning be observed that our method also reaches the SoTA. Our ResNet18 and ResNet34 achieve 65.31% and 67.40% top-1 accuracy with only 4 timesteps, which is just a little worse than GLIF Yao et al. (2022) . However, GLIF adopts the complex spiking neuron, which will increase the inference cost. It's worth noting that our models even outperform better than these SEW ResNet-based models Fang et al. (2021a) ; Guo et al. (2022c) , which transmits information with arbitrary integers. This shows the ability of our method to handle these large-scale datasets.CIFAR10-DVS. The neuromorphic dataset, CIFAR10-DVS, was also used in the paper. It is the neuromorphic version of a part of the CIFAR-10 dataset which consists of 10K images in 10 classes.We also adopted the principle to split the dataset into 9K training images and 1K test images, and resize them to 48 × 48 to evaluate the models' performance similar to Wu et al. (2019b) ; Guo et al. (2022c,b) . the random horizontal flip and random roll within 5 pixels are also adopted in the paper for augmentation Guo et al. (2022b) . To train the ANN model on the neuromorphic dataset, we change the channel number of the first layer of the ANN to 20 to contain all the temporal input at once.Other training settings are the same as that for CIFAR-10 dataset. Our method reaches to 80.10% and 80.50% for ResNet19 and ResNet20. 
conclusion: 
 This work aims at enhancing the output feature representation of SNNs. Then, the L AF Loss that improves the learning of the output feature representation of the SNN using the well-trained ANN classifier and the RepAct method which replaces the last LIF activation layer with the ReLU activation layer to generate a more powerful output feature are proposed. A series of ablation studies show that the two proposed methods can greatly increase the SNN's accuracy. The proposed methods can also be combined and will consistently outperform the other state-of-the-art methods. Justification: We find no limitation which we feel must be specifically highlighted here. Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
experiments compute resources: 
 Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?Answer: [Yes] .Justification: The computation resources description is provided in the appendix.Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage. • The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute. • The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper). 
code of ethics: 
 Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines?Answer: [Yes] .Justification: The research conducted with the NeurIPS Code of Ethics Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed? 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] .Justification: The paper poses no such risks.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] .Justification: The original paper for datasets we used are all cited.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. 
: 
 The overall workflow of the proposed method. To enhance the output feature representation, the L AF loss is used to transmit the important information of output feature of the ANN to that of the SNN counterpart and the last LIF activation neuron layer is replaced by ReLU activation in our SNN structures. Guo et al. (2022b Guo et al. ( , 2024b)) ; Wang et al. (2023 Wang et al. ( , 2022)) ; Zhou et al. (2023) . It adopts 0/1 spikes to transmit information. Benefitting from such an information processing paradigm, the multiplications of activations and weights of SNNs can be replaced by additions, thus enjoying low power consumption Guo et al. (2023a) . Furthermore, on neuromorphic hardwares Akopyan et al. (2015) , when there is no spike coming, the SNN neuron will remain inactive thus requiring significantly lower energy.Despite the SNN being more energy-efficient compared with the ANN, it suffers unsatisfactory task performance. Binary spike feature maps of the SNN will result in limited expressiveness and large accuracy degradation compared with full-precision feature maps of the ANN Guo et al. (2023a , 2022a) . Although the accuracy degradation problem can be mitigated by increasing the timesteps of the SNN, the inference burden will be increased too with the timestep increasing. It is shown crucial that output feature representation is rich and powerful for training a highly accurate model for the classification in recent studies in ANNs Chen et al. (2020) ; He et al. (2019) ; Kang et al. (2019) . Here, we also expect that improving the output feature representation is beneficial for the accuracy of the SNN. To this end, we advocate using trained ANN's rich output feature to guide the learning of output feature representation of SNN. However, different from other SNN distillation methods that align the output feature of the SNN with that of the ANN counterpart directly, we adopt the trained ANN's classifier to guide the information transfer. Using the ANN's classifier directly is inspired by some ANN-SNN methods that directly copy-paste the weight parameters from trained ANN with light modification to homogeneous SNN can obtain a well-performed SNN Han et al. (2020) ; Li et al. (2021a) . Therefore, we expect that utilizing the homogeneous ANN's some modules to guide the learning of SNN will be useful. In particular, for the same input, we wish the SNN's and ANN's feature representation to produce the same output when passed through the same ANN's classifier and the idea can be achieved by a simple Kullback-Leibler (KL)-divergence loss. Furthermore, we also suggest that the last firing activation neuron layer in SNN could be replaced by ReLU, called RepAct, thus the output feature representation will be full precision and the expression ability will be improved but with a trivial additional cost compared with the vanilla SNN. It is worth noting that spiking neural networks do have a certain number of non-binary feature maps. For example, many works use original images as the input of SNNs, thus the multiplications of the first layer can not be replaced by additions in SNNs Rathi & Roy (2020) ; Guo et al. (2022b) ; Baltes et al. (2023) ; Li & Zeng (2022) , but the accuracy would boost to a high level. With a trivial additional cost, the RepAct can also improve the SNN greatly, hence it could be seen as a practical choice for many tasks too.To sum up, we find that the expression ability of the output feature of the SNN is limited. Under this understanding, we focus on enhancing the output feature representation of the SNN in the paper. To this end, we propose two very simple but effective methods. First, we utilize the trained ANN to guide the learning of the output feature representation of the SNN. In specific, since the feature representation of the ANN is more powerful, we propose to align the output feature layer of SNN with that of ANN to learn its rich capabilities. Because direct feature representation matching does not take into account the classification task at hand and ignores the high-level representation information, thus will adversely affect SNN classification results if the matching strategy design is not good enough, we utilize the ANN's pre-trained classifier to align the output feature of the SNN with that of the ANN counterpart to enrich the SNN's feature representation, called L AF loss. This idea is based on the observation that directly copying-pasting the weight parameters from trained ANN to homogeneous SNN is always enough to obtain a well-performed SNN in ANN-SNN methods. In this sense, it can be seen as a simple and effective knowledge distillation method specially designed for SNN born in ANN-SNN. In this way, the feature representation and classification information can both be considered. Second, we replace the last LIF activation neuron layer with the ReLU activation layer to produce the full precision output feature with only a little extra computation, named RepAct. The overall workflow can be seen in Fig. 1 . Our contributions can be summarized as follows:• We propose to use the well-trained ANN to guide and improve the learning of the output feature representation of the SNN. Along with taking into account the classification task, the L AF loss is proposed to weigh the discrepancy of the two outputs that come from the ANN's and SNN's feature representations for the same image by feeding in the same trained ANN's classifier. It can be seen as a simple and effective knowledge distillation method specially designed for SNN derived from a combination of advantages of knowledge distillation and ANN-SNN. With the L AF loss, the SNN's feature representation can be extremely enhanced and results in high accuracy.• We also advise replacing the last LIF activation neuron layer in the SNN with ReLU activation layer to obtain the full precision output feature representation. This improvement will increase the expression ability of the output feature layer thus improving accuracy greatly, but with a trivial extra computation.• We evaluate our methods on both famous static and spiking datasets. Extensive experimental results show that our method consistently outperforms state-of-the-art methods in various experimental settings. 
 
 Appendix: Justification: There is no societal impact of the work performed.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.• The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 


Decision: This paper has received positive reviews. All reviewers agree that the paper has novelty, clarity and boosts the effectiveness of training for SNNs.

Reviews: review: paper_summary: paper_summary: This paper explores whether the benefits of rich output feature representations, known to enhance the accuracy of ANN models for classification, also apply to SNNs. If so, the authors seek to improve the feature representation of SNNs.

The authors address this problem in two steps: first, they align the SNN output features with the ANN output features using KL-divergence loss, similar to knowledge distillation. Second, they replace the last Leaky Integrate-and-Fire (LIF) activation layer in the SNN with a ReLU activation layer.

For feature alignment, the same input is fed to both the SNN and the ANN, and their outputs are compared and aligned using KL-divergence loss. The idea is to use the trained ANN's rich output features to guide the learning of the SNN's output feature representation.

The proposed method involves training the ANN counterpart (with ReLU activation), then replacing the LIF with the ReLU activation function in the last layer of the SNN. Finally, the modified SNN is trained with the help of the pre-trained ANN counterpart by incorporating the KL loss into the total loss of the SNN.
strength_weakness: strength_weakness: 1. The proposed method requires training the ANN counterpart first and setting it aside as a pre-trained model. Subsequently, the SNN needs to be trained. This approach essentially involves training the same framework twice, once with ANN activations and once with SNN activations.

2. The method includes replacing the last LIF activation neuron layer in the SNN with a ReLU activation layer, which results in significant improvement. However, it is unclear whether the final improvement is primarily due to this replacement of last-layer's activation function or the feature alignment process.
questions: questions: 1. In this paper, the authors try to use rich information of the weight parameters from a pretrained ANN counterpart to guide the feature representation learning of the SNN. 
Does this method require training the ANN counterpart and using it to assist in the subsequent training of the SNN? If there is true, it seems the paper trains both the ANN counterpart and the SNN, the training resources will be doubled, how to solve this problem? 


2. The authors stated that they "replace the last Leaky Integrate-and-Fire (LIF) activation layer as the ReLU activation layer in the SNN". Does the improvement mainly come from this replacement of LIF with ReLU? From results of the Ablation experiments in table 1, it shows that using "w/ RepAct" (92.50) demenstrates a larger improvement than using "w/ L_{AF}" (92.28) compared to the "baseline" (90.40) for timestep=1. For other timestep (2 and 4), the results show the same trends. Moreover, the final method using "w/ L_{AF} & RepAct" achieved accuracy of 92.66, which is only a very small increase compared to the "w/ RepAct" with 92.50. 


3. In order to check the improvement for the applications, can the authors provide the performance of "w/ RepAct" and "w/ L_{AF}", "w/ L_{AF} & RepAct", the baseline, and the ANN-counterpart for CIFAR10, CIFAR100, ImageNet, CIFAR10-DVS datasets? 



4. Grammer errors, for example Line 42-44, "Despite the SNN being more energy-efficient compared with the ANN, it suffers unsatisfactory task performance, due to binary spike feature maps of the SNN will result in limited expressiveness and large accuracy degradation compared with full-precision feature maps of the ANN",
limitations: limitations: As the above Weaknesses and Questions.
paper_summary: paper_summary: This paper introduces methods to improve the output feature representation of Spiking Neural Networks (SNNs) by utilizing knowledge distillation (KD) techniques and modifying activation functions. The approach involves aligning the SNN's output features with those of a pre-trained Artificial Neural Network (ANN) using KL-divergence loss and replacing the last Leaky Integrate-and-Fire (LIF) activation with a ReLU activation to enhance feature precision. These modifications are shown to increase the expressiveness and performance of SNNs on both static and neuromorphic datasets.
strength_weakness: strength_weakness: 1. __Novelty__: The methods, while integrating known techniques in a new context, do not present strong novelty. While the combination of techniques is original, the individual components (knowledge distillation and activation function manipulation) are well-trodden areas in SNNs. The paper could benefit from a deeper exploration of how these specific adaptations contribute uniquely to the SNNs' context.

2. __Citation Format__: The paper does not adhere to NeurIPS citation format requirements.

3. __Comparative Analysis__: The paper lacks a comprehensive comparison with other SNN training methods that use similar KD approaches, which is crucial for substantiating the claimed improvements. A broader comparison with similar approaches in the field of SNNs+KD (see [1, 2]) could better highlight this. 

4. __Significance__: The improvements, while statistically valuable, may not represent a paradigm shift in SNN training that the field might expect for a high-impact publication.
questions: questions: + Could the authors extend their comparisons to include seminal works in SNN+KD such as those by Guo et al. [1] and Deng et al. [2]? This would help in positioning their method more clearly within the current research landscape.

+ The ablation study in Table 6 is critical, and the details of the hyperparameters used in the vanilla KD need to be listed. Given that the paper mentions that "most knowledge distillation methods perform poorly without well-designed hyper-parameters," the design of KD should be further discussed. For example, the inclusion of a temperature coefficient commonly used in logits-based KD [3] should be considered (in Eq. 2, etc) to potentially improve the alignment strategy.


Reference: 

[1] Guo, Yufei, et al. "Joint a-snn: Joint training of artificial and spiking neural networks via self-distillation and weight factorization." Pattern Recognition 142 (2023): 109639.

[2] Deng, Shikuang, et al. "Surrogate module learning: Reduce the gradient error accumulation in training spiking neural networks." International Conference on Machine Learning. PMLR, 2023.

[3] Hinton, Geoffrey, Oriol Vinyals, and Jeff Dean. "Distilling the knowledge in a neural network." arXiv preprint arXiv:1503.02531 (2015).
limitations: limitations: See Weaknesses and Questions.
paper_summary: paper_summary: The authors propose a novel distillation approach (EnOF) to address the mismatch in output precision between SNN and ANN, which leads to poor distillation performance. EnOF feeds the output feature of SNN into the ANN classification head to obtain its effective mapping output P_s, and then uses distillation learning to make the P_s close to the ANN output P_a. Additionally, the paper analyzes the issue of inadequate SNN representation due to the final LIF activation layer. The authors suggest replacing it with the ReLU activation function. Through the methods presented in this paper, SNN demonstrates significant performance improvements across different datasets.
strength_weakness: strength_weakness: To illustrate the advances made by EnOF and RepAct, the authors can provide the network Hessian matrix eigenvalues or convergence speed comparisons.

An intuitive drawback of RepAct is its requirement for high-precision floating-point matrix multiplication operations in the classification head, which undoubtedly increases the computation overhead. I suggest that the authors move the overhead comparison section from the appendix to the main text.

The main text lacks the definition of the method name "EnOF.”.
questions: questions: 1. Does distillation learning have a temperature coefficient?
2. Is the optimal $\lambda$ obtained through search, and does different $\lambda$ have a significant impact on the final results?
limitations: limitations: NA
paper_summary: paper_summary: The paper presents the method to improve the representation power of SNNs. To this end, the paper proposed two new techniques to the training of spiking neural networks: 1) To guide learning of the last feature layer by an alignment loss with a pre-trained non-spiking network and 2) to replace the last spiking LIF layer with a continuously valued ReLU layer.
strength_weakness: strength_weakness: 1)	How can SNN representation be converted to float activation in eq 1? Is it averaged?
2)	Improvement with technique 2 is expected, and it makes the comparison with pure SNNs unfair.
3)	The method needs to train an ANN in addition to the SNN. So the total training time is longer.
questions: questions: Please see weaknesses.
limitations: limitations: Please see weaknesses.

Comments: title: Thanks
response
thanks
thanks
comment: Thanks very much for your reply and recognition. We are happy to see that your concerns have been addressed.
Thank you for your feedback. 
The authors have worked to improve the manuscript's readability. 
I am inclined to increase my score to 7.
Thanks very much for your reply and recognition. We are happy to see that your concerns have been addressed.
Thanks very much for your reply and recognition. We are happy to see that your concerns have been addressed.
Thank you for your response. The authors have made efforts to enhance the readability of the manuscript, and I believe that this paper will contribute positively to the development of the SNN field. I'd like to raise my score to 7.
I have reviewed the rebuttal. The revised version should clearly detail the average operation and clarify the comparisons to enhance readability.

Overall, most concerns have been addressed. I would like to adjust the score to a weak accept.