EnOF-SNN: Training Accurate Spiking Neural Networks via Enhancing the Output Feature
Authors: Yufei Guo
Weihang Peng
Xiaode Liu
Yuanpei Chen
Yuhan Zhang
Xin Tong
Zhou Jie
Zhe Ma
Venue: NeurIPS 2024 poster (2024)
Keywords: Spiking neural network
Primary area: neuroscience_and_cognitive_science

Abstract: Spiking neural networks (SNNs) have gained more and more interest as one of the energy-efficient alternatives of conventional artificial neural networks (ANNs). They exchange 0/1 spikes for processing information, thus most of the multiplications in networks can be replaced by additions. However, binary spike feature maps will limit the expressiveness of the SNN and result in unsatisfactory performance compared with ANNs. 
It is shown that a rich output feature representation, i.e., the feature vector before classifier) is beneficial to training an accurate model in ANNs for classification. 
We wonder if it also does for SNNs and how to improve the feature representation of the SNN.
To this end, we materialize this idea in two special designed methods for SNNs.
First, inspired by some ANN-SNN methods that directly copy-paste the weight parameters from trained ANN with light modification to homogeneous SNN can obtain a well-performed SNN, we use rich information of the weight parameters from the trained ANN counterpart to guide the feature representation learning of the SNN. 
In particular, we present the SNN's and ANN's feature representation from the same input to ANN's classifier to product SNN's and ANN's outputs respectively and then align the feature with the KL-divergence loss as in knowledge distillation methods, called L_ AF loss.
It can be seen as a novel and effective knowledge distillation method specially designed for the SNN that comes from both the knowledge distillation and ANN-SNN methods. Various ablation study shows that the L_AF loss is more powerful than the vanilla knowledge distillation method.
Second, we replace the last Leaky Integrate-and-Fire (LIF) activation layer as the ReLU activation layer to generate the output feature, thus a more powerful SNN with full-precision feature representation can be achieved but with only a little extra computation.
Experimental results show that our method consistently outperforms the current state-of-the-art algorithms on both popular non-spiking static and neuromorphic datasets. We provide an extremely simple but effective way to train high-accuracy spiking neural networks.

EnOF-SNN: Training Accurate Spiking Neural Networks via Enhancing the Output Feature
Yufei Guo, Weihang Peng", Xiaode Liu, Yuanpei Chen, Yuhan Zhang, Xin Tong,
Zhou Jie, Zhe Ma² Intelligent Science & Technology Academy of CASIC yfguoupku. edu. en, pengweihang8122666. com, mazhe, thuu163.
edu.
Abstract
Spiking neural networks (SNNs) have gained more and more interest as of the energy-efficient alternatives of conventional artificial neural networks (ANNs). They exchange 0/1 spikes for processing information, thus most of the multiplica- tions in networks can be replaced by additions. However, binary spike feature maps will limit the expressiveness of the SNN and result unsatisfactory performance compared with ANNs. It is shown that a rich output feature representation (i.e., the feature vector before classifier) beneficial training an accurate model ANNs for classification. We wonder if also does for SNNs and how to improve the feature representation of the SNN. To this end, we materialize this idea in two special designed methods for SNNs. First, inspired by some ANN-SNN meth- ods that directly copy-pasee the weight parameters from trained ANN with light modification to homogeneous SNN can obtain well-performed SNN, use rich information of the weight parameters from the trained ANN counterpart to guide the feature representation learning of the SNN. In particular, we present the SNN's and ANN's feature representation from the same input to ANN's classifier to product SNN's and ANN's outputs respectively and then align the feature with the KL-divergence loss as in knowledge distillation methods, called CAF loss. It can be novel and effective knowledge distillation method specially designed for the SNN that comes from both the knowledge distillation and ANN-SNN methods. Second, replace the last Leaky Integrate-and -adtee-----Frree (LIF) activation layer the ReLU activation layer to generate the output feature, thus more powerful SNN with full-precision feature representation can be achieved but with only a little extra computation. Experimental results show that our method consistently outperforms the current sate-of-thh-ar algorithms both popular non-spiking static and neuromorphic datasets.
Introduction
Recently, convolutional neural networks (CNNs) have become extremely popular due to their per- forming more and more well in diverse fields including pattern recognition He et al. (2016); (Smonnyan
& Zisserman 2014)) object detection Girshick (2015) Ren et al. Ming
(2023) language processing Chen et al. (2016), robotics Levine et al. 2015). and so on. However, these full-precision
CNN models make them power-hungrs and tedious for real-world deployment. The spiking neural
network (SNN), which aims to mimic the behavior of the human brain, has become a promising
energy-efficient architecture substitute for the CNN some specific scenarios Guo et al. (2022c)
20 Rathi et al. (2020): Zhu et al. 2022); Wu et al. (019a): Zhang et al. (2020) Jeffares et al. (202)
"Equal Contributions. Cooresponding author.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Trained ANN
Full precision map
Rel activation
Pooling
Classifiet
Lag loss
Full precision oulput feature
Full precision pre-activation
precision map
precision map
Rel nctivarion
Pooling
Classifier
Lgg loss
Pull precision output
Full precision pre pretactivaiion
Binary map
Replaced by
Vanilla SNN
Binary
FIB activation
Pooling
Classifie
Discrete output
Full precision
Hinary map
pre-activation
Figure 1: The overall workflow of the proposed method. To enhance the output feature representation, the AF loss is used to transmit the important information of output feature of the ANN to that the SNN counterpart and the last LIF activation neuron layer is replaced ReLU activation in our SNN structures.
Guo al. 22022b 2024b); Wang et al. (2023
2022); Zhou al. (2023). It adopts 0/1 spikes to transmit information. Benefitting from such information processing paradigm, the multiplications of activations and weights of SNNs can be replaced by additions, thus enjoying low power consump- tion Guo et al. (2023a). Furthermore, on neuromorphic hardwares Akopyan et al.] (2015), when there no spike coming, the SNN neuron will remain inactive thus requiring significantly lower energy.
Despite the SNN being more energy-efficient compared with the ANN, it suffers unsatisfactory task performance. Binary spike feature maps of the SNN will result in limited expressiveness and large accuracy degradation compared with full- full-irecision feature maps of the ANN Guo et al. (2023a 2022a). Although the accuracy degradation problem can be mitigated by increasing the timesteps of the SNN, the inference burden will be increased too with the timestep increasing. It is shown crucial that output feature representation rich and powerful for training highly accurate model for the classification in recent studies in ANNs Chen et al. (2020); He et al. (201)) Kang et al. (2019). Here, we also expect that improving the output feature representation beneficial for the accuracy of the SNN. To this end, we advocate using trained ANN's 's rich output feature guide the learning of output feature representation of SNN. However, different from other SNN distillation methods that align the output feature of the SNN with that of the ANN counterpart directly, we adopt the trained ANN's classifier to guide the information transfer. Using the ANN's classifier directly is inspired by some ANN-SNN methods that directly copy-paste the weight parameters from trained ANN with light modification to homogeneous SNN can obtain a well-performed SNN Han et al. (020); ; et al. 2222 Therefore, we expect that utilizing the homogeneous ANN's some modules to guide the learning SNN will be useful. In particular, for the same input, we wish the SNN's and ANN's feature representation to produce the same output when passed through the same ANN's classifier and the idea can be achieved by a simple Kullback-Leee Leibler (KL)-divereence loss.
Furthermore, we also suggest that the last firing activation neuron layer in SNN could be replaced by ReL.U, called RepAct, thus the output feature representation will be full precision and the expression ability will be improved but with a trivial additional cost compared with the vanilla SNN. It is worth

---

noting that spiking neural networks do have certain number of non-binary feature maps. For example, many works use original images as the input of SNNs, thus the multiplications of the first layer can not be replaced by additions in SNNs Rathi & Roy (2020); Guo et al. (2022b): Baltes et al. (2023); [Li & Zeng (2022), but the accuracy would boost to a high level. With a trivial additional cost, the RepAct can also improve the SNN greatly, hence it could be seen as practical choice for many tasks too.
To sum up, we find that the expression ability of the output feature of the SNN is limited. Under this understanding. we focus on enhancing the output feature representation of the SNN in the paper. To this end, we propose two very simple but effective methods. First, we utilize the trained ANN to guide the learning of the output feature representation of the SNN. In specific, since the feature representation of the ANN is more powerful, we propose to align the output feature layer of SNN with that of ANN to learn its rich capabilities. Because direct feature representation matching does not take into account the classification task at hand and ignores the high-level representation information, thus will adversely affect SNN classification results if the matching strategy design is not good enough, we utilize the ANN's pre-trained classifier to align the output feature of the SNN with that of the ANN counterpart to enrich the SNN's feature representation, called AA loss. This idea is based on the observation that directly copying-pasting the weight parameters from trained ANN to homogeneous SNN is always enough to obtain a well welll-performe SNN in ANN-SNN methods. In this sense. it can be seen simple and effective knowledge distillation method specially designed for SNN born in ANN. In this way, the feature representation and classification information can both be considered. Second, we replace the last LIF activation neuron layer with the ReLU activation layer to produce the full precision output feature with only little extra computation, named RepAct. The overall workflow can be seen in Fig [ Our contributions can be summarized as follows:
We propose to use the well-trained ANN to guide and improve the learning of the output feature representation of the SNN. Along with taking into account the classification task, the LA loss is proposed to weigh the discrepancy of the two outputs that come from the ANN's and SNN's feature representations for the same image by feeding in the same trained ANN's classifier. It can be seen as a simple and effective knowledge distillation method specially designed for SNN derived from a combination of advantages of knowledge distillation and ANN-SNN. With the CAP loss, the SNN's feature representation can be extremely enhanced and results in high accuracy.
We also advise replacing the last LIF activation neuron layer in the SNN with ReLU activation layer to obtain the full precision output feature representation. This improvement will increase the expression ability of the output feature layer thus improving accuracy greatly, but with a trivial extra computation.
We evaluate our methods on both famous static and spiking datasets. Extensive experimental results show that our method consistently outperforms state-of-the-art methods in various experimental settings.
2 Related Work
In view of the success of deep CNNs, the early development of deep SNNs benefits from the experience of CNNs. Using the trained ANN to guide the generation of high-precision SNN is one of the mainstream research at present. There are three main routes to realizing this idea.
First, a simple and effective way to use ANNs is converting a well-trained ANN model to an SNN model Han et al. (2020); Sengupta et al. (018);; Li et al. (2021a); Ding et al. (202T): Hao al. (202a b). This method trains an accurate ANN with ReLU activation first and then uses the ANN network parameters to construct a special SNN network that enjoys the same structure as the ANN and adopts the average output through multiple timesteps as the final result. The converted SNN usually performs worse than the original ANN, since there are more or fewer conversion errors between the ANN and SNN. To bridge this gap as much as possible. many effective methods have been proposed, such as long inference time Girshick (2015), threshold rescaling (Seniupta et al. (2018), soft reset Han et al. (2020), threshold shift Li et al. (2221)), and the quantization clip-floor-shi activation function Ding et al. (202T). However, the conversion method restricted to rate-coding and the Integrate- aaterateeann-Fiane -nr- (IF) model, and needs long timesteps to obtain satisfied accuracy.
3

---

Second, using the well-trained ANN model's parameters to initial the nascent SNN has been another fundamental line of research. For example, in Raathi et al. (2020) and Rathi & Roy (2020). a trained ANN is converted to an SNN first and then the weights of the converted SNN are fine-tuned further with the training method. Initialed with the trained ANN, the SNN shows a faster convergence than that from random initialization. Though this method usually needs fewer timesteps than the conversion method, it does not show obvious advantages for deep models.
Third, there are also some works that apply the knowledge distillation technique in the SNN field that to use ANN to distill the SNN Kushawaha et al. (2021); Takuya et al. (202T); Xu et al. (2023b a); Guo et al. (2023c); Zhang et al. (2024). However, as has been comprehensively and systematically studied in Tian al.J (2020), most of the knowledge distillation methods perform poorly without well-designed paperr-rrameters.. These knowledge distillation methods for SNNs have also verified this consensus. They all adopt complex multi-stage distillation manner and only report the results on small-scale datasets. Providing a simple but effective knowledge distillation for the SNN is difficult.
Despite the above methods providing us with a lot of valuable insights, we still think there are some other improvements worth further consideration. First, these prior methods merely use the parameters of the trained ANN guide the designing of the parameters of the SNN or adopt complex framework to transfer knowledge from ANN to SNN individually, designing a simple and effective knowledge distillation method specifically for SNNs combining these useful experience of SNN knowledge distillation and ANN-SNN methods simultancously are ignored. Second, these SNN methods all adopt similar architectures as ANNs that only modify the ANN activation layers as spike neuron layers simply but without considering the advantages and disadvantages of the SNNs comprehensively. In the paper, aiming at improving the output feature representation of the SNN, we will take care of these two problems well.
3 Preliminary and Methodology
This section first introduces the proposed method of aligning the SNN's output feature with the ANN's output feature. Then the Leaky antegrate-and-iin (LIF) neuron model, why its output feature's expression ability is limited, and how to replace the LIF activation layer with the ReLU activation layer will be introduced in detail. Finally, some key details for training the SNN will be given.
3.1
LAF Loss: Output Feature Alignment Loss
To improve the output feature representation ability, we advocate using the well-trained ANN to guide and improve the learning of the output feature representation of the SNN. A natural idea is to align the output feature of the SNN to that of its ANN counterpart directly like knowledge distillation methods, given by
arg min (1 h=ll)- W.
(1)
Where Ws. hg, and h2 represent the parameters of the SNN, the output feature of the ANN, and the output feature of the SNN.
However, we found that this natural idea does not always work, which has also been validated by Tian jet al. (2020). The work has investigated most of the famous knowledge distillation methods and found that they perform poorly without well-designed syper-parameters.
The reason why direct distillation performs worse in SNNs comes from that: 1) discrete output feature of the SNN makes its representation space much different from that of the full precision ANN, and directly aligning their output features simply will ignore the difference of their representation spaces and induce too much constraint in the SNN optimization, which may lead to configuration far from the global optimal: 2) this design also treats each dimension in the feature space independently, and ignores the high-level interdependencies of the feature representation; and 3) it does not take into account the classification task at all, which conflicts with the final optimization goal.
To consider the above problems comprehensively, we present a simple but effective method, that uses the ANN's classifier to guide the learning of the SNN's output feature. This is inspired by some ANN- SNN methods that directly copy-paste the weight parameters from trained ANN to homogeneous SNN can obtain a well-performed SNN. Hence, we guess that utilizing a homogeneous ANN's some elements to guide the learning of SNN may be useful. In specific, we fed the SNN's output feature to

---

the ANN's pre- rre-trained classifier to generate the output, Ps. At the same time, we obtain the ANN's output, P. from the same imput image the SNN. Then we utilize the KL-divergence loss to align the student output to the teacher output as follows:
AAF = Σ SoofMax(Pa)aaon SoftMax
(2)
Sof SoftMax(P.)
In this situation, since P. and P& come from the same ANN's pre-trained classifier, if ht is same as ha SoftMax(Pss will become same as the Sof SoftMax(Pa)) a), then the CAF will become zero and the smallest. At the same time, there still other chances that can make CA small too. That is, CAF will not lose the consideration that h2 = ha, but will also take care of other alignment situations of hg and ha Furthermore, Eq. 2 also takes the classification task into the consideration. Hence, Eq. can be seen as a more general alignment strategy than Eq. designed specifically for SNNs. Finally, taking classification loss together, the total loss can be written
Lotal = Lo + ACAF
(3)
where Lce is the cross cross-entropy loss, and À is coefficient to balance the classification loss and the alignment loss.
3.2 RepAct: Replace the Last Activation Layer
As abovementioned, we argue that the last spike neuron layer will limit the performance of the SNN. To better depict this, we give the specific form of spiking neuron first. Here, we adopt the well-known Leaky Integrate-and-Fire (LIF) neuron model for an SNN. The LIF neuron adjusts the membrane potential based on the input and its membrane potential at the previous moment as follows,
Ul, pre) = Tdeeay U(l 1) + WX(();
where Ut,, pre) is the pre-membrane potential at t-th timestep, U(t) is the membrane potential at t-th timestep, Tdecay is the membrane time constant to describe the membrane potential decaying which is set as 0.25 in the paper, W is the weight, and ((t is the binary map comes from the previous layer at t-th timestep. Since the X(t) is a binary tensor, WX(t) can realized by additions, which is more energy-efficient than multiplications.
Next, if (L) exceeds threshold, the LIF neuron will fire spike and then hard-reset to 0, given by
( 1, if U(t, pre) 2 Vth
(5)
0, otherwise
Utt) (tt.pre). (1 O(t)).
(6)
where Ve, is a given firing threshold and (() is the output of the LIF neuron.
Let set Ot,, last) as the binary output of the last firing activation layer. It will be presented to the average-pooling layer and then full full-connected layer, given by
y(t) W Avgpool (Ott, last)),
(7)
where ytt is the output of the network at t-th timestep and Wy is the weight matrix of the full- connected layer. obviously, the output feature, Avgpool (Ott. last)) will be a discrete vector with limited number of values. Hence, the representation ability of the output feature will be constrained compared to a full precision vector. We advocate for using the ReLU activation layer to replace the last LIF activation layer. Then the output of the last activation layer will be
(1, last) = ReLU (WX(()
Thus will obtain full precision output feature representation with stronger expression ability. With the rich and powerful output feature, the SNN can converge to a highly accurate model easily but with trivial cost.
3.3 Training Framework
We train the SNN with our method using the spatial-temporal backpropagation (STBP) algorithm Wu et al. 2019b), which treats the spiking neuron self-recurrent neural network thus enabling an
5

---

Algorithm Training SNN for one epoch.
Input: An SNN to be trained whose last LIF activation layer is replaced the ReLU activation layer a pre-trained ANN; timestep: T; training dataset; total training iteration in one epoch Irain- Output: The trained SNN.
for all 1, ................................ Itrain
Atriin iteration do
2: Get mini-bated training data and class label: Y*:
3: Feed the mini- batch training data into the SNN and the ANN networks;
4: Calculate the SNN output (t) and the output feature (t) of each time step;
5: Compute classification loss LCE E'= (C)(
6:
Calculate the ANN output P
Present the ht (t) to the ANN's classifier to product the output (t) of each time step;
8:
Compute alignment loss LA = +
1==1 CAF (t)
9:
Compute the total loss Total by Eq.
10:
Backpropagation using STBP algorithm Wu et al. (2019b) and the SG method (see in Eq. 9)
then update the SNN model parameters.
11: end for
(a) Vanilla
(b) CAF
(c) RepAct
(d) LAF + RepAct
Figure 2: t-SNE visualization of the output feature for random 2000 samples in CIFAR-I0. Every color represents a different class. It can be clearly seen that our method can learn better representa- tions.
error backpropagation mechanism for SNNs that follows the same principles as in conventional deep networks. As for the -in-differentiable firing activity of the spiking neuron, we choose the STE surrogate gradients solve it as doing in other surrogate gradient (SG) methods Rathi & Roy (2020); Guo et al. (2022b). Mathematically, it is defined as:
d0 dU -
{
1, if
0. otherwisc
Then, the SNN model can be trained end-to-end. As described in Zheng al. (2020) Guo al. (2022b d with the SNN going deep. the distribution of membrane potential will shift accumulatively and fall into an inappropriate range, which will cause accuracy to decrease. To deal with this problem, we adopt the threshold-depende al. (202)) The training algorithm of our method for one epoch detailed the Algo.
Batch Normalization((dBN) technique Zheng
4 Ablation Study
To understand how our method works in practice, a series of ablative studies for CAF and RepAct with spiking ResNet20 architecture along with different timesteps were conducted on the CIFAR- dataset. In the experiment, we find that when À = 0.1, LAF can lead to relatively better result and we adopt this setting. The top-1 accuracy of these models with LAF and RepAct alone and their combinations are shown in Tab. []] It's can be seen that both LAF and RepAct can greatly improve models' performances alone. furthermore, the performances can be still improved by combining these two methods. Take the first ablation experiment (i.e. timestep = 1) for example, the standard training SNN is 90.40%. we choose to use LAP loss and RepAct alone, the performance would boost to 92.28% and 92.50% which are huge improvements (more than 2.0%). Using the joint LAF

---

Table 1: Ablation experiments for RepAct and CAF.
Timestep
Methods
Accuracy
baseline
90.40%
w/ LAF
92.28%
w/ RepAct
92.50%
w/ CAF RepAct
92.66 %
baseline
92.80%
w/ CAP
93.53%
2
w/ RepAct
93.62%
w/ CAF & RepAct
93.86% %
baseline
93.85%
w/ CAF
94.39%
4
w/ RepAct
94.66%
w/ CAF & RepAct
94.74%
and RepAct loss, we get a 2.26% accuracy improvement. Moreover, we also show the output feature representation ability of these SNN models with 1 timestep using t-SNE method van der Maaten & Hinton| (0088) in Fig. 2 can be observed that LAF and RepAct are both able to learn more discriminative features, which also correlates with quantitative accuracy gains.
5 Experiments
In this section, we conducted extensive experiments using widely-used spiking ResNet20 Rathi &
Roy] (2020): Sengupta et (2018). VGG16 Rathi Roy (2020). ResNet18 FaangegtFFanggae al. (2021a).
ResNet19 Zheng et al. (2020), and ResNet34 Fang et al. (202 verify the effectiveness of the proposed methods on both static and neuromorphic datasets including CIFAR- Krizhevsky Krinevyk et al.) CIFAR-100 Krizhevsky et al.] ImageNet [eng et al. (2009), and CIFARIO-DVSI
(017). The firing threshold Vnn and the membrane potential decaying Tdecay were set as 0.5 and 0.25 respectively, For static image datasets, we fed the images into the SNN model directly and used the first layer to encode the images to spikes, as in recent works Zheng et al. (2020); Rathi & Roy (200)). For the neuromorphic image dataset, used the 0/1 spike format directly. We report the top-1 accuracy results with the mean accuracy and standard deviation of 3 trials.
CIFAR-10. The CIFAR-10 dataset consists of 60K 32 32 images in 10 classes with 50K training images and 10K test images respectively. The data normalization, random horizontal flipping, cropping, MuttAAgmeet(Cubbn Cubuk et al. (2019), and Cutout DeVries & Taylor (2017) were used for data augmentation as Li et al. (2021b); Guo et al. (2022)). We utilized the SGD optimizer to train our models for 400 epochs with the 0.9 momentum and learning rate of 0.1 cosine decayed to 0. We found that À = 0.1 can lead to a relatively better result on the CIFAR dataset and we adopt this setting in the paper. From Tab. it can be seen that models achieve better performances than other SoTA methods for three commonly used networks in this field and can greatly reduce the timesteps, outperforms the Diet-SNN Rathi & Roy (2020) with 10 timesteps by 1.31% 11% accuracy. With only 2 timesteps, our method based Resnet19 can also outperform the RecDis-SNr Zhang & Li (2020), the TET et al. 2022), and the STBP-tdBN Zheng et al. (2020) with 6 timesteps by 0.64% % 1.69%, and 3.03% accuracy, respectively. The same superiority is also shown with the ResNer20 backbone. These comparison results clearly show the efficiency and effectiveness of our method.
corresponding inference time. Our VGG16 model with only timesteps
CIFAR-100. We also verified our method CIFAR-100. The CIFAR-100 dataset is similiar with the CIFAR-100 but in 100 classes. With the same training pipeline and the architectures on CIFAR-IC 10 here, we found that our method can also achieves better accuracy than other prior works with fewer timesteps. Especially, the 1esNet19 trained with our method can achieve 82.43% top-1 accuracy with
timesteps of only 2, which outperforms the current methods, Deng al. (2022) and TEBN Duan ct al. (022) by 7.71% 711 and 6.02% but with 6 timesteps.
IeageNet. The ImageNet dataset consists of more than 1,250K training and 50K test images in 1000 classes with 224 x 224 respectively. We used standard data normalization, random horizontal flipping, and cropping for data augmentation. The SGD optimizer with the 0.9 momentum and a learning
7

---

Table 2: Comparison with SoTA methods on CIFAR-10.
Dataset
Method
Type
Architecture
Timestep
Accuracy
SpikeNorm
Sengupta al
(2018
ANN2SNN
VGG16
2500
91.55%
Hyhrid-Trai Rathi al. 2020
Hybrid training
VGGI6
200
92.02%
Spike- Mikk-basedBBPLCC. Lec
2020
SNN training
ResNer11
100
90.95%
STBP Wu 100 96
SNN training
CIFARNet
12
90.53%
TSSL- Zhang Zhang
2020
SNN training
CIFARNet
5
91.41% %
PLIP Fang ct al
20210
SNN training
PLIFNet
8
93.50%
GLIF Yao et al. 2022
SNN training
R9
2
94.44%
DSR
Meng et al 2023
SNN training
ResNer18
20
95 40%
KDS et 2025b
SNN training
ResNer18
93.41 %
TAB
Jiang et al.
2024
SNN training
ResNet19
94.76%
92.70%
VGG16
93.44%
Diet-SNN Rathi & Roy 2020
SNN training
91.78%
ResNer20
92.54% 10
2
93.13%
Dspike Li ct al.
2021b
SNN training
ResNer20
93.66% 4
CIFAR-10
94.25%
92.34%
STBP-dBNN Zheng et al.
2020
SNN training
ResNet19
92.92%
93.16%
94.16%
TET Deng 2022
SNN training
ResNer19
94.44%
94.50% 2
93.64%
RecDis-SNN Guo et al.
2022b
SNN training
1e
95.55% % 4
95.53%
95.37% 10.08
ResNet19
2
96.19% ++0.0
92.66% E0.08
Our method
SNN training
ResNer20
93.86 % #0.07
94.74% 10.08
94.07% +0.10
VGGI6
4
94.75%
Table Comparison with SoTA methods on CIFAR-100.
Dataset
Method
Type
Architecture
Timestep
Accuracy
RMP Han el al
2020
ANN2SNN
ResNet20
2048
67.82%
BinarySNN Lu Sengupta (220
ANN2SNN
VGG15
63.20%
Hybrid- Train
2020
Hybrid training
VGG11
125
67.90%
T2FSNN]Park TAB Jiag et al, 2024
2020
ANN2SNN
VGG16
680
68.80%
SNN training
ResNet19
76.81%
ResNer20
64.07%
Dict-SNN Ruthi Roy
2020
SNN training
VGG16
69.67%
71.68%
Dspike Li et
202
SNN training
ResNer20
73.35%
74.24%
72.87%
TET Deng et al.
2022
SNN training
ResNet19
74.47%
CIFAR-100
74.72%
ResNet19
74.10%
RecDis-SNY Guo al.
2022b
SNN training
VGG16
89.88%
ResNet20
66.60%
Real Spike Guo et al.
2022c
SNN training
VGG16
70.62%
75.86%
TEBN Duan et al.
2022
SNN training
ResNet19
76.13%
76.41% 1%%
VGGT6
72.53% 10.11
77.08% =0.08
ResNet19
Our method
82.43 -0.09
SNN training
71.55 10112
ResNet20
73.01 00.0.0
rate of 0.1 cosine decayed to 0 adopted. Except that we set À and epochs as 320, the other training settings consistent with CIFAR dataset. Our results presented in Tab, Again, it can

---

Table 4: Comparison with SoTA methods ImageNet.
Dataset
Method
Type
Architecture
Timestep
Accuracy
STBP-idBN Zheng et al.
2020
SNN training
ResNet34
63.72%
TET Deng et al. 2022
SNN training
ResNet34
64.79%
MS MS-ResNNll Hu al 2021
SNN training
ResNet18
63.10%
OTTT Xiao
Xiao et al. 2020
SNN training
ResNet34
63.10%
RecDis-SNN Guo et al
2022b
SNN training
ResNet34
67.33%
SlipReLUU Dhanegg
Jiang el al.
2023
ANN-SNN
ResNet34
32
66.61%
ResNet18
63.68%
ImageNet
Real Spike Guo et al. 2022c
SNN training
ResNet34
67.69%
ResNet18
63.18% 4
SEW ResNet Fang
al.
2021a
SNN training
ResNet34
67.04%
OTTT
Xiao et al. 2022
SNN training
ResNet34
65.15% 6
GLIF
Yao et al 2022
SNN training
ResNet34
67.52%
DSR
Meng cl al. 2023
SNN training
ResNet18
50
67.74%
ResNet18
65.31 00.009
Our method
SNN training
ResNet34
4
67.40 1000
Table 5: Comparison with SoTA methods on CIFARIO-DVS.
Dataset
Method
Type
Architecture
Timestep
Accuracy
Rollour Kugele ct al.
2020
Rollout
DenseNet
10
66.80%
LJAF-Ne el al. 4022
Conv3D
LIAF-Net
71.70%
LIAF-Net Wu et al. 2022
LIAF
LIAF-Net
70.40%
STBP-dBNN Zhong
2020
SNN training
ResNet19
67.80%
CIFRIOODVS
RecDis-SNN Guo
SNN training
ResNet19
72.42%
ResNet19
10
72.85%
Real Spike Guo ct al.
2022c
SNN training
ResNet22
10
78. 00%
ResNet19
10
78.40%
Temary Spike Guo al.
2024a
SNN training
ResNer20
78.70%
ResNet19
80.10%=0.21
Our method
SNN training
ResNet20
80.50 =0.20
be observed that our method also reaches the SoTA. Our ResNet18 and ResNet34 achieve 65.3 13% % and 67.40% top-1 accuracy with only 4 timesteps, which is just a little worse than GLIF Yao et al. (2022). However, GLIF adopts the complex spiking neuron, which will increase the inference cost. It's worth noting that our models even outperform better than these SEW ResNet-based models Fang al. (2021a): Guo et al. (2022c). which transmits information with arbitrary integers. This shows the ability of our method to handle these large-scale datasets.
CIFAR1O-DVS. The neuromorphic dataset, CIFARIO-DVES was also used in the paper. is the neuromorphic version of part of the CIFAR-10 dataset which consists of 10K images in 10 classes. We also adopted the principle split the dataset into 9K training images and 1K test images, and resize them to 48 x 48 to evaluate the models' performance similar to Wu et al. (2019b); Guo et al. b2)22clb). the random horizontal flip and random roll within 5 pixels are also adopted in the paper for augmentation Guo et al. (202bb). To train the ANN model on the neuromorphic dataset, we change the channel number of the first layer of the ANN to 20 to contain all the temporal input at once. Other training settings are the same as that for CIFAR-10 10 dataset. Our method reaches to 80. 10% and 80.50% for ResNet19 and ResNet20.
6 Conclusion
This work aims at enhancing the output feature representation of SNNs. Then, the CAF Loss that improves the learning of the output feature representation of the SNN using the well-trained ANN classifier and the RepAct method which replaces the last LIF activation layer with the ReLU activation layer to generate more powerful output feature are proposed. A series of ablation studies show that the two proposed methods can greatly increase the SNN's accuracy. The proposed methods can also be combined and will consistently outperform the other tht methods.

---

Acknowledgment
This work is supported by grants from the National Natural Science Foundation of China under contracts No. No.12202412 and No.12202411
References
Akopyan. F., Sawada, J., Cassidy, A., Alvarez-Icaza, R., Arthur, J., Merolla, P., Imam, N., Nakamura, Y., Datta, P., Nam, G.-J,, Taba, B., Beakes, M., Brezzo, B., Kuang, J., Manohar, R., Risk. W., Jackson, B., and Modha, D. Truenorth: Design and tool flow of 65 mw million neuron programmable neurosynaptic chip. Computer-Aided Design of Integrated Circuits and Systems, IEEE Transactions on, 34:1537-1557, 10 2015. doi: 10.1
Baltes, M., Abujahar, N., Yue, Y. Smith, C. D., and Liu, Joint ann-snn co-training for object localization and image segmentation, 2023.
Chen, Q.. Zhu, X., Ling, Z., Wei, S., Jiang, H., and Inkpen, D. Enhanced Istm for natural language inference. 2016.
Chen, T., Korbllith, S., Norouzi, M., and Hinton, simple framework for contrastive learning of visual representations. 2020.
Cubuk, E., Zoph. B., Mane, D., Vasudevan, V., and Le, Q. Autoaugment: Learning augmentation strategies from data. pp. 113-123, 06 2019. doi: 10. 109/CVPR.2019.00000
Deng, J., Dong, W., Socher, R., Li. L.-J., Li, K., and Li, F.F.. Imagenet: large-scale hierarchical image database. pp. 248,255,00 06 2009. doi: 10. 109/CC/R.200995526444
Deng, S., Li, Y., Zhang, S., and Gu, S. Temporal efficient training of spiking neural network via
gradient reiihightn In International Conference on Learning Representations, 2022. URL
htttps://openrviit nnt. http///oommidi_xtiiittdee
Vries, T. and Taylor, G. W. Improved regularization of convolutional neural networks with cutout, 2017.
Ding, J., Yu, Z., Tian, Y., and Huang. T. Optimal ann-snn conversion for fast and accurate inference in deep spiking neural networks. 2021.
Duan, C., Ding, J., Chen, S., Yu, Z., and Huang, T. Temporal effective batch normalization in spiking neural networks. In Oh, A. H., Agarwal, A., Belgrave, D., and Cho, K. (eds.), Advances in Neural Information Processing Systems, 2022. Utl.atpp::/opparreieeee.. f LL gyyQi.qz
Fang. W., Yu, Z., Chen, Y., Huang, T., and Tian, Y. Deep residual learning in spiking neural networks. 2021a.
Fang, W., yu, Z., Chen, Y., Masquelier, T., Huang, T., and Tian, Y. Incorporating learnable membrane time constant to enhance learning of spiking neural networks. 08 2021b.
Girshick, R. Fast r-enn. In 2015 IEEE International Conference on Computer Vision (ICCV), pp. 1440-1448, 04 2015. doi: 10.1109/CCV22215...0
Guo, Y., Chen, Y., Zhang, I.., Liu, X., Wang, Y., Huang, X., and Ma, Z. Im-loss: information maximization loss for spiking neural networks. Advances in Neural Information Processing Systems, 35:16611 2022a.
Guo, Y., Tong, X., Chen, Y., Zhang I.., Liu, X., Ma, 7.,, and Huang, Recdis-snn: Rectifying membrane potential distribution for directly training spiking neural networks. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 326-335, June 2022b.
Guo, Y., Zhang, I.., Chen, Y., Tong, X., Liu, X., Wang, Y., Huang, X., and Ma, Z. Real spike: Learning real-valued spikes for spiking neural networks. In European Conference on Computer Vision, pp. 52-68. Springer, 2022c.
10

---

Guo, Y., Huang, X., and Ma, Z. Direct learning- Frontiers in Neuroscience, 17:1209795, 2023a.
bearning-based based deep spiking neural networks: a review.
Guo, Y., Liu, X., Chen, Y., Zhang, L., Peng, W., Zhang, Y., Huang, X., and Ma, Z. Rmp-loss: Regularizing membrane potential distribution for spiking neural networks. In Proceedings of the IEEF/CVF International Conference on Computer Vision pp. 17391-17401 2023b.
Guo, Y., Peng, W., Chen, Y., Zhang, L.,, Liu, X., Huang, X., and Ma, Z. Joint a-snn: Joint training of artificial and spiking neural networks via ooff-istillattin and weight factorization. Pattern Recognition, 142: 109639, 2023c.
Guo, Y. Zhang, Y. Chen, Y., Peng, W., Liu, X., Zhang, L., Huang. X., and Ma, Z. Membrane potential batch normalization for spiking neural networks. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 19420-19438 2023d.
Guo, Y. Chen, Y., Liu, X., Peng, W., Zhang, Y., Huang, X., and Ma, Z. Ternary spike: Learning ternary spikes for spiking neural networks. In Proceedings of the AAAI Conference Artificial Intelligence, volume 38, pp. 12244-12252 2024a.
Guo, Y., Chen, Y., and Ma. Neuroclip: Neuromorphic data understanding by clip and snn. IEEE Signal Processing Letters, 2024b.
Han, B., Srinivasan, G., and Roy, K. Rmp-snn: Residual membrane potential neuron for enabling deeper high-accuracy and low-latency spiking neural network. pp. 13555-13564, 06 2020. doi: 10.1 00.10099CC
Hao, Z., Bu, T., Ding, J., Huang, T., and Yu, Z. Reducing ann-snn conversion error through residual membrane potential, 2023a.
Hao, Z., Ding. J., Bu, T., Huang, T., and Yu, Z. Bridging the gap between anns and snns by calibrating offset spikes, 2023b.
He, K., Zhang, X., Ren, S., and Sun, J. Deep residual learning for image recognition. pp. 770-778, 06 2016. doi: 10. 10..10...2..2066..000
He, K., Fan, H., Wu, Y., Xie, S., and Girshick, R. Momentum contrast for unsupervised visual representation learning. 2019.
Hu, Y., Wu, Y., Deng, L., and Li, G. Advancing residual learning towards powerful deep spiking neural networks. arXiv preprint 1rXiv:21 2021.
Jeffares, A., Guo, Q. Stenetorp. P., and Moraitis, T. Spike-inspired rank coding for fast and accurate recurrent neural networks. CoRR, abs/21 2021. URL //arxiv////xvv../ abs//110 02865.
2110
Jiang, H., Anumasa, S., De Masi, G., Xiong, H., and Gu, B. A unified optimization framework of ann- snn conversion: Towards optimal mapping from activation values to firing rates. In International Conference on Machine Learning, pp. 14945-14974. PMLR, 2023.
Jiang. H., Zoonekynd, V., Masi, G. D., Gu, B., and Xiong, H. TAB: Temporal accumulated batch normalization in spiking neural networks. In The Twelfth International Conference on Learning Representations, 2024. URL https: /attts::/oppnrre //tp://vieeeeeeeeeeeemeem net.
Kang, B., Xie, S.. Rohrbach, M., Yan, Z., Gordo, A., Feng. J., and Kalantidis, Y. Decoupling representation and classifier for long-taile sailet recognition. 2019.
Krizhevsky, A.. Nair, V., and Hinton, G. Cifar- 10 (canadian institute for advanced research). URL htttp://ww... cs. toronto. wooronto.ddu/kkri... edu/"kriz/ cifar .ttll
Kugele, A., Pfeil, T., Pfeiffer, M., and Chicca, E. Efficient processing of spatio-temporal data streams with spiking neural networks. Frontiers Neuroscience, 14:439, 05 2020. doi: 10.3389/fnins. 2020.00439.
11

---

Kushawaha, R. K., Kumar, S., Banerjee, B., and Velmurugan, R. Distilling spikes: Knowledge distil- lation in spiking neural networks. In 2020 25th International Conference on Pattern Recognition (ICPR), pp. 4536-4543 IEFE, 2021.
Lee, C., Sarwar, S., Panda, P., Srinivasan, G., and Roy, K. Enabling spike-based backpropagation for training deep neural network architectures. Frontiers in Neuroscience, 14:119, 02 2020. doi: 00388/fninn.2020..000 19.
Levine, S., Finn, C., Darrell, T., and Abbeel, P. End-to-end training of deep visuomotor policies. Journal of Machine Learning Research, 17(1):1334 1373, 2015.
Li, Cifarl dv-ss: An event-stream dataset for object classification. Frontiers in Neuroscience, 11, 05 2017. doi: 0..388//ninn..011..000
Li, Y. and Zeng, Y. Efficient and accurate conversion of spiking neural network with burst spikes, 2022.
Li, Y., Deng. S., Dong, X., Gong, R., and Gu, S. A free lunch from ann: Towards efficient, accurate spiking neural networks calibration. In Proceedings of the 38th International Conference on Machine Learning, pp. 6316-6325, 2021a.
Li, Y., Guo, Y., Zhang, S., Deng. S., Hai, Y., and Gu, Differentiable spike: Rethinking gradient- descent for training spiking neural networks. Advances in Neural Information Processing Systems, 34, 2021b.
Lu, S. and Sengupta, A. Exploring the connection between binary and spiking neural networks. Frontiers in Neuroscience, 14:535, 06 2020. doi: 10.3389/miiss220000055...
Meng, Q. Xiao, M., Yan, S., Wang, Y., Lin, Z., and Luo, Z.-Q. Training high-performance low-latency spiking neural networks by differentiation on spike representation, 2023.
Ming, Q. Miao, L., Ma, Z., Zhao, L., Zhou, Z.., Huang, X., Chen, Y., and Guo, Deep dive into gradients: Better optimization for 3d object detection with gradient-correcter iou supervision. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 5136-5145, 2023.
Park, S., Kim, S., Na, B., and Yoon, S. T2fsnn: Deep spiking neural networks with time-to-first-siie coding. 2020.
Rathi, and Roy, K. DIET-SNN: direct input encoding with leakage and threshold optimization in deep spiking neural networks. CoRR, abs/2008.03655. 2020. httpp:///////iiiii /
Rathi, N., Srinivasan, G., Panda, P., and Roy, K. Enabling deep spiking neural networks with hybrid conversion and spike timing dependent backpropagation. 05 2020.
Ren, S., He, K., Girshick, R., and Sun, J. Faster r-cnn: Towards real-time object detection with region proposal networks. pp. 1-10, 01 2016.
Sengupta, A., Ye, Y., Wang, R., Liu, C., and Roy, K. Going deeper in spiking neural networks: Vgg and residual architectures. Frontiers in Neuroscience, 13, 02 2018. doi: 10.3389/fnins.0111 2019.00095.
Simonyan, K. and Zisserman, A. Very deep convolutional networks for large-scale image recognition. arXiv 1409. 1556, 09 2014.
Takuya, S., Zhang, R., and Nakashima, Y. Training low-latency spiking neural network through knowledge distillation. In 2021 IEEE Symposium in Low-Power and High-Speed Chips (COOL CHIPS), pp. 1-3. IEEE, 2021.
Tian, Y., Krishnan, D., and Isola, P. Contrastive representation distillation. In International Conference on Learning Representations, 2020.
van der Maaten, L. and Hinton, G. Visualizing data using t-sne. Journal of Muchine Learning Research, 986):2579-2605, 2008 URL, http:://mm org/ wwtp///gg/eee.... papers/ / vandermaatenn08a html
12

---

Wang, S., Cheng, T. H., and Lim, M.-H. LTMD: Learning improvement of spiking neural networks with learnable thresholding neurons and moderate dropout. Oh, A. H., Agarwal. A., Belgrave, D., and Cho, K. (eds.), Advances in Neural Information Processing Systems, 2022. URL https
7/
Wang. S., Schmutz, V., Bellec, G., and Gerstner, W. Mesoscopic modeling of hidden spiking neurons, 2023.
Wu, J., Pan, Z., Zhang, M., Das, R. K., Chua, Y., and Li, H. Robust sound recognition: A neuromor- phic approach. In Interspeech, pp. 3667-3668, 2019a.
Wu, Y., Deng, L., Li, G., Zhu, J., Xie, Y., and Shi, L. Direct training for spiking neural networks:
Faster, larger, better. Proceedings of the AAAI Conference on Artificial Intelligence, 33:1311-1318, 07 2019b. doi: 10.
Wu, Z., Zhang, H., Lin, Y., Li, G., Wang, M., and Tang, Y. Liaf-net: Leaky integrate and analog fire network for lightweight and efficient spatiotemporal information processing. IEEE Transactions Neural Networks and Learning Systems, 33(11):6249 6262, 2022. doi: 10.1109/TNNLS.. 2021. 3073016.
Xiao, M., Meng, Q.. Zhang, Z., He, D., and Lin, Z. Online training through time for spiking neural networks. arXiv preprint 1rX11:2200001195, 2022.
Xu, Q. Li, Y., Fang, X., Shen, J., Liu, J. K., Tang, H., and Pan, G. Biologically inspired struc- ture learning with reverse knowledge distillation for spiking neural networks. arXiv preprint 2344::2304.95500 2023a.
Xu, Q., Li, Y., Shen, J., Liu, J. K., Tang, H., and Pan, G. Constructing deep spiking neural networks from artificial neural networks with knowledge distillation. arXiv preprint Ntw:2334.0562.., 2023b.
Yao, X., Li, F., Mo, Z., and Cheng, J. Glif: A unified gated leaky integrate-and-fire neuron for spiking neural networks. arXiv preprint 2rXiv:2210.13760 2022.
Zhang. M., Wu, J., Belatreche, A., Pan, Z., Xie, X., Chua, Y., Li, G., Qu, H., and Li, H. Supervised learning in spiking neural networks with synaptic delay-weight plasticity. Neurocomputing. 409: 103-118, 2020.
Zhang, W. and Li, Temporal spike sequence learning via backpropagation for deep spiking neural networks. 02 2020.
Zhang, Y., Liu, X., Chen, Y., Peng. W., Guo, Y., Huang, X., and Ma, 7. Enhancing representation of spiking neural networks via similarity-sensitiv contrastive learning. Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp- 16926-16934, 2024.
Zheng, H., Wu, Y., Deng, L., Hu, Y., and Li, G. Going deeper with directly-trained -trained larger spiking neural networks. 10 2020.
Zhou, Z., Zhu, Y., He, C., Wang, Y., YAN, S., Tian, Y., and Yuan, L. Spikformer: When spiking neural network meets transformer. In The Eleventh International Conference on Learning Representations, 2023. URL (tRtshtps: //opp/r/////////////// eview.
Zhu, Y., Yu, Z., Fang, W., Xie, X., Huang, T., and Masquelier, T. Training spiking neural networks with event-driven backpropagation. In 36th Conference on Neural Information Processing Systems (NeurIPS 2022). 2022.
13

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes].
Justification: We clearly state the claims made and the contributions made in both the abstract and introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [NA].
Justification:
We find no limitation which we feel must be specifically highlighted here.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes].
14

---

Justification:
We provide the full set of assumptions and complete proofs in the Section 3.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes].
Justification:
We provide the detail experiment settings in the Section 5.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
15

---

Answer: [Yes].
Justification: We provide open access to the data and code with sufficient instructions in the supplemental material.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cm ec/
for more details.
While encourage the release of code and data, understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https for more details.
//nips. //nips.cc/pil cc/public/guu httpggidddeee/eebbbiiinnneee
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes].
Justification: All implementations described in the experiments section.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes].
Justification:
We report the mean as well as the standard deviation accuracy in experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
16

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes].
Justification:
The computation resources description is provided the appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question:
Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes].
Justification:
The research conducted with the NeurlPS Code of Ethics
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [No].
Justification: There is no societal impact of the work performed.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no socictal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
17

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA].
Justification:
The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes].
Justification: The original paper for datasets used are all cited.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
18

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA].
Justification:
We adopt public datasets.
Guidelines:
The answer NA means that the paper does not release new assets,
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA].
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA].
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
19


Reviews:
[{"review_id": "yKCgHIv43k", "reviewer": "Reviewer_bdtv", "summary": "This paper explores whether the benefits of rich output feature representations, known to enhance the accuracy of ANN models for classification, also apply to SNNs. If so, the authors seek to improve the feature representation of SNNs.\n\nThe authors address this problem in two steps: first, they align the SNN output features with the ANN output features using KL-divergence loss, similar to knowledge distillation. Second, they replace the last Leaky Integrate-and-Fire (LIF) activation layer in the SNN with a ReLU activation layer.\n\nFor feature alignment, the same input is fed to both the SNN and the ANN, and their outputs are compared and aligned using KL-divergence loss. The idea is to use the trained ANN's rich output features to guide the learning of the SNN's output feature representation.\n\nThe proposed method involves training the ANN counterpart (with ReLU activation), then replacing the LIF with the ReLU activation function in the last layer of the SNN. Finally, the modified SNN is trained with the help of the pre-trained ANN counterpart by incorporating the KL loss into the total loss of the SNN.", "questions": "1. In this paper, the authors try to use rich information of the weight parameters from a pretrained ANN counterpart to guide the feature representation learning of the SNN. \nDoes this method require training the ANN counterpart and using it to assist in the subsequent training of the SNN? If there is true, it seems the paper trains both the ANN counterpart and the SNN, the training resources will be doubled, how to solve this problem? \n\n\n2. The authors stated that they \"replace the last Leaky Integrate-and-Fire (LIF) activation layer as the ReLU activation layer in the SNN\". Does the improvement mainly come from this replacement of LIF with ReLU? From results of the Ablation experiments in table 1, it shows that using \"w/ RepAct\" (92.50) demenstrates a larger improvement than using \"w/ L_{AF}\" (92.28) compared to the \"baseline\" (90.40) for timestep=1. For other timestep (2 and 4), the results show the same trends. Moreover, the final method using \"w/ L_{AF} & RepAct\" achieved accuracy of 92.66, which is only a very small increase compared to the \"w/ RepAct\" with 92.50. \n\n\n3. In order to check the improvement for the applications, can the authors provide the performance of \"w/ RepAct\" and \"w/ L_{AF}\", \"w/ L_{AF} & RepAct\", the baseline, and the ANN-counterpart for CIFAR10, CIFAR100, ImageNet, CIFAR10-DVS datasets? \n\n\n\n4. Grammer errors, for example Line 42-44, \"Despite the SNN being more energy-efficient compared with the ANN, it suffers unsatisfactory task performance, due to binary spike feature maps of the SNN will result in limited expressiveness and large accuracy degradation compared with full-precision feature maps of the ANN\",", "limitations": "As the above Weaknesses and Questions.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This paper uses the well-trained ANN to guide and improve the learning of the output feature representation of the SNN. The KL-divergence loss (L_{AF} loss) is used to measure the discrepancy of the ANN's and SNN's feature representations when feeding in the same image into the two networks. \n\n2. By replacing the last LIF activation neuron layer in the SNN with ReLU activation layer, the SNN shows a large improvement.", "weaknesses": "1. The proposed method requires training the ANN counterpart first and setting it aside as a pre-trained model. Subsequently, the SNN needs to be trained. This approach essentially involves training the same framework twice, once with ANN activations and once with SNN activations.\n\n2. The method includes replacing the last LIF activation neuron layer in the SNN with a ReLU activation layer, which results in significant improvement. However, it is unclear whether the final improvement is primarily due to this replacement of last-layer's activation function or the feature alignment process.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your efforts in reviewing our paper and your recognition of our effective method and good results. The response to your questions is given piece by piece as follows.\n\n---\n\n**W1**: The proposed method requires training the ANN counterpart first and setting it aside as a pre-trained model. Subsequently, the SNN needs to be trained. This approach essentially involves training the same framework twice, once with ANN activations and once with SNN activations.\n\n**A1**: Sorry for this confusion. The additional training cost is trivial. We take the ResNet20 with 4 timesteps on CIFAR10 to compare. **The training time of ANN is about 6.25% of that of SNN on the same GPU**, as follows: since the memory consumption of SNN is 4 times that of ANN, ANN can use a batch size that is 4 times that of SNN, which means that the number of iterations required for ANN training is 25% of that of SNN. In addition, since timestep affects SNN, the training time of ANN in the same iteration is 25% of that of SNN. Overall, the training time of ANN is 6.25% of that of SNN. We report the training cost for 400 epochs running using spiking ResNet20 (batch=128) on CIFAR10 for our methods and the training cost for 100 epochs running using ResNet20 (batch=512) with the same single V100 here.  **The costs are 8348.87s and 561.77s. It can be seen that the additional training cost is trivial.**\n\n---\n\n**W2**: The method includes replacing the last LIF activation neuron layer in the SNN with a ReLU activation layer, which results in significant improvement. However, it is unclear whether the final improvement is primarily due to this replacement of last-layer's activation function or the feature alignment process.\n\n**A2**: Sorry for the confusion. From the ablation study, it can be seen that both the RepAct method and the alignment method can gain huge improvements alone.  However, the accuracy upper of a network is limited, and the difficulty will increase quickly with the accuracy reaching the upper. Thus using them both can not get twice the profit. The same phenomenon can also be seen in many works, e.g., IM-Loss (NeurIPS2022), using IM-Loss and ESG simultaneously would not bring much gain compared with a single.\n\nGuo Y, Chen Y, Zhang L, et al. IM-loss: information maximization loss for spiking neural networks[J]. Advances in Neural Information Processing Systems, 2022, 35: 156-166.\n\n---\n\n**Q1**:  it seems the paper trains both the ANN counterpart and the SNN, the training resources will be doubled, how to solve this problem?\n\n**A1**: Sorry for this confusion. Please see our response to **W1**.\n\n---\n\n**Q2**:  Does the improvement mainly come from this replacement of LIF with ReLU?\n\n**A2**: Sorry for this confusion. Please see our response to **W2**.\n\n---\n\n**Q3**:   In order to check the improvement for the applications, can the authors provide the performance of \"w/ RepAct\" and \"w/ L_{AF}\", \"w/ L_{AF} & RepAct\", the baseline, and the ANN-counterpart for CIFAR10, CIFAR100, ImageNet, CIFAR10-DVS datasets?\n\n**A3**: Thanks for the advice. Here we add more experiments as your suggestion. Note that we use the ResNet20 architecture on CIFAR10 (with 4 timestep), CIFAR100 (with 4 timestep), and CIFAR10-DVS (with 10 timestep) and ResNet18 architecture on ImageNet (with 4 timestep).\n\n| Model\\Dataset | CIFAR10 | CIFAR100 | ImageNet | CIFAR10-DVS |\n| --- | --- | --- | --- | --- |\n| ANN | 95.91% | 75.14% | 71.04% | 81.00% |\n| Vanilla SNN  | 93.85% |  71.77% | 61.07% | 78.70% |\n| Ours w/ RepAct | 94.66% | 72.63% | 63.78% | 79.63% |\n| Ours w/ L_{AF} | 94.39% | 72.98% | 64.86% | 79.34% |\n| Ours with both  | 94.74% | 73.01% | 65.31% | 80.50% |\n\n---\n\n**Q4**:   Grammer errors, for example Line 42-44, \"Despite the SNN being more energy-efficient compared with the ANN, it suffers unsatisfactory task performance, due to binary spike feature maps of the SNN will result in limited expressiveness and large accuracy degradation compared with full-precision feature maps of the ANN\",\n\n**A4**: Thanks for the reminder. We will further recheck and polish our paper in the final version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "KuJehM8UyZ", "reviewer": "Reviewer_8VEP", "summary": "This paper introduces methods to improve the output feature representation of Spiking Neural Networks (SNNs) by utilizing knowledge distillation (KD) techniques and modifying activation functions. The approach involves aligning the SNN's output features with those of a pre-trained Artificial Neural Network (ANN) using KL-divergence loss and replacing the last Leaky Integrate-and-Fire (LIF) activation with a ReLU activation to enhance feature precision. These modifications are shown to increase the expressiveness and performance of SNNs on both static and neuromorphic datasets.", "questions": "+ Could the authors extend their comparisons to include seminal works in SNN+KD such as those by Guo et al. [1] and Deng et al. [2]? This would help in positioning their method more clearly within the current research landscape.\n\n+ The ablation study in Table 6 is critical, and the details of the hyperparameters used in the vanilla KD need to be listed. Given that the paper mentions that \"most knowledge distillation methods perform poorly without well-designed hyper-parameters,\" the design of KD should be further discussed. For example, the inclusion of a temperature coefficient commonly used in logits-based KD [3] should be considered (in Eq. 2, etc) to potentially improve the alignment strategy.\n\n\nReference: \n\n[1] Guo, Yufei, et al. \"Joint a-snn: Joint training of artificial and spiking neural networks via self-distillation and weight factorization.\" Pattern Recognition 142 (2023): 109639.\n\n[2] Deng, Shikuang, et al. \"Surrogate module learning: Reduce the gradient error accumulation in training spiking neural networks.\" International Conference on Machine Learning. PMLR, 2023.\n\n[3] Hinton, Geoffrey, Oriol Vinyals, and Jeff Dean. \"Distilling the knowledge in a neural network.\" arXiv preprint arXiv:1503.02531 (2015).", "limitations": "See Weaknesses and Questions.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. __Technical Soundness__: The experimental results support the effectiveness of the proposed methods, albeit not conclusively when considering potentially more efficient alternatives.\n\n2. __Clarity__: The paper is well-written with a clear structure. The methods are described in detail, making it easy for readers to understand the proposed enhancements.\n\n3. __Energy Estimation__: The discussion on energy estimation provides valuable insights into the practical implications of the proposed methods, highlighting their potential for energy-efficient computing.", "weaknesses": "1. __Novelty__: The methods, while integrating known techniques in a new context, do not present strong novelty. While the combination of techniques is original, the individual components (knowledge distillation and activation function manipulation) are well-trodden areas in SNNs. The paper could benefit from a deeper exploration of how these specific adaptations contribute uniquely to the SNNs' context.\n\n2. __Citation Format__: The paper does not adhere to NeurIPS citation format requirements.\n\n3. __Comparative Analysis__: The paper lacks a comprehensive comparison with other SNN training methods that use similar KD approaches, which is crucial for substantiating the claimed improvements. A broader comparison with similar approaches in the field of SNNs+KD (see [1, 2]) could better highlight this. \n\n4. __Significance__: The improvements, while statistically valuable, may not represent a paradigm shift in SNN training that the field might expect for a high-impact publication.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your efforts in reviewing our paper and your recognition of our good writing and technical soundness experimental results. The response to your questions is given piece by piece as follows.\n\n**W1**: The paper could benefit from a deeper exploration of how these specific adaptations contribute uniquely to the SNNs' context.\n\n**A1**: Thanks for your advice.  The starting point of our research is that a rich output feature representation is important for improving the accuracy of SNN first. Based on this perspective, we propose the L_{AF} Loss and RepAct. \n\nFor the L_{AF} Loss, it is not merely a distillation method. The primary motivation behind its proposal is that many ANN-SNN works do not alter the weights of neural networks yet achieve high accuracy. This insight has inspired us to incorporate certain ANN modules into SNN. However, direct migration and training are incompatible. Hence, we introduce the concept of utilizing the classifier of ANN to optimize the output feature of SNN. We have chosen to formalize our idea through distillation and, in the supplementary materials, we provide evidence of its significant advantages over direct distillation. Thus, from a systemic standpoint, the method we propose is a distillation approach specifically tailored for SNN, which fundamentally differs from simple distillation despite their superficial similarities. \n\nFor the RepAct, this method is very simple but effective. Maybe it is not very novel, but will provide a novel insight for the SNN field that enhancing the output feature is very useful for accuracy.\n\nWe will further clarify this in the revised version as your suggestion. Thanks.\n\n---\n\n**W2**: The paper does not adhere to NeurIPS citation format requirements.\n\n**A2**: Thanks for the reminder. We will further recheck and polish our paper in the final version.\n\n---\n\n**W3**: The paper lacks a comprehensive comparison with other SNN training methods that use similar KD approaches.\n\n**A3**: Thanks for your advice. Here we add some comparison results for the same models with [1] and [2] based on their reported results. It can be seen that our method works well on ImageNet with ResNet-18 and on CIFAR10 with ResNet-19, but a little worse on CIFAR10 with ResNet-18. We will report more comparisons in the revised version. Thank you. \n\n| Dataset | Model | Timestep | Accuracy |\n| --- | --- | --- | --- |\n| CIFAR10 | Our ResNet-19 | 2 | 96.19% |\n|  | Our ResNet-18 | 2 | 93.86% |\n|  | Surrogate Module ResNet-19 [2] | 4 | 95.58% |\n|  | Surrogate Module ResNet-18 [2] | 2 | 94.58% |\n|  | Joint ResNet-18 [1] | 2 | 94.01% |\n| ImageNet | Our ResNet-18 | 4 | 65.31% |\n|  | Surrogate Module ResNet-18 [2] | 4 | 64.53% |\n\n---\n\n**W4**: The improvements, while statistically valuable, may not represent a paradigm shift in SNN training that the field might expect for a high-impact publication.\n\n**A4**: Sorry for the confusion.  We think our most valuable contribution is providing insight for the SNN field that enhancing the output feature is much useful for accuracy. Our $L_{AF}$ and RepAct both are means, while our intention is to improve the output feature and prove that it is beneficial for boosting accuracy.  With verified experiments and under the perspective, we believe this work is much beneficial for the following SNN work. And many other methods to improve the output feature would be proposed, like special temporal contrastive loss or simply increasing channels for SNNs. Yet, this is not to say our methods are not novel. For RepAct, it is very simple, but effective. With a trivial additional cost, it can improve the SNN greatly. From a practical perspective, this is a really good choice for the industry. For $L_{AF}$ it is a simple and effective KD method designed for SNN, different from these straightforward KDs. In specific, we introduce its novelty in two-fold. First, providing a simple but effective KD for SNNs is difficult. Most of KDs perform poorly without well-designed hyper-parameters, which have been comprehensively and systematically studied in [1] (see Tab.1 and Tab.2 in [1]. This paper gets more than 700 cites and 1.8k stars). Most KD methods only work well on small datasets, especially in the SNN field.  Second, our $L_{AF}$ comes from both the KDs and ANN-SNN methods.  Some ANN-SNN methods that directly copy-paste the weight parameters from trained ANN to homogeneous SNN with minor changes can obtain a well-performed SNN. This inspires us that utilizing homogeneous ANN's some elements to guide the learning of SNN may be useful. Along with considering the classification and representation, we utilize ANN’s pre-trained classifier to train the SNN’s feature and it works well. Thus, we think $L_{AF}$ is new, simple, designed specifically for SNNs.\n\n[1] Tian, Y., Krishnan, D., and Isola, P. Contrastive representation distillation. In *International Conference on Learning Representations*, 2020.\n\n---\n\n**Q1**: Could the authors extend their comparisons to include seminal works in SNN+KD such as those by Guo et al. [1] and Deng et al. [2]? \n\n**A1**: Thanks for your advice. Please see our response for the **W3**. \n\n---\n\n**Q2**: The ablation study in Table 6 is critical, and the details of the hyperparameters used in the vanilla KD need to be listed. \n\n**A2**: Thanks for your advice. The hyperparameter we used in ablation is 0.1.  Here we also report the results of 0.5 and 1.0. It can be seen that the hyperparameter will affect the results. However, the effect is smaller on ours.\n\n| Dataset | Method | Timestep | hyperparameter=0.1 | hyperparameter=0.5 | hyperparameter=1.0 |\n| --- | --- | --- | --- | --- | --- |\n| CIFAR10 | Baseline | 1 |  90.40% |  90.40% |  90.40% |\n|  | Direct distillation | 1 | 91.59% | 91.03% | 90.63% |\n|  | Ours | 1 | 92.28% | 91.96% | 91.78% |\n|  | Baseline | 2 |  92.80% | 92.80% | 92.80% |\n|  | Direct distillation | 2 |  92.59% | 92.01% | 91.77% |\n|  | Ours | 2 |  93.53% | 93.13% | 92.99% |\n\n---", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "aoU1txCA2Y", "reviewer": "Reviewer_4HWX", "summary": "The authors propose a novel distillation approach (EnOF) to address the mismatch in output precision between SNN and ANN, which leads to poor distillation performance. EnOF feeds the output feature of SNN into the ANN classification head to obtain its effective mapping output P_s, and then uses distillation learning to make the P_s close to the ANN output P_a. Additionally, the paper analyzes the issue of inadequate SNN representation due to the final LIF activation layer. The authors suggest replacing it with the ReLU activation function. Through the methods presented in this paper, SNN demonstrates significant performance improvements across different datasets.", "questions": "1. Does distillation learning have a temperature coefficient?\n2. Is the optimal $\\lambda$ obtained through search, and does different $\\lambda$ have a significant impact on the final results?", "limitations": "NA", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The method proposed in this paper is simple, effective, and easy to implement. This paper presents a novel knowledge distillation learning concept for two different types of networks whose output representation abilities are different. I believe this article has a certain promoting effect on the SNN training.", "weaknesses": "To illustrate the advances made by EnOF and RepAct, the authors can provide the network Hessian matrix eigenvalues or convergence speed comparisons.\n\nAn intuitive drawback of RepAct is its requirement for high-precision floating-point matrix multiplication operations in the classification head, which undoubtedly increases the computation overhead. I suggest that the authors move the overhead comparison section from the appendix to the main text.\n\nThe main text lacks the definition of the method name \"EnOF.”.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your efforts in reviewing our paper and your recognition of our simple but effective method. The response to your questions is given piece by piece as follows.\n\n---\n\n**W1**: To illustrate the advances made by EnOF and RepAct, the authors can provide the network Hessian matrix eigenvalues or convergence speed comparisons.\n\n**A1**: Thanks for the advice. We will add the convergence speed comparisons as per your suggestion in the revised version. Thanks again.\n\n---\n\n**W2**: I suggest that the authors move the overhead comparison section from the appendix to the main text.\n\n**A2**: Thanks for the advice. We will add the overhead comparison section to the main text as your suggestion in the revised version. Thanks again.\n\n---\n\n**W3**: The main text lacks the definition of the method name \"EnOF.”.\n\n**A3**: Sorry for the confusion. “EnOF” comes from the first letter of “**Enhancing the Output Feature**” from the title. We will further clarify it in the revised version. Thanks again.\n\n---\n\n**Q1**: Does distillation learning have a temperature coefficient?\n\n**A1**: Sorry for the confusion. The distillation learning has a temperature coefficient. We set it as 0.1 for CIFAR dataset and 1 for ImageNet dataset.\n\n---\n\n**Q2**: Is the optimal $\\lambda$ obtained through search, and does different $\\lambda$ have a significant impact on the final results?\n\n**A2**: Thanks for the question. The hyperparameter we used in the paper is 0.1 for CIFAR dataset and 1 for ImageNet dataset.  We get it just from a simple search strategy that chooses several hyperparameters and compares their results. Here we also report the results of 0.5 and 1.0 for CIFAR10. It can be seen that the hyperparameter will affect the results. Nevertheless, Using the KD is better than the Baseline.\n\n| Dataset | Method | Timestep | hyperparameter=0.1 | hyperparameter=0.5 | hyperparameter=1.0 |\n| --- | --- | --- | --- | --- | --- |\n| CIFAR10 | Baseline | 1 |  90.40% |  90.40% |  90.40% |\n|  | Ours | 1 | 92.28% | 91.96% | 91.78% |\n|  | Baseline | 2 |  92.80% | 92.80% | 92.80% |\n|  | Ours | 2 |  93.53% | 93.13% | 92.99% |\n\n---", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Wu1svGp0bE", "reviewer": "Reviewer_BFUv", "summary": "The paper presents the method to improve the representation power of SNNs. To this end, the paper proposed two new techniques to the training of spiking neural networks: 1) To guide learning of the last feature layer by an alignment loss with a pre-trained non-spiking network and 2) to replace the last spiking LIF layer with a continuously valued ReLU layer.", "questions": "Please see weaknesses.", "limitations": "Please see weaknesses.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1)Those two techniques are new and interesting and lead to new SOTA on all the datasets they try. \n2)The paper is well-written.\n3)The paper provided ablation analysis to evaluate the contributions of both proposed techniques.", "weaknesses": "1)\tHow can SNN representation be converted to float activation in eq 1? Is it averaged?\n2)\tImprovement with technique 2 is expected, and it makes the comparison with pure SNNs unfair.\n3)\tThe method needs to train an ANN in addition to the SNN. So the total training time is longer.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your efforts in reviewing our paper and your recognition of our good writing, technical soundness experimental results, and novel methods. The response to your questions is given piece by piece as follows.\n\n**W1**: How can SNN representation be converted to float activation in eq 1? Is it averaged?\n\n**A1**: Sorry for the confusion. Yes, it is averaged through timesteps. We will further clarify this as per your suggestion in the revised version. Thanks again.\n\n---\n\n**W2**: Improvement with technique 2 is expected, and it makes the comparison with pure SNNs unfair.\n\n**A2**: Sorry for the confusion. Though the technique 2 will increase the energy consumption， it is very simple and effective. In the appendix, we show that the method just brings about a 1.0% energy consumption uplift. Thus the additional energy consumption could be ignored. In this sense, the comparison could be seen as fair.\n\n---\n\n**W3**: The method needs to train an ANN in addition to the SNN. So the total training time is longer.\n\n**A3**: Sorry for the confusion. The additional training cost is trivial. We take the ResNet20 with 4 timesteps on CIFAR10 to compare. **The training time of ANN is about 6.25% of that of SNN on the same GPU**, as follows: since the memory consumption of SNN is 4 times that of ANN, ANN can use a batch size that is 4 times that of SNN, which means that the number of iterations required for ANN training is 25% of that of SNN. In addition, since timestep affects SNN, the training time of ANN in the same iteration is 25% of that of SNN. Overall, the training time of ANN is 6.25% of that of SNN. We report the training cost for 400 epochs running using spiking ResNet20 (batch=128) on CIFAR10 for our methods and the training cost for 100 epochs running using ResNet20 (batch=512) with the same single V100 here.  **The costs are 8348.87s and 561.77s. It can be seen that the additional training cost is trivial.**", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper has received positive reviews. All reviewers agree that the paper has novelty, clarity and boosts the effectiveness of training for SNNs.
Decision: Accept (poster)