Enhanced Self-Distillation Framework for Efficient Spiking Neural Network Training
Authors: Xiaochen Zhao
Chengting Yu
Kairong Yu
Lei Liu
Aili Wang
Venue: NeurIPS 2025 poster ()
Keywords: Spiking Neural Networks
Self-Distillation.
Primary area: neuroscience_and_cognitive_science
TL;DR: We propose a confidence-separation self-distillation strategy and train SNNs using rate coding, achieving excellent performance with minimal training overhead.

Abstract: Spiking Neural Networks (SNNs) exhibit exceptional energy efficiency on neuromorphic hardware due to their sparse activation patterns. However, conventional training methods based on surrogate gradients and Backpropagation Through Time (BPTT) not only lag behind Artificial Neural Networks (ANNs) in performance, but also incur significant computational and memory overheads that grow linearly with the temporal dimension. To enable high-performance SNN training under limited computational resources, we propose an enhanced self-distillation framework, jointly optimized with rate-based backpropagation. Specifically, the firing rates of intermediate SNN layers are projected onto lightweight ANN branches, and high-quality knowledge generated by the model itself is used to optimize substructures through the ANN pathways. Unlike traditional self-distillation paradigms, we observe that low-quality self-generated knowledge may hinder convergence. To address this, we decouple the teacher signal into reliable and unreliable components, ensuring that only reliable knowledge is used to guide the optimization of the model. Extensive experiments on CIFAR-10, CIFAR-100, CIFAR10-DVS, and ImageNet demonstrate that our method reduces training complexity while achieving high-performance SNN training. Our code is available at https://github.com/Intelli-Chip-Lab/enhanced-self-distillation-framework-for-snn.

Enhanced Self-Distillation Framework for Efficient Spiking Neural Network Training
Xiaochen Zhao
Chengting Yu
Kairong Yu Lei Liu' Aili Wang
ZJU-UIUC Institute, Zhejiang University
2 College of Information Science and Electronic Engineering, Zhejiang University {xiaochen 24, chengting. 21, a ailiaag)nnnt zzj. edu.cn
Abstract
Spiking Neural Networks (SNNs) exhibit exceptional energy efficiency neuro- morphic hardware due to their sparse activation patterns. However, conventional training methods based surrogate gradients and Backpropagation Through Time (BPTT) not only lag behind Artificial Neural Networks (ANNs) in performance, but also incur significant computational and memory overheads that grow linearly with the temporal dimension. To enable high-performance SNN training under lim- ited computational resources, we propose an enhanced self-distillation framework, jointly optimized with rate-based backpropagation. Specifically, the firing rates intermediate SNN layers are projected onto lightweight ANN branches, and high-quality knowledge generated by the model itself is used to optimize substruc- tures through the ANN pathways. Unlike traditional self-distillation paradigms, we observe that low-quality self reff-genertted knowledge may hinder convergence. To address this, we decouple the teacher signal into reliable and unreliable compo- nents, ensuring that only reliable knowledge is used to guide the optimization of the model. Extensive experiments on CIFAR-10 10, CIFAR-100, 100, IO-DRS, and Im- ageNet demonstrate that our method reduces training complexity while achieving nigh-performance SNN training. Our code is available at https://github.cco wttpgiiththu.cmnelllliiiii Chip- www.:a/iiiiiiilaaiooteeeeeeoeeoa
Intorduction
Spiking Neural Networks, which emulate the dynamic behavior of biological neurons 52 transmit
information between neurons through binary spike events across synapses [47 [56]. This spike- based
architectural design enables remarkable energy efficiency on neuromorphic hardware N g 54). However, mainstream training for SNNs relies on time-unfolded chain rules and Backpropagation Through Time (BPTT). Due to the intrinsic temporal dynamics of SNNs, training with BPTT requires storing the entire computational graph across all time steps, resulting in significant memory and computational overhead 42
To address this issue, recent studies have begun focusing on improving the training efficiency of SNNs by decoupling the chain rule, aiming to reduce the complexity of BPTT reduce both time and memory consumption while achieving performance comparable to BPTT 86 77.. However, the simplifications introduced the gradient computation graph often lead to gradient distortions in intermediate layers, which tend to accumulate and become more pronounced as the network depth increases [77]. Recent works have SNNs exhibit feature representations based on firing rates that closely resemble those of artificial neural networks (ANNs) 44 772.. We adhere to the rate-coding assumption (77), treating the spiking rate as the fundamental unit of training and designing dedicated strategies specifically tailored to it. This work proposes introducing
35 69
68).
49
69
86
77 48). These approaches sigmificantly
Equal contribution *Corresponding author
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

lightweight auxiliary ANN branches to map the firing rate from intermediate SNN layers through the corresponding auxiliary ANN branches and guiding them with additional supervision signals. The auxiliary branches are supposed to backpropagate more accurate gradients to the associated substructures and intermediate implicit rate representations, mitigating the gradient distortion issues inherent in efficient training frameworks. Importantly, the backpropagation in branches operates solely on rate-based activations during both forward and backward passes, making it highly compatible with the rate-based framework and preserving its advantages in training efficiency on time and memory overhead. Moreover, as auxiliary branches are discarded during inference, our method incurs no additional computational cost at inference time. A quick comparison is shown in Fig. [ illustrating the positioning of our method within the landscape of direct SNN training schemes.
Knowledge Distillation (KD) is an effective strat- egy for enhancing model performance by leverag- ing soft labels provided by a teacher model [31 84
81.5
tep-1
memory/Mii
BPTT
75,35
25.30
7133.21
SM
72.49
26.10
9094.34
TWSD
79:80
2911:78
and has recently been applied in SNNs training
[22] 72 [7] 32 78. Building on logits-based KD,
accuracy (8)
30.0
80.5
Our
Our
BAGD
172
79:40
$9.20
19.40
2381.91
the self-distillation framework [2 was proposed to leverage the model's own knowledge to guide its learning process, thereby eliminating the need for an additional pre- trainrd teacher model. The idea be- hind self-distillation is that the final layer, benefiting from the entire network, can produce higher-quality predictions, which can be used teacher labels to optimize the learning of intermediate layers [82 12].
79.5
EAGD
79.0
Rate
78.0
20.0
22.5
25.0
27.5
30.0
32.5
35.0
training time (ms)
Figure 1: The performance of different train- ing methods on ResNet-18 18 with the CIFAR- 100 dataset shows that our approach achieves superior training efficiency and accuracy com- pared to current mainstream methods.
However, in the self-distillation framework for SNNs, we observe that the teacher labels are inherently as- sociated with different stages of the model's 's training [74) and their convergence rates may vary due to the varying structural complexities across layers. We introduce the concept of label reliability to measure the ability of teacher signal to effectively guide the student model toward correct convergence. Our observations indicate that the final layer's output is not always the most reliable throughout training, and statically assigning it as the teacher may lead the model to converge to a suboptimal solution. To address this issue, we propose an enhanced self-distillation framework that disentangles the reliable and unreliable components from the predictions of multiple branches, aggregating only the reliable parts to form teacher labels. The framework enables the model to iteratively leverage its own reliable knowledge while mitigating the negative impact of unreliable outputs on the student model, thereby improving the effectiveness of self-distillation in optimizing the backbone network. Our contributions can be summarized as follows:
We establish mapping between the firing rates of intermediate layers and the ANN branch, optimizing the gradient errors of the intermediate layers under rate-based backpropagation. This results in outstanding performance with extremely low training cost.
We analyze the reliability of teacher signals in the self-distillation process and propose a novel decoupling strategy that separates reliable and unreliable components, thereby constructing more stable and effective self-distillation framework.
We conduct empirical validation on standard datasets, including 10, CIFAR-100, CIFARIO-DVS and ImageNet, and perform ablation studies on various model components. Our results demonstrate that the proposed framework effectively balances training efficiency and high performance, offering clear ad advantages over existing methods.
2 Related Work
2.1 Training Methods in SNNs
The primary training approaches for Spiking Neural Networks (SNNs) include conversion-based methods and direct training methods. The former establishes a connection between SNNs and Artificial Neural Networks (ANNs) via an equivalent closed-form mapping, converting a pre-trained ANN into the target SNN model. This avoids the challenge of training SNNs from scratch. Although

---

several studies 10
26,
27
41
have achieved nearly lossless accuracy through such conversions the resulting SNNS typically require longer inference times match In contrast, direct training methods compute
28
59
57
T0
14.
the accuracy of the original ANN
S
39
30
29
gradients of discrete spikes using surrogate gradients and Backpropagation Through Time (BPTT), allowing SNNs to achieve competitive performance with very few time steps [1] 60 67 20 75 85
66 83 73.. However, since the training methods rely on temporal backpropagation. they 80 42 61
introduce both time and memory overhead that scales linearly with the number of time steps, as the computational graph must be retained during training 42 35
68
69 48 12]. Recent works have
proposed methods to decouple the temporal dimension of the backpropagation computation graph [50
48 86). Among them, [77] proposed a targeted training framework for rate- based representations, simplifying the cost of backpropagation- based training significantly. However, efficient training schemes require temporal approximations, which consequently introduce additional gradient emrors that accumulate as the network depth increases.
53 65 79
3
76 69
2.2 Knowledge Distillation in SNNs
Knowledge distillation a common transfer learning technique that facilitates the transfer of knowl-
edge from a high-capacity model to lower-capacity one [3 45 62
64
33
[1]], enabling smaller
student model to approximate the performance of a larger teacher model. In the context of Spiking
Neural Networks, typical strategy involves using pre-trained ANN or a larger SNNs to guide the
training of a smaller SNN [37 38 63 tion method based on both logits and feature representations, while 32 introduces hierarchical feature distillation framework. Although these methods have demonstrated the effectiveness of knowl- edge distillation for SNNs across various datasets, they share common limitation: the requirement for additional training on larger networks, which incurs significant computational overhead. [22 12]
81 70 24). Among them, KDSNN (71) adopts a joint distilla-
proposed a self-distillation strategy that leverages the model's own knowledge to guide the learning of different parts of the network. However, in the process of self-distillation, the quality of teacher labels is coupled with the training stage of the model. In the early training phase, statically assigned teacher labels may be unreliable, potentially misleading the student model. To address this issue, studies such as (74) have proposed normalized distillation losses and customized soft label schemes, which improve the quality of teacher labels by smoothing the target class probabilities and adjusting the distribution over non-target classes. However, when high-confidence predictions correspond to incorrect classes, the teacher labels may exert an even stronger misleading effect on the student.
3 Method
3.1 Preliminary
Inspired by the mechanism of discrete pulse transmission in the brain, Spiking Neural Networks (SNNs) utilize spiking neurons as their basic computational units. Among these models, the Leaky Amtgraaee-and-Fin (LIF) neuron is the most commonly used in SNN training. In this model, when the inputs received by a neuron raise its membrane potential to a specific threshold, a binary pulse is triggered, and subsequently, the membrane potential is reset. The equations for the membrane potential dynamics and spike generation are described as follows:
V [+++1]== A. (v' [t] Vnn 5 (t)) W', .' [t) +
st || H (v" [t+1] -Ve)
(2)
Here, À represents the membrane time constant, Ven is the membrane threshold, W' and 6' denote the weights of the neurons in the /-th layer. ['tt and S'(t] represent the membrane potential and spike emission of the neurons in the 1-th layer at time t, respectively. H--) is the step function that generates spikes, and due to its non-differentiaailiyy. gradient- based methods are used during backpropagation to propagate the error.
3.2 Model training
Distinguished from conventional BPTT, which performs single forward and backward propagation, training procedure is divided into two distinct stages (in Figure 2). In the first stage, forward
3

---

BPTT
Layer1
Layer2
Layer3
Layer4
Our
Layer1
Layer2
Layer3
Layer4
Conv1
Branch1
Branch1
Branch1
Branch1
ReLU
Conv2
Rel U
Spiking Forword
Rate Forword
Backward
Ensemble teacher labels
Compute the CE loss
0000001000
Compute the KL. loss
True Label
Teacher Label
Figure Framework Overview. Unlike the standard BPTT approach, we propose a rate-based framework that first performs a forward pass of temporal spiking activity to update the eligibility traces. In a subsequent forward pass, intermediate layer features are projected onto auxiliary ANN modules. A decoupling module then integrates teacher signals, which are jointly optimized with the ground-trune -tutt labels to supervise the corresponding substructures.
pass is conducted based temporal spike encoding without constructing the computational graph. During this phase, temporal information is utilized to accumulate the running statistics (mean and variance) for batch normalization (BN) layers, while the eligibility traces ely of the spiking neurons as; Ov.
av? ow.," is the decay factor, .S; denotes the spike output of neuron i at time step and V. represents the membrane potential. The auxiliary branch remains inactive during this stage. In the second stage, a rate-based forward pass is performed, during which the eligibility traces computed in the first pass employed approximate the backward gradients: aW,, Dr:
are simultaneously updated. Definition of eligibility trace: ct = =t
r - X
at. åt. [Eeet), where rate r; E[S[].
Rate-coding primarily captures rate-based spatial features, which are inherently aligned with the spatial inductive bias of artificial neural networks (ANNs). As such, the encoded data exhibits strong compatibility with ANN architectures, requiring no additional processing. The auxiliary branch is designed using depthwise separable convolutions and is supervised by additional loss signal, allowing more precise gradients to propagated from the ANN branches to intermediate layers of the main network, thereby promoting more effective convergence. Throughout this process, only a single-step computational graph and a small set of cligibility traces are retained, substantially reducing both memory and computational costs associated with backpropagation.
3.3 Limitations of Standard Self-Distillation
standard self-distillation frameworks, the teacher label is typically derived from the prediction of the final layer, under the assumption that deeper layers, benefiting from the full network capacity, produce higher-quality predictions. This teacher output is then used to supervise intermediate layers. Formally, for deep neural network with L branches, let the output of each branches Mi training

---

Algorithm 1 An algorithm with caption
Require: SNN model sonp. timesteps T, hyper-parameter 3, rfor rate-based backpropagation, Train datasets D {24, 3i 3e ANN block A {Ai, A2, A.....-11111 AL-1)
Ensure: Train the SNN model based on self-distillation
for each batch training data D, {æ4, 31 do
> Spiking Forward
2: Update cligibility traces e
3: end for
4: for each batch training data D, = {zi, Vi do
b Rate Forward
5: Compute each sub-model outputs {PI. P2. pr.
6: Aggregate the teacher labels Yteacher 8
7:
Compute the self-distillation loss as shown in Equation 10)
8:
Compute the CE loss as shown in Equation 2
9:
Compute the final loss using Equation (II
10:
Approximate backpropagation with eligibility traces eiy.
11: end for
iteration & be denoted Pj (1) The quality of the prediction layer I can be evaluated using the cross-entropy loss LcE(P) .) with respect to the ground truth label y.
Standard self-distillation assumes that the final layer L consistently yields the best approximation of the ground truth across all layers:
(CE(P .)) S (Cr(p) ( .y), AI E (1, fI....,I.. I. -11 1}
ensuring that the distillation process guides student layers toward a high-quality target distribution.
However, learning dynamics within deep networks are complex. Different layers may converge or reach peak performance at different training stages. In particular, shallow layers with fewer parameters may converge or saturate earlier than deeper layers. We define binary indicator to denote whether the final layer provides the best prediction iteration t:
1, if CCE(PE) (t) 9) 8(t)
min CCE(p| .)
IE(1 1e(1.-..11 L-1.
(4)
0, otherwise
The proportion of training iterations during which the final layer is not the best predictor is given by:
T
Σ 8(1)
=
In our preliminary experiments, we found that 1 = 0.507, indicating that in more than 50% of the training iterations, the final layer prediction is not globally optimal.
When suboptimal teacher predictions are used, they can lead to negative transfer for the student layers. Let (t)) denote the teacher label at time t, and P (t) denote the output of student branches M.. The knowledge distillation loss is defined as:
CRD = KL (q
(6)
When significant portion of teacher labels are inferior to the student predictions as we observed empirically with v 0.507-the expected distillation loss be decomposed as:
E,
[ct]] = (1 n)
LkD
+ 17 E [hbdd]
ggood denotes the knowledge distillation loss under standard conditions. If (C)(q .y)
CCE(PE (t) y), then the teacher (()) is logically inferior to the student and should not be used as a supervisory signal, in this case, we define the loss as Clul Since the KL divergence term guides p (t) toward q²)) this misguidance may lead to performance degradation. Moreover, when the erroneous teacher prediction has high confidence in the incorrect class, its misleading impact is further amplified.
These observations call into question the effectiveness of always using the final layer output as teacher and motivate the development of more robust teacher selection strategies during training.
5

---

3.4 Reliability Self-Distillation
In our training framework, auxiliary branches connected intermediate layers produce multiple outputs, each of which can be regarded as an independent student in an online self-distillation setup. This naturally provides a rich set of candidate teacher signals. straightforward solution is to generate the teacher label by computing weighted average of the predictions from all students. From the perspective of ensemble learning, increasing the number of voting models generally improves the stability of the final prediction. However, since the quality of teacher signals improves gradually during training. directly aggregating all predictions in the early training stages may result in unreliable teacher labels. To address this issue, propose reliability-separdde self-distillation strategy. Specifically, we hypothesize that correctly predicted samples are more likely to exhibit reliable distributions compared to incorrectly predicted ones. As shown in Figure [ we filter out samples that are incorrectly predicted by individual students and aggregate only the correctly predicted ones to form the final teacher label, following the ensemble learning principle. During loss computation, samples that are misclassified by all n students are excluded from the distillation process ensure that only reliable teacher signals participate. While do not deny that unreliable signals can still act as palllaaiieess-preeenetee categories-we apply a small standard regularization term to these samples during loss calculation. This preserves the regularization effect of the original distillation process. Our strategy thus guides the student model to focus on learning from reliable knowledge while avoiding the misleading effects of noisy or unreliable signals.
the student model from becoming overconfident in certain
1=1 11 (arg max Pl arg max y)
yteacher
(
EE=1 I (arg max P: arg max y) +
In the given formulation, prdenotes the predicted distribution of the student. The overall loss function of the model consists of two components: the hard loss computtd between each classifier's output prand the ground truth label, and the soft distillation loss Lesdeeiived from knowledge distillation. The definitions of the hard loss and the soft distillation loss are as follows:
( log (P(o))
Lee
(9)
1=1
Trad
Σ (c)
Pieacher
log
Pieacher (e)
T
#0 + 17- Ri. II Σ
(c) Preacher
0
Pt
(10)
In the aforementioned formula, represents the total number of categories, while Pieacher denotes the teacher label's prediction value for the et category of each sample. R2 represents the regularization applied to the remaining part, 17 is used to control the weight of the regularization. Additionally, we introduce a balancing factor 3 which is used to control the weight of the distillation loss.
Liorget = Lee + 83 Lead
(11)
4 Experiments
4.1 Main Results
We compare our enhanced self-distillation framework with existing direct training methods across multiple classification benchmarks, including static datasets, CIFAR- (36), CIFAR-100 (36), and
ImageNet [9] (Table []. as well neuromorphic datasets CIFARIO-DVS [40] (Table 2). We set
0.3 to control the weight of the auxiliary global distillation loss.
Results on static datasets. Experimental results demonstrate that our method achieves significant accuracy improvements over RateBP. On the CIFAR- 100 dataset, it outperforms RateBP by 1.18% and 1.3 % using ResNet-18 and ResNet- 19, respectively, and achieves 0.71% improvement on the ImageNet dataset. Our method approaches the performance of BPTT-based direct training and distillation approaches that rely on externally pre-trained artificial neural networks (ANNs). In contrast, our framework requires no external ANN teacher models; instead, leverages high-quality knowledge generated by the model's own auxiliary branches to guide and enhance the training

---

Table Comparison of top-1 accuracy (%) averaged over three runs on CIFAR-10. 10, CIFAR-I0O, and ImageNet datasets, 'indicates the use of an additional pre-trained ANN model for distillation. For all experiments on ImageNet, the ResNet-34 model consistently used for training.
Datasels
Training
Method
Architecture
Timestep
CIFARIO Top-1
CIFAR1O0 Top-1 Ace (%)
ImageNet
Top-1 Acc (%)
OTTT (9
online
VGG-11
6
93,52
71.05
65.15
OS 86
online
RosNet-19
95.20
77.86
67.54
94.25
74.24
Dspike +2
BPTT
ResNet-19
93.66
73.35
68.19
93.13
71.68
94.50
74.72
Direct-training
TFT
BPTT
ResNet-19
94.44
74.47
64.79
2
94.16
72.87
SEW-ResNet
BPTT
ResNet-34
67.04
DSR
48
one-step
Res-ei-ReNNet-18
20
95.10
78.50
67.74
RosNet-18
95.61
78.26
RateBP
one slep
94.75
75.97
70.01
96.38
80.83
RosNet-19
96.26
80.71
2
96.23
79.87
BKDSNN 71
BPTT
RosNet-19
94.64
74.95
67.21
SAKD
BPTT
ResNet-19
96.06
80.10
TKS
3
BPTT
RosNet-19
4
96.35
79.89
69.60
ResNet-18
96.04
SM
BPTT
ResNet- 19
96.82
81.70
68.25
distillation
TWKD
BPTT
ResNet-18
4
95,57
95.96
79.80
79.10
71.04
96.14
79.40
ResNet-18
95.92
78.85
7
one-step
77.06
70.64
ResNet-19
96.56
81.44
96.19 + 0.12
80.20 0.11
ResNet-18
96.5 00.92:4 + 0.03
79.30
0.21
ours
ane-step
95.29 2
0.10
77.46
0.17
70.72
96.38 06.39
0.20
2
96.31 + 0.07
80.97 0.05
process. Notably, these performance gains are achieved with constant memory consumption during backpropagation, resulting in 75.80% reduction in memory usage and a 23.30% reduction in time consumption compared BPTT training.
Results on neuromorphic datasets. On the CIFAR1O-DVS neuromorphic dataset, BPTT-hased methods and one-step training methods show differing performance characteristics. While BPTT preserves temporal dynamics more thoroughly, one-step methods focus more on spatiotemporal feature extraction. For example, methods such [72 77 may partially compress the temporal dimension. Consequently, BPTT tends to be more adaptable to neuron-based datasets. To address this challenge, we decouple the backpropagation component of the rate coding mechanism in our method and conduct fair comparison using the same encoding scheme as prior work. Compared to RateBP, our method achieves 1.50% performance gain under the same number of timesteps. Furthermore, when the rate coding component is decoupled, our method also achieves leading performance on the CIFAR10- 10-DVS dataset, indicating its effectiveness under the temporal coding framework well.
4.2 Ablation study
Ablation study on self-distillation. In this section, we analyze the effectiveness of the enhanced self-distillation method on spike-rate-coded SNNs and ANNs. We decouple the self-distillation component within the framework and examine the feedback from classifiers at different depths. Experimental results demonstrate that our self-distillation method offers significant advantages over standard self-distillatian and this observation also holds true for ANNs.
7

---

Table Performance comparison of top-1 accuracy (%) on CIFARIO-DVS and ImageNet, averaged over three experimental runs.
Training
Method
Architecture
Timestep
Top-1 ACC(%)
OTTT
69
online
VGG-11
10
76.63
RateBP
one-step
ResNet-18
10
80.40
EAGD
one-step
ResNet-19
4
80.54
TET
BPTT
VGGSNN
10
83.17
SM 12
BPTT
ResNet-18
10
83.19
Enof 22
BPTT
ResNet-19
10
80.10
TWKD
78
ВРTТ
ResNet-19
83.80 10
ResNet-18
10
81.40
one-step
ResNet-19
10
81.90
ResNet--18
10
85,70
BPTT
ResNet-19
10
85.90
84
10
$1.251
$1.32
31.
461
Basiiin (2.005)
Accursy (%)
80
Bleciine (80066h))
Accunry (%)
75
71.4
m2.47
700
%
76
72
Classifier Index
Classifier Lidex
(a) Comparison RateBP-Based Distillation Methods. (h) Comparison ANN-Based Distillation Methods.
Figure 3: Ablation bar chart of the self-distillation module ANNs and SNNs.
All distillation losses computed based KL divergence. We conduct comparative experiments the CIFAR-100 dataset using ResNet-18 18 model with temporal dimension of T=6. The following methods are compared. SD; Introduces a pre-trained ANN teacher model with the same architecture and performs standard end-to-end distillation based on logits; MB: Adds lightweight auxiliary branches after each layer facilitate the optimization of intermediate representations: ASD: Uses the final layer's predicted labels as teacher signals for intermediate layers; ESD: Decouples the reliability of student-gen geeerraee labels and aggregates them for self seff-distillation training. As shown in Figure 3a and Figure (3b] all methods outperform the original baseline. Standard end-to-end distillation provides limited beneht model training, highlighting the effectiveness of optimizing intermediate layers via auxiliary branches. The multi-branch self-distillation framework achieves superior performance without requiring any external ANN teacher models. Comparing ASD and ESD reveals that ESD avoids the misleading effects caused by unreliable predictions from the final layer in ASD, resulting in significant performance improvements. Our method improves the final classifier accuracy by 0.53%, and boosts the ANN model performance by 0.59%.
Ablation of the Regularization Component.
In general, knowledge distillation is considered to serve two main purposes. First, guides the student's predictions toward a higher-qualit dis- tribution. Second, the soft labels act as a form of regularization, preventing the model from be- coming overly confident in a single target class during training. When all classifiers make incorrect predictions for particular sample, it indicates
Baseline ESD(w/o Reg)
ESD
79.02%
80.20%
80.32%
Table 3: Ablation of the Regularization

---

Table 4: Comparison of average spike frequency across time steps on CIFARI00 using ResNet-18.
Method
T ==1
T 2
T =
T=
T
T 6
avg
BPTT
0.1799
0.2137
0.2045
0.2091
0.2018
Trained for 4 time steps
ours
0.1591
0.1709
0.1715
0.1706
0.1680
BPTT
0.1761
0.2034
0.2023
0.1966
0.2060
0.1941
0.1964
Trained for time steps
ours
0.1548
0.1560
0.1550
0.1516
0.1550
0.1532
0.1543
that the model has systematic bias in feature representation at a global level. In such cases, the model more likely assign high confidence to incorrect classes. We acknowledge that even when the teacher's 's soft labels are unreliable, they can still play role in smoothing the target distribution. Therefore, in our method, we impose a standard regularization term on this subset of samples. As shown in Table 3 the model achieves better generalization the test set after applying regularization. worth noting, that the middle later stages of training, the teacher soft labels can cover vast majority of samples, making the additional regularization relatively minor in proportion.
4.3 Performance on Energy Efficient Implementation
Impact of time expansion. We compare the proposed method with RateBP and BPTT in terms of the impact of varying time steps on training accuracy and the memory overhead during backpropagation. As shown in Figure 4b the accuracy of method continues to improve with increasing number of time steps, clearly demonstrating its scalability in the temporal dimension. Since the auxiliary branch only participates in the second forward pass and still uses spike-rate encoding, the additional memory and computational overhead introduced during backpropagation remains constant. Figure 44 further illustrates that method effectively decouples the memory of backpropagation from the time step, ensuring that training costs do not increase with larger values of T.
12
BPTT
80
BPTT
10-
Our
Our
Menury (GB)
73
77
2
8
10
2
10
Training timestep
Timestep
(a) Comparison of training cost.
(b) Comparison of test performance.
Figure Comparison of training cost and test performance of time steps.
Spike firing rate analysis. AS shown the table our method significantly reduces the spike frequency each time step compared to the BPTT-based approach. This indicates that our method achieves sparser neural activity and lower energy consumption while maintaining competitive perfor- mance, aligning better with the original design goals of Spiking Neural Networks (SNNs).
5
Conclusion
This study proposes an enhanced self-distillation framework for efficiently training spiking neural networks (SNNs). Compared to the conventional BPTT approach, our method significantly reduces memory and time consumption during training, while leveraging auxiliary ANN branches to mitigate gradient errors intermediate layers. Moreover, by selectively integrating high- and low-reliability predictions from multiple classifiers, we construct high-quality teacher signals that enable the model better absorb valuable self-generated knowledge. This addresses critical issue in standard self- distillation, where unreliable teacher labels can hinder the student model from converging in the

---

right direction. Experimental results demonstrate that our method achieves tegh-performance SNN training even under highly constrained computational settings.
6
Acknowledgments
This work was supported by the National Natural Science Foundation of China (Grant No. 62304203), international campus of ZJU international research collaboration seed project, and the ZJU-YST joint research center for fundamental science.
References
[1] Filipp Akopyan, Jun Sawada, Andrew Cassidy, Rodrigo Alvarez-Icaza, John Arthur, Paul Merolla, Nabil Imam, Yutaka Nakamura, Pallab Datta, Gi-Joon Nam, ct al. Truenorth: Design and tool flow of a 65 mw million neuron programmable neurosynaptic chip. IEEE transactions computer-aided design of integrated circuits and systems, 34(10):157--1557, 2015.
[2] Guillaume Bellec, Franz Scherr, Anand Subramoney, Elias Hajek, Darjan Salaj, Robert Legenstein, and Wolfgang Maass. A solution to the learning dilemma for recurrent networks of spiking neurons. Nature communications, 1(1):3625, 2020.
[3] Thomas Bohnstingl, Stanistaw Wožniak, Angeliki Pantazi, and Evangelos Eleftheriou. Online spatio- learningg in deep neural networks. IEEE Transactions on Neural Networks and Learning Systems, 34(1 1:8894-89908 2022.
temporal learning
[4] Tong Bu, Jianhao Ding, Zecheng Hao, and Zhaofei Yu. Rate gradient approximation attack threats deep spiking neural networks. In Proceedings of the IEEF/CVF Conference on Computer Vision and Puttern Recognition, pages 7896-7906, 2023.
[5] Tong Bu, Wei Fang, Jianhao Ding, PengLin Dai, Zhaofci Yu, and Tiejun Huang. Optimal ann-snn conver- sion for high-accuracy and ultra-low-latency spiking neural networks. arXiv preprint 2rX1:23003.0437,, 2023.
[6] Yongqiang Cao, Yang Chen, and Deepak Khosla. Spiking deep convolutional neural networks for energy- efficient object recognition. International Journal of Computer Vision, 113:54-66, 2015.
[7] Ekin D Cubuk, Barret Zoph, Dandelion Mane, Vijay Vasudevan, and Quoc V Lc. Autoaugment: Learning augmentation strategies from data. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 11--223, 2019.
[8] Mike Davies, Narayan Srinivasa, Tsung- Hann Lin, Gautham Chinya, Yongqiang Cao, Sri Harsha Choday, Georgios Dimou, Prasad Joshi, Nabil Imam, Shweta Jain, al. Loihi: A neuromorphic manycore processor with on-chip learning. leee Micro, 38-1)982,99, 2018.
[9] Jia Deng, Wei Dong. Richard Socher, Li-Jia Li, Kai Li, and Fei-Fei. Imagenet: large-scale hicrarchical image database. In 2009 IEEE conference on computer vision and puttern recognition, pages 248-255.
leee, 2009.
[10] Shikuang Deng and Shi Gu. Optimal conversion of conventional artificial neural networks spiking neural networks. arXiv preprins arXiv:2103.000 00476, 2021.
[11] Shikuang Deng, Yuhang Li, Shanghang Zhang, and Shi Gu. Temporal efficient training of spiking neural network via gradient re-weighting. arXiv preprint arXiv:2202.1 /946, 2022.
[12] Shikuang Deng, Hao Lin, Yuhang Li, and Shi Gu. Surrogate module learning: Reduce the gradient error accumulation in training spiking neural networks. In International Conference on Machine Learning, pages 7645-7657. PMLR, 2023.
[13] Peter U Diehl, Daniel Neil, Jonathan Binas, Matthew Cook, Shih-Chii Liu, and Michael Pfeiffer. Fast- classifying. aechracyrracy spiking deep networks through weight and threshold balancing. In 2015 International joint conference on neural networks (IJCNN), pages 1-8. ieee, 2015.
[14] Jianhao Ding, Zhaofci Yu, Yonghong Tian, and Tiejun Huang. Optimal ann-snn conversion for fast and accurate inference in deep spiking neural networks. arXiv preprint arXiv:2 /654, 2021.
[15] Yiting Dong. Dongeheng Zhao, and Yi Zeng. Temporal knowledge sharing enable spiking neural network learning from past and future. IEEE Transactions on Artificial Intelligence. 2024.
[16] Chaoteng Duan, Jianhao Ding, Shiyan Chen, Zhaofei Yu, and Tiejun Huang. Temporal effective batch normalization in spiking neural networks. Advances in Neural Information Processing Systems, 35: 34377-34390, 2022.
[17] Wei Fang. Zhaofei Yu, Yanqi Chen, Tiejun Huang, Timothée Masquelier, and Yonghong Tian. Deep residual learning in spiking neural networks. Advances in Neural Information Processing Systems, 34: 2021.
10

---

[18] Wei Fang, Zhaofei Yu, Yanqi Chen, Timothée Masquelier, Tiejun Huang, and Yonghong Tian. Incorporating learnable membrane time constant enhance learning of spiking neural networks. In Proceedings of the IEEE/CVF international conference on computer vision, pages 2661-2671, 2021.
[19] Wei Fang. Yanqi Chen, Jianhao Ding, Zhaofei Yu, Timothée Masquelier, Ding Chen, Liwei Huang, Huihui Zhou, Guoqi Li, and Yonghong Tian. Spikingielly: An open-source machine learning infrastructure platform for spike-based intelligence. Science Advances, 9(40):eadi 1480, 2023.
[20] Pengjic Gu, Rong Xiao, Gang Pan, and Huajin Tang. Stea: Spatio-temporal credit assignment with delayed feedback in deep spiking neural networks. In IJCAI, volume 15, pages 1366-1377 16661 2019.
[21] Qiushan Guo, Xinjiang Wang. Yichao Wu, Zhipeng Yu, Ding Liang, Xiaolin Hu, and Ping Luo. Online knowledge distillation via collaborative learning. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 1020-11029, 2020.
[22] Yufci Guo, Weihang Peng, Xiaode Liu, Yuanpei Chen, Yuhan Zhang, Xin Tong, Zhou Jic, and Zhe Ma. Enof-snn: Training accurate spiking neural networks via enhancing the output feature. The Thirty-eighth Annual Conference on Neural Information Processing Systems.
[23] Yufei Guo, Yuanpei Chen, Liwen Zhang. Xiaode Liu, Yinglei Wang. Xuhui Huang, and Zhe Ma. Im-loss: information maximization loss for spiking neural networks. Advances in Neural Information Processing Systems, 35:156- 166,-666,222 2022.
[24] Yufei Guo, Weihang Peng. Yuanpei Chen, Liwen Zhang, Xiaode Liu, Xuhui Huang, and Zhe Ma. Joint a-smn: Joint training of artificial and spiking neural networks via self-distillation and weight factorization. Pattern Recognition, 142:109633 2023.
[25] Zhen Guo, Pengzhou Zhang, and Peng Liang. Sakd: Sparse attention knowledge distillation. Image and Vision Computing, 146: 105020, 2024.
[26] Bing Han and Kaushik Roy. Deep spiking neural network: Energy efficiency through time based coding.
In European conference on computer vision, pages 388-404. Springer, 2020.
[27] Bing Han, Gopalakrishnan Srinivasan, and Kaushik Roy. Rmp-snn: Residual membrane potential neuron for enabling deeper high-accuracy and low-latency spiking neural network. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pages 1 13558-13567, 2020.
[28] Bing Han, Gopalakrishnan Srinivasan, and Kaushik Roy. Rmp-snn: Residual membrane potential neuron for enabling deeper
and low-latency spiking neural network. In Proceedings of the IEEE/CVF conference computer vision and pattrm recognition, pages 1 13558-13555 2020.
[29] Zecheng Hao, Tong Bu, Jianhao Ding. Tiejun Huang, and Zhaofei Yu. Reducing ann-snn conversion error through residual membrane potential. In Proceedings of the AAAI Conference on Artificial Intelligence. volume 37, pages 11-21, 2023.
[30] Zecheng Hao, Jianhao Ding, Tong Bu, Tiejun Huang, and Zhaofei Yu. Bridging the gap between anns and snns by calibrating offset spikes. arXiv preprint urXiv:2302. 10685 2023.
[31] Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02533 2015.
[32] Di Hong, Jiangrong Shen, Yu Qi, and Yueming Wang. Lasnn: Layer-wise ann-to-snn distillation for effective and efficient training in deep spiking neural networks. arXiv preprint (rXii:2304.09100 2023.
[33] Mingi Ji, Seungjae Shin, Seunghyun Hwang, Gibcom Park, and Il-Chul Moon. Refine myself by teaching myself:: Feature refinement via self knowledge distillation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10664-10677 2021.
[34] Haiyan Jiang, Srinivas Anumasa. Giulia De Masi, Huan Xiong, and Bin Gu. A unified optimization framework of ann-snn conversion: Towards optimal mapping from activation values to firing rates. In International Conference on Machine Learning, pages 14945-14977 PMLR, 2023.
[35] Jinseok Kim, Kyungsu Kim, and Jae-Joon Kim. Unifying activation-an and timing based learning rules for spiking neural networks. Advances in neural information processing systems, 33: 19534-19544, 2020.
[36] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers features from tiny images. 2009.
[37] Ravi Kumar Kushawaha, Saurabh Kumar, Biplab Banerjee, and Rajbabu Velmurugan. Distilling spikes: Knowledge distillation in spiking neural networks. In 2020 25th International Conference on Pattern Recognition (ICPR). pages 4536-4543. IEEE, 2021.
[38] Dongjin Lee, Seongsik Park, Jongwan Kim, Wuhyeong Doh, and Sungroh Yoon. Energy-efficient knowledge distillation for spiking neural networks. arXiv preprint arXiv:2106.0777 07177, 2021.
[39] Chen Li, Lei Ma, and Steve Furber. Quantization framework for fast spiking neural networks. Frontiers in Neuroscience, 16:918793, 2022.
[40] Hongmin Li, Hanchao Liu, Xiangyang Ji, Guoqi Li, and Luping Shi. Cifar10-dvs: an event-stream dataset for object classification. Frontiers in neuroscience, 11:309, 2017.
11

---

[41] Yuhang Li, Shikuang Deng, Xin Dong, Ruihao Gong, and Shi Gu. A free lunch from ann: Towards efficient, accurate spiking neural networks calibration. In International conference on machine learning. pages 1316-6325, PMLR, 2021.
[42] Yuhang Li, Yufei Guo, Shanghang Zhang, Shikuang Deng, Yongqing Hai, and Shi Gu. Differentiable spike: Rethinking gradient-descent for training spiking neural networks. Advances in neural information processing systems, 34:23466-23439, 2021.
[43] Yuhang Li, Tamar Geller, Youngeun Kim, and Priyadarshini Panda. Scenn: towards temporal spiking early exit ncural networks. Advances in Neural Information Processing Systems, 36:63327-6338 63342, 2023.
[44] Yuhang Li, Youngeun Kim, Hyoungseob Park, and Priyadarshini Panda. Uncovering the representation of spiking neural networks trained with surrogate gradient. arXiv preprint arXiv:2304. 13098, 2023.
[45] Yifan Liu, Ke Chen, Chris Liu, Zengchang Qin, Zhenbo Luo, and Jingdong Wang. Structured knowledge distillation for semantic segmentation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 2604-2664 2613-2263,,20 2019.
[46] Ilya Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts. arXiv preprint
arXiv:1608.0333 03983, 2016.
[47] Wolfgang Maass. Networks of spiking neurons: the third generation of neural network models. Neural networks, 10(9):655-11677 1997.
[48] Qingyan Meng. Mingqing Xiao, Shen Yan, Yisen Wang, Zhouchen Lin, and Zhi-Quan Luo. Training high- performance low-latency spiking neural networks by differentiation on spike representation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12444 12453, 2022.
[49] Qingyan Meng, Mingqing Xiao, Shen Yan, Yisen Wang, Zhouchen Lin, and Zhi-Quan Luo. Towards memory-and time-efficient backpropagation for training spiking neural networks. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 6166-6177 (1666117,,2 2023.
[50] Hesham Mostafa. Supervised learning based on temporal coding in spiking neural networks. IEEE transactions on neural networks and learning systems, 29(7):2277-3255. 2017.
[51] Emre O Neftei, Hesham Mostafa, and Friedemann Zenke. Surrogate gradient learning in spiking neural networks: Bringing the power gradient- based optimization to spiking neural networks. IEEE Signal Processing Magazine, 36(6):51 2019.
[52] Stefano Panzeri and Simon R Schultz. A unified approach to the study of temporal, correlational, and rate coding. Neural Computation, 13(6):1311-1349, 2001.
[53] A Paszke. Pytorch: An imperative style, high-performance deep learning library. arXiv preprint arXiv:192..0//
2019.
[54] Jing Pei, Lei Deng. Sen Song, Mingguo Zhao, Youhui Zhang, Shuang Wu, Guanrui Wang, Zhe Zou, Zhenzhi Wu, Wei He, et al. Towards artificial general intelligence with hybrid tianjic chip architecture. Nature, 572(7767)100-111, 2019.
[55] Nitin Rathi and Kaushik Roy. Diet-snn: A low-latency spiking neural network with direct input encoding and leakage and threshold optimization. IEEE Transactions on Neural Networks and Learning Systems, 34
(6):3174-1182, 2021.
[56] Kaushik Roy, Akhilesh Jaiswal, and Priyadarshini Panda. Towards spike-based machine intelligence with neuromorphic computing. Nature, 575(7784):607 617, 2019.
[57] Bodo Rueckaner, Iulia-Alexandra Lungu, Yuhuang Hu, Michael Pfeiffer, and Shih-Chii Liu. Conversion of continuous-valued deep networks to efficient event-driven networks for image classification. Frontiers in neuroscience. 11:294078, 2017.
[58] David E Rumelhart, Geoffrey E Hinton, and Ronald Williams. Learning representations by back- propagating errors. nature, 323(6088):533 53,-536, 1986.
[59] Abhromil Sengupta, Yuting Ye, Robert Wang, Chiao Liu, and Kaushik Roy. Going deeper in spiking neural networks: Vgg and residual architectures. Frontiers in neuroscience, 13:95, 2019.
[60] Sumit B Shrestha and Garrick Orchard. Slayer: Spike layer error reassignment in time. Advances in neural information processing systems, 31, 2018
[61] Kazuma Suetake, Shin-ichi Ikegawa, Ryuji Saiin, and Yoshihide Sawada. S3nn: Time step reduction of spiking surrogate gradients for training energy efficient single-step spiking neural networks. Neural Networks, 159:208-211 229, 2023.
[62] Siqi Sun, Yu Cheng, Zhe Gan, and Jingjing Liu. Patient knowledge distillation for bert model compression. arXiv preprint arXiv: 003550 2019.
[63] Sugahara Takuya, Renyuan Zhang, and Yasuhiko Nakashima. Training low low-latency spiking neural network through knowledge distillation. In 2021 IEEE Symposium in Low-Power and High-Speed Chips (COOL CHIPS), pages 1-3, IEEE, 2021.
12

---

[64] Lin Wang and Kuk-Jin Yoon. Knowledge distillation and student- learning for visual intelligence:
A review and new outlooks. IEEE transactions on pattern analysis and machine intelligence, 44(6): 3048-3068, 2021.
[65] Ziming Wang. Shuang Lian, Yuhao Zhang, Xiaoxin Cui, Rui Yan, and Huajin Tang. Towards lossless ann-smn conversion under ultra-low lateney with dual-phase optimization. arXiv preprint 22X.: 2022.
[66] Ziming Wang. Runhao Jiang. Shuang Lian, Rui Yan, and Huajin Tang. Adaptive smoothing gradient learning for spiking neural networks. International conference on machine learning, pages 35798-35816 PMLR, 2023.
[67] Yujie Wu, Lei Deng, Guoqi Li, and Luping Shi. Spatio-temporal backpropagation for training high- performance spiking neural networks. Frontiers in neuroscience, 12:323875, 2018.
[68] Mingqing Xiao, Qingyan Meng, Zongpeng Zhang, Yisen Wang, and Zhouchen Lin. Training feedback spiking neural networks by implicit differentiation on the equilibrium state. Advances in neural information processing systems, 34:14516- 1511-11528, 2021.
[69] Mingqing Xiao, Qingyan Meng, Zongpeng Zhang, Di Hc, and Zhouchen Lin. Online training through time for spiking neural networks. Advances neural information processing systems, 35:20717-20730 2022.
[70] Qi Xu, Yaxin Li, Jiangrong Shen, Jian K Liu, Huajin Tang, and Gang Pan. Constructing deep spiking neural networks from artificial neural networks with knowledge distillation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7886-7895, 2023.
[71] Zekai Xu, Kang You, Qinghai Guo, Xiang Wang. and Zhezhi He. Bkdsnn: Enhancing the performance of learning-based spiking neural networks training with blurred knowledge distillation. arXiv preprint arXiv:2407.09083 2024.
[72] Shu Yang, Chengting Yu, Lei Liu, Hanzhi Ma, Aili Wang. and Erping Li. Efficient ann ann- -uidguuddd distillation: Aligning rate- based features of spiking neural networks through hybrid block-wise replacement. arXiv preprint or Xrv::2503. /6572, 2025.
[73] Yukun Yang. Wenrui Zhang, and Peng Li. Backpropagated neighhorhood aggregation for accurate training of spiking neural networks. In International Conference on Machine Learning, pages 11852-11862 PMLR, 2021.
[74] Zhendong Yang. Ailing Zeng. Zhe Li, Tianke Zhang. Chun Yuan, and Yu Li. From knowledge distillation to self knowlegee distillation: A unified approach with normalized loss and customized soft labels. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 17185 17185-17194, 2023.
[75] Bojian Yin, Federico Corradi, and Sander M Bohté. Effective and efficient computation with multiple- timescale spiking recurrent neural networks. In International Conference on Neuromorphic Systems 2020. pages 1-8. 2020.
[76] Bojian Yin, Federico Corradi, and Sander M Bohté. Accurate online training of dynamical spiking neural networks through forward propagation through time. Nature Machine Intelligence, 5(5):518 (55::51--227, 2023.
[77] Chengting Yu, Lei Liu, Gaoang Wang, Erping Li, and Aili Wang. Advancing training efficiency of deep spiking neural networks through rate-based backpropagation. arXiv preprint arXiv:2410. / /4888, 2024.
[78] Chengting Yu. Xiaochen Zhao, Lei Liu, Shu Yang. Gaoang Wang, Erping Li, and Aili Wang. Effi- cient distillation of deep spiking neural networks for full-range timestep deployment. arXiv preprint arXiv:2501. 15925, 2025.
[79] Friedemann Zenke and Surya Ganguli. Superspike: Supervised learning in multilayer spiking neural networks. Neural computation, 3066)11114-1541 2018
[80] Friedemann Zenke and Tim P Vogels. The remarkable robustness of surrogate gradient learning for instilling complex function in spiking neural networks. Neural computation, 33(4):899 925, 2021.
[81] Fengzhao Zhang, Chengting Yu, Hanzhi Ma, Zheming Gu, and Er-ping Li. Knowledge distillation for spiking neural network. In 2023 5th International Conference on Robotics, Intelligent Control and Artificial Intelligence (RICAI), pages 1015-1020. IEFF, 2023.
[82] Linfeng Zhang, Jiebo Song, Anni Gao, Jingwei Chen, Chenglong Bao, and Kaisheng Ma. Be your own teacher: Improve the performance of convolutional neural networks via self distillation. In Proceedings of the IEEF/CVF international conference on computer vision, pages 3713-3722, 2019.
[83] Wenrui Zhang and Peng Li. Temporal spike sequence learning via backpropagation for deep spiking neural networks. Advances in neural information processing systems, 33: 33:12022- 2020.
[84] Borui Zhao, Quan Cui, Renjie Song. Yiyu Qiu, and Jiajun Liang. Decoupled knowledge distillation. In Proceedings of the IEEE/CVF Conference on computer vision and pattern recognition, pages 2022.
13

---

[85] Hanle Zheng, Yujie Wu, Lei Deng, Yifan Hu, and Guoqi Li. Going deeper with directly-trained larger spiking neural networks. In Proceedings of the AAAI conference artificia intelligence, volume 35, pages 11062-11070, 2021.
[86] Yaoyu Zhu, Jianhao Ding. Tiejun Huang. Xiaodong Xie, and Zhaofei Yu, Online stabilization of spiking neural networks. In The Twelfth International Conference on Learning Representations, 2024.
14

---

Appendix
A Experimental Settings
A.1 Datasets
A.1.1 CIFAR-10 and CIFAR-100
The CIFAR-10 and CIFAR-100 36 datasets consist of color images with resolution of 32x32 pixels and cover a range object categories. Both datasets are released under the MIT license. CIFAR- 10 contains 60,000 images across 10 classes, with 50,000 images for training and 10,000 for testing. CIFAR-100 includes 100 classes. Both datasets are preprocessed with zero-mean and unit-variance normalization. Data augmentation strategies follow AutoAugment 7 and Cutout [66,, consistent with recent works [41 5 23 66, 12). At each time step, the raw pixel values are directly encoded 55 and fed into the network input layer.
A.1.2 ImageNet
The ImageNet-IK K dataset [ consists 1,281,167 training images and 50,000 validation images across 1,000 distinct categories, and is available for non-commereial use. All images standardized to have zero mean and unit variance. During training, images are randomly resized and cropped to 224x224 pixels, followed by horizontal flipping. For validation, images are first resized to 256x256 pixels and then center-cropped to 224x224. Similar to the treatment of CIFAR datasets, each image converted into temporal sequence via direct encoding before being fed into the network.
A.1.3 CIFARIO-DVS
The CIFARIO-DVS dataset [40] is neuromorphic version CIFAR-10, 10, consisting 10,000 event- based images captured by Dynamic Vision Sensor (DVS) camera. The images have increased spatial resolution 128x128 pixels and are released under the CC BY 4.0 license. We split the dataset into 9,000 training images and 1,000 test images. Data preprocessing involves accumulating events into frames 19 18 and downsampling the spatial resolution to 48x48 via interpolation. Additional data augmentation includes random horizontal flipping and random translation within 5-pixel range, consistent with previous studies [49 69
A.2 Training Setup
Our experiments adopt sigmoid- based surrogate gradient method [19 to approximate the Heaviside step functtion, defined ash (z, a) = Ite where the parameter ais set to 4. We follow the time- approximate backpropagation strategy from 777, and all implementations are based on the Py PyTorch CIFAR- 10, CIFAR-100, CIFARIO-DVS. and ImageNet datasets are conducted on an NVIDIA GeForce RTX 3090 GPU. For all tasks, we use stochastic gradient descent (SGD) with momentum of 0.9 55 and apply a cosine annealing schedule [46) for learning rate adjustment. Additional hyperparameters are listed in the table 5
[53] and SpikingJelly [19] frameworks. Experiments
Table 5: Hyperparameters Settings.
CIFAR-10
ImageNer
CIFARIO-DVS
Egoch
300
500
100
300
Learnnng
0.1
0.2
0.1
Batch
128
Weight decay
Se-4
5e-4
26-5
Se-4
A.3 Network Architectures
For the CIFAR-10, CIFAR-100, and CIFARIO-DVS datasets, we adopt ResNet-18 and ResNet- as the backbone models. For the ImageNet dataset, ResNet-34 is used as the backbone model. All spiking neural network (SNN) models employ leaky integrate-and-fire (LIF) neurons, with the membrane potential decay factor uniformly set to 0.5. The implementation is based on the activation-driven paradigm proposed by [17].

---

Table 6: Performance under different using ResNet- on CIFARI00.
classifier/3
0.0
0.1
0.3
0.5
0.7
0.9
1.0
75.06
75.71
76.11
75.91
76.12
76.20
76.50
2
77.74
76.96
78.30
77.70
77.51
77.81
77.60
3
78.93
79.29
79.53
79.64
79.89
79.57
79.10
4
79.30
79.69
80.20
79.96
79.89
79.61
79.47
A.4 Surograte Branches Design
In the design of the auxiliary branches, we need to balance computational complexity and feature extraction capability. On one hand, if the auxiliary branch is too simple, it may fail effectively extract features from the intermediate layers of the model, thereby affecting training performance. On the other hand, if the auxiliary branch is too complex, it will significantly increase computational complexity, which contradicts the low training cost characteristic of rate coding and may also lead to gradient fragmentation issues.
We first consider standard convolution, whose parameter storage complexity and computational complexity given by:
Pstd = D2 x Cin x Cout
(12)
Cald D2 x Cn X Cout X D3
(13)
rerresseeetss the kernel an the number of input and output channels, respectively, and D, is the size of the output feature map. Compared to traditional convolution, depthwise separable convolution decouples computation along the spatial and channel dimensions. Depthwise convolution
extracts spatial features, followed by pointwise convolution, which captures channel-wise features. Its parameter storage complexity and computational complexity are given by:
Pasc D4 Cin Cin Cout
(14)
Case = D C. DJ + Cin Cout x D3
(15)
The ratio of the number of parameters required for depthwise separable convolution to that of standard convolution is N + DT and the ratio of computational complexity is also N + DJ This demonstrates that depthwise separable convolution requires fewer parameters and has faster computation, aligning with the low training cost characteristic of rate coding.
1
Additionally, to prevent issues such as gradient explosion and enhance training performance, we introduce residual connections in the auxiliary branches. Finally, after passing through the classifier, we obtain the soft labels of the required submodule.
B
More Results
B.1 Selection of Parameter 3
In Table 6, We fixed the weight of the hard loss to 1.0 and focused on investigating the impact of varying the self-distillation weight on model training by evaluating the ResNet-18 model on the CIFAR-100 dataset. We observe that as 39 increases from 0, the overall model performance improves progressively, with accuracy steadily rising across all classifiers. This indicates that larger values of 3 enable each classifier to better learn from the teacher's 's knowledge, thereby facilitating the training of the entire network. The model achieves optimal performance when = 0.3. However, when exceeds 0.3, we observe that while some intermediate classifiers continue to improve in accuracy, the final classifier's performance begins to decline. This suggests that an excessively large distillation weight shifts the training focus more toward the intermediate layers, diverging from the optimal learning direction of the final classifier. As a result, this imbalance leads to a gradient diversion problem, hindering the overall performance of the model.

---

Table 7: Time and memory overhead at different timesteps.
Method
Eval Metries
T=2
T=4
T=6
T=8
T=10
T=12
Time cost(s/batch)
0.076
0.161
0.253
0.360
0.474
0.606
BPTT
Memory(MB) 2645.94
4889.32
7133.21
9383.85
11636.11
13887.80
Time cost(s/batch)
0.106
0.143
0.184
0.227
0.267
0.311
RateBP
Mcmory(MB) 2120.71
2120.78
2121.18
2296.48
2747.20
3197.42
Time cost(s/batch)
0.125
0.154
0.194
0.237
0.277
0.318
Our
(cm) 2380.09
2380.76
2381.91
2383.06
2758.69
3208.4
B.2 Standard self-distillation suboptimal solution count
We evaluate the classification performance of different depth classifiers the CIFAR-100 dataset using ResNet-18 18. Instances where the deepest classifier performs worse than shallower classifiers are considered as invalid teacher label examples (as indicated by the boxed region in Figure (5). We conducted three random observation experiments, and the average proportion of invalid teacher labels was 50.7%. As analyzed Section 3.3, these invalid teacher labels hinder the student model from converging in the correct direction. More notably, as the confidence in the target class of the mispredicted teacher labels increases, and the overall proportion of invalid labels rises during training, the negative impact intensifies. This phenomenon highlights the theoretical foundation for our proposed enhanced self-distillation method.
Test Accuracy over Epochs
80
classifier1
clessifier2
dassifier3
dassifier4
60
Accuracy (%)
40
30
20
10
Epoch
Figure 5: Classification performance of classifiers with different depths during the training process.
B.3
Comprehensive Evaluation of Training Costs
As supplement Figures and 4a in the main text, provide the memory and time overhead
different timesteps for BPTT, RateBP and our proposed method, as shown Table 7
B.4 Impact on inference efficiency
We recognize SEENN 43 as highly meaningful work. It determines whether stop the inference process early by computing the confidence level over the first time steps. This allows achieve performance close to that of the full T time steps within shorter inference duration, thereby reducing

---

inference cost. As shown in Figures ba and 6b we further demonstrate that models trained under our framework are compatible with this method. We tested confidence thresholds of 0.7, 0.8, 0.9, 0.99, and 0.999. On CIFAR-10 10 and CIFAR-100, our method achieves the performance of 6 time steps using only an average 2.4214 4 and 3.5371 time steps, respectively. Compared with RateBP, the accuracy improves by 0.48% and 1.18%, with only slight increase of 0.0048 and O. 1438 time steps, respectively.
95-
Aruugyy
94-
Out
70
Out
Ratedp
68
RanefiP
OurESSEENN
88
01
Inferance timestep
Inference limesto
Performance CIFAR-10
Performance CIFAR-100
Figure 6: Comparison of SEENN performance
B.5 Visualization Analysis
We provide visual analysis comparing standard self-distillation with the enhanced self-distillation approach. We utilize t-SNE visualization to observe classifiers at different depths and compare the clustering effects between Standard Self-Distillation and our proposed method. As shown in Figure] 7 and Figure g two key observations be made: First, it is evident that the deeper the classifier, the more compact the clustering becomes. Second, classifiers trained with Enhanced Self-Distillation exhibit stronger class separability compared to those of the same depth trained with Standard Self- Distillation. These results confirm that each classifier benefits from learning high-quality teacher signals, effectively pushing the upper bound of performance.
Figure 7: Clustering Patterns of Classifiers 1-4 in the Standard Self-Distillation
Figure 8: Clustering Patterns of Classifiers 1-4 in the Enhanced Distiilllllton
B.6 Analysis of Rate Statistics
Our method is trained based on rate coding and incorporates an ANN branch to enhance training effectiveness. While this hybrid strategy improves model performance, it may also influence the

---

spiking characteristics inherent to traditional SNNs. To investigate this, we tracked the average spike frequency across different layers over multiple time steps. As shown in the figure ? the results indicate that the average firing rate remains highly consistent over time, confirming the stability of the model during inference.
0.5-
0.25
Fing
0.20
0.10
0.35
0100
(a) Rate Statistics on CIFAR-10
(b) Rate Statistics on CIFAR-100
Figure 9: Comparison of SEENN performance
C Social Impacts and Limitations
This research primarily focuses on achieving high-performance training of Spiking Neural Networks (SNNs) under limited training conditions, and therefore does not lead to direct negative social impacts. Compared to Artificial Neural Networks (ANNs). SNNs inherently offer lower energy consumption during inference, which helps reduce carbon dioxide emissions. The proposed method uses a rate coding-based training approach, which contributes to reducing the training time and hardware requirements for SNNs.
The paper acknowledges several challenges regarding the proposed method. First, by introducing lightweight artificial neural network branch based on rate encoding, the training cost is slightly higher than that of RateBP. However, these memory costs are fixed constants and do not increase linearly with the time steps, as is the case with backpropagation through time (BPTT). More importantly, the performance breakthroughs achieved through experimental evaluations make this additional cost worthwhile. Furthermore, frequency-based backpropagation is designed to efficiently capture spatiotemporal feature representations optimize training, but its performance on sequence tasks is weaker than that of BPTT. This is discussed in Section 4.1. To ensure a fair comparison, we decoupled the frequency-based backpropagation training method on dataset with neuron morphology. Experimental results show that our improved self-distillation method demonstrates outstanding performance on both frequency- and time-based backpropagation methods, as well as on artificial neural networks.

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: See Abstract and Section 1.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: See Appendix
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]

---

Justification: See Section 3
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: See Section 3 and Appendix A
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
7

---

Answer: [Yes]
Justification:
See Supplementary Materials.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cm
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/ /pipps.ccppullc//uuuii wwwiicccguie//eeeeeeeeeii CoodeSubeiiie ssi wswonnolli onnolicy)))) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
See Appendix A.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: See Section 4.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, at least for the experiments that support main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.

---

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of enrors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: See Appendix and B.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g.. preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: /tte::://euii....eee httppp/sssp/pp...ttttttttttttree
Answer: [Yes]
Justification: This research conforms the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: See Appendix C.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to

---

generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative socictal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: No such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: See Appendix A.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, wwppppswipteeeee wwworeewwthhhe....c. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's is creators.
13. New assets
Question: Are new assets introduced in the paper well documented and the documentation provided alongside the assets?
10

---

Answer:
[NA]
Justification:
We do not use new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/codd/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
We use the existing common datasets.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
We the existing common datasets.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Answer: [NA]

---

Justification:
We do not use LLM.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy httees////earrppcccc///oo/rrrees for what should or should not be described.
12


Reviews:
[{"review_id": "2rpzlci7Oa", "reviewer": "Reviewer_zqGh", "summary": "The paper proposes an enhanced self-distillation framework for efficient Spiking Neural Network training, addressing the high memory and time costs of Backpropagation Through Time. The experiments on CIFAR-10, CIFAR-100, CIFAR10-DVS, and ImageNet demonstrate that the method reduces training complexity while achieving high-performance SNN training.", "questions": "In addition to the questions above, there are also the following points：\n\n1. Unclear notation. Some symbols lack proper definitions and explanations. Such as the meanings of $L_{KD}^{good}$ and $L_{KD}^{bad}$ in Eq(7) are missing.\n\n2. Formatting Issues: The highest performance values in the tables should be highlighted in bold for better readability. Table 3 should include percentage signs (%) for clarity.\n\n3. Can you give more details about how to decouple the teacher signal into reliable and unreliable components in your methods?", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strength:\n1. The paper proposed a novel method for efficient spiking neural network training.\n\n2. This method has good empirical results. It outperforms previous methods on static/neuromorphic datasets\n\nWeakness:\n1. Is this truly self-distillation? Since it involves SNN-to-ANN conversion before distillation.\n\n2. The method's effectiveness remains unverified for modern spiking neural architectures like Spiking Transformers.\n\n3.Insufficient literature review on self-distillation. The paper missed some key references and experiment comparisons with previous self-distillation methods, weakening the contextual grounding of the proposed approach.", "was_revised": true, "final_justification": "4 Borderline accept. The authors give clear explanations of whether their method belongs to self-distillation. And they add many comparisions experiments with existing works. They solved my previous questions.", "rebuttal": "## 1.Whether our framework belongs to self-distillation. (to weakness1)\n\nThank you for your question. It is reasonable to consider whether this architecture conforms to the definition of self-distillation from the perspective of the conversion between SNN and ANN. Admittedly, our network does incorporate a transition from SNN to ANN. However, in our model, the auxiliary ANN module is integrated as a substructure within the overall architecture, which distinguishes our approach from conventional distillation methods [1][2] that typically rely on an additional, separately pre-trained teacher model. In contrast, our approach generates teacher labels directly from an internal substructure of the model itself. On this basis, we contend that our architecture is consistent with the concept of self-distillation. Furthermore, prior works such as [3][4] have adopted similar strategies, using auxiliary branches to produce teacher labels, and these methods are widely recognized in the field as instances of self-distillation. Notably, in [4], the authors also implement a transition from a backbone SNN to an ANN branch during training, leveraging the outputs of the ANN component to guide the optimization of the backbone network. Their paper explicitly describes this approach as a form of self-distillation, which further supports our rationale.\n\nIn addition, one of the contributions of our paper is addressing the issue in self-distillation frameworks where synchronous training of teacher and student signals leads to the teacher signal interfering with student learning in the early stages of training. We propose a solution that explicitly distinguishes between different levels of label reliability. This approach not only addresses the specific challenges identified in our study, but also has the potential to be generalized and applied to other self-distillation frameworks. This method enables a secondary performance breakthrough without introducing additional computational overhead, which further supports our claim that the proposed framework should be categorized as a self-distillation method.\n\n## 2.The effectiveness of our method on modern spiking neural architectures such as Spiking Transformers. (to weakness2)\nThank you for your valuable suggestion. We use Spikingformer as the baseline to reproduce three methods: RateBP [5], MB, and ESD. Through ablation studies, we demonstrate that our method remains effective on modern spiking neural architectures such as Spiking Transformers and exhibits generalizable patterns. Specifically, on the CIFAR100 dataset, with the time step set to T=4, we evaluate several configurations including Spikingformer, Spikingformer + RateBP, MB, and ESD. Here, MB introduces the same auxiliary branch as ESD but removes the self-distillation module. As shown in the experimental results, our method (ESD) achieves accuracy improvements of 0.92% and 0.81% over the baseline on the Spikingformer-4-384 and Spikingformer-4-256 models respectively, while reducing memory usage by 20.91% and 28.85%. Moreover, this advantage becomes more prominent as the number of time steps increases.\n\nIn addition, the rate encoding error leads to a decrease in accuracy compared to the baseline, but the auxiliary branch helps mitigate this degradation. This observation aligns with the pattern described in Section 4.2 “Ablation study on self-distillation” of the main paper., RateBP < MB < ESD, which indicates that our method demonstrates consistent generalizability across modern spiking neural network architectures such as Spiking Transformers.\n|          | Spikingformer | RateBP  | MB      | ESD    |\n| :-------: | :------------: | :------: | :------: | :------: |\n| 4-384    |     79.09      |  77.29   |  79.54   |  80.01   |\n| 4-256    |     77.43      |  77.29   |  77.74   |  78.24   |\n\n## 3.Citations of other self-distillation related work (to weakness3)\nThank you for your valuable suggestion. We conducted a comparative study against several state-of-the-art self-distillation methods in the SNN domain in recent years. The evaluation was performed on the CIFAR100 dataset. Models marked with * use ResNet19, while the others are based on ResNet18. According to the experimental results, our framework achieves state-of-the-art performance among self-distillation methods in the SNN field. In addition, unlike other works, our rate-based training approach significantly reduces training costs, making it feasible to train models with higher temporal resolutions under limited computational resources.\n|SAKD*[7]|SM[8]|SM*|TSSD(T=2)[9]|TKS*[10]|Our|Our*|\n| -------- | -------- | -------- |-------- |-------- | -------- |-------- |\n|80.10|79.49| 81.70| 73.40 |79.89|80.20|82.14|\n\n## 4.Regarding symbols and formatting (to question 1, 2)\nWe appreciate your feedback on the notation and tables in our paper. You are correct that some symbols, such as those in Equation (7), were not clearly defined. We will revise the manuscript to clarify these definitions and improve the overall readability.\n\nAs discussed in Section 3.3, in traditional self-distillation frameworks, the teacher and student labels are fixed. We evaluate their quality through the cross-entropy loss with the ground-truth label: $L_{\\text{CE}}(q^{(t)}, y)$ and $L_{\\text{CE}}(p^{(t)}, y)$. However, due to possible differences in convergence speed, the quality of the teacher label may become lower than that of the student during training: $L_{\\text{CE}}(q^{(t)}, y) > L_{\\text{CE}}(p^{(t)}, y)$. In this case, computing $\\text{KL}(q^{(t)} \\| p^{(t)})$ may cause the student to converge in the wrong direction. We define the KL divergence in this case as $L_{\\text{KD}}^{\\text{bad}}$. Conversely, if the teacher label is of higher quality than the student, we define it as $L_{\\text{KD}}^{\\text{good}}$.\n\nWe will revise the equations and tables you pointed out in the next version. Thank you again for your detailed review — your suggestions help make this work more rigorous and complete.\n\n\n## 5.More details about the decoupling component. (to question3)\nAs mentioned in Section 3.4 of the paper, we assume that samples correctly predicted by the network are more likely to exhibit reliable distributional characteristics compared to those that are incorrectly predicted. Specifically, suppose the network has n classification heads: A₁, A₂, A₃...Aᵢ...Aₙ. For each classification head i, we divide the predicted batch into two categories: (1) samples for which the target class k is correctly predicted (class k has the highest predicted probability among all C classes), and (2) samples where the prediction for class k is incorrect.\n\nWe then set the soft labels of the second category (incorrect predictions) entirely to zero, while keeping the soft labels of the correctly predicted samples unchanged, thus generating a new soft label set. To mitigate the issue where even correctly predicted samples may still exhibit unreliable distributions over non-target classes, we apply an averaging ensemble over the outputs of all n classification heads. This is based on ensemble learning theory [6], which suggests that increasing the number of voting models typically enhances the stability of the final prediction. In this process, only correctly predicted samples contribute to the voting; incorrectly predicted samples, having been zeroed out, contribute nothing to the final teacher label.The resulting teacher label, Teacher, is the ensemble of the n reliable predictions. In cases where all n classification heads mispredict a particular sample, the corresponding Teacher label for that sample will be all zeros. We interpret such instances as the model having likely fallen into a learning failure or ambiguity for those samples. Therefore, during the computation of the distillation loss, we use a masking mechanism to exclude these samples from distillation. However, we acknowledge that even unreliable labels can play a regularization role in the distillation process. To preserve this benefit, we apply a small regularization term to these excluded samples, preventing the model from becoming overconfident in learning from such ambiguous cases.\n\nThe implementation details can be found in the supplementary materials, specifically in the layer.py script within the model module (lines 171–191), where we provide the full process of decoupling reliable and unreliable components and computing the final loss.\n\n\n\n## Reference\n[1] Efficient Logit-based Knowledge Distillation of Deep Spiking Neural Networks for Full-Range Timestep Deployment\n\n[2] BKDSNN: Enhancing the Performance of Learning-based Spiking Neural Networks Training with Blurred Knowledge Distillation\n\n[3] Be Your Own Teacher: Improve the Performance of Convolutional Neural Networks via Self Distillation \n\n[4] Surrogate Module Learning: Reduce the Gradient Error Accumulation in Training Spiking Neural Networks \n\n[5] Advancing Training Efficiency of Deep Spiking Neural Networks through Rate-based Backpropagation\n\n[6] Online Knowledge Distillation via Collaborative Learning \n\n[7] SAKD: Sparse attention knowledge distillation\n\n[8] Surrogate Module Learning: Reduce the Gradient Error Accumulation in Training Spiking Neural Networks\n\n[9] Self-Distillation Learning Based on Temporal-Spatial Consistency for Spiking Neural Networks \n\n[10] Temporal Knowledge Sharing Enable Spiking Neural Network Learning From Past and Future"}, {"review_id": "VaQSpfnQXn", "reviewer": "Reviewer_e4GK", "summary": "This paper proposes a novel training framework for Spiking Neural Networks (SNNs) that enhances performance and efficiency by integrating self-distillation with rate-based backpropagation. The key idea is to project firing rates from intermediate SNN layers into lightweight ANN branches, enabling substructure optimization via self-generated knowledge. To address the issue of unreliable knowledge in standard self-distillation, the authors introduce a decoupling mechanism that filters out low-quality guidance, ensuring only reliable signals influence training.", "questions": "1. Could the authors clarify whether the proposed method is able to preserve any spike timing information during the knowledge transfer process, or is all temporal structure reduced to rate-based statistics?\n2. Could the proposed distillation framework remain effective if the auxiliary ANN branches were replaced with SNN-based substructures, thereby maintaining a fully spike-based architecture?", "limitations": "Yes.", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n1. Improved Training Efficiency: The proposed approach shows noticeable reductions in memory usage and training time on standard benchmarks, which may be beneficial for resource-constrained SNN training scenarios.\n2. Refined Self-Distillation Strategy: The idea of decoupling reliable and unreliable self-generated knowledge offers a thoughtful refinement to conventional self-distillation methods, potentially improving training stability.\n3. Good Empirical Coverage: The method is evaluated on multiple datasets, including both static and neuromorphic benchmarks, and shows some promise for generalization, including preliminary applicability to ANN models.\n\nWeaknesses:\n1. Limited Exploitation of Temporal Dynamics: The proposed method does not fully leverage the intrinsic temporal modeling capabilities of SNNs. Instead, it primarily relies on ANN-based distillation mechanisms and rate-based approximations to enhance performance, which may overlook the rich temporal information encoded in precise spike timing that is critical for neuromorphic tasks.\n2. Inconsistency Between SNN and ANN Pathways: Projecting SNN firing rates into ANN branches introduces a modality mismatch, as the ANN surrogate may not faithfully preserve the temporal characteristics of spiking activity.", "was_revised": false, "final_justification": "", "rebuttal": "## 1.Applicability of the distillation framework to other model structures. (to question 2)\nWe appreciate your suggestion, we replaced the auxiliary ANN module with an SNN module and evaluated the effectiveness of our self-distillation framework under both rate-based and temporally-based training schemes. The corresponding experimental results are denoted as SNN_RateBP and SNN_BPTT, respectively.\n| T=6,CIFAR100|baseline|SNN_RateBP|SNN_BPTT|Our|\n| -------- | -------- | -------- |-------- |-------- |\n|Acc(%)| 79.06| 79.46 |80.14|80.20|\n|Memory(MB)|7132.2|2906.1|7676.3|2381.91|\n|Time(s)|0.253|0.273|0.331|0.208|\n\nFrom the experimental results on CIFAR100, using rate-based training (SNN_RateBP) achieves a 0.4% accuracy improvement over the baseline. However, due to the replacement of the ANN module, the approximation error introduced by rate coding is no longer properly compensated, leading to a 0.74% drop in accuracy compared to our original method. Furthermore, the auxiliary branch must now follow the same information integration mechanism as the backbone SNN, making it less lightweight than the ANN variant. As a result, both efficiency and computational cost are inferior to our original design.\n\nThe method also proves effective under BPTT-based training (SNN_BPTT), where no approximation error from rate coding exists. This version achieves a 1.08% improvement over the baseline, showing performance comparable to our proposed method. Additionally, as reported in Table 2 of the main paper, it also outperforms previous approaches on neuromorphic datasets. However, this approach loses the low-cost advantage of rate coding, resulting in a linear increase in memory consumption with respect to the temporal dimension, which limits its scalability under constrained computational resources.\n\nIn summary, our framework achieves efficient training by utilizing temporally averaged information, focusing on spatially effective features. While we acknowledge that this may compromise the ability to model fine-grained temporal dynamics, the use of rate-based training decouples the linear dependency between the temporal dimension and memory consumption during backpropagation. This offers a reliable approach to training high-accuracy models efficiently under resource-constrained scenarios. Moreover, our proposed self-distillation framework is generalizable to other architectures, including ANNs and temporally trained SNNs.\n\n\n\n## 2.Whether any spike timing information is retained during the knowledge transfer process.（to question1）\nThank you for your question. As described in Section 3.2 of the paper, in our framework, the first forward pass is conducted based on temporal dynamics, during which the full temporal statistics are preserved and the eligibility traces are updated. In the second forward pass, both the backbone network and the auxiliary branch operate entirely based on rate information, effectively simplifying the process to rate-based statistical computation.\n\nHowever, as discussed in Question 2, the first and second contributions of the paper are independent. The self-distillation framework remains applicable to training methods based on BPTT.\n## 3.Limited utilization of temporal dynamics (to weakness1)\nThank you for your insight. Our rate-based framework achieves efficient training by integrating the mean of temporal information, with a focus on spatially effective features. As discussed in the supplementary material Section C Social Impacts and Limitations, we acknowledge that this approach compromises the model's ability to capture fine-grained temporal dynamics, which is an important area for improvement in future work.\n\nHowever, the rate-based training paradigm decouples the linear relationship between the temporal dimension and memory consumption during backpropagation. This provides a highly cost-effective solution for training high-accuracy models under resource-constrained scenarios.\n\nMoreover, as mentioned in our response to Question 2 and further validated by additional experiments, our self-distillation framework is also effective in temporally trained SNNs and ANNs, further demonstrating the general applicability of our approach.\n## 4.Inconsistencies between SNN and ANN pathways (to weakness2)\nThank you for your question. In the rate coding framework, we aggregate the mean of temporal information into rate-based representations to enable efficient training. During the first forward pass, which is based on temporal dynamics, only the statistics and eligibility traces required by the backbone network are collected. The ANN module is not involved at this stage, so there is no need to address the mapping between SNN and ANN.\n\nIn the second forward pass, since the backbone network operates on rate-based representations—and rate coding is highly compatible with ANN modules, both emphasizing spatial information—there is no issue of feature mismatch between the two.\n\nIt is important to clarify that the rate-based forward pass focuses solely on spatial-domain information, which means that the auxiliary ANN branch does not need to preserve any spike-based temporal features."}, {"review_id": "xlk9W4QuyL", "reviewer": "Reviewer_4txP", "summary": "The paper proposes a self-distillation framework based on the RateBP training framework, introducing a lightweight ANN auxiliary branch in the intermediate layers of the SNN to provide more accurate gradients for the intermediate layers, thereby alleviating the gradient distortion problem. In addition, the authors propose the ESD method, which determines the reliability of all branches’ output signals, and combines them into a final high-quality teacher signal for guiding the learning of the auxiliary branch. This paper achieves significant performance improvements and reduced training costs on all conducted datasets.", "questions": "1.The method in the paper is based on the soft reset LIF model. Can it be applied to the hard reset rules of LIF?\n\n2.In Figure 3 (b), according to the paper, self-distillation alleviates the gradient error problem. Why are methods like MB, ASD, and ESD still effective on ANNs where there is no gradient error?", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strength\n\n1.The paper achieves a simultaneous increase in training efficiency and network performance, representing an excellent attempt to solve the training problem of SNN. \n\n2.This paper identifies potential problem with standard self-distillation in SNN and proposes a simple and effective solution,\n\nWeakness\n\n1.The paper lacks a detailed discussion of the auxiliary branches, such as the structure of the auxiliary branches, the insertion location in the main branch, and the relationship between various choices of auxiliary branches and the performance improvements.\n\n2.Figure 2 is missing the caption. The current caption is a placeholder paragraph.\n\n3.This paper need to provide an in-depth theoretical analysis (or experiment) of how gradient errors are corrected by auxiliary branches.\n\n4.The definition of reliability may be overly simple. Sometimes, misclassification output can contain more useful information than correct classification output. The paper does not further explore more refined reliability measurement methods, which may still have significant room for improvement.\n\n5.The contribution 1 of the paper seems to derive from RateBP method, and a comparative analysis with it should be added.\n\n6.In Table 7, the BPTT memory cost at T=12 is excessively large. This may be caused by a transcription error.", "was_revised": true, "final_justification": "The authors provide a detailed rebuttal that completely addressed all my concerns. In the rebuttal, the authors found effective theoretical derivations for their method, enhancing the completeness of the paper. Meanwhile, I am convinced by their dual theoretical explanations for the effectiveness of the method on ANN and the brand-new ablation experiments, which successfully transformed a core concern into a highlight of the paper.", "rebuttal": "## 1.Whether it is applicable to the hard reset of LIF. (to question1)\nThank you for your question. The proposed method is applicable to the hard-reset LIF. In the code section of the supplementary materials, within the model.layer module, we define the function calcu_elig_factor_hard_reset to update the eligibility trace using the hard reset method. We have evaluated the performance of the baseline, RateBP, MB, and ESD methods separately. The detailed experimental results are as follows:\n|CIFAR100,T=6,ResNet18|baseline|RateBP|MB|ESD|\n|-|-|-|-|-|\n|Acc|78.41|78.34|79.38|79.76|\n\n## 2.Why is our method still effective on artificial thermal networks. (to q2)\nThis issue stems from the fact that our method can be understood from two theoretical perspectives.\n\nFirst, from the perspective of surrogate learning, a lossless structure (ANN in our case) provides more accurate gradients to the intermediate layers of the backbone, mitigating gradient errors. This aligns with the motivation of our method. Since ANN introduces no temporal approximation errors, the i-th auxiliary branch can be viewed as a surrogate module that supplies more accurate gradients to the preceding sub-network.\n\nSecond, from the perspective of deep supervision: inserting additional supervision signals at intermediate layers and jointly optimizing them with the main loss improves gradient flow and model generalization. This also explains why our method is effective even when both the backbone and auxiliary branches are implemented with lossless ANN structures.\n\nOur approach incorporates both the surrogate learning perspective and the deep supervision perspective, which can be validated through a simple experiment. If we replace the ANN in the auxiliary module with an SNN that matches the backbone, keeping all other settings the same, and ignore gradient approximation errors, we refer to this result as SNN_Rate and observe that accuracy still improves by 0.4% and 0.44% over the baseline and RateBP, respectively. However, it is still lower than our method. This demonstrates that deep supervision is also effective, but our method enables more comprehensive optimization under rate-coded training.\n|baseline|RateBP|SNN_Rate|ESD|\n|-|-|-|-|\n|79.06|79.02|79.46|80.20|\n\n## 3.Detailed analysis of the auxiliary branch. (to weakness1)\nExperimental setup: T=6, ResNet18, CIFAR100.\n### Structural analysis\nAs stated in Section 3.2 Model Training of the main paper, the auxiliary branch in our framework is constructed using depthwise separable convolutions. The specific architecture is as follows:DepthwiseConv (stride=2, Cin → Cin) → PointwiseConv (Cin → Cin) → BN → ReLU → DepthwiseConv (stride=1, Cin → Cin) → PointwiseConv (Cin → Cout) → BN → ReLU.\n### Why choose depthwise separable convolution\nWe compared depthwise separable convolution (DWConv) and standard convolution (StdConv). While StdConv is theoretically better at feature extraction, DWConv achieved 0.53% higher accuracy in our experiments. This supports our design, as the auxiliary module aims to optimize substructures. As discussed in Supplementary A.4, StdConv may overly focus on the auxiliary branch, shifting optimization away from the final classifier. This is why we use DWConv in auxiliary branches.\n||FC1|FC2|FC3|FC4|\n|-|-|-|-|-|\n|DWConv|76.11|78.3|79.53 |80.2|\n|StdConv|76.42| 77.91| 79.62 |79.67|\n\nIn addition, Supplementary Section A.4 analyzes the theoretical advantages of DWConv over StdConv in terms of compute and memory efficiency, supporting our goal of training high-accuracy models with limited resources.\n### Regarding the position of the auxiliary branch\nIn the paper, to facilitate evaluating the effectiveness of our framework, we uniformly set three auxiliary branches inserted after the ResNet convolutional blocks [2, 4, 6]. \n\nIn practice, the insertion positions can be treated as hyperparameters, allowing for increased design flexibility and better scalability when applied to more complex models. We also experimented with alternative insertion schemes, such as placing branches after block [4] alone, or after blocks [3, 6]. Fewer insertion points benefit training efficiency but may limit model accuracy. More flexible auxiliary branch insertion options will be provided in our forthcoming open-source code.\n||[4]|[3,6]|[2,4,6]|\n|-|-|-|-|\n|Acc(%)|79.65|79.92|80.2|\n|Time(s)|0.195|0.199|0.208|\n|Memory(MB)|2401.7|2430.10|2601.0|\n\n## 4.Figure 2 lacks a title.（to w2）\nThank you for your careful review. We will include this clarification in the next version. \n## 5.How the auxiliary branch mitigates gradient errors. (to w3)\nWe take one auxiliary branch as an example and denote its output as $y_{i}$, with the final target output as $y_{c}$. From experiments, we observe that KL$(y_{i}|y_{c})$ eventually converges, and $y_{c}$ approaches $y_{i}$. We set $y_{1}=y_{c}+\\xi$, where $\\xi$ is a small error term. Define $\\frac{\\partial y_{1}}{\\partial \\theta } = S$, $\\frac{\\partial y_{c}}{\\partial \\theta } = G$, where $G$ is the expected gradient of the backbone network.\n\nDue to rate approximation and surrogate gradient errors, we define $R\\left(\\frac{\\partial y_{c}}{\\partial \\theta }\\right) = G+\\delta$, so the error ratio is $K_{rate} = \\frac{\\delta}{G}$. We define the total loss as $L_{total} = \\alpha \\cdot L_{1}+L_{c}$, where $\\frac{\\partial L_{1}}{\\partial y_{1}}= \\frac{\\partial L_{c}}{\\partial y_{c}}+{\\xi }'$. It can be derived that the ideal total gradient of our method is\n\n$$\n\\frac{\\partial L_{total}}{\\partial \\theta}= (1+\\alpha )\\cdot \\frac{\\partial L_{c}}{\\partial y_{c}}\\cdot G+\\alpha \\cdot \\frac{\\partial L_{c}}{\\partial y_{c}}\\cdot (S-G)+\\alpha \\cdot {\\xi }'\\cdot S,\n$$\n\nwhere ${\\xi }'$ is a higher-order infinitesimal of $G$ and can be neglected, thus\n\n$$\n\\frac{\\partial L_{total}}{\\partial \\theta}= \\frac{\\partial L_{c}}{\\partial y_{c}}\\cdot (G+\\alpha \\cdot S).\n$$\n\nTherefore, under our framework, the error ratio\n\n$$\nK_{esd}=\\frac{\\delta }{G+\\alpha \\cdot S  } \\le \\frac{\\delta}{G} =K_{rate}.\n$$\n\nTherefore, under our framework, the error rate $K_{esd}$ is always less than $K_{rate}$.\nThrough experiments, we found that setting $\\alpha$=1.0 yields the best performance. An excessively large loss weight for the auxiliary branches may hinder the convergence of the backbone network.\n|α| 0.0 | 0.5 |0.7|0.9|1.0|1.2|1.5\n|-|-|-|-|-|-|-|-|\n|Acc(%)|78.96|79.17|79.21|79.24|79.30|79.28|78.89|\n\n## 6.Analysis of the Reliability of Error Labels. (to w4)\n### The quality of incorrect labels\nWe acknowledge that incorrect predictions may carry useful information. However, in self-distillation, especially early in training, correct predictions are generally closer to high-quality label distributions. We show this by comparing their KL divergence to a reliable distribution.\n\nWe introduced a ResNet34 model with an accuracy of up to 82%, which we consider to have high-quality label A. We define the correctly predicted portion of the output as B1 and the incorrectly predicted portion as B2. Across many training epochs, it is observed that KL(A, B1) is consistently smaller than KL(A, B2), especially in the early training phase. This strongly indicates that the correctly predicted part is more likely to be close to a high-quality label distribution than the incorrectly predicted part.\n|epoch|10|20|50|100|150|200|300|\n|-|-|-|-|-|-|-|-|\n|KL(A,B1)|0.2119|0.1821|0.1745|0.1678|0.1405|0.1396|0.1306|\n|KL(A,B2)|0.3117|0.297|0.2628|0.2676|0.296|0.237|0.1756|\n### A scheme for differentiating reliability based on confidence\nAdditionally, based on your suggestion, we further refined the separation of incorrect teacher labels using confidence scores. Specifically, we introduced a threshold k, if an incorrect prediction has a confidence score for the wrong class below k, it is reclassified as a reliable label. This refinement is an extension of the analysis in Section 3.3, which indicates that incorrect labels with high confidence in the wrong class are more misleading to the student. We treated k as a hyperparameter, and experimental results show that when k<0.03, the accuracy remains above 80%. However, a noticeable drop occurs when k>0.03.\n|k|0|0.02|0.03|0.04|0.05|1.0|Average Confidence|\n|-|-|-|-|-|-|-|-|\n|Acc|80.2|80.04|80.12|79.87|79.89|79.71|80.21|\n\nWe take into account that label confidence dynamically changes during training. As shown in the experiments, the average confidence increases progressively over the course of training.\n|epoch|10|50|100|150|200|250|\n|-|-|-|-|-|-|-|\n|Average Confidence|0.0355|0.0466|0.0494|0.0525|0.0556|0.0598|80.21|\n\nA fixed k cannot adapt to the changing confidence levels at different training stages. Therefore, we introduced a dynamic thresholding mechanism based on average confidence setting k=Average Confidence. This dynamic approach shows greater potential compared to a fixed threshold, but in practice, the improvement over our original results is limited.This demonstrates that our method, while simple in design, is highly effective in practice. In summary, while we do not deny that better reliability separation strategies may exist, the key contribution of our work is introducing the concept of reliability in the context of self-distillation. Defining more accurate reliability criteria remains an important direction for future research.\n## 7.Comparison with RateBP. (to w5)\nWe show performance results in Tables 1–2 and Figures 3–4(b), and training cost in Figure 1, Figure 4(a), and Table 7. Our method outperforms RateBP while keeping most of its cost benefits.\n## 8.Abnormal memory record at T=12. (to w6)\nThank you for your careful review. We found that the abnormal memory usage under the BPTT setting with T=12 was caused by redundant intermediate variables. We have re-evaluated the peak memory consumption during training using torch.cuda.max_memory_allocated, and the updated average values are shown in the table below.\n|T|2|4|6|8|10|12|\n|-|-|-|-|-|-|-|\n|Memory(MB)|2645.9|4889.3|7133.2|9383.8|11636.1|13887.8|"}, {"review_id": "oQjm8sVOvq", "reviewer": "Reviewer_mbYz", "summary": "This paper presents an enhanced self-distillation framework for efficient training of Spiking Neural Networks (SNNs). The key ideas include projecting the firing rates of intermediate SNN layers onto lightweight ANN branches and using a reliability-separated self-distillation strategy to ensure that only high-quality knowledge guides the optimization process.", "questions": "1.The paper proposes an enhanced self-distillation framework but does not analyze its convergence in detail. It is recommended that the authors theoretically analyze the convergence rate and convergence conditions of the proposed method, especially the effect on convergence when dealing with unreliable teacher signals.\n2.The influence of different β values on the model performance is shown in Table 6 in the text. It indicates that this method is relatively sensitive to the selection of β values and there exists an optimal range of β values. However, the specific method on how to select the optimal β value is not provided in the text.\n3.The author should consider whether a dynamic reliability assessment mechanism can be introduced, such as how to adjust the reliability assessment criteria of teacher signals as the training progresses.\n4.In Section 3.2, the author mentions the use of eligibility traces to approximate gradients in backpropagation. However, the accuracy and potential errors of this approximate method are not analyzed in detail in the text.", "limitations": "Please reference Weaknesses and Questions.", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 4, "originality": 3, "strengths_and_weaknesses": "Strength:\n1.The proposed enhanced self-distillation framework and the idea of using rate-based backpropagation combined with a reliability-separated self-distillation strategy are innovative and addresses a critical issue in traditional self-distillation methods where unreliable teacher labels can hinder convergence.\n2.The authors provide extensive experimental results on multiple datasets, demonstrating the effectiveness of their method. The improvements in accuracy, memory consumption, and training time compared to BPTT and other methods are substantial and well-documented.\n3.This paper presents a detailed theoretical analysis of the effects of reliable and unreliable predictions of teacher signals on student models.\nWeaknesses:\n1.The reliability separation strategy proposed in the article may ignore some samples that have potential value in the early stage of training but are currently misclassified in practical operation. For example, in Table 6, when the β value increases from 0.3 to 0.5, although the performance of the classifier in the middle layer improves, the performance of the final classifier begins to decline. This indicates that simply filtering out misclassified samples may cause the model to miss some important learning signals.\n2.It is mentioned in the text that the auxiliary branch uses depth-separable convolution to reduce the computational complexity. However, although this design has advantages in computational efficiency, it may be insufficient in feature extraction ability.\n3.The text mentions the formation of teacher labels by filtering out samples with correct predictions (see Section 3.4), but does not elaborate on how to define the criteria for \"correct predictions.\" For example, is it based on the classification accuracy rate or the confidence threshold?", "was_revised": true, "final_justification": "Thank you very much for your reply. I still maintain the original rating.", "rebuttal": "## 1.Potential Value of Error Classification Samples (to weakness1)\nThank you for your suggestions. Please refer to the response to Reviewer 4txP entitled \"6.Analysis of the Reliability of Error Labels (to w4)---The quality of incorrect labels  \".\n## 2.Introduction of a Dynamic Reliability Assessment Mechanism (to weakness1, question3)\nThank you for your suggestions. Please refer to the response to Reviewer 4txP entitled \"6.Analysis of the Reliability of Error Labels (to w4)---A scheme for differentiating reliability based on confidence \".\n## 3.Why choose depthwise separable convolution(to w2)\nWe compared depthwise separable convolution (DWConv) and standard convolution (StdConv). While StdConv is theoretically better at feature extraction, DWConv achieved 0.53% higher accuracy in our experiments. This supports our design, as the auxiliary module aims to optimize substructures. As discussed in Supplementary A.4, StdConv may overly focus on the auxiliary branch, shifting optimization away from the final classifier. This is why we use DWConv in auxiliary branches.\n||FC1|FC2|FC3|FC4|\n|-|-|-|-|-|\n|DWConv|76.11|78.3|79.53 |80.2|\n|StdConv|76.42| 77.91| 79.62 |79.67|\n\nIn addition, Supplementary Section A.4 analyzes the theoretical advantages of DWConv over StdConv in terms of compute and memory efficiency, supporting our goal of training high-accuracy models with limited resources.\n## 4.Criteria for \"Correct Prediction\" (to w3)\nThe criterion for \"correct prediction\" is classification accuracy, if the class with the highest predicted probability among the n classes matches the target label, the prediction is considered correct and thus reliable.\n## 5.Impact of Unreliable Teacher Signals on Convergence (to question1)\nThank you for your suggestion. We first define: the standard convex objective $L(\\theta) = E_{(x,y)}[\\ell(f_\\theta(x), y)],$ where $\\ell$ is the cross-entropy loss and $y$ is the ground-truth label. Its correct gradient direction is $g_{\\text{true}}(\\theta) = \\nabla_\\theta\\\\ell(f_{\\theta_t}(x), y),$ and the parameter update is: $\\theta_{t+1} = \\theta_t - \\eta\\ g_{\\text{true}}(\\theta), \\quad \\theta_t \\to \\theta^* \\quad \\text{as } t\\to\\infty.$ In distillation, by introducing a soft label to assist training, the student model learns richer inter-class distribution information; however, when this soft label is unreliable, it often backfires. We illustrate this with an extreme example: When the teacher label is incorrect and the confidence in the wrong class is too high ($p_{\\text{err}}\\to1$), the erroneous gradient can be approximated as: $g_{\\text{err}}(\\theta) = \\nabla_\\theta \\ell(f_\\theta(x), \\hat{y}\\_{\\text{wrong}}).$ When this error becomes too large, it will cause the model to deviate from its original optimization direction; this is precisely why we need to separate the reliable parts of the labels. Thank you again for your suggestion. We will include a more detailed derivation in future work.\n## 6.Optimal Range of β Values（to q2）\nThank you for your question. To determine the behavior of β, we conducted fine-grained validation experiments on CIFAR-10 and CIFAR-100. Both datasets showed similar patterns. As noted in Supplementary Section B.1, increasing β from 0 enhances the effect of self-distillation, but beyond a certain limit, the accuracy of the main classifier declines.This is likely because an excessively large loss weight shifts the training focus toward intermediate classifiers, neglecting the optimization of the main classifier. The optimal β values for CIFAR-10 and CIFAR-100 are around 0.5 and 0.3, respectively. Therefore, we believe setting β within the range [0.3, 0.5] can effectively improve training stability.\n|β|0|0.1|0.3|0.5|0.7|0.9|1.0|\n|-|-|-|-|-|-|-|-|\n|CIFAR10|95.88|96.06|96.09|96.21|96.04|95.97|95.95|\n|CIFAR100|79.30|79.69|80.20|79.96|79.89|79.61|79.47|\n## 7.Error analysis of rate approximation (to q4)\nThank you for your question. Here, we provide a concise overview of how rate coding achieves gradient approximation. A detailed exposition will be included in the supplementary materials of the forthcoming updated version. First, we define discrete time steps as $t\\in\\{1,\\dots,T\\}$ and layer index as $l$. Let membrane potential be $u^l_t$, spike output $s^l_t$, threshold $V_{th}$, decay $\\lambda$, the weight matrix of layer $l$ as $W^l$, and the Heaviside step function as $H(\\cdot)$. The network output at time $t$ is $o_t = W^L s^L_t$, the final prediction $y_{\\mathrm{pred}} = \\frac{1}{T}\\sum_{t=1}^T o_t$, and the loss function is defined as $\\mathcal{L} = \\ell\\bigl(y_{\\mathrm{pred}},\\,y\\bigr).$ The spike generation process:  $u^l_t = \\lambda\\bigl(u^l_{t-1} - V_{th}\\,s^l_{t-1}\\bigr)+ W^l\\,s^{\\,l-1}\\_{t-1},s^l_t = H\\bigl(u^l_t - V_{th}\\bigr).$\n**BPTT gradient computation for membrane potential:**  \n$$\n\\frac{\\partial \\mathcal{L}}{\\partial \\mathbf{u}^l_t}  = \\frac{\\partial \\mathcal{L}}{\\partial \\mathbf{s}^l_t} \\frac{\\partial \\mathbf{s}^l_t}{\\partial \\mathbf{u}^l_t} + \\sum_{\\tau > t} \\frac{\\partial \\mathcal{L}}{\\partial \\mathbf{s}^l_{\\tau}} \\frac{\\partial \\mathbf{s}^l_{\\tau}}{\\partial \\mathbf{u}^l_{\\tau}} \\prod_{i=\\tau-1}^t \\left( \\frac{\\partial \\mathbf{u}^l_{i+1}}{\\partial \\mathbf{u}^l_i} + \\frac{\\partial \\mathbf{u}^l_{i+1}}{\\partial \\mathbf{s}^l_i} \\frac{\\partial \\mathbf{s}^l_i}{\\partial \\mathbf{u}^l_i} \\right)\n$$\nWeight gradient accumulation: $\\nabla_{W^l}\\mathcal{L}\n= \\sum_{t=1}^T \\frac{\\partial \\mathcal{L}}{\\partial u^l_t}s^{l-1}\\_t$, and gradient propagation to the previous layer:$\\frac{\\partial \\mathcal{L}}{\\partial s^{\\_l-1}\\_t}=\\frac{\\partial \\mathcal{L}}{\\partial u^l_t}\\_W^{l\\top}.$ Under the rate-coding assumption, information in an SNN can be represented by the average firing rate. The average firing rate per layer (denoted $r^l$) is computed as the expectation of spikes over time:  $r^l = \\mathbb{E}[s^l_t] \\approx \\frac{1}{T} \\sum_{t=1}^T s^l_t.$ In the linear propagation stage, each time-step input $I^l_t = W^l s^{l-1}\\_{t-1}$ is approximated by its average $c^l = \\mathbb{E}[I^l_t] = W^l r^{l-1}$. The gradient for the linear part is straightforward:$\\frac{\\partial r^{l-1}}{\\partial c^{l}} = {W^{l}}^\\top.$ Next, we need to define the relationship between the average input $c^l$ and the firing rate $r^l$, $\\frac{\\partial r^{l}}{\\partial c^{l}}$. Starting from the exact gradient, the influence of different time-step inputs on the output is expressed as:  \n$$\n\\frac{\\partial \\mathbf{s}\\_\\tau^l}{\\partial \\mathbf{I}\\_t^l} = \\frac{\\partial \\mathbf{s}\\_\\tau^l}{\\partial \\mathbf{u}\\_t^l} = \n\\begin{cases} \n\\frac{\\partial \\mathbf{s}\\_\\tau^l}{\\partial \\mathbf{u}\\_\\tau^l} \\prod_{i=\\tau-1}^t \\left( \\frac{\\partial \\mathbf{u}\\_{i+1}^l}{\\partial \\mathbf{u}\\_i^l} + \\frac{\\partial \\mathbf{u}\\_{i+1}^l}{\\partial \\mathbf{s}\\_i^l} \\frac{\\partial \\mathbf{s}\\_i^l}{\\partial \\mathbf{u}\\_i^l} \\right) & \\text{if } \\tau \\ge t \\\\ \n\\frac{\\partial \\mathbf{s}\\_t^l}{\\partial \\mathbf{u}\\_t^l} & \\text{if } \\tau = t \\\\ \n0 & \\text{if } \\tau < t \n\\end{cases}\n$$  \nBy accumulating this dynamic process over time, we obtain the influence of a single time-step input on the overall spike output:  \n$$\n\\mathbf{\\kappa}^l_t = \\sum_\\tau \\frac{\\partial \\mathbf{s}\\_\\tau^l}{\\partial \\mathbf{I}\\_t^l} = \\left( \\frac{\\partial \\mathbf{s}\\_t^l}{\\partial \\mathbf{u}\\_t^l} + \\sum_{\\tau > t} \\frac{\\partial \\mathbf{s}\\_\\tau^l}{\\partial \\mathbf{u}\\_\\tau^l} \\prod_{i=\\tau-1}^t \\left( \\frac{\\partial \\mathbf{u}\\_{i+1}^l}{\\partial \\mathbf{u}\\_i^l} + \\frac{\\partial \\mathbf{u}\\_{i+1}^l}{\\partial \\mathbf{s}\\_i^l} \\frac{\\partial \\mathbf{s}\\_i^l}{\\partial \\mathbf{u}\\_i^l} \\right) \\right)\n$$  \nUnder rate coding, using the approximation $\\frac{1}{T} \\sum_{t=1}^{T} I^l_t \\approx c^l$, we derive the surrogate gradient of neural dynamics via mean estimation:  \n$$\n\\left( \\frac{\\partial \\mathbf{r}^l}{\\partial \\mathbf{c}^l}\\right)\\_{\\text{rate}} \\equiv \\sum_\\tau \\left( \\frac{\\partial(\\mathbb{E}[\\mathbf{s}^l_t])}{\\partial \\mathbf{I}^l_\\tau} \\frac{\\partial \\mathbf{I}^l_\\tau}{\\partial \\mathbf{c}^l} \\right) =\\frac{1}{T} \\sum_t \\sum_\\tau \\left( \\frac{\\partial \\mathbf{s}^l_t}{\\partial \\mathbf{I}^l_\\tau} \\right) =\\mathbb{E}[\\mathbf{\\kappa}^l_t]\n$$  \nUsing these compressed gradients, rate-based backpropagation can be performed by iterating only over spatial dimensions:  \n$$\n\\left(\\frac{\\partial \\mathcal{L}}{\\partial c^l}\\right)\\_\\text{rate}\n= \\left(\\frac{\\partial \\mathcal{L}}{\\partial c^L} \\prod_{i=L-1}^{l} \\left(\\frac{\\partial c^{i+1}}{\\partial r^i} \\left(\\frac{\\partial r^i}{\\partial c^i}\\right)\\_\\text{rate}\\right)\\right)\n= \\left(\\frac{\\partial \\mathcal{L}}{\\partial c^L} \\prod_{i=L-1}^{l} \\left(W^{i^\\top} \\mathbb{E}\\left[\\kappa^l_t\\right]\\right)\\right)\n$$  \nIn summary, the main difference between rate-based backpropagation and BPTT arises from the assumptions on the mean estimator representation of firing rates. By assuming temporal components are independent, the rate-coding approach establishes equivalence with BPTT. Denote $\\delta_t(s^l) = \\frac{\\partial \\mathcal{L}}{\\partial s^l_t}$ as the gradient of spike $s^l_t$ computed via BPTT, and $\\kappa_t^l = \\sum_{\\tau} \\frac{\\partial s_t^l}{\\partial I_{\\tau}^l}$ as the accumulated gradient of input $I_t^l$ on all spikes. If all layers satisfy $\\mathbb{E}\\left[\\delta_t(s^l) \\, \\kappa_t^l\\right] = \\mathbb{E}\\left[\\delta_t(s^l)\\right] \\mathbb{E}\\left[\\kappa_t^l\\right],$  \nthe gradient and accumulated response are statistically independent, then $\\mathbb{E}\\left[\\delta_t(s^l)\\right] = \\left( \\frac{\\partial \\mathcal{L}}{\\partial \\mathbf{r}^l} \\right)\\_{\\text{rate}}.$  Further, let $\\delta_t(I^l) = \\frac{\\partial \\mathcal{L}}{\\partial I_t^l}$ and assume $\\mathbb{E}\\left[\\delta_t(I^l)\\,s_t^{l-1}\\right] = \\mathbb{E}\\left[\\delta_t(I^l)\\right]\\mathbb{E}\\left[s_t^{l-1}\\right].$ Then the rate-based weight gradient equals the BPTT:$\\left( \\nabla_{W^l} \\mathcal{L} \\right)\\_{\\text{rate}} = \\frac{1}{T} \\nabla_{W^l} \\mathcal{L}.$ Thank you again for your suggestion. A more detailed process will be updated in future versions."}]

Author rebuttal:

Meta-review: The authors propose an enhanced self-distillation framework for efficient training of Spiking Neural Networks (SNNs). Reduced training complexity and high performance are demonstrated in experiments on standard visual benchmark tasks.
Strengths:
- A novel and innovative method for efficient SNN training is proposed
- It shows improved training efficiency
- Good empirical results are obtained
- Includes a theoretical analysis of effects of reliable and unreliable teacher signals
Weaknesses:
- The method relies on rate-based approximations, does not exploit temporal dynamics
- The method is based on well-known approaches, which somewhat limits its novelty.
In conclusion, the manuscript presents interesting analyses and very good performance. The method is innovative within the SNN field. There are some weaknesses, but in summary the contribution was well received by the reviewers. I therefore propose acceptance.
Decision: Accept (poster)