Synergy Between the Strong and the Weak: Spiking Neural Networks are Inherently Self-Distillers
Authors: Yongqi Ding
Lin Zuo
Mengmeng Jing
Kunshan Yang
Pei He
Tonglan Xie
Venue: NeurIPS 2025 poster ()
Keywords: Spiking Neural Networks
Neuromorphic Computing
Self-Distillation
Primary area: deep_learning

Abstract: Brain-inspired spiking neural networks (SNNs) promise to be a low-power alternative to computationally intensive artificial neural networks (ANNs), although performance gaps persist. Recent studies have improved the performance of SNNs through knowledge distillation, but rely on large teacher models or introduce additional training overhead. In this paper, we show that SNNs can be naturally deconstructed into multiple submodels for efficient self-distillation. We treat each timestep instance of the SNN as a submodel and evaluate its output confidence, thus efficiently identifying the strong and the weak. Based on this strong and weak relationship, we propose two efficient self-distillation schemes: (1) Strong2Weak: During training, the stronger "teacher" guides the weaker "student", effectively improving overall performance. (2) Weak2Strong: The weak serve as the "teacher", distilling the strong in reverse with underlying dark knowledge, again yielding significant performance gains. For both distillation schemes, we offer flexible implementations such as ensemble, simultaneous, and cascade distillation. Experiments show that our method effectively improves the discriminability and overall performance of the SNN, while its adversarial robustness is also enhanced, benefiting from the stability brought by self-distillation. This ingeniously exploits the temporal properties of SNNs and provides insight into how to efficiently train high-performance SNNs.

Synergy Between the Strong and the Weak: Spiking Neural Networks are Inherently Self-Distillers
Yongqi Ding, Lin Zuo, Mengmeng Jing, Kunshan Yang, Pei He, Tonglan Xie
School of Information and Software Engineering
University of Electronic Science and Technology of China
{yqding, ksyang, 202321090226, 2021090907006) Gastd messtc.edu. cn linzuoQuestc edu. en, 1agmeng11922gmaiinn com
Abstract
Brain-inspired spiking neural networks (SNNs) promise be low-power alter- native to computationally intensive artificial neural networks (ANNs), although performance gaps persist. Recent studies have improved the performance of SNNs through knowledge distillation, but rely on large teacher models or introduce ad- ditional training overhead. In this paper, we show that SNNs be naturally deconstructed into multiple submodels for efficient self-distillation. We treat each timestep instance of the SNN as a submodel and evaluate its output confidence, thus efficiently identifying the strong and the weak. Based on this strong and weak relationship. we propose two efficient self-distillation schemes: (1) Strong2 During training, the stronger "teacher" guides the weaker "student", effectively im- proving overall performance. (2) Weak2Strong: The weak serve the "teacher", distilling the strong in reverse with underlying dark knowledge, again yielding significant performance gains. For both distillation schemes, we offer flexible implementations such as ensemble, simultaneous, and cascade distillation. Ex- periments show that our method effectively improves the discriminability and overall performance of the SNN, while its adversarial robustness is also enhanced, benefiting from the stability brought by self-distillation. This ingeniously exploits the temporal properties of SNNs and provides insight into how to efficiently train perfooreffrm SNNs.
1 Introduction
Currently, artificial neural networks (ANNs) have shown outstanding performance in both computer vision and natural language processing 2 However, ANN inference involves intensive multiply- accumulate (MAC) operations that consume significant power, limiting its deployment power- constrained scenarios such as edge devices 3. To reduce power consumption toward sustainable artificial intelligence, brain-inspired research being conducted in anticipation of low-power artificial intelligence through neuromorphic computing [4]
Spiking Neural Networks (SNNs) are considered to be low-power alternative to ANNs by trans- mitting information through bionic binary spikes, avoiding heavy MAC operations, and requiring only energy-effin effegyeefficient accumulation (AC) operations 3 5. Moreover, spiking neurons, which model the membrane potential dynamics of biological neurons that evolve over time, are able to extract underlying temporal features, making SNNs more favorable for exploiting their potential in temporal tasks 6 7. Combined with neuromorphic chips and dynamic vision sensors, SNNs can perform various tasks with ultra-low latency and power consumption [8]. For example, the SNN on the Tianmouc chip requires only 0.7mW to perform typical vision tasks [9.
*Correspondigg author.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

However, limited by the binary information representation. there is still performance gap between SNNs and ANNs. Further improving the performance of SNNs can facilitate their widespread deployment in real-world scenarios. Previous studies have optimized the spiking neurons [10
12
13
network architectures 14 15], spike-specific BN layers [16] 17,, and residual connections [8
19
20]. significantly narrowing the performance gap. Another studies have focused on optimizing
the training process of SNNs to be compatible with a variety of neurons and architectures, have received widespread attention [21 22 [3]] Inspired by knowledge distillation, recent efforts have attempted to improve the performance of lightweight SNNs with additional teacher models (ANNs or SNNs) 224 25 However, teacher models incur additional pre- pre-training overhead and typically need to be customized for specific tasks, making the training process more cumbersome. To reduce the additional overhead, (26) extends the timestep during training and adds weak classifier for spatio-temporal seff-distillation. (27) relies on real labels to guide incorrect output with correct output, which suffers from limited efficiency and performance. Given these unresolved issues, question worth exploring is: How to distill a high-perfone nigh-peeformance SNN without additional overhead?
Following the philosophy of Occam's ' razor, we answer the above question by proposing a simple yet effective self-distillation strategy for SNNs. To begin with, we deconstruct the SNN in the temporal dimension, considering its instances at each timestep as submodel. Then, we evaluate the output confidence of each submodel to identify the strong and the weak ones, respectively. Based on the strong and weak relationships, we propose two efficient self-distillation schemes: (1) Strong2 Weak takes the strongest submodel as the "teacher" and guides the weakest "student" submodel to strengthen the discriminative ability of the weaker model, elevating the "short boards in the barrel" to improve the overall performance. (2) Weak2Strong works the other way around, where the weakest model serves "teacher" to guide the strongest "student". It is worth noting that Weak2Strong exploits the underlying dark knowledge of the weak "teacher" to augment the strong "student", e.g., an overly strong model may ignore detailed information leading to overfitting, while a weak model provides complements or acts as a regularizer, thus improving generalization [28]. Both schemes allow flexible implementations, with default one-to-one distillation and alternative ensemble, simultaneous and cascade distillation. Moreover, these two distillation schemes are compatible with various network architectures and neurons with superior generalization.
Visualizations and extensive experiments on both static and neuromorphic datasets show that the proposed self-distillation schemes narrow the performance gap between submodels of the SNN, improve overall performance, and enable inference with low latency. Meanwhile, the synergy between the strong and the weak promotes overall stability and enhances the robustness of the SNN against attacks. The contribution of this paper can be summarized as follows:
We show that SNNs can be deconstructed into multiple submodels in the temporal dimension, allowing for efficient self-distillation without any additional overhead.
We efficiently identify the strong and the weak in the deconstructed submodels by confi- dence, and propose Strong2Weat and Weak2Strong self-distillation schemes to improve the performance of SNNs.
Visualizations and experiments on both static and neuromorphic datasets show that the proposed self-distillation schemes enhance the discriminability, performance, and robustness of the SNN and enable satisfactory performance with ultra-low latency.
2 Related work
Spiking Neural Network. SNNs have received considerable attention due to their low-power 31], especially in light of the significant energy requirements associated with large language models. To improve the performance of SNNs, a plethora of studies have been conducted on spike encoding [2), neuron models [00 [2], network architectures [3] [5], and
characteristics 29 30,
training methods 21 22 23 The contributions of this paper belong training methods, compatible with various encoding, neuron models and network architectures with great generalization. Compared to other training methods, this paper deconstructs SNN into multiple submodels and identifies the strong and weak submodels for self-distillation learning, which significantly improves performance without additional overhead.
Knowledge Distillation, Knowledge distillation originated in ANNs, which attempt to boost the performance of weak student model with a strong teacher model [3), leaving only the student
2

---

for inference without compromising inference efficiency. Depending on the spatial location of the distillation signal, distillation can be categorized into logit distillation [4 351 and feature distillation [36 37 Logit distillation encourages the student to mimic the teacher's output, and feature distillation promotes intermediate feature consistency between the two. Feature distillation typically offers superior performance, but the difficulty in obtaining the intermediate features of the teacher model in many scenarios (for privacy and security reasons) makes logit distillation more versatile and easier to implement (38 39 40). To improve distillation efficiency, self-distillation methods separate the teacher and student from a single model, e.g., by using intermediate checkpoints adding multiple output heads (28 41. This paper focuses on self-distillation learning in SNNs and shows that SNNs can naturally distinguish between the strong and the weak without any checkpoints or additional output heads, and are well suited for efficient self-distillation.
Eisillation-Fnhaanee SNN. Given the remarkable gains of knowledge distillation in ANNs, the community has introduced it to train SNNs. Previous studies have shown that large ANNs and SNN teachers can improve the performance of small student SNNs [2
25
4
However, the teacher model requires addi- tional storage and computational overhead. To enhance the efficiency, self-distillation in SNNs has been explored. [27] deter- mines the correctness of the SNN output at each timestep, with the correct one guid- ing the incorrect one. However, this ne- cessitates label information and cannot ex- ploit the underlying dark knowledge in the error output, resulting in limited perfor- mance. [26] extends the training timestep and adds a additional weak classifier for spatio-temporal self-distillation, which im- proves the distillation performance but im- poses a heavier training overhead. Un- like existing methods, our self-distillation schemes require no additional overhead and exploit both strong and weak dark knowledge with superior performance. A visual comparison with other distillation methods shown in Fig.
Weak
Soft latel
H limestop extensor
Prosioms welfddisillatiin
Hand lubel
Decondrrect
strong
Sulf hubel
SNN
Mentify
The weok
Disciliaioon with teacher model
Submodek
Our self-disillation
Figure 1: Comparison to other distillation methods. Our self-distillation deconstructs an SNN into multiple sub- models and identifies the strong and weak ones for self- distillation without any additional overhead.
Deconstructing the SNN
3.1 Temporal Properties of SNNs
The core of the SNN lies the spiking neurons, which endows with low power consumption and temporal properties. Similar to biological neurons, spiking neurons maintain a membrane potential state, and the membrane potential changes continuously over time, depending on the input current received. In this paper, we use the most commonly used leaky integrate-and-fing (LIF) 45) neurons, but it is worth noting that our method can be applied to other neurons as well, since both have similar dynamics.
Let H denote the membrane potential of the LIF neuron and indicate the input current generated by the previous layer of spikes and synaptic weights, the membrane potential dynamics of the LIF neuron can be expressed as:
Hi(t) = -(111111 (1- H (t I((),
(1)
where f, L, and are the timestep, layer, and neuron indices, respectively. 7 is the membrane potential time constant, which controls the degree of leakage of the membrane potential with time, i.e. the membrane potential of the LIF gradually decreases when there is input.
If the membrane potential reaches the firing threshold 9, the spiking neuron generates spike with a value of 1, otherwise the output value is 0 (i.e. no spike is generated). After the spike is fired, the membrane potential reset. This paper adopts the soft reset mechanism [46], i.e., the membrane potential is reduced by the same amount the threshold:
3

---

R0:1)
Softmax function
Aararr The strong help the weak Lowas The weak cnhance the strong
Submodel
20
The strong
Logical deconstructin
Submodel
Confidence evalustion
Csaw
Lapas
Shared
(88)
architecture parameters
The weak
[0;T]
Submodel
Identify the strong and the weak Figure 2: Our method logically deconstructs the SNN with T timesteps into T submodels with the same architecture and parameters, and evaluates the confidence level of each submodel to identify the strong and the weak. The strong help the weak through distillation, and the weak transfer underlying dark knowledge to the strong, thus improving overall performance.
SNN with timestepp
Deconstructed T submodels
(1(() (1(() (1)0,
where S is the binary spike output. The residual membrane potential after reset remains until the next timestep to continue the dynamics (Eq.[), resulting in the initial membrane potential typically varying across timesteps. This residual membrane potential is considered carry time-dependent information, so that the SNN can be used for temporal tasks
3.2 Deconstructing SNNs from the Temporal Dimension
The output of a spike neuron at a given timestep depends not only on the initial membrane potential (t) but also on the received imput (t). The difference between the two causes the output of the SNN to vary over timestep. Existing strategies average the outputs of multiple timesteps to improve overall output stability [5. (5. However, we exploit this variability across timesteps to deconstruct an SNN into multiple submodels along the time dimension.
Specifically, we consider the instances of the SNN at each timestep as submodel. These submodels, while sharing the same architecture and parameters, can produce different outputs due to differences neuron states and input currents. Let the SNN be (f), where f(-) and 9 denote the architecture and parameters, respectively. Assuming that the SNN runs for a total of T timesteps, we deconstruct into T submodels: (f( 1), (0; 2). ... (0; T)}. as shown in Fig. 2
In this way, we deconstruct an SNN into multiple submodels with different outputs. These multiple distinct outputs form the necessary conditions for distillation learning, that self-distillation be performed without the need for other modules or strategies to generate additional outputs. Notably, this deconstruction is not limited network architecture and neuron type, and is highly generaliz- able. In the following we describe how to perform self self-distillation learning with the deconstructed submodels.
4 Self-Distillation with Deconstructed SNNs
With the deconstructed multiple submodels, the key distillation is to identify the teacher and student models. A naive distillation scheme is based on accuracy/loss, with high-accuracy/low- loss submodel as the teacher guiding the low- accuracy/high-loss student. However, this man- ner (i) requires step-by-step comparisons with labels, which complicates the distillation process; (ii) cannot be used in scenarios where labels are missing, such as streaming online learning; and (iii) cannot exploit the underlying dark knowledge,
Table Accuracy comparison (%) of different implementations.
Method
CIFARIO-DVS DVS Gesture
High-accuracy teacher
78.43:10..3
90.62+t0.6
Low-loss teacher
78.93-10333
90.3941077
Strong2Weak (Ours)
78.93:0.12
91.4310.43
Weak22trrong (Ours)
79.33=10.29
91.20+0.33
resulting in limited performance, shown in Table I To this end, propose to quantify the output

---

confidence of each submodel to efficiently evaluate the strong and weak relationships, and propose two distillation schemes based on the strong and weak relationships.
4.1 Identify the Strong and the Weak
Formally, let the output of submodel f(0;t) be o(t) (t), 02 (t). ( number of the object categories. We calculate its predicted probability distribution across the C categories as
(t)), where C' is the
p(t) softmax(o(t)).
(3)
Then, define the output confidence of the submodel as the maximum probability in p(t):
con(t) max(p(t)).
(4)
As such, con(t) reflects the degree of certainty the submodel with respect to the output, similarly to [47].
Based on the output confidence con(t), we evaluate the strong and weak relationships of these submodels. For any two submodels, we consider the one with the higher confidence to be the stronger one and the other be the weaker one. Therefore, we can identify the strongest and the weakest among the 7 submodels and thus exploit this strong and weak relationship for distillation learning. In particular, relying on confidence to evaluate strong and weak relationships does not require label information and is computationally simple without sacrificing efficiency.
4.2 The Strong Help the Weak
In classical knowledge distillation, the teacher model typically carries more parameters than the stu- dent and therefore better performance, i.e. strong teachers guide weak students [33 34). Accordingly, we propose Strong2Weak distillation: the strong submodel leads and enhances the weak submodel, thus improving overall performance.
Specifically, in each iteration evaluate the confidence of each submodel and identify the strong and weak submodels with the highest and lowest confidence. The strong submodel is considered as the teacher and the weak submodel the student, and logit distillation learning is performed between the two.
Formally, let the timestep index of the strong submodel be t, and that of the weak submodel be tw- The output logits of the two submodels are o(tz) and ottw) respectively. We soften the logit to obtain the probability distribution:
(tts)saaa
ca(t))s/a
p(t.), D caltale/a p(tw),
(5)
Ee co(tw)a/a"
where a is the softening factor, set to 2. We then use KL divergence to transfer knowledge from the strong model the weak model:
plg p(ta)i p(tn))
C a
(6)
jul
During training of the SNN, the distillation loss and the cross entropy loss synergistically optimize the parameters. The total loss is therefore:
Listat Lcr(O,Y) + Aszw Lsaw,
where Y the label information and Asaw is the coefficient, which defaults to 1.
During training, we use the rectangular surrogate gradient for optimization, since the spike activity nonddifferentiable [45). Thus, the derivative of the spike activity with respect the membrane potential can be calculated as:
aL Th (b(H)(()) 0)
sign(((I((() 01 ;)
(8)
(H! (t)
(t
where a is a hyperparameter that controls the shape of the gradient and is set to 1.0.
5

---

4.3 The Weak Enhance the Strong
In the real world, teachers are also improved as they teach their students. addition, it has been shown that weak models are able to learn underlying dark knowledge, which can contribute to the performance of stronger models [28]. For example, an overly strong model may ignore detailed information and suffer from overfitting. while weak model may provide complements act regularizer to facilitate generalization [28]. To this end, we propose the Weak2Strong distillation scheme: the weak submodel acts as the teacher, guiding the strong student fully utilize the potential dark knowledge to improve overall performance.
Similar to Strong2Weat distillation, in Weak2Strong distillation we first identify the timestep indices of the strong and weak submodels obtain their output logits o(x,) and o(t..). Then we compute the two corresponding probability distributions according to Eq. [] When calculating the distillation loss, we use p((m) as the teacher to prompt p((,,) to learn the dark knowledge in o((m)):
Σ plt
p(tw)i p(t,))
(9)
Combined with the cross-entropy loss, the final loss is obtained as:
Listat (o Y) + \w2sLwas,
(10)
where Aw2s is the hyperparameter coefficient set to
The schematic diagrams of these two distillation schemes are shown in Fig. 2 In this way, self- distillation can be achieved without additional teacher models or auxiliary modules, and we provide pseudocode for Strong Weak and Weak2Strong distillation in Alg. [ Note that in identifying the strong and weak submodels, we take the confidence mean of a batch samples rather than distilling sample by sample.
4.4 Flexible Delf-Distillation Implementation
The above describes the default implementation of the Strong2Weak (S2W) and Weak2Strong (W2S) self-distillation schemes, i.e., one-to-one distillation of the strongest and weakest submodels with KI. divergence. addition, we would like to emphasize that the deconstructed SNN allows flexible distillation implementations with great extensibility. Below we discuss two flexible components that can be reimplemented: distillation losses and teacher-student configurations. student configurations.
Remark1: Flexible distillation loss function. In Eq. 6 and Eq. 9 we use the most com- monly used KL divergence the distillation loss. However, as shown in Table 2 our method is able to seamlessly support other distillation functions (e.g.. MSE and Logit standardization 40)) with consistently superior performance. This ficxibility allows practition- ers further improve performance by incorporating specially designed loss functions (3
Table 2: Performance with flexible distillation loss functions (%).
CIFARIO-DVS -es-ure
Loss function
S2W
W2S
S2W
W2S
KL divergence MSE
78.93
79.33
91.43
91.20
76.50
76.67
92.47
91.44
Logit standardization
40
75.93
75.77
91.67
91.55
35
40
based on task requirements, establishing our method as a generalized substrate for high-performance SNNs.
Remark2: Flexible teacher and student configurations. At each iteration, we distill between the strongest and weakest submodels. Since the timestep usually greater than 2, there are options to use more submodels to compose the teacher or student in the implementation. For example, an average of the T higher/lower confidence submodels can be used as the teacher to guide the weakest/strongest. and we refer to this implementation as the ensemble teacher. Similarly, a highest/lowest confidence submodel can also guide the remaining T - submodels, i.e., the ensemble student. Alternatively,
Table 3: Performance with flexible implemen- tations (%).
CIFARIO-DVS
DVS-Gesture
Implementation
S2W
W2S
S2W
W2s
Default
78.93
79.33
91.43
91.20
Ensemble teacher
78.53
79.17
92.02
91.78
Ensemble student 78.30
78.37
91.55
91.55
Simultaneous
79.17
91.32
Cascade
79.03
79.37
93.41 92.13
both S2W and W2S distillations can be performed simultaneously, cascade distillation can be

---

performed depending on the confidence level. We detail and analyze these alternative implemen- tations in Appendix A Table T shows the performance with various implementations. indicating that flexible implementations can further improve performance, especially for cascade distillation. worth noting that the simultaneous use of S2W and W2S did not achieve significantly better performance because the excessive similarity between the submodels reduces the overall diversity of the SNN, thus limiting generalizability. How to balance diversity and similarity between multiple submodels is perennial problem in ensemble learning [48 49], and we leave it for future work.
worth noting that while our proposed method of self-distillation after deconstruction offers wide scope for extension and implementation, the performance under different implementations is often affected by various factors such as task, data, and experimental setup. Therefore, we have kept the default configuration in the following experiments to highlight the superiority of our efficient distillation methods rather than the specific implementation, leaving further extensions for future work. More importantly, experimental results show that even the default implementation, which is not deliberately tuned, can significantly improve the performance of vanilla SNNs, especially for neuromorphic tasks with temporal dimensions.
5
Experiments
We perform experiments both static images CIFARIO/100 [50]. ImageNet and neuromorphic datasets CIFAR IO-DVS [5I], DVS-Gesture [52] to confirm the effectiveness and performance advan- tages of the proposed self-distillation schemes. Both VGG and MS-ResNa [19], and Transformer architectures are used the experiments. To reduce the influence of randomness, we report the
average results of three trials. The detailed experimental setup is presented in Appendix B
5.1 Ablation Studies
In Table we compare Strong2Weak and Weak2Strong the vanilla SNN on static and neuromorphic datasets. The results show that for both VGG and MS-ResNet architectures, Strong2Weak and Weak2Strong are able to pro- vide consistent performance gains. In particular, recognition accuracy on CIFARIO-DVSS was im- proved by up to 5.26%, a more significant im- provement than on static datasets. This is due to the lack of temporal features in static images, so the performance differences between the decon- structed submodels are minimal and do not max- imize the distillation effect. Neuromorphic data, on the other hand, is rich in temporal features, taking full advantage of the mutually enhancing effect of strong and weak submodels. This sheds light the potential of our method to be applied to more temporal tasks in the future.
Table 4: Ablation studies. The proposed self- distillation schemes show consistent performance gains, especially on neuromorphic datasets with temporal properties.
Accuracy (%)
Dataset
Method
VGG-9 MS-ResNet18
Vanilla SNN
94.21
94.88
CIFARIO
Strong2
94.79+0.58
95.15
Weak2Strong
94.74...40
95.13
Vanilla SNN
74.41
76.33
CIFAR100
Strong2Weak7666 76.02.1.61
78.25.1.92
Weak2Strong 76.16.1.75
77.98+1.65
Vanilla SNN
73.97
66.40
CIFARTO-DVS
Strong2Weak
78.93 44.96
70.50
Weak2Strong
79.33.5.36
71.57.5.17
Vanilla SNN
87.85
89.35
DVS-Gesture
Strong2
91.43,3.58
90.86
Weak2Strong
91.
21.20
91.55,2,20
5.2 Submodel Output Distribution Visualization
To illustrate more intuitively how the strong and weak submodels enhance each other, the output 2D t-distributed stochastic neighbor embedding (t-SNE) distributions of the submodels for each timestep are visualized in Fig. 3 In Fig. aa), the vanilla SNN exhibits a large output gap across timesteps and is particularly weak at t = O. Excessive differences between the strong and the weak affect the overall performance. Compared
the vanilla SNN, the Strong2Weal distillation guides the weak and compensates the deficiencies, thus contributing to the overall performance. Specifically, Weak2Strong distills the weak into the strong and can improve the performance of the weak as well, embodying the concept of teaching and learning. From an overall perspective, Strong2 and Weak2Strong improve the discriminability of the submodel outputs each timestep, which improves the overall stability and performance. The visualization of the overall output can be found in Appendix
7

---

Vanilla SNN
(b) 2trongWWeak Weak
(c) Weak2Strong
Figure 3: Visualization of the output distribution of cach timestep submodel. (a) The vanilla SNN produces confusing outputs at 0, showing dramatic gaps between the strong and the weak. (b) (c) Both Strong2Weal and Weak2Strong are able to improve the output discriminability of the submodels, thus bridging the gap between strong and weak to improve the overall stability and performance.
Table 6: Improved adversarial robustness perfor- mance (%) of the proposed distillation schemes. Experiments were performed with VGG-11 on CIFARI00, T 8.
Table 5: Comparative results (%) for reduced timestep inference on CIFARIO-DVS, Our distil- lation schemes maintain decent performance even with reduced inference timesteps.
Method
Clean FGSM PGD BIM CW
Method
T=1
T=2
T=3
T=4
T=5
FEEL-SNN 54]
63.95 61.97 9.87
2.13 1.93 6,21
Vanilla SNN
10.00
60.10
69.50
73.30
74.10
AT 55
67.97 67.47 17.55
9.52 8.91 20.23
MPS [53]
66.60
74.30
75.50
75.70
76.60
RAT
[66
69.99 69.06 19.00
9.11
8.46 22.59
Strong2Weak
71.50
76.20
77.10
78.20
78.60
Strong2 Weak
70.68 69.53 21.15
9,430 23.40
Weak2Strong
73.40
76.50
77.50
78.80
79.70
Weak2Strong
70.09 68.87 21.23
10.70
9.75 24.08
5.3 Improved Performance at Low Timesteps
Timestep and inference latency positively correlated. If lower latency is required, inference can be performed with fewer timesteps. The performance of the SNN at different inference timesteps shown in Table using the pre-trained model with 5 timesteps. The performance of the vanilla SNN degrades significantly as the timestep is reduced. In particular, at timestep, the corresponding random guess. Membrane potential smoothing [53] mitigates this problem to some extent, and decent performance can be achieved low latencies. Our distillation schemes narrow the gap output across timesteps, improving overall stability and maintaining satisfactory performance even with reduced timesteps. This makes it possible to train model to apply under various latency constraints without retraining.
submodel is too weak, leading performance equivalent
5.4 Robustness Gains from Self-Distillation
The proposed self-distillation schemes improve the overall stability and performance of the SNN by guiding the strong and weak submodels towards each other. Since stability is closely related to robustness [54]. we investigate whether our methods can improve the robustness of SNNs and thus increase the reliability of deployment. We perform robustness experiments on CIFAR 100 using VGG-11, and the attack methods include common noise attack (Gaussian noise, GN) and adversarial

---

Table 7: Comparative results on ImageNet.
Table 8: Comparative results on CIFARIO/100.
Method
Architecture
T Acc.(%)
Method
Architecture
CIFARIO CIEARTOO
w/o KD
Shortcut
TAB
[7
60
ICL822
Neur
ResNe134
ResNet34
4
4
67.78
68.14
we KD
RMP-Loss
NDOTo
CLIF
12
ResNet-18
VGG-16
VGG-11
4
94.89
94,39
94.79
76.18
77.00
73.30
ESTA-SNN
AAA725
ResNet34
70.23
MS--ReNNet
19
MS-ResNet18
94.88
76,33
STAA-SNN
62
CVPR'2
ResNet34
4
70.40
Spikformer
68
Transformer
4
93.94
75.96
SEW-ResNet
NewrIPS'21 21
WWW-ReNNe34
67.04
QKFormer
SSCL 65
QKFormer
ResNet-20
94.27
96.18
72.86
81.15
IMP+LTS
64
TCLR225
SEW-ResNeth 4
68.90
EnOF
44
ResNet-20 20
4
94.74
73.01
SSCL 4.AA222
ResNet34
66.78
KDSNN
ResNet-18
BKDSNN 43
ResNet-19
4
94.64
74.95
EnOF
44
ResNet34
4
67.40
TKS
27
ResNet-19
4
96.35
79.89
KDSNN 24
PV'233
NEW-ResNer34
67.28
MS-ResNet18
95.15
78.25
MPS
53
CCR'22
SEW-ResNet34
4
69.03
Strong2Weak
ResNet-19
4
96.62
81.83
TKS
20
T4
JEW-ResNet's 4
69.60
OKForeer
4
96.42
81.27
MS-ResNet18
95.13
77.98
Week
SEW-ResNer34
70.53
Weak2Strong
ResNet-19
4
96.66
82.02
Weak2Strong
SEW-ResNe134 4
69.87
QKFormer
96.29
81.23
attacks (FGSM [55]
PGD with random start [57] BIM [58], and CW [59].. We combine the proposed
methods with RAT 56 to verify that they can maximize robustness conjunction with each other.
Experimental details are given in the Appendix B
The comparative adversarial robustness results are shown in Table 6 where our method further improves the performance of RAT (56) for both clean and adversarial samples. In particular, Weak2Strong improves robust accuracy under FGSM attacks by 2.23%, which is quite significant. This confirms that the stability provided by self-distillation can indeed improve the robustmess of the model against attacks, and we will explore further effects of distillation on robustness promotion in the future.
5.5 Comparison with Other Methods
Table 9: Comparative results (%) neuromorphic datasets.
Static datasets. The comparative results on ImageNet are shown in Table Our method achieves 70.53% accuracy at 4 timesteps, surpassing other distillation methods and specially designed modules. Compared to TKS [27]. which does not require teacher model for distillation, we outperform it by 0.93% in accuracy. Additionally, we also achieved accuracies of 96.66% and 82.02% on CIFARIO 10 and CIFAR 100, respectively. which again exceeded the other methods, as shown Table We also show results on
Method
Archi.
T CIFARIO-DVS DVS-Gesture
DSR (70
VGG-11
20
77.27
RMP-Loss [66
ResNet-20
10
75.60
NDOT (67
VGG-11
10
77.50
w/o
TAB
VGG-9
5
74.57
90.86
SLT 21
VGG-9
5
74.23
89.35
SSNN
VGG-9
5
73.63
90.74
MPS S3
VGG-9
5
76.77
93.23
TSSD 26
VGG-9
5
72.90
86.69
SSCL
65
ResNet-20
10
78.50
FnOF 14
ResNe-20
10
80.50
VGGSNN
85.30
in Table [8] where method can be combined with CILF [I2] neurons to further improve performance.
Strong2Weak
VGG-9
5
78.93
91.43
VGGSNN
10
85.60
VGG-9
5
79.33
91.20
Weak2Strong
VGGSNN
10
86.70
Neuromorphic datasets. The comparative results on neuromorphic datasets are shown Table Weak2Strong achieved 86. 70% accuracy CIFAR10 1FFAR10-DVS using the VG- GSNN architecture with T 10, which was 1.40% higher than TKS 27 In addition, our distillation schemes also enhance the SNN with Transformer architecture. The performance of both the SDT 3 and the QKFormer (69 architectures improves further, demonstrating the versatility our method.
SDT 3
Transtormer
5
72.53
93.98
QKForner
QKFormer
84.00 16
98.60
Transtormer
73.57
95.14
Strong2Weak
5
82.50
95.14
QKFomeer
16
84.60
99.08
Transformer
S
73.20
94.91
Weak2Strong
5 82.80
94.91
QKFormer
84.90
98.96
5.6 Combined with the Early Exit Dynamic Inference
Table 10 shows the performance of our method when combined with the early exit mechanism 47 for dynamic inference. used the VGG-9 model on the CIFAR10-DYY and DVS-Gesture datasets, setting the carly exit thresholds to 0.95, 0.9, 0.8, and 0.5. The results suggest that method is highly

---

Table 10: The proposed method can be combined with the early exit dynamic inference. 1 ennt is the early exit threshold, T is the average inference timestep.
Table 11: The influence of different metrics used to identify strong and weak submodels on perfor- mance. The straightforward and intuitive confi- dence metric achieved competitive performance.
CIFARIO-DVS
DVS-Gesture Acc. (%)
Method
T Acc. (%)
T
Metric
Distillation
CIFARIO-DVS WSS-eeeure
Vanilla SNN
5
73.97
5
87.85
Strong2Weak
5
78.60
92.01
Confidence Strong2Weak
78,93
91.43
+Early exit evii 0.95)
3.16
78.40
2.35
91.32
Entropy Strong2Wcak
78.80
90,97
+Early exit (1
0.90)
2.73
78.20
1.87
90.62
Margin Strong2Weak
78.83
91.55
+Early exit (dams 0.80)
2.30
77.80
1.56
90.28
Diversity Strong2Weak
78.27
90.57
+Early cxit (Wesit 0.50)
1.38
74.70
1.06
88.54
Confidence Strong2Weak
79.33
91.20
Weak2Strong
5
79.70
5
91.67
Entropy Strong2Weak
78.57
90.05
+Farly exit (ffesst 0.95)
3.19
79.50
2.22
91.67
Margin Strong2Weak
78.57
90.97
+Early exit (ferit 0.90)
2.74
79.40
1.89
90.62
Diversity Strong2Weak
78.73
91.78
+Early exit (8 0.80) 2.27
78.80
1.58
89.93
+Early exit (Beast 0.50) 1.38
75.80
1.01
86.11
compatible with the early-exit mechanism. Specifically, when the exit threshold is set to 0.8, our method achieves accuracies of 77.8% (Strong2Weak, T = 2.3) and 78.8% (Weak2Strong, T 2.27) on CIFAR1 LIFAR1O-DVS O-DVS. Compared to the vanilla SNN, our method demonstrates significantly improved performance while reducing latency by half. This further demonstrates the potential of our method for ultra-low-latens inference.
5.7 The Influence of Metrics on the Strong and the Weak
We use confidence as the default metric to distinguish between strong and weak submodels because can be efficiently calculated using only a softmax function. In addition, confidence typically reflects the uncertainty of model output results and has been shown to be related to generalization error, which to certain extent reflects performance 77]. Due to these advantages, confidence is widely used to evaluate model performance in areas such as model calibration 772 73
and early exit [74 47).
As general distillation method, we can seamlessly replace confidence with other metrics such as entropy. margin (maximum confidence difference between target category and other categories). diversity (confidence differences between all categories), or other specially designed metrics. Table IT shows the influence of different metrics on performance. The results suggest that different metries do not yield significant performance differences. This indicates that the straightforward and intuitive confidence is sufficient and effective for distinguishing between strong and weak submodels.
Conclusion
In this paper, deconstruct the multi timestep SNN into submodels and distill between the sub- models, thereby climinating the need for additional overhead. We identify the strong and the weak by evaluating the output confidence of the submodels, and propose two self-distillation schemes, Strong2Weak and Weak2Strong, respectively, where the strong helps the weak and the weak trans- fers the underlying dark knowledge to the strong. These two distillation schemes can be flexibly implemented, such as one-to-one, ensemble, simultaneous, and cascade distillation, with great ex- tensibillity. Extensive visualizations and experiments show that this efficient self- dif-diitillltiinn method effectively mitigates strong and weak gaps between submodels and significantly improves low-latency inference performance. In particular, the adversarial robustness of the model is also gained with the overall stability improvement delivered by seff-distillation. This will contribute to efficient and high-performance studies of SNNs.
Limitation. The aim of this paper is to demonstrate efficient self-distillation schemes for SNNs, and therefore there is deliberate tuning of the distillation loss functions and coefficients, which allows the performance of the proposed method to be further enhanced by improving these.
Any highly efficient, perforerform model is at risk of misuse in the real world, especially SNNs, which are known for their low power consumption and low latency. To mitigate the risk of misuse, we believe it is necessary to integrate offensive and defensive strategies and security measures into the proposed method. As an alternative, privacy-preserving mechanisms, such federated learning, should be incorporated during deployment.
10

---

Acknowledgments
This work was supported by the National Natural Science Foundation of China under Grant No. 62276054 and 62406060.
References
[1] Kunshan Yang. Lin Zuo, Mengmeng Jing, Xianlong Tian, Kunbin Hc, and Yongqi Ding. Flexible vig: Learning the self-saliency for flexible object recognition. IEEE Transactions on Circuits and Systems for Video Technology, pages 1-1, 2025.
[2] Zhiwei Hao, Jianyuan Guo, Kai Han, Han Hu, Chang Xu, and Yunhe Wang. Revisit the power of vanilla knowledge distillation: from small scale to large scale. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors, Advances in Neural Information Processing Systems, volume 36, pages Curran Associates, Inc., 2023.
101 10170-102
[3] Man Yao, JiaKui Hu, Zhaokun Zhou, Li Yuan, Yonghong Tian, Bo Xu, and Guoqi Li. Spike-driven transformer. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors, Advances in Neural Information Processing Systems, volume 36, pages 64043 64058. Curran Associates, Inc., 2023.
Kaushik Roy, Akhilesh Jaiswal, and Priyadarshini Panda. Towards spike-based machine intelligence with neuromorphic computing. Nature, 575(77844:667--61, 2019.
[5] Yongqi Ding, Lin Zuo, Mengmeng Jing, Pei Hc, and Yongjun Xiao. Shrinking your timestep: Towards low-latency neuromorphic object recognition with spiking neural networks. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 11811-11819, 2024.
[6] Wachirawit Ponghiran and Kaushik Roy. Spiking neural networks with improved inherent recurrence dynamics for sequential learning. In Proceedings of the AAAI Conference on Artificial Intelligence. volume 36, pages 8001-8008, 2022.
[7] Youngeun Kim, Yuhang Li, Hyoungseob Park, Yeshwanth Venkatesha, Anna Hambitzer, and Priyadarshini Panda. Exploring temporal information dynamics in spiking neural networks. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pages 8308-8316, 2023.
[8] Jibin Wu, Chenglin Xu, Xiao Han, Daquan Zhou, Malu Zhang, Haizhou Li, and Kay Chen Tan. Progressive tandem learning for pattern recognition with deep spiking neural networks. JEEE Transactions on Pottern Analysis and Machine Intelligence, 44(11):7824-788 7840, 2022.
[9] Zheyu Yang. Taoyi Wang. Yihan Lin, Yuguo Chen, Hui Zeng, Jing Pei, Jiazheng Wang, Xue Liu, Yichun Zhou, Jianqiang Zhang, et al. A vision chip with complementary pathways for open-world sensing. Nature, 6296 2024.
[10] Wei Fang, Zhaofei Yu, Yanqi Chen, Timothée Masquelier, Tiejun Huang, and Yonghong Tian. Incorporating learnable membrane time constant enhance learning of spiking neural networks. In Proceedings of the IEFE/CVF International Conference on Computer Vision (ICCV), pages 2661-2671, October 2021.
[11] Xingting Yao, Fanrong Li, Zitao Mo, and Jian Cheng. GLIF; unified gated leaky integrate- fire neuron for spiking neural networks. In Alice H. Oh, Alckh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems, 2022.
[12] Yulong Huang. Xiaopeng LIN, Hongwei Ren, Haotian FU, Yue Zhou, Zunchang LIU, biao pan, and Bojun Cheng. CLIF: Complementary leaky integrate- and and-fire neuron for spiking neural networks. In Forty-first International Conference Machine Learning, 2024.
[13] Yongqi Ding, Lin Zuo, Kunshan Yang, Zhongshu Chen, Jian Hu, and Tangfan Xiahou. An improved probabilistic spiking neural network with enhanced discriminative ability. Knowledge-Based Systems, 280:1 10024, 2023.
[14] Boyan Li, Luziwei Leng, Shuaijie Shen, Kaixuan Zhang, Jianguo Zhang, Jianxing Liao, and Ran Cheng. Efficient deep spiking multilayer perceptrons with multiplication-free inference. IEEE Transactions on Neural Networks and Learning Systems, pages 1-13, 2024.
[15] Xinyu Shi, Zecheng Hao, and Zhaofei Yu. Spikingresformer: Bridging resnet and vision transformer in spiking neural networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 5610-5613 June 2024.

---

[16] Chaoteng Duan, Jianhao Ding, Shiyan Chen, Zhaofei Yu, and Tiejun Huang. Temporal effective batch normalization in spiking neural networks. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems, volume 35, pages 34377-34390. Curran Associates, Inc., 2022.
[17] Haiyan Jiang. Vincent Zoonekynd, Giulia De Masi, Bin Gu, and Huan Xiong. TAB: Temporal accumulated batch normalization in spiking neural networks. In The Twelfth International Conference on Learning Representations, 2024.
[18] Wei Fang, Zhaofei Yu, Yanqi Chen, Tiejun Huang, Timothée Masquelier, and Yonghong Tian. residual learning in spiking neural networks. Advances in Neural Information Processing Systems, 34:21056-21069, 2021.
[19] Yifan Hu, Lei Deng, Yujie Wu, Man Yao, and Guoqi Li. Advancing spiking neural networks toward deep residual learning. IEEE Transactions on Neural Networks and Learning Systems, pages 1-15, 2024.
[20] Yangfan Hu, Huajin Tang, and Gang Pan. Spiking deep residual networks. IEEE Transactions on Neural Networks and Learning Systems, 34(8):5200-
2023.
[21] Srinivas Anumasa, Bhaskar Mukhoty, Velibor Bojkovic, Giulia De Masi, Huan Xiong, and Bin Gu. Enhancing training of spiking neural network with stochastic lateney. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 10900-10908, 2024.
[22] JiaKui Hu, Man Yao, Xuerui Qiu, Yuhong Chou, Yuxuan Cai, Ning Qiao, Yonghong Tian, Bo XU, and Guoqi Li. High-performance temporal reversible spiking neural networks with Samathcal (O) (O)()$ training memory and (O) inference cost. In Forty-first International Conference on Machine Learning, 2024.
[23] Lin Zuo, Yongqi Ding, Wenwei Luo, Mengmeng Jing, Xianlong Tian, and Kunshan Yang. Temporal reversed training for spiking neural networks with generalized spatio-temporal representation, 2024.
[24] Qi Xu, Yaxin Li, Jiangrong Shen, Jian K. Liu, Huajin Tang, and Gang Pan. Constructing deep spiking neural networks from artificial neural networks with knowledge distillation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 7886-7895, June 2023.
[25] Ravi Kumar Kushawaha, Saurabh Kumar, Biplab Banerjee, and Rajbabu Velmurugan. Distilling spikes: Knowledge distillation in spiking neural networks. In 2020 25th International Conference on Pattenn Recognition (ICPR). pages 4536 4543, 2021.
[26] Lin Zuo, Yongqi Ding, Mengmeng Jing, Kunshan Yang, and Yunqian Yu. Self-distillation learning based on temporal-spor -patial consistency for spiking neural networks, 2024.
[27] Yiting Dong, Dongcheng Zhao, and Yi Zeng. Temporal knowledge sharing enable spiking neural network learning from past and future. IEFF Transactions on Artificial Intelligence, 357:33554,,34, 2024.
[28] Li Yuan, Francis EH Tay, Guilin Li, Tao Wang, and Jiashi Feng. Revisiting knowledge distillation via label smoothing regularization. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2020.
[29] Qiaoyi Su, Yuhong Chou, Yifan Hu, Jianing Li, Shijie Mei, Ziyang Zhang, and Guoqi Li. Deep directly- trained spiking neural networks for object detection. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV). pages 6555 6565, October 2023.
[30] Malyaban Bal and Abhronil Sengupta. Spikingbert: Distilling bert to train spiking language models using implicit differentiation. In Proceedings of the AAAI conference on artificial intelligence, volume 38, pages 10998- 11006, 2024.
[31] Guobin Shen, Dongcheng Zhao, Tenglong Li, Jindong Li, and Yi Zeng. Are conventional snns really efficient? a perspective from network quantization. In Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 27538-27547, June 2024.
[32] Xuerui Qiu, Rui-Jie Zhu, Yuhong Chou, Zhaorui Wang, Liang-jian Deng, and Guoqi Li, Gated attention coding for training high-performance and efficient spiking neural networks. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 601 610, 2024.
[33] Geolfrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in neural network, 2015.
[34] Ying Jin, Jiaqi Wang, and Dahua Lin. Multi-level logit distillation. In 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 24276 01776-24255.02 2023.
12

---

[35] Zheng Li, Xiang Li, Lingfeng Yang, Borui Zhao, Renjie Song, Lei Luo, Jun Li, and Jian Yang. Curriculum temperature for knowledge distillation, 2022.
[36] Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang, Carlo Gatta, and Yoshua Bengio. Fitnets: Hints for thin deep nets, 2015.
[37] Martin Zong, Zengyu Qiu. Xinzhu Ma, Kunlin Yang, Chunya Liu, Jun Hou, Shuai Yi, and Wanli Ouyang. Better teacher better student: Dynamic prior knowledge for knowledge distillation. In The Eleventh International Conference on Learning Representations, 2023.
[38] Mengyang Yuan, Bo Lang, and Fengnan Quan. Student-friendly knowledge distillation, 2023.
[39] Zhendong Yang, Ailing Zeng. Zhe Li, Tianke Zhang, Chun Yuan, and Yu Li. From knowledge distillation to self-knowledge distillation: unified approach with normalized loss and customized soft labels, 2023.
[40] Shangquan Sun, Wenqi Ren, Jingzhi Li, Rui Wang. and Xiaochun Cao. Logit standardization in knowledge distillation. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 15731-15740, June 2024.
[41] Linfeng Zhang. Jiebo Song, Anni Gao, Jingwei Chen, Chenglong Bao, and Kaisheng Ma. Be your own teacher: Improve the performance of convolutional neural networks via self distillation. In 2019 IEEF/CVF International Conference on Computer Vision (ICCV), pages 3712-3721, 2019.
[42] Yufci Guo, Weihang Pong, Yuanpei Chen, Liwen Zhang, Xiaode Liu, Xuhui Huang, and Zhe Ma. Joint a-snn: Joint training of artificial and spiking neural networks via self-distillation and weight factorization. Pattern Recognition, 109699,,, 2023.
[43] Zekai Xu, Kang You, Qinghai Guo, Xiang Wang, and Zhezhi He. Bkdsnn: Enhancing the performance of learning-based spiking neural networks training with blurred knowledge distillation. Computer Vision Cham, 2024.
ECCV 2024, pages 106-123,
[44] Yufei Guo. Weihang Peng, Xiaode Liu, Yuanpei Chen, Yuhan Zhang, Xin Tong. Zhou Jie, and Zhe Ma. EnOF-SNN: Training accurate spiking neural networks via enhancing the output feature. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024.
[45] Yujie Wu, Lei Deng. Guoqi Li, Jun Zhu, and Luping Shi. Spatio-temporal backpropagation for training pigh-performance spiking neural networks. Frontiers Neuroscience, 12, 2018
[46] Bing Han, Gopalakrishnan Srinivasan, and Kaushik Roy. Rmp-snn: Residual membrane potential neuron for enabling deeper high-aceuracy and low-latency spiking neural network. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2020.
[47] Yuhang Li, Tamar Geller, Youngeun Kim, and Priyadarshini Panda. Scenn: Towards temporal spiking early exit neural networks. In A. Oh, T. Naumann, A. Globerson, K. Sacnko, M. Hardt, and S. Levine, editors, Advances in Neural Information Processing Systems, volume 36, pages 63327-63342. Curran Associates, Inc., 2023.
[48] Muhammad Ammar Ali, Yusuf Sahin, Süreyya Özögür Akyüz, Gozde Unal, and Buse Cisil Otar. Tuning diversty tradc-off in neural network ensemble via novel entropy loss function. In 2021 13th International Conference on Electrical and Electronics Engineering (ELECO), pages 365-368, 2021.
[49] Shaofeng Zhang, Meng Liu, and Junchi Yan. The diversified ensemble neural network. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 16001-16011. 1. Curran Associates, Inc., 2020.
[50] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers features from tiny images. 2009.
[51] Hongmin Li, Hanchao Liu, Xiangyang Ji, Guoqi Li, and Luping Shi. Cifar10-dvs: An event-stream dataset for object classification. Frontiers in Neuroscience, 11, 2017.
[52] Amon Amir et al. A low power, fully event-based gesture recognition system. In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 1388-7397, 2017.
[53] Yongqi Ding, Lin Zuo, Mengmeng Jing, Pei He, and Hanpu Deng. Rethinking spiking neural networks from an ensemble learning perspective. In The Thirteenth International Conference on Learning Representations, 2025.
[54] Mengting Xu, De Ma, Huajin Tang, Qian Zheng, and Gang Pan. FEEL-SNN: Robust spiking neural networks with frequency encoding and evolutionary leak factor. In The Tiitty- Annual Conference on Neural Information Processing Systems, 2024.
13

---

[55] Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. arXiv preprint arXiv:1412.6577, 2014.
[56] Jianhao Ding, Tong Bu, Zhaofei Yu, Tiejun Huang, and Jian K Liu. SNN-RAT: Robustness-enhanced spiking neural network through regularized adversarial training. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems, 2022.
[57] Aleksander Madry, Towards deep learning models resistant to adversarial attacks. arXiv preprint arXiv:/706.0000 2017.
[58] Alexey Kurakin, Ian Goodfellow, and Samy Bengio. Adversarial examples in the physical world. In Artificial intelligence safety and security, pages 112. Chapman and Hal//CRC, 2018.
[59] Nicholas Carlini and David Wagner. Towards evaluating the robustness of neural networks. In 2017 IEEE Symposium on Security and Privacy (SP), pages 39-57,201 2017.
[60] Yufei Guo, Yuanpei Chen, Zecheng Hao, Weihang Peng, Zhou Jie, Yuhan Zhang, Xiaode Liu, and Zhe Ma. Take shorteut back: Mitigating the gradient vanishing for training spiking neural networks. In The eighth Annual Conference on Neural Information Processing Systems, 2024.
[61] Kairong Yu, Tianqing Zhang, Hongwei Wang. and Qi Xu. Fsta-smnffrequenccy-bees spatial-temporal attention module for spiking neural networks, 2025.
[62] Tianqing Zhang, Kairong Yu, Xian Zhong, Hongwei Wang. Qi Xu, and Qiang Zhang. Staa-snn: Spatial- temporal attention aggregator for spiking neural networks, arXiv preprint S.Ki::2503.026899 2025.
[63] Wei Fang. Zhaofei Yu, Yanqi Chen, Tiejun Huang, Timothée Masquelier. and Yonghong Tian. Deep residual learning in spiking neural networks. In Ranzato. A. Beygelzimer. Y. Dauphin, P.S. Liang. and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems, volume 34, pages 21056-21069. Curran Associates, Inc., 2021.
[64] Hangchi Shen, Qian Zheng, Huamin Wang, and Gang Pan. Rethinking the membranc dynamics and optimization objectives of spiking neural networks. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024.
[65] Yuhan Zhang, Xiaode Liu, Yuanpei Chen, Weihang Peng, Yufei Guo, Xuhui Huang, and Zhe Ma. Enhanciig representation of spiking neural networks via similarity-sensitivee contrastive learning. In Proceedings of the AAAI Conference Artificial Intelligence, volume 38, pages 16926-16934, 2024.
[66] Yufei Guo. Xiaode Liu, Yuanpei Chen, Liwen Zhang, Weihang Peng. Yuhan Zhang, Xuhui Huang, and Zhe Ma. Rmp-loss: Regularizing membrane potential distribution for spiking neural networks. In Proceedings of the IEEE/CVP International Conference on Computer Vision (ICCV), pages 17391-17401, October 2023.
[67] Haiyan Jiang, Giulia De Masi, Huan Xiong, and Bin Gu. NDOT: Neuronal dynamies- based online training for spiking neural networks. In Forty-first International Conference on Machine Learning, 2024.
[68] Zhaokun Zhou et al. Spikformer: When spiking neural network meets transformer. In The Eleventh International Conference on Learning Representations, 2023.
[69] Chenlin Zhou, Han Zhang, Zhaokun Zhou, Liutao Yu, Liwei Huang, Xiaopeng Fan, Li Yuan, Zhengyu Ma, Huihui Zhou, and Yonghong Tian. QKFormer: Hierarchical spiking transformer using q-k attention. In The eiihtt- Annual Conference on Neural Information Processing Systems, 2024.
[70] Qingyan Meng. Mingqing Xiao, Shen Yan, Yisen Wang. Zhouchen Lin, and Zhi-Quan Luo. Training high-performance low-latency spiking neural networks by differentiation on spike representation. In Proceedings of the IEFE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 12444-1243 12453, June 2022.
[71] Wei Bian and Dacheng Tao. Constrained empirical risk minimization framework for distance metric learning. IEEE Transactions on Neural Networks and Learning Systems, 194-11055 2012.
[72] Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. On calibration of modern neural networks. In Doina Precup and Yee Whye Teh, editors. Proceedings of the 34th International Conference on Machine Learning. volume 70 of Proceedings of Machine Learning Research, pages 1321 1130. PMLR, 06 11 Aug 2017.
14

---

[73] Hongxin Wei, Renchunzi Xie, Hao Cheng, Lei Feng. Bo An, and Yixuan Li. Mitigating neural network overconfidence with logit normalization. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, Sivan Sabato, editors, Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pages 23631-23644 PMLR, 17-23 Jul 2022.
[74] Le Yang. Yizeng Han, Xi Chen, Shiji Song, Jifeng Dai, and Gao Huang. Resolution adaptive networks for efficient inference. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2020.
[75] Wei Fang. Yanqi Chen, Jianhao Ding. Zhaofei Yu, Timothée Masquelier, Ding Chen, Liwei Huang, Huihui Zhou, Guoqi Li. and Yonghong Tian. Spikingjelly: An open-source machine learning infrastructure platform for spike-based intelligence. Science Advances, 9(40) 1480, 2023.
[76] Qu Yang, Jibin Wu, Malu Zhang, Yansong Chua, Xinchao Wang, and Haizhou Li. Training spiking neural networks with local tandem learning. Koyejo, Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems, volume 35, pages 12662-12676. Curran Associates, Inc., 2022.
[77] Ziming Wang, Runhao Jiang, Shuang Lian, Rui Yan, and Huajin Tang. Adaptive smoothing gradient lcarning for spiking neural networks. In International Conference on Machine Learning, pages 35798- 35816. PMLR, 2023.
15

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The abstract and introduction accurately reflect the contribution and scope of this paper.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We explicitly point out the limitations of this paper.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
16

---

Justification:
We demonstrate the superior performance of our method through experimenta-
tion.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proof' can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification:
We clearly illustrate the experimental details.
Guidelines:
The answer NA means that the paper does include experiments. If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
17

---

Answer: [Yes]
Justification: Our method is based on open-source datasets and existing methods, and provides detailed experimental details.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.cm ec/
public/guides/ htwps///oooeeuabsssssoooooooo
for more details.
While encourage the release of code and data, understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https for more details.
//nips. //nips.c//pil wwwppbblccgguiicuceccccc wet.ggddddeeoeeebbiniinn
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We provide the details of the experiments in the Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer:
[Yes]
Justification:
We report the average results of three trials to minimize the errors.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
18

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We provide information about the computing platform.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics httpppp///orrips..ccppbbtt//t/t
Answer: [Yes]
Justification:
Our research adheres to the NeurIPS Code of Ethics in every respect.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: This paper has potential social impact.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no socictal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
19

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
This point is irrelevant to the topic of this paper.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification:
We use open-source assets and provide explicit references.
Guidelines:
The answer NA means that the paper does
existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
20

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: This point irrelevant the topic of this paper.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmoel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: This point is irrelevant to the topic of this paper.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question:
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
This point is irrelevant to the topic of this paper.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
21

---

Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
Justification: This point is irrelevant the topic of this paper.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs any important, original, non-standard components.
Please refer to our LLM policy (https://neure) httppp////////ooeeeeeee///////// for what should or should not be described.
22

---

Algorithm Pseudocode for the proposed self-distillation schemes
# x: Input data with dimension 8T,
$ y: Label information with dimension [8,00- enn:
# out: Dutput of the SXN with dimansion DT,
def [al_.distillatiooxx...
=mn(x)
atrong,eeak (sut)ify(aut)
P_R.P._ = aoftmar(atrong softnax
I_ce pross_entropy (oteroopooottnnae(()))
if Strong2Weak:
L_s2e KL_dáv(log(p_.) p_t)
[_total I_ce Lanbda+L__o2e
el5r Neak28trrng::
L_V2e KL_div dL..dv(1logpp.b) pp_))
[_total L_ce v_b
return 1_total
Submodal output confidence with dim identify(eut):
T out, shape [0]
zeros(T)
range (T)
tmp cax dim-1), dim-1)
con[t] = mean (tup)
nax piin,iinniie index argmex(con) (rngni
strong out [max_ index]
weak out [nin_index]
strong, weak
A Appendix A: Flexible Self-Distillation Implementation
While Strong2Weat Weak and 2Sak2St are implemented as one-to-one distillations by default, both can be implemented in flexible ways such as ensemble teacher, ensemble student, simultaneous distillation, and cascade distillation. Below are descriptions of alternative distillation implementations.
For SNN with T > 2 timesteps, it can be deconstructed into T submodels. In this case, one submodel taken as the student, while the remaining set of submodels ensembled as the teacher model to guide the student. When Strong2Weal is implemented, the student model is the one with the lowest confidence; when Weak2Strong is implemented, the student model the one with the highest confidence.
Ensemble teacher:
For T' 2 submodels, the ensemble student takes one of the submodels as the teacher, while the remaining ensemble submodels are distilled as the student model, which is the opposite of the ensemble teacher.
Ensemble student:
For the strong submodel with the highest confidence and the weak submodel with the lowest confidence, simultaneous distillation is performed for both Strong2 and Weak2Strong distillation, i.e., the final loss is the cross-entropy loss LCr(O, Y) + Asaw S2w
Simultaneous distillation:
>w25
Cascade distillation: For T > submodels, cascade distillation ranks them in order of decreasing confidence obtain submodels {Subi, Sub,..., two submodels with adjacent rankings. For example, for Sub, and Suby+1, the implementation of Strong2Weak takes Sub; the teacher and makes Sub++1 the student, and the opposite is true for the implementation Weak2Strong. We provide Py PyTorch-style pseudocode for cascade distillation in Alg.1 2
and then distillation is performed between
A.1 Performance Analysis of Various Implementations
The performance of different implementations on CIFARIO-DVS and DVS- WVS-Gesture is shown in Table. 12 The results show that the performance of these implementations varies slightly from dataset to dataset, where cascade distillation consistently delivers better performance. addition, we would like to emphasize that we did not deliberately adjust the hyperparameters in Table. [2] such as the coefficients of Strong2Wwend Wweak and Weak2Strong losses, which should have an impact on the performance, especially for simultancous distillation.
23

---

Algorithm 2 PyTorch-style pseudocode for implementing cascade distillation
# x: Input data with dimension [T,8,C,8,k]
$ y: Label information with dimension [8,M]- enn:
# out: Dutput the SXN with dimansion [T, def bascade_diatillettee
lanbda))
=zn(x)
loss 0
ranked_ indices ronk(out)
in (T-ne(T-1):
strong out cerranke__iddieeeeee
veak out [ranked_ indiiee [i+1]]
P_s.p_x softnax (strong) (ooamax(veaak)
StrongZleak:
loss t= (L,div(log(p_. .p_s)
olif Neakk2Srrong:
losa Ki.,div(log(p_a) .p_.)
lose /- (T-1)
L_ce getss_eatroyyoot.ncom
I_total L_ce
return L_total
con:
output confidence dim (T)
# ranked indices: Tinestop indices ranked in descending confidence order.
def rank(out):
out. shapep
con zeros([)
for t in (ange(T))
tup nax (eof[ttaaxxoutt[]]] dix-1), din-1) con(t] (tap))
ranked_inee indices aroco.argoot,c,on, drscending-/rru)
ranked_ fankediindices
Table 12: Performance with flexible distillation implementations (%). The coefficient for all dis- tillation losses is set to Performance varies slightly for different implementations, with cascade distillation consistently giving better performance.
CIFARIO-DVS
DVS-Gesture
Distillation configuration
S2W
W2S
S2W
W2S
Default
78.93
79.33
91.43
91.20
Ensemble teacher
78.53
79.17
92.02
91.78
Ensemble student
78.30
78.37
91.55
91.55
Simultaneous
79.17
91.32
Cascade
79.03
79.37
93.41
92.13
Performance Gains from Simultaneous Distillation. The simultaneous distillation of the weak by the strong and the distillation of the strong by the weak is able to utilize both the representational power of the strong and the dark knowledge of the weak, and thus should be expected yield greater gains than either alone. However, in Table. [2] simultaneous distillation does not perform optimally as expected. We attribute this to too much similarity between multiple submodels. From an ensemble learning perspective [48] 53 49), excessive similarity between members leads to decrease in overall diversity, which is unfavorable for generalization. However, how to balance similarity and diversity is perennial challenge in ensemble learning [48 49
that we leave for future work.
In addition, the adjustment of the distillation loss coefficients is expected reduce the conflict between sim ilarity and diversity, thereby improving the performance of simultaneous distillation. For example, incorporating advanced knowledge distillation loss adjustment methods and multi-task loss factor settings. Since our goal is to provide efficient self-distillation schemes for SNNs rather than carefully selecting optimal hyperparameter settings, therefore this paper we not deliberately combine other methods to improve the accuracy. When using the self-distillaiio schemes provided in this paper for performance-critie tasks, incorporating additional strategies to improve performance is further work worth exploring.
24

---

B Appendix B: Experimental Details
B.1 Datasets
We perform experiments on static images and neuromorphic datasets.
CIFARIO and CIFARI00 (50] are static image benchmark datasets containing 10 and 100 classes of 32 32 color images, respectively. Both datasets contain 50,000 training images and 10,000 test images. For CIFARIO 10 and CIFAR IOF data, preprocessed them using standard data augmentation strategies: random cropping, horizontal flipping, and normalization. We also use the AutoAugment strategy for CIFARIO.
The ImageNet dataset of 1.2 million training images, 50,000 validation images, and 150,000 test images with 1,000 categories is the most challenging object recognition benchmark. For the ImageNet dataset, we unify the images to a 224 x 224 size during training and testing, and evaluate the performance of our method on the test set.
CIFARIO-DVS dataset [51] is the neuromorphic version of the CIFARIO dataset. The CIFARIO-DVS dataset has 10,000 samples for a total of 10 object classes, and the dimension of each sample is [t. p, 3/. where the timestamp, p is the polarity of the intensity change of the corresponding pixel, and x and are the spatial coordinates of the pixel point, respectively. The spatial size of each sample in CIFARIO-DVS is 128 128, which we downsampled to 48 x 48 resolution before inputting to the SNN. Additionally, due to the high temporal resolution of the neuromorphic dataset, we integrate neuromorphic sample into T event frames [T, p. x. y] using the SpikingJelly framework [75] to match the timestep T of the SNN. For each training, we randomly divide 90% of the data as the training set and test on the remaining 10% of the data, which is by far the most common strategy [5].
The DVS-Gesture [52] dataset contains neuromorphic data for 11 hand gestures with 1 1176 training samples and 288 test samples. The dimension of each sample is [T, p, x; v), and we downsample its spatial resolution from 128 x 128 to 48 x 48 before feeding the samples into the SNN. The pre-processing of the DVS-Gesture data is the same as in CIFARIO-DVS, which also utilizes the SpikingJelly framework to obtain the event frame [T", p, I, y) by integrating it by timestep.
B.2 Implementation Details
Nvidia RTX 4090 GPU. For the VGG-9 and MesRes architectures, we follow the training strategy of [3: train the model with an initial learning rate of 0.1 for 100 epochs, reducing it by a factor of ten every 30 epochs. A stochastic gradient descent optimizer with a momentum of 0.9 and a batch size of 64 was used. The weight decays for the static and neuromorphic datasets are le-4 and 1e-3, respectively. We used the LIF neuron model with firing threshold 1 of 1.0 and membrane potential time constant of 2.0.
Our experiments are based the PyTorch package rumning
When using CLIF neurons, replace the LIF neurons in model with CLIF neurons and leave the other parameters unchanged. We use the CLIF neuron implementation from the publicly available code of the original paper [12].
When using the Spike- driken Transformer architecture, we followed the training strategy of the original model on CIFAR10-DVS 33 200 epochs were trained using the Spike-driven Transformer- 2-256 architecture, see for details.
When using the ResNet19 architecture, for fair comparison with TKS 27 we use the same training strategy as TKS and use their published code.
For the QKFromer (69 architecture, we use the proposed method directly on the officially released code and follow its training settings. However, when we used our self-distillation for QKFormer, we noticed a performance degradation when we set the loss function coefficient to 1, so we set this value to 0.01.
To reduce the influence of randomness, we repeated all our experiments three times, and the average results are reported in the paper. Notably, when evaluating the performance of different inference timesteps using the pre-trained five-timestep model, we only report the results of the evaluation on a single pre-trained model, and do not conduct multiple experiments using multiple models.
25

---

To evaluate the robustness of the proposed method, we follow the training setup of [54] and rely on its publicly available code. The parameters, architecture, etc. related to the robustness experiment are the same as in 54 For robustness results, we report the results of single trial.
C Appendix C: Additional Visualizations
C.1 t-SNE Visualization
The comparison of the 2D t-SNE visualization of the overall output of the SNN is shown in Fig. where our distillation schemes produce outputs that are more discriminative and thus have better performance.
Vanilla
Strong2
Strang
70
20
20
10
-20
-20
-20
-10
Figure 4: Visualization of the overall output of the vanilla SNN, Strong2 distillation, and Weak2Strong distillation. Strong2Weak and Weak2Strong distillation schemes provide superior performance by allowing for more discriminable outputs than the vanilla SNN.
C.2 Timestep Evolution for Strong and Weak Submodels
During training, we examined the timesteps corresponding to the strong and weak submodels and found that the weak submodel was not always located at the earliest timestep and that the strong model was not always the last. In fact, the weak submodel appears at every timestep, shown Fig. [5] However, it appears most frequently the earliest timestep and less frequently other timesteps. This is consistent with our experimental results: the vanilla SNN performs worst at the earliest timestep, while our method significantly improves the performance of the first timestep, so it is not always the weakest. The strong submodel is less likely to be selected in the first timestep, but typically selected for subsequent timesteps. This demonstrates that our method of evaluating submodels based confidence and dynamically selecting timesteps for distillation is capable of adapting to the training SNNs to a certain extent. Otherwise, would degrade to distillation with specified timesteps.
Changes timestep accuracy streng the training
timettee
Weak Limaltp
Strnn accuracy
Weak acoracy
Timesto
Accuracy
Epoch
Figure The evolution trend timestep indices corresponding strong and weak submodels during training.
26

---

Additionally, we compare distillation with specified timesteps. We construct Last2First and First2Last distillation schemes under the assumption that the first timestep is the weakest and the last timestep is the strongest. The comparative results are shown in Table 16 The results show that specifying these two timestep distillations is much less effective than our confidence selection distillation. This highlights the importance of dynamically selecting teacher and student timesteps.
D Appendix D: Additional Experiments
D.1 Influence of Loss Function Coefficients
By default, the distillation function coefficients for Strong2 and Weak2Stroog to 1.0. In Table we explore the performance for other values of the loss function coefficients. The results show that the accuracy varies slightly with the coefficients, but remains stable overall (except for the weights of 0.1 and 2.0). The adjustment of the loss coefficients based on well-established theories should further contribute the performance of the proposed methods, taking into account the relevant work in the field of knowledge distillation.
Table 13:
Influence of loss function coefficient values performance (%). slightly with the coefficients, but remains stable overall.
The accuracy varies
CIFARIO-DVS
DVS-
DVS-Gesture
Coefficient
S2W
W2S
S2W
W2S
0.1
77.27
76.43
93.06
91.67
0.5
78.30
78.57
92.13
91.55
1.0
78.93
79.33
91.43
91.20
1.5
79.13
78.53
90.51
90.16
2.0
79.17
78.17
89.24
87.96
D.2 Influence of Distillation Temperature
The default distillation temperature is set to 2.0. Table 14 shows the influence of distillation tem- perature on performance. The results indicate that adjusting the distillation temperature within reasonable range only causes slight fluctuations in performance without leading to significant degra- dation. These results also demonstrate that our method can achieve satisfactory performance without the need for deliberate adjustments to the distillation temperature. However, when the temperature or distillation weight falls outside the normal range (e.g., a temperature 0.5), the performance of our method deteriorates. Nevertheless, it consistently outperforms vanilla SNNs.
Table 14: Influence of distillation temperature performance (%).
CIFARIO-DVS
DVS-Gesture S2W
Temperature
S2W
W2S
W2S
0.5
75.30
76.77
91.09
91.32
1.0
77.63
78.43
92.13
91.44
2.0
78.93
79.33
91.43
91.20
3.0
78.27
78.43
90.86
91.21
5.0
78.75
79.03
92.01
91.67
D.3 Comparison with High Accuracy Submodel Teacher
After deconstructing the SNN into multiple submodels, naive distillation method is to select submodel with the highest accuracy among them as the teacher guiding the submodel with the lowest accuracy. However, the naive method has three weaknesses that limit its availability and performance.
The distillation process is complicated by the comparison the label on timestep basis for each training batch.
27

---

It can only be used with labeled learning, and the accuracy of each submodel cannot be evaluated if labels are missing.
Relying only accuracy to assess the strong and the weak, without being able to utilize the underlying dark knowledge, leads to limited performance. As shown in Table T5 this naive method inferior to two proposed distillation schemes on both CIFAR TO-DVS and DVS-GeSSS Gesture.
Table 15: Comparative results (%) with high-accuracy submodel teacher (Abhreviated as HAST). This method inferior to our two proposed distillation schemes both CIFARIO-DVS and DVS-Gesture.
Method
CIFAR10-DVS DVS-Gesture
HAST
78.43
90.62
Strong2Weak
78.93
91.43
Weak2Strong
79.33
91.20
D.4 Comparison with Random Submodel Distillation
Table Toble|T]shone shows the comparative results between the proposed method and random submodel distilla- tion (where two randomly selected submodels of different time steps are distilled). The experimental results show that:
(1) Random distillation outperforms the vanilla SNN, even without deliberately selecting the teacher and student. We consider this method to randomly transition between Strong2Weat Weak and Weak2Strong, achieving an effect similar to using both simultaneously. This makes our deconstruction- destillation scheme
flexible and versatile. However, distillation cannot be abused entirely between submodels. As shown by the Last-to-First and First-to-Las Last results, performing distillation only between the first and last time steps leads to significant performance degradation.
(2) Using confidence to determine teacher and student submodels for distillation consistently achieved better performance than random distillation. This shows that although confidence is straightforward, can be used to determine the performance of a model relatively well (though not absolutely). In fact, previous research empirical risk minimization has shown that confidence is usually correlated with the generalization error of a model, i.e., high confidence often corresponds to low generalization error [7I]. In contrast, random distillation cannot guarantee optimization in the direction of either strong to weak (toward low generalization error) or vice versa (toward high generalization error regularization). This easily leads conflict between consistency and diversity, similar suboptimal performance when using both simultaneously.
Table 16: Comparative results (%) between the proposed method and random submodel distillation.
Method
CIFARIO-DVS
DVS DVS-Gesture
Vanilla SNN
73.97
87.85
Last-to-First
74.75
89.12
First-to-Last
74.29
88.89
Hundom-to-Randonn
78.03
90.16
Strong2Weak
78.93
91.43
Weak2Strong
79.33
91.20
D.5 Further Analysis of the Adversarial Robustness Gains of the Proposed Method
Our method guides the student submodel using the output of the teacher submodel as soft labels, which can simultaneously achieve distillation and regularization effects promote better consistency and generalization. To investigate the source of the gains in adversarial robustness, we compared our method with those of typical label smoothing and entropy regularization (with the same settings as Table 6) VGG-11 with 8 timesteps was used on CIFAR 100). Further experimental results are shown in Table 17 We found that using regularization methods alone can achieve decent performance on clean samples. However, pure regularization methods severely degrade and are significantly inferior
28

---

to our method when faced with adversarial attacks. Therefore, we consider the source of the SNN's robustness the distillation that promotes internal consistency. making it immune to interference. However, it should be emphasized that regularization also plays role to certain extent, just as label smoothing can also achieve decent robustness.
Table 17: Comparative results (%) of the proposed method versus label smoothing and entropy regularization in terms of adversarial robustness.
Method
Clean
GN
FGSM
PGD
BIM
CW
Label smoothing
69.92
68.52
18.29
8.42
7.89
23.65
Entropy regularization
70.05
69.11
17.89
8.17
7.38
18.88
Strong2Weak
70.68
69.53
21.15
10.30
9.40
23.40
Weak2Strong 70.09
68.87
21.23
10.70
9.75
24.08
D.6
Comparative Results on Tiny-ImageNet
The results of the proposed self-distillation schemes compared other methods on Tiny- Iny-ImeeeNet are shown in Table Our method achieves competitive performance compared to other methods. In addition, our method can seamlessly integrated with CLIF [2] to further improve performance. This demonstrates the potential of our method to fuse with wider range of neurons.
Table 18: Comparative results on Timy-ImageNet dataset. * denotes self-impleementation results with open-source code.
Method
Architecture
T
Accuracy(%6)
Online LTL 76
VGG-16
16
56.87
ASGL 777
VGG-13
8
56.81
Joint A-SNN
VGG-16
55.39 4
CLIF 12
VGG-13
61.93" 4
VGG-I3(LIF)
4
59.78
Strong2Weal
(GG-I3(CLIF)
4
62.65
VGG-13(LIF)
59,40
Weak2Strong
VGG-13(CLIF)
4
62.45
29


Reviews:
[{"review_id": "IsoZynlwE2", "reviewer": "Reviewer_oyjh", "summary": "The paper proposes a self-distillation strategy for spiking neural networks to improve their performance. This method leverages the inherent temporal characteristics of spiking neural networks to deconstruct them into multiple submodels. It determines whether a submodel is strong or weak based on the logical distribution of outputs from different submodels, and performs self-distillation training among different submodels.", "questions": "1. The author mentioned dark knowledge in Weak2Strong distillation, which is not very clear. It would be better if the author could elaborate on the specific characteristics of the \"dark knowledge\" of weak submodels have (such as noise robustness, generalization ability)? Experimental results on this would be really helpful.", "limitations": "Yes.", "rating": 2, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n1. This paper describes the implementation of the method in great detail, making it easy to understand. Core contents such as the decomposition strategy and confidence calculation method are elaborated thoroughly.\n\n\nWeaknesses:\n1. The novelty of the first contribution is not clear. Previous methods, e.g. [26, 46], also treat different time steps of SNN models as different submodels.\n2. The significance of the first contribution is not clear. The SNN at each time step is part of the whole model. So it is obvious that it can be considered as 'part of the whole model', i.e. submodel. It would be better if the authors could elaborate more on why they consider this as a technical or theoretical contribution. Also, Eqn. 1 and 2 are not used in explaining why SNN at each timestep is a submodel. I suggest the author focus on proving that the SNN at each time step is submodels with natural synergy in its output. \n3. The effectiveness of identifying strong and weak submodels is also unclear. The performance of Strong2Weak and Weak2Strong is comparable. There are two possible potential issues that could benefit from experimental analysis. (1) confidence score might not rank strong and weak submodel very well, resulting in similar performance. (2) It doesn't matter which submodel we use in self-distillation, the performance will all be similar. To address these potential issues, it would be great if the auther could compare Strong2Weak and Weak2Strong with random-to-random submodel distillation. If Strong2Weak and Weak2Strong performs comparable with random-to-random, then the second contribution is not effective.", "was_revised": true, "final_justification": "The results on the significance are interesting, but the manuscript would still benefit from a rewrite to address the novelty issues.", "rebuttal": "We sincerely thank you for the time and effort you have invested in reviewing our paper, and we also appreciate your easy-to-understand positive evaluation of our method. Below, we will respond to each of the weaknesses and questions you have pointed out, and we hope to address your concerns.\n\n**Response to Weakness 1 and Weakness 2 (novelty and significance of the first contribution)**: In this paper, we explicitly deconstruct SNNs from the time dimension into multiple submodels. In particular, we point out that each timestep submodel of the SNN shares the same architecture and parameters, with the difference being the pre-charge membrane potential at the beginning of each timestep and the input current, which causes these submodels to produce different outputs. Based on this, we state that identifying strong and weak spiking submodels can utilize the temporal dynamics inherent in SNNs to efficiently self-distill, i.e., improve performance by generating teachers and students from the SNN itself, eliminating the need for additional teacher models. This provides a new perspective on the distillation of SNNs from spiking neuron dynamics.\n\nThen, we analyze the novelty of our method and previous methods. As you pointed out, [1] also utilized the concept of submodels, but did not conduct an explicit in-depth analysis or list it as a contribution. In contrast, we explicitly pointed out that the difference between each submodel lies in the initial membrane potential and input current at each timestep, which leads to differences in the output spikes. Furthermore, from a methodological perspective, [1] requires comparing output and label information step by step, while our method only requires confidence scores, making it more efficient and suitable for use in scenarios without labels. In terms of performance, our method is consistently superior to [1] on both static and neuromorphic datasets.\n\nIn addition, the paper [2] you mentioned uses an early exit strategy in SNNs. In other words, it determines whether to terminate inference early based on confidence rather than executing all timesteps. This is similar to the early exit mechanism in temporal tasks [3], and [2] does not relate its method to submodels. It is worth noting that, since there is no explicit relationship, our self-distillation method is compatible with the early exit mechanism to further improve efficiency. We used the early exit mechanism on CIFAR10-DVS and DVS-Gesture, with exit thresholds set to {0.95, 0.9, 0.8, 0.5}. The results in the table below show that this further promotes our method to achieve ultra-low latency inference.\n\nIn summary, our work has two key advantages over previous studies:\n\n(1) **A detailed analysis of spiking neuron dynamics reveals submodels in the time dimension.**\n\n(2) **An efficient, general, and competitive self-distillation method for SNNs.**\n\n| Method| Timestep| CIFAR10-DVS | Timestep | DVS-Gesture |\n| :--| :--: | :--: | :--: |:--: |\n|Strong2Weak |5 | 78.6 |5 | 92.01 |\n|Early exit (0.95) | 3.16 | 78.4 | 2.35 | 91.32 |\n|Early exit (0.9) | 2.73 | 78.2 | 1.87 | 90.62 |\n|Early exit (0.8) | 2.30 | 77.8 | 1.56 | 90.28 |\n|Early exit (0.5) | 1.38 | 74.7 | 1.06 | 88.54 |\n|Weak2Strong|5 | 79.7 |5 | 91.67 |\n|Early exit (0.95) | 3.19 | 79.5 | 2.22 | 91.67 |\n|Early exit (0.9) | 2.74 | 79.4 | 1.89 | 90.62 |\n|Early exit (0.8) | 2.27 | 78.8 | 1.58 | 89.93 |\n|Early exit (0.5) | 1.38 | 75.8 | 1.01 | 86.11 |\n|Vanilla SNN | 5 | 73.97 | 5 | 87.85 |\n\n**Response to Weakness 2 (Eq. 1 and Eq. 2 are the foundation for the deconstruction of the spiking submodel)**: Eq. 1 and Eq. 2 describe the most fundamental unit in SNNs: the dynamics of spiking neurons. In Eq. 1, $H(t-1)$ refers to the residual membrane potential from the previous timestep $t-1$. This is also the initial membrane potential before charging at the current timestep $t$. $I(t)$ represents the input current at timestep $t$. Eq. 2 describes the change in membrane potential between adjacent timesteps based on the spike output $S(t)$. It is exactly because $H(t-1)$ and $I(t)$ in Eq. 1 continuously change that the spike output $S(t)$ in Eq. 2) for each timestep is different even if the architecture and parameters are shared. This simultaneous temporal correlation but difference allows these submodels to work together synergistically. Therefore, Eq. 1 and Eq. 2 are used to illustrate the dynamics of spiking neurons in order to support the temporal  deconstruction of SNNs.\n\n**Response to Weakness 3 (the effectiveness of strong-weak distillation)**: To address your concerns, we conducted the following two experiments: (1) comparing strong-weak distillation with random distillation, and (2) using additional metrics: entropy, confidence margin (abbreviated as margin) for the target category and other categories, and confidence difference (abbreviated as diversity) between all categories. We conducted three random seed experiments using the VGG-9 model on the CIFAR10-DVS and DVS-Gesture datasets at five timesteps. The average results are shown in the tables below. The experimental results show that:\n\n(1) Random distillation outperforms the vanilla SNN, even without deliberately selecting the teacher and student. We consider this method to randomly transition between Strong2Weak and Weak2Strong, achieving an effect similar to using both simultaneously. This makes our deconstruction-after-destillation scheme more flexible and versatile. However, distillation cannot be abused entirely between submodels. In response to Reviewer vxXV, we discovered that performing distillation always between the first and last timesteps leads to significant degradation.\n\n(2) Using confidence to determine teacher and student submodels for distillation consistently achieved better performance than random distillation. This shows that although confidence is straightforward, it can be used to determine the performance of a model relatively well (though not absolutely). In fact, previous research on empirical risk minimization has shown that confidence is usually correlated with the generalization error of a model, i.e., high confidence often corresponds to low generalization error [4]. In contrast, random distillation cannot guarantee optimization in the direction of either strong to weak (toward low generalization error) or vice versa (toward high generalization error regularization). This easily leads to a conflict between consistency and diversity, similar to suboptimal performance when using both simultaneously.\n\n(3) Using metrics other than confidence alone did not yield significantly better performance, but rather resulted in slight fluctuations. This once again demonstrates that confidence is a simple and reasonable metric that allows our methods to remain efficient and high-performing.\n\n| Distillation| CIFAR10-DVS | DVS-Gesture |\n| :--| :--: | :--: |\n| Strong2Weak | 78.93 | 91.43 |\n|  Weak2Strong | 79.33 | 91.20 |\n| Random-to-Random | 78.03 | 90.16 |\n| Vanilla SNN | 73.97 | 87.85 |\n\n| Metric | Distillation| CIFAR10-DVS | DVS-Gesture |\n| :--| :--: | :--: | :--: |\n| Confidence | Strong2Weak | 78.93 | 91.43 |\n|Entropy | Strong2Weak  | 78.80 | 90.97 |\n|Margin | Strong2Weak | 78.83 | 91.55 |\n|Diversity | Strong2Weak | 78.27 | 90.51 |\n|Confidence | Weak2Strong| 79.33 | 91.20 |\n|Entropy | Weak2Strong| 78.57 | 90.05 |\n|Margin | Weak2Strong| 78.57 | 90.97 |\n|Diversity | Weak2Strong| 78.73 | 91.78 |\n\n**Response to Question (the role of dark knowledge)**: Dark knowledge refers to details that weak models can capture, but that strong models often overlook. For example, for a picture of a dog, a strong model may judge the confidence of the dog to be 0.95, while the confidence of other categories is extremely low, which can lead to overfitting. A weaker model would have slightly lower confidence in the dog (e.g., 0.7) and slightly higher confidence in the cat associated with the dog than in other unrelated categories (e.g., 0.25). At this point, dark knowledge carries the relationships between similar categories, thereby enabling the model to capture these relationships to alleviate overfitting and promote generalization [5]. In fact, current SNNs suffer from serious overfitting problems. For example, the training accuracy on CIFAR10-DVS is close to 100%, while the testing accuracy is only 70-85% [1,2,6,7], which allows dark knowledge to play a more significant role. Furthermore, reducing overfitting and promoting generalization makes the model less sensitive to potential disturbances, improving its overall stability and adversarial robustness [8]. We present the results of our experiments on adversarial robustness in Table 7 of the paper. It is obvious that Weak2Strong exhibits better robustness, i.e., retains more performance, against severe attacks such as FGSM, PGD, BIM, and CW. We believe this proves the superiority of dark knowledge in promoting generalization and robustness.\n\n**We hope that these responses adequately explain the innovation and significance of this paper and address your concerns. If you have any further questions, please reply to us for further discussion.**\n\n```\n[1] Temporal knowledge sharing enable spiking neural network learning from past and future. IEEE Transactions on Artificial Intelligence. 2024.\n[2] Towards temporal spiking early exit neural networks. NeurIPS. 2023.\n[3] Frameexit: Conditional early exiting for efficient video recognition. CVPR. 2021.\n[4] Constrained Empirical Risk Minimization Framework for Distance Metric Learning. IEEE TNNLS. 2012.\n[5] Revisiting knowledge distillation via label smoothing regularization. CVPR. 2020.\n[6] Qkformer: Hierarchical spiking transformer using q-k attention. NeurIPS. 2024.\n[7] Rethinking spiking neural networks from an ensemble learning perspective. ICLR. 2025.\n[8] Improving robust generalization by direct pac-bayesian bound minimization. CVPR. 2023.\n```"}, {"review_id": "9c2a3MO8cd", "reviewer": "Reviewer_QXfW", "summary": "The authors address the persistent challenge in Spiking Neural Networks (SNNs)—the performance gap compared to traditional Artificial Neural Networks (ANNs)—by introducing a novel, overhead-free Temporal Self-Distillation (TSD) approach. Unlike previous distillation methods that rely on external teachers or auxiliary modules, TSD ingeniously exploits the inherent temporal variability of the SNN itself by treating intermediate timesteps as individual submodels, dynamically selecting high-confidence (strong) and low-confidence (weak) submodels for mutual knowledge distillation. A notable insight is the bidirectional distillation (Strong2Weak and Weak2Strong), enabling more consistent predictions across timesteps, thus significantly enhancing low-latency inference and robustness without additional computational overhead. Empirical validation across diverse datasets  demonstrates non-trivial accuracy improvements and robustness gains.", "questions": "Your method selects \"strong\" and \"weak\" submodels based solely on prediction confidence. Could you provide deeper insight into why confidence alone is the optimal indicator for knowledge quality? Have you considered alternatives, such as entropy-based measures or diversity-aware metrics?\nSuggestion: Explicitly compare confidence-based selection with alternative metrics (e.g., entropy, margin, diversity) through a brief ablation experiment or theoretical analysis.\n\nYour simultaneous Strong2Weak and Weak2Strong distillation assumes an inherent beneficial balance between similarity and diversity among temporal submodels. How sensitive is the method to the similarity-diversity trade-off? Under what conditions does similarity become detrimental rather than beneficial?\nSuggestion: Present a brief analysis or theoretical discussion of the ideal similarity-diversity balance in temporal submodels, possibly suggesting guidelines or criteria to manage this trade-off.\n\nYou claim improvements in robustness based on adversarial evaluations. Could you clarify whether these gains stem primarily from temporal regularization or from the specific knowledge-distillation procedure itself?\nSuggestion: Perform a targeted experiment or provide deeper theoretical reasoning to isolate and explain the source of improved robustness. Also, discuss potential limitations in generalizing this method beyond the tested adversarial scenarios.", "limitations": "No.\nThe authors have addressed methodological limitations clearly but have not explicitly discussed potential negative societal impacts or ethical implications of their work, such as unintended uses of enhanced low-latency SNN models in sensitive applications (e.g., surveillance, autonomous systems). To strengthen the paper further, it would be beneficial to briefly acknowledge potential ethical considerations, especially the risks associated with deploying highly efficient and robust SNNs in critical or sensitive contexts (like in military robotics).", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 4, "originality": 4, "strengths_and_weaknesses": "The paper's major strengths lie in its elegant simplicity, impressive experimental breadth, and substantial performance uplifts, especially at reduced inference timesteps. The primary strength of the Temporal Self-Distillation (TSD) method lies in its innovative leveraging of intrinsic temporal dynamics within SNNs, and presenting an underexplored mechanism for internal knowledge transfer. Rather than introducing external supervision, the method exploits the inherent variance across timestep predictions, addressing both stability and robustness from first principles. This assumption—that intermediate temporal submodels inherently contain valuable but overlooked knowledge—proves valid, significantly enhancing both accuracy and robustness under adversarial perturbations and low-latency conditions, well-supported by extensive experiments.\nOther strengths:\nEvidence for the claim of enhanced adversarial robustness due to narrowed output distributions across timesteps is convincingly substantiated through experiments involving multiple attack methods (FGSM, PGD), adding valuable insights into robustness dynamics.\nEmpirical results strongly back the authors' claims about generalizability, demonstrating substantial improvements across diverse datasets, architectures, and neuron types, reflecting genuine versatility.\n\nWeaknesses:\nThe paper's assumptions about optimality in selecting \"strong\" and \"weak\" submodels based solely on confidence may overlook more complex interactions within the temporal dynamics of SNNs. The implicit assumption—that confident outputs are always the most informative teachers—might oversimplify scenarios where suboptimal, but complementary submodels hold richer ensemble knowledge. Further, while experiments broadly support the approach, the lack of exploration into the dynamics of hyperparameters, particularly temperature scaling and distillation weights, introduces uncertainty about whether reported improvements represent near-optimal or merely satisfactory configurations.\nOverly simplistic confidence-based teacher-student selection: The mathematical model assumes high confidence as a direct indicator of superior knowledge, neglecting potential informative trade-offs from less confident yet more diverse submodels.\nNeglected ensemble dynamics: The interplay of submodel similarity versus diversity remains insufficiently addressed mathematically and experimentally, limiting understanding of ensemble distillation effects and potentially overlooking richer interactions within the temporal model.", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank you for your efforts in the review process, and we especially appreciate your recognition of the novelty, simplicity, effectiveness, and versatility of our method. Below, we will respond point by point to the weaknesses and questions you have pointed out, and we hope to address your concerns:\n\n**The metric for identifying the strong and the weak**: We use confidence as the default evaluation metric to distinguish between strong and weak submodels because it can be efficiently calculated using only a softmax function. In addition, confidence typically reflects the uncertainty of model output results and has been shown to be related to generalization error, which to a certain extent reflects model performance [1]. Due to these advantages, confidence is widely used to evaluate model performance in areas such as model calibration [2,3] and early exit [4,5].\n\nAs a general distillation method, we can seamlessly replace confidence with other metrics such as entropy, margin (maximum confidence difference between target category and other categories), diversity (confidence differences between all categories), or other specially designed metrics. To this end, the table below presents the distillation performance of confidence in comparison with other metrics. The results show that **performance has not significantly improved, suggesting that straightforward confidence is effective enough**. In fact, to further explore the diverse information in multiple submodels, we can apply cascaded distillation. Table 3 of the paper shows that cascade distillation always outperforms single distillation. This suggests that considering diversified information is worth further exploration.\n\n| Metric | Distillation| CIFAR10-DVS | DVS-Gesture |\n| :--:| :--: | :--: | :--: |\n| Confidence | Strong2Weak | 78.93 | 91.43 |\n|Entropy | Strong2Weak  | 78.80 | 90.97 |\n|Margin | Strong2Weak | 78.83 | 91.55 |\n|Diversity | Strong2Weak | 78.27 | 90.51 |\n|Confidence | Weak2Strong| 79.33 | 91.20 |\n|Entropy | Weak2Strong| 78.57 | 90.05 |\n|Margin | Weak2Strong| 78.57 | 90.97 |\n|Diversity | Weak2Strong| 78.73 | 91.78 |\n\n**The influence of hyperparameters on performance**: The table below shows the performance of varying distillation temperatures and distillation weighting coefficients. The results indicate that **adjusting these hyperparameters within reasonable limits results in slight performance fluctuations without significant degradation**. This also demonstrates that **satisfactory performance can be achieved without deliberately adjusting the hyperparameters**. However, when the temperature or distillation weight falls outside the normal range (e.g., a temperature of 0.5 or a weight of 0.1 or 2.0), the performance of our method deteriorates. Nevertheless, it consistently outperforms vanilla SNNs.\n\n| Temperature| Distillation| CIFAR10-DVS | DVS-Gesture |\n| :--:| :--: | :--: | :--: |\n| 0.5| Strong2Weak |75.30 | 91.09 |\n|1.0| Strong2Weak  |77.63 | 92.13 |\n|2.0| Strong2Weak | 78.93 | 91.43 |\n|3.0| Strong2Weak | 78.27 | 90.86 |\n|5.0| Strong2Weak | 78.75 | 92.01 |\n| 0.5| Weak2Wtrong|76.77 | 91.32 |\n|1.0| Weak2Wtrong| 78.43 | 91.44 |\n|2.0| Weak2Wtrong| 79.33 | 91.20 |\n|3.0| Weak2Wtrong| 78.43 | 91.21 |\n|5.0| Weak2Wtrong| 79.03 | 91.67 |\n\n**Balancing similarity and diversity**: When distilling with both Stong2Weak and Weak2Strong, there was no significant improvement in performance compared to using either one alone. We attribute this to the fact that simultaneous distillation limits the overall diversity of the SNN. Next, we analyze the balance between similarity (overall stability) and diversity.\n\nWe can view the entire SNN as an ensemble by treating each submodel as a member. According to ensemble learning theory, increasing the diversity among members appropriately can promote model generalization [6,7]. This is usually done by jointly optimizing the original objective and diversity loss during training. However, existing methods still rely on manually defined coefficients to strike a balance between overall stability and diversity [7,8]. The lack of mature and feasible theoretical research has made this an unresolved challenge for the ensemble learning community [9].\n\nOur method aims to promote similarity (overall stability) between spiking submodels. The experimental results show that the single distillation scheme and the combination of the two schemes both significantly outperform vanilla SNNs. This indicates that promoting similarity in SNNs yields gains rather than degradation. Currently, however, there are no relevant studies that can accurately or even approximately quantify the threshold between similarity and diversity. This makes it difficult to determine when it is appropriate to increase similarity or enhance diversity. To further optimize the balance between the two, we can explore adjusting the optimization weights between the original object, similarity, and diversity. By changing our perspective and treating each loss as a task, we can adaptively adjust the weights and gradient ratios using multi-task learning theory [10,11]. This may allow us to identify a Pareto set that balances the overall stability and diversity of SNNs. We will explore the definition of this balance in our future work.\n\n**Robustness gains**. Our method guides the student submodel using the output of the teacher submodel as soft labels, which can simultaneously achieve distillation and regularization effects to promote better consistency and generalization. To investigate the source of the gains in adversarial robustness, we compared our method with those of typical label smoothing and entropy regularization (with the same settings as Table 7 in the paper, VGG-11 with 8 timesteps was used on CIFAR100). We found that using regularization methods alone can achieve decent performance on clean samples. However, **pure regularization methods severely degrade and are significantly inferior to our method when faced with adversarial attacks**. Therefore, we consider the source of the SNN's robustness to be the distillation that promotes internal consistency, making it immune to interference. However, it should be emphasized that regularization also plays a role to a certain extent, just as label smoothing can also achieve decent robustness.\n\n| Metric | Clean| GN | FGSM | PGD | BIM | CW |\n| :--:| :--: | :--: | :--: |  :--: | :--: | :--: |\n| Label smoothing| 69.92| 68.52 | 18.29 | 8.42 | 7.89 | 23.65 |\n|Entropy regularization| 70.05 | 69.11 | 17.89 | 8.17 | 7.38 | 18.88 |\n|Strong2Weak | **70.68** | **69.53** | 21.15 | 10.30 | 9.40 | 23.40 |\n|Weak2Strong| 70.09 | 68.87 | **21.23** | **10.70** | **9.75** | **24.08**|\n\nLimitations in other adversarial scenarios: Since adversarial robustness methods for SNNs are still in their infancy, existing SNNs struggle to maintain satisfactory performance on large-scale tasks, such as ImageNet. Therefore, the robustness of our method on large-scale and difficult downstream tasks remains to be explored.\n\n**About limitations**: Thank you for pointing out the potential risks of deploying SNNs, although our method is designed to promote low-latency, high-performance SNNs rather than their malicious use. In fact, any efficient and high-performance model faces the risk of abuse, such as some lightweight ANN models. To avoid this, we believe it is necessary to integrate attack-defense strategies, security measures, and privacy protection measures further into the method. Alternatively, privacy protection mechanisms such as federated learning should be incorporated during deployment. Thank you again for pointing this out, and we will include this content in the revised version.\n\n**We hope that the above response has addressed your concerns. If you have any further questions, please reply to us for further discussion.**\n```\n[1] Constrained Empirical Risk Minimization Framework for Distance Metric Learning. IEEE TNNLS. 2012.\n[2] On calibration of modern neural networks. ICML. 2017.\n[3] Mitigating neural network overconfidence with logit normalization. ICML. 2022.\n[4] Resolution adaptive networks for efficient inference. CVPR. 2020.\n[5] Seenn: Towards temporal spiking early exit neural networks. NeurIPS. 2023.\n[6] Neural network ensembles, cross validation, and active learning. NIPS. 1994.\n[7] The diversified ensemble neural network. NeurIPS. 2020.\n[8] Maximizing overall diversity for improved uncertainty estimates in deep ensembles. AAAI. 2020.\n[9] A Unified theory of diversity in ensemble learning. JMLR. 2023.\n[10] Pareto Multi-Task Learning. NeurIPS. 2019.\n[11] Aligned Multi Objective Optimization. ICML. 2025.\n```"}, {"review_id": "plpgaDj3tM", "reviewer": "Reviewer_5Zyo", "summary": "This paper addresses the challenge of enhancing Spiking Neural Networks (SNNs) performance via efficient self-distillation, avoiding the reliance on large teacher models or extra training overhead typically seen in conventional knowledge distillation methods. The authors propose a novel perspective: decomposing an SNN across timesteps into multiple submodels, enabling internal distillation. Based on this, they introduce two schemes: Strong2Weak (S2W) and Weak2Strong (W2S). Both methods aim to improve performance without changing the architecture or requiring external supervision.", "questions": "Line 176: What is o_t? Is it a spike or a continuous variable? If it's a spike, how is SoftMax computed from binary output?\n\nFor the Strong2Weak scheme, could simply increasing the timestep of the stronger model further improve performance? If not, why not adopt early stopping or dynamic confidence-based inference as done in [1]?\n\n[1] Chen Li, Edward Jones, Steve Furber. Unleashing the Potential of Spiking Neural Networks by Dynamic Confidence. https://arxiv.org/abs/2303.10276", "limitations": "The performance boost is not substantial, especially when both schemes are combined—this may limit practical utility.\n\nThere is no analysis of computational overhead or memory usage when applying S2W and W2S during training.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n- Viewing timestep instances of an SNN as a set of submodels offers a fresh angle on internal distillation and may inspire future research.\n\n- The approach does not introduce significant architectural changes or external training cost.\n\n- The paper is clearly structured, with well-defined motivation, methodology, and results.\n\nWeaknesses:\n\nThe improvements are relatively modest. As noted by the authors, using both S2W and W2S simultaneously does not yield further gains due to reduced submodel diversity.\n\nFigure 1 and 2 use color schemes that are too light, making interpretation difficult.\n\nFigure 3 has an overly small subtitle font, affecting readability.", "was_revised": true, "final_justification": "I keep my score. My concerns are largely solved.", "rebuttal": "We sincerely appreciate your valuable feedback and recognition of the new perspectives our work has introduced, as well as its potential to inspire future work. Below, we will respond to your questions point by point to address your concerns:\n\n**Performance gains of our method**. Although using both distillation schemes simultaneously did not further improve performance, we would like to emphasize that using one scheme alone can already yield significant gains. For instance, on CIFAR10-DVS, performance increased by 5.36% with five timesteps. Additionally, performance improved from 10% in the collapsed state to 73.4% with one timestep. Simultaneous distillation tends to cause conflicts between consistency and diversity, and how to achieve a balance between the two has remained an unsolved problem in ensemble learning theory [1,2]. Although grid search distillation weights may be able to find the optimal simultaneous distillation performance, our goal is to propose a novel, efficient self-distillation framework. Therefore, rather than using grid search, we simply set the distillation weight to 1.0. In fact, we experimented with different distillation weights and discovered that using both weights at 0.1 resulted in a 1% improvement in DVS-Gesture performance compared to using them separately. In the future, we will conduct in-depth research on the influence of consistency and diversity in SNNs. We will also explore adaptive distillation and balancing strategies to maximize their performance. Overall, our method can efficiently and significantly improve the performance of the baseline SNN, and its performance potential will be further unleashed by balancing consistency and diversity.\n\n**Combined with dynamic inference method**. In the Strong2Weak scheme, the strong model corresponds to one of the timesteps (e.g., $t\\_1$) in the SNN. Therefore, the distilled teacher output at this point is the output at time $t\\_1$. Increasing the timestep of the SNN may cause other timesteps to be considered stronger than $t\\_1$, but this will not affect the output at $t\\_1$. Therefore, increasing the timestep alone does not necessarily improve performance. This also suggests that our method can be used with dynamic inference methods. To this end, we applied early exit inference to our trained model, setting the exit threshold to {0.95,0.9,0.8,0.5}. As shown in the table below, **our method is highly compatible with the early exit mechanism, further enhancing the performance of ultra-low latency inference**. We will include this experiment in the revised version.\n\n| Method| Timestep| CIFAR10-DVS | Timestep | DVS-Gesture |\n| :--| :--: | :--: | :--: |:--: |\n|Strong2Weak |5 | 78.6 |5 | 92.01 |\n|Early exit (0.95) | 3.16 | 78.4 | 2.35 | 91.32 |\n|Early exit (0.9) | 2.73 | 78.2 | 1.87 | 90.62 |\n|Early exit (0.8) | 2.30 | 77.8 | 1.56 | 90.28 |\n|Early exit (0.5) | 1.38 | 74.7 | 1.06 | 88.54 |\n|Weak2Strong|5 | 79.7 |5 | 91.67 |\n|Early exit (0.95) | 3.19 | 79.5 | 2.22 | 91.67 |\n|Early exit (0.9) | 2.74 | 79.4 | 1.89 | 90.62 |\n|Early exit (0.8) | 2.27 | 78.8 | 1.58 | 89.93 |\n|Early exit (0.5) | 1.38 | 75.8 | 1.01 | 86.11 |\n|Vanilla SNN | 5 | 73.97 | 5 | 87.85 |\n\n**Our method requires only negligible computational overhead**. Since our method uses efficient confidence to identify the strongest and weakest submodels, it only requires the following additional steps compared to the standard SNN training process:\n1. Apply the softmax function over T timesteps;\n2. Find the maximum and minimum indices of the softmax results (the strongest and weakest);\n3. Calculate the KL divergence between the strongest and weakest.\n\nThis requires minimal training overhead and can be implemented with just a few lines of PyTorch code, as shown in Algorithm 1 of the paper. Given this simplicity, we did not observe any significant changes in GPU memory usage or training time during training.\n\n**The output of the SNN**. We use $o(t)$ to represent the output of the SNN at timestep t. Although SNNs transmit binary spikes in the intermediate layers, their output is a continuous membrane potential, which is achieved by the output layer receiving inputs without firing spikes. Currently, this is the most commonly used output strategy in SNNs [3,4]. Therefore, the softmax function can be directly applied to the output of the SNN. Thank you for pointing this out, we will describe this in the revised version.\n\n**Font color and size in Figures 1, 2, and 3**: Thank you for pointing out the font color and size issues in the figures in the paper. We will modify the font color in Figures 1 and 2 and enlarge the font size of the subtitle in Figure 3. This will contribute to understanding our method and visualization.\n\n**We hope these responses have addressed your concerns. If you have any further questions or concerns, please reply to us.**\n\n```\n[1] Neural network ensembles, cross validation, and active learning. NIPS. 1994.\n[2] The diversified ensemble neural network. NeurIPS. 2020.\n[3] Qkformer: Hierarchical spiking transformer using q-k attention. NeurIPS. 2024.\n[4] Rethinking spiking neural networks from an ensemble learning perspective. ICLR. 2025.\n```"}, {"review_id": "odDFNXiYI9", "reviewer": "Reviewer_vxXV", "summary": "This paper introduces two innovative self-distillation schemes - Strong2Weak and Weak2Strong - designed specifically for spiking neural network (SNN) training. The key innovation lies in leveraging the varying confidence levels across different timestep instances of SNNs. The authors provide comprehensive experimental validation across multiple benchmarks, demonstrating that their approach: (1) improves overall model performance, (2) achieves superior results in low timestep inference scenarios, and (3) enhances model robustness. The effectiveness of the method is further corroborated through t-SNE visualizations of the learned representations.", "questions": "- Line 279 contains a typographical error where \".5In Table 4\" should be corrected to \"In Table 4\".\n- The relationship between weak and strong submodels raises several questions. First, since both submodels share the same parameter group and are trained jointly, one would expect $L_{W2S}$ and $L_{S2W}$ to yield similar results when using symmetric distance functions like MSE. The observed performance gap in Table 2 (MSE, DVS-Gesture) therefore requires clarification. Additionally, the distinction between \"The Strong Help the Weak\" and \"The Weak Enhance the Strong\" may be artificial, as both processes occur simultaneously during self-distillation. The paper would benefit from explaining why treating them separately provides advantages over a unified approach.\n- Does the weak submodel consistently correspond to early timesteps (e.g., t=1) while the strong submodel represents later ones (e.g., t=T)?", "limitations": "yes, as discussed in the conclusion section.", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n- The core concept of leveraging confidence levels for distillation represents a novel and compelling approach. The paper convincingly demonstrates its effectiveness through significant performance improvements across multiple tasks.\n- The experimental evaluation is particularly thorough, offering rigorous validation of the method. The results provide important insights into SNN capabilities, especially under constrained timestep conditions.\n\n**Weaknesses**\n- Several methodological aspects require clarification in the main text, particularly regarding the criteria for identifying strong and weak submodels - specifically whether binary spike outputs or membrane potentials are used, and whether selection occurs across the entire dataset or within individual batches. \n- The theoretical foundation could be strengthened through deeper analysis of why the distillation approach improves performance, particularly by examining the relationship between confidence levels and generalization error.", "was_revised": true, "final_justification": "I have read all the discussion. Apart from the fact that the analysis explaining why their distillation approach works is not entirely rigorous, the other issues have been successfully addressed. Overall, I believe the paper deserves to be accepted.", "rebuttal": "We greatly appreciate your valuable comments, and we especially appreciate your recognition of the novelty and effectiveness of our method. We will respond to your questions point by point below:\n\n**Details on identifying strong and weak submodels**: The SNN outputs continuous membrane potential rather than binary spikes when identifying strong and weak submodels, so the softmax function can be used to calculate confidence. Additionally, confidence evaluation and selection of strong and weak submodels are performed on a batch-by-batch basis because this has almost no impact on training overhead. Calculating across the entire dataset, however, would require additional forward propagation to store global confidence. These details will be added in the revised version.\n\n**How can knowledge distillation improve performance**? On the one hand, according to the Vapnik-Chervonenkis Dimension theory and Rademacher complexity, the generalization error of a model usually stems from excessive complexity in the hypothesis space or insufficient data fit [1]. Knowledge distillation reduces the complexity of the hypothesis space by limiting the hypothesis space of the student model to the neighborhood of the teacher model through encouraging the student model to imitate the teacher model, thereby reducing the risk of overfitting and improving generalization ability [2]. On the other hand, we select strong and weak submodels for distillation based on their level of confidence. In the empirical risk minimization framework, high confidence typically corresponds to lower generalization error, and vice versa [3]. This makes Strong2Weak distillation effective in restricting the hypothesis space of the weak submodel to the region of low generalization error. In Weak2Strong distillation, the weak submodel acts as a form of regularization, prompting the strong submodel to learn generalized representations that include associations between different categories. For instance, if the label indicates that the object is a dog, the weak submodel's output may suggest that the image is of a cat. This encourages the strong submodel to focus on the similarities between dogs and cats.\n\n**Symmetrical MSE loss produces unequal results**. To prevent negative impacts on the performance of the teacher submodel during distillation, we apply a gradient stopping strategy to the output of the teacher model when using the symmetric MSE function as the distillation loss. This is a common strategy in knowledge distillation and representation learning; otherwise, it is easy for the model to collapse [4,5]. Therefore, even when using symmetric MSE, different performance results may occur due to differences in gradient updates. We will explain this in the revised version.\n\n**Comparison between separate distillation and simultaneous distillation**. During training, Strong2Weak distillation improves the performance of submodels with low-confidence timesteps, thereby raising the \"short boards in the barrel.\" Conversely, Weak2Strong distillation can regularize overly strong submodels by leveraging dark knowledge, thereby promoting generalization. In other words, both promote consistency among all submodels. However, a problem arises when both are used simultaneously: the conflict between similarity and diversity. This resulted in the simultaneous use of both not producing significantly better performance than either one alone. Similar issues also arise in ensemble learning, where predefined diversity metrics and loss weights are used to train the model in order to balance the diversity of the ensemble [6,7]. To avoid conflicts between similarity and diversity, as well as cumbersome weight adjustments, we prioritize similarity to improve SNN performance. Using only one of them has achieved significant gains, as shown by experiments. However, in the future, we will continue to explore methods to balance the similarity and diversity of SNN submodels to further improve performance.\n\n**The relationship between the strong-weak submodel and the timestep**. During training, we examined the timesteps corresponding to the strong and weak submodels and found that **the weak submodel was not always located at the earliest timestep and that the strong model was not always the last**. In fact, the weak submodel appears at every timestep. However, it appears most frequently at the earliest timestep and less frequently at other timesteps. This is consistent with our experimental results: the vanilla SNN performs worst at the earliest timestep, while our method significantly improves the performance of the first timestep, so it is not always the weakest. The strong submodel is less likely to be selected in the first timestep, but is typically selected for subsequent timesteps. This demonstrates that our method of evaluating submodels based on confidence and dynamically selecting timesteps for distillation is capable of adapting to the training of SNNs to a certain extent. Otherwise, it would degrade to distillation with specified timesteps.\n\nAdditionally, we compare distillation with specified timesteps. We construct Last2First and First2Last distillation schemes under the assumption that the first timestep is the weakest and the last timestep is the strongest. The comparative results are shown in the table below. The results show that specifying these two timestep distillations is much less effective than our confidence selection distillation. This highlights the importance of dynamically selecting teacher and student timesteps.\n\n| Method| CIFAR10-DVS | DVS-Gesture |\n| :--:| :--: | :--: |\n|Strong2Weak | 78.93 | 91.43 |\n|Weak2Strong| 79.33 | 91.20 |\n|Last2First| 74.75 | 89.12 |\n|First2Last| 74.29| 88.89 |\n\n**Typo correction**: We will correct the typos you pointed out and carefully review the revised version. Thank you for pointing that out.\n\n**We hope these responses address your concerns. If you have any further questions, please discuss them with us.**\n\n```\n[1] A deep connection between the Vapnik–Chervonenkis entropy and the Rademacher complexity. IEEE TNNLS. 2014.\n[2] A statistical perspective on distillation. ICML. 2021.\n[3] Constrained Empirical Risk Minimization Framework for Distance Metric Learning. IEEE TNNLS. 2012.\n[4] Decoupled kullback-leibler divergence loss. NeurIPS. 2024.\n[5] Exploring simple siamese representation learning. CVPR. 2021.\n[6] The diversified ensemble neural network. NeurIPS. 2020.\n[7] Maximizing overall diversity for improved uncertainty estimates in deep ensembles. AAAI. 2020.\n```"}]

Author rebuttal:

Meta-review: The paper proposes a temporal self-distillation view of SNNs: each inference timestep is treated as a submodel with its own confidence; “strong” (high confidence) timesteps can teach “weak” (low confidence) timesteps, and conversely weak can regularize strong. This yields more consistent per-timestep predictions, better low-timestep performance, and improved robustness without auxiliary teachers or architectural changes.

Strength
+The method reuses existing temporal predictions; rebuttal emphasizes essentially three light steps (confidence via softmax, pick max/min, apply KL/MSE), and reviewers acknowledge it as “overhead-free”/“elegant.”
+Thorough experiments across multiple benchmarks/backbones; improvements are most pronounced in low-timestep regimes that matter for latency-sensitive SNNs.
+Both reviewers and authors highlight improved adversarial/perturbation robustness attributable to self-distillation’s stabilizing effect.

Weakness
-Multiple reviewers asked for a stronger explanation of why confidence-guided temporal distillation improves generalization; the paper is empirically convincing but analytically weaker.
-Reviewers sought crisper specification of whether spikes or membrane potentials are used, and whether selection is dataset-wide or batch-wise. Some of this was clarified, but tightening in the main text would help.
-One reviewer felt uplifts, while consistent, were not large across the board; combined S2W+W2S may reduce diversity and limit benefit.
-A reviewer flagged the absence of a quantitative training-time/VRAM analysis. The authors assert negligible overhead, but numbers and wall-clock profiles would strengthen the case.

Recommendation (Accept Poster)
1) Practical impact for low-latency SNNs via improved few-timestep accuracy and compatibility with early exit (measurable latency reductions at modest accuracy cost)
2) Consensus among reviewers leans positive (one strong accept, one good accept, one borderline-accept maintaining score after rebuttal); concerns are largely addressed.

Summary after rebuttal: The added ablations (random-to-random vs confidence; alternative metrics), early-exit results, and clarifications address the most substantive concerns. While a deeper theory and a short resource-overhead table would further improve the paper, the simplicity, breadth, and practical relevance justify accept (poster).
Decision: Accept (poster)