Efficient Logit-based Knowledge Distillation of Deep Spiking Neural Networks for Full-Range Timestep Deployment
Authors: Chengting Yu
Xiaochen Zhao
Lei Liu
Shu Yang
Gaoang Wang
Erping Li
Aili Wang
Venue: ICML 2025 poster (2025)
Keywords: Spiking Neural Networks
Knowledge Distillation
Image Classification
Backpropagation
Primary area: deep_learning->everything_else
TL;DR: Considering the spatiotemporal characteristics of Spiking Neural Networks, we optimize logit-based knowledge distillation in deep SNNs training. The method is neat and effective, and we provide both theoretical and empirical analysis as evidence.

Abstract: Spiking Neural Networks (SNNs) are emerging as a brain-inspired alternative to traditional Artificial Neural Networks (ANNs), prized for their potential energy efficiency on neuromorphic hardware. Despite this, SNNs often suffer from accuracy degradation compared to ANNs and face deployment challenges due to fixed inference timesteps, which require retraining for adjustments, limiting operational flexibility. To address these issues, our work considers the spatio-temporal property inherent in SNNs, and proposes a novel distillation framework for deep SNNs that optimizes performance across full-range timesteps without specific retraining, enhancing both efficacy and deployment adaptability. We provide both theoretical analysis and empirical validations to illustrate that training guarantees the convergence of all implicit models across full-range timesteps. Experimental results on CIFAR-10, CIFAR-100, CIFAR10-DVS, and ImageNet demonstrate state-of-the-art performance among distillation-based SNNs training methods. Our code is available at https://github.com/Intelli-Chip-Lab/snn_temporal_decoupling_distillation.

Efficient Logit-based Knowledge Distillation of Deep Spiking Neural Networks for Full-Range Timestep Deployment
Chengting Yu 12f Xiaochen Zhao 21
Lei Liu?
Shu Yang? 2 Gaoang Wang²
Erping Li 12 Aili Wang | 1
Abstract
(a) SNNs by Standard Lagits- based KD
(b) SNNS by Temporal wise Distillation
BPN rousterenn
Spiking Neural Networks (SNNs) are emerging as a brain-inspired alternative to traditional Arti- ficial Neural Networks (ANNs), prized for their potential energy efficiency on neuromorphic hard- ware. Despite this, SNNs often suffer from accu- racy degradation compared to ANNS and face deployment challenges due fixed inference timesteps, which require retraining for adjust- ments, limiting operational flexibility. To ad- dress these issues, our work considers the spatio- temporal property inherent in SNNs, and pro- poses a novel distillation framework for deep SNNs that optimizes performance across full- range timesteps without specific retraining, en- hancing both efficacy and deployment adaptabil- ity. We provide both theoretical analysis and em- pirical validations to illustrate that training guaran- tees the convergence of all implicit models across full-range timesteps. Experimental results on CIFAR-10, CIFAR-100, 100, CIFAR1O-DVS, and Im- ageNet demonstrate state-of-the-art performance among distillation-basee based SNNs training methods.
Parformance
Ausurary (M)
Optimal Across
2
Local Optima Near Training Timestaps
All Timestops
-niiiesssss - Inirddddst
with 7
with T
Iniradadd T
Inference Timestops
Inference Timesteps
Figure 1. Illustration of the primary challenges and motiva- tions. (a) Standard logits-hasets hased knowledd knowledge distillation (logits-KD) KD) training suffers from large accuracy degradation and requires dif- ferent models to adapt to various inference timestep settings. (b) The proposed distillation framework reduces the gap and ensures single model for full-range timesteps.
erations generally required for synaptic connections (Roy et al., 2019), boosting both energy efficiency and speed of inference on neuromorphic hardware (Akopyan et al., 2015; Davies et al., 2018; Pei et al., 2019). In essence, SNNs can be regarded as a type of quantized model that uses binary transmission, offering the potential for low power consumption and reduced latency when implemented on neuromorphic devices (Akopyan et al., 2015; Davies et al., 2018: 8; Pei et al., 2019).
Our code is available at http://gituub.com/nnteee Chip-Lab/snn temporal decoupling distillation.
Although SNNs exhibit considerable potential, their prac- tical application is hindered by the non non-differentiabiliy of spike activity (Zuo et al., 2024; Deng et al., 2023), coupled with the limited expressiveness of binary spike feature maps (Qiu et al., 2024a). Together, these challenges result in ac- curacy degradation when compared full-precision ANNs (Xu et al., 2023b; Zuo et al., 2024; Deng et al., 2023; Guo et al., 2023a; Hu et al., 2024b). Besides, when deploying SNNs on neuromorphic hardware, non-negligible chal- lenge is that the inference timesteps of the models are fixed, aligning with those utilized during training to optimize per- formance. Altering inference timesteps based on specific needs requires retraining the models for new timesteps (Fig, la), which restricts deployment flexibility and affects opera- tional adaptability in practical settings.
1. Introduction
Spiking Neural Networks (SNNs) are modeled after biologi- cal neural systems and feature spiking neurons that replicate the dynamics of biological neurons (Maass, 1997; Roy et al., 2019). In contrast Artificial Neural Networks (ANNs), which utilize continuous data forms, SNNs employ spike- coding approach, using discrete binary spike trains for data transmission (Panzeri & Schultz, 2001). This binary sig- naling significantly reduces the multiply-acummulate op-
Equal contribution. 'College of Information Science and Electronic Engineering, Zhejiang University ZZJU-UIUC Insti- tute. Zhejiang University. Correspondence to: Aili Wang <aili- wang@intl. www.........u..n......
As the common strategy for model lightweighting, Knowl- edge Distillation (KD) (Hinton, 2015; Gou et al., 2021) has been increasingly applied to training SNNs (Xu et al., 2023b; Hong et al., 2023; Guo et al., 2023a; Qiu al.,
Proceedings of the 4nnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Efficient Logit-based Knowledge Distillation of Deep Spiking Neural Networks for Full-Range Timestep Deployment
2024a). The KD-based SNNs training leverages rich in-
follows:
formation from ANN teacher train student SNN, achieving promising results on benchmark vision datasets such as CIFARI0/100 and ImageNet with ENN-based mod- els (Xu et al., 2023b; Hong et al., 2023; Deng et al., 2023; Guo et al., 2023a; Xu et al., 2024). However, current SNNs distillation methods primarily adopt strategies from ANNs, sticking to an end-to-end framework that utilizes the SNN's ensemble outputs averaged feature maps as distillation targets (Xu et al., 2023a; Zuo et al., 2024; Xu et al., 2023b; Hong et al., 2023; Qiu et al., 2024a). Given the unique spatio-temporal characteristics of SNNs, there is a pressing need to develop distillation approaches that more effectively leverage the distinct properties. Recent studies have demon- strated improved model convergence by isolating the truth label objectives to operate independently at each timestep (Deng et al., 2022; Xiao et al., 2022; Meng al., 2023; Zhu et al., 2023). Drawing inspiration from these techniques that utilize temporally decoupled objectives, we recognize that fully leveraging the spatio-temporal characteristics inher- ent in SNNs through decoupling overall voting objectives crucial to unlocking the full potential of SNNs distilla- tion. Furthermore, inspired by self-distillation approaches (Zhang et al., 2019), recent works have shown the effec- tiveness of using additional branches based on the SNN backbone to generate extra logits for distillation (Zuo et al., 2024; Deng et al., 2023). From the perspective of ensemble learning (Allen-Zhu Zhu & Li, 2020; Wang & Yoon, 2021; Ding et al.). we further exploit the temporal properties of SNNs by considering the final voting outputs as an integration of temporal outputs across time. We recognize that the final ensemble logits can serve soft labels for self-distillation, acting as a regularization mechanism to guide model conver- gence without additional computational branches or training costs
We propose a distillation framework that emphasizes temporal-wise decoupling of objectives, which utilizes the spatio-temporal properties of SNNs and ensures implicit full-range performance without the need for retraining for specific timesteps.
We analyze the convergence of the proposed method show the superior efficiency and potential for better generalization. Both theoretical proofs and empiri- cal validations illustrate the training guarantees the convergence of all implicit models across full-range timesteps.
We conduct experiments on CIFAR-10, CIFAR- CIFAR1O-DVS, and ImageNet, achieving sfate-of-the- art results among distillation-based SNNs training methods.
2. Related Work
Learning Methods for SNNs. SNNs are typically trained using two main approaches: (1) conversion methods that create a link between SNNs and ANNs through defined closed-form mappings, and (2) direct training from scratch employing Backpropagation Through Time (BPTT). Con- version methods develop precise mathematical formulations for spike representations (Lee et al., 2016; Thiele et al., 2019; Wu et al., 2021a; Zhou et al., 2021; Wu et al., 2021b; Meng et al., 2022), which enable a smooth transition from pre-trained ANNs to SNNs and support comparable perfor- mance on extensive datasets (Cao et al., 2015; Diehl et al., 2015; Han et al., 2020; Sengupta et al., 2019; Rueckauer et al., 2017: Deng & Gu, 2021; Li et al., 2021a; Ding et al., 2021). However, the accuracy of these mappings is not al- ways guaranteed under conditions of ultra-low latency, often requiring longer durations to collect sufficient spikes and potentially reducing performance (Bu et al., 2023; Li et al., 2022; Hao et al., 2023b;a; Jiang et al., 2023). Direct training methods, on the other hand, enable SNNs to achieve robust performance with very few timesteps by using BPTT in conjunction with surrogate gradients to compute derivatives for discrete spiking events (Neftci et al., 2019; Shrestha & Orchard, 2018; Wu et al., 2018: Gu et al., 2019; Yin et al., 2020; Zheng et al., 2021; Zenke & Vogels, 2021; Li et al., 2021b; Suetake et al., 2023; Wang et al., 2023b; Zhang & Li, 2020; Yang et al., 2021). This strategy allows for the de- velopment of SNN-specific components, such optimized neurons, synapses, and network architectures, which im- prove performance (Guo et al., 2023a; Fang et al., 2021b;a; Duan et al., 2022; Yao et al., 2021; Yu et al., 2022; Guo et al., 2022a; Yao et al., 2022; Shen et al., 2023; Qiu et al., 2024b; 2025; Yao et al., 2025). Despite the advantages of re- duced latency, direct training incurs significant memory and
Based on the above considerations, we devised a distillation framework emphasizing temporal-wise decoupling while in- corporating three types of labels: truth target, teacher label, and ensemble label. The proposed distillation framework segments the overarching training objectives into timestep- specific targets, thereby promoting uniform model perfor- mance across all timesteps and alleviating the constraints of fixed timesteps during deployment. For instance, a model trained at T 6 can simultancously produce models for T = 2 and T = 4 with accuracies that rival those explic- itly trained for each timestep (see Fig. lb). To sum up, we provide an efficient distillation framework to tackle the deployment challenges of SNNs, which not only reduces the performance gap between ANNs and SNNs but also ensures that the internal full-range timestep models within the SNN are well-trained, allowing for flexible adjustment of inference timesteps upon deployment according to spe- cific requirements. Our contributions can be summarized as
2

---

Efficient Logit-based Knowledge Distillation of Deep Spiking Neural Networks for Full-Range Timestep Deployment
computational burdens due to the necessity to manage the backward computational graph (Li et al., 2021b; Kim al., 2020; Xiao et al., 2021; 2022; Meng et al., 2022; Deng et al., 2023). To reduce the training expenses associated with direct methods, several recent studies have proposed various light training strategies that have gained significant attention (Mostafa, 2017; Rathi & Roy, 2021; Wang et al., 2022: Zenke & Ganguli, 2018: Bellec al., 2020; Bohnst- ingl et al., 2022; Yin et al., 2023; Xiao et al., 2022; Meng et al., 2022; Zhu et al., 2023; Yu et al., 2024).
timestep, 2 and the output logits of the teacher ANNs, z², the bogits-based distillation loss is composed of hard and soft label components defined on the ensemble voting outputs z = ト E, z (t). The hard label corresponds to the cross-entropy loss with softmax S(.) applied to the
classification task with the true ground one-hot label y:
LSCE = LCF (S(rea)) = Σ vs log Si
(I)
Here, the softmax function S(z) - (2)(2).... "ii...,,,((,)) where Sz(z) = Σ For the soft labels in distillation, we generally use the Kullback-Leible divergence with a temperature scaling factor 7 defined as:
Knowledge Distillation for SNNs. Knowledge distilla- tion (KD) is a well-established transfer learning technique effectively utilized for model compression (Hinton, 2015;
Liu et al., 2019; Sun et al., 2019; Wang & Yoon, 2021; Wei al., 2018). Recent works have adapted KD to train SNNs (Kushawaha et al., 2021; Lee et al., 2021; Takuya et al., 2021; Zhang et al., 2023; Xu et al., 2023b; Hong et al., 2023; Guo et al., 2023b; Xu et al., 2024; Yu et al., 2025), em- ploying logits-based distillation from well-trained ANNs or compressing larger SNNs into more compact models. (Xu et al., 2023b) integrated both logits-based and feature-based knowledge distillation into SNNs. (Hong et al., 2023; Guo et al., 2023b; Xu et al., 2024) further puts forward layer- wise based ANN-to -NN distillation framework. However, SNNs' binary spike representation challenges the direct feature alignment with ANNs, making such detailed alignments potentially overly restrictive (Yang et al., 2025). This work thus focuses solely on logits-based distillation to explore its full potential. Furthermore, distiiiiilation strategies (Zhang al., 2019; Allen-Zhu & Li, 2020; Wang & Yoon, 2021) that do not rely teacher labels have been adapted for SNNs (Deng et al., 2023; Dong et al., 2024; Zuo et al., 2024; Ding et al.). (Deng et al., 2023) adds aux- iliary branches to SNNs to generate projection logits for self-distillation through KL divergence. (Zuo et al., 2024) extends inference times to use longer timestep outputs as teaching signals for shorter timesteps. Nonetheless, these strategies increase the computational burden, elevating the training costs associated with SNNs. (Ding et al.) first treats SNNs at different timesteps as submodels from an ensem- ble perspective, proposing KL divergence between adjacent steps (t and t - 1) to improve performance. While (Ding et al.) focuses on inter-submodel relations, this work fur- ther explores the link between the overall ensemble output and each submodel, aiming to fully exploit the potential of temporal decoupling within the distillation framework.
KL (S(z (S((z/T)) /T)) S ( /T) log /T) S. (,
=
=
Since the entropy regularization term in the KL divergence formula is only related to z and does not contribute to the SNNs training, it can be omitted. This simplifies to:
CSKL = CKL (S Ez (einsT)) (T., Szz /T)
= Σ -S 5 /T) log Si(z
(3)
Combining the classification and distillation losses, the to- tal loss for SNNs standard logits-based distillation can be expressed as:
LSKD = LSCE + a&sII
(4)
where is a coefficient used to balance the two losses.
Temporal-wise Distillation While standard logits- based dis- tillation typically treats SNNs as purely spatial, end-to-end models, it overlooks the unique spatio-temporal characteris- tics inherent to SNNs. Insteads of ANNs with only spatial logits, SNNs produce multiple sets of logits over time. This could offer a unique opportunity for SNNs distillation to leverage spatio-temporal features further. Inspired by en- semble learning (Allen-Zhu & Li, 2020; Wang & Yoon, 2021), viewing the mean output of SNNs as an ensemble aggregated through voting over time, it becomes apparent that the overall outcome across these temporal dimensions tends to improve as the accuracy at each individual timestep increases. This insight allows us to intuitively redefine logits- distillation targets to encompass outputs across various timesteps, thus transforming standard based distillation into temporal-wise distillation. In this context, we define temporal-wise cross-entropy (TWCE) for hard targets as:
3. Method
3.1. From Standard to Temporal-wise Distillation
Standard Logits-based Distillation: First, we consider the standard logits-based knowledge distillation setup for spik- ing neural networks. Given the output of the SNNs at each
LTWER
1 Σ CCR (S(z (1))
ECR
(5)
T
3

---

Efficient Logit-based Knowledge Distillation of Deep Spiking Neural Networks for Full- Full-Range Timestep Deployment
a) Standard Logit-based Distillation
Input
Teacher Logits n
Input
Outputs Logits
Timesteps
ANN
Ensemble Zens
Targets
SNN
convergence
KL Loss
Timesteps Reduction
Timesteps
SNN
CE Loss
Timesteps
X not guaranteed
Ensemble zens
SNN
Input
Outputs Logits z Training Stage
Input
Inference Stage
b) Temporal-wise Logit-based Distillation
Input
Teacher Logits A
Targets y
Input
Outputs Logits 2
ANN
Timesteps
Ensemble z
KL Loss
CE Loss
SNN
convergence
Timesteps
Timesteps Reduction
SNN
KL Loss
Timesteps
X
SNN
convergence
Ensemble ens
x
Input
Outputs Logits z² Training Stage
Input
Inference Stage
Figure 2. Framework overview. (a) Standard Logit-based Distillation defines targets on the final ensemble outputs, where model
convergence not guaranteed with reductions in inference timesteps. (b) Temporal-wir Logit-based Distillation decouples targets into each temporal output, resulting in the guaranteed convergence of all implicit full-range timestep models.
Similarly, temporal-wise KL divergence for soft labels from an ANN teacher is formulated as:
Accordingly, the overall training objective is formulated as:
LTWLL +
E CR.. (S (27(0))) (6)
Linal = LTWCE + aLCTWL + 3LTWSD
(9)
The overall objectives for temporal-wise distillations are thus combined form:
where a and are coefficients to balance the losses. It's worth highlighting that this self-distillation loss is highly adapted the temporal-wise distillation framework, har- moniously augmenting its efficacy using only information from the existing backbone pathway, without adding any extra feedforward computational branches. Its mechanism is structurally akin to using soft labels from the ANN, ensuring consistency across the definitions of loss.
LIWKD LIWCE + aLTWKL
3.2. Enhancing the Overall Framework through Self-Distillation with Final Ensemble Logits
In fact, the potential of the temporal-wise distillation frame- work can be further explored. Consistent with findings from student-ensemble experiments (Allen-Zhu & Li, 2020; Guo al., 2020; Wang Yoon, 2021), we observed that the effi- cacy of voting logits generally surpasses that of individual logits at separate timesteps. Consequently, propose to further incorporate the final voting logits additional set of soft labels for lel-ddisiilation-beeeeee the true labels and teacher-gee teacher-generated labels-to guide the model towards improved convergence:
3.3. Convergence of Temporall-wis Distillation
To elucidate the connection between temporal-wise distil- lation LTWKD and standard distillation LSKD, we start by examining the convergence of BPTT-based SNNs' objec- tives. (Deng et al., 2022) points the convergence chal- lenges of classification objectives and suggests optimizing outputs of each timestep to avoid falling into local minima with low prediction errors but high second-orde moments. The essential convergence proofs can be provided for the (8) temporal cross- entsopyy training objective, as in the following lemma:
1 LTWSD +
E
Lxi. (S (8(2 S S (2/n/T))

---

Efficient Logit-based Knowledge Distillation of Deep Spiking Neural Networks for Full-Range Timestep Deployment
Algorithm 1 Temporal-wise Distillation Framework for Training Deep Spiking Neural Networks
3.4. Convergence Across Full-Range Timesteps
It is worth noting that temporal-wise distillation not only enhances the overall model performance but also ensures good convergence for implicitly integrated models with fewer timesteps in the ensemble. While BPTT-based SNNs training requires a predefined number of timesteps T as a hyperparameter, with training targets defined on the fixed timesteps' voting outputs, this usually results in models that are tailored to specific timesteps and exhibit poor generaliz- ability across different timesteps during inference (see Fig. 2a). In contrast, the temporal-wis -wise distillation framework can ensure the convergence of implicit models, allowing a single trained model to handle various timestep scenarios. We refer to this capability as improving models of full-range timesteps, which greatly enhances the flexibility for model deployment.
Require: Pre-trained ANN model famn. SNN model senns timesteps T, hyper-parameter a; T, input sample x, target label y-
Ensure: Train SNN model with logits-based distillation
1: Obtain SNN temporal outputs (z"(t = fann(x);
2: Obtain ANN output logits =² = Sann( (x);
3: Compute ensemble voting output z 7 E, 25 (t);
4: Compute LWCC. = - E. LCE (S(z)())),) in Eq. (5)
by the truth target y:
5: Get LIWKL- T Ei LKL (S (f)/T) S (z (T) in
Eq. (6) by the teacher label 2d;
6: Get LTWSD = E, Lki. (S (t)/=) SS zens (T)
in Eq. (8) by the ensemble label z
7: Obtain the final objective Linal = LIWCE + aL +
TLwsD in Eq. (9);
8: Update parameters of SNN model based on Linal-
Proposition 3. CTWRD defined on timesteps T forms the scaled upper bound of inner LKD defined on Tr S T, as:
Lemma 1. LTWCe, forms the upper bound of ESCE, as:
( SKD
T (T) TWKD
(12)
Tk
ESCE = Σ y log St (z(is(t)
The proof is provided in Appendix A.3. It can be seen that (T)
(10)
1 T
LWED is not only upper bound for LKD over timesteps T as Eq. (11), but also effectively reduces the upper bound of any implicit (TT)) over timesteps Ty- S T as Eq. (12); this aligns with our empirical findings where using temporal- wise distillation with timestep T during the training phase results in good generalizability of timesteps during the in- ference stage (see Fig. 2b).
<
Vi log Si LTWCE
The proof is provided in Appendix A.I. The equality holds if z (t)= 2. for every t [1, Tj. Based on Lemma 1, using + (cr(Z(())))
instead of LCF (C y) for training can
be viewed as optimizing the upper bound of the overall training objective. Building Lemma 1, we can extend our understanding to the relationship between temporal -wie distillation and standard logits- -aased distillation:
4. Experiments
In this section, we assess the effectiveness of the proposed method through experiments CIFAR-10 (Krizhevsky et al., 2009), CIFAR-100 (Krizhevsky et al., 2009), Ima- geNet (Deng et al., 2009), and CIFAR 10-DVS (Li et al., 2017). We conduct SNNs training on the Pytorch (Paszke et al., 2019) and SpikingJelly (Fang et al., 2023) platforms, employing BPTT with sigmoid-based surrogate functions (Fang et al., 2023). All experimental details are provided in Appendix B.
Proposition 2. forms the upper bound of LSKD, as:
LSKD S LTWKD
(11)
The proof is provided in Appendix A.2. The proposition elucidates that just as the ground-truth CE objective is de- coupled over time, the sobt-label objective's 's decoupling can also ensure convergence of the upper bounds. Therefore, convergence with LIWKD implies the convergence of LSKD: once LTWKD approaches zero, the original loss function LSKD also nears zero. Furthermore, while optimizing for the decoupling objective primarily ensures convergence only to the upper bound of LSKD. the incorporation of LIWSD functions effectively as a regularization term. This could further tighten the incquality in Eq. (11), narrowing the gap between the optimization target LTWKD and LSKD, which ensures that optimizing the upper bound also effectively aids the convergence of the objective LSKD-
4.1. Performance Comparison on Benchmarks
We compare our proposed framework to both directly- trained methods and distillation-based methods on variety of classification benchmarks, as shown static datasets FIFAR-1O/CIFAR-
in Table 1, large-scale ImageNet in Table 2, and neuromorphic dynamic dataset CIFARIO-DVS in Table 3. The directly-traine methods listed in the ta- bles are based on various adaptations of the surrogate- based BPTT training scheme, which are modifications specifically
5

---

Efficient Logit-based Knowledge Distillation of Deep Spiking Neural Networks for Full-Range Timestep Deployment
Table Performance comparison of top-1 accuracy (%) CIFAR-10 and CIFAR-100 datasets, averaged over three experimental
Top-1 Ace. (%)
Method
Model
Timestep
CIFAR-10
CIFAR-100
6
93.16
STBP-wBN (Zheng et al., 2021)
ResNet-19
4
92.92
92.34
6
94.25
74.24
Dspike (Li al., 2021b)
ResNet-18
93.66
73.35
2
93.13
71.68
94.50
74.72
TET (Deng al., 2022)
ResNet-19
4
94.44
74.47
2
94.16
72.87
Direct-training
95.55
RecDis (Guo et al., 2022b)
ResNet-19
4
95.53
74.10
2
93.64
DSR (Meng al., 2022)
ResNet-18
20
95.10
78.50
SSF (Wang et al., 2023a)
ResNet-18
20
94.90
75.48
SLTT (Meng et al., 2023)
ResNet-18
6
94.4
74.38
OS (Zhu et al., 2023)
ResNet-19
4
95.20
77.86
95.90
79.02
ResNet-18
4
95.61
78.26
RateBP (Yu et al., 2024)
2
94.75
75.97
6
96.36
80.83
ResNet-19
4
96,26
80.71
2
96.23
79.87
KDSNN (Xu et al., 2023b)
ResNet-18
4
93.41
4
95.45
77.39
ResNet-18
Joint A-SNN (Guo al., 2023b)
2
94.01
75.79
96.07
79.76
ResNet-34
2
95.13
77.11
ResNet- 18
4
94.07
79.49
SM (Deng et al., 2023)
ResNet-19
4
96.82
81.70
SAKD (Qiu et al., 2024a)
ResNet-19
4
96.06
80.10
BKDSNN (Xu al., 2024)
ResNet-19
4
94.64
74.95
w/ distillation
TSSD (Zuo et al., 2024)
ResNet-18
2
93.37
73.40
TKS (Dong et al., 2024)
ResNet-19
4
96.35
79.89
EnOF (Guo al.)
ResNet-19
2
96.19
82.43
SuperSNN (Zhang et al.)
ResNet-19
95.61
77.45
2
95.08
76.49
95.96
79.80
ResNet-18
95.57
79.10
Our
2
95.11
77.32
97.00
82.56
ResNet-19
4
96.97
82.47
2
96.65
81.47
tailored to the peculiarities of SNNs. The "w/ distillation" group in the tables includes schemes that incorporate dis- tillation or self- soffdistillation on top of directly-training, The results of our approach are consistently based on the hyper- parameter settings of a 0.2, = 0.5 in Eq. (9). It should be noted, as shown in our theoretical analysis, that while scheme can train full-range timestep implicit models simultaneously in large timestep training, achieving better performance at smaller timesteps than retraining individ- ually, we have not used the method of extracting smaller
timesteps from training at larger timesteps for fair com- parison. The models presented are obtained through consis- tent maximum timesteps setting. and further discussions on full-range implicit models will follow the experimental section.
Comparing all results, can be seen that the proposed dis- tillation achieves comparable performance among all bench- marks for both directly-trained methods and distillation- based methods, proving that our approach can ensure effec-
6

---

Efficient Logit-based Knowledge Distillation of Deep Spiking Neural Networks for Full- Full-Range Timestep Deployment
Table 2. Performance comparison of top-1 accuracy (%) on Ima- geNet with single crop,
Table 4. Performance comparison on hyperparameters a, 3 set- tings using ResNet- 18 with T' = 6 on the CIFARI00 dataset.
Method
Model
Timestep
Acc.
8 0.0
a 0.1
0.2
0.3
0.5
0.8
STBP-ddN (Zheng et 2021)
ResNet-34
50
61.88
Top-1 (%)
| 79.31
79.56
79.44
79.48
79.31
Depike (Li 2021b)
34
68.19
0.2
[ = 0.1
0.2
0.3
0.5
0.8
Re:Dis (Guo et. 2022b)
34
67.33
TET (Deng et al., 2022)
ResNet-34
68.00
Top-1 (%)
79.52
79.57
79.75
79.80
79.71
OS (Zhu et al., 2023)
ResNet-34
67.54
RateBP (Yu et al., 2024)
31
70.01
KDSNN (Xu et al,, 2023b)
ResNet-34
67.18
LaSNN (Hong at sl., 2025)
ResNet-34
66.94
Table S. Performance comparison on objectives combinations us- ing ResNet- 18 on the CIFARI00 dataset.
SM (Deng et al., 2023)
RexNet-31
68.
69.
T LIwCE
w/ Crwsp
w/ CTwer.
w/ CTwKI&&cCWee
SAKD (Qia et al., 2024A)
RoiNet-34
70.04
TKS (Dong et al., 2024)
ResNet-34
69.00
78.58
78,94
79.05 79.56
79.10
ExOF (Gin et al.)
34
67.40
79.26
79.63
79.80
Our
R4
71.04
our hyperparameters.
Table Performance comparison of top-1 accuracy (%) on
CIFARIO-DVS, averaged over three experimental runs.
Ablation Study of Training Objectives. Experiments in- volving the ablation of training objectives were conducted, with three parts of labels being added sequentially to de- termine their effects. The results, summarized in Table 5, indicate that LTWKL effectively enhances performance be- yond LIwCE. which aligns with expectations and confirms the positive impact of soft labels distilled from the ANN teacher model. Furthermore, the addition of CTWSD fur- ther enhances the distillation framework, demonstrating that this oeff-disiillation setup acts as a beneficial regularization component for the framework. Overall, all objectives have contributed positively to the distillation framework and are compatible with one another.
Method
Model
Timestep
Acc. (%)
STBP-udNN (Zheng et al,, 2021)
1esNet-19
10
67.80
Depike (Li et al., 2021b)
18
10
75.40
ReeDis (Goo et al,, 20226)
Re
10
72-42
TET (Deng el al., 2022)
VGGSNN
10
83.17
SM (Deng et al., 2023)
ResNet118
10
83.19
SSF (Wang et al., 2023a)
VGG-11
20
78.00
SLTT (Meng et al., 2023)
VGG-I
10
77.17
SAKD (Qiu al., 2024a)
ResNet-19
VGG-I
80.30
81.50
Our
RosNet-18
10
4
86.40
83.50
tive convergence of the model's 's final ensemble and reduce the accuracy gap between SNNs and ANNs. This corre- sponds to our earlier conclusions. It is worth noting that based on the re-derivation of standard logits- meth- ods, the training overhead of our proposed framework is consistent with that of standard logits-based distillation. Our framework does not introduce any additional computa- tional paths, merely altering the definition location of the loss. Like logits-based knowledge distillation, our approach, compared to directly-trained BPTT schemes, only adds the overhead of ANN inference to obtain teacher labels, making it the most efficient case among ANN-guided approaches.
Comparison Study on Temporal Decoupling. Experi- ments evaluating the impact of temporal decoupling were conducted using R the CIFAR100 dataset, with results shown in Table 6. From the results, it can be concluded that temporal decoupling of cross-entropy loss (CsCE) and Kullback-Leibler divergence (Lskr.) individually enhances performance over standard logits- based distilla- tion. Furthermore, the beneficial effects of both can be additive, with the best model performance achieved when both are decoupled, which validates the effectiveness of our distillation framework based on temporal decoupling.
4.2. Ablation Study
Hy Ayperparameter Settings of a and 3. The Table 4 re- ports the Top-1 accuracy under various settings of a and 3, using the ResNet-18 model on the CIFAR 100 dataset. Initially, we demonstrate that the ANN's distillation part, CTWCE, achieves a reasonable performance gain (79.26% vs. 79.56%), shown in the upper part of the table. Sub- sequently, the lower part of the table illustrates that, with a fixed at 0.2, incorporating the self-distillation term LTWSD leads to further improvements (79.56% vs. 79.80%). While CTWSD is indispensable, the improvements are relatively sta- ble around B = 0.5, which we select as the fixed setting for
Table 6. Performance comparison of temporal decoupling on hard targets and soft labels using ResNet- 18 on the CIFARIOR 100 dataset.
T ESce
LIWCE
LSKL
LTWKL
Accuracy (%)
78.32
/
78.60
4
78.74 79.74
V
79.05
79.07
V
79.
6
v
79.32
V
79.56
7

---

Efficient Logit-based Knowledge Distillation of Deep Spiking Neural Networks for Full-Range Timestep Deployment
(e)
Standard Logits-based based Distillation
(b) Temporal wise Distillation
Table 7. Performance comparison of models trained with different
timesteps (i.e. T 2/4/6) and Top-I accuracies (%) across
various inference timesteps (i.e. T + 6).
1.4
1.2
Model
Training
Inference Timesteps
T
2
0.5
50
Epoch
200
300
100
Epoch
750
230
R8sNet-18 18 (logits-KD)
T
T=?
73.58
73.44
76.50
77.02
77.57
77.31
78.32
17.63
77.80
78.47
78,50
77.80
T
71.08
76.25
77.52
78.25
78.63
79.07
Figure Loss Trends. Results of timestep ensembles during train-
ResNet-18 (ours)
T
74.19
77.32
77.65
77.95
78.13
78.14
ing using ResNet-18 18 on the CIFAR 100 dataset.
T
75.08
77.76
78.40
79.10
79.21
79.36
T =
75.09
77.80
78.70
79.32
79.60
79.80
ResNet-19 (ours)
T=
79.40
79.37
81.47
82. 14
81.67
82.47
82.01
82,08
82. 39
82.49
82.36
(a) t-SNE of feature learned by Standard Logits- Logits-based Distillation
79.87
82.50
82.56
Cluster Visualization. As shown in Fig. 4, we present t-SNE visualizations that illustrate the clustering outcomes of two distillation strategies. The visual evidence strongly suggests that temporal-wise distillation, as depicted in Fig. 4b, results in significantly better clustering compared to the standard method shown in Fig. 4a. This enhanced cluster- ing indicates superior training effect on SNNs through temporal-wist distillation, consistent with outcomes from other experiments. Analyzing from the perspective of tem- poral ensembles, it is observed that prior to implementing temporal decoupling training (Fig. 4a), the final outcomes under different ensembles exhibit distinct separations. In the case of the proposed temporal-win -ise distillation Fig. 4b, the clustering effects of the submodels exhibit a high degree of similarity. For example, the upper, lower, central- left and central-right clusters in each subplot of Fig. 4b (marked in red boxes) all display similar pattern across the five sub-figures. This largely echoes our analysis in Section 3.4, where the loss of each submodel is essentially embedded within a larger submodel framework, resulting in uniform convergence effect in their clustering outcomes. This uniformity signifies that the internal implicit models are converging towards the features seen in the final ensem- ble, corresponding to the critical role of the self-distillation component. Furthermore, this observation also explains why embedded implicit models with reduced timesteps Tke perform better than those retrained at the maximum timestep Tg.
Final Ensemble (f s
Votes at 5
Votes 5
(b) t-SNE feature learned by Temporal-wise Distillation
Votes
Votes S
Final Ensemble (f s
Votes at 5
Votes a 5
Figure 4. Visual Results of t-SNE Projections. The features learned by (a) standard logits- logits-based distillation and (b) the pro- posed temporal-win -iie distillation. Each subfigure progressively shows cumulative voting including more timesteps, with the final ensemble shown on the right.
4.3. Analysis and Discussion
Loss Visualization. In Fig. 3, we illustrate the convergence behavior of implicit full-range models during the training process, capturing the evolution of loss across epochs. No- tably, the implementation of temporal decoupling signifi- cantly enhances the convergence of loss at each timestep. As depicted, the loss trajectories for various timestep ensem- bles not only improve but also exhibit tighter and more uniform convergence compared the standard approach. Particularly in the early phases of training with temporal decoupling, there is a notable overlap in the loss values across all timesteps. This overlapping signifies a robust syn- chronization in model performance, closely aligning with theoretical expectations where the loss approaches its theo- retical upper bounds.
Analysis of Full-Range Performance. The results shown in Table 7 offer comparisons of models trained at different timesteps (i.e. T = 2/4/6) and corresponding accuracies across various inference timesteps (i.c. T = 1 + 6). Hori- zontally, it demonstrates how inference accuracy changes when adjusting timesteps after model training. Under stan- dard logits-KLe KD, different models excel within specific in- ference timestep ranges: T model performs best at timesteps 1-2, T = 4 at times 3-4, and T = 6 at time 5-6. In contrast, ours consistently achieves optimal performance using the model trained at the maximum timestep (T' = 6), clearly demonstrating the effectiveness of our method in training robust SNNs that maintain accuracy across varying inference timesteps. Vertically, the results of our proposed
8

---

Efficient Logit-based Knowledge Distillation of Deep Spiking Neural Networks for Full-Range Timestep Deployment
method show consistent improvement in performance as the number of training timesteps increases. The improve- ment across full-range inference timesteps suggests that training with a higher timestep not only enhances model ac- curacy but also provides a more robust generalization across varying inference lengths. In practical terms, this allows for the deployment
and empirical experiments to demonstrate that our frame- work offers an effective solution this issue. Experiments standard benchmarks confirm our superior performance among distillation- based methods. By adopting temporal decoupling, our framework ensures robust model conver- gence and generalization across full-range timesteps. We hope this can pave the way for future developments in SNNs deployment and applications.
single model trained at 7 to effec- tively replace models trained with fewer timesteps (I' 2 or 4). In other words, one can utilize fixed-parane parameeree model to achieve comprehensive coverage across the full range of inference timesteps, significantly alleviating the stringent constraints on inference steps typically required at deployment. The flexibility in deployment is particularly advantageous, offering a streamlined approach to model utilization without sacrificing performance.
A Acknowledgments
This work was supported by the National Natural Science Foundation of China (Grant No. 62304203), the Natural Science Foundation of Zhejiang Province, China (Grant No. LQ22F01001 1), and the ZJU-YST joint research center for fundamental science.
Practical Significance of Temporal Robustness. We con- clude that ensuring the robustness of SNNs models at dif- ferent inference timesteps can provide the following two technical advantages:
Impact Statement
This paper presents work whose goal is to advance the field of Machine Learning. There are many potential socictal consequences of our work, none which we feel must be specifically highlighted here.
Horizontal view from Table 7. Taking the model trained with T = 6 as an example, it shows stable performance across the inference window from T to Once deployed, the model does not require ad- ditional considerations for adaptation and switching across different inference states. This allows to practically balance inference costs and performance di- rectly, providing a viable model approach for scenarios that require real-time time control of inference costs based on computational resources.
References
Akopyan, F., Sawada, J., Cassidy, A., Alvarez-Icaza, R.,
Arthur, J., Merolla, P., Imam, N., Nakamura, Y., Datta, P., Nam, G.-J,, et al. Truenorth: Design and tool flow of a 65 mw 1 million neuron programmable neurosynaptic chip. IEEE transactions on computer-aided design of integrated circuits and systems, 34(10):5537-1537, 2015.
Vertical view from Table 7. For the model trained with T' = 4, we can invest in greater training costs to distill the T = 6 model, and use the submodel at T (essentially the same model) to achieve better performance. This offers an effective and feasible way to enhance performance by leveraging surplus training resources, providing viable technical solution for scenarios where large training resources available and performance enhancement is a critical issue.
Allen-Zhu, Z. and Li, Y. Towards understanding ensem- ble, knowledge distillation and self-distillation in deep learning. arXiv preprini arXiv:2012.09886, 2020.
Bellec, G., Scherr, F., Subramoney, A., Hajek, E., Salaj, D., Legenstein, R., and Maass, W. A solution to the learn- ing dilemma for recurrent networks of spiking neurons. Nature communications, 11(1):3625, 2020.
Bohnstingl, T., Wožniak, S., Pantazi, A., and Eleftheriou, E. Online spatio-temporal learning in deep neural networks. IEEE Transactions on Neural Networks and Learning Systems, 2022.
We provide more detailed discussion in Appendix C on how our proposed method can reduce the actual infer- ence overhead of SNNs in real-world scenarios through the SEENN (Li et al., 2023) framework.
Bu, T., Fang, W., Ding, J., Dai, P., Yu, Z., and Huang, T. Optimal ann-snn conversion for high-accuracy and lltra-low-latency spiking neural networks. arXiv preprint arXiv:2303. 04000,004474.0 2023.
5. Conclusion
Leveraging the unique spatio-temporal dynamics inherent to SNNs, this work incorporates the methodology of tem- poral decoupling into the SNNs logits-based distillation framework. We address the deployment considerations of SNNs that typically require retraining models for different inference timesteps and provide both theoretical analysis
Cao, Y., Chen, Y., and Khosla, D. Spiking deep convolu- tional neural networks for energy- object recog- nition. International Journal of Computer Vision, 113: 54-6, 2015.

---

Efficient Logit-based Knowledge Distillation of Deep Spiking Neural Networks for Full-Range Timestep Deployment
Cubuk, E. D., Zoph, B., Mane, D., Vasudevan, V., and Le, Q. V. Autoaugment: Learning augmentation strategies from data. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 113-123, 2019.
Fang, W., Yu, Z., Chen, Y., Huang, T., Masquelier, T., and Tian, Y. Deep residual learning in spiking neural net- works. Advances in Neural Information Processing Sys- 2021a.
tems, 34:21056-2110
Fang, W., Yu, Z., Chen, Y., Masquelier, T., Huang, T., and Tian, Incorporating learnable membrane time constant enhance learning of spiking neural networks. In Pro- ceedings of the IEEE/CVF international conference computer vision, pp. 2661-2671, 2021b.
Davies, M., Srinivasa, N., Lin, T.-H., Chinya, G., Cao, Y., Choday, H., Dimou, G., Joshi, P., Imam, N., Jain, S., et al. Loihi: A neuromorphic manycore processor with on-chip learning. leee Micro, 38(1):82-99) 99, 2018.
Deng, J., Dong, W., Socher, R., Li, I.-J., ,, Li, K., and Fei-Fei, L. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248-255. leee, 2009.
Fang, W., Chen, Y., Ding, J., Yu, 7.., Masquelier, T., Chen, D., Huang, L.,, Zhou, H., Li, G., and Tian, Y. Spikingjelly; An open-source machine learning infrastructure platform for spike-based intelligence. Science Advances, 940)): eadi1480, 2023.
Deng, S. and Gu, S. Optimal conversion of conventional ar- tificial neural networks to spiking neural networks. arXiv preprint ar urXiv:2103.00442
Gou, J., Yu, B., Maybank, S. and Tao, D. Knowledge distillation: A survey. International Journal of Computer Vision, 129(6):1789-138) 2021.
2021.
Deng, S., Li. Y.. Zhang, S., and Gu, S. Temporal effi- cient training of spiking neural network via gradient re- weighting, arXiv preprint arXiv:2202.1 11946, 2022.
Gu, P., Xiao, R., Pan, G., and Tang, H. Stea: Spatio- temporal credit assignment with delayed feedback in deep spiking neural networks. In ICAAI, volume 15, pp. 1366- 1372, 2019.
Deng, S., Lin, H., Li, Y., and Gu, S. Surrogate module learn- ing: Reduce the gradient error accumulation in training spiking neural networks. In International Conference Machine Learning, pp. 7645-7657. PMLR, 2023.
Guo, Q., Wang, X., Wu, Y., Yu, Z., Liang, D., Hu, X., and Luo, P. Online knowledge distillation via collaborative learning. Proceedings of the IEEE/CVF Conference on Computer Vision and Pallern Recognition, pp. 11020- 11029, 2020.
Deiriss, T. and Taylor, G. W. Improved regularization of convolutional neural networks with cutout. arXiv preprint arXiv: 04552 2017.
Guo, Y., Peng, W., Liu, X., Chen, Y., Zhang, Y., Tong, X., Jie, Z., and Ma, Z. Enof-snn: Training accurate spiking neural networks via enhancing the output feature. The Thirty-eightt Annual Conference on Neural Information Processing Systems.
Diehl, U., Neil, D., Binas, J., Cook, M.. Liu, S.-C., and Pfeiffer, M. Fast-classifying, high-accuracy spiking deep networks through weight and threshold balancing. In 2015 International joint conference on neural networks (IJCNN), pp. 1-8. ieee, 2015.
Guo, Y., Chen, Y., Zhang, L., Liu, X., Wang, Y., Huang, X., and Ma, Z. Im-loss: information maximization loss for spiking neural networks. Advances in Neural Information Processing Systems, 15:156-166, 2022a.
Ding, J., Yu, Z., Tian, Y., and Huang, T. Optimal ann-snn conversion for fast and accurate inference in deep spiking neural networks. arXiv preprint arXiv:2105. 1654, 2021.
Ding, Y., Zuo, L., Jing, M., He, P., and Deng, H. Rethink- ing spiking neural networks from an ensemble learning perspective. In The Thirleenth International Conference on Learning Representations.
Guo, Y., Tong, X., Chen, Y., Zhang, L., Liu, X., Ma. Z., and Huang, X. Recdis- Rectifying membrane potential distribution for directly training spiking neural networks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp.
2022b.
Dong, Y., Zhao, D., and Zeng, Temporal knowledge sharing enable spiking neural network learning from past and future. IEEE Transactions Artificial Intelligence, 2024.
Guo, Y., Huang, X., and Ma. Z. Direct learning- based deep spiking neural networks: review. Frontiers in Neuroscience, 17: 17:1209795, 2023a.
Duan, C., Ding. J., Chen, S., Yu, Z., and Huang, T. Temporal effective batch normalization in spiking neural networks. Advances in Neural Information Processing Systems, 35: 34377-34390, 2022.
Guo, Y., Peng, W. Chen, Y., Zhang, L., Liu, X., Huang, X., and Ma, Z. Joint a-snn: Joint training of artificial and spiking neural networks via self-distillation and weight factorization. Pattern Recognition, 142: 10969, 2023b.
10

---

Efficient Logit-based Knowledge Distillation of Deep Spiking Neural Networks for Full-Range Timestep Deployment
Han, B., Srinivasan, G., and Roy, K. Rmp-snn: Residual membrane potential neuron for enabling deeper high- accuracy and low-latency spiking neural network. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 13558-13567, 2020.
Krizhevsky, A., Hinton, G., al. Learning multiple layers of features from tiny images. 2009.
Kushawaha, R. K., Kumar, S., Banerjee, B., and Velmu- rugan, R. Distilling spikes: Knowledge distillation in spiking neural networks. In 2020 25th International Con- ference on Pattern Recognition (ICPR), pp. 4536-4543. IEEE, 2021.
Hao, Z. Bu, T., Ding, J., Huang, T., and Yu, Z. Reduc- ing ann-snn conversion error through residual membrane potential. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pp. 11-21, 2023a.
Lee, D., Park, S., Kim, J., Doh, W., and Yoon, S. Energy- efficient knowledge distillation for spiking neural net- works, arXiv preprint arXiv:2106.07172, 2021.
Hao, Z., Ding, J., Bu, T., Huang, T., and Yu, Z. Bridging the gap between anns and snns by calibrating offset spikes. 2023b.
Lee, J. H., Delbruck, T., and Pfeiffer, M. Training deep spiking neural networks using backpropagation. Frontiers in neuroscience, 10:228000, 2016.
arXiv preprint arXiv:2302.0066
He, K., Zhang, X., Ren, S., and Sun, J. Deep residual learn- ing for image recognition. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 770-778, 20.
Li, C.,, Ma, L., and Furber, S. Quantization framework for fast spiking neural networks. Frontiers in Neuroscience, 16:918793, 2022.
He, K., Zhang, X., Ren, S., and Sun, J. Identity mappings in deep residual networks. In Computer Vision-ECC -Ccon-ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part IV 14, pp. 630- 645. Springer, 2016b.
Li, H., Liu, H., Ji, X., Li, G., and Shi, L. Cifar10-dvs: an event-stream dataset for object classification. Frontiers in neuroscience, 1 11:244131, 2017.
Li, Y., Deng, S., Dong, X., Gong, R., and Gu, S. A free lunch from ann: Towards efficient, accurate spiking neu- ral networks calibration. In International conference on machine learning, pp. 6316-6325. PMLR, 2021a.
Hinton, G. Distilling the knowledge in a neural network. arXiv preprint arXiv: 1003:0003305500 2015.
Hong, D., Shen, J., Qi. Y., and Wang. Y. Lasnn: Layer- wise ann-to-snn distillation for effective and efficient training in deep spiking neural networks, arXiv preprint arXiv:2304. 00101, 2023.
Li, Y., Guo, Y., Zhang, S., Deng, S., Hai, Y., and Gu, S. Dif- ferentiable spike: Rethinking gradient-descent for train- ing spiking neural networks. Advances Neural Infor- mation Processing Systems, 0122426--23339 2021b.
Hu, Y., Deng, L., Wu, Y., Yao, M., and Li, G. Advancing spiking neural networks toward deep residual learning. IEEE transactions on neural networks and learning sys- tems, 36(2):2353-2367, 2024a.
Li, Y., Geller, T., Kim, Y., and Panda, P. Seenn: towards temporal spiking early exit neural networks. Advances in Neural Information Processing Systems, 36:63327- 63342, 2023.
Hu, Y. Zheng, Q., Li, G., Tang, H., and Pan, G. To- ward large-scale spiking neural networks: A compre- hensive survey and future directions. arXiv preprint arXiv:2409.022 02/ /1, 2024b.
Liu, Y., Chen, K., Liu, C.. Qin, Z., Luo, Z., and Wang, J. Structured knowledge distillation for semantic segmen- tation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 2604 2613-2613, 2019.
Jiang. H., Anumasa, S., De Masi, G., Xiong, H., and Gu, B. unified optimization framework of ann-snn conversion: Towards optimal mapping from activation values to firing rates. In International Conference on Machine Learning, pp. 11945-14974, PMLR, 2023.
Loshchilov, I. and Hutter, F. Sgdr: Stochastic gra- dient descent with warm restarts. arXiv preprint arXiv: N00iv/660.009000 2016.
Kang, Y., Seo, M., Jo, J., Jang, H. J., Park, J., Kim, J., Lee, S., Kim, J., and Park, S. Homeostasis-awar direct spike encoding for deep spiking neural networks. In The First Workshop on NeuroAI@ NeurIPS2024.
Maass, W. Networks of spiking neurons: the third genera- tion of neural network models. Neural networks, 10(9): 1659-1671, 1997.
Meng, Q. Xiao, M., Yan, S., Wang, Y., Lin, Z., and Luo, Z.- Q. Training high-performance low-latency spiking neural networks by differentiation on spike representation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12444-1245 2022.
Kim, J., Kim, K., and Kim, J.-J. Unifying activation-and timing-basee based learning rules for spiking neural networks. Advances in neural information processing systems, 33: 19534-19544, 2020.

---

Efficient Logit-based Knowledge Distillation of Deep Spiking Neural Networks for Full-Range Timestep Deployment
Meng. Q., Xiao, M., Yan, S., Wang. Y., Lin, Z., and Luo, Z.- Q- Towards memory-and time-efficient backpropagation for training spiking neural networks. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 6166-6176, 2023.
Rumelhart, D. E., Hinton, G. B., and Williams, R. J. Learn- ing representations by mack-propagating errors. nature, 323(6088):533-536 1986.
Sengupta, A., Ye, Y., Wang, R., Liu, C., and Roy, K. Going deeper in spiking neural networks: Vgg and residual architectures. Frontiers in neuroscience, 13:95, 2019.
Mostafa, H. Supervised learning based on temporal coding in spiking neural networks. IEEE transactions on newral networks and learning systems, 29(7):327--3235, 2017.
Shen, G., Zhao, D., and Zeng, Y. Exploiting high per- formance spiking neural networks with efficient spiking patterns, arXiv preprint arXiv:2301. 2023.
Neftci, F. O., Mostafa, H., and Zenke, F. Surrogate gradient learning in spiking neural networks: Bringing the power of gradient- based optimization to spiking neural networks. IEEE Signal Processing Magazine, 36(6):51-63. 2019.
Shrestha, S. B. and Orchard, G. Slayer: Spike layer error reassignment in time. Advances in neural information processing systems, 31, 2018.
Panzeri, S. and Schultz, A unified approach to the study of temporal, correlational, and rate coding. Neural Computation, 13(6):1311-134) 2001.
Suetake, K., Ikegawa, S.-i., Saiin, R., and Sawada, Y. S3nn: Time step reduction of spiking surrogate gradients for training energy efficient single-step spiking neural net- works. Neural Networks, 159:208-225 219, 2023.
Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., et al. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, 32, 2019.
Sun, S., Cheng, Y., Gan, Z., and Liu, J. Patient knowledge distillation for bert model compression. arXiv preprint arXiv: orXiv:1908.00r 1908.09355. 2019.
Pei, J., Deng, L., Song, S., Zhao, M., Zhang, Y., Wu, S., Wang, G., Zou, Z., Wu, Z., He, W., al. Towards artificial general intelligence with hybrid tianjic chip architecture. Nature, 57277767):106-11 2019.
Takuya, S., Zhang, R., and Nakashima, Y. Training low- latency spiking neural network through knowledge dis- tillation. In 2021 IEEE Symposium in Low-Power and High-Speed Chips (COOL CHIPS), pp. 1-3. IEEE, 2021.
Qiu, H., Ning, M., Song, Z., Fang, W., Chen, Y., Sun, T., Ma, Z., Yuan, L., and Tian, Y. Self-architectural knowledge distillation for spiking neural networks. Neural Networks, pp. 106475, 2024a.
Thiele, J. C., Bichler, O., and Dupret, A. Spikegrad: An ann-equivalent computation model for implement- ing backpropagation with spikes. arXiv preprint arXiv:1906 190..00666,,, 2019.
Qiu, X., Zhu, R.-J., Chou, Y., Wang, Z., Deng, L.-j., and Li, G. Gated attention coding for training high-performance and efficient spiking neural networks. In Proceedings of the AAAI Conference on Artificial Intelligence, vol- ume 38, pp. 601-610, 2024b.
Wang. J., Song, Z., Wang, Y., Xiao, J.. Yang, Y., Mci, S., and Zhang, Z. Ssf: Accelerating training of spiking neural networks with stabilized spiking flow. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pp. 5982-5991, 2023a.
Qiu, X., Zhang. M., Zhang. J., Wei, W., Cao, H., Guo, J., Zhu, R.-J., Shan, Y., Yang, Y., and Li, H. Quantized spike- driven transformer. arXiv preprint 2Xiv:2501.13492 2025.
Wang. L. and Yoon, K.-J. Knowledge distillation and student-teacher learning for visual intelligence: A review and new outlooks. IEEE transactions on pattern analysis and machine intelligence, (16))3048-306) 2021.
Rathi, N. and Roy, K. Diet-snn: low-latency spiking neural network with direct input encoding and leakage and threshold optimization. IEEE Transactions on Neural Networks and Learning Systems, (4(6))3174-31822 2021.
Wang. Z., Lian, S., Zhang, Y., Cui, X., Yan, R., and Tang, H. Towards lossless ann-snn conversion under ultra-low latency with dual-phase optimization. arXiv preprint arXiv:2205.07473 2022.
Roy, Jaiswal, A., and Panda, P. Towards spike-based ma- chine intelligence with neuromorphic computing. Nature, 575(7784):607-611, 2019.
Wang, Z.,, Jiang, R., Lian, S., Yan, R., and Tang, H. Adaptive smoothing gradient learning for spiking neural networks. In International Conference on Machine Learning, pp. 35798-3588 356 PMLR, 2023b.
Rueckauer, B., Lungu, I.-A., Hu, Y., Pícifferr, M., and Liu, S.-C. Conversion of continuous-vall valued deep networks to efficient event-driven networks for image classification. Frontiers neuroscience, 11 11:294078, 2017.
Wei, Y., Pan, X., Qin, H., Ouyang, W., and Yan, J. Quanti- zation mimie: Towards very tiny enn for object detection.
12

---

Efficient Logit-based Knowledge Distillation of Deep Spiking Neural Networks for Full-Range Timestep Deployment
In Proceedings of the European conference on computer vision (ECCV), pp. 267-283, 2018.
Yao, M., Gao, H., Zhao, G., Wang. D., Lin, Y., Yang, Z., and Li, G. Temporal-wise attention spiking neural networks for event streams classification. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 10221-10230, 2021.
Wightman, R., Touvron, H., and Jégou, H. Resnet strikes back: An improved training procedure in timm. arxiv 2021. arXiv preprint arXiv:2110.0047..
Yao, M., Qiu, X., Hu, T., Hu, J., Chou, Y., Tian, K., Liao, J., Leng, L., Xu, B., and Li, G. Scaling spike-driven trans- former with efficient spike firing approximation training. IEEE Transactions on Pattern Analysis and Machine In- telligence, 2025.
Wu, H.., Zhang, Y., Weng, W., Zhang, Y., Xiong. Z., Zha, Z.- J., Sun, X., and Wu, F. Training spiking neural networks with accumulated spiking flow. In Proceedings of the AAAI conference on artificial intelligence, volume 35, pp- 10320-10328, 2021a.
Wu, J., Chua, Y., Zhang, M., Li, G., H., and Tan, K. C. tandem learning rule for effective training and rapid inference of deep spiking neural networks. IEEE Trans- actions on Neural Networks and Learning Systems, 34 (1):446-466) 2021b.
Yao, X., Li, F., Mo, Z., and Cheng, J. Glif: A unified gated leaky integrate-and-fin neuron for spiking neural networks. Advances in Neural Information Processing Systems, 35:32 35:32160-32171, 2022.
Yin, B., Corradi, F., and Bohté, S. M. Effective and efficient computation with multiple-timescale spiking recurrent neural networks. In International Conference on Neuro- morphic Systems 2020, pp. 1-8, 2020.
Wu, Y., Deng, L.,, Li, G., and Shi, L. Spatio-temporal backpropagation for training perforrrform spiking neural networks. Frontiers in neuroscience, 12:323875, 2018.
Yin, B., Corradi, F., and Bohté, S. M. Accurate online training of dynamical spiking neural networks through forward propagation through time. Nature Machine Intel- ligence, 5(5):518-5277 2023.
Xiao, M.. Meng, Q., Zhang, Z., Wang. Y., and Lin, Z. Train- ing feedback spiking neural networks by implicit differ- entiation on the equilibrium state. Advances in neural information processing systems, 11111-11115555 2021.
Xiao, M., Meng. Q., Zhang, Z., He, D., and Lin, Z. On- line training through time for spiking neural networks. Advances in neural information processing systems, 35: 20717-20730, 2022.
Yu, C., Gu, Z., Li, D., Wang, G., Wang, A., and Li, E. Stsc- snn: Spatio-temporal synaptic connection with temporal convolution and attention for spiking neural networks. Frontiers in Neuroscience, 16:1079357, 2022.
Xu, Q., Li, Y., Fang, X., Shen, J., Liu, J. K., Tang, H., and Pan, G. Biologically inspired structure learning with re- verse knowledge distillation for spiking neural networks. arXiv preprint arXiv:2304.099 09500, 2023a.
Yu, C.,, Liu, L., Wang. G., Li, E., and Wang, A. Advanc- ing training efficiency of deep spiking neural networks through rate-based backpropagation. arXiv preprint arXiv:2410.1 11488, 2024.
Xu, Q., Li, Y., Shen, J., Liu, J. K., Tang, H., and Pan, G. Constructing deep spiking neural networks from artificial neural networks with knowledge distillation. In Proceed- ings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, pp. 7886-7895, 2023b.
Yu, K., Yu, C., Zhang, T., Zhao, X., Yang. S., Wang, H., Zhang, Q. and Xu, Q. Temporal separation with entropy regularization for knowledge distillation in spiking neural networks. arXiv preprint arXiv:2503.03144 2025.
Zenke, and Ganguli, S. Superspike: Supervised learning in multilayer spiking neural networks. Neural computa- tion, 30(6):1514-1544 2018.
Xu, Z., You, K., Guo, Q., Wang, X., and He, Bkdsnn: Enhancing the performance of bearning-based spiking neural networks training with blurred knowledge distilla- tion. arXiv preprint arXiv:2407.09083 2024.
Zenke, F. and Vogels, T. P. The remarkable robustness of surrogate gradient learning for instilling complex function in spiking neural networks. Neural computation, 33(4): 899-925 2021.
Yang, S., Yu, C., Liu, L., Ma, H., Wang, A., and Li, E. Effi cient ann-gddded distillation: Aligning rate-based features of spiking neural networks through hybrid block-wise re- placement. arXiv preprint arXiv:2503. 16572, 2025.
Zhang. F. Yu, C.,, Ma, H., Gu, Z., and Li, E.-p. Knowledge distillation for spiking neural network. In 2023 5th In- ternational Conference on Robotics, Intelligent Control and Artificial Intelligence (RICAI), pp. 1020- IEEE, 2023.
Yang, Y., Zhang, W., and Li, P. Backpropagated neighbor- hood aggregation for accurate training of spiking neural networks. In International Conference on Machine Learn- ing, pp. 1 11852 11662. PMLR, 2021.
13

---

Efficient Logit-based Knowledge Distillation of Deep Spiking Neural Networks for Full-Range Timestep Deployment
Zhang. L., Song, Gao, A., Chen, J., Bao, C., and Ma, K. Be your own teacher: Improve the performance of convolutional neural networks via self distillation. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 3713-3722, 2019.
Zhang. Q., Ge, C.,, Chua, Y., Dou, C.,, and Wu, Supersnn: Training spiking neural networks with knowledge from artificial neural networks.
Zhang, W. and Li, Temporal spike sequence learning via backpropagation for deep spiking neural networks. Advances in neural information processing systems, 33: 12022-12033 12003, 2020.
Zheng, H., Wu, Y., Deng. L.. Hu, Y., and Li, G. Going deeper with directly-trained larger spiking neural net- works. In Proceedings of the AAAI conference artifi- cial intelligence, volume 35, pp. 11062-11070, 2021.
Zhou, C.. Yu, L., Zhou, Z., Ma. Z., Zhang, H., Zhou, H., and Tian, Y. Spikingformer: Spike-driven residual learn- ing for transformer-bast based spiking neural network. arXiv preprint arXiv:2304.1 / /954, 2023.
Zhou, S., Li, X., Chen, Y.. Chandrasekaran, S. T., and Sanyal, Temporal-coded deep spiking neural network with easy training and robust performance. In Proceed- ings of the AAAI conference on artificial intelligence, volume 35, pp. 11143-11151, 2021.
Zhu, Y., Ding, J., Huang, T., Xie. X., and Yu, Z. Online stabilization of spiking neural networks. In The Twelfth International Conference Learning Representations, 2023.
Zuo, L., Ding, Y., Jing, M., Yang, K., and Yu, Self- distillation learning based temporal-spatial consis- tency for spiking neural networks. arXiv preprint arXiv:2406.07862, 2024.
14

---

Efficient Logit-based Knowledge Distillation of Deep Spiking Neural Networks for Full-Range Timestep Deployment
A. Proof of Theorems
A.1. Proof of Lamma 1
Lemma 1. Crwce.ormn the upper bound of LSCE, as:
CSCE Σ 3 log Ss (2)n((t)
(t)) - T Σ E Ui log (t) LIwCE
(13)
Proof: Given the convex nature of the function log [د= e²t) we know that - log(Si(Z)) = log EE= ez 2;
a convex funnction. Here, the Hessian matrix of this function is given by H diag(p) - ppp where p = S(z) (2())..))))... S2(z)....} For any vector v, we have H' DAVR C2 P&UR)* 2 0, resulting from the non-negativity of variance, thus Hi is positive definite. Therefore, by the Jensen's 's Inequality, we have:
LTWCE = - Eu lo log S(( 2 - ys log St (E [ (t)) LSCE.
A.2. Proof of Proposition 2
Proposition LIWKD forms the upper bound of LSKD, as:
LSKD S LTWKD
(14)
Proof: Given that - (z(()) is a functiin, any non-negative linear combination of such functions remains convex, ie.. for all coefficients a, 2 the function obtain:
E: 0; (o)( is convex. Therefore, by applying Jensen's Inequality, we
CTWKL 72 Σ St (z /T) log St (2)())
Σ St (z/ log S, (Biz (0)/T( Lskl
Together with Lemma 1, we then derive
LSKD S LTWCR aLTwKI. LTWKD-
A.3. Proof of Proposition 3
Proposition 3. LTTXD defined on timesteps T forms the scaled upper bound of inner C(To) defined on Ta S T, as:
T
< FIWKD
(15)
SKD
Tk
Proof: Applying Jensen's 's Inequality to the segments of Eign2((1) specifically separating the terms into ExST, (t), 2 (Tx 1), (T + 2). 23 ((T)) then derive:
... CtR. LxL (S (
Tk LKL
S
Ts Σ
x (1)/T
,5 ,5
(221/7
T
k
T
T - Tx
Σ t=t+1
Ca. (s (L)T)) , (x (xA/T))
T2
1
Σ
Lx.. (S (1))))) (22A/T) = CTTKK
7
15

---

Efficient Logit-based Knowledge Distillation of Deep Spiking Neural Networks for Full-Range Timestep Deployment
With Lxi. > O, we obtain Te (KX S S Then, similar inequalities can be applied to CE-based target objectives. Thus, we can establish the following relationship for inner implicit models: 1/ (Tx)
T
SKD TWWDD
B. Experimental Details
B.1. Datasets
CIFAR-10 and CIFAR-100. The CIFAR datasets (Krizhevsky et al., 2009) consist of 32x32 color images distributed across different classes under the MIT license. CIFAR-10 comprises 60,000 images in 10 classes, split into 50,000 for training and 10,000 for testing. CIFAR-100 includes images across 100 classes. Both datasets are normalized to zero mean and unit variance, with image augmentation techniques AutoAugment (Cubuk et al., 2019) and Cutout (DeVriie & Taylor, 2017) applied. The pixel values are directly fed into the input layer each timestep as direct encoding (Rathi & Roy, 2021).
ImageNet. The ImageNet-IKK IK dataset (Deng al., 2009) features 1,281,167 training images and 50,000 validation images across 1,000 classes, normalized for zero mean and unit variance. Training images undergo random resized cropping to 224x224 pixels and horizontal flipping, while validation images are resized to 256x256 and then center-cropped to 224x224. The pixel values are directly fed into the input layer at each timestep as direct encoding (Rathi & Roy, 2021).
CIFARIO-DVS. The CIFARIO-DVS dataset (Li al., 2017) a neuromorphic adaptation CIFAR- 10, which contains 10,000 event-based images captured by the DVS camera, licensed under CC BY 4.0. The dataset is split into 9000 training images and 1000 testing images. Data preprocessing involves integrating events into frames (Fang et al., 2021b; 2023) and reducing the spatial resolution to 48x48 through interpolation. Additional data augmentation includes random horizontal flips and random rolls within a 5-pixel range, mirroring previous methods (Xiao et al., 2022; Meng et al., 2023).
B.2. Training Setup
Network Architectures. For the CIFAR-10 and CIFAR-100 datasets, we use ResNet- 18 and ResNet-19 as student SNN models (He et al.. 2016a; Zheng et al., 2021: Xiao et al., 2022; Fang et al., 2023; Wang et al., 2023b), applying 3esNet-34 with a Top-1 accuracy of 97.24% on CIFAR-10 and 81.90% on CIFAR-100 as the teacher ANN model. In the case of the neuromorphic CIFARIO-DVS dataset, we utilize Re as the teacher model, which trained on the spikes mean across the temporal dimension, with Top-1 accuracy of 83.6% for T with the corresponding timesteps. On the ImageNet dataset, our SNN model is an adapted ResNet-34 with pre-activation residual blocks (He et al., 2016b), with previous studies guiding its configuration (Xiao al., 2022; Meng et al., 2023; Zhu et al., 2023; Yu et al., 2024). The teacher ANN model for ImageNet is pre-trained ResNet-34 from the Timm library ( (Wightman et al.), which has a Top-1 accuracy of 76.32%. All SNN models incorporate the Leaky Integrate-and-FF (LIF) neurons with a consistent membrane potential decay coefficient of 0.5, implemented in activation -aaed mode (Fang et al., 2023).
and 84.4% for T 10, for ResNet- 18 SNN students
Table 8. Hyperparameters Settings.
CIFAR-10
CIFAR-100
ImageNet
CIFARIO-DVS
Epoch Learning rate
300
300
100
300
0.1
0.1
0.2
0.2
Batch size
128
128
512
32
Weight decay
5e-4
5e-4
2e-5
5e-4
Training Details. We employ sigmoid-based surrogate gradient method (Fang et al., 2023) to emulate the Heaviside step function with the equation fı(z; ox) T-e-ar and setting of a The ensemble augmentation for self-distillation implemented (Qiu et al., 2024a). The experiments conducted on the PyTorch (Paszke et al., 2019) and SpikingJelly (Fang et al., 2023) platforms. For 10, CIFAR-100, and CIFAR10-DVS we utilize a single NVIDIA GeForce RTX 3090 GPU, whereas ImageNet experiments are carried out using distributed data parallel processing across 8 NVIDIA GeForce RTX 3090 GPUs The Stochastic Gradient Descent (SGD) optimizer (Rumelhart et al., 1986) with a momentum of 0.9 is used across all datasets, combined with a cosine annealing learning rate strategy (Loshchilov & Hutter, 2016).
Detailed hyperparameters for each setup are summarized in Table 8.
16

---

Efficient Logit-based Knowledge Distillation of Deep Spiking Neural Networks for Full-Range Timestep Deployment
C. More Results and Discussion
C.I. About training costs and practical effectiveness
Table 9. Results on training costs of the standard logit-based framework and the proposed framework on CIFAR-100.
T =
T = 6
logits-KD KD
ours
logits-KD KD
ours
Time (s/batch)
0.17367
0.17443
0.26766
0.26811
Memory (MB)
6333.15
6333.20
9105.12
9105.71
As supplement the main content, provide measurements of training costs CIFAR- 100 in Table 9. Consistent with discussion, the additional training overhead introduced by method is negligible compared to the overall backbone. We conclude that our approach offers "free lunch" in both direct training frameworks (using only LTwEE + LTwsD) and direct training combined with distillation frameworks (using LTWCE + LTWSD + LTwKL.). We think that the practical advantages of our method are evident.
C.2. Statistics on firing rates
Table 10. Statistical results of Firing Rates model ResNet- 18 using the CIFARI00 dataset.
T
Method
f=
2
f t= 3
==
5
mean
ID
0.1799 0.1819
0.2137
0.2045
0.2091
/
/
0,2018 0.2044
4
ours
0.2194
0.2026
0.2138
logits KD
0.1761
0.2034
0.2023
0.1966
0.2060
0.1941
0.1964
ours
0.1775
0.2101
0.1937
0.2063
0.1952
0.1980
0.1980
We measured the average firing rate of all neurons at different time steps, shown in Table 10. It can be observed that the proposed method exhibits firing frequencies that are essentially consistent with those of the standard logits-based method.
C.3. Results more spiking-based architectures
Table 11, Results on the architecture of spikingformer on the CIFAR-100 dataset.
Arch. [T 1]
baseline
ours
trans-2-384
78.34
80.77 81.12
trans-4-384
79.09
In addition to the benchmarks presented in the main content, we also verify the effectiveness of our method on Spikingformer (Zhou et al., 2023) and MS-ResNet (Hu al., 2024a), to demonstrate its generalizability to different network architectures (Table 11 and Table 12). We note that distilling spiking transformer architectures (Qiu et al., 2025; Yao al., 2025) requires special consideration of the choice of the teacher model ANN, the design philosophy of spiking transformer architectures differs from traditional ANN architectures. Unlike the ResNet structure, it is not possible to find an ANN counterpart with the same structure, making this unique consideration for transformer distillation schemes. At this point, the differences in structural design necessitate a more ingenious and novel heuristic design for the loss objective aligned with features between the ANN teacher and the SNN teacher under the nattrre-based distillation framework, which introduces greater design complexity in practical implementations. Therefore, we think that based, end-to-end distillation offers more practical advantages in the direction of spiking transformers. As this work aptly discusses how to fully unleash the potential of logits-based distillation in exploiting the unique spatiotemporal characteristics of SNNs, it could provide solid foundation for further exploration of logits-based distillation in spiking transformer architectures.
We also conducted experiments with VGGSNN (Deng et al., 2022) on the CIFARIO-DVS dataset, as VGGSNN is a commonly used model for this dataset. The results also allow us to better investigate the interactions among different objectives on the dynamic dataset. As shown in Table 13, LIWSD demonstrates strong performance in this setting. As mentioned in Appendix A, since CIFAR1O-DVS is dynamic dataset, the ANN counterpart is trained on the mean input
17

---

Efficient Logit-based Knowledge Distillation of Deep Spiking Neural Networks for Full-Rang Timestep Deployment
Table 12. Results the architecture of MS-ResNet the CIFAR-100 dataset.
Arch. [ =
baseline
logits-KD
ours
MS-ResNet-188
76.41
79.63
80.49
Table 13. Results on objectives ablation using VGGSNN on the CIFARIO-DVS dataset.
T
LIWCE
LINce + LIwSD LIwcE LIwso + LIWKL
10
83.2
85.8
86.3
across the temporal dimension, which leads to lower accuracy compared to LIWCE + LWSDD Nevertheless, the results indicate that the KL. loss LTWKL with soft labels still provides a positive effect on model training.
We also note some works that enhance input processing by introducing strategies at the input side (Qiu et al., 2024b; Kang al.), which implement adaptive encoding of spike sequences during training via modules at the input layer. This could help improve the adaptability of SNNs to static inputs and better leverage SNNs' ability process spatio-tempon information. Interestingly, under such enhanced encoding methods, intermediate features evolve over time, and spike representations embed temporal information. In this context, standard distillation frameworks may force features across time steps to align with the same ANN targets, undermining temporal diversity. Our proposed distillation framework, designed for temporal-wir wise decoupling, is better suited for these scenarios. We think its potential in this direction merits further investigation.
C.4. Results toward real-world scenarios through SEENN
Table 14. Results of pruning inference time using the trained models under the SEENN-I framework on the CIFAR-I00 100 dataset. We compare the standard logits- based method with our method, where SNN models are trained with inference-time pruning. Here, CE denotes the confidence score mentioned in (Li et al., 2023), which is used to control the carly-exit timing and halance the trade-off hetween accuracy and inference timesteps. We use "T" avg." to indicate the average number of timesteps during inference.
and T' 6. and then transferred to the SEENN framework for
Method
Timesteps
CS
0.7
0.8
0.9
0.99
0.999
Acc. (%)
73.23
74.65
77.12
78.75
79.03
T
T avg.
1.139
1.280
1.606
2.424
3.076
logits-KD
Acc. (%)
74.14
75.53
77.53
78.28
78.32
T = 4
T' avg.
1.138
1.268
1.568
2.168
2.697
Acc. (%)
76.61
77.58
79.05
79.75
79.79
T' = 6
I' avg.
1.165
1.316
1.690
2.493
3.188
ours
Acc. (%)
76.51
77.46
78.73
79.09
79.10
T 4
T' avg.
1.164
1.306
1.620
2.211
2.752
The SEENN project (Li et al., 2023) ingeniously designed temporal pruning, achieving a trade-off improvement between inference time and performance through early exit, and provided a dynamic adjustment scheme for inference. We replicated the SEENN-I scheme, setting under logits-KII ADDts-KDDea and our method on CIFAR-100, and compressed the inference time with the results in Table 14. First, consistent with our previous observations, logits-KD training at T performed better than at T= 6 in scenarios where the inference time was significantly reduced. We think this is because the submodel T 4 is more advantageous during moments T = 1 - 4, hence at a specific time point, e.g., predicts more accurately; in this case, due to the reduction in inference time, lots of models might early exit at f = 2, leading to a more reliable performance for the T = 4 model compared to the T = 6 model. This supports the importance of time robustness as a model property, which significantly impacts optimization when pruning inference time. The compression results of ours at T = 4 and T demonstrated the advantages brought by time robustness in actual SNN scenarios. The gains from time robustness allow us achieve similar performance with even less reduced inference time, which can be used to further reduce the actual inference overhead of SNNs within the SEENN framework.
2, the model trained at 7 = 4
18


Reviews:
[{"review_id": "8YsVcmABgE", "reviewer": "Reviewer_p1Y4", "summary": "This paper proposes a Temporal-wise logit-based distillation method (TWLD), which consists with three different loss (classification loss, distilation loss, and self-distillation loss) for SNN output at each time step. This new method significantly improves the performance of SNNs and their robustness to time step change.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Response to Reviewer p1Y4 \n---\n### __R1:__ Relation of Fig. 4 to the main claim and Tab. 7 baseline results\n\nThank you for your suggestion. We use Fig. 4 to show the distinction from the standard logits-based distillation. In the case of the proposed temporal-wise distillation in Fig. 4b, the clustering effects of the submodels exhibit a high degree of similarity. For example, the upper, lower, and central-left clusters in each subplot of Fig. 4b (which will be marked in the updated images) all display a similar pattern across the five sub-figures. This largely echoes our analysis in Section 3.4, where the loss of each submodel is essentially embedded within a larger submodel framework, resulting in a uniform convergence effect in their clustering outcomes. In the updated version, we will specifically highlight the observed clustering phenomena in Fig. 4b to better complement the discussion mentioned above.\n\nAdditionally, we think the following results can more intuitively demonstrate the effectiveness of the proposed method compared to standard logits-based distillation paired with Fig 4:\n\n**Table X1. Comparison of inference performance based on models trained over different timesteps. The experiments were conducted using ResNet-18 on the CIFAR-100 dataset.**\n                \n|Method|Timesteps|T=1|T=2|T=3|T=4|T=5|T=6|\n|-|-|-|-|-|-|-|-|\n|standard logits-KD|T=2|**73.58**|**77.02**|77.31|77.63|77.80|77.80|\n||T=4|72.44|76.50|**77.57**|**78.32**|78.47|78.50|\n||T=6|71.08|76.25|77.52|78.25|**78.63**|**79.07**|\n|ours|T=2|74.19|77.32|77.65|77.95|78.13|78.14|\n||T=4|75.08|77.76|78.40|79.10|79.21|79.36|\n||T=6|**75.09**|**77.80**|**78.70**|**79.32**| **79.60**|**79.80**|\n\nThe table above shows inference results, supplementing the baseline results shown in Tab. 7. Horizontally, it demonstrates how inference accuracy changes when adjusting timesteps after model training. Under standard logits-KD, different models excel within specific inference timestep ranges: T=2 model performs best at timesteps 1–2, T=4 at times 3–4, and T=6 at time 5–6. In contrast, ours consistently achieves optimal performance using the model trained at the maximum timestep (T=6), clearly demonstrating the effectiveness of our method in training robust SNNs that maintain accuracy across varying inference timesteps.\n\nThank you for your suggestion; we will include these results and analysis in the updated version.\n\n### __R2:__ Practical significance of temporal robustness\n\nEnsuring the robustness of SNNs models at different inference timesteps can provide the following two technical advantages:\n1. **Horizontal view from Table X1.** Taking the model trained with T=6 as an example, it shows stable performance across the inference window from T=1 to 6. Once deployed, the model does not require additional considerations for adaptation and switching across different inference states. This allows us to practically balance inference costs and performance directly, providing a viable model approach for scenarios that require real-time control of inference costs based on computational resources.\n2. **Vertical View from Table X1**. For the model trained with T=4, we can invest in greater training costs to distill the T=6 model, and use the submodel at T=4 (essentially the same model) to achieve better performance. This offers an effective and feasible way to enhance performance by leveraging surplus training resources, providing a viable technical solution for scenarios where large training resources are available and performance enhancement is a critical issue.\n\n### __R3:__ Results of firing rates\n\n|R18,C100|t=1|t=2|t=3|t=4|t=5|t=6|avg|\n|-|-|-|-|-|-|-|-|\n|logits-KD [T=4]|0.1799|0.2137|0.2045|0.2091|/|/|0.2018|\n|ours [T=4]|0.1819|0.2194|0.2026|0.2138|/|/|0.2044|\n|logits-KD [T=6]|0.1761|0.2034|0.2023|0.1966|0.2060|0.1941| 0.1964|\n|ours [T=6]|0.1775|0.2101|0.1937|0.2063|0.1952|0.1980|0.1980|\n\n### __R4:__ About training costs and practical effectiveness\n\nThank you for your suggestion. We have added measurements of training costs:\n\n||logits-KD [T=4] | ours [T=4] | logits-KD [T=6] | ours [T=6] |\n|-|-|-|-|-|\n|Time (s/batch)|0.17367|0.17443|0.26766|0.26811|\n|Memory (MB)|6333.15|6333.20|9105.12|9105.71|\n\nConsistent with the discussion, the extra training overhead from ours is negligible compared to the overall backbone. We will include results in the updated version to reinforce the lightweight nature.\n\nFollowing your comments, we conclude that ours achieves a 0.73% performance improvement over logits-KD at virtually no extra cost and also introduces robustness across varying inference timesteps. We think advantages of ours in practical use are evident.\n\n### __R5:__ Include logits-KD in Fig. 1\nThank you for your suggestion. We will add the performance of logits-based KD to Fig. 1 to more clearly show effectiveness. The necessary data is already available in Table X1.", "claims_and_evidence": "Yes, the authors provide theoretical proof and sufficient experimental verification.", "methods_and_evaluation_criteria": "Yes, the proposed method has improved the performance upper bound of SNN, which may make sense for the research and application of SNN.", "theoretical_claims": "Yes, the authors provide theoretical proofs of the effectiveness of the corresponding claims.", "experimental_designs_or_analyses": "Most of the experiments provided by the this paper are reasonable and effective. However, I think the t-SNE experiment (Figure.4) is unnecessary as it does not demonstrate the effectiveness of the new method compared to Standard Logits-based Distillation.", "relation_to_broader_scientific_literature": "NA", "essential_references_not_discussed": "NA", "other_strengths_and_weaknesses": "Strength:\n1. The algorithm provided by the author is simple and effective, achieving SOTA  performance on ImageNet.\n2. The author provided enough theoretical proof and relevant experiments to demonstrate the effectiveness of the algorithm.\n\nWeakness:\n1. The author needs to explain in detail the practical significance of achieving a more robust SNN with time step changing.\n2. The author did not provide whether their method affects the important SNN factors such as firing frequency.\n3. The results presented in Table 6 show that compared to the original distillation algorithm, this paper only improved by 0.73%. The author needs to provide more detailed comparisons on the training cost of the algorithm.", "other_comments_or_suggestions": "Figure 1 should provide the comparison between the proposed method and the original distillation learning, rather than direct training.", "questions_for_authors": "NA", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "wR5JhaM2yc", "reviewer": "Reviewer_zowF", "summary": "This paper proposes a novel distillation framework for deep SNNs that optimizes performance across full-range timesteps without\nspecific retraining, enhancing both efficacy and deployment adaptability. We provide both theoretical analysis and empirical validations to illustrate that training guarantees the convergence of all implicit models across full-range timesteps.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Response to Reviewer zowF\n---\n### __R1:__ Is the proposed method still in MS-ResNet [1]?\n\n\nFollowing the experimental setup in [3], our frameword can also support the MS-ResNet [4] structure as expected, with the experimental results as follows:\n\n|CIFAR100,T=6 | baseline | logits-KD | ours |\n|-|-|-|-|\n|MS-ResNet-18|76.41|79.63|80.49|\n\n### __R2:__ Discuss the recent paper [2] using distillation in SNNs.\n\nThank you for your comment. We acknowledge that [2] is an excellent piece of work and represents a significant advancement in the study of spiking transformer architectures. We note that distilling spiking transformer architectures requires special consideration of the choice of the teacher model ANN, as the design philosophy of spiking transformer architectures differs from traditional ANN architectures. Unlike the ResNet structure, it is not possible to find an ANN counterpart with the same structure, making this a unique consideration for transformer distillation schemes. At this point, the differences in structural design necessitate a more ingenious and novel heuristic design for the loss objective aligned with features between the ANN teacher and the SNN teacher under the feature-based distillation framework, which introduces greater design complexity in practical implementations. Therefore, we believe that logits-based, end-to-end distillation offers more practical advantages in the direction of spiking transformers. Our work aptly discusses how to fully unleash the potential of logits-based distillation in exploiting the unique spatiotemporal characteristics of SNNs, thereby laying a solid foundation for further exploration of logits-based distillation in spiking transformer architectures.\n\nWe will update our paper with the above discussion to strengthen our positioning in the literature. Thank you for your suggestion.\n\n### __R3:__ Discuss the recent paper [3] in learning methods for SNNs.\n\nThe work in [3] is quite fascinating. We understand it as implementing adaptive encoding of spike sequence inputs during the network training process by adding an attention module at the input layer, which significantly enhances the adaptability of SNN architectures to static inputs and better utilizes the SNN’s capability to process spatiotemporal information. Interestingly, under this encoding method, the characteristics of the intermediate features at different times change with time, and the spike representations carry temporal dimension information. In this scenario, the standard distillation framework would cause the features at different times to fit towards the same ANN features, thus diluting the desired diversity in the temporal dimension. The prposed distillation framework considered in our paper, which is designed for temporal-wise decoupling, is better suited for such scenarios. We think the great potential of the proposed distillation framework in this direction is worth further exploration.\n\nWe will include the above discussion in the updated version. Thank you for your suggestion.\n\n---\n\n## Reference:\n[1] Quantized Spike-driven Transformer. ICLR 2025.\n\n[2] Scaling Spike-driven Transformer with Efficient Spike Firing Approximation Training. IEEE TPAMI 2025.\n\n[3] Gated attention coding for training high-performance and efficient spiking neural networks. AAAI 2024.\n\n[4] Advancing Spiking Neural Networks towards Deep Residual Learning. IEEE TNNLS 2024.", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "Yes", "theoretical_claims": "Yes", "experimental_designs_or_analyses": "Yes", "relation_to_broader_scientific_literature": "N/A", "essential_references_not_discussed": "No discussion of recent paper [1] using distillation in SNNs.\n\nNo discussion of recent paper [2] in learning methods for SNNs.\n\n[1] Quantized Spike-driven Transformer. ICLR 2025.\n\n[2] Scaling Spike-driven Transformer with Efficient Spike Firing Approximation Training. IEE T-PAMI 2025.", "other_strengths_and_weaknesses": "S:\n\na novel knowledge distillation framework is proposed. This framework optimizes the performance of SNNs over the entire time step range by means of time decoupling, without the need to retrain for specific time steps, significantly improving the deployment adaptability of SNNs.\n\nwell-written\n\n\nW:\n\nIs the proposed method still in MS-ResNet [1]?\n\nNo discussion of recent paper [2] using distillation in SNNs.\n\nNo discussion of recent paper [3] in learning methods for SNNs.\n\n[1] Quantized Spike-driven Transformer. ICLR 2025.\n\n[2] Scaling Spike-driven Transformer with Efficient Spike Firing Approximation Training. IEE T-PAMI 2025.\n\n[3] Gated attention coding for training high-performance and efficient spiking neural networks. AAAI 2024.", "other_comments_or_suggestions": "See above.", "questions_for_authors": "Is the proposed method still in MS-ResNet [1]?\n\nDiscuss the recent paper [2] using distillation in SNNs.\n\nDiscuss the recent paper [3] in learning methods for SNNs.\n\n[1] Quantized Spike-driven Transformer. ICLR 2025.\n\n[2] Scaling Spike-driven Transformer with Efficient Spike Firing Approximation Training. IEE T-PAMI 2025.\n\n[3] Gated attention coding for training high-performance and efficient spiking neural networks. AAAI 2024.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "GGkXMhYuG8", "reviewer": "Reviewer_BB72", "summary": "This paper proposes Temporal-wise Logit-based Distillation, which trains the SNN timestep by timestep using the labels and the teacher output of the ANN, and uses the overall average output as a guidance signal. The authors theoretically prove the convergence of their losses and experimentally demonstrate the performance of their method on static and DVS datasets.\n\n\n## update after rebuttal\nFurther responses from the authors demonstrate the effectiveness of their method.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Response to Reviewer BB72\n---\n### __R1:__ Contributions reclaim.\nOur contributions primarily focus on a **logits-based distillation** framework tailored for SNNs, effectively leveraging spatio-temporal features **without incurring additional overhead**. Specifically, we address two critical deployment concerns: **accuracy degradation from ANNs** and **retraining costs when adjusting timestep settings**, achieving this through a lightweight, end-to-end solution validated by theoretical and empirical analysis. Furthermore, the trained SNNs demonstrate a notable advantage in terms of **robustness across inference timesteps**, as exemplified by stable performance from T=1 to T=6 (Tab. 7), enabling practical, flexible trade-offs between inference cost and accuracy. Overall, our clearly articulated methodology, novel conceptualization of temporal robustness, theoretical and empirical validation meaningfully advance the field of SNN distillation. Connections and differences with related works are further detailed in responses **R1.1**-**R1.3**, positioning our contributions within the existing literature.\n### __R1.1:__ Relation to TET [1] and its underlying concept\nOur work relates to TET [1], which significantly advanced direct SNN training by highlighting temporal-wise decoupling of the classification objective. Inspired by this underlying concept, our paper introduces, for the first time, a clear definition and adaptation of temporal-wise decoupling into logits KD domain. Specifically, we demonstrate—both theoretically and empirically—the advantage of designing logits KD aligned with the spatio-temporal characteristics inherent to SNNs.\n\nUnlike TET, which focuses exclusively on decoupling cross-entropy loss and convergence boundary analysis, our method emphasizes applying temporal decoupling explicitly within logits KD, highlighting that performance improvements can be obtained without additional complexity (“free lunch”, **see R4 of Reviewer p1Y4**). Moreover, our work uniquely emphasizes the robustness across different inference timesteps, showcasing stable convergence for submodels (Sec. 3.4 and Tab. 7), an advantage previously unexplored by TET or related studies. Additionally, by integrating self-distillation within our temporal-decoupling framework, we fully utilize supervision signals at all timesteps without introducing extra overhead, further extending beyond TET.\n### __R1.2:__ Relation and comparison with TSSD [2]\nWe regard TSSD as representative work in self-distillation, exploring soft-label generation from temporal and spatial perspectives. Unlike TSSD, which introduces overhead through auxiliary classifiers and extra forward processes, our method achieves improvements by using existing supervision signals (teacher, ensemble, and task labels in logits-KD framework) within a fixed backbone, requiring no additional branches or training overhead.\n### __R1.3:__ Ensemble perspective with [3]\nWe recognize [3] as an excellent contemporaneous study that similarly explores submodels within SNNs from ensemble perspectives. While [3] constructs KL between adjacent timesteps (t and t-1), our method builds KL between the overall ensemble output and each individual time t. We will include discussion in the updated version, enabling readers to more comprehensively grasp ensemble learning in SNNs. We plan a detailed comparison once [3] becomes publicly available.\n### __R2:__ About experiment setup.\nIn setting up, our work has maintained strict alignment with recent SNN studies [OTTT, SLTT, ASGL, rateBP]. We replicated logits KD, TKS on C100 setup:\n||base|logits-KD|TKS|ours|\n|-|-|-|-|-|\n|R18,C100,T=6|78.62|79.07|79.15|79.80|\n\nOn Imagenet and C10-DVS (Tab. 2 & 3), base models do not exhibit special advantage. The result on large-scale __Imagenet__ is indicative of performance.\n### __R3:__ Ablation study on CIFAR10-DVS\n||$L_{TWCE}$|$L_{TWCE}+L_{TWSD}$|$L_{TWCE}+L_{TWSD}+L_{TWKD}$|\n|-|-|-|-|\n|vggssn,C10DVS,T=10|83.2|85.8|86.3|\n### __R4:__ Results on spikingformer [4]\n|Arch|Spikingformer|+ours|\n|-|-|-|\n|trans-2-384,C100,T=4|78.34|80.77|\n|trans-4-384,C100,T=4|79.09|81.12|\n### __R5:__ Ablation on TWSD\n||$L_{SCE}$|$L_{SCE}+L_{TWSD}$|$L_{TWCE}$| $L_{TWCE}+L_{TWSD}$ |\n|-|-|-|-|-|\n|R18,C100,T=4|77.95|78.50|78.58|78.94|\n|R18,C100,T=6|78.62|79.15|79.26|79.63|\n### __R6:__ Relation of Fig. 4 to the main claim and Tab. 7 baseline results\nThank you for suggestions. Please refer to **R1 of Reviewer p1Y4**.\n### __R7:__ Regarding reproducibility and minor errors\nFollowing modifications will ensure the code runs correctly:\n1. In`util.py`, change`args.arch().lower()`to `args.stu_arch.lower()`within`get_model_name`function.\n2. Delete`init_model(model)`in`bptt_model_setting`function.\n3. In run command, replace`--epoch`with`--num_epoch`.\n\nThe appendix provides necessary details. We plan to open-source complete codes and ensure reproducibility in future releases. Minor errors will be fixed, and we appreciate your feedback.", "claims_and_evidence": "The authors claim in the t-SNE visualization in Figure 4 that their method has better clustering than the standard SNN training method, but this effect is hard to perceive.", "methods_and_evaluation_criteria": "The proposed method is incremental rather than innovative:\n1. The temporal-wise cross-entropy (TWCE) loss proposed in this paper is the same as the TET loss (see Eq. 9 in the TET paper [1]);\n2. The timestep-by-timestep distillation method is similar to the spatial self-distillation loss in TSSD [2], with the difference that this paper uses a pre-trained ANN, whereas TSSD uses the final output of the SNN to bootstrap intermediate outputs.\n\nIn addition, the authors consider the total output of the SNN as an ensemble for self-distillation, suggesting the inclusion of a discussion and comparison with [3]. Moreover, [3] also showed that their method is able to achieve better inference performance with less than the training timestep, so I suggest a fair comparison with [3].\n\n\n```\n[1] Temporal Efficient Training of Spiking Neural Network via Gradient Re-weighting. In ICLR. 2022.\n[2] Self-Distillation Learning Based on Temporal-Spatial Consistency for Spiking Neural Networks. In Arxiv. 2024.\n[3] Rethinking Spiking Neural Networks from an Ensemble Learning Perspective. In ICLR. 2025.\n```", "theoretical_claims": "There are no obvious weaknesses in the theoretical proofs in this paper.", "experimental_designs_or_analyses": "About the experiment:\n1. Although the proposed method shows a clear performance advantage in comparative experiments, I think that **the performance advantage claimed by the authors is mainly due to the experimental setup (the performance of the baseline model) rather than to their method**. For example, using ResNet-18 at T=6, the baseline of this paper achieves an accuracy of 79.26% on CIFAR100, exceeding the performance of other methods with the same architecture. The addition of the proposed method increases the accuracy by only 0.53%, which does not indicate the performance advantage of its method. I suggest that the authors make a fair comparison with other methods under the same conditions.\n2. I would like the authors to provide ablation experiments on the DVS dataset, as the use of an ANN teacher may not promote the temporal properties that are important for SNNs. In addition, I suggest that the authors add the results using VGGSNN on CIFAR10-DVS, as this is a commonly used architecture for SNN methods, and TKS [1] achieved an accuracy of 85.3% using VGGSNN on CIFAR10-DVS with 10 timesteps.\n3. Could the method proposed in this paper be applied to the Transformer architecture, which might be able to improve the performance of the SOTA SNN?\n4. The authors should provide ablation experiments using only $\\mathcal{L}_{TWSD}$, as this is one of their most innovative points.\n5. The visualization in Figure 4 does not reflect the advantages of the method proposed in this paper, and it is recommended that the authors provide more comparisons or other visualizations.\n6. It is recommended that the authors report the performance of the baseline model in Table 7 to reflect the performance advantages of their method at different timesteps.\n\n```\n[1] Temporal Knowledge Sharing Enable Spiking Neural Network Learning From Past and Future. IEEE Transactions on Artificial Intelligence. 2024.\n```", "relation_to_broader_scientific_literature": "I think the main contribution of this paper is similar to [1,2] with a weak core innovation.\n\n```\n[1] Temporal Efficient Training of Spiking Neural Network via Gradient Re-weighting. In ICLR. 2022.\n[2] Self-Distillation Learning Based on Temporal-Spatial Consistency for Spiking Neural Networks. In Arxiv. 2024.\n```", "essential_references_not_discussed": "This paper's view of the final output of the SNN as an ensemble (in Section 3.2) is similar to [1]'s, and I suggest that the authors discuss the differences between the two.\n```\n[1] Rethinking Spiking Neural Networks from an Ensemble Learning Perspective. In ICLR. 2025.\n```", "other_strengths_and_weaknesses": "I think this paper's method of using the overall output of the SNN to distill the output on a timestep basis is concise and somewhat original, and I suggest that this is the main contribution of this paper. It is recommended that the authors include ablation of this loss in the experimental section to verify its validity.", "other_comments_or_suggestions": "Authors should compare fairly with other methods in their experiments, not just achieve improved accuracy with a high baseline model.\n\nIn addition, there are some minor errors in the paper that need to be improved, such as an incomplete sentence in line 78 and a double citation of [1] in the references section.\n\n```\n[1] Surrogate Module Learning: Reduce the Gradient Error Accumulation in Training Spiking Neural Networks. In ICML. 2023.\n```", "questions_for_authors": "I have no further questions. I am willing to adjust my score if the author clearly demonstrates the innovation and effectiveness of their method.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper introduces a temporal-wise logit-based distillation (TWLD) framework for training Spiking Neural Networks (SNNs), with the goal of improving robustness across a range of inference timesteps. The method incorporates classification loss, distillation loss, and self-distillation loss applied at each timestep. The authors provide theoretical analysis on convergence and evaluate the approach on standard benchmarks, including CIFAR-10, CIFAR-100, CIFAR10-DVS, and ImageNet. The method aims to enhance deployment flexibility without requiring retraining for different timesteps.

## Strengths  
- **Timestep robustness**: The proposed method addresses a practical challenge in SNN deployment by training models to maintain performance across varying inference timesteps.
- **Low overhead**: The approach introduces minimal computational and memory overhead, with results showing consistent, if modest, performance improvements over standard KD baselines.
- **Broad applicability**: Experimental results cover both static and neuromorphic datasets, and the framework is shown to generalize to architectures such as MS-ResNet and Transformer-based SNNs.
- **Rebuttal clarifications**: The authors provided detailed responses to reviewer questions, including new results, ablation studies, baseline alignment, and discussion of related work.

## Weaknesses  
- **Incremental novelty**: The overall structure of the method shares similarities with prior work in temporal decoupling (e.g., TET), self-distillation (e.g., TSSD), and ensemble-based training. While the authors differentiate their method conceptually, the core components are viewed by some reviewers as extensions of existing techniques.
- **Comparative baselines**: Some reviewers noted that stronger or more consistent baselines (e.g., TKS, TSSD) could have been used in the initial submission. These were addressed in the rebuttal, but not all were included in the original manuscript.
- **Literature positioning**: Recent concurrent or closely related works, such as those accepted at CVPR and ICLR 2025, were not initially discussed. The authors acknowledged these papers and included comparisons in the response.

## Rebuttal Assessment  
The rebuttal was comprehensive and addressed most reviewer concerns with additional experiments and analysis. Reviewers appreciated the clarification of methodological distinctions and the added evaluations on Transformer architectures and SEENN-based pruning scenarios. Several reviewers revised their evaluations positively following the response.
Decision: Accept (poster)