Many Eyes, One Mind: Temporal Multi-Perspective and Progressive Distillation for Spiking Neural Networks
Authors: Kai Sun
Peibo Duan
Yongsheng Huang
Nanxu Gong
Levin Kuhlmann
Venue: ICLR 2026 Poster (2026)
Keywords: Spiking Neural Networks
Knowledge Distillation
Neuromorphic Computing
Primary area: applications to neuroscience & cognitive science
TL;DR: We propose MEOM, a unified temporal distillation framework that improves SNN accuracy and time flexibility across timesteps via multi-perspective supervision and progressive alignment to full-length prediction.

Abstract: Spiking Neural Networks (SNNs), inspired by biological neurons, are attractive for their event-driven energy efficiency but still fall short of Artificial Neural Networks (ANNs) in accuracy. Knowledge distillation (KD) has emerged as a promising approach to narrow this gap by transferring ANN knowledge into SNNs. Temporal-wise distillation (TWD) leverages the temporal dynamics of SNNs by providing supervision across timesteps, but it applies a constant teacher output to all timesteps, mismatching the inherently evolving temporal process of SNNs. Moreover, while TWD improves per-timestep accuracy, truncated inference still suffers from full-length temporal information loss due to the progressive accumulation process. We propose **MEOM** (**M**any **E**yes, **O**ne **M**ind), a unified KD framework that enriches supervision with diverse temporal perspectives through mask-weighted teacher features and progressively aligns truncated predictions with the full-length prediction, thereby enabling more reliable inference across all timesteps. Extensive experiments and theoretical analyses demonstrate that MEOM achieves state-of-the-art performance on multiple benchmarks. Code is available at https://github.com/KaiSUN1/MEOM.

Published as a conference paper at ICLR 2026
MANY EYES, ONE MIND: TEMPORAL MULTI- PERSPECTIVE AND PROGRESSIVE DISTILLATION FOR SPIKING NEURAL NETWORKS
Kai Sun', Peibo Duan'*, Yongsheng Huang", Nanxu Gong", Levin Kuhlmann Department of Data Science and AI, Monash University, Melbourne, Australia 2 School of Software, Northeastern University, Shenyang, China
3 School of Computing and Augmented Intelligence, Arizona State University, Arizona, USA (pe beeib..doe duan, levin. kuhlmann mem.mmmmnnmmmaaahhhaa edu
ABSTRACT
Spiking Neural Networks (SNNs), inspired by biological neurons, are attractive for their event-driven energy efficiency but still fall short of Artificial Neural Networks (ANNs) accuracy. Knowledge distillation (KD) has emerged as a promising approach to bridge this gap by transferring knowledge from ANNs to SNNs. Temporal-wise distillation (TWD) leverages the temporal dynamics SNNs by providing supervision across timesteps, but it applies constant teacher output to all timesteps, mismatching the inherently evolving temporal process SNNs. Moreover, while TWD improves per-timestep accuracy, truncated infer- ence still suffers from full-length temporal information loss due to the progres- sive accumulation process. We propose MEOM (Many Eyes, One Mind), unified KD framework that enriches supervision with diverse temporal perspec- lives via mask-weighted teacher features and progressively aligns truncated pre- dictions with the full-length prediction, thereby enabling reliable inference at arbi- trary timesteps. Extensive experiments and theoretical analyses demonstrate that MEOM achieves otate-of-the-ee art performance on multiple benchmarks. Code is available https: / thu.. com/ Ka SUNI /MROM
INTRODUCTION
Spiking Neural Networks (SNNs), often regarded as the third generation of neural models (Maass 1997]. differ from traditional Artificial Neural Networks (ANNs) by transmitting information through discrete, sparse, and temporal spikes. The asynchronous, event-driven nature of SNNs enables efficient spatio-temporal pattern encoding with low energy consumption, making them well- suited for deployment resouree-limited platforms such edge devices and neuromorphic hard- ware (Indiveri & Liu 2015) Davies et al., 2018). Operating over discrete timesteps, an SNN incre- mentally processes input spikes and integrates them temporally into a decision.
Despite these advantages. SNNs typically face a performance gap compared with ANNs, which achieve high accuracy through precise maating-point operations (Roy al.) (209)). To narrow this gap, two main strategies have been explored. The first is ANN-to-SNN conversion, which transfers accuracy effectively but often requires a large number of inference steps, limiting practicality under latency or energy constraints (Rueckauer et al. 2017 Bu et al. (023). The second is direct training with knowledge distillation (KD), where methods such as KDSNN (Xu et al., 2023) and LaSNN (Hong et al. 2023) distill features logits from teacher ANN, enabling competitive results with tewer timesteps. However, these approaches largely overlook the intrinsic temporal dynamics of SNNs. To address this, temporal-wise distillation (TWD) (Yu et al. 2025a was proposed, which treats outputs across timesteps as a temporal ensemble and supervises each timestep directly. By improving the accuracy of individual timesteps, TWD enhances both overall accuracy and truncated inference performance, which is required under strict latency and energy constraints but inevitably entails accuracy degradation (Li et al. 2023b
"Corresoonding author

---

Published as a conference paper at ICLR 2026
A Probability Density
0.4
T- T-2
T-3
T-4
Logits
-0.4-
-0.8
T-1
T-2
-2
-1.2
2
8
0.0
0.4
0.6
1.0
Index
Membrane Potential (mV)
(a) Diffcrential logits relative to
T= 1.
(b) Differential voltage distribution rolative to T
Figure 1: Visualization of the temporal difference in SNNs.
Nevertheless, in TWD the teacher ANN's outputs serve as constant supervisory signal across all timesteps, while the SNN itself evolves dynamically. In practice, outputs across timesteps do not remain identical, as discussed in Appendix(A] The final logits themselves also exhibit temporal varia- tion, as illustrated in Figure Therefore, relying fixed ANN target to supervise every timestep may not fully capture the inherent temporal dynamics SNNs. A more effective strategy would incorporate diverse temporal supervisory signals, aligning more naturally with the time-varying be- havior of SNNs, analogous to "many eyes", focusing on different temporal perspectives.
Meanwhile, truncated inference introduces another challenge. As shown in Figure 1b membranc potentials at timesteps. As a result, truncated inference only exploit these early and incomplete signals, in- evitably leading to accuracy degradation. Although TWD improves accuracy at individual timesteps, truncated predictions still suffer from temporal information loss. To mitigate this issue, it is impor- tant to amplify the density of information within the truncated timesteps during training, allowing the predictions approximate those obtained from full-timestep inference. Regardless of the extent of truncation, the intended outcome is that the model produces identical predictions, analogous to "one mind", ensuring consistency across both truncated and full-length inference scenarios.
are concentrated at lower values, and they progressively accumulate at later
The absence of "many eyes" and "one mind" in KD-based SNN training arise from insufficient learning along the temporal dimension. To address these limitations, we introduce Temporal Multi- Perspective Distillation (TMPD), generating diverse teacher outputs from a single ANN to pro- vide heterogeneous supervision that captures diverse temporal perspectives, Temporal Progressive Distillation (TPD), enforcing predictions at each timestep to progressively align with the full-length prediction, ensuring temporal flexibility under truncated inference. To- gether, TMPD and TPD form unified framework, MEOM (Many Eyes, One Mind). The main contributions of this work are threefold.
the complementary
We introduce MEOM, a unified KD framework that integrates TMPD to provide diverse temporal supervision and TPD to align truncated predictions with the full-length prediction, enabling reliable inference across all timesteps.
We provide theoretical analyses showing that MEOM achieves stable convergence across all timesteps, leading to higher accuracy and better generalization.
We validate MEOM through comprehensive experiments on 10, CIFAR-100, and ImageNet, achieving state-of-the-art performance, together with ablation studies and anal- yses of energy efficiency and visualization.
RELATED WORK
2.1 KNOWLEDGE DISTILLATION FOR SNNS
Knowledge distillation (KD), which transfers supervision from a nigh-performing teacher to a stu- dent, has become a widely adopted strategy for training SNNs without resorting to ANN-to-SNN conversion. Early ANN-to-SNN distillation studies leveraged ANN teachers to provide interme-
diate feature guidance and/or logit targets (Xu al. 2023 Hong al. 2023
Qiu al. 2024
2

---

Published as a conference paper at ICLR 2026
2023 Xu al. 2024b), achieving competitive accuracy with fewer approaches. Building on this, SNN-to-SNN distillation emerged, such as using longer-timestep SNN to guide shorter one via temporal-spaie self- distillation (TSSD) (Zuo et al. 2024), or transferring compact yet informative spike representations from pruned model in Sparse- KD (Xu al. 2024a), enabling latency reduction without sacrificing representational quality. In parallel, TWD treats the spiking sequence as a collection of temporal sub-networks and applies temporal-wise supervision from single ANN teacher, improving both per-timestep accuracy and the final output Yu et al., 2025b Konstantaropoulos et al. (025). De- spite these advances, it supervises all timesteps with the same teacher signal, overlooking the intrin- sic temporal diversity of spiking dynamics and tending to homogenize intermediate predictions. By contrast, the ANN, multi-teacher ensemble distillation has been shown to improve both general- ization and robustness by aggregating complementary inductive biases from teachers with different architectures or parameterizations (Zhang et al., 2018: Wen et al., 2024 Yang et al. 2025). Moti- vated by this principle, design a distillation strategy where each temporal sub-network is guided by distinct teacher features reweighted by masks, restoring diverse temporal perspectives ("many eyes") while achieving the effect of multi-teacher ensembles without training multiple teachers."
Hong & Wang 2025 Guo et al. timesteps compared to conversion
2.2 TIME FLEXIBILITY FOR SNNS
SNNs accumulate information over multiple timesteps, but real-world neuromorphic deployment often demands low-latency and energy-efficient inference, making it essential to preserve accuracy when the number of available timesteps reduced. This has motivated methods that improve tempo- ral flexibility from complementary perspectives. Adaptive truncation approaches, such as SEENN (Li et al. 2023c) that terminates inference based on confidence (SEENN-I) or a learned policy thresholding (Li et al.) 2023a). dynamically decide when to stop. Training-based strategies aim to strengthen early-timestep predictions without dynamic con- trol, including HSD (Zhong et al. (024) with task-specific fine-tuning, MTT (Du et al. 2025) that
(SEENN-IL II), and prretoffront-basee
treats timesteps as random variables for augmentation, and SSNN (Ding et al. (022)) that progres- sively reduces the number of timesteps lower latency. Consistency-orientee methods further encourage stable predictions across timesteps, such as viewing each timestep as an independent sub-model for ensembling (Yu al., (025a). enforcing all all-pair consistency (Zhao al. (025),, or aligning temporally adjacent sub-networks (Ding et al. 2025). However, these approaches still rely on pairwise or stepwise consistency, which does not guarantee that early truncated predictions ap- proximate the full-length prediction. As a result, when inference is stopped early. the model may produce immature or inconsistent results. To address this issue, we propose strategy that progres- sively aligns truncated outputs with the full-length prediction ("one mind"), ensuring that even under truncated inference, the outputs converge toward the same prediction.
3 METHODOLOGY
3.1 PRELIMINARIES
Two strategies are commonly used for ANN-to-SNN distillation: 'eeppora--veragee Distillation (TAD), aligning with the average prediction, and Temporal- Wise Distillation (TWD), aligning with each timestep output, as shown in Figure 2a and 2b
Temporal-Averagee Distillation (TAD). Let C' denote the number of classes and T the number of discrete timesteps. At each step E {l...., ., T3. the student SNN outputs a pre-softmax logit vector S C.C. These logits are averaged across time to produce single representative vector Save T E z The student probability distribution is given by the temperature-scatt
softmax with exp(x, 9ars /r)
0 as the temperature parameter: Snw (o
Supervision
Eg exp(s, (ave (r)
on the aggregated output includes hard-label and soft-label components. The hard-label loss uses cross-entropy with ground- truth y (0, 1)::
C
LIAD-CE = CE(p Swe
Σ
Vi log P.
Save.
(1)
3

---

Published as a conference paper at ICLR 2026
Clucffration fead
ANVY
Loghle
JNNY
Legit:
Ciosifostion
ClouniRcation
XNW
SUN
Inpul
Akergee oupul
oages
CE
(a) Temporal-Averaged Distillation
(b) Temporal- Wise Distillation
TMPD
Franurr
Classtiaaiion Joad
Legle
Apper
Clasoication
Arangy
Original Faston
Zmanlogee
TPD
Avrrggy
Anorae Til
(c) MEOM (Ours)
Truo label
Annage T1:
Figure 2: Comparison of temporal knowledge distillation approaches: (a) Temporal- Averrage Dis- tillation, (b) Temporal-Wol Wise Distillation, and (c) Our Proposed MEOM with Related Components.
Soft-label supervision uses the Kullback Leibler (KL) divergence to align the student's 's distribution with that of the teacher. Let pd denote the teacher ANN's probability distribution. The KL loss is
C
STAD-KI (IoSSa) Σ
P
log
P
Save
Σ
p. logp,
Suva
Suve
H)))),
i=1
where the entropy term - -i P." log p: is constant with respect to the student param- eters and is omitted during training. The total TAD loss combines both terms,
LAAD a LTAD-CE + + LTAD-KL.
(3)
where a and are hyperparameters controlling the trade-off between hard-label and soft-label.
-emporal-Wise Distillation (TWD). TWD leverages the temporal nature of SNN outputs by su- pervising the student network at each timestep rather than collapsing temporal information into an aggregated vector as in TAD. Let p denote the probability distribution of the student at timestep t. By supervising each timestep directly, this fine-grained strategy enhances per-timestep accuracy, resulting in more stable outputs and more reliable final prediction. Formally, both hard-label and soft-label losses are applied at every timestep and then averaged over all T steps:
LTWD-CE T
1
CE PE(p..y) y)
1
Σ
Σ
Ue log pi,
(4)
T
1=1
t=1
T
LTWD-KL
Σ
1
p log
logp's
(5)
T
T
1
The final loss combines the two objectives
LTWD LTWD.CE LTWD.L.
4

---

Published as a conference paper at ICLR 2026
3.2 TEMPORAL CULTI-PERSPECTIVE AND PROGRESSIVE DISTILLATION
While TWD applies identical teacher targets to all timesteps and optimizes them in isolation, it ne- glects both temporal diversity and truncated information loss. To address this, we introduce MEOM, which integrates TMPD for diverse temporal-wer wwise supervision and TPD for progressive alignment, as shown in Figure 2c
Temporal Multi- Perspiperr Distillation (TMPD). SNNs produce a sequence of outputs over time, reflecting their intrinsic temporal dynamies. Conventional KD, however, reuses single static teacher output to supervise all timesteps, which cannot adequately capture the temporal variation in SNN predictions. To overcome this limitation without relying on multiple pre-trained teach- ers, TMPD draws inspiration from multi-teacher distillation and introduces lightweight mask transformations different timesteps. These transformations diversiffy the teacher signals while pre- serving semantic consistency, thereby providing richer and temporally aligned supervision for the student across timesteps.
For each timestep t E {1...., T) we add a lightweight time-indexed mask m. ensuring semantic consistency to form masked feature f.* f my, where @ denotes the Hadamard product. Passing both the original and masked features through the same classifier yields the original logits AA and the masked logits E This process can be written
= W Wf fA²) W.S + Walf* ④ m: =4 8z
=:
where W. is the classification weight matrix and 5z²² = WolfA @ m.) denotes the perturbation to the teacher logits timestep f. Finally, the original and masked logits linearly combined to produce temporal-wise perturbed teacher logits:
(8) temperature-scalle softmax is applied to 2 to obtain the teacher distribution D which is com- pared with the student distribution p at each timestep using KL divergence averaged over T steps: T
2 = -1 A)= +
T c
P
L KL
Σ T
kL 11 p
Σ
log Pi.
T
In addition to this KL term, TMPD also adopts the temporal-wise cross-entropy loss LTWD-CE (Eq. 4p. By introducing mask weighting, TMPD delivers a form of multi-perspective supervision ("many eyes"). where each timestep receives a slightly perturbed but semantically consistent teacher signal, in contrast to TWD's s identical targets across steps.
Temporal Progressive Distillation (TPD). In neuromorphic deployment, SNNs are often con- strained to truncated timesteps due to strict latency and energy requirements, forcing predictions to be made with incomplete temporal evidence. Under such conditions, TAD averages outputs and dilutes discriminative signals, while TWD improves per-timestep accuracy but does not ensure that earlier predictions converge toward the full-length prediction, leading immature incon- sistent results under truncation. Direct alignment with the full-length output is unstable due to the large discrepancy, so TPD aligns cumulative average predictions of consecutive timesteps, provid- ing smoother guidance and ensuring progressive convergence to a unified decision. We compute the running mean logits S and probabilities p as
P softmax, (ES.).
(10)
TPD encourages the cumulative prediction up to step to align with that of step + 1, driving predictions to converge progressively over time. Specifically, for every consecutive pair (t, 1) with E (1....,77 T 1}. we define
1 T-1 C
1 CE(P. - set+.c log P
CTPD
T
(11)
Using the later window as the CE target proved more stable than the KL. In this way, the distillation process progressively transfers information from more stable cumulative averages to earlier, noisier predictions, guiding outputs toward the full-length prediction ("one mind") and ensuring reliable prediction under truncated inference.
5

---

Published as a conference paper at ICLR 2026
Overall Training Objective. The overall loss combines the standard classification loss with TMPD and TPD into a single objective:
aal = a CTWD-CE + 8 LIMPD-LL + CTPD.
(12)
where we set a 1, B 0.5, and 7 = 0.3.
3.3 THEORETICAL ANALYSIS
To theoretically substantiate our findings, we highlight three key properties. First, TMPD intro- duces an implicit regularization effect on the distillation objective (Theorem []. which strengthens the temporal supervision by preventing the student from overfitting to individual timestep logits. Second, this stronger supervision allows TMPD to achieve a strictly lower final error floor than TWD, thereby yielding higher ultimate accuracy (Theorem 2 Third, TPD mitigates the accuracy drop under truncated inference, outperforming no-consisteney (NC), which ignores temporal align- ment, and neighbor-step consistency (NSC), which only matches adjacent steps without cumulative averaging (Theorem 3
Theorem (Regularization Effect of TMPD). TMPD induces an implicit regularization effect on the distillation objective.
Theorem 2 (Convergence Robustness Bound of TMPD). The final ervor of TMPD, denoted by ēTMPD, forms strictly lower bound of STWD under the same learning rute, as:
ETMPD < ETWWD
(13)
The proof is provided in Appendix [ and B.2 respectively. Together, these results show that TMPD introduces an implicit regularization effect by perturbing the teacher logits each timestep (Theorem I). Instead of matching a single deterministic teacher value, the student must remain con- sistent over local neighborhood perturbed teacher signals, resulting in denser and thus stronger temporal supervision. This stronger supervision reduces the correlation of gradient contributions across timesteps (Theorem []. lowers the effective variance without introducing significant bias, and ultimately leads TMPD to converge to strictly lower error floor and achieve higher accuracy.
Theorem (Truncation Robustness Bound TPD). Denote by ( the truncation accuracy drop at step & < T. Then the truncation accuracy drop of TPD is strictly smaller than that of NSC, and NSC is strictly smaller than NC:
A
TPD(() aCc
<
A NSC acc (1)
< A NC acc
(t).
(14)
The proof is provided in Appendix B.3 For any truncation step t, the bound establishes a strict ordering where TPD yields a smaller accuracy drop than NSC, which in turn outperforms NC. This guarantees reliability and higher accuracy under truncated inference. TMPD and TPD provide com- plementary benefits since TMPD improves full-length accuracy through richer temporal supervision and reduced gradient variance, while TPD enhances truncated inference by exponentially bound- ing accuracy loss. When combined, the stronger final accuracy achieved by TMPD tightens TPD's bound and yields consistently superior performance across both full-length and truncated timesteps.
EXPERIMENTS
In this section, we evaluate the effectiveness of our proposed model across different datasets and net- work architectures, followed by an examination of its temporal flexibility under truncated timesteps. We then conduct ablation studies to analyze the impact of key components, and conclude with an assessment of energy efficiency and supporting visualizations.
4.1 EXPERIMENTAL SETTINGS
Experiments are conducted on nodes with AMD EPYC 7742 CPUs (128 cores, 2.25 GHz) and NVIDIA A100 GPUs, with GPU count adjusted according to the experiment scale. LIF neurons with surrogate gradient backpropagation (Huh & Sejnowski 2018) are implemented using SpikingJelly (Fang et al. 2023). We evaluate the proposed model ResNet- 18/19 and Spiking-Transfonn Transformer architectures over CIFAR- and ImageNet. Reported results are averaged over three runs for reliability. Detailed settings are provided in Appendix

---

Published as a conference paper at ICLR 2026
Table Top-1 accuracy (%) on CIFAR-10/100 with and without KD across different timesteps T.
METHOD
MODEL
T= 2
4
T =
TET Deng et al. 2022
ResNet-19
94.16 72.87
94.44 / 74.47 74.47
94.50 / 74.72
STBP-tdBN Zheng al. 2021
ResNet-19
92.34 69.41
93.16/71. 92.92/70.86 70.86
Dspike Li et al. 2021
ResNet-18
93.13/71.68
93.66 73.35
94.25 74.24
w/o KD
ResNet-18
94.15/74.
94.67 1667/70 76.42
94.88 / 77.28
GLIF Yao et al.
2022
ResNet-19
94.44/75..
94.85 77.05
95.03 / 77.35
ReeDis-SNN Guo al.
2022
ResNet-19
93.64
95.53/74.10
95.55
ResNet-18
75.97 94.75
95.61 78.26
95.90 79.02
RateBP Yu et al.
2024
ResNet-19
96.23 79.87
96.26 / 80.71
96.36 80.83
KDSNN Xu et al.
2023
ResNet-18
93.41
ResNet-18
75..79 94.01
95.45 77.39
Joint A-SNN Guo al.
2023
ResNet-34
95.13 777111
96.07 79.76
KD
BKDSNN (uu et al. 2024b)
ResNet-19
94.95 /4.95/4455 74.95
HTA-KL Zhang et al.
2025
ResNet-19
80.51 96.68
96.51
96.76 81.03
ResNet-18
95.11
95.57
79.10
95.96
79.80
TWSNN Yu et 2025a
ResNet-19
96.65 / 81.47
96.97
82.47
97.00 / 82.56
ResNet-18
95.51 / 77.99
96.07 79.66 10.05
96.36 / 80.42
1:0.66 + 0.08
1. 0.04
10.10 1 0.06
MEOM (Ours)
ResNet-19
96.65 81.82
97.13 82.85 10.07
97.08 83.22 10.08
10.11 + 0.05
+ 0.04
1 0.14
ANN
ANN
ANN
50
82
96
92
KDSNN
MuilllSNN
74
76
KDSNN
VanileSNN
95
KDSNN
VarilaSNN
KDSNN
VanilANNN
Oues
TWSNN
Ous
TWSNN
Our.
TWSNN
76
Curs
TWSNN
2
6
2 3
6
2
3
Inference Timestep
Inference Timestop
Inftrence Timestep
Infereace Timestep
(a) C10-RI8
(b) C100-RI8
(c) C10-R19
(d) C100-RI9
Figure 3: Inference accuracy when models are trained with T = 6 and evaluated with truncated timesteps (T 1-5). Here, C10/C100 denote CIFAR- and R18/R19 denote ResNet-18/19.
4.2 PERFORMANCE COMPARISON
For CIFAR-10 and CIFAR-100 (Table D. MEOM consistently surpasses directly-trained and distillation-based SNN methods across timesteps and model depths. On CIFAR-100, it even slightly outperforms the ANN teacher and achieves clear gains over the strongest SNN baselines, while under low-latency settings (T 2) it still maintains competitive performance. On ImageNet (Ta- 2). MEOM generalizes effectively across architectures, from convolution- based ResNet-34 to the spiking transformer model, consistently outperforming all existing SNN baselines under the same timestep budget and narrowing the gap to AÑN teachers. By overcoming the limitations of a single repeated teacher and isolated timestep training, MEOM provides more informative temporal guid- ance, enabling the student to learn richer intermediate representations. Combined with consistent gains on both small- and large-scale datasets, this enhanced representational transfer confirms the effectiveness of our method.
4.3 TIME FLEXIBILITY
To assess the time flexibility of our approach, we trained all SNNs with a full six tix-timestep schedule (T = 6) on CIFAR- 10 and CIFAR-100 using ResNet backbones, and compared against VanillaSNN, KDSNN, and TWSNN. In evaluation, the rollout was progressively truncated from T = 5 down to
T = 1, shown in Figure MEOM consistently achieved the highest accuracy across all truncation
7

---

Published as a conference paper at ICLR 2026
Table 2: Top-1 accuracy (%) on ImageNet with and without KD. All methods evaluated
Timestep 7 = 4. S-8-384 denotes the Spiking Transformer architecture encoder layers and 384-dimensional hidden states.
(Zhou et al. 2023) with
MODEL
METHOD
ResNet-34
S-8-384
TET Deng al.
2022
68.00
Dspike (Li
2021
68.19
GLIF Yao et al.
2022
67.52
w/o KL
RecDis-SNN Guo et al. 2022)
67.33
RateBP Yu ct al. 2024)
70.01
Spikformer Zhou et
2022
70.24
Spikingformer (Zhou et al.
2023
72.36
Spike-driven Transformer
Yao et al.
2023
72.28
KDSNN Xu et al.
2023)
67.18
74.62
w/ KD
LaSNN Hong et al.
2023
66.94
73.85
BKDSNN Xu et al. 2024b
67.21
75.48
TWSNN Yu et al.
2025a)
71.04
MEOM (Ours)
71.64
76.77
Table 3: Ablation of components on CIFAR-10/100 with different T, with temporal variance mea- sured T on CIFAR-10.
TAD
TWD
TMPD
TPD
T=
T= 4
T=
Var 4
X
X
X
X
94.67 75.16
95.00/77.22
95.74/78.37
0.493
X
X
X
94.97 76.01
95.31
96.09 79.28
0.485
X
J
X
X
95.11/77.55
95.57
96.16/79.91
0,212
X
J
V
X
95.43/77.78
95.98 79.58
96.31 80.13
0.133
X
V
X
V
95.35/77.72
75.84/79.50
96.20 80.05
0.199
X
V
V
95.51 77.99
96.07 79.66
96.36 80.42
0.150
points, and on CIFAR-100 with ResNet-19 at T it even surpassed the ANN teacher. At 1, although accuracy inevitably declined, it still outperformed all baselines. These results demonstrate the effectiveness of TPD guiding early predictions to converge toward the full-length prediction, thereby enabling reliable prediction under truncated inference.
4.4 ABLATION STUDY
Ablation on TMPD and TPD. To evaluate the effectiveness of our proposed components, we con- duct an ablation study on CIFAR-10 and CIFAR-100 with ResNet-18 with different T, shown in Table Starting from the TAD and TWD baselines, progressively add TMPD and TPD. Incorpo- rating these components consistently improves accuracy across datasets and timesteps. addition to improving accuracy, these modules also shape the model's temporal variance, which measures how much the outputs change across timesteps. TMPD reduces variance by smoothing noisy devi- ations, and TPD lowers variance by improving the consistency of temporal predictions. When used together, the two components preserve meaningful temporal differences while limiting unnecessary variation, and this balanced temporal behavior leads the highest accuracy in all settings.
Ablation on Temporal Step Selection in TPD. Our method, denoted as Full, applies temporal consistency across all six timesteps, with the consistency loss computed as the cumulative average over the selected steps. For the reduced-ster step configurations, three timesteps). F-6-R-2 (always using t 6 plus two random steps), and F-2-4-6 (using 6). As shown in Table Full achieves the highest accuracy, indicating the benefit of leveraging
evaluate R-3 (randomly choosing
8

---

Published as a conference paper at ICLR 2026
Table 4: Comparison of different temporal step selection strategies in TPD.
R-3
F-6 R-2
F-2-4-6
F-6 A-5
NSC
FULL
CIFAR-10
96.19
96.24
96.14
95.99
96.19
96.36
CIEAR-100
80.30
80.35
80.16
80.11
80.36
80.42
Table 5: Ablation study of hyperparameters and y..
8 wwith 7
0.1
0.3
0.5
0.7
0.9
ACC (%)
95.62 79.33
79.33
79.33
/ /9.4/79.554
95.87 / 79.35
95.89
79.35
7 with 3 = 0.5
0.1
0.3
0.5
0.7
0.9
ACC (%)
95.90 79.56
96.07 79.66
95.95
79.56
95.85 79.33
95.69 / 79.46
all steps. Among the reduced-step variants, F-6-R-2 performs best, suggesting that coupling the final step with varied earlier ones is more effective than relying on fixed subset. In addition to these reduced-step variants, include two broader baselines; F-6 A-5 (comparing the prediction at t = with all five earlier timesteps) and NSC (comparing only adjacent timestep pairs). Both F-6 local-only comparison
5 and NSC perform worse than Full, indicating that neither global-only sufficient. Using all timesteps in a progressive manner yields the most stable temporal behavior and the best overall performance.
Ablation hyperparameters. Table reports the hyperparameter ablation under the protocol where we first fix a = to select the optimal 33, and then fix this B to determine the best 7- Introducing TMPD with the selected 3 consistently improves accuracy over the TWD bascline, indicating that the hyperparameters produce stable and predictable effects. We also note that the two modules interact with each other, and selecting a compatible set of hyperparameters plays key role in obtaining the best performance. We adopt = 0.5 and y = 0.3 in the experiments.
4.5 ENERGY EVALUATION AND VISUALIZATION
Energy Evaluation. As shown in Table the firing rate (FR) of MEOM slightly higher than that of the other methods, and its difference from TWSNN is regarded as normal fluctuation. Temporal- wise comparison requires each step to contribute predictive evidence rather than concentrating at later steps, which shifts some activity earlier and slightly increases the overall FR. Energy is computed from accumulation (ACs) and multiply-accumulate (MACs), and it grows correspond- ingly with the increase in firing rate. The increase in training time (TT, per batch) and GPU hours mainly comes from TMPD, as introduces temporally masked teacher logits and multiple KL- divergence computations across timesteps, with no increase in GPU memory, and this remains much cheaper than training multiple teachers. The detailed energy calculation procedure provided in Appendix D. and additional experiments shown in Appendix
Visualization. We utilize t-SNE visualize features CIFAR-108 100, with ResNet-34 the teacher and ResNet-18 as the student. As shown in Figure 4 our method generates more compact and well-separated clusters compared to prior SNN distillation methods, indicating better feature discriminability and knowledge transfer. In contrast, other methods show more overlapping clus- ters.
5 DISCUSSION
To address the limitations of existing SNN distillation methods that rely on single teacher signal and overlook full temporal information under truncated inference, we propose MEOM, a unified framework that fully exploits the temporal dimension of SNNs. MЕOM integrates TMPD, which introduces diverse teacher supervision to capture multiple temporal perspectives, and TPD, which progressively aligns truncated predictions with the full-length prediction to ensure temporal flex-

---

Published as a conference paper at ICLR 2026
Table 6: Comparison of energy consumption, computation operations, training time, GPU hours,
GPU memory, and accuracy on CIFAR-10 with ResNet-18 18 with timestep T 6.
FRs (%)
ACs
MACs
ENERGY
TT
GPU h
GPU Mem
ACC
METHOD
(M)
(M)
(nJ)
(ms)
(h)
(GB)
(%)
ANN
0.557
549.1
1757.2
1.07
1.99
97.06
VanillaSNN
14.21
72.500
3.342
17.946
123
4.25
11.50
95.74
KDSNN
14.90
78.641
3.342
18.560
129
4.45
11.88
95.85
TWSNN
15.66
83.915
3.342
19.087
130
4.49
11.88
96.16
Ours
15.98
85.644
3,342
19.260
163
5.63
11.88
96.28
ANN
Vanilla SNN
KDSNN
LBSNN
Ours
Figure 4: I-SNE visualization of feature representations learned by teacher ANN, vanilla SNN, and various KD methods.
ibility. Extensive theoretical analyses and experiments on multiple benchmarks demonstrate that MEOM substantially improves both accuracy and time flexibility.
Implications for "one mind". For the declaration of express, the final full-length output chosen as the "one mind." In fact, enforcing single consistent output across timesteps is both pursuit and a necessity for SNNs, since using fewer timesteps corresponds to lower energy consumption. When the number of timesteps becomes large, accuracy does not always increase monotonically, and earlier timesteps may even outperform the final output. In such cases, the one-mind princi- earlier
ple still holds, the "one mind" can be interpreted the best output that emerges timestep. At the population level, some prefix satisfies q1.- q- The alignment constraints im- pose ( (1:44)) O, yielding æ1.1
QT.. Since one prefix matches q", all must, giving h.r = q²' Thus the final average is Bayes-optimal because any global minimizer collapses all prefix averages to the same optimal predictor.
Learnable mask for distillation. Although TMPD diversifies supervision, the masks at each timestep are randomly sampled and therefore do not adapt to the evolving temporal dynamics of spiking sequences. Intuitively, learnable masks could better capture fluctuations in the student's s pre- dictions. To evaluate this hypothesis, we compared a learnable parameter mask, a gating mask, and gumbel-sigmoid mask with random masks, as shown in Table 7 Our results show that random masks consistently achieve the best and most stable performance. We argue that learnable masks tend to adapt to the student's intermediate predictions, which are not yet reliable and therefore intro- duce biased perturbations. In contrast, simple random masks offer lightweight, unbiased temporal perturbations that better serve the purpose of TMPD. Future work may explore temporally aware models that more effectively capture the underlying temporal variations.
Table 7: Comparison of different mask operations CIFAR-10/ CIFAR-100.
Mask Operation Random (Ours)
Parameter
Gating
Gumbel-Sigmoid
Accuracy (%)
96.07 / 79.66
95.99 79.34 95.85 0855/79.38
95.89/79.44 79.44
ACKNOWLEDGMENTS
The work supported in part by start-up funds with No. MSRI8001004 and No. MSRI9002005.
10

---

Published as a conference paper at ICLR 2026
REFERENCES
Ahmad Ajalloeian and Sehastian U Stich. On the convergence of sgd with biased gradients. arXiv preprint arXiv:2008. 00051, 2020.
Tong Bu, Wei Fang, Jianhao Ding, PengLin Dai, Zhaofei Yu, and Tiejun Huang. Optimal ann- snn conversion for high-accuracy and oltra-low-latency spiking neural networks. International Conference Learning Representations, 2023.
Mike Davies, Narayan Srinivasa, Tsung- Man Lin, Gautham Chinya, Yongqiang Cao, Sri Harsha Choday, Georgios Dimou, Prasad Joshi, Nabil Imam, Shweta Jain, et al. Loihi: A neuromorphic manycore processor with on-chip learning. IEEE Micro, 38(1):82- 99, 2018.
Shikuang Deng, Yuhang Li, Shanghang Zhang, and Shi Gu. Temporal efficient training of spiking neural network via gradient re- re-weighting. In International Conference on Learning Representa- fions, 2022.
Luc Devroye, László Györfi, and Gábor Lugosi. A probabilistic theory of pattern recognition, vol- ume 31. Springer Science & Business Media, 2013.
Yongqi Ding, Lin Zuo, Mengmeng Jing, Pei He, and Yongjun Xiao. Shrinking your timestep: To- wards low-latency neuromorphic object recognition with spiking neural networks. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 11811-11819, 2024.
Yongqi Ding, Lin Zuo, Mengmeng Jing, Pei He, and Hanpu Deng. Rethinking spiking neural networks from an ensemble learning perspective. International Conference on Learning Repre- sentations, 2025.
Kangrui Du, Yuhang Wu, Shikuang Deng, and Shi Gu. Temporal flexibility in spiking neural net- works: Towards generalization across time steps and deployment friendliness. International Con- ference Learning Representations, 2025.
Wei Fang, Yanqi Chen, Jianhao Ding. Zhaofci Yu, Timothée Masquelier, Ding Chen, Liwci Huang, Huihui Zhou, Guoqi Li, and Yonghong Tian. Spikingjelly: An open-source machine learning infrastructure platform for spike-based intelligence. Science Advances, (40) 1480, 2023.
Robert M Gower al. Sgd for structured nonconvex functions: Linear convergence stationary point. In Proceedings of the 38th International Conference on Machine Learning, pp. 3799-3808. PMLR, 2021.
Yufei Guo, Xinyi Tong, Yuanpei Chen, Liwen Zhang, Xiaode Liu, Zhe Ma, and Xuhui Huang. Recdis-snn: Rectifying membrane potential distribution for directly training spiking neural net- works. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition,
pp. 326 326-335, 2022.
Yufei Guo, Weihang Peng, Yuanpei Chen, Liwen Zhang, Xiaode Liu, Xuhui Huang, and Zhe Ma. Joint a-snn: Joint training of artificial and spiking neural networks via diff-disillation and weight factorization. Pattern Recognition, 142:109639, 2023.
Song Han, Jeff Pool, John Tran, and William Dally. Learning both weights and connections for efficient neural network. Advances in neural information processing systems, 28, 2015.
Di Hong and Yueming Wang. Self-attentive spatio-temporal calibration for precise intermediate layer matching in ann-to-snn distillation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pp. 17240-17248, 2025.
Di Hong, Jiangrong Shen, Yu Qi, and Yueming Wang. Lasnn: Layer-wise ann-to-smm distillation for effective and efficient training in deep spiking neural networks. arXiv preprint arXiv:2304.0910, 2023.
Dongsung Huh and Terrence J Sejnowski. Gradient descent for spiking neural networks. Advances in neural information processing systems, 31, 2018

---

Published as a conference paper at ICLR 2026
Giacomo Indiveri and Shih-Chij Liu. Memory and information processing in neuromorphic systems. Proceedings of the IEEE, 103(8):1379-1397 2015.
Orestis Konstantaropoulos, Theodoris Mallios, and Maria Papadopouli. Dynamic activa- tion with knowledge distillation for energy-efficient spiking nn ensembles. arXiv preprint arXiv:2502. 14023, 2025.
Chen Li, Edward G Jones, and Steve Furber. Unleashing the potential of spiking neural networks with dynamic confidence. In Proceedings of the IEEE/CVF International Conference on Com- puter Vision, pp. 13350-13360, 2023a.
Chen Li, Edward G Jones, and Steve Furber. Unleashing the potential of spiking neural networks with dynamic confidence. In Proceedings of the IEEE/CVE International Conference Com- puter Vision, pp. 13350-133333 13360, 2023b.
Yuhang Li, Yufei Guo, Shanghang Zhang, Shikuang Deng, Yongqing Hai, and Shi Gu. Differ- entiable spike: Rethinking gradient-descent for training spiking neural networks. Advances in neural information processing systems, 34:23426 24-23426-23439, 2021.
Yuhang Li, Tamar Geller, Youngeun Kim, and Priyadarshini Panda. Seenn: Towards temporal spiking early exit neural networks. Advances in Neural Information Processing Systems, 36: 63327-63342, 2023c.
Wolfgang Maass, Networks of spiking neurons: the third generation of neural network models. Neural networks, 1009):1559-1671, 1997.
Haonan Qiu, Munan Ning, Zeyin Song, Wei Fang, Yanqi Chen, Tao Sun, Zhengyu Ma. Li Yuan, and Yonghong Tian. Self-architectural knowledge distillation for spiking neural networks. Neural Networks, 178:106475, 2024.
Kaushik Roy, Akhilesh Jaiswal, and Priyadarshini Panda. Towards spike- based machine intelligence with neuromorphic computing. Nature, 575(7784)607-6177, 2019.
Bodo Rueckauer. Julia-Alexandis Lungu, Yuhuang Hu, Michael Pfeiffer, and Shih-Clis Liu. Con- version of continuous- deep networks to efficient event-driven networks for image classifi- cation. Frontiers neuroscience, 11:682, 2017.
Haitao Wen, Lili Pan, Yu Dai, Heqian Qiu, Lanxiao Wang, Qingho Wu, and Hongliang Li. Class incremental learning with multi-teacher distillation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 28443-28452, 2024.
Qi Xu, Yaxin Li, Jiangrong Shen, Jian K Liu, Huajin Tang, and Gang Pan. Constructing deep spiking neural networks from artificial neural networks with knowledge distillation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 7886-7895, 2023.
Qi Xu, Yaxin Li, Xuanye Fang. Jiangrong Shen, Qiang Zhang, and Gang Pan. Reversing structural pattern learning with biologically inspired knowledge distillation for spiking neural networks. In Proceedings of the 32nd ACM International Conference on Multimedia, pp. 3431-3439, 2024a.
Zekai Xu, Kang You, Qinghai Guo, Xiang Wang, and Zhezhi He. Bkdsnn: Enhancing the perfor- mance of learning-based spiking neural networks training with blurred knowledge distillation. In European Conference on Computer Vision, pp. 106-123. Springer, 2024b.
Chuanguang Yang. Xinqiang Yu, Han Yang, Zhulin An, Chengqing Yu, Libo Huang, and Yongjun Xu. Multi-teacher knowledge distillation with reinforcement lcarning for visual recognition. In Proceedings of the AAAI Conference Artificial Intelligence, volume 39, pp. 9148-91566 2025.
Man Yao, Jiakui Hu, Zhaokun Zhou, Li Yuan, Yonghong Tian, Bo Xu, and Guoqi Li. Spike-driven transformer. Advances in neural information processing systems, 36:64043-64055 2023.
Xingting Yao, Fanrong Li, Zitao Mo, and Jian Cheng. Glif: A unified gated leaky integrate-and neuron for spiking neural networks. Advances in Neural Information Processing Systems, 35: 32160-32171, 2022.
12

---

Published as a conference paper at ICLR 2026
Chengting Yu, Lei Liu, Gaoang Wang, Erping Li, and Aili Wang. Advancing training efficiency of deep spiking neural networks through rate-based backpropagation, Advances in Neural Informa- tion Processing Systems, 37:1 11755766-11111 2024.
Chengting Yu, Xiaochen Zhao, Lei Liu, Shu Yang, Gaoang Wang, Erping Li, and Aili Wang. Ef- ficient logit-based knowledge distillation of deep spiking neural networks for full-range timestep deployment. In Proceedings of the 42nd International Conference on Machine Learning. 2025a.
Kairong Yu, Chengting Yu, Tianqing Zhang. Xiaochen Zhao, Shu Yang. Hongwei Wang. Qiang Zhang, and Qi Xu. Temporal separation with entropy regularization for knowledge distillation in spiking neural networks. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 8806-8816, 2025b.
Tianqing Zhang, Zixin Zhu, Kairong Yu, and Hongwei Wang. Head-tail-aware kl divergence in knowledge distillation for spiking neural networks. arXiv preprint arXiv:2504.20445, 2025.
Ying Zhang, Tao Xiang, Timothy M Hospedales, and Huchuan Lu. Deep mutual learning. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4320-4328, 2018.
Dongcheng Zhao, Guobin Shen, Yiting Dong. Yang Li, and Yi Zeng. Improving stability and perfor- mance of spiking neural networks through enhancing temporal consistency. Pattern Recognition, 159: 1094, 2025.
Hanle Zheng, Yujie Wu, Lei Deng, Yifan Hu, and Guoqi Li. Going deeper with directly-trainee larger spiking neural networks. In Proceedings of the AAAI conference on artificial intelligence, volume 35, pp. 11022-11070, 2021.
Xian Zhong, Shengwang Hu, Wenxuan Liu, Wenxin Huang, Jianhao Ding, Zhaofei Yu, and Tiejun Huang. Towards low-latency event-based visual recognition with hybrid step-wise distillation spiking neural networks. In Proceedings of the 32nd ACM international conference on multime- dia, pp. 9828-9836, 2024.
Chenlin Zhou, Liutao Yu, Zhaokun Zhou, Zhengyu Ma, Han Zhang, Huihui Zhou, and Yonghong Tian. Spikingformer: Spike-driven residual learning for transformer-basee spiking neural net- work. arXiv preprint vrXiv:2304.11954 2023.
Zhaokun Zhou, Yuesheng Zhu, Chao He, Yaowei Wang, Shuicheng Yan, Yonghong Tian, and Li Yuan. Spikformer: When spiking neural network meets transformer. arXiv preprint arXiv:2209. 15425, 2022.
Lin Zuo Yonggi Ding, Mengmeng Jing, Kunshan Yang, and Yumqian Yu. Self-distillation learning based on temporal-spatial consistency for spiking neural networks. arXiv:2406.07862, 2024.
arXiv preprint
13

---

Published as a conference paper at ICLR 2026
A STRUCTURAL TEMPORAL FLUCTUATIONS IN LIF DYNAMICS
The leak-integrate-rrsst mechanism of LIF neurons inherently produces temporal fluctuations in membrane potentials, even under constant input. Consider a single LIF neuron receiving a fixed current Its membrane potential evolves as
U++1 = an + I - Vin St,
E {0,1},
(15)
where < is the leak factor and Vaa 0 is the firing threshold. any non-trivial fir- ing regime (neither always silent nor always firing), the neuron exhibits both spike and non-spike timesteps. Consequently, the update increments take two distinct values:
We+1 (a- 1) +
(no spike),
(16)
- U4 = (a 1) ur + I Vih
(spike).
(17)
These two increments differ by exactly Vín > 0, so the sequence {ue} cannot remain constant over time and must visit at least distinct membrane-potente levels with non-zero frequency. This implies strictly positive temporal variance, showing that temporal fluctuations arise naturally from the LIF update rule.
B PROOF OF THEOREMS
B. REGULARIZATION EFFECT OF TMPD
Theorem 1. TMPD induces an implicit regularization effect on the distillation objective.
Proof. Under TWD, the teacher provides a fixed logit vector A4, and the student minimizes
LTWD A
(18)
where ((.) denotes the distillation loss (KL divergence).
Under TMPD, the teacher logit is perturbed by a temporal mask:
3 (1 + Am)) = z +E,
(19)
where is a zero-mean random mask vector and 2 @ (Am) satisfies B[e] 0. Since mask is drawn at every iteration, the student minimizes the expected loss
LTMPD = [E( + ++)]
(20)
For distillation losses with KL divergence, f(-) convex with respect to the teacher logit vector. Thus, Jensen's inequality yields
((= = [EE(= + +e)) s <((e +c)
(21)
Consequently, the TMPD objective be decomposed
CTMPD = {(=A) + [E[((z> + +e]] {((=1) LTWD + R(zA),
(22)
where the residual term R(2A) 0. This quantity measures the increase in loss under small pertur- bations of the teacher logits. Minimizing the TMPD objective therefore penalizes solutions that are overly sensitive such perturbations, which constitutes implicit regularization effect.
Hence, TMPD can be viewed as optimizing the original TWD loss together with an additional regularization term.
14

---

Published as a conference paper at ICLR 2026
B.2 CONVERGENCE ROBUSTNESS BOUND OF TMPD
Theorem 2. The final error of TMPD, denoted by εTмтD, forms a strictly lower bound of that of TWD, denoled by ETWD, under the same learning rate,
ETMPD < ETWD-
(23)
Proof. Let (0) denote the training objective function parameterized by 0, and let J be its mini- mum value achieved at the optimal parameter 0* Assume J(0) is L-smooth and satisfies the PL condition with parameter μ > 0.
For SGD with constant step size S L, the standard bound is
[()(),) - JJ] - (1 mn)** (J(0g) J*) +
(24)
where e a 62 b with a, b. > 0 coefficients, o² denoting the gradient variance and 3 the gradient bias term standard SGD-PL. bounds (Ajalloeian & Stich [020 Gower et al., (202T). This shows that the geometric factor (1 nu)* determines the convergence rate, while the asymptotic floor determines the final accuracy. Since TWD and TMPD share the same rate, it suffices to compare STMPD and eTwD-
Let 9: denote the gradient contribution at timestep t. For two different timesteps s and t, we write g, and gt. The variance of the time-averaged gradient + Σ decomposes as
T gt = T
T
Var
Σ t=1
Var(gi) + 7
Σ
Covlge 9t).
(25)
T
ISACIST
In TWD, all gr are computed using the same teacher output, which induces positive correlations across timesteps; hence the covariance terms are positive, inflating the effective variance ot
In TMPD, the teacher outputs are perturbed independently across timesteps, making the cross-time covariances vanish up O(2²), while per-timestep variances change by most O()²). Therefore, for some constant > O:
T S oTWD c₂A².
(26)
Since the masks are drawn from mean-zero distribution, the induced gradient bias is O()²) in norm, leading to BIMPD O(*), negligible compared to the variance reduction. Thus the error floor satisfies
TTMPD = a iMmp O(x*) < a Two O(x*) = ETWD-
(27)
Since both methods share the same geometric factor (1 1H4), the strictly smaller error floor of TMPD directly implies better final performance than TWD.
B.3 TRUNCATION ROBUSTNESS BOUND OF TPD
Lemma (Distributional difference bound). Let dp,( = 111pp-11 qll denote the total variation distance. For any sequence of predictive distributions (To: Fi Ty...,, ,
n-1
drv(ro. rw)) S Σ drv((
S
Σ j=0
(28)
1=0
where Q is either the KL divergence or the cross-entropy.
Proof. Since dry metric, it satisfies the triangle inequality:
Tw) S
Σ drv((r,
(29)
1=
'L-smooth: The gradient is Lipschitz continuous, i.c., - < L - -0a]|.
VJ((2)]
PI. condition: weaker condition than strong convexity, expressed [ 2p(J(() J*).
15

---

Published as a conference paper at ICLR 2026
By Pinsker's inequality, for each j we have
drv(T,,T5t)) - v PRu(rrlllstH)
(30)
Furthermore, cross-entropy satisfies Hp, q) = Dxa,(ll))))) so clearly Hp, g) 2 (xl( Hence the same bound holds if we replace KL with cross-entropy:
drv(rg,7)))) & V ((1)))41)).
(31)
Summing over j and applying Manchy-Schwarzeeee then gives
n-1
dTv(T,)34))) S
7
(32)
1=0
Theorem 3. Denote by (1) the truncation accuracy drop at step T. Then the truncation accuracy drop of TPD is strictly smaller than that of neighbor-step step consistency (NSC), and NSC is strictly smaller than no-consistency (NC):
TPD acc
(1)
A DCC
NSC
(1) <
NC
(33)
Proof. The truncation accuracy drop at step is defined as the difference between the accuracy at the final prediction !r and the truncated prediction qe. The difference in accuracies is upper bounded by the total variation (Devroye et al. 2013):
(t)( S drv(gt, QT).
(34)
Since the truncation accuracy drop at step / is upper bounded by the total variation distance between the truncated distribution qe and the final distribution @r, it is sufficient compare the corresponding total variation distances when analyzing truncation errors across different methods.
For NSC, the training objective INSC(0)) is
JNS (0)
supervised (0s)(0) + (0) (0))
7 > 0.
(35)
k=1
Applying Lemma D to the chain (g....., ( yields
drv(go, (T) <
D(gn, (c+1)-
(36)
Since the supervised loss is nonnegative, and the penalty term vanishes when the objective is mini- mized (so that NNCC mine JNNC((O) we obtain
( (0) - JNSC
(gg,, (k+1)
T (1k, (k+1) < ( (N)-.)) (37)
k=1
A=1
Therefore,
drv (T) <
I (JNSC(O) - ( JN
(38)
At this point, the truncation error has been linked to the optimization suboptimality (NC JNSC. According to Eq. 24 we suppress these terms since our focus is on the geometric conver- gence rate and the noise contributions are identical across methods. Thus, gradient descent with step
size 77 < L/L ensures
JASC (0m)
J. JNCC < (1 --p,)" (JNSC (OS)C(On) NNS
(39)
16

---

For NC, the objective reduces to Published as a conference paper at ICLR 2026
Combining the inequalities shows that the truncation error in NSC decays geometrically:
NSC
(t)
S
(1 - ( (JNSC(Ou) - JN
(40)
acc
24
For TPD, we define averaged predictions q5k
1 1e Σx q. The training objective is
J supervised los)(0) +
Σ (q (5+-1())))))))))
(41)
Applying Lemma to the averaged chain (Āst..... (gT) gives
drv(āgr TTT)
(J) - - J)
(42)
By applying the same reasoning as in the NSC case above, the truncation error also decays geomet- rically:
TPD (t) S V I (1-nn,)) (O)P - -.PDD*. ace
(43)
Temporal averaging reduces variance across steps, smoothing the discrepancy signal and increasing the PL. constant, so that j,, 2 pd. Consequently, the truncation error TPD decays strictly faster than in NSC, yielding
TPD (t)
A
NSC
(t).
(44)
N (0) = supervised loss(0).
(45)
However, unlike in NSC or TPD, the sum of divergences Σx (q.. 9++1) is not penalized in the objective and hence not reduced during optimization. These discrepancies may remain large even as the supervised loss decreases, so the truncation error does not benefit from geometric decay. As result, the error in NC is strictly larger than in NSC:
! NSC aOC
(t)
A NC (t).
(46)
Putting the three cases together, we obtain the strict ordering
IPD ace
(t)
<
A
NSC
(t) < (t).
(47)
acc
Moreover, each divergence term ((gn, gk+ 1) arises from stochastic parameter updates, and its size ultimately controlled by the gradient variance and bias that also determine the error floor in TMPD.
Finally, recall that the temporal error at step decomposes
e())
= Efull + Ance (1).
(48)
TMPD reduces the full-length error term Eiull by lowering the variance of the teacher signal (Theo- rem 2), while TPD reduces the truncation term (t)e()) by tightening the temporal alignment (The- orem 3). Since these two quantities enter the error additively and affect disjoint parts of the decom- position, their benefits are complementary rather than redundant.
Thus, when TMPD and TPD are combined in MEOM, the error at step f can therefore written as
E MEOM
(t)
TMPD Full
+ A TTD (t)
(49)
TWD
+ ANC (t) e haselinee (t):
Eall
term (through TMPD) and the truncation term (through TPD). this sense, the two components act on complementary parts of the temporal error decomposition and their effects add up, yielding uniformly better performance across timesteps compared with using either component alone.
which shows that MEOM simultaneously reduces the full-length
17

---

Published as a conference paper at ICLR 2026
C EXPERIMENTAL SETTINGS
Dataset. CIFAR-I0 and CIFAR-100 100 are benchmark datasets for image classification. CIFAR-I0
consists of 10 classes with 60,000 32 32 color images, divided into 50,000 training and 10,000 testing samples. CIFAR-100 extends this to 100 classes grouped into 20 superclasses, with the same total number of images but fewer per class, making it more challenging. Direct encoding is utilized to convert image pixels into time series, with pixel values repeatedly fed into the input layer at each timestep.
ImageNet is a large-scale image classification dataset containing over 1.2 million training images and 50,000 validation images spanning 1,000 object categories. Each image varies in resolution and visual complexity, providing a more challenging and diverse benchmark compared to CIFAR datasets. To align with the experimental setup, images are resized and center-cropped a resolution of 224 224 before being processed into time series representations through direct encoding.
Experiment Details. All models are trained using stochastic gradient descent (SGD) with mo- mentum of 0.9, combined with a cosine annealing learning rate schedule. The experiments are implemented PyTorch. For the CIFAR datasets, training is conducted on a single NVIDIA A100 GPU, while ImageNet experiments employ distributed data parallel training across eight A100 GPUs to maximize computational efficiency, accelerate convergence, and ensure training stability. Differ- ent student architectures paired with their corresponding teacher models for training, and all hyperparameter settings and network architectures are summarized Table
Table Training settings and architecture across datasets. Here, LR denotes the learning rate, and WD denotes the weight decay.
BATCH SIZE
STUDENT ARCH.
TEACHER ARCH.
TEACHER ACC(%)
DATASET
EPOCHS
LR
WD
ResNet-18 ResNet-19 19
ResNet-34
97.06
CIFAR-10
128
300
0.1
5e-4
ResNet-19
97.20
ResNet-18 ResNet-19
ResNet-34
81.31
CIFAR-100
128
300
0.1
5e-4
ResNet-19
82.57
512
100
0.2
2e-5
ResNet-34 S-8-384
ResNet-34
71.24
IMAGENET
64
300
0.1
5e-2
ViT-Base
81.78
D ENERGY CONSUMPTION ANALYSIS
To evaluate the energy efficiency SNNs, we adopt standard neuromorphic computing method- ology that estimates the total synaptic operation power (SOP) based on the number of fundamental operations and their associated energy (Zhou al. 2022). The SOP is defined as:
SOP- & L'xc AC, + EMAC AAC,
(50)
where EAC and EMAC represent the energy consumption per accumulation (AC) and per multiply- accumulate (MAC) operation, respectively. Following the energy model in (Han et al. 2015), we assume that each 32-bit floating-pon point addition consumes 0.9 picojoules (pJ). while each MAC op- eration consumes 4.6 pJ.
In SNNs, neurons transmit binary spike signals, { E (0, 1). indicating whether neuron in layer 1 fires at timestep f. A firing spike activates all its outgoing synapses, with each synapse performing an addition. a neuron has f outgoing connections (fan-out), the total number AC operations across the network can be expressed as:
T I-1 N
AC.
Σ
ΣΣΣN
(51)
where T the total number of timesteps, L is the number of layers, and No denotes the number of neurons in layer
18

---

Published as a conference paper at ICLR 2026
In contrast, ANNs operate without temporal dynamics. Each neuron performs single forward pass, requiring a fixed number MAC operations determined by its synaptic connections:
L-1 N
MAC, = ΣΣ
(52)
By combining the counts of AC and MAC operations with their respective energy costs, the total SOP for any given network configuration can be effectively estimated.
E ENERGY EVALUATION
Table presents the comparison of energy consumption, computation, training time, and accuracy across different methods. Energy is computed from accumulation operations (ACs) and multi- ply-accumulate operations (MACs), where MACs remain identical because each LIF neuron ap- plies fixed leak-factor multiplication at every timestep. independent of spiking input. Thus, the main variation comes from ACs, which are directly determined by the firing rate (FR). Compared with VanillaSNN and KDSNN, both TWSNN and our method exhibit higher FR. This arises from the temporal-wise comparison mechanism, which forces the network to increase responsiveness at earlier timesteps, rather than relying predominantly By shifting part of the activity forward, the average FR rises, leading to correspondingly higher ACs and slightly greater energy. In terms of training time (TT, measured per batch), our method shows moderate increase compared with other SNNs, mainly due to the mask meightee perturbed teacher logits introduced in TMPD. At each timestep, TMPD requires generating perturbed logits and com- puting the corresponding distillation losses, which adds extra overhead during training. However, this additional cost only occurs the training phase and does not affect inference latency or energy efficiency, making it limited and acceptable.
later steps accumulate predictive evidence.
Table 9: Comparison of energy consumption, computation operations, training time, GPU hours, and accuracy on CIFAR-10 and CIFAR-100 with ResNet-18 and ResNet-19 19 at timestep T 6.
FR
ACs
MACs
ENERGY
TT
GPU h
ACC
MODEL
METHOD
(%)
(M)
(M)
(pJ)
(ms)
(h)
(%)
ANN
0.557
549.1
1757.3
1.07
97.06
VanillaSNN
14.21
72.500
3.342
17.946
123
4.25
95.74
C10-R18
KDSNN
14.90
78.641
3.342
18.560
129
4.45
95.85
TWSNN
15.66
83.915
3.342
19.087
130
4.49
96.16
Ours
15.98
85.644
3.342
19.260
163
5.63
96.28
ANN
1.442
2268.6
7259.7
1.10
97.20
VanillaSNN
13.06
285.764
8.652
56.264
308
10.64
96.71
C10-R19
KDSNN
12.30
272.512
8.652
54.939
317
10.95
97.02
TWSNN
15.35
310.799
8.652
58.767
318
10.98
97.05
Ours
15.20
311.080
8.652
58.796
363
12.54
97.08
ANN
0.557
549.2
1757.4
0.90
81.31
VanillaSNN
17.33
93.829
3.342
20.080
128
4.42
78.37
C100-R18
KDSNN
17.80
96.175
3.342
20.315
135
4.66
79.28
TWSNN
18.83
103.222
3.342
21.020
135
4.66
80.05
Ours
17.68
97.703
3.342
20.468
170
5.87
80.42
ANN
1.442
2268.6
7259.7
235
8.11
82.57
VanillaSNN
16.08
350.697
8.653
62.759
324
11.19
81.27
C100-R19
KDSNN
16.32
359.646
8.653
63.654
332
11.47
82.12
TWSNN
17.33
369.744
8.653
64.664
333
11.50
83.10
Ours
17.11
366.015
8.653
64.291
381
13.17
83.22
19

---

Published as a conference paper at ICLR 2026
F ABLATION STUDY ON IMAGENET
To assess the contribution of each component on a large-scale dataset, we conduct an ablation study on ImageNet using the ResNet-34 SNN backbone with T As shown in Table I0 incorporating TMPD and TPD yields additional improvements over the TWD baseline, and the full configuration achieves the highest accuracy.
Table 10: Ablation of components on ImageNet under the ResNeet-3 SNN backbone (T 4).
TAD
TWD
TMPD
TPD
ACC (%)
X
X
X
X
66.93
A
XX
68.22
71.04
ANKA
V
X
71.55
V
X
V
71.34
V
V
71.64
G USE OF LARGE LANGUAGE MODELS
We used LLMs only to polish the writing (e.g., grammar and readability). All ideas, experiments, and analyses are entirely the authors' own.
20


Reviews:
[{"review_id": "a0nZiaPhYB", "reviewer": "Reviewer_BP7W", "summary": "This paper introduces MEOM (Many Eyes, One Mind), a unified knowledge distillation framework for spiking neural networks (SNNs). It integrates two complementary modules: Temporal Multi-Perspective Distillation (TMPD), which enhances temporal diversity via masked teacher features, and Temporal Progressive Distillation (TPD), which gradually aligns truncated and full-length predictions to improve temporal consistency.", "questions": "see weaknesses", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.The paper clearly identifies the weaknesses of prior temporal distillation approaches and motivates the need for temporal diversity and consistency.\n2.The unification of TMPD (“Many Eyes”) and TPD (“One Mind”) provides an intuitive and theoretically supported structure.\n3.Experiments cover multiple benchmarks, showing both performance improvement and robustness under truncated inference.\n4.The paper’s exposition and proofs (information gain, convergence robustness) provide reasonable theoretical support for the design choices.", "weaknesses": "1. TMPD uses random, static masks. While effective, the design choice appears heuristic. Could adaptive or learnable masks yield further benefit? Consider comparing random vs. learned mask distributions or adding mask diversity analysis. \n2. TPD enforces progressive step-wise alignment but may neglect long-range temporal dependencies. A long-range consistency variant or additional analysis on global alignment would make the argument more complete.\n3. Assuming the final timestep is globally optimal may not hold universally. A calibration or stability analysis across timesteps could validate this assumption, or adaptive target-timestep selection could be explored.\n4. Evaluation focuses on ResNet-style SNN backbones, generalization to more complex or recent SNN backbones (e.g., Spiking Transformer, hybrid architectures) remains unexplored.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Td6MB5ahNH", "reviewer": "Reviewer_tdth", "summary": "The paper proposes a knowledge distillation method for SNN training, called MEOM (Many Eyes, One Mind). The method consists of two parts: Temporal Multi-Perspective Distillation (TMPD), which creates the teacher's feature for each time-step, and Temporal Progressive Distillation (TPD), which averages activation across different time-steps and calculates CE losses between the average results and targets.\nExperiment shows that the method has better accuracy than BKDSNN with Spikformer-8-384 on ImageNet.", "questions": "Please see the weaknesses.", "limitations": "", "rating": 2, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The paper organization is clear.\n2. The proposed distillation methods are understandable.\n3. The results in CIFAR10/100 is good.", "weaknesses": "1. Limited novelty:\nThe proposed work introduces two techniques to enhance the performance of knowledge distillation. However, the TMPD method only provides a marginal improvement (around 0.3%) on CIFAR-10/100, which is within the range of training variance and thus not convincing. Moreover, the TPD approach is rather straightforward and not inherently tied to the distillation framework. Therefore, it could also be applied to conventional BPTT-based training, which weakens its novelty.\n\n2. Insufficient experimental validation on ImageNet:\nThe experiments on ImageNet lack sufficient ablation studies, making it difficult to attribute the performance gain solely to the proposed techniques. In addition, the authors should consider evaluating on a larger model, such as the S-8-768 structure, where BKDSNN achieves 79.9% accuracy, to strengthen the evidence.\n\n3. Missing analysis of training overhead:\nThe proposed distillation-based methods are expected to introduce additional training overhead, especially due to the TMPD methods. The authors should include a comparison of the GPU memory footprint and total GPU hours between the proposed methods and standard distillation baselines.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ldB0NoFLIQ", "reviewer": "Reviewer_dxXD", "summary": "This paper introduces MEOM, a novel SNN distillation framework that tackles two key issues: static teacher supervision and poor truncated inference. It employs Temporal Multi-Perspective Distillation (TMPD) to generate diverse teacher signals and Temporal Progressive Distillation (TPD) to align predictions across timesteps. MEOM achieves state-of-the-art results, demonstrating significant improvements in both final accuracy and performance under truncated inference.", "questions": "1. Regarding the mask design in TMPD, ehe random mask strategy is simple and effective. Have you explored other, more sophisticated masking approaches, such as learnable masks or masks that are dynamically adapted based on the timestep? Do you think such strategies could offer additional performance gains?\n2. Regarding the progressive alignment in TPD, the TPD achieves progressive consistency by aligning the \"cumulative average prediction\" of consecutive timesteps. How does this compare to a simpler strategy of directly aligning the cumulative prediction at each step with the final prediction (at time T)? Do you think the \"step-wise\" smoothing is crucial for training stability?\n3. Regarding the synergy between TMPD and TPD, as TMPD is designed to introduce \"diversity\" and TPD is designed to enforce \"consistency,\" two goals that could somehow be in tension. Could you elaborate on a deeper level how these two seemingly opposing goals work synergistically within the MEOM framework?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper is well written, clearly structured, and logically sound. \n2. The critique of existing TWD methods is insightful. Identifying the \"static teacher vs. dynamic student\" mismatch and the \"information loss in truncated inference\" as two practical and important problems is a key contribution. \n3. The experimental results are convincing. The authors demonstrate state-of-the-art results across multiple datasets (CIFAR, ImageNet) and architectures (ResNet, Spiking Transformer), while a dedicated \"time flexibility\" experiment shows the method's advantage in truncated inference. Furthermore, thorough ablation studies clearly isolate the contributions of each proposed component (TMPD and TPD) confirming their complementary benefits.", "weaknesses": "1. Masking strategy in TMPD. The use of random masks in TMPD is a simple and effective way to generate diverse teacher signals. However, this might not be the optimal strategy. The mask generation is based on a fixed random sampling and does not adapt to the student's learning state or the characteristics of different timesteps. Exploring the connection between masking strategies and temporal-wise features could potentially lead to further performance improvements.\n2. Interplay between TMPD and TPD. The paper presents TMPD and TPD as two complementary components. However, TMPD is designed to introduce \"diversity\" while TPD is designed to enforce \"consistency,\" two goals that could, at some level, be seen as being in tension. While the experiments show their combination is effective, the paper could benefit from a deeper discussion on how these components work synergistically rather than counteracting each other.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jfqT5QnCzH", "reviewer": "Reviewer_uLvn", "summary": "This paper presents MEOM (Many Eyes, One Mind), a Knowledge distillation (KD) framework for spiking neural networks (SNNs). It employs Temporal Multi-Perspective Distillation (TMPD) to introduce temporal variances to the ANN teacher output. It also utilizes Temporal Progressive Distillation (TPD) to align with the full-length prediction for truncated inference progressively.", "questions": "1. Please analyze the temporal variance of SNNs and its effect on SNN performance.\n2. Please analyze the impact and robustness of the hyperparameter introduced in MEOM.", "limitations": "", "rating": 2, "confidence": 5, "soundness": 1, "presentation": 2, "contribution": 2, "strengths": "1. The proposed MEOM considers the temporal variances of SNNs.\n2. This paper provides theoretical analyses for the proposed submodules.", "weaknesses": "1. The motivation of TMPD requires further clarification. Lines 51-69 claim that it is improbable for outputs across timesteps to remain identical. A more effective strategy would incorporate diverse temporal supervisory signals. However, as illustrated in Fig. 1, both final logits and membrane potential distribution do not exhibit significant differences across timesteps. Furthermore, the impact of such variations on SNN performance has not been verified.\n2. The proposed TMPD is more like a data augmentation method that introduces perturbations in the temporal dimension, rather than providing richer temporal supervision. Theorems 1 and 2 merely demonstrate that introducing perturbations results in higher temporal covariance and lower gradient variance, thereby proving the effectiveness of this data augmentation approach. They do not prove that TMPD provides richer temporal supervision. I believe this data augmentation method is not only applicable to KD but also equally effective for general SNN training.\n3. The proposed MEOM method does not show significant performance improvement over state-of-the-art methods. Compared to TWSNN, its accuracy gains across various datasets are less than 1%.\n4. The overall training objective introduces three hyperparameters $\\alpha$, $\\beta$, and $\\gamma$. The impact of hyperparameter settings on the effectiveness of the proposed method remains unclear. The robustness of hyperparameter settings across different tasks is also unclear.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)