Temporal Flexibility in Spiking Neural Networks: Towards Generalization Across Time Steps and Deployment Friendliness
Authors: Kangrui Du
Yuhang Wu
Shikuang Deng
Shi Gu
Venue: ICLR 2025 Poster (2025)
Keywords: spiking neural networks
direct training
event-driven friendliness
Primary area: applications to neuroscience & cognitive science
TL;DR: We 1) introduced mixed-timestep training to optimize SNNs for event-driven chip deployment and energy-accuracy trade-offs, and 2) validated our approach on GPU-based framework, real neuromorphic hardware and event-driven chip simulator.

Abstract: Spiking Neural Networks (SNNs), models inspired by neural mechanisms in the brain, allow for energy-efficient implementation on neuromorphic hardware. However, SNNs trained with current direct training approaches are constrained to a specific time step. This "temporal inflexibility" 1) hinders SNNs' deployment on time-step-free fully event-driven chips and 2) prevents energy-performance balance based on dynamic inference time steps. In this study, we first explore the feasibility of training SNNs that generalize across different time steps. We then introduce Mixed Time-step Training (MTT), a novel method that improves the temporal flexibility of SNNs, making SNNs adaptive to diverse temporal structures. During each iteration of MTT, random time steps are assigned to different SNN stages, with spikes transmitted between stages via communication modules. After training, the weights are deployed and evaluated on both time-stepped and fully event-driven platforms. Experimental results show that models trained by MTT gain remarkable temporal flexibility, friendliness for both event-driven and clock-driven deployment (nearly lossless on N-MNIST and 10.1\% higher than standard methods on CIFAR10-DVS), enhanced network generalization, and near SOTA performance. To the best of our knowledge, this is the first work to report the results of large-scale SNN deployment on fully event-driven scenarios.

Published as a conference paper at ICLR 2025
TEMPORAL FLEXIBILITY IN SPIKING NEURAL NET- WORKS: TOWARDS GENERALIZATION ACROSS TIME
STEPS AND DEPLOYMENT FRIENDLINESS
Kangrui Du* Yuhang Wu Shikuang Deng', Shi Gu 1.2 'University of Electronic Science and Technology of China Shenzhen Institute for Advanced Study, UESTC *College of Computing, Georgia Institute of Technology kangruiduegatech. edu, gus gusQuestc.
edu.cn en
ABSTRACT
Spiking Neural Networks (SNNs), models inspired by neural mechanisms the brain, allow for energy-efficient implementation neuromorphic hardware. However, SNNs trained with current direct training approaches are constrained to specific time step. This "temporal inflexibility" 1) hinders SNNs' deployment on time-step-free fully event-driven chips and 2) prevents energy- mheryy-pefformance bal- ance based on dynamic inference time steps. In this study, we first explore the fea- sibility of training SNNs that generalize across different time steps. We then intro- duce Mixed Time-step Training (MTT), a novel method that improves the tempo- ral flexibility of SNNs, making SNNs adaptive to diverse temporal structures. Dur- ing each iteration of MTT, random time steps are assigned to different SNN stages, with spikes transmitted between stages via communication modules. After train- ing, the weights are deployed and evaluated on both time-stepped and fully event- driven platforms. Experimental results show that models trained by MTT gain re- markable temporal flexibility, friendliness for both event-driven and clock-driven deployment (nearly lossless N-MNIST and 10.1% higher than standard meth- ods on CIFAR 10-DVS), enhanced network generalization, and near SOTA perfor- mance. To the best of our knowledge, this is the first work report the results of large-scale SNN deployment on fully event-driven scenarios. Codes are available at: htttps://ggtuub.oommbrrtee
wawelignnnce-aattprrllllll lab/eeemporalffeiiiiiitty flexibility.innnn SNN
INTRODUCTION
As deep learning continues to evolve, the field has witnessed numerous groundbreaking advance- ments that have made unprecedented strides across diverse applications. However, deploying these huge neural networks on low-power edge devices presents substantial challenges. In addition to the typical solutions, including network quantization (Rastegari al. (016)), pruning (He al.. 2017). and distillation (Hinton et al. 2015), the Spiking Neural Networks, known as one of the 3rd gener- ation of neural networks, have emerged as a compelling candidate due to their unique bio-inspired characteristics (Fang et al., 2021 Guo et al. 2022; Yao et al. 2023). SNNs mimic the behavior of biological neurons by accumulating membrane potentials and transmitting sparse spikes, thereby circumventing the need for computationally expensive multiplications (Roy et al. 2019), which presents promising solution for energy-efficient neuromorphie computation.
The SNN community has flourished in recent years. One of the most significant driving factors the invention of direct training methods with time-step-based iterative neurons (Wu et al., 2018 2019)- with an RNN-like backpropagation method, the introduction of time steps (T) successfully brings SNN training into mainstream deep learning platforms like PyTorch, which allows for fast GPU-accelerated training for large-scale SNNs.
While training at a specific time step has become a prevalent paradigm for following works, the obtained SNNs perform well only at specific T but generalize poorly to others. This temporal inflexibility posts constraints on SNNs' deployment on neuromorphic chips, where the energy ad- vantage is truly exploited. For example, it posts constraints on time step adjustment and thus hinders
"Equal Contribution
Corresponning author

---

Published as a conference paper at ICLR 2025
Updaz valgle
Vali
stages
Deploysenn
Fally cran-divess
iffosmas
Figure 1: The workflow of MTT pipeline. We first partition SNN into G stages. In each iteration, sample s temporal configs g())_ ((0) E R each assigning set of random time steps to different stages (for j-th sampled config, T, = t ) These configurations create partitioned SNNs with distinct temporal structures, all sharing the same woights. To update the shared weights, we backpropagate the sum of the losses to obtain the gradient. Models trained with MTT exhibit temporal flexibility, which leads to their adaptation any time step and friendliness with fully event-driven chips.
SNNs' on-device energy- perryy-performance balance by dynamic time step algorithms (Li et al. 2023d) and hardware (Li et al. 2023d). Another more conspicuous issue occurs when deploying SNNs on fully event-driven neuromorphic accelerators. Fully event-driven accelerators are ideal always-on edge platforms for SNNs with ultra-low energy consumption (Li et al. 2023b) and bio pio-plausibility (Richter et al. 2024). On these time-step-free platforms, all neurons operate asynchronously, and auxiliary for GPU-friendly training rather than real-world typer-parameed
the time step here
In response to these deployment issues caused by temporal inflexibility, we propose a novel training method, Mixed Time-step Training (MTT), improve temporal flexibility of SNNs. The workflow is shown in Fig. The parameters trained with this method are decoupled from temporal structures used during training and are compatible with wide range of time steps. This not only enhances the model's 's compatibility with fully event-driven hardware but also enables dynamic on-chip time step adjustment and corresponding energy perrgy-performance trade-off strategies. Our main contributions are as follows:
We identified the temporal inflexibility caused by the standard training method and its potential solution, based on which we further designed Mixed Time-step Training (MTT) to enhance the temporal flexiblility of SNNs.
Intensive experiments are carried out on platforms including GPU-accelerated servers, neu- romorphic chips, and our fiigh-performance event-driven simulator, testifying MTT's effee- tiveness on static and event datasets. To our knowledge, this is the first work to evaluate large-scale SNNs in fully event-driven scenarios.
Models trained with MTT demonstrate remarkable temporal flexibility while achieving performance comparable to other state-of-the-art approaches.
2 RELATED WORK
Direct Training The direct training approach stems from the idea that SNNs can be viewed as vari- ant RNNs and trained using BPTT as long as the non-differentiable activation term is replaced with a surrogate gradient. [u et al. (2018) first proposes the STBP method and trains SNNs using an ANN framework. Further, (Wu et al. 2019) and /Zeng et al. (202T) suggest novel NeuronNorm and BatchNorm strategies to facilitate large-scale SNN training, respectively. Recently, the performance of SNN on neuromorphic datasets has been substantially enhanced with the advent of specially de- veloped algorithms, including TET (Deng et al. 2002)) and TCJA-SNN (Zhu et al. (022a).. On static datasets, various methods (Li et al. the gap between SNNs and ANNs. Notably, Guo et al. (2022) first reported SNNs with accuracy exceeding the corresponding ANN counterpart, demonstrating the strong potential of SNNs. More-
2021b Guo et al. 2022] Yao et al. 2022) are proposed to close
over, the recent emergence of spike- based transformer architectures (Zhou al. 2024
2023
Yao
2

---

Published as a conference paper at ICLR 2025
al. 2024 b Zhou et al. 2022) has propelled the performance of large-scale SNNs to unprece- dented levels. However, weights obtained by existing direct training methods are only applicable to specific time step, which posts constraints for deployment and entails further fine-tuning.
ANN-SNN Conversion ANN-SNN conversion uses SNN firing rates to approximate the activation of ANN. Specifically, parameters are first directly copied from pre-trained ANN to the target SNN and then fine-tundd to mimic the original ANN activation. Techniques have been proposed to reduce the minimum convertible time step, including the subtraction mechanism (Rueckauer al. 2016 Han et al. 2020 Han and Roy 2020), spike-norm (Sengupta et al., 2019), threshold shift (Deng and Gu 2021 layer-wise calibration (Li et al. 2021a) and activation quantization (Bu al. 2023 Although SNNs obtained by conversion show some temporal flexibility for large time steps (e.g., above 100), they do not exhibit temporal flexibility for ultra-low time steps. Additionally, the conversion method is unable to handle DVS datasets and can only procure SNNs with IF neurons.
Dynamic Inference Time Step Recent research explores inference- wise varied time steps to reduce the average inference cost by skipping steps when the network is confident enough. Li et al. (2023c) introduced SEENN, which determines the exit time step using confidence scores (SEENN-I) or policy network (SEENN-II). Li et al. (2023a) introduced another confidence-basee dynamic model, identifying the optimal confidence threshold using a Pareto front. Our models provides ideal weights for these methods due to their ability to infer at different time steps without extra onnettuning
Fully Event-driven Neuromorphic Chips Although iterative neurons successfully integrate direct training into modern backpropagation frameworks (Wu et al.] 2018), clock-driven neuromorphic hardware based on this framework may not be suitable for always-on real-time devices at edge plat- forms due the constant state updates even in the absence of input spikes (Dampthoffer et al.) 2022). Recently, time-step-free, fully event-driven SNN implementations have received increasing attention from researchers (Deng al., 2024] Кoopman al., 2024) due to their ultra-low energy consumption, compatibility with real-time edge scenarios, and better similarity with biological neu- rons (Richter et al. 2024] Li et al.] (023bb) Models trained with our method are well well-suited for deployment on this fully event-driven hardware.
3
PRELIMINARIES
3.1
SPIKING NEURON MODEL
The most popular model for SNN neurons in recent studies (Wu et al. 2019 Zheng et al. 202] Xiao
al. 2022. Deng al. 2023 Zhou al. 2022,] Yao al. 2024b) the Leaky Integrate-anc and -nd--Fire
(LIF) model. Its dynamics can be characterized by
du di
u u <<< VaA
(1)
fire a spike and u + R(u), u 2 Vvn
(2)
where u is the membrane potential, To is membrane constant, is pre-synaptic input, Vú, is mem- brane threshold, and R(-) ) reset function (Wu et al., 2019). For hard reset I((u) and for soft reset It(u) = Vea- The two simulation methods of these equations, the time-stepped and the event-based simulation, have given rise two popular SNN models: the clock-driven synchronous model and the event-driven asynchronous model.
3.2 TIME-STEPPED SIMULATION, CLOCK-DRIVEN LIF/IF MODEL AND HARDWARE
Let's first consider Eq. 1 After applying the forward Euler method to Eq. ] replacing 1 dt/To
with 7, and absorbing dt To scaling factor into the synapse weight (Wu et al. 2019). we have
u(t + dt) ru(t) I(t)
(3)
where T is decay factor. We then incorporate fire and reset (Eq. 2 mechanism into the simulation by adding a temporary variable e and have
ot + dt) = ru(t) + I(t) s(t + dt) 0(v(t + dt) Ven)
(4)
(5)
w(t + dl) = x(1 + di) R(v() + d()) + [1 - s( (t + ddt)
(6)
where vtt) is the temporary variable, stt) is the output spike at time t, O(x)) the Heaviside step
function where (xx) = for 2 and ((x) 0 for x Eq. 46 is the single iteration
3

---

Published as a conference paper at ICLR 2025
of the time-stepped simulation. The iterative Integrate-and-Fire (IF) model is special case of LIF where r = or To - +00. Time-stepped simulation can be properly accelerated by GPU due to its compatibility with mainstream deep learning framework, enabling efficient training of large- scale SNNs. Clock-driven neuromorphic hardware simply mimics the iterative process of time- stepped models, which can thus be deployed such hardware with little performance loss. Time step of clock-driven neuromorphic inference serves as a hyperparameter that controls the temporal granularity and forward time complexity, directly influencing the trade-off between performance and energy consumption. For experiments on time-stepped platforms in this paper, we use LIF neuron with Ven T 0.5.
3.3 EVENT-BASED SIMULATION, EVENT-DRIVEN LIF/IF MODEL AND HARDWARE
Rethinking the original neuron dynamics, (()) only has discrete spikes, which can be seen as weighted Dirac delta functions. This enables a precise simulation of LIF neurons based on events. Let's start with < Vaa again and analyze (tt = and [(t) # 0 separately. For cases where where t' is the moment the last spike arrives. For / [(t) & 0 cases, we let dt + O, absorb di To into I as we did in Eq. 3 and then have du = I where du is the instantaneous increment of u. The term -dt/ u is eliminated because is a finite value. Combining the two cases discussed, we have the formula
T(t) O, we directly solve the differential equation and have u(t) u(t')e-(t<t))
=
1((--1) + I(4)
where t is the timestamp of the current i-th spike and 44-1 is the timestamp of the last spike (Wu jet al.] (2088) Similar the time-stepped scenario, add an intermediate variable enable fire and reset behaviors.
4--4-1
e(t,) TO =((t-1))+ + I(t;)
(8)
s(t,) O(u(t,) - Vinl
(9)
u(t;)) = s(t,) R(tt))( + (1- (1--s(ti))) s(ti))u(tt)
(10)
where utt) is the temporary variable, s(t) is the output spike at time t, O(x) is the Heaviside step function. The event-based Integrate-and-Fing (IF) model is a special case of LIF where To - 400.
The discrete, sequential nature of events hinders parallelizing event-driven SNN training. While some event-driven methods have been explored, they either remain constrained within the time- stepped framework (Zhu al.) (022b) or face challenges in scaling to large models due to in- sufficient GPU support (Engelken 2023). Since event-driven simulation under spiking input pre- cisely replicates the original dynamics, the industry uses time-stepped simulation to approximate event-driven behavior and directly deploys weights trained in the timestteped framework onto fully event-driven hardware (Richter al.) (023). In fact, Eq. 4 and Eq. 88 10 share the same form, and thus theoretically, the event-driven inference can be approximated by time-stepped inference as di - 0, equivalently as time step T +oo. This aligns with the intuition-an sufficiently high T will result in at most one event per time step, effectively mirroring the event-drived paradigm where neurons update only upon event arrival. All trainings for event-driven platforms in this paper are conducted by Speck deployment toolkit- /Tonic We use IF neurons with Ve, = 1 for all event- based experiments because most of the existing fully event-driven neuromorphic chips mainly support IF for its suitability to asynchronous scenarios.
3.4 SURROGATE GRADIENT
The time-step-based direct training method computes gradients for parameters by spatiotemporal backpropagation (STBP) (Wu et al. 2018):
0L
Σ aL ds(t) @u(t) aI(t)
(11)
ow
ds(() dv(() (f ow
When backpropagating, all terms apart from the term
Be(t) Sw(()) can be easily caleulated. However, the
8a(t) ae(u) is the derivative of the Dirac delta function and does not exist. To solve this term (f)(F) de
problem, surrogate gradient(SG) is used to approximate the original gradient. For all time-stepped
experiments in this work, we adopt a triangular surrogate gradient (Rathi and Roy 2020). which can be formulated as
ds(t) 1 Dv(()
max h - -VaA - e(t)),
(12)
4

---

Published as a conference paper at ICLR 2025
where h is a constant controlling the sharpness. In this work, we apply h=1 to most experiments. For experiments on event -eent-driven platforms, we follow Speck handbook and use the default single exponential SG (Shrestha and Orchard 2018) in Tonic (Lenz et al. 202I).
METHODOLOGY
4.1 IDENTIFYING TEMPORAL INFLEXIBILITY AND POTENTIAL SOLUTION
As discussed in Sec. 3 current mainstream methods for training SNNs, whether clock or event-driven, rely on time-stepped frameworks. To avoid confusion, we refer to these conventional time-step- training Standard Direct Training (SDT). Our study identifies critical limitation of SDT, which we named "temporal inflexibility". models trained with SDT perform ideally only the same time-step setting at training, and perform evidently lower other time step configu- rations than models trained at that time step configurations specifically. To showcase this limitation, we perform experiments with ResNet18 on CIFAR 100 (Krizhevsky al. 2009). An SNN was first trained at T = 6, and its inference accuracy was evaluated across five distinct time step settings (see SDT in Tab. D However, the temporal structures of SNNs on platforms deployed often differ from those used during training. For example, event-driven platforms lack explicit time steps, and the time-stepped training is only an approximation to event-driven models, leading to a substantial gap with on-chip scenarios (see Sec. (.3). In clock-driven systems, time steps may also be dynamically adjusted to meet varying snergy-performance requirements (see Sec. on "Dynamic Inference Time Steps"). The temporal inflexibility, however, prevents model from generalizing to temporal structure on deployment platforms and thus poses significant challenges for SNN deployment.
To alleviate temporal inflexibility, we start with a straightforward method named Naive Mixture Training (NMT). In each iteration of NMT, it ran- domly selects 3 time steps from a predefined range (1.2....,6) 6} and performs forward passes. The th sampled time step T; is then used for the i-th forward pass of the iteration. The parameters are then updated collectively after completing all for- ward passes within an iteration. We test the NMT- trained model at different time steps and compare it with SDT-trained models. The results are shown at Tab. D Although NMT is relatively simple method, a single NMT-trained model performs compa- rably to models separately trained by SDT for each time step (denoted as SDT*). This indicates that NMT effectively mitigates models' temporal inflexibility, enabling the model to generalize across different time-step configurations. We term this generalization to temporal structures beyond those used during training as temporal flexibility, which can be measured by its performance on unseen temporal structures compared to models trained specifically on those structures. Given the simplicity and effectiveness of NMT, we further investigate underlying mechanisms.
Table I: Inference accuracy ResNet18 on CI- FARI00 100 by naive mixture training vs. standard direct training. "SDT=": SNNs independently trained with SDT at each T. "SDT": single SNN trained T=6 and infers at other T.
Methods
T-2
T=3
T=4
T=5
T=6
SDT
70.08
72.77
74.17
75.09
75.63
SDT*
72.86
73.86
74.77
74.96
75.63
NMT
73.47
74.17
75.11
75.34
75.77
4.2 ANALYSIS ON NAIVE MIXTURE TRAINING
Temporal Flexibility. As mentioned earlier, the models trained with SDT are only optimized for a specific T. This will render overfitting towards the single temporal structure and cause poor temporal flexibility (see Tab. D. NMT successfully mitigates this overfitting by training temporal structures at the same time so that the model learns how to keep the performance with different time structures and thus gets better generalization across temporal structures. Due to the enhanced temporal flexibility, NMT-trained models can change their inference time structures with reduced performance degradation. For time-step-basse based SNNs, this property also detaches training time steps from the event sensor's 's framing time steps, which is relevant to specific application scenarios and is often too high for GPU training platforms.
Event-driven Friendliness. On fully event-driven hardware platforms, all neurons operate asyn- chronously, and the time step is no longer a hardware hyperparameter that determines the number of iterations for each inference. In such systems, the time step becomes irrelevant to the on-chip inference phase, serving only in the training phase to simulate the on-chip model's operation in a GPU-compatible manner. However, we analyzed in Sec. 3.3 and 4.1 there is substantial gap between time-stepped simulation and real event-driven dynamics, and temporal inflexibility caused by SDT prevents models from generalizing to event-driven temporal structures, which can be seen
5

---

Published as a conference paper at ICLR 2025
NMT add to model's temporal flexibility and largely decouple network outputs from the time-step-bare base- temporal structure. This makes NMT-trained models ideal candidates for event-driven deployment.
a time-stepped model whose T is infinity (see Sec. 3.3
Network Generalization. SNN training falls into local minimum point, once NMT samples time step that is far away from the current one, the SNN's output may change significantly. In this scenario, the new training loss may not converge, leading SNN to jump out the local minimum point. Eventually, the SNN will be trained towards flatter minimum point. Another perspective is that since the sampling space is 6, NMT is equivalent to training similar SNNs simultaneously. This similar applying new kind of dropout to the SNN, which improves the network's ' generalization. This is why NMT significantly improves SNN's performance when T is small. We verify our theory through experiments Sec. 5.1 and loss landscapes in See. A.10
4.3 MIXED TIME-STEP TRAINING
The success of NMT lies in its incorporation of diverse temporal structures during training. A straightforward idea to improve is to include more temporal structures. However, the number of temporal structures in NMT scales linearly with Tmaz: and excessively large T cannot be trained on current GPUs. To introduce more temporal structures while keeping Tiaz not increasing, we propose Mixed Timestep Training (MTT). In MTT, A normal SNN is first partitioned into stages as shown in Fig. 2 In each iteration, each stage is assigned with different time steps. By assigning stage-wise different time steps, MTT successfully expands the number of temporal structures from Tiax to (T max) where the number of stages. We then introduce each step MTT.
stage:
staggu-3
Tear
Visa
Figure 2: SNN partitioned for Mixed Timestep Training.
4.3.1 NETWORK PARTITIONING
The first step of MTT to divide the whole network into stages which will then be set to different time steps for each forwarding process. We denote the setting of time steps of different stages in each forwarding process MTT by temporal configuration vector = (L,,...,4m) (n), where 4 denotes the time step of the i-th stage and is the total number of stages. The forwarding of partitioned SNN Sp can be denoted by (z, t), where x is an input and is the temporal configuration vector.
4.3.2 TEMPORAL TRANSFORMATION MODULE
We then design the inter-block com- munication rule between adjacent stages with different simulation time steps-temporal transformation mod- ule (TTM). When adjacent stages are of the same number of time steps,
A b1
by
B
b2
TTM becomes an identity transfor- mation. Otherwise, TTMs can be cat- egorized into types as illustrated in Fig. dooonnsmmlgng TTM and upsampling TTM. In downsampling TTMs, we borrow from the pooling layer and divide input time frames into fout groups of adjacent frames. We then sum up the frames within each group to form fout time frames. By contrast, the upsampling type of TTM replicates each input time frame and assigns it to all output frames in the corresponding group, using the grouping policy same as the downsampling type of TTM with lout frames of input and tan frames of output. Now, the task at hand is to identify a suitable policy to partition I frames into l groups, where / k. A natural idea is to group them evenly possible to minimize temporal mismatch. According to this idea, we use the following policy shown in Eq.
3
Figure 3: (A) Downsampling TTM when fin=5 and lowr=3. (B) Upsampling TTM when =3 and foue=5.
13 where b; denotes the index of the first frame of group i. Explanation of our design is in Sec. A.4

---

Published as a conference paper at ICLR 2025
b; (i- ( 1)-1 +liE k).
(13)
k:
4.3.3 MIXED TIME-STEP TRAINING
Overall MTT Framework With TTM and partitioned network, we can now develop mixed time- step training (MTT). Mathematically, the goal of MTT is to minimize loss for all temporal structures:
N
MMTT(vvealll =
Σ
Σ
[(Sp(as,))) yk)
(14)
k=1 LE (Tminin... Tmaz)
where is any loss function, is dataset size, G is the number of stages, Tmin and Taa are the minimum and maximum time steps respectively, Sp is partitioned SNN, and & is any possible temporal configuration vector. Since directly optimizing the overall loss is too expensive, we sample 8 vectors z(1) x()...... t(() E (Tmin:...;; Tman)) } for each iteration and optimize the estimated loss function instead:
LTT Σ Σ
L(Sp(ak, (), y))
(15)
(E
where is batch size. To better illustrate our method, the training pipeline of one epoch is detailed in Algo !
In this work, Twin is set to for all experiments.
Algorithm 1 Mixed time step training for one epoch
Input: SNN model Sp; training dataset; training iteration I; sample number S in one iteration;
stage number G: minimum and maximum time step Tuins Toaas
1: for all 1,2.... I-th iteration do
2: Get the training data and labels
3: for all = 1,2, ..... 5-th sample do
4:
Sample vector ((*) with G numbers in the range (Tminn T mar] 5: Calculate the loss function C(Sp(x, t))), y:)
6:
Backpropagation and collect the gradient 7: end for
8: Update the model weights with collected gradients
9: end for
BN Layer Calibration MTT implemented with the standard BN technique suffers significant ac- curacy degradation. When training with mixed time steps, drastic structural changes will cause intensive variations batch statistics. Therefore, the running mean and variance calculated during training are inaccurate for models trained with MTT. To address this problem, we either lock BN layers (when fine-tuning, Sec. AA.3) or calibrate BN statistics similar to Yu and Huang (2019) from few training batches after optimizing and fixing weights (when training). In our experiments in Sec. correcting BN statistics with as few as 10 batches proved sufficient. Therefore, we used this setting in all experiments. We also found that the BN statistics of Taxx apply to other time steps. This saves us extra calibrations when switching to different inference (details in Sec. A.6).
5.2
4.4 TESTS ON FULLY EVENT-DRIVEN SCENARIOS
To verify the event-driven friendliness of our TFSNN, we employed Synsense Speck2e as the testing platform. The Speck series chips are among the most advanced real-time, fully event-driven neu- romorphic chips available today, boasting extremely low power consumption and latency (Richter et al. 2023: Li et al.) 2023b). However, due to their limited size, Speck cannot support the main- stream backbones used for DVS datasets in recent studies (e.g. VGGSNN). Therefore, we developed an easy-to-use parallelized
driven chip software simulator and aligned it with Speck on small datasets. To quantify the mismatch between a given output and the real hardware output, we define spike difference (SD)
[so[]] slill
SD 8)
(16)
-== 50[6]
where So is the real hardware output spikes, is the output spikes to compare with, suli] denotes the total spike counts of the i-th neuron, and Ny is the dimention of output. We then used this simulator test TESNN our knowledge, our work is the first to report the performance of large-scale datasets and models on fully event-driven scenario.
larger datasets and mainstream backbones (see Sec. 5. To the best of

---

Published as a conference paper at ICLR 2025
5 EXPERIMENTS
In this section, first conduct validation experiments for analyses in Sec. 4.2 These experi- ments aim to examine if the models trained with MTT improves temporal flexibility, event-driven friendliness, and network generalization as claim. Then, a comparison between our method and other current training methods is made to demonstrate the effectiveness of our method in terms of temporal flexibility and model performance. Finally, well-designed ablation studies are carried out to prove the effectiveness of network partitioning and BN calibration, the two main compo- nents of our method. The datasets involved in this work include static datasets like CIFARIO, CI- FAR10N 100 (Krizhevsky et al. 2009). and ImageNet (Deng et al. 20009), and event-hased datasets such as CIFARTO-DVSS {Li et al., 2017) and N-Caltech101 (Orchard et al., 2015). We also tested our method on sequence task and audio task (see Sec. A2 The model structures used in this paper are ResNet-18 (He et al. 2016), ResNet-19 (Zheng et al. 2021 ResNet-34 (He et al. 2016), VGG.
5.1 VALIDATION EXPERIMENTS
Temporal Flexibility Across Time Steps We first test models under wider range of inference time steps on the datasets and network structures. On static datasets, inputs are repeated for times as previous works where T is the inference time step. MTT-trained models perform fairly well at different time steps as shown in Tab. 2 On DVS datasets, input events are split into frames according their timestamps for time-stepped training and inference. See training details in Sec. A.3 We find the model's temporal flexibility is significantly enhanced by MTT especially at high T as shown in Fig. and inference high T is similar to event-driven scenario (Sec. B.3). while SDT and TET both fail to generalize across time steps. We then further compare our method with recent ANN-SNN conversion methods Tab. to demonstrates the temporal flexibility brought by MTT. Although these conversion methods involve post-conversion finetuning at inference T and MTT doesn't, I, MTT still outperforms them significantly at T=1 and T=2 while remaining comparable to them for higher time steps. For a fair comparison, adopt the same data augmentation policy as these methods. To further demonstrate the temporal flexibility that MTT brings, and to show how temporal flexibility benefits driven deployment we claimed, we combine SEENN (Li et al., 2023c) with ResNet19 trained by MTT on CIFARIO (Tab. 4. Since the time step in SEENN- is dynamically decided by confidence scores, the T ultimately contains decimals, which stand for the average number of time steps needed for each sample in the test set. We observed a performance boost under the same average inference T when compared with their reported results trained by TET, testifying the suitability of MTT-trained models to dynamic time step inference methods.
Table 2: Accuracy of different inference time steps.
Dataset
Method
Backbone
T=2
T=3
T=4
T=5
T=6
70
CIARR100
MIT
19
81.14
81,51
81.73
81.88
CIFAR10
MIT
ResNet-19
96.20
96.62
96.75
96.76
96.84
Accuracy
ImaggNNet
MIT
ResNot-34
65.23
67.58
67.54
68.02
68.34
80
Method
Table 3: Compare with SOTA ANN-SNN conversion methods on
50
SDT
100, ResNet18. Tmar = 6 is used for MTT.
Method
T-1
T-2
T-4
T-8
T-16
T-32
T-64
40
TET
QQFFS Nu (0123)
70.29
75.67
78.48
79,48
79.62
79.34
50 100 150
200
SlipReLU Diang atal. 2023
71.51
73.91
74.89
75.
75.41
75.30
74.98
Number time steps
MTT
72.09
76.54
78.47
78.90
79.17
79.25
79.42
Figure 4: MTT-trained model has uni- versally high performance across differ- ent number of time steps for VGGSNN on CIFAR10-DVS, T = 10 is used for SDT, TET and Traz 10 is used for MTT.
Table 4: Combine MTT with SEENN.
Method
T=1.20
1=1.99
SUENNI LL al.
96.07
SPENN-I FMIT
96.58
96.08
Event-driven Friendliness Since MTT-trained models have improved temporal flexibility, they are more suitable for deployment on fully event-driven chips as we claimed. To prove this, we deploy and test our MTT-trained models on event-driven neuromorphic systems. We first train two net- works on NMNIST using SDT and MTT respectively, and then deploy them directly on Speck2e Devkit (Richter et al., 2023). We then develop an easy-to-use software simulator for event-driven chips to test MTT large-scale datasets and models. Experiments show that our simulator accu- rately mimics the chip behavior. On the NMNIST dataset, the spike difference (SD, see Eq. [6) between the simulator's output and the actual on-chip output is only 3.92%. %, comparable to the
8

---

Published as a conference paper at ICLR 2025
2.81% SD between two identical model tests the same chip, much lower than the 28.77% SD be- tween time-step- step-based inference and the on-chip output. Our supportive results in Tab. 5 on various datasets and backbones strongly demonstrate the event-driven friendliness of MTT-traned models. In this section, our models show slightly lower PyTorch test accuracy compared to models with similar backbones because they are bias-free to be deployed on chips. Fully event-driven chips like Speck eliminate time-step-wise operations, including clocked bias addition, making them extremely energy-efficient and ideal for always-on scenarios. See Sec. AA3 for more training details.
Table S: Comparison model performance across different datasets and methods.
Dataset
Backbone
Param Size
Method
Torch
Simulator
Speck
N-MNIST
3CIFC(W16)
6160
MTT (Tmax=10)
99.16
98.56
98.57
SDT (T=10)
98.09
93.07
92.77
DVS-Gesture
4C2FC(W32)
48768
MTT (Tmax=40)
88,28
81.82
SDY (T=40)
88.67
80.68
MTT (Tmax=10)
75.2
58.5
CIFARIO-DVS VGGSNN
9228416
SDT (T=10)
74.7
48.4
Network Generalization As men- tioned carlier in the Sec. [4. our method makes the network param- eters robust against network struc- ture changes and adds to the mod- els' generalization. verified this through experiments with 18 on CIFAR100, A common method to measure the model's generalization is noise in- jection. First, we randomly in- ject Gaussian noise (N0 o²) to weights, where o² is the variance of the noise. For each o², run the experiment times and plot the mean, maximal, and minimal accuracy in Fig. Results show that the weights trained by MTT are more robust against noise. Then, inject Gaussian noise into the inputs instead, and also run the experiment 5 times each o², the results are shown in Fig. 6 In addition, solidify the conclusion, we also inspect the generalization in other metries (see Sec. A.I.p. All the experiments indicate that the model obtained by MTT performs better generalization.
2:41
2.63
Accunecy
We further
342
HTT 5OT
30
0.06
0.32
Norse Variarce (x20"))
Figure Accuracy of mod- Figure 6: Accuracy of models els with weights injected with with inputs injected with Gaus- Gaussian noise
sian noise
5.2 ABLATION STUDY
Evolution from SDT to NMT to MTT Our improvement to NMT mainly lies in dividing networks into timesste--iifferent stages. In this section, we test MTT with different partition granularity on CIFARIO0 with R8sNet-18 to validate the effectiveness of network partition- ing. We define granularity constant g as the number of blocks per stage. NMT can be now seen as a special case where g=8 (there are 8 blocks in ResNet-18). Then, we train 4 SNNs with MTT and g=1,2,4,8 respectively, and an- other single SNN with SDT for comparison. Fi- nally, we assess test accuracy for each model with T=2,3.4.5,6. The results are shown in Fig. (A). As expected, with g continued to reduce and more temporal structures added to the opti- mization space, the overall performance is gen- crally improved.
A
B
0.8
0.6-
72-
0.7
ON caleracion Wif bratton
70
Twd
0.0
LONTO
(019T) SUT
100
200
300
Number of blocks stape
Numbbrr of spochh
Figure 7: (A) Accuracy of ResNet-18 with different granularity where g=8 denotes NMT and g=1 denotes MTT. SDT denotes single model trained T=6 and tested across T=2.3.4.5. (B) Training ResNet-18 on FAR100 and tracing the test accuracy with and without BN calibration.
Effectiveness of BN Calibration In this section, we studied the necessity of BN calibration. We first trained two ResNet-18 on CIFARI00 and tracked their accuracy, one with BN calibration on 10 batches before testing and one simply using running means and variances. The results are shown in Fig. (B). Our experiment indicates that without correct BN statistics, the model suffers huge accuracy degradation and that BN calibration effectively ameliorates the degradation.

---

Published as a conference paper at ICLR 2025
Tablee 6: Compare with existing works static image datasets. denotes introducing additional floating point multiplications
Datasel
Model
Methods
Architecture
TimeStep
Accuracy
90.
Guo al.
Guo 2022
InfLoR-SNN
ResNet-19
26.07700.07
94.44:10.08
91.5011.07
Dong al.
Dong all
2022
TET
ResNet-19
94.44::0005
14.06.00.33
CIFARIO
95, 95.03++0.08
Yao al.
Yio
2022
GLIF
ResNet-19
94.85-00.07
96.34100:03
Our Method
MIT
ResNet- 19
96.75110:44
96.20+00007
Li Lictal al.
Li
2021b
Dspike
ResNet-18
73.35.10.14 0.144
14.2400.10
al.
InfLoR-SNN
ResNet-19
79.51
0.00
71172:10.26
CIFAR 100
Dong al.
Dong al
2023
TET
ResNet-19
74.47:10.15
Yao ot al
2022
GLIF
ResNet-19
77.05++0.2
D7.35++0X077
Our Method
MTT
ResNet-19
Zheng et
Zheng ol 202
STBP-WIN
ResNet-44
6
63.72
Deng
TET
ResNet-34
64,79
Fang
Fang 0021
SEWI
SEW-ReNNe-34
67.04
ImageNet
Chen Chen 2023
MPSNN
DSNN-34
67.52
Our Method
MTT
ResNet-34
68.34
67.54
Table 7: Compare with existing works on DVS datasets. *denotes introducing additional floating-pots multi-
plications
Dataset
Model
Methods
Architecture
Accuracy
GLIf
WB-wideNet
78.10
Guo al.
(Gib et
2012
InfLoR-SNN
ResNet-19
10
75.50=+0. 12
CIPARIO-DVS
Zhu
TCIA-SNNI
VGGSNN
80.7
Deng al.
[Teng m.
2022
TET
VGGSNN
10
83.17+0.1 IS
Our Method
MTT
Re
10
82.8 10 54(83.5)
Kim et al. (Kim und Pancar 2021
SALT
VGGI
20
35.0
N-Caltech101
Li et eL 2039
NDA
VGG11
10
78.2
Zhu et al. Zhu et 2022a
TCIA-SNNI
VGGSNN
14
78.5
Our Metiod
MIT
ResNeF18
10
81774007382.30
5.3 COMPARISON TO EXISTING WORKS
While our work mainly focuses on improving the temporal flexibility of networks, the models trained by MTT maintain performance par with other SOTA methods. See Sec. A.2 for training details. Here, we compare the accuracy of models trained by MTT with existing works. Remarkably, to demonstrate the superior temporal flexibility of MTT-trained networks, for one backbone and one dataset in the table, we trained only once and tested for all different T. For all experiments, we apply the sampling number s unless otherwise specified. The results of static and neuromorphic datasets are provided in Tab. [aand Tab. 7 We repeat the experiment three times to report the mean and standard deviation (see Sec. A.2 for details).
CONCLUSION
In this paper, we have identified "temporal inflexibility", ", a side effect caused by the prevailing train- ing paradigms. This issue, which has yet received sufficient attention, can lead significant challenges when deploying GPU-trained models on neuromorphic devices. To respond to the de- ployment issues, we propose novel training method, Mixed Time-step Training, enhance SNN's temporal flexibility. We conduct intensive experiments on GPU, neuromorphic chips, and event- driven simulator to prove effectiveness. The results indicate that MTT significantly enhances temporal flexibility, improves compatibility with event-driven systems while maintaining perfor- mance comparable to state-of-the-art approaches. The idea of introducing diverse training temporal structures in MTT can also be applied to other training forms, such randomly accumulating input frames or adjusting time steps in online training (Xiao et al. 2022 Bellec et al., 2220). We believe methods pave path to the promising future of nearly lossless event-driven hardware deploy- ment and will inspire other impressive designs. Finally, we sincerely hope that our work will draw more academic attention to the deployment issues of SNNs event-driven neuromorphic chips.
10

---

Published as a conference paper at ICLR 2025
7 ACKNOWLEDGEMENTS
This project is supported by NSFC Key Program 62236009, Shenzhen Fundamental Research Pro- gram (General Program) JCYJ 108044444407099, NSFC General Program 61876032, and Key Laboratory of Data Intelligence and Cognitive Computing, Longhua District, Shenzhen.
REFERENCES
Guillaume Bellec, Franz Scherr, Anand Subramoney, Elias Hajek, Darjan Salaj, Robert Legenstein, and Wolfgang Maass. A solution the learning dilemma for recurrent networks of spiking neu- rons. Nature communications, 11(1):3625, 2020.
Tong Bu, Wei Fang, Jianhao Ding, PengLin Dai, Zhaofei Yu, and Tiejun Huang. Optimal ann- snn conversion for high-aceuracy and lltra-low-latencyy spiking neural networks. arXiv preprint arXiv:2303.04432 043447 2023.
Guangyao Chen, Peixi Peng, Guoqi Li, and Yonghong Tian. Training full spike neural networks via auxiliary accumulation pathway. arXiv preprint arXiv:2301. 1929, 2023.
Benjamin Cramer, Yannik Stradmann, Johannes Schemmel, and Friedemann Zenke. The heidelberg spiking data sets for the systematic evaluation of spiking neural networks. IEEE Transactions Neural Networks and Learning Systems, 33(7):2744-2757 2020.
Ekin D Cubuk, Barret Zoph, Dandelion Mane, Vijay Vasudevan, and Quoe Le. Autoaugment: Learning augmentation policies from data. arXiv preprint (rXXi::8055005011 2018
Manon Dampfhoffer, Thomas Mesquida, Alexandre Valentian, and Lorena Anghel. Are snns ally more energy-efficient than anns? an in-depth hardware-aware study. IEEE Transactions on Emerging Topics in Computational Intelligence, 7(3):71-774,, 2022.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hi- erarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248 255. leee, 2009.
Shikuang Deng and Shi Gu. Optimal conversion of conventional artificial neural networks to spiking neural networks. arXiv preprint arXiv:2103.00477 2021.
Shikuang Deng, Yuhang Li, Shanghang Zhang, and Shi Gu. Temporal efficient training of spiking neural network via gradient re- re-weighting. arXiv preprint arXiv:2202.11946 2022.
Shikuang Deng, Hao Lin, Yuhang Li, and Shi Gu. Surrogate module learning: Reduce the gradient error accumulation in training spiking neural networks. In International Conference Machine Learning, pages 7655-7657. PMLR, 2023.
Shikuang Deng, Yuhang WU, Kangrui Du, and Shi Gu. Spiking token mixer: An event-driven friendly former structure for spiking neural networks. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024.
Terrance Deiries and Graham W Taylor. Improved regularization of convolutional neural networks with cutout. arXiv preprint 1708:::1700045522 2017.
Rainer Engelken. Sparseprop: Efficient event-based simulation and training of sparse recurrent spiking neural networks. Advances in Neural Information Processing Systems, 36:3638-3657, 2023.
Wei Fang, Zhaofei Yu, Yanqi Chen, Tiejun Huang, Timothée Masquelier, and Yonghong Tian. Deep residual learning in spiking neural networks. Advances in Neural Information Processing Systems, 24.21956-21069. 2021.
Yufei Guo, Yuanpei Chen, Liwen Zhang, YingLei Wang, Xiaode Liu, Xinyi Tong, Yuanyuan Ou, Xuhui Huang, and Zhe Ma. Reducing information loss for spiking neural networks. In Com- puter Rision-ECCV 2022: 17th European Conference, Tel Aviy, Israel, October 23-27, 2022, Proceedings, Part XI, pages 36-52. Springer, 2022.

---

Published as a conference paper at ICLR 2025
lyass Hammouamri, Ismail Chalfaoui-Hassane and Timothée Masquelier. Learning delays in spiking neural networks using dilated convolutions with learnable spacings. arXiv preprint arXiv:2306.17670 2023.
Bing Han and Kaushik Roy. Deep spiking neural network: Energy efficiency through time based coding. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 28,28, 2020, Proceedings, Part X, pages 388-404. Springer, 2020.
Bing Han, Gopalakrishnan Srinivasan, and Kaushik Roy. Rmp-snn: Residual membrane potential neuron for enabling deeper high-accuracy and low-latency spiking neural network. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 13558-13566) 2020.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 770-778, 2016.
Yihui He, Xiangyu Zhang, and Jian Sun. Channel pruning for accelerating very deep neural net- works. In Proceedings of the IEEE international conference on computer vision, pages 1389- 2017.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in neural network. arXiv preprint arXiv:1503.0253// 2015.
Haiyan Jiang, Srinivas Anumasa, Giulia De Masi, Huan Xiong, and Bin Gu. A unified optimization framework of ann-snn conversion: Towards optimal mapping from activation values to firing rates. 2023.
Youngeun Kim and Priyadarshini Panda. Optimizing deeper spiking neural networks for dynamic vision sensing. Neural Networks, 144:686-698 2021.
Roel Koopman, Amirreza Yousefzadeh, Mahyar Shahsavari, Guangzhi Tang, and Manolis Sifalakis. Overcoming the limitations of layer synchronization in spiking neural networks. arXiv preprint arXiv:2408. 055000 05098, 2024.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Gregor Lenz, Kenneth Chaney, Sumit Bam Shrestha, Omar Oubari, Serge Picaud, and Guido Zarrella. Tonic: event-based datasets and transformations.. July 2021. URL
httt::://doi.rrg/// 0ww51/2811222ee 5079802 Documentation available under https://oniccreaatheedosios
Chen Li, Edward Jones, and Steve Furber. Unleashing the potential of spiking neural networks by dynamic confidence. arXiv preprint arXiv:2303. 10276 2023a.
Guoqi Li, Lei Deng, Huajing Tang, Gang Pan, Yonghong Tian, Kaushik Roy, and Wolfgang Maass.
Brain inspired computing: systematic survey and future trends. Authorea Preprints, 2023b.
Hao Li, Zheng Xu, Gavin Taylor, Christoph Studer, and Tom Goldstein. Visualizing the loss land- scape of neural nets. Advances neural information processing systems, 31, 2018.
Hongmin Li, Hanchao Liu, Xiangyang Ji, Guoqi Li, and Luping Shi. Cifar an event-stream dataset for object classification. Frontiers in neuroscience, 11:309, 2017.
Yuhang Li, Shikuang Deng, Xin Dong, Ruihao Gong, and Shi Gu. A free lunch from ann: Towards efficient, accurate spiking neural networks calibration. In International Conference on Machine Learning. pages 6316-6325. PMLR, 2021a.
Yuhang Li, Yufei Guo, Shanghang Zhang, Shikuang Deng, Yongqing Hai, and Shi Gu. Differen- tiable spike: Rethinking gradient-descent for training spiking neural networks. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Process- ing Systems, 2021b. wtt.ntttss///peeeeetee http:/ee..eeee.eeeeeeeeeeeeeee http://ewweeeeeeerrroooo/oeeeeee
12

---

Published as a conference paper at ICLR 2025
Yuhang Li, Youngeun Kim, Hyoungseob Park, Tamar Geller, and Priyadarshini Panda. Neuromor- phic data augmentation for training spiking neural networks. In Computer Esion-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022. Proceedings, Part VII, pages 631-649. Springer, 2022.
Yuhang Li, Tamar Geller, Youngeun Kim, and Priyadarshini Panda. Seenn: Towards temporal spiking early-exit neural networks. arXiv preprini arXiv:2304.01230, 2023c.
Yuhang Li, Abhishek Moitra, Tamar Geller, and Priyadarshini Panda. Input-aware dynamic timestep spiking neural networks for efficient in-memory computing. In 2023 60th ACM/IEEE Design Automation Conference (DAC), pages 1 1-6. IEEE, 2023d.
Garrick Orchard, Ajinkya Jayawant, Gregory Cohen, and Nitish Thakor. Converting static image datasets to spiking neuromorphic datasets using saccades. Frontiers in neuroscience, 9:437, 2015.
Mohammad Rastegari, Vicente Ordonez, Joseph Redmon, and Ali Farhadi. Xnor-net: Imagenet classífication using binary convolutional neural networks. In Computer Vision-ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part IV, pages 525-542. Springer, 2016.
Nitin Rathi and Kaushik Roy. Diet-smm: Direct input encoding with leakage and threshold optimiza- tion in deep spiking neural networks. arXiv preprint arXiv:2008. 03658, 2020.
Ole Richter, Yannan Xing, Michele De Marchi, Carsten Nielsen, Merkourios Katsimpris, Roberto Cattaneo, Yudi Ren, Qian Liu, Sadique Sheik, Tugba Demirci, et al. Speck: A smart event-based vision sensor with low latency 327k neuron convolutional neuronal network processing pipeline. 2023.
arXiv preprint arXiv:2304.0673
Ole Richter, Chenxi Wu, Adrian M Whatley German Köstinger, Carsten Nielsen, Ning Qiao, and Giacomo Indiveri. Dynap-se2: a scalable multi-core dynamic neuromorphic asynchronous spik- ing neural network processor. Neuromorphic Computing and Engineering. 4(1):014003, 2024.
Kaushik Roy, Akhilesh Jaiswal, and Priyadarshini Panda. Towards spike- based machine intelligence with neuromorphic computing. Nature, 575(7784):607 617, 2019.
Bodo Rueckauer, lulia-Alexandi Lungu, Yuhuang Hu, and Michael Pfeiffer Theory and tools for the conversion of analog to spiking convolutional neural networks. arXiv preprint arXiv:1612.04052. 0400222 2016.
Abhronil Sengupta, Yuting Ye, Robert Wang, Chiao Liu, and Kaushik Roy. Going deeper in spiking neural networks: Vgg and residual architectures. Frontiers in neuroscience, 13:95, 2019.
Haibo Shen, Yihao Luo, Xiang Cao, Liangqi Zhang, Juyu Xiao, and Tianjiang Wang. Train- ing robust spiking neural networks on neuromorphic data with spatiotemporal fragments. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1-5. IEFE, 2023.
Sumit B Shrestha and Garrick Orchard. Slayer: Spike layer error reassignment in time, Advances in neural information processing systems, 31, 2018.
Yujie Wu, Lei Deng, Guoqi Li, Jun Zhu, and Luping Shi. Spatio-temporal backpropagation for
training high-performance spiking neural networks. Frontiers in neuroscience, 12:331, 2018.
Yujie Wu, Lei Deng, Guoqi Li, Jun Zhu, Yuan Xie, and Luping Shi. Direct training for spiking neural networks: Faster, larger, better. In Proceedings of the AAAI conference on artificial intelligence, volume 33, pages 1311-1318. 2019.
Mingqing Xiao, Qingyan Meng, Zongpeng Zhang, Di He, and Zhouchen Lin. Online training through time for spiking neural networks. Advances in neural information processing systems, 35:20717-20730. 2022.
Man Yao, Guangshe Zhao, Hengyu Zhang, Yifan Hu, Lei Deng, Yonghong Tian, Bo Xu, and Guoqi Li. Attention spiking neural networks. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023.
13

---

Published as a conference paper at ICLR 2025
Man Yao, Jiakui Hu, Tianxiang Hu, Yifan Xu, Zhaokun Zhou, Yonghong Tian, Bo Xu, and Guoqi Li. Spike-driven transformer v2: Meta spiking neural network architecture inspiring the design of next next-generation neuromorphic chips. arXiv preprint arXiv:240..036613 2024a.
Man Yao, Jiakui Hu, Zhaokun Zhou, Li Yuan, Yonghong Tian, Bo Xu, and Guoqi Li. Spike-driven transformer. Advances in neural information processing systems, 36, 2024b.
Xingting Yao, Fanrong Li, Zitao Mo, and Jian Cheng. Glif: A unified gated leaky integrate-and-fre neuron for spiking neural networks. arXiv preprint arXiv:2210. 13768, 2022.
Jiahui Yu and Thomas Huang. Universally slimmable networks and improved training techniques. In Proceedings of the IEEE/CVF international conference computer vision, pages 1803-1811,
2019.
Hanle Zheng. Yujie Wu, Lei Deng, Yifan Hu, and Guoqi Li. Going deeper with directly- larger spiking neural networks. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 1062--1177,, 2021.
Chenlin Zhou, Liutao Yu, Zhaokun Zhou, Zhengyu Ma, Han Zhang, Huihui Zhou, and Yonghong Tian. Spikingformer: Spike-driven residual learning for transformer-based spiking neural net- work. arXiv preprint arXiv:2304.11954 2023.
Zhaokun Zhou, Yuesheng Zhu, Chao He, Yaowei Wang, Shuicheng Yan, Yonghong Tian, and Li Yuan. Spikformer: When spiking neural network meets transformer. arXiv preprint arXiv:2209. 15425, 2022.
Zhaokun Zhou, Kaiwei Che, Wei Fang, Keyu Tian, Yuesheng Zhu, Shuicheng Yan, Yonghong Tian, and Li Yuan. Spikformer v2: Join the high accuracy club on imagenet with an snn ticket. arXiv preprint arXiv:2401. 000020000..0000 2024.
Rui-Jie Zhu, Qihang Zhao, Tianjing Zhang, Haoyu Deng, Yule Duan, Malu Zhang, and Liang-Jian Deng. Teja-snn: Temporal-channe joint attention for spiking neural networks. arXiv preprint arXiv:2206. 10177. 2022a.
Yaoyu Zhu, Zhaofei Yu, Wei Fang, Xiaodong Xie, Tiejun Huang, and Timothée Masquelier. Train- ing spiking neural networks with event-driven backpropagation. Advances in Neural Information Processing Systems, 35:30528-3088 30541, 2022b.
14

---

Published as a conference paper at ICLR 2025
A APPENDIX
A.1 EXPERIMENTS ON VGG STRUCTURES
We evaluated our method on the CIFARI00 dataset using VGG architectures, besides ResNets. We treated each layer as stage in the VGG series. During experimentation, we observed that VGGI6 with three fully connected (fc) layers could not be trained effectively using the standard direct training approach (its accuracy remained limited at 1%). To tackle this issue, we merged the last three fe layers of VGG16 into one and named the resulting architecture VGG14. We set Tmaz = 5, 3, and computed the mean and standard deviation of three runs. We used the SGD) optimizer to train the model, with learning rate of 0.1, a weight decay of 0.0005, and a batch size of 256. The results, presented in Tab. indicate the effectiveness of our approach on VGG structures. Table 8: Accuracy of VGG on CIFAR100
Methods
Model
T=2
T=3
T=4
T=5
MTT
VGG14
73.53
7..57
75.27
75.72
InFLoR-SNN
VGG16
71.5610.10
A.2 TRAINING DETAILS FOR NON-EVENT-DRIVEN EXPERIMENTS
CIFAR The CIFARIO/CIFARION dataset comprises 50K training images and 10K test images with 32x32 pixel resolution. For CIFAR100, we train a ResNet-19 using the MTT pipeline for 300 epochs with a batch size of 256 and a Tmaz of 6. Following the practice in GLIF (Yao et al.) 2022). the last fully connected layers of ResNet-19 are replaced with a single fully connected layer. We employ the SGD optimizer with a weight decay of 0.0005 and a learning rate of 0.1 cosine decayed to 0. To make a fair comparison with the state-of-the thtthhear art (SOTA) work (Li et al. 2021b Guo al., 2022 Yao et al. 2022). AutoAugment (Cubuk et al. 2018) and Cutout VDev and Taylor (017) are applied both CIFARIO and CIFARIO0 datasets. However, these augmentation techniques are only used for comparative experiments and temporal flexibility experiments, and not for other experiments.
ImageNet ImageNet (Deng et al. 2009) contains more than 1 1280k training images and 50k test im- ages. We use the standard data processing flow to crop each image to a size of 224x224. We deploy the ResNet-34 structure, however, with the removal of the first max- maxx-poling layer and changing the stride of the first basic block from to 2 (Zheng et al.) 2021 Yao et al. 2022). We train the model for 160 epochs with a batch size of 512 and T mar of 6. We utilize the AdamW optimizer with weight decay of 0.02 and a learning rate of 0.004 cosine decayed to O.
DVS-Dataset There are training settings on DVS-datasets in this work, one is for compari- son with existing -taee-of-the-arts, and another is for all other experiments on event-driven datasets where ensure the models deployable on fully event-driven chips. However, the data pro- cessing of these two settings are the same. The event datasets involved in this paper, NMNIST, DVS-Gesture, CIFARIO-DVS and N-Caltech001, are neuromorphic datasets widely used in SNN experimentation. For NMNIST, we don't do any data augmentation and remain the resolution. For -ys-Gest we perform random horizontal flip and roll the frames up to 20 pixels. For CI DVS and 1-Caltech we divide the dataset into a 9:1 ratio, which, similar to previous work (Li et al., 2021b Deng et al.] 2022), we resize to 48 48. For both these datasets, we adopt a random horizontal flip and rotate the frames up to 5 pixels
augmentation techniques. We employ the additional temporal inversion policy (Shen et al. 2023) uniquely for N-Caltech101.
We then introduce the settings of SOTA comparison experiments (see next section for event-driven settings), which is for time-stepped inference and only used in See. 5.3 Following previous works, we merge all events to form ten frames. The optimizer we choose SGD, with a weight decay of 0.0005, and a learning rate of 0.1, which we cosine decay to O. For both datasets, we use a T man of 10, a batch size of 50, and train standard ResNet-18 model for 300 epochs. We take only the first I. frames of the ten frames where I. denotes the time step of the input stage, to feed into the network.
A.3 TRAINING DETAILS FOR EVENT-DRIVER EXPERIMENTS
Here, we provide the settings used by all other experiments on event-driven datasets than SOTA comparison. As mentioned, the settings introduced in the last section is only applicable to SOTA
15

---

Published as a conference paper at ICLR 2025
comparions (experiments in Sec. 5.3 All other experiments related to event datasets (e.g. Sec. use the settings described in this section, including experiments in Fig. and Tab. 5
5.1
We carefully learned Synsense's documentation. To obtain Spiking Neural Networks (SNNs) more suitable for deployment on asynchronous chips, we utilized a soft reset mechanism and multistep- IF neurons during training (neurons emit mem/Vth spikes per event to simulate multiple event trans- missions and generations). Note that, however, the neurons realistically employed on our simulator and Speck chip are still IF neurons.
Since asynchronous event-driven chips do not support linear or convolutional layers containing bi- ases, and networks with Batch Normalization (BN) layers inevitably introduce bias terms after ab- sorbing BN into the convolutional layers, we adopted more indirect method to obtain deployable networks. For 3CIFC(W166 16) (xCyFC(Wz) denotes VGG-like structure with x convolution layers, y fully connected layers, and maximum convolution channel z) on N-MNIST, For larger and deeper networks like 4C2FC(W32) for DVS-Gesture and VGGSNN for CIFARIO-DVE we first train with networks that include BN layers with SDT to establish a baseline model, then absorb BN into the convolutional layers, remove the bias, and further fine-tune to achieve a network without bias terms. MTT is applied to the bias-removal finetuning process with BN locked.
For NMNIST, our final layer an IF voting layer; for DVS-Gesture and CIFARIO-DVS, we find crucial to replace the final layer with membrane potential voting layer.
A.4 DESIGN TTM GROUPING POLICY
We previously stated our policy's 's objective is to partition / frames into is groups (1 2 k) as evenly as possible. In this section, mathematically interpret the design. We will start by describing the grouping process in a different manner. The 7 frames are viewed as adjacent intervals of length over the rational number domain with the i-th frame starting at 1 and ending at i. We define oa as the boundary between group and group i - 1. Here, ranges from 1 to k, and ei is O. Ideally, C (i- 1)--k) is set group frames most evenly. Nevertheless, this strategy produces non-integer Ca which results in atomic frames' division when I is not a multiple of k. To solve this problem, we retreat and set e, to the nearest integer and get
1)-1 (1-11))---
G =
(17)
k
where e is small constant used to determine ea when the distances to the closest two integers are equal. As by must be the frame directly following the boundary e (b. C, + 1), we obtain Eq. [
A.5 DERIVATION OF THE BACKPROPAGATION FORMULA FOR LIF
In this section, we derive the backpropagation formula for LIF from the forwarding formula. We derive @u(t)/Ou(t) from Eq. first:
du(l)
= 1 - s(1) - v()) Os(() Du(t)
(18)
Ov(t)
Then, we consider the derivation of @u(t)/Ou(t- 1)).
@u(t) du(t) dv(() Ou(t 1) Ou(() Øu(( 1)
(19)
Ou(t 1) @u(t) du(t - 1) Ov(t= - ad((t) du(t 1)
By combining multiple Eq.
19 we get
du(t)
Ou(i) Oo(i)
n
(20)
Ou(t - n) imt-m
Finally, we get the complete expression for @s(t)/OI(t n) follows
ds(t) Os(t) du(t) du(t 1) delt n)
01(1 ([[-n)
To(() Ou(l - 1) Oo(t n) 0I( n)
Os(1).
II [(1 s(i) v(1) 0s((), dv(i)
1-1
(21)
Do(t)
i=t-x
16

---

Published as a conference paper at ICLR 2025
A.6 T= MAX BN STATISTICS VS. RECALCULATED BN STATISTICS
As previously mentioned, we discovered that the BN statistics of the T=T network can be applied to other temporal structures with uniform across blocks. In this section, we provide experimental verification for this observation. Our experiment involves testing the accuracy of one of our trained ResNet- 19 models with two distinct BN layer information approaches. The first approach utilizes the statistics of the T=T max, while the second approach recalculates the BN statistics individually for each time step T. For the latter approach, we calibrate the BN layer three times and report the average accuracy. Our experimental results demonstrated in Tab. 10 show that the mean and variance calculated at T=T is applicable directly to other values. Therefore, we can utilize the BN statistics of T=T max for other T directly, which saves the time required to calibrate the BN layers for other values.
Table 9: Accuracy given sampling number and training epochs e.
Table 10: Test accuracy of single model with kinds of BN statistics.
Sampling Num
é. X
300
600
900
1200
TimeStep
Method
75.61
76.13
76.23
75.78
4
5
2
75.53
76.37
76.31
76.36
T=T moce stat
80.21
81.06
81.51
81.82
75.25
76.45
76.47
76.44
Recalculated stat
80.21
81.23
81,44
81.85
74.81
75.74
76.41
76.42
A.7 IMPACT OF DIFFERENT SAMPLING NUMBER AND TRAINING EPOCHS
In most previous experiments, we employed sampling number 3. In this section, we experiment with varying values of s, assess their effects at different epochs, and explain why we chose = 3. We train ResNet- 18s with Tmaz = on CIFAR1O0 with varied and list their test accuracy at T=6. The results are displayed in Tab. 9 When X is constrained, the model requires more epochs to converge, necessitating lower s. However, if trained across sufficient number of epochs, sampling of structurally diverse networks can smooth the optimization of network parameters and improve performance. Specifically, we find s = performs well and adopt 8 = 3 for most of the experiments.
78
0.56
== 0.844
75
O. 0.898
bedicaee 20 lnndccas
Teat Aucansey
0.945
8444--72
0.000w+00
0.84-
Teat Accurity
73
72
0.325
88
-8420-2 433--00-2--- 6.300 5.00 0228
0.93
0.505
0.04
0.346
0.56
72 Estimolo Accusayy
Estimate Accuracy
Figure 10: The ID landscapes of
Figure 8: Correlation curve for timate accuracy and test accuracy on CIFARIO
Figure 9: Correlation curve for estimate accuracy and test accuracy on CIFAR100
ResNet18 88 trained by SDT/MTT on CIFAR100.
A.8 PARTITIONED SNN ACCURACY ESTIMATION FOR DIFFERENT TEMPORAL CONFIGURATIONS ON 1IFAR10/100
As MTT divides SNN into stages of different when training, we are interested in the performance of each possible temporal configuration t. Although the stage-wise different time steps are only applicd to the MTT training process to improve the temporal flexibility of SNNs and are not for inference, the relationship between t and inference performance can inspire SNN structure designs. However, is impossible to test all possible situations directly (e.g.. there are 6 cases for ResNet- 18). Therefore, we propose the following hypotheses estimate the accuracy of each t: 1) The expressive power of each stage contributes differently and positively to the final network accuracy. 2) The expressive power of each stage is related to the information content of its selected time T, such K / /lgg T. where the square root is due the information content of spike sequence with
17

---

Published as a conference paper at ICLR 2025
time step T cannot exceed logg T because the arrangement of spikes is regular, e.g.. tending to be uniformly distributed. Then we assume that the accuracy equation of different temporal configura- tions is Ex= K; /oy2 Ti + c, where the K2 is the contribution of each block, and с is a constant. Our experiment demonstrates that we only need to sample a very small number of ts to infer the parameters of the equation, and then able to estimate accuracies of all temporal configurations with this equation.
Here, we randomly sample 18 temporal configuration vectors (different time step combinations) and obtain their test accuracy for solving the hypothesis equation in Sec. AA.] The weight parameters we obtained are (0.93, 0.53, 0.59, 0.67, 1. 1.22, 0.48, 1.36, 0.18), 18), and the constant value e. is 67.11. This result supports our hypothesis 1). which suggests that all blocks' time step increment positively contributes to the accuracy of the final network. Some blocks, such as blocks 1, 5, and 7, have greater contribution. Then, we resample 1000 configurations and validate their estimated accuracy effectively predict the actual testing accuracy of different temporal configurations. Finally, we use the spike frequency and accuracy estimation build combinatorial optimization equation for searching the optimal configuration (see Sec [A.9 for detail). For example, by setting the energy cost that is lower than default (the time step of all blocks 3), we discover the optimal combination of block time steps is {3, 2, 2, 3, 5, 3, 6, 2}. The selected configuration acquires an accuracy of 75.38%, which is 0.62% higher than the default.
and their test accuracy. The result (Fig. 7 (A)) shows that method
In addition to CIFARI00 100, we also conducted experiments with ResNet- 18 on CIFAR10/100, We ran- domly sample 18 temporal configurations as usual and solve the equation in Sec. A.9 The weight parameters are {0.0049, 0.0028, 0.0017, 0.0037, 0.0037, 0.0004, 0.0005, 0.0017). and the constant value is 92.13. Notes that block 1,4,5 have a higher contribution, and the 1,5 blocks are also highly weighted on CIFAR1O0 100, which may imply that the weights are partly related to the network structure. We then resample 1000 temporal configurations and plot their estimate accuracy and test accuracy CIFARIO in Fig. g and CIFARIO 100 in Fig respectively. We also use the aforemen- tioned combinatorial optimization strategy to search the optimal temporal configuration under the energy consumption of T=3 and find {4, 2, 2, 3, 5, 2, 2, 4), which achieves accuracy of 94.70%, 0.21% % higher than its T=3 counterpart.
A.9 DETAILS OF COMBINATORIAL OPTIMIZATION
As previously mentioned, the accuracy formula of different temporal structures given by the ex- pression Σl= K; Vlogz t4 + c, where K, represents the contribution of each block, is the number of blocks, and is a bias. We randomly select 18 distinct temporal configurations (t) and evaluate their accuracies on the test set, resulting in 18 pairs of temporal configurations and their corre- sponding accuracies. Using the least squares method, collected data. Next, we estimate the average firing rate (R₂) of each block in a unified SNN of T=6. Then, the energy consumption of a specified temporal configuration can be approximated as Σ/,4 4 Rg For example, the estimated energy consumption of a unified SNN with T=4 is calculated as EC2 = E/= 4R, Based on this, we can obtain a group of temporal configurations with lower energy consumption (EC) for given T-T,- and we aim to identify the config with the maximum estimated accuracy from this set. This is formulated as the following optimization problem:
compute the values of K; and from the
maximize
ACC esttmmmteed
Σ Ki Vlgg +
Σ 4. Rí S F.CT,
s.t.
T minn
S t1, .t. S T Tmas:
where t,is the i-th component of temporal configurations t and ECT, is the given uper bound of energy.
18

---

Published as a conference paper at ICLR 2025
In order to solve the above problem, we adopt the depth- search (DFS) algorithm to search in the solution space. To obtain a more accurate accuracy for each temporal configuration t, we perform three times of BN calibrations and take the average of the accuracies.
A. LOSS LANDSCAPES OF MTT AND SDT
To visually confirm the flatter minimum achieved by the model trained with MTT. we trained two ResNer18 using SDT and MTT respectively on CIFAR100 for 300 epochs and plotted their loss landscapes in Fig. TO The ID landscape is plotted using the code provided by L al. (2018), whose basic idea is view the trained weights as sighddimnssoon points and plots its surroundings. We observed that MTT led the model to flatter minimum which indicates improved generalizability.
A.11 VERIFYING GENERALIZABILITY THROUGH GRADIENT METRICS
Apart from noise injection, another famous metric that indicates the generalizability is the length 2f
AC ||. For l1 ala ||. we evaluate the length of of the gradient on weights sw || and the inputs al
ac
the gradient of loss over the entire training set for the convolution layers. For || [1 I]. we calculate the mean value of the length of each input gradient. The model trained by MTT exhibits a shorter
gradient of both weights and inputs (see Tab. TTI. which implies the model's ' strong robustness and generalizability.
Table 11:
The gradi- Table 12: Results on Table 13: Results on Spiking Table 14: Results on Spiking Speech Com- mands
ent statistics of the model SeqMNIST. trained by SDT and MTT.
Heidelherg Digits
Methods
Acc
Methods
Acc
Methods
Our RNN
56.22
Hammouamii al. 2023
EJ Repro by code of
75.26
Methods
Aec
MTT
SDT
38.08
11.99
7.78
1.1
SNN MTT
SNN SDT
64.56
55.75
1=3 our MIT
Im our S17T
79.68
74.43
Our MTT 1m3
Our SDT E $7.75
60.15
A. EXPERIMENTS ON AUDIO AND SEQUENTIAL DATASETS
Our research reveals that models trained by MTT can function effectively as a time encoder when the temporal configuration vectors used for training are monotonically non- non-increasing.
To illustrate this adaptability, we present the performance on three distinct temporal tasks: seqM- NIST, Spiking Heidelberg Digits and Spiking Speech Commands.
For the sequential task SeqMNIST, we utilized simple fc LIF SNN with hidden layers of width 64 and set the time constant r to 0.99. We also trained an RNN with 2 hidden layers of width 64 for comparison. The results are as shown in Tab. 12
For the Spiking Heidelberg Digits, we adopt the plain 3-layer feed-forwand SNN architecture pro- posed Cramer et al. (2020), fully connected SNN with an input width of 70 and 128 LIF neurons in each of the 3 hidden layers. The timestep of the first layer is fixed to the input timestep, while the timesteps of subsequent layers are restricted to monotonically non-increasing. We set 7 0.9753, which is equivalent to the parameter À in the work of Cramer et al. (2020), namely 1 - 1/7 in most other articles, and train the model for 150 epochs. To ensure the validity of our results, we also re- produce the result using the code provided by Hammouamri et al. 2023 The results as shown in Tab. 13
Spiking Speech Commands (SSC) (Cramer et al. 2020) is a spiking dataset converted from Google Speech Commands v0.2 and is tailored for SNN. For SSC, we continue using the same architecture and the same parameters as we used in SHD, except that here we only train the model for 60 epochs. The results are shown in Tab. [4]
A.13 COMPUTATIONAL COST ANALYSIS FOR MTT
Time Complexity When training with GPUs, the time required for single forward and backward pass of normal SNNs is proportional to the time steps T.. Therefore, the time cost for normal SNN with I' time steps within single iteration can be expressed as:
C(T) T k
19

---

Published as a conference paper at ICLR 2025
where k is the time cost of single time step. Now consider the cost of MTT. Before inserting the TTM module, the time cost at stage can be expressed as C,(T) = T k; where = E, kss Let CTTи denote the total time cost of all TTM modules. The total time cost for one iteration of MTT can then be expressed as:
CMIT CrIM +
kj
Here, 8 is the sampling times, and is the total number of stages. Note that BN calibration is only needed before inference, which requires only few forward passes and incurs negligible overhead during training.
Due to the randomness in temporal configuration sampling, calculate the expectation of the time cost under given min and max as follows:
Tiin + T max
((CMrr(TTnae mmmx)) E(Crim(Tminn mmax))
+ )
2
Since single TTM module involves at most Timx tensor multiplications and additions, its cost is negligible compared to the main model. After ignoring the CTTM term, the time cost expectation is:
ECCMTT (Tmin: Tmax)) 2e s min + T Tmax
2
Thus, the time cost ratio between MTT and SDT is approximately:
(T min + Tmax)
2T
We verified this analysis by testing the first-epoch time of SDT and MTT various datasets and models. All the experiments were conducted with RTX3090 GPUs and data- duta-parallel training. The results are shown in T5
Table 15: Experimental results of first-epoch training time for both MTT and SDT. MTT config is denoted by o[ Tiii, Tmax] where s is sampling times each iteration, [nai, T max] Is the sampling range of
Model
Dataset
GPUs
Batch Size
MIT 2 4 Tadl
MIT Time
SDT T
SDT Time
Actual
Theory
ResNet19
CIFART00
3
256
3[1.6]
193s
6
10%
1.71x
L.75x
ResNet19
CIFAR100
256
3[1. 10]
1.65x
ResNet18
CIFARIO-DVS
2
50
3[1. 10]
1238
10
77s
1.60x
1.65x
As shown, the experimental results align well with the theoretical analysis. According to analy- sis, MTT's overhead is approximately 1.5 times that of SDT when not too small.
Space Complexity MTT performs immediate backward passes after forward passes and accumu- lates gradients of all temporal configurations sampled within a single iteration. Because the com- putation graph and temporary tensors instantly released after backpropagation, the theoretical maximal memory usage of MTT is comparable to standard SDT. However, since the maximal mem- ory usage only occurs when the time steps of all stages are set to T, the intermediate memory usage of MTT may be smaller than SDT. We tested the GPU memory usage at the end of the first epoch, and the results are shown in Tab. [6 The experimental results confirm that MTT's memory usage is consistent with theoretical expectations.
Table 16: Experimental results of first-epoch memory usage for hoth MTT and SDT, 8 Tuinn Tmax] denotes MTT samples s temporal configs each iteration, each time step is sampled from [Tmi,, T Timmx
Model
Datasct
GPUs
Method
MTT Memory (per GPU)
ResNet18
CI-ARI
MTT 311. 10)
6640MiB
ResNet18
CIFARIO-DVS
MIT 3[10, 10]
7025MiB
ResNet18
CIFARIO-DVS
SDT 7' 10
7295MBB
20


Reviews:
[{"review_id": "aStBggN2Pa", "reviewer": "Reviewer_wNg9", "summary": "This paper introduces a training method, Mixed Time-step Training (MTT), aimed at improving the temporal flexibility of Spiking Neural Networks (SNNs) for more versatile deployment. The training method allows SNNs to adapt across diverse time steps by assigning random time steps to different stages of the network in each iteration. After training, TFSNNs are deployed and evaluated on both time-step-based and event-driven platforms. The authors compare their method on static and DVS datasets with ANN-SNN conversion methods (Jiang et al.2023) and SEENN (Li et al., 2023), emphasize its generalization to different time steps.", "questions": "Please refer to the weaknesses section. Additionally, there are two more questions:\n1. The authors suggest that their models can infer across various time steps without additional fine-tuning. However, whether flexibility across all time steps is necessary, especially outside of event-driven scenarios, remains an open question. Given the potential complexity of the proposed training approach (see Weakness 1 for details), it may be more efficient and focused to fine-tune the model for specific possible platforms rather than attempting universal temporal flexibility.\n2.The authors compare their method to the current state-of-the-art (SOTA) in ANN-to-SNN conversion (Table 3) and report improved performance when T is low. However, performance declines slightly as T increases. Likewise, in Table 4, although naïve mixture training demonstrates some advantage over standard direct training at smaller T values, this benefit diminishes as T approaches 5 or 6. This raises an question: given that SNNs have limited capacity to capture temporal dynamics across time steps when T is very small, is this improvement practically significant?", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1.\tThe paper introduces a novel training method to address the side effects of temporal \ninflexibility caused by the prevailing training paradigms.\n2.\tThe paper conducts intensive experiments on GPU, neuromorphic chips, and event-\ndriven simulator to testify to its effectiveness.", "weaknesses": "1. The proposed method aims to enhance the model's temporal flexibility through mixture training. However, the authors do not provide a clear analysis of the training complexity or computational costs involved. Table 9 highlights the relationship between the sampling frequency and training epochs, yet further details are needed to elucidate these aspects comprehensively.\n2. Some performance improvements reported by the authors appear less substantial upon closer examination. For example, in Table 4, the addition of MTT has a very limited effect on improving overall performance. Similarly, in the generalization comparison involving Gaussian noise-injected inputs (Figure 5), while the accuracy of the MTT method consistently exceeds that of SDT, the margin is minimal compared to the overall drop in accuracy as noise intensity increases. These observations make it challenging to substantiate the claim that the model’s generalization is significantly enhanced.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "VJaeGtgDh3", "reviewer": "Reviewer_cDVC", "summary": "The paper proposes the Mixed Time - step Training (MTT) method, which is utilized to train Spiking Neural Networks, thereby achieving generalization across distinct time steps. This method empowers SNNs to accommodate diverse temporal structures. Furthermore, the paper conducts validation experiments to confirm the effectiveness of MTT. The method devises specific loss, a temporal transformation module, and network partitioning to fulfill the above-mentioned objective.", "questions": "Could you provide a clear definition of V(t) when it is first introduced in section 3.1, and explain its significance in the context of the neuron model？", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "The paper presents a promising SNN training approach, namely MTT, which attains remarkable outcomes in terms of temporal adaptability and model performance, possessing definite innovation and practical value. The co-training method involving multiple time steps is both rational and effective. The experiments are comprehensive, having been conducted on diverse datasets, and there are also tests on event - driven neuromorphic systems.", "weaknesses": "1.The writing in this paper has room for improvement. The authors ought to place greater emphasis on the key points throughout the article. For instance, concerning the temporal flexibility across time steps, the experiment should stress that different time steps perform well in general. Avoid delving too deeply into which particular time step works well, as this may obscure the aim of the work. The authors should focus on presenting the overall performance trend across various time steps instead of detailing the results for each separate time step. This would enhance the emphasis on the key point of generalization across time steps.\n\n2. Some additional details regarding the experiment and the method are required. Specifically, an explanation must be given as to why the time steps in Table 4 are represented as decimals. Moreover, the meanings of the symbols used should be clearly elucidated.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "2WT9UQQSXG", "reviewer": "Reviewer_spLP", "summary": "This paper identifies the temporal flexibility problem for spiking neural networks (SNNs) that is important for SNNs’ deployment on time-step-free fully event-driven chips, and proposes a novel mixed time-step training method to alleviate the problem under current direct training approaches. For evaluation, the trained models are tested under both time-step-based and fully event-driven settings, where the latter includes both the Speck chip and a developed simulator. Experiments show promising performance for temporal flexibility, robustness, deployment to the fully event-driven setting, and commonly used static and neuromorphic datasets.", "questions": "To alleviate the influence of time steps, will it be better to consider the combination with some online-through-time training methods with instantaneous losses [1,2] rather than backpropagation through time, since the former can naturally consider information from different time steps and may be suitable for on-chip training to better adapt to real settings?\n\n[1] Online training through time for spiking neural networks. NeurIPS, 2022.\n\n[2] A solution to the learning dilemma for recurrent networks of spiking neurons. Nature Communications, 2020.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. This paper considers an important problem in bridging algorithms of SNNs and the real deployment on fully event-driven neuromorphic hardware. It is significant for real applications.\n\n2. Experiments are comprehensive, covering not only the commonly adopted GPU simulation settings but also real chips or a similar event-driven simulator. Various datasets including static images, neuromorphic images, and audio, have been considered to verify the good performance of the proposed method.\n\n3. Results show that the proposed method can achieve near SOTA performance and has much better performance when deployed under the fully event-driven setting.", "weaknesses": "1. The paper lacks sufficient details for the considered fully event-driven setting. For example, what are the details of the Speck chip and the developed simulator? How is input or output formulated, and how does asynchronization influence the network? This can affect some claims, for example, “large-scale SNNs on fully event-driven scenarios”, since only N-MNIST is verified on the real chip and other experiments are on the simulator. These details should be included to enable justification if simulator experiments can support the claim.\n\n2. For the presentation, there are some not fully discussed logical gaps. \n\nFirst, there is a gap between the identified temporal inflexibility problem and deployment on fully event-driven chips, because the former is still in the time-step-based setting while the latter is in the time-step-free setting. It is better to add more explanations about why the considered flexibility under the synchronized setting can certainly improve time-step-free settings, e.g., why flexibility can alleviate the problem caused by asynchronization. \n\nSecond, the motivation from NMT to MTT is missing.\n\nThird, there is no formal and rigorous definition for temporal flexibility. Even for SNNs trained with a specific T, they can naturally run for different time steps, just with a drop in performance. To what extent can a model be called flexible or inflexible? For the proposed method, there is also a performance drop and the improvement is to reduce it rather than introducing a new property. The concept is mainly a quantitative comparison instead of a qualitative one, so I think it is not rigorous to claim the proposed method to “exhibit temporal flexibility”.\n\n3. This paper mainly focuses on empirical verifications while theoretical analysis is limited.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0hq25sVX4P", "reviewer": "Reviewer_g9wC", "summary": "This work first addresses the limitations of using fixed time steps in conventional SNN training, which lead to low generalization capability and performance degradation during practical deployment. Additionally, it highlights the challenges in dynamically balancing energy-performance trade-offs under this constraint. Starting from the Naive Mixture Training, this manuscript incrementally develops a framework that incorporates hybrid time-step and temporal flexibility training methods. By applying a strategic up/down rounding technique, the authors group the stages of networks like VGG and ResNet into multiple dynamic time-step segments for training. Notably, the paper presents results on neuromorphic hardware (Speck V2) and simulation results on asynchronous platforms, offering fresh insights into the practical advantages of dynamic time-step strategies.\nThe manuscript is logically clear and coherent, with the methodology presented in a gradual and systematic manner.\nHowever, I still have a few questions and suggestions for modifications (and additions).", "questions": "1. To my knowledge, the highest-performing architecture in the SNN field is the Spiking Transformer [1-5]. Please discuss whether the proposed method can be effectively applied to the Spiking Transformer. If feasible, please provide some preliminary experimental results.\n\n[1] Zhou, Z., Zhu, Y., He, C., Wang, Y., Shuicheng, Y. A. N., Tian, Y., & Yuan, L. Spikformer: When Spiking Neural Network Meets Transformer. In The Eleventh International Conference on Learning Representations.\n\n[2] Yao, M., Hu, J., Zhou, Z., Yuan, L., Tian, Y., Xu, B., & Li, G. (2024). Spike-driven transformer. Advances in neural information processing systems, 36.\n\n[3] Zhou, C., Yu, L., Zhou, Z., Ma, Z., Zhang, H., Zhou, H., & Tian, Y. (2023). Spikingformer: Spike-driven residual learning for transformer-based spiking neural network. arXiv preprint arXiv:2304.11954.\n\n[4] Zhou, Z., Che, K., Fang, W., Tian, K., Zhu, Y., Yan, S., ... & Yuan, L. (2024). Spikformer v2: Join the high accuracy club on imagenet with an snn ticket. arXiv preprint arXiv:2401.02020.\n\n[5] Yao, M., Hu, J., Hu, T., Xu, Y., Zhou, Z., Tian, Y., ... & Li, G. Spike-driven Transformer V2: Meta Spiking Neural Network Architecture Inspiring the Design of Next-generation Neuromorphic Chips. In The Twelfth International Conference on Learning Representations.", "limitations": "", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The manuscript is well-structured, logically coherent, and clearly articulated.\n\n2. The figures are well-designed.\n\n3. The defined spike difference has practical significance, offering a valuable metric for optimizing GPU training and on-chip deployment.\n\n4. Experimental results demonstrate that the proposed MTT-enabled TFSNN outperforms the baseline in accuracy and exhibits stronger generalization capabilities.", "weaknesses": "1. In line 234 of the manuscript, the time step range is described as \"$T_{max}$ to $G^{T_{max}}$\" Should  $G^{T_{max}}$ be corrected to ${T_{max}}^{G}$ here?\n\n2. What is the difference between $T=a_i$ and $T=t_i$ in Figure 1? Are they referring to different sample sets? If so, please clarify this in the caption.\n\n3. The time step serves as a search space, which has a certain relationship with Neural Architecture Search (NAS). Please provide some discussion on the connection between the two.\n\n4. Although it is understood that the focus of this paper is on reducing the gap between training and deployment, the sampling of samples and the grouped calculation of loss evidently increase training overhead. Please provide some discussion on this in the appendix.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "c3n0SW5Ry6", "reviewer": "Reviewer_ksDE", "summary": "The manuscript introduces a new method, MTT, for training spiking neural networks (SNNs) that perform well across different numbers of timesteps during inference. These SNNs are referred to as Temporally Flexible SNNs (TFSNNs). The proposed method divides an SNN model into multiple stages, assigns each stage a random number of timesteps, and conducts multiple forward passes on the same data batch. The outputs are aggregated into a single loss function, and backpropagation through time (BPTT) is used to update the weights. Experimental results demonstrate that this approach enables the SNN model to perform consistently across varying timesteps during inference. Additionally, models trained using this method show enhanced generalization and robustness to noise, as well as high accuracy across multiple static and dynamic image classification datasets.", "questions": "- What specific problem is the manuscript addressing, deployment on event-driven hardware or inference consistency across multiple timesteps? How does the proposed method align with this problem?\n- What are the differences between the LIF models in Section 3.1 and those used in event-driven simulations?\n- How does the method generalize to values of  $T $ outside the range used during training ($ [T_{\\text{min}}, T_{\\text{max}}] $) on event-based datasets?\n- How was Figure 9 generated?\n- What is the distinction between SDT and SDT* in Table 1?\n- Could you clarify the \"specific application scenarios\" referenced in line 208?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "The manuscript identifies an interesting issue with direct training methods for SNNs: their limited performance when performing inference under a different timestep configuration than the one used during training. It proposes a solution that allows models to perform well across a range of timesteps, enhancing their flexibility and robustness. The models trained under this approach demonstrate improved generalization and competitive accuracy compared to SOTA direct training methods. Furthermore, the manuscript is overall well-written, with a clear and accessible description of the proposed method.", "weaknesses": "The problem definition and the proposed method's alignment with it remain somewhat unclear. The manuscript introduces the problem of \"temporal inflexibility\" in standard direct training methods, suggesting that this limitation could affect SNN deployment on fully event-driven hardware. However, the manuscript's focus is primarily on synchronous discrete models trained on static image datasets, with only a minor experiment (Table 5) dedicated to event-driven models, which lacks sufficient detail. Thus, the manuscript does not fully address the implications of \"temporal inflexibility\" for event-driven hardware deployment, instead centering on the need for inference performance consistency across multiple timesteps, a goal that may not directly relate to event-driven applications.\n\nIf the objective is to facilitate SNN deployment on event-driven hardware, the manuscript should clarify the differences between models described in Section 3.1 and fully event-driven models. For instance, Section 5.1 mentions that SPECK removes time-step-wise operations like clocked bias addition. How does the LIF model change after removing such operations? Conversely, if the goal is multi-timestep inference, further explanation is needed to justify its relevance for static image classification tasks. In these cases, where temporal information is absent, the optimal approach would be to achieve accurate inference with the minimum number of timesteps (ideally $T=1$, as in ANNs).\n\nThe experimental setup and results presentation require further clarification. For example, in Section 5.1, \"Temporal Flexibility Across Time Steps\", Table 2 is not referenced in the text, making it unclear what these results indicate. Additionally, in Table 3, which compares the proposed method with SOTA ANN-SNN methods, the settings for the ANN-SNN methods are unclear, was the conversion applied once for a single $T$ value, then tested across various $ T $ values, or was the conversion performed individually for each $ T $ value in Table 3? Similar issues with experimental descriptions and discussions are present in Sections 5.1 and 5.3.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)