Temporal Efficient Training of Spiking Neural Network via Gradient Re-weighting
Authors: Shikuang Deng
Yuhang Li
Shanghang Zhang
Shi Gu
Venue: ICLR 2022 Poster (2022)
Keywords: Spiking Neural Networks
Direct Training
Surrogate Gradient
Generalizability

Abstract: Recently, brain-inspired spiking neuron networks (SNNs) have attracted widespread research interest because of their event-driven and energy-efficient characteristics. It is difficult to efficiently train deep SNNs due to the non-differentiability of its activation function, which disables the typically used gradient descent approaches for traditional artificial neural networks (ANNs). Although the adoption of surrogate gradient (SG) formally allows for the back-propagation of losses, the discrete spiking mechanism actually differentiates the loss landscape of SNNs from that of ANNs, failing the surrogate gradient methods to achieve comparable accuracy as for ANNs. In this paper, we first analyze why the current direct training approach with surrogate gradient results in SNNs with poor generalizability. Then we introduce the temporal efficient training (TET) approach to compensate for the loss of momentum in the gradient descent with SG so that the training process can converge into flatter minima with better generalizability. Meanwhile, we demonstrate that TET improves the temporal scalability of SNN and induces a temporal inheritable training for acceleration. Our method consistently outperforms the SOTA on all reported mainstream datasets, including CIFAR-10/100 and ImageNet. Remarkably on DVS-CIFAR10, we obtained  83% top-1 accuracy, over 10% improvement compared to existing state of the art.

Published as a conference paper at ICLR 2022
TEMPORAL EFFICIENT TRAINING OF SPIKING NEURAL NETWORK VIA GRADIENT RE- RE-WEIGHTING
Shikuang 2 Yuhang Li², Shanghang Zhang & Shi Gu 1.250
University of Electronic Science and Technology of China, Shenzhen Institute for Advanced Study, UESTC 3 Yale University, "Peking University, Peng Cheng Laboratory Lengsk119@stdr uesto. edu. cn, yuhang. li@yale..dee.. gus@uestc.edu. edu.ceedducoo
ABSTRACT
Recently, brain-inspired spiking neuron networks (SNNs) have attracted widespread research interest because of their event-driven and energy-efficient characteristics. Still, it is difficult to efficiently train deep SNNs due to the non- differentiability of its activation function, which disables the typically used gradi- ent descent approaches for traditional artificial neural networks (ANNs). Although the adoption of surrogate gradient (SG) formally allows for the back- pack-propagation of losses, the discrete spiking mechanism actually differentiates the loss landscape
SNNs from that ANNs, failing the surrogate gradient methods to achieve comparable accuracy for ANNs. In this paper, first analyze why the current direct training approach with surrogate gradient results in SNNs with poor gen- eralizabilitty. Then introduce the temporal efficient training (TET) approach to compensate for the loss momentum the gradient descent with SG so that the training process can converge into flatter minima with better generalizabil- ity. Meanwhile, we demonstrate that TET improves the temporal scalability of SNN and induces a temporal inheritable training for acceleration. Our method consistently outperforms the SOTA on all reported mainstream datasets, includ- ing CIFAR-10/100 and ImageNet. Remarkably on DVS-CIFARIO, we obtained 83% top-1 accuracy, over 10% improvement compared to existing state of the art. Codes are available at https: / gith.b..h...eeeg com/ Gus Lab/ watuuss-abtttttrrrrreeeee temporal_ afficient_trainiiee
INTRODUCTION
The advantages of Spiking neuron networks (SNNs) lie in their energy-saving and fast-inference computation when embedded on neuromorphic hardware such as TrueNorth (DeBole et al., 2019) and Loihi (Davies et al., 2018). Such advantages originate from the biology-inspired binary spike transmitted mechanism, by which the networks avoid multiplication during inference. On the other hand, this mechanism also leads to difficulty in training very deep SNNs from scratch because the mon-differentiable spike transmission hinders the powerful back-propagation approaches like gradient descents. Recently, many studies on converting artificial neuron networks (ANNs) to SNNs have demonstrated SNNs' comparable power in feature representation ANNs (Han & Roy, 2020; Deng & Gu, 2020; Li et al., 2021a). Nevertheless, it is commonly agreed that the direct training method for perforaeffe SNN is still crucial since it distinguishes SNNs from converted ANNs, especially on neuromorphic datasets.
The output layer's spike frequency or the average membrane potential increment is commonly used as inference indicators in SNNs (Shrestha & Orchard, 2018; Kim et al., 2019). The current standard direct training (SDT) methods regard the SNN RNN and optimize inference indicators' distri- bution (Wu et al., 2018), 8). They adopt surrogate gradients (SG) to relieve the non-differentaabiliy (Lee al., 2016; Wu et al., 2018; Zheng et al., 2021). However, the gradient descent with SG does not match with the loss landscape in SNN and is easy to get trapped in local minimum with low generalizability. Although using suitable optimizers and weight decay help ease this problem, the
Corresponding author

---

Published as a conference paper at ICLR 2022
Standard Direct Training
Leaky Integrate- Fire
Post-spike
Output Integrate
Label
Omeax
4(Omvon: y)
Pre naplic Triput
Class
FC
2010
Σ, ((00,))
(0.00
Surrogite Cradient
Temporal Efficient Training
Forward
Rackward
Temporal Efficient Training Forward
RificieenTrraiingg
Figure 1:
Workflow of temporal efficient training (TET). To obtain more generalized SNN, modify the optimization target to adjust each moment's 's output distribution.
performance of deep SNNs trained from scratch still suffers a big deficit compared to that of ANNs Deng al. (2020). Another training issue is the memory and time consumption, which increases linearly with the simulation time. Rathi & Roy (2020) initializes the target network by a converted SNN to shorten the training epochs, indicating the possibility of performnrf SNN with limited activation time. The training problem due to the non-diffeeenniaabe activation function has become the main obstruction of spiking neural network development.
In this work, we examine the limitation of the traditional direct training approach with SG and propose the temporal efficient training (TET) algorithm. Instead of directly optimizing the integrated potential, TET optimizes every moment's pre-synaptic inputs. As a result, it avoids the trap into local minima with low prediction error but a high second-order moment. Furthermore, since the TET applies optimization each time point, the network naturally has more robust time scalability. Based on this characteristic, we propose the time inheritance training (TTT), which reduces the training time by initializing the SNN with smaller simulation length. With the help of TET, the performance of SNNs has improved on both static datasets and neuromorphic datasets. Figure depicts the workflow of our approach.
The following summarizes our main contributions:
We analyze the problem of training SNN with SG and propose the TET method, new loss and gradient descent regime that succeeds in obtaining more generalizable SNNs.
We analyze the feasibility of TET and picture the loss landscape under both the SDT and TET setups to demonstrate TET's advantage in better generalization.
Our sufficient experiments on both static datasets and neuromorphic datasets prove the effectiveness of the TET method. Especially DVS- CVS-CIFARIO, we report 83.17% top-1 accuracy for the first time, which is over 10% hetter than the current state-of-the-ard result.
RELATED WORK
In recent years, SNNs have developed rapidly and received more and more attention from the re- search community. However, lots of challenging problems remain to be unsolved. In general, most works on SNN training have been carried out two strategies: ANN-to-SNN conversion and direct training from scratch.
ANN-to-SNN Conversion. Conversion approaches avoid the training problem by trading high ac- curacy through high latency. They convert ligh-eerfrming ANN to SNN and adjust the SNN parameters w.r. the ANN activation value layer-by-layer (Diehl et al., 2015; 2016). Some special techniques have been proposed to reduce the inference latency, such the subtraction mechanism
2

---

Published as a conference paper at ICLR 2022
(Rueckauer et al., 2016; Han et al., 2020), robust normalization Rueckauer et al. (2016), spike-norm (Sengupta et al., 2018), and channel- wiie normalization (Kim et al., 2019). Recently, Deng & Gu (2020) decompose the conversion error to each layer and reduce it by bias shift. Li et al. (2021a) suggest using adaptive threshold and layer-wise calibration to obtain high-performance SNNs that require simulation length of less than 50. However, converted methods significantly extend the inference latency, and they are not suitable for neuromorphic data (Deng et al., 2020).
Direct training. In this area, SNNs are regarded as special RNNs and training with BPTT (Neftci et al., 2019). On the backpropagation process, The non-differentiable activation term is replaced with a surrogate gradient (Lee al., 2016). Compared with ANN-to-SNN conversion, direct train- ing achieves high accuracy with few time steps but suffers more training costs (Deng et al., 2020).
Several studies suggest that surrogate gradient (SG) is helpful to obtain high-performance SNNs on both static datasets and neuromorphic datasets (Wu et al., 2019; Shrestha & Orchard, 2018; Li et al., 2021b). On the backpropagation process, SG replaces the Dirac function with various shapes of curves. Exceptionally, Wu et al. (2018) first propose the STBP method and train SNNs on the ANN programming platform, which significantly promotes direct training development. Zheng et al. (2021) further proposes the tdBN algorithm to smooth the loss function and first realize training large-scale SNN on ImageNet. Zhang& & Li (2020) proposes TSSL-BP to break down error back- propagation across two types of inter-neuron and intra-neuron dependencies and achieve low-latency and high accuracy SNNs. Recently, Yang et al. (2021) designed a neighborhood aggregation (NA) method to use the multpple perturbed membrane potential waveforms the neighborhood com- pute the finite difference gradients and guide the weight updates. They significantly decrease the required training iterations and improve the SNN performance.
PRELIMINARY
3.1 ITERATIVE LIF MODEL
We adopt the Leaky Integrate-and-Fire (LIF) model and translate it to an iterative expression with the Euler method (Wu et al., 2019). Mathematically, the membrane potential is updated as
u(l +1+ 1) = ru(() + I()),
(1)
where 7 is the constant leaky factor, u(t) is the membrane potential at time t, and I(t) denotes the pre-synaptic inputs, which is the product of synaptic weight W and spiking input at).. Given specific threshold VaA, the neuron fires spike and utt) reset to O when the utt) exceeds the threshold. So the firing function and hard reset mechanism can be described as
a(t 1) = 1 + 1) Vin)
(2)
u(t 1) u + (1- a(t +
(3)
where O denotes the Heaviside step function. The output spike a(t + 1) will become the post synaptic spike and propagate to the next layer. In this study, we set the starting membrane u0)) to O. the threshold VAA to 1, and the leaky factor 7 to 0.5 for all experiments.
The last layer's spike frequency is typically used as the final classification index. However, adopting the LIF model on the last layer will lose information on the membrane potential and damage the performance, especially on complex tasks (Kim et al., 2019). Instead, we integrate the pre-synaptic inputs I (t) with no decay or firing (Rathi & Roy, 2020; Fang et al., 2021). Finally, we the average membrane potential as the classification index and calculate the cross-entropy loss for training.
3.2 SURROGATE GRADIENT
Following the concept of direct training, we regard the SNN as RNN and calculate the gradients through spatial-temporal backpropagation (STBP) (Wu et al., 2018):
ÖL
aL da(t) du(t) aI(t)
(4)
aw
da(t) du(t) aI(1) OW
da(t) where the term Du(lJ is the gradient of the non-differentiabilii step function involving the derivative
of Dirac's 5-function that is typically replaced by surrogate gradients with derivable curve. So far,
3

---

Published as a conference paper at ICLR 2022
there various shapes of surrogate gradients, such rectangular (Wu al., 2018; 2019), triangle (Esser et al., 2016; Rathi Roy, 2020), and exponential (Shrestha Orchard, 2018) curve. this work, choose the surrogate gradients shaped like triangles. Mathematicallyy, it can describe
@a(1) 1 Du(t) 72
max(0, /e((t) Vial):
(5)
where the denotes the constraint factor that determines the sample range to activate the gradient.
3.3 BATCH NORMALIZATION FOR SNN
Batch Normalization (BN) (loffe & Szegedy, 2015) is beneficial to accelerate training and increase performance since it can smooth the loss landscape during training (Santurkar al., 2018). Zheng et al. (2021) modified the forward time loop form and proposed threshold-dependent Batch Nor- malization (tdBN) normalize the pre-synaptic inputs I in both spatial and temporal paradigms so that the BN can support spatial-temporal input. We adopt this setup with the extension of the time dimension to batch dimension !. In the inference process, BN layer will be merged into pre-convolutional layer, thus the inference rule of SNN remain the same but with modified weight:
W - w b 3 b 2)
(6)
where Jr, a are the running mean and standard deviation on both spatial and temporal paradigm, 7, B the affine transformation parameters, and W, are the parameters of the pre-convolutional layer.
METHODOLOGY
4.1 FORMULA OF TRAINING SNN WITH SURROGATE GRADIENTS
Standard Direct Training, We use Ott) to represent pre-synaptic input [(t) of the output layer and calculate the cross-entropy loss. The loss function standard direct training LSDT is:
LSM LSM== = LCR( Σ o()).).
where 7' is the total simulation time, LeE denotes the cross-entropy loss, and y represents the target label. Following the chain rule, we obtain the gradient of W with softmax S(.) inference function
dL Iis(Omenn (5(Omean) --] OW
T
80(t)
(8)
T tm1
2W
where Omean denotes the average of the output O()) over time, and is the one-hot coding of y.
Temporal Efficient Training. In this section, we come up with a new kind of loss function LIET to realize temporal efficient training (TET). It constrains the output (pre-synaptic inputs) at each moment to be close to the target distribution. It is described as:
T Σ
1
LIET
T Cce(O(t), v).
(9)
Recalculate the gradient of weights under the loss function CIET, and we have:
TLT 1
80(()
Σ (((0())) - 8w
(10)
OW
T
hhttps://gi thw..ttthhhhoo www. fangwei /23456/spikingie11

---

Published as a conference paper at ICLR 2022
4.2 CONVERGENCE OF GRADIENT DESCENT FOR SDT V.S. TET
In the of SDT, the gradient consists of two parts, the error term (S(O mean) - ÿ) and the partial derivative of output (1(1)/ow) When the training process reaches near a local minimum, the term (S(Omeea) ÿ) approximates for all & 1,.... 7, ignorant of the term (t) (t) For traditional ANNs, the accumulated momentum may help get out of the local minima (e.g. saddle point) that typically implies bad generalizability (Kingma & Ba, 2014; Kidambi et al., 2018). However, when the SNN is trained with surrogate gradients, the accumulated momentum could be extremely small, considering the mismatch of gradients and losses. The fact that the activation function a step one while the SG is bounded with integral constraints. This mismatch dissipates the momentum around a local minimum and stops the SDT from searching for flatter minimum that may suggest better generalizability.
In the case of TET, this issue of mismatch relieved by reweighting the contribution of (t Indeed, considering the fact that the first term (S(O(t)) ý) is impossible to be 0 at every moment of SNN since the early output accuracy on the training set is not 100%. So TET needs the second term (t close to 0 to make the LIET convergence. This mechanism increases the norm of gradients around sharp local minima and drives the TET to search for a flat local minimum where the disturbance of weight does not cause huge change in O(t).
Further, to ensure that the convergence with TET implics the convergence of SDT, we prove the following lemma:
Lemma 4.1. LSDT upper bounded by LTET-
Proof. Suppose (z) and y; denote the i-th component of O() and ÿ. respectively. Expand Eqn.9,
we have:
1
1
CIET
U log S(O,(t)) = T
Σ
y log S(O,(t)))
T
[=1
Σ Úi log(
((O,())))
Σ
n
U log
1
Σ s(O.()))
T 1=1
t=1
T
2 yi log(S( Σ O.())) = LSDT;
(11)
t=1
where the first inequality is given by the Arithmetic Geometee Mean Inequality, and the second one is given by Jensen Inequality since the softmax function is convex. As corollary, once the LrEr gets closed to zero, the original loss function LSDT also approaches zero.
Furthermore, the network output (t) a particular time point may be a particular outlier that dra- matically affects the total output since the output of the SNN has the same weight at every moment under the rule of integration. Thus it is necessary to add a regularization term like LMSE loss to confine each moment's output to reduce the risk of outliers:
T
LMSE
Σ
MSE ()))
(12)
T
where ó is a constant used to regularize the membrane potential distribution. And we set ó Va in our experiments. practice, we use hyperparameter adjust the proportion of the regular term, we have:
LOTTAL = (1 - LTTT ALMSE-
(13)
It is worth noting that only changed the loss function in the training process and did not change SNN's inference rules in the testing phase for a fair comparison. This algorithm is detailed in Algo.
5

---

Published as a conference paper at ICLR 2022
Algorithm 1: Temporal efficient training for one epoch
Input: SNN model; Simulation length: T: Threshold: VnA: Training dataset; Validation dataset;
total training iteration in one epoch: Itrain; total validation iteration in one epoch: Fea
for all i 1, 1.2,...Trrin iteration do
Get mini- training data, and class label: Y
Compute the SNN output O' (1)) of eatch time step;
Calculate loss function: LTOTAL (1 TET + AS
(1 E (O' (t), Y*) + + E (S)(()()),
Backpropagation and update model parameters;
end
for all i = 1, 1,.2,...vall iteration do
Get mini-batch validation data, and class label: Y
Compute the SNN average output O O T Ei= (t) over all time step;
Compare the classification factor O mcan and Y for classification;
end
4.3 TIME INHERITANCE TRAINING
SNN demands simulation length long enough obtain a satisfying performance, but the training time consumption will increase linearly as the simulation length grows. So how to shorten the training time is also an essential problem the direct training field. Traditional loss function LSDT only optimizes the whole network output under a specific T, so its temporal scalability is poor. Unlike the standard training. TET algorithm optimizes each moment's s output, enabling us to extend the simulation time naturally. We introduce Time Inheritance Training (TIT) to alleviate the training time problem. We first use long epochs to train an SNN with a short simulation time T, e.g., 2. Then, increase the simulation time to the target value and retrain with short epochs. We discover that TIT performs better than training from scratch on accuracy and significantly saves the training time. Assuming that training an SNN with simulation length T 1 cost is time per epoch, the SNN needs 300 epochs to train from scratch, and the TIT needs 50 epochs for finetuning. So we need 1800 time to train SNN with T = 6 from scratch, but following the TTT pipeline with the initial T only requires 900ts. As result, the TIT can reduce the training time cost by half.
5 EXPERIMENTS
We validate our proposed TET algorithm and compare it with existing works on both static and neuromorphic datasets. The network architectures in this paper include ResNet-19 (Zheng et al., 2021), Spiking-ResNet33 (Zheng et al., 2021). VGGSNN. SNN-5 lutional SNN suitable for multiple runs to discover statistical rules (Figure A. 7). The architec- ture of VGGSNN FC) is based on VGG11 with two fully connected layers removed as we found that additional fully connected layers were unnecessary for neuromorphic datasets.
(Fang et al., 2021), SNN-5, and is simple convo-
12C3-11223-A22-5122--
5.1 MODEL VALIDATION AND ABLATION STUDY
Effectiveness of TET over SDT with SG. We first examine whether the mismatch between SG and loss causes the convergence problem. For this purpose, we set the simulation length to 4 and change the spike function in Eqn.2 to Sigmoid aKk input). We find that the TET and SDT achieved similar accuracy (Table 2) when &- 1, 10, 20. This indicates that both TET and SDT work when the gradient and loss function match each other. Next, we compare the results training with LSDT and LTET on SNNs (ResNet-19 on CIFAR 100) training with surrogate gradient for three runs. As shown in Table 1, our proposed new TET training strategy dramatically increases the accuracy by 3.25% when the simulation time is 4 and 3.53% when the simulation time is 6. These results quantitatively support the effectiveness of TET in solving the mismatch between gradient and loss in training SNNs with SG.

---

Published as a conference paper at ICLR 2022
Staded Bied 1ET
SEGGE Trakig
501 Lese
127
Figure 2: Loss landscape of VGGSNN. The 2D landscape of LSeT and LrET from two different training methods.
Table 1: Comparison between SDT and TET. We adopt the SNN architecture ResNet-19 with
Table 2: Comparison of SDT and TET with sig- moid function ok (fk-inpu).. We fix the simulation length to 4 and record the results of CNN-5 un- der three different & on CIFAR10.
SG on CIFAR 100 and record the results with three different simulation lengths 2, 4, and 6.
Method
T=2
T=4
T=6
Method
k=
1=10
k=20
Direct training
70.86:10222 72.37+00.1
69.41::0.88
7L12#0.37
Direct training
88 00
88.3340.2
88 50.00.332
TET
74.11+0.118
74.65+0.112
TET 87,63+0.38
89.31+0.115
88.64+0.28
Loss Landscape around Local Minima. We further inspect the 2D landscapes (Li et al., 2018) of LSDT and LTET around their local minima (see Figure. 2) demonstrate why TET generalizes better than SDT and how TET helps the training process jump out of the sharp local minima typically found by SDT. First, comparing Figure. 2 A and C. we can see that although the values of local minima achieved by SDT and TET are similar in LSDT, the local minima of TET (Figure. 2 C) flatter than that of SDT (Figure. A). This indicates that the TET is effective finding flatter minima that are typically more generalizable even w.r.t the original loss in TET. Next, examine the two local minima under LTET to see how it helps jump out the local minima found by SDT. When comparing Figure. B and D. observe that the local minima found by SDT (Figure. B) not only sharper than that found by TET (Figure. 2 D) under LSDT but also maintains a higher loss value. This supports our claim that TET loss cannot be easily minimized around sharp local minima (Figure. B), thus preferable to converge into flatter local minima (Figure. 2 D). Put together, the results here provide evidence for our reasoning in Section 4.2.
Training from SDT to TET. In this part, we further validate the ability of TET to escape from the local minimum found by SDT. We adopt the VGGSNN with 300 epochs training on DVS-CIFARIO. First, we optimize LSDT for 200 epochs and then change the loss function to LreT after epoch 200. Figure demonstrates the accuracy and loss change on the test set. After 200 epochs training, SDT
10
60
SDT LewiChunge TET) LeTLoaaCTeegal le TETT
30-
20-
150
100
150
1.50
2.0
Opoche
Figure 3: TET helps to jump out the local minimum point. We provide the test accuracy (A) and loss (B) change after changing the SDT to TET at epoch 200. TET efficiently improves the test performance and reduces two kinds of loss.

---

Published as a conference paper at ICLR 2022
A
B
75.0
750
14.5
74.0
73.5
Accunes
73.0
Thoning Tit
7200
Plactaating Infrec manrag
Thetig I
Thoning Tm4
72.0
Thening from acaads
71.0
T+4
T-3
T+2
11.0
70.
69.0
Stmulation leugth
Exergy (mn))
Figure 4: Time scalability robustness and network efficiency of ResNet-19 on CIFARI00, (A) The comparison of training from scratch (dots) and inheriting from small simulation length (lines). (B) SNN network performance changes with energy consumption.
gets trapped into local minimum, and the LSDT no longer decreases. The CrET is much higher than LSDT since SDT does not optimize it. Nevertheless, after we change the loss function to LIET, the LTET and LSDT on the test set both have a rapid decline. This phenomenon illustrates the TET ability to help the SNN efficiently jump out of the local minimum with poor generalization and find another flatter local minimum.
Time Scalability Robustness. Here, we study the time scalability robustness of SNNs trained with TET (LTET). First, we use 300 epochs to train small simulation length ResNet-19 on CIFARI00 as the initial SNN. Then, we directly change the simulation length from 2 to 8 without finetuning and report the network accuracy on the test set. Figure. 4. displays the results after changing the simulation length. We use 2, 3, and 4, respectively,
the simulation length of the initial network. When we increase the simulation length, the accuracy of all networks gradually increases. After the simulation time reaches certain value, the network performance will slightly decrease. Interest- ingly, SNNs trained from scratch (T=4 and T=6) are not as good as those trained following the TTT procedure.
Network Efficiency. In this section, we measure the relationship between energy consumption and network performance. SNN avoids multiplication on the inference since its binary activation and event-based operation. The addition operation in SNN costs 0.9p./ energy while multiplication operation consumes 4.6p./ measured in 45nm CMOS technology (Rathi Roy, 2020). our SNN model, the first layer has multiplication operations, while the other layers only have addition operations. Figure B summarizes the results of different simulation times. In all cases, the SNN obtained by TET has higher efficiency.
5.2 COMPARISON TO EXITING WORKS
In this section, we compare our experimental results with previous works. We validate the full TIT algorithm (CrOtal) both on the static dataset and neuromorphic dataset. All of the experiment results are summarized in Table 5.2. We specify all the training details in the appendix A.1.
CIFAR. We apply TET and TIT algorithm on CIFAR (Krizhevsky al., 2009), and report the mean and standard deviation of 3 runs under different random seeds. The À is set to 0.05. On CIFARIO, our TET method achieves the highest accuracy above all existing approaches. Even when T 2, there is a 1.82% increment compare to STBP-ddBN with simulation length T = 6. It is worth noting that our method is only 0.47% lower than the ANN performance. TET algorithm demonstrates more excellent ability on CIFARI00. It has accuracy increase greater than 3% all report simulation lengths. In addition, when = 6, the reported accuracy is only 0.63% lower than that of ANN. We CIFAR100, where the generalizability the model distinguishes a lot among minima with different flatness.
that the proposed TET's improvement is even higher complex data like
8

---

Published as a conference paper at ICLR 2022
Table 3: Compare with existing works.
Our method improves network performance across all tasks. results. denotes data augmentation (Li et al., 2022).
denotes
Dataset
Model
Methods
Architecture
Simulation Length
Accuracy
Rathi al. (2019)
Hybrid training
ResNct-20
92.22
Rathi Roy (2020)
Diet-SNN
ResNct-20
92.54
Wu al. (2018)
STRP
CIFARNet
12
89.83
Wu et al. (2019)
STBP NeuNorm
CIFARNel
12
90.53
Zhang & Li (2020)
TSSL-BP
CIFARNet
91.41
93.16
CIFARIO
Zheng et (2021)
STBP-dBN
ResNet-19
92.34
94,50-10.07
our model
TET
ResNct-19
94.44:10.08
94.16+0003
ANN
ANN
ResNet-19
94.97
Rathi al. (2019)
Hybrid training Dict-SNN
VGG-II
125
67.87
Rathi & Roy (2020)
ResNet-20
64.07
71.12+00.57
Zheng et al. (2021y
STBP-dBN
ResNet-19
70.86+t0.22
CIFAR100
69.41110.08
74.72:10.28
our model
TET
ResNet-19
74.47 1515
72.87
ANN
ResNet-19
75.35
Rathi al. (2019)
Hybrid training
ResNet-34
61.48
Sengupta al. (2018)
SPIKE-NORM
ResNct-34
2500
69.96
Zheng et al. (2021)
STBP-dBN
Spiking-ResNet-2. ResNet-34
63.72
ImageNet
Fang et al. (2021)
SEW ResNet
SEW- Ne44Re
67.04
TET
Spiking- NesNet-34
Spiking-ResNete
64,79
model
TET
SBW-ResNet-3d
68.00
Zheng et (202T)
STEP
ResNet-19
67.8
Kugelé al. (2020)
Streaming Rollout
DenseNet
66.8
DVS-CIFARIO
Wu al. (2021)
Conv3D
LIAF-Net
10
71.70
Wu et al. (2021)
LIAF
LJAF-Net
10
70.40
TET
VGGSNN
77.33
model
TET
VGGSNN
83.17.10.15
ImageNet. The training set of ImageNet (Krizhevsky et al., 2012) provides 1.28k training samples for each label. We choose the two most representative ResNet-34 verify our algorithm on Ima- geNet with À = 0.001. SEW-ResNer34 is not a typical SNN since it adopts the IF model and mod- ifies the Residual structure. Although we only train our model for 120 epochs, the TET algorithm achieves 1.07% increment Spiking-ResNet- and 0.96% increment on NeW-ResNe
DVS-CIFARIO. The neuromorphic datasets suffer much more noise than static datasets. Thus the well-trained SNN is easier overfit these datasets than static datasets. DVS-CIFARI (Li al., 2017), which provides each label with 0.9k training samples, is the most challenging mainstream neuromorphic dataset. Recent works prefer deal with this dataset complex architectures, which are more susceptible to overfitting and do not result in very high accuracy. Here, we adopt VGGSNN on the DVS-CIFARID 10 dataset, set À 0.001, and report the mean and standard deviation of runs under different random seeds. Along with data augmentation methods, VGGSNN can achieve an accuracy of 77.4 77.4%. Then we apply the TET method to obtain more generalizable optima. The accuracy rises to 83.17%. Our TET method outperforms existing state-of-ther art by 11.47% accuracy. Without data augmentation methods, VGGSNN obtains 73.3% accuracy by SDT and 77.3% accuracy by TET.
CONCLUSION
This paper focuses on the SNN generalization problem, which described as the direct training SNN performs well the training set but poor the test set. We find this phenomenon is due to the incorrect SG that makes the SNN easily trapped into a local minimum with poor generalization. To solve this problem, periments verify that our proposed method consistently achieves better performance than the SDT process. Furthermore, TET significantly improves the time scalability robustness of SNN, which
propose the temporal efficient training algorithm (TET). Extensive

---

Published as a conference paper at ICLR 2022
enables propose the time inheritance training (TIT) to significantly reduce the training time consumption by almost a half.
7
ACKNOWLEDGMENT
This project is supported by NSFC 61876032 and JCYI20210034140807001 Li completed this work during his prior research assistantship in UESTC.
REFERENCES
Mike Davies, Narayan Srinivasa, Tsung-Han Lin, Gautham Chinya, Yongqiang Cao, Sri Harsha Choday, Georgios Dimou, Prasad Joshi, Nabil Imam, Shweta Jain, et al. Loihi: neuromorphic manycore processor with on-chip learning. leee Micro, 38(1):82-99)) 2018.
Michael V DeBole, Brian Taba, Arnon Amir, Filipp Akopyan, Alexander Andreopoulos, William P Risk, Jeff Kusnitz, Carlos Ortega Otero, Tapan Nayak, Rathinakumar Appuswamy, et al. Truenorth: Accelerating from zero to 64 million neurons in 10 years. Computer, 52(5):20-29, 2019.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hi- erarchical image database. In 2009 IEEE conference computer vision and pattern recognition, pp. 248-255. leee, 2009.
Lei Deng, Yujie Wu, Xing Hu, Ling Liang, Yufei Ding, Guoqi Li, Guangshe Zhao, Peng Li, and Yuan Xie. Rethinking the performance comparison between snns and anns. Neural Networks, 1 307, 2020.
Shikuang Deng and Shi Gu. Optimal conversion of conventional artificial neural networks to spiking neural networks. In International Conference Learning Representations, 2020.
Peter U. Diehl, Daniel Neil, Jonathan Binas, Matthew Cook, and Shih Chii Liu. Fast-classifying. high-accuracy spiking deep networks through weight and threshold balancing. In Neural Net- works (IJCNN), 2015 International Joint Conference on, 2015.
Peter U Diehl, Guido Zarrella, Andrew Cassidy, Bruno U Pedroni, and Emre Neftci. Conversion of artificial recurrent neural networks to spiking neural networks for low-power neuromorphic hardware. In 2016 IEEE International Conference on Rebooting Computing (ICRC), pp. 1-8. IEEE, 2016.
Steven K. Esser, Paul A Merolla, John V Arthur, Andrew S Cassidy, Rathinakumar Appuswamy, Alexander Andreopoulos, David Berg, Jeffrey McKinstry, Timothy Melano, Davis R Barch, et al. Convolutional networks for fast, energy-efficient neuromorphic computing. Proceedings of the national academy of sciences, 1 113(41): 1441--11111111111 1446, 2016.
Wei Fang, Zhaofei Yu, Yanqi Chen, Tiejun Huang, Timothée Masquelier, and Yonghong Tian. Deep residual learning in spiking neural networks. arXiv preprini arXiv:2102.04159 2021.
Priya Goyal, Piotr Dollár, Ross Girshick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, An- drew Tulloch, Yangqing Jia, and Kaiming He. Accurate, large minibatch sgd: Training imagenet in 1 hour. arXiv preprint arXiv: 1oX.v:1706.02677. 2017.
Bing Han and Kaushik Roy. Deep spiking neural network: Energy efficiency through time based coding. In European Conference on Computer Vision, 2020.
Bing Han, Gopalakrishnan Srinivasan, and Kaushik Roy. Rmp-snn: Residual membrane potential
Proneennnneeeeeeeeeeeeee neuron for enabling deeper high-accuracy and low-latency spiking neural network. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 33558-13567, 2020.
Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International conference on machine learning, pp. 448-456. PMLR, 2015.
10

---

Published as a conference paper at ICLR 2022
Rahul Kidambi, Praneeth Netrapalli, Prateek Jain, and Sham Kakade. On the insufficiency of ex- isting momentum schemes for stochastic optimization. In 2018 Information Theory and Applica- fions Workshop (ITA), pp. 1-9. IEEE, 2018.
Seijoon Kim, Seongsik Park, Byunggook Na, and Sungroh Yoon. Spiking-yolo: Spiking neural network for energy-efficient object detection. arXiv preprint axXiv:1903.065300 2019.
Youngeun Kim and Priyadarshini Panda. Visual explanations from spiking neural networks using inter-spike intervals. Scientific reports, 11(1):1-14, 2021.
Youngeun Kim, Yeshwanth Venkatesha, and Priyadarshini Panda. Privatesnn: Fully privacy- preserving spiking neural networks. arXiv preprint arXiv:2104.03414 2021.
Youngeun Kim, Yuhang Li, Hyoungseob Park, Yeshwanth Venkatesha, and Priyadarshini Panda. Neural architecture search for spiking neural networks, arXiv preprint arXiv:2201.10355 2022.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Alex Krizhevsky, I. Sutskever, and G. Hinton. Imagenet classification with deep convolutional neural networks. Advances in neural information processing systems, 25(2), 2012.
Alexander Kugele, Thomas Pfeil, Michael Pfeiffer, and Elisabetta Chicca. Efficient processing of spatio-temporal data streams with spiking neural networks. Frontiers in Neuroscience, 14:439, 2020.
Jun Haeng Lee, Tobi Delbruck, and Michael Pfeiffer. Training deep spiking neural networks using backpropagation. Frontiers in neuroscience, 10:508, 2016.
Hao Li, Zheng Xu, Gavin Taylor, Christoph Studer, and Tom Goldstein. Visualizing the loss land- scape of neural nets. In Proceedings of the 32nd International Conference on Neural Information Processing Systems, pp. 6391-6401, 2018.
Hongmin Li, Hanchao Liu, Xiangyang Ji, Guoqi Li, and Luping Shi. Cifar10-dvs: event-stream dataset for object classification. Frontiers in neuroscience, 11 11:309, 2017.
Yuhang Li, Shikuang Deng, Xin Dong, Ruihao Gong, and Shi Gu. A free lunch from ann: Towards efficient, accurate spiking neural networks calibration. arXiv preprint arXiv:2106. 069006944. 2021a.
Yuhang Li, Yufei Guo, Shanghang Zhang, Shikuang Deng, Yongqing Hai, and Shi Gu. Differ- entiable spike: Rethinking gradient-descent for training spiking neural networks. Advances in Neural Information Processing Systems, 34, 2021b.
Yuhang Li, Youngeun Kim, Hyoungseob Park, Tamar Geller, and Priyadarshini Panda. Neuromor- phic data augmentation for training spiking neural networks. arXiv preprint arXiv:2203.066 06145, 2022.
Emre O Neftci, Hesham Mostafa, and Friedemann Zenke. Surrogate gradient learning in spiking neural networks: Bringing the power of gradient- based optimization to spiking neural networks. IEEE Signal Processing Magazine, (666):51-633 2019.
Nitin Rathi and Kaushik Roy. Diet-snn: Direct input encoding with leakage and threshold optimiza- tion in deep spiking neural networks. arXiv preprint arXiv:2008.0568 2020.
Nitin Rathi, Gopalakrishnan Srinivasan, Priyadarshini Panda, and Kaushik Roy. Enabling deep spiking neural networks with hybrid conversion and spike timing dependent backpropagation, In International Conference Learning Representations, 2019.
Bodo Rueckauer, Alexa--aaa Lungu, Yuhuang Hu, and Michael Pfeiffer. Theory and tools for the conversion of analog to spiking convolutional neural networks. arXiv: Statistics/Machin Learning, (1612.04052):0-0, 2016.
11

---

Published as a conference paper at ICLR 2022
Ali Samadzadeh, Fatemeh Sadat Tabatabaci Far, Ali Javadi, Ahmad Nickabadi, and Morteza Haghir Chehreghani. Convolutional spiking neural networks for -patio-temporal feature extraction, arXiv preprint arXiv:2003.12346. 2020.
Shibani Santurkar, Dimitris Tsipras, Andrew Ilyas, and Aleksander Madry. How does batch nor- malization help optimization? In Proceedings of the 32nd international conference on neural information processing systems, pp. 8488-2490 2018.
Abhronil Sengupta, Yuting Ye, Robert Wang, Chiao Liu, and Kaushik Roy. Going deeper in spiking neural networks: Vgg and residual architectures. Frontiers in Neuroence, 13, 2018.
Sumit Bam Shrestha and Garrick Orchard. Slayer: Spike layer reassignment in time. In Advances in Neural Information Processing Systems, pp. 1412-1421, 2018.
Yujie Wu, Lei Deng, Guoqi Li. Jun Zhu, and Luping Shi. training high-performance spiking neural networks. Frontiers in neuroscience, 12:331, 2018.
backpropagation for
Yujie Wu, Lei Deng, Guoqi Li, Jun Zhu, Yuan Xie, and Luping Shi. Direct training for spiking neural networks: Faster, larger, better. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pp. 1311-1318, 2019.
Zhenzhi Wu, Hehui Zhang, Yihan Lin, Guoqi Li, Meng Wang. and Ye Tang. Liaf-net: Leaky inte- grate and analog fire network for lightweight and efficient spatiotemporal information processing. IEEE Transactions on Neural Networks and Learning Systems, 2021.
Yukun Yang, Wenrui Zhang, and Peng Li. Backpropagated neighborhood aggregation for accu- rate training of spiking neural networks. In International Conference on Machine Learning, pp- 11852-11866 1162.. PMLR, 2021.
Wenrui Zhang and Peng Li. Temporal spike sequence learning via backpropagation for deep spiking neural networks. arXiv preprint arXiv:2002. 10085, 2020.
Hanle Zheng, Yujie Wu, Lei Deng, Yifan Hu, and Guoqi Li. Going deeper with directly-trained larger spiking neural networks. In Proceedings of the AAAI Conference Artificial Intelligence, volume 35, pp. 11062-11070, 2021.
12

---

Published as a conference paper at ICLR 2022
A APPENDIX
A.1 DATASET AND TRAINING DETAIL
CIFAR. The CIFAR dataset (Krizhevsky et al., 2009) consists of 50k training images and 10k testing images with the size of 32 32. We use ResNet-19 for both CIFARIO and CIFARI00. Moreover, random horizontal flip and crop are applied to the training images the augmentation. First, we use 300 epoch train the SNN with the simulation length T 2. We use Adam optimizer with a learning rate of 0.01 and cosine decay to O. Next, following the TIT algorithm, we increase the simulation time (to 4 and 6) and continue training the SNN for only 50 epochs, with the learning rate changing to le- - 4.
ImageNet. ImageNet (Deng et al., 2009) contains more than 1250k training images and 50k valida- tion images. We crop the images to x24x222 and using the standard augmentation for the training data. We use SGD optimizer with 0.9 momentum and weight decay 4e 5. The learning rate set to 0.1 and cosine decay to O. We train the SEW-ResNetts (Fang et al., 2021) with T = for 120 epochs. As for the
ResNet34 (Zheng et al., 2021), use TIT algorithm train 90 epochs with T 4 first, then change the simulation time to 6 and finetune the network for 30 epochs. We adopt an Adam optimizer on the finetune phase and change the learning rate to le TIT algorithm significantly reduces the training time consumption since training the Spiking-ResNet24 is extremely slow.
DVS-CIFARIO. DVS-CIFARI0 (Li et al., 2017), the most challenging mainstream neuromor- phic data set, is converted from CIFARIO. It has 10k images with the size 128x 128. Following Samadzadeh et al. (2020), divide the data stream into 10 blocks by time and accumulate the spikes in each block. Then, we split the dataset into 9k training images and 1k test images and reduce the spatial resolution to 48x48. Random horizontal flip and random roll within 5 pixels taken as augmentation (li et al., 2022). We adopt VGGSNN architecture with 300 epochs training on this classification task. And we use Adam optimizer with the learning rate 1e 3 and cosine decay to 0, As for the case that does not apply any augmentation, we add a weight decay of 5e-4 to the optimizer.
A.2 LSDT LOSS LANDSCAPE OF RESNET-19
Here we compare the classification loss (CsDr) landscapes of ResNet-19 on CIFARIO0. The po- sition around the local minimal value found by the SDT (CSDt) is very sharp. However, the area around the local minimum found by TET (ChET) is much smoother (Figure 5), which indicates that TET effectively improves the network generalization. Such improvements could be further utilized to other techniques like privacy-pre sewaac-preserving data generalization (Kim et al., 2021) and neural archi- tecture search (Kim et al., 2022).
1.00
1.0
0.5
3.5
0.04
0.50
3.5
420
Standard Training
Tauporal Efficient Trinning
SDT Loss
Figure STD loss landscape of ResNet-19 on CIFARI00 from different training approaches.
A.3 EFFECT OF LMSE
In this part, we examine the effect of the regular term LMSE with 5 different levels of X. Figure 6 Summarizes the final results. The regular term LMsE effectively increases the performance of
13

---

Published as a conference paper at ICLR 2022
both ResNet-19 CIFARIOO and VGGSNN DVS-CIFARI The static dataset CIFARIO0 is more suitable for larger A, while smaller À is suitable for NVS-CIFARIO. Theoretically, it hard to obtain satisfying performance the early simulation moment due to the sparseness of neuromorphic datasets. So too large regular term LAS not suitable for the neuromorphic dataset. Furthermore, find that high À may harm the early training phase ImageNet, especially if zero-initialize (Goyal et al., 2017) is not performed. As a result, we set À to 5e for CIFARIO and CIFAR 100, le for ImageNet and DVS-CIFARIO.
Pe CIPARI00
VOGSSK DYSCIFARD
56-
Figure The accuracy under different levels of A.
A.4 STATISTICAL RESULTS
Here we provide statistical results (Figure 7) to prove that the total SNN accuracy is positively associated with every average of moment's output test accuracy. We train CNN-5 on CIFAR1O for a total of 20 runs with SDT and 5 runs with TET.
90.4
z- 0.809
g- 4.302e
90.2
SMN
89.6
89.4
86.5
87
87.5
25.5
59
89.5
Avrerage accuracy of single time stops
Figure 7: Statistical results. The overall performance of SNN highly positively associated with the average accuracy of each moment. The standard training obtains the green dots, while the red dots are trained by the TET method.
A.S TIME SCALABILITY ROBUSTNESS OF SDT AND TET.
Here we first show the test accuracy (ResNet19 on CIFAR100 100) of the membrane potential increment each moment instead of the integrated membrane potential. We set the initial simulation length of the SNNs to 3 or 4 and trained them for full 300 epochs. Then expand their simulation length to 8. As shown in table 4, TET (CtEt) makes the membrane potential increment at each moment have a higher classification ability than SDT (LsDr). And TET (1.41 and 0.08) also acquires a low accuracy variance than SDT (3.81 and 4.04).
Then compare the time scalability robustness between SDT (Lsvr) and TET (CIET). We set the initial simulation length of 9 SNNs to 2, 3, 4 and train with SDT or TET. Then we gradually increase SNN simulation length to 64 and record test accuracy of the integrated membrane potential.
14

---

Published as a conference paper at ICLR 2022
1.08
74.0
1.06
Accuracy relutive growih rate
1.02
Accuracy
70.0
1.00
0.98
68.0
TETwith 2-2
TET with I [-3
0.96
TET with T T-3
THT with =
66.0
64.0
SDT with =
SDT with 3-3
TET with 4-4
SDT with --4
0.94
0.92
SIYT
SDT with T-3
SDT with T-2
TET with --4
0.90
62.0
0.88
10
20
30
40
50
10
20
30
Simulation Length
Simulation Length
Figure 8: The accuracy after increasing the simulation length. We first train the SNN with TET (only use LTTT) and SDT (LSDT) with simulation length (T) is 2, 3, or 4. Then, we increase the simulation 64 without finetuning and record the test the classification accuracy (A) and the accuracy relative growth rate (B) of the total SNN output (integrate membrane potential) at each simulation time.
As increase the simulation length, all the SNNs' accuracy will first increase and then be stable in certain area. Meanwhile, TET (1.80) has small accuracy variance than the SDT (11.13) after increasing the simulation length. This phenomenon indicates that the initialization steps of TIT only need a small simulation length SNN for TET but sufficiently large simulation (or enough epochs for finetuning step) for SDT.
Table Accuracy each moment's membrane potential increment. We use LSDT LIET train the networks with simulation length 3 or 4. Then directly increase their simulation length to 8 and record each moment's ' potential increment test accuracy.
Method
T=1
T=2
T=3
T=4
T=5
T=6
T=7
T=8
SDT (T=3)
55.61
57.95
56.87
55.09
57.56
53.54
57.72
54.04
SDT (T=4)
37.96
61.78
55.03
56.64
57.47
54.24
58.74
55.48
TET (T=3)
65.97
72.22
71.78
70.55
71.90
69.57
72.15
69.78
TET (T=4)
62.17
71.57
71.05
72.08
71.77
71.23
71.81
71.36
15


Reviews:
[{"review_id": "uqIFOIZhfzy", "reviewer": "Reviewer_A3wt", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "$\\textbf{Strengths:}$ \nThe high classification accuracy on DVS-CIFAR10 is very impressive.\n\n$\\textbf{Weaknesses:}$\nIn spike of the excellent classification records achieved in this work, the authors’ work leaves a number of uncertainties that should be thoroughly addressed to not mislead the readers. My main concerns are as follows.\n\n1.\tI wonder if the use of TET is the main cause of the improvement in classification accuracy. The authors actually used $L_{TOTAL}=(1-\\lambda)L_{TET}+\\lambda L_{MSE}$ rather than $L_{TET}$. If the use of TET is the main cause, the comparable results should be achieved using solely $L_{TET}$. Given the use of $L_{TOTAL}$, the feature of TET such that loss evaluation at every timestep is undermined because $L_{MSE}$ is evaluated at the last timestep. Additionally, the loss landscape analysis is also undermined because the actually used loss is $L_{TOTAL}$ not $L_{TET}$. Therefore, the authors should clarify the effect of TET only if TET is the key to the present work.\n\n2.\tI wonder if the time inheritance training is the unique feature of TET. The authors should discuss if the same method can successfully apply to SDT as well. This method may be the main cause of the accuracy improvement. The authors should provide the general framework of the time inheritance training method to identify its scalability to different networks on different tasks rather than naively discussing about a single hypothetical case. \n\n3.\tThe authors repeatedly remarked the mismatch of gradient and loss for SDT. I wonder on what ground the mismatch is assured and how the use of TET can resolve the mismatch. Also, the point that “TET needs the second term $\\partial O/\\partial W$ close to 0 to make the $L_{TET}$ convergence.” lacks its theoretical grounds, which should be proven. If this is true, the accuracy should be independent of timestep length T for training; yet, Figure 4 identifies that this is not the case. Even if it is true, this point cannot directly explain the accuracy improvement because learning was done with $L_{TOTAL}$\n\n4.\tThere are many statements that may mislead the readers.\n“Another training issue in the memory and time consumption …” on Page 1. Please specify when this is the truth.\n“Furthermore, since the TET applies optimization on each time point…” on Page 2. Is this true for the use of $L_{TOTAL}$?\n“TET algorithm optimizes each moment’s output, enabling us to extent…” on Page 6. I am very puzzled about this statement. What did the authors mean by “naturally extend the simulation time”?\n\n5. The improvement in classification accuracy on DVS-CIFAR10 is impressive. However, for the moment, it is unclear the direct cause of the improvement for the reason that I remarked above.\n\nMinor concerns:\nThere are many statements that may mislead the readers.\n“Another training issue in the memory and time consumption …” on Page 1. Please specify when this is the truth.\n“Furthermore, since the TET applies optimization on each time point…” on Page 2. Is this true for the use of $L_{TOTAL}$?\n“TET algorithm optimizes each moment’s output, enabling us to extent…” on Page 6. I am very puzzled about this statement. What did the authors mean by “naturally extend the simulation time”?", "summary_of_the_paper": "The authors propose the temporal efficient training (TET) method that features loss evaluation at every timestep. As such, the main difference between TET and SDT lies in the fact that TET evaluates loss at every timestep with the correct label vector whereas SDT evaluates loss at the last timestep. But unfortunately, this could not be done because of the use of $L_{TOTAL}=(1-\\lambda)L_{TET}+\\lambda L_{MSE}$, where $L_{MSE}$ is evaluated at the last moment. The proposed method is simple but its efficacy outperforms the SOTA results for various networks on various datasets.\n", "summary_of_the_review": "Based on the main concerns listed above, I recommend for reject for the current form of manuscript. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "1: The main claims of the paper are incorrect or not at all supported by theory or empirical results.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "sum0m3zSeNZ", "reviewer": "Reviewer_HqdL", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "While the proposed improvements, TET and TIT, may appear only as minor tweaks as compared to previous work, the demonstrated improvements in speed and accuracy suggest that the paper may have substantial impact for SNN research community. ", "summary_of_the_paper": "The paper focusses on an important problem of improving supervised training of spiking neural nets (SNNs), as the current state of SNN training is lacking as compared to that for standard ANNs. The authors propose a couple simple but effective methods for improving both training efficiency and classification accuracy:  temporal efficient training (TET) which simply accumulates error gradients for each time step, and time inheritance training (TIT), which is a repeated training schedule with increasing simulation times.  In extensive experiments the authors demonstrate that their approach improves training time and accuracy. ", "summary_of_the_review": "Efficient supervised learning of deep SNNs is a problem that has resisted research efforts for many decades, so even incremental improvements are welcome in this area. The results of the paper offer a much desired step towards better SNN backprop algorithms that promise to unleash the many advantages of SNNs.  \nThe paper is lucidly written and provided experiments strongly support the claims. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "hq-BQSzvT92", "reviewer": "Reviewer_KVXN", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "There are some main issues to be addressed:\n1. The TET method and the TIT method can be viewed as two individual parts. The TET method is a new temporal loss function and the TIT is a fine-tune method. If so, an ablation study is required to show the contribution of these two parts in temporal efficiency individually. Furthermore, can the TIT method be applied to other SNNs as long as these SNNs are trained with temporal loss (e.g., the STCA loss [1])? \n2. In the section 4.3, this paper claims the TIT method could significantly reduce the training time. Here are some questions.\n(1) the time spent per epoch is not linearly proportional to the simulation length, so it is inappropriate to claim ‘As a result, the TIT can reduce the training time cost by half.’\n(2) based on the above analysis, actually, it seems difficult to select the training epoch and simulation length for the TIT method to achieve temporal efficiency for training SNNs.\n3. In table 3, the network performance across static datasets and neuromorphic datasets, why the simulation length of two networks with the TET on the Imagenet are different? Is it from original paper or based on the optimization?\n4. For time scalability, it’s better to supplement the result of relative growth rate for both SDT and TET. \n5. Some minor comments, such as the unclear figure legends, the missing curve of “Training from scratch” in Fig 4A, inconsistent decimal places of accuracies and so on.\n\n[1] Gu, et al. STCA: Spatio-Temporal Credit Assignment with Delayed Feedback in Deep Spiking Neural Networks. IJCAI 2019. \n\n", "summary_of_the_paper": "This paper proposes a new training approach, the temporal efficient training (TET). This algorithm utilizes a new loss function to improve the generalizability of SNNs. Further, a new training pipeline is presented to reduce the simulation time of SNNs. This work outperforms the SOTA on the static datasets and neuromorphic datasets.", "summary_of_the_review": "The novelty is limited, as the core idea and the relative derivations are straightforward.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "GB_xgaD9KDI", "reviewer": "Reviewer_5F8z", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength: The results are good compared to existing works. The paper also has a comprehensive comparison between the proposed TET and regular SDT.\nWeakness: The proposed method only changes the loss function. The novelty may not be enough. It is still a kind of surrogate gradient method and cannot solve the approximation made by smoothing the gradient of the discontinuous spiking function.", "summary_of_the_paper": "This paper proposed a new loss function called TET for directly training SNNs.", "summary_of_the_review": "1. Missing reference: There’re some recent papers that should be discussed in the related works, e.g. [1]\n2. I'd like to learn more about the details of data preprocessing. For example, does the real value in CIFAR and ImageNET datasets directly used as the inputs? In DVS-CIFAR10, how does the resolution reduced? How is the time length shrunk to 10 time steps? \n3. Performance comparison: In CIFAR10, can the authors also provide the performance of the 5 layers CIFARNet for better comparison with existing works? In DVS-CIFAR10, do other works also use data augmentation, e.g.[2]? Does the network adopted in this paper have a similar number of parameters to the existing works in the comparison?\n\n\n\n[1] Yang, Y., Zhang, W., & Li, P. (2021, July). Backpropagated Neighborhood Aggregation for Accurate Training of Spiking Neural Networks. In International Conference on Machine Learning (pp. 11852-11862). PMLR.\n[2] Zhenzhi Wu, Hehui Zhang, Yihan Lin, Guoqi Li, Meng Wang, and Ye Tang. Liaf-net: Leaky integrate and analog fire network for lightweight and efficient spatiotemporal information processing. IEEE Transactions on Neural Networks and Learning Systems, 2021.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes a new loss function for the training of spiking neural networks leading to significant improvements in generalization performance across a variety of datasets and network architectures. While conceptually simple, the approach leads to substantial performance gains, and some intuition is provided to explain its success.

The reviewers are split on the issue of significance of the paper, in part due to the simplicity of the proposed loss function. Still, good results speak for themselves, and the effectiveness of the technique has been demonstrated thoroughly.
Decision: Accept (Poster)