Temporal Effective Batch Normalization in Spiking Neural Networks
Authors: Chaoteng Duan
Jianhao Ding
Shiyan Chen
Zhaofei Yu
Tiejun Huang
Venue: NeurIPS 2022 Accept (2022)
Keywords: Spiking Neural Networks
Batch Normalization
Spatio-Temporal Representation

Abstract: Spiking Neural Networks (SNNs) are promising in neuromorphic hardware owing to utilizing spatio-temporal information and sparse event-driven signal processing. However, it is challenging to train SNNs due to the non-differentiable nature of the binary firing function. The surrogate gradients alleviate the training problem and make SNNs obtain comparable performance as Artificial Neural Networks (ANNs) with the same structure. Unfortunately, batch normalization, contributing to the success of ANNs, does not play a prominent role in SNNs because of the additional temporal dimension. To this end, we propose an effective normalization method called temporal effective batch normalization (TEBN). By rescaling the presynaptic inputs with different weights at every time-step, temporal distributions become smoother and uniform. Theoretical analysis shows that TEBN can be viewed as a smoother of SNN's optimization landscape and could help stabilize the gradient norm. Experimental results on both static and neuromorphic datasets show that SNNs with TEBN outperform the state-of-the-art accuracy with fewer time-steps, and achieve better robustness to hyper-parameters than other normalizations.

Temporal Effective Batch Normalization in Spiking Neural Networks
Chaoteng Duan"
Jianhao Ding
School of Electronic and Computer Engineering
School of Computer Science
Peking University
Peking University Beijing, China 100871 djh01998@stu, pku. edu. cn
Beijing, China 100871
duanchaoteng@stu. pwtpsesttu.uuuu... www.euu.kuuuuuuuuu....u edu.
Shiyan Chen
Zhaofei Yu Institute for Artificial Intelligence School Computer Science
School of Electronic and Computer Engineering
Peking University
Beijing, China 100871
Peking University Beijing, China 100871 yuzf 12@pku. edu
strerrichia002p@stu. pku. edu. cn
Tiejun Huang
School of Computer Science
Peking University Beijing, China 100871 tjjhuang@pku.co edu. en
Abstract
Spiking Neural Networks (SNNs) are promising neuromorphic hardware owing to utilizing apatio-tempont information and sparse event wenntdriven signal processing. However, challenging to train SNNs due the non-differentialle nature of the binary firing function. The surrogate gradients alleviate the training problem and make SNNs obtain comparable performance as Artificial Neural Networks (ANNs) with the same structure. Unfortunately, batch normalization, contributing to the success ANNs,, does not play a prominent role in SNNs because the additional temporal dimension. To this end, we propose effective normalization method called temporal effective batch normalization (TEBN). By rescaling the presynaptic inputs with different weights every time-step, temporal distributions become smoother and uniform. Theoretical analysis shows that TEBN can viewed as a smoother of SNN's optimization landscape and could help stabilize the gradient norm. Experimental results on both static and neuromorphic datasets show that SNNs with TEBN outperform the state-of-the-an accuracy with fewer time-steps, and achieve better robustness to hyper-parameters than other normalizations.
Introduction
As the third generation of neural network models, Spiking Neural Networks (SNNs) are closer the neuronal circuits of the brain and have recently received substantial attention in artificial intelligence and neuroscience 32 40 SNNs process spatio-temporal information with asynchronous discrete
"These authors contributed equally to this work.
CCoresponding author
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

events 22 52 53 54], which is altra-low-power and highly compatible with neuromorphic and FPGA devices. However, due to the non non-differentiable nature of the membrane potential at the firing time, it remains challenging to train nigh-performance deep SNNs (2 44
There are two main methodologies address the training problems. One is the conversion- -based method, which converts pre-trained Artificial Neural Networks (ANNs) to SNNs by converting ReLU activation spiking neurons and retaining the weights (5] 36 43). The converted SNNs usually require a large number of time-steps to achieve comparable performance as source ANNS 4 [8]. and can not be applied to neuromorphic datasets [9]. The second is end-to-end training with back- propagation, which is promising as it only needs a few time-steps [49]. In the end-to-end training, the forward path achieved with -in-eefferent spiking function, and the network back- propaggass through surrogate gradient functions [35] or temporal dependencies [35]
Batch normalization (BN) contributes to the success of ANNS by flattening the loss landscape and reducing the internal covariate shift (ICS) [24]42]. Whereas when it comes to SNNs, the existing normalization methods rarely take full advantage of the additional temporal dimension, which restricts the performance of SNNs [50]. Furthermore, the temporal dynamics of spiking neuron is tightly in connection with the neuronal parameters. Therefore, when these parameters are changed, the performance of some methods may be degraded [3]
In this paper, propose Temporal Effective Batch Normalization (TEBN), a novel and effective normalization approach that regularizes the temporal distribution through weighting the presynaptic inputs of the spiking neurons. We prove that our approach could smooth the optimization landscape of SNN and help stabilize the gradient norm. The experiment results on CIFAE-10, CIFAR- and DVS-CIFAR 10 datasets show that our approach achieves better classification performance and than other BN methods. Besides, the proposed method outperforms
robustness to the state-of-the-ar learning methods, using fewer time-steps.
2 Related Works
2.1
Learning of Spiking Neural Networks
In recent years, many learning methods have been developed to train deep SNNs with excellent
performance, which can be classified into ANN to SNN conversion (ANN2SNN)
4.
5
9
17
18
20
22
33
43 [5) and end-to-end backpropagation methods 6 7 I2 16 21 29
29
31
35
57 ANN2SNN pre-trains a source ANN and then converts it to an SNN by changing
the artificial neuron model the spiking neuron 5 The main idea is to use the firing rates [18]
or average postsynaptic potentials [9] of spiking neurons to approximate the ReLUU activations of artificial neurons. Although some advanced conversion methods have achieved almost loss-less
accuracy on large-scale datasets with ResNet and VGG-16 structures 4
9
1 43), they overlook
the rich temporal dynamic characteristies of SNNs and demand many time steps to approach the accuracy of pre-trained ANNs. In contrast, the backpropagation method directly trains SNNs by unfolding the network over the time-steps and then computing the gradient on both spatial and temporal domains, which borrows the idea of training Recurrent Neural Networks (RNNs) with backpropagation through time 49). As the gradient of spikes with respect to membrane potential is non-differentiable, the surrogate gradient is proposed by approximating the gradient with smooth functions 35 55) The backpropagation method with surrogate gradients can be applied both static and neuromorphic datasets, and requires far fewer time-stepe steps than ANN2SNN methods. Another kind of backpropagation method is timing-based 12 8 26, 56), which directly computes the gradient of the timing of firing spikes to the membrane potential and does not require unfolding the network over the stme-steps However, they are usually limited to shallow networks that are less than 15 layers 3
2.2 Normalization in Spiking Neural Networks
The training of well-behaved ANNs made possible by using normalization approaches, including batch normalization (24], group normalization [51], and layer normalization [I]. Therefore, the researchers try to incorporate normalization approaches in ANNs to enhance the learning of SNNs. For example, NeuNorm [50) constructed auxiliary feature maps receiving the lateral inputs from the same layer to control the strength of the stimulus, which normalized the data along the channel dimension. Recently, some methods have modified batch normalization from temporal perspective

---

to fit the training of SNNs. Zheng et al. [57] proposed a threshold dependent batch normalization (tdBN) method, which extended the scope of BN to the additional temporal dimension. tdBN is stated to be able to alleviate the vanishing or explosion of gradients. Just as vanilla BN, the utilization of shared parameters may neglect the negative impact brought by the unusual temporal distributions. PSP-BN (23 used unique statistics, the second raw moment, as the denominator in normalization
and it can be inserted right after the spiking functions. BNTT [27] regulated the firing rates by utilizing separate sets of BN parameters diverse scenarios 47.. The complexity of BN parameters is higher compared to other BN methods and could break the temporal coherence of information. Inspired by these methods, aim develop a modified BN method with low complexity of parameters and take advantage of the temporal distribution of presynaptic inputs in the meantime.
different time-steps, and can be used for learning in
3
Preliminary
3.1 Spiking Neuron Model
Unlike traditional ANNs, SNNs use binary spike trains to transmit information. Here we consider the widely used Seaky-Integrate anr-Fire (LIF) [15] model emulate the dynamics of spiking neurons, which is formulated as a first-order differential equation.
(t) = (t) + RIR (t),
Fm
(1)
dt
where (t( denotes the membrane potential of the i-th neuron at time f, Tm represents the membrane time constant, R and (t) denote the linear resistor and the input current the i-th neuron at time f. For numerical simulations of LIF neurons, we need to consider a discrete version of the parameters dynamics. Similar to (9), the membrane potential u, [t] of the i-th neuron at time-step t is represented as:
u. [t] ru, [t
1]+
Σ jEpre(s)
Wijo [t].
(2)
Here j denotes the index of pre-synaptic neuron and 0y[t] denotes the binary spike of neuron at time-step I, which equals if there spike and otherwise. The membrane potential u, [4]
decreases with leak factor that is no more than 1, and increases with the summation of inputs spikes from the pre-synaptic neurons pre(i) through synaptic weight w.j- When the membrane potential [t] exceeds a specific threshold 0, the neuron will fire a spike and then the membrane
potential is reset to O. By combining the sub-threshold dynamics (Eq. ( and hard reset mechanism, the whole iterative LIF model can be determined by:
u: [t] = ru; [t - 1] - o, - (1)- + Σ Wijo [t]
(3)
jEDre(s)
o [t] H uu |t| 0)
(4)
where Hl denotes the Hcaviside step function.
3.2 Batch Normalization
Batch Normalization (BN) has been a typical strategy for training deep neural networks. By fixing the distribution of inputs, BN can reduce the Internal Covariate Shift (ICS) during training, allowing stable convergence of deeper neural networks [24]. Considering an analog neuron with input
where m is a mini-batch size. BN first normalizes each activation with: = (1) x (2) ,
(b))
(b)
b= 1,2, M,
where the expectation and the variance 2² are computed by 1 Σm (b) and a = i represent the identity transformation, the learnable parameters 7 and are introduced to scale and shift the normalized vector, and the output of BN is:
1 Em (z)(0) 11) respectively. To enhance the representation capability and ensure BN can
BN (b)) = 7 (b) + 3.
(6)
3

---

(-0
10 1
HAN
11
BN
BNTT
IdB
TEBN
Figure 1: Illustrations of different BN methods for SNNs.
4 Methodology
4.1 Covariate Shift in the Temporal Dimension
ICS has been widely regarded as the key to the effectiveness of BN. In SNNs, synaptic currents fed into spiking neurons sequentially, where BN is still the key to enforcing effective SNN training. Intuitively, the spikes triggering asynchronous currents may strengthen the internal covariate shift in the temporal domain. We refer to this phenomenon as Temporal Covariate Shift (TCS), which can be viewed special case of ICS. Default BN actually updates its statistics at each time-step, approximating the statistics of total time-step inputs. Here we build the relationship between the expectation and variance across all time-steps (Heotat o iotat) and those of the single time-step (u(t). (t 1.2, T):
Ptotll
T i=1
Σ
ulii,
Σ
T
2
Σ
(8)
T
T
==1
By supposing that n4({] is approximately identical for each time-step I,
T
[i].
total
T
Hence, the Gaussian distribution across all time-steps can be seen as the composition of independent distributions. Fig. illustrates three typical BN methods used in SNN, including default BN (24 BNTT 27 and tdBN [57]. The statisties and parameters these methods use are listed in Tab. [
Table 1: Statistics and parameters of presynatic inputs with different BN methods
1=2
t=T
BNTT 14
11.0
12
2
1
11
2
7
Mataai o iotal
IdBN
Y, B
Mooaals goata
TEBN
411 11
2
2
4(7),87)
As shown in Tab. 1 BNTT exploits separate sets of mean, variance, and the scale and shift parameters to model the distribution of presynaptic input at each time-step. It establishes a delicate model of distributions while ignoring the time dependence of the input spikes. In addition, BNTT takes

---

large amounts of computation in implementation as it needs the same number of BN modules as the time-step. Different from tdBN calculates the overall expectation and variance for all T time-steps. and the scale and shift parameters are also shared for all time-steps. The difference between tdBN and default BN lies in the implementation. As illustrated in Fig- [ the input statistics of tdBN consider the statistics of all the batches and time-steps, while default BN uses the momentum update to approximate the total statistics. The merit of using Mrotats iotal in tdBN is that when doing backpropagation, there is only one set of input statistics that may contribute to the alignment of temporal distributions. Actually, IdBN considers the time dependence of the input spikes, but ignores the varying temporal characteristics at different time-steps.
4.2 Temporal Effective Batch Normalization
To model the temporal distributions in SNN training without using too many parameters, combining the findings of Eq. / we treat the distribution of presynaptic currents at each time step as a scaling of the overall distribution. To this end, we propose Temporal Effective Batch Normalization (TEBN), which assigns different weights to each time-step to distinguish their effect on the final result. The weights are learnable find an optimum for every time-step implement efficient inference. Given that - [4] is the input to layer at time-step 1, the spiking neuron with TEBN can be described as:
x² [ W 1 [(),
(10)
u' Tu' [t 1](1 - o' - 1l) + a² (t].
(11)
k(t = TEBN (z [t = 7t- x|t] 4 + B[t).
where
(12)
Va² +c
{0 7 x p|d], [ = B x pit].
(13)
Here u' and o' [t] denote the membrane potential and binary output spikes of all neurons in layer ( at time-step f. W is the synaptic weights between layer - 1 and layer 1, and #'(t] represents the output of TEBN in layer at time-step I. We calculate the mean l and variance o² from samples of all time-steps. is small constant, which is added guarantee numerical stability. Each TEBN layer processes time-invariant BN parameters and ,3 and learnable weight parameters p|(]. Since the parameters pt 0, 1, 1..., I) are self-adapting to adjust the distribution of æ jt). TEBN will help relieve the problem of TCS.
4.3 Detailed Learning Rule in SNNs
We apply TEBN to SNNs and derive the detailed learning rule. Actually, the problem is to compute the gradients of the loss function I' with respect to the weights aw! iy and with respect to the parameter pit]. where ow; 1y denotes the weight between the j-th neuron in layer / - 1 and the i-th neuron in layer l. Similar the previous works [27 57 we compute the gradient by unfolding the network over the simulation time-steps, and get:
O.
dL do
OL
Du! [t +
(14)
Out [t]
0o [ du
tt]
Du
[t
[++] 1] Ou, [+]
a.
df
tL
0g 1-1 [4]
of
[
ou [1] Ox 1-1 [4] oj
1-1
[t]
(15)
ow 0a!-1 14
1j
where u([t] and o[ denote the membrane potential and output spikes of the i-th neurons in layer I at
time-step t, respectively, [][ and [ represent the i-th input and output of TEBN in layer / at time-
step f. As the derivative of spike with respect to the membrane potential do is non-differentiable,
au[[[]
the surrogate gradient method [35 49 is used to smooth the Heaviside step function.
The weight parameter p[t] allows the network to capture temporal dynamics when training synaptic connections, and the gradient of the loss function L to the weight parameter p in layer I is:
dL
i du t
aL Ou
Σ
OL
7
x
1 + B3 +
(16)
3p t
due
[t]
V
where μd and (o denote the mean and variance of the i-th neuron from samples of all time-steps.
7/ and Bi are the i-th parameters of TEBN.
5

---

4.4 Related to Learnable Input Resistance of LIF Neuron
Like Long Short-Tere Memory (LSTM) [9], the LIF neuron has a similar function in controlling the memory. As illustrated in Eq. [ the LIF neuron is able to remember current input information and forget some information from the past. which is under the regulation of the relative scale of membrane time constant and linear resistor R. A larger Tm or a smaller R typically indicates more reinforced memory. The resistor usually regarded as a latent variable absorbed in the weights (c.f. Eq. 2 Fixing the time constant has its problem. First, when given the time-step number for inference, the LIF neuron cannot maintain a memory to enhance the internal representation. Second, the accuracy of SNN is subjected to the chosen [13]. In this regard, our TEBN uses learnable hyperparameter p|t] to rescale the presynaptic inputs at time-step t. which can be regarded as optimizing learnable input resistance R. The learnable p-{] can share the benefit of the learnable T proposed in [S] to improve the representation. Different from [3]. the time- varying characteristic further enables the network to find an optimal proportion of remembrance to input each time-step, which endows the network with better fitting ability and robustness to hyperparameters.
5 Smoothing Optimization: A Theoretical Analysis
The proposed normalization methods suggest diverse disposal for different time-steps of neurons. The mini- batch training makes each neuron i have a dimension of analysis that the special design of TEBN can not only flatten the loss landscape with regard to the input signal, but also help absorb the decay of gradient norm of mini-batches in adjacent time-steps.
We will show in our
Theorem 1. For the time-dependent imput signal ac; [t] with t as the current time-step and as the neuron index, the relation between the gradient of V.L w.r.f (t in TEBN network V v= (a(()) and the corresponding gradient V vz (ac L is as follows:
(1]
in non-TEBN counterpart network which has a loss
9:14
V a. [c] eI <
(17)
where [1] = TEBN (a: (t),, 5t [0] the i-th element of 9(1] in TEBN. Note that the L2 norm is calculated in the dimension of mini-batches
The Lipschitzness of the loss with TEBN blocks is shown to exploit a better Lipschitz constant in Theorem I which contributes to the restraint of the amplification of gradient update steps. Theorem I implies that the TEBN block performs as traditional BN in ANNs. TEBN actually smooths the gradient of the temporal-varying currents. With favored optimization landscape, SNNs can also have the opportunity to flatten the optimization. Moreover, as the existence of the time-dependent % [4] the bound of the Lipschitz constant is non-static. By setting p[{]s equal 1, TEBN degenerates to traditional BN. This may decrease the capability of controlling the temporal Lipschitz constant in practice.
As illustrated in Sec. 4.44 the TEBN block scaling presynaptic inputs can be seen as learnable resistance in SNN. We further check the Lipschitzness of the gradient between time-steps t 1 and t. Our analysis can help understand why extra scaling on the time-step is necessary.
Theorem 2. For the time-dependent input signal a. [1] with as the current time-step and as the neuron index, denote - [t] as the hidden output of TEBN without scaling presynaptic inputs by p' ('( Then, the relation between the gradient of AL w.E.. -/--! [t] and z. !-1 [t 1] is as follows:
IV V VE
p' 1] P'[
TVT 08 max
|| x
L'l + + 1a II [/t 1f-f (18)
where li max is the maximum that the surrogate gradient function can take, and ks the mini-batch size.
Theorem depicts the magnification of the gradient norm with regard to time. When setting all p|t]s equal one, TEBN can provide the effects of traditional BN. The Lipschitz constant of gradients for adjacent time is constant T (1 + 0fimaxx, which may cause the norm to be unstable while training. The use of trainable p[t), in this regard, can help alleviate this situation and stabilize the gradient norm, which is essential for training procedure.

---

To summarize, directly applying the traditional resolution of Batch Norm into SNN can benefit the training process: smoothing the optimization landscape. We conclude from Theorem I that our proposed normalization method has at least the same ability to preserve gradient norm. Besides, we also discover that the Lipschitz constant of BN depends on some neuron parameters like the leak factor т and the threshold / in Theorem 2 dimension will mitigate the gradient norm from explosion and vanishing.
We believe a trainable scaling factor in the temporal
Experiments
In this section, we validate the effectiveness of our proposed TEBN for classification tasks on both static and neuromorphic datasets. We first compare algorithm with BN methods demonstrate the advantages of TEBN. Further, we compare our method with other state-of-the approaches. Finally, we visualize the distribution of TEBN outputs and demonstrate that the proposed method is robust to hyper-pane paremeeaaeetrss..... We follow the same data pre-processing protocol as previous work (50) More details of the configurations are provided in the supplementary.
6.1 Comparison with other BN Methods
We first evaluate the performance of the proposed TEBN and other normalization methods. For fair comparison, do not advanced data augmentation like cutout [10]. Tab. 2 reports the test accuracy on both traditional static CIFAR-10, CIFAR- 100 (28) datasets and neuromorphic DVS-CIFARI0 (00 dataset. On the CIFAR-10 dataset, our TEBN achieves 94.57% top-1 accuracy with ResNet-19 19 network using only 2 time-steps. When the network structure the same, TEBN outperforms the other BN methods, even with fewer time-steps T. We get the conclusion other datasets. For example, on CIFAR- 100, our method achieves better performance (74.37% v.s. 66.6%) and fewer time-steps v.s. 50) than BNTT (27). On INS-CIFARIO, the accuracy of our method is 14.6% (75.1% % v.S. 60.5%) and 11.9% (75.1% v.s. 63.2%) higher than NeuNorm and BNTT, which use 4 times and times many simulation time-steps T, respectively. Moreover, we achieve better performance (64.29% v.s. 63.72%) and fewer time-steps (4 v.s. 6) than tdBN [57] on ImageNet. All the accuracies of other methods reported in the tables result from the literature.
Table 2: Comparisons with different normalization methods.
Dataset
Model
Methods
Architecture
Time-steps
Accuracy( (%)
SPIKE-NORN 43
ANN2SNN
VGG-16 16
2500
91.55
NeuNorm 50
Surrogate Gradient
CIFARNet
12
90.53
BNTT 27
Surrogate Gradient
VGG-9
90.30
6
93.16
ddBN 57
Surrogate Gradient
Re
4
92.92
CIFAR-10
2
92.34
VGG-9
4
92.81
94.71
TEBN
Surrogate Gradient
ResNet-19
94.70
2
94.57
SPIKE-NORM 43
ANN2SNN
VGG-16
2500
70.90
BNTT 2
Surrogate Gradient
VGG-11
50
66.60
VGG-11
74.37
CIFARI00
76.41
TEBN
Surrogate Gradient
ResNet-19
76.
2
75.86
NeuNorm 50
Surrogate Gradient
7-layer CNN
40
60.50
BNTT 27
Surrogate Gradient
7-layer CNN
63.20
DVS- -VS-CIFARIO
IdBN
Surrogate Gradient
ResNet-19
67.80
TEBN
Surrogate Gradient
7-layer CNN
75.10
SPIKE-NORM 43
ANN2SNN
ResNet-34
2500
69.96
ImageNet
dBN (57
Surrogate Gradient
ResNet-34 34
6
63.72
TEBN
Surrogate Gradient
ResNet-34
64.29
7

---

default
Figure 2: Distribution of presynaptic inputs in layer 2 with different BN methods on CIFAR-10.
TEBN Layer 5
TEBN Layer 6
TEBN 7
TEBN Layer B
Figure 3: Distribution of presynaptic inputs with TEBN on DVS-CIFARI0
6.2 Comparison with the State-of-the-Art
We compare our method with other state-of-the-an learning methods for SNNs on CIFAR-10, CIFAR- 100, and DVS-CIFARE 10 datasets and report the results in Tab. 3 With our reproduction of the CNN model, our TEBN outperforms HC 93.96% v.s. 92.94% VGG-11) and TSSL-BBP (92.65% % v.s. 91.41% %. 7-layer CNN) with fewer time- We contribute this to the better representational capacity of TEBN. On CIFAR-100, 100, our TEBN achieves an accuracy of 75.86% with time-steps on ResNet-19, which is better than results of ANN2SNN and hybrid training. On DVS-CIFARIO method performs better than PLIF and TET using the same architecture. Besides, we also adopt the cutout augmentation on static datasets denoted by "*" in the table. Details of the data augmentation are presented in the supplementary. Cutout helps TEBN to realize better performance. Compared with other surrogate gradient methods such as TET and Dspike, better. For instance, We achieve the top-1 accuracy of 78. 76% on CIFAR- 100 dataset with ResNet- (T=6), which is 4.04% higher than that of TET with the same network structure and time-steps. Moreover, our method outperforms the state-of-the-ar learning methods on ImageNet dataset. All the accuracies of other methods reported in the tables result from the literature.
TEBN model performs consistently
6.3 Distribution of TEBN Outputs
We train 9-layer CNN models to visualize the temporal distribution of normalization layers with default BN, BNTT, tdBN, and the proposed TEBN. Fig. 2 illluttraees the results of inputs in layer 2 on CIFAR- 10 with 3 time-steps. The default BN shows an irregular shape instead of a standard normal distribution. On the other hand, BNTT. tdBN, and TEBN reshape the presynaptic inputs into bell-shaped distribution. Among the three methods, the temporal distribution of TEBN appears more homogeneous. Unlike the static datasets, neuromorphic datasets contain rich temporal information, which means presynaptic inputs in different time-steps may have varied distributions. Here we train 9-layer CNN model on DVS-CIFARI with TEBN and visualize the temporal distribution of the normalization layers in Fig. 3 The last five time-steps are selected for better visual effects. All the chosen time-steps have almost identical distribution. We can conclude that our TEBN can play role in adjusting the current distribution of DVS data.
6.4 Robustness on Hyper-parameters
Previous work has revealed that improper initial value of membrane time constant will hamper the learning of SNN [13]. Here we compare the performances of different BN methods with various т. We train 9-layer CNN models on CIFAR-10 dataset with 2 time-steps and values chosen from 10.1, 0.25, 0.5, 0.75, 1.0]. As shown in Fig. (a) the models with TEBN maintain stable accuracy when т changes, whereas others fluctuate. The result indicates the robustness of our method to various T.
In fact, the parameters p(t] are self-adapted to find the optimum when 7 changes. We conduct additional experiments with TEBN. The time-step is set to 6, and varies from 0.25 to 1.0 with step
8

---

Table 3: Compare with state-of-the-ar supervised training methods. denotes using cutout.
Dataset
Model
Methods
Architecture
Time-steps
Accuracy($0)
RMP [18
ANN2SNN
ResNet-20
2048
91.36
Opt. 9
ANN2SNN
ResNet-20
128
93.56
PTT. 48
ANN2S
VGG-11
16
91.24
QCFSA
ANN2SNN
ResNet-20
92.35
HC 37
Hybrid Training
VGG-11
2500
92.94
TC 58
Time-based Gradient
VGG-16
92.68
TSSL-BP 56
Time-based Gradient
7-1ayer CNN
5
91.41
6
94.25
Dspike* 31
Surrogate Gradient
ResNet-18
4
93.66
93.13
CIFAR-10
6
94.50
TET 10
Surrogate Gradient
ResNet-19
4
94.44
2
94.16
7-layer CNN VGG-11
4
92.65
93.96
TEBN
Surrogate Gradient
94.71
ResNet-19
4
94.70
2
94.57
6
95,60
TEBN *
Surrogate Gradient
ResNet-19
4
95.58
2
95.45
RMP 18
ANN2SNN
ResNet-20
2048
67.82
Opt. 9
ANN2SNN
ResNet-20
512
72.34
QCFSA 4
ANN2SNN
ResNet-20
128
70.55
HC 37
Hybrid Training
VGG-11
2500
70.94
74.24
Dspike* 31
Surrogate Gradient
ResNet-18
73.35
2
71.68
6
74.72
CIFARION 100
TET 10
Surrogate Gradient
ResNet-19
4
74.47
2
72.87
VGG-TT
4
74.37
76.41
TEBN
Surrogate Gradient
ResNet-19
4
76.13
2
75.86
6
78.76
TEBN
Surrogate Gradient
ResNet-19
4
78.71
78.07
PLIF
Surrogate Gradient
6-layer CNN
20
74,80
Dsipke 3
Surrogate Gradient
ResNet-18 18
10
75,40
DVS-CIFAR10
TET 10
Surrogate Gradient
VGGSNN
10
83.17
l-yayer CNN
10
80.00
TEBN
Surrogate Gradient
VGGSNN
10
84,90
SEW 12
Surrogate Gradient
SEW ResNet-34
4
67.04
ImageNet
TET 10
Surrogate Gradient
SEW ResNet-34
68.00
TEBN
Surrogate Gradient
SEW ResNet-34
4
68.28
0.25. As shown in Fig. 4(b) 4(c)
smaller makes p converge to a bigger value and vice versa. Note that when 1.0, the LIF neuron degenerates the IF neuron, which affects the optimization of p.
7 Conclusion
An effective and robust normalization method TEBN is proposed for SNN in this paper. The main idea is utilizing time-specific learnable weights to rescale the presynaptic inputs, which serves to normalize the temporal information to smooth and conform temporal distribution. Optimizing the learnable parameters p([] can regarded as adjusting the input resistance, which helps to enhance the fitting ability and robustness to specific hyper-parameters of spiking neurons. TEBN promotes the potential advantages backpropagation methods for future practical applications of SNNs. future researches, aim to dive deeper into the temporal distribution of SNNs with BN and investigate

---

(a) Accuracy
Layer
(c) Layer 5
Figure 4: The accuracy of different BN methods and p[t] change with respect to
more robust SNNs. Besides, recent works have shown that short plasticity (STP) 14 39 46
can be incorporated into ANNs to enhance efficiency and computational power [3 38 As STP performs function of temporal filtering similar TEBN, how use the biologically-realistic filter-TPP in SNNs is the future direction.
8 Acknowledgements
This work was supported by the National Natural Science Foundation of China Grants 62176003 and 62088102.
References
[1] Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E. Hinton. Layer normalization. arXiv preprint 1677:06600004450 2016.
[2] Sander M Bohte, Joost N Kok, and Han La Poutre. Brror-backpropagation in temporally encoded networks of spiking neurons. Neurocomputiing 48(1-4):17-37, 2002.
[3] Tong Bu, Jianhao Ding, Zhaofei Yu, and Tiejun Huang. Optimized potential initialization for low-latency spiking neural networks. In In Proceedings of the AAAI Conference on Artificial Intelligence (AAAI), pages 11-20. 2022.
[4] Tong Bu, Wei Fang, Jianhao Ding, et al. Optimal ANN-SNN conversion for high-accuracy and lltraelow-latencyy spiking neural networks. In International Conference on Learning Representations (ICLR), 2022.
[5] Yongqiang Cao, Yang Chen, and Deepak Khosla. Spiking deep convolutional neural networks for energy-efficient object recognition. International Journal of Computer Vision (IJCV), 113(1):54-66 66, 2015.
[6] Yanqi Chen, Zhaofei Yu, Wei Fang, Tiejun Huang, and Yonghong Tian. Pruning of deep spiking neural networks through gradient rewiring. In Proceedings of the Thirtieth International Joint Conference Artificial Intelligence (IJCA), pages 1713-1721, 2021.
[7] Yanqi Chen, Zhaofei Yu, Wei Fang, Zhengyu Ma, Tiejun Huang, and Yonghong Tian. State tran- sition of dendritic spines improves learning of sparse spiking neural networks. In International Conference on Muchine Learning (ICML), pages 3701-3715. PMLR, 2022.
[8] Iulia M Comsa, Krzysztof Potempa, Luca Versari, Thomas Fischbacher, Andrea Gesmundo, and Jyrki Alakuijala. Temporal coding in spiking neural networks with alpha synaptic function. In International Conference Acoustics, Speech and Signal Processing (ICASSP), pages 8529 8533, 2020.
[9] Shikuang Deng and Shi Gu. Optimal conversion of conventional artificial neural networks to spiking neural networks. In International Conference on Learning Representations (ICLR), 2020.
10

---

[10] Shikuang Deng, Yuhang Li, Shanghang Zhang, and Shi Gu. Temporal efficient training of spiking neural network via gradient re-weighting. In International Conference on Learning Representations (ICI.R), 2021.
[11] Jianhao Ding, Zhaofei Yu, Yonghong Tian, and Tiejun Huang. Optimal ANN-SNN conversion for fast and accurate inference in deep spiking neural networks. In International Joint Conference on Artificial Intelligence (IJCAI), pages 2328-22222 2336, 2021.
[12] Wei Fang, Zhaofei Yu, Yanqi Chen, Tiejun Huang, Timothée Masquelier, and Yonghong Tian. Deep residual learning in spiking neural networks. Advances in Neural Information Processing Systems (NeurIPS), 34:2 10221216666999 2021.
[13] Wei Fang, Zhaofei Yu, Yanqi Chen, Timothee Masquelier, Tiejun Huang, and Yonghong Tian. Incorporating learnable membrane time constant enhance learning of spiking neural networks. In Proc. IEEE International Conference on Computer Vision (ICCV), pages 2661-2671, 2021.
[14] Eric S Fortune and Gary Rose. Short-term synaptic plasticity as a temporal filter. Trends in Neurosciences, 24(7):381-335 2001.
[15] Wulfram Gerstner, Werner M Kistler, Richard Naud, and Liam Paninski. Neuronal dynamics: From single neurons to networks and models of cognition. Cambridge University Press, 2014.
[16] Yufei Guo, Xinyi Tong. Yuanpei Chen, Liwen Zhang. Xiaode Liu, Zhe Ma, and Xuhui Huang. RecDis-SNN: Rectifying membrane potential distribution for directly training spiking neural networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 326-335 2022.
[17] Bing Han and Kaushik Roy. Deep spiking neural network: Energy efficiency through time based coding. In European Conference on Computer Vision (ECCV), pages 388 388-404,25 2020.
[18] Bing Han, Gopalakrishnan Srinivasan, and Kaushik Roy. RMP-SNN: Residual membrane potential neuron for enabling deeper high-accuracy and low-latency spiking neural network. In Proc. IEEE Conference on Computer Vision and Patern Recognition (CVPR), pages 13555- 13564, 2020.
[19] Sepp Hochreiter and Jurgen Schmidhuber. Long short-term memory. Neural Computation, 1835:1735 1997.
[20] Yangfan Hu, Huajin Tang, and Gang Pan. Spiking deep residual networks. IEEE Transactions on Neural Networks and Learning Systems (TNNLS), 2021.
[21] Dongsung Huh and Terrence J Sejnowski. Gradient descent for spiking neural networks. Advances neural information processing systems (NeurIPS), pages 1450, 2018.
[22] Eric Hunsberger and Chris Eliasmith. Spiking deep networks with LIF neurons. arXiv preprint arXiv:1510.08829, 2015.
[23] Shin-ichi Ikegawa, Ryuji Saiin, Yoshihide Sawada, and Naotake Natori. Rethinking the role of normalization and residual blocks for spiking neural networks. Sensors, 22(8):2876, 2022.
[24] Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift, In International Conference on Machine Learning (ICML), pages 446. 2015.
[25] Saeed Reza Kheradpisheh, Mohammad Ganjtabesh, Simon J Thorpe, and Timothée Masque- lier. based spiking deep convolutional neural networks for object recognition. Neural Networks, 99:56-69 67, 2018.
[26] Saeed Reza Kheradpisheh and Timothée Masquelier. Temporal backpropagation for spiking neural networks with one spike per neuron. International Journal of Neural Systems (IJNS), 30(06):2050027, 2020.
[27] Youngeun Kim and Priyadarshini Panda. Revisiting batch normalization for training low-latency deep spiking neural networks from scratch. Frontiers Neuroscience, 17:773954-773554 2021.
11

---

[28] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
[29] Jun Haeng Lee, Tobi Delbruck, and Michael Pfeiffer. Training deep spiking neural networks using backpropagation. Frontiers in Neuroscience, 10:508, 2016.
[30] Hongmin Li, Hanchao Liu, Xiangyang Ji, Guoqi Li, and Luping Shi. CIFARIO-DVS: An event-stream dataset for object classification. Frontiers in Neuroscience, 11, 2017.
[31] Yuhang Li, Yufci Guo, Shanghang Zhang, Shikuang Deng, Yongqing Hai, and Shi Gu. Differ- entiable spike: Rethinking gradient-descent for training spiking neural networks, Advances in Neural Information Processing Systems (NeurIPS), 34:23426-23439, 2021.
[32] Wolfgang Maass. Networks of spiking neurons: The third generation of neural network models. Neural Networks, 10(9):659-11711 1997.
[33] Qingyan Meng, Shen Yan, Mingqing Xiao, Yisen Wang, Zhouchen Lin, and Zhi-Quan Luo. Training much deeper spiking neural networks with small number of time-steps. Neural Networks, 153:254-266 2022.
[34] Timoleon Moraitis, Abu Sebastian, and Evangelos Eleftheriou. Optimality of short-term synaptic plasticity in modelling certain dynamic environments. arXiv preprint arXiv:2009. 06808, 2020.
[35] Emre O Neftci, Hesham Mostafa, and Friedemann Zenke. Surrogate gradient learning in spiking neural networks:
Bringing the power of maudient-based optimization to spiking neural networks. IEEE Signal Processing Magazine, 36(6):51 (-66)::1663, 63, 2019.
[36] Dichl PU, Neil D, Binas J, Cook M, Liu SC, and Pfciffer M. Fast-classifying, high-accuracy spik- ing deep networks through weight and threshold balancing. In International Joint Conference on Neural Networks((JCNN), pages 1-8, 2015.
[37] Nitin Rathi, Gopalakrishnan Srinivasan, Priyadarshini Panda, and Kaushik Roy. Enabling deep spiking neural networks with hybrid conversion and spike timing dependent backpropagation. In International Conference on Learning Representations (ICLR), 2020.
[38] Hector Garcia Rodriguez, Qinghai Guo, and Timoleon Moraitis. Short-term plasticity neurons learning to learn and forget. International Conference on Machine Learning (ICML.), pages 1 18704-18722. PMLR, 2022.
[39] Robert Rosenbaum, Jonathan Rubin, and Brent Doiron. Short term synaptic depression imposes a frequency dependent filter on synaptic information transfer. PLoS Computational Biology. 86))xc002557, 2012.
[40] Kaushik Roy, Akhilesh Jaiswal, and Priyadarshini Panda. Towards spike-based machine intelligence with neuromorphic computing. Nature, 575(7784):607-617, 2019.
[41] Bodo Rueckauer, Illia-Alexaddra Lungu, Yuhuang Hu, Michael Pfeiffer, and Shih-Chii Liu. Conversion of continuous- valued deep networks efficient event-driven networks for image classification. Frontiers in Neuroscience, 1 11:682, 2017.
[42] Shibani Santurkar, Dimitris Tsipras, Andrew Ilyas, and Aleksander Madry. How does batch nor- malization help optimization? Advances in Neural Information Processing Systems (NeurIPS), pages 2488-2498, 2018.
[43] Abhronil Sengupta, Yuting Ye, Robert Wang, Chiao Liu, and Kaushik Roy. Going deeper in spiking neural networks: VGG and residual architectures. Frontiers in Neuroscience, 13:95, 2019.
[44] Sumit Bam Shrestha and Garrick Orchard. Slayer: Spike layer error reassignment in time. In Advances in Neural Information Processing Systems (NeurIPS), pages 1419-1428, 2018.
[45] Christoph Stöckl and Wolfgang Maass. Optimized spiking neurons can classify images with high accuracy through temporal coding with two spikes. Nature Machine Intelligence, 3(3):200-238, 2021.
12

---

[46] Misha V Tsodyks and Henry Markram. The neural code between neocortical pyramidal neurons depends on neurotransmitter release probability. Proceedings of the National Academy of Sciences, 94(2):719-723, 1997.
[47] Yeshwanth Venkatesha, Youngeun Kim, Leandros Tassiulas, and Priyadarshini Panda. Federated learning with spiking neural networks. IEEE Transactions on Signal Processing, 69:61 83-6194, 2021.
[48] Jibin Wu, Chenglin Xu, Xiao Han, Daquan Zhou, Malu Zhang, Haizhou Li, and Kay Chen Tan. Progressive tandem learning for pattern recognition with deep spiking neural networks. IEEE Trunsuctions on Puttern Analysis and Machine Intelligence (TPAMI), 2021.
[49] Yujie Wu, Lei Deng, Guoqi Li, Jun Zhu, and Luping Shi. Spatio-temporal backpropagation for training bigh-performance spiking neural networks. Frontiers in Neuroscience, 12:331, 2018.
[50] Yujie Wu, Lei Deng, Guoqi Li, Jun Zhu, and Luping Shi. Direct training for spiking neural networks: Faster, larger, better. In Proceedings of the AAAI Conference on Artificial Intelligence (AAAT), pages 1311-1318, 2019.
[51] Yuxin Wu and Kaiming He. Group normalization. In Proceedings of the European conference on computer vision (ECCV), pages 3-19, 2018.
[52] Qi Xu, Jianxin Peng, Jiangrong Shen, Huajin Tang, and Gang Pan. Deep CovDenseSNN:
A hierarchical event-driven dynamic framework with spiking neurons in noisy environment. Neural Networks, 121:512-519, 2020.
[53] Qi Xu, Yu Qi, Hang Yu, Jiangrong Shen, Huajin Tang, Gang Pan, et al. Csnn: an augmented spiking based framework with perceptron-inceetion. In International Joint Conference on Artificial Intelligence (IJCAI), pages 1652, 2018.
[54] Friedemann Zenke, Sander M Bohté, Claudia Clopath, Iulia M Comsa, Julian Göltz, Wolfgang Maass, Timothée Masquelier, Richard Naud, Emre O Neftci, Mihai A Petrovici, et al. Visual- izing joint future of neuroscience and neuromorphic engineering. Neuron, 109(4):571-575, 2021.
[55] Friedemann Zenke and Tim P Vogels. The remarkable robustness of surrogate gradient learning for instilling complex function in spiking neural networks. Neural Computation, 33(4):899
2021.
[56] Wenrui Zhang and Peng Li. Temporal spike sequence learning via backpropagation for deep spiking neural networks. Advances in Neural Information Processing Systems (NewrIPS), 2020.
[57] Hanle Zheng Yujie Wu, Lei Deng, Yifan Hu, and Guoqi Li, Going deeper with directly- trained larger spiking neural networks. In Proceedings of the AAAI Conference on Artificial Intelligence(AAAI), pages 1062-11070, 2021.
[58] Shibo Zhou, Xiaohua Li, Ying Chen, Sanjeev T Chandrasekaran, and Arindam Sanyal. Temporal- coded deep spiking neural network with easy training and robust performance. In Proceedings of the AAAI Conference on Artificial Intelligence (AAAI), pages 11143-11151, 2021.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes]
(c) Did you discuss any potential negative societal impacts of your work? [Yes]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you including theoretical results...
13

---

(a) Did you state the full set of assumptions of all theoretical results? [Yes]
(b) Did you include complete proof' of all theoretical results? [Yes] The proof is available in the supplementary.
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as a URL)? [Yes] Our code will be available in the supplementary.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] All implementations are described in the experiments section and supplementary.
(c) Did you report error bars (e.g.. with respect to the random seed after running experi- ments multiple times)? [No] Fixed random seeds are used.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes]
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes] We use CIFAR10/100 and CIFAR1O-DVE [30]
(b) Did you mention the license of the assets? [No] We adopt public datasets.
(c) Did you include any new assets either in the supplemental material or as a URL? [Yes]
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [No]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [No]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
14


Reviews:
[{"review_id": "wv9-2WbDfa8", "reviewer": "Reviewer_RAfA", "summary": "The manuscript presents a technique (TEBN) for batch normalization in SNNs that takes into account the temporal dimension of the spiking input to rescale and recenter the input distribution differently at each input timestep, even though it still estimates the mean and variance of the whole batch, throughout all timesteps. The paper's theory shows that this smoothens the loss landscape. Experiments show that TEBN improves accuracy, robustness, and latency compared to other batchnorm methods and compare to the state of the art in the SNN literature, in image recognition of common benchmark datasets, including recordings of static images through neuromorphic vision sensors.", "questions": "Could the authors please respond to the above points, and ideally address them with experiments?\nAre the accuracies of other methods reported in the tables results from the authors own experiments, or from the literature? Please clarify in the paper.", "limitations": "The authors have not discussed any limitations of the work.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper targets a true problem in the field of SNNs, and appears to provide a truly improved solution for certain important cases. In addition, it provides theoretical analysis, and experimental insights deeper than merely end accuracies, showing the impact of the method on input distributions and on robustness to hyperparameters.\n\nOn the other hand, the paper:\n- does not address issues related to truly sequential data. SNNs are considered as most suitable for temporal tasks, therefore a complete evaluation of the method should involve such tasks. An example is keyword spotting on the Speech Commands dataset. Currently, all tested datasets in the paper only include static images, even if in some cases the images are recorded through a DVS.\n- does not compare to layer normalization, a technique known as suitable for recurrent networks and temporal datasets, even though the authors' input includes a temporal dimension, and even though LIF SNNs are recurrent networks.\n- does not explain how its results compare to the true state of the art, i.e. beyond SNNs. For example, non-spiking networks can also reduce latency and computation while maintaining their higher accuracy than SNNs (Jeffares et al., ICLR 2022 https://openreview.net/forum?id=iMH1e5k7n3L). But more generally, it would be good to put the results back in the context of the broader ML field, and discuss the differences e.g. in accuracy, latency, or computational efficiency.\n- does not mention the method's strong similarity to short-term plasticity (STP), which is a long-known property of biological SNNs for temporal filtering of inputs (see e.g. Fortune & Rose, 2001 https://roselab.biology.utah.edu/publications/trends2001.pdf; Rosenbaum et al., 2012 https://journals.plos.org/ploscompbiol/article?id=10.1371/journal.pcbi.1002557). Temporal filtering is what the authors' TEBN also applies. Of course, STP applies a fixed, e.g. exponential kernel, whereas TEBN learns the shape of the temporal kernel. On the other hand, short-term plasticity is also learnable (Garcia-Rodriguez et al., ICML 2022 https://arxiv.org/abs/2206.14048), which then makes STP similar to what the authors here achieve, but in several aspects with more flexibility than what the authors here propose. Moreover, STP has been shown to be a powerful property of SNNs, such that SNNs with STP can even outperform LSTMs and CNNs in accuracy (Moraitis et al., 2021 https://arxiv.org/abs/2009.06808). Even though the authors do not have to perform experimental comparisons of their method against STP, STP's relevance through these works must be mentioned, and discussed as a potential alternative for future work, that is also more biologically-realistic.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "ky4EkCirrE", "reviewer": "Reviewer_gvBu", "summary": "The paper proposes a temporal BN method to train SNNs with high accuracy,", "questions": "My main concern is teh technical novelty of this paper with respect to previous works. The authors have performed a holistic comparison. But, can they comment on how their work is technically novel? At this time, in my opinion, the paper reads to be more incremental effort.", "limitations": "Please see the above comments on weaknesses.", "rating": 4, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "+ The methos is simple and effective as suggested by author's results.\n-The main comment I ahve is on the novelty of this work. The authors have cited many relevant works such as BNTT, TDBN etc. I think the paper's results are very similar to that of BNTT. Except for the fact that the accuracy is better in the datasets that authros have exeprimented with, I don't thinkt here is a lot of technical contribution or novelty. \n-Further, authors of BNTT and TDBN experimented on larger datasets like Tiny Imagenet. Can the authors commnet on the scalability of their approach?\n-BNTT and TDBN like methods were adapted for learning in diverse scenarios, liek Federated learning [1], Segmentation [2], large scale DVS datasets, NAS optimization[3] among others. Maybe the authors can include a commentary or discussion on how their approach can be applied to more diverse learning scenarios.\n\n[1] Venkatesha, Yeshwanth, et al. \"Federated learning with spiking neural networks.\" IEEE Transactions on Signal Processing 69 (2021): 6183-6194.\n[2] Kim, Youngeun, et al. \"Beyond classification: directly training spiking neural networks for semantic segmentation.\" arXiv preprint arXiv:2110.07742 (2021).\n[3]Kim, Youngeun, et al. \"Neural architecture search for spiking neural networks.\" arXiv preprint arXiv:2201.10355 (2022).", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "k_PfUt6I8gk", "reviewer": "Reviewer_VwQw", "summary": "The authors proposed a batch normalization scheme considering the temporal domain, TEBN. The proposed TEBN requires all time-step information to estimate the desired expectation and variance. The authors provided theoretical analysis and validated the effectiveness of the proposed TEBN with classification tasks. ", "questions": "Besides the questions, I listed above, the authors must need to describe the detailed architecture of the models used in their experiments. They cannot assume the audiences know them or let audiences play a guessing game. ", "limitations": "The validation is not convincing at all from my perspective. Only LIF and classification tasks cannot show the generalization of the proposed method. Without revealing the performance of the tasks that rely on temporal information, I do not think the proposed scheme is meaningful to the community. \n\nAs such, I suggest the authors show that the proposed scheme works with time-sensitive tasks. ", "rating": 3, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n+1. The authors showed good efforts in theoretical analysis, which is desired for the NeurIPS audience. \n\nWeaknesses:\n\n-1. Requiring all time steps information is not practical at all, especially for time-dependent tasks, e.g., tracking. For classification tasks, it may make sense. However, using SNN for classification tasks lacks justification as the tasks do not really need temporal information.\n\n-2. The experimental setup is unclear, and validations are weak. For VGG, ResNet, and other architectures, which parts are based on SNNs? For the ResNet-19, did the authors only replace the ReLU with SNNs?\n* The experimental results did not show the impacts of the following factor on the classification tasks.\n     * mini-batch size\n     * spiking threshold\n     * higher time steps: The authors claimed that fewer time steps are enough for the tested classification tasks. However, how do the larger time steps impact the results?\n\n-3. The experimental results are only based on LIF. What about other SNN neuron models, e.g., SRM, IF?\n\n\n\n ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "RNY5iYFOaL5", "reviewer": "Reviewer_Ljtf", "summary": "This paper presents an efficient batch normalization (BN) method to smooth and uniform the temporal distributions of the presynaptic input in the Spiking Neural Networks (SNN). By combing the BN with temporal features, the proposed TEBN could alleviate the gradient vanishing problem to some extent. The proposed TEBN shows better classification accuracy and robustness to hyper-parameters on CIFAR-10, CIFAR-100 and DVS-CIFAR10 with fewer time-steps.", "questions": "\n1) Line 40, the sentence, ‘Theoretical analysis shows that our approach can be viewed as a smoother of SNN’s optimization landscape and could help stabilize the gradient norm.’ need a revision as it is repetitive with the statement in the abstract. \n2) Line 65, please add a citation to support the statement (less than 15 layers). \n3) Page9, Figures 2 and 3. The X and Y limit should be the same for both figures since the authors could compare distributions. Besides, I expect the author to conduct some quantitative measures to compare the distributions, rather than directly state that TEBN appears more homogeneous.\n4) I would recommend the author validate the efficiency of mitigating gradient explosion and vanishing (Line 215) with the proposed TEBN using a deeper model (e.g., ResNet-50).\n", "limitations": "The efficiency of mitigating gradient explosion and vanishing (Line 215) with the proposed TEBN is not validated efficiently in this work. And what is the biological meaning of the hyper-parameters in LIF model? Or if the authors explain how different hyper-parameters influence TEBN with theoretical analysis not only experimental evaluations, the results would be more sound.", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: \n1) The writing is clear and the motivation is clarified clearly. Besides, the theoretical grounding and experimental evaluation are also sufficient to show its originality and significance.  \n2) Experimental results also demonstrate the superiority of the proposed TEBN method. Through Fig. 1, we can see the advantages of the proposed TEBN Intuitively compared to tdBN.\n3) This paper gives a detailed illustration of how TEBN processes in a surrogate-gradient SNN training method. If the authors show the input and output simultaneously with or without TEBN, the effectiveness maybe better in a visual way. \n4) The spiking nondifferentiable leads to a non-smooth with the loss landscape, the authors analyze the smoothing optimization in sec. 5 which I think is very important for using the surrogate-gradient or ANN2SNN method training a deep SNN model.\n\nWeaknesses: \nHow does internal covariate shift appear in SNN? In my opinion, because of the different scales from each layer's output, ICS in ANN  is obvious, however, in this paper, when you used the time-series LIF neuron model, the ICS should be changed in the time window. When using rate-based coding, the spikes are more intensively compared to temporal coding, how does TCS behave in addition to sec. 4.1? \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept