TAB: Temporal Accumulated Batch Normalization in Spiking Neural Networks
Authors: Haiyan Jiang
Vincent Zoonekynd
Giulia De Masi
Bin Gu
Huan Xiong
Venue: ICLR 2024 poster (2024)
Keywords: Temporal Batch Normalization
Spiking Neural Networks
Primary area: applications to neuroscience & cognitive science

Abstract: Spiking Neural Networks (SNNs) are attracting growing interest for their energy-efficient computing when implemented on neuromorphic hardware. However, directly training SNNs, even adopting batch normalization (BN), is highly challenging due to their non-differentiable activation function and the temporally delayed accumulation of outputs over time.
    For SNN training, this temporal accumulation gives rise to Temporal Covariate Shifts (TCS) along the temporal dimension, a phenomenon that would become increasingly pronounced with layer-wise computations across multiple layers and multiple time-steps. 
    In this paper, we introduce TAB (Temporal Accumulated Batch Normalization), a novel SNN batch normalization method that addresses the temporal covariate shift issue by aligning with neuron dynamics (specifically the accumulated membrane potential) and utilizing temporal accumulated statistics for data normalization. 
    Within its framework, TAB effectively encapsulates the historical temporal dependencies that underlie the membrane potential accumulation process, thereby establishing a natural connection between neuron dynamics and TAB batch normalization. 
    Experimental results on CIFAR-10, CIFAR-100, and DVS-CIFAR10 show that our TAB method outperforms other state-of-the-art methods.

Published as a conference paper at ICLR 2024
TAB: TEMPORAL ACCUMULATED BATCH NORMALIZA- TION IN SPIKING NEURAL NETWORKS
Haiyan Jiang' Vincent Zoonekynd Giulia De Masi
Bin Gu
Huan Xiong 1,5+
Mohamed bin Zayed University of Artificial Intelligence, UAE
2 Abu Dhabi Investment Authority, UAE TTechnooogy Innovation Institute, UAE 'Sant' Anna School of Advanced Studies, Italy HHarbin Institute of Technology, China School of Artificial Intelligence, Jilin University, China
haiyan. j fiang@mbzual ac. vincent. zoonekynd@ adwa.
1 gmail. com
(bin. huan. muan.xiong] @mbzuai. ac
ABSTRACT
Spiking Neural Networks (SNNs) are attracting growing interest for their energy- efficient computing when implemented on neuromorphic hardware. However, directly training SNNs, even adopting batch normalization (BN), is highly challeng- ing due to their non-differentiable activation function and the temporally delayed accumulation of outputs over time. For SNN training, this temporal accumulation gives rise to Temporal Covariate Shifts (TCS) along the temporal dimension, a phenomenon that would become increasingly pronounced with layer-wise compu- tations across multiple layers and multiple time-steps. In this paper, we introduce TAB (Temporal Accumulated Batch Normalization), novel SNN batch normal- ization method that addresses the temporal covariate shift issue by aligning with neuron dynamics (specifically the accumulated membrane potential) and utilizing temporal accumulated statistics for data normalization. Within its framework, TAB effectively encapsulates the historical temporal dependencies that underlie the membrane potential accumulation process, thereby establishing natural con- nection between neuron dynamics and TAB batch normalization. Experimental results on CIFAR-10 10, CIFAR-100, and DVS- DVS-CIFARII 10 show that our TAB method outperforms other state-of-the-arr methods.
INTRODUCTION
Spiking Neural Networks (SNNs) are known to be biologically inspired artificial neural networks (ANNs) and have recently attracted great research interest (Chowdhury et al., 2022; Ding al., 2022). The attraction of SNNs lies in their ability to deliver energy-efficient and fast-inference computations when implemented on neuromorphic hardware such as Loihi (Davies et al., 2018) and TrueNorth (Akopyan et al., 2015; DeBole et al., 2019). These advantages arise from the fact that SNNs utilize spikes to transmit information between layers, whereby the networks circumvent multiplication during inference (Roy al., 2019). However, the discrete and non-differentiable nature of the binary firing functions makes it difficult to directly train deep SNNs. ANN-to-SNN conversion (Diehl et al., 2015; Bu al., 2022; Jiang et al., 2023) and directly training with surrogate gradients back-propagation (Neftci et al., 2019; Deng et al., 2022; 2023) are two typical solutions.
Batch Normalization (BN) has found extensive use in ANNs and has seen tremendous success in boosting their performance by reducing the internal covariate shift (ICS) and flattening the loss landscape (Ioffe & Szegedy. 2015; Santurkar et al.. 2018). ANNs, ICS refers changes in the distribution of layer inputs caused by updates of preceding layers, while in SNNs, the Temporal Covariate Shift (TCS) phenomenon (Duan et al., 2022) has been identified due to updates of preceding layers and prior time-steps, which transpires along the additional temporal dimension. Within SNNs, synaptic currents are sequentially fed into spiking neurons, with spike-triggent asynchronous currents accumulating in the membrane potential. Whenever this accumulated membrane potential exceeds a threshold, a spike is generated. This temporal dependency on membrane accumulation
"Corresponding authors. Codes available httph:t/ttthhieeeee com/ yanJ ang/ SNA-

---

Published as a conference paper at ICLR 2024
has the potential to amplify the internal covariate shift across the temporal domain. The intertwining of this temporal dependency with the TCS phenomenon, presents a significant challenge in direct training of SNNs especially for the integration of BN techniques into SNNs.
When it comes to BN techniques for SNNs, only a few methods have been proposed. These methods either normalize data jointly by aggregating data across the temporal dimension or perform independent normalization at each discrete time-step. For example, Kim & Panda (2021) conducts independent batch normalization separately at each time-step. However, this approach uses separate sets of mean, variance, and scale and shift parameters at each time-step, failing to account for the temporal dependencies of the input spikes. While Zheng et al. (2021) merges the data along the time dimension and utilizes shared batch statistics across all time-steps for normalization. Nonetheless, introducing such overall statisties may limit the flexibility to capture varying temporal characteristics at different time-steps. On the other hand, Duan et al. (2022) attempts to tackle the TCS issue by assigning different weights to each time-step, while still utilizing shared batch statistics across all time-steps for normalization. Although these methods improve upon the performance of the SNN models, they do not significantly address the alignment with the neuron dynamics, i.e., the membrane accumulation dependency, or provide a potential to do so.
In this paper, we propose TAB (Temporal Accumulated Batch Normalization) as a solution to effectively address these challenges by closely aligning with the neuron dynamics, specifically the accumulated membrane potential, and providing more accurate batch statistics. This alignment establishes natural connection between neuronal dynamics and batch normalization in SNNs. Neuron dynamics refer to the changes in the membrane potential of a neuron over time as it integrates input signals and generates spikes. Here, "aligning with neuron dynamics" means that TAB is tailored to mimic or capture neurons' behavior as closely as possible, normalizing data in line with the temporal dependencies and information accumulation within neurons. This alignment ensures that TAB's 's normalization process corresponds well with how neurons naturally operate in SNNs, thus leading to improved performance by addressing the temporal covariate shift problem.
BACKGROUND
2.1 RELATED WORK
SNN Learning Methods. Many works have recently emerged and focused on the supervised training of SNNs (Wu et al., 2021a; Zhou et al., 2021; Meng et al., 2022; Xiao et al., 2021). These SNN learning methods can be mainly categorized into two classes: ANN-to-SNN conversion (Dichl et al., 2015; Deng & Gu, 2021; Ding et al., 2021; Han et al., 2020; Li al., 2021 la; Bu et al., 2022; Hao et al., 2023; Lv et al., 2023) and end-to-end training with back pack-propagation (Fang et al., 2021a; Zhang & Li, 2020; Deng et al., 2022; Xiao et al., 2022; Guo et al., 2022: Meng al., 2023). ANN-10-SNN conversion takes a pre-trained ANN and converts it into an SNN by preserving the weights and replacing the ReLU activation function with a spiking activation function. This approach can be efficient in obtaining an SNN since the ANN has already been trained and the weights can be directly copied to the SNN. However, the resulting performance of the converted SNN may not be as good as that of the original source ANN. It usually requires a large number of time-steps for the converted SNN to achieve performance comparable to the source ANN. Direct end-to-end training usually employs the surrogate gradients (Wu et al., 2018; 2019; Neftci et al., 2019; Zheng et al., 2021; Eshraghian et al., 2021) method to overcome the non-differentiable nature of the binary spiking function to directly train SNNs from scratch. This method can yield comparable performance that of traditional ANÑS with a few time-steps.
BN Method in ANNs. Batch normalization methods have significantly contributed to the success of ANNs by boosting their learning and inference performance (Ioffe & Szegedy, 2015; Xiong et al., 2020; Bjorck et al., 2018). BN is a technique used to stabilize the distribution (over mini-hatch) of inputs to each network layer during training. This is achieved by introducing additional BN layers which set the first two moments (mean and variance) of the activation distribution zero and one. Then, the batch-normalized inputs are scaled and shifted using learnale/trainable parameters to preserve model expressiveness. This normalization is performed before the linearitty is applied. The BN layer can be formulated as,
D
H
BN(x:) = +8, 84
i=
i=
Vo++e
2

---

Published as a conference paper at ICLR 2024
The mini-batch mean μ and variance x² are computed by μ Ei= x, and o² = E (2; 22?
BN Method SNNs. Due to the additional temporal dimension, several recent studies have proposed modifications to batch normalization to fit the training of SNNs. The threshold-dependen Batch Normalization (tdBN) method (Zheng et al., 2021) is introduced to alleviate the gradient vanishing or explosion during training SNNs. The tdBN utilizes shared BN statistics and parameters (as the conventional BN) by merging the data along the temporal dimension. Similar to tdBN, the TEBN method (Duan al.. 2022) employs shared BN statistics by merging the data along the temporal dimension, then scales using different weights to capture temporal dynamics. Different from them, BNTT (Kim & Panda, 2021) uses separate BN statistics and parameters at each time-step t independently, however, it ignores the temporal dependencies of the input spikes. Differently, our TAB method leverages the accumulated pre-synaptic inputs in the temporal domain, which is in alignment with the membrane potential accumulation in the LIF model.
2.2 SPIKING NEURON DYNAMICS AND NEURON MODEL
SNNs use binary spike trains to transmit information between layers. Each neuron maintains its membrane potential dynamics (t( over time, "integrates" the received input with a leakage (much like an RC circuit), and fires spike if the accumulated membrane potential value exceeds a threshold. We adopt the widely used -et--nnttteead-fiiii (LIF) model. Neuron dynamics refer to the changes in the membrane potential of neuron over time as it integrates input signals and generates spikes, which can be formulated first-order differential equation (ODE),
du,(t) LIF Neuron Dynamics: T dt
(t) + Rt)(),
un (t) Vah.
(1)
where (t) is the injected input current to the i-th neuron at time i, (t) is the membrane potential of the i-th neuron at time t in the current layer, Vea is the membrane threshold, and denotes the membrane time constant, and R denotes the resistor. For numerical simulations of LIF neurons, we consider discrete version of the neuron dynamics. Similar Wu & He (2018), the membrane potential u, ft of the i-th neuron at time-step (discrete) is represented as:
u [t] Xu,,
+|+
Σ jEpro
W
We adopt simple current model RI =
EE [t with R absorbed in weights Wiy- Here, ot denotes the binary spike of neuron i at time-step [t], taking value of 1 when spike occurs and 0 otherwise. The index j refers to pre-synaptic neurons. The membrane potential u,(t] increases with the summation of input spikes from all the pre-synaptic neurons pre(i) connecting the current i-th neuron through synaptic weight Wiy. also decreases with a leak factor (0 < X & 1),
where À and the time constant are related by X e A The discrete LIF model degenerates to the IF model when À = 1, therefore in the following, we only use the LIF model with 0 < X < 1. When the neuron's 's membrane potential u,(t] exceeds the threshold Via the neuron will fire spike with o; [4] = 1 and then reset the membrane potential to O. By combining the sub-threshold dynamics Eq. (2) and hard reset mechanism, the whole iterative LIF model can be formulated by:
Discrete LIF Neuron Model:
u. [4] = Xu,(1 1](1 0, - 1]) + Σ WyjofM4
(3)
jEpre(f)
o [t] H (u.(t) Vin),
where H(x) is the Heaviside step function, i.e., the non-differentiable spiking activation function. H(x) = 1f > 0 and (xx) otherwise.
PROPOSED TAB METHOD
In this section, we will present our TAB method. We begin by introducing the Temporal Dependencies and Temporal Covariate Shift SNNs which motivate our method. Following this, we introduce TAB method, which addresses these challenges. Finally, we establish theoretical connection between the neural dynamics and the TAB method by deriving the closed-form solution of LIF dynamics ODE.
3

---

Published as a conference paper at ICLR 2024
3.1 MOTIVATION: TEMPORAL DEPENDENCIES AND TEMPORAL COVARIATE SHIFT
Temporal dependencies in SNNs arise naturally from the sequential nature of spike events, where synaptic currents (also known as spike trains) are sequentially fed into spiking neurons, playing pivotal role in capturing the dynamic evolution of input spikes over time. These networks model the dynamics of biological neurons through ODEs and utilize spikes to transmit information (Eshraghian et al., 2021). In SNNs, each neuron maintains a membrane potential, continuously 'integrating' and accumulating received spikes over time. emits spike only when its accumulated membrane potential exceeds a threshold, remaining inactive otherwise in the current time-step (I.i et al., 2021 a)). This process highlights the intrinsic influence of temporal dynamics on the temporally delayed accumulation of the membrane potential. We refer to this accumulation dependency over the time dimension as temporal dependencies.
In SNNs, a phenomenon known as Temporal Covariate Shift (TCS) has been identified (Duan et al., 2022), which represents ICS (Internal Covariate Shift) (loffe & Szegedy, 2015) across the additional temporal dimension, and it refers to changes in the distribution of layer inputs caused by updates of preceding layers, and prior time-steps. Within the framework of SNNs, synaptic currents are sequentially fed into spiking neurons, and srike-triggered asynchronous currents are accumulated into the membrane potential which will trigger a spike when it exceeds the membrane threshold. This temporal dependency on membrane potential accumulation intensifies the internal covariate shift along the temporal domain. This temporal dependency, together with the TCS phenomenon, presents significant challenge when integrating BN techniques into SNNs.
Our motivation comes along these lines, how to perform batch normalization in training of SNNs, but keeping in mind the temporal dependency of the data, as well the temporal covariate shift. A simple, yet elegant, method that aligns closely with this underlying neuron dynamics comes with Temporal Accumulated Batch nor- malization (TAB). Generally speaking, our TAB method addresses the temporal covari- ate shift issue by aligning with the inherent temporal dependencies in SNNs. Fig. illustrates the temporal dependencies and neuron dynamics and showcases the in- volvement of our proposed TAB method.
u 1]
4 [4]
u'( + 1]
t S t - 1]
3' [t]
a' + 1]
+ TAB
TAAB
TAB
x² | 1]
ae'4]
xe(t + 1] + W
+ W
W
- [t- 1]
[c]
o 1]
H
H
H
uu [t
ue" [c]
[t
Neuronal dynamics refers to the change in membrane potential over time as a neu- ron integrates input signals and generates spikes. This temporal accumulation of the membrane potential in SNNs enables neu- rons process input data by taking into account both past and current time-steps (with no access to future information be- yond /), and the TAB method aligns closely with this underlying neuron dynamics and alleviates the TCS issue.
The temporal dependencies and neuron dy- namics in SNNs. specifically the temporal dependency associated with the accumulation of membrane potential in the discrete LIF model. The black arrows represent the temporally delayed accumulation over time, while the red arrows indicate the information flow along the spatial layers (vertical axis) and the temporal domain (horizontal axis).
Figure 1:
3.2 TEMPORAL ACCUMULATED BATCH NORMALIZATION (TAB)
To address the temporal covariate shift issue and to model the temporal distributions in SNNs, our TAB method aligns with the inherent temporal dependencies by utilizing the temporal accumulated batch statistics (Hi.e: )i.)) over an expanding window [1, t). To achieve this, we establish the relationship between the expectations and variances across accumulated time-steps (J1a,, ff.) and those of the
4

---

Published as a conference paper at ICLR 2024
single time-step (1ft), ((t), follows:
P1 Σ
i
1 i
Σ
0²
Our proposed TAB method utilizes Temporal Accumulated Statistics (Wiss o.)) for data normaliza- tion, and then assigns different learnable weights w/(] >> 0 to each time-step to distinguish their effect on the final result. The TAB method is given by
x Pri iit +e
MIR F:
2.[1 = (ZB((z (z) = wit]
7(1]
+ B[t] = -|t]]
B[t]
w(t] > 0. (6)
1 +
Given the pre-synaptic inputs x' [t] to layer at time-step t, the spiking neuron with TAB is as follows,
x [1] w o [
u 111(] = Au'(( - 1](1 - 0'11 11) (0()),
(8)
s'( = TAB(a'(1) = -911 2014 Pla + 3
where
(9)
Voi. +
Here u'(t and o'[t] denote the membrane potential and binary spike outputs of all neurons in l-th layer at time-step I, and W denotes the synaptic weights between layer - and layer I. We assign different positive weights w'1 >> 0 to each time-step which is different from Deng et al. (2022) and [tt = ( (t). [[ = w [t). The weights w(t] and parameters > [t),, ['(]] are learnable, which are trained during the training process. For details, refer to Append. A and Append. B. Refer to Append. C for the learning rules compute the gradients.
Computation of the temporal accumulated statisties is dynamically performed, a moving averaging fashion, without the need to store batch data from all previous time-steps. This not only saves memory, but is also an important feature of our novel approach. For the algorithm details of the TAB method, please refer algorithm in the Appendix.
The rationale behind employing this accumulated spatial- temporal information in TAB comes from the sequential processing and temporal dependency characteristics intrinsic spiking neurons. The TAB method utilizes the accumulated batch statistics (Ju:,, af.) over an expanding window [1, Fig. 2 illustrates overview of four typical BN methods used in SNNs: default BN (loffe & Szegedy, 2015), BNTT (Kim & Panda, 2021), tdBN (Zheng et al., 2021) I), and TEBN (Duan et al., 2022). A comprehensive overview of statisties and parameters used by these methods is summarized in Table SI in the Append. B.
As shown in Table S1, BNTT (Kim & Panda, 2021) considers BN statistics each time-step individually and calculates different BN statistics (ufE), (²()) and BN parameters (y(tt) at each time-step, which ignores the temporal dependencies of the input spikes. In contrast, tdBN (Zheng et al., 2021) computes the same overall BN statistics (41.7, (i.r) and BN parameters (7, B) across all time-steps, but overlooking the temporal differences. Similarly, TEBN (Duan al., 2022) employs the same overall BN statistics (M1.T, o:r)) tdBN, but introduces distinct weight parameters i)] at each time-step to capture time-specific variations. However, both tdBN and TEBN, computing BN statistics over T' time-steps, implicitly assume access to data from all T stme-steps, that is, even if the current time-step is t < T, future information up to time-step T can also be obtained, which is not true for the temporal accumulation of membrane potential nor the neural dynamics. As illustrated in Fig. 2, the input statistics of tdBN and TEBN consider the statistics of all the time-steps and all effective batches, while BNTT considers BN statistics at each time-step. Despite these differences, none of the existing methods have addressed the aligmment with the membrane potential accumulation.
3.3 THEORETICAL CONNECTION BETWEEN TAB METHOD AND THE NEURAL DYNAMICS
TAB is tailored to capture the temporal dependencies of neurons as closely as possible by aligning with the neuron dynamics. To explore the theoretical connection between the TAB method and the neural dynamics, we need to delve into the LIF dynamics from the perspective of differential equations. In SNNs, each neuron maintains the dynamics of its membrane potential U (c) over time,
5

---

Published as a conference paper at ICLR 2024
T
w|T
T
&&W ANN
P[1
& W &N
H&W&N
H& &N
p(2]
wi
t ==
2
2
H&W &N
p/27
& &N
H&W &N
& &N
H&WAN &N
H&W &NN
H1 H&W &N
H&W &N
==1
& w &N
H&W &N
H& W &N
BN
BNTT
tdBN
TEBN
TAB
Figure 2: Comparison of different Batch Normalization methods with one given channel. In conven- tional BN, there is time dimension. BNTT independently normalizes data at each time-step. The IdBN jointly normalizes data across all time-steps. TEBN shares a similar approach with tdBN but incorporates per-time-step scaling of the normalized data. In contrast, our TAB normalizes data using temporal accumulated statistics up to time-step step and subsequently applies scaling.
by "integrating" the received input current (t) with leakage term until spike is triggered. This is described as a first-order linear differential equation (ODE),
dU(()) -U() + RI(t), U(t) < Vsh di
Neuron Dynamics
ODE:
(10)
where [(t) represents the input current injected into the neuron at time t, and it is a function of t (note that ((t) is not a constant value). The closed-form solution of the LIF neuron dynamics (as an ODE) can be derived with analytical and theoretical methods. Additional details are available in Append. D.I and Append. D.2.
Lemma 1. The analytical closed-form solution for the first-order IVP (Initial Value Problem) of the LIF dynamics ODE as follows (Gerstner al., 2014),
Utt) exp (ocpp-() (.
R -
(7) ds Uo Uo)
+UO)+
(11)
Remark 1. When the neuron initiates at the value Uo with no further input. i.e., I(t) = 0, the closed-form solution of the ODE Eq. (11) shows that the membrane potential (t) will start at Un and exponentially decay with time constant T, U(t) Usexp (-+)- Consequently, can determine the membrane potential ratio, often referred to as the leak factor, denoted by A.
as A U(t+At) Upexp( (+A()
exp ). This relationship enables to formulate the discretization scheme as: Ult + (t)
D())
(bexp(-5)
This remark provides insights into the behavior of the membrane potential in the absence input and establishes the discretization principle used for LIF modeling.
Lemma 2. Through applying integration by parts, we derive another equivalent form of the closed- form solution for the LIF dynamics, denoted as:
exponential decay term
input current mode!
(t) (Uo- RIo)exp 1
-S Rexp dI(s)
RI(t)
(12)
commonly considered in the discrete LIF model
absent discrete LIF model
With the application of the Riemann-Stieltjer integral, the discretization version of the closed-form solution is represented as:
(Uo (Uo-Rlo)ex(($5)
WORL =
UMD = AU
Σ 9
(13)
TAB method

---

Published as a conference paper at ICLR 2024
In this formulation Eq. (13), the first exponential decay term, AU 11, captures the temporal dependency of the membrane potential from the preceding time-step. The second term, a simple current input model, R W [t]] incorporates spikes from the pre-connected neurons at the current time-step [4]- Significantly, the third term, representing the temporal accumulated input across all previous time-steps through a weighted sum of the input currents X [[s;] with associated weights Pi, t denotes a partition of the time
introduces novel concept. Here 0 = So < ... < < ... < interval [0. f] with finite sequence of numbers. Refer to Append. D.3 for the details. Importantly, note that this accumulation mechanism of the inputs is foundational component of the TAB method, providing link that connects the TAB method and the neural dynamies.
Remark 2.
The commonly used discrete LIF model in Eq. (2), as denoted by Uit] -
is derived from the first two terms of the discretization version of the closed-form solution Eq. (13). The third term, representing the temporal accumulated input across all previous time-steps, however, is not incorporated into the discrete LIF models typically used in practice.
Remark 3. Note that the recursive application of the discrete LIF model, as denoted by U[[] =
At 1] + X[t). yields the temporal evolution of the membrane potential Utt = XU
Σ. X This result shows the temporal dependency of the membrane potential accumulation in LIF neuron dynamics.
Recalling the TAB method introduced in Sect. 3.2, our TAB method normalizes data utilizing temporal accumulated batch statistics (Mize ai.) across an expanding window [I.t], where pa and o. represent the temporal accumulated information up to time-step [4]. The utilization of the temporal accumulated batch statistics aligns well with the accumulation mechanism of the membrane potentialthrough Eq. (13). Consequently, it alleviates the temporal covariate shift issue which refers to the changes in the distribution of layer inputs resulting from updates of preceding layers and prior time-steps. The entire TAB method procedure and membrane updates can be linked through Eq. (13), derived by solving the LIF dynamics ODE. This equation naturally connects TAB batch normalization to neuron dynamics, as evident in Eq. (13).
Upon comparing the commonly used discrete LIF model in Eq. (2) with the discrete closed-form solution in Eq. (13), it shows that the TAB method reintroduces the accumulation term into the normalization procedure. This is achieved by using temporal accumulated batch statistics from time-step 1 to t. While the temporal accumulated batch statistics employed by the TAB method do not replicate the exact term in Eq. (13), but as an approximation. Thus, there exists no one-to-one functional mapping between the two. The adjustment within TAB method brings the discrete LIF model closer to its analytical closed-form counterpart, thus, TAB can work well in addressing the temporal covariate shift issue This establishes natural connection between neuron dynamics and batch normalization.
4 EXPERIMENTS
In this section, we conduct extensive experiments on large-scale static and neuromorphic datasets, CIFAR-10 (Krizhevsky et al., 2009), CIFAR 100 (Krizhevsky et al., 2009), and DVS-CIFAR 10 (Li et al., 2017), to verify the effectiveness of our proposed TAB method. We utilize the VGG network architecture and ResNet architecture. Firstly, we perform comparative analysis of our TAB method with other BN methods in the context of SNNs. Further, we compare our TAB method with other state-of-the-an aart--rrttt--ttteee approaches. For implementation details, refer to Append. E.
4.1 COMPARISON WITH OTHER BN METHODS
We conduct our evaluation by comparing the performance of the proposed TAB method and other batch normalization methods in the context of SNNs. To ensure fairness our comparisons, we do not employ advanced data augmentation techniques like cutout VDevs & Taylor, 2017). Table 1 provides a comprehensive overview of the the test accuracy on both traditional static dataset CIFAR- 10, CIFAR- and neuromorphic dataset DVS-CIFAR10 10. On the CIFAR-10 dataset, our TAB method demonstrates remarkable performance improvement, achieving a top-1 accuracy of 94.73% with the ResNet-19 19 network using only 2 time-steps. Notably, this surpasses the performance of TEBN using 6 time-steps. Furthermore, when using the same network architecture, TAB consistently outperforms other BN methods, even with fewer time-steps T. This pattern holds true for other
7

---

Published as a conference paper at ICLR 2024
Table 1: Comparison between the proposed TAB method and other BN methods in SNNs.
Dataset
Model
Method
Architecture
Time-steps
Accuracy (%)
SPIKE-NORM (Sengupta et al., 2019)
ANN-to-SN
VGG-16
2500
91.55
NeuNorm (Wu al., 2019)
Surrogate Gradient
CIFARNet
12
90.53
BNTT (Kim & Panda, 2021)
Surrogate Gradient
VGG-9
20
90.30
IdBN (Zheng al., 2021)
Surrogate Gradient
ResNet-19
6/4/2
93.16
92.92 92.34
CIFAR-10
VGG-9
4
92.81
TEBN (Duan et al., 2022)
Surrogate Gradient
ResNet-19
6/4/2
94.71
94.70/ 99.557
VGG-9
4
93.41
TAB (Ours)
Surrogate Gradient
ResNet-19
6/4/2
94.81
99.776 94.73
SPIKE-NORI (Sengupta et al., 2019)
ANN-to-SNN
VGG-16
2500
70.90
BNTT (Kim & Panda, 2021)
Surrogate Gradient
VGG-11
50
66.60
TEBN (Duan al., 2022)
Surrogate Gradient
VGG-11
4
74.37
CIFAR-100
TEBN (Duan al., 2022)
Surrogate Gradient
ResNet-19
6/4/2
76.41
76.13 75.86
VGG-11
4
75.89
TAB (Ours)
Surrogate Gradient
ResNet-19
6/4/3
76.82
76.81 76.31
NeuNorm ( (Wu et al., 2019)
Surrogate Gradient
7-layer CNN
40
60.50
BNTT (Kim & Panda, 2021)
Surrogate Gradient
7-layer CNN
20
63.2
DVS- OVS-CIFARIO
1dBN (Zheng al., 2021)
Surrogate Gradient
ResNet-19
10
67.8
TEBN (Duan al., 2022)
Surrogate Gradient
7-layer CNN
10
75.10
TAB (Ours)
Surrogate Gradient
7 7-1ayer CNN
4
76.7
SlipReLU (Jiang al., 2023)
ANN-to-SNN
ResNet-34
32
66.61
IdBN (Zheng al., 2021)
Surrogate Gradient
ResNet-34
63.72
ImageNet
TEBN (Duan et al., 2022)
Surrogate Gradient
ResNet-34
4
64.29
ResNet-34
4
67.78
TAB (Ours)
Surrogate Gradient
ResNet-34
2
65.94
datasets as well. For instance, on the DVS-CIFARI dataset, our TAB method achieves 1.6% better performance (76.7% v.s. 75.10% while utilizing fewer time-steps v.s. 10) than TEBN. Similarly, on CIFAR-100, our method exhibits 0.55% increase in accuracy (76.31% v.s. 75.86%) compared to TEBN when both use 2 time-steps. All the accuracy values for other methods reported in the table are drawn from the existing literature.
4.2 COMPARISON ON LARGE-SCALE IMAGENET DATASET
In this section, we investigate the effectiveness of our TAB method on the ImageNet dataset, renowned for its extensive collection of more than 1.25 million training images and 50, 000 test images (Deng et al., 2009). The training set of ImageNet offers 1, 280 training samples for each label, and we apply standard preprocessing and augmentation techniques (He et al., 2016) to the training data. Test data is centered and cropped to dimensions of 224 x 224. The evaluation employs the ResNet-34 architecture, a widely recognized model. The network is trained using the AdamW optimizer with an initial learning rate of 0.00002 and weight decay of 0.02. Training occurs on an NVIDIA RTX A6000 with 4 GPUs, each handling batch size of 24. To ensure unbiased statisties, we follow Zheng et al. (2021) and synchronize batch mean and variance across devices.
The results, presented in Tables Table and Table $4, reveal the efficacy of our TAB method. Notably, even with a modest training duration of 80 epochs for T = 4, the TAB method exhibits a 3. 3.29% improvement on ResNet-34 over TEBN at (TAB with 67. 78% vs. TEBN 64. 29%). Impressively, with only time-steps (T' 2), our TAB method achieves an accuracy of 65.94% on ImageNet, showcasing its promising performance.
4.3 COMPARISON WITH THE STATE-OF- THE-ART APPROACHES
In this section, we present comprehensive comparison of our TAB method with other ofate-of-the-an learning methods for SNNs using CIFAR-10 the benchmark dataset, as illustrated in Table 2.
8

---

Published as a conference paper at ICLR 2024
On the VGG-11 architecture, our TAB method achieves impressive accuracy of 94.73% while utilizing 4 time-steps, outperforming all the ANN-to to-SNN conversion and hybrid training methods that require more stmes Besides, we follow TEBN (Duan et al., 2022) and adopt the cutout augmentation (DeVries & Taylor, 2017) on static datasets denoted by **" in the table. Compared to other surrogate gradient methods, TAB method consistently performs better. On ResNet- our TAB method achieves an accuracy of 96.09% with 6 time-steps, which is better than Dspike (94.25%) TET (94.5%). TEBN (95.6%) while using the same number of time-steps. Even when using only 2 time-steps T 2, our TAB method on ResNet-19 achieves a higher accuracy than TEBN (Duan et al., 2022) which utilizes 6 time-steps. We contribute this elevated performance the better representation capability of TAB, achieved by its alignment with the neuron dynamies, thereby bridging the gap between the discrete LIF model and the underlying neuron dynamics and making the two closer. For clarity, all reported accuracy values for other methods in the tables are sourced from the literature. Further experimental results on CIFAR- 100 and WVS-CIFARIO datascts are detailed in Table S3 from Append. E. For comprchensive comparison with state-of-the-an (SOTA) methods on ImageNet, please consult Table S4 provided in Append. E.5.
Table 2: Comparison between the proposed TAB and other state-of-the-ar approaches CIFAR-10
Model
Method
Architecture
Time-steps
Accuracy (%)
RMP (Han al., 2020)
ANN-to-SNN
ResNet-20
2048
91.36
RTS (Deng Gu, 2021)
ANN-1o-SNN
ResNet-20
128
93.56
QCFS (Bu et al., 2022)
ANN-to-SNN
ResNet-20
16
91.62
PTL (Wu et al., 2021b)
ANN-to-SNN
VGG-11
16
91.24
HC (Rathi et al., 2020)
Hybrid Training
VGG-11
2500
92.94
TC (Zhou et al., 2021)
Time-based Gradient
VGG-16
92.68
TSSL-BP (Zhang & Li, 2020)
Time-based Gradient
7 7-yyer CNN
91.41
Dspike (Li et al., 2021b)
Surrogate Gradient
ResNet-18 ResNet-199
6/4/2
94.25
19.666 /9.39.11 13
TET (Deng et al., 2022)
Surrogate Gradient
6 6/4/2
94.50
94.44 / 16.16
TEBN (Duan et al., 2022)
Surrogate Gradient
VGG-11
93.96
TEBN (Duan et al., 2022)
Surrogate Gradient
ResNet-19
6/4/2
95.60 / 995.58 /95.45
VGG-11
94.73
TAB (Ours)
Surrogate Gradient
ResNet-19
6/4/2
96.09 / 995.94/ 95.62
5 CONCLUSION
Directly training SNNs is extremely challenging, even when adopting BN techniques enable more stable training. The presence of the Temporal Covariate Shift (TCS) phenomenon, coupled with the intrinsic temporal dependency of neuron dynamics, further compounds these challenges for directly training SNNs. To tackle this, we have introduced TAB (Temporal Accumulated Batch Normalization), a novel SNN batch normalization approach. TAB closely aligns with the neuron dynamics, normalizing data using temporal accumulated statistics, effectively capturing historical temporal dependencies similar to that of the accumulation process of the membrane potential in the LIF neuron model. Neuron dynamics refer to the changes in the membrane potential of neuron time as it integrates input signals and generates spikes. The alignment with the neuron dynamics means that the TAB method tailored to mimic capture the behavior of neurons as closely as possible. It aims to normalize the data in a manner that is coherent with the temporal dependencies and accumulation of information that occur within neurons they process input signals. This aligmment ensures that TAB's ' normalization process corresponds well with the way neurons naturally operate in SNNs, thereby leading to improved training and performance by addressing the temporal covariate shift problem.
ACKNOWLEDGEMENTS
This work is part of the research project ("Energy-basee probing for Spiking Neural Networks". Contract No. TII/ARR/2003//0221 collaboration between Technology Innovation Institute (TII, Abu Dhabi) and Mohamed bin Zayed University of Artificial Intelligence (MBZUAI, Abu Dhabi).

---

Published as a conference paper at ICLR 2024
REPRODUCIBILITY STATEMENT
The experiments and results presented in this research are reproducible, with all code, data, and detailed methodologies available in the supplementary materials. The codebase has been documented extensively, ensuring clarity and ease of implementation for future researchers, The datasets used in this study, including CIFAR-10, CIFAR-101 100, and DVS-CIFARII 10, are publicly accessible, and we provide precise instructions on data preprocessing and augmentation procedures. Additionally, the hardware and software specifications utilized for conducting experiments are thoroughly documented, enabling researchers to replicate our results under similar computational environments. We are committed to supporting the scientific community's efforts in validating and building upon our work, thus promoting transparency and trustworthiness in the field of spiking neural networks and batch normalization techniques.
ETHICS STATEMENT
This research strictly adheres to ethical standards and guidelines governing scientific inquiry. All experiments involving living subjects or animals were not a part of this study, eliminating any ethical concerns in that regard. terms of data usage, we employed publicly available datasets, ensuring no breach of privacy data protection regulations. In terms of research conduct, this study promotes openness and transparency by making all code, data, and methodologies accessible to the wider scientific community. We also acknowledge and properly cite prior work, respecting intellectual property rights and academic integrity. Furthermore, this research focuses on improving the efficiency and effectiveness spiking neural networks, which could potentially contribute to more energy- efficient AI applications. We are committed to upholding the highest ethical standards in research and encourage responsible and transparent scientific practices within the field.
REFERENCES
Filipp Akopyan, Jun Sawada, Andrew Cassidy, Rodrigo Alvarez-Icaza, John Arthur, Paul Merolla, Nabil Imam, Yutaka Nakamura, Pallab Datta, Gi-Joon Nam, et al. Truenorth: Design and tool flow of a mw million neuron programmable neurosynaptic chip. IEEE transactions on computer-aided design of integrated circuits and systems, 34(10))1537-1553 2015.
Nils Bjorek, Carla P Gomes, Bart Selman, and Kilian Q Weinberger. Understanding batch normaliza- tion. Advances in Neural Information Processing Systems, 31, 2018.
Léon Bottou. Stochastic gradient descent tricks. Neural Networks: Tricks of the Trade, pp- 421-436. Springer, 2012.
Tong Bu, Wei Fang. Jianhao Ding, PengLin Dai, Zhaofei Yu, and Tiejun Huang. Optimal ANN-SNN conversion for high-accuracy and ultra-low-latency spiking neural networks. In International Conference on Learning Representations, 2022.
Sayeed Shafayet Chowdhury, Nitin Rathi, and Kaushik Roy. Towards ultra low latency spiking neural networks for vision and sequential tasks using temporal pruning. In Computer Vision-ECCS 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XI, pp. 709-726. Springer, 2022.
Mike Davies, Narayan Srinivasa, Tsung-Han Lin, Gautham Chinya, Yongqiang Cao, Sri Harsha Choday, Georgios Dimou, Prasad Joshi, Nabil Imam, Shweta Jain, et al. Loihi: A neuromorphic manycore processor with on-chip learning. leee Micro, 38(1):82-99, 2018.
Michael V DeBole, Brian Taba, Amon Amir, Filipp Akopyan, Alexander Andreopoulos, William Risk, Jeff Kusnitz, Carlos Ortega Otero, Tapan K Nayak, Rathinakumar Appuswamy, et al. Truenorth: Accelerating from zero to million neurons in 10 years. Computer, 52(5):20-29, 2019.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Fei-Fei Li. Imagenet: A large-scale hierarchical image database. In 2009 JEEE conference on computer vision and pattern recognition, pp. 248-255. leee, 2009.
10

---

Published as a conference paper at ICLR 2024
Shikuang Deng and Shi Gu. Optimal conversion of conventional artificial neural networks to spiking neural networks. In International Conference on Learning Representations, 2021.
Shikuang Deng, Yuhang Li, Shanghang Zhang, and Shi Gu. Temporal efficient training of spiking neural network via gradient re- weighting. In International Conference on Learning Representations, 2022.
Shikuang Deng, Hao Lin, Yuhang Li, and Shi Gu. Surrogate module learning: Reduce the gradient error accumulation in training spiking neural networks. 2023.
Terrance DeVries and Graham W Taylor. Improved regularization convolutional neural networks with cutout. arXiv preprint arXiv: 04552, 2017.
Peter Diehl, Daniel Neil, Jonathan Binas, Matthew Cook, Shih-Chii Liu, and Michael Pfeiffer Fast-classifying, high-accuracy spiking deep networks through weight and threshold balancing. In 2015 International Joint Conference on Neural Networks (IJCNN), pp. 1-8. icee, 2015.
Jianhao Ding, Zhaofei Yu, Yonghong Tian, and Tiejun Huang. Optimal ANN-SNN conversion for fast and accurate inference deep spiking neural networks. Proceedings of the Thirtieth International Joint Conference Artificial Intelligence, IJCAI-21, pp. 2328 2336,, 8 2021. doi: 10. /1996///jcii22221/333 URL https: http///oiioo//oo///ooo wrt. 10 24w.0..24333388 jiceeiijaaa... 2021 321. Main Track.
Jianhao Ding, Tong Bu, Zhaofei Yu, Tiejun Huang, and Jian Liu. SNN-RAT: Robustness-enhance spiking neural network through regularized adversarial training. Advances in Neural Information Processing Systems, 01-24780-24795 2022.
Chaoteng Duan, Jianhao Ding, Shiyan Chen, Zhaofei Yu, and Tiejun Huang. Temporal effective batch normalization in spiking neural networks. In Advances in Neural Information Processing Systems, 2022.
Jason K Eshraghian, Max Ward, Emre Neftci, Xinxin Wang. Gregor Lenz, Girish Dwivedi, Mo- hammed Bennamoun, Doo Seok Jeong, and Wei D Lu. Training spiking neural networks using lessons from deep learning. arXiv preprint arXiv:2 109. 1098944409 2021.
Wei Fang. Yanqi Chen, Jianhao Ding. Zhaofei Yu, Timothée Masquelier, Ding Chen, Liwei Huang, Huihui Zhou, Guoqi Li, Yonghong Tian, et al. Spikingjelly, 2020.
Wei Fang, Zhaofei Yu, Yanqi Chen, Tiejun Huang, Timothée Masquelier, and Yonghong Tian. Deep residual learning in spiking neural networks. Advances in Neural Information Processing Systems, 2021a.
Wei Fang, Zhaofei Yu, Yanqi Chen, Timothée Masquelier, Tiejun Huang, and Yonghong Tian. Incorporating learnable membrane time constant to enhance learning of spiking neural networks. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 2661-2667 2021b.
Wulfram Gerstner, Werner M Kistler, Richard Naud, and Liam Paninski. Neuronal dynamics: From single neurons networks and models of cognition. Cambridge University Press, 2014.
Yufei Guo, Yuanpei Chen, Liwen Zhang, Xiaode Liu, Yinglei Wang, Xuhui Huang, and Zhe Ma. Im- loss: information maximization loss for spiking neural networks. Advances in Neural Information Processing Systems, 35:156-166 166, 2022.
Bing Han, Gopalakrishnan Srinivasan, and Kaushik Roy. RMP-SNN: Residual membrane potential neuron for enabling deeper high-accuracy and law-natency spiking neural network. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 13557713567, 2020.
Zecheng Hao, Jianhao Ding, Tong Bu, Tiejun Huang, and Zhaofei Yu. Bridging the gap between anns and snns by calibrating offset spikes. In The Eleventh International Conference on Learning Representations, 2023.
11

---

Published as a conference paper at ICLR 2024
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. International Conference on Machine Learning, pp. 448-456. PMLR, 2015.
Haiyan Jiang, Srinivas Anumasa, Giulia De Masi, Huan Xiong, and Bin Gu. A unified optimization framework of ann-snn conversion: Towards optimal mapping from activation values to firing rates. 2023.
Youngeun Kim and Priyadarshini Panda. Revisiting batch normalization for training low- low-latency deep spiking neural networks from scratch. Frontiers neuroscience, pp. 1638, 2021.
Alex Krizhevsky, Geoffrey Hinton, al. Learning multiple layers of features from tiny images. 2009.
Hongmin Li, Hanchao Liu, Xiangyang Ji, Guoqi Li, and Luping Shi. Cifar10-DVS: event-stream dataset for object classification. Frontiers in neuroscience, 11:309, 2017.
Yuhang Li, Shikuang Deng, Xin Dong, Ruihao Gong, and Shi Gu. A free lunch from ANN: Towards efficient, accurate spiking neural networks calibration. In International Conference Machine Learning, pp. 6316-6325. PMLR, 2021a.
Yuhang Li, Yufei Guo, Shanghang Zhang, Shikuang Deng, Yongqing Hai, and Shi Gu. Differentiable spike: Rethinking gradient-descens for training spiking neural networks. Advances in Neural Information Processing Systems, 34:23426-23439, 2021b.
Xiangru Lian and Liu. Revisit batch normalization: New understanding and refinement via composition optimization. In International Conference Artificial Intelligence and Statistics, 2019.
Ilya Loshchilov and Frank Hutter. SGDR: Stochastic gradient descent with warm restarts, In International Conference on Learning Representations, 2017. URL net fert/Eormm2aa iid--dd--i waw.imiidddkss9soomo SSxxX
Changze Lv, Jianhan Xu, and Xiaoqing Zheng. Spiking convolutional neural networks for text classification. In The Eleventh International Conference Learning Reprexentations, 2023.
Barbara D MacCluer, Paul S Bourdon, and Thomas L Kriete. Differential Equations: Techniques, Theory, and Applications. American Mathematical Soc., 2019.
Qingyan Meng. Mingqing Xiao, Shen Yan, Yisen Wang, Zhouchen Lin. and Zhi- Quan Luo. Training nigh-performance low-latency spiking neural networks by differentiation on spike representation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp- 22444-12453 2022.
Qingyan Meng, Mingqing Xiao, Shen Yan, Yisen Wang, Zhouchen Lin, and Zhi- Quan Luo. Towards memory-and time-efficient backpropagation for training spiking neural networks. arXiv preprint arXiv:2302. 143/1, 2023.
Emre O Neftci, Hesham Mostafa, and Friedemann Zenke. Surrogate gradient learning in spiking neural networks:
Bringing the power of gradient- optimization spiking neural networks. IEEE Signal Processing Magazine, 36(6):51-633 2019.
Nitin Rathi, Gopalakrishnan Srinivasan, Priyadarshini Panda, and Kaushik Roy. Enabling deep spiking neural networks with hybrid conversion and spike timing dependent backpropagation. In International Conference on Learning Representations, 2020.
Kaushik Roy, Akhilesh Jaiswal, and Priyadarshini Panda. Towards spike- machine intelligence with neuromorphic computing. Nature, 575(7784):607-66) 2019.
Ali Samadzadeh, Fatemeh Sadat Tabatabaei Far, Ali Javadi, Ahmad Nickabadi, and Morteza Haghir Chehreghani. Convolutional spiking neural networks for Processing Letters, pp. 1-17, 2023.
feature extraction. Neural
12

---

Published as a conference paper at ICLR 2024
Shibani Santurkar, Dimitris Tsipras, Andrew Ilyas, and Aleksander Mądry. How does batch nor- malization help optimization? In Proceedings of the 32nd International Conference on Neural Information Processing Systems, volume 31, pp. 1888-2498,, 2018
Abhronil Sengupta, Yuting Ye, Robert Wang, Chiao Liu, and Kaushik Roy. Going deeper in spiking neural networks: VGG and residual architectures. Frontiers in neuroscience, 13:95, 2019.
Nitish Srívastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: simple way to prevent neural networks from overfitting. The Journal of Machine Learning Research, 15(1):1929-1958 2014.
Hao Wu, Yueyi Zhang, Wenming Weng, Yongting Zhang, Zhiwei Xiong. Zheng-Jun Zha, Xiaoyan Sun, and Feng Wu. Training spiking neural networks with accumulated spiking flow. In Proceedings of the AAAI Conference Artificial Intelligence, volume 35, pp. 10320-10328, 2021a.
Jibin Wu, Chenglin Xu, Xiao Han, Daquan Zhou, Malu Zhang, Haizhou Li, and Kay Chen Tan. Progressive tandem learning for pattern recognition with deep spiking neural networks. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(11):7824-784 2021b.
Yujie Wu, Lei Deng, Guoqi Li, Jun Zhu, and Luping Shi. Spatio-temporal backpropagation for training high-performance spiking neural networks. Frontiers in neuroscience, 12:331, 2018.
Yujie Wu, Lei Deng, Guoqi Li. Jun Zhu, Yuan Xie, and Luping Shi. Direct training for spiking neural networks: Faster, larger, better. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pp.
1311-1318, 2019.
Yuxin Wu and Kaiming He. Group normalization. Proceedings of the European Conference on Computer Vision (ECCV), pp. 3-19, 2018.
Mingqing Xiao, Qingyan Meng, Zongpeng Zhang, Yisen Wang, and Zhouchen Lin. Training feedback spiking neural networks by implicit differentiation on the equilibrium state. Advances in Neural Information Processing Systems, 34: 34:14516-14577 2021.
Mingqing Xiao, Qingyan Meng, Zongpeng Zhang, Di He, and Zhouchen Lin. Online training through time for spiking neural networks. Advances in Neural Information Processing Systems, 35:20717-20730, 2022.
Ruibin Xiong, Yunchang Yang, Di He, Kai Zheng, Shuxin Zheng, Chen Xing, Huishuai Zhang, Yanyan Lan, Liwei Wang, and Tieyan Liu. On layer normalization in the transformer architecture. In International Conference on Machine Learning, pp. 10524-10533, PMLR, 2020.
Wenrui Zhang and Peng Li. Temporal spike sequence learning via backpropagation for deep spiking neural networks. Advances in Neural Information Processing Systems, 33: 13:12022-1222 12033, 2020.
Hanle Zheng, Yujie Wu, Lei Deng, Yifan Hu, and Guoqi Li. Going deeper with directly- larger spiking neural networks. In Proceedings of the AAAI conference on artificial intelligence, volume 35, pp. 11062-11070, 2021.
Shibo Zhou, Xiaohua Li, Ying Chen, Sanjeev T Chandrasekaran, and Arindam Sanyal. Temporal- coded deep spiking neural network with easy training and robust performance. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pp. 11143-11151, 2021.
13

---

Published as a conference paper at ICLR 2024
Appendices
SUPPLEMENTARY MATERIAL FOR "TAB: TEMPORAL ACCUMULATED BATCH NORMALIZATION IN SPIKING NEURAL NETWORKS"
NOTATIONS
Compared to ANNs, SNNs utilize binary activations, or spikes, in each layer. To compensate for this limited representation capacity of the binary spiking activation, the time dimension, also known as latency T, is introduced in SNNs. For the forward pass in SNNs, inputs presented as streams of events and the forward pass is repeated for 7 time-steps to produce the final result.
To make it clear, refer Eq. (1) as the LIF neuron dynamics which described as differential equations, and refer to Eq. (2) as the discrete LIF neuron model.
Let æ=]]] be the spike representation of the traditional features time-step denoted æ[t] SpikeRep(æ,, t). Here, let (æ, y) be samples from a traditional dataset D with æ representing the image and y the label. Note that x: and [1] share the same dimensionality, w.l.o.g., let E xx have C' as the channel dimension, (H, W) as the spatial dimension, so æ=4 E RCXHXW Let Dr represent the set of spiking representations time-step t, which can be expressed as:
D {a=(tt) v). with 2 [t] Spike and V (æ, D)
Due to the temporal processing nature of SNNs, the training dataset D consists of sets of spiking representations from all time-steps, which satisfies D = Dj U... U Dr, and any two are disjoint with others, D₂ D 0 if s.
Algorithm 1: The Algorithm of TAB Method
Data: The mini-batch data at current time-step t, {B.} = {a, (1)
Input: Parameters to he learned: w(f), >[t] ([1), where weight w[t] > 0.
Output: ft(t) (A:((z:(f)
for 1, T do
Calculate Mean/Variance for mini-batch B, at current time-step n(t] = Σ. =;(1], o3[(] = (ar() - -x(t)?
Calculate Temporal Accumulated Mcan/Variance
1 Pia 11:(t11) + & fnlt), of L- oi [f)
L
The TAB Method
x|{{ + 7/4 7:0 2,((t]
Mia + B1q, for i = , Vo +
[][{] = [1] [t] TAB(a,(t))
w|t] > 0.
Outputs: @[1] = [+]]...[[}}
A BN OF CONVOLUTIONAL LAYERS
For convolutional layers, we additionally want the normalization obey the convolutional property. that is, different elements the same feature map (or the same channel), at different locations, are normalized in the same way. Specifically, let x [t] denote the input variable to the layer I at time-step t, where x' [t] is a tensor with four dimensions N x x H W. Denote x a'[t] E R Xxxxxxx XXXXXXXX HxW the corresponding batch input data the layer at time-step t. We let B, be the set of all values in feature map across both the elements of a mini-bated and spatial locations. So for a mini-batch of size N and feature maps of size H W. the effective mini-batch of size b B, NHW.
14

---

Published as a conference paper at ICLR 2024
The mini- data can be written as æ' (x][t].- denotes the input to the layer at time-step t for e-th channel (or c-th feature map). Since the normalization is applied independently to each dimension over the samples in a mani-batch x [t] let us focus on the mini-batch data of a particular dimension a and omit e for clarity. Consider a spatial-temporal mini-batch B2 {ar [t] #f...b(f] of size b at each time-step t.
,t))),,,,,,,.)),, where each x'[t] R N
B DETAILS OF TAB SPIKING NEURAL NETWORKS
Our proposed TAB normalizes data along the channel dimension before passing it through the activation function, then this normalized data is fed as an input to the activation function. Let [t] denote the input variable layer / time-step t. Without loss of generality, we assume that the layer input x' has d-dimensions, which can be written x x'(t] (x'(')t)... xd tt). Here x (())((t represents the k-th dimensional input to the (-th layer at time-step t. Since the normalization is performed independently each dimension x! [t), we can focus specific dimension denoted as 2 (t] and omit the ((k) notation for clarity.
We investigate Spiking Convolutional Neural Networks (SCNNs), which are Spiking Neural Networks that incorporate convolutional layers and can process both spatial and temporal information. Our main focus is primarily on applying Batch Normalization for Spiking Convolutional Neural Networks. In SNNs, there is an additional temporal dimension, indexed by time-step [t]. Thus, at the current time-step [t]. all the accumulated spatial-temporal mini-batches, denoted as B = {Ba)'=1 are available, where the temporal information is accumulated over i time-steps. Here, B, (. represents a spatial-temporal mini-batch at time-step with batch index channel index c, height index h, and width index w. For notation simplification. we can put all the batch dimension N and all spatial locations H, W into one dimension with b NWW and indexed by i, and we also omit the channel index for a given channel. In particular, let x; [t] represent the i-th sample at time-step t in the pratial-tempore mini-batch B,.
Let o' [t] denote the spiking outputs of all neurons in 1-th layer at time-step t, and W' denote the synaptic weights between layer 1 and layer I. The pre-synaptic inputs to the LIF neurons, denoted as x [t] can be expressed as
x [t] ' o'
In Spiking Convolutional Neural Networks, both the spikes and the pre-synaptic inputs at time- step are 3-dimensional tensors. Specifically, we refer to o d' = 1-1 of E NxCx
the spikes tensor and ar(( == s.ll R-NxCxHxW as the pre-synaptic inputs tensor in mini-batch with batch size N. Here, the indices b, h w provide a precise description of the dimensions, representing the batch axis N, the channel axis C, and the two spatial dimensions and
W, respectively. In the case of input images to the neural network, the channels correspond the RGB channels.
The spatialltemporal mini-batch B, at time-step I, with batch size N. is defined the set of all values in feature map across both the elements of mini-batch and spatial locations. For mini-batch of size N and feature maps size H W. the effective mini-batch of size is b b= B, |= N NWW. We learn pair of parameters, (k) and gik), per feature map, where k 1, ... C, rather than per activation.
TAB with layer index. The TAB Layer normalizes the pre-synaptic inputs ('[(] in the I-th layer of the spiking neuron networks at time-step f, and it can be formulated as
x [t] W o''o' o'
u' [t] Xu' [t 1](1 o'it 1)) vl
where
y' = TAB (z [t)) = w'(
w'I x
a it >' x! [t] ph. + (
VVL. +e
Here w'(t) and o' [t] represent the membrane potential and binary spike outputs, respectively, of all neurons in 1-th layer at time-step 1. The variable x represents the input variable layer 1, and
15

---

Published as a conference paper at ICLR 2024
w denotes the synaptic weights between layer / - and layer 1. Each TAB Layer incorporates
time-dependent learnable parameters, namely w [t), r(t) 3 it. for each layer. These parameters
C'-channel tensors defined as ('[t] = (w1 [t). ... [t), r' (fi ... (c(t)), (t
(8 [t] ... (t)). The learnable parameters w ['t,
[1] it,
(t = 1, ,, 1... L) are
trained during the training process.
Here the accumulated mean pi and the accumulated variance V (oxx)) are computed as follows,
p Σ
[s]
Vl. 3 Σ (o)(s))
where
11 1
1 b [c][[]]
= (d"
11 (s)2
If we expand the channel dimension, we will have the following. Here the accumulated mean wia = (wi. -c,,.)) where each p has the same definition on each channel, and the accumulated variance Vi (Vi, C,1a) is defined analogously. Specifically,
1 M/.- t Σ p'(s))
V ( (o'ls))
where notations [l[s] = (M) . sllss) and (o'(s))² = ((o] ... (((()))).....((((s))))) and Spe- cially,
pl[ss = NHW
Σ b,h.w
(o'l))) alchhal NHW
Σ (a/.c...ls) (!(s))? b,
TAB without layer index. As here we normalize all the neurons in I-th layer at time-step t, we can omit the layer index In the batch normalization process, we will independently normalize each scalar feature along the channel dimension, by making it have zero mean and the variance of over mini-batch. In our TAB method, the pre-synaptic inputs x (t] will be normalized and both the input and output of a TAB layer are four-dimensional tensors. Our proposed TAB applies the same normalization for all activations given channel (along the channel dimension).
yo + woll nal4) Db.c, [t] Mo.lit + Bcll
(S.1)
VoL.aa +
The details of the TAB
Yb [t] TAB (o, woltt w.[t] > 0,
(S.2)
where Bs = (o(t] wblchh.tt - Me,lit + Bc(t),
(S.3)
Vo2 +c
Me,ld
Σ Hc[s], Mc[s]= = NHW Σ 1b.c
(S.4)
t ==1
buhac
1 t
22
(o.(1)) (ar()))² NHW
Σ (0 1e, (S.5)
b,b,
Here, TAB subtracts the accumulated mean activation 1e, from all input activations in channel c, where Bi, contains all activations in channel over the accumulated time-step from 1 to f across all features in the entire mini-batch and all spatial h, w locations. Subsequently, TAB divides the centered activation by the accumulated standard deviation o2 (plus for numerical stability) which is calculated analogously. Normalization is followed by a channel- wise affine transformation and scaling parametrized through 7e(4] 3e we (t. > 0) at different time-step f, which are
16

---

Published as a conference paper at ICLR 2024
learned during training. During testing, running averages of the accumulated mean and accumulated variances are used. Refer to algorithm for the algorithm of our TAB method.
The reason for considering all the accumulated spatial-tempora mini-bateh over expanding window [1, t is enabled by the nature of the spiking neurons to process both spatial and temporal information. In SNNs, the synaptic currents are fed into spiking neurons sequentially, the membrane potential u, (t) accumulates over time t. The temporal information in SNNs is accumulated over multiple time-steps, allowing the neurons consider the input data in context of the past time-steps well as the current time-step. Therefore, we expect it to be beneficial for batch normalization to consider the temporal accumulated information over an expanding window [1, t).
Table S1: Statistics and parameters of pre-synaptic inputs with different BN methods.
1=1 1
1
1=T
o11),2211) >/1
a(t), (
[TT),
BNTT (Kim & Panda, 2021)
7/4
7[1]
Protal P1.T, Fiotal Fi.
tdBN (Zheng et al., 2021)
7
Piotal = PI.,, Pinsa oi.r
TEBN (Duan et al., 2022)
8[1),8[1]
1(1),
4[77,
Me. of.
TAB (ours)
[1]x[1]
(T
C DETAILED LEARNING RULES OF TAB IN SNNS
When applying TAB in SNNs, we derive the detailed learning rules by computing the gradients of the loss function with respect to the weights aW and with respect to the parameter dw (t). Following the previous work (Kim & Panda, 2021; Zheng et al., 2021; Duan et al., 2022), we compute the gradient by unfolding the network along the time dimension and get,
ac
ac 00
OC
@u![t]
0o @u![t]
Ouli 1]
Ou!
0C
dc
OC dā.
- 0, (-1
M1]
Twe 19
dx:- [t]
dut Ox!
where w/[4] and o|[4] denote the membrane potential and output spikes of the i-th neuron at time-step t in layer 1, a! [t] and [t] are the input and output of the TAB layer. As the derivative of spikes with
respect to the membrane potential [o is non-differentiable due to the Heaviside step function, we
(u.(] use the surrogate gradient methods (Neftci et al., 2019; Wu et al., 2018; 8: Eshraghian et al., 2021) to address this problem.
The weight parameter w allows the TAB layer to emphasize the temporal dynamics along the temporal dimension. The gradient of the loss function L with respect to the weight parameter Ow'(l,
dc Iw
Σ
OC Dulit it]
Σ
ac
7/14
x
whe + 3:4
Du
aw'
Dul
V(o +
where ur and (ai denote the mean and variance of the i-th neuron from samples from time-step 1 to I, A and Bi(l] are the i-th parameters of TAB at time-step l.
D THEORETICAL POOF
In this section, we provide the theoretical proof for Lemma 1 and Lemma in the main paper. We first give short introduction the first-order differential equations and the Initial Value Problem
17

---

Published as a conference paper at ICLR 2024
(IVP). Then derive the analytical closed-form solution of the first-order differential equation underlying the LIF neuron dynamics and give detailed proof for the main lemma.
D.1 FIRST-ORDER DIFFERENTIAL EQUATIONS
We start by considering equations in which only the first derivative of the function appears. Definition 1 (First-Order Differential Equation). A first-order differential equation is an equation of the form F(t, y, y²) 0. A solution of first-order differential equation is function f(t) that makes Ft,, [t), f( (t)) = O for every value of t.
The term "first-order" means that the first derivative of y appears, but higher order derivatives do. Note that the derivative y' will explicitly appear in the equation.
Definition 2 (First-Ore -rderr Initial Value Problem). first-order initial value problem a system of equations of the form [t, y/) O, y(to) yoo Here to is a fixed time and yo is number. A solution of an initial value problem a solution f(t) of the differential equation that also satisfies the initial condition f(4) yu- D
The general first-order equation is rather too general. We focus on specific kinds of first-order differential equations. For example, equations of the form y' = o(t, y) where & is function of the two variables and y. Under reasonable conditions on ø, such an equation has a solution and the corresponding initial value problem has a unique solution.
Definition 3 (First-Order Linear Differential Equation). A first-order differential equation is linear when it can be written as
y' + P(tyy Q(t)
(S.6)
Where P(t) and Qt)) are functions of f. 0
We now need to derive the general solution of a linear first-order differential equation. We start by revisiting two lemmas of first-order differential equation (MacCluer al., 2019).
Lemma S.1. (Homogeneous Solution (MacCluer et al., 2019).) If P(t) is continuous on (a, b), then the general solution of the homogeneous equation
y' + P(tyy =0,
on (a, b) is
ce P(s)ds
Proof. Now we show some key steps to derive the solution,
1/ + P()) 0
dy + P()y = 0
complementary equation
di
dy + P(t)dt =0
/ dy + / P())dt
= - -s P())de +
In(y)
ly| :-- P(s)da
étt if
(a,
= _ if y < on (a. b)
Denote the integrating factor A(t) y1 (t) exp (- P(s)ds), then the solution Eq. (S.6) is
(t) (ʃ Q(s) ds + +c) = A(1) ( Q(s) ds +
(s)
A(s)
18

---

Published as a conference paper at ICLR 2024
Lemma S.2. (Solution to the Initial Value Problem (MacChuer al., 2019).) to is an arbitrary point in (a, b) and yo an arbitrary real number, then the initial value problem
y' +
y(to) yo-
(S.7)
has the unique solution
Q(s)
y (t)
30
ds
91 (to)
to 8/
D.2 CLOSED-FORM SOLUTION OF THE LIF DYNAMICS AS FIRST-ORDER LINEAR DIFFERENTIAL EQUATION
For the Analytical Closed-form Solution for the IVP the LIF Dynamics, we have the following lemma in the main paper.
Lemma 3. The analytical closed-form solution for the first-order IVP (Initial Value Problem) of the LIF dynamics ODE as follows (Gerstner al., 2014),
U()) = exp (-+) (1. ( (-) ds + U0)
exxp
(S.8)
Proof. If the injected current input (t) is a function depending on t, the LIF neuron dynamics becomes
dU(() U(1) + RI(L.
d
Write into standard first-order differential equation
y/ P(ty Q(t).
Consider the LIF neuron dynamics, and follow Lemma S.1
dU(l) + (1) = T(1)
R
dt
1 T
R
P(t) Q(t) = (()
A(I) exp (-) P(()d)) = exp (-+) (A(0) - 1)
/ Q(s)
U(t) = A(t)
ds +
A(s)
( -
R (ss) A(s)
1
U(I) = A(I)
ds +c
U(t) = exp (-+) (ʃ I (s)exp
(2) dol
If the initial condition ((0)) Up, then closed-form solution is
e
Up A(O]
Un. and by following Lemma S.2, the final
Utt) exp (-)
(
R (s)exp
( ds + Uo
Note: If the neuron starts at some value Up with no further current input, i.c., I(t) 0, the solution of the linear differential equation above is as follows
(() = Uoexp
19

---

Published as a conference paper at ICLR 2024
In the absence of imput current, the membrane potential will start at Up and exponentially decay with a time constant т. Therefore, we can determine the membrane potential ratio, often referred to as the
leak factor. denoted by A, as A = U(++ Uoexp( (12) +)
exp (-
U(E)
Unexp
This relationship enables to formulate the discretization scheme U (U This provides insights into the behavior of the membrane potential in the absence of input and establishes the discretization principle used for LIF modeling.
Let's s consider a more practical case for the input current (t), where [(t) is a function of t, not zero other constant values.
Lemma 4. Through applying integration by parts, we derive another equivalent form of the closed- form solution for the LIF dynamics, denoted as:
exponential decay term
input current model
RI(t) -S Rexp
(t) = (Uo Rlo)exp
dl(s)
(S.9)
commonly considered discrete LIP model
absent discrete model
With the application of the Riemann-Stielliee integral, the discretization version of the closed-form solution is represented as:
(U6 (o--8)oxxx(-5)
U14 = AU +
X
Σ 9
(S.10)
TAR method
For the term So R (s))ep (+) ds, we can make some simplifications using rigorous mathe- matical derivation, and have
Proof.
+
R
(())xxp
8
ds
RT (dxx)
(-)
7 16 1- Rexp (-) dI(s)
(8)
RI(s)exp
R1( /-¹ Rexp
[I(()exp
di(s).
Then, have
(-1) (J/ T(s)exp (-) ds + Un
U(1) exp
U(() exp (-+) [RI())eX
( -RI(O) S: Rexp
( dI(s) Un!
(-4) exp
U(t) RT(() (U, RT(O))exp
Rexp
dT(s)
1' Rexp
8 dT(ss)
- U(l) (Un - (O((0))
+ (R((I)
8 dl(s)
U(t) (Uo (xx))) + RI(()
Rexp
comsidered communlyy discrete model
Finally, we have derived the closed-form solution of the LIF model,
인 +
I Rexp
$ dl(s)
U(t) = (Uo - (i(O))expp
RI(t)
(S.11)
injected input currents
exponential docay
discrete LIF model
20

---

Published as a conference paper at ICLR 2024
Specifically.
exponential docay term
input current model
(tt) (Ug RIo) exxp
RI(t)
Rexp
dI
(S.12)
commonly considered in the discrete LIF model
absent in the discrete LIF model
As the third term is bit complicated with an integral time, cannot drop it without any declaration for just simplification. To understand more about the third term, we consider applying integration by parts, and derive discrete version of the closed-form solution it can be used in spiking neural network.
D.3 DISCRETIZING THE CONTINUOUS CLOSED-FORM SOLUTION OF THE LIF MODEL
By the definition of the Riemann Stieltjes integral, we can discretize the continuous integral by replacing the integral with Riemann sum.
Riemann-Stieltjer
Denote partition of an interval [0, t] with finite sequence of numbers of the form 0 = 8o 81 < .... < 8, <..... < 8n = t.
The Riemann sum of the function with respect to the tagged partition Su, s Sn is
Rexp
dI(s)
7
n-1
8: - )
(R[ - (s:))) ( denote X(ss] = RT(s)))
exp
n-1
8, -
* (X[s:ta] - -(ssi)
exp
1=0
80 -
91 91--t
exp
(X[s1) - X( + exp
(X[s2) ((ssi)
8n-1 -1
+ .... exp
(X
X (8n-11)
80
50 -
81 1
(Sn-8 -t x [sn]
exp
X [(so] + exp
exp
X[s1] + eexp
n
gi ([s]]
where S: exp
8i-1 /
exp
8t
-,,..,, 1, and gi for
7
7
5o
Sn
9o exp
0, In exp
T
T
Note that 9o > 0 and 9.. = 1, weights gi can be positive or negative for i = 1,... n- 1. It makes no difference for positive or negative weights, the signs can be absorbed into the learnable parameters of the TAB method.
Finally, from the continuous version of closed-form solution of the LIF model,
(tt) (Uo (x) (-+)
RI(() S Rexp dl dI(s).
we have the discrete version of the closed-form solution of the LIF model,
(-+) + RI(d] Σ X[s]]
Uit = (Uo - cxp
1=0
21

---

Published as a conference paper at ICLR 2024
In summary, we have
U(1) (Un (x) (-1) + RI((1) - S: Rexp (5-+) dT(ss)
(S.13)
Ult] AUit 1] RI[t]
Σ [1:
(S.14)
f=0
U ut - 1] + X
9: X:X((s)
(S.15)
==
Then we have the discrete version of the closed-form solution of the LIF model,
(Uo (Uo-Rlo)exp(-3
wolt=
12
U
AU
X
9:X
(S.16)
TAB method
In this formulation, the first exponential decay term, AU 1]. captures the temporal dependency of the membrane potential from the preceding time-step. The second term, a simple current input model, R/[t] W Ott), incorporates spikes from the pre-connected neurons at the current time-step [4. Significantly, the third term, representing the temporal accumulated input across all previous time-steps through weighted sum of the input currents X [ss]] with associated weights gi, introduces novel concept. This accumulation mechanism of the inputs is foundational component of the TAB method, providing a link that connects the TAB method and the neural dynamics.
Comparing the commonly used discrete LIF model with the discrete closed-form solution, we observe the absence of the accumulation term in the former. Our TAB method reintroduces this accumulation term in the normalization procedure. In this paper, we still employ the commonly used discrete LIF neuron model where the third term Ei 9; [s,] is simplified/omitted, but in the TAB method, we consider this term back during the normalization step.
The TAB method normalizes the data using temporal accumulated information from time-step 1 to f, which takes into account historical temporal dependencies but does not look ahead to future temporal information. This adjustment also brings the discrete LIF model closer to the analytical closed-form counterpart and establishes natural connection between neuron dynamics and batch normalization.
E EXPERIMENTS DETAILS
E.
OPERATING ENVIRONMENTS
All of our models are trained on the PyTorch platform. Experiments are conducted on an NVIDIA
RTX A6000 GPU.
E.2 DATASETS
The CIFAR dataset (Krizhevsky et al., 2009) consists of 50, 000 training images and 10, 000 testing images each with the size of 32 x 32. There are 10 classes in CIFAR- 10 dataset and 100 classes in CIFAR-10 dataset.
The 10 dataset (Krizhevsky et al., 2009) consists of 60, 000 color images each with image size of 32 x 32 in 10 classes of objects such as airplanes, cars, and birds, with 6, 000 images per class. There are 50, 000 samples in the training set and 10, 000 samples in the test set.
CIFAR-10:
CIFAR-100: The CIFAR- 100 dataset (Krizhevsky et al., 2009) consists of 60, 000 32 32 color images in 100 classes with 6, 000 images per class. There are 50, 000 samples in the training set and 10, 000 samples in the test set.
DVS-CIFAR10: DVS-CIFAR10 (Li al., 2017), challenging mainstream neuromorphic data set, used in this study. AWS-CIFARIO is converted from CIFARIO. The DVS-CIFARE 10 consists of 10, 000 images with size 128 128.
22

---

Published as a conference paper at ICLR 2024
E.3 DATA PRE-PROCESSING
For CIFAR- 10 and CIFAR-I00 100 datasets, common data normalization and some data pre-processing techniques are used the experiments. For example, we resize the images in the II/CR-I 100 datasets into 32 x 32. We apply data normalization to the CIFAR-10 and CIFAR- 100 datasets. We apply random horizontal flipping and random cropping to the training images as data augmentation. We these above data pre-processing techniques the datasets in comparison experiments with other BN methods in SNNs. In addition, we add cutout operations (DeVries & Taylor, 2017) as TEBN (Duan et al., 2022) and Dspike (Li et al., 2021b) did ResNet, when comparing our TAB method with other state-of of-the-art methods.
Our DVS-CIFARIO dataset is loaded from the Spikingjelly framework (Fang al., 2020). Follow- ing Samadzadeh et al. (2023); Duan et al. (2022), we split the dataset into 9, 000 training images and 1, 000 testing images, and reduce the spatial resolution from the original 128 x 128 to 48 x 48.
E.4 NETWORK ARCHITECTURES AND TRAINING CONFIGURATIONS
In comparison experiments with existing BN methods, we reproduce the same architectures as BNTT (Kim & Panda, 2021) and TEBN (Duan et al., 2022): VGG-9 network architecture (64C3- on CIFAR-10, VGG-11 on CIFAR- 100, and 7-layer CNN (64C3-AP2-188C3-1286) -LC.CCAAPP--222226-----2222-0000
ACC--AP2-128C3- MCC-APP--BCC--18CCAA-P----CCC
on DVS-CIFARIL Following TEBN (Duan et al., 2022), we add Dropout (Srivastava et al., 2014) layers before fully-connected layers to enhance generalization, but we do not use a voting layer after the last cully-comnected layer in DVS-CIFAR10 experiments in Duan et al. (2022). We use the standard ResNet-19 19 for both CIFAR-10 10 and CIFAR-100.
The Stochastic Gradient Descent (SGD) optimizer (Bottou, 2012) is used in the experiments with momentum parameter of 0.9. We use a cosine decay scheduler (1 (Loshchilov & Hutter, 2017) 7) to adjust the learning rate with weight decay 5 10-4 for CIFAR-1O/CIFAR- trained for 200 epochs. We set the initial learning rate to = 0.01 for CIFAR-10 10 and CIFAR-100.
datasets. All models
As for the input to the first layer and the output of the last layer of the SNN, do not employ any spiking mechanism as in Li et al. (2021a); Bu et al. (2022). We use constant input when evaluating the SNNs. We directly encode the static image temporal dynamic spikes input to the first layer, which can prevent the undesired information loss introduced by the Poisson encoding. For the last layer output, we only integrate the pre-synaptic inputs and do not fire any spikes, so that the output neurons only output the temporal average of the pre-synaptic inputs of all time-steps (m1, m2, ... mc) with C as the number of classes. The output then fed into a Softmax layer to compute the cross-entropy loss with the true labels (111 12, yC).
C
e
LCE = - y log 1=1
Li=1 e
To allow for effective BP training, the triangle-shaped surrogate gradients are used
@o,[1] []
max {0, |u: [t] - -
@u,(t]
Other hyperparameters can be found in Table $2.
Table S2: Hyperparameters optimization for training
Dataset
Optimizer
Scheduler
Epochs
Learning Rate
Batch Size
CIFAR-10
SGD
CosineeAneellingL (T==20000
200
0.02
64
CIFAR- 100
SGD
CosineAAnnealing
200
0.02
64
DVS-CIFARIO
SGD
CosineAmmealnggLR
200
0.05
64
23

---

Published as a conference paper at ICLR 2024
E.5 MORE RESULTS
We compare our TAB method with other state-of-the-art approaches in SNNs, including RMP (Han al., 2020), ReL.U-Threshold-Shiff (RTS) (Deng & Gu, 2021), quantization clip-floor-shift (QCFS) (Bu et al., 2022), progressive tandem learning (PTL) (Wu et al., 2021b), Hybrid Con- version (HC) (Rathi al.. 2020). Temporal-Coded method (TC) (Zhou et al., 2021), Temporal Spike Sequence Learning Back-Propp Proca-Prppgttion (TSSL-BP) (Zhang & Li, 2020), Differentiable Spike (Dspike) (Li et al., 2021b), Temporal Efficient Training (TET) (Deng et al., 2022), TEBN (Duan et al., 2022). We compare our method with other -tate-of-the-art learning methods for SNNs on CIFAR-100, and DVS-CIFARIO datasets and report the results in Table S3. Results on ImageNet dataset are reported in Table S4, with other sate-of-the-ane learning methods.
Table S3: Comparison between the proposed TAB and other state-of-the-ee approaches SNNs.
Dataset
Model
Method
Architecture
Time-steps
Accuracy (%)
RMP (Han et al., 2020)
ANN-to-SNI
ResNet-20
2048
67.82
RTS (Deng & Gu, 2021)
ANN-to-SNN
ResNet-20
512
72.34
QCFS (Bu al., 2022)
ANN-to-SNN
ResNet-20
128
70.55
HC (Rathi al., 2020)
Hybrid Training
VGG-11
2500
70.94
CIFAR-100
Dspike (Li et al., 2021b)
Surrogate Gradient
ResNet-18
6 6/4/2 2
74.24 / 73.35 71.68
TET (Deng et al., 2022)
Surrogate Gradient
ResNet-19
6/4/2
74.72
74.47 72.87
TEBN (Duan et al., 2022)
Surrogate Gradient
VGG-11
74.37
TEBN (Duan et al., 2022)
Surrogate Gradient
ResNet-19
6 6/4/2
76,41
76.13 75.86
VGG-11
4
75.89
TAB (Ours)
Surrogate Gradient
ResNet-19
6/4/2
76.82 / 76.81 / 76.31
PLIF (Fang et al., 2021b)
Surrogate Gradient
6-layer CNN
20
74.80
Dspike (Li ct al., 2021b)
Surrogate Gradient
ResNet-18 18
10
75.4
TET (Deng et al., 2022)
Surrogate Gradient
VGGSNN
10
83.17
DVS-CIFARIO
TEBN (Duan al., 2022)
Surrogate Gradient
6-layer CNN
10
80.00
TEBN (Duan et al., 2022)
Surrogate Gradient
VGGSNN
84.90
7-layer CNN
4
76.7
TAB (Ours)
Surrogate Gradient
6-layer CNN
84.21
Table S4: Comparison between the proposed TAB and other state-of-the-arr approaches on ImageNet.
Model
Method
Architecture
Time-steps
Accuracy (%)
SPIKE-NORM (Sengupta et al., 2019)
ANN-to-SNN
ResNet-34
2500
69.96
RTS (Deng & Gu, 2021)
ANN-to-SNN
VGG-16
16
55,80
QCFS (Bu al., 2022)
ANN-to-SNN
ResNet-34
16
59.35
SlipReLU (Jiang et al., 2023)
ANN-to-SNN
ResNet-34
66.61
SNNC-AP (Li et al., 2021a)
ANN-to-SNN
ResNet-34
64,54
Hybrid Conversion (Rathi et al., 2020)
Hybrid Training
ResNet-34
250
61,48
TET (Deng et al., 2022)
Surrogate Gradient
Spiking-ResNet-34
64.79
tdBN (Duan et al., 2022)
Surrogate Gradient
ResNet-34
6
63.72
TEBN (Duan et al., 2022)
Surrogate Gradient
ResNet-34
64.29
ResNet-34
67.78
TAB (Ours)
Surrogate Gradient
ResNet-34
65.94
THE OPTIMIZING OBJECTIVE FUNCTION OF BN METHODS IN SNNS
In the main paper, we have developed the TAB method to address the temporal covariate shift problem by investigating the neuron dynamics, specifically by aligning with the accumulated membrane potential. The TAB method utilizes the temporal accumulated statistics for normalization. As the accumulation process aligns naturally with the membrane potential accumulation procedure, the TAB method is conceived.
24

---

Published as a conference paper at ICLR 2024
In fact, we can take more general approach by directly exploring the additional temporal dimension, which opens up possibilities for developing different variants of BN methods for SNNs. To facilitate this adaptability and the development of range of BN variants in SNNs, introduce optimizing objective function that highlights the distinctions in defining BN methods along the time dimension. This framework empowers the creation of broader spectrum of BN methods tailored to the unique characteristics of SNNs.
We draw inspiration from BN techniques used in ANNs (loffe & Szegedy, 2015; Lian & Liu, 2019), and we establish a rigorous mathematical optimization objectíve function for BN methods in SNNs. Given the presence of an additional temporal/time dimension in SNNs,,t is imperative handle the time dimension meticulously when formulating the objective function for BN in SNNs. Refer to Sect. for notation details.
Let us define the normalization operator ( (-) as the mapping of function to a function, associated with mini-batch data samples B, an activation function Φ. and parameters W. Additionally, we introduce the function g(-) defined as g(-) (g1 (.), (n((),, 1)", where each gi(-) maps a tensor scalar value. The normalization operator fw (.) can be defined by
91 (9. W
mean (gn: B) var (gn: B)
)))
fW B, o g(()) = BB (o) -
var(gi, B)
where mean(g. B) = 1
[B] EvEB (b( and var(g, B) == mean B) (mean B))?
Note that the argument of B.. is function g(-), and SW is another function, the first argument of mean B) and var (gi, B) a function and the second augment is a set of sample from mini-batch B.
An m-layer spiking neural network with BN can be represented by a function
(+) FB (W, (1) = BB.0m pm J......
pB..m o .... B.or o I(-),
where I(+) is the identical mapping function [(x)) x. and W (W denote all the learnable weights in the neural network, and ø5's are the activation functions for each layer. Using similar notations, represent SNN without BN by another function fü
Fw(.) = FlW51y=1 (-)
pór 1(-), john p60-1 ....o
where f is defined by f o() (WWg()))
Besides the difference of operator functions fW... and f. B.om their optimization objective functions with BN and without BN also different.
Given the training dataset D, the optimization objective function without BN is defined by
( 1 T
min W
Σ
LCE
{>Σ
Z Fa(()))),
(S.17)
D
(M(t)
where LCE(..) is predefined Cross Cross-Entroppy loss function, D, is spiking representation at time-step is the training dataset consisting of spiking representations from time-steps. Specially
D, {xx[t) v). with xtt] = Spike t) and V (x, v) D).
At each time-step t, feed spiking data x [ into the network Fw(+), which produces output Fw (x())). The final prediction is obtained by aggregating results over the time dimension, resulting in 1 ET= Fw (x:t)).
The optimization objective function with BN is
BNN
1
Σ
LeE
FB (artt), v)
(S.18)
B
B
BE
where B, C. D, represents a subset of the dataset at time-step t, and B a partition of the whole dataset D. that is, the set of mini-batch sets, with EBeR | The mini-bater B is used to
25

---

Published as a conference paper at ICLR 2024
calculate the mini- batch mean and variance. The objective function of SNN with BN is very different from that without BN, due to the inclusion of BN.
The presence of the additional temporal dimension in SNNs gives rise to differentt BN variants, depending on the chosen approaches for defining mini- mini-batches along the time dimension. Therefore, BN methods in SNNs are more varied than in ANNs because of the additional temporal dimension in SNNS.
The mini-batch sets B in SNNs differ significantly from those in ANNs due to the presence of the additional temporal dimension, as the mini-batch set B is used to calculate the mini-batch mean and variance. Specifically, the mini-batch sets B can be viewed a partition function of the subset 1= along the temporal dimension, represented as
B = r(B1, B2. ... BE, Br).
Denote the partition B {B C B, B r(Bi, B,. Br), Br C. Dr}. Moreover, it is noteworthy that at each time-step t, the data (xlt), y) always come from B₁.
The definition and partitioning of mini-batches along the time dimension are critical factors that shape BN methods in SNNs and their associated objective functions. This leads us to the following important observation: Different partition/sampling strategies along the temporal dimension give rise to distinct BN methods in SNNs.
F.1 SPECIAL CASES OF THE OPTIMIZING OBJECTIVE FUNCTION
For the optimizing objective function, the mini mini- set B (or the sampling strategy) plays an important role, as it is used to calculate the mini-batch mean and variance. Research works on BN methods in SNNs primarily differ in their partition strategies along the temporal dimension. Notably, the proposed optimizing objective function encompasses most existing BN methods in SNNs as special cases, including our proposed TAB method. Refer to Append. F.2 for details.
In the BNTT method (Kim & Panda, 2021), data are normalized independently at each time-step where the mini-batch B is defined lemporal-indeppendnntly with data at each time-step t, i.e., B A By. In TEBN (Duan et al., 2022) and tdBN (Zheng al., 2021), data are normalized temporal- jointly across all T time-steps. The mini-bated B is defined with data across all T time-steps, i.c., B 2 BL. However, the underlying assumption of sampling strategy BAT implicitly assumes that at each time-step t, we have access to data spanning all T time-steps, including the current time- step t < T even future time-steps T. Nevertheless, this assumption may not hold true in practical situations. In contrast, our proposed TAB method takes a different partition approach. employs strategy to jointly normalize the data using temporal accumulated information from time-step 1 to f, denoted as B A Bl.t. By incorporating the temporal accumulated information, TAB ensures efficient utilization of all available information up to the current time-step f, while it does not look ahead to the future information beyond f such as T' The data normalization process in TAB follows an accumulative-joint scheme, which aligns with the analytical closed-form solution of the neuron dynamics.
F.2 SPECIAL CASES OF BN METHODS IN SNNS
For the optimizing objective function of BN methods for SNNs, the mini-batch set B (or the sampling strategy) plays an important role, as it is used to calculate the mini-batch mean and variance. The research works on BN methods in SNNs primarily differ in their approaches to sampling strategies. Notably, the proposed optimizing objective function encompasses most existing BN methods in SNNs as special cases, including our proposed TAB method.
Current research works on batch normalization methods for SNNs differ from each other from the perspective of sampling strategies.
A typical B can be defined temporal-independennt using data at each time-step f, i.e.
T
= B, B- = -B == (B.) = B2, Bz.... ,Br with B, C Dr, where (-1
B B. = ((0= y), where an Cxx A (= E =.)
26

---

Published as a conference paper at ICLR 2024
As D = 0 s, then B, n n, = if s, and 2 B As the mini-bater set B only uses Bi, we denote - B The objective function becomes,
(BN)
min T
Σ B
LCE (d[t),))
Σ
W (a[(1),
Be
T
By using this sampling strategy, Kim & Panda (2021) proposed the BNTT method, in which the data are normalized independently each time-step t.
Another common construction of is defined temporal-jointly using information from all
T time-steps, by 2 = U with B C D where
B ((E1. v), where æurr (2[11 222
xT]]
E ( (=lt). E D.)
As the mini-batch set B uses all B, at all time- steps from = to tot = T, we denote
B ByT. The objective function becomes,
min Σ
LOE (=)
Σ
T E r w (=]l))))))))))))))))))) v
B
B
Using this sampling strategy, researchers have proposed the TEBN method (Duan et al., 2022) and the tdBN method (Zheng al., 2021). In these methods, the data are temporal- jointly normalized across all time-steps T'
However, choosing 2 in this way implicitly assumes that all nodes at each time-step can access the same dataset over all time-steps, which may not be true in practice.
Another possible B can be defined jointly using the temporal accumulated information from all previous time-steps up to the current time-step t, represented as B == {Bi, B12, B with
B = B = {B1 ... U UB, c Dt}.
The objective function becomes,
T Σ T
min W
Σ BRE
LCF (a
Σ
B (x((4), y
B
From optimization perspective, this sampling strategy aligns with proposed TAB method, where TAB normalizes the data using the accumulated information over all time-steps from 1 to I, i..., B = Bi.t-
Moreover, the optimizing objective function highlights the distinctions in defining BN methods in SNNs along the time/temporal dimension, by which we can optimize and develop more BN variants along the time dimension.
G LIMITATIONS AND SOCIETAL IMPACT
While achieving better performance compared to existing methods, our method has only been tested on rate-encodingg-asee SNNs with surrogate gradients for pockkpropaggtion. Rate-encoding is popularly used in conventional SNNs where each neuron fires at most once, which is very different from timing-basse based encoding. Since timing- based encoding methods encode information in the relative timing across individual neurons without single-spike restrictions. Therefore, they allow each neuron fire multiple times and bring better asynchronism. In our future research, we intend investigate the application of our method within timing- based SNNs
terms of social impacts, our research primarily focuses on the direct training of high-performance and low-latency SNNs. which does not bring obvious negative effects. Furthermore, with the increasing adoption of SNNs due to their energy-saving consumption properties, SNNs will become indispensable in edge computing applications.
27


Reviews:
[{"review_id": "ltENPgGdPA", "reviewer": "Reviewer_ZLsc", "summary": "In Artificial Neural Networks (ANNs), Internal co-variate shift denotes alterations in the input distribution due to preceding layer updates. In Spiking Neural Networks (SNNs), the Temporal Covariate Shift (TCS) problem arises from both previous layer updates and prior time steps, extending along the temporal dimension. Within SNNs,  neurons accumulate incoming spikes in their membrane potential over time and only generate a spike when this accumulation surpasses a threshold, staying inactive otherwise within the current time-step.\nHence, the accumulation of synaptic currents in the membrane potential is affected by temporal dependencies, which could magnify the internal covariate shift as time progresses. The central concept of this study revolves around addressing the challenge of applying batch normalization during SNN training while considering the temporal data dependencies and the issue of temporal covariate shift. TAB (Temporal Accumulated Batch Normalization) is proposed which closely follows neuron dynamics, especially the accumulated membrane potential, enhancing the accuracy of batch statistics. This alignment establishes a natural link between the behavior of neurons and the application of batch normalization in Spiking Neural Networks (SNNs). TAB uses Temporal Accumulated Statistics (dynamically using a moving averaging approach) for data normalization, introducing learnable weights to differentiate the impact of each time-step on the final result. The study also provides a theoretical connection between TAB method and the neural dynamics.", "questions": "You have used trainable weights for the each point in the neurons sequence, however it is not explain well how the weights for a new data can be used. Imagine you see a shift in the data distribution are these weights still valid?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper is well written and easy to read. The authors presented an extensive set of experimental results to evaluate the proposed approach. The theoretical connection between the underlying neural dynamics and the propose normalization technique is very well explained and presented.", "weaknesses": "The experimental results does not show a significant improvement over SOTA for example TEBN, I would suggest including a demsar plot to show the improvement also explain fairly why the improvement is very small. Taking into account all the added complexity regarding trainable weights for sequential neurons, etc.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "l7eUAsSvKd", "reviewer": "Reviewer_Y1dw", "summary": "The authors point out the challenging issue currently recognized in SNN-based models, which involves non-differentiable activation functions and temporally delayed accumulation of outputs over time. In this manuscript, the authors introduce TAB (Temporal Accumulated Batch Normalization), a novel SNN batch normalization method that addresses the temporal covariate shift issue by aligning with neuron dynamics (specifically the accumulated membrane potential) and utilizing temporal accumulated statistics for data normalization.", "questions": "Please see details of weaknesses.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. A very detailed theoretical analysis discusses the challenging problems of directly training SNN models through BN structures.\n2. The authors introduce TAB, a novel SNN batch normalization method that addresses the temporal covariate shift issue by aligning with neuron dynamics and utilizing temporal accumulated statistics for data normalization.", "weaknesses": "1. The authors mention the phenomenon of covariance concept drift encountered by SNN models, so why don't they verify it on time-domain related datasets?\n2. We can see that the improvement on the CIFAR-10 and CIFAR-100 datasets is very small, even just 0.6%. Can the current approach be regarded as a theoretically feasible strategy?\n3. In addition, the authors have not looked at the impact of other network skeletons, such as ResNet34.\n4. Another issue reviewer is concerned about is the relationship between TAB and neuron dynamics. Does the exponential nonlinear operation of the SNN-based model on the time coefficient lead to an error in approximating a first-order linear ODE?\n\nFeedback: The author added a large number of revisions in the limited time, I think I have no new concerns.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "lB8vpvv0bi", "reviewer": "Reviewer_34HL", "summary": "In the paper, the authors introduce a Temporal Accumulated Batch Normalization to address the temporal covariate shift issue by aligning with neuron dynamics and utilizing temporal accumulated statistics for data normalization.  They did experiments on CIFAR10, CIFAR100, and DVS-CIFAR10.", "questions": "Please see weakness.", "limitations": "", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Ideas on low-bit precision neural networks are important and should be encouraged.\nThe paper focuses on temporal Covariate Shifts along the temporal dimension for SNN, which is an important problem in SNN.", "weaknesses": "The authors should provide an ablation study of the TAB, so we can know if the TAB can really increase the accuracy and how much the method improves. \nRecent work all did experiments on ImageNet, while the paper ignore this.\nSome references are missing.\n[1] GLIF: A Unified Gated Leaky Integrate-and-Fire Neuron for Spiking Neural Networks.\n[2] Reducing Information Loss for Spiking Neural Networks\n[3] Surrogate Module Learning: Reduce the Gradient Error Accumulation in Training Spiking Neural Networks", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "3bjqAo8GVv", "reviewer": "Reviewer_vi7C", "summary": "The paper discusses the challenges of training SNNs, such as non-differentiability of the spiking activation function and the temporal nature of data. It proposes a method called Temporal Accumulated Batch Normalization (TAB) which uses temporal accumulated statistics for data normalization to solve a well-established problem called Temporal Covariate Shift, enhancing the efficiency of SNN training. The paper presents experimental results on CIFAR-10, CIFAR-100, and DVS-CIFAR10 datasets, demonstrating the effectiveness of TAB in improving accuracy. The paper's strength lies in its novelty, detailed explanation of the method, and well-presented experimental results.", "questions": "What is the impact of TAB on energy efficiency, and how does it compare to other methods for training SNNs? While the authors mention that SNNs are attractive for their energy-efficient computing when implemented on neuromorphic hardware, they do not provide a detailed analysis of the impact of TAB on energy efficiency. This is an important question, as it would help to assess the potential impact of TAB on energy efficiency and identify potential areas for future research.\n\n\n\n\nCan you provide a more detailed analysis of the computational complexity of TAB and how it compares to other methods?\n\n\n\n\nCan you provide evidence that TAB solves TCS? I think the illustration of distribution is capable of demonstrating the solution.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "This paper has the following strengths:\n\nThe authors relate the proposed algorithm with the closed-form of the LIF dynamics and give principled insights into what components in SNN should be normalized.\n\nThe authors obtain better performance than previous algorithms.\n\nThe proposed method only records the past information, which can be some basic support to online training.", "weaknesses": "The paper lacks a detailed analysis of the computational complexity of TAB, which could be crucial for practical applications. Although it mentions that TAB requires more computations than other normalization techniques, it does not provide a detailed cost analysis or the impact of different hyperparameters on computational complexity, making it difficult to evaluate its practicality in real-world applications.\n\nThere seems to be a gradual improvement on the SNN batch normalization technique with the suggested method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)