Deep Residual Learning in Spiking Neural Networks
Authors: Wei Fang
Zhaofei Yu
Yanqi Chen
Tiejun Huang
Timothée Masquelier
Yonghong Tian
Venue: NeurIPS 2021 Poster (2021)
Keywords: Spiking Neural Networks
SNN
spike-based backpropagation
ImageNet

Abstract: Deep Spiking Neural Networks (SNNs) present optimization difficulties for gradient-based approaches due to discrete binary activation and complex spatial-temporal dynamics.  Considering the huge success of ResNet in deep learning, it would be natural to train deep SNNs with residual learning. Previous Spiking ResNet mimics the standard residual block in ANNs and simply replaces ReLU activation layers with spiking neurons, which suffers the degradation problem and can hardly implement residual learning. In this paper, we propose the spike-element-wise (SEW) ResNet to realize residual learning in deep SNNs. We prove that the SEW ResNet can easily implement identity mapping and overcome the vanishing/exploding gradient problems of Spiking ResNet. We evaluate our SEW ResNet on ImageNet, DVS Gesture, and CIFAR10-DVS datasets, and show that SEW ResNet outperforms the state-of-the-art directly trained SNNs in both accuracy and time-steps.  Moreover, SEW ResNet can achieve higher performance by simply adding more layers, providing a simple method to train deep SNNs. To our best knowledge, this is the first time that directly training deep SNNs with more than 100 layers becomes possible. Our codes are available at https://github.com/fangwei123456/Spike-Element-Wise-ResNet.

Deep Residual Learning in Spiking Neural Networks
Wei Fangl.2, 1.2, Zhaofei Yu!.2*, Yanqi Chen
Tiejun Huang!.? 1.2, Timothée Masquelier Yonghong Tian
1 Department of Computer Science and Technology. Peking University Peng Cheng Laboratory, Shenzhen 518055, China *Centre de Recherche Cerveau Cognition, UMR5549 CNRS Univ, Toulouse 3, Toulouse, France
Abstract
Deep Spiking Neural Networks (SNNs) present optimization difficulties for gradient-based approaches due to discrete binary activation and complex spatial- temporal dynamics. Considering the huge success of ResNet in deep learning, it would be natural to train deep SNNs with residual learning. Previous Spiking ResNet mimics the standard residual block ANNs and simply replaces ReLU activation layers with spiking neurons, which suffers the degradation problem and can hardly implement residual learning. In this paper, we propose the spike- element-wise (SEW) ResNet realize residual learning in deep SNNs. We prove that the SEW ResNet can easily implement identity mapping and overcome the vanishing/exploding gradient problems Spiking ResNet. We evaluate our SEW ResNet on ImageNet. DVS Gesture, and CIFARIO-DVE datasets, and show that SEW ResNet outperforms the state-of-the the-te- directly trained SNNs in both ac- curacy and time-steps. Moreover, SEW ResNet can achieve higher performance by simply adding more layers. providing simple method to train deep SNNs. To our best knowledge, this is the first time that directly training deep SNNs with more than 100 layers becomes possible. Our codes are available at tps
com //5stthu..ccm//aaneee111333333/2 Sttp/////ppkeeeee Element Wise ResNet
Introduction
Artificial Neural Networks (ANNs) have achieved great success in many tasks, including image classification [28 52, 55), object detection [9 34 44), machine translation [2], and gaming [37 51).
One of the critical factors for ANNs' success is deep learning (29), which uses multi-layers to learn representations of data with multiple levels of abstraction. It has been proved that deeper networks have advantages
shallower networks in computation cost and generalization ability (3). The function represented by a deep network can require an exponential number of hidden units by shallow network with one hidden layer (38]. In addition, the depth of the network is closely related to the network's 's performance in practical tasks 52
55 27 52
Nevertheless, recent evidence 13 53 14
reveals that with the network depth increasing, the accuracy gets saturated and then degrades rapidly. To solve this degradation problem, residual learning is proposed [14] [5] and the residual structure is widely exploited in "very deep" networks that achieve the leading performance [22 59 18 57].
Spiking Neural Networks (SNNs) are regarded as a potential competitor of ANNs for their high biological plausibility, event-driven property, and low power consumption 45). Recently, deep learning methods are introduced into SNNs, and deep SNNs have achieved close performance as ANNs in some simple classification datasets [[56], but still worse than ANNs in complex tasks, e.g., classifying the ImageNet dataset 47 To obtain higher performance SNNs, it would be natural to
"Corresponding author
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

explore deeper network structures like ResNet. Spiking ResNet [2 60 21
17
49
12 30 64 48
42 45 as the spiking version of ResNet, proposed by mimicking the residual block in ANNS and replacing ReLU activation layers with spiking neurons. Spiking ResNer converted from ANN achieves state-of-the-art accuracy on nearly all datasets, while the directly trained Spiking ResNet has not been validated to solve the degradation problem.
In this paper, we show that Spiking ResNet inapplicable all neuron models to achieve identity mapping. Even if the identity mapping condition is met, Spiking ResNet suffers from the problems of somishinn//xpllddigg
gradient. Thus, we propose the Spike- Spike-Elemeen Element- Wise (SEW) ResNet to realize residual learning in SNNs. We prove that the SEW ResNet can easily implement identity mapping and overcome the vanishing/explodng gradient problems the same time. We evaluate Spiking ResNet and SEW ResNet on both the static ImageNet dataset and the neuromorphic DVS Gesture dataset M.
CIFAR RO-DVV dataset [32. The experiment results are consistent with our analysis, indicating that the deeper Spiking ResNet suffers from the degradation problem the deeper network has higher training loss than the shallower network, while SEW ResNet can achieve higher performance by simply increasing the network's 's depth. Moreover, we show that SEW ResNet outperforms the state-of-the-ar art directly trained SNNs in both accuracy and time-stes. To the best of our knowledge, this is the first time to explore the directly-trained deep SNNs with more than 100 layers.
2 Related Work
2.1 Learning Methods of Spiking Neural Networks
and backpropagation with surrogate gradient [40] are the two main methods to get deep SNNs. The ANN2SNN method firstly trains ANN with ReLU activation, then converts the ANN to SNN by replacing ReLU with spiking neurons and adding scaling operations like weight normalization and threshold balancing. Some recent conversion methods have achieved near loss-less accuracy with VGG-16 and ResNet [12] 6 33 However, the converted SNN needs a longer time to rival the original ANN in precision as the conversion is based on rate-coding [46), which increases the SNN's s latency and restricts the practical application. The backpropagation methods can be classified into two categories (26). The method in the first category computes the gradient by unfolding the network over the simulation time- 30 40), which is similar to the idea of backpropagation through time (BPTT). As the gradient with respect to the wreshold-triggeree firing non-differentiable, the surrogate gradient is often used. The SNN trained by the surrogate method is not limited to rate-coding, and can also be applied on temporal tasks, e-g., classifying neuromorphic datasets [58 g 16]. The second method computes the gradients of the timings of existing spikes with respect to the membrane potential at the
ANN SNN conversion (ANN2SNN) 20
6 54 33
steps [31 19 58, 50
spike timing [5] 39 24 65
2.2 Spiking Residual Structure
Previous ANN2SNN methods noticed the distinction between plain feedforward ANNs and residual ANNs, and made specific normalization for conversion. Hu et al. [17] were the first to apply the residual structure in ANN2SNN with scaled shortcuts SNN to match the activations of the original ANN. Sengupta et al. [49] proposed Spike-Norm to balance SNN's threshold and verified their method by converting VGG and ResNet to SNNs. Existing backpropagation-bar based methods use nearly the same structure from ResNet. Lee et al. (30] evaluated their custom surrogate methods on shallow ResNets whose depths are no more than ResNet-11. Zheng et al. (64) proposed the pareshold-dependeed batch normalization (td-BN) to replace naive batch normalization (BN) (23) and
successfully trained Spiking ResNet-34 and Spiking ResNet-50 directly with surrogate gradient by adding td-BN in shortcuts.
3 Methods
3.1
Spiking Neuron Model
The spiking neuron is the fundamental computing unit of SNNs. Similar to Fang et al. [ we use unified model to describe the dynamics of all kinds of spiking neurons, which includes the following

---

x'
s'(=]
['[c]
Conv
Conv
Canv
BN
BN
BN
F (x)
ReLU
F((
SN
F (+)
Conv
Conv
Conv
BN
SN
A
ReLU
SN
['[c]
O'[c]
(a) Basic block in ResNet
(b) Basic block in Spiking ResNet (c) Spike-El Element- Wise block
Figure 1: Residual blocks in ResNet, Spiking ResNet and SEW ResNet.
discrete-time equations:
H = [(V[) - 1],
(1)
S(1 ((H[ ViA),
(2)
V #| (1- - S(t)) + Vreset S[t],
(3)
where [[{] is the input current at time-step 1, (1) and V(4] denote the membrane potential after neuronal dynamics and after the trigger of spike at time-step f, respectively. VaA is the firing threshold, ((x) is the Heaviside step function and is defined by ((z) for 2 0 and +(x) there a spike and 0 otherwise. Vreser denotes the reset potential. The function f() in Eq. 4 describes the neuronal dynamics and takes different forms for different spiking neuron models. For example, the function f(-) for the Integrate-ante and--Free (IF) model and Leaky Integrate-and and-Fire (LIF) model can be described by Eq. (4) and Eq. (5), respectively.
for x < 0. S[t] is the output spike at time-step t, which equals
H V - 1] + X[t),
(4)
H Vlt +] 1 (x]i] (Vlt 1] Vreset))
(5)
where 7 represents the membrane time constant. Eq. 2 and Eq. (3) describe the spike generation and resetting processes, which are the same for all kinds of spiking neuron models. this paper, the surrogate gradient method is used define O'xx) 2 o((x)) during error back- pack-propagation, with (x)
denoting the surrogate function.
3.2 Drawbacks of Spiking ResNet
The residual block is the key component of ResNet. Fig. a[) shows the basic block in ResNet [4]. where X', y' are the input and output of the 1-th block in ResNet, Conv is the convolutional layer, BN denotes batch normalization, and ReLU denotes the rectified linear unit activation layer. The basic
block of Spiking ResNet used in 64 17 30 simply mimics the block ANNs by replacing ReLU
activation layers with spiking neurons (SN), which is illustrated in Fig. 1 bb). Here [*[4, O'[L] are the
input and output of the (-th block in Spiking ResNet at time-step t. Based on the above definition, we will analyze the drawbacks of Spiking ResNet below.
Spiking ResNet is inapplicable all neuron models to achieve identity mapping. One of the critical concepts in ResNet is identity mapping. He et al. [14] noted that if the added layers implement the identity mapping, deeper model should have training eгror no greater than its shallower counterpart. However, it is unable to train the added layers to implement identity mapping in feasible time, resulting in deeper models performing worse than shallower models (the degradation problem). To solve this problem, the residual learning is proposed by adding shortcut connection (shown in Fig. Ia)) If we use Ft to denote the residual mapping, egg., a stack of two convolutional layers, of the (-th residual block in ResNet and Spiking ResNet, then the residual block in Fig Ia]
3

---

and Fig can be formulated as
y = ReL.U(F(X)) + +x),,
(6)
O [t] (S))F + (t)))
(7)
The residual block of Eq. (6) make it easy to implement identity mapping in ANNs. To see this, when FXX) 0, Y ReLU(X)) ( In most cases, X is the activation of the previous ReLU layer and X 220 0. Thus, y ReLU(X') = X', which is identity mapping.
Different from ResNet, the residual block in Spiking ResNet (Eq.
7 restricts the models of spiking
When.F'SS F((S [4] = 0, O' (N(S"(() + S*((), To transmit S'(4 and make SN((S [4] = [4]. the last spiking neuron (SN) in the 1-th residual block needs to fire a spike after receiving a spike, and keep silent after receiving no spike at time- step t. It works for IF neuron described by Eq. 4 Specifically, we can set 0 < Vth S and Vlt 1] = 0 to ensure that x = leads to H [t] 2 th. and X [t] = leads to H [t] < Vua. However, when considering some spiking neuron models with complex neuronal dynamics, it is hard to achieve (t)
neuron to implement identity mapping. When
S' For example, the LIF neuron used in [66, 8 61] considers learnable membrane time constant the neuronal dynamics of which can be described with Eq. (5). When X[t] and Vlt 1]
H[4 - J. It is difficult to find a firing threshold that ensures H[4 > Vo as r is being changed in training by the optimizer.
Spiking ResNet suffers from the problems of vanishing/exploding gradient. Consider spiking
ResNet with sequential blocks to transmit (*[t), and the identity mapping condition is met, e.g.. the spiking neurons the IF neurons with 0 Vin S 1, then we have S'[t] S =...
g = 1 Denote the j-th element in (*(4) and O'(4 as S/10] and O [4] respectively,
the gradient of the output of the (1 + k 1)-th residual block with respect to the input of the 1-th residual block can be calculated layer by layer:
0, 0. < (S] Vin) <
a05
[4]
k-1
k-1
II'' 1=0
Van) 7
1. Van) =
1 10
a
if Vva)
(8)
where ((x)) is the Heaviside step function and 6' (zx is defined by the surrogate gradient. The second equality hold as 0 = SN(S)= [t)). In view of the fact that L can only take 0 or ('( Vnn) = 1 is not satisfied for commonly used surrogate functions mentioned [40]. Thus, the vanishing/exploding gradient problems are prone happen in deeper Spiking ResNet.
Based on the above analysis, we believe that the previous Spiking ResNet ignores the highly nonlinear caused by spiking neurons, and can hardly implement residual learning. Nonetheless, the basic block in Fig. Tbb) is still decent for ANN2SNN with extra normalization [17 49], as the SNN converted from ANN aims to use firing rates to match the origin ANN's activations.
3.3 Spike-Element- Wise ResNet
Here we propose the -eee---eeeeetteee (SEW) residual block realize the residual learning in SNNs, which can easily implement identity mapping and overcome the vanishing/exploding gradient problems the same time. As illustrated in Fig. [c), the SEW residual block can be formulated as:
O'[t] = ((N((F (S ())) 5' ('()) = ((A
where g represents an element-wise function with two spikes tensor as inputs. Here we use A'[ to denote the residual mapping to be learned as A'lt] = (S
SEW ResNet can easily implement identity map- ping. By utilizing the binary property of spikes, we can find different eliment-wise functions g that sat- isfy identity mapping (shown in Tab. []. To be spe- cific, when choosing ADD and IAND as element-wise functions g, identity mapping is achieved by setting A = O, which can be implemented simply by setting
Name
Expression of gi ()
ADD
A +
AND
IAND
Table 1: List of element-wise functions

---

5'(c
['[[]
Conv
Conv
.F
FFi
BN
BN
SN
SN
SN
['[
O' [c]
(a) Downsample basic block
(b) Downsample SEW block
Figure 2: Downsample blocks in Spiking ResNet and SEW ResNet.
the weights and the bias of the last batch normalization layer (BN) in F to zero. Then we can get O'(t] g( [1) (t)) (SN(O)) S" [t)) g(0, [t)) = [')).. This is applicable to all neuron models. When using AND the element-wise function g. we A' It can be implemented by setting the last BN's weights to zero and the bias to large enough con- stant to cause spikes, e.g.. setting the bias as Vin when the last SN is IF neurons. Then we have O'( = [t] = S [t]. Note that using AND may suffer from the same problem as Spiking ResNet. is hard to control some spiking neuron models with complex neuronal dynamies to generate spikes at specified time-step.
get identity mapping.
Formulation of downsample block. Remarkably, when the input and output of one block have different dimensions, the shortcut is set as convolutional layers with stride > 1, rather than the identity connection, perform downsampling. The ResNet and the Spiking ResNet utilize (Conv- B without ReLU in shortcut (Fig. 2 a)). In contrast, we add a SN in shortcut (Fig. b)).
SEW ResNet can overcome vanishing/exploding gradient. The SEW block is similar ReLU before addition (RBA) block [15] in ANNs, which can be formulated as
y ReLU(F (X')) X
(10)
The RBA block is criticized by He et al. [5 for X Y X', which will cause infinite outputs in deep layers. The experiment results in [5] also showed that the performance of the RBA block is worse than the basic block (Fig(aa) To some extent, the SEW block is an extension of the RBA block. Note that using AND and TAND will output spikes (i.e. binary tensors), which means that the infinite outputs problem ANNS will never occur in SNNs with SEW blocks, since all spikes are less equal than 1. When choosing ADD g. the infinite outputs problem can be relieved as the output of sequential SEW blocks will be no larger than + In addition, a downsample SEW block will regulate the output to be no larger than 2 when y is ADD.
When the identity mapping is implemented. the gradient of the output of the (I+ k 1)-th SEW block with respect to the input of the (-th SEW block can be calculated layer by layer:
k-1 a(0+s/+(t) (+)
if g -AD
a07+h-111 a
k 1 Og( 0g(A4+*100
[1] $4,,5 (4)
801- (
as - if g AND
= (II)
1= as astt1
3(1-0))s(t1))
if = LAND
as
The second equality holds as identity mapping is achieved by setting A [t] for g AND, and ^++((| = 0 for g = ADD/ANND, Since the gradient in Eq. (11) is a constant, the SEW ResNet can overcome the vanishing/cxploding gradient problems.
Experiments
4.1 ImageNet Classification
As the test server ImageNet 2012 is no longer available, we can not report the actual test accuracy. Instead, we use the accuracy the validation set as the test accuracy, which is the same as [17 64).
5

---

BRM Ru.der
BRX Rodkeca
mq Booke-4
SKW Rodk-1111
eq Bolke-111
SRW
5C1 Rabii-11
Ralk Radii
Radia--1 14
Solkk
Sokka Radiint11
SCW Badii 130
(a) Training loss
(h) Training accuracy
587 Io 03-6-191
Salin Boolicca Salce
290
1p30
(c) Test accuracy
Figure 3: Comparison of the training loss, training accuracy and test accuracy ImageNet.
Network
SEW ResNet (ADD)
Spiking ResNet
Acc@ 1(%)
Acc@5(%)
Ace@ ((%)
Ace@5(%)
ResNet-18
63.18
84.53
62.32
84.05
ResNet-34
67.04
87,25
61.86
83.69
ResNet-50
67.78
87.52
57.66
80.43
ResNet-101
68.76
88.25
31.79
54.91
ResNet-152
69.26
88.57
10.03
23.57
Table 2: Test accuracy on ImageNet.
He et al. [1 evaluated the 18/34/50/101/152-aayee ResNets on the ImageNet dataset. For comparison, we consider the SNNs with the same network architectures, except that the basic residual block (a)) is replaced by the spiking basic block (b) and SEW block (Fig(Tc)) with as ADD, respectively. We denote the SNN with the basic block as Spiking ResNel and the SNN with the SEW block as SEW ResNet. The IF neuron model is adopted for the static ImageNet dataset. During training on ImageNet, we find that the Spiking ResNNet-00/10//15 can not converge unless we use zero initialization [10], which sets all blocks to be an identity mapping at the start of training. Thus, the results of Spiking ResNet- 18/34/50///111////555 reported in this paper are with zero initialization.
Spiking ResNet vs. SEW ResNet. We first evaluate the performance of Spiking ResNet and SEW ResNet. Tab, reports the test accuracy on ImageNet validation. The results show that the deeper 34-layer Spiking ResNet has lower test accuracy than the shallower 18-layer Spiking ResNet. As the layer increases, the test accuracy of Spiking ResNet decreases. To reveal the reason, we compare the training loss, training accuracy, and test accuracy of Spiking ResNet during the training procedure, which is shown Fig. 3 We can find the degradation problem of the Spiking ResNet the deeper network has higher training loss than the shallower network. In contrast, the deeper 34-layer SEW ResNet has higher test accuracy than the shallower 18-layer SEW ResNet (shown in Tab. 2 More importantly, it can be found from Fig. that the training loss of our SEW ResNet decreases and the training/test accuracy increases with the increase of depth, which indicates that we can obtain higher performance by simply increasing the network's depth. All these results imply that the degradation problem is well addressed by SEW ResNet.
Comparisons with State-of-the-ar Methods. In Tab. we compare SEW ResNet with previous Spiking ResNets that achieve the best results on ImageNet. To our best knowledge, the SEW ResNet- 101 and the SEW ResNet-155 152 are the only SNNs with more than 100 layers to date, and there are no other networks with the same structure to compare. When the network structure the same, our SEW ResNet outperforms the state-of-the-an accuracy of directly trained Spiking ResNet, even with fewer

---

Network
Methods
Accuracy( (%)
SEW ResNet-34
Spike-based BP
67,04
4
Spiking ResNet-34(large) with Id-BN 66
Spike-based BP
67.05
6
Spiking ResNet-34 with Id-BN [64
Spike-based BP
63.72
6
Spiking ResNet-34 [2
ANN2SNN
69.89
4096
Spiking ResNet-34
ANN2SNN
65.47
2000
Spiking ResNet-34
[33
ANN2SNN
74,61
256
Spiking ResNNet:34 [4
ANN2SNN and Spike-based BP
61.48
250
SEW ResNet-50
Spike-based BP
67.78
4
Spiking ResNeet-50 with Id-BN 64
Spike-based BP
64.88
6
Spiking ResNet-50 [7
ANN2SNN
72.75
350
SEW ResNet--101
Spike-based BP
68.76
SEW ResNet-152
Spike-based BP
69.26
Table 3: Comparison with previous Spiking ResNet on ImageNet. f has the same network structure the standard Spiking ResNet-34, but uses four times as many the number of convolution kernels.
04
019-
600
Mode adio!
Figure 4: Firing rates of A in SEW blocks on ImageNet.
time-steps T. The accuracy of SEW ResNet-34 is slightly lower than Spiking ResNet-34 (large) with td-BN (67.049 .0%% v.s. 67. 67.05%) which uses 1.5 times as many simulating steeessess T v.s. 4) and times many the number of parameters (85.5M v.s. 21. 21.8M), compared with our SEW ResNet. The state-of-the-art ANN2SNN methods [33 17 have better accuracy than our SEW ResNet, but they respectively use 64 and 87.5 times as many time-steps as ours.
Analysis of spiking response of SEW blocks. Fig. shows the firing rates of A' in SEW ResNet- 18/34/50/101/15/ on ImageNet. There are 7 blocks in SEW ResNet-18, 15 blocks in SEW ResNet-34 and SEW ResNet-50, 33 blocks SEW Res,ett1101, and 50 blocks in SEW ResNet-152. The downsample SEW blocks are marked by the triangle down symbol V. As we choose ADD as element- wise functions g. a lower firing rate means that the SEW block gets closer to implementing identity mapping, except for downsample blocks. Note that the shortcuts of downsample blocks are not identity mapping, which is illustrated in Fig. b). Fig. 4 shows that spiking neurons SEW blocks have low firing rates, and the spiking neurons in the last two blocks even have firing rates of almost zero. As the time-steps T is 4 and firing rates are no larger than 0.25, all neurons SEW GesNeet-18344555 fire average more than one spike during the whole simulation. Besides, all firing rates in SEW ResNet- not more than two spikes. In general, the firing rates of A' in SEW blocks are at low level, verifying that most SEW blocks act as identity mapping.
are not larger than 0.5, indicating that all neurons fire on average
Gradients Check on ResNet-152 Structure. Eq. (8) and Eq. analyze the gradients of multiple blocks with identity mapping. To verify that SEW ResNet can overcome vanishing/exploding gradient, we check the gradients of Spiking ResNet- 152 and SEW ResNet-152, which are the deepest standard ResNet structure. We consider the same initialization parameters and with//wthout zero initialization.
As the gradients of SNNs are significantly influenced by firing rates (see A.4). we analyze the firing rate firstly. Fig. a) shows the initial firing rate of 1-th block's output O The indexes of downsample blocks are marked by vertical dotted lines. The blocks between two adjacent dotted lines represent the identity mapping areas, and have inputs and outputs with the same shape. When using zero initialization, Spiking ResNet, SEW AND ResNet, SEW IAND ResNet, and SEW ADD ResNet have the same firing rates (green curve), which is the zero init curve. Without zero initialization, the silence problem happens in the SEW AND network (red curve), and is relieved by the SEW IAND network (purple curve). Fig. () shows the firing rate of A', which represents the output of last SN in 1-th block. It can be found that although the firing rate of O' in SEW ADD ResNet increases linearly the identity mapping areas, the last SN each block still maintains a stable firing rate. Note that when g is ADD, the output of the SEW block is not binary, and the firing
7

---

rate is actually the mean value. The SNs of SEW IAND ResNet maintain an adequate firing rate and decay slightly with depth (purple curve), while SNs deep layers of SEW AND ResNet keep silent (orange curve). The silence problem can be explained as follows. When using AND, A O'='(1] I ('- Since it is hard to keep SN(F (1)) = at each time-step I, the silence problem may frequently happen in SEW ResNet with AND as g. Using
O'(4 = (N(F
IAND as a substitute of AND can relieve this problem because it is easy to keep SN SN(F [d)) =
at each time-step
0.50
0.50
0.25
0.22
(hing
5.19
Sprinng Rec
EEVAAN AND
ADD
0.10
0:35
SPV WMD
SEW AND
BEW MANU
0.00
1
10
20
20
40
20
40
40
10
20
30
40
brook ndoe
brook Index
block indox
2
(a) Firing rate of output O'
(b) Firing rate of A'
Figure 5: The initial firing rates of output O' and A in /-th block on 152-layer network.
The surrogate gradient function we used in all experiments is (xx) arctan (5rr) + 1.2 thus d'xx)
2( When Vvs- = a, = 2, the gradient amplitude || as of each block is shown in Fig. Note that 2, o' (x:) S d' a Vva) 1 and o o(( Via.) ) 0.092 1. It can he found that the gradients in Spiking ResNet- decay from deeper layers to shallower layers in the 1. It is worth noting that the decay also happens in Spiking ResNet-152 with zero initialization. The small convex A near the dotted lines is caused by the vanishing gradients of those S = 0. After these gradients decays to 0 completely, a'(1 - Vv.) 1, which When referring to SEW ResNet- 152 with zero initialization, can be found that all gradient-inde -ndex curves similar matter what 9 choose. This is caused by that in the identity mapping areas, St is constant for all index I, and the gradient also becomes a constant as it will not flow through SNs. Without zero initialization, the vanishing gradient happens in the SEW AND ResNet- 152, which is caused by the silence problem. The gradients of SEW ADD, IAND network increase slowly when propagating from deeper layers to shallower layers, due to the adequate firing rates shown in Fig. 5
identity mapping areas without zero initialization, which is caused by o' o'(x)
|| aL 75 will be constant because the rest gradients are calculated by it 1 and also explain why the gradient-inde index curve is horizontal at some areas.
When Ven = 0.5, a = 2. d'(0 - Voa) = a'(1 - ViA) = 0.288 < 1, indicating that transmitting spikes to SNs is prone to causing vanishing gradient, shown in Fig. 7 With zero initialization, the decay in Spiking ResNet-152 is more serious because gradient from F can not contribute. matter what g we choose. When Vn 1, = 3, d- Vv,) = 1.5 > 1, indicating that transmitting spikes to SNs is prone to causing exploding gradient. Fig. X shows the gradient in this situation. Same with the reason in Fig. 6 the change of surrogate function will increase gradients of all networks without zero initialization, but not affect SEW Re5Net-152 with zero initialization. The Spiking ResNet-152 meets exploding gradient, while this problem in SEW ADD, IAND ResNet-152 is not serious.
The SEW ResNet-152 will not be affected
4.2 DVS Gesture Classification
The origin ResNet, which is designed for classifying the complex ImageNet dataset, is too large for the DVS Gesture dataset. Hence, we design tiny network named 7B-Net, whose structure is c32k3sl B-833sI-BN-PLFEE RN-PLIF-/SEW Block-MPk2s2)
Here c32k3sl means the convolutional layer with channels 32, kernel size 3, stride 1. MPk2s2 the max pooling with kernel size 2, stride 2. The symbol ()*7 denotes seven repeated structure, and PLIF denotes the Parametric Inter--nttaat Fire Spiking Neuron with a learnable membrane time constant, which is proposed in S and can be described by Eq. (5) See A.1 for AER data processess details.
8

---

0.0
198
Splingg Heaieerl Spiing Harter parpes
Anl Rar 14-522
SCH1A20 Bashar (a
Acl Ban 1.000
bock
Figure 6: Gradient amplitude
of 1-th block when ViA 11 a as!
0250
90W Peria-112
116
ALL Hantee mauwaad
Lo00
Cor1
3002
2214 Pas Caao- B01)
0400
20
2000
280
100
10
0008
34
brock
birh wibee
Bookinaeer
Figure 7: Gradient amplitude
||
01. DS
of 1-th block when Vn 0.5, 2.
Spiking ResNet vs. SEW ResNet. We first compare the performance of SEW ResNet with ADD element-wee wise function (SEW ADD ResNet) and Spiking ResNet by replacing SEW blocks with basic blocks. As shown in Fig. 1 and Tab. 4 although the training loss of Spiking ResNet (blue curve) is lower than SEW ADD ResNet (orange curve), the test accuracy is lower than SEW ADD ResNet (90.97% v.s. 97. 97.92%) which implies that Spiking ResNet is easier to overfit than SEW ADD ResNet.
Evaluation of different element-wise functions and plain block. As the training cost of SNNs on the DVS Gesture dataset is much lower than ImageNet, carry out more ablation ex- periments on the DVS Gesture dataset. We replace SEW blocks with the plain blocks (no short- connection) and test the performance. We also evaluate all kinds of element wise functions in Tab. I Fig. shows the training loss and training/test accuracy on DVS Gesture. The sharp fluctuation during early epochs is caused by the large learning rate (see Sec We can find that the training loss SEW IAND-Spiking ResNet< ADDx<Plin NeW AND. Due the overfitting problem, lower loss does not guarantee higher test accuracy. Tab. shows the test accuracy of all net- works. The SEW ADD ResNet gets the highest accuracy than others.
Network
Element-Wii Function
Accuracy %)
SEW ResNet
ADD
97.92
ResNet
IAND
95.49
Comparisons with State-of-the-art Methods. Tab. compares our network with SOTA methods. It can be found that our SEW ResNet outperforms the SOTA works in accuracy, parameter numbers, and simulating time-steps.
Plain Net
91.67
Spiking ResNet
90.97
SEW ResNet
AND
70.49
Table 4: Test accuracy on DVS Gesture. The networks' order is ranked by accuracy.
Network
Accuracy %%
Parameters
T
BEL Block (c32) -MP1232) (7B-Net) [c128K3s1 MPP22///L-MMM2
(7B-N))
97.92
0.13M
16
97.57
1.70M
20
Spiking 17 with id-BN 64
96.87
11.T8M
40
93.40
23.23M
60
Table 5; Comparison with the state-of-the-art (SOTA) methods on DVS Gesture dataset.
4.3 CIFARIO-DVS Classification
We also report SEW ResNet on the CIFARIO-DVS dataset, which is obtained by recording the moving images of the CIFAR- 10 dataset on LCD monitor by a DVS camera. As DIFARIO-DYY more complicated than DVS Gesture, we use the network structure named Wide-7B-Net, which is similar to 7B-Net with more channels. The structure of Wide-7B-Net is -64K331-RR-PIIFF-SS -BN--LII--SS-P Block (cJ28))
Block (c64)

---

ring Famrie 16
201 Sarvee
6o8
2-A Wy Wo-to
2P17 ICC Rorddd WdS-oaad
0000
100
9000
121
12
23
33
block Jade
mbo
20ch
Figure 8: Gradient amplitude
01 as
of /-th block when Ven 1, 3.
SEWINDD
spoch
(a) Training loss
(b) Training accuracy
(c) Test accuracy
Figure 9: Comparison of the training loss, training accuracy and test accuracy on DVS Gesture
dataset.
Network
Accuracy %)
Parameters
T
BL-LI---SW Block (c64)-MPK222)) 2411222833
UN-PLIF-(SEN) Block (C22)-MPP222)"=--CIII (Wide-7B-Net)
64.8, 70.2, 74.4
1.19M
4,8,
74,8
17.4M
20
FC100-PLIF COO-PLIF-APK100 APK10s
Spiking ResNet-19 with Id-BN [64
67.8
11.18M
10
Table Comparison with the -tate-of-the-art (SOTA) methods on CIFARIO-DVS dataset.
In Tab 6 we compare SEW ResNet with the previous Spiking ResNet. One can find that our method achieves better performance (70.2% v.s. 67.8%) and fewer time-steps (8 10) than the Spiking ResNet [64]. We also compare our method with the state-of offte--f-hh-aat ttthhh-rtt art (SOTA) supervised learning methods on CIFARIO-DVS. The accuracy of our Wide-7B-Net is slightly lower than the current SOTA method [8] (74.4% v.s. 74.8%), which uses 1.25 times as many simulation time-steps T (20 v.s. 16) and 14.6 times as many the number of parameters (17.4M v.s. 1.19)). Moreover, when reducing T shapely to T = 4, our Wide-7B-No can still get the accuracy of 64.8%.
5 Conclusion
In this paper, analyze the previous Spiking ResNet whose residual block mimics the standard block ResNet, and find that can hardly implement identity mapping and suffers from the problems of vanishing/exploding gradient. To solve these problems, propose the SEW residual block and prove that it can implement the residual learning. The experiment results on ImageNet, DVS Gesture, and CIFARIO-DVS datasets show that our SEW residual block solves the degradation problem, and SEW ResNet can achieve higher accuracy by simply increasing the network's 's depth. Our work may shed light the learning of "very deep" SNNs.
Acknowledgment
This work is supported by grants from the National Natural Science Foundation of China under contracts No. 6002622770 10.61825101, and 10.62088102.
10

---

References
[1] Arnon Amir, Brian Taba, David Berg, Timothy Melano, Jeffrey McKinstry, Carmelo Di Nolfo, Tapan Nayak, Alexander Andreopoulos, Guillaume Garreau, Marcela Mendoza, Jeff Kusnitz, Michael Debole, Steve Esser, Tobi Delbruck, Myron Flickner, and Dharmendra Modha. A low power, fully event-based gesture recognition system. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition (CVPR), pages 7243-7252, 2017.
[2] Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning align and translate. In International Conference Learning Representations (ICLR), 2015.
[3] Yoshua Bengio, Yann LeCun, et al. Scaling learning algorithms towards ai. Large-scale Kernel Machines, 34(5):1-41, 2007.
[4] Yongqiang Cao, Yang Chen, and Deepak Khosla. Spiking deep convolutional neural networks for energy-efficient object recognition. International Journal of Computer Vision, 1 113(1)54466, 2015.
[5] Iulia M Comsa, Krzysztof Potempa, Luca Versari, Thomas Fischbacher, Andrea Gesmundo, and Jyrki Alakuijala. Temporal coding in spiking neural networks with alpha synaptic function. In International Conference on Acousties, Speech and Signal Processing (ICASSP), pages 8529-8555 8533. IEEE, 2020.
[6] Shikuang Deng and Shi Gu. Optimal conversion of conventional artificial neural networks to spiking neural networks. In International Conference on Learning Representations (ICLR), 2021.
[7] Wei Fang. Yanqi Chen, Jianhao Ding, Ding Chen, Zhaofei Yu, Huihui Zhou, Yonghong Tian, and other contributors. Spikingjelly.
com
spikingjelly
[8] Wei Fang, Zhaofei Yu, Yanqi Chen, Timothee Masquelier, Tiejun Huang, and Yonghong Tian. Incorporating learnable membrane time constant to enhance learning of spiking neural networks. In Proceedings of the IEEE/CVF International Conference Computer Vision (ICCV), pages 2661-2671, 2021.
[9] Ross Girshick, Jeff Donahue, Trevor Darrell, and Jitendra Malik. Rich feature hierarchies for accurate object detection and semantic segmentation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 580-587, 2014.
[10] Priya Goyal, Piotr Dollár, Ross Girshick. Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, Andrew Tulloch, Yangqing Jia, and Kaiming He. Accurate, large minibatch sgd: Training imagenet in hour. arXiv preprint arXiv: / 706.002677 2017.
[11] Bing Han and Kaushik Roy. Deep spiking neural network: Energy efficiency through time based coding. In European Conference on Computer Vision (ECCV), pages 388-404, 2020.
[12] Bing Han, Gopalakrishnan Srinivasan, and Kaushik Roy. Rmp-snn: Residual membrane potential neuron for enabling deeper high-accuracy and low-latency spiking neural network. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 13558 13567, 2020.
[13] Kaiming He and Jian Sun. Convolutional neural networks at constrained time cost. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 2015.
[14] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition (CVPR), pages 770-778, 2016.
[15] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Identity mappings in deep residual networks. In European Conference on Computer Vision (ECCV), pages 630-645. Springer, 2016.
11

---

[16] Weihua He, YuJie Wu, Lei Deng, Guoqi Li. Haoyu Wang, Yang Tian, Wei Ding. Wenhui Wang. and Yuan Xie. Comparing smns and mns on neuromorphic vision datasets: Similarities and differences. Neural Networks, 132: 1021008-220 2020.
[17] Yangfan Hu, Huajin Tang, Yueming Wang, and Gang Pan. Spiking deep residual network. arXiv preprint Xr0: 0352,, 2018.
[18] Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. Densely connected convolutional networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 4700-4708, 2017.
[19] Dongsung Huh and Terrence J Sejnowski. Gradient descent for spiking neural networks. In Advances Neural Information Processing Systems (NeurlPS), pages 1440-1450, 2018.
[20] Eric Hunsberger and Chris Eliasmith. Spiking deep networks with lif neurons. arXiv preprint arXiv:1510.08829, 2015.
[21] Sungmin Hwang, Jeesoo Chang, Myn Oh, Kyung Kyu Min, Taejin Jang. Kyungchul Park, Junsu Yu, Jong-Ho Lee, and Byung-Good Gookk Park. Low-latency spiking neural networks using pre-charged membrane potential and delayed evaluation. Fronfiers Neuroscience, 15:135, 2021.
[22] Forrest N landola, Song Han, Matthew W Moskewicz, Khalid Ashraf, William J Dally, and Kurt Keutzer. Squeezenet: Alexnet-level accuracy with 50x fewer parameters and< 0.5 mb model size. arXiv preprint arXiv: 2016.
[23] Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International Conference on Muchine Learning (ICML), pages 448-456, 2015.
[24] Saeed Reza Kheradpisheh and Timothée Masquelier. Temporal backpropagation for spik- ing neural networks with one spike per neuron. International Journal of Neural Systems, 30(06):2050027, 2020.
[25] Jachyun Kim, Heesu Kim, Subin Huh, Jinho Lee, and Kiyoung Choi. Deep neural networks with weighted spikes. Neurocomputing, 311:373-386, 2018
[26] Jinseok Kim, Kyungsu Kim, and Jae-Joon Kim. Unifying activation- and timing-based learning rules for spiking neural networks. In Advances in Neural Information Processing Systems (NeurIPS), pages 19534-19544, 2020.
[27] Alex Krizhevsky. One weird trick for parallelizing convolutional neural networks arXiv preprint arXiv: 1904,597, 2014.
[28] Alex Krizhevsky, llya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep con- volutional neural networks. In Advances in Neural Information Processing Systems (NeurIPS), pages 1097-1105, 2012.
[29] Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. Deep learning. Nature, 521((553):4366 444, 2015.
[30] Chankyu Lee, Syed Shakib Sarwar, Priyadarshini Panda, Gopalakrishnan Srinivasan, and Kaushik Roy. Enabling spike-based backpropagation for training deep neural network architec- tures. Frontiers in Neuroscience, 14, 2020.
[31] Jun Haeng Lee, Tobi Delbruck, and Michael Pfeiffer. Training deep spiking neural networks using backpropagation. Frontiers in Neuroscience, 10:508, 2016.
[32] Hongmin Li. Hanchao Liu, Xiangyang Ji, Guoqi Li, and Luping Shi. Cifar10-dvs: An event- stream dataset for object classification. Frontiers in Neuroscience, 11:309, 2017.
[33] Yuhang Li, Shikuang Deng, Xin Dong, Ruihao Gong, and Shi Gu. A free lunch from ann: Towards efficient, accurate spiking neural networks calibration. In International Conference on Machine Learning (ICML), volume 139, pages 1316-6325, 2021.
12

---

[34] Wei Liu, Dragomir Anguelov, Dumitru Erhan, Christian Szegedy, Scott Reed, Cheng- Yang Fu, and Alexander C Berg. Ssd: Single shot multibox detector. In European Conference on Computer Vision (ECCV), pages 21-37. Springer, 2016.
[35] Iyy Loshchilov and Frank Hutter. SGDR: stochastic gradient descent with warm restarts, In International Conference on Learning Representations (ICI.R), 2017.
[36] Paulius Micikevicius, Sharan Narang, Jonah Alben, Gregory Diamos, Erich Elsen, David Garcia, Boris Ginsburg, Michael Houston, Oleksii Kuchaiev. Ganesh Venkatesh, et al. Mixed precision training. In International Conference on Learning Representations (ICLR), 2018.
[37] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Mare G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. Nature, 51 518 ((7540)) 533-333, 2015.
[38] Guido F Montufar, Razvan Pascanu, Kyunghyun Cho, and Yoshua Bengio. On the number of linear regions of deep neural networks. In Advances in Neural Information Processing Systems (NeurIPS), pages
2014.
[39] Hesham Mostafa. Supervised learning based on temporal coding in spiking neural networks. IEEE Transactions on Neural Networks and Learning Systems, 29(7):3227-32355 2017.
[40] Emre O Neftei, Hesham Mostafa, and Friedemann Zenke. Surrogate gradient learning in spiking neural networks: Bringing the power of gradient- optimization to spiking neural networks. IEEE Signal Processing Magazine, 36(6):51 63, 2019.
[41] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang. Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-performance deep learning library. In Advances in Neural Information Processing Systems (NeurIPS), pages 8026-8037, 2019.
[42] Nitin Rathi and Kaushik Roy. Diet-snn: Direct input encoding with leakage and threshold optimization in deep spiking neural networks. arXiv preprint arXiv:2008. 036558 2020.
[43] Nitin Rathi, Gopalakrishnan Srinivasan, Priyadarshini Panda, and Kaushik Roy. Enabling deep spiking neural networks with hybrid conversion and spike timing dependent backpropagation. International Conference on Learning Representations (ICLR), 2020.
[44] Joseph Redmon, Santosh Divvala, Ross Girshick, and Ali Farhadi. You only look once: Unified, real-time object detection. Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition (CVPR), pages 779-788, 2016.
[45] Kaushik Roy, Akhilesh Jaiswal, and Priyadarshini Panda. Towards spike-based machine intelligence with neuromorphic computing. Nature, 575(7784);607-617, 2019.
[46] Bodo Rueckauer, Iulia-Alexandra Lungu, Yuhuang Hu, Michael Pfeiffer, and Shih-Chii Liu. Conversion of continuous-valuue deep networks efficient event-driven networks for image classification. Frontiers in Neuroscience, 11:682, 2017.
[47] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. International Journal of Computer Vision, 115(3):211-252 2015.
[48] Ali Samadzadeh, Fatemeh Sadat Tabatabaei Far, Ali Javadi, Ahmad Nickabadi, and Morteza Haghir Chehreghani. Convolutional spiking neural networks for spatio-temporal feature extraction. arXiv preprint arXiv:2003. /2346, 2020.
[49] Abhronil Sengupta, Yuting Ye, Robert Wang, Chiao Liu, and Kaushik Roy. Going deeper in spiking neural networks: Vgg and residual architectures. Frontiers Neuroscience, 13:95, 2019.
13

---

[50] Sumit Bam Shrestha and Garrick Orchard. Slayer: Spike layer error reassignment in time. In Advances in Neural Information Processing Systems (NeurIPS), pages 1419-1428, 2018.
[51] David Silver, Aja Huang, Chris J Maddison, Arthur Guez, Laurent Sifre, George Van Den Driess- che, Julian Schrittwieser, loannis Antonoglou, Veda Panneershelvam, Mare Lanctot, et al. Mas- tering the game of go with deep neural networks and tree search. Nature, :29(7587):484-488) 2016.
[52] Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. In International Conference on Learning Representations (ICLR), 2015.
[53] Rupesh Kumar Srivastava, Klaus Greff, and Jürgen Schmidhuber. Highway networks. arXiv preprint arXiv::505. 003877 2015.
[54] Christoph Stöckl and Wolfgang Maass. Optimized spiking neurons can classify images with high accuracy through temporal coding with two spikes. Nature Machine Intelligence, 3(3):230-238, 2021.
[55] Christian Szegedy. Wei Liu, Yangging Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 1-9, 2015.
[56]
Amirhossein Tavanaei, Masoud Ghodrati, Saeed Reza Kheradpisheh, Timothée Masquelier, and Anthony Maida. Deep learning in spiking neural networks. Neural Networks, 111:47-63, 2019.
[57] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, L ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems (NeurIPS), pages 5998-6008, 2017.
[58] Yujie Wu, Lei Deng, Guoqi Li, Jun Zhu, and Luping Shi. training perforreform spiking neural networks. Frontiers in Neuroscience, 12:331, 2018.
backpropagation for
[59] Saining Xie, Ross Girshick, Piotr Dollár, Zhuowen Tu, and Kaiming He. Aggregated residual transformations for deep neural networks. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pages 5987-5995, 2017.
[60] Fu Xing, Ye Yuan, Hong Huo, and Tao Fang. Homeostasis-based cnn-to-snn conversion of inception and residual architectures. In International Conference on Neural Information Processing, pages 173-184. Springer, 2019.
[61] Bojian Yin, Federico Corradi, and Sander M Bohté. Effective and efficient computation with multiple-timescate spiking recurrent neural networks. In International Conference on Neuromorphic Systems, pages 1-8, 2020.
[62] Friedemann Zenke and Tim P Vogels. The remarkable robustness of surrogate gradient learning for instilling complex function in spiking neural networks. Neural Computation, 33(4):999-925, 2021.
[63] Wenrui Zhang and Peng Li. Temporal spike sequence learning via backpropagation for deep spiking neural networks. In Advances in Neural Information Processing Systems (NeurIPS), pages 10023, 2020.
[64] Hanle Zheng, Yujie Wu, Lei Deng, Yifan Hu, and Guoqi Li. Going deeper with directly- trained larger spiking neural networks. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 11062-1 1070, 2021.
[65] Shibo Zhou, Xiaohua Li, Ying Chen, Sanjeev T. Chandrasekaran, and Arindam Sanyal. Temporal-coded deep spiking neural network with easy training and robust performance. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 11 1114--1155,, 2021.
[66] Romain Zimmer, Thomas Pellegrini, Srisht Fatch Singh, and Timothée Masquelier. Technical report: supervised training of convolutional spiking neural networks with pytorch. arXiv preprint 191.12491,,,014,, 2019.
14


Reviews:
[{"review_id": "uwpA9qbzd9W", "reviewer": "Reviewer_2Q6H", "summary": "This work proposes spike-element-wise ResNet, in an effort to achieve identity mapping. The original Spiking ResNet put the element-wise operation before the activation function. However, with the LIF activation function, the activation will be binarized to spikes and thus the identity mapping is not fulfilled. Thus, the authors put the element-wise operation after the LIF activation. \n\n", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Post Rebuttal**\n\nI have seen the author's response. Overall, I still think the author's response does not fully address my first question (why not use QNN if the neuromorphic HW can support up to 8-bit spikes?). Their responses referring that 1) SNN uses binary activation, 2) multiplication-free computation and 3) higher adversarial robustness all are the property of conventional SNN rather than the multi-bit SNN. However, I also acknowledge that the authors made adequate efforts for rebuttal, which I indeed appreciate. I will increase my score to 6 and I hope authors can rigorously discuss the point of this multi-bit spike in the next version. \n\nPS: According to your baseline paper, the tdBN-Spiking-ResNet-50 has 65% accuracy while your result is 58%?\n\n-----\n\n\nOverall, the authors propose a reasonable architecture to address the residual mapping issue in the spiking networks. However, the overall algorithm seems to be a minor solution, as the only difference is the swap of two operations. This work main experiment with  ImageNet experiments, which I appreciate.  \n\n\n**Pros**:\n\n+ The topic of this paper is interesting, the residual path is an important technique in ANN, and I am happy to see a paper study this architecture in SNN. \n\n+ The experiments in ImageNet show improvements. \n\n**Cons**:\n\n- My biggest concern of this work is the non-binary spikes in the network. Although the authors propose two variant element-wise ops: namely AND and IAND, they finally adopt the ADD operation, which means the spikes after the operation is not binarized. And the bit-width that is used to represent this spike will continue to increase. This seems to be an undesired characteristic in SNN since the neuromorphic hardware operates with binary activation. Thus the energy consumption may be significantly higher than the spiking resnet.\n\n- In my view, the spiking resnet is more biology-plausible. Is there any biological evidence that two spikes can be fused together? ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "My concerns are listed above. In my opinion, it would be natural that multi-bit spikes outperform binary spikes, therefore I would reject this submission for now. If the authors can clarify this, I will change my scores. ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}, {"review_id": "N3HH0g77cwc", "reviewer": "Reviewer_Up2E", "summary": "In this paper, the authors proposed the spike-element-wise (SEW) ResNet to train deep SNNs with ResNet modules.  They achieved very excellent results compared to the SOTA. Although simple, the proposed methods turned out to be very effective in the practice. ", "questions": "", "limitations": "", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "1. The methods and results were written clearly except for some minor grammar issues. Please check more carefully in the later version. \n\n2. As the authors also state in lines 166-176, the SEW block is an extension of the RBA block. Thus, it suffers the same shortcoming that it leads to a non-negative output from the transform $\\mathcal{F}$, which would impact the representational ability. \n\n3. For the comparison with other methods, the simulation for SEW is with T=4. Is there any reason for not using T=6, the same as Spiking ResNet-34? Does the proposed method require more memory for training? Another issue here is that the references to the ANN2SNN methods are not accurate here. For example, Deng & Gu, 2021 ICLR can get 67.54% with T=128 and Li, 2021 ICML can get 71.12\\% with T = 64. \n\n4. The demonstration of gradient amplitude is a bit misleading. I suggest that the authors subtract the gradient calculated by $S_{j}^l[t]=1$ and $\\sigma'(1-V_{th})=1$ out so that $0$ indicates vanishment. \n\n5. The problem of SpikeNet may be relieved by choosing surrogate gradients for the identity map. This setup with the same training strategy as the SEW-Net can serve as a benchmark to compare as well. \n\n6. It would be more informative to see further results on DVS-CIFAR-10. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Not included. The authors may add a paragraph to discuss the limitation and potential negative societal impact of their work. ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "10"}, "extra_text": {}}, {"review_id": "CuZrrakfhB", "reviewer": "Reviewer_uG22", "summary": "The present paper is concerned about a residual architecture for spiking neural networks. First, the authors point out two drawbacks of the existing Spiking ResNet. The first drawback is that Spiking ResNet cannot easily represent identity mapping for some neuronal dynamics. The second drawback is that even if a block of Spiking ResNet represents the identity mapping, the gradient of its output with respect to the input is not equal to 1, and therefore, a sequence of Spiking ResNet blocks suffers from vanishing or exploding gradients. These two drawbacks hinder us from training a deep Spiking ResNet.\nThe authors present the spike-element-wise (SEW) residual block to address the drawbacks mentioned above. Since the drawbacks are caused by the architecture where the skip connection is fed into a spiking neuron, the idea is to skip the spiking neuron and directly add/multiply/IAND the residual mapping and the input spike sequence. The authors discussed that the SEW block can address the drawbacks.\nThe authors present two sets of experiments, one uses ImageNet and the other uses DVS Gesture data set. The experimental results suggest that the proposed SEW ResNet successfully addresses the drawbacks and the deeper the network is the better the performance is.\n", "questions": "", "limitations": "", "rating": 7, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I cannot judge whether the proposed architecture is novel or not, because I am working on a learning algorithm of SNN rather than its architecture. However, provided that the proposed architecture is novel, I think this paper is good because it clearly states the drawbacks of the existing method and the proposed method successfully addresses it.\n\nOne minor concern is that there are many typos;\n- l.271 \"indicating that indicating that\" -> \"indicating that\"\n- l.275 \"Fig. 7 shows\" -> \"Fig. 8 shows\" ?\n- l.276 \"Fig. 8\" -> \"Fig. 6\" ?\nto name a few. I would recommend the authors to go through the paper to improve the quality of the paper.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "N/A", "review_text": "", "extra_scores": {"ethical_concerns": "N/A", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4 hours"}, "extra_text": {}}, {"review_id": "A-INdh9qIj5", "reviewer": "Reviewer_96mQ", "summary": "This paper wants to apply the residual block into deep spiking neural networks to achieve better classification accuracy. They proposed a spike element-wise (SEW) to realize residual learning. This top and idea are very important in training a robust deep spiking neural network and the residual block is approved useful in the deep learning field. The authors evaluated the SEW on several datasets such as ImageNet and DVS gesture datasets, the experimental results show the proposed method can achieve better performance. \n", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "A spiking neural network (SNN) is considered a promising approach to build an intelligent machine as efficient as the human brain. This paper investigated the learning of deep SNN by converting residual network which allowed a network to go extremely deep and has achieved great success in pattern classification tasks. This paper analyzed the drawbacks of some current spiking resnet methods, aiming at this, the paper introduced SEW block as formulated as equation 9. Through the identity mapping with ADD, AND, and IAND, the binary spiking signals could be calculated and communicated between neural layers. \n\nDifferent from ANN2SNN methods, this paper applied a spiking bp-based method to training the proposed network model. Table 3 shows the experimental results on ImageNet, the authors reported impressive results that the classification accuracy is positive with fewer time steps. \n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "This paper still existed some problems that I hope the authors could illustrate in a clearer way. \n\n1.\tThe authors argued that they were the first time to directly training deep SNNs with more than 100 layers. I don’t think this is the core contribution in this paper, because of the residual block, the spiking could be deeper. In my opinion, SEW structure is the most important point in this paper, and directly training a 50-layer and 100-layer snn is not a huge breakthrough. Otherwise, if they could give a more detailed analysis about why other methods can’t train a 100-layer snn except section 3.2, it may be more reasonable. \n2.\tWhy the RBA block can be seen as a special case of the SEW block? I mean SEW is another kind of RBA with binary input and output. \n3.\tEqu. 11 is wonderful, how about other bit operations?\n4.\tFig. 5 a seems strange, please give more explanations. \n5.\tWhen the input is aer format, how did you deal with DVS input? \n6.\tIf you can analyze the energy consumption as reference[15] did, this paper would be more solid. \n", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "1"}, "extra_text": {}}]

Author rebuttal:

Meta-review: By allowing multi-bit spiking, a novel residual block structure is proposed for neuromorphic deep learning. Thorough experiments were run to compare performance and power consumption. Reviewers generally agreed that this contribution is interesting for the spiking neural network community. They also agreed that the writing can be improved. Please make sure to include the results included in the rebuttal in the main paper as well.
Decision: Accept (Poster)