Deep Residual Learning in Spiking Neural Networks

Deep Spiking Neural Networks ( SNNs ) present optimization difficulties for gradient-based approaches due to discrete binary activation and complex spatialtemporal dynamics . Considering the huge success of ResNet in deep learning , it would be natural to train deep SNNs with residual learning . Previous Spiking ResNet mimics the standard residual block in ANNs and simply replaces ReLU activation layers with spiking neurons , which suffers the degradation problem and can hardly implement residual learning . In this paper , we propose the spikeelement-wise ( SEW ) ResNet to realize residual learning in deep SNNs . We prove that the SEW ResNet can easily implement identity mapping and overcome the vanishing/exploding gradient problems of Spiking ResNet . We evaluate our SEW ResNet on ImageNet , DVS Gesture , and CIFAR10-DVS datasets , and show that SEW ResNet outperforms the state-of-the-art directly trained SNNs in both accuracy and time-steps . Moreover , SEW ResNet can achieve higher performance by simply adding more layers , providing a simple method to train deep SNNs . To our best knowledge , this is the first time that directly training deep SNNs with more than 100 layers becomes possible . Our codes are available at https : //github.com/fangwei123456/Spike-Element-Wise-ResNet . 1 Introduction . Artificial Neural Networks ( ANNs ) have achieved great success in many tasks , including image classification [ 28 , 52 , 55 ] , object detection [ 9 , 34 , 44 ] , machine translation [ 2 ] , and gaming [ 37 , 51 ] . One of the critical factors for ANNs ’ success is deep learning [ 29 ] , which uses multi-layers to learn representations of data with multiple levels of abstraction . It has been proved that deeper networks have advantages over shallower networks in computation cost and generalization ability [ 3 ] . The function represented by a deep network can require an exponential number of hidden units by a shallow network with one hidden layer [ 38 ] . In addition , the depth of the network is closely related to the network ’ s performance in practical tasks [ 52 , 55 , 27 , 52 ] . Nevertheless , recent evidence [ 13 , 53 , 14 ] reveals that with the network depth increasing , the accuracy gets saturated and then degrades rapidly . To solve this degradation problem , residual learning is proposed [ 14 , 15 ] and the residual structure is widely exploited in “ very deep ” networks that achieve the leading performance [ 22 , 59 , 18 , 57 ] . Spiking Neural Networks ( SNNs ) are regarded as a potential competitor of ANNs for their high biological plausibility , event-driven property , and low power consumption [ 45 ] . Recently , deep learning methods are introduced into SNNs , and deep SNNs have achieved close performance as ANNs in some simple classification datasets [ 56 ] , but still worse than ANNs in complex tasks , e.g. , classifying the ImageNet dataset [ 47 ] . To obtain higher performance SNNs , it would be natural to ∗Corresponding author 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . explore deeper network structures like ResNet . Spiking ResNet [ 25 , 60 , 21 , 17 , 49 , 12 , 30 , 64 , 48 , 42 , 43 ] , as the spiking version of ResNet , is proposed by mimicking the residual block in ANNs and replacing ReLU activation layers with spiking neurons . Spiking ResNet converted from ANN achieves state-of-the-art accuracy on nearly all datasets , while the directly trained Spiking ResNet has not been validated to solve the degradation problem . In this paper , we show that Spiking ResNet is inapplicable to all neuron models to achieve identity mapping . Even if the identity mapping condition is met , Spiking ResNet suffers from the problems of vanishing/exploding gradient . Thus , we propose the Spike-Element-Wise ( SEW ) ResNet to realize residual learning in SNNs . We prove that the SEW ResNet can easily implement identity mapping and overcome the vanishing/exploding gradient problems at the same time . We evaluate Spiking ResNet and SEW ResNet on both the static ImageNet dataset and the neuromorphic DVS Gesture dataset [ 1 ] , CIFAR10-DVS dataset [ 32 ] . The experiment results are consistent with our analysis , indicating that the deeper Spiking ResNet suffers from the degradation problem — the deeper network has higher training loss than the shallower network , while SEW ResNet can achieve higher performance by simply increasing the network ’ s depth . Moreover , we show that SEW ResNet outperforms the state-of-the-art directly trained SNNs in both accuracy and time-steps . To the best of our knowledge , this is the first time to explore the directly-trained deep SNNs with more than 100 layers . 2 Related Work . 2.1 Learning Methods of Spiking Neural Networks . ANN to SNN conversion ( ANN2SNN ) [ 20 , 4 , 46 , 49 , 12 , 11 , 6 , 54 , 33 ] and backpropagation with surrogate gradient [ 40 ] are the two main methods to get deep SNNs . The ANN2SNN method firstly trains an ANN with ReLU activation , then converts the ANN to an SNN by replacing ReLU with spiking neurons and adding scaling operations like weight normalization and threshold balancing . Some recent conversion methods have achieved near loss-less accuracy with VGG-16 and ResNet [ 12 , 11 , 6 , 33 ] . However , the converted SNN needs a longer time to rival the original ANN in precision as the conversion is based on rate-coding [ 46 ] , which increases the SNN ’ s latency and restricts the practical application . The backpropagation methods can be classified into two categories [ 26 ] . The method in the first category computes the gradient by unfolding the network over the simulation timesteps [ 31 , 19 , 58 , 50 , 30 , 40 ] , which is similar to the idea of backpropagation through time ( BPTT ) . As the gradient with respect to the threshold-triggered firing is non-differentiable , the surrogate gradient is often used . The SNN trained by the surrogate method is not limited to rate-coding , and can also be applied on temporal tasks , e.g. , classifying neuromorphic datasets [ 58 , 8 , 16 ] . The second method computes the gradients of the timings of existing spikes with respect to the membrane potential at the spike timing [ 5 , 39 , 24 , 65 , 63 ] . 2.2 Spiking Residual Structure . Previous ANN2SNN methods noticed the distinction between plain feedforward ANNs and residual ANNs , and made specific normalization for conversion . Hu et al . [ 17 ] were the first to apply the residual structure in ANN2SNN with scaled shortcuts in SNN to match the activations of the original ANN . Sengupta et al . [ 49 ] proposed Spike-Norm to balance SNN ’ s threshold and verified their method by converting VGG and ResNet to SNNs . Existing backpropagation-based methods use nearly the same structure from ResNet . Lee et al . [ 30 ] evaluated their custom surrogate methods on shallow ResNets whose depths are no more than ResNet-11 . Zheng et al . [ 64 ] proposed the threshold-dependent batch normalization ( td-BN ) to replace naive batch normalization ( BN ) [ 23 ] and successfully trained Spiking ResNet-34 and Spiking ResNet-50 directly with surrogate gradient by adding td-BN in shortcuts . 3 Methods . 3.1 Spiking Neuron Model . The spiking neuron is the fundamental computing unit of SNNs . Similar to Fang et al . [ 8 ] , we use a unified model to describe the dynamics of all kinds of spiking neurons , which includes the following discrete-time equations : H [ t ] = f ( V [ t− 1 ] , X [ t ] ) , ( 1 ) S [ t ] = Θ ( H [ t ] − Vth ) , ( 2 ) V [ t ] = H [ t ] ( 1− S [ t ] ) + Vreset S [ t ] , ( 3 ) where X [ t ] is the input current at time-step t , H [ t ] and V [ t ] denote the membrane potential after neuronal dynamics and after the trigger of a spike at time-step t , respectively . Vth is the firing threshold , Θ ( x ) is the Heaviside step function and is defined by Θ ( x ) = 1 for x ≥ 0 and Θ ( x ) = 0 for x < 0 . S [ t ] is the output spike at time-step t , which equals 1 if there is a spike and 0 otherwise . Vreset denotes the reset potential . The function f ( · ) in Eq . ( 1 ) describes the neuronal dynamics and takes different forms for different spiking neuron models . For example , the function f ( · ) for the Integrate-and-Fire ( IF ) model and Leaky Integrate-and-Fire ( LIF ) model can be described by Eq . ( 4 ) and Eq . ( 5 ) , respectively . H [ t ] = V [ t− 1 ] +X [ t ] , ( 4 ) H [ t ] = V [ t− 1 ] + 1 τ ( X [ t ] − ( V [ t− 1 ] − Vreset ) ) , ( 5 ) where τ represents the membrane time constant . Eq . ( 2 ) and Eq . ( 3 ) describe the spike generation and resetting processes , which are the same for all kinds of spiking neuron models . In this paper , the surrogate gradient method is used to define Θ′ ( x ) , σ′ ( x ) during error back-propagation , with σ ( x ) denoting the surrogate function . 3.2 Drawbacks of Spiking ResNet . The residual block is the key component of ResNet . Fig . 1 ( a ) shows the basic block in ResNet [ 14 ] , whereX l , Y l are the input and output of the l-th block in ResNet , Conv is the convolutional layer , BN denotes batch normalization , and ReLU denotes the rectified linear unit activation layer . The basic block of Spiking ResNet used in [ 64 , 17 , 30 ] simply mimics the block in ANNs by replacing ReLU activation layers with spiking neurons ( SN ) , which is illustrated in Fig . 1 ( b ) . Here Sl [ t ] , Ol [ t ] are the input and output of the l-th block in Spiking ResNet at time-step t. Based on the above definition , we will analyze the drawbacks of Spiking ResNet below . Spiking ResNet is inapplicable to all neuron models to achieve identity mapping . One of the critical concepts in ResNet is identity mapping . He et al . [ 14 ] noted that if the added layers implement the identity mapping , a deeper model should have training error no greater than its shallower counterpart . However , it is unable to train the added layers to implement identity mapping in a feasible time , resulting in deeper models performing worse than shallower models ( the degradation problem ) . To solve this problem , the residual learning is proposed by adding a shortcut connection ( shown in Fig . 1 ( a ) ) . If we use F l to denote the residual mapping , e.g. , a stack of two convolutional layers , of the l-th residual block in ResNet and Spiking ResNet , then the residual block in Fig.1 ( a ) and Fig.1 ( b ) can be formulated as Y l = ReLU ( F l ( X l ) +X l ) , ( 6 ) Ol [ t ] = SN ( F l ( Sl [ t ] ) + Sl [ t ] ) . ( 7 ) The residual block of Eq . ( 6 ) make it easy to implement identity mapping in ANNs . To see this , when F l ( X l ) ≡ 0 , Y l = ReLU ( Xl ) . In most cases , X l is the activation of the previous ReLU layer and X l ≥ 0 . Thus , Y l = ReLU ( X l ) = X l , which is identity mapping . Different from ResNet , the residual block in Spiking ResNet ( Eq . ( 7 ) ) restricts the models of spiking neuron to implement identity mapping . When F l ( Sl [ t ] ) ≡ 0 , Ol [ t ] = SN ( Sl [ t ] ) 6= Sl [ t ] . To transmit Sl [ t ] and make SN ( Sl [ t ] ) = Sl [ t ] , the last spiking neuron ( SN ) in the l-th residual block needs to fire a spike after receiving a spike , and keep silent after receiving no spike at time-step t. It works for IF neuron described by Eq . ( 4 ) . Specifically , we can set 0 < Vth ≤ 1 and V [ t− 1 ] = 0 to ensure that X [ t ] = 1 leads to H [ t ] ≥ Vth , and X [ t ] = 0 leads to H [ t ] < Vth . However , when considering some spiking neuron models with complex neuronal dynamics , it is hard to achieve SN ( Sl [ t ] ) = Sl [ t ] . For example , the LIF neuron used in [ 66 , 8 , 61 ] considers a learnable membrane time constant τ , the neuronal dynamics of which can be described with Eq . ( 5 ) . When X [ t ] = 1 and V [ t− 1 ] = 0 , H [ t ] = 1τ . It is difficult to find a firing threshold that ensures H [ t ] > Vth as τ is being changed in training by the optimizer . Spiking ResNet suffers from the problems of vanishing/exploding gradient . Consider a spiking ResNet with k sequential blocks to transmit Sl [ t ] , and the identity mapping condition is met , e.g. , the spiking neurons are the IF neurons with 0 < Vth ≤ 1 , then we have Sl [ t ] = Sl+1 [ t ] = ... = Sl+k−1 [ t ] = Ol+k−1 [ t ] . Denote the j-th element in Sl [ t ] and Ol [ t ] as Slj [ t ] and O l j [ t ] respectively , the gradient of the output of the ( l + k − 1 ) -th residual block with respect to the input of the l-th residual block can be calculated layer by layer : ∂Ol+k−1j [ t ] ∂Slj [ t ] = k−1∏ i=0 ∂Ol+ij [ t ] ∂Sl+ij [ t ] = k−1∏ i=0 Θ′ ( Sl+ij [ t ] − Vth ) → 0 , if 0 < Θ′ ( Slj [ t ] − Vth ) < 1 1 , if Θ′ ( Slj [ t ] − Vth ) = 1 +∞ , if Θ′ ( Slj [ t ] − Vth ) > 1 , ( 8 ) where Θ ( x ) is the Heaviside step function and Θ′ ( x ) is defined by the surrogate gradient . The second equality hold as Ol+ij [ t ] = SN ( S l+i j [ t ] ) . In view of the fact that S l j [ t ] can only take 0 or 1 , Θ′ ( Slj [ t ] − Vth ) = 1 is not satisfied for commonly used surrogate functions mentioned in [ 40 ] . Thus , the vanishing/exploding gradient problems are prone to happen in deeper Spiking ResNet . Based on the above analysis , we believe that the previous Spiking ResNet ignores the highly nonlinear caused by spiking neurons , and can hardly implement residual learning . Nonetheless , the basic block in Fig . 1 ( b ) is still decent for ANN2SNN with extra normalization [ 17 , 49 ] , as the SNN converted from ANN aims to use firing rates to match the origin ANN ’ s activations .

Summary: This work proposes spike-element-wise ResNet, in an effort to achieve identity mapping. The original Spiking ResNet put the element-wise operation before the activation function. However, with the LIF activation function, the activation will be binarized to spikes and thus the identity mapping is not fulfilled. Thus, the authors put the element-wise operation after the LIF activation.