Accelerating Training of Deep Spiking Neural Networks with Parameter Initialization

1 INTRODUCTION . To date , deep learning has contributed to outstanding performance on various tasks ( Hinton et al. , 2012 ; Hosu & Rebedea , 2016 ; He et al. , 2016 ; LeCun et al. , 2015 ) . Most state-of-the-art models are based on analog neurons . Each analog value conveys activation ; thus , the inference processes of analog neural networks ( ANNs ) consist of massive matrix manipulations . To overcome this , spiking neural networks ( SNNs ) are increasing attracting from researchers ( Maass , 1997 ; Maass & Markram , 2004 ; Yang et al. , 2019 ) . The spiking neurons in an SNN emit sparse 0,1 spikes when activated . Compared with existing ANNs , SNNs are considered to encode spatiotemporal patterns and have better power efficiency . Hence , implementations of SNNs are highly appealing in powerlimited scenarios . In addition , SNNs have exhibited computational capabilities as powerful as those of conventional artificial neural networks ( Lee et al. , 2016 ) . Despite SNNs ’ powerful capabilities , these networks lack efficient supervised learning algorithms . The discrete spikes prevent the direct use of backpropagation ( BP ) training . Recently proposed supervised algorithms have mainly focused on two methodologies : ANN-SNN conversion and BP through time ( BPTT ) with surrogate functions . Compared to ANN-SNN conversion ( Diehl et al. , 2015 ; Sengupta et al. , 2019 ; Deng & Gu , 2021 ; Ho & Chang , 2020 ; Ding et al. , 2021 ) , BPTT is more friendly to the event-based datasets ( Lee et al. , 2016 ; Zheng et al. , 2021 ; Neftci et al. , 2019 ) . However , Wu et al . ( 2018 ) has reported that BPTT is only feasible to shallow architectures . Therefore , BPTT still can not be regarded as a mature SNN training algorithm . We ascribe the immaturity of BPTT training to the influence of the gradients . One common reason is gradient vanishing and explosion . This is natural since SNNs can be regarded as a type of recurrent neural networks ( RNNs ) ( He et al. , 2020 ; Fang et al. , 2020 ) . In the wisdom of RNN methodology , researchers have managed to provide a simple solution by rethinking weight initialization ( Le et al. , 2015 ) . The success of RNN training inspires consideration of whether there may also be a simple solution for SNNs . Another reason is early gradient generation , which has rarely been mentioned in the previous literature . Most surrogate functions produce gradients when the membrane potential is in the neighborhood of the threshold . Some produce gradients only when spikes are fired ( Lee et al. , 2020a ) . Thus , the flow of the gradients strongly depends on the number of firing neurons and will influence the optimization of the network . A small experiment can help illustrate how early gradient generation affects training . As shown in Fig . 1 , with an inadequate weight initialization method , the neurons receive inputs that are too small and can not fire initially , leading to stall of the training process because gradients fail to be produced . Until some neurons fire , the loss does not begin to decrease , and the neurons must suffer a latter uncoordinated training schedule . Therefore , in general , good weight initialization is also important for efficient SNN training . At present , initialization for SNN training is mainly achieved by following and adapting methods originally developed for ANNs . However , unlike the rectified linear unit ( ReLU ) activation function conventionally used in ANNs , spiking neurons respond very differently , and recent improvements in models have rarely focused on the unique properties of spiking neurons . In this paper , we investigate the theoretical response of spiking neurons to develop a better SNN initialization strategy . The main contributions of this paper are summarized as follows : • We use iterative systems to model first-order integrate-and-fire neurons and theoretically investigate the response curve . A slant asymptote is derived to approximate the actual neuron response . • We examine the difficulty of training deep SNNs and develop an initialization method based on the slant asymptote . The proposed initialization method relies on adequate scaled weights and nonzero biases to enable early gradient training . • Experiments with rate coding and event coding in classification tasks show that our method can effectively improve the model training speed and accuracy . In addition , empirical analyses suggest that our asymptote initialization method offers versatility and superiority over a vast range of training hyperparameters , such as different neuron types , surrogate functions , and optimizers . 2 THEORETICAL ANALYSIS OF SNNS . 2.1 SPIKING NEURONS AND SURROGATE FUNCTIONS . With the development of neuroscience , many neuron models have been proposed ( Hodgkin & Huxley , 1952 ; Gerstner et al. , 2014 ) , commonly in the form of differential systems . However , in the field of computing , analyses are often based on first-order differential models because of computational efficiency . Typically , leaky and nonleaky integrate-and-fire ( LIF and IF ) models can be formulated in terms of the dynamics of the membrane potential v as shown in Eq . 1 ( a ) and ( b ) . τ dv ( t ) dt = − ( v ( t ) − vreset ) +Ri ( t ) ( a ) τ is time constant . di ( t ) dt = 1 C i ( t ) ( b ) ( 1 ) By utilizing the Euler method , we can convert these dynamic systems into an iterative expression that computers prefer ( Eq . 2 ) . Note that vreset is usually 0 in the literature ; hence , it is neglected in latter analyses . When an element of vt exceeds the firing threshold θ , the corresponding neuron will emit a spike of 1 , and membrane potential will then be reset to 0 . Without loss of generality , we use the following equations to describe the iterative systems for first-order integrate-and-fire neurons . ut = kvt + λit ot = ut > θ vt+1 = ut ( 1− ot ) ( 2 ) In Eq . 2 , k and λ are hyperparameters that control the neuron type and firing pattern ( k ∈ ( 0 , 1 ] ) . When k is set to 0 , the iteration of the membrane potential over time is canceled , and the network is not a standard SNN . Many neuron models used in the existing literature on SNN training can be regarded as the model given in Eq . 2 with different settings of the two parameters . Table 1 illustrates some of these settings . In Eq . 2 , ot = ut > θ is the non-differentiable part . It can also be rewritten as ot = H ( ut − θ ) where H ( · ) is the Heaviside step function . H ( x ) = 0 when x ≤ 0 , and H ( x ) = 1 when x > 0 . The heaviside function has a gradient of 0 almost everywhere . Surrogate function methods first emerged in network quantization to cope with non-differentiable problems ( Mairal , 2013 ; Hubara et al. , 2016 ) . Accordingly , four surrogate functions ( sigmoid , tanh , hardtanh , arctan ) are presented in this paper , parameterized by the parameter a , as listed in Table A3 . These functions are the shifted and scaled version of the original functions , as shown in Fig . A7 in the appendix . The parameter a can be adjusted to rescale the output gradient . The gradient functions are typically symmetric w.r.t . an input of zero . An exception has been proposed by Lee et al . ( 2020a ) , such that only the production of spikes can trigger nonzero gradients . 2.2 RESPONSE OF SPIKING NEURONS AND ASYMPTOTES . It is not easy to establish the relationship between the current input and the output based on an iterative equation . Considering that the difference between the iterative equation and the differential equation is small when stepsize = 1 , and that the differential equation can be integrated to obtain the response curve , Theorem 1 gives the response of a first-order integrate-and-fire neurons . Theorem 1 . For a first-order integrate-and-fire neuron as expressed in Eq . 2 ( ut = kvt + λit , k ∈ ( 0 , 1 ] ) , the model takes a constant current it = i as input and produces spikes as output . Let the discrete output firing rate w.r.t . i be expressed as f = TN , where T is the total inference time and N is the spike count . Then , the input-output response curve ( f -i curve ) is estimated as follows : If k = 1 : f = 0 , i ≤ 0 , λ θ i , 0 < i < θ λ , 1 , i ≥ θλ . If k ∈ ( 0 , 1 ) : f = 0 , i ≤ 0 , ( k − 1 ) / ln ( 1 + ( k−1 ) θλi ) , 0 < i < θ λ · k−1 ek−1−1 , 1 , i ≥ θλ · k−1 ek−1−1 . ( 3 ) The solid lines in Fig . 2 represent the response curves when k = 1 and k ∈ ( 0 , 1 ) in Theorem 1 . Fig . 2 also presents scatter plots of the actual response of a spiking neuron given the corresponding parameters ( marked with circular dots ) . The actual neuron responses match the theoretical f − i curve for most inputs . Consider the region in which f ∈ ( 0 , 1 ) . When k ∈ ( 0 , 1 ) , it can be intuitively found that the f -i curve , which has a non-differentiable point , seems to exhibit a slant asymptote . Theorem 2 manifests the existence of this slant asymptote and provides a uniform expression for it . In Fig . 2 , this asymptote approximately crosses the actual neuron response distribution . In this case , the slant asymptote of f -i can better describe the relationship between the expected input current and the output firing rate . Theorem 2 . For a first-order integrate-and-fire neuron as expressed in Eq . 2 ( ut = kvt + λit , k ∈ ( 0 , 1 ] ) , the slant asymptote of the f -i curve satisfying f ∈ ( 0 , 1 ) exists and can be uniformly expressed as follows : fa = λ θ i+ k − 1 2 . ( 4 ) 3 WEIGHT INITIALIZATION FOR SNN TRAINING . The gradient of the weights is related to the spike output . To better illustrate this , given an L-layered SNN , we denote the input current , membrane potential , intermediate variable , and spike output of the lth layer by il , t , vl , t , ul , t , and ol , t , respectively . The variables for the neurons in one layer satisfy Eq . 2. il , t is the weighted input consisting of the previous layer ’ s spike output ( il , t = W lol−1 , t+bl ) , and the first layer takes the encoded spike o0 , t as input . When we backpropagate the loss ε through time T , the gradients w.r.t . the weights of the lth layer can be expressed as shown in Eq . 5 . An inappropriate ol−1 , t , such as an output of zero , will lead to a poor gradient . ∂ε ∂W l = T∑ t=1 ∂ε ∂ul , t ∂ul , t ∂il , t ∂il , t ∂W l = λ T∑ t=1 ∂ε ∂ul , t ( ol−1 , t ) T ( 5 ) Therefore , the main concern is how to produce spikes to trigger the learning process . Thankfully , the clipped version of the slant asymptote can replace the analysis of complex iterative functions , allowing us to focus on the scale of the parameters : îl = W lôl−1 + bl ôl = clip ( λ θ îl + k − 1 2 , 0 , 1 ) ( 6 ) where îi and ôi are the expected current and firing rate vector , respectively , of the ith layer with ni neurons and W i and bi are the corresponding parameters . Note that W i is an ni × ni−1 matrix . λ , k and θ represent the neuron parameters . First , all weights in the SNN are initialized to be independent and identically distributed ( i.i.d. ) . Let us assume that the input current variances are the same . Then , we have : Var ( ̂i l ) = nl−1Var ( W lôl−1 ) ( 7 ) where î l , Wl and ôl−1 denote random variables of elements in îl , W l and ôl−1 , respectively . Considering Var ( Wlôl−1 ) , Var ( Wlôl−1 ) = Var ( Wl ) E [ ( ôl−1 ) 2 ] + Var ( ôl−1 ) E [ Wl ] 2 , let Wl be of zero mean ; then : Var ( ̂i l ) = nl−1Var ( W l ) E [ ( ôl−1 ) 2 ] . ( 8 ) In the function ô = clip ( λθ î+ k−1 2 , 0 , 1 ) , i = θ ( 1−k ) 2λ is the transition point between the nonspiking region and the linear response region . When bl is initialized as θ ( 1−k ) 2λ ( l = 1 , 2 , · · · , L ) , a zerocentered W l × ôl−1 produces a symmetric distribution around zero . With bl initialized as θ ( 1−k ) 2λ , E [ ( ôl−1 ) 2 ] = 1 2 Var ( λ θ î l−1 ) = 1 2 λ2 θ2 Var ( ̂i l−1 ) . ( 9 ) By plugging E [ ( ôl−1 ) 2 ] into Eq . 8 , we can obtain : Var ( ̂i l ) = 1 2 λ2 θ2 nl−1Var ( W l ) Var ( ̂i l−1 ) ( 10 ) To produce spikes is to maintain E [ ( ôl ) 2 ] , which is related to Var ( ̂i l ) according to Eq . 9 . Therefore , we determine the distribution of Wl as follows ( for l = 2 , 3 , · · · , L ) : Var ( ̂i l ) Var ( ̂i l−1 ) = 1 2 λ2 θ2 nl−1Var ( W l ) = 1 ( 11 ) Accordingly , W l is initialized from a Gaussian distribution N ( 0 , 2θ 2 λ2nl−1 ) , where nl−1 is the number of input neurons in layer l. As done in other initializers , we apply the initialization to the first layer for simplicity . In previous work on ANNs , there is no need for the neurons to accumulate input . Thus , the biases are often set to zero in an ANN . For SNNs , because the response asymptote indicates a nonzero nondifferentiable point , it is natural for the biases in an SNN to be initialized to that non-differentiable value . We validate the impact to the mean firing rate of our initialization method in Fig . 3 . Our method can effectively trigger spikes when compared with ANN-based methods .

Summary: The paper investigates the effects of weight initialization for deep spiking neural networks (SNNs) on training speed and final accuracy. The main finding is that popular initialization methods for conventional ANNs and RNNs do not match the specific dynamics of SNNs, in particular they ignore the need to have sufficient firing early on to generate good gradients. The paper derives a theoretical first-order approximation on the response curve, and from it a weight initialization scheme that initializes the weights in a region where neurons fire from the first epoch on, but also avoid explosion or reduction of activities throughout the network. In various experiments on standard and spiking variants of MNIST and CIFAR, the initialization scheme shows good performance and fast convergence for different neuron types (e.g. non-leaking vs. fast leaking), and different encoding schemes. For CIFAR10 the results are significantly better than for other initialization schemes.