Optimal ANN-SNN Conversion for High-accuracy and Ultra-low-latency Spiking Neural Networks

Spiking Neural Networks ( SNNs ) have gained great attraction due to their distinctive properties of low power consumption and fast inference on neuromorphic hardware . As the most effective method to get deep SNNs , ANN-SNN conversion has achieved comparable performance as ANNs on large-scale datasets . Despite this , it requires long time-steps to match the firing rates of SNNs to the activation of ANNs . As a result , the converted SNN suffers severe performance degradation problems with short time-steps , which hamper the practical application of SNNs . In this paper , we theoretically analyze ANN-SNN conversion error and derive the estimated activation function of SNNs . Then we propose the quantization clipfloor-shift activation function to replace the ReLU activation function in source ANNs , which can better approximate the activation function of SNNs . We prove that the expected conversion error between SNNs and ANNs is zero , enabling us to achieve high-accuracy and ultra-low-latency SNNs . We evaluate our method on CIFAR-10/100 and ImageNet datasets , and show that it outperforms the stateof-the-art ANN-SNN and directly trained SNNs in both accuracy and time-steps . To the best of our knowledge , this is the first time to explore high-performance ANN-SNN conversion with ultra-low latency ( 4 time-steps ) . 1 INTRODUCTION . Spiking neural networks ( SNNs ) are biologically plausible neural networks based on the dynamic characteristic of biological neurons ( McCulloch & Pitts , 1943 ; Izhikevich , 2003 ) . As the third generation of artificial neural networks ( Maass , 1997 ) , SNNs have attracted great attention due to their distinctive properties over deep analog neural networks ( ANNs ) ( Roy et al. , 2019 ) . Each neuron transmits discrete spikes to convey information when exceeding a threshold . For most SNNs , the spiking neurons will accumulate the current of the last layer as the output within T inference time steps . The binarized activation has rendered dedicated hardware of neuromorphic computing ( Pei et al. , 2019 ; DeBole et al. , 2019 ; Davies et al. , 2018 ) . This kind of hardware has excellent advantages in temporal resolution and energy budget . Existing work has shown the potential of tremendous energy saving with considerably fast inference ( Stöckl & Maass , 2021 ) . In addition to efficiency advantages , the learning algorithm of SNNs has been improved by leaps and bounds in recent years . The performance of SNNs trained by backpropagation through time and ANN-SNN conversion techniques has gradually been comparable to ANNs on large-scale datasets ( Fang et al. , 2021 ; Rueckauer et al. , 2017 ) . Both techniques benefit from the setting of SNN inference time . Setting longer time-steps in backpropagation can make the gradient of surrogate functions more reliable ( Wu et al. , 2018 ; Lee et al. , 2016 ; Neftci et al. , 2019 ; Zenke & Vogels , 2021 ) . However , the price is enormous resource consumption during training . Existing platforms such as TensorFlow and PyTorch based on CUDA have limited optimization for SNN training . In contrast , ANN-SNN conversion usually depends on a longer inference time to get comparable accuracy as the original ANN ( Sengupta et al. , 2019 ) because it is based on the equivalence of ReLU activation and integrate-and-fire model ’ s firing rate ( Cao et al. , 2015 ) . Although longer inference time can further reduce the conversion error , it also hampers the practical application of SNNs on neuromorphic chips . The dilemma of ANN-SNN conversion is that there exists a remaining potential in the conversion theory , which is hard to be eliminated in a few time steps ( Rueckauer et al. , 2016 ) . Although many methods have been proposed to improve the conversion accuracy , such as weight normalization ( Diehl et al. , 2015 ) , threshold rescaling ( Sengupta et al. , 2019 ) , soft-reset ( Han & Roy , 2020 ) and threshold shift ( Deng & Gu , 2020 ) , tens to hundreds of time-steps in the baseline works are still unbearable . To obtain high-performance SNNs with ultra-low latency ( e.g. , 4 time-steps ) , we list the critical errors in ANN-SNN conversion and provide solutions for each error . Our main contributions are summarized as follows : • We go deeper into the errors in the ANN-SNN conversion and ascribe them to clipping error , quantization error , and unevenness error . We find that unevenness error , which is caused by the changes in the timing of arrival spikes and has been neglected in previous works , can induce more spikes or fewer spikes as expected . • We propose the quantization clip-floor-shift activation function to replace the ReLU activation function in source ANNs , which better approximates the activation function of SNNs . We prove that the expected conversion error between SNNs and ANNs is zero , indicating that we can achieve high-performance converted SNN at ultra-low time-steps . • We evaluate our method on CIFAR-10 , CIFAR-100 , and ImageNet datasets . Compared with both ANN-SNN conversion and backpropagation training methods , the proposed method exceeds state-of-the-art accuracy with fewer time-steps . For example , we reach top-1 accuracy 91.18 % on CIFAR-10 with unprecedented 2 time-steps . 2 PRELIMINARIES . In this section , we first briefly review the neuron models for SNNs and ANNs . Then we introduce the basic framework for ANN-SNN conversion . Neuron model for ANNs . For ANNs , the computations of analog neurons can be simplified as the combination of a linear transformation and a non-linear mapping : al = h ( W lal−1 ) , l = 1 , 2 , ... , M ( 1 ) where the vector al denotes the output of all neurons in l-th layer , W l denotes the weight matrix between layer l and layer l − 1 , and h ( · ) is the ReLU activation function . Neuron model for SNNs . Similar to the previous works ( Cao et al. , 2015 ; Diehl et al. , 2015 ; Han et al. , 2020 ) , we consider the Integrate-and-Fire ( IF ) model for SNNs . If the IF neurons in l-th layer receive the input xl−1 ( t ) from last layer , the temporal potential of the IF neurons can be defined as : ml ( t ) = vl ( t− 1 ) +W lxl−1 ( t ) , ( 2 ) where ml ( t ) and vl ( t ) represent the membrane potential before and after the trigger of a spike at time-step t. W l denote the weight in l-th layer . As soon as any element mli ( t ) of m l ( t ) exceeds the firing threshold θl , the neuron will elicit a spike and update the membrane potential vli ( t ) . To avoid information loss , we use the “ reset-by-subtraction ” mechanism ( Rueckauer et al. , 2017 ; Han et al. , 2020 ) instead of the “ reset-to-zero ” mechanism , which means the membrane potential vli ( t ) is subtracted by the threshold value θl if the neuron fires . Based on the threshold-triggered firing mechanism and the “ reset-by-subtraction ” of the membrane potential after firing discussed above , we can write the uplate rule of membrane potential as : sl ( t ) = H ( ml ( t ) − θl ) , ( 3 ) vl ( t ) = ml ( t ) − sl ( t ) θl . ( 4 ) Here sl ( t ) refers to the output spikes of all neurons in layer l at time t , the element of which equals 1 if there is a spike and 0 otherwise . H ( · ) is the Heaviside step function . θl is the vector of the firing threshold θl . Similar to Deng & Gu ( 2020 ) , we suppose that the postsynaptic neuron in l-th layer receives unweighted postsynaptic potential θl if the presynaptic neuron in l − 1-th layer fires a spike , that is : xl ( t ) = sl ( t ) θl . ( 5 ) ANN-SNN conversion . The key idea of ANN-SNN conversion is to map the activation value of an analog neuron in ANN to the firing rate ( or average postsynaptic potential ) of a spiking neuron in SNN . Specifically , we can get the potential update equation by combining Equation 2 – Equation 4 : vl ( t ) − vl ( t− 1 ) = W lxl−1 ( t ) − sl ( t ) θl . ( 6 ) Equation 6 describes the basic function of spiking neurons used in ANN-SNN conversion . By summing Equation 6 from time 1 to T and dividing T on both sides , we have : vl ( T ) − vl ( 0 ) T = W l ∑T i=1 x l−1 ( i ) T − ∑T i=1 s l ( i ) θl T . ( 7 ) If we use ϕl−1 ( T ) = ∑T i=1 x l−1 ( i ) T to denote the average postsynaptic potential during the period from 0 to T and substitute Equation 5 into Equation 7 , then we get : ϕl ( T ) = W lϕl−1 ( T ) − v l ( T ) − vl ( 0 ) T . ( 8 ) Equation 8 describes the relationship of the average postsynaptic potential of neurons in adjacent layers . Note that ϕl ( T ) ⩾ 0 . If we set the initial potential vl ( 0 ) to zero and neglect the remaining term v l ( T ) T when the simulation time-steps T is long enough , the converted SNN has nearly the same activation function as source ANN ( Equation 1 ) . However , high T would cause long inference latency that hampers the practical application of SNNs . Therefore , this paper aims to implement high-performance ANN-SNN conversion with extremely low latency . 3 CONVERSION ERROR ANALYSIS . In this section , we will analyze the conversion error between the source ANN and the converted SNN in each layer in detail . In the following , we assume that both ANN and SNN receive the same input from the layer l − 1 , that is , al−1 = ϕl−1 ( T ) , and then analyze the error in layer l. For simplicity , we use zl = W lϕl−1 ( T ) = W lal−1 to substitute the weighted input from layer l − 1 for both ANN and SNN . The absolute conversion error is exactly the outputs from converted SNN subtract the outputs from ANN : Errl = ϕl ( T ) − al = zl − v l ( T ) − vl ( 0 ) T − h ( zl ) , ( 9 ) where h ( zl ) = ReLU ( zl ) . It can be found from Equation 9 that the conversion error is nonzero if vl ( T ) − vl ( 0 ) ̸= 0 and zl > 0 . In fact , the conversion error is caused by three factors . Clipping error . The output ϕl ( T ) of SNNs is in the range of [ 0 , θl ] as ϕl ( T ) = ∑T i=1 x l ( i ) T =∑T i=1 s l ( i ) T θ l ( see Equation 5 ) . However , the output al of ANNs is in a much lager range of [ 0 , almax ] , where almax denotes the maximum value of a l. As illustrated in Figure 1a , al can be mapped to ϕl ( T ) by the following equation : ϕl ( T ) = clip ( θl T ⌊ alT λl ⌋ , 0 , θl ) . ( 10 ) Here the clip function sets the upper bound θl and the lower bound 0 . ⌊·⌋ denotes the floor function . λl represents the actual maximum value of output al mapped to the maximum value θl of ϕl ( T ) . Considering that nearly 99.9 % activations of al in ANN are in the range of [ 0 , a l max 3 ] , Rueckauer et al . ( 2016 ) suggested to choose λl according to 99.9 % activations . The activations between λl and almax in ANN are mapped to the same value θ l in SNN , which will cause conversion error called clipping error . Quantization error ( flooring error ) . The output spikes sl ( t ) are discrete events , thus ϕl ( T ) are discrete with quantization resolution θ l T ( see Equation 10 ) . When mapping a l to ϕl ( T ) , there exists unavoidable quantization error . For example , as illustrated in Figure 1a , the activations of ANN in the range of [ λ l T , 2λl T ) are mapped to the same value θl T of SNN . Unevenness error . Unevenness error is caused by the unevenness of input spikes . If the timing of arrival spikes changes , the output firing rates may change , which causes conversion error . There are two situations : more spikes as expected or fewer spikes as expected . To see this , in source ANN , we suppose that two analog neurons in layer l − 1 are connected to an analog neuron in layer l with weights 2 and -2 , and the output vector al−1 of neurons in layer l − 1 is [ 0.6 , 0.4 ] . Besides , in converted SNN , we suppose that the two spiking neurons in layer l − 1 fire 3 spikes and 2 spikes in 5 time-steps ( T=5 ) , respectively , and the threshold θl−1 = 1 . Thus , ϕl−1 ( T ) = ∑T i=1 s l−1 ( i ) T θ l−1 = [ 0.6 , 0.4 ] . Even though ϕl−1 ( T ) = al−1 and the weights are same for the ANN and SNN , ϕl ( T ) can be different from al if the timing of arrival spikes changes . According to Equation 1 , the ANN output al = W lal−1 = [ 2 , −2 ] [ 0.6 , 0.4 ] T = 0.4 . As for SNN , supposing that the threshold θl = 1 , there are three possible output firing rates , which are illustrated in Figure 1 ( b ) - ( d ) . If the two presynaptic neurons fires at t = 1 , 3 , 5 and t = 2 , 4 ( red bars ) respectively with weights 2 and -2 , the postsynaptic neuron will fire two spikes at t = 1 , 3 ( red bars ) , and ϕl ( T ) = ∑T i=1 s l ( i ) T θ l = 0.4 = al . However , if the presynaptic neurons fires at t = 1 , 2 , 3 and t = 4 , 5 , respectively , the postsynaptic neuron will fire four spikes at t = 1 , 2 , 3 , 4 , and ϕl ( T ) = 0.8 > al . If the presynaptic neurons fires at t = 3 , 4 , 5 and t = 1 , 2 , respectively , the postsynaptic neuron will fire only one spikes at t = 5 , and ϕl ( T ) = 0.2 < al . Note that the clipping error and quantization error have been proposed in Li et al . ( 2021 ) . There exist interdependence between the above three kinds of errors . Specifically , the unevenness error will degenerate to the quantization error if vl ( T ) is in the range of [ 0 , θl ] . Assuming that the potential vl ( T ) falls into [ 0 , θl ] will enable us to estimate the activation function of SNNs ignoring the effect of unevenness error . Therefore , an estimation of the output value ϕl ( T ) in a converted SNN can be formulated with the combination of clip function and floor function , that is : ϕl ( T ) ≈ θl clip ( 1 T ⌊ zlT + vl ( 0 ) θl ⌋ , 0 , 1 ) . ( 11 ) The detailed derivation is in the Appendix . With the help of this estimation for the SNN output , the estimated conversion error Ẽrr l can be derived from Equation 9 : Ẽrr l = θl clip ( 1 T ⌊ zlT + vl ( 0 ) θl ⌋ , 0 , 1 ) − h ( zl ) ≈ Errl . ( 12 )

Summary: This paper proposes a quantization clip-bottom-shift activation function to replace the ReLU activation function in ANNs, so as to better approximate the activation function of SNNs. The authors also prove that the expected error of ANN-SNN conversion can be reduced to 0 by using this method. The reported results on CIFAR-10/100 and ImageNet dataset show that this work archives state-of-the-art accuracy with fewer time-steps.