A SPIKING SEQUENTIAL MODEL: RECURRENT LEAKY INTEGRATE-AND-FIRE

1 INTRODUCTION . The terms of deep learning and the corresponding artificial neural networks ( ANNs ) derivatives have been dominating in subject of computer science and keep the current state-of-the-art performance in a widespread of machine learning ’ s application scenario such as computer vision ( Simonyan & Zisserman , 2014 ) , natural language processing ( Collobert & Weston , 2008 ) , speech/audio recognition ( Hinton et al. , 2012 ) , video understanding ( Ye et al. , 2015 ) since the first arising of the AlexNet ( Krizhevsky et al. , 2012 ) , even some of them has beat the humans ’ cognitive level in certain tasks . However , ANNs fail to uptake the advantages of the Neuronal Dynamics , which instantiates as high-power consumption , relatively low responses and etc . Spiking Neuron Networks ( SNNs ) ( Maass , 1997 ) , with inspiration for the propagation of the cortex neurons ( Perrett et al. , 1982 ; Tuckwell , 1988 ) , have been presented continuous attention as a new , power-efficient and hardware friendly technology . In contrast to the mere implementation of spatial information and complicated float point computation of ANNs , SNNs utilize spatial-temporal dynamics to mimic the bio-behavior of neurons , as well as its dyadic-valued computation whose feeding electrical sequential impulses ( i.e. , spikes ) , belong to the binary-like set of { 0,1 } . Benefit from the capabilities of processing binary-spiking signal and consequential effectiveness , there is an alternative for SNNs that has a feasibility of further development of machine learning and neuromorphic application , which has been long-term significantly deployed in many neuromorphic hardware including SpiNNaker ( Furber et al. , 2014 ) , TrueNorth ( Akopyan et al. , 2015 ) and Loihi ( Davies et al. , 2018 ) . In contrast to the ANNs ’ well advanced , salient , proficient training methodology that indicate the conception of BackPropagation ( BP ) ( LeCun et al. , 1998 ) along with its derivatives that consequently give rise to the convergence of ANNs and diverse categories of frameworks ( ie . TensorFlow , PyTorch , et al . ) that make it succinct and available to train more deeper networks . However , for one thing , there are not so much theoretically supported or potent procedure for tackling the issue of training SNNs , which limits SNNs from going deeper , therefore SNNs hardly fulfill the ability in real-world complex missions , such as video-based recognition/detection , natural language pro- cessing et al .. For another thing , there no exit practical auxiliary frameworks that are capable to promote the mature structure of SNNs , which leads to the consequence of few application and rare forward-step development of SNNs . There are still various efforts to make progress in training , deepening the depth and applications of SNNs , whereas many obstacles block the development of SNNs at the same time . As for training , there are many circumvention ways to strengthen the accuracy of SNNs , except for neuromorphic methodology such as spike-timing-dependent plasticity ( STDP ) ( Serrano-Gotarredona et al. , 2013 ) , winner-taken all ( WTA ) ( Makhzani & Frey , 2015 ) . In the first alternative scheme , an ANN is trained firstly , then it is transformed into the SNN version whose network structure is the same as the abovementioned ANN , and neurons analog the behavior of ANN neurons ( Diehl et al. , 2015 ) . The other is the direct supervised learning , also called Gradient descend , which is a superior , prevalent optimization method for this learning procedure . In order to solve the issue of the non-differential problems of spikes , ( Lee et al. , 2016 ) proposed an alternate that treats membrane potential as differential signals and directly uses BP algorithm to train deep SNNs . To act as more bio-behavior , ( Ponulak & Kasiński , 2010 ) introduced the remote supervised STDP-like rule to be capable of the learning of sequential output spike . Besides , ( Urbanczik & Senn , 2009 ) proposed a novel leaning rule whose information will be embedded into the spatio-temporal information during learning of the spike signals . Nevertheless , most of the learning methods presented above are merely engaged in a single aspect of either spatial or temporal information . The applications started to spring up due to the incoming of the event-based cameras composed of Dynamic Visual Sensors ( DVS ) ( Shi et al. , 2018 ) . The mechanism of DVS can be outlined as a simulation of the visual path structures and functionalities of the biological visual systems whose neurons asynchronously communicate and encode the visual information from environment as spatiotemporally sparse light intensity change in the form of spikes . On the strength of the event-based cameras , diverse event-based datasets were acquired such as Poker-DVS , MNIST-DVS ( Serrano-Gotarredona & Linares-Barranco , 2015 ) and CIFAR10-DVS ( Wu et al. , 2019 ) . Embracing the event-based cameras and their derived datasets , a variety of monographs demonstrate the different methodologies whose intentions are to make a plausibility of the application of accordingly components . ( Peng et al. , 2016 ) proposed an event-based classification based on static learning method , named Bag of Events ( BOE in short ) . This method denotes the events of corresponding to the activated pixel of the DVS as joint probability distribution . Moreover , this method tests on multiple datasets such as NMNIST , MNIST-DVS , Poker-DVS , and it reveals that BOE can significantly achieve competitive results in real-time for feature extraction and implementation time as well as the classification accuracy . ( Neil & Liu , 2016 ) proposed a deep CNN to pre-process spiking data from DVS , which is used in various deep network architecture and is also used to achieve an accuracy of 97.40 % on N-MNIST datasets , in spite of its complicated pre-processing approach . In terms of SNNs , ( Indiveri et al. , 2015 ) proposed a SNN architecture , named Feedforward SNN , which is based on spike-based learning and temporary learning , and it achieves 87.41 % accuracy on MNIST-DVS datasets . ( Stromatias et al. , 2015 ) proposed a composite system , including convolutional SNNs , non-spiking fully connected classifier , and spiking output layer with its performance of 97.95 % of accuracy . Together with improving the performance and enhancing the convergence rate of SNNs , the goal that whether a method that can absorb both advantages of ANNs and SNNs can be achieved . To this end , we propose RLIF with both low computational complexity and biological plausibility , to explore its usage in real-world tasks . In summary , the major contributions of this paper can be listed as follows : • We propose RLIF , which absorbs the biological traits from SNNs , follows the unroll structure of RNNs , and enables a seamless way to insert into any sequential model in common deep learning frameworks . • A mass throughput can be implemented through the transition of binary information be- tween an interlayer of RLIF and other sequential layers , which meets the basic principle that the emission of neuron trains are binary values . Furthermore , RLIF can be easily extended into neuromorphic chips since its peculiarity of hardware-friendly . • The experiments conducted in general DVS-based datasets ( MNIST-DVS , CIFAR10-DVS ) and Chinese text summarization ( LCSTS-2.0 ) show that our RLIF is capable of capturing key information through time and has lower parameters compared to its counterparts . 2 PREMISE OF UNDERSTANDING RLIF . As mentioned before , the core idea in our architecture is about how to absorb the biological traits of SNN into RNN . To this end , learning algorithm in SNN will be introduced first and then we do a simple analysis on basic LIF neuron model , which aims to highlight the most relevant parts to our RLIF . 2.1 LEARNING ALGORITHM FOR SNN . To the best of our knowledge , the learning algorithm for SNN could be divided into two categories : i ) unsupervised learning algorithms represented by spike timing dependent plasticity ( STDP ) and ii ) direct supervised learning algorithms represented by gradient-based backpropagation . Classical STDP and its reward-modulated variants ( Legenstein et al. , 2008 ; Frémaux & Gerstner , 2016 ) , the typical SNN learning method which only use the local information to update the weights of model , surrender to difficulties in the convergence of models with many layers on complex datasets ( Masquelier & Thorpe , 2007 ; Diehl & Cook , 2015 ; Tavanaei & Maida , 2016 ) . Illuminated by observing the huge success of backpropagation in ANN , researchers start to explore a new way about how can backpropagation be used in training SNN under the end-to-end paradigm . ( Lee et al. , 2016 ; Jin et al. , 2018 ) have introduce spatial backpropagation method into training SNN which mainly based on conventional backpropagation . As to imitate the temporal characteristics of SNN , ( Wu et al. , 2018 ) pioneered the use of backpropagation in both spatial and temporal domains to train SNN directly , through which it achieved the state-of-the-art accuracy on MNIST and N-MNIST datasets . ( Huh & Sejnowski , 2018 ) introduce a differentiable formulation of spiking dynamics and derive the exact gradient calculation to achieve this and ( Neftci et al. , 2019 ) use surrogate gradient methods to conquer the difficulties associated with the discontinuous nonlinearity . As a step further to increase the speed of training , ( Wu et al. , 2019 ) convert the leaky integrateand-fire ( LIF ) model into an explicitly iterative version so as to train deep SNN with tens of times speedup under backpropagation through time ( BPTT ) . 2.2 LIF NEURON MODEL . Leaky Integrateand-Fire ( LIF ) is the most common and simple model which can modeling neuron operations and some basic dynamic traits effectively with low computational costs . In general , we describe LIF neuron ( layer l and index i ) in differential form as τmem dU li dt = − ( U li − Urest ) +RI li ( 1 ) where Ui refers to the membrane potential , Urest is the resting potential , τmem is the membrane time constant , R is the input resistance , and Ii is the input current ( Gerstner et al. , 2014 ) . When the membrane voltage of neuron reaches it firing threshold ϑ , spikes was released to communicate their output to other neurons . After each spike , Ui is reset to the original resting potential Urest . since the input current is typically generated by synaptic currents triggered by the arrival of presynaptic spikes Slj , ( Neftci et al. , 2019 ) model the dynamics of operations during approximating the time course as an exponentially decaying current following each presynaptic spike by dI li dt = − I l i τsyn︸ ︷︷ ︸ decay + ∑ j W lij · Sl−1j︸ ︷︷ ︸ feed forward + ∑ j V lij · Slj︸ ︷︷ ︸ recurrent ( 2 ) Based on this , the simulation of single LIF neuron can be decomposed into solving two linear differential equations . As RNN , who accepts both the current input xt and the previously hidden state ht−1 and updates the current state via non-linear activation function σ ( ... ) , the basic form is yt = σ ( Wx · xt +Wh · ht−1 + b ) ( 3 ) Apparently , Equation 2 has the similar structure with basic RNN , which provides an insight about paraphrasing LIF into recurrent paradigm .

Summary: Recently, it has been shown that spiking neural networks (SNN) can be trained efficiently, in a supervised manner, using backpropagation through time. Indeed, the most commonly used spiking neuron model, the leaky integrate-and-fire neuron (LIF), obeys a differential equation which can be approximated using discrete time steps, leading to a recurrent relation for the potential. The firing threshold causes a non-differentiability issue, but it can be overcome using a surrogate gradient. In practice, it means that SNNs can be trained on GPUs using standard deep learning frameworks such as PyTorch or TensorFlow.