A SPIKING SEQUENTIAL MODEL: RECURRENT LEAKY INTEGRATE-AND-FIRE
Authors: Daiheng Gao
Hongwei Wang
Hehui Zhang
Meng Wang
Zhenzhi Wu
Venue:  (2020)
Keywords: spiking neural network
RNN
spiking mode
brain-inspired
text summarization
DVS

Abstract: Stemming from neuroscience, Spiking neural networks (SNNs), a brain-inspired neural network that is a versatile solution to fault-tolerant and energy efficient information processing pertains to the ”event-driven” characteristic as the analogy of the behavior of biological neurons. However, they are inferior to artificial neural networks (ANNs) in real complicated tasks and only had it been achieved good results in rather simple applications. When ANNs usually being questioned about it expensive processing costs and lack of essential biological plausibility, the temporal characteristic of RNN-based architecture makes it suitable to incorporate SNN inside as imitating the transition of membrane potential through time, and a brain-inspired Recurrent Leaky Integrate-and-Fire (RLIF) model has been put forward to overcome a series of challenges, such as discrete binary output and dynamical trait. The experiment results show that our recurrent architecture has an ultra anti-interference ability and strictly follows the guideline of SNN that spike output through it is discrete. Furthermore, this architecture achieves a good result on neuromorphic datasets and can be extended to tasks like text summarization and video understanding.

Under review as a conference paper at ICLR 2020
A SPIKING SEQUENTIAL MODEL: RECURRENT LEAKY INTEGRATE-AND-FIR
Anonymous authors Paper under double-blind review
ABSTRACT
Stemming from neuroscience, Spiking neural networks (SNNs), a brain-inspired neural network that is versatile solution to fault-tolerant and energy efficient in- formation processing pertains to the "event-driven" characteristic as the analogy of the behavior of biological neurons. However, they are inferior artificial neu- ral networks (ANNs) in real complicated tasks and only had it been achieved good results in rather simple applications. When ANNs usually being questioned about it expensive processing costs and lack of essential biological plausibility, the tem- RNN-based architecture makes it suitable to incorporate SNN inside imitating the transition of membrane potential through time, and a brain-inspired Recurrent Leaky Integrate-and-Fire (RLIF) model has been put forward to overcome series of challenges, such discrete binary output and dy- namical trait. The experiment results show that our recurrent architecture has an ultra inffinttereernce ability and strictly follows the guideline of SNN that spike output through it is discrete. Furthermore, this architecture achieves a good result on neuromorphic datasets and can be extended to tasks like text summarization and video understanding.
poral characteristic
INTRODUCTION
The terms of deep learning and the corresponding artificial neural networks (ANNs) derivatives have been dominating in subject of computer science and keep the current state-of-the-an performance in widespread of machine learning's application scenario such computer vision (Simonyan & Zisserrann 2014), natural language processing (Collobert & Weston 2008), speech/audio recog- nition (Hinton et al.] (012). video understanding (Ye et al.) (215)) since the first arising of the AlexNet (Krizhessky et al. 2012), even some of them has beat the humans' cognitive level in certain tasks. However, ANNs fail to uptake the advantages of the Neuronal Dynamics, which instantiates as high-power consumption, relatively low responses and etc.
Spiking Neuron Networks(SNNs) (Maass 1997). with inspiration for the propagation of the cortex neurons (Perrett et al. 1982; Tuckwell] 1988). have been presented continuous attention as a new, power-efficient and hardware friendly technology. In contrast the mere implementation of spa- tial information and complicated float point computation of ANNs, SNNs utilize spatial-temporal dynamics to mimic the bio-behavior of neurons, well as its dyadic-valued computation whose feeding electrical sequential impulses (i.e., spikes), belong to the binary-like set of {0,1). Benefit from the capabilities of processing binary-spiking signal and consequential effectiveness, there an alternative for SNNs that has feasibility of further development of machine learning and neuromor- phic application, which has been long-term significantly deployed in many neuromorphic hardware including SpiNNaker (Furber et al. 2014), TrueNorth (Akopyan et al. 2015) and Loihi (Davies) et al. 2018)
In contrast to the ANNs' well advanced, salient, proficient training methodology that indicate the conception BackPropagation(BP) (LeCun et al. 1998) along with its derivatives that consequently give rise to the convergence of ANNs and diverse categories of frameworks(ie. TensorFlow, Py- Torch, et al.) that make it succinet and available to train more deeper networks. However, for one thing, there are not much theoretically supported or potent procedure for tackling the issue of training SNNs, which limits SNNs from going deeper, therefore SNNs hardly fulfill the ability in real-world complex missions, such as video-based recognition//detection, natural language pro-

---

Under review as a conference paper at ICLR 2020
cessing et al.. For another thing, there no exit practical auxiliary frameworks that are capable to promote the mature structure of SNNs, which leads to the consequence of few application and rare forward-step development of SNNs.
There are still various efforts to make progress in training, deepening the depth and applications of SNNs, whereas many obstacles block the development of SNNs at the same time. As for training, there are many cireumvention ways to strengthen the accuracy of SNNs, except for neuromorphic methodology such as spike-timing-depentee winner-taken all (WTA) (Makhzani & Frey 2013). In the first alternative scheme, an ANN is trained firstly, then is transformed into the SNN version whose network structure is the same as the above- mentioned ANN, and neurons analog the behavior of ANN neurons (Dichl et al.] 2015). The other the direct supervised learning, also called Gradient descend, which is a superior, prevalent op- timization method for this learning procedure. In order to solve the issue of the non-differential problems of spikes, (Lee et al.] 2016) proposed an alternate that treats membrane potential as dif- ferential signals and directly uses BP algorithm to train deep SNNs. To act more bio-behavior, (Ponulak & Kasiñski 2010) introduced the remote supervised STDP-like rule to be capable of the learning of sequential output spike. Besides, (Urbanczik & Senn 2009) proposed novel leaning rule whose information will be embedded into the spatio-temporal information during learning of the spike signals. Nevertheless, most of the learning methods presented above are merely engaged in a single aspect of either spatial or temporal information. The applications started to spring up due to the incoming of the event- beset cameras composed of Dynamic Visual (ens))s(DVS) (Shi al. 2018). The mechanism of DVS can be outlined as simulation of the visual path structures and functionalities of the biological visual systems whose neurons asynchronously communicate and encode the visual information from environment as spatiotemporally sparse light intensity change in the form of spikes. On the strength of the event-based cameras, diverse event-based datasets were acquired such as Poker-DVS, MNIST-DVS (Serrano-Gotarredona & CIFARIO-DVE (Wu et al. 2019).
plasticity (STDP) (Serrano-Gotarredonna al.) 2013).
2015) and
Embracing the event-based cameras and their derived datasets, variety of monographs demon- strate the different methodologies whose intentions are to make a plausibility of the application of accordingly components. (Peng et al.) 2016) proposed event-based classification based on static learning method, named Bag of Events (BOE in short). This method denotes the events of corre- sponding to the activated pixel of the DVS as joint probability distribution. Moreover, this method tests on multiple datasets such as NMNIST, MNIST-DVS, Poker-DVS, and reveals that BOE can significantly achieve competitive results in real-time for feature extraction and implementation time as well as the classification accuracy. (Neil & Liu 2016) proposed a deep CNN to pre-process spik- ing data from DVS. which is used in various deep network architecture and is also used to achieve an accuracy of 97.40% on N-MNIST datasets, in spite of its complicated pre-processing approach. In terms of SNNs, (Indiveri et al. 2015) proposed a SNN architecture, named Feedforward SNN, which is based on spike-based learning and temporary learning, and it achieves 87.41 accuracy on MNIST-DVS datasets. (Stromatias et al. (015)) proposed a composite system, including convolu- tional SNNs, non-spiking fully connected classifier, and spiking output layer with its performance of 97. 95% of accuracy.
Together with improving the performance and enhancing the convergence rate of SNNs, the goal that whether a method that can absorb both advantages of ANNs and SNNs can be achieved. To this end, we propose RLIF with both low computational complexity and biological plausibility, to explore its usage in real-world tasks. In summary, the major contributions of this paper can be listed as follows:
We propose RLIF, which absorbs the biological traits from SNNs, follows the unroll struc- ture of RNNs, and enables a seamless way to insert into any sequential model in common deep learning frameworks.
A mass throughput can be implemented through the transition of binary information be- tween an interlayer of RLIF and other sequential layers. which meets the basic principle that the emission of neuron trains are binary values. Furthermore. RLIF can be easily extended into neuromorphic chips since its peculiarity of hardware-frient friendly.
The experiments conducted in general DVS-based datasets (MNIST-DVS. CIFAR1O SHEAR1O-DVS) and Chinese text summarization (LCSTS-2..) 2.00) show that our RLIF is capable of capturing key information through time and has lower parameters compared to its counterparts.
2

---

Under review as a conference paper at ICLR 2020
2 PREMISE OF UNDERSTANDING RLIF
As mentioned before, the core idea in our architecture is about how to absorb the biological traits of SNN into RNN. To this end, learning algorithm in SNN will be introduced first and then we simple analysis on basic LIF neuron model, which aims to highlight the most relevant parts to our RLIF.
2.1 LEARNING ALGORITHM FOR SNN
To the best of our knowledge, the learning algorithm for SNN could be divided into two categories:
i) unsupervised learning algorithms represented by spike timing dependent plasticity (STDP) and
ii) direct supervised learning algorithms represented by gradient-based backpropagation. Classi-
cal STDP and its modullted variants (Legenstein et al., 2008 Frémaux & Gerstner 2016),
the typical SNN learning method which only use the local information to update the weights of model, surrender difficulties in the convergence of models with many layers complex datasets (Masquelier & Thorpe 2007 Diehl & Cook) 2015 Tavanaei & Maida 2016).
Illuminated by observing the huge success of backpropagattion in ANN, researchers start to explore a new way about how can backpropagation be used in training SNN under the end-to-end paradigm. (Lee al., 2016: Jin et al. 2018)) have introduce spatial backpropagation method into training SNN which mainly based on conventional backpropagation. As to imitate the temporal characteristics of SNN, (Wu et al.) 2018) pioneered the use of backpropagation both spatial and temporal do- mains to train SNN directly, through which it achieved the state-of-the accuracy on MNIST and N-MNIST datasets. (Huh & Sejnowski 2018) introduce differentiable formulation of spiking dynamics and derive the exact gradient calculation to achieve this and (Neftci et al.) (219)) use sur- rogate gradient methods to conquer the difficulties associated with the discontinuous nonlinearity. As step further to increase the speed of training, (Wu et al., 2019] convert the leaky integrate- and-fire (LIF) model into an explicitly iterative version so as to train deep SNN with tens of times speedup under backpropagation through time (BPTT).
2.2 LIF NEURON MODEL
Leaky Integrateand- Fire (LIF) is the most common and simple model which can modeling neuron operations and some basic dynamic traits effectively with low computational costs. In general, we describe LIF neuron (layer 1 and index i) in differential form
du! = -U! Urest) RI!
Tiem dt
where U, refers the membrane potential, Urest is the resting potential, Tmem the membrane time constant, R is the input resistance, and I, is the input current (Gerstner et al., 2014). When the membrane voltage of neuron reaches firing threshold 0, spikes was released to communicate their output to other neurons. After each spike, U, is reset to the original resting potential Urest-
since the input current typically generated by synaptic currents triggered by the arrival of presy- naptic spikes St. (Neftei et al. 2019) model the dynamics of operations during approximating the time course as an exponentially decaying current following each presynaptic spike by
di! I Tayn
Σ W/y Sf
Σ
V'J S
(2)
dt
docay
ford.forward
recurreat
Based on this, the simulation of single LIF neuron can be decomposed into solving two linear dif- ferential equations. As RNN, who accepts both the current input 2a and the previously hidden state h4-1 and updates the current state via non-linear activation function (...), the basic form is
Dt oWa, 34 Wh h4- + +b)
(3)
3

---

Under review as a conference paper at ICLR 2020
Discrete Spke Trains
ap
RLIF Cell
RLIF Cell
Nembrane Potentials
Figure 1: A diagram of RLIF cell.
Apparently, Equation 2 has the similar structure with basic RNN, which provides an insight about paraphrasing LIF into recurrent paradigm.
RLIF ARCHITECTURE
this section, we will present the architecture the Recurrent Leaky Integrate-and-Fe model (RLIF). The principle idea we hold is enable RLIF with more biologically properties and achieve high computational efficiency meanwhile. As our architecture following by the paradigm of ANN, treat the synaptic current of SNN continuous probability distribution whereas keep the spike as discrete through a novel gradient broaden strategy, which allows the standard backpropagation through time in RLIF.
3.1 RLIF DEFINITION
Based on Equation 2 and of LIF, bring it into recurrent neural network's paradigm and the fusion form was described as follows:
V = U -
F² = vt 2 Vihres
(5)
u/y pr Vt Vreeett11 +1F" o V
a' = M' +
(7)
where Vt refers to the membrane potential with regard to the current voltage at timestep t and recurrent membrane potential at timestep t- 1, Fr denotes whether the current voltage of neurons has reaches its own firing threshold Vi it reaches its firing threshold then label this neuron with otherwise O. Next, we reset the firing neurons to its resting potential and let the membrane voltage of other neurons remain unchanged, are thus retrieved. Then we calculate u' with more biological plausibility as to mimic the random noise and accumulate with leakage.
shown in Equation the processed membrane potential un
Fd
Here, the information firing between layers is YJ (binary output, depicted in Figure I we denote this mode as spike). At current timestep f, where X denotes the input, U refers to the calculation of current voltage and Ms represents the updating process of membrane potential. V Virset is the reset voltage which produces the same effect (Lee et al. 2016) like Vreset in Equation 6 as to simulate the inhibitory response of neurons and V Vhhre is the firing threshold targeting at whether neuron is fire or not. Besides, we propose two patterns in the calculation of U and M FC (short for Fully Connected) and Conv (short for Convolution):
4

---

Under review as a conference paper at ICLR 2020
(xx
(x))
0
V tures
X
V thres
(a) original gradient of F
(b) broad gradient of F*.
The gradient of F. (red line refers to points that have gradient): (a) since F; is retrieved from a common step function, its gradient only exists one point which leads to rather harsh conditions in updating learnable weights before it. (b) we use novel gradient broading tactic to solve this problem with a hyperparameter a.
Figure 2:
{
W wolt X + byolts Wealt X bwolts
FC Conv
U²_
(9)
M a u/ + Ummm=
FC
(10)
U b mems
Conv
where represents matrix multiplication, C refers to the hadamard product whereas @ refers to the convolution product. a refers to the leakage to accumulate membrane potentials of each discrete timestep and B is the mechanism of simulating random noise in mammal neurons.
As depicted in Figure [ which is rather simple to further extend it into real-world complex tasks. Therefore, we set V Equation 4 the hidden state ht of LSTM (Hochreiter & Schmidhuber
1997) (as shown in Equation [T). Then followed by the same procedure as Equation Equation g The attention needs here that replace ht membrane potential u and keep the cell state unchanged. The usage of this RLIF variant (LIF-LSTM) will be introduced in the experiment of text summarization.
f oW, X' w- 1-1 +-1+b) bb)
z (W. + 2
11 +bb)
of a (W. X + Ug "1-1 bo)
(11)
c² tanh W. X + Uo - 26-1 bo)
ht of @ tanh(c"
3.2 GRADIENT BROADING: broadgrad()
Since critical problem has arisen due the spi ke output mode: the nondifferentiable property of the binary spike trains output. Here, a rectangular function (Wu et al. 2018) grad(...) was chosen broaden the range of spike derivatives on the backward phase.
grad(Ft) = 1, [Vt thres a, Vt shres + a] other
(12)
0,
As shown Figure 2 hyperparameter a essential to determinate the range of grad(F") which further exhibits considerable influence the convergence of network.

---

Under review as a conference paper at ICLR 2020
Model
Method Composite system
MNIST-DVS
CIFARTO-DVS
Zhao et al.
2014
88.14%
Stromatias et al.
2017)
Composite system HOTS Lightweight Statistical
97.95%
Lagorce et
2016)
27.10%
Shi et al. 2018
78.08%
31.20%
Paulun al. 2018)
NeuCube
92.03%
Cannicı ct al.
2019
Attention Mechanisms
44.10%
Sironi et al.
2018)
HATS
98.40%
52.40%
Ours
RLIF
98.43%
56.93%
Table 1: The comparison of accuracy of RLIF and other methods two neuromorphic datasets.
Model
Total number of samples used
Acc
(Paulun et al.
2018
10.000 (scale-4)
90.56%
Paullun al.
2018
10.000 (scale-8)
92.03%
(aullun et al.
2018
10.000 (scale-16)
86.09%
Ours
10.000 (scale-4)
97. 97.82%
Ours
10.000 (scale-8)
98.43%
Ours
10,000 (scale-16)
92.46%
Table 2: The comparison of the impact of different timesteps on MNIST-DVS.
EXPERIMENTS
We evaluate proposed RLIF on image classification task to verifyy its effectiveness compared with other brain-inspired methods. Moreover, we extend it into text summarization, classical natural language processing task, the experiment shows that RLIF and its variant, with pluggability and flexibility inside, could be applied successfully into complex real-world tasks.
4.1 CLASSIFICATION ON NEUROMORPHIC DATASETS
Here, we used two neuromorphic datasets, MNIST-DVS and CIFARIO-DVS, with pixel resolution of 128 128 to verify the classification performance of RLIF Both event-based datasets are taken from the original dataset by the DVS sensors, which in particular takes samples through moving along a fixed trajectory in front of the LCD monitor. MNIST-DVS dataset contains 30,000 event- stream records from handwritten digits 0 to 9, among which 80% used for training and 20% for testing. The CIFAR1O-DVS dataset contains 10 categories as well, totaling 10,000 event-stream records, each category containing 1,000 records.
DATA PREPARATION The design of the pre-processing part is based on the statistical information of DVS data, which can effectively represent temporal and spatial information. First of all, the pre-processing algorithm slides on the original event-streams sorted by timestamp according specific length event-window. The sliding step of the event-window is equal to its length. When the went-windoww slides, new event-stream thus generated represent the data with the number of event-stream recordings as the event-wing window. Finally, each new event-stream is expanded into a dimeesiimnsi data frame which call event-frame. Therefore, event- event-frame converting from a set of events, which represents the information of recording data at one timestep. After T times of processing, record with timestep T can be obtained, which contains both spatial and temporal information of the original event-stream data, and its dimension is (T, 128, 128, 2).
retrieved by
NETWORK STRUCTURE As shown in Figure 3 the network receives the event-frame record of T timesteps followed by the pre-processing module and performs feature extraction. The addition of RIF in our network the highlight, which serves as key for high-efficient use temporal information. The special layer (we denote it as SumLayer), which ultimately transformed the discrete binary event-stream into a continuous representation for overall prediction, through a way of integrating information of all timesteps.

---

Under review as a conference paper at ICLR 2020
bot-Sheat Reands
Event Frume Recard W9T Taxedea
RUP Lupe (Dor))
Lope (Due))
HLT Laper (tens)
syylipolo.
6
192
Figure 3: Our proposed network structure used in neuromophic dataset classification.
Most worthy of mention is, our model does not require complex pre-processing of the DVS raw event stream, and it can achieve better performance. 'abble | compares the performance of our model and the state-of-the thrt methods the MNIST-DVS and CIFARIO-DVS dataset, and our model achieves relatively high accuracy in the test set. We obtained 98. 98.43% accuracy on MNIST- DVS dataset, which is similar to the performance of ordinary convolutional network. Compared to the MNIST-DVS dataset, the CIFARIO-DVS dataset is more complex and contains much more information and noise than MNIST-DVS, but we ultimately achieved an accuracy of 56.93% that are better than all the SOTAs.
In order to verify the feasibility of the system, we compared the results scale-4, scale-8, scale-16, in which the same preprocessing tacties conducted on MNIST-DVS and trained with the same network model. The final test results are shown Table
4.2 TEXT SUMMARIZATION
Here, proposed sequence-to-sequenee model (Seq2Seq) with LIF-LSTM (RLIF's variant) on LCSTS dataset.
DATASET LCSTS is large-scale Chinese short text summarization dataset, consisting of pairs of (short text, summary) collected by (Hu et al., 2015). The whole dataset, which consists of more than 2,400,000 pairs, was split into three parts under the same process (Li et al.) 2017: Ma et al.) 2018) described. The noteworthy part is we only reserve pairs with scores no less than 3, thus we take PART for training, filtered PART II for validation, and filtered PART III for testing. During experiments, word segmentation was excluded whereas we only take Chinese character sequence as input.
the most common metric in evaluating the effect of text summarization: ROUGE score (Lin) 2004). The core idea of ROUGE is to compute the number of overlapping units between generated summaries and its reference summaries, including n-grams, word sequences, and word pairs. We use ROUGE- (unigram), ROUGE-2 (bi-gram) and ROUGE-L (LCS) with previous exercises (Hu at. 2015 al 2017 results.
EVALUATION METRIC Here,
Ma et al. 2018) in the experimental
NETWORK STRUCTURE As shown in Figure our network is based on the senneeccettosequeeee model where encoder is a stack Layer Normalization (LN) and Bi-LSTM and decoder similar to encoder of which Bi-LSTM is replaced by Uni-L.FFLLSTM. In general, we use the final decoder layer and the final encoder layer output for obtaining the recurrent attention context through multi- head attention (Vaswani et al. 2017) and teacher-forcing strategy supervise the learning of the representation of the source content with the corresponding summary. Under the assumption that word appears in the summary may existed in the text, prior distribution is adopted here to make model prefer the word in text rather than others.
As the experiment result depicted in Table our LIF-LSTM appears to be good substitute for LSTM but step further of its biological plausibity, which demonstrates the feasibility of the usage of LIF-LSTM into real-world tasks.

---

Under review as a conference paper at ICLR 2020
Soltmes
FC
Prior
Classiflel
LeakyHeLu (elphoi
Onehot StaleShitt
Concat
Layer
Bi-LSTM
Normalinat.oo
Uri-LIF-LSTM
Enceder
Layer Normaizzaton
Dacader
Multi Head Altention
Nocmalizat.og
Layer
B-LSTM
Uni-LiF-LLTM
Layer
Normalisation
Hormalitat
Maq_Max
Embedding
Enladdiing
Short text
Summary
Figure 4: Model architecture for text summarization. Thee encoder on the left side has 2 blocks of LN and Bi-LSTM, the decoder on the bottom right, Uni-LIF-LSTM interspersed throughout LN. Com- mon strategies like teacher-foreing, multi-head attention and the introduction of prior distribution have been taken to improve the performance as well as setting topK of beam search to 10.
Model
R-1
R-2
R-L
RNN (Hu et al. 2015)
21.50
8.90
18.60
RNN-context (Hu et al.
2015
29.90
17.40
27.20
SRB (Ma al.) 2017)
33.30
20.00
30.10
CopyNet (Gu al. 2016)
34.40
21.60
31.30
DRGD (L ct al. 2017
37.00
24.20
34.20
Seq2Seq (superAE) (Ma et al.
2018
39.20
26.00
36.20
Ours
37.22
24.64
34.45
Table 3: ROUGE-FI on LCSTS test set (R-1, R-2, and R-L are short for ROUGE-I, ROUGE-2, and ROUGE-L, respectively).
BOOGE-,,ROUGE2..
CONCLUSION
In this paper, we propose the Recurrent Leaky Integrate-and-Fire (RLIF) model which has comple- mentary advantages of ANNs and SNNs. RLIF is a more in-depth simulation on mammal neuron and can be easily plugged into the prevalent ANNs framework with the advantages of BPTT. The hybrid network of the combination of traditional ANNs module and RLIF converges easier than conven- tional SNNs method. The experiments show that our RLIF and its variant are of good application prospects due to their adaptability and stability, especially reflected in the text summary task. We believe that RLIF and variant can be applied to many real- weal-world challenging tasks such as neural machine translation, video understanding, which may lead shift in the public view about SNNs.
8

---

Under review as a conference paper at ICLR 2020
REFERENCES
Filipp Akopyan, Jun Sawada, Andrew Cassidy, Rodrigo Alvarez-lcaza, John Arthur, Paul Merolla, Nabil Imam, Yutaka Nakamura, Pallab Datta, Gi-Joon Nam, et al. Truenorth: Design and tool flow of 65 mw 1 million neuron programmable neurosynaptic chip. IEEE Transactions Computer- Design of Integrated Circuits and Systems, 34(10):1537-1555, 2015.
Marco Cannici, Marco Ciccone, Andrea Romanoni, and Matteo Matteucei. Attention mechanisms for object recognition with event-based cameras. In 2019 IEEE Winter Conference on Applica- fions of Computer Vision (WACV), pp. 1127-1136. IEEE, 2019.
Ronan Collobert and Jason Weston. A unified architecture for natural language processing: Deep neural networks with multitask learning. In Proceedings of the 25th international conference on Machine learning, pp. 160-167. ACM, 2008.
Mike Davies, Narayan Srinivasa, Tsung Han Lin, Gautham Chinya, Yongqiang Cao. Sri Harsha Choday, Georgios Dimou, Prasad Joshi, Nabil Imam, Shweta Jain, et al. Loihi: A neuromorphic manycore processor with on-chip learning. IEEE Micro, (881):822-8 29,9, 2018.
Peter U. Diehl and Matthew Cook. Unsupervised learning of digit recognition using spike-timing- dependent plasticity. Frontiers in Computational Neuroscience, 9(429):99, 2015.
Peter U Dichl, Daniel Neil, Jonathan Binas, Matthew Cook, Shih-Chii Liu, and Michael Pfeiffer. Fast-classifying, high-accuracy spiking deep networks through weight and threshold balancing. In 2015 International Joint Conference Neural Networks (IJCN), pp. 1-8. IEEE, 2015.
Frémaux and W Gerstner. Neuromodulated spike-timing-dependemn plasticity, and theory of three- factor learning rules. Frontiers in Neural Circuits, 9(172):85, 2016.
Steve Furber, Francesco Galluppi, Steve Temple, and Luis A Plana. The spinnaker project. Pro- ceedings of the IEEE, 102(5)x652-665, 2014.
Wulfram Gerstner, Werner M Kistler, Richard Naud, and Liam Paninski. Neuronal dynamics: From single neurons networks and models of cognition. Cambridge University Press, 2014.
Jiatao Gu, Zhengdong Lu, Hang Li, and Victor OK Li. Incorporating copying mechanism in sequence-to-sequenee learning. arXiv preprint arXiv: /00:://03.00000 06693, 2016.
Geoffrey Hinton, Li Deng. Dong Yu, George Dahl, Abdel-rahman Mohamed, Navdeep Jaitly, An- drew Senior, Vincent Vanhoucke, Patrick Nguyen, Brian Kingsbury, et al. Deep neural networks for acoustic modeling in speech recognition. IEEE Signal processing magazine, 29, 2012.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8): 1735-1780, 1997.
Baotian Hu, Qingcai Chen, and Fangze Zhu. Lests: A large scale chinese short text summarization dataset. arXiv preprint arXiv:1506.000 05865, 2015.
Dongsung Huh and Terrence J Sejnowski. Gradient descent for spiking neural networks. In Ad- vances in Neural Information P Processing Systems, pp. 1433-1443, 2018.
Giacomo Indiveri, Federico Corradi, and Ning Qiao. Neuromorphic architectures for spiking deep neural networks. In 2015 JEEE International Electron Devices Meeting (IEDM), pp. 4-2. IEEE, 2015.
Yingyezhe Jin, Wenrui Zhang, and Peng Li. Hybrid macro/micro level backpropagation for training deep spiking neural networks. In Advances in Neural Information Processing Systems, pp. 7005- 7015, 2018.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convo- lutional neural networks. In Advances in neural information processing systems, pp. 1097-1105, 2012.

---

Under review as a conference paper at ICLR 2020
Xavier Lagorce, Garrick Orchard, Francesco Galluppi, Bertram E Shi, and Ryad B Benosman. Hots: a hierarchy of event- based time-surfaces for pattern recognition. IEEE transactions on pattern analysis and machine intelligence, 39(7):1346-13555 2016.
Yann LeCun, Léon Bottou, Yoshua Bengio, Patrick Haffner, al. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(1 (K:127)278223 2324, 1998.
Jun Haeng Lee, Tobi Delbruck, and Michael Pfeiffer. Training deep spiking neural networks using backpropagation. Frontiers in Neuroscience, 10, 2016.
Legenstein, D Pecevski, and W Maass. learning theory for reward-modulated spike-timing- dependent plasticity with application biofeedback. Plos Computational Biology, 4(10): c1000180, 2008
Piji Li, Wai Lam, Lidong Bing, and Zihao Wang. Deep recurrent generative decoder for abstractive text summarization. arXiv preprint arXiv: (o0:::708.0233 2017.
Chin-Yers Lin. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, pp. 74-81, 2004.
Shuming Ma, Xu Sun, Jingjing Xu, Houfeng Wang, Wenjie Li, and Qi Su. Improving semantic relevance for sequence-to--sequence learning of chinese social media text summarization. arXiv preprint arXiv:1706.0245: 2017.
Shuming Ma, Xu Sun, Junyang Lin, and Houfeng Wang. Autoencoder as assistant supervisor: Improving text representation for chinese social media text summarization. arXiv preprint arXiv: 04069, 2018
Wolfgang Maass. Networks of spiking neurons: the third generation of neural network models. Neural networks, 10(9):1659-1671, 1997.
Alireza Makhzani and Brendan Frey. Winner-take-all autoencoders. In Advances neural infor- mation processing systems, pp. 2791-2799 2015.
T Masquelier and S. J. Thorpe. Unsupervised learning of visual features through spike timing dependent plasticity. Plos Computational Biology, 3())=31,, 2007.
Emre O Neftei, Hesham Mostafa, and Friedemann Zenke. Surrogate gradient learning in spiking neural networks. arXiv preprint 1901::901.0094.. 2019.
Daniel Neil and Shih-Chii Liu. Effective sensor fusion with event-based sensors and deep network architectures. In 2016 IEEE International Symposium on Circuits and Systems (ISCAS), pp. 2282- 2285. IEEE, 2016.
Lukas Paulun, Anne Wendt, and Nikola Kasabov. A retinotopic spiking neural network system for accurate recognition of moving objects using neucube and dynamic vision sensors. Frontiers in Computational Neuroscience, 12:42, 2018.
Xi Peng, Bo Zhao, Rui Yan, Huajin Tang, and Zhang Yi. Bag of events: An efficient probability- based feature extraction method for aer image sensors. IEEE transactions on neural networks and learning systems, (3(4))791-803 2016.
David I Perrett, Edmond T Rolls, and Woody Caan. Visual neurones responsive to faces in the monkey temporal cortex. Experimental brain research, 47(3))329-3444 1982.
Filip Ponulak and Andrzej Kasiński. Supervised learning in spiking neural networks with resume: sequence learning, classification, and spike shifting. Neural computation, 22(2):467-510, 2010.
Teresa Geraano-Go and Bernabé Barranc. Poker-dvs and mnist-dvs. their history, how they were made, and other details. Frontiers in neuroscience, 9:481, 2015.
Teresa Cerrano-Gotarrede
Timothée Masquelier, Themistoklis Prodromakis, Giacomo Indiveri, Stdp and stdp variations with memristors for spiking neuromor-
and Bernabe phic learning systems. Frontiers in neuroscience, 7:2, 2013.
10

---

Under review as a conference paper at ICLR 2020
Cong Shi, Jiajun Li, Ying Wang. and Gang Luo. Exploiting lightweight statistical learning for event-based vision processing. IEEE Access, 6:19396-1940,, 2018.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv: 1556, 2014.
Amos Sironi, Manuele Brambilla, Nicolas Bourdis, Xavier Lagorce, and Ryad Benosman. Hats: Histograms of averaged time surfaces for robust wrent-based object classification. I Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1731-1740, 2018.
Evangelos Stromatias, Daniel Neil, Francesco Galluppi, Michael Pfciffer, Shih-Chii Liu, and Steve Furber. Scalable energy-efficieme low-latency implementations of trained spiking deep belief networks on spinnaker. 2015 International Joint Conference on Neural Networks (IJCNN), pp- 1-8. IEEE, 2015.
Evangelos Stromatias, Miguel Soto, Teresa Serrano-Gotarredonna, and Bernabé Linares-Barranco An event-driven classifier for spiking neural networks fed with synthetic or dynamic vision sensor data. Frontiers in neuroscience, 11:350, 2017.
Amirhossein Tavanaei and Anthony Maida. Bio-inspired spiking convolutional neural network using layer-wise sparse coding and stdp learning. arXiv preprint weXiv:161..03000 2016.
Henry C Tuckwell. Introduction theoretical neurobiology: volume 2, nonlinear and stochastic theories, volume 8. Cambridge University Press, 1988.
Robert Urbanczik and Walter Senn. A gradient learning rule for the tempotron. Neural computation, 21(2):340 3520003322220000 2009.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pp. 5998-6008, 2017.
Yujie Wu, Lei Deng, Guoqi Li, Jun Zhu, and Luping Shi. Spatio-temporal backpropagation for training high-performance spiking neural networks. Frontiers in Neuroscience, 12:331-, 2018.
Yujie Wu, Lei Deng, Guoqi Li. Jun Zhu, Yuan Xie, and Luping Shi. Direct training for spiking neural networks: Faster, larger, better. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pp. 1311-1318. 2019.
Hao Ye, Zuxuan Wu, Kui-WeiZee Zhao, Xi Wang, Yu-Gang Jiang, and Xiangyang Xue. Evaluating two- stream for video classification. In Proceedings of the 5th ACM on International Conference on Multimedia Retrieval, pp. 435-442 ACM, 2015.
Bo Zhao, Ruoxi Ding, Shoushun Chen, Bernabe Linares- Linares-Barranco, and Huajin Tang. Feedforward categorization motion events using cortex-like features in spiking neural network. IEEE transactions on neural networks and learning systems, 1669):1963-197 2014.

---

Under review as a conference paper at ICLR 2020
A SUMMARIZATION EXAMPLE OF OUR MODEL AND OTHER WORKS.
Source:
Last night, several people were caught to smoke on a flight of China United Airlines from Chendu to Beijing. Later the flight temporarily landed on Taiyuan Airport.
Some passengers asked for security check but were denied by the captain, which led to a collision between crew and passengers.
Reference:
Several people smoked flight which led collision between crew and passengers.
Seq2Seq (Ma et al. 2018):
China United Airlnnes exploded in the airport, leaving several people dead.
Seq2Seq + superAE (Ma et al., 2018):
Several people smoked on a light from Chendu to Beijing,
which led to collision between crew and passengers.
Ours:
China United Airlines flight diverted to Chendu airport due smoking conflict.
Table 4: The comparison of our LIF-LSTM model with example.
(Ma et al. 2018 text summarization
12


Reviews:
[{"review_id": "SyxiwuB_cH", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Recently, it has been shown that spiking neural networks (SNN) can be trained efficiently, in a supervised manner, using backpropagation through time. Indeed, the most commonly used spiking neuron model, the leaky integrate-and-fire neuron (LIF), obeys a differential equation which can be approximated using discrete time steps, leading to a recurrent relation for the potential. The firing threshold causes a non-differentiability issue, but it can be overcome using a surrogate gradient. In practice, it means that SNNs can be trained on GPUs using standard deep learning frameworks such as PyTorch or TensorFlow.\n\nHere the authors extend this approach by proposing two variations of the LIF model, called RLIF and LIF-LSTM. However, the presentation of these models is not clear at all.\nFor example:\n* what is U^t in Equation 4?\n* what is M^t in Equation 7?\n* what is the difference (if any) between u^t and u_d^t?\nEquation 8 is even more obscure. Why bothering defining a new variable Y if it is equal to F? What is index j, and why is it used only on the left hand side of the equation?\n\nThe description of the LIF-LSTM is even more obscure, nothing is defined.\n\nFigure 2 has an error. On the left, with the heavyside activation function, the gradient is actually defined everywhere (with a value of 0) but on the red segment!!!\n\nIn addition, the experiments are not convincing. I am not an expert in NLP, so I will focus on the vision experiments.\nTable 1 is incomplete. Wu et al 2019 (which they cite elsewhere!!!), reached 60.5% on DVS-CIFAR10, which is much better than this paper (56.93%)\n\nFor all these reasons, I recommend rejection.\n\n", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}, {"review_id": "SJlM3Xs49r", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "####\nA. Summarize what the paper claims to do/contribute. Be positive and generous.\n####\nThe paper translates the Leaky Integrate and Fire model of neural computation via spike trains into a discrete-time RNN core similar to LSTM. The architecture would be readily amenable to the modern deep learning toolkit if not for the non-differentiability of the hard decision to spike or not. The hard decision is made by thresholding. The paper adopts a simple approximation of backpropagating a \"gradient\" of 1.0 through the operation if the threshold is within a neighbourhood [thresh - a, thresh + a], and otherwise 0.0, so the system can be trained by backpropagation.\n\nThe architecture is tested on a few \"neuromorphic\" video classification datasets including MNIST-DVS and CIFAR-DVS. Experiments are also run on a text summarization task.\n\n####\nB. Clearly state your decision (accept or reject) with one or two key reasons for this choice.\n####\n\nThe reviewer thinks the paper should be rejected in its current state. \n\nThe proposed architecture is a straightforward change to a standard LSTM core. Thus it should be compared head-to-head to LSTM on standard datasets for these models (e.g. classic synthetic tasks, language modeling, speech recognition, machine translation, etc) with everything else held constant (hidden size, learning rate, sequence length, etc etc).\n\nIt also doesn't really carry over any of the benefits of Spiking Neural Nets even though it is inspired by Leaky Integrate and Fire because it operates in discrete time like a normal RNN, just with an extra binary output produced by spiking. It's unclear that a spiking inductive bias is actually useful, even though event-driven computation could in theory allow much less computation, the proposed method does not have that property. \n\nSo the paper doesn't really provide evidence to back up their claim that the proposed model combines the complimentary advantages of Deep Learning and Spiking Neural Nets. \n\n####\nC. Provide supporting arguments for the reasons for the decision.\n\nWhile the proposed method is in-spirit inspired by the leaky integrate and fire model, it is operated/trained in discrete time which does not allow it to achieve the benefits of continuous time integrate-and-fire models which allow for less computation and time-discretization-invariance. \n\nThe conversion of the spiking model to the deep learning framework is rather crude, as the differentiable approximation to the non-differentiable threshold operation is biased and not well-motivated either empirically, intuitively, or theoretically (i.e. there are no comparisons to alternative choices).\n\nThere are new techniques for marrying continuous-time models and deep learning which seem more promising to investigate to this end (e.g. Neural ODE).\n\nSo in summary, the method doesn't have the computational benefits of a biologically plausible spiking algorithms and is not well-tested against competing deep learning methods, making it hard to verify the motivation of pushing toward a performant yet biologically plausible algorithm.\n####\n\n####\nD. Provide additional feedback with the aim to improve the paper. Make it clear that these points are here to help, and not necessarily part of your decision assessment.\n####\nThere are many grammatical and word-choice mistakes which make the paper hard to read.\n\nMainly, from a practical perspective, the paper would be much-improved by showing what benefit the spiking inductive bias confers over a standard LSTM on standard tasks in the deep learning community.\n\nThe method/landscape should be developed and studied in further detail until claims can be made about combining the strengths of spiking and deep-learning models.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I made a quick assessment of this paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "HkgH65Z1qH", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a brain-inspired recurrent neural network architecture, named Recurrent Leaky Integrate-and-Fire (RLIF). Computationally, the model is designed to mimic how biological neurons behave, e.g. producing binary values. The hope is that this will allow such computational models to be easily implemented on neuromorphic chips and the solution will be more energy-efficient. On neuromorphic MNIST and CIFAR, the proposed model achieves higher classification accuracy than other listed methods. On ROGUE, a text summarization benchmark, the proposed model achieves competitive performance. \n\nI am leaning towards rejecting this paper. The main advantage of the proposed computational model was not supported by evidence in the paper. The presented evidence only suggests that the computational model has the capacity for learning to solve real-world tasks to a degree that is on par with other existing computational models. But what supposedly distinguishes the proposed one from the rest, i.e. being more hardware-friendly and energy-efficient, was not demonstrated. ", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_thoroughness_in_paper_reading": "I made a quick assessment of this paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This work extends Leaky Integrate and Fire (LIF)  by proposing a recurrent version.
All reviewers agree that the work as submitted is way too preliminary. Prior art is missing many results, presentation is difficult to follow and incomplete and contains errors. Even if these concerns were addressed, the benefit of the proposed method is unclear. Authors have not responded.
We thus recommend rejection.
Decision: Reject