High performance RNNs with spiking neurons
Authors: Manu V Nair
Giacomo Indiveri
Venue:  (2020)
Keywords: RNNs
Spiking neurons
Neuromorphics

Abstract:   The increasing need for compact and low-power computing solutions for machine learning applications has triggered a renaissance in the study of energy-efficient neural network accelerators. In particular, in-memory computing neuromorphic architectures have started to receive substantial attention from both academia and industry. However, most of these architectures rely on spiking neural networks, which typically perform poorly compared to their non-spiking counterparts in terms of accuracy. In this paper, we propose a new adaptive spiking neuron model that can also be abstracted as a low-pass filter. This abstraction enables faster and better training of spiking networks using back-propagation, without simulating spikes. We show that this model dramatically improves the inference performance of a recurrent neural network and validate it with three complex spatio-temporal learning tasks: the temporal addition task, the temporal copying task, and a spoken-phrase recognition task. Application of these results will lead to the development of powerful spiking models for neuromorphic hardware that solve relevant edge-computing and Internet-of-Things applications with high accuracy and ultra-low power consumption.

Under review as a conference paper at ICLR 2020
HIGH-PERFORMANCE RNNs WITH SPIKING NEURONS
Anonymous authors Paper under double-blind review
ABSTRACT
The increasing need for compact and low-power computing solutions for machine learning applications has triggered significant interest energy-efficient neuromor- phic systems. However, most of these architectures rely on spiking neural networks, which typically perform poorly compared to their non non-spiking counterparts in terms adaptive spiking neuron model that
of accuracy. In this paper, propose can be abstracted as a low-pass filter. This abstraction enables faster and hetter training of spiking networks using pack-propagation, without simulating spikes. We show that this model dramatically improves the inference performance of a re- current neural network and validate it with three complex spatio-temporal learning tasks: the temporal addition task, the temporal copying task, and -hosee recognition task. We estimate at least 500x higher energy- eneggy-efficiency using our models on compatible neuromorphic chips in comparison to Cortex-M4, popular embedded microprocessor.
INTRODUCTION
Exponential growth in computational power and efficiency have played a vital role the development of neural networks and their training algorithms. However, it has also led to higher design complexity and increasing difficulty to keep up with Moore's 's law (Schaller, 1997 Waldrop 2016). Recent years have also seen the movement of computation from data-centres to compact, distributed, and portable embedded systems. These factors have created a demand for energy-efficien Al-capable devices, leading to the development of dedicated and optimized von Neumann-style Artificial Neural Network (ANN) accelerators (Aimar et al.) 2018 Cavigelli and Benini 2016 Chen et al. (0016) and
renewed interest in neuromorphic systems (Chicca et al., 2014 Frenkel et al.) 2019 Davies et al.,
2018
Moradi et al., 2018: Akopyan et al. 2015 Qiao et al. 2015 Neckar et al.
2019)
A key difference between neural networks deployed von Neumann systems and most neuromorphic platforms is the use of spikes or train of pulses to represent signals in the latter. Such networks are called Spiking Neural Networks (SNNs). Spiking neuromorphic systems have number of features that inhibit their use in real-world problems: (1) mixed-signal circuits suffer from Complementary Metal-Oxide-Semiine Semiconductor (CMOS) mismatch (Pelgrom et al. 1989) that degrades performance; (2) rate-based SNNs generate large number of spikes to represent signals that reduces their energy benefit; (3) complex spiking dynamics makes it difficult to train them using gradient-descent methods. In this paper, describe new neuron model that addresses these problems and discuss how its Low-Pass Filter (LPF) abstraction enables training spiking Recurrent Neural Networks (RNNs) using the backpropagation algorithm (or Backprop). This is significant breakthrough as it enables training and deployment of energy-efficient spiking neural network devices without simulating complex spiking dynamics.
PROCESSING-N--MEMON FOR RNNs
Consider an RNN layer with nodes. At each time-step, the processor computes one or several matrix products of the form = W.x, where y and æ are vectors of length n, and W is a 2-D matrix von Neumann system with batch-size 1, as is common in most
of size n. When operating edge applications, the bottleneck in throughput and energy-efficiency is the O(n²) memory fetches of W at every timestep. An in-memory matrix multiplier addresses this problem. It is module where a "read" from the memory location of the W matrix, using "X" as the "address" gives out "Y", without ever moving W. This leads to quadratic reduction in energy consumption. Processing

---

Under review as a conference paper at ICLR 2020
in-memory systems have been implemented for various tasks such as DNA sequencing (Ghose et al., 2018), graph processing (Ahn et al. 2015), etc and with dramatic reduction in energy consumption.
The energy reduction from in-memory computing is well established, but the key challenge with deploying such systems is the absence of compatible algorithms. In this paper, we propose an RNN model for such a system. The implementation of the in-memory module depends how x and y are encoded and transported. It can be synchronous or asynchronous and analogue or digital. We adopt an asynchronous digital approach as it offers some implementation advantages. Encoding information in binary digital format is less susceptible to noise in comparison to analogue. Asynchronous signalling allows the consumptonn to scale in proportion chip activity. while also permitting low- latency response. Chips implementing such schemes have been published in literature (Qiao et al.) 2015, Moradi et al.) 201 8j. The model presented this paper designed to integrate on similar chips (A reference framework is described in supplementary section D However, most of the a algorithmic ideas presented in this paper are general and applicable a range of compute systems.
THE SPIKING NEURON MODEL
Spiking neuron models for encoding signals typically use rate- or time-coded schemes (Dichl et al. 2015 Rueckauer et al. 2017; Bohte 2012. Mostafa 2017). In rate-coding, the firing rate of the neuron proportional to the input signal. Therefore, achieving high data-resolution with rate-coding requires a large number of spikes, which is not energy-effiiient (Nair and Indiveri 2019). To address this problem, several time-coding schemes have been proposed. In this work, we build on existing models to propose an Adaptive Integrate and Fire (al&F) neuron model, which can also be interpreted an asynchronous ΣA circuit (Nair and Indiveri 2019 Bohte 2012) Yoon 20166). This mechanism reduces the spike count by only transmitting the error between an internal state and the input. The aI&F neuron model implemented with current-mode neuromorphic circuits (Nair and
Indiveri 2019) can be described by the following equations:
dlmem Tmem dt
aI, - -mem) +i
(1)
ds
Tw
dt
0g (Imem IL)
(2)
mem = 0. when mmem A
where the currents Tmem and It represent the "membrane potential" and "leak reversal potential" variables. The term 8 represents the neuron adaptation current, the input current, Tmem the membrane time constant, az a gain factor, A the threshold, a, the adaptation coupling parameter and Tu is the adaptation time constant. The al&F model is feedback loop that tries to decrease the difference between the i(t) and st). The difference, i(t) - s(). is filtered with gain, &L, and time constant, Tmem- When the output of this filter, m exceeds the spiking threshold, A, I is reset and spike is generated. The ΣA circuit model used in this work is different from the al&F model in the computation of the feedback term. Instead of filtering memms we operate on the spike train generated by the spiking neuron. This ensures that the noise inserted by the spike-generation mechanism is also suppressed by the ΣA feedback. The modified feedback equation is as follows: ds
(4) Where, 6, indicates the spike train and 4i is a programmable maximum current value that the analogue filter implementation can generate. The product 8,44 models the feedback filter that integrates current I,n for the duration of the spikes. Figure showss block diagram of the circuit implementation of the Equation 3 with the modification described by Equation 4 In this diagram, F(s) is first order LPF that receives inputs to the neuron. Hss) is a first-order low-pass filter that produces s(L) in response to spikes generated by the neuron. E(s) is also a first-order low-pass filter on the difference between the input current i(l) and the feedback signal sl). When the output of E(s). Imem. exceeds the spiking threshold (A) of the neuron, spike-event is produced. With each svike-event, s(t) increases and i(t) - s(t) decreases. Figure shows the asynchronous EA feedback loop in action for test-case. It can be shown that the Laplace domain representation of the output spike train can be expressed by the equation:
= a. (8,I - Tk
- -1)- -
dt
((s) (())
N(s)
Y(s) 1+ H ((s)
(5)
1+ (()
2

---

Under review as a conference paper at ICLR 2020
Ieper Eite
Spile peerebe
Current (in nA)
20
Input, Feedback current, s6
Drotbakk ERtr
1.0
Banoder
0.00
0.05
0.10
0.15
0.20
0.25
seconds)
(a) Block diagram of a EA neuron.
(b) Transient behaviour of the Σ'A neuron.
Figure (a) The block marked pulse A encoder is similar to the al&F neuron model. The LPF stage at the input makes it an asynchronous ΣA loop. (b) Evolution of the feedback signal, s(t), over time. The sudden jumps in s()) correspond to spike events, yt).
where, (s)) indicates the Laplace-domain representation of the noise introduced into the loop, for example by the spike generation mechanism. The ΣA loop described by Equation [ similar to continuous-time ΣA modulation loop (Pavan et al. 2017) with the key difference being that the output spikes are unipolar. This is valuable because the output of conventional ΣA loop is always active (+1 or 1), whereas the asynchronous model is only active the time of a spike event, making the model more energy-efficient.
The input signals to the neuron may be encoded as spikes trains or as continuous analogue values from a sensor. The transmitted analogue signal is reconstructed from spike train by simply low-pass filtering it. An advantage of the EA neuron model that it comes with a low-pass filter in its input stage. Therefore, ΣA neuron is "codec" It can both encode analogue signal into spike train and decode an incoming spike train back to the transmitted analogue signal. As the low-pass filter at the input stage is agnostic of the type of input to it, ΣA encode and decode both types of signals spike trains or continuous analogue ones. A description of the biological motivation and noise-fitterig properties of the model is provided in supplementary section A The circuit implementing this model has been fully characterized in Nair and Indiveri (2019).
4 TRAINING A SPIKING RNN WITHOUT SIMULATING SPIKES
The neuron model introduced in the previous section allows us to train a recurrent SNN by treating the spiking neurons LPFs and modifying the recurrent ANN equations suitably. We will show that the trained weight parameters of the recurrent ANN model can be mapped to recurrent SNN, without additional training. We measure the effectiveness of this mapping procedure by comparing the temporal dynamics of the neurons in the recurrent ANN to the low-pass filtered spike trains generated by the spiking neurons in the corresponding recurrent SNN. In this demonstration, we use high precision synaptic weights. This is typically not available in most spiking neuromorphic platforms. However, the same mapping procedure can be used for mapping ANNs trained with binary or noisy weights. Before introducing the mapping procedure, we describe three operations that are needed for it.
Input re-scaling: When implementing SNN in mixed-signal neuromorphic systems, the state variables of the neuron are represented by voltages or currents that are of the order of mV or nA. Using such small values when training ANN in software may lead to computational instability. To avoid this we train the network with normalized input signals and re-scale the parameters and activation functions after training. For example, if single layer calculation is represented as y = ont(W x). where anr is a non-linear activation function, æ, W and are the inputs, weights, and outputs from the layer, respectively, then, to re-scale the inputs by factor 7, the activation function used in the ANN will be modified, during inference, as y o T I)), where, (. a.l
Low-pass filtering: The input signal can be reconstructed from the spikes trains generated by the EA neuron if s(t) tracks i(t) (Equation 4 This is because s(t) is obtained by filtering the neuron spike train. This is why a ΣA neuron can be modelled as an LPF with time constant Tu. The LPF approximation ignores the high-frequency components injected by the spiking mechanism, they
3

---

Under review as a conference paper at ICLR 2020
suppressed by feedback loop (the (s)) term in the transfer function of ΣA neuron, Equation 5
We model this by using a discrete-time Euler approximation to incorporate a LPF-term at the output of the RNN stage. This results in the Low-Pass Recurrent Neural Network (IpRNN) cell by simple tweak to the classical equation:
yt a 3 y1-1 11 @ (Wree V--1 + Win 34 + b)
(6)
where, o, @ and denote non- con-linearity, element-wise Hadamard product and matrix multiplication functions, respectively. The variables a, Yns Zn, Wrees Win. and represent the retention ratio vector, input vector, output vector, recurrent connectivity weight matrix, input connectivity weight matrix, and biases, respectively. The subscripts on variable y and x: indicate the time step. a models the time
constant of the recurrent ANN, and it is matched the SNN time constant by setting it to a =Ix where, ss is the time-step of the input data-stream fed to the recurrent ANN, and T2 is the feedback time constant of the >A neurons used in the desiredSNN.
For example, if the recurrent ANN is being trained to detect speech from an audio-signal, then T's should be set equal to the time difference between the consecutive samples. The value of Tg. for the ANN set to min(Tw (Tu: mmem) in the Σλ equations. T. must therefore be chosen such that the signals being transmitted lie well within the pass-band of the feedback filter. This ensures that all the in-band components are transmitted well, even when different neurons in the systems have different values of T., for example, due to mismatch. This is an important observation for mixed-signal systems, where mismatch effects may result in different neurons to have differing time-constants. We will demonstrate that the effect of device mismatch is well-tolerated for most practical cases and leads to gradual degradation in performance
increases. It must be noted that while computing Ts the bandwidth of an audio or sensor measurement is easy, it is not trivial for data-sets sets such as text.
Saturating non-linearity: It has already been shown in the literature that the Rectified Linear Unit (ReL.U) non-linearity is a good non-linear model of the & neuron (Yoon) 2016 [Bohte| 2012). However, the ΣA neuron also filters incoming spike trains using a low-pass filter which limits its maximum output current to Ln (see Equation 4 To model this effect we can either set Ln in the SNN to the largest activation output found in the ANN simulation or clamp the maximum output of the activation function in the ANN simulation to Jin In our experiments, we do the latter.
4.1 THE MAPPING PROCEDURE
First, the recurrent ANN cell is modified by replacing the RNN units with the IpRNN. The modified network is then trained using Backprop with conventional Autograd tools provided by libraries such as Py PoTorch or Tensorflow. This gives us the synaptic weights for the recurrent SNN. Then, the largest value attained by the state variables in the trained network is mapped to Inn- This ensures that the spiking neurons do not saturate. Finally, the inputs to the SNN are re-scaled to suitable currents or voltage values as described earlier.
Limitation: A recurrent SNN is a continuous-time system that spikes hundreds to thousands of times per second to achieve the necessary transmission accuracy. Therefore, the time step of the transient simulation needs to be made very fine. The mapping algorithm assumes that the mapped SNN operates on the same sequence that the original ANN is trained on. This is a problem. Training a recurrent ANN on long sequence using back proppaattantnoon is computationally expensive and often intractable because of vanishing and exploding gradients. For example, with speech signals sampled at the standard rate of 44.1 kHz, even a short utterance is thousands of samples long. If we are unable to train an ANN for the desired task, the mapping mechanism is useless. Our approach to addressing this issue is train the ANN with sub- sub-sampled signals. After we compute the desired weights, rescale the time-constants of the network before mapping it to the SNN. If the simulation time-step for the SNN is T SgNN and that of ANN is TsANy. then the time constants of the two simulations JagNN.
TAANN
We only rescale the time constants without changing the weights of the mapped network, introducing inaceuracies in the mapped network. The mismatch arises because single time-step of the ANN corresponds to several simulation time-steps
given by OANNN e
and OSNN N e
in the mapped SNN = T ). The mapping is exact for a first-order LPF because of the Lincar Invariat (LTI) property. However, even though an RNN is non-linear, by making the low-pass filtering effect more dominant (for example, with a = 0.99), we observe that the mapped dynamics match well.
4

---

Under review as a conference paper at ICLR 2020
5 EXPERIMENTAL RESULTS
All the spiking simulations in the following sections are run on a custom transient mixed- signal modeling library, called spiking simulator for systems of Ist-order LPFs (Spiker). The motivation for design and operation of Spiker is described in supplementary section B
5.1 ENCODING PERFORMANCE OF THE SIGMA-DELTA NEURON MODEL
The ΣΛ neuron model used in the mapped SNN is not ideal transmitter of information as the spiking mechanism introduces error akin to quantization noise. This is analogous to use of low bit-precision in conventional ANNs. It is important measure how much precision is available using a metric that is meaningful for the proposed spiking architecture. We measure this using a Segal-to-Distortin
ratio (SDR) metric when encoding a sinusoidal input and reconstructing it with an LPF. The SDR is the ratio between the energy contained in the transmitted signal and total energy in all other frequency components generated by the distortions introduced in the signal chain. The results of these experiments are shown in Figure 2 We note in Figure 2a that highest SDR of the ΣA neuron 55dB, with 20 dB/decade roll-off as a function of frequency with a pole corresponding of the SDR. We note in Figure 2b
to Tmem. Figure 2b highlights the input amplituee-deenndeee that the SDR improves as a logarithmic function of the input amplitude and then drops suddenly. The logarithmic improvement in SDR is because the error component corresponding to the spiking threshold, A, becomes a smaller fraction of the input amplitude. The sudden drop occurs when the input amplitude approaches and exceeds i,, in Equation 4 This is because the maximum attainable value of the feedback term, s() in Equation 4 is Tin- Under these conditions, meem is always greater than st), causing the neuron fire at very high rate. For the rest of the SNN simulations this paper, the ΣA neuron settings listed in Figure 2 caption are used.
1000
Inod Amp Sex
27.0
SDR
25
40
20
Fring Renggrat dr HO4G
36
28.0
20
90
26.5
200
10"
10'
10" 10"
Arpiitaile
(a) SDR vs input sinusoid frequency.
(b) SDR vs input sinusoid amplitude.
Figure 2: SDR of the ΣA neuron as a function of sinusoidal input parameters. The ΣA neuron is fed a single-tone sinusoid riding on a DC bias to ensures that the input is non- negative.. The transient simulations are run with a simulation time step of Lus. This is the reason for the saturation in the firing rate in (b) The SDR ratio is reported after subtracting the DC component. The neuron parameters for the simulation are Tmem = 0.007s, Tu = 0.0014s, ag = 5000, a₂ = 1, A = 0.1n4,
I, A0nA, I On.A.
5.2 DEMONSTRATION OF THE MAPPING MECHANISM
We demonstrate the mapping mechanism using multi-layer RNNs. The ANN dynamics are compared against signal obtained by low-pass filtering the spike trains generated by the ΣA neuron. Our assertion is that the mapping mechanism will map any recurrent ANN to equivalent recurrent SNN. Therefore, instead of demonstrating the mapping for a particular task, we set synaptic weights to random samples from a Gaussian distribution. We then compare the dynamics of all the neuron units in the mapped and original networks. To ensure that our nodes do not saturate, we constrain the largest eigenvalue of the recurrent weight matrices to 1.4. The motivation for this trick was from obtained from Echo-State Networks (JaNs))J 2002) Jaeger et al., 2007). The quality or goodness of fit is measured using an Normalized mean square error (NMSH) metric, which measures
5

---

Under review as a conference paper at ICLR 2020
the mean square error normalized by the signal power. It is used to compare two time series signals Fref and x using the following formulation:
| x|
NMSE = -1 ||Iref mean(Ere)) (Tre))]l
where, |I-I| indicates the 1.2 norm. The NMSE metric lies between and -o0, with indicating a perfect match and -oo indicating very bad fit. If NMSF = O, then is at least as good fit as straight line at free .. In our results, we report the mean and standard deviation in the NMSE scores for all the units in a layer.
The effectiveness of the mapping mechanism is demonstrated using a four-layer RNN. The input and output stages are implemented as fully-connected feed-forward layers, and the recurrent layers are also interleaved with conne-connected layers. The input feature dimension was set to two and the output weighted sum of sinusoidal signals that were aand-limited to 50 Hz
to three. The imput data and sampled at 1 MHz for 0.2 seconds. The high sampling rate is necessary to accurately capture the dynamics of the SNN, whose neuron models are highly non-linear, in transient simulation. The length of the simulation is key consideration as we want our mapped SNN implementation to remain matched for arbitrarily long sequences. Computational considerations limited the duration of our simulations, but this should be tested before large scale deployment in real-world use.
Fully-digital neuromorphic platforms, such as Intel Loihi, IBM TrueNorth, or SpinNaker, do not suffer for mismatch issues. However, mixed-signal -iggaal neuromorphic chips, such as QQao et al. (2015); Neckar et a (2019); [Schemmel al. (2012), are potentially more energy-efficient than their digital counterparts but suffer from device mismatch. A ΣA feedback loop naturally compensates for such effects (Pavan et al., (207)) but there are many components in the model that lies outside the feedback loop. To study this, we add the effect of mismatch in our simulations by sampling the parameters, p of the mapped SNN:
p p= p- Co, (N( 1))
(8)
where, Cr is the coefficient of variation (= standord deviation in the parameter, p. To understand mean the statistics in the quality of the mapping mechanism, we generate multiple samples of network parameters and measure the quality of fit. These results are tabulated in Tables and 2, where we list the measured mean of and standard deviation in NMSE values for 4-layer RNN with 51 and 500 units per layer, respectively. With no mismatch effects, the reconstruction is very good to all layers, in both cases. Furthermore, we observe nearly perfect reproduction of the network dynamics for up to layers, and gradual degradation as the size, depth and mismatch the network increases. We note that the mapping mechanism is robust for Cup 0.2. Reduced mismatch sensitivity is useful for design of neuromorphic chips because it simplifies the design, and that, in turn, reduces the energy and area consumed by these chips. Visualization of the transient dynamics of all the nodes in the original and mapped RNNs is provided in supplementary section F Finally, the performance of the mapping algorithm comparing the dynamics of the ANN with sub-sampled data that of the SNN is shown in Table 3 The length of the SNN simulation is 0.2s, translating to input sequence lengths, L. We note that the mapping technique works well for fairly high sub-sampling ratios and shows significant degradation only for L = 20. Note that a near perfect match (NMSE* > 0.5) is achieved up for depth of two. This restricts the models used for benchmarking in Section 5.3
5.3 LEARNING PERFORMANCE OF THE LPRNN CELL
The key idea behind enabling the mapping between an recurrent ANN to its spiking equivalent was the addition of low pass filter to the the state variables. It must be highlighted that the idea of using a low-pass filter model for neurons fairly old and has been studied in various contexts (Beer) 1995, Mozer 1992 Jaeger et al. (200)). The novelty in this work is in identifying its use in the mapping mechanism and in the study of its learning properties. The mapping procedure would be of no use the resulting ANN was unable to perform as well as their unfiltered counterparts in learning tasks, and it is the focus of this section.
in Equation by sampling from distribution with common mean value shared by all the neuron units in the network. This simplifies the design of the neuromorphic system by eliminating the need to create precise tunable time constants in the neuron implementations.
In our experiments, we

---

Under review as a conference paper at ICLR 2020
Layer
UNMSE
ONMSE
Cup
Cup
0.2
Cop =
Cop =2
Cop = 0
Cup = 0.2
Cup
Cop 2
Rec. layer
1.0
0.9
-5.1
-4.0
0.1
0.3
55.1
15.5
Rec. layer 2
1.0
0.5
-8.8
-17.5
0.1
1.1
54.4
41.4
Rec. layer
0.9
-1.5
-36.8
-100.2
0.1
6.4
117.1
300.0
Rec. layer 4
0.9
-3.7
-107.5
-278.9
0.2
8.2
202.4
535.9
Table I: Mapping a four layer network with units per layer for different mismatch values.
Layer
PNMSE
ONEE
Cep =0
Cup
0.2
Cop =
Cop =2
Cep = 0
Cup = 0.2
Cup
Cop =
Rec. layer
1.0
1.0
0.4
0.6
0.1
0.1
1.8
1.3
Rec. layer
0.9
-1.0
-48.9
-132.0
0.1
1.6
52.1
143.7
Rec. layer 3
0.8
-6.8
-185.1
-683.6
0.2
2.9
73.2
295.9
Rec. layer
0.2
-6.3
-133.7
-416.3
0.9
3.5
64.0
203.6
Table 2: Mapping four layer network with 500 units per layer for different mismatch values.
Layer
JNMSE
ONMSE
TEANN = 10
100μss 2000
lms
10ms
10 μrs
100gss 2000
lms
10ms
20000
200
20
20000
200
20
Rec. layer
1.0
1.0
0.8
-0.4
0.0
0.0
0.1
0.4
Rec. layer 2
0.9
1.0
0.7
-1.7
0.8
0.0
0.2
5.2
Rec. layer
0.9
0.9
0.4
-1.6
0.5
1.0
2.2
Rec. layer 4
0.9
0.9
0.1
-2.0
0.3
0.3
1.2
2.2
Table 3: Performance when mapping resampled data for four layer network with 128 units per layer.
Results in this work
Reference literature
Task
SimpleRNN
IpRNN
LSTM
Temporal addition
40 steps
642 steps
0
5000 steps (Li al. 2018)
Temporal copy
30 steps
120 steps
200 steps
500 steps (Arjovsky et al. 2016
Spoken phrase (acc.)
27%
93% %
92%
90% (Sainath and Parada 2015
Table The performance of the IpRNN cell on three benchmarks. The reference literature column reports the best results (to our knowledge) with neural networks with less than 100K parameters. IpRNN and SimpleRNN networks for the add and copy tasks used a single layer with 128 hidden units. Arjovsky et al. (2016) unitary recurrent weight matrices and Li et al. (2018) use two layers of 128-unit IndRNN layers. The spoken phrase task reference is a CNN model released as Tensorflow example (Google 2019).
First, we study the short-tern memory capabilities the IpRNNs using the synthetic addition and copying tasks (Hochreiter and Schmidhuber 1997 Arjovsky et al. 2016 Le et al. 2015). Next, compare the performance of the IpRNN cell vs SimpleRNN cell in speech recognition task. We summarize the key observations in this section and details are in the supplementary material.
Temporal addition task: The addition task (Le et al. 2015) involves processing two parallel input data streams of equal length. The first stream comprises random numbers E (0, 1) and the second is full of zeros except at two time steps. The network is trained generate the sum of the two numbers in the first data stream corresponding to the time-steps when the second stream had non-zero entries. The baseline to beat is mean square error (mse) 0. 1767, corresponding to a network that always generates 1. We adopt curriculum learning (Bengio al. 2009) procedure for this task, by first training the RNN cell on a short sequence and progressively increasing the number of time steps. Each curriculum used 10,000 training and 1000 test samples. The length of the task incremented when the mse went below 0.001. The SimpleRNN cell failed to converge beyond sequences length 40, while the IpRNN cell converged to mse 0.001 for sequences up 642 steps. Interestingly, the

---

Under review as a conference paper at ICLR 2020
Long Short-Tern Memory (LSTM) cell learnt general solution when trained by this procedure and could solve arbitrarily long sequences, even with just two hidden units in the cell!
Temporal copying task: We train the RNN cells on varying length copying task as defined in (Graves et al., 2014) instead of the original definition (Hochreiter and Schmidhuber, 1997. Arjovsky et 2016). This problem is harder to solve than the temporal addition task. The network receives sequence of up to S symbols (in the original definition, S is fixed) drawn from an alphabet of size K. At the end of S symbols, a sequence of T blank symbols ending with a trigger symbol is passed. The trigger symbol indicates that the network should reproduce the first S symbols in the same order. We adopt a curriculum learning procedure here too and first train the network on short sequence (T=3) and gradually increase it (T=))00). The sequence length is incremented when the categorical aceuracy is better than 99%. The SimpleRNN cell failed at this task even for T=30. The IpRNN cell was able to achieve 99% accuracy for up to 120 time steps. After that, it generates T blank entries accurately but the accuracy of the last S symbols drops (For T=200, it was 96%).
Spoken phrase classification: typically aasuurce-oonstraiee. due to power, memory, and area restrictions. Therefore, we test the performance of the IpRNN cell on a problem that is compatible with such systems (see supplementary section D). We chose a limited vocabulary spoken commands detection task using the Google commands dataset ( (Warden 2018), which comprises 36 classes of short-length spoken commands such as "left", "up" or "go" The dataset was created for hardware and algorithm developers to evaluate their low footprint neural network models in limited dictionary speech recognition tasks. The neural network receives a Mel-spectrogram as inputs, and comprises, from input to output, two fully-connected dense layers with 128 and 32 units with batch normalization, two RNN layers with 128 units, two fully-connected dense layers, topped by softmax readout. In total, the network has about 80K trainable parameters. We use Adam optimizer (Kingma and Ba, 2014), with a learning rate of 0.001 for training. We note that all IpRNN variants massively outperform the SimpleRNN and slightly outperform the LSTM variants, all with the same number of parameters. We further note that the performance of the network peaks for certain values of and the random sampling case. The high performance of the IpRNN cell is a crucial result because not only did the addition of the low-pass filter enable mapping of the recurrent ANN model to neuromorphic platforms, it also resulted in much- much-improved performance.
The target for the IpRNN cell are neuromorphic platforms that are
[0.1.1]
0.1
0.5
0.8
0.9
0.99
1
Accuracy
93% %
26%
25%
69%
92%
93%
90%
4%
Table 5: Performance of the IpRNN cell on the Google commands classification task with different filtering coefficients, a. 0 is equivalent to SimpleRNN and a is MLP.
Energy-efficiency estimation: We compare the energy cost of computing two-layer RNN with 128 hidden units (used in the audio task) for an in-memory IpRNN system with Σλ neurons against Cortex-M4, a popular low-power microprocessor used in milli Watt-rage applications. We conserva- tively estimate that our implementation yields 500x better energy weegy-efficieency in this instance. Larger networks will see dramatic increase in this difference. (see supplementary section]].
6 CONCLUSION AND OUTLOOK
We presented a novel F spiking neuron model and discussed its spike-coding and noise-tolerance benefits. Then, we discussed how the LPF abstraction of the aI&F neuron model enables training of recurrent SNNs without having to simulate the complex dynamics of a large network of spiking neurons. This strategy enables training a recurrent SNN using standard optimization algorithms such as backpropagation. The mapping technique that allowed us to achieve this result is studied in detail, including its limitations.. We observe a dramatic improvement in the performance of the RNN cell with the addition of the low-pass filtering term. We think that this improvement is because the LPF acts as a temporal regularizer (see supplementary section C. Current spiking RNNs are benchmarked on much simpler tasks than presented in this paper, often due to the computational cost and lack of algorithms for training them. The importance of this work lies in proposing algorithmic solutions to address this key problem, enable new generation of ultra-low-poot power neuromorphic chips.
8

---

Under review as a conference paper at ICLR 2020
REFERENCES
Ahn, J., Yoo, S., Mutlu, O., and Choi, K. (2015). Pim-enabled instructions: a low-overhead, locality-aware beocesing-in-meenee architecture. In 2015 ACM/IEEE 42nd Annual International Symposium on Computer Architecture (ISCA), pages 536-348, IEEE
Aimar, A., Mostafa, H., Calabrese, E., Rios-Navarro, A., Tapiador-Morales, R., Lungu, I.-A., Milde, M. B., Corradi, F., Linares-Barranco, A., Liu, S.-C., et al. (2018). Nullhop: A flexible convolutional neural network accelerator based on sparse representations of feature maps. IEEE transactions neural networks and learning systems.
Akopyan, F., Sawada, J., Cassidy, A., Alvarez-Icaza, R., Arthur, J., Merolla, P., Imam, N., Nakamura, Y., Datta, P., Nam, G.-J., et al. (2015). Truenorth: Design and tool flow of a 65 mw million neuron programmable neurosynaptic chip. IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems, (e1100)5537-155
Arjovsky, M., Shah, A., and Bengio, (2016). Unitary evolution recurrent neural networks. In International Conference Machine Learning, pages 11 1120-112.
ARM (2019). Arm cortex. Datasheet Cortex-MA Accessed on 2019-11-07.
Bai, S., Kolter, Z., and Koltun, V. (2018). An empirical evaluation of generic convolutional and recurrent networks for sequence modeling. arXiv preprint arXiv:1803.01271.
Beer, R. D. (1995). On the dynamics of small cattinuous-tmme recurrent neural networks. Adaptive Behavior, 3(4):469-509.
Bengio, Y., Louradour, J., Collobert, R., and Weston, J. (2009). Curriculum learning. In Proceedings of the 26th annual international conference on machine learning, pages 41-48. ACM.
Bohte, S. M. (2012). Efficient spike-coding with multiplicative adaptation a spike response model. In Advances in Neural Information Processing Systems, pages 1835-1843.
Brette, and Gerstner, W. (2005). Adaptive exponential integrate-and-fire model as an effective description of neuronal activity. Journal of neurophysiology. 94(5):33773362..
Cavigelli, L. and Benini, L. (2016). Origami: A 803-gop/s/w convolutional network accelerator. IEEE Transactions on Circuits and Systems for Video Technology, 27(11):2461-2475
Chan, V., Liu, S.-C., and van Schaik, A. (2007), Aer ear: A matched silicon cochlea pair with address event representation interface. IEEE Transactions on Circuits and Systems I: Regular Papers, 54(1):48-59.
Chen, Y.-H., Krishna, T., Emer, S., and Sze, V. (2016). Eyeriss: An energy-efficient reconfigurable accelerator for deep convolutional neural networks. IEEE Journal of Solid-State Circuits, 52(1):127~ 138.
Chicca, E., Stefanini, F., Bartolozzi, C., and Indiveri, G. (2014). Neuromorphic electronic circuits for building autonomous cognitive systems. Proceedings of the IEEE, 13279))336711388
Chung, J., Gulcehre, C., Cho, K., and Bengio, Y. (2014). Empirical evaluation of gated recurrent neural networks on sequence modeling. arXiv preprint 412.5544
Davies, M., Srinivasa, N., Lin, T.-H., Chinya, G., Cao, Y., Choday, S. H., Dimou, G., Joshi, P., Imam, N., Jain, S., et al. (2018). Loihi: A neuromorphic manycore processor with on-chip learning. IEFE Micro, 38(1):82-99
Diehl, P. U., Neil, D., Binas, J., Cook, M., Liu, S.-C., and Pfeiffer. M. (2015). Fast-classifying, high- accuracy spiking deep networks through weight and threshold balancing. 2015 International Joint Conference on Neural Networks (IJNN), pages 1-8. IEFE.
Frenkel, C., Lefebvre, M., Legat. J.-D., and Bol, D. (2019). A 0.086-mm2 12.7 12.7-pj/sop 64k-synapse 256 256-neuron online-learnin learning digital spiking neuromorphic processor in 28-nm cmos. IEEE transac- fions biomedical circuits and systems,

---

Under review as a conference paper at ICLR 2020
Gers, F. A., Schmidhuber, J., and Cummins, F. (2000). Learning to forget: Continual prediction with 1stm. Neural Computation, 12(10):4451-2471
Gerstner, W. and Kistler, W. M. (2002). Spiking neuron models: Single neurons, populations, plasticity. Cambridge university press.
Gewaltig, M.-O. and Diesmann, M. (2007). Nest (neural simulation tool). Scholarpedia, 2(4))1430.
Ghose, S., Hsich, K., Boroumand, A., Ausavarungnirun, R., and Mutlu, O. (2018). Enabling the adoption of processing in-memory: Challenges, mechanisms, future research directions. arXiv preprint arXiv:/802.00322
Google (2019). Audio recognition. Github source code for Tensorflow vl.x. Accessed 2019-1 1--077
Graves, A., Wayne, G., and Danihelka, I. (2014). Neural turing machines. CoRR, abs/1410.5401.
Hamilton, J., Jin, C., Van Schaik, A., and Tapson, J. (2008). An active 2-d silicon cochlea. IEEE Transactions biomedical circuits and systems, 2(1):30 43.
Henaff, M., Szlam, A., and LeCun, (2016). Recurrent orthogonal networks and long-memon tasks. arXiv preprint arXiv:
Hochreiter, S. and Schmidhuber, J. (1997). Long short-term memory. Neural computation, 978): 1780.
Hu, Y., Huber, A., Anumula, J., and Liu, S.-C. (2018). Overcoming the vanishing gradient problem in plain recurrent networks. arXiv preprint arXiv:1801.06105
Jaeger, H. (2002). Tutorial on training recurrent neural networks, covering BPPT, RTRI, EKF and the" echo state network" approach. GMD--orschungszeenrum Informationsteehnik.
Jaeger, H., Lukoševičius, M., Popovici, D., and Siewert, U. (2007). Optimization and applications of echo state networks with inttg-integratorr neurons. Neural networks, 20(3):335-32..
Kim, Y., Jernite, Y., Sontag, D., and Rush, A. M. (2016). Character-aware neural language models. In Thirtieth AAAI Conference on Artificial Intelligence.
Kingma, D. P. and Ba, J. (2014), Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6983
Le, Q. V., Jaitly, N., and Hinton, G. E. (2015). A simple way to initialize recurrent networks of rectified linear units. arXiv preprint arXiv: 554.00941.
Li, S,, Li, W., Cook, C., Zhu, C., and Gao, Y. (2018). Independently recurrent neural network (indrun): Building longer and deeper In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pages 5457-5466.
Marcus, M., Kim, G., Marcinkiewicz, M. A., Maclntyre, R., Bies, A., Ferguson, M., Katz, K., and Schasherger, (1994). The penn treebank: annotating predicate argument structure. In Proceedings of the workshop on Human Language Technology, pages 114-119. Association for Computational Linguistics.
Moradi, S., Qiao, N., Stefanini, F., and Indiveri, G. (2018). A scalable multicore architecture with heterogeneous memory structures for dynamic neuromorphic asynchronous processors (DYNAPs). Biomedical Circuits and Systems, IEEE Transactions on, 12(1):106-122.
Mostafa, H. (2017). Supervised learning based on temporal coding in spiking neural networks. IEEE transactions on neural networks and learning systems, 29(7):327-3335.
Mozer, M. C. (1992). Induction of multiscale temporal structure. In Advances in neural information processing systems, pages 275-282.
Nair, M. V. and Indiveri, G. (2019). An ultra-low power sigma-delta neuron circuit. In 2019 IEEE International Symposium Circuits and Systems (ISCAS), pages 1-5.
10

---

Under review as a conference paper at ICLR 2020
Neckar, A., Fok. S., Benjamin, B. V., Stewart, T. C., Oza, N. N., Voelker, A. R., Eliasmith, C., Manohar, R., and Boahen, K. (2019). Braindrop: mixed-signal neuromorphic architecture with dynamical systems-based programming model. Proceedings of the IEEE, 107(1):444164.
Neil, D., Pfeiffer, M., and Liu, S.-C. (2016). Phased Istm: Accelerating recurrent network training for long event-based sequences. In Advances neural information processing systems, pages 3882-3890.
Oliphant, (2006). Guide to NumPy.
Pascanu, R., Mikolov, T., and Bengio, Y. (2013). On the difficulty of training recurrent neural networks. In International Conference on Machine Learning, pages 1310-1318.
Paszke, A., Gross, S., Chintala, S., Chanan, G., Yang, E., DeVito, Z.,, Lin, Z.,, Desmaison, A., Antiga,
I.,, and Lerer, (2017). Automatic differentiation in PyTorch. NIPS Autodiff Workshop.
Pavan, S., Schreier, R., and Temes, G. C. (2017). Understanding delta-sigma data converters. John Wiley & Sons.
Pelgrom, M., Duinmaijer, A., and Welbers, A. (1989). Matching properties of MOS transistors. IEEE Journal of Solid-State Circuits, 24(5):1433-1440.
Qiao, N., Mostafa, H., Corradi, F., Osswald, M., Stefanini, F., Sumislawska, D., and Indiveri, G. (2015). A reconfigurable on-line learning spiking neuromorphic processor comprising 256 neurons and 128k synapses. Frontiers in neuroscience, 9:141.
Rethinagiri, S. K., Palomar, O., Ben Atitallah, R., Niar, S., Unsal, O., and Kestelman, A. C. (2014). System-level power estimation tool for embedded processor based platforms. In Proceedings of the 6th Workshop on Rapid Simulation and Performance Evaluation: Methods and Tools, page 5. ACM.
Rueckauer, B., Lungu, I.A.,, Hu, Y., Pfeiffer, M., and Liu, S.-C. (2017). Conversion of continuous- valued deep networks efficient event-driven networks for image classification. Frontiers in neuroscience, 11:682.
Sainath, and Parada, C. (2015). Convolutional neural networks for small-footprint keyword spotting.
Sarpeshkar, R. (1998). Analog versus digital: Extrapolating from electronics to neurobiology. Neural Computation, 10(7):1601-1688.
Schaller, R. R. (1997). Moore's law: past, present and future. IEEE spectrum, 34(6):52-59.
Schemmel, J., Grübl, A., Hartmann, S., Kononov, A., Mayr, C., Meier, K., Millner, S., Partzsch, J., Schiefer, S., Scholze, S., Schüffny, R., and Schwartz, M. (2012). Live demonstration: A scaled-down version of the BrainScaleS wafer-scale neuromorphic system. In IEEE International Symposium on Circuits and Systems ISCAS 2012, page 702.
Srivastava, N., Hinton, G., Krizhevsky, A., Sutskever, I., and Salakhutdinov, R. (2014). Dropout: simple way to prevent neural networks from overfitting. The journal of machine learning research, 15(1):1929-1953
Stimberg, M., Brette, R., and Goodman, D. (2019). Brian 2: an intuitive and efficient neural simulator. BioRxiv, page 595710.
Waldrop, M. M. (2016). The chips are down for moore's 's law. Nature News, 530(7589):144.
Wang, Y. and Tian, (2016). Recurrent residual learning for sequence classification. Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 138-943.
Warden, P. (2018). Speech commands: A dataset for -ioited-vocabulary speech recognition. arXiv preprint ar Xiv: X (004 00400320.
11

---

Under review as a conference paper at ICLR 2020
Xingjian, S., Chen, Z., Wang, H., Yeung, D.-Y., Wong, W.-K., and Woo, W.-c. (2015). Convolutional 1stm network: A machine learning approach for precipitation nowcasting. In Advances in neural information processing systems, pages 802- 810.
Yoon, Y. C. (2016). Lif and simplified srm neurons encode signals into spikes via a form of asynchronous pulse sigma-delta modulation. IEEE transactions on neural networks and learning systems,
12

---

Under review as a conference paper at ICLR 2020
A MOTIVATION FOR THE USE OF THE SIGMA-DELTA NEURON MODEL
A.1 CODING MECHANISM
neuron in deep learning has one primary function the non-linear transformation of its input. However, a biological neuron and its neuromorphic counterpart have the added job of encoding information in spike trains. The most popular model for this encoding mechanism is rate coding, where the neuron fires at rate proportional to the incoming signal. This mechanism is not efficient for transmitting high- nigh-resolution data. For example, to transmit a signal at 8-bit resolution, it will require O(256) spikes for each sample. An improvement this coding mechanism is theal& model (Brette and Gerstner 2005), which can be interpreted as an asynchronous delta-sigma (AE) loop (Yoon 2016| 2012 Nair and Indiveri, 2019). The AΣ mechanism is time-coding model that is more efficient in its use of spike trains (Nair and Indiveri 20119) than cate-coding models. Other time-coding schemes have also been proposed in literature (Rueckauer al. 2017 Gerstner and Kistler 2002) However, the advantage of the A interpretation is that it leads to highly power-efficient circuit implementations (Nair and Indiveri 2019) that is tolerant mismatch and noise effects. Furthermore, the model allows to treat the neuron state as an analogue variable and ignoring the specific timing details of the encoding spike trains (Nair and Indiveri| 2019). The independence from the monitoring precise spike-tine times is beneficial because state-dependency, noise and device mismatch cause different neurons to generate spikes at different times for the same input. Modelling it is computationally expensive. The AΣ feedback loop ensures that they all represent the same signal with the same accuracy in spite of their differences (Nair and Indiveri 2019). This abstraction enables the network designer to only look at the internal state of the neuron when optimizing the network weights for an SNN. It is a crucial enabler for this paper as simulating and training mismatch-prone SNNs is computationally much more expensive than ANNs.
A.2 BIOLOGICAL NEURAL NETWORKS ARE LOW PASS FILTERING
Activation functions used in neural networks and apply a non-linearity weighted sum of input signals. However, ANNs assume that when the input changes, the internal state of the neuron or dendrites can also change immediately reflect the new input. This behaviour ignores the fact that biological neuronal channels are LPFs (Gerstner and Kistler 2002). Modelling the inertial or low- ww-pass filtering property is essential to implement and study recurrent neural networks in any neuromorphic system as the transitional dynamics deviate completely in its absence. The AΣ neuron models the filtering behaviour with a first-order low pass filter. We argue that not only is this modelling essential, it is also useful constraint to impose on RNNs.
A.3 CHANNEL NOISE AND RECONSTRUCTION ACCURACY IN SPIKING NEURON
The YA scheme encodes the information in the relative timing of the spikes. This implies that any noise in the spike timing, i.e. jitter, introduced during transmission of the spikes will result in distortion of the reconstructed signal, for example, in situations when the transmitting and receiving neurons are on separate chips. The distortions are modelled by a random variable A., which is sampled from a normal random distribution, A, ~ (N j3) with probability distribution function Pr(A.). Note that fixed delay does not contribute to distortion and is ignored. To compute the effect of spike timing noise, we calculate the Laplace domain representation of the transmitted signal as spike train,
follows. If In is the Dirac delta function, the transmitted spike train y(t). y' (1), the desired filtered spike train, s(), and the filtered version of the jitter-affected spike-train,
13

---

Under review as a conference paper at ICLR 2020
Mognitutelin dB)
-40
Sggal
Nose
Chann
136
10
10a
10a
10a
102
10f
Frequency (H2)
Figure A.1: The different transfer function of the ΣA neuron model. Red: Signal transfer function,
Blue: noise transfer function, Green: Signal transfer function with spike timing noise.
r(t) can be expressed as,
N v(() = Σ
[n( - (b)
(9)
k=
(()
[p(t -4 1 A.)
(10)
k=
s(t) = v(t) * h(t))
Σ h(t tx)
(11)
N
= S(s) = Σ (s)e
sta
(12)
k=
(t) = 5/) h(t)
h(t tk A.)
(13)
The mean value of r(t) can then be computed
F(I) F(r(t)) Σ
((t Pr(A.) h(t 160-14 14 -ii)
k=
A==00
N
Σ
Pr hilt tx)
(14)
K=U
R(s) ([(())
Σ (()) P(s)S(s) K=
(15)
= جي R(s)) is plotted in Figure A.I as the recovery transfer function (RTF) with ja = Imss. We see that for sufficiently band lind-limited signal, the channel noise does not affect the signal transmission accuracy.
where, P(s) = 2 (Pr(t)
B COMPARISON TO OTHER RNN MODELS
The low pass filtering behaviour of neurons is well-known in neuro-scientific literature (and in other fields) and has been studied in the past with RNNs as Beer 1995 Mozer 1992 Jaeger et al. 2007). For example, ESNs proposed by Herbert Jaeger (Jaeger al. 2007) has identical formulation to the IRRNN where the recurrett layer uses leaky integration units. In ESNs, the spectral radius of the initialization values of the recurrent kernel is constrained confers an "echo-state
14

---

Under review as a conference paper at ICLR 2020
property" to the network. The recurrent or input connectivity weights not trained during the learning process. Instead, only the read-out linear classifier is trained. In the IpRNN model, the spectral radius of the recurrent kernel is not constrained and all the weight matrices, including the retention ratios if required, are trained. IRRNN also shares similarities with recurrent residual networks proposed by Yiren Wang (Wang and Tian 2016), which are described by the following equations
yt = f(((--)) + 031-1, Lt, W)
(16)
where W denotes input and recurrent kernels, and other symbols have the same meaning as equation In equation 16 and f are identity and a hyperbolic tangent functions, respectively. A comparison can also be made with the LT-RNN model proposed by Mikael Henaff (Henaff et al. 2016), whose update equations are:
hr oWWn I x 6) + V- h4-1
(17)
3/ Ww -.
(18)
where W and V are 2-D transition matrices that are learned during the training process. In this case, it is possible that the LT-RNN cell reduces to an IpRNN, but is unlikely to occur in practice. Similar analogies
also be made to the IndRNN model (Li et al. 2018) and recurrent identity networks (Hu et al.) 2018). Generally speaking. the main difference between the IpRNN cells and popular RNN models in use today is jndrouiinoooittciion of the filtering term into the RNN model with impositions on boundary and train-ability conditions. We see that in addition to enabling their use in neuromorphic platforms, this results in more stable convergence properties due temporal regularization effect, as described in the Section
C MEMORY IN AN LPRNN
We can analyze the evolution of an IpRNN cell by using an approach similar to the power iteration method, described by Razvan Pascanu (Pascanu et al. 2013). To do this analysis, we approximate the IpRNN update equation as:
V/ -0 Wt-1
a) (Wree rec Dt-1 + sn 2e +
(19)
where, for simplicity, also make the added assumption that all units of the IpRNN layer have the same retention factor, a. The gradient terms during tack-propagation through time can now be expressed as a product of several terms that have the form:
dy: = [11 a)) Tec + a]
(20)
dy:
where, / and k; are time step indices with / > k and 1 = 1 k. If an eigenvalue of the WT rec matrix is A, then the corresponding eigenvalue of the matrix [(1 - a W
rec + a can be written as
(1-a)A+ a
(21)
Looking at the eigenvalue of the gradient terms as computed in equation 21 temporal regularizer on the eigenvalues of the recurrent network. It can also be seen that by scaling to lie between 0 and 1, the operation of the network shifts between that of purely non-inertial
note that acts like
recurrent to a completely inertial network stuck in its initial state, respectively. This insight helps us understand why IpRNNs perform well in long memory tasks.
Hochreiter (Hochreiter and Schmidhuber 1997) defined the constant error carousel (CEC) as a central feature of the LSTM networks that allowed it to remember past events. In crude sense, this corresponds to setting the retention ratio, a = 1. Forget gates were subsequently added by Felix A. Gers (Gers et al.) (000) the original LSTM structure, that allowed the network to also crase unnecessary events that were potentially trapped in the CEC. This means that the average effective the CEC was made 1. A randomly initialized set of a values
weight of the self-connection with reasonably large number of cells appears to have similar functionality. By setting a < 1, the network is guaranteed to lose memory over time, but if some of the as are close to I, it may retain the information for longer time frame. Moreover, the regularization effect of as also prevents the eigenvalues of the recurrent network from becoming too small, ensuring that memory is never lost immediately. We expect that the IpRNN model has reduced representational power than gated RNN cells, not simply because it has 4x fewer parameters, but because the IpRNN state is guaranteed to fade with time whereas gated cell can potentially store state indefinitely.
15

---

Under review as a conference paper at ICLR 2020
D THE NEUROMORPHIC SIGNAL CHAIN
The AE mapping mechanism requires defining suitable time constants for the IpRNN cell being trained by backprop. This can be derived for continuous-time signals from sensors or real-world signals such as an audio input by taking into account the bandwidth of the incoming signals as described earlier. We illustrate a reference neuromorphic signal chain for processing audio input in Figure D.2 The data received from the audio sensor is first filtered by audio filtering stage such as the cochlea chips (Chan et al., 2007] Sarpeshkar, [998] Hamilton et al. (008). These systems typically implement mel-spaced filter banks. A neural network processes the filter outputs and drives an actuator system. minimal configuration implement the IpRNN cell in a neuromorphic platform is illustrated in F.2 is a memory array with spiking neurons attached to the periphery of the system. Each memory cell acts as transconductance stage it receives voltage spikes and generates scaled output current. These currents are summed by the Kirchoff's ' current law and integrated by the neurons. Readers familiar with memory design and computer architecture may identify this as an in-memory computational unit. An in-memory neural network accelerator is energy-e energy-efficient primarily because it eliminates movement of synaptic weights (Ghose et al.) 2018 Qiao et al. 2015) from the memory to a far-away processing module. Instead, the activations of the neurons network. The computation is no longer memory-bowe bound unlike RNN computation on von Neumann style architecture. Figure D.2 implements single spiking RNN stage (equivalent to IpRNN), with green and red boxes highlighting the input and recurrent kernels, respectively. The architecture can be modified implement fully connected layer by eliminating recurrent connections. Note that an equivalent configuration can be also set up in fully-digital neuromorphic systems such as (Frenkel al., 2019; Davies al., 2018 Akopyan et al. 2015) that do not suffer from noise and mismatch issues, but may consume more area and power.
weights and connectivity required to
transmitted to the other nodes in the
Sensor
Filtering
Spiking NN
Actuator//
UU
Valtage ipprr
Current aupout - wV
K
Voltage spiless next
Voltage spike train from previous stage
Input kemel
Spiking low
Figure D.2: Top: A neuromorphic signal chain. Bottom: Architecture of an SNN accelerator implementing an IpRNN.
E SPIKER A SYSTEM-LEVEE SPIKING SIMULATOR
Spiker is crunsenn-siimlltorr for simulating large networks of spiking neurons and synapses us- ing the Basic Linear Algebra Subprogams (BLAS) libraries. It is written in Python and uses the
16

---

Under review as a conference paper at ICLR 2020
Numpy (Oliphant 2006) library. There is also PyTorch (Paszke al. 2017) version that supports Graphical Processing Unit (GPU) acceelecccr....nn.. Unlike spiking simulation tools like Brian2 (Stimberg et al. 2019) or NEST (Gewaltig and Diesmann 2007). which are general-purpose solvers of Ordnary Differential Equations (ODEs), Spiker is a highly-specific simulator for systems where the only differential equation implemented is that of a first-order LPFs. The simulator is not designed to solve any differential equation. Instead, it allows us to run massive simulations of large networks comprising neuron and synapse models whose building blocks include first-order LPFs.
E.
HOW DOES SPIKER WORK?
Key idea #1 Constraining the support to first-order LPFs has the advantage that it allows us to create closed-form solutions to the differential equations at all time-step resolutions without loss of accuracy. The differential equation corresponding to a first-order LPF is the following:
dx di
T
x
(22)
The closed-fore form solution to this, ignoring the initial conditions takes the form
I =
(23)
useful property of the exponential function is that --1++22 ee Therefore, we have a simple closed-form method to compute the state of a LPF at any arbitrary time, given an initial condition. This implies that if all the building blocks of a system are made of modules which have an exponential solution, then, given their initial state, it is possible to compute the state of the entire system at some arbitrary time the future, precisely. This is a well-known property of all LTI systems.
Key idea #2 However, spiking system is not LTI. Therefore, it is not possible to predict the state of a spiking neural network at an arbitrary time in the future. Instead, the Spiker simulator takes tiny temporal steps and computes the state of the network variables at each step using the closed-form solution. At the end of each time-step, the neuron models check if it should spike and then resets the corresponding variable zero. The spiking input to the synapses is train of 1 1-bit values corresponding to the presence or absence of an incident spike from an upstream neuron. This implies that the precision of the spike-event is limited the resolution of the simulation time-step. However, the key insight here is that, if the signals being transmitted have bandwidth much smaller than the simulation time-step, the higher-order effects can be safely assumed to be gone.
Moreover, in spiking neurons, small amounts of jitter in the spike-timing is well-tolerated. This is because of the noise-cancelling property of the feedback loop. Therefore, the Spiker simulator allows the user to set the simulation time-step to a large value that offers fast transient simulations fast-simulation trades-off against the precision of the simulated spike-timing.
Finally, the simulator also allows us to program and simulate noise and mismatch effects in the neuron parameters, such as mismatch the spiking threshold and time-constants. The Spiker simulator was used to run all the SNN simulations in this paper.
F VISUALIZING MAPPING OF A RECURRENT SNN
The top row of Figure F33 shows the mapping mechanism in action for single test case. We note that the depth of the network increases, the quality of fit degrades, but is still close to perfect as measured by the NMSE metric. The bottom row of Figure shows the mapping mechanism in action for devices with very large C., = 1. We note that even in such cases, the performance in lower layers remains fairly stable and only gradually degrades as the depth increases.
G CONVERGENCE PLOTS FOR THE GOOGLE SPOKEN COMMANDS TASK
Figure G.4 shows the convergence plots for various values of a.
17

---

Under review as a conference paper at ICLR 2020
Recurrent boyer 1. Ca -. MMSE 0.971
Rocurrntt byer 2. Co NNSE 0.951
Outout feed forword layer. G, NMSE 0.569
nV n
ANN output
SMN output
ARN output
SNN output
ANN oucput
SNN outbut
Rocurrent layer Co, 1. MNSE 1.000.
Recurrnt layer 1. NNSE 0.485.
Outout food forword layer. G, NMSE 0.802.
m
MAT
Figure F.: Dynamics in two-layer RNN with 51 units per layer. The SNN output shown in the figures is the trace obtain by filtering the spike trains. The NMSE measures the quality of fit with indicating a perfect match and -o very bad fit as described in Equation
2.5
0.4
[1.11 Bosts 89
0.80 Baa 82
Besti.26
0:0.0 Bald
000.00
0.
1.0
0.2
Figure G.4: Performance of the IpRNN cell on the Google commands detection task: Categorical accuracy (Left) and Cross entropy loss (Right). We note that as higher values of and random sampling results in faster convergence and better accuracy.
H MAPPING THE GOOGLE SPOKEN COMMANDS NETWORK
Each 1-second recording from the dataset is transformed using the Mel-spectrogram into 25 frequency and 128 temporal bins. This sequence represents a single speech command for the ANN. Mapping the trained ANN an SNN is challenging because of the limitation described earlier; short length sequence does not give the SNN enough time to generate the spikes required to transmit information. On the other hand, long. To address this issue, we train the ANN using the short sequence with 128 bins and demonstrate the mapped SNN by feeding it the same spectrogram data that is up-sampled to MHz. The quality of the mapped recurrent SNN is demonstrated using the weights from the trained recurrent ANN for a single case in Figure H.5 The bottom row of Figure H.5 also shows the mapping mechanism in action for RNN layers that affected by mismatch with Cr 0.2. The results indicate excellent fit between the mapped and the original networks, even with mismatch. The prediction made by the SNN also matched that of the ANN. Unfortunately, it is computationally prohibitive compare the
too difficult to train longer length sequence that several thousand samples
18

---

Under review as a conference paper at ICLR 2020
Recurrent bye 1 G2 ap MMSE- 0.9085.
Recurrent baye 2. 5g-0. 0. KNSE 6489
Output dense loyer Co -0 NMSE 0 0.0647
ANN output
SMN output
ARN ousput
SNN output
ANN ourput
SNN OUtOUt
Recurrett eyer 1. Co 02. NNSE 0.8566
Recurrenc oyer 0.2 NHSE 2.0607.
Output derse layer, 0.2 MISE 0.58122
ARN
output
output
OUEDUE
Figure H.5: Mapping 2-layer 128 unit recurrent ANN trained to discriminate commands from the Google speech dataset to an equivalent recurrent SNN. The top row shows the mapping in action for neuron models unaffected by mismatch and the bottom row demonstrates it for mismatched units. The IpRNN cells have a 0.99.
accuracy results of the SNN to the reference model on the full dataset. We only demonstrate the mapping accuracy for the two recurrent layers of the network in Figure|H.5) for a single command in this paper. A complete analysis of the accuracy performance will require testing on a neuromorphic system and will be the focus of a follow-up work.
I EXTENDING THE LOW-PASS FILTERING IDEA TO OTHER RNN MODELS
Our goal with introducing IpRNN cell was to enable faster and better training of SNNs. However, the analysis performed here indicates that low pass filtering also provides temporal regularization features which can benefit ANN-RNNs such as LSTMs. Therefore, we propose to extend the LSTM formulation by applying a low pass filter at the output (h), and call it an IpLSTM cell:
Forget gate: ft Sigmoid yxz + Wree, +ott+00 by)
Input gate: it Sigmoid W,
+ Wree, (W, + bi)
Output gate: = Sigmoid(W.oo + 1r + bo)
State ot = Se ② a-1 + i @ Relu(W.It + 1--1 be)
Output he = or @ Relu(ce)
Filtered Output he = a @ h (1 (1-aa a) he
(24)
where, Wrees, W.. by indicate the recurrent kernel, input kernel, and bias for the corresponding gate or state. Similar formulations for other RNN cells such as GRU (Chung et al. 2014), IndRNN (Li
al. 2018), Phased-LSTMs (Neil et al. 2016), Convolutional LSTMs (Xingjian al. 2015).
can be easily made.
I.1 EXPERIMENTAL RESULTS FOR THE LPLSTM CELL
In this section, we benchmark the Low-Pass Long Short-Tern Memory ((pLSTM) cells to their unfiltered variants. In our experiments, the learning rate was set to 0.005 and normalized gradient was clipped to 1. Current works describe use of various task-specific initialization constraints to solve the addition and copying tasks better (Henaff et al. 2016 Le et al. 2015). Instead of that, we use
19

---

Under review as a conference paper at ICLR 2020
data-driven curriculum learning protocol in our experiments and are able to obtain dramatically improved performance on these tasks. The networks used for the copying and addition tasks are fairly small. We also test it on a character-level language modelling task with the Penn Treebank dataset (Marcus et al. 1994) using a large network with roughly 19M parameters(Kim et al.) (016)) We swap the IPL cells with LSTM cells in experiments and the network architecture and typer-parameter settings were left unchanged from the values reported by the original authors. summary of our observations are as follows: The IpRNN cell exhibits a dramatically improved performance over the SimpleRNN cell. The low pass filter appears to have temporal stabilization effect even for LSTM cells. However, when other regularization and stabilization techniques such as Srivastava Dropout(Srivastava et al. 2014) are introduced, the benefit appears muted. It is possible that the large networks with low-pass RNN layers require network architecture tweaks to benefit from the filtering property, but it was not investigated in this work.
1.1.1 TEMPORAL ADDITION TASK
The addition task (Le et al. 2013) involves processing two parallel input data streams of equal length. The first stream comprises random numbers E (0, 1) and the second is full of zeros except at 2 time steps. At the end of the sequence, the network should output the sum of the two numbers in the first data stream corresponding to the time-steps when the second stream had non entries. The baseline beat is a mean square error (mse) of 0.1767, corresponding to a network that always generates 1.
We first train the RNN cell being tested on a short sequence and progressively increase the length. Each curriculum used 10,000 training and 1000 test samples. The results are shown in Figures L6 where each stage of the curriculum learning process marked with bands of different colours. The width of the band indicates the number of epochs taken for convergence. The length of the task was incremented when the mse went below 0.001. With random initialization, the SimpleRNN cell failed to converge beyond sequence length 40, even with curriculum learning. On the other hand, both the IpRNN and LSTM cells benefit from the curriculum learning protocol. The IpRNN cell was able to transfer learning for sequences shorter than 150 steps. While, the benefits of curriculum learning appears to have reduced beyond that, the IpRNN cells were able to achieve better than 0.001 mse for sequences up to 642. The performance of the IpRNN cell is at par or slightly inferior other works in literature (Hochreiter and Schmidhuber) 1997; Arjovsky et al. 2016 [e et al. 2015 Hu al., 2018)) we achieved this result purely by random initialization. Another interesting outcome of this experiment was the effectiveness of a 2-umit LSTM cell in solving this task. It was was able to add sequences much longer(we we tested up to 100K) than any reported work (where the networks are only able to solve the task for about 1/100th of the sequence length).
Given the effectiveness of the training protocol, we made the task more complex by allowing the second stream to have up to 10 unmasked entries during training. The trained cell was tested with a data stream having more than 10 masked entries. The LSTM cell was successful in solving this problem mse less than 1e-3, indicating that it learnt a general add and accumulate operation. Figure [.6]shoosss the evolution of the gating functions, internal state, and the state variables of an LSTM cell that was trained only fixed length sequence of 100 with mse less than 0.001. Contrast this against the stable dynamics of the network trained by curriculum learning in Figure [.dd] in a 100K sequence with mse less than 1e-3 (Figure [66] indicating almost perfect long-term memory and addition. To our knowledge, this kind of generalized learning by an LSTM cell has not been shown before.
I.1.2 TEMPORAL COPYING TASK
We train the RNN cells on a varying length copying task as defined in (Graves et al. 2014) instead of the original definition (Hochreiter and Schmidhuber] 1997) Arjovsky et 20., This problem is harder to solve than the temporal addition task. The network receives a sequence of up to symbols (in the original definition, S is fixed) drawn from an alphabet of size K. At the end of S symbols, a sequence of T blank symbols ending with a trigger symbol is passed. The trigger symbol indicates that the network should reproduce the first S symbols in the same order. We first train the network on a short sequence (T=3) and gradually increase it (T<)00). The sequence length is incremented when the categorical accuracy is better than 99%.
20

---

Under review as a conference paper at ICLR 2020
Length of the masked addition task
Length of the masked addition task
23
310
312
446
585
644
oow.eeeerooooee 2 ENgSN SoNeONNAAtinn -as.aaaasaa-mmmmmmmmo me & @ Segeannenniiieee
0.04
1.0
0.8
0.03
MSE
0.6
0.4
0.01
0.2
0.0
0.00
250
500
750
1000
1250
1500
1750
80
100
120
140
160
Epoch
Epoch
(a) Curriculum learning with a 128 unit IpRNN
(b) Curriculum learning a 2 hidden unit LSTM
1.0
Forget gate
~
Forget gate
Input gate
Input gate
Dutput gate
Output gatn
0.8
Internal
Internal
Output
Output
Mask
Mask
0.6
Value
Value
0.4
0.2
0.0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
D.4
0.0
0.8
1.0
Time step
102
Time step
165
(c) Plain I.STM without curriculum learning
(d) Plain LSTM with curriculum learning
Figure 1.6: Curriculum learning the masked addition task. LSTM cell trained without curriculum learning results in unstable state variables [c) When trained with curriculum learning it looks much
more stable (d) Stars in and (d) indicate value of the add mask.
21

---

Under review as a conference paper at ICLR 2020
(8+8, Ke8) with (00,
Copy task (S=8, K= with E13.200)
300
0.
Categorical cross entropy loss
0.35
Categorital cross entopy lose
0.150
0.30
0. 125
0.25
0.100
0.20
0.076
0.060
0.10
0.05
0.000
0.00
400
Epoch
Epoch
Copy task (SHR, T-2) with T 500)
Copy task (8r8, T-235) with e [10.25]
2
0.010
Calegerical cIoes entropy lose
Categorcal cross entropy (oss
0.000
0.15
0.006
0.004
0.08
0.002
0.00
0
50
100
1.00
200
250
300
350
0
20
60
100
120
1.0
160
Epoch
Epoch
Figure 1.7: Curriculum learning on the variable length copying task for a 256 unit IpRNN (top left)
and 128 unit LSTM cell (top right) and 128 unit IpLSTM cell(bottom left and right).
The SimpleRNN cell failed at this task even for T=30 with categorical accuracy dropping to 84% when it predicted only S S T blank symbols. The IpRNN cell was able to achieve 99% accuracy for up to 120 time steps. After that, it generates T blank entries accurately but the accuracy of the last symbols drops (For T=200, was 96%). However, the LSTM cell achieves more than 99.5+ % accuracy for all tested sequence lengths, highlighting the advantage of curriculum learning. This is big improvement over reported results (Arjovsky et al. 2016 Graves et al. 2014 Bai et al. 2018) where LSTM cells solved the task for much smaller values of S and T.
We observed stability issues when training an LSTM cell for sequences longer than 30, even if eventually converged by using smaller learning rates and gradient norm scaling. This makes good test case to validate the temporal regularization property of the IpLSTM cell. In our tests, the IpLSTM cell converged without instability with categorical accuracy higher than 99.5% for all tested values of T(E [3, 500). It also to generalized larger values of than the other cells (< 25). The IpLSTM cell exhibited gradual degradation in performance for larger values of S. We stop our simulations when the categorical accuracy fell below 96%. These results are summarized in Figure I.7
I.1.3 PENN TREEBANK (PTB) CHARACTER MODEL
We studied temporal regularization in a network trained on the PTB dataset (Marcus et al. 1994) by replacing the LSTM cells by its low pass variants. We choose model with 19M parameters (Kmm et al. 2016) and trained all variants using the same settings as described in (Kim et al. 2016) for 25 epochs. We note that both IpLSTM cells converge to better score the training set and marginally poorer score on the train/validation set (refer T). The IpLSTM cell with relu activation also converges unlike the plain relu LSTM cell validating our claim temporal regularization.
22

---

Under review as a conference paper at ICLR 2020
Table 6: Impact of temporal regularization on the Penn Treebank model.
Activation
Train perplexity
Validation Perplexity
Test Perplexity
LSTM
relu tanh tanh 41.0545
approx. 641
approx. 641
Fails to converge
46.0948
83.9807
80.0873
IpLSTM
84.6127
81.7519
relu
43.0602
84.1484
80.6946
J ENERGY CONSUMPTION ESTIMATION
In this section, we describe the procedure used for comparing the power consumption of the in- memory architecture against Cortex-M4 processor. This processor chosen it is of the most common low-power MCU platforms in use today.
In our analysis, we make a highly-optimistic estimate for the performance of the Cortex-M4 (ARM 2019). We assume that there are no cache misses, that multiply and add operations take one clock cycle, the read from DRAM only consumes pJ/bit, and also assume that the MCU fully available for RNN computation. In particular, note that the memory cost DRAM access should also include the address and data bus power consumption. This has been completely ignored in this analysis to keep things highly optimal on the Cortex-Me M4 side. We see that in such configuration, the Cortex-M4 consumes only few mW of power. In practice, the active power consumption of such processors tend to be in hundreds of mW (Rethinagiri et al. 2014).
estimate performance of the in-memory unit (also 180nm technology for known neuron implementation (Nair and Indiveri 2019)). These results are then tabulated and system activity is modelled for two RNN models in Figure J.8a (2 layer RNN with 128 units/layer) and Figure J.8b (4
layer RNN with 500 units/layer).
The energy cost for the Cortex-M4 is modelled by the following equation:
Esot (Clkmmd Nmut + Clkiadd Nadd) Edlk + M Emem
(25)
where
Ero: Total power consumed
Clkimut Number of clocks for multiply
Nmut: Number of multiply operations in the task.
Clkiaaa Number of clocks for addition.
Nodd: Number add operations the task.
Eax: Energy consumed by the processor per clock.
M: Number of memory bit accesses
Emem: Energy cost of accessing single bit.
The energy cost for the in-memory architecture is modelled by the following equation:
Erst (N Espike + M) Nepikes
(26)
where
N: Number of neurons
Expike: Energy per spike
Nepikes: Total number of spikes. This is computed for the ΣA model by computing the average firing rate as a function of the desired bit precision. This is approximating by equating the desired firing to achieve desired Signal to Noise Ratio (SNR) (Pavan et al. 2017).
of the ΣA neuron that of oversampled clock necessary
M: Memory access cost. This modelled by the the product of the read current while
spike is active.
23

---

Under review as a conference paper at ICLR 2020
Each of the terms in the computation of the power consumption of the Cortex and in me-oryy systems are turn calculated based on number of hardware and operational assumptions that are listed in the tables. We note improved energy ehergy-efficiency of several hundred times across the board in both configurations We also note that the mnergy-savinge is much higher for the larger network. This is simply because the number of sequential compute operations increases quadratically.
Input
Sampling layer
Number Layers
Output
a-bit SRAM arca factor
NN cathaa misess in Carties a Meroory cosd only includes 3RAM access exciu des datta bus
properties Hardeare
4775--C
assumptions
DRAM, Energy fbit 3a 192E-10
ARM Cortec CEnerggy gy
ARM Cortect (Emergy/ clock!
Multiply clocks
Addition clocks
Energy per spikk
Spike width
Per neuron leakago wasted
Desired SDR
5S5--44
LEUE-10
1.00--11
LO0E-06
1.06-00
5.002 di
Computation cast per kample
Computational cost
Clocks needad for multiply
linput stagel
Clocka reeded for add linput
stage)
Clocks needee
for muttilly Piiden!
Clocks needed for add Ihidder)
Clocks newdad for muttiply loulputi
for add loutpuli
Clocks needee
Tobal clocks per sumple
Total clocks
190f
3
3
Paill-03
7
Corlex
access (per sampli,, Inpub
(per sample,
hiddonns
access four sample. Niddens)
tread sertie)
accesses miW
Total (evclucting memoryl in niW
including memory lin mWW
excluding
DRAM lin
mmza)
3Rdfi-02
3,30
78:E
LobE-co
::36-60
280E-00
Overssmpling
Average flring
Total number nearr
Total number
Blased current amplitude
Memory read
Neuron powerr comsumption
Total power m1#7
Chip area On
computation
In-memory
spikka
1.66-00g
294
4316-05
3206-07
1.39-115
4.60E-c6
4806-33
74gB-64
energy energy-fflcknncy
564.40
(compute
rmeemoyy
Factor
improvement
anargy-fficieecy
compute, pattom ideal)
317.71
(a) For two-layer RNN with 128 units per layer
Data, network properties and Hardware assumptions destto UL.I
dimersion
Sampling rabe
Hiddden size
hädden
Output size
factor unaa
includes
sosia
Meroory only
Energy/
DRAM:
Cortex
Cortex
Mutttply
Energy per
Spike
Per neuron
teakago
1826-i 3t
1.3E-004 04
100E
1006--00
1/0EE Up
Computational
Clocks needed for multigly
Clocks needed udd linput
Clocks needed multiply
Clockls
Clocks needed multipl
needed (output)
Total per sample
1.00-og
5:00--02
10E-006
1 100E-00
193E-04
1BoEro4
2ocE-08
Hemory cast
sample,
acciss (per
Number
memory
memory ect scceess
(per sample,
Number
access (per
Number
memory
cample.
Total number of memory accesses
Energy cost of memory accesses
Total power (axcludding memoryl in
Total power including memory lin
Chip area, escluding
1.0E+og
103E-00
BoE--o
204Er08
382ExD3
3 3.8E-00
7.00--01
0.44
In-mamory
Oversampling
amplitude read
mnW0
mmz?
Factor of improvement
margy-afficeeny (computo
2,195.46
Factor of improvement (tee compute.. memory accees pattern idwal)
1,230.42
(b) For four-layer RNN with 500 units per layer
Figure J.8: Energy consumption comparison between in-memory RNN accelerator implementing the IpRNN model against a Cortex M4.
24


Reviews:
[{"review_id": "rkxbmCrTKS", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper deals with neuromorphic computing architectures for solving deep learning problems. Given that the energy-efficiency is a critical need for modern machine learning tools, this is an important study in my opinion. Until now, most of the spiking neuron based works don't really apply for realistic deep model learning. Therefore, advancements such as these would be very useful. \n\nHowever, my biggest concern is that the paper may be a bit difficult to grasp for the general ML audience. I myself am not an expert in this topic and had quite a bit of difficulty as an ML researcher in understanding parts of the paper. As a matter of fact, some of the supplementary material was quite helpful to understand the entire context. So, I suggest to move some of the basics from there to the main paper. Also, some of the performance comparison results in the supplementary section are more convincing and clear compared to the results in the main paper. \n\nBased on my understanding, the algorithmic approach seems logical and the empirical results are convincing. However, I am not in a position to assess the level of novelty. So, I am giving the benefit of the doubt to the authors, given that this is a critical topic for the ML community.   ", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "Bkx31xO0YB", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper promises efficient training of spiking neuron models using back-propagation. The authors say that this is important because spiking networks offer significant energy savings, yet they typically perform poorly compared to prevalent artificial neural networks (ANNs). They aim to support this with several synthetic and one real-world task.\n\nI selected to reject the paper based on the support for the main claim of the paper being weak and inadequately demonstrated. No comparisons are to baselines are presented in the text. Reproducibility is hampered by a great deal of the experimental details missing.\n\nOn the experimental results:\n* In section 4.1, some justification is needed for why these experiments are relevant. \n* In sections 4.4 and 4.5, experiments are discussed, but results are not presented.\n* If space is a limitation, I would strongly prefer a table of actual results for sections 4.4 and 4.5 rather than Figure 2.\n* Since the encoding of the task in the spiking and non-spiking cases is so drastically different, making that crystal clear would be great.\n\nOther comments:\n* The goal of energy savings is very appealing, so it would help to give *some* quantitative measure of savings promised.\n* Similarly, the claim about the poor performance of spiking neurons would benefit from some quantitative measures.\n* A figure showing the operation of the novel SigmaDelta neuron, compared to the standard aI&F neuron, would be welcome.\n\nOverall I think the paper would benefit from focussing much more on the results, making the claims as quantitatively apparent as possible. \n\n", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "BJgkNLcAtr", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The submission describes an adaptive spiking neuron model that is based on the Laplace transform of the model output.\nThis reformulation allows to train a recurrent neural network of spiking neural network with good performances.\nThree tasks are proposed to assess the recurrent net, on synthetic data and real signal.\n\nI think this contribution could not be accepted for a methodological problem: the main idea is to use neuromorphic chips that use spiking neural networks with an approximation that is basically similar to an artificial neural network. The authors need to introduce a complex rescaling of the time step, repercussion on the computation. This point should be shown experimentally. The authors introduced a modification of the aI&F and are applying low pass filtering the spike trains. \n\nAs all the results are obtained on a python simulator, it is difficult to assess the interest and the applicability to real neuromorphic chips.\nAs pointed out by the authors, the main interest of these architectures are the lower energy consumption than CPU/GPU-based architecture.\nUnfortunately, it is not possible to assess if the proposed neuron model is working on energy efficient architecture.\n\n\n[1] Nair, M. V. and Indiveri, G. (2019). An ultra-low power sigma-delta neuron circuit. In 2019 IEEE International Symposium on Circuits and Systems (ISCAS), pages 1–5.\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents a new mechanism to train spiking neural networks that is more suitable for neuromorphic chips.
While the text is well written and the experiments provide an interesting analysis, the relevance of the proposed neuron models to the ICLR/ML community seems small at this point. My recommendation is that this paper should be submitted to a more specialised conference/workshop dedicated to hardware methods.
Decision: Reject