High performance RNNs with spiking neurons

1 INTRODUCTION . Exponential growth in computational power and efficiency have played a vital role in the development of neural networks and their training algorithms . However , it has also led to higher design complexity and increasing difficulty to keep up with Moore ’ s law ( Schaller , 1997 ; Waldrop , 2016 ) . Recent years have also seen the movement of computation from data-centres to compact , distributed , and portable embedded systems . These factors have created a demand for energy-efficient AI-capable devices , leading to the development of dedicated and optimized von Neumann-style Artificial Neural Network ( ANN ) accelerators ( Aimar et al. , 2018 ; Cavigelli and Benini , 2016 ; Chen et al. , 2016 ) and a renewed interest in neuromorphic systems ( Chicca et al. , 2014 ; Frenkel et al. , 2019 ; Davies et al. , 2018 ; Moradi et al. , 2018 ; Akopyan et al. , 2015 ; Qiao et al. , 2015 ; Neckar et al. , 2019 ) . A key difference between neural networks deployed on von Neumann systems and most neuromorphic platforms is the use of spikes or train of pulses to represent signals in the latter . Such networks are called Spiking Neural Networks ( SNNs ) . Spiking neuromorphic systems have a number of features that inhibit their use in real-world problems : ( 1 ) mixed-signal circuits suffer from Complementary Metal-Oxide-Semiconductor ( CMOS ) mismatch ( Pelgrom et al. , 1989 ) that degrades performance ; ( 2 ) rate-based SNNs generate a large number of spikes to represent signals that reduces their energy benefit ; ( 3 ) complex spiking dynamics makes it difficult to train them using gradient-descent methods . In this paper , we describe a new neuron model that addresses these problems and discuss how its Low-Pass Filter ( LPF ) abstraction enables training spiking Recurrent Neural Networks ( RNNs ) using the backpropagation algorithm ( or Backprop ) . This is a significant breakthrough as it enables training and deployment of energy-efficient spiking neural network devices without simulating complex spiking dynamics . 2 PROCESSING-IN-MEMORY FOR RNNS . Consider an RNN layer with n nodes . At each time-step , the processor computes one or several matrix products of the form y = W.x , where y and x are vectors of length n , and W is a 2-D matrix of size n× n. When operating on a von Neumann system with batch-size 1 , as is common in most edge applications , the bottleneck in throughput and energy-efficiency is the O ( n2 ) memory fetches of W at every timestep . An in-memory matrix multiplier addresses this problem . It is a module where a “ read ” from the memory location of the W matrix , using “ X ” as the “ address ” gives out “ Y ” , without ever moving W . This leads to a quadratic reduction in energy consumption . Processing in-memory systems have been implemented for various tasks such as DNA sequencing ( Ghose et al. , 2018 ) , graph processing ( Ahn et al. , 2015 ) , etc and with dramatic reduction in energy consumption . The energy reduction from in-memory computing is well established , but the key challenge with deploying such systems is the absence of compatible algorithms . In this paper , we propose an RNN model for such a system . The implementation of the in-memory module depends on how x and y are encoded and transported . It can be synchronous or asynchronous and analogue or digital . We adopt an asynchronous digital approach as it offers some implementation advantages . Encoding information in binary digital format is less susceptible to noise in comparison to analogue . Asynchronous signalling allows the energy-consumption to scale in proportion to chip activity , while also permitting lowlatency response . Chips implementing such schemes have been published in literature ( Qiao et al. , 2015 ; Moradi et al. , 2018 ) . The model presented in this paper is designed to integrate on similar chips ( A reference framework is described in supplementary section D ) . However , most of the algorithmic ideas presented in this paper are general and applicable to a range of compute systems . 3 THE SPIKING NEURON MODEL . Spiking neuron models for encoding signals typically use rate- or time-coded spike-generation schemes ( Diehl et al. , 2015 ; Rueckauer et al. , 2017 ; Bohte , 2012 ; Mostafa , 2017 ) . In rate-coding , the firing rate of the neuron is proportional to the input signal . Therefore , achieving high data-resolution with rate-coding requires a large number of spikes , which is not energy-efficient ( Nair and Indiveri , 2019 ) . To address this problem , several time-coding schemes have been proposed . In this work , we build on existing models to propose an Adaptive Integrate and Fire ( aI & F ) neuron model , which can also be interpreted as an asynchronous Σ∆ circuit ( Nair and Indiveri , 2019 ; Bohte , 2012 ; Yoon , 2016 ) . This mechanism reduces the spike count by only transmitting the error between an internal state and the input . The aI & F neuron model implemented with current-mode neuromorphic circuits ( Nair and Indiveri , 2019 ) can be described by the following equations : τmem dImem dt = αL ( IL − Imem ) − s+ i ( 1 ) τw ds dt = αs ( Imem − IL ) − s ( 2 ) Imem = 0 , when Imem > ∆ ( 3 ) where the currents Imem and IL represent the “ membrane potential ” and “ leak reversal potential ” variables . The term s represents the neuron adaptation current , i the input current , τmem the membrane time constant , αL a gain factor , ∆ the threshold , αs the adaptation coupling parameter and τw is the adaptation time constant . The aI & F model is a feedback loop that tries to decrease the difference between the i ( t ) and s ( t ) . The difference , i ( t ) − s ( t ) , is filtered with gain , αL , and time constant , τmem . When the output of this filter , Imem , exceeds the spiking threshold , ∆ , Imem is reset and a spike is generated . The Σ∆ circuit model used in this work is different from the aI & F model in the computation of the feedback term . Instead of filtering Imem , we operate on the spike train generated by the spiking neuron . This ensures that the noise inserted by the spike-generation mechanism is also suppressed by the Σ∆ feedback . The modified feedback equation is as follows : τk ds dt = αs ( δiIin − IL ) − s ( 4 ) Where , δi indicates the spike train and Iin is a programmable maximum current value that the analogue filter implementation can generate . The product δiIin models the feedback filter that integrates a current Iin for the duration of the spikes . Figure 1a shows a block diagram of the circuit implementation of the Equation 3 with the modification described by Equation 4 . In this diagram , F ( s ) is a first order LPF that receives inputs to the neuron . H ( s ) is a first-order low-pass filter that produces s ( t ) in response to spikes generated by the neuron . E ( s ) is also a first-order low-pass filter on the difference between the input current i ( t ) and the feedback signal s ( t ) . When the output of E ( s ) , Imem , exceeds the spiking threshold ( ∆ ) of the neuron , a spike-event is produced . With each spike-event , s ( t ) increases and i ( t ) − s ( t ) decreases . Figure 1b shows the asynchronous Σ∆ feedback loop in action for a test-case . It can be shown that the Laplace domain representation of the output spike train can be expressed by the equation : Y ( s ) = X ( s ) F ( s ) E ( s ) 1 +H ( s ) E ( s ) + N ( s ) 1 +H ( s ) E ( s ) ( 5 ) where , N ( s ) indicates the Laplace-domain representation of the noise introduced into the loop , for example by the spike generation mechanism . The Σ∆ loop described by Equation 5 is similar to a continuous-time Σ∆ modulation loop ( Pavan et al. , 2017 ) with the key difference being that the output spikes are unipolar . This is valuable because the output of a conventional Σ∆ loop is always active ( +1 or−1 ) , whereas the asynchronous model is only active at the time of a spike event , making the model more energy-efficient . The input signals to the neuron may be encoded as spikes trains or as continuous analogue values from a sensor . The transmitted analogue signal is reconstructed from a spike train by simply low-pass filtering it . An advantage of the Σ∆ neuron model is that it comes with a low-pass filter in its input stage . Therefore , a Σ∆ neuron is a “ codec ” - It can both encode an analogue signal into a spike train and decode an incoming spike train back to the transmitted analogue signal . As the low-pass filter at the input stage is agnostic of the type of input to it , a Σ∆ can encode and decode both types of signals - spike trains or continuous analogue ones . A description of the biological motivation and noise-filtering properties of the model is provided in supplementary section A . The circuit implementing this model has been fully characterized in Nair and Indiveri ( 2019 ) . 4 TRAINING A SPIKING RNN WITHOUT SIMULATING SPIKES . The neuron model introduced in the previous section allows us to train a recurrent SNN by treating the spiking neurons as LPFs and modifying the recurrent ANN equations suitably . We will show that the trained weight parameters of the recurrent ANN model can be mapped to a recurrent SNN , without additional training . We measure the effectiveness of this mapping procedure by comparing the temporal dynamics of the neurons in the recurrent ANN to the low-pass filtered spike trains generated by the spiking neurons in the corresponding recurrent SNN . In this demonstration , we use high precision synaptic weights . This is typically not available in most spiking neuromorphic platforms . However , the same mapping procedure can be used for mapping ANNs trained with binary or noisy weights . Before introducing the mapping procedure , we describe three operations that are needed for it . Input re-scaling : When implementing an SNN in mixed-signal neuromorphic systems , the state variables of the neuron are represented by voltages or currents that are of the order of mV or nA . Using such small values when training an ANN in software may lead to computational instability . To avoid this we train the network with normalized input signals and re-scale the parameters and activation functions after training . For example , if a single layer calculation is represented as y = σnl ( W · x ) , where σnl is a non-linear activation function , x , W and y are the inputs , weights , and outputs from the layer , respectively , then , to re-scale the inputs by a factor γ , the activation function used in the ANN will be modified , during inference , as y = σnl ( W · γ · x ) , where , σnl ( . ) = σnl ( . γ ) Low-pass filtering : The input signal can be reconstructed from the spikes trains generated by the Σ∆ neuron if s ( t ) tracks i ( t ) ( Equation 4 ) . This is because s ( t ) is obtained by filtering the neuron spike train . This is why a Σ∆ neuron can be modelled as an LPF with time constant τw . The LPF approximation ignores the high-frequency components injected by the spiking mechanism , as they are suppressed by feedback loop ( the N ( s ) term in the transfer function of Σ∆ neuron , Equation 5 ) . We model this by using a discrete-time Euler approximation to incorporate a LPF-term at the output of the RNN stage . This results in the Low-Pass Recurrent Neural Network ( lpRNN ) cell by a simple tweak to the classical equation : yt = α yt−1 + ( 1− α ) σ ( Wrec · yt−1 +Win · xt + b ) ( 6 ) where , σ , and · denote non-linearity , element-wise Hadamard product and matrix multiplication functions , respectively . The variables α , yn , xn , Wrec , Win , and b represent the retention ratio vector , input vector , output vector , recurrent connectivity weight matrix , input connectivity weight matrix , and biases , respectively . The subscripts on variable y and x indicate the time step . α models the time constant of the recurrent ANN , and it is matched to the SNN time constant by setting it to α = e −Ts τs where , Ts is the time-step of the input data-stream fed to the recurrent ANN , and τs is the feedback time constant of the Σ∆ neurons used in the desiredSNN . For example , if the recurrent ANN is being trained to detect speech from an audio-signal , then Ts should be set equal to the time difference between the consecutive samples . The value of τs , for the ANN set to min ( τw , τmem ) in the Σ∆ equations . τs must therefore be chosen such that the signals being transmitted lie well within the pass-band of the feedback filter . This ensures that all the in-band components are transmitted well , even when different neurons in the systems have different values of τs , for example , due to mismatch . This is an important observation for mixed-signal systems , where mismatch effects may result in different neurons to have differing time-constants . We will demonstrate that the effect of device mismatch is well-tolerated for most practical cases and leads to gradual degradation in performance as it increases . It must be noted that while computing Ts or the bandwidth of an audio or sensor measurement is easy , it is not trivial for data-sets such as text . Saturating non-linearity : It has already been shown in the literature that the Rectified Linear Unit ( ReLU ) non-linearity is a good non-linear model of the aI & F neuron ( Yoon , 2016 ; Bohte , 2012 ) . However , the Σ∆ neuron also filters incoming spike trains using a low-pass filter which limits its maximum output current to Iin ( see Equation 4 ) . To model this effect we can either set Iin in the SNN to the largest activation output found in the ANN simulation or clamp the maximum output of the activation function in the ANN simulation to Iin . In our experiments , we do the latter .

Summary: This paper deals with neuromorphic computing architectures for solving deep learning problems. Given that the energy-efficiency is a critical need for modern machine learning tools, this is an important study in my opinion. Until now, most of the spiking neuron based works don't really apply for realistic deep model learning. Therefore, advancements such as these would be very useful.