Spike-inspired rank coding for fast and accurate recurrent neural networks

1 INTRODUCTION . Neuromorphic computing is the study and use of computational mechanisms of biological neural networks in mathematical models , software simulations , and hardware emulations , both as a tool for neuroscience , and as a possible path towards improved machine intelligence ( Indiveri , 2021 ) . In fact , much of the recent progress in machine learning ( ML ) is attributed to artificial neural networks ( ANNs ) , which share certain characteristics with biological neural networks . These biological analogies of ANN models include a connectionist graph-like structure ( Rosenblatt , 1958 ) , parallel computing over multiple synaptic weights and neurons ( Indiveri & Horiuchi , 2011 ) , and the nonvon Neumann collocation of memory and processing at each synapse and neuron ( Sebastian et al. , 2020 ) . On the other hand , state-of-the-art ( SOTA ) ANNs for ML often miss several other neuromorphic mechanisms that are fundamental in biological neural systems . A characteristic example is that of “ spikes ” , i.e . the short stereotypical pulses that biological neurons emit to communicate ( Maass , 1997 ; Ponulak & Kasinski , 2011 ) . This is a principal neuromorphic feature , characterizing the brain and a category of bio-plausible models known as spiking neural networks ( SNNs ) , but not the most successful ANNs , suggesting unexploited potential and limited understanding of spike-based ∗Corresponding author approaches . Despite the stereotypical , unmodulated shape of spikes , spiking neurons can encode continuous values , for example in their average firing rate ( Brette , 2015 ) , which is abstracted into the continuous activation of conventional artificial neurons ( Pfeiffer & Pfeil , 2018 ) . Perhaps more interestingly , spikes can also carry information in their specific timing , i.e . through temporal coding that modulates when spikes are fired ( Brette , 2015 ) . Partly because individual spikes can encode information sparsely and rapidly , biological nervous systems and spiking neuromorphic systems can be extremely energy-efficient and fast in the processing of their input stimuli ( Qiao et al. , 2015 ; Davies et al. , 2018 ; Zhou et al. , 2021 ; Yin et al. , 2021 ) . This efficiency and speed are key motivations for much of the research on SNNs for potential applications in ML and inference . Moreover , owing to neuronal and synaptic dynamics , SNNs are more powerful computational models than certain ANNs in theory ( Maass , 1997 ; Moraitis et al. , 2020 ) . In practice , SNNs have recently surpassed conventional ANNs in accuracy in particular ML tasks , by virtue of short-term synaptic plasticity ( Leng et al. , 2018 ; Moraitis et al. , 2020 ) . Spike coding itself can also add computational power to SNNs by increasing the dimensionality of neuronal responses ( Izhikevich , 2006 ; Moraitis et al. , 2018 ) . However , in practical ML terms , firstly , spike coding poses difficulties to precise modelling and training that require ad hoc mitigation ( Mostafa , 2017 ; Pauli et al. , 2018 ; Pfeiffer & Pfeil , 2018 ; Woźniak et al. , 2020 ; Comşa et al. , 2021 ; Zhang et al. , 2021 ) . Secondly , SNNs are particularly difficult to analyse mathematically and rigorous ML-theoretic spiking models are scarce ( Nessler et al. , 2013 ; Moraitis et al. , 2020 ) . Thirdly , they require unconventional neuronal models , which do not fully benefit from the mature theoretical and practical toolbox of conventional ANNs ( Bellec et al. , 2018 ; Woźniak et al. , 2020 ; Comşa et al. , 2021 ) . As a result , the efficiency and speed benefits of temporal coding for ML have been hard to demonstrate with SOTA accuracy in real-world tasks . For instance , very recent literature on temporal coding with SNNs ( Comşa et al. , 2021 ; Zhang et al. , 2021 ; Zhou et al. , 2021 ; Mirsadeghi et al. , 2021 ; Göltz et al. , 2021 ) uses rank coding ( RC ) , i.e . the temporal scheme where the first output neuron to spike encodes the network ’ s inferred label ( Thorpe & Gautrais , 1998 ) , and it applies it to speed up inference on tasks such as hand-written digit ( MNIST ) ( Lecun et al. , 1998 ) recognition . However , when applied ( Zhou et al. , 2021 ) to more difficult datasets such as Imagenet ( Deng et al. , 2009 ) , the accuracy is significantly lower than in non-spiking versions of the same network ( Szegedy et al. , 2015 ) . Moreover , in these demonstrations there are no directly measured benefits compared to non-spiking ANNs . Even though temporal coding is usually not described in terms comparable with non-spiking ANNs , there are in fact ANN architectures with certain analogies to RC , when viewed from a particular angle . Namely , in an SNN that receives a sequence example of several input steps , e.g . in a sequenceclassification task , one implication of RC is that the computation for each sequence example can be halted after the first output neuron has spiked and produced an inferred label , even if several steps of the input sequence still remain unseen . Therefore , this is an adaptive type of processing that dynamically chooses the time and computation to be dedicated to each sequence . From this perspective , RC is related to ANN techniques such as Self-Delimiting Neural Networks ( Schmidhuber , 2012 ) , Adaptive Computation Time ( Graves , 2016 ) , and PonderNet ( Banino et al. , 2021 ) , which are also concerned with when computation should halt . However , these methods do not aim to adaptively reduce the processed length of an input sequence as RC does , but rather to adaptively add timesteps of processing to each step of the input sequence . A possibly more deeply related method is that of adaptive early-exit inference ( Laskaridis et al. , 2021 ) . In this case , the forward propagation of an input throughout layers in a deep neural network at inference time is forwarded from an early layer directly to a classifier head , skipping the layers that remain higher in the hierarchy , if that early layer crosses a confidence threshold . In certain cases , early exit has been applied to sequence classification with recurrent neural networks ( RNNs ) , where the threshold-crossing dictates when in the sequence of an input example the network should output its inference , saving indeed in terms of time and computation ( Dennis et al. , 2018 ; Tan et al. , 2021 ) . This timing decision based on the first cross of a threshold is similar to inference with RC . However , these early-exit models were not specifically trained to learn a rank code . It is conceivable that this mismatch between training and inference is suboptimal . In addition , this non-RC training of early-exit inference models does not apply the computational savings and speed benefits also to the training process . Taking together the limitations and benefits of SNNs and of other approaches , what is needed is a strategy that introduces aspects of RC into conventional , non-spiking ANNs , including during training . This would potentially reap the speed and efficiency benefits of this neuromorphic scheme , without abandoning the real-world usability and performance of ANNs . In addition , these insights into neural coding could feed back to neuroscience . Here we describe and demonstrate such a strat- egy for temporal coding in deep learning and inference with ANNs . The general concept is simple . Namely , even though ANN activations are continuous-valued and therefore neurons do not have to rely on time to encode information as in SNNs , ANNs too could time their outputs and , importantly , they could learn to do so . To achieve this in an RNN such as long short-term memory ( LSTM ) ( Hochreiter & Schmidhuber , 1997 ) , we back-propagate through time ( BPTT ) from a strategic and early time step in each training example ’ s sequence . The time step is decided by the rank-one , i.e . first , output neuron activation to cross a threshold . As a result of this Rank Coding ( RC ) during training , the network learns not only to minimize the loss , but implicitly also to optimize its outputs ’ timing , reducing time to insight as well as computational demands of training and inference . In our experiments , we provide several demonstrations , with advantages compared to SNNs from the literature as well as compared to conventional ANNs , including in MNIST classification and speech recognition . Moreover , we show that our method could be applied to SNNs directly as well . 2 DEEP LEARNING OF RANK CODING . Algorithm 1 RC-training Given : a training set of N example sequences Si = { xi0 , ... , xiT } and corresponding labels yi ; an RNNR ; and a threshold θ . 1 : i = 0 2 : while i < N do . iterate over training examples 3 : i++ ; t = 0 ; Ti = duration of sequence Si 4 : tsp , i ← Ti . latest possible first “ spike ” 5 : while t < tsp , i do . iterate through input sequence steps until first spike 6 : t++ 7 : activation ŷit = Rt ( Si ) 8 : for all output neurons ŷj do 9 : if ŷjit ≥ θ then . Inferred label=j 10 : tsp , i ← t . rank-one spike time 11 : ŷi = ŷit . activation at tsp is considered asR ’ s overall output from Si 12 : BPTT ( R , Loss ( ŷit , yi ) ) . BPTT from tsp only 13 : break . Done with this sequence Algorithm 1 shows the RC-training process . The network decides the rank-one spike timing tsp of its outputs based on its output-layer activations and a threshold θ ( line 10 ) . The loss function can be a common on such as cross-entropy . There is no explicit parametrization on time , and only one instantaneous output ŷit is used in the loss ( line 12 ) . Therefore , RC-training does not explicitly optimize the timing tsp of the network ’ s outputs . Thus , it is not obvious that learning of the timing aspect can emerge . Nevertheless , timing is implicitly , albeit indeed optimized , as seen in what follows . With random initialization , the activations are nearly uniformly distributed across output neurons , and are smaller than the threshold , throughout the sequence . Without an earlier cross of a threshold , the algorithm applies BPTT from the last step of the sequence ( line 4 : tsp = T ) . As training advances , minimizing the error between the outputs and the labels minimizes the entropy of the output distribution , i.e . causes the activations at the end of each sequence example to be concentrated around one of the output neurons , such that the maximum activation ŷmaxtsp is maximized . Through BPTT from that last time step , credit is assigned to earlier time steps as well . Progressively through training , this causes outputs to cross the threshold earlier and earlier , under the condition that relevant , credit-assigned input signal does exist earlier . This conditional acceleration causes the optimization of output timing . This is also shown mathematically in Appendix A . Importantly , the insight that it is through the minimization of entropy ( Eq . 4 ) that timing is minimized , gives us access to a mechanism for balancing between minimizing the loss and minimizing the timing . Specifically , we introduce to the loss function a regularization term , weighted by a hyperparameter β , such that minimization of the loss rewards entropy H of the outputs ŷtsp : LRC ( ŷtsp , y ) = L ( ŷtsp , y ) − βH ( ŷtsp ) . ( 1 ) RC-inference after RC-training is similar to Algorithm 1 but the backward pass ( line 12 ) is not applied . It should be noted that the inference stage on its own performed in this manner , where a threshold decides the timing of the output , is a version of what has been called early exit or early inference . In our implementation , the expectation is that the model learns to encode information in the rank order of its output ’ s timing , because that timing is integrated into the learning process . In this sense , at inference , the model does not merely exit early , but it does so through an underlying learned rank code ( RC ) . Our experimental demonstrations confirmed that the loss is minimized through RC-training , although BPTT ’ s application time-step varies between training examples , and that timing is also minimized down to an optimal floor , as the conditional arguments above ( and Eq . 8 in the Appendix ) predict .

Summary: The authors introduce a new way to train RNNs using rank order coding (ROC). With ROC the label is given by the first readout unit to reach a threshold. As soon as this happens, the processing is stopped, and BPTT is used from that particular time step, using the predictions at that particular time step and the ground truth. This will encourage the neuron with the right label to be as active as possible at that particular time step, and thus its threshold will tend to be reached earlier in the future. This is desirable, as the latency of the decision will decrease. Furthermore, the speed-accuracy trade-off is tunable by varying the threshold.  The authors validate their idea using LSTMs on two toy problems, and then on MNIST and on the Google Speech Command dataset.