Online Learning Of Neural Computations From Sparse Temporal Feedback

1 Introduction . In biological and artificial spiking neurons , information processing depends on synaptic connectivity as well as intrinsic variables . How a neuron integrates synaptic inputs over time changes its inputoutput function , leading to a large variety of neural computations [ 1 ] . In biology , the input-output function of a neuron is modified not only by synaptic plasticity [ 2 ] , but also by alterations of the neuron ’ s electrophysiology , e.g. , through changes in the composition of trans-membrane ion channels , so-called intrinsic plasticity [ 3 , 4 ] . Synaptic interactions influence the amount of neurotransmitter released from the presynaptic neuron or the amount of receptors in the postsynaptic terminal and therefore how strongly neurons are connected to each other . Intrinsic properties of the neuron , on the other hand , affect the neuron ’ s threshold and resting potential and how strongly postsynaptic potentials are amplified [ 4 ] . Correspondingly , the computational properties of an artificial neuron depend on the values of its synaptic weights and intrinsic parameters . For instance , the neural computation performed by a leaky integrate-and-fire neuron [ 5 ] or a leaky resonate-and-fire neuron [ 6 ] with identical synaptic inputs and synaptic weights but different intrinsic parameters varies strongly within and across neuron models ( Figure 1A ) . Learning in artificial neurons is thus the process of modifying a set of synaptic weights and intrinsic parameters to reach a target input-output function . However , unlike their biological counterparts , most computational approaches to learning in spiking neurons are limited to effects of 1 . Department of Experimental Psychology , University of Oxford , Oxford , United Kingdom 2 . Bernstein Center for Computational Neuroscience Berlin , Berlin , Germany 4 . Institute of Science and Technology Austria , Klosterneuburg , Austria 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . alterations in synaptic connectivity . Here , we derive a gradient-based online learning rule for leaky integrate-and-fire neurons and leaky resonate-and-fire neurons that can learn the synaptic connectivity along their intrinsic parameters from sparse temporal feedback in the teacher-student paradigm . 1.1 Related Work . Applying gradient-based optimisation and the backpropagation algorithm [ 7 , 8 ] to spiking neurons is difficult since the gradient of spike trains is ill-defined : it is zero everywhere except at spike times , where it is infinite . Methods to overcome the problem of non-differentiable spike trains include using gradients with respect to the probability density of spike-times [ 9 ] , minimising the temporal distance between output and target spike times [ 10 ] , temporal smoothing of synaptic-currents [ 11 ] or the discrete error signals caused by target-spikes [ 12 , 13 , 14 ] or by temporal gradient accumulation using the adjoint method [ 15 ] . In contrast to approaches that are directly optimising output spike times , there are also approaches that optimise the activity of spiking neurons through their network activity [ 16 , 17 ] , firing rate response function [ 18 ] or via the auxiliary objective of generating a fixed amount of output spikes for certain input spike patterns [ 19 , 20 ] . Alternatively , so-called `` surrogate gradient '' methods replace the non-differentiable threshold function with a smooth surrogate function during the backpropagation of errors [ 21 , 22 , 23 ] . Using surrogate gradients and backpropagation through time [ 24 ] , multilayer spiking neural networks can be successfully trained on a wide range of supervised learning tasks [ 25 , 26 , 27 ] , even when errors are backpropagated at spike times only [ 28 ] . Our learning rule is closely related to the work by Gütig and Sompolinsky [ 19 ] and Memmesheimer et al . [ 29 ] . The tempotron learning rule [ 19 ] trains leaky integrate-and-fire neurons to emit an output spike for some input spike patterns , but to remain silent for others . Memmesheimer et al . [ 29 ] adapted this learning rule to learn synaptic weights that map input spike trains to a target output spike train . In contrast to the tempotron learning rule , the feedback signal does not indicate the class membership of an input spike pattern but target spike times : on the occurrence of each missed or false output spike the distance to the spiking threshold is minimised or respectively , maximised . If an output spike does not fall within a tolerance window around a target spike , a single parameter update is performed and the training procedure and neuron model are reset to the beginning of the training sequence . Consequently , the target spike train is learned one by one , from the first to the last spike of the target sequence , eventually requiring many replays of the sequence until the algorithm converges . We extend the learning rule by Memmesheimer et al . [ 29 ] by an event-dependent scaling factor , such that synaptic weights and intrinsic parameters can be learned mutually and online , i.e. , from a continuous stream of inputs without resetting the training procedure after each error and without relying on a tolerance window . Like Memmesheimer et al . [ 29 ] , we train without backpropagating errors through time [ 24 ] . We hypothesise that a learning signal that indicates when a neuron should spike , paired with a gradient-based online parameter update rule , is sufficient to learn the complex temporal input-output relationships of a neuronal computation . 2 Methods . 2.1 Teacher-student paradigm . Learning in artificial neurons relies on finding the set of parameters that facilitates a certain target computation . One approach to test an algorithm ’ s learning capability is to test its ability to map randomly sampled input spike patterns to randomly sampled output spike trains ( e.g . used in [ 12 , 13 , 23 ] and in [ 30 ] to compare algorithms ) . However , under this testing paradigm , the difficulty of the task is directly dependent on the statistics ( e.g . dimensionality and firing rate ) of the input spike patterns and target spikes . Therefore , the size and shape of the solution space and thus the difficulty of the task are unknown and strongly fluctuate with the choice of hyperparameters . As a consequence , the set of parameters that solves the problem may be large and thus even suboptimal learning algorithms can still yield good performance . For example , some of the solutions to a learning task may not be anticipated , like networks with biologically implausible high firing rates [ 27 ] . Further , using random input-output spike patterns of fixed length is unsuitable for evaluating an online-learning algorithm since , as the length of the random pattern increases , the probability of a solution existing is vanishing to zero . The teacher-student paradigm [ 31 , 32 ] circumvents this problem by generating target spike times using a model ( i.e . the teacher neuron ) . The learning task is then to recover the teacher ’ s parameter that determine its neural computation with a student neuron ( e.g . in [ 29 ] “ Reconstruction of Synaptic Weights ” ) . Like this , the learning problem is well defined , as a single point in parameter space . In our case , learning signals are derived from comparing the output spike train of the teacher neuron with the output spike train of the student ( Figure 1B ) . The teacher-student paradigm has several advantages , e.g. , over the test performance in a standardised supervised learning task . First , if the teacher and student neuron have the same architecture , it is guaranteed that a unique solution to the training problem exists . Second , the student ’ s parameters , and thus performance , can be compared to the solution during and after the training process . Finally , since the teacher is used to generate training samples , an infinite amount of unique and continuous training data can be sampled . In summary , by controlling the solution space of the task , the teacher-student paradigm can be used to precisely test the capability of performing spike-pattern recognition and the validity of assumptions of a learning algorithm . 2.2 Error signals . As the dynamics of the teacher and student neuron evolve over time , three possible events can occur when comparing their output spike trains ( Figure 1B ) . First , the student may generate a spike at the same time as the teacher ( a hit ) . Second , the teacher may generate a spike when the student does not ( a miss , denoted by d− = −1 ) . Third , the student may generate a spike when the teacher does not ( a false positive , denoted by d+ = 1 ) . Since false positives can be detected by the student neuron without feedback , only missed spikes require error signals between teacher and student neurons that are ungraded , unsigned and therefore `` unary '' . Intuitively , a miss occurs when the student ’ s membrane potential V is too low and a false positive occurs when the student ’ s membrane potential is too high . We base learning on this observation and update the student ’ s parameters θ such that the distance to the threshold is reduced for each miss , but increased for each false positive spike [ 19 , 29 ] . 2.3 The event-dependent scaling online learning rule . Our learning rule consists of a gradient-based parameter update like in Memmesheimer et al . [ 29 ] ( for a derivation see appendix A ) and an additional event-dependent scaling factor λ ( ∆d± ) . We thus call it the Event-Dependent Scaling ( EDS ) rule . More specifically , we use the partial derivative of the membrane potential with respect to a parameter and perform a gradient ascent step on each miss ( d− = −1 ) and a gradient descent step on each false positive ( d+ = 1 ) spike : ∆θ ∝ −λ ( ∆d± ) d± ∂V ( t ) ∂θ . ( 1 ) Here , the EDS factor λ ( ∆t ) scales parameter updates according to the truncated sigmoidal function λ ( ∆d± ) = 1000− 1000 exp ( ln ( 0.5 ) ( min ( ∆d± , ∆θ ) m ) b ) , ( 2 ) where ∆θ is a cutoff parameter , m determines the midpoint and b the slope of the sigmoidal and ∆d± denotes the time since the last parameter update . Thus , the size of the parameter update is event-dependent . In practice , we use ∆θ = 75 , m = 500 and b = 4 , which is plotted in figure 2A . In contrast to the EDS rule , surrogate gradient methods [ 23 , 26 , 27 ] use a scaling function λ ( V ( t ) ) that is based on the current membrane potential . We compare the EDS factor λ ( ∆d± ) to the the SuperSpike surrogate gradient λ ( V ) = ( β|V − VThr|+ 1 ) −2 [ 23 ] . Using the EDS rule , parameter updates are temporal , stochastic and online . Updates are temporal since the underlying function that we optimise is a function evolving over time i.e . the parameterised function that describes the membrane potential . They are stochastic since updates are performed for a single teacher-student pair and not as averages over multiple initialisations . And lastly , they are online since gradients are calculated for the current point in time only , without propagating errors through time [ 24 ] and therefore without taking the recurrent nature of neural dynamics into account ( cf . [ 26 ] ) . To speed up learning , we use the Adam optimiser [ 33 ] with default parameters ( β1 = 0.9 , β2 = 0.999 , = 1e−8 ) and per parameter learning rates , which are scaled according to the range from which target values are sampled ( ηw = 35e−6 , ητs = 7e −4 , ητm = 28e −4 , ηVr = 7e −5 and ηw̃ = 8e −5 , ηb = 15e−6 , ηω = 33e−7 , ηṼr = 8e −5 ηĨr = 8e −5 ) .

Summary: This paper introduces the event-dependent scaling (EDS) rule to use in neural network training that allows neural computations to be learned online - i.e. without error propagation. It presents a teacher-student paradigm for problem formulation, then characterizes the error signal and EDS rule. It then describes its architectures (leaky integrate-and-fire or leaky resonate-and-fire neurons, Poisson-distributed input spike trains, and other standard parameters), and metrics (of convergence and robustness). Next, it discusses resulting learning dynamics and analyses the effectiveness of EDS empirically, and concludes by acknowledging concerns behind lack of theoretical backing and a rarefied testing paradigm.