Kernel RNN Learning (KeRNL)

1 INTRODUCTION . Animals and humans excel at learning tasks that involve long-term temporal dependencies . A key challenge of learning such tasks is the problem of spatiotemporal credit assignment : the learner must find which of many past neural states is causally connected to the currently observed error , then allocate credit across neurons in the brain . When the time-dependencies between network states and errors are long , learning becomes difficult . In machine learning , the current standard for training recurrent architectures is Backpropagation Through Time ( BPTT , Rumelhart et al . ( 1985 ) , Werbos ( 1990 ) ) . BPTT assigns temporal credit or blame by unfolding a recurrent neural network in time up to a horizon length T , processing the input in a forward pass , and then backpropagating the error back in time in a backward pass ( see Fig . 1a ) . From a biological perspective , BPTT – like backpropagation in feedforward neural networks – is implausible for many reasons . For each weight update , BPTT requires using the transpose of the recurrent weights to transmit errors backwards in time and assign credit for how past activity affected present performance . Running the network with transposed weights requires that the network either has two-way synapses , or uses a symmetric copy of the feedforward weights to backpropagate error . In either case , the network must alternatingly gate its dynamical process to run forward or backward , and switch from nonlinear to linear dynamics , depending on whether activity or errors are being sent through the network . From both biological and engineering perspectives , there is a heavy memory demand : the complete network states , going T timesteps back in time , must be stored . The time-complexity of computation of the gradient in BPTT scales like T , making each iteration slow when training tasks with long time scale dependencies . Although T should match the length of the task or the maximum temporal lag between network states and errors for unbiased gradient learning , in practice T is often truncated to mitigate these computational costs , introducing a bias . The present work is another step in the direction of providing heuristics and relaxed approximations to backpropagation-based gradient learning for recurrent networks . KeRNL confronts the problems of efficiency and biological plausibility . It replaces the lengthy , linearized backward-flowing backpropagation phase with a product of a forward-flowing temporal eligibility trace and a spatial sensitivity weight . Instead of storing all details of past states in memory , synapses integrate their past activity during the forward pass ( see Fig . 1b ) . The network does not have to recompute the entire gradient at each update , as the time-scales of the eligibility trace and the sensitivity weight are learned over time . 2 RELATED WORK . In recent years , much work has been devoted to implementing backpropagation algorithms in a more biologically plausible way , partly in the hope that more plausible implementations might also be simpler . The symmetry requirement between the forwards and backwards weights can be alleviated by using random return weights ( Lillicrap et al . ( 2016 ) and Nøkland ( 2016 ) ) , however , learning still requires a separate backward pass through a network with linearized dynamics . Neurons may be able to extract error information in the time derivative of their firing rates using an STDP-like learning rule ( Bengio et al . ( 2015 ) ) , with error backpropagation computed as a relaxation to equilibrium ( Scellier & Bengio ( 2017 ) ) , at least for learning fixed points . Other work has focused on replacing batch learning with online learning . Typically , BPTT is implemented in a setting where data is prepared into batches of fixed sequence length T and used to perform learning in a T -step unrolled graph ; however , online learning , with a constant stream of data error signals , is a more natural description of how the world supplies a learning system with data . BPTT without truncation struggles with online learning , as it must repeatedly backpropagate the error all the way through a continuously expanding graph . Since computation of the unbiased gradient scales with the length of the graph , gradient computation increases linearly with time . For a task with T timesteps , the total computation of the gradients scales like T 2 . Real Time Recurrent Learning ( RTRL , Williams & Zipser ( 1989 ) ) and Unbiased Online Gradient Optimization ( UORO , Tallec & Ollivier ( 2017 ) , Ollivier et al . ( 2015 ) ) deal with this issue by keeping track of how the synaptic weights affect the hidden state in a feedforward way . Decoupled Neural Interfaces ( DNI Jaderberg et al . ( 2016 ) ) estimates the truncated part of the gradient by continually predicting the future loss with respect to the hidden state . KeRNL offers this same advantage , in addition to other benefits . RTRL requires that the network keep track of an unwieldy rank-3 tensor , which could not be stored by any known biological entities . UORO factorizes this into rank-2 objects but still requires non-local computations like vector norm operations . Finally , DNI requires an entire separate network to keep track of the synthetic gradient . KeRNL is distinguished by its simplicity , requiring only rank-2 tensors . All computations are local , and synapses need to integrate over only a few relevant quantities . 3 THE LEARNING RULE . Consider a single-layer RNN in discrete time ( indexed by t ) with readout , input , and hidden layer activations given by yt , xt , and ht , respectively ( boldface represents vectors , with vector entries denoting the activity of individual units ) . The dynamics of the recurrently connected hidden units are given by : ht+1 = σ ( gt+1 ) = σ ( W recht +W inxt + b ) ( 1 ) where W rec , W in are the recurrent and input weights , b are the hidden biases , σ is a general pointwise non-linearity , and gt represents the summed inputs ( pre-nonlinearity ) to the neurons at time t. The readout is given by yt = σout ( W outht + bout ) . The objective function is C = C ( { ( x0 , y0 ) , · · · ( xT , yT ) , ŷT } ) , where ŷT is the target output , in the case where error feedback is received at the end of an episode of length T , and C = ∑T t=0 C t when errors Ct = C ( { ( x0 , y0 ) , · · · ( xt , yt ) , ŷt } ) are received continuously over the episode . The parameters W in , W rec , W out , b , bout are trainable . Assuming the readout weights and biases ( W out , bout ) are trained in the usual way , the KeRNL update rule for the RNN weights W = { W in , W rec } is simply : ( ∆Wjk ) t = −η ∑ i δtiβije t jk , ( 2 ) where δti ≡ dC t dhti = ∑ l dCt dytl dytl dhti is the gradient of the cost with respect to the current hidden state , βij is a set of learned sensitivity weights , and etjk ≡ ∑ τ K ( τ , γj ) σ ′ ( u ) ∣∣∣∣ gt−τj st−τk ( 3 ) is a local eligibility trace ( Williams ( 1992 ) ) consisting of a temporally filtered version of the product of presynaptic activation and a postsynaptic activity factor . The temporal filter or kernel , K , in the eligibility has learnable time-scales ; in this manuscript we use the simplest version of a lowpass temporal filter , a decaying exponential with a single time-constant γj per neuron : K ( τ , γj ) = exp ( −γjτ ) , though one can imagine many other function choices with multiple timescales . The role of the eligibility is to specify how strongly a synapse Wjk should be held responsible for any errors in neuron j at the present time , on the basis on how far in the past the presynaptic neuron k was active . Here st−τk = { x t−τ k , h t−τ−1 k } stands in for the activation of the neuron presynaptic to the synapse being updated . 1 . Since the eligibility trace can be computed during the forward pass , KeRNL does not require backpropagating the error through time . Furthermore , KeRNL only uses at most rank-2 tensors , so neurons and synapses could plausibly do all of the required computation . The contrast between BPTT and KeRNL is depicted in Fig . 1a , b . KeRNL emerges from the following Ansatz : ∂hti ∂ht−τj ≈ βijK ( τ , γj ) . ( 4 ) We call ∂hti/∂h t−τ j , a key term in the computation of the gradient , the sensitivity tensor in an extension of the usage in Fiete & Seung ( 2006 ) ) . This sensitivity describes how the activity of neuron j at a previous time t − τ affects the activity of neuron i at the current time t. While the true sensitivity is a 4-index tensor summarizing many interactions based on the many paths through which activity propagates forward in a recurrent network , we approximate it with a product of a ( learnable ) rank-2 sensitivity weight matrix β and a temporal kernel K with ( learnable ) inversetime coefficients γ . The sensitivity weights βij describe how strongly neuron j affects neuron i on average , while the temporal kernel describes how far into the future the activity of a neuron affects the other neurons for learning . We describe how to learn these parameters ( β , γ ) in the next section . We arrive at KeRNL by using our Ansatz for the sensitivity ( 4 ) in the computation of a gradientbased weight update , instead of using the true sensitivity . First , we write down the full gradient rule for a recurrent network . If the parametersWij are treated as functions that can vary over time during a trial , then the derivative can be written as a functional derivative : dC dWjk = 1 T T∑ t=0 δC δWjk ( t ) . ( 5 ) This is simply mathematical notation for the “ unfolding-in-time ” trick , in which the network and weights are assumed to be replicated for each time-step of the dynamics of a recurrent network , 1The rule to update the hidden layer biases is given by ( ∆bi ) t = −η ∑ i δ t iβije t j where e t j =∑ τ exp ( −γjτ ) σ ′ ( gt−τj ) integrates over a clamped input . and separate gradients are computed for each time-replica of the weights ; the actual weight updates are simply the average of the separate weight variations for each time-replica . We next apply the sensitivity lemma Fiete & Seung ( 2006 ) to express gradients with respect to weights as gradients with respect to input activations , times the presynaptic activity : δCδWjk ( t ) = δC δgj ( t ) sk ( t ) . Thus , dC dWjk = ∑ τ δC δgj ( t− τ ) sk ( t− τ ) = ∑ i ∑ τ δC δhi ( t ) δhi ( t ) δgj ( t− τ ) sk ( t− τ ) ( 6 ) = ∑ i t∑ τ=0 δC δhi ( t ) δhi ( t ) δhj ( t− τ ) σ′ ( u ) ∣∣∣∣ gj ( t−τ ) sk ( t− τ ) ( 7 ) By replacing the sensitivity δhi ( t ) /δhj ( t− τ ) with our Ansatz ( 4 ) , we arrive at our learning rule , KeRNL ( 2 ) . The time-dependent part of the computation– a leaky integral of the product of the presynaptic activity multiplied by the instantaneous change in the postsynaptic activity– can be computed during the forwards pass , without any backpropagation of activity or error signals .

Summary: The proposed kernel recurrent learning (KeRL) provides an alternative way to train recurrent neural network with backpropagation through time (BPTT) where the propagation of gradients can be skipped over different layers. The authors directly assume the sensitivity function between two layers with a distance of tau in a form of Eq. (7). The algorithm of BPTT is then approximated due to this assumption. The model parameters are changed to learn the network dynamics. The optimization problem turns out to estimate beta and gamma of the kernel function. The learned parameters are intuitive. There are a set of timescales to describe the memory of each neuron and a set of sensitivity weights to describe how strongly the neurons interact on average. The purpose of this study is to save the memory cost and to reduce the time complexity for online learning with comparable performance.