1 The NoBackTrack algorithm 1 . 1 The rank-one trick : an expectation-preserving reduction

We introduce the “NoBackTrack” algorithm to train the parameters of dynamical systems such as recurrent neural networks. This algorithm works in an online, memoryless setting, thus requiring no backpropagation through time, and is scalable, avoiding the large computational and memory cost of maintaining the full gradient of the current state with respect to the parameters. The algorithm essentially maintains, at each time, a single search direction in parameter space. The evolution of this search direction is partly stochastic and is constructed in such a way to provide, at every time, an unbiased random estimate of the gradient of the loss function with respect to the parameters. Because the gradient estimate is unbiased, on average over time the parameter is updated as it should. The resulting gradient estimate can then be fed to a lightweight Kalman-like filter to yield an improved algorithm. For recurrent neural networks, the resulting algorithms scale linearly with the number of parameters. Small-scale experiments confirm the suitability of the approach, showing that the stochastic approximation of the gradient introduced in the algorithm is not detrimental to learning. In particular, the Kalman-like version of NoBackTrack is superior to backpropagation through time (BPTT) when the time span of dependencies in the data is longer than the truncation span for BPTT. Consider the problem of training the parameters θ of a dynamical system over a variable h ∈ Rn subjected to the evolution equation h(t + 1) = f(h(t), x(t), θ) (1) where f is a fixed function of h and of an input signal x(t), depending on parameters θ. The goal is online minimization of a loss function ∑︀ t lt(y(t), y(t)) between a desired output y(t) at time t and a prediction1 y(t) = Y (h(t), φ) (2) 1The prediction y may not live in the same set as y. Often, y encodes a probability distribution over the possible values of y, and the loss is the logarithmic loss l = − log py(y). 1 ar X iv :1 50 7. 07 68 0v 2 [ cs . N E ] 2 0 N ov 2 01 5 computed from h(t) and additional parameters φ. A typical example we have in mind is a recurrent neural network, with activities ai(t) := sigm(hi(t)) and evolution equation hi(t + 1) = bi + ∑︀ k rkixk(t) + ∑︀ j Wjiaj(t), with parameter θ = (bi, rki, Wji)i,j,k. If the full target sequence y(t)t∈[0;T ] is known in advance, one strategy is to use the backpropagation through time algorithm (BPTT, see e.g. [Jae02]) to compute the gradient of the total loss LT := ∑︀T t=0 lt with respect to the parameters θ and φ, and use gradient descent on θ and φ. However, if the data y(t + 1) arrive one at a time in a streaming fashion, backpropagation through time would require making a full backward computation from time t + 1 to time 0 after each new data point becomes available. This results in an Ω(t2) complexity and in the necessity to store past states, inputs, and outputs. A possible strategy is to only backtrack by a finite number of time steps [Jae02] rather than going back all the way to t = 0. But this provides biased gradient estimates and may impair detection of time dependencies with a longer range than the backtracking time range. By contrast, methods which are fully online are typically not scalable. One strategy, known as real-time recurrent learning (RTRL) in the recurrent network community,2 maintains the full gradient of the current state with respect to the parameters: G(t) := ∂h(t) ∂θ (3) which satisfies the evolution equation G(t + 1) = ∂f(h(t), x(t), θ) ∂h G(t) + ∂f(h(t), x(t), θ) ∂θ (4) (by differentiating (1)). Knowing G(t) allows to minimize the loss via a stochastic gradient descent on the parameters θ, namely,3 θ ← θ − ηt ∂lt ∂θ ⊤ (5) with learning rate ηt. Indeed, the latter quantity can be computed from Gt and from the way the predictions depend on h(t), via the chain rule ∂lt ∂θ = ∂lt(Y (h(t), φ), y(t)) ∂h G(t) (6) However, the full gradient G(t) is an object of dimension dim h × dim θ. This prevents computing or even storing G(t) for moderately largedimensional dynamical systems, such as recurrent neural networks. 2This amounts to applying forward automatic differentiation. 3We use the standard convention for Jacobian matrices, namely, ∂x/∂y is the matrix with entries ∂xi/∂yj . Then the chain rule writes ∂x ∂y ∂y ∂z = ∂x ∂z . This makes the derivatives ∂lt/∂θ into row vectors so that gradient descent is θ ← θ − (∂lt/∂θ).