Training Recurrent Neural Networks Online by Learning Explicit State Variables

Recurrent neural networks ( RNNs ) allow an agent to construct a staterepresentation from a stream of experience , which is essential in partially observable problems . However , there are two primary issues one must overcome when training an RNN : the sensitivity of the learning algorithm ’ s performance to truncation length and and long training times . There are variety of strategies to improve training in RNNs , the mostly notably Backprop Through Time ( BPTT ) and by Real-Time Recurrent Learning . These strategies , however , are typically computationally expensive and focus computation on computing gradients back in time . In this work , we reformulate the RNN training objective to explicitly learn state vectors ; this breaks the dependence across time and so avoids the need to estimate gradients far back in time . We show that for a fixed buffer of data , our algorithm—called Fixed Point Propagation ( FPP ) —is sound : it converges to a stationary point of the new objective . We investigate the empirical performance of our online FPP algorithm , particularly in terms of computation compared to truncated BPTT with varying truncation levels . 1 INTRODUCTION . Many online prediction problems are partially observable : the most recent observation is typically insufficient to make accurate predictions about the future . Augmenting the inputs with a history can improve accuracy , but can require a long history when there are long-term dependencies back in time . Recurrent Neural Networks ( RNNs ) ( Elman , 1990 ; Hopfield , 1982 ) learn a state which summarizes this history . Specifically , RNNs contain recurrent connections to their hidden layers which allow past information to propagate through time . This state need not correspond to a true underlying state ; rather , the state is subjective and constructed to facilitate prediction . RNNs have been widely used , in speech recognition ( Hinton et al. , 2012 ; Graves et al. , 2013 ; Miao et al. , 2015 ; Chan et al. , 2016 ) , image captioning ( Mao et al. , 2014 ; Lu et al. , 2016 ; Vinyals et al. , 2014 ) , speech synthesis ( Mehri et al. , 2016 ) and reinforcement learning ( Hochreiter and Schmidhuber , 1997 ; Düll et al. , 2012 ) . Despite these success , there are significant stability and computational issues in training RNNs online ( Pascanu et al. , 2013 ; Tallec and Ollivier , 2017 ) . In the online setting , the agent faces an unending stream of data and on each step the agent must update its parameters to make a new prediction . RNNs are typically trained either using Backpropagation-through-time ( BPTT ) ( Werbos , 1990 ) or approximations to an algorithm called Real-Time Recurrent Learning ( RTRL ) ( Williams and Zipser , 1989a ; Pearlmutter , 1995 ) , although there are methods that appeal to other principles ( see Murray ( 2019 ) for instance ) . The update for BPTT is a variant of standard backpropagation , computing gradients all the way back in time . This approach is problematic because the computational cost scales linearly with the number of time-steps . A more common alternative is truncated BPTT ( T-BPTT ) ( Williams and Peng , 1990 ) which only computes the gradient up to some maximum number of steps : we truncate how far back in time we unroll the network to update the parameters . This approximation , though , is not robust to long-term dependencies ( Tallec and Ollivier , 2017 ) . Approximate gradients can also be computed online by RTRL ( Williams and Zipser , 1989b ) . This online algorithm , however , has high computational complexity per step and therefore is not commonly used in practice . Recently , there have been some efforts towards approximating gradients for back-propagation , both for feedforward NNs and RNNs . Synthetic gradients and BP ( λ ) ( Jaderberg et al. , 2017 ) use an idea similar to returns from reinforcement learning : they approximate gradients by bootstrapping off estimated gradients in later layers ( Jaderberg et al. , 2017 ; Czarnecki et al. , 2017 ) . There are several methods that approximate RTRL—which is itself an approximation of the true gradient back in time—including NoBackTrack ( Ollivier and Charpiat , 2015 ) , Unbiased Online Recurrent Optimization ( UORO ) ( Tallec and Ollivier , 2017 ) which uses an unbiased rank-1 approximation to the full matrix gradient , and Kronecker Factored RTRL ( Mujika et al. , 2018 ) which uses a Kronecker product decomposition to approximate the RTRL update for a class of RNNs . Finally , there are some methods that use selective memory back in time to compute gradients for the most pertinent samples , using skip connections ( Ke et al. , 2017 ) . All of these methods , however , attempt to approximate the gradient back in time , for the current observation and state , and so suffer to some extent from the same issues as BPTT and RTRL . In this paper , we investigate an alternative optimization strategy that does not attempt to approximate the gradient back in time . Instead we learn the state variables in the RNN explicitly . These new variables are optimized to both improve prediction accuracy , and to maintain consistency in producing the next learned state variables . This second constraint is a fixed-point formula for the states under the given RNN dynamics.1 We develop a provably sound stochastic update for the new fixed-point objective , which we then use to develop an online algorithm for training RNNs . The algorithm explicitly optimizes state vectors and RNN parameters with many efficient one-step—or short term multi-step updates—across a buffer . Instead of focusing computation to get a more accurate gradient estimates for this time-step , our algorithm , called Fixed Point Propagation ( FPP ) , can more effectively use computation to update prediction accuracy across states . We demonstrate that the algorithm is effective on several problems with long-term dependencies , and improves over T-BPTT , particularly in terms of stability and computation . 2 PROBLEM SETTING AND BACKGROUND . We consider a partially observable online setting , where an immediate observation is not sufficient for prediction . More formally , assume there is a sequence of n observations , o1 , . . . , on , which provide only partial information about an unknown underlying sequence of states . After obtaining an observation oi , the agent makes a prediction ŷi and sees the actual outcome yi . The goal is to minimize this prediction error . Given only oi , the agent is unlikely to make accurate predictions about yi , because oi is not a sufficient statistic to predict yi : p ( y|oi , oi−1 , oi−2 , . . . ) 6= p ( y|oi ) . The agent could obtain a lower prediction error using a history of observations . Unfortunately , the agent may require a prohibitively long history , even if this history could have been summarized compactly . An alternative is to construct state using a Recurrent Neural Network ( RNN ) , by learning a stateupdate function . Given the current ( constructed ) state st−1 ∈ Rk , and a new observation ot ∈ Rd , the parameterized state-update function fW : Rk×Rd → Rk , with parameters W , produces the next ( constructed ) state st = fW ( st−1 , ot ) . For example , fW could be a linear weighting of st−1 and ot , with a ReLu activation : fW ( st−1 , ot ) = max ( [ st−1 , ot ] W,0 ) . More complex state-updates are possible , like the gating in Long Short-Term Memory ( LSTM ) ( Hochreiter and Schmidhuber , 1997 ) . The objective is to adapt these parameters W for the state-update to minimize prediction error . For the current state st , a prediction is made by parameterized function gβ : Rk → Rm with learned parameters β . For example , the prediction could be a linear weighting of the state , gβ ( s ) = β > s . We denote the prediction error as ` β : Rk × Rm → R for a given β . For example , this loss could be ` β ( st ; yt ) = ‖gβ ( st ) − yt‖22 . 1Recurrent Backpropagation and related variants ( Almeida , 1987 ; Pineda , 1987 ; Scellier and Bengio , 2017 ; Liao et al. , 2018 ) also use fixed points for their optimization , but in a different way . These algorithms only address a restricted class of RNNs , that assume a fixed input and converge to a single low-energy state—a fixed point of the dynamics for that given input . These RNNs are actually highly related to Graph NNs ( Scarselli et al. , 2008 ) , because the temporal nature only arises from cyclic connections , rather than from temporal data . Their problem setting is fundamentally different from our online prediction setting , and is usually used for associative memory with Hopfield networks or semi-supervised problems . Recurrent Backpropagation can not be used for our setting and so we do not further discuss this class of RNN algorithms . The goal in RNNs training is to minimize , for some start state s0 , min β , W n∑ i=1 ` β ( fW ( ... fW ( fW ( s0 , o1 ) ︸ ︷︷ ︸ s1 , o2 ) , ... , oi ) ; yi ) . ( 1 ) Computing gradients for this objective , however , can be prohibitively expensive . A large literature on optimizing RNNs focuses on approximating this gradient , either through approximations to RTRL or improvements to BPTT . RTRL ( Williams and Zipser , 1989b ) uses a recursive gradient form , which can take advantage of gradients computed up until the last observation to compute the gradient for the next observation . This estimate , however , is only exact in the offline case and thus RTRL is an approximation of the true gradient in our online setting . Further , in either online or offline , RTRL requires O ( k4 ) computation per observation ( recall k is the dimension of the state ) . In BPTT , gradients are computed back in time , by unrolling the network . In the online setting , it is infeasible to compute gradients all the way back to the beginning of time . Instead , this procedure is truncated T steps back in time . T-BPTT is suitable for the online setting , and requires O ( Tk2 ) computation per step , i.e. , for each observation . Arguably the most widely-used strategy is T-BPTT , because of its simplicity . Unfortunately , T-BPTT has been shown to fail in settings where dependencies back in time are further than T ( Tallec and Ollivier , 2017 ) , as we affirm in our experiments . Yet , the need for simple algorithms remains . In this work , we investigate an alternative direction for optimizing RNNs that does not attempt to estimate the gradients of ( 1 ) . Note that in addition to a variety of optimization strategies , different architectures have also been proposed to facilitate learning long-term dependencies with RNNs . The most commonly used are LSTMs ( Hochreiter and Schmidhuber , 1997 ) , which use gates to remember and forget components of the state . Other architectures include clockwork RNNs ( Koutník et al. , 2014 ) , phased LSTMs ( Neil et al. , 2016 ) , hierarchical multi-scale RNNs ( Chung et al. , 2016 ) , dilated RNNs ( Chang et al. , 2017 ) , and skip RNNs ( Campos et al. , 2017 ) . In this work , we focus on a general purpose RNN algorithm , that could be combined with each of these architectures for further improvement .

Summary: Background: The authors consider the problem of training RNNs in an online fashion. The authors note that RNNs are trained using BPTT, which prevents them from being trained in an online fashion.  There have been various approximations which has been proposed which are based on RTRL or approximations to RTRL, as current approximations based on RTRL has high computational complexity.