On the Curse of Memory in Recurrent Neural Networks: Approximation and Optimization Analysis

1 INTRODUCTION . Recurrent neural networks ( RNNs ) ( Rumelhart et al. , 1986 ) are among the most frequently employed methods to build machine learning models on temporal data . Despite its ubiquitous application ( Baldi et al. , 1999 ; Graves & Schmidhuber , 2009 ; Graves , 2013 ; Graves et al. , 2013 ; Graves & Jaitly , 2014 ; Gregor et al. , 2015 ) , some fundamental theoretical questions remain to be answered . These come in several flavors . First , one may pose the approximation problem , which asks what kind of temporal input-output relationships can RNNs model to an arbitrary precision . Second , one may also consider the optimization problem , which concerns the dynamics of training ( say , by gradient descent ) the RNN . While such questions can be posed for any machine learning model , the crux of the problem for RNNs is how the recurrent structure of the model and the dynamical nature of the data shape the answers to these problems . For example , it is often observed that when there are long-term dependencies in the data ( Bengio et al. , 1994 ; Hochreiter et al. , 2001 ) , RNNs may encounter problems in learning , but such statements have rarely been put on precise footing . In this paper , we make a step in this direction by studying the approximation and optimization properties of RNNs . Compared with the static feed-forward setting , the key distinguishing feature †Equal contribution ‡Corresponding author here is the presence of temporal dynamics in terms of both the recurrent architectures in the model and the dynamical structures in the data . Hence , to understand the influence of dynamics on learning is of fundamental importance . As is often the case , the key effects of dynamics can already be revealed in the simplest linear setting . For this reason , we will focus our analysis on linear RNNs , i.e . those with linear activations . Further , we will employ a continuous-time analysis initially studied in the context of feed-forward architectures ( E , 2017 ; Haber & Ruthotto , 2017 ; Li et al. , 2017 ) and recently in recurrent settings ( Ceni et al. , 2019 ; Chang et al. , 2019 ; Lim , 2020 ; Sherstinsky , 2018 ; Niu et al. , 2019 ; Herrera et al. , 2020 ; Rubanova et al. , 2019 ) and idealize the RNN as a continuous-time dynamical system . This allows us to phrase the problems under investigation in convenient analytical settings that accentuates the effect of dynamics . In this case , the RNNs serve to approximate relationships represented by sequences of linear functionals . On first look the setting appears to be simple , but we show that it yields representative results that underlie key differences in the dynamical setting as opposed to static supervised learning problems . In fact , we show that memory , which can be made precise by the decay rates of the target linear functionals , can affect both approximation rates and optimization dynamics in a non-trivial way . Our main results are : 1 . We give a systematic analysis of the approximation of linear functionals by continuoustime linear RNNs , including a precise characterization of the approximation rates in terms of regularity and memory of the target functional . 2 . We give a fine-grained analysis of the optimization dynamics when training linear RNNs , and show that the training efficiency is adversely affected by the presence of long-term memory . These results together paint a comprehensive picture of the interaction of learning and dynamics , and makes concrete the heuristic observations that the presence of long-term memory affects RNN learning in a negative manner ( Bengio et al. , 1994 ; Hochreiter et al. , 2001 ) . In particular , mirroring the classical curse of dimensionality ( Bellman , 1957 ) , we introduce the concept of the curse of memory that captures the new phenomena that arises from learning temporal relationships : when there is long-term memory in the data , one requires an exponentially large number of neurons for approximation , and the learning dynamics suffers from exponential slow downs . These results form a basic step towards a mathematical understanding of the recurrent structure and its effects on learning from temporal data . 2 RELATED WORK . We will discuss related work on RNNs on three fronts concerning the central results in this paper , namely approximation theory , optimization analysis and the role of memory in learning . A number of universal approximation results for RNNs have been obtained in discrete ( Matthews , 1993 ; Doya , 1993 ; Schäfer & Zimmermann , 2006 ; 2007 ) and continuous time ( Funahashi & Nakamura , 1993 ; Chow & Xiao-Dong Li , 2000 ; Li et al. , 2005 ; Maass et al. , 2007 ; Nakamura & Nakagawa , 2009 ) . Most of these focus on the case where the target relationship is generated from a hidden dynamical system in the form of difference or differential equations . The formulation of functional approximation here is more general , albeit our results are currently limited to the linear setting . Nevertheless , this is already sufficient to reveal new phenomena involving the interaction of learning and dynamics . This will be especially apparent when we discuss approximation rates and optimization dynamics . We also note that the functional/operator approximation using neural networks has been explored in Chen & Chen ( 1993 ) ; Tianping Chen & Hong Chen ( 1995 ) ; Lu et al . ( 2019 ) for nonrecurrent structures and reservoir systems for which approximation results similar to random feature models are derived ( Gonon et al. , 2020 ) . The main difference here is that we explicitly study the effect of memory in target functionals on learning using recurrent structures . On the optimization side , there are a number of recent results concerning the training of RNNs using gradient methods , and they are mostly positive in the sense that trainability is proved under specific settings . These include recovering linear dynamics ( Hardt et al. , 2018 ) or training in overparameterized settings ( Allen-Zhu et al. , 2019 ) . Here , our result concerns the general setting of learning linear functionals that need not come from some underlying differential/difference equations , and is also away from the over-parameterized regime . In our case , we discover on the contrary that training can become very difficult even in the linear case , and this can be understood in a quantitative way , in relation to long-term memory in the target functionals . This points to the practical literature in relation to memory and learning . The dynamical analysis here puts the ubiquitous but heuristic observations - that long-term memory negatively impacts training efficiency ( Bengio et al. , 1994 ; Hochreiter et al. , 2001 ) - on concrete theoretical footing , at least in idealized settings . This may serve to justify or improve current heuristic methods ( Tseng et al. , 2016 ; Dieng et al. , 2017 ; Trinh et al. , 2018 ) developed in applications to deal with the difficulty in training with long-term memory . At the same time , we also complement general results on “ vanishing and explosion of gradients ” ( Pascanu et al. , 2013 ; Hanin & Rolnick , 2018 ; Hanin , 2018 ) that are typically restricted to initialization settings with more precise characterizations in the dynamical regime during training . The long range dependency within temporal data has been studied for a long time in the time series literature , although its effect on learning input-output relationships is rarely covered . For example , the Hurst exponent ( Hurst , 1951 ) is often used as a measure of long-term memory in time series , e.g . fractional Brownian motion ( Mandelbrot & Ness , 1968 ) . In contrast with the setting in this paper where memory involves the dependence of the output time series on the input , the Hurst exponent measures temporal variations and dependence within the input time series itself . Much of the time series literature investigates statistical properties and estimation methods of data with long range dependence ( Samorodnitsky , 2006 ; Taqqu et al. , 1995 ; Beran , 1992 ; Doukhan et al. , 2003 ) . One can also combine these classic statistical methodologies with the RNN-like architectures to design hybrid models with various applications ( Loukas & Oke , 2007 ; Diaconescu , 2008 ; Mohan & Gaitonde , 2018 ; Bukhari et al. , 2020 ) . 3 PROBLEM FORMULATION . The basic problem of supervised learning on time series data is to learn a mapping from an input temporal sequence to an output sequence . Formally , one can think of the output at each time as being produced from the input via an unknown function that depends on the entire input sequence , or at least up to the time at which the prediction is made . In the discrete-time case , one can write the data generation process yk = Hk ( x0 , . . . , xk−1 ) , k = 0 , 1 , . . . ( 1 ) where xk , yk denote respectively the input data and output response , and { Hk : k = 0 , 1 , . . . } is a sequence of ground truth functions of increasing input dimension accounting for temporal evolution . The goal of supervised learning is to learn an approximation of the sequence of functions { Hk } given observation data . Recurrent neural networks ( RNN ) ( Rumelhart et al. , 1986 ) gives a natural way to parameterize such a sequence of functions . In the simplest case , the one-layer RNN is given by hk+1 = σ ( Whk + Uxk ) , ŷk = c > hk . ( 2 ) Here , { hk } are the hidden/latent states and its evolution is governed by a recursive application of a feed-forward layer with activation σ , and ŷk is called the observation or readout . We omit the bias term here and only consider a linear readout or output layer . For each time step k , the mapping { x0 , . . . , xk−1 } 7→ ŷk parameterizes a function Ĥk ( · ) through adjustable parameters ( c , W , U ) . Hence , for a particular choice of these parameters , a sequence of functions { Ĥk } is constructed at the same time . To understand the working principles of RNNs , we need to characterize how { Ĥk } approximates { Hk } . The model ( 2 ) is not easy to analyze due to its discrete iterative nature . Hence , here we employ a continuous-time idealization that replaces the time-step index k by a continuous time parameter t. This allows us to employ a large variety of continuum analysis tools to gain insights to the learning problem . Let us now introduce this framework . Continuous-time formulation . Consider a sequence of inputs indexed by a real-valued variable t ∈ R instead of a discrete variable k considered previously . Concretely , we consider the input space X = C0 ( R , Rd ) , ( 3 ) which is the linear space of continuous functions from R ( time ) to Rd that vanishes at infinity . Here d is the dimension of each point in the time series . We denote an element in X by x : = { xt ∈ Rd : t ∈ R } and equip X with the supremum norm ‖x‖X : = supt∈R ‖xt‖∞ . For the space of outputs we will take a scalar time series , i.e . the space of bounded continuous functions from R to R : Y = Cb ( R , R ) . ( 4 ) This is due to the fact that vector-valued outputs can be handled by considering each output separately . In continuous time , the target relationship ( ground truth ) to be learned is yt = Ht ( x ) , t ∈ R ( 5 ) where for each t ∈ R , Ht is a functional Ht : X → R. Correspondingly , we define a continuous version of ( 2 ) as a hypothesis space to model continuous-time functionals d dt ht = σ ( Wht + Uxt ) , ŷt = c > ht , ( 6 ) whose Euler discretization corresponds to a discrete-time residual RNN . The dynamics then naturally defines a sequences of functionals { Ĥt ( x ) = ŷt : t ∈ R } , which can be used to approximate the target functionals { Ht } via adjusting ( c , W , U ) . Linear RNNs in continuous time . In this paper we mainly investigate the approximation and optimization property of linear RNNs , which already reveals the essential effect of dynamics . The linear RNN obeys ( 6 ) with σ being the identity map . Notice that in the theoretical setup , the initial time of the system goes back to −∞ with limt→−∞ xt = 0 , ∀x ∈ X , thus by linearity ( Ht ( 0 ) = 0 ) we specify the initial condition of the hidden state h−∞ = 0 for consistency . In this case , ( 6 ) has the following solution ŷt = ∫ ∞ 0 c > eWsUxt−sds . ( 7 ) Since we will investigate uniform approximations over large time intervals , we will consider stable RNNs , where W ∈ Wm with Wm = { W ∈ Rm×m : eigenvalues of W have negative real parts } . ( 8 ) Owing to the representation of solutions in ( 7 ) , the linear RNN defines a family of functionals Ĥ : = ∪m≥1Ĥm , Ĥm : = { { Ĥt ( x ) , t ∈ R } : Ĥt ( x ) = ∫ ∞ 0 c > eWsUxt−sds , W ∈ Wm , U ∈ Rm×d , c ∈ Rm } . ( 9 ) Here , m is the width of the network and controls the complexity of the hypothesis space . Clearly , the family of functionals the RNN can represent is not arbitrary , and must possess some structure . Let us now introduce some definitions of functionals that makes these structures precise . Definition 3.1 . Let { Ht : t ∈ R } be a sequence of functionals . 1 . Ht is causal if it does not depend on future values of the input : for every pair of x , x′ ∈ X such that xs = x′s for all s ≤ t , we have Ht ( x ) = Ht ( x′ ) . 2 . Ht is linear and continuous if Ht ( λx + λ′x′ ) = λHt ( x ) + λ′Ht ( x′ ) for any x , x′ ∈ X and λ , λ′ ∈ R , and supx∈X , ‖x‖X≤1 |Ht ( x ) | < ∞ , in which case the induced norm can be defined as ‖Ht‖ : = supx∈X , ‖x‖X≤1 |Ht ( x ) | . 3 . Ht is regular if for any sequence { x ( n ) ∈ X : n ∈ N } such that x ( n ) t → 0 for Lebesgue almost every t ∈ R , limn→∞Ht ( x ( n ) ) = 0 . 4 . { Ht : t ∈ R } is time-homogeneous if Ht ( x ) = Ht+τ ( x ( τ ) ) for any t , τ ∈ R , where x ( τ ) s = xs−τ for all s ∈ R , i.e . x ( τ ) is x whose time index is shifted to the right by τ . Linear , continuous and causal functionals are common definitions . One can think of regular functionals as those that are not determined by values of the inputs on an arbitrarily small time interval , e.g . an infinitely thin spike input . Time-homogeneous functionals , on the other hand , are those where there is no special reference point in time : if the time index of both the input sequence and the functional are shifted in coordination , the output value remains the same . Given these definitions , the following observation can be verified directly and its proof is immediate and hence omitted . Proposition 3.1 . Let { Ĥt : t ∈ R } be a sequence of functionals in the RNN hypothesis space Ĥ ( see ( 9 ) ) . Then for each t ∈ R , Ĥt is a causal , continuous , linear and regular functional . Moreover , the sequence of functionals { Ĥt : t ∈ R } is time-homogeneous .

Summary: This paper reports a mathematical study of approximation properties of linear RNNs. The first part  reports a universal approximation theorem, and presents an analysis of how efficient the approximation is. In particular, it is shown that approximating a slowly decaying, power-law temporal filter requires a large number of neurons, a property the authors refer to as the "curse of memory".