Stable Recurrent Models

1 INTRODUCTION . Recurrent neural networks are a popular modeling choice for solving sequence learning problems arising in domains such as speech recognition and natural language processing . At the outset , recurrent neural networks are non-linear dynamical systems commonly trained to fit sequence data via some variant of gradient descent . Stability is of fundamental importance in the study of dynamical system . Surprisingly , however , stability has had little impact on the practice of recurrent neural networks . Recurrent models trained in practice do not satisfy stability in an obvious manner , suggesting that perhaps training happens in a chaotic regime . The difficulty of training recurrent models has compelled practitioners to successfully replace recurrent models with non-recurrent , feed-forward architectures . This state of affairs raises important unresolved questions . Is sequence modeling in practice inherently unstable ? When and why are recurrent models really needed ? In this work , we shed light on both of these questions through a theoretical and empirical investigation of stability in recurrent models . We first prove stable recurrent models can be approximated by feed-forward networks . In particular , not only are the models equivalent for inference , they are also equivalent for training via gradient descent . While it is easy to contrive non-linear recurrent models that on some input sequence can not be approximated by feed-forward models , our result implies such models are inevitably unstable . This means in particular they must have exploding gradients , which is in general an impediment to learnibility via gradient descent . Second , across a variety of different sequence tasks , we show how recurrent models can often be made stable without loss in performance . We also show models that are nominally unstable often operate in the stable regime on the data distribution . Combined with our first result , these observation helps to explain why an increasingly large body of empirical research succeeds in replacing recurrent models with feed-forward models in important applications , including translation ( Vaswani et al. , 2017 ; Gehring et al. , 2017 ) , speech synthesis ( Van Den Oord et al. , 2016 ) , and language modeling ( Dauphin et al. , 2017 ) . While stability does not always hold in practice to begin with , it is often possible to generate a high-performing stable model by imposing stability during training . Our results also shed light on the effective representational properties of recurrent networks trained in practice . In particular , stable models can not have long-term memory . Therefore , when stable and unstable models achieve similar results , either the task does not require long-term memory , or the unstable model does not have it . 1.1 CONTRIBUTIONS . In this work , we make the following contributions . 1 . We present a generic definition of stable recurrent models in terms of non-linear dynamical systems and show how to ensure stability of several commonly used models . Previous work establishes stability for vanilla recurrent neural networks . We give new sufficient conditions for stability of long short-term memory ( LSTM ) networks . These sufficient conditions come with an efficient projection operator that can be used at training time to enforce stability . 2 . We prove , under the stability assumption , feed-forward networks can approximate recurrent networks for purposes of both inference and training by gradient descent . While simple in the case of inference , the training result relies on non-trivial stability properties of gradient descent . 3 . We conduct extensive experimentation on a variety of sequence benchmarks , show stable models often have comparable performance with their unstable counterparts , and discuss when , if ever , there is an intrinsic performance price to using stable models . 2 STABLE RECURRENT MODELS . In this section , we define stable recurrent models and illustrate the concept for various popular model classes . From a pragmatic perspective , stability roughly corresponds to the criterion that the gradients of the training objective do not explode over time . Common recurrent models can operate in both the stable and unstable regimes , depending on their parameters . To study stable variants of common architectures , we give sufficient conditions to ensure stability and describe how to efficiently enforce these conditions during training . 2.1 DEFINING STABLE RECURRENT MODELS . A recurrent model is a non-linear dynamical system given by a differentiable state-transition map φw : Rn×Rd → Rn , parameterized by w ∈ Rm . The hidden state ht ∈ Rn evolves in discrete time steps according to the update rule ht = φw ( ht−1 , xt ) , ( 1 ) where the vector xt ∈ Rd is an arbitrary input provided to the system at time t. This general formulation allows us to unify many examples of interest . For instance , for a recurrent neural network , given weight matrices W and U , the state evolves according to ht = φW , U ( ht−1 , xt ) = tanh ( Wht−1 + Uxt ) . Recurrent models are typically trained using some variant of gradient descent . One natural—even if not strictly necessary—requirement for gradient descent to work is that the gradients of the training objective do not explode over time . Stable recurrent models are precisely the class of models where the gradients can not explode . They thus constitute a natural class of models where gradient descent can be expected to work . In general , we define a stable recurrent model as follows . Definition 1 . A recurrent model φw is stable if there exists some λ < 1 such that , for any weights w ∈ Rm , states h , h′ ∈ Rn , and input x ∈ Rd , ‖φw ( h , x ) − φw ( h′ , x ) ‖ ≤ λ ‖h− h′‖ . ( 2 ) Equivalently , a recurrent model is stable if the map φw is λ-contractive in h. If φw is λ-stable , then ‖∇hφw ( h , x ) ‖ < λ , and for Lipschitz loss p , ‖∇wp‖ is always bounded ( Pascanu et al. , 2013 ) . Stable models are particularly well-behaved and well-justified from a theoretical perspective . For instance , at present , only stable linear dynamical systems are known to be learnable via gradient descent ( Hardt et al. , 2018 ) . In unstable models , the gradients of the objective can explode , and it is a delicate matter to even show that gradient descent converges to a stationary point . The following proposition offers one such example . The proof is provided in the appendix . Proposition 1 . There exists an unstable system φw where gradient descent does not converge to a stationary point , and ‖∇wp‖ → ∞ as the number of iterations N →∞ . 2.2 EXAMPLES OF STABLE RECURRENT MODELS . In this section , we provide sufficient conditions to ensure stability for several common recurrent models . These conditions offer a way to require learning happens in the stable regime– after each iteration of gradient descent , one imposes the corresponding stability condition via projection . Linear dynamical systems and recurrent neural networks . Given a Lipschitz , point-wise nonlinearity ρ and matrices W ∈ Rn×n and U ∈ Rn×d , the state-transition map for a recurrent neural network ( RNN ) is ht = ρ ( Wht−1 + Uxt ) . If ρ is the identity , then the system is a linear dynamical system . Jin et al . ( 1994 ) show if ρ is Lρ-Lipschitz , then the model is stable provided ‖W‖ < 1Lρ . Indeed , for any states h , h ′ , and any x , ‖ρ ( Wh+ Ux ) − ρ ( Wh′ + Ux ) ‖ ≤ Lρ ‖Wh+ Ux−Wh′ − Ux‖ ≤ Lρ ‖W‖ ‖h− h′‖ . In the case of a linear dynamical system , the model is stable provided ‖W‖ < 1 . Similarly , for the 1-Lipschitz tanh-nonlinearity , stability obtains provided ‖W‖ < 1 . In the appendix , we verify the assumptions required by the theorems given in the next section for this example . Imposing this condition during training corresponds to projecting onto the spectral norm ball . Long short-term memory networks . Long Short-Term Memory ( LSTM ) networks are another commonly used class of sequence models ( Hochreiter & Schmidhuber , 1997 ) . The state is a pair of vectors s = ( c , h ) ∈ R2d , and the model is parameterized by eight matrices , W ∈ Rd×d and U ∈ Rd×n , for ∈ { i , f , o , z } . The state-transition map φLSTM is given by ft = σ ( Wfht−1 + Ufxt ) it = σ ( Wiht−1 + Uixt ) ot = σ ( Woht−1 + Uoxt ) zt = tanh ( Wzht−1 + Uzxt ) ct = it ◦ zt + ft ◦ ct−1 ht = ot · tanh ( ct ) , where ◦ denotes elementwise multiplication , and σ is the logistic function . We provide conditions under which the iterated system φrLSTM = φLSTM ◦ · · · ◦ φLSTM is stable . Let ‖f‖∞ = supt ‖ft‖∞ . If the weights Wf , Uf and inputs xt are bounded , then ‖f‖∞ < 1 since |σ| < 1 for any finite input . This means the next state ct must “ forget ” a non-trivial portion of ct−1 . We leverage this phenomenon to give sufficient conditions for φLSTM to be contractive in the ` ∞ norm , which in turn implies the iterated system φrLSTM is contractive in the ` 2 norm for r = O ( log ( d ) ) . Let ‖W‖∞ denote the induced ` ∞ matrix norm , which corresponds to the maximum absolute row sum maxi ∑ j |Wij | . Proposition 2 . If ‖Wi‖∞ , ‖Wo‖∞ < ( 1− ‖f‖∞ ) , ‖Wz‖∞ ≤ ( 1/4 ) ( 1 − ‖f‖∞ ) , ‖Wf‖∞ < ( 1− ‖f‖∞ ) 2 , and r = O ( log ( d ) ) , then the iterated system φrLSTM is stable . The proof is given in the appendix . The conditions given in Proposition 2 are fairly restrictive . Somewhat surprisingly we show in the experiments models satisfying these stability conditions still achieve good performance on a number of tasks . We leave it as an open problem to find different parameter regimes where the system is stable , as well as resolve whether the original system φLSTM is stable . Imposing these conditions during training and corresponds to simple row-wise normalization of the weight matrices and inputs . More details are provided in Section 4 and the appendix . 3 STABLE RECURRENT MODELS HAVE FEED-FORWARD APPROXIMATIONS . In this section , we prove stable recurrent models can be well-approximated by feed-forward networks for the purposes of both inference and training by gradient descent . From a memory perspective , stable recurrent models are equivalent to feed-forward networks—both models use the same amount of context to make predictions . This equivalence has important consequences for sequence modeling in practice . When a stable recurrent model achieves satisfactory performance on some task , a feed-forward network can achieve similar performance . Consequently , if sequence learning in practice is inherently stable , then recurrent models may not be necessary . Conversely , if feed-forward models can not match the performance of recurrent models , then sequence learning in practice is in the unstable regime . 3.1 TRUNCATED RECURRENT MODELS . For our purposes , the salient distinction between a recurrent and feed-forward model is the latter has finite-context . Therefore , we say a model is feed-forward if the prediction made by the model at step t is a function only of the inputs xt−k , . . . , xt for some finite k. While there are many choices for a feed-forward approximation , we consider the simplest one— truncation of the system to some finite context k. In other words , the feed-forward approximation moves over the input sequence with a sliding window of length k producing an output every time the sliding window advances by one step . Formally , for context length k chosen in advance , we define the truncated model via the update rule hkt = φw ( h k t−1 , xt ) , h k t−k = 0 . ( 3 ) Note that hkt is a function only of the previous k inputs xt−k , . . . , xt . While this definition is perhaps an abuse of the term “ feed-forward ” , the truncated model can be implemented as a standard autoregressive , depth-k feed-forward network , albeit with significant weight sharing . Let f denote a prediction function that maps a state ht to outputs f ( ht ) = yt . Let ykt denote the predictions from the truncated model . To simplify the presentation , the prediction function f is not parameterized . This is without loss of generality because it is always possible to fold the parameters into the system φw itself . In the sequel , we study ∥∥yt − ykt ∥∥ both during and after training .

Summary: In this paper, the authors study the stability property of recurrent neural networks. Adopting the definition of stability from the dynamical system literature, the authors present a generic definition of stable recurrent models and provide sufficient conditions of stable linear RNNs and LSTMs. The authors also study the "feed-forward" approximation of recurrent networks and theoretically show that the approximation works for both inference and training. Experimental studies compare the performance of stable and unstable models on various tasks.