Lipschitz Recurrent Neural Networks

1 INTRODUCTION . Many interesting problems exhibit temporal structures that can be modeled with recurrent neural networks ( RNNs ) , including problems in robotics , system identification , natural language processing , and machine learning control . In contrast to feed-forward neural networks , RNNs consist of one or more recurrent units that are designed to have dynamical ( recurrent ) properties , thereby enabling them to acquire some form of internal memory . This equips RNNs with the ability to discover and exploit spatiotemporal patterns , such as symmetries and periodic structures ( Hinton , 1986 ) . However , RNNs are known to have stability issues and are notoriously difficult to train , most notably due to the vanishing and exploding gradients problem ( Bengio et al. , 1994 ; Pascanu et al. , 2013 ) . Several recurrent models deal with the vanishing and exploding gradients issue by restricting the hidden-to-hidden weight matrix to be an element of the orthogonal group ( Arjovsky et al. , 2016 ; Wisdom et al. , 2016 ; Mhammedi et al. , 2017 ; Vorontsov et al. , 2017 ; Lezcano-Casado & MartinezRubio , 2019 ) . While such an approach is advantageous in maintaining long-range memory , it limits the expressivity of the model . To address this issue , recent work suggested to construct hidden-tohidden weights which have unit norm eigenvalues and can be nonnormal ( Kerg et al. , 2019 ) . Another approach for resolving the exploding/vanishing gradient problem has recently been proposed by Kag et al . ( 2020 ) , who formulate the recurrent units as a differential equation and update the hidden states based on the difference between predicted and previous states . In this work , we address these challenges by viewing RNNs as dynamical systems whose temporal evolution is governed by an abstract system of differential equations with an external input . The data are formulated in continuous-time where the external input is defined by the function x = x ( t ) ∈ Rp , and the target signal is defined as y = y ( t ) ∈ Rd . Based on insights from dynamical systems theory , we propose a continuous-time Lipschitz recurrent neural network with the functional form { ḣ = AβA , γAh+ tanh ( WβW , γW h+ Ux+ b ) , y = Dh , ( 1a ) ( 1b ) where the hidden-to-hidden matrices Aβ , γ ∈ RN×N and Wβ , γ ∈ RN×N are of the form { AβA , γA = ( 1− βA ) ( MA +MTA ) + βA ( MA −MTA ) − γAI WβW , γW = ( 1− βW ) ( MW +MTW ) + βW ( MW −MTW ) − γW I , ( 2a ) ( 2b ) where βA , βW ∈ [ 0 , 1 ] , γA , γW > 0 are tunable parameters and MA , MW ∈ RN×N are trainable matrices . Here , h = h ( t ) ∈ RN is a function of time t that represents an internal ( hidden ) state , and ḣ = ∂h ( t ) ∂t is its time derivative . The hidden state represents the memory that the system has of its past . The function in Eq . ( 1 ) is parameterized by the hidden-to-hidden weight matrices A ∈ RN×N and W ∈ RN×N , the input-to-hidden encoder matrix U ∈ RN×p , and an offset b . The function in Eq . ( 1b ) is parameterized by the hidden-to-output decoder matrix D ∈ Rd×N . Nonlinearity is introduced via the 1-Lipschitz tanh activation function . While RNNs that are governed by differential equations with an additive structure have been studied before ( Zhang et al. , 2014 ) , the specific formulation that we propose in ( 1 ) and our theoretical analysis are distinct . Treating RNNs as dynamical systems enables studying the long-term behavior of the hidden state with tools from stability analysis . From this point of view , an unstable unit presents an exploding gradient problem , while a stable unit has well-behaved gradients over time ( Miller & Hardt , 2019 ) . However , a stable recurrent unit can suffer from vanishing gradients , leading to catastrophic forgetting ( Hochreiter & Schmidhuber , 1997b ) . Thus , we opt for a stable model whose dynamics do not ( or only slowly do ) decay over time . Importantly , stability is also a statement about the robustness of neural units with respect to input perturbations , i.e. , stable models are less sensitive to small perturbations compared to unstable models . Recently , Chang et al . ( 2019 ) explored the stability of linearized RNNs and provided a local stability guarantee based on the Jacobian . In contrast , the particular structure of our unit ( 1 ) allows us to obtain guarantees of global exponential stability using control theoretical arguments . In turn , the sufficient conditions for global stability motivate a novel symmetric-skew decomposition based scheme for constructing hidden-to-hidden matrices . This scheme alleviates exploding and vanishing gradients , while remaining highly expressive . In summary , the main contributions of this work are as follows : • First , in Section 3 , using control theoretical arguments in a direct Lyapunov approach , we provide sufficient conditions for global exponential stability of the Lipschitz RNN unit ( Theorem 1 ) . Global stability is advantageous over local stability results since it guarantees non-exploding gradients regardless of the state . In the special case where A is symmetric , we find that these conditions agree with those in classical theoretical analyses ( Lemma 1 ) . • Next , in Section 4 , drawing from our stability analysis , we propose a novel scheme based on the symmetric-skew decomposition for constructing hidden-to-hidden matrices . This scheme mitigates the vanishing and exploding gradients problem , while obtaining highly expressive hidden-to-hidden matrices . • In Section 6 , we show that our Lipschitz RNN has the ability to outperform state-of-theart recurrent units on computer vision , language modeling and speech prediction tasks . Further , our results show that the higher-order explicit midpoint time integrator improves the predictive accuracy as compared to using the simpler one-step forward Euler scheme . • Finally , in Section 7 ) , we study our Lipschitz RNN via the lens of the Hessian and show that it is robust with respect to parameter perturbations ; we also show that our model is more robust with respect to input perturbations , compared to other continuous-time RNNs . 2 RELATED WORK . The problem of vanishing and exploding gradients ( and stability ) have a storied history in the study of RNNs . Below , we summarize two particular approaches to the problem ( constructing unitary/orthogonal RNNs and the dynamical systems viewpoint ) that have gained significant attention . Unitary and orthogonal RNNs . Unitary recurrent units have received attention recently , largely due to Arjovsky et al . ( 2016 ) showing that unitary hidden-to-hidden matrices alleviate the vanishing and exploding gradients problem . Several other unitary and orthogonal models have also been proposed ( Wisdom et al. , 2016 ; Mhammedi et al. , 2017 ; Jing et al. , 2017 ; Vorontsov et al. , 2017 ; Jose et al. , 2018 ) . While these approaches stabilize the training process of RNNs considerably , they also limit their expressivity and their prediction accuracy . Further , unitary RNNs are expensive to train , as they typically involve the computation of a matrix inverse at each step of training . Recent work by Lezcano-Casado & Martinez-Rubio ( 2019 ) overcame some of these limitations . By leveraging concepts from Riemannian geometry and Lie group theory , their recurrent unit exhibits improved expressivity and predictive accuracy on a range of benchmark tasks while also being efficient to train . Another competitive recurrent design was recently proposed by Kerg et al . ( 2019 ) . Their approach is based on the Schur decomposition , and it enables the construction of general nonnormal hidden-to-hidden matrices with unit-norm eigenvalues . Dynamical systems inspired RNNs . The continuous time view of RNNs has a long history in the neurodynamics community as it provides higher flexibility and increased interpretability ( Pineda , 1988 ; Pearlmutter , 1995 ; Zhang et al. , 2014 ) . In particular , RNNs that are governed by differential equations with an additive structure have been extensively studied from a theoretical point of view ( Funahashi & Nakamura , 1993 ; Kim et al. , 1996 ; Chow & Li , 2000 ; Hu & Wang , 2002 ; Li et al. , 2005 ; Trischler & D ’ Eleuterio , 2016 ) . See Zhang et al . ( 2014 ) for a comprehensive survey of continuous-time RNNs and their stability properties . Recently , several works have adopted the dynamical systems perspective to alleviate the challenges of training RNNs which are related to the vanishing and exploding gradients problem . For nonsequential data , Ciccone et al . ( 2018 ) proposed a negative-definite parameterization for enforcing stability in the RNN during training . Chang et al . ( 2019 ) introduced an antisymmetric hidden-tohidden weight matrix and provided guarantees for local stability . Kag et al . ( 2020 ) have proposed a differential equation based formulation for resolving the exploding/vanishing gradients problem by updating the hidden states based on the difference between predicted and previous states . Niu et al . ( 2019 ) employed numerical methods for differential equations to study the stability of RNNs . Another line of recent work has focused on continuous-time models that deal with irregular sampled time-series , missing values and multidimensional time series . Rubanova et al . ( 2019 ) and De Brouwer et al . ( 2019 ) formulated novel recurrent models based on the theory of differential equations and their discrete integration . Lechner & Hasani ( 2020 ) extended these ordinary differential equation ( ODE ) based models and addresses the issue of vanishing and exploding gradients by designing an ODE-model that is based on the idea of long short-term memory ( LSTM ) . This ODE-LSTM outperforms the continuous-time LSTM ( Mei & Eisner , 2017 ) as well as the GRU-D model ( Che et al. , 2018 ) that is based on a gated recurrent unit ( GRU ) . The link between dynamical systems and models for forecasting sequential data also provides the opportunity to incorporate physical knowledge into the learning process which improves the generalization performance , robustness , and ability to learn with limited data ( Chen et al. , 2019 ) . 3 STABILITY ANALYSIS OF LIPSCHITZ RECURRENT UNITS . One of the key contributions in this work is that we prove that model ( 1 ) is globally exponentially stable under some mild conditions on A and W . Namely , for any initial hidden state we can guarantee that our Lipschitz unit converges to an equilibrium if it exists , and therefore , gradients can never explode . We improve upon recent work on stability in recurrent models , which provide only a local analysis , see e.g. , ( Chang et al. , 2019 ) . In fact , global exponential stability is among the strongest notions of stability in nonlinear systems theory , implying all other forms of Lyapunov stability about the equilibrium h∗ ( Khalil , 2002 , Definitions 4.4 and 4.5 ) . Definition 1 . A point h∗ is an equilibrium point of ḣ = f ( h , t ) if f ( h∗ , t ) = 0 for all t. Such a point is globally exponentially stable if there exists some C > 0 and λ > 0 such that for any choice of initial values h ( 0 ) ∈ RN , ‖h ( t ) − h∗‖ ≤ Ce−λt‖h ( 0 ) − h∗‖ , for any t ≥ 0 . ( 3 ) The presence of a Lipschitz nonlinearity in ( 1 ) plays an important role in our analysis . While we focus on tanh in our experiments , our proof is more general and is applicable to models whose nonlinearity σ ( · ) is an M -Lipschitz function . Specifically , we consider the general model ḣ = Ah+ σ ( Wh+ Ux+ b ) , ( 4 ) for which we have the following stability result . In the following , we let σmin and σmax denote the smallest and largest singular values of the hidden-to-hidden matrices , respectively . Theorem 1 . Let h∗ be an equilibrium point of a differential equation of the form ( 4 ) for some x ∈ Rp . The point h∗ is globally exponentially stable if the eigenvalues of Asym : = 12 ( A + A T ) are strictly negative , W is non-singular , and either ( a ) σmin ( Asym ) > Mσmax ( W ) ; or ( b ) σ is monotone non-decreasing , W +WT is negative definite , and ATW +WTA is positive definite . The two cases show that global exponential stability is guaranteed if either ( a ) the matrix A has eigenvalues with real parts sufficiently negative to counteract expanding trajectories in the nonlinearity ; or ( b ) the nonlinearity is monotone , both A and W yield stable linear systems u̇ = Au , v̇ = Wv , and A , W have sufficiently similar eigenvectors . In practice , case ( b ) occasionally holds , but is challenging to ensure without assuming specific structure onA , W . Because such assumptions could limit the expressiveness of the model , the next section will develop a tunable formulation for A and W with the capacity to ensure that case ( a ) holds . In Appendix A.1 , we provide a proof of Theorem 1 using a direct Lyapunov approach . One advantage of this approach is that the driving input x is permitted to evolve in time arbitrarily in the analysis . The proof relies on the classical Kalman-Yakubovich-Popov lemma and circle criterion from control theory — to our knowledge , these tools have not been applied in the modern RNN literature , and we hope our proof can illustrate their value to the community . In the special case whereA is symmetric and x ( t ) constant , we show that we can also inherit criteria for both local and global stability from a class of well-studied Cohen–Grossberg–Hopfield models . Lemma 1 . Suppose that A is symmetric and W is nonsingular . There exists a diagonal matrix D ∈ RN×N , and nonsingular matrices L , V ∈ RN×N such that an equilibrium of ( 4 ) is ( globally exponentially ) stable if and only if there is a corresponding ( globally exponentially ) stable equilibrium for the system ż = Dz + Lσ ( V z + Ux+ b ) . ( 5 ) For a thorough review of analyses of ( 5 ) , see ( Zhang et al. , 2014 ) . In this special case , the criteria in Theorem 1 coincide with those obtained for the corresponding model ( 5 ) . However , in practice , we will not choose A to be symmetric .

Summary: Considering a continuous time RNN with Lipschitz-continuous nonlinearity, the authors formulate sufficient conditions on the parameter matrices for the network to be globally stable, in the sense of a globally attracting fixed point. They provide a specific parameterization for the hidden-to-hidden weight matrices to control global stability and error gradients, consisting of a weighted combination of a symmetric and a skew-symmetric matrix (and some diagonal offset). The authors discuss numerical integration by forward-Euler and RK2, and thoroughly benchmark their approach against a large set of other state-of-the-art RNNs on various tasks including versions of MNIST and TIMIT.  Finally, they highlight improved stability of their RNN against parameter and input perturbations.