AntisymmetricRNN: A Dynamical System View on Recurrent Neural Networks

1 INTRODUCTION . Recurrent neural networks ( RNNs ) ( Rumelhart et al. , 1986 ; Elman , 1990 ) have found widespread use across a variety of domains from language modeling ( Mikolov et al. , 2010 ; Kiros et al. , 2015 ; Jozefowicz et al. , 2016 ) and machine translation ( Bahdanau et al. , 2014 ) to speech recognition ( Graves et al. , 2013 ) and recommendation systems ( Hidasi et al. , 2015 ; Wu et al. , 2017 ) . Modeling complex temporal dependencies in sequential data using RNNs , especially the long-term dependencies , remains an open challenge . The main difficulty arises as the error signal back-propagated through time ( BPTT ) suffers from exponential growth or decay , a dilemma commonly referred to as exploding or vanishing gradient ( Pascanu et al. , 2012 ; Bengio et al. , 1994 ) . Vanilla RNNs as originally proposed are particularly prone to these issues and are rarely used in practice . Gated variants of RNNs , such as long short-term memory ( LSTM ) networks ( Hochreiter & Schmidhuber , 1997 ) and gated recurrent units ( GRU ) ( Cho et al. , 2014 ) that feature various forms of “ gating ” are proposed to alleviate these issues . The gates allow information to flow from inputs at any previous time steps to the end of the sequence more easily , partially addressing the vanishing gradient problem ( Collins et al. , 2016 ) . In practice , these models must be paired with techniques such as normalization layers ( Ioffe & Szegedy , 2015 ; Ba et al. , 2016 ) and gradient clipping ( Pascanu et al. , 2013 ) to achieve good performance . Identity and orthogonal initialization is another proposed solution to the exploding or vanishing gradient problem of deep neural networks ( Le et al. , 2015 ; Mishkin & Matas , 2015 ; Saxe et al. , 2013 ; Chen et al. , 2018a ) . Recently , Arjovsky et al . ( 2016 ) ; Wisdom et al . ( 2016 ) ; Hyland & Rätsch ∗Work performed while interning at Google Brain . ( 2017 ) ; Xie et al . ( 2017 ) ; Zhang et al . ( 2018a ) advocate going beyond initialization and forcing the weight matrices to be orthogonal throughout the entire learning process . However , some of these approaches come with significant computational overhead and reportedly hinder representation power of these models ( Vorontsov et al. , 2017 ) . Moreover , orthogonal weight matrices alone do not prevent exploding and vanishing gradients , due to the nonlinear nature of deep neural networks as shown in ( Pennington et al. , 2017 ) . Here we offer a new perspective on the trainability of RNNs from the dynamical system viewpoint . While exploding gradient is a manifestation of the instability of the underlying dynamical system , vanishing gradient results from a lossy system , properties that have been widely studied in the dynamical system literature ( Haber & Ruthotto , 2017 ; Laurent & von Brecht , 2017 ) . The main contributions of the work are : • We draw connections between RNNs and the ordinary differential equation theory and design new recurrent architectures by discretizing ODEs . • The stability of the ODE solutions and the numerical methods for solving ODEs lead us to design a special form of RNNs , which we name AntisymmetricRNN , that can capture longterm dependencies in the inputs . The construction of the model is much simpler compared to the existing methods for improving RNN trainability . • We conduct extensive simulations and experiments to demonstrate the benefits of this new RNN architecture . AntisymmetricRNN exhibits well-behaved dynamics and outperforms the regular LSTM model on tasks requiring long-term memory , and matches its performance on tasks where short-term dependencies dominate with much fewer parameters . 2 RELATED WORK . Trainability of RNNs . Capturing long-term dependencies using RNNs has been a long-standing research topic with various approaches proposed . The first group of approaches mitigates the exploding or vanishing gradient issues by introducing some forms of gating . Long short-term memory networks ( LSTM ) ( Hochreiter & Schmidhuber , 1997 ) and gated recurrent units ( GRU ) ( Cho et al. , 2014 ) are the most prominent models along this line of work . Parallel efforts include designing special neural architectures , such as hierarchical RNNs ( El Hihi & Bengio , 1996 ) , recursive neural networks ( Socher et al. , 2011 ) , attention networks ( Bahdanau et al. , 2014 ) , dilated convolutions ( Yu & Koltun , 2015 ) , recurrent batch normalization ( Cooijmans et al. , 2017 ) , residual RNNs ( Yue et al. , 2018 ) , and Fourier recurrent units ( Zhang et al. , 2018b ) . These , however , are fundamentally different networks with different tradeoffs . Another direction is to constrain the weight matrices of an RNN so that the back-propagated error signal is well conditioned , in particular , the input-output Jacobian having unitary singular values . Le et al . ( 2015 ) ; Mikolov et al . ( 2014 ) ; Mishkin & Matas ( 2015 ) propose the use of identity or orthogonal matrix to initialize the recurrent weight matrix . Followup works further constrain the weight matrix throughout the entire learning process either through re-parametrization ( Arjovsky et al. , 2016 ) , geodesic gradient descent on the Stiefel manifold ( Wisdom et al. , 2016 ; Vorontsov et al. , 2017 ) , or constraining the singular values ( Jose et al. , 2017 ; Kanai et al. , 2017 ; Zhang et al. , 2018a ) . It is worth noting that , orthogonal weights by themselves do not guarantee unitary Jacobians . Nonlinear activations still cause the gradients to explode or vanish . Contractive maps such as sigmoid and hyperbolic tangent lead to vanishing gradients . Chen et al . ( 2018a ) offer an initialization scheme taking into account the nonlinearity . However , the theory developed relies heavily on the random matrix assumptions , which only hold at the initialization point of training . Although it has been shown to predict trainability beyond initialization . Dynamical systems view of recurrent networks . Connections between dynamical systems and RNNs have not been well explored . Laurent & von Brecht ( 2017 ) study the behavior of dynamical systems induced by recurrent networks , and show that LSTMs and GRUs exhibit chaotic dynamics in the absence of input data . They propose a simplified gated RNN named the chaos free network ( CFN ) , that has non-chaotic dynamics and achieves comparable performance to LSTMs and GRUs on language modeling . Tallec & Ollivier ( 2018 ) formulate RNNs as a time-discretized version of ODE and show that time invariance leads to gate-like mechanisms in RNNs . With this formulation , the authors propose an initialization scheme by setting the initial gate bias according to the range of time dependencies to capture . Dynamical systems view of residual networks . Another line of work that is closely related to ours is the dynamical systems view on residual networks ( ResNets ) ( He et al. , 2016 ) . Haber & Ruthotto ( 2017 ) ; Chang et al . ( 2018a ; b ) ; Lu et al . ( 2018 ) propose to interpret ResNets as ordinary differential equations ( ODEs ) , under which learning the network parameters is equivalent to solve a parameter estimation problem involving the ODE . Chen et al . ( 2018b ) parameterize the continuous dynamics of hidden units using an ODE specified by a neural network . Stable and reversible architectures are developed ( Haber & Ruthotto , 2017 ; Chang et al. , 2018a ) from this viewpoint , which form the basis of our proposed recurrent networks . 3 ANTISYMMETRICRNNS . 3.1 ORDINARY DIFFERENTIAL EQUATIONS . We first give a brief overview of the ordinary differential equations ( ODEs ) , a special kind of dynamical systems that involves a single variable , time t in this case . Consider the first-order ODE h′ ( t ) = f ( h ( t ) ) , ( 1 ) for time t ≥ 0 , where h ( t ) ∈ Rn and f : Rn → Rn . Together with a given initial condition h ( 0 ) , the problem of solving for the function h ( t ) is called the initial value problem . For most ODEs , it is impossible to find an analytic solution . Instead , numerical methods relying on discretization are commonly used to approximate the solution . The forward Euler method is probably the best known and simplest numerical method for approximation . One way to derive the forward Euler method is to approximate the derivative on the left-hand side of Equation 1 by a finite difference , and evaluate the right-hand side at ht−1 : ht − ht−1 = f ( ht−1 ) . ( 2 ) Note that for the approximation to be valid , > 0 should be small by the definition of the derivative . One can easily prove that the forward Euler method converges linearly w.r.t . , assuming f ( h ) is Lipschitz continuous on h and that the eigenvalues of the Jacobian of f have negative real parts . Rearranging it , we have the forward Euler method for a given initial value ht = ht−1 + f ( ht−1 ) , h0 = h ( 0 ) . ( 3 ) Geometrically , each forward Euler step takes a small step along the tangential direction to the exact trajectory starting at ht−1 . As a result , is usually referred to as the step size . As an example , consider the ODE h′ ( t ) = tanh ( Wh ( t ) ) . The forward Euler method approximates the solution to the ODE iteratively as ht = ht−1 + tanh ( Wht−1 ) , which can be regarded as a recurrent network without input data . Here ht is the hidden state at the t-th step , W is a model parameter , and is a hyperparameter . This provides a general framework of designing recurrent network architectures by discretizing ODEs . As a result , we can design ODEs that possess desirable properties by exploiting the theoretical successes of dynamical systems , and the resulting recurrent networks will inherit these properties . Stability is one of the important properties to consider , which we will discuss in the next section . It is worth mentioning that the “ skip connection ” in this architecture resembles the residual RNN ( Yue et al. , 2018 ) and the Fourier RNN ( Zhang et al. , 2018b ) , which are proposed to mitigate the vanishing and exploding gradient issues . 3.2 STABILITY OF ORDINARY DIFFERENTIAL EQUATIONS : ANTISYMMETRICRNNS . In numerical analysis , stability theory addresses the stability of solutions of ODEs under small perturbations of initial conditions . In this section , we are going to establish the connections between the stability of an ODE and the trainability of the RNNs by discretizing the ODE , and design a new RNN architecture that is stable and capable of capturing long-term dependencies . An ODE solution is stable if the long-term behavior of the system does not depend significantly on the initial conditions . A formal definition is given as follows . Definition 1 . ( Stability ) A solution h ( t ) of the ODE in Equation 1 with initial condition h ( 0 ) is stable if for any > 0 , there exists a δ > 0 such that any other solution h̃ ( t ) of the ODE with initial condition h̃ ( 0 ) satisfying |h ( 0 ) − h̃ ( 0 ) | ≤ δ also satisfies |h ( t ) − h̃ ( t ) | ≤ , for all t ≥ 0 . In plain language , given a small perturbation of size δ of the initial state , the effect of the perturbation on the subsequent states is no bigger than . The eigenvalues of the Jacobian matrix play a central role in stability analysis . Let J ( t ) ∈ Rn×n be the Jacobian matrix of f , and λi ( · ) denotes the i-th eigenvalue . Proposition 1 . The solution of an ODE is stable if max i=1,2 , ... , n Re ( λi ( J ( t ) ) ) ≤ 0 , ∀t ≥ 0 , ( 4 ) where Re ( · ) denotes the real part of a complex number . A more precise proposition that involves the kinematic eigenvalues of J ( t ) is given in Ascher et al . ( 1994 ) . Stability alone , however , does not suffice to capture long-term dependencies . As argued in Haber & Ruthotto ( 2017 ) , Re ( λi ( J ( t ) ) ) 0 results in a lossy system ; the energy or signal in the initial state is dissipated over time . Using such an ODE as the underlying dynamical system of a recurrent network will lead to catastrophic forgetting of the past inputs during the forward propagation . Ideally , Re ( λi ( J ( t ) ) ) ≈ 0 , ∀i = 1 , 2 , . . . , n , ( 5 ) a condition we referred to as the critical criterion . Under this condition , the system preserves the long-term dependencies of the inputs while being stable . Stability and Trainability . Here we connect the stability of the ODE to the trainability of the RNN produced by discretization . Inherently , the stability analysis studies the sensitivity of a solution , i.e. , how much a solution of the ODE would change w.r.t . changes in the initial condition . Differentiating Equation 1 with respect to the initial state h ( 0 ) on both sides , we have the following sensitivity analysis ( with chain rules ) : d dt ( ∂h ( t ) ∂h ( 0 ) ) = J ( t ) ∂h ( t ) ∂h ( 0 ) . ( 6 ) For notational simplicity , let us define A ( t ) = ∂h ( t ) /∂h ( 0 ) , then we have dA ( t ) dt = J ( t ) A ( t ) , A ( 0 ) = I . ( 7 ) Note that this is a linear ODE with solution A ( t ) = eJ·t = P eΛ ( J ) tP−1 , assuming the Jacobian J does not vary or vary slowly over time ( We will later show this is a valid assumption ) . Here Λ ( J ) denotes the eigenvalues of J , and the columns of P are the corresponding eigenvectors . See Appendix A for a more detailed derivation . In the language of RNNs , A ( t ) is the Jacobian of a hidden state ht with respect to the initial hidden state h0 . When the critical criterion is met , i.e. , Re ( Λ ( J ) ) ≈ 0 , the magnitude of A ( t ) is approximately constant in time , thus no exploding or vanishing gradient problems . With the connection established , we next design ODEs that satisfy the critical criterion . An antisymmetric matrix is a square matrix whose transpose equals its negative ; i.e. , a matrix M ∈ Rn×n is antisymmetric if MT = −M . An interesting property of an antisymmetric matrix M is that , the eigenvalues of M are all imaginary : Re ( λi ( M ) ) = 0 , ∀i = 1 , 2 , . . . , n , making antisymmetric matrices a suitable building block of a stable recurrent architecture . Consider the following ODE h′ ( t ) = tanh ( ( Wh −W Th ) h ( t ) + Vhx ( t ) + bh ) , ( 8 ) where h ( t ) ∈ Rn , x ( t ) ∈ Rm , Wh ∈ Rn×n , Vh ∈ Rn×m and bh ∈ Rn . Note that Wh −W Th is an antisymmetric matrix . The Jacobian matrix of the right hand side is J ( t ) = diag [ tanh′ ( ( Wh −W Th ) h ( t ) + Vhx ( t ) + b ) ] ( Wh −W Th ) , ( 9 ) whose eigenvalues are all imaginary , i.e. , Re ( λi ( J ( t ) ) ) = 0 , ∀i = 1 , 2 , . . . , n. In other words , it satisfies the critical criterion in Equation 5 . See Appendix B for a proof . The entries of the diagonal matrix in Equation 9 are the derivatives of the activation function , which are bounded in [ 0 , 1 ] for sigmoid and hyperbolic tangent . In other words , the Jacobian matrix J ( t ) changes smoothly over time . Furthermore , since the input and bias term only affect the bounded diagonal matrix , their effect on the stability of the ODE is insignificant compared with the antisymmetric matrix . A naive forward Euler discretization of the ODE in Equation 8 leads to the following recurrent network we refer to as the AntisymmetricRNN . ht = ht−1 + tanh ( ( Wh −W Th ) ht−1 + Vhxt + bh ) , ( 10 ) where ht ∈ Rn is the hidden state at time t ; xt ∈ Rm is the input at time t ; Wh ∈ Rn×n , Vh ∈ Rn×m and bh ∈ Rn are the parameters of the network ; > 0 is a hyperparameter that represents the step size . Note that the antisymmetric matrix Wh −W Th only has n ( n − 1 ) /2 degrees of freedom . When implementing the model , Wh can be parameterized as a strictly upper triangular matrix , i.e. , an upper triangular matrix of which the diagonal entries are all zero . This makes the proposed model more parameter efficient than an unstructured RNN model of the same size of hidden states .

Summary: This paper introduces antisymmetric RNN, a novel RNNs architecture that is motivated through ordinary differential equation (ODE) framework. Authors consider a first order ODE and the RNN that results from the discretization of this ODE. They show how the stability criteria with respect to perturbation of  the initial state results in an ODE  in a trainability criteria for the corresponding  RNN. This criteria ensures that there are  no exploding/vanishing gradients. Authors then propose a specific parametrization, relying on antisymmetric matrix to ensure that the stability/trainability criteria is respected. They also propose a gated-variant of their architecture.  Authors evaluate their proposal on pixel-by-pixel MNIST and CIFAR10 where they show they can outperforms an LSTM.