RNNs Incrementally Evolving on an Equilibrium Manifold: A Panacea for Vanishing and Exploding Gradients?

Recurrent neural networks ( RNNs ) are particularly well-suited for modeling longterm dependencies in sequential data , but are notoriously hard to train because the error backpropagated in time either vanishes or explodes at an exponential rate . While a number of works attempt to mitigate this effect through gated recurrent units , skip-connections , parametric constraints and design choices , we propose a novel incremental RNN ( iRNN ) , where hidden state vectors keep track of incremental changes , and as such approximate state-vector increments of Rosenblatt ’ s ( 1962 ) continuous-time RNNs . iRNN exhibits identity gradients and is able to account for long-term dependencies ( LTD ) . We show that our method is computationally efficient overcoming overheads of many existing methods that attempt to improve RNN training , while suffering no performance degradation . We demonstrate the utility of our approach with extensive experiments and show competitive performance against standard LSTMs on LTD and other non-LTD tasks . 1 INTRODUCTION . Recurrent neural networks ( RNNs ) in each round store a hidden state vector , hm ∈ RD , and upon receiving the input vector , xm+1 ∈ Rd , linearly transform the tuple ( hm , xm+1 ) and pass it through a memoryless non-linearity to update the state over T rounds . Subsequently , RNNs output an affine function of the hidden states as its prediction . The model parameters ( state/input/prediction parameters ) are learnt by minimizing an empirical loss . This seemingly simple update rule has had significant success in learning complex patterns for sequential input data . Nevertheless , that training RNNs can be challenging , and that performance can be uneven on tasks that require long-term-dependency ( LTD ) , was first noted by Hochreiter ( 1991 ) , Bengio et al . ( 1994 ) and later by other researchers . Pascanu et al . ( 2013b ) attributed this to the fact that the error gradient back-propagated in time ( BPTT ) , for the time-step m , is dominated by product of partials of hiddenstate vectors , ∏T−1 j=m ∂hj+1 ∂hj , and these products typically exhibit exponentially vanishing decay or explosion , resulting in incorrect credit assignment during training and test-time . Rosenblatt ( 1962 ) , on whose work we draw inspiration from , introduced continuous-time RNN ( CTRNN ) to mimic activation propagation in neural circuitry . CTRNN dynamics evolves as follows : τ ġ ( t ) = −αg ( t ) + φ ( Ug ( t ) +Wx ( t ) + b ) , t ≥ t0 . ( 1 ) Here , x ( t ) ∈ Rd is the input signal , g ( t ) ∈ RD is the hidden state vector of D neurons , ġi ( t ) is the rate of change of the i-th state component ; τ , α ∈ R+ , referred to as the post-synaptic time-constant , impacts the rate of a neuron ’ s response to the instantaneous activation φ ( Ug ( t ) +Wx ( t ) + b ) ; and U ∈ RD×D , W ∈ RD×d , b ∈ RD are model parameters . In passing , note that recent RNN works that draw inspiration from ODE ’ s ( Chang et al. , 2019 ) are special cases of CTRNN ( τ = 1 , α = 0 ) . Vanishing Gradients . The qualitative aspects of the CTRNN dynamics is transparent in its integral form : g ( t ) = e−α t−t0 τ g ( t0 ) + 1 τ ∫ t t0 e−α t−s τ φ ( Ug ( s ) +Wx ( s ) + b ) ds ( 2 ) This integral form reveals that the partials of hidden-state vector with respect to the initial condition , ∂g ( t ) ∂g ( t0 ) , gets attenuated rapidly ( first term in RHS ) , and so we face a vanishing gradient problem . We will address this issue later but we note that this is not an artifact of CTRNN but is exhibited by ODEs that have motivated other RNNs ( see Sec . 2 ) . Shannon-Nyquist Sampling . A key property of CTRNN is that the time-constant τ together with the first term −g ( t ) , is in effect a low-pass filter with bandwidth ατ−1 suppressing high frequency components of the activation signal , φ ( ( Ug ( s ) ) + ( Wx ( s ) ) + b ) . This is good , because , by virtue of the Shannon-Nyquist sampling theorem , we can now maintain fidelity of discrete samples with respect to continuous time dynamics , in contrast to conventional ODEs ( α = 0 ) . Additionally , since high-frequencies are already suppressed , in effect we may assume that the input signal x ( t ) is slowly varying relative to the post-synaptic time constant τ . Equilibrium . The combination of low pass filtering and slowly time varying input has a significant bearing . The state vector as well as the discrete samples evolve close to the equilibrium state , i.e. , g ( t ) ≈ φ ( Ug ( t ) +Wx ( t ) + b ) under general conditions ( Sec . 3 ) . Incremental Updates . Whether or not system is in equilibrium , the integral form in Eq . 2 points to gradient attenuation as a fundamental issue . To overcome this situation , we store and process increments rather than the cumulative values g ( t ) and propose dynamic evolution in terms of increments . Let us denote hidden state sequence as hm ∈ RD and input sequence xm ∈ Rd . For m = 1 , 2 , . . . , T , and a suitable β > 0 τ ġ ( t ) = −α ( g ( t ) ± hm−1 ) + φ ( U ( g ( t ) ± hm−1 ) +Wxm + b ) , g ( 0 ) = 0 , t ≥ 0 ( 3 ) hm , h β·τ m , g ( β · τ ) Intuitively , say system is in equilibrium and−α ( µ ( xm , hm−1 ) ) +φ ( Uµ ( xm , hm−1 ) +Wxm+b ) = 0 . We note state transitions are marginal changes from previous states , namely , hm = µ ( xm , hm−1 ) − hm−1 . Now for a fixed input xm , as to which equilibrium is reached depends on hm−1 , but are nevertheless finitely many . So encoding marginal changes as states leads to “ identity ” gradient . Incremental RNN ( iRNN ) achieves Identity Gradient . We propose to discretize Eq . 3 to realize iRNN ( see Sec . 3 ) . At time m , it takes the previous state hm−1 ∈ RD and input xm ∈ Rd and outputs hm ∈ RD after simulating the CTRNN evolution in discrete-time , for a suitable number of discrete steps . We show that the proposed RNN approximates the continuous dynamics and solves the vanishing/exploding gradient issue by ensuring identity gradientIn general , we consider two options , SiRNN , whose state is updated with a single CTRNN sample , similar to vanilla RNNs , and , iRNN , with many intermediate samples . SiRNN is well-suited for slowly varying inputs . Contributions . To summarize , we list our main contributions : ( A ) iRNN converges to equilibrium for typical activation functions . The partial gradients of hiddenstate vectors for iRNNs converge to identity , thus solving vanishing/exploding gradient problem ! ( B ) iRNN converges rapidly , at an exponential rate in the number of discrete samplings of Eq . 1 . SiRNN , the single-step iRNN , is efficient and can be leveraged for slowly varying input sequences . It exhibits fast training time , has fewer parameters and better accuracy relative to standard LSTMs . ( C ) Extensive experiments on LTD datasets show that we improve upon standard LSTM accuracy as well as other recent proposals that are based on designing transition matrices and/or skip connections . iRNNs/SiRNNs are robust to time-series distortions such as noise paddings ( D ) While our method extends directly ( see Appendix A.1 ) to Deep RNNs , we deem these extensions complementary , and focus on single-layer to highlight our incremental perspective . 2 RELATED WORK . Gated Architectures . Long short-term memory ( LSTM ) ( Hochreiter & Schmidhuber , 1997 ) is widely used in RNNs to model long-term dependency in sequential data . Gated recurrent unit ( GRU ) ( Cho et al. , 2014 ) is another gating mechanism that has been demonstrated to achieve similar performance of LSTM with fewer parameters . Some recent gated RNNs include UGRNN ( Collins et al. , 2016 ) , and FastGRNN ( Kusupati et al. , 2018 ) . While mitigating vanishing/exploding gradients , they do not eliminate it . Often , these models incur increased inference , training costs , and model size . Unitary RNNs . Arjovsky et al . ( 2016 ) ; Jing et al . ( 2017 ) ; Zhang et al . ( 2018 ) ; Mhammedi et al . ( 2016 ) focus on designing well-conditioned state transition matrices , attempting to enforce unitary-property , during training . Unitary property does not generally circumvent vanishing gradient ( Pennington et al . ( 2017 ) ) . Also , it limits expressive power and prediction accuracy while also increasing training time . Deep RNNs . These are nonlinear transition functions incorporated into RNNs for performance improvement . For instance , Pascanu et al . ( 2013a ) empirically analyzed the problem of how to construct deep RNNs . Zilly et al . ( 2017 ) proposed extending the LSTM architecture to allow step-tostep transition depths larger than one . Mujika et al . ( 2017 ) proposed incorporating the strengths of both multiscale RNNs and deep transition RNNs to learn complex transition functions . While Deep RNNs offer richer representations relative to single-layers , it is complementary to iRNNs . Residual/Skip Connections . Jaeger et al . ( 2007 ) ; Bengio et al . ( 2013 ) ; Chang et al . ( 2017 ) ; Campos et al . ( 2017 ) ; Kusupati et al . ( 2018 ) feed-forward state vectors to induce skip or residual connections , to serve as a middle ground between feed-forward and recurrent models , and to mitigate gradient decay . Nevertheless , these connections , can not entirely eliminate gradient explosion/decay . For instance , Kusupati et al . ( 2018 ) suggest hm = αmhm−1 + βmφ ( Uhm−1 +Wxm + b ) , and learn parameters so that αm ≈ 1 and βm ≈ 0 . Evidently , this setting can lead to identity gradient , observe that setting βm ≈ 0 , implies little contribution from the inputs and can conflict with good accuracy , as also observed in our experiments . Linear RNNs . ( Bradbury et al. , 2016 ; Lei et al. , 2018 ; Balduzzi & Ghifary , 2016 ) focus on speeding up RNNs by replacing recurrent connections , such as hidden-to-hidden interactions , with light weight linear components . This reduces training time , but results in significantly increased model size . For example , Lei et al . ( 2018 ) requires twice the number of cells for LSTM level performance . ODE/Dynamical Perspective . Few ODE inspired architectures attempt to address stability , but do not end up eliminating vanishing/exploding gradients . Talathi & Vartak ( 2015 ) proposed a modified weight initialization strategy based on a dynamical system perspective on weight initialization process to successfully train RNNs composed of ReLUs . Niu et al . ( 2019 ) analyzed RNN architectures using numerical methods of ODE and propose a family of ODE-RNNs . Chang et al . ( 2019 ) , propose Antisymmetric-RNN . Their key idea is to express the transition matrix in Eq . 1 , for the special case α = 0 , τ = 1 , as a difference : U = V − V T and note that the eigenspectrum is imaginary . Nevertheless , Euler discretization , in this context leads to instability , necessitating damping of the system . As such vanishing gradient can not be completely eliminated . Its behavior is analogous to FastRNN Kusupati et al . ( 2018 ) , in that , identity gradient conflicts with high accuracy . In summary , we are the first to propose evolution over the equilibrium manifold , and demonstrating identity gradients . Neural ODEs ( Chen et al. , 2018 ; Rubanova et al. , 2019 ) have also been proposed for time-series prediction to deal with irregularly sampled inputs . They parameterize the derivative of the hidden-state in terms of an autonomous differential equation and let the ODE evolve in continuous time until the next input arrives . As such , this is not our goal , our ODE explicitly depends on the input , and evolves until equilibrium for that input is reached . We introduce incremental updates to bypass vanishing/exploding gradient issues , which is not of specific concern for these works .

Summary: In this paper, the authors propose the incremental RNN (iRNN), which is inspired by the continuous-time RNN (CTRNN). Theoretically, the equilibrium point of iRNN exists and is unique. Furthermore, the norm of the Jacobian between two hidden states is always one, provided that the Euler iterations converge. The authors proved this property as well as the exponential convergence rate of the Euler iteration. These properties avoid the vanishing/exploding gradient problem typical in RNN with long sequences in theory. Empirically, the proposed method is compared with multiple RNN architecture on various tasks.