RNNs Incrementally Evolving on an Equilibrium Manifold: A Panacea for Vanishing and Exploding Gradients?

Recurrent neural networks ( RNNs ) are particularly well-suited for modeling longterm dependencies in sequential data , but are notoriously hard to train because the error backpropagated in time either vanishes or explodes at an exponential rate . While a number of works attempt to mitigate this effect through gated recurrent units , skip-connections , parametric constraints and design choices , we propose a novel incremental RNN ( iRNN ) , where hidden state vectors keep track of incremental changes , and as such approximate state-vector increments of Rosenblatt ’ s ( 1962 ) continuous-time RNNs . iRNN exhibits identity gradients and is able to account for long-term dependencies ( LTD ) . We show that our method is computationally efficient overcoming overheads of many existing methods that attempt to improve RNN training , while suffering no performance degradation . We demonstrate the utility of our approach with extensive experiments and show competitive performance against standard LSTMs on LTD and other non-LTD tasks . 1 INTRODUCTION . Recurrent neural networks ( RNNs ) in each round store a hidden state vector , hm ∈ RD , and upon receiving the input vector , xm+1 ∈ Rd , linearly transform the tuple ( hm , xm+1 ) and pass it through a memoryless non-linearity to update the state over T rounds . Subsequently , RNNs output an affine function of the hidden states as its prediction . The model parameters ( state/input/prediction parameters ) are learnt by minimizing an empirical loss . This seemingly simple update rule has had significant success in learning complex patterns for sequential input data . Nevertheless , that training RNNs can be challenging , and that performance can be uneven on tasks that require long-term-dependency ( LTD ) , was first noted by Hochreiter ( 1991 ) , Bengio et al . ( 1994 ) and later by other researchers . Pascanu et al . ( 2013b ) attributed this to the fact that the error gradient back-propagated in time ( BPTT ) , for the time-step m , is dominated by product of partials of hiddenstate vectors , ∏T−1 j=m ∂hj+1 ∂hj , and these products typically exhibit exponentially vanishing decay or explosion , resulting in incorrect credit assignment during training and test-time . Rosenblatt ( 1962 ) , on whose work we draw inspiration from , introduced continuous-time RNN ( CTRNN ) to mimic activation propagation in neural circuitry . CTRNN dynamics evolves as follows : τ ġ ( t ) = −αg ( t ) + φ ( Ug ( t ) +Wx ( t ) + b ) , t ≥ t0 . ( 1 ) Here , x ( t ) ∈ Rd is the input signal , g ( t ) ∈ RD is the hidden state vector of D neurons , ġi ( t ) is the rate of change of the i-th state component ; τ , α ∈ R+ , referred to as the post-synaptic time-constant , impacts the rate of a neuron ’ s response to the instantaneous activation φ ( Ug ( t ) +Wx ( t ) + b ) ; and U ∈ RD×D , W ∈ RD×d , b ∈ RD are model parameters . In passing , note that recent RNN works that draw inspiration from ODE ’ s ( Chang et al. , 2019 ) are special cases of CTRNN ( τ = 1 , α = 0 ) . Vanishing Gradients . The qualitative aspects of the CTRNN dynamics is transparent in its integral form : g ( t ) = e−α t−t0 τ g ( t0 ) + 1 τ ∫ t t0 e−α t−s τ φ ( Ug ( s ) +Wx ( s ) + b ) ds ( 2 ) This integral form reveals that the partials of hidden-state vector with respect to the initial condition , ∂g ( t ) ∂g ( t0 ) , gets attenuated rapidly ( first term in RHS ) , and so we face a vanishing gradient problem . We will address this issue later but we note that this is not an artifact of CTRNN but is exhibited by ODEs that have motivated other RNNs ( see Sec . 2 ) . Shannon-Nyquist Sampling . A key property of CTRNN is that the time-constant τ together with the first term −g ( t ) , is in effect a low-pass filter with bandwidth ατ−1 suppressing high frequency components of the activation signal , φ ( ( Ug ( s ) ) + ( Wx ( s ) ) + b ) . This is good , because , by virtue of the Shannon-Nyquist sampling theorem , we can now maintain fidelity of discrete samples with respect to continuous time dynamics , in contrast to conventional ODEs ( α = 0 ) . Additionally , since high-frequencies are already suppressed , in effect we may assume that the input signal x ( t ) is slowly varying relative to the post-synaptic time constant τ . Equilibrium . The combination of low pass filtering and slowly time varying input has a significant bearing . The state vector as well as the discrete samples evolve close to the equilibrium state , i.e. , g ( t ) ≈ φ ( Ug ( t ) +Wx ( t ) + b ) under general conditions ( Sec . 3 ) . Incremental Updates . Whether or not system is in equilibrium , the integral form in Eq . 2 points to gradient attenuation as a fundamental issue . To overcome this situation , we store and process increments rather than the cumulative values g ( t ) and propose dynamic evolution in terms of increments . Let us denote hidden state sequence as hm ∈ RD and input sequence xm ∈ Rd . For m = 1 , 2 , . . . , T , and a suitable β > 0 τ ġ ( t ) = −α ( g ( t ) ± hm−1 ) + φ ( U ( g ( t ) ± hm−1 ) +Wxm + b ) , g ( 0 ) = 0 , t ≥ 0 ( 3 ) hm , h β·τ m , g ( β · τ ) Intuitively , say system is in equilibrium and−α ( µ ( xm , hm−1 ) ) +φ ( Uµ ( xm , hm−1 ) +Wxm+b ) = 0 . We note state transitions are marginal changes from previous states , namely , hm = µ ( xm , hm−1 ) − hm−1 . Now for a fixed input xm , as to which equilibrium is reached depends on hm−1 , but are nevertheless finitely many . So encoding marginal changes as states leads to “ identity ” gradient . Incremental RNN ( iRNN ) achieves Identity Gradient . We propose to discretize Eq . 3 to realize iRNN ( see Sec . 3 ) . At time m , it takes the previous state hm−1 ∈ RD and input xm ∈ Rd and outputs hm ∈ RD after simulating the CTRNN evolution in discrete-time , for a suitable number of discrete steps . We show that the proposed RNN approximates the continuous dynamics and solves the vanishing/exploding gradient issue by ensuring identity gradientIn general , we consider two options , SiRNN , whose state is updated with a single CTRNN sample , similar to vanilla RNNs , and , iRNN , with many intermediate samples . SiRNN is well-suited for slowly varying inputs . Contributions . To summarize , we list our main contributions : ( A ) iRNN converges to equilibrium for typical activation functions . The partial gradients of hiddenstate vectors for iRNNs converge to identity , thus solving vanishing/exploding gradient problem ! ( B ) iRNN converges rapidly , at an exponential rate in the number of discrete samplings of Eq . 1 . SiRNN , the single-step iRNN , is efficient and can be leveraged for slowly varying input sequences . It exhibits fast training time , has fewer parameters and better accuracy relative to standard LSTMs . ( C ) Extensive experiments on LTD datasets show that we improve upon standard LSTM accuracy as well as other recent proposals that are based on designing transition matrices and/or skip connections . iRNNs/SiRNNs are robust to time-series distortions such as noise paddings ( D ) While our method extends directly ( see Appendix A.1 ) to Deep RNNs , we deem these extensions complementary , and focus on single-layer to highlight our incremental perspective . 2 RELATED WORK . Gated Architectures . Long short-term memory ( LSTM ) ( Hochreiter & Schmidhuber , 1997 ) is widely used in RNNs to model long-term dependency in sequential data . Gated recurrent unit ( GRU ) ( Cho et al. , 2014 ) is another gating mechanism that has been demonstrated to achieve similar performance of LSTM with fewer parameters . Some recent gated RNNs include UGRNN ( Collins et al. , 2016 ) , and FastGRNN ( Kusupati et al. , 2018 ) . While mitigating vanishing/exploding gradients , they do not eliminate it . Often , these models incur increased inference , training costs , and model size . Unitary RNNs . Arjovsky et al . ( 2016 ) ; Jing et al . ( 2017 ) ; Zhang et al . ( 2018 ) ; Mhammedi et al . ( 2016 ) focus on designing well-conditioned state transition matrices , attempting to enforce unitary-property , during training . Unitary property does not generally circumvent vanishing gradient ( Pennington et al . ( 2017 ) ) . Also , it limits expressive power and prediction accuracy while also increasing training time . Deep RNNs . These are nonlinear transition functions incorporated into RNNs for performance improvement . For instance , Pascanu et al . ( 2013a ) empirically analyzed the problem of how to construct deep RNNs . Zilly et al . ( 2017 ) proposed extending the LSTM architecture to allow step-tostep transition depths larger than one . Mujika et al . ( 2017 ) proposed incorporating the strengths of both multiscale RNNs and deep transition RNNs to learn complex transition functions . While Deep RNNs offer richer representations relative to single-layers , it is complementary to iRNNs . Residual/Skip Connections . Jaeger et al . ( 2007 ) ; Bengio et al . ( 2013 ) ; Chang et al . ( 2017 ) ; Campos et al . ( 2017 ) ; Kusupati et al . ( 2018 ) feed-forward state vectors to induce skip or residual connections , to serve as a middle ground between feed-forward and recurrent models , and to mitigate gradient decay . Nevertheless , these connections , can not entirely eliminate gradient explosion/decay . For instance , Kusupati et al . ( 2018 ) suggest hm = αmhm−1 + βmφ ( Uhm−1 +Wxm + b ) , and learn parameters so that αm ≈ 1 and βm ≈ 0 . Evidently , this setting can lead to identity gradient , observe that setting βm ≈ 0 , implies little contribution from the inputs and can conflict with good accuracy , as also observed in our experiments . Linear RNNs . ( Bradbury et al. , 2016 ; Lei et al. , 2018 ; Balduzzi & Ghifary , 2016 ) focus on speeding up RNNs by replacing recurrent connections , such as hidden-to-hidden interactions , with light weight linear components . This reduces training time , but results in significantly increased model size . For example , Lei et al . ( 2018 ) requires twice the number of cells for LSTM level performance . ODE/Dynamical Perspective . Few ODE inspired architectures attempt to address stability , but do not end up eliminating vanishing/exploding gradients . Talathi & Vartak ( 2015 ) proposed a modified weight initialization strategy based on a dynamical system perspective on weight initialization process to successfully train RNNs composed of ReLUs . Niu et al . ( 2019 ) analyzed RNN architectures using numerical methods of ODE and propose a family of ODE-RNNs . Chang et al . ( 2019 ) , propose Antisymmetric-RNN . Their key idea is to express the transition matrix in Eq . 1 , for the special case α = 0 , τ = 1 , as a difference : U = V − V T and note that the eigenspectrum is imaginary . Nevertheless , Euler discretization , in this context leads to instability , necessitating damping of the system . As such vanishing gradient can not be completely eliminated . Its behavior is analogous to FastRNN Kusupati et al . ( 2018 ) , in that , identity gradient conflicts with high accuracy . In summary , we are the first to propose evolution over the equilibrium manifold , and demonstrating identity gradients . Neural ODEs ( Chen et al. , 2018 ; Rubanova et al. , 2019 ) have also been proposed for time-series prediction to deal with irregularly sampled inputs . They parameterize the derivative of the hidden-state in terms of an autonomous differential equation and let the ODE evolve in continuous time until the next input arrives . As such , this is not our goal , our ODE explicitly depends on the input , and evolves until equilibrium for that input is reached . We introduce incremental updates to bypass vanishing/exploding gradient issues , which is not of specific concern for these works .

Summary: The authors present a novel work to address the problem of signal propagation in the recurrent neural networks. The idea is to build a attractor system for the signal transition from state h_{k-1} to h_k. If the attractor system converges to a equilibrium, then the hidden to hidden gradient is an identity matrix. This idea is elegant. The authors verify the performance of Increment RNN on long-term-dependency tasks and non-long-term-dependency tasks.