Coupled Oscillatory Recurrent Neural Network (coRNN): An accurate and (gradient) stable architecture for learning long time dependencies

1 INTRODUCTION . Recurrent neural networks ( RNNs ) have achieved tremendous success in a variety of tasks involving sequential ( time series ) inputs and outputs , ranging from speech recognition to computer vision and natural language processing , among others . However , it is well known that training RNNs to process inputs over long time scales ( input sequences ) is notoriously hard on account of the so-called exploding and vanishing gradient problem ( EVGP ) ( Pascanu et al. , 2013 ) , which stems from the fact that the well-established BPTT algorithm for training RNNs requires computing products of gradients ( Jacobians ) of the underlying hidden states over very long time scales . Consequently , the overall gradient can grow ( to infinity ) or decay ( to zero ) exponentially fast with respect to the number of recurrent interactions . A variety of approaches have been suggested to mitigate the exploding and vanishing gradient problem . These include adding gating mechanisms to the RNN in order to control the flow of information in the network , leading to architectures such as long short-term memory ( LSTM ) ( Hochreiter & Schmidhuber , 1997 ) and gated recurring units ( GRU ) ( Cho et al. , 2014 ) , that can overcome the vanishing gradient problem on account of the underlying additive structure . However , the gradients might still explode and learning very long term dependencies remains a challenge ( Li et al. , 2018 ) . Another popular approach for handling the EVGP is to constrain the structure of underlying recurrent weight matrices by requiring them to be orthogonal ( unitary ) , leading to the so-called orthogonal RNNs ( Henaff et al. , 2016 ; Arjovsky et al. , 2016 ; Wisdom et al. , 2016 ; Kerg et al. , 2019 ) and references therein . By construction , the resulting Jacobians have eigen- and singular-spectra with unit norm , alleviating the EVGP . However as pointed out by Kerg et al . ( 2019 ) , imposing such constraints on the recurrent matrices may lead to a significant loss of expressivity of the RNN resulting in inadequate performance on realistic tasks . In this article , we adopt a different approach , based on observation that coupled networks of controlled non-linear forced and damped oscillators , that arise in many physical , engineering and biological systems , such as networks of biological neurons , do seem to ensure expressive representations while constraining the dynamics of state variables and their gradients . This motivates us to propose a novel architecture for RNNs , based on time-discretizations of second-order systems of non-linear ordinary differential equations ( ODEs ) ( 1 ) that model coupled oscillators . Under verifiable hypotheses , we are able to rigorously prove precise bounds on the hidden states of these RNNs and their gradients , enabling a possible solution of the exploding and vanishing gradient problem , while demonstrating through benchmark numerical experiments , that the resulting system still retains sufficient expressivity , i.e . ability to process complex inputs , with a competitive performance , with respect to the state of the art , on a variety of sequential learning tasks . 2 THE PROPOSED RNN . Our proposed RNN is based on the following second-order system of ODEs , y′′ = σ ( Wy + Wy′ + Vu + b ) − γy − y′ . ( 1 ) Here , t ∈ [ 0 , 1 ] is the ( continuous ) time variable , u = u ( t ) ∈ Rd is the time-dependent input signal , y = y ( t ) ∈ Rm is the hidden state of the RNN with W , W ∈ Rm×m , V ∈ Rm×d are weight matrices , b ∈ Rm is the bias vector and 0 < γ , are parameters , representing oscillation frequency and the amount of damping ( friction ) in the system , respectively . σ : R 7→ R is the activation function , set to σ ( u ) = tanh ( u ) here . By introducing the so-called velocity variable z = y′ ( t ) ∈ Rm , we rewrite ( 1 ) as the first-order system : y′ = z , z′ = σ ( Wy + Wz + Vu + b ) − γy − z . ( 2 ) We fix a timestep 0 < ∆t < 1 and define our proposed RNN hidden states at time tn = n∆t ∈ [ 0 , 1 ] ( while omitting the affine output state ) as the following IMEX ( implicit-explicit ) discretization of the first order system ( 2 ) : yn = yn−1 + ∆tzn , zn = zn−1 + ∆tσ ( Wyn−1 + Wzn−1 + Vun + b ) −∆tγyn−1 −∆t zn̄ , ( 3 ) with either n̄ = n or n̄ = n− 1 . Note that the only difference in the two versions of the RNN ( 3 ) lies in the implicit ( n̄ = n ) or explicit ( n̄ = n− 1 ) treatment of the damping term − z in ( 2 ) , whereas both versions retain the implicit treatment of the first equation in ( 2 ) . Motivation and background . To see that the underlying ODE ( 2 ) models a coupled network of controlled forced and damped nonlinear oscillators , we start with the single neuron ( scalar ) case by setting d = m = 1 in ( 1 ) and assume an identity activation function σ ( x ) = x . Setting W = W = V = b = = 0 leads to the simple ODE , y′′ + γy = 0 , which exactly models simple harmonic motion with frequency γ , for instance that of a mass attached to a spring ( Guckenheimer & Holmes , 1990 ) . Letting > 0 in ( 1 ) adds damping or friction to the system ( Guckenheimer & Holmes , 1990 ) . Then , by introducing non-zero V in ( 1 ) , we drive the system with a driving force proportional to the input signal u ( t ) . The parameters V , b modulate the effect of the driving force , W controls the frequency of oscillations and W the amount of damping in the system . Finally , the tanh activation mediates a non-linear response in the oscillator . In the coupled network ( 2 ) with m > 1 , each neuron updates its hidden state based on the input signal as well as information from other neurons . The diagonal entries of W ( and the scalar hyperparameter γ ) control the frequency whereas the diagonal entries of W ( and the hyperparameter ) determine the amount of damping for each neuron , respectively , whereas the non-diagonal entries of these matrices modulate interactions between neurons . Hence , given this behavior of the underlying ODE ( 2 ) , we term the RNN ( 3 ) as a coupled oscillatory Recurrent Neural Network ( coRNN ) . The dynamics of the ODE ( 2 ) ( and the RNN ( 3 ) ) for a single neuron are relatively straightforward . As we illustrate in Fig . 6 of supplementary material SM§C , input signals drive the generation of ( superpositions of ) oscillatory wave-forms , whose amplitude and ( multiple ) frequencies are controlled by the tunable parameters W , W , V , b . Adding a tanh activation does not change these dynamics much . This is in contrast to truncating tanh to leading non-linear order by setting σ ( x ) = x− x3/3 , which yields a Duffing type oscillator that is characterized by chaotic behavior ( Guckenheimer & Holmes , 1990 ) . Adding interactions between neurons leads to further accentuation of this generation of superposed wave forms ( see Fig . 6 in SM§C ) and even with very simple network topologies , one sees the emergence of non-trivial non-oscillatory hidden states from oscillatory inputs . In practice , a network of a large number of neurons is used and can lead to extremely rich global dynamics . Hence , we argue that the ability of a network of ( forced , driven ) oscillators to access a very rich set of output states may lead to high expressivity of the system , allowing it to approximate outputs from complicated sequential inputs . Oscillator networks are ubiquitous in nature and in engineering systems ( Guckenheimer & Holmes , 1990 ; Strogatz , 2015 ) with canonical examples being pendulums ( classical mechanics ) , business cycles ( economics ) , heartbeat ( biology ) for single oscillators and electrical circuits for networks of oscillators . Our motivating examples arise in neurobiology , where individual biological neurons can be viewed as oscillators with periodic spiking and firing of the action potential . Moreover , functional circuits of the brain , such as cortical columns and prefrontal-striatal-hippocampal circuits , are being increasingly interpreted by networks of oscillatory neurons , see Stiefel & Ermentrout ( 2016 ) for an overview . Following well-established paths in machine learning , such as for convolutional neural networks ( LeCun et al. , 2015 ) , our focus here is to abstract the essence of functional brain circuits being networks of oscillators and design an RNN based on much simpler mechanistic systems , such as those modeled by ( 2 ) , while ignoring the complicated biological details of neural function . Related work . There is an increasing trend of basing RNN architectures on ODEs and dynamical systems . These approaches can roughly be classified into two branches , namely RNNs based on discretized ODEs and continuous-time RNNs . Examples of continuous-time approaches include neural ODEs ( Chen et al. , 2018 ) with ODE-RNNs ( Rubanova et al. , 2019 ) as its recurrent extension as well as E ( 2017 ) and references therein , to name just a few . We focus , however , in this article on an ODE-inspired discrete-time RNN , as the proposed coRNN is derived from a discretization of the ODE ( 1 ) . A good example for a discrete-time ODE-based RNNs is the so-called anti-symmetric RNN of Chang et al . ( 2019 ) , where the RNN architecture is based on a stable ODE resulting from a skew-symmetric hidden weight matrix , thus constraining the stable ( gradient ) dynamics of the network . This approach has much in common with previously mentioned unitary/orthogonal/nonnormal RNNs in constraining the structure of the hidden-to-hidden layer weight matrices . However , adding such strong constraints might reduce expressivity of the resulting RNN and might lead to inadequate performance on complex tasks . In contrast to these approaches , our proposed coRNN does not explicitly constrain the weight matrices but relies on the dynamics of the underlying ODE ( and the IMEX discretization ( 3 ) ) , to provide gradient stability . Moreover , no gating mechanisms as in LSTMs/GRUs are used in the current version of coRNN . There is also an increasing interest in designing hybrid methods , which use a discretization of an ODE ( in particular a Hamiltonian system ) in order to learn the continuous representation of the data , see for instance Greydanus et al . ( 2019 ) ; Chen et al . ( 2020 ) . Overall , our approach here differs from these papers in our use of networks of oscillators to build the RNN . 3 RIGOROUS ANALYSIS OF THE PROPOSED RNN . An attractive feature of the underlying ODE system ( 2 ) lies in the fact that the resulting hidden states ( and their gradients ) are bounded ( see SM§D for precise statements and proofs ) . Hence , one can expect that a suitable discretization of the ODE ( 2 ) that preserves these bounds will not have exploding gradients . We claim that one such structure preserving discretization is given by the IMEX discretization that results in the RNN ( 3 ) and proceed to derive bounds on this RNN below . Following standard practice we set y ( 0 ) = z ( 0 ) = 0 and purely for the simplicity of exposition , we set the control parameters , = γ = 1 and n̄ = n in ( 3 ) leading to , yn = yn−1 + ∆tzn , zn = zn−1 1+∆t + ∆t 1+∆tσ ( An−1 ) − ∆t1+∆tyn−1 , An−1 : = Wyn−1 + Wzn−1 + Vun + b . ( 4 ) Analogous results and proofs for the case where n̄ = n− 1 and for general values of , γ are provided in SM§F . Bounds on the hidden states . As with the underlying ODE ( 2 ) , the hidden states of the RNN ( 3 ) are bounded , i.e . Proposition 3.1 Let yn , zn be the hidden states of the RNN ( 4 ) for 1 ≤ n ≤ N , then the hidden states satisfy the following ( energy ) bounds : y > n yn + z > n zn ≤ nm∆t = mtn ≤ m. ( 5 ) The proof of the energy bound ( 5 ) is provided in SM§E.1 and a straightforward variant of the proof ( see SM§E.2 ) yields an estimate on the sensitivity of the hidden states to changing inputs . As with the underlying ODE ( see SM§D ) , this bound rules out chaotic behavior of hidden states . Bounds on hidden state gradients . We train the RNN ( 3 ) to minimize the loss function , E : = 1 N N∑ n=1 En , En = 1 2 ‖yn − ȳn‖22 , ( 6 ) with ȳ being the underlying ground truth ( training data ) . During training , we compute gradients of the loss function ( 6 ) with respect to the weights and biases Θ = [ W , W , V , b ] , i.e . ∂E ∂θ = 1 N N∑ n=1 ∂En ∂θ , ∀ θ ∈ Θ . ( 7 ) Proposition 3.2 Let yn , zn be the hidden states generated by the RNN ( 4 ) . We assume that the time step ∆t < < 1 can be chosen such that , max { ∆t ( 1 + ‖W‖∞ ) 1 + ∆t , ∆t‖W‖∞ 1 + ∆t } = η ≤ ∆tr , 1 2 ≤ r ≤ 1 . ( 8 ) Denoting δ = 11+∆t , the gradient of the loss function E ( 6 ) with respect to any parameter θ ∈ Θ is bounded as , ∣∣∣∣∂E∂θ ∣∣∣∣ ≤ 32 ( m+ Ȳ√m ) , ( 9 ) with Ȳ = max 1≤n≤N ‖ȳn‖∞ be a bound on the underlying training data . Sketch of the proof . Denoting Xn = [ yn , zn ] , we can apply the chain rule repeatedly ( for instance as in Pascanu et al . ( 2013 ) ) to obtain , ∂En ∂θ = ∑ 1≤k≤n ∂En ∂Xn ∂Xn ∂Xk ∂+Xk ∂θ︸ ︷︷ ︸ ∂E ( k ) n ∂θ . ( 10 ) Here , the notation ∂ +Xk ∂θ refers to taking the partial derivative of Xk with respect to the parameter θ , while keeping the other arguments constant . This quantity can be readily calculated from the structure of the RNN ( 4 ) and is presented in the detailed proof provided in SM§E.3 . From ( 6 ) , we can directly compute that ∂En∂Xn = [ yn − ȳn , 0 ] . Repeated application of the chain rule and a direct calculation with ( 4 ) yields , ∂Xn ∂Xk = ∏ k < i≤n ∂Xi ∂Xi−1 , ∂Xi ∂Xi−1 = [ I + ∆tBi−1 ∆tCi−1 Bi−1 Ci−1 ] , ( 11 ) where I is the identity matrix and Bi−1 = δ∆t ( diag ( σ ′ ( Ai−1 ) ) W − I ) , Ci−1 = δ ( I + ∆tdiag ( σ′ ( Ai−1 ) ) W ) . ( 12 ) It is straightforward to calculate using the assumption ( 8 ) that ‖Bi−1‖∞ < η and ‖Ci−1‖∞ ≤ η+ δ . Using the definitions of matrix norms and ( 8 ) , we obtain : ∥∥∥∥ ∂Xi∂Xi−1 ∥∥∥∥ ∞ ≤ max ( 1 + ∆t ( ‖Bi−1‖∞ + ‖Ci−1‖∞ ) , ‖Bi−1‖∞ + ‖Ci−1‖∞ ) ≤ max ( 1 + ∆t ( δ + 2η ) , δ + 2η ) ≤ 1 + 3∆tr . ( 13 ) Therefore , using ( 11 ) , we have∥∥∥∥∂Xn∂Xk ∥∥∥∥ ∞ ≤ ∏ k < i≤n ∥∥∥∥ ∂Xi∂Xi−1 ∥∥∥∥ ∞ ≤ ( 1 + 3∆tr ) n−k ≈ 1 + 3 ( n− k ) ∆tr . ( 14 ) Note that we have used an expansion around 1 and neglected terms of O ( ∆t2r ) as ∆t < < 1 . We remark that the bound ( 13 ) is the crux of our argument about gradient control as we see from the structure of the RNN that the recurrent matrices have close to unit norm . The detailed proof is presented in SM§E.3 . As the entire gradient of the loss function ( 6 ) , with respect to the weights and biases of the network , is bounded above in ( 9 ) , the exploding gradient problem is mitigated for this RNN . On the vanishing gradient problem . The vanishing gradient problem ( Pascanu et al. , 2013 ) arises if ∣∣∣∂E ( k ) n∂θ ∣∣∣ , defined in ( 10 ) , → 0 exponentially fast in k , for k < < n ( long-term dependencies ) . In that case , the RNN does not have long-term memory , as the contribution of the k-th hidden state to error at time step tn is infinitesimally small . We already see from ( 14 ) that ∥∥∥∂Xn∂Xk ∥∥∥∞ ≈ 1 ( independently of k ) . Thus , we should not expect the products in ( 10 ) to decay fast . In fact , we will provide a much more precise characterization of this gradient . To this end , we introduce the following order-notation , β = O ( α ) , for α , β ∈ R+ if there exists constants C , C such that Cα ≤ β ≤ Cα . M = O ( α ) , for M ∈ Rd1×d2 , α ∈ R+ if there exists constant C such that ‖M‖ ≤ Cα . ( 15 ) For simplicity of notation , we will also set ȳn = un ≡ 0 , for all n , b = 0 and r = 1 in ( 8 ) and we will only consider θ = Wi , j for some 1 ≤ i , j ≤ m in the following proposition . Proposition 3.3 Let yn be the hidden states generated by the RNN ( 4 ) . Under the assumption that yin = O ( √ tn ) , for all 1 ≤ i ≤ m and ( 8 ) , the gradient for long-term dependencies satisfies , ∂E ( k ) n ∂θ = O ( ĉδ∆t 3 2 ) +O ( ĉδ ( 1 + δ ) ∆t 5 2 ) +O ( ∆t3 ) , ĉ = sech2 ( √ k∆t ( 1 + ∆t ) ) , k < < n. ( 16 ) This precise bound ( 16 ) on the gradient shows that although the gradient can be small , i.e O ( ∆t 32 ) , it is in fact independent of k , ensuring that long-term dependencies contribute to gradients at much later steps and mitigating the vanishing gradient problem . The detailed proof is presented in SM§E.5 . Summarizing , we see that the RNN ( 3 ) indeed satisfied similar bounds to the underlying ODE ( 2 ) that resulted in upper bounds on the hidden states and its gradients . However , the lower bound on the gradient ( 16 ) is due to the specific choice of this discretization and does not appear to have a continuous analogue , making the specific choice of discretization of ( 2 ) crucial for mitigating the vanishing gradient problem .

Summary: Firstly, this paper conducts the rigorous analysis of the coRNN via the formula deduction to verify the bound. Then the coRNN is proved to mitigate the exploding and vanishing gradient problem and this is also validated in a series of experiments. Also, the performance of the coRNN is comparable or better compared to state-of-the-art models. This paper provides a new idea to address the exploding and vanishing gradient problem, which hinders the development of deeper neural networks tremendously. In my opinion, this coRNN model is meaningful for practical application, especially for the extension of more complicated neural networks.