SGORNN: Combining Scalar Gates and Orthogonal Constraints in Recurrent Networks

1 INTRODUCTION . Recurrent Neural Networks ( RNN ) are models for sequence processing that have found applications in many areas , such as statistical machine translation ( Cho et al. , 2014 ) , credit score calculation ( Babaev et al. , 2019 ) , vehicle mobility prediction ( Liu & Shoji , 2019 ) , and music generation ( Agarwala et al. , 2017 ) . Despite their wide applicability , RNNs have long suffered from exploding gradients , mainly due to the repeated application of the network ’ s hidden weights matrix during gradient computation ( Hochreiter & Schmidhuber , 1997 ) . Several strategies have been proposed to solve this problem , including models with orthogonal/unitary constraints on the hidden weights ( Taylor-Melanson et al. , 2021 ; Casado & Martı́nez-Rubio , 2019 ; Zhang et al. , 2018 ; Vorontsov et al. , 2017 ; Arjovsky et al. , 2016 ; Jing et al. , 2016 ) . Such orthogonal models usually have fewer parameters than standard RNN architectures , which reduces the representation capacity and leads to less accurate models . Specifically , Taylor-Melanson et al . ( 2021 ) noted that a stack of two RNN cells is required to accurately model the addition problem for RNNs using the orthogonal VPRNN architecture when sequences become very long , and Rusch & Mishra ( 2020 ) noted that the EXPRNN architecture ( Casado & Martı́nez-Rubio , 2019 ) fails to converge on the addition problem for very long sequences . To combat the problem of reduced representation capacity , Jing et al . ( 2019 ) proposed an LSTM-based architecture called the gated orthogonal recurrent unit ( GORU ) which was experimented with using the EURNN orthogonal parametrization ( Jing et al. , 2016 ) . While the GORU architecture is a very useful and novel orthogonal RNN construction , it adds many parameters when compared to typical orthogonal RNNs , and thus may not be suited for embedded or mobile applications . Recently , Kusupati et al . ( 2019 ) proposed the FastRNN architecture , which adds a residual connection to a standard recurrent network using a scalar gating mechanism . FastRNN outperforms several well-known unitary RNN models on many classification tasks . These scalar gated RNN models have a generalization error gap which scales with the product of the length of the sequences being processed and the non-residual scalar gate , which is established through constraints introduced on the scalar gates of the network that are generally not enforced while training . Because of this , the assumptions required to prove the generalization of FastRNN can not be guaranteed to be satisfied in practice under the original formulation of the architecture . Although the VPRNN architecture and many other RNNs alleviate exploding gradients by utilizing orthogonal constraints , the problem of increasing representation capacity for orthogonal RNNs without greatly increasing parameter counts still remains . Moreover , even with the benefit from the weighted residual connection , FastRNN can still suffer from exploding gradients when applied to long sequences as shown in Section 3.1 , and Rusch & Mishra ( 2020 ) noted that FastRNN fails to converge when training on long sequences for the RNN addition problem . In order to solve these problems , we propose the Scalar Gated Orthogonal Recurrent Neural Network ( SGORNN ) , which is a variant of FastRNN that uses orthogonal linear transformations at the recurrent step . The constraints on the scalar gates of FastRNN required for generalization are dependent on bounds of the Frobenius norm of the hidden weights matrix , and of the sequence length of the data being processed . Both of these quantities may not be well-defined in practice , making it difficult to ensure generalization . In contrast with FastRNN constraints , we present a constraint that relies only on the scalar gates of the SGORNN model , which is feasible to be employed in practice and results in a generalization gap with similar growth to that of FastRNN . Our theoretical results also confirm that SGORNN is immune to the ( exponentially ) exploding gradients so often seen when training RNN models . In summary , SGORNN can ensure its theoretical generalization by applying the constraints we provide , which also alleviates exploding gradients . Our addition problem experiment shows that SGORNN converges for long sequences with a single RNN cell , outperforming both predecessor architectures despite having far fewer parameters than FastRNN . Using real-world data , SGORNN shows superior generalization to its predecessor models and comparable performance to LSTM models with far more parameters . The remaining sections of this paper are organized as follows . In Section 2 , we describe the proposed SGORNN architecture , followed by the theoretical formulation on generalization and exploding gradients . Experiments are presented and discussed in Section 3 . In Section 4 , we present the conclusions obtained from the experiments and formulation , as well as proposed future work . Finally , Appendix A contains detailed information on the mathematical formulation used to provide the generalization bound for SGORNN , as well as the alleviation of exploding gradients . 2 SGORNN . The proposed Scalar Gated Orthogonal Recurrent Neural Network ( SGORNN ) uses orthogonal linear transformations at the recurrent step . Given an input sequence { xt } Tt=1 ⊂ Rn of length T > 0 , we define the hidden states ht ∈ Rh of a SGORNN at time t as : ht = ασ ( Wxt + φ ( Θ ) ht−1 + b ) + βht−1 . ( 1 ) where Θ holds the recurrent parameters of the architecture , φ is a mapping from recurrent parameters into the orthogonal matrices , W ∈ Rh×n and U = φ ( Θ ) ∈ Rh×h are matrices , b ∈ Rh is a bias vector , and σ is a nonlinear activation function , such as rectified linear unit ( ReLU ) . We use the notation U = φ ( Θ ) for consistency with Kusupati et al . ( 2019 ) during analysis . As with FastRNN , the parameters α , β ∈ ( 0 , 1 ) are set by the sigmoid function . For simplicity , we assume h0 is a vector of zeros , though , in theory , it can be made to be a trainable value as well . For the sake of experimentation , we choose φ ( Θ ) to be defined as the hidden weights matrix of the VPRNN architecture ( Taylor-Melanson et al. , 2021 ) , a recently developed orthogonal RNN which is known to outperform several other unitary/orthogonal RNNs on classification tasks . It is worth mentioning that , although we use the VPRNN mapping in our experiments with SGORNN , an alternative orthogonal constraint such as that of EXPRNN ( Casado & Martı́nez-Rubio , 2019 ) ( with φ = exp ) can also be applied to the proposed architecture . Following from Taylor-Melanson et al . ( 2021 ) , the recurrent matrix is defined as : φ ( Θ ) = R1Q1 . . .Rk−1Qk−1RkQk , ( 2 ) which is a case of SGORNN with dom ( φ ) = Rn/2×k , such that Rj performs pairwise 2D rotations of the input vector using parameters from the jth column of Θ , and Qj is a fixed permutation matrix . This formulation is inspired by a feed-forward predecessor ( MacDonald et al. , 2021 ) . It should be noted that , in the publicly available implementation of the VPNN layers from MacDonald et al . ( 2021 ) written in Keras ( Chollet et al. , 2015 ) that we require to implement SGORNN1 , a modification to the permutation structure of U is present in the code.2 This detail is ignored to simplify notation , but is justified in the supplementary material for the sake of proving Corollary 1 . The constraint we propose for the scalar gates of SGORNN to ensure generalization takes the following form : β ≤ 1− 2α . ( 3 ) At first glance , this constraint may seem to limit the representation capacity of the model as FastRNN takes β = 1 − α > 1 − 2α by assumption during theoretical analysis . However , typically α 1 , in which case β + 2α ≈ β + α , showing that an approximation to the standard formulation of the scalar gates remains . This constraint is empirically justified in later sections . Because our proposed constraints on α and β are independent of all other parameters ( or bounds ) , they are very simply enforced at training time by applying Equation 3 . In practice , the parameter β is clipped after applying the sigmoid function , so that it lies in the half-open interval ( 0 , 1− 2α ] . 2.1 GENERALIZATION BOUND . Initially , we state the constraint required for the generalization of FastRNN , to contrast with the constraints we require to guarantee the generalization of SGORNN and avoid exploding gradients . It is assumed for FastRNN that ‖U‖F ≤ RU for some RU > 0 , and the required constraint takes the form : α ≤ min ( 1 4T |D ‖U‖2 − 1| , 1 4TRU , 1 T | ‖U‖2 − 1| ) . ( 4 ) In the above equation , D is a real number derived from the parameters and activation function of the model . From this , Kusupati et al . ( 2019 ) established an O ( αT ) generalization bound for FastRNN which becomes independent of T since α = O ( 1/T ) . However , establishing this constraint in practice is difficult for several reasons . First , the constraint is dependent on the sequence length T of the model , which may be ill-defined in real-world scenarios , and can lead to vanishing α for very long sequences . Second , it relies on restricting the function space to those RNNs with ‖U‖F ≤ RU , so that RU is well-defined for use in Equation 4 . In addition , in many cases , choosing such an RU before starting the training may decrease representation capacity drastically or , again , lead to vanishing α . Finally , the constraint becomes undefined for orthogonal matrices ( ‖U‖2 = 1 ) . These issues motivate the use of our constraint from Equation 3 for SGORNN , which depends only on the use of the scalar gates α and β , and can be enforced at training time through a simple clipping operation without any prior assumptions about the sequence length or other parameters of the RNN . We define the class FT of SGORNN networks with ReLU activations as follows . Given bounds 1The implementation of SGORNN and experiment code is provided in a supplementary ZIP archive 2See the VPNN Python implementation at https : //github.com/wtaylor17/keras-vpnn RW , Rx > 0 , any fixed α , β ∈ ( 0 , 1 ) with β ≤ 1− 2α , a mapping φ from parameters to orthogonal matrices , and an input space X of sequences { xt } Tt=1 where ‖xt‖2 ≤ Rx , the SGORNN function class is defined as : FT = { ( xT , hT−1 ) 7→ ασ ( WxT + φ ( Θ ) hT−1 ) + βhT−1 : ‖W‖F ≤ RW , Θ ∈ dom ( φ ) } . ( 5 ) It is implicit in the definition of the SGORNN function class that bias vectors are omitted or absorbed into the weights matrix , σ is the ReLU activation , and that the dimensions of hT−1 , xT , and W are appropriate . The Rademacher complexity of this function class FT leads to the following theorem , which describes the generalization ability of SGORNN models with ReLU activation functions and with a single output neuron using any 1-Lipschitz activation function . This provides the equivalent of the FastRNN generalization gap result for our new proposed architecture : Theorem 1 Let hT denote the final hidden state of a SGORNN model from the function class FT for predetermined bounds RW , Rx . For any vector v ∈ Rh with ‖v‖2 ≤ Rv , and any 1-Lipschitz activation function σy , denote f ∈ σy ◦ v ◦ FT as a single-neuron output model such that f ( X ) = σy ( v > hT ) . Then , denoting L : R × R → [ 0 , BL ] as any bounded 1-Lipschitz loss function for some BL > 0 , we have with probability at least 1 − δ over the draw of an i.i.d . sample S = { ( X ( i ) , y ( i ) ) } mi=1 from any input-output distribution , the following generalization gap holds : E [ L ( y , f ( X ) ) ] ≤ 1 m m∑ i=1 L ( y ( i ) , f ( X ( i ) ) ) + 4C αT√ m + 3BL √ log 2/δ m ( 6 ) where C = RxRWRv represents the boundedness of the model parameters and input data , as in the analysis of FastRNN ( Kusupati et al. , 2019 ) , and the expectation is taken over the input-output distribution . The presented theoretical result for SGORNN shows that the generalization of the model is independent of U and Θ , which were not used to compute the bounds on α and β and do not appear in the definition of C. This is because the spectral norm for matrices is consistent with the Euclidean vector norm and ‖U‖2 = 1 for SGORNN . Further , the constraints on the scalar gates from Equation 3 do not depend on the sequence length T of the model . A proof of Theorem 1 is provided in Appendix A.1 .

Summary: This work proposes to approach and partially solve (under certain constraints) the exploding gradient problem of RNN. In practice, the authors combined two existing methods (residual connection + unitary RNNs) in a single novel model. Demonstrations are given for the exploding issue and for the motivation of parameter constraints. The new model, called SGORNN, is then tested on two toy tasks and on a largish one (language modelling). SGORNN outperform models that are based either on residual connection or unitary recurrent connections only.