Understanding Generalization in Recurrent Neural Networks

1 INTRODUCTION . The Recurrent Neural network ( RNN ) is a neural sequence model that has achieved the state-ofthe-art performance on numerous tasks , including natural language processing ( Yang et al. , 2018 ; Mikolov & Zweig , 2012 ) , speech recognition ( Chiu et al. , 2018 ; Graves , 2013 ) and machine translation ( Wu et al. , 2016 ; Kalchbrenner & Blunsom , 2013 ) . Unlike feedforward neural networks , RNNs allow connections among hidden units associated with a time delay . Through these connections , RNNs can maintain a ” memory ” that summarizes the past sequence of inputs , enabling it to capture correlations between temporally distant events in the data . RNNs are very powerful , and empirical studies have shown that they have a very good generalization property . For example , Graves ( 2013 ) showed that deep LSTM RNNs achieved a test error of 17.7 % on TIMT phoneme recognition benchmark after training with only 462 speech samples . Despite of the popularity of RNNs in practice , their theory is still not well understood . A number of recent works have sought to shed light on the effective representational properties of recurrent networks trained in practice . For example , Oymak ( 2018 ) studied the state equation of recurrent neural networks and showed that SGD can efficiently learn the unknown dynamics from few observations under proper assumptions . Miller & Hardt ( 2019 ) tried to explain why feed-forward neural networks are competitive with recurrent networks in practice . They identified stability as a necessary condition and proved that stable recurrent neural networks are well approximated by feed-forward networks for the purpose of both inference and training by gradient descent . Despite of the impressive progress in understanding the training behavior of RNNs , there is no generalization guarantee in these works . Understanding the generalization performance in machine learning has been a central problem for many years and revived in recent years with the advent of deep learning . One classical approach to proving generalization bound is via notions of complexity . For deep neural networks , numerous complexity measures have been proposed to capture the generalization behavior such as VC dimension ( Harvey et al. , 2017 ) and norm-based capacity including spectral norm ( Bartlett et al. , 2017 ; Neyshabur et al. , 2019 ) , Frobenius norm ( Neyshabur et al. , 2015b ; a ; 2018 ) and lp-path norm ( Neyshabur et al. , 2015b ; Bartlett & Mendelson , 2002 ; Golowich et al. , 2018 ) . These existing normbased complexity measures depend on the number of hidden units of the network explicitly and thus can not explain why neural networks generalize so well in practice , despite that they operate in an overparametrized setting ( Zhang et al. , 2017 ) . Neyshabur et al . ( 2019 ) proved generalization bounds for two layer ReLU feedforward networks , which decreased with the increasing number of hidden unit in the network . However their results only applied to two layer ReLU networks and some specific experiments . More recently , a new generalization bound based on Fisher-Rao norm was proposed ( Liang et al. , 2017 ) . This notion of Fisher-Rao norm is motivated by information geometry and has good invariance properties . But they proved the bound only for linear deep neural networks . There are also some works about the generalization of recurrent neural networks ( Zhang et al. , 2018 ; Chen et al. , 2019 ; Allen-Zhu & Li , 2019 ) . However these bounds also depend on the size of networks , which makes them vacuous for very large neural networks . Our main contributions are summarized as follows . • We define the Fisher-Rao norm for RNNs based on its gradient structure and derive new Rademacher complexity bound and generalization bound for recurrent neural networks based on Fisher-Rao norm and matrix 1-norm . In contrast to existing results such as spectral norm-based bounds , our bound has no explicit dependence on the size of networks . • We prove a generalization bound for RNNs when training with random noises . Our bound applies to general types of noises and can potentially explain the effect of noise training on generalization of recurrent neural networks as demonstrated by our empirical results . • We propose a new technique to decompose RNNs with ReLU activation into a sum of linear network and difference terms . As a result , each term in the decomposition can be treated independently and easily when estimating the Rademacher complexity . This decomposition technique can potentially be applied to other neural networks architectures such as convolutional neural networks , which might be of independent interest . The remainder of this paper is structured as follows . We define the problem and notations in Section 2 . The notion of Fisher-Rao norm for RNNs is introduced in Section 3.1 . We prove the generalization bound for RNNs in Section 3.2 , and the generalization bound for training with random noise is derived in Section 3.3 . Section 3.4 gives a detailed analysis of our generalization bound . Finally we conclude and discuss future directions . 2 PRELIMINARIES . We focus on the vanilla RNNs with ReLU activation . Let U ∈ Rm×d , V ∈ Rk×m and W ∈ Rm×m be the weight matrices . Given the input sequence x = ( x1 , x2 , · · · , xL ) ∈ RLd where each xi ∈ Rd and L is the input sequence length , the vanilla RNNs can be described as follows . gt = Uxt +Wht−1 , ht = ρ ( gt ) , yt = V ht , ( 1 ) where gt and ht ∈ Rm represents the input and output of hidden layer at step t , ρ ( · ) is the ReLU function and yt ∈ Rk denotes the output value at step t. For simplicity , in this paper , we only consider the final output yL . We assume that data ( x , y ) is drawn i.i.d . from some unknown distribution D over RLd × Y where Y represents the label space { 1 , 2 , · · · , k } . The RNNs above define a mapping yL ( x ) from RLd → Rk , where k is the number of classes . We convert yL ( x ) to a classifier by selecting the output coordinate with the largest magnitude , meaning x→ argmaxi [ yL ( x ) ] i , where [ · ] i represents the i-th element of a vector . This naturally leads to the definition of margin MyL ( x , y ) of the output yL at a labeled example ( x , y ) : MyL ( x , y ) = [ yL ( x ) ] y −max y′ 6=y [ yL ( x ) ] y′ . Thus , yL misclassifies ( x , y ) if and only ifMyL ( x , y ) ≤ 0 . The quality of the prediction made by yL is measured by the expected risk defined as E ( x , y ) ∼D [ 1MyL ( x , y ) ≤0 ] . Since the underlying distribution D is unknown to us , we instead consider the empirical error on sample data given by 1 n n∑ i=1 ( 1MyL ( xi , yi ) ≤α ) . The generalization error is then the difference between expected risk and empirical risk , defined as E ( x , y ) ∼D [ 1MyL ( x , y ) ≤0 ] − 1 n n∑ i=1 ( 1MyL ( xi , yi ) ≤α ) . Our goal in this paper is to study the generalization error for RNNs theoretically . To establish the generalization bound , a little bit of notations are necessary . For a vector , we denote the lp norm by ||v||p = ( ∑ |vi|p ) 1/p and the l∞ norm by ||v||∞ = max |vi| . For a matrix , we denote the matrix p-norm as ||A||p = max||x||p=1 ||Ax||p , the matrix 1-norm by ||A||1 = maxj { ∑ i |aij | } and the Frobenius norm by ||A||2F = trace ( AAT ) . The smallest eigenvalue of a matrix A is given by λmin ( A ) . The activation function ρ and its derivative ρ′ are entrywise , i.e. , ρ ( A ) = ( ρ ( aij ) ) ij and ρ′ ( v ) = ( ρ′ ( vi ) ) i . We denote c = ( L + 1 , L , · · · , 2 ) T , η ( θ ) = [ V diag ( ρ′ ( gL ) ) ... Wdiag ( ρ ′ ( g1 ) ) Ux1 , V diag ( ρ ′ ( gL ) ) ... Wdiag ( ρ ′ ( g2 ) ) Ux2 , · · · , V diag ( ρ′ ( gL ) ) UxL ] ∈ Rk×L and τ ( θ ) = ( VWL−1Ux1 , V WL−2Ux2 , · · · , V UxL ) where θ = ( U , W , V ) and diag converts a vector into a diagonal matrix . 3 MAIN RESULT . In this section , we prove a generalization bound for RNNs with ReLU activation . Our new bound is based on Fisher-Rao norm and matrix 1-norm . We first define the Fisher-Rao norm for RNNs . 3.1 FISHER-RAO NORM FOR RNNS . We adapt the notion of Fisher Rao norm to recurrent neural networks . To begin with , we establish the following structural result for RNNs . Lemma 1 . Given an input x = ( x1 , x2 , · · · , xL ) , consider the recurrent neural network in ( 1 ) , we have the identity ∑ a , b ∂yL ∂vab vab + ∑ i , j ∂yL ∂wij wij + ∑ p , q ∂yL ∂upq upq = η ( θ ) c. The notion of Fisher-Rao norm is motivated by Fisher-Rao metric of information geometry and is defined as follows . Definition 1 ( ( Liang et al. , 2017 ) , Definition 2 ) . The Fisher-Rao norm for a parameter θ is defined as ||θ||2fr : = < θ , I ( θ ) θ > , where I ( θ ) = E ( ∇l ( yLθ ( x ) , y ) ⊗∇l ( yLθ ( x ) , y ) ) and l ( . , . ) is the loss function . The following lemma gives the explicit formula of Fisher-Rao norm in RNNs . We can see that the notion of Fisher-Rao norm relies mainly on the gradient structure of RNNs . Lemma 2 . Assume that the loss function l ( . , . ) is smooth in the first argument . Then the following identity holds for the RNN in ( 1 ) , ||θ||2fr = E ( 〈 η ( θ ) c , ∂l ( yLθ ( x ) , y ) ∂yLθ 〉2 ) . Remark 1 . We observe that each term V diag ( ρ′ ( gL ) ) ... Wdiag ( ρ′ ( gi ) ) Uxi in η ( θ ) is actually the gradient component in Backpropagation through time ( BPTT ) . Therefore , the Fisher-Rao norm can be regarded as a measure of gradient . As will be shown later , we can build a relationship between generalization and trainability in RNNs via Fisher-Rao norm . For the linear activation function and margin loss l ( yLθ ( x ) , y ) = Φα ( MyL ( x , y ) ) where α > 0 is the margin parameter , one might upper bound the Fisher-Rao norm in Lemma 2 by ||θ||2fr ≤ 4 α2 E ( max i [ ( τ ( θ ) c ) i ] 2 ) , since 〈 τ ( θ ) c , ∂l ( yLθ ( x ) , y ) ∂yLθ 〉2 ≤ 4α2 maxi [ ( τ ( θ ) c ) i ] 2 by definition of MyL ( x , y ) and lipschitz property of Φα ( · ) . We define this upper bound as ||θ||2fs : = E ( max i [ ( τ ( θ ) c ) i ] 2 ) , ( 2 ) and still call it ” Fisher-Rao norm ” in the paper by slightly abusing the terminology as they are equivalent for k = 1 . In the rest of the paper , we will use this Fisher-Rao norm || · ||fs to derive generalization bound for RNNs .

Summary: In this paper, the authors investigate the topic of theoretical generalizability in recurrent networks. Specifically, they extend a generalization bound using matrix 1-norm and the Fisher-Rao norm, proving the effectiveness of adding noise to input data for generalizability. These bounds have no dependence on the size of networks being investigated. The authors also propose a technique for representing RNNs as a decomposition into a sum of linear networks with a difference term, which allows for easier estimation of Rademacher complexity. The authors claim this is a representation that can be extended to other neural network architectures such as convolutional networks.