On the Provable Generalization of Recurrent Neural Networks

Recurrent Neural Network ( RNN ) is a fundamental structure in deep learning . Recently , some works study the training process of over-parameterized neural networks , and show that over-parameterized networks can learn functions in some notable concept classes with a provable generalization error bound . In this paper , we analyze the training and generalization for RNNs with random initialization , and provide the following improvements over recent works : ( 1 ) For a RNN with input sequence x = ( X1 , X2 , ... , XL ) , previous works study to learn functions that are summation of f ( βTl Xl ) and require normalized conditions that ||Xl|| ≤ ϵwith some very small ϵ depending on the complexity of f . In this paper , using detailed analysis about the neural tangent kernel matrix , we prove a generalization error bound to learn such functions without normalized conditions and show that some notable concept classes are learnable with the numbers of iterations and samples scaling almost-polynomially in the input length L. ( 2 ) Moreover , we prove a novel result to learn N-variables functions of input sequence with the form f ( βT [ Xl1 , ... , XlN ] ) , which do not belong to the “ additive ” concept class , i , e. , the summation of function f ( Xl ) . And we show that when either N or l0 = max ( l1 , .. , lN ) − min ( l1 , .. , lN ) is small , f ( βT [ Xl1 , ... , XlN ] ) will be learnable with the number iterations and samples scaling almost-polynomially in the input length L . 1 Introduction . In Deep Learning , the recurrent neural network ( RNN ) is well-known as one of the most popular models to model sequential data and is widely used in practice for tasks in natural language processing ( NLP ) . One of the characters of RNN is that it performs the same operation for all the input of the sequence . Consider a input sequence x = ( X1 , X2 , ... , XL ) . A RNN with the form hl ( x ) = ϕ ( Whl−1 +AXl ) , ( 1 ) ∗Corresponding Author . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . is trying to learn functions fl ( X1 , X2 , ... Xl ) as h1 ( x ) = f1 ( X1 ) h2 ( x ) = f2 ( X1 , X2 ) ... hL ( x ) = fL ( X1 , X2 , ... XL ) ( 2 ) Due to the complex nonlinearity , the loss is generally non-convex , and it is very difficult to give a theoretical guarantee . Recently , there are some works Allen-Zhu et al . ( 2019b ) ; Cao and Gu ( 2019 ) ; Allen-Zhu et al . ( 2019a ) ; Du et al . ( 2019 ) ; Arora et al . ( 2019 ) ; Allen-Zhu et al . ( 2019c ) trying to give a theoretical explanation that why gradient descent can allow an overparametrized network to attain arbitrarily low training error and ample generalization ability . These papers show that , under some assumptions , we have : • Multi-layer feed-forward networks Allen-Zhu et al . ( 2019b ) ; Du et al . ( 2019 ) and recurrent neural networks Allen-Zhu et al . ( 2019c ) with large hidden size can attain zero training error , regardless of whether the data is properly labeled or randomly labeled . • For multi-layer feed-forward networks , functions with the form F ∗ ( x ) =∑C r=1 ϕr ( β T r X ) , X ∈ Rd , βr ∈ Rd , ||βr|| = 1 are learnable i.e . fitting the train- ing data with a provably small generalization error , if ϕ is analytic and the “ complexity ” is low enough Allen-Zhu et al . ( 2019a ) ; Arora et al . ( 2019 ) ; Cao and Gu ( 2019 ) . • The “ complexity ” of function ϕ can be measured by a matrix derived from the NTK ( Neural Tangent Kernel ) of the network Arora et al . ( 2019 ) ; Cao and Gu ( 2019 ) . • For recurrent neural networks Allen-Zhu and Li ( 2019a ) , if the input sequence is normalized , i.e. , x = ( X1 , X2 , ... , XL ) , ||X1|| = 1 , ||Xl|| = ϵ with ϵ very small , functions with the form F ∗ ( x ) = ∑L l=1 ∑Cl r=1 ϕl , r ( β T l , rXl ) are learnable , where m is the size of matrix W , and C = ∑∞ i=0 aiR i is a series representing the complexity of learnable functions . These works show the provable learning ability of deep learning . But there are still some important issues that were not addressed . • Firstly , for RNNs , the method in Allen-Zhu and Li ( 2019a ) requires a normalized condition for A andXl in ( 1 ) that ||AXl|| ≤ ϵx for all l ≤ L and shows that for a function F ∗ ( x ) with the complexity C , it is learnable with error O ( ϵ1/3x C ) . Thus ||Xl|| ( or equally , ||A|| ) should be very small and the scale is dependent on the complexity of functions . The dependence of ||AXl|| on C makes the results unrealistic in practice since generally the norm of input will not be so small . • Secondly , the result in Allen-Zhu and Li ( 2019a ) shows that RNNs can learn functions which are the summation of functions like ψ ( βTl Xl ) . But this is only a linear combination of the functions of the input at different positions and does not consider the nonlinear interaction of the inputs . One may ask , since hL ( x ) is a function of { X1 , X2 , ... XL } , is it possible to go beyond and learn more complex functions ? In order to study these problems , we consider the binary classification problem : for every input xi , the label ( +1 or −1 ) of xi can be expressed by the sign of a target function F ∗ ( xi ) . We consider Elman recurrent neural networks with ReLU activation hl ( x ) = ϕ ( Whl−1 +AXl ) f ( W , x ) = BThL ( x ) ∈ R. x = ( X1 , X2 , ... , XL ) , Xl ∈ Rd , W ∈ Rm×m , A ∈ Rm×d , B ∈ Rm , ϕ ( x ) = max ( x , 0 ) ( 3 ) to learn two types of target functions : • Additive Concept Class : F ∗ ( x ) = L∑ l=1 ∑ r=1 ψl , r ( β T l , rXl/||Xl|| ) , ψl , r ( x ) = ∞∑ i=0 cix i , ( 4 ) • N-variables Concept Class : F ∗ ( x ) = ∑ r ψr ( ⟨βr , [ Xl1 , ... , XlN ] ⟩ ) , ψr ( x ) = ∞∑ i=0 cix i . ( 5 ) For these two types of function , we study the following questions : • Can RNN learn additive concept class functions ( 4 ) without the normalized condition with reasonable complexity on the sequence size L ? • Can RNN learn functions in N-variables Concept Class ( 5 ) which can not be written as the summation of f ( Xl ) with reasonable complexity on N and L ? Our Result . We answer the two questions and give a provable generalization error bound . Our results are stated as follows : Theorem 1 ( Informal ) For a function F ∗ ( X1 , X2 , ... , XL ) with the form as in ( 4 ) or ( 5 ) , there is a power series named the complexity C ( F ∗ ) dependent on the Taylor expansion coefficient in ( 4 ) and ( 5 ) . For ( 4 ) , C ( F ∗ ) is almost-polynomial in L. For ( 5 ) , when N or l0 = max ( l1 , .. , lN ) − min ( l1 , .. , lN ) is small , C ( F ∗ ) is almost-polynomial in L. Under this definition of complexity C ( F ∗ ) , F ∗ is learnable using RNN with m hidden nodes and ReLU activation in ( 3 ) in O ( C ( F ∗ ) 2 ) steps with O ( C ( F ∗ ) 2 ) samples if m ≥ poly ( L , C ( F ∗ ) ) . Contribution . We summarize the contributions as follows : • In this paper , we prove that RNN without normalized condition can efficiently learn some notable concept classes with both time and sample complexity scaling almost polynomially in the input length L. • Our results go beyond the “ additive ” concept class . We prove a novel result that RNN can learn more complex function of the input such as N-variables concept class functions . And “ long range correlation functions ” with small N ( e.g . N = 2 , f ( βT [ Xl , Xl+l0 ] ) ) are learnable with complexity scaling almost polynomially in the input length L and correlation distance l0 . • Technically , we study the “ backward correlation ” of RNN network . In RNN case , using a crucial observation on the degeneracy of deep network , we show that the “ backward correlation ” 1m ⟨Backl ( xi ) , Backl ( xj ) ⟩ will decay polynomially rather than exponentially in input length L. This shows the complexity of learning RNN with ReLU activation function is polynomial in the size of input sequence L. Notions . For two matrices A , B ∈ Rm×n , we define ⟨A , B⟩ = Tr ( ATB ) . We define the asymptotic notations O ( · ) , Ω ( · ) , poly ( · ) as follows . an , bn are two sequences . an = O ( bn ) if lim supn→∞ |an/bn| < ∞ , an = Ω ( bn ) if lim infn→∞ |an/bn| > 0 , an = poly ( bn ) if there is k ∈ N that an = O ( ( bn ) k ) . Õ ( · ) , Ω̃ ( · ) , p̃oly ( · ) are notions which hide the logarithmic factors in O ( · ) , Ω ( · ) , poly ( · ) . || · || and || · ||2 denote the 2-norm of matrices . || · ||1 denote the 1-norm . || · ||F is the Frobenius-norm . || · ||0 is the number of non-zero entries . For elements Ai.j , Bi , j of symmetric matrix A , B . We abuse the notion Ai.j ⪰ Bi.j to denote A ⪰ B , i.e . A−B is a positive semidefinite matrix . 2 Preliminaries . 2.1 Function Complexity . For a analytic function ψ ( z ) , we can write it as ψ ( z ) = c0 + ∑∞ i=1 ciz i . We define the following notion to measure the complexity to learn such functions . C ( ψ , R ) = 1 + ∞∑ i=1 i · |ci|Ri . ( 6 ) CN ( ψ , R ) = 1 + ∞∑ i=1 L1.5NCN1 · √ CN , i · ( i/N ) N · |ci|Ri ( 7 ) where C1 > 100 is an large absolute constant and CN , i is the largest combination number i ! n1 ! n2 ! ... nN ! for n1 , n2 ... nN > 0 , n1 + n2 + ... nN = i , Example 2.1 Arora et al . ( 2019 ) Consider ψ ( z ) = arctan ( z/2 ) . Then ψ ( z ) = ∑ i=1 ( −1 ) i−121−2i 2i− 1 z2i−1 ( 8 ) In this case , C ( ψ , 1 ) = 1 + ∞∑ i=1 i · |ci| ≤ 1 + ∞∑ i=1 21−2i ≤ O ( 1 ) . Example 2.2 In the case N = 2 , C2 , i = i , ( i/2 ) 2 ≤ i2 . ψ ( z ) = exp ( z ) C2 ( ψ , 1 ) ≤ 1 + ∞∑ i=1 L3C21πi 2.5/i ! ≤ O ( 1 )

Summary: This paper considers when the training data are separable by certain concept class, the generalization of overparameterized RNN trained by SGD. The generalization error depends on a function complexity measure and converges at the rate $1/n$.  The treatment and results in the paper share the flavor of [Cao and Gu, 2019, Arora et al., 2019, Allen-Zhu et al., 2019]. Several comparisons are made in the paper to justify the contribution: 1) Compared to [Cao and Gu, 2019, Arora et al., 2019], no data distribution assumption is made in the current manuscript; 2) No normalized condition is needed compared to [Allen-Zhu and Li, 2019a].  Organization of the paper is clear and easy to follow, despite the proof sketch in Section 4.1 reads a bit dense. I appreciate the authors provided a proof road map in supplementary. No experiments are provided, yet the focus is purely on theory.