Learning and Generalization in RNNs

1 Introduction . Simple Recurrent Neural Networks [ 1 ] also known as Elman RNNs or vanilla RNNs ( just RNNs henceforth ) along with their more advanced versions such as LSTMs [ 2 ] and GRU [ 3 ] are among the most successful models for processing sequential data , finding wide-ranging applications including natural language processing , audio processing [ 4 ] and time series classification [ 5 ] . Feedforward networks ( FFNs ) model functions on inputs of fixed length , such as vectors in Rd . In contrast , RNNs model functions whose input consists of sequences of tokens x ( 1 ) , x ( 2 ) , . . . , where x ( i ) ∈ Rd for each i. RNNs have a notion of memory ; formally it is given by the hidden state vector which is denoted by h ( t ) after processing the t-th token . RNNs apply a fixed function to h ( t ) and x ( t+1 ) to compute h ( t+1 ) and the output . This fixed function is modeled by a neural networks with one hidden-layer . Compared to FFNs , new challenges arise in the analysis of RNNs : for example , the use of memory and the same function at each step introduces dependencies across time and RNN training suffers from vanishing and exploding gradients [ 6 ] . Studies aimed at understanding the effectiveness of RNNs have been conducted since their introduction ; for some of the early work , see , e.g. , [ 7 , 8 ] . These works take the form of experimental probing of the inner workings of these models as well as theoretical studies . The theoretical studies are often focused on expressibility , training and generalization questions in isolation rather than all together—the latter needs to be addressed to approach full understanding of RNNs and appears to be far more challenging . While experimental probing has continued apace , e.g. , [ 9 , 10 ] , progress on theoretical front has been slow . It is only recently that training and generalization are starting to be addressed in the wake of progress on the relatively easier case of FFNs as discussed next . RNNs are closely related to deterministic finite automata [ 11 , 9 ] as well as to dynamical systems . With finite precision and ReLU activation , they are equivalent to finite automata [ 11 ] in computational power . In the last few years progress was made on theoretical analysis of overparameterized FFNs ∗Work done as a research fellow in Microsoft Research India 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . with one-hidden-layer , e.g. , [ 12 , 13 , 14 , 15 , 16 , 17 , 18 ] . Building upon these techniques , [ 19 ] proved that RNNs trained with SGD ( stochastic gradient descent ) achieve small training loss if the number of neurons is sufficiently large polynomial in the number of training datapoints and the maximum sequence length . But the gap between our understanding of RNNs and FFNs remains large . [ 20 , 21 ] provide generalization bounds on RNNs in terms of certain norms of the parameters . While interesting , these bounds shed light on only a part of the picture as they do not consider the training of the networks nor do not preclude the possibility that the norms of the parameters for the trained networks are large leading to poor generalization guarantees . RNNs can be viewed as dynamical systems and many works have used this viewpoint to study RNNs , e.g. , [ 22 , 23 , 24 , 25 ] . Other related work includes relation to kernel methods , e.g. , [ 26 , 27 , 28 ] , linear RNNs [ 29 ] , saturated RNNs [ 30 , 31 , 32 ] , and echo state networks [ 33 , 34 ] . Several other works talk about the expressive power of the novel sequence to sequence models Transformers [ 35 , 36 ] . Due to a large number of works in this area it is not possible to be exhaustive : apart from the references directly relevant to our work we have only been able to include a small subset . [ 37 ] gave the first “ end-to-end ” result for RNNs . Very informally , their result is : if the concept class consists of functions that are sums of functions of tokens then overparameterized RNNs trained using SGD with sufficiently small learning rate can learn such a concept class . They introduce new technical ideas , most notably what they call re-randomization which allows one to tackle the dependencies that arise because the same weights are used in RNN across time . However , an important shortcoming of their result is limited expressive power of their concept class : while this class can be surprisingly useful as noted there , it can not capture problems where the RNN needs to make use of the information in the past tokens when processing a token ( in their terminolgy , their concept class can adapt to time but not to tokens ) . Indeed , a key step in their proof shows that RNNs can learn to ignore the hidden state h ( t ) . ( The above concept class comes up because it can be learnt even if h ( t ) is ignored . ) But the hidden state h ( t ) is the hallmark of RNNs and is the source of information about the past tokens—in general , not something to be ignored . Thus , it is an important question to theoretically analyze RNNs ’ performance on general concept classes and it was also raised in [ 37 ] . This question is addressed in the present paper . As in previous work , we work with sequences of bounded length L. Without loss of generality , we work with token sequences x ( 1 ) , . . . , x ( L ) of fixed length as opposed to sequences of length up to L. Informally , our result is : Overparameterized RNNs can efficiently learn concept classes consisting of one-hidden-layer neural networks that take the entire sequence of tokens as input . The training algorithm used is SGD with sufficiently small step size . By the universality theorem for one-hidden-layer networks , such RNNs can approximate all continuous functions of x ( 1 ) , . . . , x ( L ) —though naturally the more complex the functions in the class the larger the network size required . We note that the above result applies to all three aspects mentioned above : expressive power , training and generalization . We illustrate the power of our result by showing that some regular languages such as PARITY can be recognized efficiently by RNNs . In a concurrent work [ 38 ] , the authors also study related problems . The results there are not directly comparable : On the one hand , they do not normalize the input unlike our work . On the other hand , the concept classes treated in their work appear to be weak . The authors show that RNNs can learn concept classes that include functions of the form f ( x ( ` 1 ) , x ( ` 2 ) , . . . , x ( ` N ) ) , where ` 1 , . . . , ` N ∈ [ L ] . However , the number of SGD iterations and training samples necessary depend exponentially on the minimum of the two parameters , N and ` 0 = max ( ` 1 , . . . , ` N ) −min ( ` 1 , . . . , ` N ) . 2 Preliminaries . Let Sd−1 : = { x ∈ Rd | ‖x‖2 = 1 } be the unit sphere in Rd . For positive integer n define [ n ] : = { 1 , 2 , . . . , n } . Given a vector v , by vi we denote its i-th component . Given two vectors a ∈ Rd1 and b ∈ Rd2 , [ a , b ] ∈ Rd1+d2 denotes the concatenation of the two vectors . 〈· , ·〉 denotes the standard dot product . Given a matrix M , we will denote its i-th row as mi and the element in row i and column j as mij . Given two matrices A ∈ Ra1×a2 and B ∈ Rb1×b2 with a1 = b1 let [ A , B ] r ∈ Ra1× ( a2+b2 ) denote the matrix whose rows are obtained by concatenating the respective rows of A and B . Similarly , [ A , B ] c ∈ R ( a1+b1 ) ×a2 ( assuming a2 = b2 ) denotes the matrix whose columns are obtained by concatenating the columns of A and B. O ( · ) and Ω ( · ) hide absolute constants . Similarly , poly ( · ) denotes a polynomial in its arguments with degree and coefficients bounded by absolute constants ; different instances of poly ( · ) may refer to different polynomials . Writing out explicit constants would lead to unwieldy formulas without any new insights . Let σ : R→ R , given by σ ( x ) : = max { x , 0 } = x Ix≥0 , be ReLU activation function . ReLU can be extended to act on vectors by coordinate-wise application : σ ( ( x1 , . . . , xd ) ) : = ( σ ( x1 ) , . . . , σ ( xd ) ) . Note that ReLU is a positive homogenous function of degree 1 , that is to say σ ( λx ) = λσ ( x ) for all x and all λ ≥ 0 . To be learnable efficiently , the functions in the concept class need to be not too complex . We will quantify this with the following two complexity measures which are weighted norms of the Taylor expansion and intuitively can be thought of as quantifying network size and sample complexities , resp. , needed to learn φ up to error . Definition 2.1 ( Function complexity [ 15 ] ) . Suppose that function φ : R→ R has Taylor expansion φ ( z ) = ∑∞ i=0 ciz i . For R , > 0 , define Cε ( φ , R ) : = ∑∞ i=0 ( ( C∗R ) i + ( √ log ( 1/ε ) √ i C∗R ) i ) |ci| , Cs ( φ , R ) : = C ∗∑∞ i=0 ( i+ 1 ) 1.75Ri |ci| , where C∗ = 104 . As an example , if φ ( z ) = zd for positive integer d , then Cs ( φ , R ) = O ( Rd ) and Cε ( φ , R ) = O ( Rd logd/2 ( 1ε ) ) . For φ ( z ) = sin z , cos z , e z , we have Cs ( φ , R ) = O ( 1 ) and Cε ( φ , R ) = poly ( 1/ε ) . We have Cs ( φ , R ) ≤ Cε ( φ , R ) ≤ Cs ( φ , O ( R ) ) × poly ( 1/ε ) for all φ and for φ ( z ) = sin z , ez or constant degree polynomials , they only differ by o ( 1/ε ) . See [ 15 ] for details . Note that φ itself is not a member of our concept class but functions like it will be used to construct members of our concept class .

Summary: In this paper, the authors try to prove that overparametrized RNNs can efﬁciently learn concept classes consisting of one-hidden-layer neural networks that take the entire sequence of tokens as input. The training algorithm used is SGD with sufﬁciently small step size. Conceptually and technically, the introduced new ideas enable us to extract information from the hidden state of the RNN, which addresses a crucial weakness in previous work. In the end, the authors illustrate their results on some regular language recognition problems.