The Recurrent Neural Tangent Kernel

1 INTRODUCTION . The overparameterization of modern deep neural networks ( DNNs ) has resulted in not only remarkably good generalization performance on unseen data ( Novak et al. , 2018 ; Neyshabur et al. , 2019 ; Belkin et al. , 2019 ) but also guarantees that gradient descent learning can find the global minimum of their highly nonconvex loss functions ( Du et al. , 2019b ; Allen-Zhu et al. , 2019b ; a ; Zou et al. , 2018 ; Arora et al. , 2019b ) . From these successes , a natural question arises : What happens when we take overparameterization to the limit by allowing the width of a DNN ’ s hidden layers to go to infinity ? Surprisingly , the analysis of such an ( impractical ) DNN becomes analytically tractable . Indeed , recent work has shown that the training dynamics of ( infinite-width ) DNNs under gradient flow is captured by a constant kernel called the Neural Tangent Kernel ( NTK ) that evolves according to a linear ordinary differential equation ( ODE ) ( Jacot et al. , 2018 ; Lee et al. , 2019 ; Arora et al. , 2019a ) . Every DNN architecture and parameter initialization produces a distinct NTK . The original NTK was derived from the Multilayer Perceptron ( MLP ) ( Jacot et al. , 2018 ) and was soon followed by kernels derived from Convolutional Neural Networks ( CNTK ) ( Arora et al. , 2019a ; Yang , 2019a ) , Residual DNNs ( Huang et al. , 2020 ) , and Graph Convolutional Neural Networks ( GNTK ) ( Du et al. , 2019a ) . In ( Yang , 2020a ) , a general strategy to obtain the NTK of any architecture is provided . In this paper , we extend the NTK concept to the important class of overparametrized Recurrent Neural Networks ( RNNs ) , a fundamental DNN architecture for processing sequential data . We show that RNN in its infinite-width limit converges to a kernel that we dub the Recurrent Neural Tangent Kernel ( RNTK ) . The RNTK provides high performance for various machine learning tasks , and an analysis of the properties of the kernel provides useful insights into the behavior of RNNs in the following overparametrized regime . In particular , we derive and study the RNTK to answer the following theoretical questions : Q : Can the RNTK extract long-term dependencies between two data sequences ? RNNs are known to underperform at learning long-term dependencies due to the gradient vanishing or exploding ( Bengio et al. , 1994 ) . Attempted ameliorations have included orthogonal weights ( Arjovsky et al. , 2016 ; Jing et al. , 2017 ; Henaff et al. , 2016 ) and gating such as in Long Short-Term Memory ( LSTM ) ( Hochreiter & Schmidhuber , 1997 ) and Gated Recurrent Unit ( GRU ) ( Cho et al. , 2014 ) RNNs . We demonstrate that the RNTK can detect long-term dependencies with proper initialization of the hyperparameters , and moreover , we show how the dependencies are extracted through time via different hyperparameter choices . Q : Do the recurrent weights of the RNTK reduce its representation power compared to other NTKs ? An attractive property of an RNN that is shared by the RNTK is that it can deal with sequences of different lengths via weight sharing through time . This enables the reduction of the number of learnable parameters and thus more stable training at the cost of reduced representation power . We prove the surprising fact that employing tied vs. untied weights in an RNN does not impact the analytical form of the RNTK . Q : Does the RNTK generalize well ? A recent study has revealed that the use of an SVM classifier with the NTK , CNTK , and GNTK kernels outperforms other classical kernel-based classifiers and trained finite DNNs on small data sets ( typically fewer than 5000 training samples ) ( Lee et al. , 2020 ; Arora et al. , 2019a ; 2020 ; Du et al. , 2019a ) . We extend these results to RNTKs to demonstrate that the RNTK outperforms a variety of classic kernels , NTKs and finite RNNs for time series data sets in both classification and regression tasks . Carefully designed experiments with data of varying lengths demonstrate that the RNTK ’ s performance accelerates beyond other techniques as the difference in lengths increases . Those results extend the empirical observations from ( Arora et al. , 2019a ; 2020 ; Du et al. , 2019a ; Lee et al. , 2020 ) into finite DNNs , NTK , CNTK , and GNTK comparisons by observing that their performance-wise ranking depends on the employed DNN architecture . We summarize our contributions as follows : [ C1 ] We derive the analytical form for the RNTK of an overparametrized RNN at initialization using rectified linear unit ( ReLU ) and error function ( erf ) nonlinearities for arbitrary data lengths and number of layers ( Section 3.1 ) . [ C2 ] We prove that the RNTK remains constant during ( overparametrized ) RNN training and that the dynamics of training are simplified to a set of ordinary differential equations ( ODEs ) ( Section 3.2 ) . [ C3 ] When the input data sequences are of equal length , we show that the RNTKs of weight-tied and weight-untied RNNs converge to the same RNTK ( Section 3.3 ) . [ C4 ] Leveraging our analytical formulation of the RNTK , we empirically demonstrate how correlations between data at different times are weighted by the function learned by an RNN for different sets of hyperparameters . We also offer practical suggestions for choosing the RNN hyperparameters for deep information propagation through time ( Section 3.4 ) . [ C5 ] We demonstrate that the RNTK is eminently practical by showing its superiority over classical kernels , NTKs , and finite RNNs in exhaustive experiments on time-series classification and regression with both synthetic and 56 real-world data sets ( Section 4 ) . 2 BACKGROUND AND RELATED WORK . Notation . We denote [ n ] = { 1 , . . . , n } , and Id the identity matrix of size d. [ A ] i , j represents the ( i , j ) -th entry of a matrix , and similarly [ a ] i represents the i-th entry of a vector . We use φ ( · ) : R → R to represent the activation function that acts coordinate wise on a vector and φ′ to denote its derivative . We will often use the rectified linear unit ( ReLU ) φ ( x ) = max ( 0 , x ) and error function ( erf ) φ ( x ) = 2√ π ∫ x 0 e−z 2 dz activation functions . N ( µ , Σ ) represents the multidimensional Gaussian distribution with the mean vector µ and the covariance matrix Σ. Recurrent Neural Networks ( RNNs ) . Given an input sequence data x = { xt } Tt=1 of length T with data at time t , xt ∈ Rm , a simple RNN ( Elman , 1990 ) performs the following recursive computation at each layer ` and each time step t g ( ` , t ) ( x ) = W ( ` ) h ( ` , t−1 ) ( x ) +U ( ` ) h ( ` −1 , t ) ( x ) + b ( ` ) , h ( ` , t ) ( x ) = φ ( g ( ` , t ) ( x ) ) , where W ( ` ) ∈ Rn×n , b ( ` ) ∈ Rn for ` ∈ [ L ] , U ( 1 ) ∈ Rn×m and U ( ` ) ∈ Rn×n for ` ≥ 2 are the RNN parameters . g ( ` , t ) ( x ) is the pre-activation vector at layer ` and time step t , and h ( ` , t ) ( x ) is the after-activation ( hidden state ) . For the input layer ` = 0 , we define h ( 0 , t ) ( x ) : = xt . h ( ` ,0 ) ( x ) as the initial hidden state at layer ` that must be initialized to start the RNN recursive computation . The output of an L-hidden layer RNN with linear read out layer is achieved via fθ ( x ) = V h ( L , T ) ( x ) , where V ∈ Rd×n . Figure 1 visualizes an RNN unrolled through time . h ( 2,2 ) h ( 2,1 ) ( x ) h ( 2,3 ) ( x ) h ( 1,2 ) ( x ) h ( 1,1 ) ( x ) h ( 1,3 ) ( x ) W ( 2 ) W ( 2 ) W ( 1 ) W ( 1 ) U ( 2 ) U ( 2 ) U ( 2 ) h ( 2,0 ) ( x ) h ( 1,0 ) ( x ) W ( 2 ) W ( 1 ) x1 x2 x3 U ( 1 ) U ( 1 ) U ( 1 ) Neural Tangent Kernel ( NTK ) . Let fθ ( x ) ∈ Rd be the output of a DNN with parameters θ . For two input data sequences x and x′ , the NTK is defined as ( Jacot et al. , 2018 ) Θ̂s ( x , x ′ ) = 〈∇θsfθs ( x ) , ∇θsfθs ( x′ ) 〉 , where fθs and θs are the network output and parameters during training at time s. 1 Let X and Y be the set of training inputs and targets , ` ( ŷ , y ) : Rd × Rd → R+ be the loss function , and L = 1|X | ∑ ( x , y ) ∈X×Y ` ( fθs ( x ) , y ) be the the empirical loss . The evolution of the parameters θs and output of the network fθs on a test input using gradient descent with infinitesimal step size ( a.k.a gradient flow ) with learning rate η is given by ∂θs ∂s = −η∇θsfθs ( X ) T∇fθs ( X ) L ( 1 ) ∂fθs ( x ) ∂s = −η∇θsfθs ( x ) ∇θsfθs ( X ) T∇fθs ( X ) L = −ηΘ̂s ( x , X ) ∇fθs ( X ) L. ( 2 ) Generally , Θ̂s ( x , x′ ) , hereafter referred to as the empirical NTK , changes over time during training , making the analysis of the training dynamics difficult . When fθs corresponds to an infinite-width MLP , ( Jacot et al. , 2018 ) showed that Θ̂s ( x , x′ ) converges to a limiting kernel at initialization and stays constant during training , i.e. , lim n→∞ Θ̂s ( x , x ′ ) = lim n→∞ Θ̂0 ( x , x ′ ) : = Θ ( x , x′ ) ∀s , which is equivalent to replacing the outputs of the DNN by their first-order Taylor expansion in the parameter space ( Lee et al. , 2019 ) . With a mean-square error ( MSE ) loss function , the training dynamics in ( 1 ) and ( 2 ) simplify to a set of linear ODEs , which coincides with the training dynamics of kernel ridge regression with respect to the NTK when the ridge term goes to zero . A nonzero ridge regularization can be conjured up by adding a regularization term λ 2 2 ‖θs − θ0‖ 2 2 to the empirical loss ( Hu et al. , 2020 ) .

Summary: This paper extends NTK to RNN to explain behavior of RNNs in overparametrized case. It’s a good extension study and interesting to see RNN with infinite-width limit converges to a kernel. The paper proves the same RNTK formula when the weights are shared and not shared. The proposed sensitivity for computationally friendly RNTK hyperparameter tuning is also insightful.