Framing RNN as a kernel method: A neural ODE approach

1 Introduction . Recurrent neural networks ( RNN ) are among the most successful methods for modeling sequential data . They have achieved state-of-the-art results in difficult problems such as natural language processing ( e.g. , Mikolov et al. , 2010 ; Collobert et al. , 2011 ) or speech recognition ( e.g. , Hinton et al. , 2012 ; Graves et al. , 2013 ) . This class of neural networks has a natural interpretation in terms of ( discretization of ) ordinary differential equations ( ODE ) , which casts them in the field of neural ODE ( Chen et al. , 2018 ) . This observation has led to the development of continuous-depth models for handling irregularly-sampled time-series data , including the ODE-RNN model ( Rubanova et al. , 2019 ) , GRU-ODE-Bayes ( De Brouwer et al. , 2019 ) , or neural CDE models ( Kidger et al. , 2020 ; Morrill et al. , 2020a ) . In addition , the time-continuous interpretation of RNN allows to leverage the rich theory of differential equations to develop new recurrent architectures ( Chang et al. , 2019 ; Herrera et al. , 2020 ; Erichson et al. , 2021 ) , which are better at learning long-term dependencies . On the other hand , the development of kernel methods for deep learning offers theoretical insights on the functions learned by the networks ( Cho and Saul , 2009 ; Belkin et al. , 2018 ; Jacot et al. , 2018 ) . Here , the general principle consists in defining a reproducing kernel Hilbert space ( RKHS ) —that is , a function class H — , which is rich enough to describe the architectures of networks . A good example is the construction of Bietti and Mairal ( 2017 , 2019 ) , who exhibit an RKHS for convolutional neural networks . This kernel perspective has several advantages . First , by separating the representation of the data from the learning process , it allows to study invariances of the representations learned by the network . Next , by reducing the learning problem to a linear one in H , generalization bounds can be more easily obtained . Finally , the Hilbert structure of H provides a natural metric on neural networks , which can be used for example for regularization ( Bietti et al. , 2019 ) . ∗Equal contribution 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . Contributions . By taking advantage of the neural ODE paradigm for RNN , we show that RNN are , in the continuous-time limit , linear predictors over a specific space associated with the signature of the input sequence ( Levin et al. , 2013 ) . The signature transform , first defined by Chen ( 1958 ) and central in rough path theory ( Lyons et al. , 2007 ; Friz and Victoir , 2010 ) , summarizes sequential inputs by a graded feature set of their iterated integrals . Its natural environment is a tensor space that can be endowed with an RKHS structure ( Király and Oberhauser , 2019 ) . We exhibit general conditions under which classical recurrent architectures such as feedforward RNN , Gated Recurrent Units ( GRU , Cho et al. , 2014 ) , or Long Short-Term Memory networks ( LSTM , Hochreiter and Schmidhuber , 1997 ) , can be framed as a kernel method in this RKHS . This enables us to provide generalization bounds for RNN as well as stability guarantees via regularization . The theory is illustrated with some experimental results . Related works . The neural ODE paradigm was first formulated by Chen et al . ( 2018 ) for residual neural networks . It was then extended to RNN in several articles , with a focus on handling irregularly sampled data ( Rubanova et al. , 2019 ; Kidger et al. , 2020 ) and learning long-term dependencies ( Chang et al. , 2019 ) . The signature transform has recently received the attention of the machine learning community ( Levin et al. , 2013 ; Kidger et al. , 2019 ; Liao et al. , 2019 ; Toth and Oberhauser , 2020 ; Fermanian , 2021 ) and , combined with deep neural networks , has achieved state-of-the-art performance for several applications ( Yang et al. , 2016 , 2017 ; Perez Arribas , 2018 ; Wang et al. , 2019 ; Morrill et al. , 2020b ) . Király and Oberhauser ( 2019 ) use the signature transform to define kernels for sequential data and develop fast computational methods . The connection between continuous-time RNN and signatures has been pointed out by Lim ( 2021 ) for a specific model of stochastic RNN . Deriving generalization bounds for RNN is an active research area ( Zhang et al. , 2018 ; Akpinar et al. , 2019 ; Tu et al. , 2019 ) . By leveraging the theory of differential equations , our approach encompasses a large class of RNN models , ranging from feedforward RNN to LSTM . This is in contrast with most existing generalization bounds , which are architecture-dependent . Close to our point of view is the work of Bietti and Mairal ( 2017 ) for convolutional neural networks . Mathematical context . We place ourselves in a supervised learning setting . The input data is a sample of n i.i.d . vector-valued sequences { x ( 1 ) , . . . , x ( n ) } , where x ( i ) = ( x ( i ) 1 , . . . , x ( i ) T ) ∈ ( Rd ) T , T ≥ 1 . The outputs of the learning problem can be either labels ( classification setting ) or sequences ( sequence-to-sequence setting ) . Even if we only observe discrete sequences , each x ( i ) is mathematically considered as a regular discretization of a continuous-time process X ( i ) ∈ BV c ( [ 0 , 1 ] , Rd ) , where BV c ( [ 0 , 1 ] , Rd ) is the space of continuous functions from [ 0 , 1 ] to Rd of finite total variation . Informally , the total variation of a process corresponds to its length . Formally , for any [ s , t ] ⊂ [ 0 , 1 ] , the total variation of a process X ∈ BV c ( [ 0 , 1 ] , Rd ) on [ s , t ] is defined by ‖X‖TV ; [ s , t ] = sup ( t0 , ... , tk ) ∈Ds , t k∑ j=1 ‖Xtj −Xtj−1‖ , where Ds , t denotes the set of all finite partitions of [ s , t ] and ‖ · ‖ the Euclidean norm . We therefore have that x ( i ) j = X ( i ) j/T , 1 ≤ j ≤ T , where X ( i ) t : = X ( i ) ( t ) . We make two assumptions on the processes X ( i ) . First , they all begin at zero , and second , their lengths are bounded by L ∈ ( 0 , 1 ) . These assumptions are not too restrictive , since they amount to data translation and normalization , common in practice . Accordingly , we denote by X the subset of BV c ( [ 0 , 1 ] , Rd ) defined by X = { X ∈ BV c ( [ 0 , 1 ] , Rd ) |X0 = 0 and ‖X‖TV ; [ 0,1 ] ≤ L } and assume therefore that X ( 1 ) , . . . , X ( n ) are i.i.d . according to some X ∈ X . The norm on all spaces Rm , m ≥ 1 , is always the Euclidean one . Observe that assuming that X ∈X implies that , for any t ∈ [ 0 , 1 ] , ‖Xt‖ = ‖Xt −X0‖ ≤ ‖X‖TV ; [ 0,1 ] ≤ L. Recurrent neural networks . Classical RNN are defined by a sequence of hidden states h1 , . . . , hT ∈ Re , where , for x = ( x1 , . . . , xT ) a generic data sample , h0 = 0 and hj+1 = f ( hj , xj+1 ) for 0 ≤ j ≤ T − 1 . At each time step 1 ≤ j ≤ T , the output of the network is zj = ψ ( hj ) , where ψ is a linear function . In the present article , we rather consider the following residual version , which is a natural adaptation of classical RNN in the neural ODE framework ( see , e.g. , Yue et al. , 2018 ) : h0 = 0 and hj+1 = hj + 1 T f ( hj , xj+1 ) for 0 ≤ j ≤ T − 1 . ( 1 ) The simplest choice for the function f is the feedforward model , say fRNN , defined by fRNN ( h , x ) = σ ( Uh+ V x+ b ) , ( 2 ) where σ is an activation function , U ∈ Re×e and V ∈ Re×d are weight matrices , and b ∈ Re is the bias . The function fRNN , equipped with a smooth activation σ ( such as the logistic or hyperbolic tangent functions ) , will be our leading example throughout the paper . However , the GRU and LSTM models can also be rewritten under the form ( 1 ) , as shown in Appendix A.1 . Thus , model ( 1 ) is flexible enough to encompass most recurrent networks used in practice . Overview . Section 2 is devoted to framing RNN as linear functions in a suitable RKHS . We start by embedding iteration ( 1 ) into a continuous-time model , which takes the form of a controlled differential equation ( CDE ) . This allows , after introducing the signature transform , to define the appropriate RKHS , and , in turn , to show that model ( 1 ) boils down , in the continuous-time limit , to a linear problem on the signature . This framework is used in Section 3 to derive generalization bounds and stability guarantees . We provide some experiments in Section 4 before discussing our results in Section 5 . All proofs are postponed to the supplementary material . 2 Framing RNN as a kernel method . Roadmap . First , we quantify the difference between the discrete recurrent network ( 1 ) and its continuous-time counterpart ( Proposition 1 ) . Then , we rewrite the corresponding ODE as a CDE ( Proposition 2 ) . Under appropriate conditions , Proposition 4 shows that the solution of this equation is a linear function of the signature of the driving process . Importantly , these assumptions are valid for a feedforward RNN , as stated by Proposition 5 . We conclude in Theorem 1 . 2.1 From discrete to continuous time . Recall that h0 , . . . , hT denote the hidden states of the RNN ( 1 ) , and let H : [ 0 , 1 ] → Re be the solution of the ODE dHt = f ( Ht , Xt ) dt , H0 = h0 . ( 3 ) By bounding the difference between Hj/T and hj , the following proposition shows how to pass from discrete to continuous time , provided f satisfies the following assumption : ( A1 ) The function f is Lipschitz continuous in h and x , with Lipschitz constants Kh and Kx . We let Kf = max ( Kh , Kx ) . Proposition 1 . Assume that ( A1 ) is verified . Then there exists a unique solution H to ( 3 ) and , for any 0 ≤ j ≤ T , ‖Hj/T − hj‖ ≤ c1 T , where c1 = KfeKf ( L + sup ‖h‖≤M , ‖x‖≤L ‖f ( h , x ) ‖eKf ) and M = sup ‖x‖≤L ‖f ( h0 , x ) ‖eKf . Moreover , for any t ∈ [ 0 , 1 ] , ‖Ht‖ ≤M . Then , following Kidger et al . ( 2020 ) , we show that the ODE ( 3 ) can be rewritten under the form of a CDE . At the cost of increasing the dimension of the hidden state from e to e+ d , this allows us to reframe model ( 3 ) as a linear model in dX , in the sense that X has been moved ‘ outside ’ of f . Proposition 2 . Assume that ( A1 ) is verified . Let H : [ 0 , 1 ] → Re be the solution of ( 3 ) , and let X̄ : [ 0 , 1 ] → Rd+1 be the time-augmented process X̄t = ( X > t , 1−L2 t ) > . Then there exists a tensor field F : Rē → Rē×d̄ , ē = e+ d , d̄ = d+ 1 , such that if H̄ : [ 0 , 1 ] → Rē is the solution of the CDE dH̄t = F ( H̄t ) dX̄t , H̄0 = ( H > 0 , X > 0 ) > , ( 4 ) then its first e coordinates are equal to H . Equation ( 4 ) can be better understood by the following equivalent integral equation : H̄t = H̄0 + ∫ t 0 F ( H̄u ) dX̄u , where the integral should be understood as Riemann-Stieljes integral ( Friz and Victoir , 2010 , Section I.2 ) . Thus , the output of the RNN can be approximated by the solution of the CDE ( 4 ) , and , according to Proposition 1 , the approximation error is O ( 1/T ) . Example 1 . Consider fRNN as in ( 2 ) . If σ is Lipschitz continuous with constant Kσ , then , for any h1 , h2 ∈ Re , x1 , x2 ∈ Rd , ‖fRNN ( h1 , x1 ) − fRNN ( h2 , x1 ) ‖ = ‖σ ( Uh1 + V x1 + b ) − σ ( Uh2 + V x1 + b ) ‖ ≤ Kσ‖U‖op‖h1 − h2‖ , where ‖ · ‖op denotes the operator norm—see Appendix A.3 . Similarly , ‖f ( h1 , x1 ) − f ( h1 , x2 ) ‖ ≤ Kσ‖V ‖op‖x1 − x2‖ . Thus , assumption ( A1 ) is satisfied . The tensor field FRNN of Proposition 2 corresponding to this network is defined for any h̄ ∈ Rē by FRNN ( h̄ ) = ( 0e×d 21−Lσ ( Wh̄+ b ) Id×d 0d×1 ) , where W = ( U V ) ∈ Re×ē . ( 5 )

Summary: The authors consider a continuous data/neural ode model for recurrent neural networks. They prove convergence of the discrete model to the continuum model as the length of input sequences increases (assuming that sequences are obtained by finer and finer discretizations of the same process). They then show that the neural ODE can be represented alternatively as a controlled ODE, and that a solution of the controlled ODE can be expressed in terms of the signature of the input process. This allows the authors to equate the RNN to a kernel method for a specific kernel. Finally, they use Rademacher complexity bounds for the RKHS to estimate the generalization properties of RNNs in terms of the length of the input sequence/fineness of discretization. They use the continuum model to design a penalty term which improves stability as seen in numerical experiments.