Framing RNN as a kernel method: A neural ODE approach

1 Introduction . Recurrent neural networks ( RNN ) are among the most successful methods for modeling sequential data . They have achieved state-of-the-art results in difficult problems such as natural language processing ( e.g. , Mikolov et al. , 2010 ; Collobert et al. , 2011 ) or speech recognition ( e.g. , Hinton et al. , 2012 ; Graves et al. , 2013 ) . This class of neural networks has a natural interpretation in terms of ( discretization of ) ordinary differential equations ( ODE ) , which casts them in the field of neural ODE ( Chen et al. , 2018 ) . This observation has led to the development of continuous-depth models for handling irregularly-sampled time-series data , including the ODE-RNN model ( Rubanova et al. , 2019 ) , GRU-ODE-Bayes ( De Brouwer et al. , 2019 ) , or neural CDE models ( Kidger et al. , 2020 ; Morrill et al. , 2020a ) . In addition , the time-continuous interpretation of RNN allows to leverage the rich theory of differential equations to develop new recurrent architectures ( Chang et al. , 2019 ; Herrera et al. , 2020 ; Erichson et al. , 2021 ) , which are better at learning long-term dependencies . On the other hand , the development of kernel methods for deep learning offers theoretical insights on the functions learned by the networks ( Cho and Saul , 2009 ; Belkin et al. , 2018 ; Jacot et al. , 2018 ) . Here , the general principle consists in defining a reproducing kernel Hilbert space ( RKHS ) —that is , a function class H — , which is rich enough to describe the architectures of networks . A good example is the construction of Bietti and Mairal ( 2017 , 2019 ) , who exhibit an RKHS for convolutional neural networks . This kernel perspective has several advantages . First , by separating the representation of the data from the learning process , it allows to study invariances of the representations learned by the network . Next , by reducing the learning problem to a linear one in H , generalization bounds can be more easily obtained . Finally , the Hilbert structure of H provides a natural metric on neural networks , which can be used for example for regularization ( Bietti et al. , 2019 ) . ∗Equal contribution 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . Contributions . By taking advantage of the neural ODE paradigm for RNN , we show that RNN are , in the continuous-time limit , linear predictors over a specific space associated with the signature of the input sequence ( Levin et al. , 2013 ) . The signature transform , first defined by Chen ( 1958 ) and central in rough path theory ( Lyons et al. , 2007 ; Friz and Victoir , 2010 ) , summarizes sequential inputs by a graded feature set of their iterated integrals . Its natural environment is a tensor space that can be endowed with an RKHS structure ( Király and Oberhauser , 2019 ) . We exhibit general conditions under which classical recurrent architectures such as feedforward RNN , Gated Recurrent Units ( GRU , Cho et al. , 2014 ) , or Long Short-Term Memory networks ( LSTM , Hochreiter and Schmidhuber , 1997 ) , can be framed as a kernel method in this RKHS . This enables us to provide generalization bounds for RNN as well as stability guarantees via regularization . The theory is illustrated with some experimental results . Related works . The neural ODE paradigm was first formulated by Chen et al . ( 2018 ) for residual neural networks . It was then extended to RNN in several articles , with a focus on handling irregularly sampled data ( Rubanova et al. , 2019 ; Kidger et al. , 2020 ) and learning long-term dependencies ( Chang et al. , 2019 ) . The signature transform has recently received the attention of the machine learning community ( Levin et al. , 2013 ; Kidger et al. , 2019 ; Liao et al. , 2019 ; Toth and Oberhauser , 2020 ; Fermanian , 2021 ) and , combined with deep neural networks , has achieved state-of-the-art performance for several applications ( Yang et al. , 2016 , 2017 ; Perez Arribas , 2018 ; Wang et al. , 2019 ; Morrill et al. , 2020b ) . Király and Oberhauser ( 2019 ) use the signature transform to define kernels for sequential data and develop fast computational methods . The connection between continuous-time RNN and signatures has been pointed out by Lim ( 2021 ) for a specific model of stochastic RNN . Deriving generalization bounds for RNN is an active research area ( Zhang et al. , 2018 ; Akpinar et al. , 2019 ; Tu et al. , 2019 ) . By leveraging the theory of differential equations , our approach encompasses a large class of RNN models , ranging from feedforward RNN to LSTM . This is in contrast with most existing generalization bounds , which are architecture-dependent . Close to our point of view is the work of Bietti and Mairal ( 2017 ) for convolutional neural networks . Mathematical context . We place ourselves in a supervised learning setting . The input data is a sample of n i.i.d . vector-valued sequences { x ( 1 ) , . . . , x ( n ) } , where x ( i ) = ( x ( i ) 1 , . . . , x ( i ) T ) ∈ ( Rd ) T , T ≥ 1 . The outputs of the learning problem can be either labels ( classification setting ) or sequences ( sequence-to-sequence setting ) . Even if we only observe discrete sequences , each x ( i ) is mathematically considered as a regular discretization of a continuous-time process X ( i ) ∈ BV c ( [ 0 , 1 ] , Rd ) , where BV c ( [ 0 , 1 ] , Rd ) is the space of continuous functions from [ 0 , 1 ] to Rd of finite total variation . Informally , the total variation of a process corresponds to its length . Formally , for any [ s , t ] ⊂ [ 0 , 1 ] , the total variation of a process X ∈ BV c ( [ 0 , 1 ] , Rd ) on [ s , t ] is defined by ‖X‖TV ; [ s , t ] = sup ( t0 , ... , tk ) ∈Ds , t k∑ j=1 ‖Xtj −Xtj−1‖ , where Ds , t denotes the set of all finite partitions of [ s , t ] and ‖ · ‖ the Euclidean norm . We therefore have that x ( i ) j = X ( i ) j/T , 1 ≤ j ≤ T , where X ( i ) t : = X ( i ) ( t ) . We make two assumptions on the processes X ( i ) . First , they all begin at zero , and second , their lengths are bounded by L ∈ ( 0 , 1 ) . These assumptions are not too restrictive , since they amount to data translation and normalization , common in practice . Accordingly , we denote by X the subset of BV c ( [ 0 , 1 ] , Rd ) defined by X = { X ∈ BV c ( [ 0 , 1 ] , Rd ) |X0 = 0 and ‖X‖TV ; [ 0,1 ] ≤ L } and assume therefore that X ( 1 ) , . . . , X ( n ) are i.i.d . according to some X ∈ X . The norm on all spaces Rm , m ≥ 1 , is always the Euclidean one . Observe that assuming that X ∈X implies that , for any t ∈ [ 0 , 1 ] , ‖Xt‖ = ‖Xt −X0‖ ≤ ‖X‖TV ; [ 0,1 ] ≤ L. Recurrent neural networks . Classical RNN are defined by a sequence of hidden states h1 , . . . , hT ∈ Re , where , for x = ( x1 , . . . , xT ) a generic data sample , h0 = 0 and hj+1 = f ( hj , xj+1 ) for 0 ≤ j ≤ T − 1 . At each time step 1 ≤ j ≤ T , the output of the network is zj = ψ ( hj ) , where ψ is a linear function . In the present article , we rather consider the following residual version , which is a natural adaptation of classical RNN in the neural ODE framework ( see , e.g. , Yue et al. , 2018 ) : h0 = 0 and hj+1 = hj + 1 T f ( hj , xj+1 ) for 0 ≤ j ≤ T − 1 . ( 1 ) The simplest choice for the function f is the feedforward model , say fRNN , defined by fRNN ( h , x ) = σ ( Uh+ V x+ b ) , ( 2 ) where σ is an activation function , U ∈ Re×e and V ∈ Re×d are weight matrices , and b ∈ Re is the bias . The function fRNN , equipped with a smooth activation σ ( such as the logistic or hyperbolic tangent functions ) , will be our leading example throughout the paper . However , the GRU and LSTM models can also be rewritten under the form ( 1 ) , as shown in Appendix A.1 . Thus , model ( 1 ) is flexible enough to encompass most recurrent networks used in practice . Overview . Section 2 is devoted to framing RNN as linear functions in a suitable RKHS . We start by embedding iteration ( 1 ) into a continuous-time model , which takes the form of a controlled differential equation ( CDE ) . This allows , after introducing the signature transform , to define the appropriate RKHS , and , in turn , to show that model ( 1 ) boils down , in the continuous-time limit , to a linear problem on the signature . This framework is used in Section 3 to derive generalization bounds and stability guarantees . We provide some experiments in Section 4 before discussing our results in Section 5 . All proofs are postponed to the supplementary material . 2 Framing RNN as a kernel method . Roadmap . First , we quantify the difference between the discrete recurrent network ( 1 ) and its continuous-time counterpart ( Proposition 1 ) . Then , we rewrite the corresponding ODE as a CDE ( Proposition 2 ) . Under appropriate conditions , Proposition 4 shows that the solution of this equation is a linear function of the signature of the driving process . Importantly , these assumptions are valid for a feedforward RNN , as stated by Proposition 5 . We conclude in Theorem 1 . 2.1 From discrete to continuous time . Recall that h0 , . . . , hT denote the hidden states of the RNN ( 1 ) , and let H : [ 0 , 1 ] → Re be the solution of the ODE dHt = f ( Ht , Xt ) dt , H0 = h0 . ( 3 ) By bounding the difference between Hj/T and hj , the following proposition shows how to pass from discrete to continuous time , provided f satisfies the following assumption : ( A1 ) The function f is Lipschitz continuous in h and x , with Lipschitz constants Kh and Kx . We let Kf = max ( Kh , Kx ) . Proposition 1 . Assume that ( A1 ) is verified . Then there exists a unique solution H to ( 3 ) and , for any 0 ≤ j ≤ T , ‖Hj/T − hj‖ ≤ c1 T , where c1 = KfeKf ( L + sup ‖h‖≤M , ‖x‖≤L ‖f ( h , x ) ‖eKf ) and M = sup ‖x‖≤L ‖f ( h0 , x ) ‖eKf . Moreover , for any t ∈ [ 0 , 1 ] , ‖Ht‖ ≤M . Then , following Kidger et al . ( 2020 ) , we show that the ODE ( 3 ) can be rewritten under the form of a CDE . At the cost of increasing the dimension of the hidden state from e to e+ d , this allows us to reframe model ( 3 ) as a linear model in dX , in the sense that X has been moved ‘ outside ’ of f . Proposition 2 . Assume that ( A1 ) is verified . Let H : [ 0 , 1 ] → Re be the solution of ( 3 ) , and let X̄ : [ 0 , 1 ] → Rd+1 be the time-augmented process X̄t = ( X > t , 1−L2 t ) > . Then there exists a tensor field F : Rē → Rē×d̄ , ē = e+ d , d̄ = d+ 1 , such that if H̄ : [ 0 , 1 ] → Rē is the solution of the CDE dH̄t = F ( H̄t ) dX̄t , H̄0 = ( H > 0 , X > 0 ) > , ( 4 ) then its first e coordinates are equal to H . Equation ( 4 ) can be better understood by the following equivalent integral equation : H̄t = H̄0 + ∫ t 0 F ( H̄u ) dX̄u , where the integral should be understood as Riemann-Stieljes integral ( Friz and Victoir , 2010 , Section I.2 ) . Thus , the output of the RNN can be approximated by the solution of the CDE ( 4 ) , and , according to Proposition 1 , the approximation error is O ( 1/T ) . Example 1 . Consider fRNN as in ( 2 ) . If σ is Lipschitz continuous with constant Kσ , then , for any h1 , h2 ∈ Re , x1 , x2 ∈ Rd , ‖fRNN ( h1 , x1 ) − fRNN ( h2 , x1 ) ‖ = ‖σ ( Uh1 + V x1 + b ) − σ ( Uh2 + V x1 + b ) ‖ ≤ Kσ‖U‖op‖h1 − h2‖ , where ‖ · ‖op denotes the operator norm—see Appendix A.3 . Similarly , ‖f ( h1 , x1 ) − f ( h1 , x2 ) ‖ ≤ Kσ‖V ‖op‖x1 − x2‖ . Thus , assumption ( A1 ) is satisfied . The tensor field FRNN of Proposition 2 corresponding to this network is defined for any h̄ ∈ Rē by FRNN ( h̄ ) = ( 0e×d 21−Lσ ( Wh̄+ b ) Id×d 0d×1 ) , where W = ( U V ) ∈ Re×ē . ( 5 )

Summary: This paper provides a rigorous framework to connect learning with RNNs and kernel methods, exploiting the idea that the output of a RNN can be viewed as a linear function of the signature of the input sequence in the continuous-time setting. Working within the framework (embedding the RNN into a RKHS), generalization bounds and stability guarantees are provided. Experiments are also provided to support the theory.