Neural CDEs for Long Time Series via the Log-ODE Method

1 INTRODUCTION . Neural controlled differential equations ( Neural CDEs ) ( Kidger et al. , 2020 ) are the continuous-time analogue to a recurrent neural network ( RNN ) , and provide a natural method for modelling temporal dynamics with neural networks . Neural CDEs are similar to neural ordinary differential equations ( Neural ODEs ) , as popularised by Chen et al . ( 2018 ) . A Neural ODE is determined by its initial condition , without a direct way to modify the trajectory given subsequent observations . In contrast the vector field of a Neural CDE depends upon the time-varying data , so that the trajectory of the system is driven by a sequence of observations . 1.1 CONTROLLED DIFFERENTIAL EQUATIONS . We begin by stating the definition of a CDE . Let a , b ∈ R with a < b , and let v , w ∈ N. Let ξ ∈ Rw . LetX : [ a , b ] → Rv be a continuous function of bounded variation ( which is for example implied by it being Lipschitz ) , and let f : Rw → Rw×v be continuous . Then we may define Z : [ a , b ] → Rw as the unique solution of the controlled differential equation Za = ξ , Zt = Za + ∫ t a f ( Zs ) dXs for t ∈ ( a , b ] , ( 1 ) The notation “ f ( Zs ) dXs ” denotes a matrix-vector product , and if X is differentiable then∫ t a f ( Zs ) dXs = ∫ t a f ( Zs ) dX ds ( s ) ds . If in equation ( 1 ) , dXs was replaced with ds , then the equation would just be an ODE . Using dXs causes the solution to depend continuously on the evolution ofX . We say that the solution is “ driven by the control , X ” . 1.2 NEURAL CONTROLLED DIFFERENTIAL EQUATIONS . We recall the definition of a Neural CDE as introduced in Kidger et al . ( 2020 ) . Consider a time series x as a collection of points xi ∈ Rv−1 with corresponding time-stamps ti ∈ R such that x = ( ( t0 , x0 ) , ( t1 , x1 ) , ... , ( tn , xn ) ) , and t0 < ... < tn . LetX : [ t0 , tn ] → Rv be some interpolation of the data such thatXti = ( ti , xi ) . Kidger et al . ( 2020 ) use natural cubic splines . Here we will actually end up finding piecewise linear interpolation to be a more convenient choice . ( We avoid issues with adaptive solvers as discussed in Kidger et al . ( 2020 , Appendix A ) simply by using fixed solvers . ) Let ξθ : Rv → Rw and fθ : Rw → Rw×v be neural networks . Let ` θ : Rw → Rq be linear , for some output dimension q ∈ N. Here θ is used to denote dependence on learnable parameters . We define Z as the hidden state and Y as the output of a neural controlled differential equation driven by X if Zt0 = ξθ ( t0 , x0 ) , with Zt = Zt0 + ∫ t t0 fθ ( Zs ) dXs and Yt = ` θ ( Zt ) for t ∈ ( t0 , tn ] . ( 2 ) That is – just like an RNN – we have evolving hidden state Z , which we take a linear map from to produce an output . This formulation is a universal approximator ( Kidger et al. , 2020 , Appendix B ) . The output may be either the time-evolving Yt or just the final Ytn . This is then fed into a loss function ( L2 , cross entropy , . . . ) and trained via stochastic gradient descent in the usual way . The question remains how to compute the integral of equation ( 2 ) . Kidger et al . ( 2020 ) let gθ , X ( Z , s ) = fθ ( Z ) dX ds ( s ) , ( 3 ) where the right hand side denotes a matrix multiplication , and then note that the integral can be written as Zt = Zt0 + ∫ t t0 gθ , X ( Zs , s ) ds . ( 4 ) This reduces the CDE to an ODE , so that existing tools for Neural ODEs may be used to evaluate this , and to backpropagate . By moving from the discrete-time formulation of an RNN to the continuous-time formulation of a Neural CDE , then every kind of time series data is put on the same footing , whether it is regularly or irregularly sampled , whether or not it has missing values , and whether or not the input sequences are of consistent length . Besides this , the continuous-time or differential equation formulation may be useful in applications where such models are explicitly desired , as when modelling physics . 1.3 CONTRIBUTIONS . Neural CDEs , as with RNNs , begin to break down for long time series . Training loss/accuracy worsens , and training time becomes prohibitive due to the sheer number of forward operations within each training epoch . Here , we apply the log-ODE method , which is a numerical method from stochastic analysis and rough path theory . It is a method for converting a CDE to an ODE , which may in turn be solved via standard ODE solvers . Thus this acts as a drop-in replacement for the original procedure that uses the derivative of the control path . In particular , we find that this method is particularly beneficial for long time series ( and incidentally does not require differentiability of the control path ) . With this method both training time and model performance of Neural CDEs are improved , and memory requirements are reduced . The resulting scheme has two very neat interpretations . In terms of numerical differential equation solvers , this corresponds to taking integration steps larger than the discretisation of the data , whilst incorporating substep information through additional terms1 . In terms of machine learning , this corresponds to binning the data prior to running a Neural CDE , with bin statistics carefully chosen to extract precisely the information most relevant to solving a CDE . 1For the reader familiar with numerical methods for SDEs , this is akin to the additional correction term in Milstein ’ s method as compared to Euler-Maruyama . 2 THEORY . We begin with motivating theory , though we note that this section is not essential for using the method . Readers more interested in practical applications should feel free to skip to section 3 . 2.1 SIGNATURES AND LOG-SIGNATURES . The signature transform is a map from paths to a vector of real values , specifying a collection of statistics about the path . It is a central component of the theory of controlled differential equations , since these statistics describe how the data interacts with dynamical systems . The log-signature is then formed by representing the same information in a compressed format . We begin by providing a formal definition of the signature , and a description of the log-signature . We will then give some intuition , first into the geometry of the first few terms of the ( log- ) signature , and then by providing a short example of how these terms appear when solving CDEs . Signature transform Let x = ( x1 , ... , xn ) , where xi ∈ Rv . Let T > 0 and 0 = t1 < t2 < ... < tn−1 < tn = T be arbitrary . Let X = ( X1 , ... , Xd ) : [ 0 , T ] → Rd be the unique continuous function such that X ( ti ) = xi and is affine on the intervals between ( essentially just a linear interpolation of the data ) . Letting2 Si1 , ... ika , b ( X ) = ∫ ... ∫ 0 < t1 < ... < tk < T k∏ j=1 dXij dt ( tj ) dtj , ( 5 ) then the depth-N signature transform of X is given by SigNa , b ( X ) = ( { S ( X ) ( i ) } d i=1 , { S ( x ) ( i , j ) } d i , j=1 , . . . , { S ( x ) ( i1 , ... , iN ) } d i1 , ... , iN=1 ) . ( 6 ) This definition is independent of the choice of T and ti ( Bonnier et al. , 2019 , Proposition A.7 ) . We see that the signature is a collection of integrals , with each integral defining a real value . It is a graded sequence of statistics that characterise the input time series . In particular , ( Hambly & Lyons , 2010 ) show that under mild conditions , Sig∞ ( X ) completely determines X up to translation ( provided time is included in a channel in X ) . Log-signature transform However , the signature transform has some redundancy : a little algebra shows that for example S1,2a , b ( X ) + S 2,1 a , b ( X ) = S 1 a , b ( X ) S 2 a , b ( X ) , so that for instance we already know S2,1a , b ( X ) provided we know the other three quantities . 2This is a slightly simplified definition , and the signature is often instead defined using the notation of stochastic calculus ; see Definition A.2 . The log-signature transform is then essentially obtained by computing the signature transform , and throwing out redundant terms , to obtain some ( nonunique ) minimal collection . Starting from the depth-N signature transform and removing some fixed set of redundancies produces the depth-N log-signature transform.3 We denote this LogSigNa , b , which is a map from Lipschitz continuous paths [ a , b ] → Rv into Rβ ( v , N ) , where β ( v , N ) denotes the dimension of the log-signature . The precise procedure is a little involved ; both this and a formula for β ( v , N ) can be found in Appendix A. Geometric intuition In figure 2 we provide a geometric intuition for the first two levels of the log-signature ( which have particularly natural interpretations ) . ( Log- ) Signatures and CDEs ( Log- ) signatures are intrinsically linked to solutions of CDEs . Let Df denote the Jacobian of a function f . Now expand equation ( 1 ) by linearising the vector field f and neglecting higher order terms : Zt ≈ Za + ∫ t a ( f ( Za ) +Df ( Za ) ( Zs − Za ) ) dX dt ( s ) ds = Za + ∫ t a ( f ( Za ) +Df ( Za ) ∫ s a f ( Zu ) dX dt ( u ) du ) dX dt ( s ) ds ≈ Za + f ( Za ) ∫ t a dX dt ( s ) ds+Df ( Za ) f ( Za ) ∫ t a ∫ s a dX dt ( u ) du dX dt ( s ) ds = Za + f ( Za ) { S ( X ) ( i ) } di=1 +Df ( Za ) f ( Za ) { S ( X ) ( i , j ) } d i , j=1 . ( 7 ) This gives a Taylor expansion of the solution , and moreover the coefficients involve the terms in the signature . Higher order Taylor expansions results in corrections using higher order signature terms . We refer the reader to section 7.1 of Friz & Victoir ( 2010 ) for further details .

Summary: .** The authors describe how to apply a log signature to temporal datasets. This operation reduces dimensionality along the time axis at the price of adding some dimensionality to the spatial dimension. Then they train a neural controlled differential equation (Neural CDE) on the transformed dataset and show that their model learns more quickly and achieves better test generalization. They report results on two real-world datasets (EigenWorms and the TSR vitals dataset).