Neural CDEs for Long Time Series via the Log-ODE Method
Authors: James Morrill
Patrick Kidger
Cristopher Salvi
James Foster
Terry Lyons
Venue:  (2021)
Keywords: CDE
neural differential equation
time series
long time series
log-ODE

Abstract: Neural Controlled Differential Equations (Neural CDEs) are the continuous-time analogue of an RNN, just as Neural ODEs are analogous to ResNets. However just like RNNs, training Neural CDEs can be difficult for long time series. Here, we propose to apply a technique drawn from stochastic analysis, namely the log-ODE method. Instead of using the original input sequence, our procedure summarises the information over local time intervals via the log-signature map, and uses the resulting shorter stream of log-signatures as the new input. This represents a length/channel trade-off. In doing so we demonstrate efficacy on problems of length up to 17k observations and observe significant training speed-ups, improvements in model performance, and reduced memory requirements compared to the existing algorithm.

Under review as a conference paper at ICLR 2021
NEURAL CDES FOR LONG TIME SERIES VIA THE LOG-ODE METHOD
Anonymous authors Paper under double-blind review
ABSTRACT
Neural Controlled Differential Equations (Neural CDEs) are the continuous-time analogue of an RNN, just as Neural ODEs are analogous to ResNets. However just like RNNs, training Neural CDEs can be difficult for long time series. Here, we propose to apply a technique drawn from stochastic analysis, namely the log- procedure
ODE method. Instead of using the original input sequence, marises the information over local time intervals via the log-signature map, and uses the resulting shorter stream of log-signatures sents a length/channel trade-off. In doing so we demonstrate efficacy problems of length up to 17k observations and observe significant training speed-ups. provements in model performance, and reduced memory requirements compared to the existing algorithm.
the new input. This repre-
INTRODUCTION
Neural controlled differential equations (Neural CDEs) Kidger al. 2020) are the continuous-time analogue to a recurrent neural network (RNN), and provide a natural method for modelling temporal dynamics with neural networks.
Neural CDEs are similar to neural ordinary differential equations (Neural ODEs), as popularised by Chen al. (2018). A Neural ODE is determined by its initial condition, without direct way to modify the trajectory given subsequent observations. In contrast the vector field a Neural CDE depends upon the time-varying data, so that the trajectory of the system is driven by sequence of observations.
1.1 CONTROLLED DIFFERENTIAL EQUATIONS
We begin by stating the definition of a CDE.
Let a, b E IK with a < h, and let v, w N. Let E I.. Let X: [a, -> R continuous function of bounded variation (which is for example implied by it being Lipschitz), and let f g be continuous.
g
Then we may define Z [a, R the unique solution of the controlled differential equation
2= Zn [ [(Z,)dX, for E (a, ba,
2, = E,
(1)
The notation "I(Z,)dX," " denotes matrix-vector product, and if X is differentiable then St. fZ,)dXx = Sf f(z,) dx (s)ds.
If in equation ① d.X, was replaced with ds, then the equation would just be an ODE, Using d.X, causes the solution depend continuously on the evolution of X. We say that the solution is "driven by the control, x
1.2 NEURAL CONTROLLED DIFFERENTIAL EQUATIONS
We recall the definition of Neural CDE introduced in Kidger et al. 2020

---

Under review as a conference paper at ICLR 2021
Consider time series collection of points E R=-1 with corresponding time-stamps' t2 E IR such that x = ((to. Øo), (11, 21),,.. (fn, In)), and to <.... tn.
Let X: Ito, fn] + _ be some interpolation of the data such that Xt (t4, xz). Kidger et al. 2020 use natural cubic splines. Here we will actually end up finding piecewise linear interpolation to be more convenient choice. (We avoid issues with adaptive solvers discussed in Kidger et al. (2020) Appendix A) simply by using fixed solvers.)
Let Es: R + R and fo: T Roxe be neural networks. Let lo R" - R2 be linear, for some output dimension E N. Here 0 is used to denote dependence on learnable parameters.
We define Z the hidden state and Y as the output of neural controlled differential equation driven by X if
Zi, Eo(to, 2a); with Z = Zeo [ fo(Z.)dx, and Y, = Eo fort E (to, tn]. (2)
That is just like RNN have evolving hidden state Z, which take linear map from to produce an output. This formulation is a universal approximator (Kidger et al. 2020 Appendix B). The output may be either the time-evolving Y, or just the final Y,,. This then fed into loss function cross entropy. ... and trained via stochastic gradient descent in the usual way.
The question remains how compute the integral of equation (2. Kidger al. 2020) let
dX
g0.x s)
fo
(s).
(3)
ds where the right hand side denotes matrix multiplication, and then note that the integral can be written as
Z, = Zto + I' ge.x s)ds.
This reduces the CDE to an ODE, so that existing tools for Neural ODEs may be used to evaluate this, and to backpropagate.
By moving from the discrete-tima formulation of an RNN to the continuous-tima formulation of a Neural CDE, then every kind of time series data is put the same footing, whether it is regularly or irregularly sampled, whether or not it has missing values, and whether or not the input sequences of consistent length.
Besides this, the continuous-tins or differential equation formulation may be useful in applications where such models are explicitly desired, when modelling physics.
1.3 CONTRIBUTIONS
Neural CDEs, as with RNNs, begin to break down for long time series. Training loss/accuracy worsens, and training time becomes prohibitive due the sheer number of forward operations within each training epoch.
Here, we apply the log-On log-ODE method, which numerical method from stochastic analysis and rough path theory. It is method for converting a CDE to an ODE, which may in turn be solved via standard ODE solvers. Thus this acts as a drop-in replacement for the original procedure that uses the derivative of the control path.
In particular, we find that this method particularly beneficial for long time series (and incidentally does not require differentiability of the control path). With this method both training time and model performance of Neural CDEs are improved, and memory requirements are reduced.
The resulting scheme has two very neat interpretations. In terms of numerical differential equation solvers, this corresponds to taking integration steps larger than the discretisation the data, whilst incorporating substep information through additional terms"" In terms of machine learning, this corresponds to binning the data prior to running a Neural CDE, with bin statistics carefully chosen to extract precisely the information most relevant to solving CDE.
Forr the reader familiar with numerical methods for SDEs, this is akin to the additional correction term in Milstein's method compared Fuler-Maruyama.
2

---

Under review as a conference paper at ICLR 2021
Hidden state Z;
Irogruticn
Hidden state Zt
Log signature puth
000000000009000
0O000000000
Integaaiion cpp
LogS
Path X
0.00
o00
90D
08
000
Time
Time
Figure I: Left: The original Neural CDE formulation. The path X is quickly varying, meaning lot of integration steps are needed resolve it. Right: The log-ODE method. The log-signature path is more slowly varying (in a higher dimensional space), and needs fewer integration steps to resolve.
THEORY
We begin with motivating theory, though we note that this section is not essential for using the method. Readers more interested in practical applications should feel free skip to section
2.1 SIGNATURES AND LOG-SIGNATURES
The signature transform is a map from paths vector of real values, specifying a collection of statistics about the path. It is a central component of the theory of controlled differential equations, since these statistics describe how the data interacts with dynamical systems. The log-signature is then formed by representing the same information in compressed format.
We begin by providing a formal definition of the signature, and a description of the log-signature. We will then give some intuition, first into the geometry of the first few terms of the (log- Jsignature, and then by providing a short example of how these terms appear when solving CDEs.
Signature transform Let x = (I1, En), where 2, E R", Let T 0 and = t1 < t2 < tn-1 < tn T be arbitrary. Let X (X), ...,.))) [0. + Rd be the unique continu- ous function such that Xt,)) = H; and is affine on the intervals between (essentially just a linear interpolation of the data). Letting
dx df (ty),
Shi...d 0.b
(X)
(5)
then the depth-N signature transform of is given by Si (X) = (1 (15(x) f Jij=- ((xx)(
(6)
This definition is independent of the choice 7 and 4 (Bonnier et al.
2019 Proposition A.7).
We that the signature is collection of integrals, with each integral defining real value. a graded sequence of statistics that characterise the input time series. In particular, (Hambly & Lyons] 2010) show that under mild conditions, Sig 0 (X) completely determines X up to translation (provided time is included in a channel in X).
Log-signature transform However, the signature transform has some redundancy: a little algebra shows that for example 51.2 (X) + 52.1 (X) = _,,( (x), (X), so that for instance we already know 2 2.6 (X) provided we know the other three quantities.
This is slightly simplified definition, and the signature is often instead defined using the notation of stochastic caleulus; Definition A.2

---

Under review as a conference paper at ICLR 2021
Data,
Path, X
log-signature
X2
Ax!
Degth
Ax7
Degth
A
A+
AX!
Ax²
Higher order
x
Xx
Figure 2: Geometric intuition for the first two levels of the log-signature for a 2-dimensional path. The depth I terms correspond the change in each of the coordinates the interval. The depth term corresponds to the Lévy area of the path, this being the signed area between the curve and the chord joining its start and endpoints.
The log-signature transform is then essentially obtained by computing the signature transform, and throwing out redundant terms, to obtain some (nonunique) minimal collection.
Starting from the depth-N signature transform and removing some fixed of redundancies pro- duces the depth-N log-signature transform B We denote this LogSig which is map from Lip- schitz continuous paths [a, b] - R" into R,,,,, where (vv .N) denotes the dimension of the log-signan signntuat. The precise procedure is little involved: both this and formula for ((v, .N) can be found in Appendix
Geometric intuition In figure 2 we provide geometric intuition for the first two levels of log-signature (which have particularly natural interpretations).
(Log-) Siinntiinss and CDEs Logg))signaturs are intrinsically linked to solutions of CDEs. Let D denote the Jacobian of function f. Now expand equation O by linearising the vector field and neglecting higher order terms:
Z. Za J" (I(Za) Dy(Za) -Za) dX
(s)
Zn (((2)) + (7() S s(7m)) dX
dX
dt (u) du
(s)
ds
dt
2 Zn + f(2x) /
dx
1"' 1"a
dX
dX
+ (D()))
(w)
du
(s)
di
di
Za f(Z.) (Zz (") +2, Jij=1
This gives a Taylor expansion of the solution, and moreover the coefficients involve the terms in the signature. Higher order Taylor expansions results in corrections using higher order signature terms. We refer the reader to section 7.1 Friz & Victoir (2010) for further details.
2.2
THE LoG-ODE METHOD
Recall for X: [a, b] + R= that LogSig E ( The log-ODE method states
Zo ~ Zo where -. //" f(Z.) LogSig ds, and Z, = Za.
(8)
where Z is defined in equation and the relationship between f to f is given in Appendix A
That the solution of the CDE may be approximated by the solution an ODE. In practice, we go further and pick some points rÜ such that = To < ... < Tm = b. We split up the CDE of
Similar terminoloyy such as "step-N log-signature" is also used in the literature.
4

---

For the Under review as a conference paper at ICLR 2021
equation (1 into an integral over (ro, r1], an integral over P2l, and on, and apply the log-ODE method to each interval separately.
See Appendix A for more details and Appendix for proof of convergence.
Also see Janssen (201 I); Lyons (2014); Boutaib et al. (2014) for other discussions the log-ODE
method. See Gaines & Lyons 1997): 7 Gyurkó & Lyons (2008); Flint & Lyons 2015); Foster et al. 2020 for applications of the log-ODe method to stochastic differential equations (SDEs).
METHOD
move on to discussing the application of the log-ODE method to Neural CDEs.
Recall that we observe some time series a piecewise linear interpolation X: [to, tn - I2" such that Xx, (t),
((4,, vo), (41, (1). ..., (ly: Fn)), and have constructed
We now pick points r; such that to To < ri < Tm fx. In principle these can be variably spaced but in practice we will typically space them equally far apart. The total number of points m should be much smaller than
In section 2 the log- signature transform was introduced. To recap, for X: [to, tn] + R" and to S T++1 tn the depth-N log-signature of X over the interval [ra, Ti41 is some collection of statistics LogSig,, (X) (ee.N)
In particular, these statistics are precisely those most relevant for solving the CDE equation
3.1 UPDATING THE NEURAL CDE HIDDEN STATE EQUATION VIA THE LOG-ODE METHOD
Recall how the Neural CDE formulation of equation log-ODE approach replace 33) with the piecewise
2
was solved via equations 3
(Z, s) ZZ) LogSigh.. (X) for s [ri, r=1)s
(9)
T++1 Ti
where To: R (v, is an arbitrary neural network, and the right hand side denotes matrix-vector product between fo and the log- signairae Equation (4) then becomes
Z, = Zea + [" To.x (Z,, s)ds.
(10)
This may now be solved as a (neural) ODE using standard ODE solvers.
3.2 RELATIONSHIP TO THE ORIGINAL METHOD
Suppose we happened to choose r; t, and ri+1 1441.
Then the log-signature term is
LogSig (X)
N
(1I)
4+ -4
The depth 1 the log-sis signature is just the increment of the path over the interval, and this becomes
AX (1tttalll t+ t
dinner
dt (t+) for [ta, (444,
(12)
that is to the same as obtained via the original method if using linear interpolation.
3.3 DISCUSSION
Ease of Implementation This method straightforward to implement using pre-existing tools. There are standard libraries available for computing the log-signature transform; use Signatory (Kidger & Lyons 20206). Then. as equation (10) is an ODE, it may be solved directly using tools such torchdi f ffeq (Chen 2018).

---

Under review as a conference paper at ICLR 2021
As alternative, note that the formulation in equation (1]) can be written in precisely the same form as equation 3). with the driving path taken to be piecewise linear in log-signature space. Computation of the rignsinaatress a sequence of logsignatures. From this we may construct a path in log-signature space, and apply existing tools for neural CDEs. This idea is summarised in figure We make this approach available in the [redacted] open source project.
therefore be considered
preprocessing step, producing
Structure of / The description here aligns with the log-ODE scheme described in equation (8) There is one discrepancy: we do not attempt model the specific structure of f. This is in prin- ciple possible, but is computationally expensive. Instead, model f as neural network directly. This need not necessarily exhibit the requisite structure, but as neural networks are universal ap- proximators (Pinkus 1999 Kidger & Lyons 2020a) then this approach is least general from modelling perspective.
Lossy Representation The log- signatre transform can be thought of as lossy representation for time series. This is made rigorous in Diehl et al. (2020), where it is shown that the log-signature can be obtained by iterating "area" operation between paths. For CDEs, these geometric features precisely encode the interaction between the data and the system.
Length/Channel Trade-Off The sequence of log-signatures is now of length which was cho- sen to be much smaller than n. As such, it much more slowly varying over the interval [fo, tn] than the original data, which was of length The differential equation it drives is better behaved, and so larger integration steps may be used in the numerical solver. This is the source of the speed-ups of this method; we observe typical speed-ups by factor of about 100.
Each element is a log-signature of size B(u, N) 2 o; the additional channels are higher-order cor- rections compensate for the larger integration steps.
Generality of the Log-ODE Method If depth = and steps r; = t, are used, then the above formulation exactly reduces onto the original Neural CDE formulation using linear interpolation. Thus the log-ODE method in fact generalises the original approach.
Applications In principle the log-ODE method may be applied to solve any Neural CDE. In prac- tice, the reduction length (from n to m), coupled with the loss information (from using the log-signature summary statistic) makes this particularly useful for long time series.
Memory Efficiency Long sequences need large amounts of memory to perform backpropagation- through-time (BPTT). As with the original Neural CDEs, the log-ODE approach supports memory- efficient backpropagation via the adjoint equations, alleviating this issue. See Kidger al. (2020).
The Depth and Step Hyperparameters To solve a Neural CDE accurately via the log-ODE method, we should be prepared take the depth N suitably large, or the intervals T++1 - r, suitably small. Accomplishing this would realistically require that they are taken very large or very small, respectively. Instead, we treat these as hyperparameters. This makes use of the log-ODE method modelling choice rather than an implementation detail.
Increasing step size will lead faster (but less informative) training by reducing the number of operations in the forward pass. Increasing depth will lead to slower (but more informative) training, as more information about each local interval is used each update.
4 EXPERIMENTS
We investigate solving a Neural CDE with and without the log-ODE method on four real-world problems. Every problem was chosen for its long length. The lengths are in fact sufficiently long that adjoint-based backpropagation (Chen et al. 2018) was needed to avoid running out memory any reasonable batch size. Every problem is regularly sampled, take I

---

Under review as a conference paper at ICLR 2021
Model
Step
Test Accuracy (%)
Time (Hrs)
Memory (Mb)
1
62.4 1: 12.1
22.0
176.5
8
64.1 + 13.3
3.1
24.3
NCDE,
32
64.1 14.3
0.5
8.0
128
48.7 = 2.6
0.1
3.9
8
77.8 + 5.9
2.1
94.2
NCDE₂
32
67.5 12.1
0.7
28.1
128
76.1 = 5.9
0.2
7.8
8
70.1 = 6.5
1.3
460.7
NCDEa
32
75.2
0.6
134.7
128
68.4 +
53.3
Table 1: Mean and standard deviation of test set accuracy (in %) over three repeats, as well as memory usage and training time, the EigenWorms dataset for depths 1-3 and small selection of step sizes. The bold values denote that the model was the top performer for that step size.
depth
1.00
0.50
025
2
18
52
64
128
258
512
1024
2048
0.00
Step
log(Step)
Figure 3: Left: Heatmap of normalised accuracies on the EigenWorms Worms dataset for differing step sizes and depths. Right: Log-log plot of the elapsed time of the algorithm against the step size.
We will denote Neural CDE model with log-ODE method, using depth N and step NCDE Taking = 1 (and any s) corresponds to not using the log-ODE method, with the data subsampled rate 1/s, as per section 3.3 Thus we use NCDE { as our benchmark: no subsampling, no log-ODE method.
In principle we could compare against RNN variants. This is for simple practical reasons: RNN- based models do not fit in the memory of the GPU resources have available. This is of the main advantages of using differential equation models in the first place, for which adjoint backprop- agation is available. (As per the first paragraph of this section.)
Each model is run three times and we report the mean and standard deviation of the test metrics along with the mean training times and memory usages.
For each task, the hyperparameters were selected by performing grid search on the NCDE; model, where chosen that the length of the sequence 500 steps. This was found to create reasonable balance between training time and sequence length. (Doing hyperoptimisation on the baseline NCDE model would have been more difficult due to the larger training times.)
Precise details of the experiments be found in Appendices C and
4.1 CLASSIFYING EIGENWORMS
Our first example uses the EigenWorms Worms dataset from the UEA archive from Bagnall et al. (2017) This consists of time series of length 17 984 and 6 channels (including time), corresponding the movement roundworm. The goal is to classify each worm as either wild-type or one of four mutant-type classes.

---

Under review as a conference paper at ICLR 2021
I,2
Time (H)
Depth
Step
Memory (Mb)
RR
HR
SpO2
RR
HR
SpO₂
2.79 1: 0.04
9.82 -: 0.34
2.83 0.27
23.8
22.1
28.1
56.5
2.80 0.06
10.72 = 0.24
3.13
3.0
2.6
4.8
14.3
NCDE,
32
2.53 0.23
12.23
2.18
1.9
0.9
2.2
9.8
128
=
11.98 0.37
2.06
0.2
0.2
0.3
8.7
2.63 0.12
8.63 = 0.24
2.88 0.15
2.1
3.4
3.3
21.8
NCDE₂
32
1.90
7,90
1.69
1.2
1.1
2.0
13.1
128
1.86 0.03
6.77
1.95
0.3
0.4
0.7
10.9
8
2.42
= 0.19
7.67 0.40
2.55 0.13
2.9
3.2
3.1
43.3
NCDEs
32
1.67
0.01
4.50
0.70
1.61
0.05
1.3
1.8
7.3
20.5
128
1.51
0.08
2.97
0.45
1.37
0.22
0.5
1.7
1.7
17.3
Table 2: Mean and standard deviation of the 1,2 losses on the test set for each of the vitals signs prediction tasks (RR, HR, SpO2) the BIDMC dataset, across three repeats. Only mean times are shown for space. The memory usage is given as the mean over all three of the tasks as was approximately the same for any task for a given depth and step. The bold values denote the algorithm with the lowest test set loss for fixed step size for each task.
RR
HR
3p02
32
64
120
250
512
162
2010
135
206
512
1224
20-6
125
200
512
1224
2046
100
Figure 4: Heatmap of normalised losses on the three BIDMC datasets for differing step sizes and depths.
See Table We see that the straightforward NCDE, model takes roughly a day to train. Using the log-ODE method (NCDE2, NCDE3) speeds this up take roughly minutes. Doing so addi- tionally improves model performance dramatically, and reduces memory usage. Naive subsampling approaches (NCDE), NCDE,", NCDE,"*) only achieve speed-ups without performance improve- ments, this be seen in the NCDE, column which corresponds to naive subsampling for step size greater than 1.
We notice that the NCDE; model has faster training times than the depth model (and sometimes better then depth 1) over each step size. This is due to the fact we imposed a stopping criterion if the loss failed to decrease after 60 epochs, meaning that the NCDEg has converged with less epochs (time per epoch will still be larger though).
See also Figure 3 which we summarise results for a larger range of step sizes.
4.2 ESTIMATING VITALS SIGNS FROM PPG AND ECG DATA
Next we consider the problem of estimating vital signs from PPG and ECG data. This comes from the TSR archive (Tan et al.) (220) using data from the Beth Israel Deaconess Medical Centre (BIDMC). We consider three separate tasks, in which we aim to predict person's 's respiratory rate (RR), their heart rate (HR), and their oxygen saturation (SpO2). This data is sampled at 125Hz with each series having a length of 4 000. There are 7 749 training samples, and 3 channels (including time).
We train a model on each of the three vitals sign prediction tasks. The metric used to evaluate performance is the 2 loss. The results
range of step sizes presented in table 2 We also provide heatmaps in Figure 4 for each dataset containing the loss values (normalised to [0, 1) for each task. The full results over all step sizes may be found in Appendix
We find that the depth model is the top performer for every task at any step size. What's more, it does so with significantly reduced training time. We attribute the improved performance to
8

---

Under review as a conference paper at ICLR 2021
the log-ODE model being better able to learn long-term dependencies due the reduced sequence length. Note that the performance of the 2 NCDE; models actually improves as the step size is increased. This is in contrast to NCDEj, which degradation in performance.
5 LIMITATIONS OF THE LOG-ODE METHOD
Number of hyperparameters Two new hyperparameters truncation depth and step size with substantial effects on training time and memory usage must now also be tuned.
Number of input channels The log-ODE method is most feasible for low numbers of input chan- nels, as the number log-signature channels ((v, N) grows exponentially in v.
RELATED WORK
There has been some work on long time series for classic RNN (GRU/LSTM) models.
Wisdom et al. (2016); Ding et al. (2019) show that unitary or orthogonal RNNs can mitigate the vanishing/cpploding gradients problem. However, they are expensive train due to the need to compute a matrix inversion at each training step. Chang et al.] (017)) introduce dilated RNNs with skip connections between RNN states, which help improve training speed and learning of long- term dependencies. Campos et al. (2017) introduce the 'Skip-RNN" model, which extend the RNN by adding an additional learnt component that skips state updates. Li et al. (2018) introduce the 'IndRNN' model, with particular structure tailored to learning long time series.
One important comparison is to hierarchical subsampling as in Graves 2012); De Mulder et al. (015)). There the data is split into windows, RNN is over each window, and then additional RNN is run over the first RNN's outputs; we may describe this as an RNN/RNN pair. [iao et al. (2019) then perform the equivalent operation with log- Liggsiggnauur//RNN pair. In this context, of log-ODE method may then be described as a Iog-signature/NCDE pair.
In comparison Liao al. (2019), this means moving from inspired choice of pre-processing to an actual implementation of the log-ODE method. In doing so the differential equation structure preserved. Moreover this takes advantage of the synergy between log-signatures (which extract statistics on how data drives differential equations), and the controlled differential equation it then drives. Broadly speaking these connections are natural: at least within the signature/CDE/rough path community, it is well-known but poorly-published fact that (log- bignaatrgs, RNNs and (Neural) CDEs are all related; see for example Kidger et al. (2020) for little exposition on this.
CNNs and Transformers have been shown to offer improvements over RNNs for modelling long- term dependencies (Bai et al., 2018 Li et al. 2019). However, both can be expensive in their own right; Transformers are famously O (2) in the length of the time series L. Whilst several approaches have been introduced to reduce this, for example Li et al. (2019) reduce this to OL (log L)²). this still be difficult with long series. Extensions specifically to long sequences do exist (Sourkov) 2018), but these typically focus on language modelling rather than multivariate time series data.
De Brouwer al. 2019); Lechner & Hasani 2020) amongst others consider continuous time
logues of GRUs and LST TMs, going some way to improving the learning of long-term dependencies. Voelker al. (2019; Gu et al. (2020) consider links with ODEs and approximation theory, to im- prove the long-term memory capacity of RNNs.
7 CONCLUSION
We demonstrate how to effectively apply Neural CDEs to long (17k) time series, via the log-ODE method. The model may still be solved via ODE methods and thus retains adjoint backpropagation and continuous dynamics. In doing so we see significant training speed-ups, improvements in model performance, and reduced memory requirements.

---

Under review as a conference paper at ICLR 2021
REFERENCES
Anthony Bagnall, James Lines, Aaron Bostrom, James Large, and Eamonn Keogh, The great time series classification bake off: a review and experimental evaluation of recent algorithmic ad- vances. Data Mining and Knowledge Discovery, 31:606-660 (6000-66,2200 2017.
Shaojie Bai, J Zico Kolter, and Vladlen Koltun. An empirical evaluation of generic convolutional and recurrent networks for sequence modeling. arXiv preprint arXiv: 1803.0117, 2018.
Patric Bonnier, Patrick Kidger, Imanol Perez Arribas, Cristopher Salvi, and Terry Lyons. Deep Signature Transforms. Advances in Neural Information Processing Systems, 2019.
Youness Boutaib, Lajos Gergely Gyurkó, Terry Lyons, and Danyu Yang. Dimension-free free Euler estimates of rough differential equations. Revue Roumaine de Mathmatiques Pures et Appliques, 59, 2014.
Víctor Campos, Brendan Jou, Xavier Giró-i Nieto, Jordi Torees, and Shih-Fu Chang. Skip RNN: Learning to Skip State Updates Recurrent Neural Networks. arXiv preprint 1wxXiv:778..000 2017.
Shiyu Chang, Yang Zhang, Wei Han, Mo Yu, Xiaoxiao Guo, Wei Tan, Xiaodong Cui, Michael Witbrock, Mark A Hasegawa-Johnson, and Thomas S Huang. Dilated recurrent neural networks. In Advances Neural Information Processing Systems, pp. 77-87, 2017.
Ricky T. Q. Chen.
torchdiffeg. 2018.
https://github.c com http:///ggttub..ccoorgg....c..it
torchdiffeq
Ricky T. Q. Chen, Yulia Rubanova, Jesse Bettencourt, and David Duvenaud. Neural Ordinary Dif-
ferential Equations. Advances in Neural Information Processing Systems, 2018.
Edward De Brouwer, Jaak Simm, Adam Arany, and Yves Moreau. Gru-ode-bayes: Continuous modeling of sporadically-ooserve time series. In Advances in Neural Information Processing Systems, pp. 1379-7390, 2019.
Wim De Mulder, Steven Bethard, and Maric-Francine Moens. A survey on the application of re- current neural networks to statistical language modeling. Computer Speech & Language, 30(1): 61-98, 2015.
Joscha Dichl, Terry Lyons, Rosa Preiß, and Jeremy Reizenstein. Areas of areas generate the shuffle algebra. arXiv preprint arXiv:2002.02338. 2020.
Guy Flint and Terry Lyons. Pathwise approximation of SDEs by coupling piecewise abelian rough paths. arXiv preprint arXiv: 1505.0/298, 2015.
James Foster, Harald Oberhauser, and Terry Lyons. An optimal polynomial approximation of Brow- nian motion. SIAM Journal on Numerical Analysis, 1393-1113 2020.
Peter K Friz and Nicolas B Victoir. Multidimensional stochastic processes as rough paths: theory and applications, volume 120. Cambridge University Press, 2010.
Jessica G Gaines and Temr Lyons. Variable step size control in the numerical solution of stochastic differential equations. SIAM Journal Applied Mathematics, 57(5):1455-144 1484, 1997.
Alex Graves. Supervised sequence labelling. In Supervised sequence labelling with recurrent neural networks, pp. 5-13. Springer, 2012.
Albert Gu, Tri Dao, Stefano Ermon, Atri Rudra, and Christopher Re. HiPPO: Recurrent Memory with Optimal Polynomial Projections. 2XXi:2008..07663 2020.
Lajos Gergely Gyurkó. Numerical methods for approximating solutions to rough differential equa- tions. DPhil thesis, University of Oxford, 2008
Lajos Gergely Gyurkó and Terry Lyons. Rough paths based numerical algorithms in computational finance. Mathematics in Finance: UIMP-RSS RMME Lluis A. Santaló Summer School, 2008
10

---

Under review as a conference paper at ICLR 2021
Ben Hambly and Terry Lyons. Uniqueness for the signature of path of bounded variation and the reduced path group. Annals of Mathematics, 171, 2010.
Arend Janssen. Order book models, signatures and numerical approximations of rough differential equations. PhD thesis, University of Oxford, 2011.
Li Jing, Caglar Gulcehre, John Peurifoy, Yichen Shen, Max Tegmark, Marin Soljacic, and Yoshua Bengio. Gated orthogonal recurrent units: On learning to forget. Neural computation, 31(4): 765-783, 2019.
Patrick Kidger and Terry Lyons. Universal Approximation with Deep Narrow Networks. COLT 2020, 2020a.
Patrick Kidger and Terry Lyons Signatory: differentiable computations of the signature and logsignature transforms, on both CPU and GPU. arXiv:2001.00706, 2020b. URL https:
Patrick Kidger, James Morrill, James Foster, and Terry Lyons. Neural controlled differential equa- tions for irregular time series. arXiv preprint Stt::20005009226, 2020.
Mathias Lechner and Ramin Hasani. Learning long-term dependencies in tregularly-samplee time series. arXiv preprint arXiv:2006.04mm 2020.
Shiyang Li, Xiaoyong Jin, Yao Xuan, Xiyou Zhou, Wenhu Chen, Yu-Xiang Wang, and Xifeng Yan. Enhancing the locality and breaking the memory bottleneck of transformer on time series forecasting. In Advances in Neural Information Processing Systems, pp. 5243-5253, 2019.
Shuai Li, Wanqing Li, Chris Cook, Ce Zhu, and Yanbo Gao. Independently recurrent neural network (indmnn): Building a longer and deeper rnn. In Proceedings of the IEFE conference on computer vision and pattern recognition, pp. 5457-5466, 2018
Shujian Liao, Terry Lyons, Weixin Yang, and Hao Ni. Learning stochastic differential equations using RNN with log signature features. arXiv preprint arXiv: 082866 2019.
Terry Lyons. Rough paths, signatures and the modelling of functions on streams. Proceedings of the International Congress of Mathematicians, 4, 2014.
Terry Lyons, Caruana Michael, and Lévy Thierry. Differential equations driven by rough paths. In École d' éééé de probabilités de Saint-Flour XXXIV-2004, edited by J. Picard in Volume 1908 of Lecture Notes in Mathematics, Berlin. Springer, 2007.
Allan Pinkus. Approximation theory of the MLP model in neural networks. Acta Numer, 8:143-
195, 1999.
Jeremy Reizenstein. Calculation of Iterated- Iterated-Integral Integral Signatures and Log Signatures. arXiv preprint arXiv:1712.02257, 2017.
Raymond Ryan. Introduction to Tensor Products of Banach Spaces. Springer Monographs in Mathematics, Springer, 2002.
Vsevolod Sourkov. Igloo: Slicing the features space to represent sequences. arXiv preprint arXiv:1807.03402, 2018.
Chang Wei Tan, Anthony Bagnall, Christoph Bergmeir, Eamonn Keogh, Francois Petitjean, and Geoffrey I. Webb. Monash university, uca, ucr time series regression archive, 2020. http:
//Linesereesreeresssso.... org/]
Aaron Voelker, Ivana Kajić, and Chris Eliasmith. Legendre memory units: Continuous-time repre- sentation in recurrent neural networks. In Advances in Neural Information Processing Systems 32. Curran Associates, Inc., 2019.
Scott Wisdom, Thomas Powers, John Hershey, Jonathan Le Roux, and Les Atlas. Full-capacity unitary recurrent neural networks. In Advances in neural information processing systems, pp. 4880-4888, 2016.
Eugene Wong and Moshe Zakai. On the Convergence of Ordinary Integrals to Stochastic Integrals. Annals of Mathematical Statistics, 3665):1560-1564, 1965.
11

---

Under review as a conference paper at ICLR 2021
Supplementary material
sections and we give a more thorough introduction to solving CDEs via the log-ODE method.
In section C we discuss the experimental details such as the choice of network structure, computing infrastructure and hyperparameter selection approach.
In section D we give full breakdown of every experimental result.
A AN INTRODUCTION TO THE LOG-ODE METHOD FOR CONTROLLED DIFFERENTIAL EQUATIONS
The log-ODE method is effective method for approximating the controlled differential equation:
dY, f(Y) dX2,,
(13)
Yo E,
where :], Rd has finite length, c R and f R" + L(R RT) is function with certain smoothness assumptions so that the CDE (13) is well posed. Throughout these appendices, L(U, V) denotes the space of linear maps between the vector spaces U and V. rough path theory, the function f is referred to the "vector field" of (13) and usually assumed to have Lip(r) regularity (see definition 10.2 in Friz & 2010)), this section, we assume one of the below conditions on the vector field:
f is bounded and has bounded derivatives.
2. f is linear.
In order to define the log-ODE method, we will first consider the tensor algebra and path signature.
Definition A.1 We say that T ((Ra) R Rd (RR))22 .... the tensor algebra of Rd and T((B") == {a- (ao;a1, ..,...) ag (R.) tk 20 is the set formal series of tensors of TR'. Moreover, (RR) and T (FR can be endowed with the operations of addition and multiplication. Given (ao; .., and (bo, .... have
a + b = (ao- + bo, a1 + b1, b,,
(14)
a 2 b = (co, C1, ..,,
(15)
where for 0, the n-th term c, E (Id) can be written using the usual tensor product
Cn Σ ak & bn-k.
The operation @ given by [55 often referred to as the "tensor product".
Definition A.2 The signature of finite length path X : [0, + Rd over the interval [s,t] is defined as the following collection of iterated Stiemann-Stielgiee integrals:
(x) = (1,x(1) x(3) ..)..... T(RR)),
(16)
where 2 1,
X (n)
dX., .... &... (R on
Similarly, define the depth-N (or truncated) signature of the path X on [s as
S.Y. = (1. J.c dx
f f... dX,,, 2 & dXax
/
T (RR),
(17)
s< CuNEE
where N (Rd) R Rdd (Rd))22 @ (Rd)8N denotes the truncated tensor algebra.
12

---

Under review as a conference paper at ICLR 2021
The (truncated) signature provides natural feature set that describes the effects path X has systems that can be modelled by (13). That said, defining the log-ODE method actually requires the so-called tog-siinattre" which efficiently encodes the same integral information the signature. The log-signature is obtained from the path's signature by removing certain algehraic redundancies, such
J. ax;ax]+ [... dx/dx;= =
for j E {1,-. d). which follows by the integration-by-Pee parts formula. To this end, we will define the logarithm map the depth-N truncated tensor algebra 7 (RR) IR + IRd
+ ..... @ (R
Definition A.3 (The logarithm of formal series) For (ag. a1. )...)E T (((*)) with ag 0,
define log(a) to be the element Tf ((R4) given by the following series:
(-1)
1 a
log(a) == log(as)
+
(18)
ag
where 1 = (1, O..,,..)) is the unit element of T ((R")) and log(ao) is viewed as log(ag)1
Definition A.4 (The logarithm of a truncated series) For a (ao, 01 .N) E (((((R)) with ag 0, define log" N (a) to be the element T N (IRd) defined the logarithm map 18 as
log" (a) == PN (log(ā)),
(19)
where à = (ag, 01.".. aN. .........))................ E T (R*) and PN denotes the standard projection map from T( T(RR)) onto T (IR.).
Definition A.S The log-signature of finite length path X [0,T] + I& over the interval [s, I] is
defined as LogSigg, (X) (X) == log(S.. (X)), where (X) denotes the path signature of X given by Definition A.2 Likewise, the depth-N (or truncated) log- signairee of X is defined for each N 2 as LogSig.. g. (X) ))= = log" (X).
The sig-signa is a map from X: [0, - Rd Ro(d,), The exact form of B(d, N) is given by
N
((d
4
s/K
with μ the Möbius function. We note that the order of this remains open question.
The final ingredient we use to define the log-ODE method are the derivatives of the vector field f.
It is worth noting that these derivatives also naturally appear in the Taylor expansion of [3
Definition A.6 (Vector field derivatives) We define foh: R. + L((R))&) RR" recursively by
(o (y) = y,
(o()) (v) = =((),
fo(k+1) (y) = D (u)(())))().
for y E R", where D(jok) denotes the Fréchet derivative of fok.
Using these definitions, we can describe two closely related numerical methods for the CDE
13
Definition A.7 (The Taylor method) Given the CDE B3 we can use the path signature of to approximate the solution Y on an interval [s, f) via its truncated Taylor expansion. That is, we use
TYy
Σ pok (Sx, K=D
(20)
approximation for Y, where each T N (1Rd) (1 k is the projection map onto (I2
13

---

Under review as a conference paper at ICLR 2021
Action of f signature of
Ys X [s, 1] - Rd
Y Taylor
Ys (((+)))8)) (X)
2
Action of f on log-signature of X
Log-ODE method
z' = (zg) (X) 200 = Y
Solve ODE [0, 1]
YL = (11
Figure 5: Illustration of the log-ODE and Taylor methods for controlled differential equations.
Definition A.8 (The Log-ODE method) Using the Taylor method 20
define the function
f: F. + L(TNN "ee)),").. by f(2) == Taylor(z, f..)) By applying f the truncated log-signature of the path X over an interval [s, tl. we can define the following ODE on [0, 1]
dz ((s)LogSig...XX
(21)
du
z(0)= = Y,
Then the log-ODE approximation of Yt (given Y. and L (X)) is defined as
LogODE(Y,, LogSi N (X)) =(1).
(22)
Remark A.9 Our assumptions of f ensure that + f( LogSi (X) either globally bounded and Lipschitz continuous or linear. Hence both the Taylor and log-ODE methods are well defined.
Remark A.10 It is well known that the log-signature of path X lies in a certain free Lie algebra (this is detailed in section 2.2.4 of Lyons et al.) (2007)). Furthermore, it is also a theorem that the Lie bracket of two vector fields is itself vector field which doesn't 't depend
choices of basis. By expressing LogSig N (X) using basis of the free Lie algebra, it can be shown that only the vector field and its (iterated) Lie brackets are required to construct the log-ODE vector field ((o).ogSi (X). In particular, this leads to our construction of the log-ODE (8) using the Lyndon basis of the free Lie algebra (see Reizenstein (2017) for a precise description the Lyndon basis).
We direct the reader to Lyons (204)) and Boutaib et al. (2014) for further details on this Lie theory.
To illustrate the log-Ole ODE method, we give two examples:
Example A.11 (The "increment-only"" log-ODE method) When N = 1, the ODE 21 becomes
dz
(())
z(0) Y..
Therefore that this "increment-only" log- -DE method is equivalent to driving the original CDE (13 by a piecewise linear approximation of the control path X. This is a classical approach for stochastic differential equations (i.e. when X, (t, Wt) with W denoting Brownian motion) and is an example of Wong-Zakai approximation (see Wong & Zakai 19655 for further details).
Example A.12 (An application for SDE simulation) Consider the following affine SDE,
dY, = a(b y1) dt + ay o dW,,
(23)
(0) Yo R
14

---

Under review as a conference paper at ICLR 2021
where a, b 2 are the mean reversion parameters, 2 is the volatility and W denotes standard real-valued Brownian motion. The means that this SDE is understood in the Stratonovich sense. The SDE (3 is known in the literature as Inhomogeneous Geometric Brownian Motion (or IGBM).
Using the control path = [(t,, W2) and setting = 3, the log-ODE (2) becomes
du a(b - h + a a., abo A,, + 2 (1) + a bo L
dz
=00) y
where h t - denotes the step size and the random variables As,t, Lige L.y are given by
Ao,1 I. W., dr
1 hW
2
1.. [.² S" W.D o oW dr , W Wa,tAss
1 nww
L 1, J." S W du dr
1 nAs W2
In Foster et al. (2020), the depth-3 log-signature of X = {(t, Wo) was approximated so that the above log- ODE method became practical and this numerical scheme exhibited state of-the-art convergence rates. For example, the approximation evvor produced by 25 steps of the high order log-ODE method was similar to the error of the "increment only" log-ODE: method with 1000 steps.
B CONVERGENCE OF THE LOG-ODE METHOD FOR ROUGH DIFFERENTIAL EQUATIONS
In this section, we shall present "rough path" error estimates for the log-ODE method. In addition, we will discuss the case when the vector fields governing the rough differential equation are linear. We begin by stating the main result of Boutaib et al. (2144) which quantifies the approximation error of the log-ODE method in terms of the regularity of the systems vector field f and control path X. Since this section uses a number of technical definitions from rough path theory, we recommend Lyons et al. (2007) as introduction to the subject.
For > O, we will use the notation Dy = {(,,t) [0,7]3 i < t} to denote a rescaled 2-simplex.
Theorem B.1 (Lemma 15 in Boutaib et al. (2014) Consider the rough differential equation
dY, fY) dx,,
(24)
Yo = E,
where we make the following assumptions:
X is geometric p-rough path in Rd. that is X Ar + T(1l (Rd) is a continuous path in the tensor algebra T (IRd) R + Rd @ (Rd) 62 ......
+ (Rd) @ with increments
X (1, (1,x!
((p)))
(25)
8.
x(k) = n&k (Xe,):
where Te Tinl (R*) + ( @k is the projection map onto (Rd) such that there
exists a sequence of continuous finite variation paths In [0, T gd whose truncated
signatures converge to X in the p-variation metric:
ds(ss [(n))))))) (zn). +0, 0,
(26)
as + 0c, where the p-variation between two continuous paths 2 and 22 in Tlpl (12)) is
- (2t. 11))
$
(z) -2,
max
sup D
Σ
nk (2) (2t, Tu
(27)
I
LED
15

---

Under review as a conference paper at ICLR 2021
where the supremum is taken over all partitions D of [0, T) and the norms must satisfy (up to some constant)
||a @ bl I ||l|||b||
for (Rd)&" and (Rd))m. For example, we can take to be the projective or injective tensor norms (see Propositions 2.1 and 3.1 in Ryan (2002)
The solution Y and its initial value & both take their values in R
The collection of vector fields {fis... Sa3 on R" are denoted by f -R" L(R", IR"), where L(R", 12, is the space of linear maps from Rn to Rdd We will assume that f has Lip(y) regularity with v > p. That is, il is bounded with [r] bounded derivatives, the last being Hölder continuous with exponent (7 Lr1).. Hence the following norm is finite:
[Iflltpp6)
max 0SkSiai
fI v V|lD fll fll
(28)
where D is the k-th (Fréchet) derivative of f and || lla-Hot is the standard a-Hölder norm with a E (0,1).
The RDE 24 is defined the Lyon's 's sense. Therefore by the Universal Limit Theorem (see Theorem 5.3 in Lyons et al. (2007), there exists unique solution Y [0, R".
We define the log-Oli Prg-ODEffor for approximating the solution Y over interval [s, 4 C [0, [ as follows:
1. Compute the depth-l7] log-signature of the control path X over [s, t]. That is, we obtain LogSigh (X) == log (S)7) (X)) Tiil (IRS), where log (+) is defined by projecting
the standard tensor logarithm map onto {a E Tl To(a) > 0}.
2. Construct the following (well-posed) ODE on the interval (0, 1].
dz
F F(=*tt)
(29)
20 YY.
where the vector field F 1Rn + RK" is defined from the log-signature as
bx]
F(z)
yok (T)
(30)
Recall that foh, Rd + L((R (R was defined previously in Definition A.6
Then we can approximate Y, using the u solution of (29). Moreover, there exists a universal constant Cp.y depending only on p and such that
|| 2 & Cps/l/ (ipp))
|X|] ( (-vorr(s.)))
(31)
where || [[pvvar(ad)] is the p-variation norm defined for paths in [ (TRe) by
Σ x Srh.tenll 111)
[ [I ==
max I<kS
sup
(32)
E.D
with the supremum laken over all partitions D of (s,
Remark B.2 the vector fields (fi,... fá} are linear, then immediately follows that linear.
Although the above theorem requires some sophisticated theory, it has a simple conclusion namely that log-ODEs approximate controlled differential equations. That said, the estimate (31) does not directly apply when the vector fields {ff} are linear as they would be unbounded. Fortunately, it is well known that linear RDEs are well posed and the growth of their solutions be estimated.
16

---

Under review as a conference paper at ICLR 2021
Theorem B.3 (Theorem 10.57 in Friz & Victoir (2010)) Consider the linear RDE on [0, T]
dY, f(Y) dX,, Yo = E,
where X is geometric p-rough path in IRd, E R and the vector fields Ufehsisa take the form Sa(y) = Aay + where {As} and {B:} are TI X n. matrices. Let K denote an upper bound on max; [| Then unique solution Y [0, T] IR exists. Moreover, it bounded and
there exists constant Cp depending only such that
1 Y, S C2(1 + [lEl]) K X p-vor](s.1] exp (C, poaorfa.))
(33)
for all 0 < SIST.
When the vector fields of the RDE (24) are linear, then the log-ODE (29 also becomes linear. Therefore, the log-ODE solution exists and is explicitly given as the exponential of the matrix F.
Theorem B.4 Consider the same linear RDE on [0, T] as in Theorem B.3
dY, f dX,,
Yo = E.
Then the log-ODE vector field F given by 30 is linear and the solution of the associated ODE 29 exists and satisfies
b
< ||a|| exp
Σ
K
Tm
(LogSig! (x))
(34)
m=1
for [0, and all 0 S SttST.
Proof B.5 Since F is a linear vector field on R", we can view it as an n X n matrix and so for [0.
= exp(uFF==
where exp denotes the matrix exponential. The result now follows by the standard estimate (exp(F))l S FlI.
Remark B.6 Due to the boundedness of linear RDES 33 and log-ODEs (34). the arguments that established Theorem B. will hold in the linear setting as (llip()) would be finite when defined on the domains that the solutions Y and lie in.
Given the local error estimate (31) for the log-ODE method, we can now consider the approximation error that exhibited by log-ODE numerical solution to the RDE 24 Thankfully, the analysis required to derive such global error estimates was developed by Greg Gyurkó in his PhD thesis. Thus the following result straightforward application of Theorem 3.2.1 from Gyurkó 2008).
Theorem B.7 Let X, f and Y satisfy the assumptions given by Theorem
and suppose that
(0 to <h .... Ix TT is partition of [0, T) with max & X|I p-varr/( (x=rl sufficiently small. We can construct numerical solution {Y JoSKEN of 24 by setting y- == Yo and for each k E 10,1, ,N - 11), defining ylog be the solution
of the following ODE:
d
F (2ftttt11).
(35)
du
20 te.211 = V-ogg
where the vector field F is constructed from the log lig-signature of X. over the interval [tx, txt1l according 30). Then there exists a constant depending only on p. 7 and [[ such that
|| Y.
||
| |
(36)
2=0
for < k N.
17

---

Under review as a conference paper at ICLR 2021
layers
Matrix multiplication
h x
Hidden layer n
Output. 2rr43
Input. Z,,
Hidden layer
LogSige..res
ReLU
ReLU
Tanh
Linear reshupe
ODE Solve
fo(Z.)
Logsig factor
Figure 6: Overview of the hidden state update network structure. We give the dimensions at cach layer in the top right hand corner of each box.
Remark B.8 The above error estimate also holds when the vector field f is linear (by Remark B.6
Since ly] is the truncation depth of the log-signatures used to construct each Bog-ODE vector field, see that high convergence rates be achieved through using more terms in each log-signature. is also unsurprising that the error estimate (36) increases with the "roughness" the control path. So just in our experiments, we see that the performance of the log-ODE method can be improved by choosing an appropriate step size and depth of log-signature.
C EXPERIMENTAL DETAILS
Code
The code reproduce the experiments is available at [redacted; see supplementary material]
Data splits Each dataset was split into training, validation, and testing dataset with relative sizes 70%//5%%/15%
Normalisation
The training splits each dataset were normalised to zero mean and unit variance. The statistics from the training were then used normalise the validation and testing datasets.
Architecture We give a graphical description of the architecture used for updating the Neural CDE hidden state in f6gure The input first run through a multilayer perceptron with n layers of size h, with with n, h being hyperparameters. ReLU nonlinearities are used each layer except the final one, where we instead use a tanh nonlinearity. The goal of this to help prevent term blow-up over the long sequences.
Note that this a small inconsistency between this work and the original model proposed in Kidger al. (2020). Here, we applied the tanh function as the final hidden layer nonlinearity, whilst in the original paper the tanh nonlinearity is applied after the final linear map. Both methods are used to constrain the rate of change of the hidden state; do not know of reason prefer over the other.
Note that the final linear layer in the multilayer perceptron is reshaped produce matrix-valued output, of shape v X p. (As fo matrix-valued, matrix- multiplication with the log- signature then produces the vector field for the ODE solver.
ODE Solver All problems used the 'rk4' solver as implemented by torchdiffeg (Chen 2018 version 0.0.1.
18

---

Under review as a conference paper at ICLR 2021
Computing infrastructure All Eigen Worms experiments three GeForce RTX 2080 Ti's. All BIDMC experiments were run on a computed with two GeForce RTX 2080 Tī's and two Quadro GP100's.
computer equipped with
Optimiser All experiments used the Adam optimiser. The learning rate was initialised at 0.032 divided by batch size. The batch size used was 1024 for EigenWorms and 512 for the BIDMC problems. If the validation loss failed to decrease after 15 epochs the learning rate was reduced by factor of 10. the validation loss did decrease after 60 epochs, training was terminated and the model was rolled back to the point at which it achieved the lowest loss the validation set.
Hyperparameter selection Hyperparameters were selected to optimise the score of the NCDE₁ model the validation set. For each dataset the search performed with step size that meant the total number of hidden state updates was equal to 500, as this represented a good balance between length and speed that allowed this was short enough that we could train using the non-adjoint training method which helped to speed this section up. The hyperparameters that were considered were:
complete the search in reasonable time-frame. In particular,
Hidden dimension: [16, 32, 64] The dimension of the hidden state Ze.
Number of layers: [2, 3, 4] The number of hidden state layers.
Hidden hidden multiplier: [1, 2, 3] Multiplication factor for the hidden hidden state, this being the 'Hidden layer k.' in figure The dimension of each of these 'hidden hidden' layers with be this value multiplied by 'Hidden dimension'.
We ran each of these 27 total combinations for every dataset and the parameters that corresponded were used as the parameters when training over the full depth and step grid. The full results from the hyperparameter search are listed tables with bolded values to show which values were eventually selected.
D EXPERIMENTAL RESULTS
Here include the full breakdown of all experimental results. Tables 5 and include all results from the Eigen Worms and BIDMC datasets respectively.
19

---

Under review as a conference paper at ICLR 2021
Validation accuracy
Hidden dim
Num layers
Hidden hidden multiplier
Total params
33.3
16
2
3
5509
43.6
16
2
2
5509
56.4
16
1
4453
64.1
16
2
8869
38.5
16
3
8869
51.3
16
1
6517
82.1
16
2
12741
35.9
16
4
3
12741
53,8
16
1
8581
35,9
32
2
3
21253
74.4
32
2
21253
43.6
32
2
1
17093
53,8
32
3
34629
87.2
32
2
34629
64.1
32
1
25317
35,9
32
3
50053
71.8
32
4
1
33541
79.5
32
2
50053
41.0
64
2
3
83461
64.1
64
2
83461
48.7
64
3
3
136837
59.0
2
136837
51.3
64
2
1
66949
56.4
64
2
198405
64.1
64
4
3
198405
64.1
64
1
99781
51.3
64
132613
Table 3: Hyperparamter selection results for the EigenWorms dataset. The blue values denote the selected hyperparameters.
20

---

Under review as a conference paper at ICLR 2021
Validation loss
Hidden dim
Num layers
Hidden hidden multiplier
Total params
RR
HR
SpO2
1.72
6.10
2.07
16
2
1
2209
1.57
5.58
1.97
16
2
3265
1.55
6.10
1.33
16
2
3
3265
1.80
5.16
2.05
16
1
3249
1.61
5.22
1.62
16
2
5601
1,56
3.34
1.18
16
3
5601
1.57
3.86
1.97
16
4
1
4289
1.45
3.54
1.25
16
4
2
8449
1.54
3.93
1.09
16
4
3
8449
1,56
6.81
1.87
32
2
1
8513
1.42
3.11
1.43
32
2
12673
1.54
3.60
1.11
32
2
3
12673
1.54
3.52
1.57
32
12641
1.39
2.96
1.03
32
3
2
21953
1.47
2.95
1.05
32
3
21953
1,55
3.00
2.00
32
4
1
16769
1.38
3,20
1.07
32
2
33281
1.43
2.58
1.01
32
4
3
33281
1,51
3.21
1.10
64
1
33409
1.43
2.22
1.00
64
2
2
49921
1,51
3.34
0.94
64
2
3
49921
1.55
3.24
2.09
64
1
49857
1.32
2.53
0.88
64
3
2
86913
1.25
2.57
0.73
64
3
86913
1.43
5.78
1.43
64
4
1
66305
1.28
2.26
0.93
64
2
132097
1.32
2.46
1.15
64
4
3
132097
Table 4: Hyperparameter selection results for each problem of the BIDMC dataset. The bold values denote the selected hyperparameters for each vitals sign problem. Note that RR and SpO2 had the same parameters selected, hence why only two lines given in bold.
21

---

Under review as a conference paper at ICLR 2021
Model
Step
Test Accuracy
Time (Hrs)
Memory (Mb)
62.4 + 12.1
22.0
176.5
69.2 + 4.4
14.6
90.6
66.7 11.8
5.5
46.6
65.8 = 12.9
2.6
31.5
64.1 13.3
3.1
24.3
16
64.1 16.8
1.5
13.4
64.1
8.0
NCDE₁
64
6.84
0.4
5.2
128
2.6
0.1
3,9
256
43.0
0.1
3.2
512
4.3
0.0
2.9
1024
41.9 14.6
0.0
2.7
2048
5.1
0.0
2.6
76.1 + 13.2
9.8
354.3
83.8 3.0
2.4
180.0
76.9
2.0
82.2
5.9
2.1
94.2
16
7.96
1.3
50.2
NCDE₂
32
67.5 12.1
0.7
28.1
64
73.5
0.4
17.2
128
5.9
0.2
7.8
256
72.6 - 12.1
0.1
8.9
512
69.2 + 11.8
0.0
7.6
1024
6..0
0.0
6.9
2048
3.9
0.0
6.5
6..7
7.4
1766.2
76.9
2.8
856.8
1.5
1.4
606.1
6.5
1.3
460.7
3.0
243.7
NCDE3
32
+.0
0.6
134.7
74.4 11.8
0.3
81.0
128
68.4
0.1
53.3
256
60.7 8.2
0.1
40.2
512
62.4 + 10.4
0.0
33.1
1024
59.8 + 3.9
0.0
29.6
2048
61.5 + 4.4
0.0
27.7
Table S: Mean and standard deviation of test set accuracy (in %) over three repeats, as well as memory usage and training time, on the Eigen Worms dataset for depths 1-3 and small selection of step sizes. The bold values denote that the model was the top performer for that step size.
22

---

Under review as a conference paper at ICLR 2021
L
Time (H)
Depth
Step
Memory (Mb)
RR
HR
SpO₂
RR
HR
SpO₂
2.79
0.04
9.82 1 0.34
2.83
0.27
23.8
22.1
28.1
56.5
2
2.87
0.03
11.69
3.36
19.3
9.6
8.8
32.6
4
2.92
0.08
11.15 0.49
0.09
5.3
5.7
3.2
20.2
2.06
10.72
0.17
3.0
2.6
4.8
14.3
16
2.22
0.61
0.11
1.7
1.4
32
2.53 +
0.43
2.68 0.12
1.9
0.9
2.2
9.8
NCDE;
64
2.63 0.11
0.09
2.88 +
0.2
0.3
0.4
9.1
128
0.18
1
2.86 2.8
0.2
0.2
0.3
8.7
0.04
12.
0.1
8.3
512
2.53 0.03
0.11
2.98 0.04
0.1
0.0
0.1
8.4
1024
2.67 0.12
11.55
2.91
0.1
0.1
0.1
8.4
2048
2.48 + 0.03
+
3.25 0.01
0.0
0.1
0.0
8.2
2
2.91 0.1
11.11 0.23
3.89 0.44
12.7
9.3
8.2
58.3
4
2.92 0.04
1.214
4.57
18.1
5.0
3.4
34,0
8.24
288
3.4
3.3
21.8
16
1.07
0.43
1.21
2.2
1.4
2.5
16.0
32
1.9
1.0
1.2
1.2
1.1
2.0
13.1
NCDE₂
64
0.04
5.44
0.07
0.3
0.3
1.7
11.6
128
0.03
6.77
0.15
0.3
0.4
0.7
10.9
0.09
5.64
0.19
512
1.11
5.05
2.17
0.1
0.2
0.4
10.3
1024
0.11
6.0
2.07
0.1
0.1
0.2
10.2
2048
2.03 0.03
7.7
0.03
0.1
0.1
0.1
10.2
2
2.82 + 0.08
11.01 0.28
4.1
8.8
9.4
6,9
125.2
2.97 0.23
10.13 0.62
3.56
71.6
2.42 0.19
7.67
0.13
2.9
3.2
3.1
43.3
16
1.74 0.05
4.1++..6 0.61
1.06
1.4
1.4
6.5
29.1
32
1.67 + 0.01
4.5
1.61
1.3
1.8
7.3
20.5
NCDE,
64
1.53
0.08
3,05 0.36
1.48
0.4
1.9
3.3
17.9
1.51 0.08
2.77
1.22
1.7
17.3
256
1.51 1 0.06
3.4 0.74
0.3
0.7
0.6
16.6
512
1.49 + 0.08
3.46 + 0.13
1.29 1.9+
0.3
0.4
0.4
15.4
1024
1.83 0.33
5.58 + 2.5
1.72 0.31
0.2
0.1
0.1
15.7
2048
2.31 0.27
9.77 1.53
2.45 0.18
0.1
15.6
Table 6: Mean and standard deviation of the 1.2 losses on the test set for each of the vitals signs prediction tasks (RR, HR, SpO2) on the BIDMC dataset, across three repeats. Only mean times are shown for space. The memory usage is given as the mean over all three of the tasks as was approximately the same for any task for given depth and step. The bold values denote the algorithm with the lowest test set loss for fixed step size for each task.
23


Reviews:
[{"review_id": "w79Veac2Xyw", "reviewer": "AnonReviewer5", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "### **Summary and Contributions of Paper**\nThis paper proposes a new method for computing Neural CDEs via the signature transform, which transforms a path integral into log signatures, i.e. a collection of iterated integrals. Then standard ODE tools are applied to each piecewise log signature.\n\n### **Strengths**\n- The writing quality is rigorous.\n- The approach seems motivated and based on a clever mathematical trick via the signature transform.\n- Experiments are convincing and sound.\n- Appendix provides proof of the approximation properties of the clipped-term signature transform (which originally requires infinite basis for exact approximation)\n\n### **Weaknesses**\n- The signature transform seems somewhat esoteric and nonstandard for readers without specific knowledge in this field. It would be very good if the authors could give more intuitive/pictorial views of this transform (I needed to read online surveys multiple times to understand the intuition behind this). For instance, I read this in detail: https://arxiv.org/pdf/1905.08494.pdf (NeurIPS 2019), which provides a much cleaner explanation of the signature transform, but also demonstrates that an entire paper is needed to simply explain the method. \n- While the authors claim that there is an ease of implementation via pre-existing tools, the larger bottleneck seems to be actually understanding the method itself (which seems to also be a function of how the paper treats this material). While I have no doubt that this work would be great for a very mathematically minded community, I am unsure of its merits for the ICLR conference community. I think the authors should provide more high level overview of the signature transform, and keep the strict math in the appendix.\n\nI am not an expert on these types of methods, so my confidence will not be as high, but I believe that this paper contributes via its insight with the signature transform, and thus my rating is marginally above the acceptance threshold.\n\nIf the authors could perhaps make the work more approachable, I would be happy to raise my score.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "b1e5JLCcY-d", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "**Summary.** The authors describe how to apply a log signature to temporal datasets. This operation reduces dimensionality along the time axis at the price of adding some dimensionality to the spatial dimension. Then they train a neural controlled differential equation (Neural CDE) on the transformed dataset and show that their model learns more quickly and achieves better test generalization. They report results on two real-world datasets (EigenWorms and the TSR vitals dataset).\n\n**Strong points.** This paper is technically sound and the method shows clear improvement over “no preprocessing.”\n\n**Weak points.** The authors are proposing a simple method of reparameterizing time series data so that information is transferred from the time dimension to the space dimension. Strangely, the title “Neural CDEs for Long Time Series via the Log-ODE Method” implies that they are proposing a new model. They are not. Rather, they are proposing a data preprocessing technique that they apply to a dataset _before_ they train an ML model on it. It’s also worth noting that this feature engineering technique is already being used on time series data (eg Liao, 2019, “Learning Stochastic…”). The only difference is that here the authors are using a continuous-time analogue of an RNN. But since this feature engineering trick is applied to the dataset independently of what model is used, I’m not sure what new scientific insights are to be gained. This work seems very similar to Liao et al (2019); compare Figure 1 in the two papers, for example. To the authors: am I missing a critical new contribution? In what ways is the “log signature” feature engineering trick significantly different in the context of CDEs compared to RNNs?\n\nAnother issue with this paper is that it presents the theory in an unnecessarily mathematical manner. Some math is useful. But whenever the authors introduce terms that an ML audience is not familiar with, they should offer a sentence of intuition regarding what that term means. From my notes on this paper, here are phrases that I found confusing and were not accompanied by any explanation or intuition: “Riemann–Stieltjes integral,” “n-fold iterated integral,” “tree-like equivalence,” “Holder continuous paths,” “Mobius function,” “Lie brackets whose foliage is a Lindon word,” and “Magnus expansion.” Again, it’s ok to introduce these concepts, but 1) you should give some reasonable intuition for what they are and why they are relevant and 2) they should be directly relevant to the main contributions of the paper.\n\nFor other reviewers/readers who want an intuitive introduction to log signatures, I thought that [this README](https://github.com/kormilitzin/the-signature-method-in-machine-learning) did a very nice job. To the authors: are there ways that we can simplify the theory section so that it remains technically correct while also being readable and accessible? Are there specific reasons that you chose to introduce the \"log signature” operation in the way you did?\n\nThe experiments are technically sound and the results are presented well. However, it is surprising that the authors did not compare to the RNN/RNN model that they mentioned in the second paragraph of related work. That would seem to be the natural baseline; using the non-preprocessed dataset is a rather trivial baseline, as the performance of recurrent models degrades quite seriously across such long sequences.\n\n**Recommendation.** 4: Ok but not good enough - rejection\n\n**Reasoning.** The Log Signature method has already been shown to be a viable preprocessing technique for time series data. The main contribution of this paper is to show that it also works with CDEs. Since the log signature method is model-independent, it does not seem surprising that this is the case. The experimental results are technically sound but would be improved by adding a stronger baseline. It would make more sense to compare to the RNN/RNN model described in the Related Work section, or to another method of data preprocessing that reduces dimensionality along the time axis.\n\n**To improve the paper.** When using a mathematical concept that will be new to an ML audience, give a one-sentence intuition for what it does and how it is relevant. Try and give a simple, intuitive example of how one might apply the log signature to a short, example time series. Make sure the reader is able to quickly grasp how a log signature of, say, order 2, works in practice. You might consider transitioning to a deeper theoretical treatment after doing so, or possibly refer the reader to a relevant tutorial such as (Chevyrev and Kormilitzina, 2016).\n\nOne way to improve the experimental results would be to compare to a stronger baseline such as RNN/RNN. If you don't want to compare to RNN/RNN, you could even just compare to a simple tensor reshape operation that folds the time dimension into the space dimension (eg., given a dataset with axes [\"num_examples\",\"time\",\"space] = [a,b * c,d], reshape to [a,b,c * d]). There are probably more clever and effective approaches than that. But the point is that, instead of comparing your dataset preprocessing method to \"no dataset preprocessing at all\", try comparing it to \"another dataset preprocessing method\" that does something similar. In this case, that method would involve transferring some of the dataset dimensionality from time to space).", "extra_scores": {}, "extra_text": {}}, {"review_id": "Gqp3DXjacpS", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summarizing the paper claims\n------------------------------------------\nThe paper introduces an approach that allows training Neural Controlled Differential Equations (CDEs) for long time series. In contrast to the Neural ODE that is determined to its initial condition, Neural CDE produces a trajectory dependent on time-varying data. The authors propose to use log-signature as input instead of the original time series. Log-signature can be understood as a lossy representation for time series, which has a much smaller length and varies slower over the same time interval. Hence, larger steps may be used in the numerical solver, and that yields to the training speed-up.\n\nIn a little more detail, the log-signature is constructed as a sequence of statistics computed using the original time series. It is characterized by the depth N, where N states for the maximal length of paths used to calculate statistics. By applying log-signature transform, the solution of the CDE may be approximated by the solution of the ODE.\n\nStrong points\n-------------------\n- The paper is clearly written.\n- The experiments are conducted for four real-wold problems (worms classification, predicting a person's heart rate/respiratory rate/oxygen saturation). \n- The authors provide an ablation study for the (log-signature length) / (number of channels in time series) trade-off,  investigate an influence of log-signature depth and solver's number of steps to the performance. \n- The paper states the limitations of the proposed method.\n\nWeak points\n-----------------\nThe paper provides an overview of many RNN based models used for long time series; however, it doesn't compare with any of them. That would be interesting to see how good is a proposed method comparing to the RNN based one in terms of test accuracy/training time/memory usage.\n\nRecommendation (accept or reject)\n------------------------------------------------\nI recommend accepting the paper. The paper provides a detailed theoretical formulation and demonstrates a significant training time speed-up for various real-world problems.\n\nUpdate: The authors thoroughly addressed all the questions, the experiments demonstrate an improvement, the theory coincides with experiments. From my perspective, that would be useful for the neural ODE community to know more about the proposed log-signature-based technique. I increase the score. \n\nQuestions\n--------------\n- Which type of ODE solvers has been used for the experiments? Does the solver's order influence the neural CDE performance?\n- What order is the value of  $\\beta(v, N)$ in the experiments?\n- In section 4.1, it is written that the naive subsampling achieves speed-up without performance improvements. Could you provide time and accuracy for these experiments?\n-  What architecture is used for $\\hat{f}$ neural network? How the choice of architecture affects performance?\n- The standard deviation in Table 1 has quite large values. Haven't you tried to tune training hyperparameters to reduce it?\n- What is the stopping criterion for the training? I'm curious why in Table 1, for the same number of solver's steps, the training time for $NCDE_2$ is longer than for  $NCDE_3$? (It seems that for the same number of training epochs, the time should be shorter because we do less preprocessing computations for log-transformation) \n\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents a method for improving the learning of neural controlled differential equation (CDE) models. Neural CDE models provide a number of advantages over neural ODE models in terms of their ability to incorporate continuous-time observations. The primary strength of this paper is that it proposes a mathematically rigorous approach to enable neural CDE models to be learned more efficiently from long time series by converting the CDE to an ODE via the log-ODE method. The results are promising in that the method is able to simultaneously improve accuracy, reduce running time and reduce memory required during learning. 

The paper has two main weaknesses. First, the authors claim that due to the problems they are solving (time series with up to 17,000 steps), there are no viable baselines outside of the family of methods that they are proposing. As was noted in the reviews, it would be advisable to consider even very basic baselines for these experiments in addition to current benchmark results. For example, the EigenWorms data set was used in the time series classification benchmark described in Bagnall et al. and there are benchmark results available that appear to outperform those shown in Table 2 (see mean test accuracy results reported here: http://www.timeseriesclassification.com/results/AllAccuracies.zip). The authors are also encouraged to consider even coarse RNN approximations such as partitioning the time series into tractable blocks for learning. It is not clear that the data sets actually have long-range dependencies despite being long. 

The second weakness is that the representation that underlies the log-ODE method (the log-signature transform) has been used in previous work in conjunction with discrete-time RNNs. It can be viewed as a preprocessing method in a sense, as was noted by a reviewer. However, it is much more fundamentally integrated with methods for solving CDE's than its prior application to RNNs indicates. 

Overall,  support for the paper did not rise to the bar required for acceptance, but we encourage the authors to revise and re-submit the work to a future venue. 

Decision: Reject