On the Choice of Interpolation Scheme for Neural CDEs
Authors: James Morrill
Patrick Kidger
Lingyi Yang
Terry Lyons
Venue: Accepted by TMLR ()

Abstract: Neural controlled differential equations (Neural CDEs) are a continuous-time extension of recurrent neural networks (RNNs), achieving state-of-the-art (SOTA) performance at modelling functions of irregular time series. In order to interpret discrete data in continuous time, current implementations rely on non-causal interpolations of the data. This is fine when the whole time series is observed in advance, but means that Neural CDEs are not suitable for use in \textit{online prediction tasks}, where predictions need to be made in real-time: a major use case for recurrent networks. Here, we show how this limitation may be rectified. First, we identify several theoretical conditions that control paths for Neural CDEs should satisfy, such as boundedness and uniqueness. Second, we use these to motivate the introduction of new schemes that address these conditions, offering in particular measurability (for online prediction), and smoothness (for speed). Third, we empirically benchmark our online Neural CDE model on three continuous monitoring tasks from the MIMIC-IV medical database: we demonstrate improved performance on all tasks against ODE benchmarks, and on two of the three tasks against SOTA non-ODE benchmarks.

Published in Transactions on Machine Learning Research (09/2022)
On the Choice of Interpolation Scheme for Neural CDEs
James Morrill
Comes.morrill.. 6.0gmadllll...mmmmmmmm. www.aooiiiiollllooalll.oommmmmo. com
Deportment of Mothematics
University of Oxford
Patrick Kidger
kidger@maths..co or or.ai.....r........
Department of Mathematics University of Oxford
Lingyi Yang
yanglämaths.or..co
Deportment of Mathematics University of Orford
Terry Lyons
tlyons@mati ww..etthsss...........o
Department of Mathemalics University of Oxford
Reviewed on OpenReview: https: // openreview. met/forunt forum? id= caRBFhaXIG
Abstract
Neural controlled differential equations (Neural CDFs) are continuous-time extension of recurrent neural networks (RNNs), achieving state-of-the-art (SOTA) performance at modelling functions of irregular time series. In order to interpret discrete data in continuous time, current implementations rely non-causal interpolations of the data. This is fine when the whole time series is observed in advance, but means that Neural CDEs are not suitable for use in online prediction tasks, where predictions need to be made in real-time: a major use case for recurrent networks. Here, we show how this limitation may be rectified. First, we identify several theoretical conditions that control paths for Neural CDEs should satisfy, such as boundedness and uniqueness. Second, we use these to motivatc the introduction of new schemns (hat address these conditions, offering in particular measurability (for online prediction), and smoothness (for speed). Third, we empirically benchmark online Neural CDE model three continuous monitoring tasks from the MIMIC-IV medical database: demonstrate improved performance on all tasks against ODE benchmarks, and on two of the three tasks against SOTA non-ODE benchmarks.
1 Introduction
Neural differential equations are an elegant formulation combining continuous-time differential equations with the high-capacity function approximation of neural networks. This makes them an appealing methodology for handling irregular time series. Recent examples include Rubanova al. (2019); Jia Benson (2019);
De Brouwer al. (2019); Kidger et al. (2020); Herrera al. (2021); Morrill amongst others. A comprehensive overview of the subject can be found in Kidger (2022).
(2021); Kidger al. (2021b)
Our particular focus is the Neural Controlled Differential Equation (Neural CDE) of Kidger et al. (2020). These were introduced as the general continuous-time limit of arbitrary RNNs. Besides these appealing theoretical connections and indeed recent. work on RNNs has often explicitly designed them around differentialleeuuaion--iii structures (Chang et al., 2019) - Neural CDEs have additionally been shown to demonstrate excellent empirical performance. In particular, they have been shown outperform similar Neural ODE or RNN models modelling functions of irregular time series in offline prediction tasks, where all data is observed in advance (Kidger et al., 2020; Bellot & van der Schaar, 2021).

---

Published in Transactions on Machine Learning Research (09/2022)
However despite these appealing properties, Neural CDEs cannot yet be used to learn and predict in real-time (where new data arrives during inference), due to the solution trajectory exhibiting dependence on future data, In contrast, other Neural ODE variants such as the ODE-RNN (Rubanova et al., 2019) can already handle online processing.
In this work, we show how this may be rectified, so that Neural CDEs may be adapted to apply to online problems.
1.1 Neural controlled differential equations
Suppose that we observe some time series x ((to, (o).... (f)) with f, R denoting the timestamp of the observation vector 3; E (IR {+})²; where is used to denote that some information may be missing, and to < <... < tn. Let Xx: [0, n] + R" be (contimuous, bounded variation) interpolation such that Xx(1) = (4), where '=' denotes equality up to missing data. We refer to Xx as the control path.'
If timestamps are irregular or data are missing, then the frequency of observations may carry information, which simple interpolation would obscure. This is well known to be true of medical ICU data (Che et al., 2018). In such cases, we can replace each (t,, 2x) H> (fi, where & E NJ counts the number of times the channels in 2, have been observed up to t.. Let fol: : R" + "=xx" and Çoz: R - R" be neural networks depending on learnable parameters 01, 0z. Here is a hyperparameter that descríbes the size of the hidden state and corresponds to the dimension of the information propagated along the solution trajectory.
Provided Xx is piecewise contimuuusy differentiable (as will always be the case for us), then the Neural CDE model is defined as the solution x to
z =(6) Soz (4o, vo), =(1) =(1)= =(4)) + S se (2(1))
dXx ds for I E (n: ds
and as such, the model can be interpreted and solved ordinary differential equation. Here "fo, (=(s)) dXx denotes a matrix-vector product. The solution is said to be the response of CDE driven or controlled by Xx.
de
The evolving z(t) E R" is analogous to the hidden state in an RNN, now operating in continuous time. Typically, the output of the model will be a lincar map on this hidden state: either applied =(t) for all times & [n] to produce a time-dependent output path, or on just x(l,.) for a single output such for classification.
1.2 Continuous time control signals for Neural CDEs
Neural CDEs act on and require a continuous-time embedding Xx of the observed data This provides number of benefits. Firstly, this simplifies the handling of 'messy" (irregularly sampled with missing data) time scries, by enabling it to be interpreted in the same way as regular data. Additionally, as this results in an ODE-like model, the model produces a continuously defined solution, has memory-effeient continuous adjoint methods, and the utilisation of modern ODE solvers offers trade-offs between error and computation.
Current implementations take the map x Xx to be cither a natural cubic spline or a lincar interpolation (Kidger al., 2020; Morrill et al., 2021), of which neither can be used in an online fashion. This is because any segment of the control path depends on all data values, and therefore predictions at time &; depend on future, unbbeeeu data at t > 4.
1.3 Contributions
First, we formalise the requirements for what it means to be an ideal Neural CDE control path by introducing four theoretical conditions that it should satisfy: adopled measurabilily, smoothness, boundedness, and uniqueness.
We actually have Xx: [su, sn] Rd with Xx(e;) = x;) for any su 8n, as the reparameterisation invariance property of CDEs means there no change the solution, see Appendix A.2. For example Kidger et (2020) tonk 8; = 4. We prefer 8; as is an easier choice when batching data.
2

---

Published in Transactions on Machine Learning Research (09/2022)
Continuously online
Discretely online
Offline
Online predittions
staee 1t0
Contml, X.
fn
to
Figure 1: Graphical description of the measurability definition for Neural CDEs. Arrows indicate the direction of time datapoints can influence and indicates an online prediction can be made al that point. Left: a continuously online model where no information is passed backwards in time, resulting in an online solution at all points in time. Middle: a discretely online scheme where information can be passed backwards, but not further than the preceding observation. Right: an offline scheme where information passed backwarde in time further than the preceeding observation.
Second, we use these conditions as a guide introduce two new control signals, namely the rectilinear control and cubic Hermite splines with backward differences. These are designed to satisfy the previous thcoretical conditions, well as to address the drawbacks of previously considered schemes. In particular, by utilising the invariance of control paths to reparameterisation, the rectilinear control we construct can be used in all online prediction tasks. This enables neural CDEs to be as flexible in application as RNNs.
Third, we show that this expansion in the domain of application does not come at the cost of performance. We verify the empirical behaviour our new schemes for constructing control paths, and provide straightforward recommendations into which scheme use when. We run benchmark experiments on three continuous monitoring tasks, drawn from the medical time series MIMIC-IV database. In this regime, demonstrate that Neural CDFs exhibit improved performance on all tasks against ODE benchmarks, and on two of the three tasks against non-ODE models.
2
What makes good control signal?
We now introduce four conditions that a good control signal should satisfy.
2.1 Adapted measurability
A model which can learn and predict in real-time is often referred to online model in machine learning. This is analogous to the concept of an adapted measurable process in the language of probability theory (Williams, 1991). This is the primary properly of interest to us here, which is not satisfied by existing implementations of Neural CDEs. Fulfilment of this property is what will enable Neural CDEs to be deployed in real-world real-time scenarios, such as continuous monitoring in ICU settings.
T- mcasurable Let T C [o, We say that the Neural CDE solution =t) (of Equation (1)) is T- measurable for all E T we have that x(t) function of only observations (4,, x;;) for those 4 E (4,, t]. That is, for all t it is possible to obtain prediction in an online setting.
Continuously online If the solution is T meaasurabee for T online. That is, an online prediction task can be defined for any time, and the existing control path will not be affected by new observations. The incorporation of new data bx41 means that we update =(t) for 2 1 (with 2(t), < tt+1 unchanged).
= [to, tul, then we say that continuously
Discretely online Suppose that the solution 2(t) -emasuraable for (to...., tnd (that is, only the observation times). For times between the observation times, & in (1z, (11,4)1), suppose =(1) may also
3

---

Published in Transactions on Machine Learning Research (09/2022)
Natural cubic splines
Linear
2.
A.C'
How bend the here Depends
Discontinuous first dertvative hand to resolve
since relles the next determine the
Cannot make online predictions between points
Cubic Hermite s splines with backward differences
Rectilinear
Discontinucus first hard to resolve
Continuous change gradient enforced
Linear Interpolation
Online predictions
Matth linear gradient observation
Viruull time
Figure Graphical comparison of the four control signals: natural cubic splines (top lcft), lincar control (Lop right), cubic Hermite splines with backward differences (bottom left), rectilinear control (bottom right).
depend on the onc-step ahead observation, so xtt) is function of ((fo, (fi41, Then we call the solution discrelely online. In other words, =(1) is online only at the discrete observation times and the incorporation of the next observation involves recomputing z(t) for t t..
We note that if there are missing data then this will usually destroy the discretely online property. To see why this is the case, consider the following data observations
x = ((lo vo), (1,, *), (12, 22));
(2)
with denoting missing data. The control at time t is dependent on the value Ty at t2, and as such, it cannot define discretely online solution at I1-
Offline If the solution is not at least discretely online then we say it is offline. This means that in prediction lasks, the arrival of a new data point will cause changes in the control path such that =(1) has to be recomputed for all t.
We choose to sub-categorise the definitions in this way since well separates existing models. For example, standard RNNs are discretely online, ODE-RNN (Rubanova. et. al., 2019) is continuously online, and existing Neural CDEs are offline.
A graphical depiction of this is shown in Figure 1.
2.2 Smoothness
To be able to apply numerical solvers to the integral in Equation (1) we require that the integrand be sufficiently smooth, and thus that the control be sufficiently smooth. As minimum (using Euler's method), require that Xx be piecewise twice continuuusy differentiable with bounded second derivative. In practice to use a higher-order numerical method, such as Dormand-Prince, then additional smoothness is desirable.
When using an adaptive solver and a control that is piecewise smooth, but not smooth, the solver should be informed about the jumps between pieces so that its integration steps aligns with them. Without this the solver must locate the discontinuities on its own, slow down to resolve them, and then speed up again, which numerically expensive. For example this may be done using the jump_t t argument for torchdiffeq (Chen al., 2018), or the d s.discontiimitiee. argument for Differential/quuaiions., (Rackauckas & Nie, 2017).
4

---

Published in Transactions on Machine Learning Research (09/2022)
2.3 Boundedness
We require that Xx should behave "reasonably". It should not introduce any spurious oscillations grow unboundedly from bounded data. This condition ensures that Xx is good representative of x, so that the Neural CDE can learn from the data.
Formally, let T2 = t++1 - & for each i. Then we require that there exists some continuous wt R
+
such that
dXx
dXx
1x +
+
< (Inax Tr, min T,, max (z:1),
(3)
d
dt
BV
where 1 denotes the bounded variation seminorm.
This condition is required to attain the universal approximation property for Neural CDFs. A formal proof is given in Appendix B, but the main idea is as follows. Consider collection of time series X' for which supxex (mmxx, min, max, (x:l) < co. Then (3) implies that 3 = {Xx x } is relatively compact ||ax/aelll, and hence also with respect to the topology
with respect to the topology generated by | + generated by | + +|X| As this then satisfies the compactness condition of Kidger et al. (2020, Theorem B.7), Neural CDEs driven by E X are universal approximators on x E
We note that this property is non-trivial: for example, quadratic splines exhibit a resonance property that may rosult in unbounded oscillations as time progresses.
2.4 Control signal uniqueness
Given a collection of time series X, we say that control Xx is unique with respect to X' if
+ (zo: Signature(Xx)) is injective with respect to X.
(4)
This is required, along with the boundedness property, for universal approximation of Neural CDEs to hold. Signature denotes the signature transform, which is central to the study of CDEs (Lyons, 1998; Lyons et al., 2007: Bonnier al., 2019; Morrill et al., 2020; Kidger & Lyons, 2021; Morrill, 2022).
For the reader unfamiliar with signature transforms, this may intuitively be approximated by the simplified (but not completely accurate) condition
x + Xx is injective with respect to X.
(5)
That is, we require every possible unique set of data to have a unique control path.
If the datasct is rogularly sampled, then (5) is immodiately satisfied (by the Xx(1) = (f;, property). If the dataset irregularly sampled, then this may fail. One way to recover (5) is to include the observational frequencies q. See Appendix B.2.
3 Control signals for Neural CDEs
As mentioned in Section 2, previous work has not explored the choice of control in much detail. Here we overview existing controls and identify the theoretical properties from Section 2 that each possesses. We then introduce two new controls- cubic Hermite splines with backward differences, and rectilinear controls that address issues with existing control schemes.
Natural cubic splines Natural cubic splines were used in the original Neural CDE paper by Kidger et al. (2020). This control signal requires the full time series to be available prior to construction, as =(() depends on all datapoints ([to, zo)... (tn, zn)). Any change in one datapoint has a small effect on the entire construction, even at carlier times. As such, natural cubic splines cannot be used in an online fashion. an offline scheme is sufficient, then these do still make good choice: they are relatively smooth and slowly varying, making them fast to integrate numerically.
5

---

Published in Transactions on Machine Learning Research (09/2022)
Properties
Control signal
Mcasurability
Smoothness Boundedness
Uniqueness
Natural cubic
Linear -----
(discrete) (discrete)"
(piecewise)
(with s) (with ci) (with (c)
Cubic Hermite Rectilinear
(piecewise)
Table 1: Summary of the interpolation schemes and the properties they hold for irregular and partially observed data. A superscript denotes that the property holds only if no data is missing, and ~ the property is unknown. Proofs for previously unknown properties are given Appendix B
Linear control This is arguably the simplest and most intuitive control signal, whereby we apply linear interpolation between observations. For fully observed data, the linear control defines discretely online control path, and thus has the same online properties as an RNN.
Whilst it has better online properties, the linear control is generally slower than natural cubic splines. This is due to the need to resolve derivative discontinuities at the knots (t;, æ4), as mentioned Section 2.2.
Cubic Hermite splines with backward differences This scheme smooths the discontinuities in the linear control, whilst retaining the same online properties. We achieve this by joining adjacent timepoints with cubic spline where the additional degrees of freedom are used to smooth gradient discontimuities. This leads to faster integration times than linear controls.
This differs from natural cubic splines as it solves a single equation on each [ i 1) piece independently. As a result, is more quickly varying than natural cubic splines (see Figure 2) and so produces slower integration times than natural cubic splines.
For each interval [ 1), we ensure that Xx(1) = (t,, xz), Xx(i + 1) (1441, T241), and that the gradient each node matches the backward finite difference
dXx
(i) 2; Ii-1;
di
dXx (i + 1) Ti+1
dt
The result of this is a control signal with continuous derivatives that is discretely online.
Rectilinear control Each the previous schemes need to look at least one time-step ahead to construct. the control between time points, so they are not continuously online.
To resolve this issuc, propose the following procedure. Let F; denote the forward fill of x; with respect to the missing data *. We then let Xx: [0,2m] IR" be piecewise linear such that Xx(2i) (t,, c.) for E (O....,n) n) and X2i 1) (1:41, G.) for i nI,...,n), where e is the observational frequency.
This produces a control signal the rectilinear control that updates the time and feature channels separately in a lead-lag fashion. While channels are not observed, time is increased as normal. When a channel is observed, we interpolate between channel values with the real time held fixed. We term the interpolation between channel values the virtual time, since real time fixed but the parameterisation moved forward. We have converted the time parameterisation onto space-time parameterisation. This is shown pictorially in the bottom right of Figure 2.
Note that this step is analogous to the concept of innovation in iitering/signa processing problems (Bain & Crisan, 2009). ODE-RNNs (Rubanova et al., 2019) may in fact. be seen as a special case of a Neural CDE with a rectilinear control.
Evolving the neural CDE along the virtual time at the instance data arrives enables us to update our solution at the instant data arrives before continuing with the standard time evolution. This methodology has hitherto

---

Published in Transactions on Machine Learning Research (09/2022)
Tables to 5: Control signal comparison for the regularly sampled datasets using the 'dopri5' adaptive solver. The top performer for each datasct is given in bold. NFEs represents the number of function evaluations per epoch.
Control
RMSE
NFEs x x10
Control
RMSE
NFEsx 10
Natural cubic
53.3 0.3
4.6 0.1
Natural cubic
77.6 = 1.7
4.5 + 0.1
Linear
51.7
0.8
5.4 + 0.3
Linear
77.5 + 2.4
5.7 + 0.1
Cubic Hermite
52.5 + 0.7
4.8 - 0.1
Cubic Hermite
78.5 = 1.6
4.7 + 0.1
Rectilinear
52.0 + 1.3
16.3 + 0.4
Rectilinear
79.0 = 0.7
15.7 + 0.4
Table 2: BeijingPM2.5
Table 3: BeijingPM10
Control
ACC (%)
NFEsx 10s
Control
ACC (%)
NFEsx 10a
Natural cubic
83.6 + 6.1
1.0 + 0.2
Natural cubic
92.9 + 0.3
2.67 + 0.36
Linear
97.6 + 1.5
2.2 - 0.1
Linear
93.3 = 0.7
6.35 = 0.50
Cubic Hermite
99.3 = 0.0
1.9 0.0
Cubic Hermite
92.5 = 0.4
4.16 = 0.03
Rectilinear
98.6 + 0.5
7.9 0.4
Rectilinear
93.7 + 0.8
11.3 = 0.67
Table 4: Character
Table 5: SpeechCommands
not been used for neural CDEs and is what enables the model to be used in online prediction tasks. The downside of this scheme is that the paramoterisation twice as long (domain is [0, 2n] rather than [0, n]), with correspondingly many derivative discontimuitess. This means that the model takes longer to evaluate, and train.
In Table 1, we summarise each of the aforementioned schemes, and the properties from Section 2 that each holds. Corresponding proofs are given in Appendix B.
4 Experiments
We start by illustrating the effect of using offline method simulated toy problem to predict whether Brownian path terminates above zero. We then evaluate each of the controls across range of both regularly sampled and irregularly sampled datasets. Finally, we benchmark the online Neural CDE against a collection of benchmarks on MIMIC-IV prediction tasks.
A Apart from the toy example, each model used hyperparamoters that were chosen by Bayesian optimisation strategy with 20 trials. The ranges of values the hyperparameters were optimised over as well the final configuration for every model is given in Appendix C. Full experimental details including optimisers, learning rates, normalisation, architectures and can be found in Appendix C. The code for reproducing all results is available at [redacted for anonymity].
4.1 Simulated data prediction problem
We construct loy example to demonstrate why the ouline property warrants particular emphasis. Simulating standard Brownian paths starting at the origin, predict whether the paths will be >> 0 at t = 1 at prediction times (0, 0.5, 1).
Theoretically, given information up to the current time (this is Markov by independent Gaussian increments), an online prediction cannot do better than 75% accuracy. However, as we see in Table 6, the natural cubic spline is used to construct the control path, then there some inevitable information bleed, therefore we obtain an accuracy of 81.7%. It is precisely this feature/but that makes the previous implementation of Neural CDFs unsuitable for online problems. Further details can be found in Appendix C.1.
7

---

Published in Transactions on Machine Learning Research (09/2022)
Control
(%)
Natural cubic
81.7 + 0.007
Linear
75.0 + 0.003
Cubic Hermite
74.8 + 0.006
Rectilinear
74.3 + 0.006
Table Predicting location of Brownian path.
4.2 Practical Datasets
We now overview the datasets used in our analysis. These all had 70%/15%/155 train//allidation//ees split, with (as required by the Neural CDE formulation) time included as a channel.
We bogin with four regularly sampled and fully observed datasets.
BeijingPM2.5, BeijingMM10: Both contain 10 channels and 16966 total samples. The aim to predict the PM2.5/PMIL level (two pollutant indexes) 12 different air-quality monitoring sites in Beijing (Tan et al., 2020). The performance metric is taken be the RMSE against the true value.
SpeechCommands Contains 11 channels and 34975 samples of one-second audio recordings of individual spoken words such as 'yes', 'no', 'left', and 'right' (Warden, 2018). The performance metric is taken to be classification accuracy of the spoken words.
TrajacterTrajccooriess Contains channels and 2872 samples of the z, y position and the force applied by the pen tip whilst writing letter from the Latin alphabet a single stroke (Bagnall al., 2018). The performance metrie used is the classification accuracy for the different characters.
In addition to these, additionally examine three tasks for the MIMIC-IV database (Johnson et al., 2021; Goldberger et al., 2000) related to continuous patient health monitoring. The database contains 76540 de-identified admissions to intensive care units at the Beth Israel Deaconess Medical Center. The data is highly irregular and channels have lots of missing data. Each of the three tasks has its own set of exclusion criteria; these are outlined in full in Appendix C.4.
Mortality: We predict the likelihood of mortality within an ICU stay from some initial data. The performance metric is the AUC of prediction of eventual mortality.
LOS: We estimate the length of stay of a patient given their first 24 hours of data. The performance metric is the RMSE against the true value in days.
Sepsis: We predict the risk of sepsis along a paticnt's stay. The performance metric is the AUC against the true state of sepsis that defined according to the Sepsis-3 definition (Singer al., 2016) (full details are given in Appendix C.4).
In terms of metrics, lower is better for BeijingPM2.5, BeijingPM10, and LOS, whilst higher is better for the hers..
4.3 Empirical study control signals
In Tables 2 to we compare the performance of the control signals on the regularly sampled datasets.
The NFEs column (Number of Function Evaluations per epoch) shows that natural cubic splines are uniformly the fastest choice. For example, on the SpeechCommands dataset, natural cubic splines require an average of 2.55 x 10 evaluations per epoch, piecewise cubic nearly double on 4.16 x 104, linear nearly triple on 6.35 x 10º, and recuilinear significantly more on 1.08 x 10
However, the linear, cubic Hermite, and rectilincar schemes all produce better RMSE and accuracies, generally similar each other. As they are the fastest from this group, we thus recommend cubic Hermite splines with backward differences the best choice regular datasets.
8

---

Published in Transactions on Machine Learning Research (09/2022)
Tables 7 and 8: Control signal comparison for the irregularly sampled Mortality and LOS MIMIC-IV tasks using the 'dopri5' adaptive solver. The top performer for cach datasct is given in bold. NFEs represents the number of function evaluations per epoch.
Control
AUC
NFEsx 103
Control
RMSE
NFEsx 10
Natural cubic
0.859 = 0.003
14.5 + 0.1
Natural cubic
0.241 + 0.005
4.0 0.0
Linear
0.910 + 0.003
36.9 + 0.8
Linear
0.149 + 0.037
7.8 0.2
Cubic Hermite
0.909 = 0.002
26.0 + 0.6
Cubic Hermite
0.138 = 0.025
4.2 0.1
Rectilinear
0.906 + 0.002
96.3 + 0.1
Rectilinear
0.109 + 0.008
11.9 + 1.0
Table 7: Mortality
Table 8: LOS
MIMIC-IV
Model
Mortality
LOS
Sepsis
GRU
0.846 t 0.05
0.236 0.028
0.791 + 0.002
GRU-dt
0.912 t 0.003
0.149 0.008
0.8 0.001
GRU-dt-intensity GRU-D
0.924 i 0.014
0.142 0.013
0.801 + 0.002
0.93 = 0.002
0.148 + 0.002
0.801 = 0.002
ODE-RNN
0.524 + 0.004
0.154 + 0.004
0.793 = 0.001
Online Neural CDE
0.908 + 0.003
0.11 + 0.009
0.77 = 0.002
Online Neural CDE w/ observational freq
0.943 + 0.003
0.099 0.001
0.795 = 0.004
Table 9: Benchmarking online Neural CDEs (Neural CDEs with rectilinear interpolation) using the 'rk4' solver with and without observational frequency against a range of algorithms the MIMIC-IV tasks. The top performer for each problem is shown in bold.
We see a similar story on the irrogularly sampled MIMIC-IV tasks in Tables 7 and 8 (sco also Appendix C.4 for results on the sepsis task). Natural cubic splines are again the fastest, but again their AUC/RMSE performance is poor. Linear Dinear/piecewise cubic performs best on the Mortality prediction tasks, whereas rectilinear is the best on LOS. Note that only the rectilinear model can actually be deployed in continuous ICU monitoring scenario.
4.4 Benchmarking on MIMIC-IV
Finally, we benchmark the online (rectilinear) Neural CDE on the three tasks from the MIMIC-IV database.
We highlight that Neural CDEs have been tested before on such problems before (notably Kidger et al. (2020) sepsis detection tasks); however, this is the first such benchmarking of Neural CDE model that actually be deployed in an online, real-time, in-hospital environment.
Our benchmarks include: vanilla GRU; GRU-dt, which is a GRU that additionally includes the time difference between observations; GRU-dt-intensity, the same as a GRU-dt but also includes the observational intensity; GRU-D (Che et al., 2018), incorporates both time differences and observational intensity but in a more sophisticated manner; and ODE-RNN (Rubanova et al., 2019). The ODE-RNN chosen as it represents a continuously online ODE-benchmark, whereas all other models operate only discretely online.
The results over three runs are presented in Table 9. We that upon inclusion of the measurement intensity matrix, the causal Neural CDE becomes extremely competitive with the GRU-D benchmark, achieves convincing improvements in performance in the LOS and mortality tasks, and is only narrowly beaten in the sepsis detection task. We also see across-the-board superior performance to the ODE-RNN.
It already known that Neural CDEs are effective modelling irregular functions on time series. However, these results, in particular those on rectilinear controls, are the first demonstration that Neural CDEs can be adapted for in real-world online scenario whilst retaining performance.
9

---

Published in Transactions on Machine Learning Research (09/2022)
5
Related work
Several works have now studied Neural CDEs in some capacity.
Bellot & van der Schaar (2021) develop a SOTA counterfactual estimation method in continuous time for irregular data using the Neural CDF. They state that misaligned observation times had been a problem for previous methods, and none had been able operate in continuous time.
Morrill et al. (2021) apply techniques from rough path theory so that Neural CDEs may better handle very long time series.
Zhuang et al. (2021) apply a reversible ODE solver so that both optimise-then-discrelin and discretise-then optimise methods produce the same gradients. Meanwhile Kidger et al. (2021a) tweak the numerical solver methods.
to approximately double the speed of training Neural CDEs via
More broadly, there has been much interest applying neural differential equations to time series.
Rubanova et al. (2019) introduced ODE-RNNS, which are neural jump ordinary differential equation with jumps at each observation. Meanwhile De Brouwer et al. (2019); Herrera et al. (2021) take very similar approaches to each other, using neural ODEs to perform contimuous-tin time filtering.
Jia. & Benson (2019); Li et al. (2020); Kidger et al. (2021b) amongst others consider Neural SDEs for time series modelling. The distinction that Neural CDEs are used to model functions of time series (typically supervised learning), whilst Neural SDEs seek to model the time series themselves (typically unsupervised learning)-
Other neural differential equation based unsupervised approaches to time series modelling use random ODEs (Noreliffe et al., 2021) and spatio-temporal point processes (Chen et al., 2021b).
Several works have studied using neural diffcrential equations to model time series arising from physical systems, for example chemical kinetic modelling (Kim et al., 2021), oscillatory dynamical systems (Norcliffe et al., 2020), systems with switching behaviour (Chen et al., 2021a), and those arising in general scientific modelling (Rackauckas al., 2020).
6 Discussion and limitations
Novelty
1. The extension of Neural CDEs to online problems is a qualitative step-change in their appli- cability. Whereas they have previously been restricted only to offline problems, they may now be applied as flexibly as RNNs, which are their discrete- time counterparts.
2. The experimental results demonstrate significant accuracy /loss improvements on all datasets over the previous (natural cubic spline) implementation, even for offline tasks.
Recommendations for the control signal Considering the results from Section 4 and the properties from Section 2, we make the following recommendations for Neural CDE control paths:
If the problem is online and requires the solution to be continuously online, then use rectilinear controls. This the only scheme that is continuously online.
Likewise if the problem is online, has missing data, and requires the solution to be discretely online, then use rectilincar controls. This is the only scheme that discretely online in the presence of missing data.
If the problem online, has no missing data, and requires the solution be discretely online, then use cubic Hermite splines with backward differences. We see in Tables to 5, and 8 that the performance is in-line with both linear and rectilinear, but faster speeds.
Likewise, these also recommended if the problem is offline.
10

---

Published in Transactions on Machine Learning Research (09/2022)
3. If speed is of greater importance than accuracy, and the task is offlin,, then use natural cubic splines. This is evidenced by the results in Tables 2 to 5 which show natural cubic splines to be significantly faster than other alternatives.
As such our findings recommend either rectilinear control both introduced in this paper, for most cases.
or cubic Hermite splines with backward differences,
The main limitation of the techniques introduced here come from rectilinear controls. These are typically slow, and if trained with discretis-then--ptimee techniques come with high memory usage. Despite this, in many online cases these are 'the only game town'.
Limitations
Implementation
To help facilitate adoption, both rectilinear controls and cubic Hermite splines with backward differences have been implemented in the torchcde open-source library for CDEs.
7 Conclusion
We formalised the properties that ideal Neural CDE control schemes should have. In doing we identified two new control schemes that address issues with existing implementations, in particular with respect to online predictions and speed. Having performed both a theoretical and empirical study into the schemes' behaviour, we provide recommendations regarding which scheme to use when. This included benchmarking the online Ncural CDE on three continuous monitoring ICU tasks, in which improved and statc-of-tho-art performance is demonstrated against similar ODE or RNN based approaches.
References
Ax adaptive experimentation platform, 2021. URL
Anthony Bagnall, Hoang Anh Dau, Jason Lines, Michael Flynn, James Large, Aaron Bostrom, Paul Southam, and Eamonn Kcogh. The UEA multivariate time series classification archive. arXiv:1811.00075 2018.
Alan Bain and Dan Crisan. Fundamentals of stochastic filtering. Springer, 2009.
Alexis Bellot and Mihaela van der Schaar. Policy analysis using synthetic controls in contimuous-time. arXiv:2102.01577, 2021.
Patric Bonnier, Patrick Kidger, Imanol Perez Arribas, Cristopher Salvi, and Terry Lyons. Deep signature transforms. Advances in Neural Information Processing Systems, 2019.
Bo Chang, Minmin Chen, Fldad Haber, and Ed H Chi. AntisymmetricRNN: dynamical system view on recurrent neural networks. International Conference on Learning Representations, 2019.
Zhengping Che, Sanjay Purushotham, Kyunghyun Cho, David Sontag, and Yan Liu. Recurrent noural networks for multivariate time series with missing values. Scientific reports, 8(1):1-12, 2018.
Ricky T. Q. Chen, Yulia Rubanova, Jesse Bettencourt, and David Duvenaud. Neural Ordinary Differential Equations. Advances in Neural Information Processing Systems, 2018
Ricky Q. Chen, Brandon Amos, and Maximilian Nickel. Learning Neural Event Functions for Ordinary Differential Equations. In Internotional Conference on Learning Representations, 2021a.
Ricky T. Q. Chen, Brandon Amos, and Maximilian Nickel. Neural Spatio- Pentio-Temporal Point Processes. In
International Conference on Learning Representations, 2021b.
Edward De Brouwer, Jaak Simm, Adam Arany, and Yves Moreau. GRU-ODE-Bayes: Continuous modeling of sporadically-observed time series. Advances in Neural Information Processing Systems, 2019.
11

---

Published in Transactions on Machine Learning Research (09/2022)
T Desautels, J. Calvert, J. Hoffman, M. Jay, Y. Kerem, L. Shich, D. Shimabukuro, U. Chettipally, M. D. Feldman, C. Barton, D. J. Wales, and R. Das. Prediction of sepsis in the intensive care unit with minimal electronic health record data: A machine learning approach. JMIR medical informatics, 4(3), 2016.
Ary L Goldberger, Luis AN Amaral, Leon Glass, Jeffrey M Hausdorff, Plamen Ch Ivanov, Roger Mark, Joseph Mietus, George B Moody, Chung-Kang Peng, and Eugene Stanley. Physiobank, physiotoolkit, and physionet: components of new research resource for complex physiologic signals. circulation, 101 (23): e215 e220, 2000.
Ben Hambly and Terry Lyons. Uniqueness for the signature of a path of bounded variation and the reduced path group. Annals of Mathematics, pp. 109 167, 2010.
Calypso Herrera, Florian Krach, and Josef Teichmann. Neural jump ordinary differential equations: Consistent continuous prediction and filtering. In International Conference on Learning Representations, 2021.
Junteng Jia and Austin R Benson. Neural jump stochastic differential equations. Advances in Newral Information Processing Systems, 2019.
Alistair Johnson, Lucas Bulgarelli, Tom Pollard, Steven Horng, Leo Anthony Celi, and Roger Mark. The mimic-iv clinical database. https://doi. org/10. 10. 10026/366606xx29000 2021.
Patrick Kidger. On neural differential equations. arXiv preprinl orXi::2202.02435, 2022.
Patrick Kidger and Terry Lyons. Signatory: disferentiable computations of the signature and logsignature transforms, on both CPU and GPU. In International Conference on Learning Representations, 2021. https://github.
Patrick Kidger, James Morrill, James Foster, and Terry Lyons. Neural Controlled Differential Equations for Irregular Time Series. Advances in Neural Information Processing Systems, 2020.
Patrick Kidger, Ricky T. Q. Chen, and Terry Lyons. "Hey, that's not an ODE": Faster ODE Adjoints via Seminorms. International Conference on Machine Learning, 2021a.
Patrick Kidger, James Foster, Xuechen Li, Harald Oberhauser, and Terry Lyons. Neural SDEs as Infinite- Dimensional GANs. International Conference on Machine Learning, 2021b.
Suyong Kim, Weiqi Ji, Sili Deng, and Christopher Rackauckas. Stiff Neural Ordinary Differential Equations. arXiv:2103.15341, 2021.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv:1412.690 2014.
Xuechen Li, TTin-KKan Leonard Wong, Ricky T. Q. Chen, and David Duvenaud. Scalable gradients for stochastic differential equations. In International Conference on Artificial Intelligence and Statistics, pp. 3870-3882 PMLR, 2020.
Terry J Lyons. Differential equations driven by rough signals. Revista Matemática Tberoumericana, 14(2): 215-310, 1998.
Terry Lyons, Michael Caruana, and Thierry Lévy. Differential equations driven by rough paths. Springer, 2007.
James Morrill. Machine learning applications of controlled differential equations to streamed data. PhD thesis, University of Oxford, 2022.
James Morrill, Adeline Fermanian, Patrick Kidger, and Terry Lyons. A generalised signature method for time scries. arXiv:2006.0 00873, 2020.
James Morrill, Cristopher Salvi, Patrick Kidger, James Foster, and Terry Lyons. Neural rough differential equations for long time series. International Conference Machine Learning, 2021.
12

---

Published in Transactions on Machine Learning Research (09/2022)
Alexander Norcliffe, Cristian Bodnar, Ben Day, Nikola Simidjievski, and Pietro Lió. On Second Order Behaviour in Augmented Neural ODEs. In Adeances in Neural Informalion Processing Systems, volume 33, pp. 9911-921. Curran Associates, Inc., 2020.
Alexander Norcliffe, Cristian Bodnar, Ben Day, Jacob Moss, and Pietro Liò. Neural ODE Processes. Tn International Conference on Learning Representations, 2021.
Christopher Rackauckas and Qing Nie. Differentialequations.. jl performant and feature-rich ecosystem for solving differential equations in julia. Journal of Open Research Software, 5(1), 2017.
Christopher Rackauckas, Yingbo Ma, Julius Martensen, Collin Warner, Kirill Zubov, Rohit Supekar, Dominic Skinner, Ali Ramadhan, and Alan Edelman. Universal differential equations for scientific machine learning. arXiv:2001.44385, 2020.
Yulia Rubanova, Ricky T. Q. Chen, and David Duvenaud. Latent ordinary differential equations for irregularly-sampled time series. Advances in Neural Information Processing Systems, 2019.
Nemati S, Razmi Holder A, Clifford G Stanley M, and Buchman T. An interpretable machine learning model for accurate prediction of sepsis in the icu. Critical Care Medicine., 46(4):547-53, 2018.
Mervyn Singer, Clifford S. Deutschman, Christopher Warren Seymour, Manu Shankar-Hari, Djillali Annane, Michael Bauer, Rinaldo Bellomo, Gordon R. Bernard, Jean-Daniel Chiche, Craig M. Coopersmith, Richard S. Hotchkiss, Mitchell M. Levy, John C. Marshall, Greg S. Martin, Steven M. Opal, Gordon D. Rubenfeld, Tom der Poll, Jean-Louis Vincent, and Derek C. Angus. The Third International Consensus Definitions for Sepsis and Septic Shock (Sepsis-3). JAMA, 315(8):801 810, 02 2016. ISSN 0098-7484.
Chang Wei Tan, Christoph Bergmeir, Francois Petitjean, and Geoffrey Webb. Monash University, UEA, UCR Time Series Regression Archive. arXiv:2006. 10996, 2020.
J. L. Vincent, R. Moreno, J. Takala, S. Willatts, A. De Mendonça, H. Bruining, C. K. Reinhart, P. M. Suter, and L. G. Thijs. The sofa (sepsis-related organ failure assessment) score to describe organ dysfunction/ failure. on behalf of the working group on sepsis-related problems of the european society of intensive care medicine. Intensive Care Medicine, 22:707-710, 1996.
Pete Warden. Speech commands: A dataset for limited-vocabulary speech recognition. arXiu:1804.03209, 2018.
David Williams. Probability with Martingales. Cambridge University Press, 1991.
Juntang Zhuang, Nicha C Dvornek, sekhar tatikonda, and James s Duncan. MALI: memory efficient and reverse accurate integrator for Neural ODEs. In International Conference on Learning Representations, 2021.
13

---

Published in Transactions on Machine Learning Research (09/2022)
A Neural controlled differential equations
A.1 RNNs are a special case of Neural CDEs
This is easily shown by noting that an RNN of the form
-++1 (z(
is an explicit Euler discretisation with step unit. length of
z(t) = =(o)) + S. ho (z(s), =ss) ds,
which is a special case of Neural CDE (Kidger et al., 2020, Theorem C.1).
A.2 Reparameterisation invariance of Neural CDEs
CDEs exhibit a reparamcterisation inveriance property.
Let : |a, b] + [c, d] be differentiable, increasing, and surjective, with vb(a) = c and (bbb) = d. Let T E [c, d],
let 2 = 2 o th, let X = th, and (TT). Then substituting t = (T() into a CDE (and using a standard
change of variables):
((S) x(T)
(6)
==(c)+ + S f(=(I))((())
(7)
= =x(c) + [' f(((1)) dX
di (f)d
(8)
z(b(a)) + f(=(}(T))) dx
(T)
dt (V(T)) del (r)dr dr
(9)
(2) + (T)(T)
f((=o))(T)) d(X o v) (r)dr
(10)
dr
(2) + f((=o0)())d(( (
(11)
=(c)+ + { f(((T))(8(T))
(12)
From this we can see that & satisfies the Neural CDE equation, now with X as the control.
B Theoretical conditions on the control
B.1 Boundedness
B.1.1
Boundedness is required for universal approximation
Let X' be some set of time series such that
sup w 74, min 7,, max (x,])) 00.
xex
Let X = (Xx x E X}. By equation (3), then
dx
dX
sup
|X|| +
00.
Xex
dt
dt
2In fact, they also exhibit. tree-like invariance property Hambly Lyons (2010), which slight generalisation.
14

---

Published in Transactions on Machine Learning Research (09/2022)
Let = {ax/ael X &}. Then &' is bounded in Wl.
[to, tx] and relatively compact in [tu, and x' is bounded in ['a (n] and so relatively compact in T. [lo In]. Therefore X x X' is relatively compact in [o, [n] x L [oo, tn]-
Let X = {(X, ( Then X x &' so X is also relatively compact in L [oo tnl x L L'tto tnl. This implies that relatively compact with respect to the topology generated by || + ||dX/aell: and hence also with respect to the topology generated by || + +|X| [X
This now satisfies the compactness condition of Kidger et al. (2020, Theorem B.7), implying that Neural CDEs driven by X E X are universal approximators on E X.
B.1.2 Boundedness of cubic Hermite splines with backward differences
The boundedness result for natural cubic splines is given in Kidger et al. (2020, Lemma B.9). We now give a corresponding proof for cubic Hermite splines with backward differences.
Let Xx: : [to, tx] + R be the cubic Hermite splines with backward differences such that (x,t,) z;. Note that it is sulficient to consider Xx E R here since each dimension is interpolated separately. Tet the ith piece of Xx, on the interval [t., t++1] be denoted by X, Without loss of generality, translate each piece onto the interval [0, Tı] where = 4++1 - f4 so that X;: [; + R. Let (t) = a; + bet + c,t² + d,t for some coefficients ag by, C, de and i E {0..... n 1}.
For cubie Hermite splines with backward differences we enforce X2(0) Ij. (T,T,) 2141, along with the derivative conditions
x((0) = 3i Ti
(13)
T3-1
X (T;) Titl
(14)
Ti
Letting Ar, = I; - 24-1, we find
0;
(15)
b; T Ari
-1
(16)
Q: 2T, 2, T (Ti- Arit1 - T(Da)),
(17)
d, T 3 3--1 (TAZ, T Az14))
(18)
Letting Fmax = maxx (xal.
Tuax = max Ts, Twin min it can be shown that
X loc S C1 (Fmax +
Tmax Tmax
(19)
Tiin
dxx
I C2
Emax Tmin
(20)
dt
dXxx
C3
Fmax
(21)
dt
BV
T
(22)
for fixed constants C1, C2, C3 E RR This proves the boundedness property from Equation (3) for cubic Hermite splines with backward differences.
B.2
Uniqueness
B.2.1
Interpolation schemes are non-unioe non-unique in general
Here we give simple counterexample prove non-uniqueness of the interpolation schemes only (t, x4), and CG included as information.
15

---

Published in Transactions on Machine Learning Research (09/2022)
Let X1 and X2 be two time series such that
X1 ((to. (t)) X2 ((to.z)) (tr, (o))
(23)
(24)
where to < ta < b. Clearly, as one has an additional observations, the two time series contain different information. However, all interpolation schemes given in Section 3 result in the interpolation Xx(!) = (1, 20)
for t E [to, t1]. As such, the map + Xx is not injective.
B.2.2 Observational frequencies guarantee uniqueness
First, we define the measurement intensity vector Cn.j to be a count of the number of observations of the j-th feature up until time n. Explicitly we have
-=E1p t=0
(25)
[x
This is oflen also termed the measurement intensily vector.
Suppose that we have two time series, x= =((o,, To. (6) c (In: c.) and y ((lo: yo. d..... (In: Uue x)),, where of denotes the corresponding mcasurement intensity vector. Assuming that x * then we must have that either
1. There exists some t; for which # y/-
There exists at least one i for which x measured and y is not (we take x over y here wlog).
This proof proceeds in two parts. Firstly, we show injectivity of the map x + Xx by showing the interpolations from two different sets of data must be different. Secondly, we prove uniqueness of the map Xx + Signature(X; (Xx) by showing that the schemes from Section cannot contain tree-like pieces (see Hambly & Lyons (2010)).
Let to denote the first time for which one of the two conditions above is violated. If 1 holds, then F; # y and so (xa+;) # (4) there. Similarly, if 2 holds, then will have that of * of there and again (x.):) + (4)).. This then proves that
Xx is injective with respect to X,
(26)
provided we include observational intensities. Here X' is as defined in Section 2.
To show additionally that we have injectivity of the signature, tree-like pieces. For a path to contain a tree-like piece, the path must exactly trace itself backwards which is a strong condition Hambly & Lyons (2010). This can only be true if there is a point of turnaround for all dimensions simultaneously, we will now argue why this cannot the case for any of our proposed control paths if the observational intensities e; are included.
need to argue that the path cannot contain
Recall that c E R increments by one according the channels of z; that measured at Let cay R denote the tth channel of c;. By definition, at every time at least one feature of c, is updated. Therefore we can find jn, with each ja E (1,....o} such that G.j, is incremented.
For linear and cubic Hermite controls, there is always a dimension in that is increasing. This is a sufficient condition for there to be no tree-like pieces. Similarly, for rectilincar, always at least one of the time dimonsion or & dimension increasing, and so rectilinear controls too have no tree-like pieces. This condition is unknown however for natural cubic splines.
Together, this shows that provided we include e; then all of our considered control schemes result in
+ (xo, Signature(Xx)) is injective with respect to X,
(27)
16

---

Published in Transactions on Machine Learning Research (09/2022)
holding for linear, cubic Hermite, and rectilinear controls.
Note that inclusion of G is by no means the only way to guarantee this property, however, it represents a sensible approach from which uniqueness can be achieved.
B.3 Brownian motion toy example
Each Brownian path, x[() was simulated so that is observed at times 0.5, 1}. The labels at each time point is whether zt (2) > O (in which case the label is 1) or not (in which case the label is 0). Essentially the Neural CDE should learn the Gaussian distribution. Due to the independent Gaussian increments of Brownian motion, an online method can do no better than 75%.
We have training size of 4096 and test size of 1024 and the model was trained with the Adam optimiser batches of 64 across 100 epochs. For the neural network, there 10 hidden channels, with a one hidden layer of 256 neurons.
The ODE solver used was the Dormand-Prince 5(4) ("dopri5") scheme with an absolute tolerance of 10 and a relative tolerance of 10-7. All simulation and training took place on a computer with two Quadro GP100's.
Code is available al [redacted].
C Experimental details
C.1 Brownian motion toy example
Each Brownian path, (6) was simulated so that it is observed at times 0.5, 1}. The labels at cach time point whether a;() > (in which case the label is 1) or not (in which case the label 0). Essentially the Neural CDE should learn the Gaussian distribution. Due to the independent Gaussian increments of Brownian motion, online method can do no better than 75%.
We have training size of 4096 and test size of 1024 and the model was trained with the Adam optimiser on batches of 64 across 100 epochs. For the neural network, there are 10 hidden channels, with a one hidden layer of 256 neurons.
The ODE solver used was the Dormand-Prince 5(4) ("dopri5") scheme with an absolute tolerance of 10-9 and a relative tolerance of 10-7, All simulation and training took place on computer with two Quadro GP100's.
Code is available at [redacted].
C.2 General notes for practical datasets
We begin with details common to all experiments.
Code All code is available at http://wwo..o.oooooooooeeeeerraa
Normalisation For all datasets, each channel was normalised to have zero mean and unit variance.
Data splits For all problems we randomly split the dataset into 3 (stratified by label for classification problems) with 10%/15%/1555 15% training/valiin validation/teettngg respectively.
ODE Solvers When comparing between interpolation schemes the integral in Equation (1) was solved using the Dormand-Prince 5(4) ("dopri5") scheme with an absolute tolerance of 10-5 and a relative tolerance of 10-3, For the MIMIC-IV benchmarking results (Table 9) we used the Runge-Kutta-4 ("rk4") scheme.
Optimiser
The optimiser used for every problem was Adam (Kingma & Ba (2014)).
17

---

Published in Transactions on Machine Learning Research (09/2022)
Training details All models run with batch size 1024 for up to 1000 epochs. If training loss stagnated for 15 epochs the learning rate was decreased by factor of 10. If training loss stagnated for 60 epochs then model training was terminated and the model was rolled back to the point of lowest validation loss.
Computing infrastructure All experiments were two computers. One equipped with two Quadro GP100's, the other with one NVIDIA A100-PCIB-40GB.
C.3 Hyperparameter selection
Hyperparameters for all datasets and models were found using a Bayesian optimisation with 20 trials using the Adaptive Experimentation Platform (ax, 2021) framework.
For Neural CDEs, hyperparameters were optimised separately for the the natural cubic, linear, and rectilinear schemes, with non-natural cubic inheriting the hyperparameters from linear.
We admit a slight error in missing off the learning rate optimisation for the Neural CDE models (it was fixed at 0.005), however if anything, this biases against the Neural CDE results.
The search space for all regular datasets was hidden_ dim in range [32, 256], hidden hidden, dim in range [32, 192], num num_layers in range [1, 4], and learning_raan ratt 005.
The search space was modified slightly for the MIMIC-IV tasks so as work within the memory re- quirements of the GPUs. We used hidden dim in range [32, 128], hidden hidden dim in range [32, 128], num_ layers in range [1, 4], and learning_rate rate 0. 005 The learning rate was dropped by factor of 10 for training the actual to account for observed overfit.
For the GRU variants (GRU, GRU-dt, GRU-dt-intensity, GRU-D) we used hidden dim in range [32, 512], learning_rate in range [0.0001, 0.1].
For the ODE-RNN we used hidden_diie dim in range [32, 256], hidden _idden_hidden___ dim in range [32, 196], num num_layers in range [1, 4], learning_rate in range [0.0001, 0.01].
Final values for all hyperparameters are given in Tables 10 and 11
C.4 MIMICII
The Medical Information Mart for Intensive Care (MIMIC) IV dataset comprises of de-identified patient-level eloctronic health record (EHR) data from over 50,000 patients that stayed in ICUs at the Beth Israel Deaconess Medical Center, Boston, Massachuselus between 2008 and 2019. MIMIC-IV builds on ils predecessor MIMIC- I1I. Improvements include approximate year of admission has been made available (previously obfuscated pat of de-identification) and there more granular information on intake of medications through use of new tables.
Specifically, MIMIC-IV contains information on 53,150 patients, across 69,211 hospital admissions and 76,540 ICU stays. These are stored in relational tables in SQL format. Our first stage of data processing involves merging the information and pivoting the data such that we have time se- ries data with each channel/dimension separate variable of interest. We extracted data on vi- tal signs, laboratory reading, ventilation devices. The full list of extracted features can be seen httpa/////oiiieeeooooooaceteee//
We use the MIMIC-IV dataset on three problems of interest, namely mortality prediction, length of stay (LOS) prediction and sepsis prediction.
For all tasks we removed patients whose stay length was greater than 72 hours (as the data has a very long tail in this regard), and additionally required a stay length greater than hours and measurements least 4 different times. As the problems are different, the exclusion criteria are slightly different each case. The exclusion concept diagram can be seen in Figure
18

---

Published in Transactions on Machine Learning Research (09/2022)
Dataset
Model
Interpolation
Hidden
Hidden hidden
Num layers
NCDE
Natural cubic
256
168
0.0005
BeijingPMIL
NCDE
Linear
252
120
0,0005
NCDE
Rectilinear
249
170
0.0005
NCDE
Natural cubic
180
32
0,0005
PMij
NCDE
Linear
256
196
0.0005
NCDE
Rectilinear
256
171
0.0005
NCDE
Natural cubic
238
194
0.0005
Charlraaj
NCDE
Linear
249
170
0,0005
NCDE
Rectilinear
148
89
0.0005
NCDE
Natural cubic
108
40
0.0005
NCDE
Linear
65
119
0.0005
LOS
NCDE
Rectilincar
32
126
0.0005
ODE-RNN
32
168
0,002268
NCDE
Natural cubic
41
0.0005
NCDE
Lincar
33
91
0.0005
Mortality
NCDE
Rectilinear
70
55
0.0005
ODE-RNN
32
123
0.1
NCDE
Natural cubic
0.0005
NCDE
Linear
0.0005
Sepsis
NCDE
Rectilinear
82
65
0.0005
ODE-RNN
215
98
0,001083
NCDE
Natural cubic
0.0005
SpeechCommant
NCDE
Linear
0.0005
NCDE
Rectilincar
113
124
0.0005
Table 10: Final hyperparameter values for the ODE models (Neural CDE and ODE-RNN).
Dataset
Model
Hidden dim
Learning rate
LOS
GRU
435
0.017858
LOS
GRU-dt
299
0.001359
LOS
GRU-dt-intensity
294
0.000753
LOS
GRU-D
408
0.000143
Mortality
CRU
432
0.025304
Mortality
GRU-dt
508
0.000158
Mortality
GRU--at-intensity
196
0.017743
Mortality
GRU-D
398
0.000161
Sepsis
GRU
362
0.000697
Sepsis
GRU-dt
362
0.000697
Sepsis
GRU-dt-intensity
362
0.000697
Sepsis
GRU-D
362
0.000697
Table 11: Final hyperparameter values for the GRU variants.
19

---

Published in Transactions on Machine Learning Research (09/2022)
MIMIC-IV v1.0 (n 76540 ICU stays)
Exclude patients without any vari- able of interest (Excluded ICU stay)
Extract vari-
ables of interest
(n 76539 ICU stays)
Exclude ICU stays which are less than 4 hours or longer than 72 hours
(Excluded 25031 ICU stays)
Keep only the ICU stays which are greater than hours but less than 72 in length (n 51508 ICU stays) )
Exclude patients based on task spe- cific exclusion criteria
Keep only those ICU stays which are relevant to the task that we predicting, each individual subsection for details.
Figure 3: Study flow diagram
Mortality Here we attempted to predict in hospital mortality for each patient. We required measurements at at least 4 unique times, and at least 4 hours worth of data. This resulted in 50,538 ICU stays.
LOS We predicted length of stay given the first 24 hours of data for patients who had length of stay between 24 and 72 hours. This resulted in 16,054 ICU stays.
Sepsis Historically, there have been many standards used to define sepsis from the data a way to operationalize research. One of the latest and commonly used definition is Sepsis-3 (Singer et al., 2016) which describes sepsis as "life-threatening organ dysfunction caused by a dysregulated host response to infection." For the purpose of operationalization, organ dysfunction is identified by an increase in Sequential Organ Failure Assessment (SOFA) score (Vincent et al., 1996) by 2 points or more consequent to the infection.
As the focus in (Singer et al., 2016) was not the detection of onset, the onset time was not made explicit. As result, there have been various interpretations, for example at the point of SOFA score increase, the earlier of time of suspected infection and time SOFA score increase in (Desautels et al., 2016) and (S et al., 2018) respectively. From a clinical perspective, it makes sense to define the onset of sepsis as the time when we see organ dysfunction (as indicated by SOFA score) provided that we see evidence of suspicions of infection around the same time (thorough the use of antibiotics and cultures). Therefore this is the way we have chosen to define the time of onset of sepsis, line with Desautels et al. (2016).
To determine the time of sepsis onset, we looked for a suspicion of infection and a deterioration in the SOFA score. To identify suspicion of infection, first find the times that lab cultures were taken. We also find the start of new antibiotic treatments, and require that there are two doses of antibiotics given to the patient within a 96 hour window, otherwise that time is not used in suspicion of infection. If an antibiotic treatment has commenced within 24 hours before or 72 hours after taking a culture sample, then the earlier of the two times defines a time of suspected infection. We then look at a patient's SOFA scores and see there an increase within 24 hours before or 48 hours after the time of suspected infection. We note that the choice of these time windows are aligned with Singer et al. (2016) but other choices are used in the literature. The time of sepsis onset is then given by the earlier time, i.e. min(time of suspected infection, time of SOFA deterioration).
20

---

Published in Transactions on Machine Learning Research (09/2022)
Control
AUC
NFEsx 10
Natural cubic
0.779 + 0.003
16.2 + 0.1
Linear
0.784 + 0.002
27.8 + 0.3
Cubic Hermite
0.782 + 0.002
20.2 + 0.4
Rectilinear
0.772 + 0.004
136.7 + 4.5
Table 12:
Comparison the control signals on the sepsis dataset. This complements Tables and and is included in the appendix for completeness since it does not alter the narrative of the paper.
The older, more obsolcte EHR data from the CarcVue system (2003-2008) has been removed in MIMIC-IV. This has meant that more patients can be included in the sepsis analysis as previously the CareVue data lacked the necessary antibiotic information required for the Sepsis-3 definition.
As well using a cut off 72 hours, also removed paticnts who prescribed antibiotics before their ICU admission time and those developed sepsis within 4 hours of their ICU stay. This resulted in 45,218 ICU stays.
C.5 Additional results
We give the interpolation results for the sepsis task in Appendix C.4. This is left out of the main paper only for reasons of space, but we see that it tells much the same story as the Mortality and LOS datascts. One note is that the rectilinear interpolation batch size had to be reduced from 1024 to 512 as it would not fit within GPU memory.
21


Reviews:
[{"review_id": "rcAAGkpBJy", "reviewer": "Reviewer_TQQf", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Overall, this paper is well written and the presentation of the results are generally clear. The subject matter is important to practictioners who wish to develop a continuous-in-time prediction model for time-series analysis tasks that cannot look at future observational values. This is a common scenario. The results are well substantiated with different experimental settings. In particular, the paper gives some guidelines on how to choose the appropriate interpolation scheme depending on the requirements of the problem. \n\nThe main shortcoming is the lack of theoretical understanding of why different interpolations give different performance, but given the variety of factors that may come into play (choice of integrator, properties of dataset, choice of learning/optimization algorithm, choice of neural network architectures etc.), this is probably a difficult task. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper considers interpolation schemes for neural CDEs, defined as a sequence regression model where the hidden state is a dynamical system of the form $z(t) = z(0) + \\int f_\\theta (z(s)) dX_x(s)$. The last term being a suitable interpolation of the observed discrete time-series data $x$. This paper studies the choice of interpolation schemes that forms $X_x$. A primary issue under consideration is the application to online prediction, where continuous predictions need to be generated without looking at future discrete observation points. The authors defines these requirements as 1) continuous online (the entire prediction of $z(t)$ only depends on observations of $t_i < t$) or the weaker 2) discretely online (the prediction of $z(t_j)$ only depends on observations $t_i < t_j$). The authors show that the original choice of natural cubic splines do not satisfy these requirements. Thus, the authors propose two other types of interpolation schemes: 1) cubic hermite splines with backward differences (splines coefficients only depend on the neighboring knots, and additional degrees of freedom fixed by matching the slopes of linear interpolation) and 2) rectilinear interpolation (piece-wise constant interpolation). They demonstrate that these choices indeed satisfies the discretely (resp. continuously) online properties, and have favorable performance with respect to baselines such as natural cubic splines interpolation, as well as non-CDE benchmarks. ", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "Below, I have some questions for the authors. If they are addressed adequately, I am happy to recommend acceptance.\n\n1. Eq 1. Would it not be more precise to write the integral as $\\int f_{\\theta_1} dX_x(s)$ instead of including the derivative $dX_x(s)/ds$. The interpolation schemes considered subsequently in this paper include non-differentiable (pw linear) and discontinuous ones (rectilinear).\n2. Integration time: it is mentioned several times that \"integration time\" varies when different interpolation schemes are used. Can the authors make this statement precise, since different integrators in general will have different computational time, and their dependence on properties such as smoothness/gradient norms are all different. For example, the computation time of RK4 with or without adaptive step size selection will vary differently with respect to changing signal smoothness. Moreover, since the model has to be trained, wouldn't the training time (affected by efficiency of gradient evalutions with respect to $\\theta_1,\\theta_2$) also play a role?\n3. Observational frequencies $c_i$ definition \"counts the number of times the channels in $x_i$ have been observed up to $t_i$.\" By channels, do the authors mean the components of the vector $x_i$? By \"number of times\" do the authors mean the number of $i$ for which $x_{i,j}$ (jth component) is not $*$? I suggest the authors be a little more precise with the definition (say with a formula), since it is an important object in this study\n4. Rectilinear control: how is the proposed interpolation scheme different from a simple piece-wise constant (and left continuous) interpolation (with missing data forward-filled)? Why is there a need to do a linear interpolation holding $t$ constant? It does not appear to affect the integrator how you interpolate in this region. The authors may consider adding more explanation here.\n5. Natural extrapolation schemes: while only the rectilinear interpolation is the continuously online choice considered in this paper, one may wonder if a simple natural extrapolation scheme (say matching up to a number of derivatives) may produce different results. This would also be continuously online. Have the authors tested these types of schemes? I suppose they also have the discountinuity issues, and hence slow down integration?\n6. Table 6 and Tables 7-8 (NFE) column did not boldface the best result. Is this intentional?\n", "broader_impact_concerns": "None identified.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "Y3Qr9dA7XR", "reviewer": "Reviewer_cpiv", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength.\n\nThe inability to process online streams of data is an important limitation for Neural CDE and the motivation for this work is clear. I appreciate the rigor in defining the criteria that the interpolation strategies should statisfy. The experimental section is solid, with various data sets and tasks and emphasizing the computational speed trade-offs. The paper is also well written.\n\nWeaknesses.\n\nI understand that natural splines would get better performance to the leakage of future information in the stochastic process experiment. However, I don't understand why this effect is not present in the other experiments. Indeed, in the other tasks, Neural splines consistently underperforms the online interpolation strategies, which is counter-intuitive. I would recommend to expand on this in the main text or explaining the specificities of the experimental setup that leads to this effect. \n\nTo me, there is little motivation for the continuous online setup in irregular classification or regression exeperiments. Indeed, in between observations, you don't really care about the part of the time series  between now and the last observation because there is just not information there. The only case where I would imagine a difference would be in forecasting (which you don't address here). I think this is worthwhile discussing this in the paper as well, as it puts the relevance of the rectilinear approach into perspective.\n\nIt seems that in your experiments, you are computing the hermite splines a priori (which makes sense). However, in a real continuous setup, we would have to compute this on the fly. Could you maybe elaborate on the extra computational complexity that this entails, if any ?\n\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper investigates different strategies for allowing online sequential processing of data with neural CDEs. The authors first show why the classical Neural CDE implementation fails to deal with online streams of data due to the global temporal dependencies of cubic splines. They then go on with providing desirable properties for an online interpolation strategy. \n\nThe authors then propose 3 strategies : linear, cubic hermite and rectilinear. \n\nThey evaluate the strategies on regression and classification tasks both from synthetic and real-world data.", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "I've described changes i'd like to see above but in summary : \n\n- Elucidating why the natural splines performs worse than the continuous interpolation methods.\n- Being more explicit about the relevance of a continuous online setup for classification and regression.\n- Impact of interpolation strategy from a computational perspective.\n\nWhat is more, in Figure 1, I'm also not convinced that future values of the control $X$ will impact the current value of $z(t)$ as the ODE is by definition causal. I'd like the author to either explain me why this is the case or to adapt the figure accordingly.  ", "broader_impact_concerns": "I have no broader impact concern. ", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "VNR7cEJYiL", "reviewer": "Reviewer_T4bA", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "# Strengths \n\n- The paper addresses an important problem that would be of interest to anyone working with time series: making Neural CDE models applicable to online prediction tasks. \n- The four theoretical properties for control paths that the paper introduces offer a nice framework for anchoring the discussion in the entire paper. Furthermore, the paper does a good job of justifying these properties theoretically, which could also be helpful for discussing and comparing Neural CDE models in the future.\n- The paper proposes two new simple and effective interpolation schemes that satisfy the discretely online and, respectively, continuously online properties. While the cubic Hermite splines with BD are discretely online, unlike linear control, it is smooth, which leads to NFE improvements. At the same time, rectilinear control is the first interpolation scheme for Neural CDEs that is fully online (i.e. continuously online). \n- The paper contains many empirical experiments showing the benefits of the proposed schemes in terms of the prediction error and the number of function evaluations compared to other interpolation schemes. \n- Figures one and two are very good at explaining the intuition behind the introduced concepts. \n- The paper is well written and strikes the right balance between mathematical rigour and readability.\n\n#  Weaknesses\n\n- The presentation of the rectilinear control method can be significantly improved to better explain the intuition behind what is going on. This is partially achieved via the last subfigure of Figure 2 which uses the notion of \"virtual time\". I think this is quite a nice way to understand what the method is doing, however, what is meant by \"virtual time\" is not explained textually anywhere and the term appears only in the figure.  \n- The paper does not explain how the dataset is split more precisely. It is only mentioned that \"70%/15%/15%\" splits are performed. Are these random splits for the points in each time series in the dataset or is the splitting performed in a particular way? I also checked the appendix, but I could not find more details there either.\n- Something that could have been useful to see is time-series extrapolation experiments (even synthetic ones). I think the newly proposed interpolation schemes are more suitable for time-series extrapolation than existent NeuralCDEs, which can only make predictions up to the time of the maximum available time-point. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "- The paper analyses the choice of the interpolation schemes that power Neural Controlled Differential Equations (NCDEs). To that end, the paper proposes four theoretical properties that control paths should ideally satisfy: adapted measurability, smoothness, boundedness, and uniqueness.\n- Of these four, the paper is particularly focused on the first property, which (informally) requires that the NCDE model is able to make online predictions. Existent interpolation schemes (e.g. cubic splines) are known to violate this property and, therefore, existent Neural CDEs cannot be used for online prediction tasks. This can also be interpreted as existent NCDEs being implicitly non-causal (outputs at time $t$ rely on the value of the time-series at a later time $t' > t$). \n- The paper proposes two new interpolation schemes, cubic Hermite splines with backward difference and rectilinear control, which satisfy the four properties (to various degrees). Most importantly, these schemes can be used for different types of online tasks. \n- The experimental analysis shows improved performance on medical time-series data and other benchmarks. \n- Finally, the paper proposes a set of guidelines for practitioners explaining which interpolation scheme should be chosen for a given task. ", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "All the points in the weaknesses section above are easily addressable and I believe would strengthen the paper. They are not however critical for acceptance. ", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: All reviewers appreciated the importance of the topic to practitioners, as well as the experiments and guidelines discussed. The reviewers noted several areas that needed clarification or further discussion.  After the response and discussion, all reviewers were satisfied and recommended acceptance. The authors should prepare a revision of the paper according to the review and response.

Decision: Accept (poster)