Stable Neural Stochastic Differential Equations in Analyzing Irregular Time Series Data
Authors: YongKyung Oh
Dongyoung Lim
Sungil Kim
Venue: ICLR 2024 spotlight (2024)
Keywords: Neural Ordinary Differential Equations
Neural Stochastic Differential Equations
Irregular time series data
Primary area: general machine learning (i.e., none of the above)
TL;DR: Stable Neural Stochastic Differential Equations

Abstract: Irregular sampling intervals and missing values in real-world time series data present challenges for conventional methods that assume consistent intervals and complete data. Neural Ordinary Differential Equations (Neural ODEs) offer an alternative approach, utilizing neural networks combined with ODE solvers to learn continuous latent representations through parameterized vector fields. Neural Stochastic Differential Equations (Neural SDEs) extend Neural ODEs by incorporating a diffusion term, although this addition is not trivial, particularly when addressing irregular intervals and missing values. Consequently, careful design of drift and diffusion functions is crucial for maintaining stability and enhancing performance, while incautious choices can result in adverse properties such as the absence of strong solutions, stochastic destabilization, or unstable Euler discretizations, significantly affecting Neural SDEs' performance. In this study, we propose three stable classes of Neural SDEs: Langevin-type SDE, Linear Noise SDE, and Geometric SDE. Then, we rigorously demonstrate their robustness in maintaining excellent performance under distribution shift, while effectively preventing overfitting. To assess the effectiveness of our approach, we conduct extensive experiments on four benchmark datasets for interpolation, forecasting, and classification tasks, and analyze the robustness of our methods with 30 public datasets under different missing rates. Our results demonstrate the efficacy of the proposed method in handling real-world irregular time series data.

Published as a conference paper at ICLR 2024
STABLE NEURAL STOCHASTIC DIFFERENTIAL EQUA- TIONS IN ANALYZING IRREGULAR TIME SERIES DATA
YongKyung Oh; Dong-Young Lim; & Sungil Kim
Ulsan National Institute of Science and Technology, Republic of Korea (yongkyungoh, dl d1im, sungil 1 kim) @unist. ac. kr
ABSTRACT
Irregular sampling intervals and missing values in real-world time series data present challenges for conventional methods that assume consistent intervals and complete data. Neural Ordinary Differential Equations (Neural ODEs) offer an alternative approach, utilizing neural networks combined with ODE solvers learn continuous latent representations through parameterized vector fields. Neural Stochastic Differential Equations (Neural SDEs) extend Neural ODEs by incor- porating diffusion term, although this addition is not trivial, particularly when addressing irregular intervals and missing values. Consequently, careful design of drift and diffusion functions is crucial for maintaining stability and enhancing performance, while incautious choices can result adverse properties such as the absence of strong solutions, stochastic destabilization, or unstable Fuler discretiza- tions, significantly affecting Neural SDEs' performance. In this study, we propose three stable classes of Neural SDEs: Langevin-type SDE, Linear Noise SDE, and Geometric SDE. Then, we rigorously demonstrate their robustness maintaining excellent performance under distribution shift, while effectively preventing overfit- ting. To assess the effectiveness of our approach, we conduct extensive experiments four benchmark datasets for interpolation, forecasting, and classification tasks, and analyze the robustness of our methods with 30 public datasets under different missing rates. Our results demonstrate the efficacy of the proposed method in handling real-world irregular time series data.
INTRODUCTION
Conventional deep learning models, such as Recurrent Neural Network (RNN) (Rumelhart et al. 1986
Medsker & Jain 1999). Long Short-term Memory (LSTM) (Hochreiter Schmidhuber 1997) and
Gated Recurrent Unit (GRU) (Chung al. 2014). consider time series data as consecutive discrete subsets, struggling with irreuularly-smpple or partially-observe data (Mozer et al. 2017). To better reflect the underlying continuous process of the time series, researchers have proposed Neural Differential Equation (NDE)-based methods that enable deep learning models to learn continuous-
time dynamics and the underlying temporal structure (Chen et al. 2018 Tzen & Raginsky 2019 Jia
& Benson 2019 Liu et al. 2019 Kidger et al.] 2020 2021
Ansari et al. 2023).
Unlike discrete representations from the conventional methods, neural ordinary differential equations (Neural ODEs) (Chen et al. 2018) directly learn continuous latent representation (or latent state) based on a vector field parameterized by a neural network. Kidger et al. (2020) introduced neural controlled differential equations (Neural CDEs), which are continuous-tin time analogs of RNNs that employ controlled paths to represent irregular time series.
As extension of Neural ODEs, neural stochastic differential equations (Neural SDEs) have been introduced with a focus on aspects such as gradient computation, variational inference for latent spaces, and uncertainty quantification. For example. Tzen & Raginsky (2019) considers variation inference for the diffusion limit in deep latent Gaussian models. Li et al. (2020) develops reverse-mode automatic differentiation for solutions of SDEs. Neural SDEs that incorporate various regularization
'These two authors are equal contributors to this work and designated as co- co-first authors.
1 "oorrsponniig Author

---

Published as a conference paper at ICLR 2024
mechanisms such as dropout and noise injection were introduced in Liu al. (2019). Kong et al. (2020) utilized a framework of Neural SDEs to quantify uncertainties in deep neural networks.
Despite advancements in the literature, there still exist unresolved issues and lack of understanding in the modeling and robustness of Neural SDEs under distribution shift, motivating us to extend Neural SDE approaches into two directions. Firstly, illustrated in motivating example, the naïve implementation
Neural SDEs often fails to train. This due to the fact that the behavior of the solution SDEs can vary significantly depending on drift and diffusion functions, emphasizing the need for a study on an optimal selection of drift and diffusion functions. Secondly, irregular time series data, due to the nature of time variables, irregularity, and missingness, is prone to experiencing distribution shifts, which requires a substantial demand on model robustness (Li et al. 2021 Zhou
et al. 2023).
The performance of deep learning models often deteriorates when applied to data distributions that are different from the original training distributions (Ovadia al., (019). This vulnerability under distribution shif has been widely observed across various domains including healthcare, computer vision, and NLP (Leek et al.J 2010 Bandi et al. 2018 Lazaridou et al., 2021; Miller et al., 2020 202Tb. Therefore, theoretical guarantees to maintain robustness against distribution shift is a critical concern within modern deep learning communities. However, to the best of our knowledge, it has yet
been explored to address this specific issue in the context of Neural SDEs.
To address these research gaps, this study introduces three classes of Neural SDEs to capture complex dynamics and improve robustness under distribution shift in time series data. While the drift and diffusion terms in the existing Neural SDEs are directly approximated by neural networks, the proposed Neural SDEs are trained based on theoretically well-defined SDEs. As result, the new framework achieves state-of-the-arr results in extensive experiments.
The main contributions of this study are summarized as follows. Firstly, present three stable classes of Neural SDEs based on Langevin-type SDE, Linear Noise SDE, and Geometric SDE, and then combine the concept of controlled paths into the drift term of the proposed Neural SDEs to effectively capture sequential observations. Secondly, we show the existence and uniqueness of the solutions of these SDEs. In particular, we show that the Neural Geometric SDE shares properties with deep ReLU networks. Thirdly, we theoretically demonstrate that our proposed Neural SDEs maintain excellent performance under distribution shift due to missing data, while effectively preventing overfitting. Lastly, we conduct extensive experiments on four benchmark datasets for interpolation, forecasting, and classification tasks, and analyze the robustness of our methods with 30 public datasets under different missing rates. The proposed Neural SDEs show not only powerful empirical performance in terms of various measures but also remain robust to missing data.
RELATED WORK & PRELIMINARIES
Notations. Let (S,F,P) be a probability space. Fix integer d > 1. For an -aaluee random variable X, its law is denoted by L(X). |- is the Euclidean norm where the dimension of the space may vary depending on the context. We denote by (P() the set of X with 4/ < co. Let P(RR)) denote the set of probability measures on ((Rd). For two probability measures μ. v E P(R let C(,,v) v) denote the set of probability measures II on ((R22) such that its respective marginals are is defined as
u and v. Then, the Wasserstein distance of order
(Ee
Jee |x "o-''""((dr,,d") 1/p
Wp(u,D) inf nEC(HN))
2.1 NEURAL DIFFERENTIAL EQUATION METHODS
Neural ODEs. Let & E dd denote the input data where d, is its dimension. Consider latent representation =(1) E dR at time I, which is given by
(t) = z(0) + / f(s, 0y): with =0) h(x:; 0x),
This phenomenon occurs when the statistical properties of the target distribution that the model aims to predict shift over time, leading to discrepancies hetween the distributions of training data and test data.
2

---

Published as a conference paper at ICLR 2024
where h Rd Rd is affine function with parameter On. In Neural ODEs, f(t, (t); By)
is a neural network parameterized by O to approximate di Equation I Neural ODEs rely ODE solvers, such the explicit Euler method (Chen al. 2018). Since we can freely choose the upper limit of the integration, we can predict 2 at any time That is, once l(;; Op) and ((:);0)) have been learned, then we able compute =(1) for any t 1.2 O. Note that the extracted features x11) are used for various tasks such as classification and regression. However, the solution to Neural ODE is determined by its initial condition, making it inadequate for incorporating incoming information into differential equation.
dx(1) To solve the integral problem in
To address this issue, Kidger et al. (2020) proposed Neural CDEs as a continuous analogue of RNNs by combining controlled path X(1) of the underlying time-series data with Neural ODEs. Specifically, given the sequential data æ = (2o. Ti,..., Tn). (t) is determined by
Neural CDEs.
<() = =0)) + / [(s, ((s))(()) with z0) h(ro: 0n);
where the integral is the Riemann-Stieltjes integral and (() is chosen natural cubic spline
path (Kidger al. 2020) or hermite cubic splines with backward differences (Morrill et al. [022 of the underlying time-series data. Differently from Neural ODEs, / (1,, =(); Oy)) called CDE function,
a neural network parameterized by Of to approximate
(x dæ(() The integral problem Equation ?
can be solved by using existing ODE solvers since dz(() = [(,, = (();0,)
dx de
dt
Neural SDEs. Neural SDEs are an extension of Neural ODEs, which allows for describing the
stochastic evolution of sample paths, rather than the deterministic evolution (Han al.] 2017 Tzen & Raginsky 2019, Jia & Benson 2019 Liu et al. 2019, Kidger et al.) 202 (0 Park et al. 202 The
latent representation xt) of Neural SDEs is governed by the following SDE:
=(1) = =())= /'' [(s,2(0):0))) + S' g'os, =s): with z(0) = (r;; On),
(3)
where (W:) [W.0 is a d: -d-dieeniional Brownian motion, 0g) ) the drift function, g(s ; 0g) is the diffusion function and the second integral on the right-hand side is the Itô integral. Neural SDEs assume that drift and diffusion functions are represented by neural networks.
2.2 LIMITATIONS OF NAIVE NEURAL SDES
1.6
The naive implementation of Neural SDEs with- out sufficient consideration can lead to adverse results such as the absence of unique strong solution (Mao 2007 Øksendal & Øksendal 2003), stochastic destabilization (Mao 1994: Appleby et al.] 2008), and/or unstable Fuler dis- cretization (Hutzenthaler et al.) 2011 Roberts & Tweedie, 1996). Therefore, careful design of the diffusion term is essential to improve the stability and efficacy of Neural SDEs.
41) 20033
(11 ztrs
1.2
(II) de
(Iv) dylt)
0.8
(v) auft)a(t)
(vi) gult, z(8): 6,)
20
40
60
80
100
Epoch
Figure 1: Comparison of test losses for Neural SDEs with six different diffusion functions on the 'BasicMotions' dataset at 50% missing rate
The significance of a well-defined diffusion term becomes evident in the following illustrative example. Figure [ displays a comparison of test losses for Neural SDEs employing six distinet diffusion functions gtt, (t);: 0g)): (i) /z(t) (Hölder continuity with the exponent 1/2), (ii) [=())] (non-Lipschitz continuity), (iii) ae (constant), (iv) time-dependent function (1; 0₂) (additive noise), (v) time-dependent function a (1; 0, (z()) (multiplicative noise), and (vi) neural network go (1); 0g) (naīve Neural SDE). As depicted in the figure, the performance of Neural SDEs can vary significantly depending on the choice of diffusion functions.
particular, we observe that the non-Lipschitz continuous diffusion function (z) and the naïve diffusion function are inadequate for stabilizing Neural SDEs. However, surprisingly, the influence of drift function and diffusion function design on the performance of Neural SDEs has been underexplored and remains poorly understood.
"The numerical approximation of Fuler discretization for SDEs may explode in finite time.
3

---

Published as a conference paper at ICLR 2024
3 METHODOLOGY
In this section, our objective is to explore the characteristics of well-designed Neural SDEs. To achieve this, we propose three distinct classes Neural SDEs, each with unique drift and diffusion functions, and conduct comprehensive investigation of their theoretical properties. These three classes of Neural SDEs include Neural Langevin-type SDE (LSDE), Neural Linear Noise SDE (LNSDE), and Neural Geometric SDE (GSDE).
3.1 THE PROPOSED NEURAL SDES
Langevin-type SDE. Neural LSDE, motivated by a class of Langevin SDEs, is defined by
dz(()= = (z(1);0,)dt + ol1;0,)dW(I) with z() h(x;; 0,);
(4)
where the initial condition h-:: 0x,) an affine function with parameter 0,., the drift function 7(: 0x) neural network with parameter 0, and the diffusion function a(; 0,) is neural network with parameter Og. Note that the drift function of Neural LSDE is not explicitly dependent on time. While the Langevin SDE is popular tool in recent years for Markov Chain Monte Carlo (MCMC) and stochastic optimization, it has been never explored in the context of Neural SDEs.
Assume that there exists U such that (z) = ((; 0.)) for all æ E Fd, and 4oo (t: 0,) do for some positive constant do. Then, it is well known that, under mild conditions the Langevin SDE in Equation
44 admits a unique invariant measure т. which is indeed the Gibbs measure
T.-S. Chiang & Sheu 1987 Raginsky et al. 2017). The existence a unique invariant measure of the Langevin SDEs may be suitable for illustrating the phenomenon where the distributions of latent state values become invariant as the depth of the layers increases.
(2) X exp
20(Z)
Linear Noise SDE. Neural LNSDE governed by the following SDE with linear multiplicative noise:
dz(() = >(, (1; + (((:))))((()WW()) with =(0) h((:;; 0n);
(5)
where the initial condition ln;: 0x) an affine function with parameter 02. the drift function 7(, 0,,) is neural network with parameter 0,, and the diffusion function ø (-: 0,) is neural network with parameter 9g. The form of Neural LŇSDE was also proposed in Liu et al. (2019), which merely reiterates the well- results for stability such exponential stability. In contrast, work provides a novel theoretical analysis on the robustness of the trained model under drift shift, a crucial topic in recent deep learning research.
Geometric SDE.
Neural GSDE considers the following SDE:
dz(() (t)
= (f), + (t; (.) with x(0) = h(x; 0n),
(6)
where the initial condition h::; 0%.) is an affine function with parameter 0,. the drift function 7(, + 0,,) a neural network with parameter 0g and the diffusion function (:: 0,) ) is neural network with parameter 0g. Due to its exponential form, Neural GSDE has distinct characteristics associated with deep ReLU networks such as a unique positive solution and an absorbing state of O, which are not observed in Neural LSDE and Neural LNSDE (see Proposition 3.4 (ii)).
3.2 PROPERTIES OF THE PROPOSED NEURAL SDES
We impose some assumptions on neural networks used in Neural SDEs as follows.
Assumption 3.1. For any neural network s(t, ar; 0.) R4 Rd - Rd with parameter 0,, there exists positive constant L, > 0 such that for all t 2 0 and x, x' E R4
stt, 0.) stt, x 0.)l < L. x x'|.
For example, he nonconvex and have superlincar gradients Raginsky al. 2017 Lim al. 2023a
4

---

Published as a conference paper at ICLR 2024
Neural networks with activation functions such tanh, ReLU, and sigmoie iimoidd functions generally satisfy the Lipschitz continuity condition in Assumption 3.1 (Chen et al. 2018 Liu et al. 2019 Virmaux & Scaman 2018 Fazlyab et al. 2019 Latorre et al. 2020).
Assumption 3.2. For any neural network st, z; 0.) R x R - Rd with parameter 6g where the last layer is the ReLU linear function, there exists positive constant K, > 0 such that
(tt, xz; 0.)l S (,(1 + [=]),, for all / 0, Rdi
On the other hand, when tanh function or sigmoid function is applied at the last layer, we have [s(t, xz; 0.)l I K,, for all t 2 0, x E IR...
(7)
Assumption 3.2 implies that neural networks satisfy the linear growth condition. Assumptions
and 3.2 are commonly imposed in the literature on NDE-based methods to ensure the existence and
uniqueness of solutions for SDEs (Chen et al. 2018 Liu et al., 2019 Kong et al. 2020).
Assumption 3.3. For any neural network st,, 0.) IR+ x IRd, - Rd with parameter 0,, there
exist positive constants m, b such that for all 0, x Rd4
(s(t, (,,,0;0)) 0.), x) S -m|x|" +
The condition in Assumption 3.3 is typical assumption necessary for ensuring the uniform bounded- ness of the solutions to SDEs and diffusion approximation (Mattingly et al., 2002 Raginsky et al., 2017,Xu et al. 2018 Zhang al. 2017). We emphasize that do not impose convexity the drift and diffusion functions.
Under Assumptions 3.1
show that SDEs in Equations strong solutions. In particular, the geometric SDE has interesting properties that are suitable for modeling deep ReLU networks.
3.2
and 66 have their unique
Proposition 3.4. Let Assumptions 3.7 and 3.2 hold. Then, we have
(i) The SDEs in Equation 4 and Equation ( have their unique strong solutions.
(ii) Assume that the activation function in r is either a tanh si sigmois i function. Then, the SDE in Equation (6) has a unique strong solution ztt), which is nonnegative almost surely, i.e., P(z(t) 2 0 for all t 0) 1 a.s. Furthermore, state 0 is absorbing state.
The proof for Proposition 3.4 can be found Appendix A Intuitively, Proposition 3.4 (ii) implies that the latent representation 2 (t) of the Neural GSDE has always nonnegative values. Furthermore, once the one values of the latent representation reaches 0, it remains there forever. This property is consistent with that of ReLU networks which have positive values for activated neurons and zeros for deactivated neurons. In particular, the fact that deactivated neurons never turn to activated neurons corresponds to the property of state being absorbing state.
3.3 ROBUSTNESS UNDER DISTRIBUTION SHIFT
This subsection provides insight into why the proposed Neural SDES remain robust to input pertur- bations, such as missing data, and prevent overfitting caused by differences between training and test datasets. To obtain our main results, we heavily rely on the analysis of stochastic stability. An overview stochastic stability and relevant results are summarized in Appendix [B.]
Let æ denote the input data, and its law be represented by L(x), i.e., is valuedd random variable. Consider ø to be perturbed version of the input data, with the law L(x), such that
E 2 P.
(8)
where > 0 represents the degree of distribution shift. Suppose that we consider the task of classification regression for time series data and use feed need-forward neural network F composed of two fully connected layers as follows:
((=(T))0p), "- )()
where y, represent the predictions from the extracted feature (TT) and its perturbed version ε(T). respectilvl.. Note that (T) and æ(T) are outputs of Neural SDEs with input data and æ. respectively. For our stability analysis, we assume ott 0₂) to be either a constant og or to have a limit such that lim (1:0g) 00-
5

---

Published as a conference paper at ICLR 2024
Theorem 3.5. (Robustness of Neural LSDE) Let Assumptions 3.1 3.2 and 3.3 hold. Let
(S))
and E (Q2) denote the input data and its perturbed version satisfying Equation outputs y. of Neural LSDEs in Equation 99, we have
For the
(((C(y),C()) V3LpLage V(5+ 2|| + 21 x '1))
()(C())<C()) Lee 2/ (5 2||[ 2 +1] 1/4 vp,
where LE is the Lipschitz constant of the neural network F defined in Equation 9 Ln, is the Lipschitz constant of the initial condition h in Equation 4 and positive constants e₁ C2 are independent of T.
Theorem 3.6. (Robustness of Neural LNSDE and Neural GSDE) Let Assumptions 3.2 and 3.3
hold. Let æ E (22) and i E (2) denote the imput data and its perturbed version satisfying Equation and y and denote their outputs defined in Equation 9
(i) (Neural LNSDE) For lar|? > 2L, where Lyy is the Lipschitz constant of the neural network Y and for sufficiently large T, we have
Wi ((T)) S
(ii) (Neural GSDE) For loo12 > 2Kg where K2 is the upper bound of the neural network
defined in Equation 7 and for sufficiently large T, we have
Wi Wi ((T) < LF expe(((((off 2K, (1 P))
The proofs for Theorem 3.5 and Theorem 3.6 can be found in Appendix B.2 and B33 respectively. Theorem 3.5 and Theorem 3.6 provide non-asymptotic upper bounds for the differences between the output distributions of the original input data and its perturbed version, influenced by the degree of distribution shift p and the depth of the Neural SDEs T. Smaller perturbations and larger depths yield smaller differences. Thus, the proposed Neural SDEs do not undergo abrupt changes with respect to p, exhibiting robust performance even if the input distribution shifts. In contrast, the lack of such stability in other Neural SDEs can yield dramatically different solutions from small changes in the input data, causing dramatic performance degradation overfitting. The experimental findings presented in Section and Appendix D.4 provide empirical support for our theoretical results.
3.4 INCORPORATING A CONTROLLED PATH TO NEURAL SDES
To further improve empirical performance and effectively capture sequential observations like time- series data, we propose z(t) that incorporates controlled path in nonlinear way as follows:
x(1) = <(t,, (())
(10)
where X(t is the controlled path and Ç is neural network parameterized by 0ç. Then, replace (t) in the drift functions of the proposed Neural SDEs with æ(t) as presented in Equation (10). The effectiveness of utilizing E(1) is confirmed through ablation study in Section 4.3
Remark 3.7. When combined with (t), the proposed Neural SDEs, including Neural LSDE, Neural
LNSDE, and Neural GSDE, have their unique strong solutions. We refer to Appendix A.1 for detailed discussion and verification.
EXPERIMENTS
4.1 SUPERIOR PERFORMANCE WITH REGULAR AND IRREGULAR TIME SERIES DATA
In this section, we conducted interpolation and classification experiments and evaluated the proposed Neural SDEs using three real- sal-world datasets: PhysioNet Mortality (Silva al. 2012), PhysioNet Sepsis (Reyna et al. 2019), and Speech Commands (Warden 2018). For forecasting experiments, we utilized the MuJoCo (Tassa et al. 2018) dataset, and the results are detailed in Appendix B
Datasets.
The PhysioNet Mortality dataset contains multivariate time series data from 37 variables of Intensive Care Unit (ICU) records, with irregular measurements taken within the first 48 hours of admission. For our interpolation experiments, we all 4,000 labeled and 4,000 unlabeled instances.

---

Published as a conference paper at ICLR 2024
The PhysioNet Sepsis dataset includes 40,335 ICU patient cases and 34 demeddepnddent variables, aiming to classify each case according to the sepsis-3 criteria. This dataset represents irregular time series data, as only 10% of the values are sampled with their respective timestamps for each patient. We considered two scenarios: one with observational intensity (0I) and one without. In the scenario with OI, the observation index is attached to every value in the time series.
The Speech Commands dataset is a one-second--onn audio data recorded with 35 distinct spoken words with the background noise. To create balanced classification problem, ten labels" were selected from 34,975 time-series samples. Each preprocessed sample, utilizing Mel-frequency cepstral coefficients, has time-series length of 161 and a 20-dimensional imput size.
Experimental Protocols. We followed the recommended pipeline for interpolation experiments with the PhysioNet Mortality dataset, as outlined in In[Shukla & Marlin (202T) and its corresponding GitHub repository For classification experiments involving the PhysioNet Sepsis and Speech Commands datasets, we adhered to the experimental protocols described in Kidger et al. (2020) and its GitHub repository Our experiments were performed using server on Ubuntu 22.04 LTS, equipped with an Intel(R) Xeon(R) Gold 6242 CPU and six NVIDIA A100 40GB GPUs The source code can be accessed at httpps://ggtiieie
com/ hoam///ongyunn--oii. Stabbee-eeeeerree-eeeee SDEs
Models. We considered a range of models, including state-of- of-thee-rr art models for both interpolation and classification tasks. These are RNN-based models RNN (Rumelhart al.. 1986 Medsker
& Jain 1999). LSTM (Hochreiter & Schmidhuber] 1997). GRU (Chung et al.] 2014). GRU-AV,
and GRU-D (Che et al. (0188). Attention-based models MATAANShukle & Marlin (202T). and
MeAM et al. (2022)), Neural ODEs: Neural ODEs (Chen et al. 2018). GRU-ODE (De Brouwer
ct al. 2019), ODE-RNN (Rubanova et al. 2 ODE-LSTM (Lechner & Hasani 2020). Latent-
ODE (Rubanova et al. 2019), Augmented-On ODE (Dupont et al., 2019), and ACE-NODE (Thin
ct al. 2021])), Neural CDEs (Neural CDE Kidger et al. (2020), Neural RDE (Morrill et al. (021)).
ANCDE (Jhin et al. 2023b), EXIT (Jhin al.] (022), and LEAP (Jhin al. (023a)), and Neural
SDEs
(Neural SDEs (Tzen & Raginsky 2019), and Latent SDE Li et al. (2020)).
Results. The results of the interpolation and classification experiments summarized in the following tables. We have highlighted the best and the second best methods in the result tables.
comparess interpolation performance from 50% to 90% observed values in the test dataset. The proposed Neural SDEs consistently outperform all benchmark models, including the state-of-the- method mTAAND-Full (MTAN MTCN decoder model), at all observed time point settings.
Table 1: Interpolation performance versus percent observed time points on PhysioNet Mortality
Methods
Mean Squared Error (x10-3)
VAF
13,418 0.008
12.594 0.004
11,887 0.005
11.133 0.007
11.470 0.006
L-ODE-RNN
8.132 0.020
8.140 0.018
8.171 + 0.030
8,143 = 0.025
8.402 0.022
L-ODE-ODE
6.721 : 0.109
6816 2 0,045
6,798 : 0.143
6,850 = 0.066
7.142 : 0.066
mTAND-Full
4.139 0.029
4.018 +: 0.048
4.157 0.053
4.410 0.149
4.798 0.036
LatentSDF
8.862 + 0.036
8.864 + 0.058
8.686 0.122
8.716 = 0.032
8.435 0.077
Neural SDE
8,592 0.055
8.591 + 0.052
8.540 0.051
8,318 = 0.010
8.252 0.023
Neural LSDE
3.799 0.055
3,584 0.055
3.457 0.078
3.262 0.032
3.111 0.076
Neural LNSDE
3,808 0.078
3.617 + 0.129
3,405 # 0.089
3,269 0.057
3.154 a 0.084
GSDE
3.824 0.088
0.079
3.493 0.024
0.070
3.118
Observed %
50%
60
70%
80%
90%
Tables and 3 demonstrate that the proposed Neural SDEs consistently outperform all other methods in classification tasks. For PhysioNet Sepsis, we report AUROC rather than accuracy due to dataset imbalance. While CDE-based methods typically benefit from OI (Kidger et al.) 2020), our models perform well even without OI. This indicates that the proposed Neural SDEs leverage the advantages of incorporating a controlled path in the drift function and introducing stochasticity through the diffusion function. In Table 3
we do not report the performance of the naïve Nerual SDE due to
"They include "yes', 'no', 'down' 'down', 'left', 'right', 'on', 'off", 'stop', and 'go'.
htttps://qithub..co wwm http:://ggithhb..co///tttrrr
/Neuraleuaaa CDR
7

---

Published as a conference paper at ICLR 2024
training instability. Note that the proposed methods are less memory- compared to CDE-based methods, and we leave memory usage improvement for future research.
Table 3: Accuracy and memory usage (in MB) on Speech Commands
Table 2: AUROC and memory usage (in MB) on PhysioNet Sepsis
Methods
Test Accuracy
Memory
Test AUROC
Memory
Methods
RNY hused
RNN
0.197 0.006
1905
01
No OI
01
No Ol
LSTM
0.684 0.034
4080
GRU
0.747 0.050
4609
GRU-At
0,878 0.006
0.840 0.007
817
826
GRU A
0.453 0.313
1612
GRU-D
0.871 = 0.022
0,850 = 0.013
889
878
RM
GRU-D
0.346 0.286
1717
GRU-ODE
0.852 0.010
0.771 0.024
454
273
ODE-RNN
0,874 2 0,016
0,833 a 0.020
696
686
GRU-ODE
0.487 0.018
171
Latent-
0.787 0.011
0.495
126
ODE-RNN
0.678 0.276
1472
ACE-NODE
0,804 + 0,010
0,514 0.003
194
218
Lateni ODE
0.912 a: 0.006
2668
Neural CDE
0,880 = 0,006
0,776 0.009 0.009
244
122
Augmented
0.911 0.008
2626
ANCDE
0,900 0.002
0,823 0.003
285
129
NDE. based
ACE-NODE
0.911 + 0.003
3046
Neural CDE
0.025
175
Neural SDE
0.799 0.007
0.796
ANCDE
0.807 0.075
180
Neural LSDDE
0.909 + 0.004
0.879 + 0.008
373
436
LEAP
0.922 2 0.002
391
Neural LNSDE
0,911 0.002
0,881 0.002
341
445
Neural LSDE
0.927 + 0.004
1187
Neural GSDE
0,909 0.001
0,884 0.002
588
280
Neural LNSDE
0.923 0.001
1164
Neural GSDE
0.913 0.001
1565
4.2 ROBUSTNESS TO MISSING DATA
We evaluated the performance of the proposed Neural SDEs on 30 datasets. including 15 univariate and 15 multivariate datasets, from the University of East Anglia (UEA) and the University of California Riverside (UCR) Time Series Classification Repository (Bagnall al., 2018). We assess our models under regular (0% missing rate) and three missing rate conditions (30%, 50%, and 70%). adhering to the protocol by Kidger et al. (2020). Details on datasets and protocols along with the implementation of benchmark methods are summarized in Appendices C and (D] We compare the average classification performance of each method using average accuracy and average rank.
Table 4: Classification performance on 30 datasets with regular and three missing rates (The values within the parentheses indicate the average of 30 individual standard deviations.)
Regular datasets
Missing datasets (30%)
Missing datasets (50%)
Missing datasets (70%
Methods
Accurney
Rank
Accuracy
Rank
Accuracy
Rank
Accuracy
Rank
RNN
0.582 (0.064)
13.9
0.513 (0.087)
15.3
0.485 (0.088)
16.6
0.472 (0.072)
15.6
LSTM
0.633 (0.053)
11.2
0.595 (0.060)
12.1
0.567 (0.061)
12.9
0.558 (0.058)
12.5
GRU
0.672 (0.059)
8.3
0.621 (0.063)
10.2
0.610 (0.055)
10.3
0.597 (0.062)
10.3
GRU-AA
0.641 (0.070)
10.3
0.636 (0.066)
8.9
0.634 (0.056)
8.7
0.618 (0.065)
10.2
GRU-D
0.648 (0.071)
(0.075)
10.4
(0.073)
0.604 (0.067)
10.8
MTAN
0.648 (0.080)
12.0
0.618 (0.099)
10.7
0.618 (0.091)
10.1
0.607 (0.078)
9.8
MIAM
0.623 (0.048)
11.0
0.603 (0.066)
11.1
0.589 (0.063)
12.2
0.569 (0.056)
12.3
GRU-ODE
0.671 (0.067)
9.8
0.663 (0.064)
9.5
0.666 (0.059)
8,3
0.655 (0.062)
7.8
ODE-RNN
0.658 (0.063)
9.1
0.635 (0.064)
9.3
0.636 (0.067)
8.2
0.630 (0.055)
8.5
ODE-LSTM
0.619 (0.063)
11.4
0.584 (0.064)
12.1
0.561 (0.065)
13.3
0.530 (0.085)
12.9
Neural CDE
0.709 (0.061)
8.2
0.706 (0.073)
6.4
0.696 (0.064)
6.5
0.665 (0.072)
7.6
Neural RDE
0.607 (0.071)
(0.064)
14.9
(0.068)
0.415 (0.077)
16.3
ANCDE
0.693 (0.067)
7.8
0.687 (0.068)
7.2
0.683 (0.078)
6.9
0.655 (0.067)
7.1
EXIT
0.636 (0.073)
11.1
0.633 (0.078)
10.2
0.616 (0.075)
10.6
0,599 (0.075)
11.1
LEAP
0,444 (0.068)
15.2
0.401 (0.078)
16.3
0.425 (0.073)
14.9
0.414 (0.070)
14.7
Latent SDE
0.456 (0.073)
16.6
0.455 (0.073)
15.4
0.455 (0.069)
15.0
0.446 (0.066)
15.1
Neural SDE
0,526 (0.068)
13.4
0.508 (0.066)
13.1
0.517 (0.058)
13.2
0.512 (0.066)
12.9
Neural LSDE
0.717 (0.056)
5.6
0.690 (0.050)
6.4
0.686 (0.051)
6.1
0.682 (0.067)
5.2
Neural LNSDE
0.727 (0.047)
3,4
0.723 (0.050)
5.0
0.717 (0.054)
4.3
0.703 (0.054)
Neural GSDE
(0.065)
(0.069)
5.3
(0.063)
0.689 (0.056)
Table shows that the proposed Neural SDEs consistently achieve top-tier performance in accuracy and rank across different missing rates, maintaining stable accuracy even with increased missing rates. Conversely, traditional RNN-based methods, such as RNN, LSTM, and GRU, experience noticeable performance drops as the missing rate increases. Furthermore, the proposed methods converge more rapidly and achieve lower loss values than the naïve Neural SDE, as depicted in Figure 2 Particularly, when comparing Figure a) with [i it becomes evident that the proposed methods overcoming the stability limitations of the naive Neural SDE.
8

---

Published as a conference paper at ICLR 2024
2.0
3.0
1.5
2.0
3.5
3.5
130
20
40
63
170
150
20
40
65
80
1.0
Spech
Spoch
Epo.21
Epoc)
GRIAD
GRAD
Roure L2G5
06--0
1a0re 1204
GLAD
10.7 1200
Searel 200
Ne mage
Searei
SWire 0.005
Mirvicoe
Sw. 0500
(a) BasicMotions
(b) CharacterTrajeettrriss
(c) SpoieenArabicDggiis
(d) Trace
Figure 2: Comparing stability of test loss during model training with the four datasets at a 50%
missing rate with the selected methods (Training 100 epochs without early-stopping.)
4.3 ABLATION STUDY
We conduct ablation experiments to show the performance impact of incorporating a controlled path into the drift function and using nonlinear neural network for the diffusion function. Table 5 illustrates the effectiveness of a nonlinear neural network compared to an affine function in the diffusion function. The results reveal a fundamental performance difference between the three proposed Neural SDEs and the naïve Neural SDE, with the latter unable to bridge the gap by adopting a controlled path and a nonlinear diffusion function. The superior and consistent performance of our models across 15 univariate and 15 multivariate datasets underscores their robustness and generality. Ablation studies for the controlled path design and solver for Neural SDEs are described in Appendix D Additionally, detailed results for each individual dataset are provided in Appendix F
Table 5: Comparison of average accuracy and average cross-entropy loss from the ablation study (C indicates whether the drift function incorporates the controlled path or not. 'N' or 'L' denote the architecture of diffusion function a with an affine function or a nonlinear neural network.)
Univariate datasels (15)
Multivariate datasets (15)
All datasets (34)
Methods
c
Accuracy
Loss
Accuracy
Loss
Accuracy
Loss
N
0.615 (0.090)
0,736 (0.089)
0.760 (0.036)
0.618 (0.092)
0.688 (0.063)
0.677 (0.091)
Neural SDE
L
0.535 (0.095)
0.838 (0.094)
0.752 (0.040)
0.633 (0.087)
0.643 (0.068)
0.736 (0.090)
N
0.516 (0.084)
0.914 (0.069)
0.515 (0.045)
1.248 (0.085)
0.516 (0.064)
1,081 (0:077)
X
L
0.498 (0:087)
0.929 (0.078)
0,514 (0.050)
1.255 (0.081)
0.506 (0.068)
1.092 (0.080)
N
0.604 (0.081)
0,752 (0.084)
0.783 (0.031)
0.572 (0.080)
0.694 (0.056)
0.662 (0.082)
Neural LSDE
0.533 (0.089)
0.877 (0.086)
0.745 (0.038)
0.668 (0.085)
0.639 (0. (0.064)
0.772 (0.085)
0.530 (0.069)
0.856 (0.060)
0.527 (0.054)
1.177 (0.082)
0.528 (0.060)
1.039 (0:074)
X
0,505 (0.064)
0,912 (0.061)
0,518 (0.057)
1.210 (0.101)
0,512 (0.059)
1.083 (0.088)
0.654 (0.073)
0.701 (0.091)
0.780 (0.029)
0.577 (0.070)
0.717 (0.051)
0.639 (0.080)
Neural LNSDE
0.586 (0.087)
0.765 (0.083)
0.764 (0.044)
0.617 (0. 108)
0.675 (0.066)
0.691 (0.096)
0.532 (0.077)
0,878 (0.060)
0.502 (0.051)
1.254 (0.082)
0.517 (0.064)
1,066 (0:071)
0.528 (0.085)
0,890 (0.069)
0,510 (0.047)
1.257 (0.089)
0,519 (0.066)
1.074 (0.079)
0.633 (0.091)
0,742 (0.086)
0.772 (0.036)
0.598 (0.077)
0.703 (0.063)
0.670 (0.081)
Neural GSDE
0.572 (0.083)
0.796 (0.084)
0.748 (0.039)
0.653 (0.094)
0.660 (0.061)
0.724 (0.089)
0.531 (0.074)
0.868 (0.052)
0.510 (0.045)
1,261 (0.093)
0.520 (0.060)
1,064 (0:072)
0,525 (0.078)
0,893 (0.072)
0,509 (0.052)
1.261 (0.093)
0,517 (0.065)
1.073 (0.083)
5 CONCLUSION
In this study, we proposed three stable classes of Neural SDEs Langevin-type SDE, Linear Noise SDE, and Geometric SDE with the aim of capturing complex dynamics and improving stability in time series data. While the drift and diffusion terms in the existing Neural SDEs are directly approximated by neural networks, the proposed Neural SDEs are trained based on theoretically well-defined SDEs. We investigated the theoretical properties of the proposed Neural SDE classes, particularly their robustness under distribution shift, and corroborated their effectiveness in handling real-world irregular time series data through extensive experiments. As result, the proposed Neural SDEs achieved state-of-the-an results in a wide range of experiments. However, it is important to acknowledge that methods require more computational resources compared to Neural CDE-based models. Despite this, we found that our methods significantly enhance the stability of Neural SDE training and improve classification performance under challenging circumstances.
9

---

Published as a conference paper at ICLR 2024
ETHIC STATEMENT
We commit conducting our research with integrity, ensuring ethical practices and responsible use of technology in alignment with established academic and scientific standards.
REPRODUCIBILIJT STATEMENT
We make our code and implementation details publicly accesssbbl for reproducibility purposes. Code is available at https:: github.thuuu.g.ggggg com/ cttp://caacckyaaaaoooo///aalaala
ACKNOWLEDGEMENT
We thank the teams and individuals for their efforts in dataset preparation and curation for our research, especially the UEA & UCR repository for the 30 datasets we extensively analyzed.
This work was partly supported by the Korea Health Technology R&D Project through the Korea Health Industry Development Institute (KHIDI), funded by the Ministry of Health and Welfare, Republic of Korea (Grant number: H119C1095), the National Research Foundation of Korea (NRF) grant funded by the Korea government (MSIT)(No.R) CTTTT/NN..-22---00000 2023-00253002). the National Research wt-.oo2s--000002111110
Foundation of Korea (NRF) grant funded by the Korea government and the Institute of Information & communications Technology Planning & Evaluation (IITP) grant funded by the Korea government (MSIT) (N0.2020-0-01336, Artificial Intelligence Graduate School Program (UNIST)).
REFERENCES
Abdul Fatir Ansari, Alvin Heng, Andre Lim, and Harold Soh. Neural continuous-discrete state space models for imeeullrll-samplle time series. arXiv preprint arXiv:2301.11308 2023.
J. A. D. Appleby, X. Mao, and A. Rodkina. Stabilization and destabilization of nonlinear differential equations by noise. IEEE Transactions Automatic Control, 53:683-691, 2008.
Anthony Bagnall, Hoang Anh Dau, Jason Lines, Michael Flynn, James Large, Aaron Bostrom, Paul Southam, and Eamonn Keogh. The uea multivariate time series classification archive, 2018. arXiv preprint 1eXiv:1811.000 2018.
P. Bandi, O. Geesink, M. Manson, Q. Van Dijk, M. Balkenhol, M. Hermsen, B. E. Bejnordi, K. Lee, B. Paeng. and A. Zhong. From detection of individual metastases to classification of lymph node status the patient level: the camelyon17 challenge. IEEE transactions on medical imaging, 38(2), 2018.
N. H. Chau, F. Moulines, M. Rasonyi, S. Sotirios, and Y. Zhang. On stochastic gradient I Langevin dynamics with dependent data streams: The fully nonconvex case. SIAM Journal Mathematics of Data Science, 3()): 959-986, 2021.
Zhengping Che, Sanjay Purushotham, Kyunghyun Cho, David Sontag, and Yan Liu. Recurrent neural networks for multivariate time series with missing values. Scientific Reports, 8(1)x6085, 2018.
Ricky TQ Chen, Yulia Rubanova, Jesse Bettencourt, and David K Duvenaud. Neural ordinary differential equations, Advances neural information processing systems, 31, 2018
Edward Choi, Mohammad Taha Bahadori, Andy Schuetz, Walter F Stewart, and Jimeng Sun. Doctor ai: Predicting clinical events via recurrent neural networks. In Machine learning for healthcare conference, pp. 301-318. PMLR, 2016.
Junyoung Chung, Caglar Gulcehre, KyungHyun Cho, and Yoshua Bengio. Empirical evaluation of gated recurrent neural networks sequence modeling, arXiv preprint arXiv:1412.3555, 2014.
Edward De Brouwer, Jaak Simm, Adam Arany, and Yves Moreau. Gru-ode-bayes: Continuous modeling of sporadically-observed time series. Advances in neural information processing systems, 32, 2019.
Emilien Dupont, Arnud Doucet, and Yee Whye Teh. Augmented neural odes. Advances in Neural Information Processing Systems, 32, 2019.
Mahyar Fazlyab, Alexander Robey, Hamed Hassani, Manfred Morari, and George Pappas. Efficient and accurate estimation of lipschitz constants for deep neural networks. Advances in Neural Information Processing Systems, 32, 2019.
10

---

Published as a conference paper at ICLR 2024
Yifeng Gao and Jessica Lin. Efficient discovery of variable-length time series motifs with large length range in million scale time series, arXiv preprint arXiv: 04883, 2018.
Jiequn Han, Aruuff Jentzen, et al. Deep learning- numerical methods for high-dimensional paraholic partial differential equations and backward stochastic differential equations. Communications mathematics and statistics, 5(4):349-380, 2017.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 96)::1335-1780, 1997.
M. Hutzenthaler. A. Jentzen, and P. E. Kloeden. Strong and weak divergence in finite time of euler's method for stochastic differential equations with non-globally lipschitz continuous coefficients, Proceedings of the Royal Society A: Mathematical, Physical and Engineering Sciences, 467:1563-155 1563-1576, 2011.
Sheo Yon Jhin, Minju Jo, Taeyong Kong, Jinsung Jeon, and Noseong Park. Ace-node: Attentive co-cvolving neural ordinary differential equations. Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining, pp. 736-745, 2021.
Sheo Yon Jhin, Jaehoon Lee, Minju Jo, Seungji Kook. Jinsung Jeon, Jihyeon Hyeong, Jayoung Kim, and Noseong Park. Exit: Extrapolation and interpolation-base based neural controlled differential equations for time-series classification and forecasting. In Proceedings the ACM Web Conference 2022, pp. 3102-3112, 2022.
Sheo Yon Jhin, Minju Jo, Seungji Kook, Noseong Park, Sungpil Woo, and Sunhwan Lim. Learnable path in neural controlled differential equations. arXiv preprint arXiv:2301.0433 2023a.
Sheo Yon Jhin, Heejoo Shin, Sujie Kim, Scoyoung Hong, Minju Solhec Park, Noscong Park, Scungbeom Lee, Hwiyoung Maeng, and Seungmin Jeon. Attentive neural controlled differential equations time-series classification and forecasting. Knowledge and Information Syxtems, pp. 1-31, 2023b.
Junteng Jia and Austin R Benson. Neural jump stochastic differential equations. Advances in Neural Information Processing Systems, 32, 2019.
Eamonn Kcogh. Efficiently finding arbitrarily scaled patterns in massive time series databases. In European Conference on Principles of Data Mining and Knowledge Discovery, pp. 253 263-265. Springer, 2003.
Rafail Khasminskii. Stochastic Stability of Differential Equations. Springer, 2011.
Patrick Kidger, James Morrill, James Foster, and Teryy Lyons. Neural controlled differential equations for irregular time series. Advances Neural Information Processing Systems, 33:6696-6707, 2020.
Patrick Kidger, James Foster, Xuechen Li, and Terry J Lyons. Neural sdes infinite-dimensine dimensional gans, In International Conference on Machine Learning, pp. 5453-5463. PMLR, 2021a.
Patrick Kidger, James Foster, Xuechen Chen Li, and Terry Lyons. Efficient and accurate gradients for neural sdes. Advances Neural Information Processing Systems, 34: 34:18747-18761, 2021b.
Peter E Kloeden and Andreas Neuenkirch. The pathwise convergence of approximation schemes for stochastic differential equations. LMS journal of Computation and Mathematics, 10:235-253, 2007.
L. Kong. J. Sun, and C. Zhang. Sde-net: Equipping deep neural networks with uncertainty estimates. Interna- tional Conference Machine Learning, 37, 2020.
Fabian Latorre, Paul Rolland, and Volkan Cevher. Lipschitz constant estimation of neural networks via sparse polynomial optimization. arXiv preprint arXiv:2004. 008688004006888, 2020.
A. Lazaridou, A. Kuncoro, E. Gribovskaya, D. Agrawal, A. Liska, T. Terzi, T. Gimenez, M. Kocisky, S. Ruder, K. Yogatama, S. Cao, Young, and P. Blunsom. Mind the gap: Assessing temporal generalization in neural language models. Advances in neural information processing systems, 2021.
Mathias Lechner and Ramin Hasani. Learning long-term dependencies in tregularly-samplee time series, arXiv preprint arXiv:2006. 04488, 2020.
Yurim Lee, Eunji Jun, Jaehun Choi, and Heung-Il Suk. Multi-view integrative attention- based deep representation learning for irregular clinical iine-series data. IEEE Journal of Biomedical and Health Informatics, 2022.
J. T. Leek, R. B. Scharpf, H. C. Bravo, D. Simcha, B. Langmead. W. E. Johnson, D. Geman, K. Baggerly, and R. A. Trizarry. Tackling the widespread and critical impact of batch effects in high-throughput data. Nature Reviews Genetics, 11(10), 2010.
11

---

Published as a conference paper at ICLR 2024
A. Li, A. Boyd, P. Smyth, and S. Mandt. Detecting and adapting to irregular distribution shifts in bayesian online learning. Advances in neural information processing systems, 2021.
Xuechen Li, Ting Kam Leonard Wong, Ricky TQ Chen, David Duvenaud. Scalable gradients for stochastic differential equations. International Conference on Artificial Intelligence and Statistics, pp. 3870-3882. PMLR, 2020.
Richard Liaw, Eric Liang, Robert Nishihara, Philipp Moritz, Joseph E Gonzalez, and Ion Stoica. Tune: A research platform for distributed model selection and training. arXiv preprint 1807.05/ 2018.
Kong-Young Lim and Sotirios Sahanis. Polygonal unadjusted langevin algorithms: Creating stable and efficient adaptive algorithms for neural networks. arXiv preprint arXiv:2105.13933 2021.
Dong- Young Lim, Neufeld Ariel, Sotirios Sabanis, and Ying Zhang. Non-asympiotic estimates for tusla algorithm non-convex learning with applications to neural networks with relu activation function. IMA Journal of numerical analysis, 2023a.
Dong- Young Lim, Ariel Neufeld, Sotirios Sabanis, and Ying Zhang. Langevin dynamics based algorithm e-thæo poula for stochastic optimization problems with discontinuous stochastic gradient. arXiv preprint 1301132 2023b.
Xuanqing Liu, Tesi Xiao, Si Si, Qin Cao, Sanjiv Kumar, and Cho-Jui Hsieh. Neural sde: Stabilizing neural ode networks with stochastic noise. arXiv preprint vrXiv:/906.02335 2019.
Markus Löning, Authony Bagnall, Sajaysurya Ganesh, Viktor Kazakov, Jason Lines, and Franz Kirâly. sktime: unified interface for machine learning with time series. arXiv preprint arXiv:1909.078% 2019.
X. Mao. Stochastic stabilisation and destabilisation. Systems and Control Letters, 23:279-290, 1994.
Xuerong Mao. Stochastic differential equations and applications. Elsevier, 2007.
C. Mattingly, A. M. Stuart, and Higham. Ergodicity for sdes and approximations: locally lipschitz vector fields and degenerate noise. Stochastic Processes and Their Applications, 101(2):185 12222,2,,, 2002.
Laryy Medsker and Lakhmi C Jain. Recurrent neural networks: design and applications. CRC press, 1999.
J. Miller, K. Krauth, B. Recht, and L. Schmidt. The effect of natural distribution shift on question answering models. International Conference on Machine Learning, 2020.
J. Miller, R. Taori, A. Raghunathan, S. Sagawa, P. W. Koh, V. Shankar, and I., Schmidt. Accuracy on the line: on the strong correlation between out-of-distribution and in-distribution generalization. International Conference on Machine Learning, 2021.
Philipp Moritz, Robert Nishihara, Stephanie Wang. Alexey Tumanov, Richard Liaw, Eric Liang, Melih Elibol, Zongheng Yang. William Paul, Michael Jordan, et al. Ray: distributed framework for emerging (AI} applications. 13th (USENIX) Symposium on Operating Systems Design and Implementation (ŞOSDI) 18), pp. 561-577, 2018.
James Morrill, Cristopher Salvi, Patrick Kidger, and James Foster. Neural rough differential equations for long time series. In International Conference Machine Learning, pp. 7829-7838. PMIR M.RR 2021.
James Morrill, Patrick Kidger, Lingyi Yang, and Tenry Lyons, On the choice of interpolation scheme for neural cdes. Transactions Machine Learning Research, 2022(9), 2022.
Michael C. Mozer, Denis Kazakov, and Robert V Lindsey. Discrete event, continuous time mns. arXiv preprint arXiv:1710.04110. 2017.
Bernt Øksendal and Bernt Øksendal, Stochastic differential equations. Springer, 2003.
Yaniv Ovadia, Emily Fertig, Jie Ren, Zachary Nado. D Sculley, Sebastian Nowozin, Joshua Dillon. Balaji Lakshminarayanan, and Jasper Snoek. Can you trust your model's uncertainty? evaluating predictive uncertainty under dataset shift. Advances in neural information processing systems, 33, 2019.
Sung Woo Park, Kyungjac Loe, and Junscok Kwon. Neural markov controlled sde: Stochastic optimization for continuous-tim time data. In International Conference Learning Representations, 2021.
M. Raginsky, A. Rakhlin, and M. Telgarsky. Non-convex learning via stochastic gradient Langevin dynamies: nonasymptotic analysis. In In Conference on Learning Theory, pp. 1674-1703. PMLR, 2017.
12

---

Published as a conference paper at ICLR 2024
Matthew A Reyna, Chris Josef, Salman Seyedi, Russell Jeter, Supreeth P Shashikumar, M Brandon Westover, Ashish Sharma, Shamim Nemati, and Gari D Clifford. Early prediction of sepsis from climical data: the physionet/computing in cardiology challenge 2019. In 2019 Computing in Cardiology (CinC), pp. Page-1. IEEE, 2019.
G. O. Roberts and L. Tweedie, Exponential convergence of langevin distributions and their discrete approxi- mations. Bernoulli, 2(4))341-363 1996.
Andreas RöBler. Runge kutta methods for stratonovich stochastic differential equation systems with commutative noise. Journal of computational and applied mathematics, 164:613 627, 2004,
Yulia Rubanova, Ricky TQ Chen, and David K Duvenaud. Latent ordinary differential equations for irregularly- sampled time series. Advances in neural information processing systems, 32, 2019.
David E Rumelhart, Geoffrry E Hinton, and Ronald Williams. Learning representations by back- propppropggtiing errors. nature, (33(6088):533-555) 536, 1986.
Satya Narayan Shukla and Benjamin M Marlin, Multi-time attention networks for irregularly sampled time series. arXiv preprint arXiv:2101.10314 2021.
Ikaro Silva, George Moody, Daniel Scott, Leo A Celi, and Roger G Mark. Predicting in-hospital mortality of icu patients: The physionet/computing in cardiology challenge 2012. In 2012 Computing in Cardiology. pp. 245-248, IEEE, 2012.
C.-R. Hwang T.-S. Chiang and S.J. Sheu. Diffusion for global optimization in r", SIAM Journal on Control and Optimization, 25(3):737- 753, 1987.
Chang Wei Tan, François Petitjean, Eamonn Keogh, and Geoffrey I Webb. Time series classification for varying length series. arXiv preprint 191v:1910.044444 2019.
Yuval Tassa, Yotam Doron. Alistair Muldal, Tom Erez, Yazhe Li, Diego de Las Casas, David Budden, Abbas Abdolmaleki, Josh Merel, Andrew Lefrancq, al. Deepmind control suite. arXiv preprint 1011v:1801.00690. 2018.
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for JEEE/RSJ international conference on intelligent robots and systems. pp. 5026 5033. JEEE, 2012.
control. In 2012
Belinda Tzen and Maxim Raginsky. Neural stochastic differential equations: Deep latent gaussian models in the diffusion limit, arXiv preprint arXiv:1905. 09883, 2019.
Ashish Vaswani, Noam Shazeer, Niki Parmar. Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30. 2017.
Aladin Virmaux and Kevin Scaman. Lipschitz regularity of deep neural networks: analysis and efficient estimation. Advances Neural Information Processing Systems, 31, 2018.
Pete Warden. Speech commands: A dataset for limited-vocabulary speech recognition. arXiv preprint arXiv:1804.03200 2018.
Xu, Chen, D. Zou, and Q. Gu. Global convergence of Langevin dynamics based algorithms for nonconvex optimization. Advances in Neural Information Processing Systems, 32, 2018
Dragomir Yankov, Eamonn Keogh, Jose Medina, Bill Chiu, and Victor Zordan. Detecting time series motifs under uniform scaling, Proceedings of the 13th ACM SIGKDD international conference on Knowledge discovery and data mining, pp. 844 853, 2007.
Zhang, Liang, and Charikar. A hitting time analysis of stochastic gradient langevin dynamics. In In Conference on Learning Theory, pp. 1980-2022. PMLR, 2017.
Helen Zhou, Sivaraman Balakrishnan, and Zachary Lipton. Domain adaptation under missingness shift. International Conference on Artificial Intelligence and Statistics, 2023.
13

---

Published as a conference paper at ICLR 2024
A PROOFS FOR SECTION 3.2
It is classical result that SDE has unique strong solution when the drift and diffusion functions Lipschitz continuous and at most linearly growing as stated in the following Theorem.
Theorem A.1. (Mao 2007 Consider the following d-dimensional stochastic differential equation
dx(() X(U)dd + g(t, ((I)ddW()
where (!!) is the d-dimensional Brownian motion. Assume that f(t,2), g(t,2) are Lipschitz continuous in (t,z) and satisfy the linear growth condition for every finite subinterval [0, T), i.e. there exist constants Kr, Lr > 0 such for all t E [0,T] and x, x' Rd,
(f(t,z) - f(t,,')) +(1,2)))+9((t,z))) (t,2')1 Lrke 2/),
and
S Kr [=])-
Then, there exists unique strong (global) solution
(tttssiossrris satisfying
V. Ix())pdl < co.
I
Proof of Proposition 3.4.
(i) For the Langevin SDE and linear noise SDE. from Assumptions 3.1 3.2 the drift and diffusion coefficients are Lipschitz continuous and at most linearly growing for every finite subinterval E [0, T). Thus, they have their unique strong solutions by Theorem A.1
(ii) Recall that we consider tanh or sigmoid functions as activation functions for the GSDE. Hence, the drift and diffusion coefficients in Equation (6) satisfy the conditions stated in Theorem A. Therefore, Equation (6) has unique strong global solution.
To prove the nonnegativity of the solution, consider stochastic process (1(()) defined on the same probability space, which is governed by the following SDE:
(ty(t) = (7(t. ew(t,
(
dt a
(1I)
where diag(A) is the vector of the diagonal elements of matrix A. Since the diffusion term of Equa- tion (TT) Lipschitz continuous and is at most linearly growing, from Theorem A.I Equation 111 has a unique strong solution, so {v(t) h2o is well-defined
Let xtt) exp(y(t)), i.e. (t) exp(3i(t) (t)) for 1,2,..... where z; and y, are the i-th components of z and y. respectively. For all i = 2. dz. we note that 2; (t) is nonnegative almost surely for all 2 by construction. Also, using Ito's 'ss formula we have
(z, = 2. 2. + v. c. ((y.()))
=x(tt)x(t.=(t))0))) + 2,(t) Σ (o(t
where TV is the i-th element of y and (o(1; ))s represents the (i, jit element of (I; 0g), and (t), (t) is the quadratic variation of (t) up to for = 1,2,.... de. In the matrix form, x(t) satisfies
dz(t) (l),,(()); o1; (l: (L) (1)
Therefore, ztt) is the solution of Equation 6 Let Tj = inf(t > (t) = 0} be the first hitting time state 0 with (0) > 0 for i 1, 2. de where 2, is the i-th component of =(t). Since dz²(t) = for 2 Tí then 2t()) = for all > Tí- This implies that once (t) reaches 0, remains there forever. Therefore, is absorbing state.
14

---

Published as a conference paper at ICLR 2024
A.1 DETAILS TO REMARK 3.7
To incorporate a controlled path of the underlying time-series data, we replace =(1) in the drift
functions of the proposed Neural SDEs with =(1) defined in Equation (10) More specifically, when combined with 2(), the proposed Neural SDEs are given by
(Neural LSDE)
(t) = 71 (= (o)
(12)
(Neural LNSDE)
(z((t) (2 = ((t
(13)
dz(t)
(Neural GSDE)
z(1) 72(t, (f)), ,, + (W)
(14)
where 91 (=)); 0c) 7 (x()):0,) x(t))) (()) and (t); 0c) y (t,z(t); = (t, ((t,, for given (X and Ç is a neural network with parameter Q5.
We highlight that the proposed Neural SDEs defined in Equations 12 and have their unique strong solutions. To show this, it is enough show that 7i (: 0, 0g) and T2(:: 1 By, 0g) are Lipschitz continuous and at most linearly growing (or bounded for GSDE).
First of all, one can show that 71 (:3 8, 02) is Lipschitz continuous for 2, E Rd since
M (z;; By 02) 7i (v: 7r 0c)l = (1(((t, ((t): -
v X(); - (t):). < 1,
where Ly and Le are Lipschitz constants for the neural networks y and Ç. respectively. In addition, 51 (1 0y, 0c) is at most linearly growing since for all & E IRd.
17 (z; (,;;0,,0))), 0c)l = ly (c(=;0e); 0c);
- (1, + IS(2;; 0c)l
K, - K, (1 +1++1+++11 [x|)
S (1 Kc) [x]).
Therefore, due to Theorem A.1 Neural LSDE combined with (ff) defined in Equation (12) has unique strong solution. The proofs for Neural LNSDE and Neural can be shown the same manner.
B STOCHASTIC STABILITY AND PROOFS FOR SECTION 3.3
B. BACKGROUND STABILITY OF SDES
Stability is an important concept that describes the behavior of solution in a given differential equation with respect to changes in the initial conditions. It is well-known that stability of ODEs can be determined without solving the equation by using the Lyapunov direct method. Fortunately, turns out that the Lyapunov technique for analyzing stability of ODEs can be applied to study stability of SDEs with slight modifications. This is significantly useful since explicit solutions for SDEs cannot be obtained except special cases. In this Appendix, we provide brief overview of stochastic stability. We refer to Mao (2007); Khasminskil (201T) for more details.
Consider the d-dimensional SDE:
dX(t)) = X())dd + g(t, (()ddW())
(15)
with the initial value xo Rd We assume that Equation 15) has a unique global solution denoted by X xt; for all f 2 0 with the initial value zg = O. Furthermore, assume that
f(t = and 0)
for Then, X(1; ) = the solution of Equation 15) and is called the trivial solution. For V E (TR, x Rd). we define the infinitesimal generator G of z(1):
gV(t,z) = (t. + (a(t, x) f(t, x) + Trig(t, (tg(t, x) Van(t,
15

---

Published as a conference paper at ICLR 2024
where Ve is the gradient of V with respect to and Var is the Hessian of V with respect to
In a stochastic system driven by SDE, stability can be defined in several ways such as stability in probability, almost sure exponential stability, and moment exponential stability, For our purpose, focus on almost sure exponential stability.
Definition B.I. The trivial solution of Equation 15 is almost surely exponential stable if for all Zo E R
1 lim sup log [X (t: zo)l < 0 a.s. to0
The almost surely exponential stability implies that almost all sample paths of the solution will converge to x = 0 exponentially fast. We record a criterion (sufficient condition) for the almost surely exponential stability.
Theorem B.2. MMo 2007 Assume that there is a function V (tt,z) x), E C2,2(R4+ x )R and constants
p > 01,
E. C3 such that
(i) a|21 Vt,,
(i) GV(L,z) S ((,2))
(iil) [Va(t,z)g(tt, 2²²² (t,
for all (t,z) E R4 Rdd Then,
C3 - -222
lim sup t-
log (t: -
a.s.
2p
for all zo Rd. In particular, if 202, the given SDE is almost surely exponentially stable.
A function (tt, r) that satisfies the stability conditions of Theorem B.2 is called a Lyapunov function.
Theorem B.2 states that one can determine stability given SDEs by specifying a suitable Lyapunov function without solving the solution explicitly.
B.2 LANGEVIN SDE.
The Langevin SDE has been extensively studied in the field of stochastic optimization and MCMC algorithmss because it admits unique invariant measure, which is indeed Gibbs measure. For more detailed discussions, we refer to Raginsky et al. (017): Chau et al. (2021) 1: Lim & Sabanis (021):
Lim et al. 2023a b).
Recall that we consider the following SDE:
dz(() = (z(()) + oldW((),
(16)
where ag R, which admits the unique invariant measure
For 2 1, we define a Lyapunov function V,, by
Vx(x) = (1 /2/3)//2
(17)
for x E 12d= Let Pv, (IR)) be the set of μ E P(IR)) satisfying J.a ((d)) We then introduce
functional which shows geometric ergodicity of the Langevin SDE. For p 2 p,, Pv, define
2'll (1 -
(1, == inf neC(u.))
(z(() + (=)))) II(dr, dz')
Rd
Notice that one can easily show that W; (p,v) S W (H. v). The key analyzing stability of the Neural SDE based on the Langevin SDE is contraction property of W1.2-
Proposition B.3. (Chau et al. (2027)) Let Assumptions 3.7 3.2 and 3.3 hold. Let 2))), 21) be the solutions Equation (16) with the initial values 2(0), (0) L. respectively. Then, there exists positive constants C1, C2 such that
w/ (())()))) cie cat
for all i O.
16

---

Published as a conference paper at ICLR 2024
Now derive our main result using the contraction property stated in Proposition B.3
Proof of Theorem 3.5 Fix T > O. Recall that y. y represent the outputs of the Neural LSDE with
the inputs æ, ã. respectively. Assume that v [ क| < p. For notational convenience, we drop
the dependence on B5 in Equation 9 Due to Assumption 3.1 there exists Lp 0 such that for all I, . Rdd
F(x)) F(x') - -r|x-x'1 1'1.
Then, we observe that
(iCC() = wi WI(C(PE((T))
= =inf (Jr
nEe RRd [F(z) |1(=)) - ((3)
- LEWi (C(x(T)), (((()))
where C. is the set of probability measures such that its respective marginals are ((=(T)) and (T)(
Furthermore, one can obtain the bound for Wi (C(z ((T())) using Proposition B.3 Cauchy-
Schwartz inequality, Jensen's inequality and the definition of V2(+)
Wi ((<=(T))
w1, (L(x(TT)
s eie caf 11 (C(2())))
-
< ozl w1)((
S
S Incre of VB
I VBLACIE of /E - - - क | (2)
S V3LACIe T
PV 1
S VILace- oT
V 2B 2B + x $1))p
where L, is the Lipschitz constant of the initial condition h defined in Equation Combining the two inequalities above, it follows that
((() I o2T v (5 + 27 + 2E p.
Now focus on deriving the upper bound for (((()) Observe that (V w (CCF(
(fr
inf THE Jd F(()) [F()) F()) PI[(zz dž)
13 ((E(TT),,(E((T)))
(18)
From the definition of the Wassersten-2 distance, we write for μt, v E (RZ(RA)
W2(,v)) <
x x 2I dz')
Rd
Rd
( ( lx -
1 x' /1 dr') + Joe Jge
dx')
3d
2d
{ [ 1 of (|0r [x')) p<> ((d,, dx')
2
(|012 + | M(d, dxy)) 2
RM JRd
Ra
2 2
[1 L - Vz(2) + V2(a" (2(a'))ll(ddr, dx').
Jgd Jdd
Taking infimum over II C(),v), we have
Walu, v) S v 222.22(1.))
17

---

Published as a conference paper at ICLR 2024
Therefore, one further calculates that from Proposition B.3 Cauchy-Schward inequality and Jensen's inequality
W/ (C(x((T),
v 2 (E(2(T)),
2 2a1e of 11 (4((()))))
< 2LECIE oz' w/ (C))
< 2LACrE E (1
x (1+
< 2V3LACIE oz oz BB 2
S <23ILnCE of V5 + 2E x
2E 2
(19)
From Equation 18 and Equation 19). we derive
(i(C()) s L -22 V(5+ 2 211
B.3
LINEAR NOISE SDE AND GEOMETRIC SDE
Lemma B.4. Consider the following d-dimensional SDE:
dt f(t, + oaX (t)(W))
with the initial condition (0) E L22 Assume that I satisfies the conditions in Assumplions 3. and 3.2 Furthermore, the drift function f satisfies the condition Assumption 3.3 positive constants > 0 and b 2 0 such that
That is, there exists
(f(t, x) S -m/=/
(20)
for all x E Rdd Then,
sup ((X))) [|X(0)]+ b
do 2m
exp
120
and
do
sup [[(((())=== - V [E[XX())]+
exp
120
2m
Proof of Lemma
B.4 Let (()) = (t) for all t 2 0. Then, Itô's formula gives for 2me y (t)dt 2m (f(t, X(t) 2mt 2o (T(((I) 2(ao X())WW,
d (
(21)
which yields
(()) = 2md ( 2m) + 2m ( 2m y + 2
2 2mm X(s))ds
+ do (' (22 200 2md [ Y(s)dW..
(22)
Due to Equation 200 we have
f 2m (f(s (s)), X(s))ds -m f" 2m(s-t) [X + f' 2m(
(f(s,
b (1 _011 2mn). (23)
- 2mm(s-t)
2m
18

---

Published as a conference paper at ICLR 2024
Substituting Equation 23 into Equation (2) have
Y(t) e 2md (0)
(1 e 2mm)
d / () 200e 2mt S Y(s)da...
and then taking expectations yields
([) S 2m ( + - - 22m0) + dog f" BB (s)]ds.
Using Gronwall's inequality leads to
dop /'" _ 2 ds
B (+) 5 [O) (1 e
2mL)
exp
dog
W VI
BOY (1 2md
exp
(1 e emmm)
2m
[[Y + b
do
exp
2m
Lastly, we have
v( [EIY(0)]
do 2m
B|X(()| S VBIX(()
exp
m
Proof of Theorem 3.6 (i) Linear Noise SDE, Using the same argument in the Proof of Theorem
3.5
we have
(I(C))) = wi (.(CFF((())))CCCCEE))))
inf nee Jud -
(24)
s LpWi (C(=(T))
(25)
where C is the set of probability measures such that its respective marginals are L(z(T)) and ((²(T)).
Now, we show that the solution of Equation (5) is almost surely moment exponential stable defined in Definition B.1 Let 2 (t), Z(t) be the solutions of Equation (5) with the initial conditions =00) Z(0).
respectively. Define E(t) = x(t) - (t) for 2 O. Then, E(t) the solution of the following SDE:
ds(t) (7(t, (t(t,=(();0,)) (() 0.)) dt og (z(t) #()) (W
= =oa(t.c(t)ddd++aatt((()))))
(26)
where
(s(t, = (t, + = - (tt,
(tt,
and
(t))) = ogs(())))))))
Note that 72 (t, 0) and (t 0) for all Hence, Equation 26 has the trivial solution
e(t) = 0 when e(0) = 0 is given. Then, due to Assumptions 3. and 3.2 one can show that for all x, Rd
(ta 7a(1,2') < T., x 2'1, (t, =s(t,x') < 0g z'|
ha(t, x) $ L, |/].
(t,, a /z)
19

---

Published as a conference paper at ICLR 2024
yielding that the Equation 266 has unique strong solution from Theorem A.1 We introduce a Lyapunov function V(t, x) = |x|2 Then, we have
(V(t, = vz( 2 (t, x)) 1 ()) Ve)(t,
S (2 + |,01 (2t, + +00l
VV (ftt,))ostt,a x)) = 4/00| |x| = 4/00 4/00/ (t,,x))
Using Theorem B.2
obtain
|o -2L
sup log (tt: S t
{
a.s.
In other words, for laol? > 2L,, Equation 26 is almost surely exponential stable. Observe that
sup Bl| El|=((t < sup [ (+)) [=())]] < oo,
120
t20
due to Lemma B.4 Thus, by Egorov's ' theorem, there exists T > 0 and E F such that
21
log (=(])(())) <
2 WEE,
T
with P(E) 2 1 - 6, and
1
|0012 -22,
T log |e(TTa(0))
w E",
2
with P(F9) I o for T T Therefore, write
(())(T;=(0))]] = e( al + Blle(T;
oop"
2
I exp
([)
2
Ioo 2 2L
S exp
sup ( 120
(27)
Hence, for any 0, we choose sufficiently large T satisfying
B[le(Z);=(0)1] < - 1 +e,
(28)
which leads to
(())
(29)
Combining Equation 25 and Equation 29 we have the desired result
S Lr ex 2T,))723(11 + p).
for sufficiently large T.
(ii) Geometric SDE. We take the same argument in the of Linear Noise SDEs and adopt the
same notations. The only difference is that the drift term 7A (t, <()) for e(t) is given by
Ta (1)),((1) = (1, #(1) +(()0)))))))))00)))) + =()) - 7(, ((),
satisfying that
(ta K, 1
due to the use of the bounded activation functions. The remaining part can be proved in the same way of the case of Linear Noise SDEs.
L
20

---

Published as a conference paper at ICLR 2024
C DESCRIPTION OF DATASETS
PhysioNet Mortality. The 2012 PhysioNet Mortality dataset (Silva et al. 2012) is a compilation of multivariate time series data, encompassing 37 distinct variables derived from Intensive Care Unit (ICU) records. Each dataset entry comprises irregular and infrequent measurements taken during the initial 48-hour period post ICU admission. Adhering to the methodologies proposed by Rubanova et al. (2019). we ve adjusted the observation timestamps, rounding them to the closest minute, resulting in a potential 2880 distinct measurement instances for each time series.
For our interpolation experiments, we utilized all 8000 instances with the experiment pipeline, which suggested by (hhkka & Marlin (20TT). We base our predictions on a selected subset of data points and aim to determine values for the remaining time intervals. This interpolation is executed with an observation range that spans from 50% to 90% of the total data points. During testing, the models conditioned on the observed values and tasked with deducing the values for the remaining time intervals within the test set. The efficacy of the models is evaluated using the Mean Squared Error (MSE) with five random initializations of model parameters.
The MuJoCo dataset, which stands for Multi-Joint dynamics with Contact (oooroy al. 2012), ) employed in this study utilizes the Hopper model from the DeepMind control suite, as detailed in Tassa et al. (2018). The dataset comprises 10,000 simulations of the Hopper model, each forming a 14-dimensional time series with 100 regularly sampled data points.
MuJoCo.
In our experimental setup, the standard training and testing horizon involves analyzing the first 50 observations in a sequence to forecast the subsequent 10 observations. To simulate more complex scenarios, we introduce variability by randomly omitting 30%, 50%, and 70% of the values in each sequence, as Jhin et al. (2023b). This approach creates a range of challenging environments, particularly focusing irregular lar time-series forecasting. The MSE metric is utilized for evaluation.
PhysioNet Sepsis. The 2019 PhysioNet/ / Computing in Cardiology (CinC) challenge on Sepsis prediction serves as the foundation for time-series classification experiments (Reyna et al.. 2019). Sepsis, a tife-threatening condition triggered by bacteria or toxins in the blood, is responsible for significant number of deaths in the United States of America. The dataset used in these experiments contains 40,335 cases of patients admitted to intensive care units, with 34 dependent variables such as heart rate, oxygen saturation, and body temperature. The primary objective is to classify whether each patient has sepsis or not according to the sepsis-3 definition.
The PhysioNet dataset is an irregular time series dataset, as only 10% of the values are sampled with their respective timestamps for each patient. To address this irregularity, two types of time-series classification are performed: (i) classification using observation intensity (OI), and (ii) classification without using observation intensity (no OI). Observation intensity is measure of the degree of illness, and when incorporated, an index number is appended to each value in the time series. Due to the imbalanced nature of the data, the Area Under the Receiver Operating Characteristic curve (AUROC) score is employed to evaluate the performance.
Speech Commands. The Speech Commands dataset an extensive collection of one-second audio recordings that encompass spoken words and background noise (Warden) 2018). This dataset is comprised of 34,975 time-series samples, representing 35 distinct spoken words. To create a balanced classification problem, ten labels (including 'yes', 'no', 'up", 'down', 'left', 'right', 'on', 'off', 'stop', and 'go") were selected from the dataset. The dataset is preprocessed by calculating Mel-frequency cepstral coefficients, which are used as features to better represent the characteristics of the audio recordings and improve the performance of machine learning algorithms applied to the dataset. Each sample in the dataset has a time-series length of 161 and input size of 20 dimensions.
Robustness to missing data. We examined the performance of the proposed Neural SDEs on 30 datasets from the University of East Anglia (UEA) and the University of California Riverside (UCR) Time Series Classification Repository (Bagnall et al. 2018) using the python library sktime (Löning et al., 2019). The archive was comprised of univariate and multivariate time series datasets from various real-world applications.
http::/wwwe http://www..iiieeeeeeeescceccsca ficti com/
21

---

Published as a conference paper at ICLR 2024
Table 6: Data description for Robbsstesss missing data' experiments
Dataset
Total number of samples
Number classes
Dimension of time series
Length of time series
ArrowHead
211
251
Car
120
577
Coffee
56
286
GunPoint
150
Herring
128
2
512
Lightning2
Lightning?
143
319
Meat
120
448
OliveOil
60
4
570
Rock
70
2844
SmbstaSubspace
ToeSegmeatationl
268
2
277
TueSegmentation2
Trace
200
4
275
Wine
111
2
234
575
25
144
BasicMotions
CharacterTrajectoricc
2858
20
60 180
Cricket
180
1197
Epileppy
275
4
206
ERiin
300
6
Conenttoonnnttaation
524
1751
EyesOpenShut
98
2
14
128
FingerMovementt
Handwriting
1000
26
3
152
Japanese Vowels
640
7-26
Libras
360
NATOPS
360
6
24
RacketSports
SpokenAArabicDigiis
8798
10
13
4-93
According to T the datasets have distinct sample sizes, dimensions, lengths, the number of classes. Some datasets contain variable-length samples. To tackle the issue of varying time series lengths, we applied uniform scaling [Keogh] 2003) Yankov et al.) 2007] Gao & Lin 2018) Tan et al.) 2019) match all series to the length of the longest one. Subsequently,
generated random missing observations for each time series and then combined the modified series. The data was divided into train, validation, and test in 0.70/0.1 150..15 15 ratio. This done after aggregating the original split as the training and testing splits provided by Bagnall et al. (2018) had inconsistent ratios across different datasets, After that, random missing observations for each variable were generated, and the modified variables were combined. In each set of cross validation, we used different random seed to investigate the robustness of the methods.
D DETAILS OF EXPERIMENTAL SETTINGS
experiments were performed using a server on Ubuntu 22.04 LTS, equipped with an Intel(R) Xcon(R) Gold 6242 CPU and six NVIDIA A100 40GB GPUs. The source code for our experiments can be accessed at
httpreeettttaeeeeeeea----------- Neural-SDES.
D.1 BENCHMARK METHODS
Conventional recurrent neural networks including RNN Rumel hart al.) 1986 Medsker & Jain 1999). LSTM (Hochreiter & Schmidhuber 1997) and
RNN-based methods:
GRU (Chung et al.) 2014). When dealing with iregularly-sampled or missing data, we apply mean imputation for the missing values. Furthermore, Choi al. (2016) proposed model that incorporates lapses in time between observations, alongside the observations themselves. In Che et al. (2018), GRU-D uses sequence of observed values, missing data indicators, and elapsed times between observations inputs and learns exponential decay between observations.
Attention-based methods: Shukla & Marlin (2021) proposed technique known Multi- Time Attention Networks (MTAN), which involves learning an embedding of continuous time values and utilizing attention mechanism. Similarly, Lee et al. (2022) introduced
22

---

Published as a conference paper at ICLR 2024
method named the Multi-Integraan Attention Module (MIAM) for extracting intricate information from irregular time series data with additional attention mechanism.
Neural ODEs: Neural ODEs (Chen al. 2018), which are widely utilized for learning
continuous latent representations, come in various forms including GRU-ODE (De Brouwer
et al., 2019), ODE-RNN (Rubanova et al., 2019). ODE-LSTM (Lechner & Hasami (020).
Latent-ODE (Rubanova et al. 2019), Augmented-ODE (Dupont et al. 2019). and Attentive
co-evolving neural ordinary differential equations (ACE-NODE) (Jhin al. (02T).
Neural CDEs: Kidger et al. (2020) introduced the Neural CDE consider continuous change of the entire input over time. Neural Rough Differential Equation (Neural RDE) (Mor rill et al. 202I). make use of log-signature transformations to directly incorporate time series into the path space. We used depth 2 with mean imputation for the Neural RDE, Attentive Neural Controlled Differential Equation (ANCDE) (Jhin et al. 2023b), employ dual Neural CDEs to compute attention scores. EXtrapolation and In lelpollatoon-baa based model (EXIT) TJhi et al., 2022) and LEArnable Path-based model (LEAP) (Jhin al., 2023a) utilize an explicit encoder-decoder structure to create the latent control path.
Neural SDEs: Neural SDEs have been proposed to model genuine random phenomena. SDEs, which represent a logical progression from ODEs, can be applied the analysis of continuously evolving systems and accommodate uncertainty using drift and diffusion terms (Tzen& & Raginsky [2099] Jia & Benson 2019; Liu al., 2019). On the other hand, Li et al. (2020) introduced a method called Latent SDE that uses an adjoint method, which can be considered as an instantaneous analog of the chain rule for solving Neural SDEs. We incorporate Kullback-Leible (KL) divergence loss for training Latent SDE.
D.2 NETWORK ARCHITECTURE
There are multiple neural networks in the proposed method, as shown in Equations (22
and (1): 7, (ztt); 0, 0c), (2,t,, xt); fy 0c), and 0t: The drift terms with control include the mapping function <: R2 x I x Rd. + Hd,, which uses the concatenated value of the latent value xt) and the controlled path X (f) as input. Both the drift functions 71 and T22 are implemented as Multi-layer Perceptrons (MLPs) with ReLU activation. The diffusion function has two options: linear affine and nonlinear neural network. Our approach involves constraining the SDEs to exhibit diagonal noise, a decision aimed satisfying the commutativity property as delineated by Rööler (2004). Consequently, our designated networks under Itô -Taylor schemes ensure the wish convergence from any given fixed starting point (Kloeden & Neuenkirch, 2007 Li et al. 2020)
In practice, we implemented several design choices to optimize the model's performance and stability. Firstly, we utilized the idea of sinusoidal positional encoding, as proposed by Vaswani al. (2017). for the time variable, ensuring that each time step possesses a unique encoding. Secondly, we opted for the tanh nonlinearity as the final operation for drift, diffusion, and all other vector fields, following the recommendation by Kidger et al.) 2020). This choice is intended to prevent the model from experiencing issues related to excessively large values or gradients. Lastly, we employed layer-wise learning rates for the model's final layer (e.g. x100), facilitating more precise and adaptive learning for the classification task. This approach can result in improved classification outcomes, as each layer can adjust its learning rate according to the complexity of the features it aims to capture.
D.3 TRAINING STRATEGY
In addressing datasets characterized by irregular sampling or missing values, the initial value of the process is established based on available observations. The initial value, denoted Zig interpolated from the observed data x and subsequently used to define the mapping for z()). This methodology is consistent with the approaches outlined by Kidger et al. (2020) and subsequent research.
Subsequently, the latent trajectory value at z()) is specifically utilized for designated tasks. This approach ensures that both the estimation of the SDE and the model's 's output are co-optimized. In our task of classifying time series data, we utilize the cross- cross-entropy loss function, taking the final value of the latent representation z(T) as input. We employ an MLP, comprising two fully connected layers with Re activation, to process the extracted feature z [0, T), as follows:
ÿ (z)(2(TT;; 0MP))
(30)
23

---

Published as a conference paper at ICLR 2024
where ÿ is the predicted label of the given time series sample. To mitigate the risk of overfitting and regularize the model, we incorporate a dropout rate of 10%. Also, we employed an carly-stopping mechanism, ceasing the training when the validation loss didn't 't improve for 10 successive epochs. The training approach for our proposed Neural SDEs for classification is outlined in Algorithm [ The adjoint sensitivity method's gradient computation facilitates the integration of mini-batch algorithms in solving Neural SDEs. For this purpose, python library t torchsde (Li et al. 2020) is employed, which is adept at handling both naive Neural SDEs and the proposed Neural SDEs our research.
Algorithm Train procedure for classification task
1: Divide training data into train set Derain and validation set Deat- Set the maximum iteration epockmer.
2: Initialize the parameters for the control neural network 0g, the drift term 0y, and the diffusion term 0g.
3: Initialize the parameters for the MLP classifier OMLP.
4: for i = to epoch do
5: Train parameters O [0c, 0y G,, Omeel using Dirain and classification loss C(y, 9).
arg min L(y, 0)-
6: Validate using Dost and update parameters.
7: Find the best parameters 0" to minimize validation loss.
8: end for
9:
Return 07 87 07 0ha.
D.4 EMPIRICAL STUDY
Table 7: Analysis of sensitivity for Neural SDEs' depth T and Classification performance utilizing the 'BasicMotions' dataset under each scenario (* denoting the final value)
(a) Neural SDE
Depth
Regular datasets
Missing datasets (30%)
Missing datasets (50%)
Missing datasets (70%)
10
0.417 + 0.132
0.433 = 0,070
0.467 0.095
0,417 = 0,132
50
0.383 4 0.112
0.417 = 0.144
0.417 0.118
0.383 0,162
90
0.483 0. 124
0.383 0.095
0.483 0.091
0,383 0.151
100*
0.417
# 0.000
0.333 + 0.059
0,500 = 0.059
0.467 = 0.095
(b) Neural LSDE
Depth
Regular datasets
Missing datasets (30%)
Missing datasets (50%)
Missing datasets (70%)
10
0.433 + 0. 137
0.417 + 08
0.300 + 0.095
0.450 0.173
50
1.000 0.
0.967 0.046
0.950 0.075
0.867 0.046
90
1.000 + 0.000
0.967 # 0.046
0.967 = 0.046
0.917 0.000
100*
0.000
0.000
0.037
(c) Neural LNSDE
Depth
Regular datasets
Missing datasets (30%)
Missing datasets (50%)
Missing datasets (70%)
10
0.333
0.400 1
0.139
0.367 0.139
50
1.000 a 0.000
0.967 + 0.046
0,933 +: 0.070
0.900 0.070
90
0.037
0.037
0,933 0.070
100*
1.000 + 0.000
1.000 0.000
1.000 0.000
0.983 0.037
(d) Neural GSDE
Depth
Regular datasets
Missing datasets (30
Missing datasets (50%)
Missing dutasets (70%)
10
0.483 + 0. 124
0.417 + 0,102
0,400 0.070
0.417 = 0.144
50
0.983 + 0.037
0,950 + 0.046
0.933 0.070
0.883 0.095
90
1.000 0.000)
0.967 1 0.046
0,983 0.037
0.917 0.059
100*
1.000 4 0.000
0.983 + 0.037
0.983 + 0.037
0.983 0.037
https:://githhb..cmmm www.aleeerreeaeeeeeeeeeeeeee
24

---

Published as a conference paper at ICLR 2024
Impact of the Neural SDEs' depth T. Theorems 3.5 and 3.6 establish non-asymptotic upper bounds on the variance between output distributions of original and perturbed input data. To empirically verify this hypothesis. the impact of varying I on Neural SDEs' performance is examined.
Table 7 illustrates the classification results obtained from the 'BasicMotions' dataset at various depth levels. In alignment with theoretical expectations, it is observed that the model's 's efficacy consolidates and its robustness intensifies as the depth parameter is increased, substantiating the durability of the proposed methods in scenarios of varying depths.
Table 9: Classification performance on the 'BasiMMotions' dataset at a 50% missing rate (Average taken over five iterations. Runtime, in seconds, for 100 epochs without early-stopping.
Table 8: Classification performance on the 'BasicMotions'' dataset at a 50% missing rate using different solvers: the explicit Euler, the Milstein, and the Stochastic Runge-Kutte Kutta (SRK) (Average taken over five iterations. Runtime, in seconds, for 100 epochs without early-stopping)
Accuracy
Runtime
RNN
0.600 = 0.070
0.1
LSTM
0.700 0.095
2.
GRU
3.00..0
Solvers
Accuracy
Runtime
0,967 =
GRE-D
0,933 0.070
4.3
Fuler
0,500 0.059
0.3
MTAN
0.717 0,247
6.2
Neural SDE
Milstein
0:483
0.091
MIAM
0,933 0,109
0.6
SRK
0,500
0.059
749.4 a 3.4
GRU-ODE
0,983 = 0.037
0.3
ODE-RNN
1.000 0.000
0.2
Euler Milstein
1.000 0.000
89.4
ODE: ODE-LSTM
0,717 0.126
0.7
Neural LSDE
0.983
0.037
112.6 + 1.0
0,983
102.5 0.3
SRK
1.000 0.000
748.5
Neural RDE
0,717 0.046
93.8
Fuler Milstein
1.000
ANCDE
0.983 0.037
360.5 0.5
Neural LNSDE
0.983
0.037
135.3 + 0.8
EXIT
0.417
0.3
SRK
0.983 0.037
822.5 + 2.2
LEAP
=317=0.122
163.0 0.6
0,350
Euler
0.983 0.037
96.4 0.6
Neural
Milstein
0,983
Neural SDE
0,500 = 0.059
87.4 # 0.3
SRK
0.967 0.046
839.1 4.1
Neural LSDE
1,000 0.000
89.4 : 0.5
Neural LNSDE
0.000
95.0
Neural GSDE
0,983 = 0.037
96.4 + 0.6
Choice of solver. We considered three distinct numerical solvers for Neural SDFs: the explicit Fuler-Maruyama method, the Milstein method, and the Stochastic Runge-Kutte (SRK) method. Each solver presents varying degrees of convergence and computational efficiency: the Euler method demonstrates a convergence order of 0.5, while the Milstein and SRK methods exhibit higher orders of 1.0 and 1.5, respectively, indicating progressively enhanced accuracy.
For our study involving high- nigh-dimensional SDEs, the explicit Euler method was selected due to its superior computational efficiency, despite its lower accuracy, in comparison to the more computation- ally intensive Milstein and SRK methods. This choice was motivated by the necessity to manage the computation computational demands of high-dimensional data effectively. As shown in Table 8 this selection is further validated by the average runtimes observed using the 'BasicMotions' dataset.
Recognizing the potential concerns about numerical stability in explicit numerical solutions, our Neural SDE models were carefully crafted to ensure robust performance. This design forethought crucial in maintaining numerical stability throughout the computations utilizing the explicit Euler method. Table 9 provides an investigation into both the classification accuracy and the average runtime, reflecting the efficiency and effectiveness of our chosen methodology versus benchmarks.
E DETAILED RESULTS OF THE BENCHMARK DATASETS
For the interpolation task, we employed the dataset pipeline recommended for PhysioNet Mortal- ity (Silva al., 2012). as outlined by Shukla & Marlin (021). with additional resources available in its GitHub repository The forecasting task utilized the Mu.oooo dataset (Tassa et al. 2018).
following the experimental procedures and resources found in Jhin et al. (2023b) and its corre- sponding GitHub repository [0 For the datasets PhysioNet Sepsis (Reyna et al. 2019), and Speech
httpps://github..com////
ANCOR
25

---

Published as a conference paper at ICLR 2024
Commands (Warden 2018), we adopted the preprocessing and experimental protocols detailed in Kidger et al. (2020) and its corresponding Github repository We recommend referring to the onginal paper for further details regarding the data and experiment protocols.
PhysioNet Mortality. We compare the proposed techniques with the benchmark method delineated in Shukla & Marlin (202)) The following models are considered: RNN-VAL model (Chen et al. 2018), employs Variational AutoEncoder (VAE) framework wherein both the encoder and decoder are instantiated as conventional RNN architectures. L-ODE-RNN model (Chen et al.. 2018), is a Latent ODE variant that utilizes an RNN encoder and a Neural ODE decoder. L-ODE-ODE (Rubanoval et al. 2019), is another Latent ODE variant, but both its encoder and decoder are based on ODE-RNN and Neural ODE respectively. MTAN (Shukla & Marlin 2021), integrates time attention mechanism alongside Bidirectional RNNs to encapsulate temporal features and interpolate data.
Table 10: Results of hyperparameter tuning for PPhysioNet Mortality': Mean Squared Error (MSE, scaled by 10 10-3) for interpolation tasks across various observation levels
(a) Neural SDE
(b) Neural LSDE
Observed %
Observed %
F92
73
50%
60%
70%
80%
90%
50%
60%
70%
80%
90%
16
8.500
8.892
8.666
8,593
5.315
4.654
32
8.953
8.785
8.719
8,603
8.377
32
5,081
5.039
4.596
4,632
3.732
8.739
8.718
8,392
4,623
3.347
128
8,770
8,686
8,560
8,464
8.296
128
3,822
3,658
3,516
3,355
3,098
16
8.901
8.835
8.706
8.624
8,445
5.399
5.286
5.153
5.104
4.336
32
8.861
8.858
8.669
8,550
8.446
32
4,711
4.520
4.332
4.097
4,000
2
8.829
8,518
4,200
128
8,826
8:705
8.526
8,430
8,305
128
4,310
4.167
4.130
4.057
3,106
16
8.899
8.782
8. 8.652
8.517
8.427
16
5.585
5.367
5.295
5,242
4.683
32
8.812
8.829
8.607
8.470
8.3114
32
4,806
4.567
4.220
3,98%
3.789
3
8.687
128
8.696
8.652
8.533
8,412
8.312
128
3.793
3.645
3.350
3.209
3.089
16
8.902
8.690
8.715
8:521
8,473
16
5,223
5.296
4.964
4.811
4.665
32
8.854
8.657
8.571
8,398
8.367
32
4.682
4.751
4.107
4,046
3.848
4
8.788
128
8.592
8.591
8.540
8.318
8.252
128
3.799
3.584
3.457
3.262
3.111
(c) Neural LNSDE
(d) Neural GSDE
Observed %
Observed
7
72
50%
60%
70%
80%
90%
50%
60%
70%
80%
90%
16
5.367
5.362
4,914
4,516
4.224
16
5.408
5.245
5.159
5.090
4.499
32
4,980
4.309
4.674
4,616
3,766
32
5.093
4.961
4.320
4,888
3.968
64
4,490
4,591
4,356
4,531
3.503
4,699
4.191
4.470
4.464
3.477
128
3,800
3.763
3.492
3,293
3.060
128
4.149
3.882
4.219
4.072
3.195
16
5,297
5.374
4.842
5,030
4.703
16
5.508
5.491
5.090
4,650
4.541
2
64
32
4.171
4.645
4.00 4.044
4.628
3.838
4.531
3.705
4.197
3.303
4.060
2
32
4.215
4,832
4.078
4.766
4.595
3.936
4.196
3.702
4.094
3.427
128
4,309
4,199
4.130
3,535
3.162
128
4,369
4.274
4.222
3.952
3,158
16
5.455
5.411
5.074
5.152
4.766
16
5.591
5.498
5.311
4,896
4.754
3
64
32
4,082
4,726
3.905
4.383
3.656
4.052
3.457
3,963
3.395
4.040
3
32
4,273
4.684
4.046
4.345
4.224
3.760
4,0%
3,663
3,869
3,486
128
3.829
3,600
3.353
3.193
3.040
128
3.863
3,661
3,453
3.296
3.089
16
5.365
5.119
4,809
4,760
4.608
16
5,449
4.918
4.911
4,780
4.649
4
32
4.674
4.180
4,260
3,945
4.064
4
32
4.672
4.496
4.377
4.145
3.902
64
4.1
3.823
3.689
3.458
3.398
4.242
3,966
3.753
3.555
3,497
128
3.808
3.617
3. 3.405
3.269
3.154
128
3.824
3.667
3.493
3.287
3.118
For the proposed methodology, the training process spans 300 epochs, employing batch size of 64 and a learning rate of 0.001. To train our models on a dataset comprising irregularly sampled time series, adopt strategy from [hhukla & Marlin (202T) This involves the modified VAE training method, where we optimize a normalized variational lower bound of the log marginal likelihood,
26

---

Published as a conference paper at ICLR 2024
grounded on the evidence lower bound (ELBO). Hyperparameter optimization is conducted through a grid search, focusing on the number of layers in vector field ny of {16, 32, 64, 128) and hidden vector dimensions n2 of {16, 32, 64, 128]. The optimal hyperparameters for the proposed methods are remarked in bold within Table TO encompassing observed data ranging from 50% to 90%.
MuJoCo. We evaluate the proposed model against the performance provided on Jhin et al. (023b)
The methods we compared include GRU-At (Choi et al. 2016). GRU-D (Che et al. 2018). GRU- ODE (De Brouwer et al. 2019), ODE-RNN (Rubanova et al.] 2019p. Latent-ODE (Rubanova et al.
2019),
Augmented-ODI ODE (Dupont et al., 2019), ACE-NODE (Jhin et al. 202T), Neural CDE (iidger
et al. 2020). ANCDE (Jhin et al. 2023b), b,, EXIT (Thin et al. 2022). and LEAP (Thin et al. 2023a)
Table 11: Results of hyperparameter tuning for 'MuloCo' with regular dataset
Neural SDE
Neural LSDE
Neural LNSDE
Neural GSDE
Test
Memory
Test
Memory
Test
Memory
Test
Memory
16
0.057 0.009
66
0.031 0.001
72
0,032 0.001
78
0.031 0.004
32
0.040 0.001
118
0.022 + 0.001
127
0.021 0.002
139
0.019 0.001
157
64
0.041 + 0.008
218
0.018 + 0.001
237
0.016 0.001
261
0.016 0.001
298
0.002
0.017 0.001
0.016 0.001
0.013 0.001
16
0.054 0.011
69
0.041 0.001
75
0.033 a: 0.002
81
0.034 & 0.000
32
0.038 0.003
124
0.024 0.002
133
0.024 0.002
141
0.025 0.001
160
64
0.032 + 0.000
234
0.017 + 0.001
249
0.017 + 0.001
273
0.018 0.001
306
0,031 0.002
0.013 0,000
0.012 0.000
0,015 0.001
16
0.054 & 0.012
72
0.043 &: 0.000
78
0.036 0.003
0.039 0.003
0.061 0.012
130
0.030 0.000
139
0.029 0.000
151
0.029 0.001
162
0.031 + 0.001
246
0.023 + 0.000
261
0.023 + 0.002
286
0.022 + 0.003
310
0,028 0,001
0.018 0.001
509
0.017 0.001
0,018 0.001
16
0.065 0.004
75
0.046 a: 0.000
81
0.044 a: 0.005
0.036 2 0.001
92
0.061 0.010
136
0.035 + 0.002
145
0.037 0.001
157
0.031 0.000
169
0.031 0.003
259
0.027 + 0 0.000
274
0.028 0.000
298
0.027 + 0.000
322
0.031 0.004
307
0.023 0.002
534
0.023 0.001
583
0.023 0.001
632
For the proposed methodology, we spanned 500 epochs with a batch size of 1024 and a learning rate set at 0.001. We performed hyperparameter optimization using a grid search, focusing specifically on the number of layers in vector field ny of {16, 32, 64, 128} and hidden vector dimensions n, of {16, 32, 64, 128). The optimal hyperparameters, as identified for our methods, are highlighted in bold in Table IT for regular datasets. For scenarios with 30%, 50%, and 70% missing data, we applied the same hyperparameters.
Table 12: Forecasting performance versus percent observed time points on MuloCo
Test MSE
Memory Usage (MB)
Methods
Regular
30% dropped
50% dropped
70% dropped
GRU-At
0.223 0.020
0.198 0.036
0.193 0.015
0,196 0.028
533
GRU-D
0.578 0.042
0.608 0.032
0.587 2 0.039
0,579 2 0.052
569
GRU-ODE
0.856 0.016
0.015
0.015
0.015
146
ODE-RNN
0.328 2 0.225
0,274 1 0.213
0,237 0.110
0.267 0.217
115
Latent-Ou ODE
0.029 = 0.011
0.056 0.001
0.055 0.004
0.058 a 0.003
314
Augmented- -DE
0.055 0.004
0.056 0.004
0.057 0.005
0.057 0.005
286
ACE-NODE
0.039 0.003
0.053 = 0.007
0.053 + 0.005
0.052 + 0.006
423
NCDE
0.028 0.002
0.000
0.001
0.026 0.001
52
ANCDE
0.026 0.001
0.025 2 0.001
0.025 a 0.001
0.024 0.001
79
EXIT
0.000
0.004
0.000
0.026
127
LEAP
0.022 0.002
0.022 + 0.001
0.022 0.002
0.022
144
Neural SDE
0.028 0.004
0.029 0.001
0.029 0.001
0.027 0.000
234
Neural LSDE
0.013 0.000
0.014 = 0.001
0.014 = 0,000
0.013 2 0.001
249
Neural LNSDE
0.001
0.014 = 0.001
0.014 0.001
0.014 0.000
273
Neural GSDE
0.013+0.000
0.013 = 0.001
0.013 = 0.000
0.014 + 0.000
306
Table 12 presents the forecasting performance across varying data drop ratios. Our methods consis- tently demonstrate lower MSE scores, indicating their superior forecasting capabilities. However, the proposed methods demand higher computational resources due to the complex architecture.
27

---

Published as a conference paper at ICLR 2024
PhysioNet Sepsis. We evaluate the proposed model against the performance provided on Jhin et al. (2023b). The methods we compared include GRU-At (Choi et al., 2016), GRU-D (Che et al. 2018).
BRU-ODE (De Brouwer al.) 2019), ODE-RNN (Rubanova et al. 22199. Latent-ODa (Rubanoval
et al. 2019), ACE-NODE (Jhin et al. 202T), Neural CDE (Kidger et al. 2020), and ANCDE (Jhin al. 2023b). We examine the results both with and without the observational intensity (OI), which
is determined by attaching a mask indicating whether an observation was made or not to each input.
Table 13: Results of hy hyperparameter tuning for 'PhysioNet Sepsis' with observation intensity
Neural SDE
Neural LSDE
Neural LNSDE
Neural GSDE
Test AUROC
Memory
Test AUROC
Memory
Test AUROC
Memory
Test AUROC
Memory
16
0.792 0.008
304
0.907 0.003
314
0.910 + 0.002
340
0.909 0.002
338
32
0.780 0.006
369
0.904 0.002
353
0.899 0.003
366
0.902 0.004
369
64
0.790
= 0.006
442
0.895 + 0.002
421
0.881 0.007
485
0.894 2 0.007
509
128
0.769 0.008
736
0.890 0.010
0.870 0.015
0.883 0.007
16
0.793 0.004
338
0.903 0.005
344
0.911 + 0.002
341
0.907 0.001
298
32
0.786 0.010
379
0,909 0.004
373
0.903 0.005
351
0,903 0.002
332
64
0.763 + 0.005
486
0.907 + 0.003
489
0.898 0.006
522
0,900 + 0.007
536
0.769 0.010
0.906 0.004
0.867 0.007
0.882 0.009
16
0.793 0.003
346
0.905 0.001
341
0.909 0.002
338
0,904 #: 0.006
281
32
0.799 0.007
368
0.908 0.004
359
0 0.003
341
0,906 0.002
347
64
0.006 0.776
491
0.901 + 0.006
481
0.902 # 0.007
495
0.901 0.002
511
0.776 0.006
0.902 0.005
0.004
0.003
16
0.782 0.011
333
0.902 0.007
299
0.906 0.005
341
0.893 0.007
247
32
0.784 0.012
376
0.902 0.001
385
0.908 0.003
345
0,906 0.004
362
0.780 0.005
488
0.907 +: 0.004
476
0.899 0.005
541
0.909 0.001
588
0.764 0.013
0.900 0.002
0.900 0.004
0.001
Table 14: Results of hyperparameter tuning for 'PhysioNet Sepsis' without observation intensity
Neural SDE
Neural LSDE
Neural LNSDE
Neural GSDE
Test AUROC
Memory
Test AUROC
Memory
Test AUROC
Memory
Test AUROC
Memory
0.782 0.008
171
0.867 0.004
183
0.873 0.006
173
0,875 0.002
32
0.787 0.008
216
0.869 0.006
256
0.868 0.010
248
0.873 0.003
267
0.790 = 0.005
353
0.870 0.004
353
0.832 0.012
388
0.852 0.011
402
128
0.773 0.005
659
0.829 : 0.010
712
0,775 0.020
783
0.809 0.009
868
0.795 0.009
191
0.872 0.007
181
0.874 0.004
188
0,880 0.002
184
32
0.776 0.007
233
0.867 0.002
245
0.877 0.006
245
0,880 + 0.005
260
0.782 0.012
394
0.866 0.004
447
0.865 0.007
0.870 0.006
128
0.775 0.006
592
0.867 0.006
761
0,815 0.011
772
0.854 : 0.007
833
16
0.787 0.009
198
0. 0.865 0.001
181
0,869 0.006
0.872 0.006
164
32
0.797 0.006
240
0.867 0.006
248
0.878 0.006
261
0.884 # 0.0022
280
0.783 0.007
386
0.868 0.004
420
0.863 0.005
0.878 0.001
128
0.764 0.004
704
0.837 + 0.015
802
0,846 0.011
780
0,870 0.004
968
16
0.762 0.005
184
0.859 0.007
189
0.865 0.003
178
0,868 0.007
186
32
0.781 0.006
254
0.862 0.009
266
0.872 0.002
284
0.877 0.004
304
4
0.777 0.005
0.008
0.002
0.878
0.003
128
0.752 0.017
789
0.866 0.006
747
0.859
0.005
850
0,875 0.003
994
In the proposed method, we train for 200 epochs with a batch size of 1024 and learning rate of 0.001. The hyperparameters are optimized by grid search in the number of layers in vector field n; of (16, (1, 64, 128} and hidden vector dimensions ni, of (16, 32, 64, 1283. The best hyperparameters highlighted with bold font in Table [3] and [4] Due to varying input channel counts in each scenario, the best hyperparameters might differ. The interplay between hyperparameters and data traits is vital for model efficacy and adaptability.
Speech Commands. We evaluate the proposed model against the performance provided on Thin et al. (2023a). The methods we compared include RNN (Medsker & Jain| 1999). LSTM (Hochreiter & Schmidhuber 1997). GRU (Chung et al. 2014),
GRU-At (Choi et al., 2016). GRU-D (Che
et 2018). GRU-ODE (De Brouwer et al. 2019 ODE-RNN (Rubanova et al. 2019), Latent-
ODE (Rubanova et al. 2019). Augmented-ODE (Dupont et al. 2019). ACE-NODE (Jhin et al. 2021
Neural CDE (Kidger al. 2020), ANCDE Thin al. 2023b), and LEAP (Jhin al. 2023a
28

---

Published as a conference paper at ICLR 2024
Table 15: Results of hyperparameter tuning for 'Speech Commands'
Neural SDE
Neural LSDE
Neural LNSDE
Neural GSDE
Accuracy
Memory
Accuracy
Memory
Test Accuracy
Memory
Accuracy
Memory
16
0. 0.105 + 0.001
238
0.732 + 0.013
246
0.749 0.009
250
0.739 = 0.009
261
32
0.110 + 0.005
309
0.837 0.011
347
0.846 0.004
352
0.837 = 0.005
436
1
64
0.105 a 0.001
521
0,866 =: 0,008
543
0.892 0.005
656
0.876 = 0.005
698
128
0.104
880
0.037
0.924 0.000
1164
16
0. 0.106+ + 0.003
213
0.839 0.016
259
0.757 0.007
198
0,746 = 0.002
252
32
0.105 0.002
315
0.886 0.009
353
0.859 0.003
377
0.848 0.002
476
2
64
0.109 a 0.005
535
0,900 = 0,0066
599
0.903 0,003
649
0.895 = 0.002
734
128
0.109 +:
995
0.005
1142
0.915 0.003
1211
0.003
1431
16
0.108 + 0.004
215
0.847 0.009
237
0.700 0.020
240
0.711 = 0.005
263
32
0.107 + 0.002
343
0,904 0.001
398
0.838 0.007
456
0,844 0.003
467
3
64
0.103 + 0.002
577
0,919 = 0.003
557
0.898 0.006
731
0.895 = 0.004
843
128
0.107 a
0.004
0.922 0.001
1277
0.001
16
0.108 + 0.004
233
0.847 = 0.013
243
0.383 0.069
267
0.632 = 0.019
276
32
0.107 0.001
363
0,900 0.002
387
0.782 0.031
423
0,800 0.016
482
64
0.106 + 0.001
573
0,916 = 0.003
630
0.887 2 0.009
692
0.872 = 0.007
812
128
0.106
1157
0.005
0.912 0.004
1372
0.899
In the proposed method, we train for 200 epochs with a batch size of 1024 and learning rate of 0.001. The hyperparameters are optimized by grid search in the number of layers in vector field of {16,32 64, 128} and hidden vector dimensions vi, of {16, 32, 64, 128]. The best hyperparameters are highlighted with bold font in Table Relative to the naïve Neural SDE, our suggested approaches deliver enhanced results through a more intricate model framework.
Robustness to missing data. We utilized the Python library torchede Kidger et al. Morrill et al. 2021) for the interpolation scheme and the Python library torchsde T3 (Li et al.
2020
2020)
to solve Neural SDEs for both the Neural SDE and the proposed methods. This study utilized the
original Neural CDE source codo 14 "idger et al. 2020 for benchmark methods including GRU-At,
GRU-D, GRU-ODE, ODE-RNN, and Neural CDE Additionally, we employed ODF-LSTM using
its original source code 15| (1 (Lechner & Hasani 2020). as well as Neural RDF (Morrill et al. 2021
A (Jhin et al. 2023b), "X (Jhin et al. 2022), and LEAP @Jhin et al. 2023a). using their
original source code. For Latent SDE, we formulate adjoint SDE proposed by Li et al. 2020 and train backpropagation incorporated with the original KL divergence.
To ensure fair comparison, this study used the original architecture across all methods. However, because the optimal hyperparameters vary between the methods and data, this study used the Python library ray 200 Moritz et al. 2018 Liaw al. 2018)) for hyperparameter tuning. With the use of this library, hyperparameters are optimally tuned to minimize validation loss automatically, contrasting with previous research that required manual tuning for each dataset and model. In our experiments, we identified the optimal hyperparameters for regular time series and applied them to irregular time series. For all methods, we employed the (explicit) Euler method as the ODE solver.
In each model and dataset, the following hyperparameters are optimized to minimize validation loss: hidden vector dimensions nh,, and the number of layers n,. For RNN-based methods such as RNN, LSTM, and GRU, the fully-connected layer employs hyperparameters. For NDE-based methods, they are utilized to embed layer and vector fields. The hyperparameters are optimized as follows: the learning rate Ir from 10-4 to 10-1 using log-uniform search; from 2, 4] using grid search; and ny from (16, 32, 64, 128) using grid search. The batch size was selected from (16,32, 64, 128) with respect to total data size. All models were trained for 100 epochs, and the best model was
29

---

Published as a conference paper at ICLR 2024
selected based on the lowest validation loss, ensuring that the chosen model generalizes well to unseen data. These design choices and training strategies aim to create a robust, well- model that effectively addresses the given classification tasks.
F DETAILED RESULTS OF 'ROBUSTNESS TO MISSING DATA' EXPERIMENTS
Our analysis involved thorough evaluation of 30 datasets, considering various scenarios such as regular time series and time series with missing data rates of 30%, 50% and 70%. Also, we considered different characteristics of time series, such as univariate or multivariate.
We tested Neural CDE using various controlled paths in Table 16 linear, rectilinear, natural cubic spline, and hermite cubic splines. Hermite cubic spline demonstrated superior performance compared to the Neural CDE with other control paths, leading us to select it for our proposed methods.
Table T6 presents ablation study conducted to assess the performance of the proposed methods: Neural LSDE, Neural LNSDE, and Neural GSDE. It also compares the naïve Neural SDE with various components. As expected, the careful design of Neural SDEs can improve the classification performance. Also, we can observe that incorporating the control path into the model leads to signif- icant improvement in classification performance. Furthermore, employing nonlinear neural network for the diffusion term further enhances the performance compared to using an affine transformation.
Table 16: Comparison of average accuracy and average cross-entropy loss from the ablation study. For Neural CDEs, different controlled paths were examined: (i)inear, (r)ectilinear, (n)atural cubic spline, and (h)ermite cubic splines. For Neural SDEs, < indicates whether the drift function incorporates the controlled path or not. 'N' or 'L' denote the architecture of diffusion function a with affine function nonlinear neural network. The best and the second best are highlighted.)
Univariate datasets (15)
Multivariate datasets (15)
All datasets (34)
Methods
C
Accuracy
Loss
Accuracy
Loss
Accuracy
Loss
Neural CDE ())
0.593 (0.094)
0.809 (0. 107)
0.770 (0.037)
0.595 (0.086)
0.682 (0.065)
0.702 (0.097)
Neural CDE (O)
0,550 (0.09)
0,879 (0.090)
0.716 (0.045)
0,715 (0.105)
0.633 (0.071)
0.797 (0.098)
Neural CDE (n)
0.597 (0.083)
0.795 (0.084)
0.773 (0.042)
0.590 (0.095)
0.685 (0.063)
0.692 (0.089)
Neural CDE
0.611 (0.092)
0.804 (0.119)
0.777 (0.044)
0.549 (0.101)
0.694 (0.068)
0.676 (0.110)
0.615 (0.090)
0.736 (0.089)
(0.036)
0.618 (0.092)
0.688 (0.063)
0.677 (0.091)
Neural SDE
L
0.535 (0.095)
0,838 (0.094)
0.752 (0.040)
0.633 (0.087)
0.643 (0.068)
0.736 (0.090)
N
0.516 (0.084)
0.914 (0.069)
0.515 (0.045)
1.248 (0.085)
0,516 (0.064)
1,081 (0.077)
X
0.498 (0.087)
0.929 (0.078)
0.514 (0.050)
1.255 (0.081)
0.506 (0.068)
1.092 (0.080)
(0.081)
0.752 (0.084)
(0.031)
0.572 (0.080)
0.694 (0.056)
0.662 (0. (0.082)
0.533 (0.089)
0,877 (0.086)
0.745 (0.038)
0.668 (0.085)
0.639 (0.064)
0.772 (0.085)
Neural LSDE
0.530 (0.069)
0.856 (0.060)
0.527 (0.054)
1.177 (0.082)
0.528 (0.060)
1.039 (0.074)
X
0.505 (0.064)
0,912 (0.061)
0,518 (0.057)
1.210 (0. 101)
0.512 (0.059)
1.083 (0.068)
0.654 (0.073)
(0.091)
(0.029)
0.577 (0.070)
(0.051)
0.639 (0.080)
0.586 (0.087)
0,765 (0.083)
0,764 (0.044)
0.617 (0.10)
0.675 (0.066)
0.691 (0.096)
Neural LNSDE
0.532 (0.077)
0,878 (0.060)
0.502 (0.051)
1.254 (0.082)
0.517 (0.064)
1.066 (0.071)
X
0.528 (0.085)
0.890 (0.069)
0,510 (0.047)
1.257 (0.089)
0.519 (0.066)
1.074 (0.079)
0.633 (0.091)
(0.086)
(0.036)
0.598 (0.077)
0.703 (0.063)
0.670 (0.081)
(0.083)
0.796 (0.084)
0.748 (0.039)
0.653 (0.094)
0.660 (0.061)
0.724 (0.089)
Neural GSDE
0.531 (0.074)
0.868 (0.052)
0.510 (0.045)
1.261 (0.093)
0.520 (0.060)
1.064 (0.072)
X
0.525 (0.078)
0.893 (0.072)
0.509 (0.052)
1,261 (0.093)
0.517 (0.065)
1.077 (0.083)
Figures B] and 4 display the classification outcomes for all datasets over the four missing rate scenarios. Each point in figure showcases the average classification score for every dataset and missing rate per method. The results highlight the variability in performance based on the dataset and its specific characteristics, emphasizing the importance of selecting an appropriate model that suits the specific properties of the time series data. These characteristics may include the degree of missing data, the presence of noise, the complexity of patterns, and the distribution of the data value.
In conclusion, it is vital consider the inherent properties of the time series data when choosing the most appropriate model for given dataset. By doing so, we can develop robust and accurate models that address the specific challenges of diverse time series datasets, resulting in better generalization and performance across various applications and domains.
30

---

Published as a conference paper at ICLR 2024
0.43
3.00
0.0
0.23
0:20
620-
Missing
1000 (19)
73%
1ng 15g44mm74
SEN. Niosing
uni
2004
002
044
1833
ArrowHedd
Car
Coffee
3.00
3.00
0.62
6.00
00.00
E:02
Masiing
Mising
Nissing rals
GunPoint
Herring
Lighting2
1.50
3.00
0.82
0.22
5200
18p.4a13%
rega e1
37%
33%
31% Nissing Fatl
Missing ratl
Miss nisiing 1092
M.08
8005
4m
1824
40T PMM
Lightning?
Meat
OliveOil
0.82
0.50
0.33
9:20
0:2
0.89
0.07
A/D
45%
Rack
SmowthSubspucee
ToeSegmentationn
3.55
3.00
0.43
0.00
0.42
600
Vissing rate
Mising rote
Nissng Fate
10M4
008
ToeSegmentation2
Trace
Wine
Figure 3: Classification result of the 15 univariate datasets with all four settings
31

---

Published as a conference paper at ICLR 2024
0.63
3.00
0.0
0.23
0:20
620
17344 1134
75%
1ng 44
5EN.
Missing
Niosing
400
002
04.
000
Start 1833
Articulary SoniulayyWodReoogniiion
Mations
CharacterTrnjectniiee
3.00
0.42
6.0
00.00
E:02
Masiing
Mising rate
Nissing rats
Cricket
Epilopsy
ERing
1.00
3.00
0.82
0.22
18p.4a13%
37%
33%
1epularoS)
33% Nlcsing FatL
Missing ratl
Mis
4m
40C PM
FihhnooCConentraation
OpenOttt
FingerMavements
0.82
0.00
0.33
9:20
0:2
0.82
0.07
-5%
A/D
45%
Handwriting
Jupanese Vowels
Libras
3.55
3.00
0.03
0.00
0.43
840
600
Vissing rate
Mising rote
Nising Fate
1014
028
05
02F mM
NATOPS
RacketSports
SpokenArabic-iiiiss
Figure 4: Classification result of the 15 multivariate datasets with all four settings
32


Reviews:
[{"review_id": "dRzq95ht0o", "reviewer": "Reviewer_8YRs", "summary": "This paper addresses the challenges posed by irregular sampling intervals and missing values in real-world time series data. The authors propose three classes of Neural Stochastic Differential Equations (Neural SDEs) to improve robustness under distribution shifts in time series data. The proposed Neural SDEs include Langevin-type SDE, Linear Noise SDE, and Geometric SDE. The study demonstrates the robustness of these Neural SDEs theoretically and through extensive experiments, showing their effectiveness in handling real-world irregular time series data and maintaining excellent performance under distribution shifts due to missing data.", "questions": "The details of the proposed models seem unclear.  Can you clarify which neural networks are used in the diffusion and drift functions of each Neural SDE?  \n\nAdditionally,  how are the proposed neural SDEs solved, especially the neural GSDE?   Do you employ numerical solvers for the Neural SDEs? If so, which specific solver was utilized, and was there any ablation study conducted to evaluate its effectiveness?\n\nHow much computational time is required for training the proposed Neural SDEs, and what is the complexity of these models?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The authors provide a solid theoretical foundation for the proposed Neural SDEs, including the existence and uniqueness of solutions. \n\nThe robustness section in the paper provides valuable insights into the proposed Neural SDEs' resilience against distribution shifts and input perturbations. \n\n The paper conducts extensive experiments to validate the effectiveness of the proposed Neural SDEs. The models are tested on various datasets, and their robustness is analyzed under different missing rates, providing a comprehensive evaluation.", "weaknesses": "I didn’t identify major weakness in this paper; however, there are a few minor concerns that I would like to address:\n\n1. Section 3.4 could be more explicit in detailing how the controlled path is incorporated into the Neural SDEs. More comprehensive explanations or illustrations could help in understanding the model’s architecture and functionality better.\n\n2.  While section 3.3 discusses the robustness of the proposed Neural SDEs under distribution shifts, it might benefit from a more thorough exploration or comparison with other neural SDEs' robustness aspects in related works.\n\n3. What is the $\\| \\sigma_\\theta  \\|$ in Theorem 3.6 ?\n\n4. In the experiments of missing data (Table 4), the proposed Neural SDEs show only marginal improvements when compared to the Neural CDE model. How do the theoretical bounds in Theorems 3.5 and 3.6 relate to the robustness of the Neural SDEs in practical implementations? Are these bounds tight or rather loose in actual application scenarios?\n\n5. The implementation code is not provided.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "EwwXgJy0K2", "reviewer": "Reviewer_Q7tH", "summary": "This paper introduces three stable classes of Neural SDEs (Langevin-type SDE, Linear Noise SDE, and Geometric SDE) to capture complex dynamics and improve robustness under distribution shifts in time series data. Theoretically, this paper shows the existence and uniqueness of the solutions of these SDEs, and presents their performance guarantee under distribution shifts. Extensive experiments are conducted to validate the good performance.", "questions": "Detailed discussions on the training procedure are needed, especially for dealing with irregular time steps and missing data. For instance, with different irregular time steps across different sample time series, is it still possible to train the algorithm using mini-batch optimization; and is there a way to improve the computational efficiency in practice? When there is missing data in the sequence, how do we deal with missing values in the training phase -- are these missing values being imputed or ignored during the pre-processing step?\n\nMore explanations are needed for Figure 1 — line (i) exhibits a constant loss, and in fact, most of the loss trajectories are not satisfactory, with unstable trends and not decaying with the increase of epochs.\n \nFrom Table 5, it is a bit confusing why there is no result for LSDE, LNSDE, and GSDE-‘+Z’.\n\nThe downstream tasks considered in this paper are interpolation and classification; it could also be meaningful and worthwhile to consider the prediction task for time series data as well.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Topic-wise, modeling time series data with irregular sampling intervals and missing values is an essential research topic and of great importance in practice.\n\nTheory-wise, this paper proves the existence and uniqueness of the solutions of the proposed three SDEs, and shows their robustness to input data. \n\nAdditionally, extensive numerical results are presented to compare the proposed method with existing algorithms for time series modeling.", "weaknesses": "The computational complexity of the proposed method is certainly high. \n\nThis paper lacks sufficient details on the implementation of the method, especially when there are irregular time steps and missing data in the series.\n\nFor time series data, in addition to interpolation and classification tasks, it would also be meaningful to consider forecasting tasks as well, which seems to be absent in this paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ayw3Ytu5VA", "reviewer": "Reviewer_gG9F", "summary": "The problem of learning neural stochastic differential equations (NSDE) to solve classification/interpolation tasks in the context of (irregularly sampled) time series data is considered. The authors focus on the analysis of theoretically well-defined SDE classes that exhibit desirable properties in terms of parameterization of drift and diffusion coefficients by neural networks, e.g., in so-called NSDEs. In contrast to naive NSDEs, which can (theoretically) learn almost arbitrary classes of functions, the authors restrict themselves to classes of SDEs for which (i) a (uniquely) strong solution exists, (ii) which can be approximated in a numerically stable manner, and (iii) which remain robust to input perturbations. In addition, they build on concepts from the field of controlled differential equations, which are known to improve model performance on irregularly sampled time series data.  Extensive experiments with established benchmark data sets are included. These provide empirical evidence for the proposed improvements.", "questions": "1. For example, to evaluate robustness empirically, *explicit Euler* is used for all experiments (see page 25). What are the reasons for this choice? I am very curious about the impact of different numerical solution methods on these and other results. Can you explain this in more detail?\n2. Are you planning to release your Code which would unlock reproducibility of the results?\n\nMinor:\n\n3. Aren't the initial state in the Eq. (2) and (3) supposed to be vectors and therefore should be bold?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "I appreciate the idea of taking a step back from the state of unlimited expressiveness in NSDE and instead concentrating on sub-classes of SDEs that have favorable properties that combine well with the functional class properties of neural networks. The authors reveal shortcomings associated with the use of unrestricted parameterization of drift and diffusion coefficients by standard neural networks. In turn, an ablation study empirically supports the assumption that careful design of drift and diffusion coefficients is indeed reflected in improved model performance.\nThe content of the paper is well organized, original to the best of my knowledge and shows no obvious spelling or grammatical flaws.\nLast but not least, I enjoyed the theoretically analysis of robustness under distribution shift.", "weaknesses": "1. I don't see the necessity to include the details on Neural ODE and CDE into the main manuscript. However, thats more ore less a \nmatter of taste.\n2. (Section 4.2) Comparing such a rich variety of models is challenging. The main difficulty arises due to major differences in model structure. E.g., vanilla RNN based methods are by nature not capable to process irregularly sampled time series. As reported, data imputation strategies must be applied additionally. Another challenge arises from comparing methods that include a control mechanism (e.g., Neural CDE) with methods that do not (e.g., NSDE). The former are able to continuously correct the sampled trajectories over time during learning, while the latter can largely only adjust the initial state. However, the authors elaborate on the latter problem in Table 11, where the proposed methods nevertheless showed their advantage. However, I can't escape the impression that the built-in control mechanism is a big part of the success; because Neural CDE often takes the closely followed second place.\n\nNevertheless, I am on the positive side of this work.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (spotlight)