Learning Continuous-Time Dynamics by Stochastic Differential Networks
Authors: Yingru Liu
Yucheng Xing
Xuewen Yang
Xin Wang
Jing Shi
Di Jin
Zhaoyue Chen
Venue:  (2021)
Keywords: Continuous-time Stochastic RNN
Neural SDE

Abstract: Learning  continuous-time  stochastic  dynamics  is  a  fundamental  and  essential problem in modeling sporadic time series, whose observations are irregular and sparse in both time and dimension.  For a given system whose latent states and observed data are high-dimensional, it is generally impossible to derive a precise continuous-time  stochastic  process  to  describe  the  system  behaviors.  To  solve the above problem, we apply Variational Bayesian method and propose a flexible continuous-time stochastic recurrent neural network named Variational Stochastic Differential Networks (VSDN), which embed the complicated dynamics of thesporadic time series by neural Stochastic Differential Equations (SDE). VSDNs capture the stochastic dependency among latent states and observations by deep neural networks.  We also incorporate two differential Evidence Lower Bounds to efficiently  train  the  models. Through  comprehensive  experiments,  we  show  that VSDNs  outperform  state-of-the-art  continuous-time  deep  learning  models  and achieve remarkable performance on prediction and interpolation tasks for sporadic data.

Under review as a conference paper at ICLR 2021
LEARNING CONTINUOUS-TIME DYNAMICS BY STOCHASTIC DIFFERENTIAL NETWORKS
Anonymous authors
Paper under double-blind review
ABSTRACT
Learning continuous-time stochastic dynamics is a fundamental and essential problem in modeling sporadic time series, whose observations are irregular and sparse in both time and dimension. For a given system whose latent states and observed data are multivariate, is generally impossible to derive pre- cise continuous-time stochastic process to describe the system behaviors. To solve the above problem, we apply Variational Bayesian method and propose a flexible cittinuuus-tmee stochastic recurrent neural network named Variational Stochastic Differential Networks (VSDN), which embeds the complicated dynam- ies of the sporadic time series by neural Stochastic Differential Equations (SDE). VSDNs capture the stochastic dependency among latent states and observations by deep neural networks. We also incorporate differential Evidence Lower Bounds to efficiently train the models. Through comprehensive experiments, we show that VSDNs outperform state-of-the-ard continuous- time deep learning mod- els and achieve remarkable performance on prediction and interpolation tasks for sporadic time series.
INTRODUCTION AND RELATED WORKS
Many real-world systems experience complicated stochastic dynamics over continuous time pe- riod. The challenges on modeling the stochastic dynamics mainly come from two sources. First, the underlying state transitions many systems are often uncertain, as they are placed in unpredictable environment with their states continuously affected by unknown disturbances. Second, the monitor- ing data collected may sparse and at irregular intervals as a result the sampling strategy or data corruption. The sporadic data sequence loses a large amount of information and system behaviors hidden behind the intervals of the observed data. order to accurately model and analyze dynamics of these systems, it is important to reliably and efficiently represent the continuous-time stochastic process based on the discrete-time observations.
In some domains, the derivation of the contiuoosstimme stochastic model relies heavily on human knowledge and many studies focus on its inference problem (Ryder et al.] 2018 Särkkä et al.) 2015). But in tection (Rubanova et al. 2019)), is difficult and sometimes intractable to derive an accurate model capture the underlying temporal evolution from the collected sequence of data. Although some studies have been made on approximating the stochastic process from the data collected, the ma- jority of these methods define the system dynamics with linear model (Macke et al. 2011 Yu et al., 2009b ab. which can not well represent high-dimensional data with nonlinear relationship. Recently, the Neural Ordinary Differential Equation (ODE) studies (Chen et 2018 Rubanova et al. 2019, Jia & Benson, 2019 De Brouwer et al. 2019 Yildiz et al. 2019: Kidger et al. 2020) introduce deep learning models to learn an ODE and apply to approximate continuous-time dy- namics. Nevertheless, these methods generally neglect the randomness of the latent state trajectories and posit simplified assumptions on the data distribution (c.g. Gaussian), which strongly limits their capability of modeling complicated continuous -time stochastic processes.
domains (e.g., video analysis (Vondrick et al., (016)) and human activity de-
Compared to ODE, Stochastic Differential Equation (SDE) (Jorgensen et al. 2020) is a more prac- tical solution in modeling the continuous-tiim time stochastic process. Recently there have been some studies on bridging the gap between deep neural networks and SDEs (Ha et al. 2018). In (Hegde
et al. 2019 Liu al. 2020 Peluchetti & Favaro 2020 Wang et al. 2019 Kong 2020 SDEs

---

Under review as a conference paper at ICLR 2021
introduced to define more robust and accurate deep learning architectures for supervised learning problems (e.g. classification and regression). These studies focus on the design of neural network architectures, and are orthogonal to our work on the modeling of sporadic time series. In (Tenn & Raginsky, 2019a b) the authors studied the theoretical guarantees of the optimization and inference problems of Neural SDEs. In (Li et al. 2020), stochastic adjoint method is proposed efficiently compute the gradients for neural SDEs.
In this paper, we propose a new continuous-time stochastic recurrent network called Variational Stochastic Differential Network (VSDN) that incorporates SDEs into recurrent neural model to effectively model the continuous-time stochastic dynamics based only on sparse or irregular obser- vations. Taking advantage of the capacity of deep neural networks, VSDN has higher flexibility and generalizability in modeling the nonlinear stochastic dependency from nigh-dimensional observa- tions.
Compared to Neural ODEs, VSDN incorporates the latent state trajectory to capture the underlying factors of the system dynamics. The trajectory helps to more flexibly model the data distribution and more accurately generate the output data than Neural ODEs. Parallel to the theoretical analysis
(Tze & Raginsky 2019a b) and gradient computations (Li et al. 22200), our study focuses more on exploring the feasible variational loss and flexible recurrent architecture for the Neural SDEs to model the sporadic data.
The contributions of this paper are three-fold:
1. We incorporate the continuous-time variants VAE and IWAE losses into VSDN to train the continuous- time stochastic neural networks with latent state trajectories.
2. We propose the efficient and flexible network architecture of VSDN which can model the complicated stochastic process under high-dimensional sporadic data sequences.
3. We conduct comprehensive experiments to show that VSDN outperforms state-of- mate-of-thh-art deep learning methods on modeling the continuous-time dynamics and achieves remarkable performance in the prediction and interpolation of irregular or sporadic time series.
The rest of this paper is organized follows. In Section we first present the continuous-time variants of VAE loss, and then derive a continuous-time IWAE loss to train continuous-ting state- space models with deep neural networks. In Section B we propose the deep learning structures of VSDN. Comprehensive experiments are presented in section and and conclusion is given in section 5
CONTINUOUS- TIME VARIATIONAL BAYES
In this section, we first introduce the basic notations and formulate our problem. We then define the continuous-time variants of the Variational Auto-Encoding (VAB) and Importance- Importance-Weighted Auto- Encoding (IWAE) lower bounds to enable the efficient training of our models. Due to the page limit, we present all deductions in Appendix A
2.1 BASIC NOTATIONS AND PROBLEM FORMULATION
Throughout this paper, we define X; E Rd as the continuous-time latent state at time and Yn, e dd the neeh tiscrete-time observed data time fn. d1 and d2 are the dimensions of the latent state and observation, respectively. Xet is the continuous trajectory before time t and X is the path up to time Yny:ng is the sequence of data points and X1,, :fng the continuous-time state trajectory from bng to lng. Di = (Yal,, < l} is the historical observations before and W, = (Yx, 2 1} is the current and future observations. For simplicity, we also assume that the initial value of the latent state is constant. The results this paper can be casily extended to the situation that the initial states are also random variables. Given K data sequences {y(t) }. 1. K, the target of our study to learn an accurate continuous-time generative model G that maximizes the likelioo:ooo::
K
G arg max
log Py Pv(v
(1)
K
g
2

---

Under review as a conference paper at ICLR 2021
For Multivariate sequential data, there exists a complicated nonlinear relationship between the ob- served data and the unobservable latent state, which can be either the physical state of a dynamic system the low-dimensional manifold of data. In our study, the latent state evolves in the contin- uous time domain and generates the observation through some transformation.
2.2 CONTINUOUS- TIME VARIATIONAL INFERENCE
In order to capture the underlying stochastie process from sporadic data, we design the generative model as neural continuous-time state-space model, which consists of latent Stochastic Differ- ential Equation (SDE) and a conditional distribution of the observation. The latent SDE describes the stochastic process of the latent states and the conditional distribution depicts the probabilistic dependency of the current data with the latent states and historical observations:
dX; -X, (x; ()d + R)()W,)),,
(3)
where Hgg and Rg are the drift and diffusion functions of the latent SDE. W, denotes the Wiener process, which is also called standard Brownian motion. To integrate the information of the observed data, Hg is the function of the current state X₁ and the historical observations V/. However, Rg only uses the historical data as input. It not beneficial to include X, as the input of the diffusion function, as it will inject more noise into gradients of the network parameters. A detailed example and analysis of the noise injection problem is given in Appendix B (--) a parametric family of distributions over the data and f()) is the function to compute the parameters of &. With the advance of deep learning methods, we parameterize Hg. Rg and f(-) by deep neural networks.
Continuous-Tine Auto-Encoding Variational Bayes: The exact log- legliiklihhodd of the generative model is given as
T log log log (X - Xt,dXX
which does not have the closed-form solution in general. Therefore, G can not be directly trained by maximizing lig-liiel To overcome this difficulty, an inference model Q is introduced to depict the stochastic dependency of the latent state observed data. Similar the generative model, Q consists of a posterior SDE:
dx, Ho Mr, t.;t)tt + t)dW,
(5)
where HQ is the posterior drift function. Different from Hg. He also uses the future observation Y, as the input and therefore the inference model Q induces the posterior distribution (e,
Based on Auto-Encodee euco-En Variational Bayes (Kingma & Welling 2014), it straightforward to introduce a continuous time variant of the VAE lower bound of the ligelioidhood:
AV = 3KL(Po)
Σ Epo(x,)) log Po(u-Vimn-,,,, Xt.).
(6)
==1
1 t. KI(PolIPo) = [/
(x) ( H&) Rg R (Hse)-Hs))))))))))))) HHs)
where (g( and Pe(Xct,) are the probability density of the latent states induced by the prior SDE Eq. (2p and the posterior SDE Eq. (5). (-(--l-) denotes the KL divergence between two distributions and / is hyper-parameter to weight the effect of the KI. terms. In this paper, we fix as 1.0 and AV is the original VAE objective (Kingma Welling 2014). 3-VAE (Higgins) ctt al. 2017 Burgess et al. 2018). is shown that larger 3 encourage the model learn more efficient and disentangled representation from the data. Eq. (5) is restricted to having the same diffusion function Eq. A feasible LV AE not be defined train VSDN-SDE without this restriction, as the KI. divergence of two SDES with different diffusions will be infinite (Archambeau et al. 2008).
The VAE objective has been widely used for discrete- tieeettee-mmeeeee stochastic recurrent modals, such as LFADS (Sussillo al. 2016), VRNN (Chung al., 2015) and SRNN (Fraccaro et al.) 2016). The
3

---

Under review as a conference paper at ICLR 2021
major difference between these models and our work is that incorporate continuous- time latent state into our model while the latent states the discrete-tin tetrett--imeeeee models evolve only at distinct and separate time slots.
Time Importance Weighted Variational Bayes: Lv (31i:)) equals the exact log- likelihood when Pe. of the inference model is identical to the exact posterior distribution induced by the generative model. The errors of the inference model can result in the looseness of the VAE loss for the model training. Under the framework of werrehhee-Weighee Auto-Encoode (IWAE) (Burda et al. 2016 Cremer et al. 2017), we can define a tighter evidence lower bound:
Contimuous-
K
C/w AE(1m)) =B L St,," 2 (olsst.))
log
wk II Po(ulvim-
(8)
K
k=1
where the importance weights satisfy the following SDE:
d log wh =dlog Po(a 1 (He Ho) (Rg (Ho Hggdt
Pa
- -(He [Rgg
(9)
Given the variational auto-encoding lower bound LVVR(-) and the importance weighted auto- encoding lower bound C A( for the continuous-time generative model, the tightness of the lower bounds are given by the following inequality:
log (V1i:) 2
[K+1 AIW 2 JK LFW) LvvE()),
(10)
for any positive integer K. Consequently, ZK (.()) is infinite if the diffusions of Eq. (2) and Eq. different. In implementation,
notice that the training of our models by LRw AE is
not stable, possibly due to the drawbacks importance sampling and the Signal- Signal-To-Noise To-Noise problem
(Rainforth et al. 2018). To alleviate the problem, we train our model by a convex combination of the VAE and IWAE losses:
(1 (1) + oC oL AE a E (0,
(11)
With the use reparameterizztion (Kingma & Welling 2014), both Ly ( and (M))) differentiable with respect the parameters of the generative and inference models. Therefore, they can be applied to train continuous-time stochastic models with deep learning components.
3 VARIATIONAL STOCHASTIC DIFFERENTIAL NETWORKS
We propose a new continuous-timg stochastic recurrent network called Variational Stochastic Dif- ferential Network (VSDN) (Figure [T. VSDN introduces the latent state to capture the underlying unobservable factors that generate the observed data, and incorporates efficient deep learning struc- tures to compute the components in the generative model Eq.
and inference model Eq.
X
X.
a-
Forward ODF-RNN
Nan
Backward
Re,pre
Re.past
h
Decoder
D
D
N.
Dense Network
Yn+1
Y
Yg
Y
V,
Y
Figure 1: Model Architectures of (a) VSDN-F (filtering) (b) VSDN-S (smoothing).
4

---

Under review as a conference paper at ICLR 2021
Generative Model G: Inside the generative model, the latent SDE Eq. (2) depicts the dynamics of the latent state trajectory controlled by the historical observations V/. Both the drift and diffusion functions have the dependency V/. Therefore, we first apply a forward ODE-RNN (Rubanoval et al. 2019) to embed the information of historical data into the hidden feature h t.pre- Two feed- forward networks are defined to compute drift and diffusion respectively. The decoder network further computes the parameters of the conditional distribution in Eq. (3) by the concatenation of the latent state and forward feature
Hg NXx, T. (D:: RS = exp(NNaa exp(Nars((Repmm))) (Re.pr)))
Po(YID),, 2YY, = -ov.I=D((AA)) D([X, Th p.peee)...
(12)
Inference Model @: We propose two types of inference models in VSDN; a filtering model, and a smoothing model. AVAE(UIm and C/ AF (111m) equal the exact log-likelihood when (o(Xxt,) is identical to the exact posterior distribution Pe the whole data sequence compute He at time. According to d-separation (Bishop 2006), the latent state X₁ is dependent on both the historical data Yi and future observations Y. Therefore, we first define Q as a smoothing model by introducing backward ODE-RNN embed the information of the future observations into a hidden feature h The drift function is computed as:
The inference model must process the
He NNraaff((e h pyre + I)), h ODE-RNN, (Y,;
(13)
In real-world applications, it is sometimes possible have close performance in inference with- out processing the future observations. Besides, the future measurements are intractable in online systems. Therefore, also design filtering inference model that infers the latent state from the historical and current data. The drift of the filtering model is given as:
He {
Hg Narist((x,, Ti re + r.pose)) if there is y at otherwise
(14)
where h post is the post-observation updated feature of the forward ODE-RNN (Rubanova et al, 2019). The filtering model does not have to include a backward RNN to process the huture observa- tions and thus its running speed is faster.
The whole architectures of VSDN with filtering Q (VSDN-F) and smoothing Q (VSDN-S) are shown in Figure (a) (b). The inference model and the generative model share the drift network. This strategy can force the ODE-RNNs to embed more information into the hidden features and reduce the model complexity.
VSDN consists of generative model and an inference model. The generative model is an online predictive model which can recurrently predict the future values of the sequence. The inference models can be applied to either filtering or smoothing problems of the latent states accord- ingly. Furthermore, the smoothing inference model infers the latent state trajectory from the whole sequence, which can be further used in Eq. (3) to synthesize missing data. Therefore, the smoothing inference model is capable of offline interpolation. The motivation of this paper is to design effi cient continuous-time stochastic recurrent model. Therefore, VSDNs only use the generative model recurrently predict the future values in the experiments.
Applications:
Discussions: VSDN has higher flexibility and model capability than current continuous-ting deep learning models in modeling the sporadic sequences. LatentODE (Chen et al. 2018) and ODE (Yildiz et al. 2019) encode the information of the time series into the initial values of the latent state trajectories and neglect the variance in the latent state transition. This strategy is impractical and inefficient in real-world applications, as it requires the initial latent states to disentangle the property of the long sequence. Furthermore, LatentODE, ODE2VAE are offline models, as the encoder used during training of these models can not be directly used for online prediction. In contrast, VSDN defines a latent SDE controlled by the historical observations and recurrently integrates the informa- tion of the sequence along the time axis. It is more efficient than the initial state embedding and is also applicable in online prediction. GRU-ODE (De Brouwer et al. 2019). ODE-RNN (Rubanoval et al. 2019) and NCDE (Kidger et al. 2020 also utilize recurrent scheme but does not explicitly model the stochasticity of the underlying latent state. Therefore, they are less capable than VSDN in modeling the complicated stochastic process of the irregular data.
5

---

Under review as a conference paper at ICLR 2021
4 EXPERIMENTS
In this section, we conduct comprehensive experiments to validate the performance our models and demonstrate its advantages in real- weal-world applications. We compare the performance of VSDN with state-of- of-thh-art continuous time recurrent neural networks (i.e. ODE-RNN (Rubanova et al, 2019) and GRU-ODE (De Brouwer al. 2019)), LatentODE (Chen al. 2018 and LatentSDE (Li et al. 2020
4.1 HUMAN MOTION ACTIVITIES
We first evaluate the performance of different models on the prediction and interpolation problems for human motion capturing. For a given sequence of data points sampled irregular time intervals, the prediction task is defined to estimate the next observed data in the time axis, and the interpo- lation task is defined to recover the missing parts of the whole data trajectory. In both prediction and interpolation tasks, only the generative models of VSDNs are evaluated. The experiments are conducted on the following datasets:
Human3.6M (Ionescu et al., 2014): We apply the same data pre-processing as (Martinez et al.) 2017), after which the data frame each time is 51-dimensional vector. The long data sequences are further segmented by 248 frames.
We follow the data pre-processing in (Liu et al. 2019). In each data frame, human activity is represented as 62-dimensional vector and each dimension of the frames is normalized by global mean and standard deviation. The long data sequences are further segmented by 300 frames.
CMU MoCar
After data pre- rre-processing randomly remove half of the frames in the data sequence as missing data. To quamtify the model performance, we consider two evaluation metries: one is the negative log- ligliikei (NLL) per frame; the other is the frame-level mean square error (MSE) between the ground-true and estimated values. The model configurations are given Appendix C The model performance is shown in Tables and
VSDN incorporates SDE to model the stochastic dynamics, and also applies a recurrent structure embed the information of the irregular time series into the whole latent state trajectory. With these advances, VSDN outperforms the baseline models in both the prediction and interpolation tasks. VSDN has much smaller negative log- likelihoo, which indicates that can better model the underlying stochastic process of the data. Furthermore, VSDN trained by IWAE losses has similar and sometimes better performance than those with VAE losses. As the latent state the inference model has stochastic dependency on the future observations, VSDN-S using the smoothing model has slightly lower NLL and better choice than VSDN-F using filtering model.
Table I: Model performance on Human3.8 3.6M dataset
Prediction
Interpolation
NLL
MSE
NLL
MSE
LatentODE
45.34 + 0.85
1.6421 + 0.041
-45.31 + 0.85
1.6477 + 0.041
LatentSDE
-63.01 + 1.26
0.8278 + 0.059
-62.93 + 1.28
0.8311 + 0.060
GRU-ODE
-93.70
2.34
+ 0.062 0.3201
-93.71
2.34
0.3207 - 0.062
ODE-RNN
-93.78
1.48
0.2981 + 0.075
-93.78
= 1.48
0.2984 + 0.076
0.076
VSDN-F (VAE)
122.64 2.79
0.2373 1 0.064
122.62 1 2.79
0.2367 1 0.064
VSDN-S (VAE)
126.93 3.35
0.2374 + 0.086
126.88
3.35
0.2368 + 0.086
VSDN-F (IWAE)
125.55
6.64
0.1751 1 0.092
-12..51 6.62
0.1746 1.746400000 0.092
VSDN-S (TWAE)
127.12 1 5.19
0.1797 0.073
5.17 -17.008
0.1790 1 0.073
hhtp://mnccup.esccm..duu

---

Under review as a conference paper at ICLR 2021
Table 2: Model performance on MoCap dataset
Prediction
Interpolation
NLL
MSE
NLL
MSE
DE
14.99 + 1,64
49.51 + 0.38
14,91 + 1.81
49.88 + 0.39
LD
-59.83 - 2.13
30.11 - 0.28
-60.13 + 2.59
30.43 - 0.31
GRU-ODE
-51.83 = 0.48
32.77 0.11
-51.91 = 0.49
33.17 0.12
ODE-RNN
-51.75 1.16
31.81 + 0.21
-51.82 1.18
33.22 0.21
VSDN-F (VAE)
110.71 3.92
20.64 + 0.39
-111.40 3.88
20.95 1 0.39
VSDN-S (VAE)
114.31:4.44
19.05 0.59
4.44
19.35 = 0.59
VSDN-F (TWAE)
20.47 1 0.53
-110.441 5.33
20.76 1 0.53
VSDN-S (IWAE)
114.57 1 2.58
19.84 0.18
-115.24 2.50
20.12 1 0.18
Visualization: We further compare different models qualitatively through the visualization of the interpolated human skeletons in Figure VSDN models are able to generate vivid skeletons that are closer to the ground-true ones. Instead, ODE-RNN and GRU-ODE can not interpolate the postures correctly (c.g the angles of arms in each frame are significantly different from the ground-trus truuddtruu ones). We also observe that the motions generated by VSDNs are smooth and closer to the real data, while there large vibration in the movements generated by the baseline models. The videos of these human motions are provided in supplementary materials.
Ground-Trus
ODE-RNN
GRU-ODE
F(WWE)
Figure Visualization for human skeleton interpolation of different models.
4.2 TOY SIMULATION AND CLIMATE PREDICTION
We conduct additional experiments on two sporadic time series datasets in (De Brouwer et al. 2019);
Double-Oui The Double-OU dataset consists of data sequences synthesized by 2- dimensional Comstein-Ulleneee process, which is classic stochastic differential equa- tions in finance and physics.
htnp://ggtthaaccmm/ddeeowwwngggg ode bayes

---

Under review as a conference paper at ICLR 2021
USHCND The United State Historical Climatology Network (USHCN) dataset contains daily measurements of 5 climate variables from the meteorological stations in United States. In our experiment, use the pre-processed subset of the data given in (De Brouwer et al. 2019).
Compared with the previous experiments, the data in Double-OU and USHCN are not only sampled at irregular times, but also have missing dimensions at each sampled frames. That a data sequence is sparse in both time axis and frame dimension. We evaluate the model performance in predicting future values based on the sporadic observations.
The results are shown in Table 3 All VSDN models outperform the baseline ones. On the USHCN dataset, VSDN-S has better NLL than VSDN-F when using either VAE or IWAE losses in the training processes. VSDNs trained by IWAE loss also have smaller NLL than those trained by VAE loss. However, when running on the Double-OU dataset, the training with IWAE performs slightly worse than the training using the VAE loss. This possibly caused by the randomness of the training process, as Double-OU process is a very simple stochastic differential equation and all VSDNs have the smallest errors in the prediction tasks.
Table 3: Model Performance on Sporadic Time Series
Doublc-OU
USHCN
NLL
MSE
NLL
MSE
LatentODE
0.351 + 0.023
0.1201 + 0.0071
1.319 + 0.156
0.772 + 0.099
Lat
0.334 0.010
0.1118 + 0.0029
1.304 + 0.083
0.748 + 0.116
GRU-ODE
-0.997 0.021
0.0080 1 0.0004
0.940 0.058
0.443 0.067
ODE-RNI
-1.002 +
0.0082 + 0.0003
0.866 + 0.057
0.397 + 0.064
VSDN-F (VAE)
1.025
0.0065 + 0.0003
0.384 - 0.060
VSDN-S (VAE)
-1.145 0.013
0.0065 0.0002
0.716 0.113
0.390 0.057
VSDN-F (IWAE)
-..14 + 0.027
0.0065 + 0.0004
0.661
+ 0.096
0.370 + 0.062
VSDN-S (IWAE)
- -1.139 0.019
0.0065 + 0.0003
0.654 + 0.084
0.381 + 0.058
NumTraj-S
NumTraa -10
NumTraj-115
NumTraj-20
13500
12230
10003
30000
1530
7003
0.00
1530
VSDN- (300)
VSON Sholt)
VSBK Fivie)
VSBN SAAC
4000
3000
5000
5000
5003
Epoch
Epuch
Epoch
Epoch
NumTrej-5
NumTrr:-11 -10
NumTraj-15
NumTraj-20 20
33000
11200
10003
20000
5300
0030
3002
5000-
$300
1590
5300
5000
15.00 (AVAL)
5002
4300
4230
464
8X0
Epoch
Epoch
Epoch
Epach
Figure 3: Training processes of our models with respect to the different number of sampled latent state trajectories. (UP: training set; Bottom: validation set)
4.3 QUANTITATIVE STUDIES
In order to better understand the properties of VAE and IWAE losses in training VSDNs, we conduct comprehensive quantitative evaluation by varying the number sampled trajectories when comput-
hhtps:/cciac.essseeeee////// www.eiiiiiiessiiieeee.... www.-eeiie.......oveeiie.eee hww.sddwwww.lgoooee//////iaaaana doc.html
8

---

Under review as a conference paper at ICLR 2021
ing these losses. We visualize the AV and L AE of VSDN trained for 40 epoches the Human3.6M dataset in Figure 3 As the VSDN-S contains both forward and backward ODE-RNNs, it is more difficult to train than VSDN-F. The looseness of Lv AE further increases the training dif- ficulty and results a worse lower bound of VSDN-S (VAE). Therefore, VSDN-S (VAE) requires more epochs converge during the training. For the other cases, we observe that the LK LIWAE is tighter than Lv AE in training when the number of trajectories is small. Therefore, L/W AE has faster convergence in training our models. For large number of trajectories, LV has similar tight- ness as L AE in training set.
5 CONCLUSIONS
In this paper, we propose a continuous-time stochastic recurrent neural network called VSDN to learn the continuous-time stochastic dynamics from irregular or even sporadic data sequence. We provide two variants, one is VSDN-F whose inference model is filtering model, and the other is VSDN-S with smoothing inference model. The continuous-time variants of the VAE and IWAE losses are incorporated to efficiently train model. We demonstrate the effectiveness of VSDN through evaluations studies on different datasets and tasks, and our results show that VSDN can achieve much better performance than state-of-the-an continuous-time time deep learning models. In the future work, we will investigate along several potential directions: First, we will apply our models to higher dimensional and more complicated data, such as vidcos, which are more challenging model yet, especially under the premise of increasing demand for producing videos in high resolution and frame-per-seede
seee--ee-ssccodd second (FPS); Second, as stochastic differential equations are the base of many significant control methodologies, we will try to further extend the capacity of our models such that they can be used in precise control scenarios.
REFERENCES
Cédric Archambeau, Manfred Opper, Yuan Shen, Dan Cornford, and John S. Shawe-taylor. Varia- tional inference for diffusion processes. In Advances in Neural Information Processing Systems 20. 2008.
Christopher M. Bishop. Pattern Recognition and Machine Learning (Information Science and Statis- fics). Springer-Verlag, Berlin, Heidelberg, 2006.
Yuri Burda, Roger Grosse, and Ruslan Salakhutdinov. Importance weighted autoencoders. ArXiv, fbss/1509000199 2016.
Christopher Burgess, Irina Higgins, Arka Pal, Loie Matthey, Nick Watters, Guillaume Desjardins, and Alexander Lerchner. Understanding disentangling in VAVE 2018.
Tian Qi Chen, Yulia Rubanova, Jesse Bettencourt, and David K Duvenaud. Neural ordinary dif- ferential equations. In Advances in Neural Information Processing Systems 31, pp. 6571-6583. 2018.
Junyoung Chung, Kyle Kastner, Laurent Dinh, Kratarth Goel, Aaron C Courville, and Yoshua Ben- gio. A recurrent latent variable model for sequential data. In Advances in Neural Information Processing Systems 28, pp. 2980-2988. 2015.
Chris Cremer, Quaid Morris, and David Duvenaud. Reinterpreting importance-weighted autoen- coders. In International Conference Learning Representations (ICLR)- Workshop Track, 2017.
Edward De Brouwer, Jaak Simm, Adam Arany, and Yves Moreau. GRU-ODE-Bayes. Continuous modeling of sporadically--bsereed time series. In Advances Neural Information Processing Systems 32, pp. 7379-7390. 2019.
Michaël Defferrard. Kirell Benzi, Pieme Vandergheynst, and Xavier Bresson. FMA: A dataset for music analysis. In The 18th International Sociery for Music Information Retrieval Conference (ISMIR), 2017.

---

Under review as a conference paper at ICLR 2021
Marco Fraccaro, Søren Kaac So nderby, Ulrich Paquet, and Ole Winther. Sequential neural models with stochastic layers. In Advances in Neural Information Processing Systems 29, pp. 2199-2207, Curran Associates, Inc., 2016.
Jung-Su Su Ha, Young-Jin Park, Hyeok-Joo Chae, Soon-Seo Park, and Han-Lim Choi. Adaptive path- integral autoencoders: Representation learning and planning for dynamical systems. In Advances in Neural Information Processing Systems 31, pp. 8927-8938. 2018.
Pashupati Hegde, Markus Heinonen, Harri Lähdesmäki, and Samuel Kaski. Deep learning with differential gaussian process flows. In 22nd International Conference on Artificial Intelligence and Statisties, volume 89, pp. 1812-1821, Apr 2019.
Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. 3-VAE: Learning basic visual concepts with a con- strained variational framework. In International Conference on Learning Representations (ICLR), 2017.
Catalin Ionescu, Dragos Papava, Vlad Olaru, and Cristian Sminchisescu. Human3.6m Large scale datasets and predictive methods for 3d human sensing in natural environments. IEEE Transactions Pattern Analysis and Machine Intelligence, 36(7):1325 130-1-3399999,,, jul 2014.
Junteng Jia and Austin R Benson. Neural jump stochastic differential equations. In Advances in Neural Information Processing Systems 32, pp. 9843-9854. 2019.
M. Jørgensen, M. Deisenroth, and Hugh Salimbeni. Stochastic differential equations with variational wishart diffusions. ArXiv, abs/2006. 14899000..484.,... 2020.
Patrick Kidger, James Morrill, James Foster, and Terry Lyons. Neural Controlled Differential Equa- tions for Irregular Time Series. ArXiv, abs/2005.08926, 2020.
Diederik P Kingma and Max Welling. Auto-encoding variational bayes. In International Conference on Learning Representations (ICLR), 2014.
Lingkai Kong. Jimeng Sun, and Chao Zhang. SDE-Net: Equipping deep neural network with un- certainty estimates. In Proceedings of the 37th International Conference on Machine Learning, 2020.
Xuechen Li, Kam Leonard Wong, Ricky Q. Chen, and David Duvenaud. Scalable gradients for stochastic differential equations. In 23rd International Conference on Artificial Intelligence and Statistics, pp. 3870-3882, Aug 2020.
Xuanqing Liu, Tesi Xiao, Si Si, Qin Cao, Sanjiv Kumar, and Cho-Jui Hsieh. How does noise help robustness? explanation and exploration under the neural sde framework. In IEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR), June 2020.
Yingru Liu, Dongliang Xie, and Xin Wang. Generalized boltzmann machine with deep neural struc- ture. In The 22nd International Conference on Artificial Intelligence and Statistics (AISTATS), volume 89, pp. 926-934, Apr 2019.
Jakob H Macke, Lars Buesing, John P Cunningham, Byron M Yu, Krishna V Shenoy, and Maneesh Sahani. Empirical models of spiking in neural populations. In Advances in Neural Information Processing Systems 24, pp. 1350-132 1358. 2011.
Julieta Martinez, Rayat Hossain, Javier Romero, and James J. Little. A simple yet effective base- line for 3d human pose estimation. In IEEE/CVF International Conference on Computer Vision (ICCV), 2017.
Stefano Peluchetti and Stefano Favaro. Infinitely deep neural networks as diffusion processes. In 23rd International Conference on Artificial Intelligence and Statistics, volume 108, pp. 1126 1136, Aug 2020.
Tom Rainforth, Adam R. Kosiorek, Tuan Anh Le, Chris J. Maddison, Maximilian Igl, Frank Wood, and Yee Whye Teh. Tighter variational bounds are not necessarily better. In In Proceedings of the 35th International Conference on Machine Learning, pp. 4274-2244 2018.
10

---

Under review as a conference paper at ICLR 2021
Yulia Rubanova, Tian Qi Chen, and David K Duvenaud. Latent ordinary differential equations for -reeularly-sampled time series. In Advances in Neural Information Processing Systems 32, pp- 5321-5331. 2019.
Tom Ryder, Andrew Golightly, A. Stephen McGough, and Dennis Prangle. Black-box variational inference for stochastic differential equations. In Proceedings of the 35th International Confer- ence on Machine Learning. pp. 4423 1423-4432, 2018.
Simo Särkkä, Jouni Hartikainen, Isambi Sailon Mbalawata, and Heikki Haario. Posterior inference parameters of stochastic differential equations via non-linear gaussian filtering and adaptive MCMC. Statistics and Computing, 25(2)427-437, Mar 2015.
David Sussillo, Rafal Józefowicz, L. F. Abbott, and Chethan Pandarinath. LFADS latent factor analysis via dynamical systems. ArXiv, 0bs//600.003155 2016.
Belinda Tzen and Maxim Raginsky. Theoretical guarantees for sampling and inference in generative models with latent diffusions. In 32nd Annual Conference on Learning Theory, volume 99, pp. 3084-31 14, Jun 2019a.
Belinda Tzen and Maxim Raginsky. Neural stochastic differential equations: Deep latent gaussian models in the diffusion limit. ArXiv, abs/1905.00883. 2019b.
Carl Vondrick, Hamed Pirsiavash, and Antonio Torralba. Generating videos with scene dynamics. In Advances in Neural Information Processing Systems 29, pp. 613-621. 2016.
Bao Wang, Zuoqiang Shi, and Stanley Osher. Resnets ensemble via the feynman-kac formalism to improve natural and robust accuracies. In Advances Neural Information Processing Systems 32, pp. 1657-1667. 2019.
Cagatay Yildiz, Markus Heinonen, and Harri Lahdesmaki. vae: Deep generative second order odes with bayesian neural networks. Advances in Neural Information Processing Systems 32, pp. 13412-13424 2019.
Byron M. Yu, John Cunningham, Gopal Santhanam, Stephen Ryu, Krishna V. Shenoy, and Maneesh Sahani. Gaussian-process factor analysis for low-dimensional single-trial analysis of neural population activity. Journal of Nearophysiology, 102(1):614-635, 2009a.
Byron M Yu, John P Cunningham, Gopal Santhanam, Stephen I. Ryu, Krishna Shenoy, and Maneesh Sahani. Gaussian- factor analysis for low-dimensional single-trial analysis of neural population activity. In Advances Neural Information Processing Systems 21, pp. 1881- 1888. 2009b.

---

Under review as a conference paper at ICLR 2021
A DEDUCTIONS OF CONTINUOUS-TINE EVIDENCE LOWER BOUND
A.1 PRELIMINARIES OF STOCHASTIC DIFFERENTIAL EQUATIONS
During the model design and implementation, will use the Euler Maruyama method to discretize the stochastic differential equation. The details are given as follows.
Lemma (Discretization of SDE). For a SDE dX = H(X, t)dt + R(t)dW, we can discretize it as
X Xx + H(X,, la A + R(4))VAl,,
(15)
where & (0, tx kåt and At the sampling interval. Eq. 15 converges to the original SDE when At + 0.
Lemma 2. The state XR+1 in Eq.
15 follows the conditional Gaussian distribution
P(X Xx) = NiXx + H(X))At, AIR(X
( R(x)).. The joint distribution of the sate se-
quence X1.K of Eq. (15) is given by
R-1
P(XEX Xo) x xexp (-0.5 0.5 Σ
(Xa+1 mmx) = (Xx+1 - -ma)).
(16)
h=0
where mi X + A()))AL and Eg AIR(Xx))
A.2 DEVIATION OF LVAE
The proof is similar the evidence lower bound in (Archambeau al. 2008). By applying Jensen's inequality, can obtain that:
/ st II X1,
"
logPg(yim)) log
= =log / Po(xg())
Po(Xst.) IT 11. -1, X1, dx dK
Pe(X
Po(Xs log Po(X [[ Py dxst
Po(Xst.)
-/ Po(X ) log
Pg(Ne dXa, + / Po(X log
log
II I.
Pe(Xst.)
- + + Epo(X,)) log 1o
The next step derive the KL divergence term for the prior and inference SDEs. After discretiza- tion into K points via Lemma
[ the KL divergence of the two SDEs in VSDN-SDE will be:
KL(PolPP) f Po(X log Po(XI:K)
dXi:k
(x(X
K-1
Po(Xi.)) log Po(X Xx) (11KK
K-1
Σ
/
Po(Xi:K log ( (8x)
Po(Xx+1lXp dXi:K
Po(N Xx)
log Po(Xu+1|Xa dX1.x Pe (
Pe(
/ [Pe(xH+lX))P(Xx) log Po(Xe+1lXp dl dXxdXX
Po(X Xx,)
/ Po(X)) KL XP dXx
KX PaP
12

---

Under review as a conference paper at ICLR 2021
where Po(Xss is the marginal distribution of X in the inference SDE. According lemma 2 and the KL divergence of two Gaussian distribution, we further have
KL (Po(Xx+1 det Ik.g - d) +log det k
(X+) (Ir(E (ma& m) " (ma. " mx,0)
1 (Te Ro Rb) + At(Hg - (Rg
R d)
RR - Ho) + log R
where d is the dimension of Xk+1.
When we restrict Rg Rgg we have
K-1
KL(PelIPg) -2 (x (Rg (1lg HO)
Σ
When we set At + O, the discretized SDEs converge to the original SDEs and K L con- verges to:
PT) Epoix. (14e- (oo0-HH)) (Ho Hq)) dl
The expectation operator removed Hg. He and Rg are independent with X If Rg does not equal to Rg, we have
PL(Po] = lim Exxx)a(Xi) ((HS - H²) (RG 112) Al At,
1
const
k=0
=0
A.3 DEVIATION OF LIWAE
Given X = -xx + HQAt + Rg R AtE = mx,e + Rgv Ate, we have:
log log Pg(Is(a)
K-1
Po(Xet1lAd
Σ
log
Po(osta)
Po(Xx+1|Xp
k=0
K-1
(X++1 - m E (X - mp.g) + (X++1 -mm,mm mx.e)* " mx.e)
K
- Ho)oi+ RoV ROV [AtRG " ( -1) RoVAte
K=
(RoV Ate) [AtR Rov (RoVAte) (ROVV-
K-1
(Ho (Rg (Ho Hg) - 2(H (Ro VAis (17)
2 K=
Let Al + 0, we have
log w / (He - (H) (Rg - Hg)dt (HH--Ho) dWe,
(18)
which is equivalent
d llg w = (Hg - H₂) (2) H&)dt (Ho - (Ro)
(19)
B ILLUSTRATION OF THE NOISE INJECTION OF R(X,)
In the section, give example illustrate the noise injection problem when we include X, as the input for the diffusion function R(X,) in Neural SDE. For simplicity, we consider the scalar case (i.e. X, IR).
13

---

Under review as a conference paper at ICLR 2021
B.1
CASE A: R. INDEPENDENT OF X
Consider the following neural SDE:
dX, Hol (Xx:tdtt + Ra(t)dW,
(20)
where H& and Rg are neural networks. & denotes the parameters of the drift network and 0 denotes the parameters of the diffusson network.
Xst
X Xzat
X X3ar
Hą. Re
Hg. Re
L
Figure 4: A example to show the noise injection problem of R(X,).
Now consider the following example (shown in Figure that we have to compute the gradient of the loss at t = 3åt with respect to the network parameters, where the neural SDE discretized by Fuler-Maruyama method:
Xa: =Xo Ho 0)4/ (o Ale,
(21)
X2s: =XA: + (T)x + Ro(A Ate
(22)
X3s: -X2si H& (X 2 Ro (2At)) v Atss.
(23)
where en ~ ~N(0,1). straight forward to prove the following lemma. For notation simplicity,
define Hş(n) (X (x (n - 1) and Rn(n) Ro((n 1)
Lemma 3. Eqs. 27 23 follows the following relationship of the gradients:
8X.A
AL (H₂(n)
(24)
0x (n
0X(n-1)A
Therefore, the gradients of the parameters in the drift and diffusion functions can be given by:
OC
OL
0Xast
OL ONasr 0X2s:
OL 0Xast 0X2st 0X
06
0X3s
0XsAr OX2sr
O6
0X OX2se OXst OΦ
OL
(3)(3)
dL
(3)
(2)
At 0X:A
+ At
At
OX3Ml
d6
0X2s1
OL
1 At (3)
OH, (2) (H₂(1)
+ 0X3AL
1 At
At.
(25)
0X2s:
0XA:
06
and
0C
OC 0X3s:
OL ONasc 0X2s:
OC 0Xasr 0X2si 0X
+ 0X OX2sc
00
0Nast 00
00
0X ON2st ON 00
aL dRg(3)
OL
(H)
@Ra(2) VAtez
VAtes+ + 0X3st
1 At
OXasc
00
0X
00
+ 0X3At
dL
1+ At aH
[+ At
R (1) (2)
V
(26)
0X2st
OX
20
According to Eq. (25) and Eq. (66), the gradient of & of the drift network is deterministic except the weight of first hidden layer and the gradient of 0 of the diffusion network obstructed by Gaussian noise terms / M/8,, VAle, and VAleg.
14

---

Under review as a conference paper at ICLR 2021
B.2 CASE B: R USES X₁ AS INPUT
Now we consider the case when the diffusion network R also use X, as input. Eq. 24 the following equation:
will change
0Xnst = 1 + At
(Ho(n) 0X (m-1)A
+ Vtte @Ro(n)
(27)
0x (n 1)Ar
0X (1-
Inserting Eq. (27 into Eq.
(5) and Eq. 26 we have
OC
OL 0x3A.
dc 0X3sr 0X2sr + 0Xsar Ox2si 0x 06
OC 0N 0X2s1 OX
dΦ 0X3se do
0Xsar 0X2sc do
dL
(3)
UC
At (H₂(3) 0X2st
(3)(3) (2
At +
+ VAtts 0X2st
At
0X3st dΦ Uc
0X3sc
0Φ
1+ At (3)
aRo (3)
+ At (2)
+ V RRo
OH At.
VAtes 0x281
0X3ar
0X2ss
0Xa:
ds (28)
and
dc
Oc OX321
Oc OX3st 0X2s!
ac 0X32t 0X2sr OXst
+ 0Xasr 0X2st
+ 0Xasr 0x2s: 0Xar 00
00
0Xasr 00
00
0L
(R)
VAts++ dL
(H)(3)
VVAes RRo dRo V A/22
1 At 0X2st
0Xasr 00
0xast
0x2st
00
OL [1+ At
(H
Ro (3)
At (2)
Ro
RRo(1)
+ 0X3st
+ V
1 ++41
+ Vt
V Ate1
0X2sr
0X2s1
0XA:
0X
00
(29)
According to Eq. 28 the gradient of á is now also corrupted by noise terms (i.e. V Aea and AtegE3), while in previous case it is deterministic. What's worse, more noise terms are added into the gradient of When we train our models in long data sequence, these injected noise terms will cause a large variance of the parameters' gradients. Therefore, we can conclude that introducing X, into the diffusion function is not beneficial.
C MODEL CONFIGURATION
C.I HUMAN MOTION ACTIVITIES
For all the models, the as 0.25. The dimension of hidden features of ODE-RNN and GRU-ODE 512 and the dimension of latent states 128. A single-layer feed-forward network with 128 Relu units is defined to compute the initial states of the latent state. For LatentSDE, the posterior initial state is computed by using the encoding feature of backward ODE-RNN. The number of latent state trajectories generated to compute VAE and IWAE losses is 5.
network contains one hidden layer with 256 Relu units. At is set
All models are trained by Adam optimizer with learning rate 0.0001 and weight-decay 0.0005. The batch size is 64. Early stopping with 10 epoch tolerance is applied.
C.2 TOY SIMULATION AND CLIMATE PREDICTION
For all the models, the feed-forward network contains one hidden layer with 25 Relu units. At is set 0.1 for USHCN and 0.01 for Double-OU. The dimension of hidden features of ODE-RNN and GRU-ODE is 15 and the dimension of latent states 15 as well. A single-layer feed-forward network with 128 Relu units is defined to compute the initial states of the latent state. The number of latent state trajectories generated to compute VAE and IWAE losses is 5.
All models are trained by Adam optimizer with learning rate 0.0001 and weight-decay 0.0001. The batch size is 500 for USHCN and 250 for Double-OU, Early stopping with 25 epoch tolerance is applied.
15

---

Under review as a conference paper at ICLR 2021
C.3 SPECTROGRAM MODELING
To further evaluate the performance of our model in high-dimensional data, we conduct a brief ex- periment the spectrogram data extracted from the FMA dataset (Defferrard et 2017), which is a large collection of music and songs. We transform the first 500 songs in FMA-small into spectro- gram and then split the spectrogram into segments. Each segment has 100 frames and each frame is 1025 dimensional. Each dimension of the data frame corresponds to specific frequency component of STFT. The definitions of prediction and interpolation tasks are same as those in Section 4.1.
Table 4: Model performance on Spectrogram dataset
Prediction
Interpolation
NLL
MSE
NLL
MSE
LatentODE
0.1356
0.066
0.1369
0.066
LatentSDE
00.1403
0.065
-0.1291
0.066
GRU-ODE
-0.1610
0.058
-0.1603
0.059
ODE-RNN
0.1677
0.058
0.1688
0.058
VSDN-F (VAE)
-0.1969
0.059
-0.1947
0.059
VSDN-S (VAE)
-0.1994
0.059
-0.1981
0.059
VSDN-F (IWAE)
-0.1931
0.059
-0.1935
0,059
VSDN-S (IWAE)
-0.2020
0.059
00.2117
0.059
For all the models, the feed-forward network contains hidden layer with 256 Relu units. The dimension of hidden features of ODE-RNN and GRU-ODE 256 and the dimension of latent states 64. A single-layer feed-forward network with 128 Relu units is defined compute the initial states of the latent state. For LaE the posterior initial state is computed by using the encoding feature backward ODE-RNN. The number latent state trajectories generated compute VAE and IWAE losses is 5. The batch size is 32.
The NLL and MSE (per dim) are shown in Table 4 Our model has similar mean square errors with baselines but has much better NLL, which indicates that our model can better estimate the stochastic process of the data.
16


Reviews:
[{"review_id": "tHdM7JkhOD8", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In this paper the author/s study/ies the fundamental problem of learning continuous-time stochastic dynamics, in the case where the available time series data suffer irregularity and sparseness. The paper assumes that high dimensional data are generated from a system where latent states are observed. The paper tackles the problem \nIn such a settings, it is generally impossible to derive a continuous-time stochastic process which precisely describes the many behaviors of the system under study.\nThe paper develops a method base on Variational Bayes applied to learn flexible continuous-time stochastic recurrent neural network, they call Variational Stochastic Differential Networks. The particualr feature of such model is to capture the stochastic dependency among latent states and observations.\nThe paper provides theoretichal tool, under the form of lower bounds for the efficient training of the neural model.\nTHe author/s state that the results from a rich set of numerical experiments witnesses that the proposed approach \noutperforms state-of-the-art continuous-time deep learning models to solve the prediction and interpolation tasks, in the particular case when irrelugar and sporading time series data is available.\n----------------------------------------------------------------------------------------------------------------------\n\nReason for Score:\nOverall, I vote for rejecting. I like the idea to mix parametric and non parametric components to learn time series dynamics. However, I have many concerns that are made explicit in the cons section.\nHopefully the authors can address my concerns in the rebuttal period. \n\n----------------------------------------------------------------------------------------------------------------------\nPros.\n1) the paper tackles a relevant problem.\n2) I appreciate the idea to combine parametric and non parametric models\n3) the proposed model is clear\n3) numerical experiments, on the selected data sets, witness in favour of the proposed method\n----------------------------------------------------------------------------------------------------------------------\n\nCons.\n1) why other models for time series modeling and forecasting have not been taken into accout? what about dynamic Bayesian networks, hidden Markov models, contiuous time Bayesian networks, and many other ...\n\n2) the paper states the interest is related to high dimensional time series, but the data sets used for numerical experiments, in my humble opinion, are not as such, they are of very small or moderate dimensionality, in terms of number of variables.\n\n3) I found no discussion about the choice of the dimension of the latent space, maybe my fauls and if this is the case I apologize, but still I think that more attention and more motivations must be given in the numerical experiments section.\n\n4) I'm under the impression that the proposed solution is a sensible combination of existing results and thus the work is somewhat of incremental nature, which I do not know whether is relevant for ICLR. \n\n5) I'm confused on the formulation of the inference problems, the paper considers both filtering and smoothing. However, later in the paper, the numerical experiments section, I read about \"prediction\" and \"interpolation\" which in my uderstanding is the same as filtering and smoothing. Thus, I would like the authors to better frame their problem.\n\n6) at page 2, I can not understand the difference between path and history, if they are used as synonims, please do not do it, it increases confusion to the interested reader, try to be more formal and clear as possible to let interested reader to appreciate your contributions.\n\n7) at page 3 I read \"However, RG only use the historical data as input, as we observe no improvement by including the\ncurrent state.\", I would ask to motivate, personally I'm not that surprised by this but I would like the paper to help understand the why of this occuring.\n\n8) In equation (5) you are using the entire trajectory and thus you are performing smoothing, I think the problem/s tackled must be clearly described.\n\n9) at page 5 I read \"In real-world applications, the latent state may not have strong dependency on future observations\". This is somewhat surprising me, which has not strong dependency on which? the latent does not influence the future? the future does not tell much about latent? This sentence in obscure to my elementary mind, and thus I kindly ask to better clarify on it.\n\n10) Table 1 and 2, please call the macro column filtering and smoothing in place of prediction and interpolation, maybe I completely miss the meaning of the difference you made between prediction and filtering and the difference between smoothing and interpolation. If this is the case, please better clarify on these aspects.\n\n11) NLL and MSE of VSDN-S (VAE) are always better that those achieved by VSDN-F (VAE), this is not that strange, one is using more info than the other, as well as I understood. The same does not apply to VSDN-S (IWAE) and VSDN-F (IWAE), and this is surprising in my humble opinion, maybe not found the optimum?\n\n12) In table 2, NLL is positive for LatentSDE, can you comment on this?\n\n13) The same as 12) in Table 3, I kindly ask to clarify under which circumstances this happens.\n----------------------------------------------------------------------------------------------------------------------\nQuestions during rebuttal period: \n\nPlease address and clarify the cons above \n----------------------------------------------------------------------------------------------------------------------\nSome typos: \n\npage 3: I read \"However, RG only use the historical\", it should read \"However, RG only uses the historical\"\n        I read \"HQ also use the\", it should read \"HQ also uses the\"\n----------------------------------------------------------------------------------------------------------------------\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "hYlZXFgJoUD", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper introduces Variational Stochastic Differential Networks to filter and smooth sporadically observed time series.\n\nThe authors adopt a Bayesian perceptive on the smoothing problem for time series living a latent space and irregularly observed.\nIn particular, the random evolution of the process in latent space is clearly accounted for in the paper by embedding an SDE into an RNN.\n\nThe authors derive a variational loss for their model and describe in fact multiple losses with different improvements such as importance sampling. In the end the authors do report issues caused by excess variance in training and settle for a convex combination of the two losses they propose.\n\nThe paper becomes quite interesting in its experimental part as the authors show the superiority of their method as compared to previous approaches on data sets concerned with Mocap, synthetic OU process data and meteorological data.", "extra_scores": {}, "extra_text": {}}, {"review_id": "KIijs-bxFDt", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper introduces a latent variable model for high dimensional stochastic time-series. The model is akin to a VAE with RNNs that incorporate time-series data. The authors introduce two variants of the model, one which only contains a feedforward RNN (filtering) and another that contains feedforward and feedback RNNs (smoothing).  The authors use two inference procedures for the model, one the standard VAE, and the other importance weighted IWAE.  The work is reasonably clearly presented and the experiments are multiple data sets are a nice addition.\n\n\nI have two primary concerns -- 1) I do not see how the choice of the two inference procedures (which makes up a fair portion of the submission) is well motivated, and  2) I find the arguments for them achieving state-of-the-art performance unconvincing. \n\nfor 1)\nThe use of these two inference methods feels ad hoc. Why not just choose one? Using importance waiting and the standard ELBO does not seem to add to the paper. I see the primary contribution of the paper to be the model, not the inference procedure. If there is an important distinction between the inference methods, and the authors feel that is important to presenting their work, I would devote more space to explaining why there are two approaches --  the details of the objectives themselves could be moved to an appendix if needed.\n\nAdditionally, regarding the inference procedure, it looks like a beta-VAE objective is used (equation 6), but the authors do not say why the hyperparameter beta is needed, and do not cite the Beta-VAE paper (Higgins et al. 2017)\n\nFor 2)  I would like to see more comparing the performance of this model to a stochastic high dimensional. The authors discuss neural ODEs, but there is no comparison against them for performance in the experiment section. If the comparison cannot be made for some specific reason, the authors should explain why.\n\nAnother model which comes to mind that might warrant some discussion/comparison is lfads (Sussillo et al. 2016). \n\nOne other point: I feel similarly to the choice of two models (VSDN-F and VSDN-S) as I do about the IWAE and VAE. I think it would be better to hone in on a stronger take home point -- do one of the methods (say, VSDN-S) achieve better performance than existing methods. Showing this on more tasks or compared to more models would, in my mind, make this a stronger submission.\n\nSome other concerns:\n\nIt is hard to get anything out of figure two. The human skeletons are very small and there are many of them. If the authors wish to show how the distinct positions for each of the methods differ from ground truth, they should remove some of the panels (again, not sure here why the different inference procedure warrants a different panel. I don't think the authors want to communicate how the different inference procedures make an important difference, do they?) An additional task or more clear plot showing accuracy of recovered latents, or accuracy of generated time-series data would be useful.\n\n\nTypo in the final sentence of the first paragraph of the experiments section. You say \"two\" but compare to three things.", "extra_scores": {}, "extra_text": {}}, {"review_id": "7zIteFtLpE_", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper aims to model the complicated continuous time-series by using SDE for modeling the latent state trajectories. The authors claim that using SDE instead of ODE for the latent states has higher flexibility to capture more complex dynamics. Then they propose a continuous-time versions of variational evidence lower bounds (ELBO) which can be trained using ODE-RNNs as the inference networks. The proposed VSDN model has the capability to capture the latent state stochasticity not via the initial states but rather via SDEs, as opposed to other methods like latent ODE and latent SDE. \n\nOverall, the paper is interesting as it employs many recent advances in the field to introduce a richer model for the continuous complex time-series with the capability of taking into account the stochasticity of the latent state dynamics. Nevertheless, I have some questions as follows:\n\n1- The first motivation of the paper to define the VSDN is that the methods based on the neural ODEs cannot model complicated time-series. I wonder why is this claim true? When the underlying latent state is an ODE, the stochasticity of the time-series could be modeled via rich conditional observation distributions. Why do we need to have a double stochasticity, one in latent state and one in the observation data?\n\n2- How many dimensions can VSDN handle? I am wondering how it will work in a high dimensional regime (say dim is several thousands). How many dimension are used for states and observation data?\n\n3- As the VSDN is using SDEs for the latent states, I wonder if it can quantify the uncertainty of the inferred latent state trajectories? \n\n4- Why in eq. (13), $h_{t,pre}$ and $h_t$ are summed? Could they be concatenated instead of summation? \n\n5- Minor comment: I guess $W_t$ in eq. (2) refers to the Wiener process, right? But there is no definition on the text. So, the authors should define it in the paper so that the readers know what it is. \n\n\n########## EDIT ##########\nThe authors have addressed all my questions.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper sits right at the borderline: the reviewers agree that it is interesting and addresses a relevant problem. On the negative side, the presentation could be improved (including some incorrect claims), and the experiments could be strengthened (both in terms of baselines and datasets used). Ultimately, the paper will probably require another round of reviews before it is ready for publication.
Decision: Reject