Neural Markov Controlled SDE: Stochastic Optimization for Continuous-Time Data
Authors: Sung Woo Park
Kyungjae Lee
Junseok Kwon
Venue: ICLR 2022 Poster (2022)
Keywords: controlled stochastic differential equation
time-series prediction

Abstract: We propose a novel probabilistic framework for modeling stochastic dynamics with the rigorous use of stochastic optimal control theory. The proposed model called the neural Markov controlled stochastic differential equation (CSDE) overcomes the fundamental and structural limitations of conventional dynamical models by introducing the following two components: (1) Markov dynamic programming to efficiently train the proposed CSDE and (2) multi-conditional forward-backward losses to provide rich information for accurate inference and to assure theoretical optimality. We demonstrate that our dynamical model efficiently generates a complex time series in the data space without extra networks while showing comparable performance against existing model-based methods on several datasets.

Published as a conference paper at ICLR 2022
NEURAL MARKOV CONTROLLED SDE: STOCHASTIC OPTIMIZATION FOR CONTINUOUS- DATA
Sung Woo Park', Kyungjae Lee?, JJuseok Kwon
School of Computer Science and Engineering, Chung-Ang University, Korea 1,2,3 Artificial Intelligence Graduate School, Chung-Ang University, Korea pswkikk email 1 com, 2kyungjae www.aleelaa...... cau. acuuac @jskwon@cau.
ABSTRACT
We propose novel probabilistic framework for modeling stochastic dynamics with the rigorous use of stochastic optimal control theory. The proposed model called the neural Markov controlled stochastic differential equation (CSDE) overcomes the fundamental and structural limitations of conventional dynamical models by introducing the following two components: (1) Markov dynamic programming efficiently train the proposed CSDE and (2) multi-conditional losses to provide information for accurate inference and to assure theoretical optimality. We demonstrate that our dynamical model efficiently generates a complex time series in the data space without extra networks while showing comparable performance against existing model-based methods on several datasets.
INTRODUCTION
Recently, there has been interest in using continuous dynamical systems to approximate complex time series, Neural Chen et al. (018)), which opened the way for continuous representation of neural networks, have been widely investigated and thoroughly analyzed by Massaroli et al. (2020). As the stochastic generalization of ODE, Neural SDEs Li et al. (2020) have been proposed by regarding intrinsic stochasticity data representations (e.g., stock market data). Since the conventional Neural ODE/SDEs only utilize the initial information of trajectories when propagating dynamics, modelling complex time-series with naive Neural ODE/SDEs has been regarded as inefficient and undesirable choices, as pointed out by Kidger et al. (2020).
To address these problems, Rubanova et al. (2099) presented an auto-regressive model to generalize recurrent neural networks (RNNss to have continuous hidden dynamics with neural ODE. Furthermore, Chen et al. (2018) proposed an encoder-decoder structure with Neural ODE the latent space to irronstruct/preditt complex data representation. Although the aforementioned approaches produce remarkable results, they focus on suggesting additional probabilistic structures rather than improving the learnability of the Neural ODE model itself. Compared to aforementioned approaches, focus on solving the fundamental issues of Neural ODE/SDES. First, we raise two important questions.
Q1) How require additional recurrent networks to model complex time series?
construct an efficient network architecture for Neural ODE/SDE models that do not
Q2) How can we train Neural ODE/SDEs that can utilize richer information of observed sequences accurately generate complex time series?
As SDEs can be posed as stochastic generalizations of ODEs, we focus on a stochastic framework and adopt the stochastic optimal control theory as our primary analysis tool for the rigorous and systematic analysis of the aforementioned problems. Keeping this in mind, the contributions of our paper are to answer the above two questions.
A1) Novel probabilistic framework for stochastic dynamics. We propose a novel neural controlled stochastic differential equation (CSDE) to model the complex stochastic time series, where multiple control agents are defined to construct local dynamics in their private temporal states. With this property, the proposed CSDE incorporates Markov dynamic programming, enables our model to directly infer the complex trajectory data space rather than the latent space without any extra network (e.g., encoders/decodeers). and shows remarkable efficiency compared to existing methods.
A2) Novel conditional losses. We introduce novel Markov forward conditional (MFcond) loss to utilize multi-conditioned dynamics instead of the conventional dynamics determined by partial initial conditions. The proposed MFcood loss makes our method model the complex information of

---

Published as a conference paper at ICLR 2022
time-series data. To impose regularization and to ensure the optimality of control agents, also suggest a novel Markov backward conditional (MBcond) loss.
RELATED WORK
ODE As Latent Probabilistic Model. Rubanova et al. (2019) suggested an ODE-RNN by com- bining RNN with the latent dynamics induced by the Neural ODE. To deal with irregular time-stamps. exponential-decaying of the hidden states was also discussed by Che et al.] (2018). De Brouwer et al. (2019) assumed that the observations are sampled from the stochastic dynamics induced from SDEs and introduced GRU-ODE to approximate the observed stochastic time series.
SDE As Latent Probabilistic Model. Liu al. 2021) incorporated Neural SDEs with recurrent models as primary probabilistic dynamical model to generate stochastic continuous-tiig time latent variables. While this SDE model could describe the stochastic dynamics on the latent space with recurrent structures (e.g., RNN encoder/decoder), it required whole sequence of historical observa- tions as inputs to the model. Unfortunately, this type of formulation leads to non-Markov types of SDEs, which makes it difficult to analyze the probabilistic characteristics of the dynamics. Unlike this model, we focus on the Markov SDEs while maintaining identical objectives.
Neural CDE and RDE. Kidger et al. (2020) proposed data-driven neural controlled differential equation called Neural CDE to incorporate a rough-path analysis theory and model complex time series. Morrill et al.
20021 extended the bough-path theory with Neural RDE to deal with the continuous time series over long time.
Generative SDE Models. Recently, Kidger et al. (222) suggested SDE-based generative adversarial networks (GANs). Park et al. (2021) utilized the temporal conditional Wasserstein distance to construct GANs for time-series generation.
Please refer to Appendix A.1 for additional discussion on related works.
MARKOV NEURAL CONTROLLED SDE
we introduce novel SDE model that considers temporally private agents. In Section 3.2 we propose the Markov-DP-TT TP framework efficiently solve the stochastic optimal control problem with the proposed neural SDE model. Finally, we suggest novel Markov conditional forward and backward losses in Section 3.3 and 3.4 respectively. the Appendix, we provided the detailed technical definitions.
In Section 3.1
3.1 CONTROLLED STOCHASTIC DIFFERENTIAL EQUATIONS
The basic object of our interest a controlled Fe-adapted process X; with multiple control agents = fa MM E A where A denotes the set of admissible control agents. In particular, the stochastic process X. is defined as solution to the following CSDE:
dxi- Σ
wr)t) Xt. a²) dt Σ wilt)o' (t, a²) dw,,
(1)
where and σ [0, xd XA Rd are the drift and diffusion functions, respectively. Each control agent a² [ x IRd x Rm a' = (t,, Xx; "", lI i M is defined as a Markov closed-loop feedback control, which is parameterized by the neural network 0'. While every agent defined as closed-loop feedback-type Carmona (2016b), the solution to the CSDE above, X is the Markov process, which means that process X propagated using the information of the current state.
Let T= (4ehseeN be a set of ordered time such that 0 = 4 <...... < L < ... <N indicator function the intervals, w. w;(() Legees with predetermined starting/ending points tk, tę in T. We call this function temporal privacy (TP) because represents each agent's attention on different sub-intervals. Overall, in (I) the stochastic process X is propagated by summing M-number of individual agent's weighted
The set of functions (w:({)ssiMM is defined
attentions
Σd w.bb(, a²), Σd (n a²)) To understand the behavior of the proposed CSDE more deeply, we consider the following detailed example:
'The time interval dt A. It2 &/| for any k he regularlyliregularly in method.
2

---

Published as a conference paper at ICLR 2022
Role of Temporal Privacy. We define (s) Lssasu t, u E T with I M. Then, Xg in given X2 at an interval [t, u] can be equivalently rewritten in the integration form:
M
wi(s)b' (s, X5, a2)) J. M
xa=la' ma= "a
=x-- 2 I
Σ
I (s) a'W,,
= xx + [ (x)
(s, dds + m.)(s)
d'(s, = X
In 2). the activated control agent to evaluate the stochastic process X for the interval [t, u] is only a" (i.e., X = =x=") owing to the definition of the weighting function w() (t). This means that the remaining control agents { not used for the evaluation of the stochastic process in the sub-interval [t, u.. While each agent a' is activated at its own private sub-interval, this leads our method to adopt dynamic programming (DP) to train Neural CSDEs in the form of ( this paper, we aim to solve the optimal control problem via DP with multiple agents, where each agent specializes in solving particular sub- uub-problem its private interval.
3.2 MARKOV DYNAMIC PROGRAMMING PRINCIPLES
The dynamic programming principle is one of the fundamental philosophies for dealing with stochastic optimal control problems. Its basic idea is to consider family of sub-problems with different initial times/states and establish the relation among the sub-problems to systemically solve them. Using mathematical property of the proposed CSDE with TP, we present efficient learning strategy to solve stochastic optimal control problems via Markov dynamic programming (Markov-DP),
In this paper, we aim to solve the stochastic optimal control problem by training control agents a = [a p , " and minimizing the cost functional ((,, [0, x E + RR::
Xt, = E L. ((s, X. + ((X)) Fe = [. ((s + Juu x X
(3)
where : [0, x I2d + R$ is the running cost (e.g., L2 loss) that computes the discrepancy between the propagated process X; and the observed data point ye at each time t, V (.x)) : R - RR the terminal cost that estimates the discrepancy between the terminal state and the data yr. To evaluate the cost functional (1, X at time with control agents a, the running cost is integrated over the time interval [t, 7') conditioned on filtration F, Note that the expectation conditioned on F, in (3) can be substituted to the expectation conditioned on X in light of the Markov property presented in Section A.2 and the cost functional time only depends the current state of the process X;"
Markoy-DE with Temporal Privacy. By combining the tower property of the conditional expecta- tions with the dynamic programming principle and Itô's formula (Oksendal (1992)), one show that a minimization problem can be recursively decomposed into sub- sub-problems owing to the property of TP in our proposed CSDE:
f.
Vtt, X;) & inf Xf) inf E
1(s, X Juu, Xa) Xx) X
(A)
(4)
inf J. ((s X
+ inf (x,
(B)
where V is an optimal cost functional (i.e., value function), a" denotes the r-th control agent, and (-r) [aj..... O.... asd indicates the set of remaining agents (the r-th component is zero). In the minimization problem (A) over a is divided into two sub-problems using the dynamic programming principle, which are (B) and (B'). Because the minimization problem (B) is only dependent on the control agent a" parameterized by the neural network f", we compute the gradient descent of H" to solve the sub-problem (B):
i
[(mnn (0, ds X
oet 0%
E
(5)
30
3

---

Published as a conference paper at ICLR 2022
where (s) Lssesu is the TP function at interval [t, u] and k is the index for the learning iterations. In 1 the r-th control agent a' minimizes the cost functional using the gradient descent scheme at its own temporal sub-interval. As the remaining sub-problem (B") over agents af can also be recursively decomposed into smaller sub-problent using the dynamic programming principle, the original problem (A) is solved separately with M number of control agents a = {a', . . ax with the M-number of gradient descent schemes. This indicates that we can obtain the set of agents a, = =faa(-a-- (..(..;;,,)) by collecting individual optimal agents with sub-problent bob--roblems..
In this paper, combine the Markov-DP with M gradient descent schemes in 5 and CSDE with TP in ① and introduce a novel Markov-DP-11 framework. In the numerical experiments in Section 4.4 show that the proposed Markov-DP-TP framework remarkably increases the model efficiency compared to conventional non-DP naive approaches, which makes our method directly model the complex time series in the data space. However, despite the improvements with our novel Markov-DP-TT framework, there exist remaining practical/theoreha issues that should be addressed to solve the optimal control problem with complex datasets.
1) Conditional Dependency. The main practical issue in implementing the Markov-DP-TT frame- work is that explicit conditional states are not given, e.g., X in (5). As different initial/terminal conditions of SDE lead to totally different behaviors of induced dynamics, well-designed conditional information a crucial factor in training the Neural CSDE for specific applications. In Section 3.3 we introduce the Markov Forward conditional (MFcond) loss to train the Neural CSDE with well-posed conditional information that ensures accurate network predictions.
2) Theoretical Optimality, In the optimal control theory, there are well-known partial differential equations called Hamiltonian- Jacooiiii---a-i---e-e Jacobi-Belllm (HJB) equations, which assure the theoretical opti- mality of control agents. If the control agents solve the HJB equation, the proposed method attains the optimal state Vx(X)) = inf, (x = Jx(,")). However, the optimal agents a, of the proposed CSDE with gradient descent are not generally equivalent the solution the HJB equation. In Section 3.4 we propose the Markoy Backward conditional (MBeond) loss to assure the optimality of control agents and to provide information in backward dynamics for regularization.
3.3 MARKOV FORWARD CONDITION
In this section, first raise the important question: Why is the well-posed conditional estimation in cost functional important to accurately train Neural SDE (CSDE) models? To elucidate the importance of this question, we consider the following minimization problem with the cost functional with naive partial information:
[ I(s, (ss + ( Xo = yo
inf C(a) = inf Eyo
(6)
a
where y() = =ye denotes set of observed data, and yo the initial data at time t In (6). the conditional expectation is taken to the single initial state Xo = yo. and the control agents minimize the accumulated losses using this partial information. As pointed out by Kidger al.) (2020), this naive cost functional causes a problem when dealing with high-dimensional complex datasets. This is because the Neural CSDE should disentangle the inherent latent information of complex digh-siimesionall data generate accurate results, but the control agents are trained with only the restrictive and partial information of the observed data (i.e., initial condition X,- = yo). To solve this problem, introduce novel loss function called the MFcond loss that can fully exploit the information of the given observed data Y()) while keeping the Markov structure of X
Definition 1. (MFcond loss) We define the prediction operator T& as follows, for <t,,
M wibi (u, X3 addu + S km
M
TX
Σ mei(s.t)
X +
Σ 1m1
E'an Eiro' (u, Xü, amdw( x ytm
[((,,t)
where I(s,t) := {m : tm <}. [[stt)] t) is the cardinality of I(s,t), and {w (s.t().)
denotes the Wiener processes with respect to time u. Let us define a random stopping time Ta such that Ts := inf,(t: : l(t, TT.)= > <) for the pre-determined threshold g Then, we can define the FFcond
loss with the stopping time T()) as follows:
Please refer to Appendix
A.7 for detailed information
4

---

Published as a conference paper at ICLR 2022
X
T....
ToTn
Ye.0
X7. T
byns
Lk
(a) Network Training with the MFcond loss Figure Conceptual illustration of the MFfood loss and network inference. (a) The bold green lines indicate single trajectory
(b) Network Inference
proposed CSDE, while other trajectories are shown shaded regions. The MFcond loss estimates the conditioned losses I a Ts.r. (e.g.. red vertical lines) at 7s. The empty parts of the black lines indicate the irregularly sampled (unobserved) data representation. (b) The average decision X (e.g.. green cross) is estimated to approximate the true test data Dc, (e.g.. red cross) using multiple past observations. This figure shows the failure case as process Xú hardly approximates di..
L (o) = Eycs [ I To..) x(s)ds +
where x()) is indicator function that produces values the observed time (i.e., X(s) 1 if y, observed at 8; otherwise, x()) = 0). This function is used to consider the irregularly sampled data points. In (8). naive running cost of (6) is replaced with T which the MFcond loss recursively accumulates the expected future losses T conditioned on multiple observations.
At each time s, stopping time 7. decides the future time stop the CSDE propagation by determining if the accumulated losses are larger than the predetermined threshold not. While the proposed loss requires set of multiple conditions on the Markov process X;" train control agents, information is utilized to generate time-series data, and complex dynamics can be expressed. A conceptual illustration the proposed MFcond loss shown in Figure (a).
The main idea of our MFcond loss in X is to minimize the differences between the future estimations X for any given S u. other words, the proposed CSDE is trained to generate an identical future estimation of X given any past initial conditions X 21-)- i.e., (Xa.* so X Vs < I S u) to estimate network inference with multiple conditions in the test time. This idea is used to introduce novel inference procedure to overcome the raised issues on the partial information.
Network Inference. Let (3t.., } be the observed data sequences until the current time t in the test dataset. Our objective is to predict the future points (0ra 1. (tm S t < tg). Our model generates the stochastic estimation X to approximate St, at a future time given multiple initial conditions dt..:
Network Inference
T Tomit
(t,b((t Xi, a')dt + Σ S mo x x yawww x
(9)
In each control agent makes decisions on its specialized temporal state and collaborates to generate a stochastic conditional estimation Xç and approximate 9r.- As our MFcond loss induces identical estimations X g.. for any tm. Xç utilizes multiple conditions {ú... and fully exploits the past information to predict/estimate future values. A conceptual illustration of the network
5

---

Published as a conference paper at ICLR 2022
inference is shown in Figure (b). While the proposed inference mechanism utilizes enlarged information" compared to a single initial condition, it can model the complex time-series data.
If the control agents are trained with the naive cost functional, the terminal states X (conditioned on initial state X, = ys) and X (conditioned on initial state X, = yt) are largely different, which causes problems when we generate complex time-series data during the test time, whereas inference mechanism introduced in (9) utilizes averaged multi-decisions X given different initial conditions. Thus, the MFcond loss is essential for utilizing the proposed inference procedure.
Unlike the dynamical auto-regressive probabilistic models (e.g., ODE-RNNs) that encode whole (or partial) data sequences, shown in (T). the proposed Markovian CSDE model only uses the current observation to propagate stochastic dynamics. An additional inference mechanism coordinates the multi-conditioned trajectories utilize information and produces complex time series.
3.4 MARKOV BACKWARD CONDITION
In the previous section, we suggested the Markov forward conditional loss that exploits the entire information of time-series data generate accurate results. Aside from its empirical benefits to some applications, no theoretical/empiriaal optimality of (4) is assured by minimizing the MFcond loss in general. To tackle this problem, in this section, we further introduce the additional stochastic dynamics relating optimality of proposed CSDE-TP.
Let define the auxiliary process 7, V(t, X²") with value function V, where a. denotes the optimal control agents. Subsequently, we consider the following hewarddbackweer stochastic differential equations (FBSDEs):
FBSDEs
d.aaa -EE Seelubble (1, Xx, at) - EE wlo'(l, Xi "f",," J.W,,, dZ: -ls, Xa )at + EM (t,,, (i)woo" (t, xa at ddW
(x,', 7t) =
(10)
7T (
The first SDE (I.e., x called the forward SDE has an identical form of and propagates stochastic evaluation in the forward direction with optimal control agents. The second SDE (i.e., Z,) called backward SDE recursively subtracts the running cost from the terminal state V ((xx)) in the backward direction using forward estimations X,* and cancels the effect of martingales in the diffusion term. We utilize the property of backward dynamics Ze to train the control agents for the following reasons.
1) Backward Multi-conditions Like the MFcond loss with multi-conditions in the forward direction, we want to provide additional information to backward dynamics to train the control agents.
2) Approximated Solution of HJBE. The auxiliary process Z. gives the theoretical optimality for control agents related to the HJB equation based on the results developed in Yong & Zhou (1999);
Pardoux & Tang (199)). where the process Z, V(t, -) admits solution of the HJB equation in (11) and induces an optimal solution for the minimization problem inf / in
1 ol,, VV VVV VV((,)) b(,, 1(t,x))) 0, + (V, Öt
(11)
where V(T,v) (xr). In we want approximate Z, using control agents for optimality. However, the process Z, requires optimal control agents a, that cannot be obtained during the training time. To overcome this problem, approximate the auxiliary process Z, with Z parameterized by neural control agents a(,,, f), which is defined as the modified version of Z, In particular, Z, can be expressed in the following integral form:
z = (XX)) - 1 I (s)(ss
M
['Σ (x)ss ('J (12)
where J is the cost functional defined in 33. and VJ denotes the gradient of the cost functional with respect its spatial axis. Using the proposed process Z introduce a novel loss function called the MBcond loss to satisfy the two objectives discussed above.
Please refer to detailed explanation in Appendix A.3

---

Published as a conference paper at ICLR 2022
Algorithm 1 Neural Markov CSDB-TP
Require: 0.95, for k = 1 to K (i.e., the total number of training iterations) do
1) Simulate forward controlled SDE with Markov control agents
1-1) dx;+* = Em=1 wx "
.aj ajtdt + Σm w;o' (t, X ,t) 1-2) Evaluate each decision control agents aj = X,, ((,,)))
1-3) Compute the MFcond loss for Mccontrol agents [Cylai 0%)) with stopping time T()
1-4) Update threshold for random stopping time, 6k-11 4- 2 max / (f, Tel (v+)
2) Simulate backward controlled SDE
2-1) dZi = -E wrl(s, X7 dt + Ee VJL,, Jun
2-2) Evaluate the MBcond loss for M -cnorroll agents, {Co(ails (1)),
3) Update control agents with Markov-DP
3-1) f = 0 Ly 8))) - -01 (1 end for
Definition 2. (MBcond loss) Let us define the auxiliary process Z as the solution to 12 Then, the MBcond loss can be defined follows:
Ca(a) By (e [
yr
(13)
Theoretically. we optimize the MBcond loss (13 according to the proposed backward dynamics Z the PDE reformulation of backward dynamics, called Non-linear Feynmann- Kac, have the identical solutior to HJB equation in (11 Thus, our method can attain the optimal solution of original problem posed in section 3.2
Intuitively saying, one can show that the MBeond loss is equivalent to the reformulation of the minimization problems (4) using Itô's formula. Thus, solving the minimization problem inf, Ls induces an identical effect to solve the original problem inf J. The only difference is that we utilize multiple conditions to provide conditional information on the backward dynamics Z for the regularization of control agents trained with forward conditional dynamics and to impose constraints on control agents, which induces an approximated solution to the HJB equation.
3.5 OBJECTIVE FUNCTION
In this section, we describe the overall training procedure, which incorporates all the proposed components (i.e., Markov-DP with CSDE-TP, MFcond loss, and MBcond loss) follows:
inf L(a) a=(a (o ar 2Ly(a)+ + Co(a)
inf
MFBeond
MBeund
(14)
M CSDE-IP Σ inf (Ia' (0)
[11 (11 (-D]
where Cy and Ls are defined in and 13
respectively, and 7 is a balancing hyperparameter. are trained with convex combination of MFcond
In (14), the control agents and MBcond losses. Using the property of CSDE-TP with Markov-DP the original problem is approximated with the collection of M sub-problemss and each control agent is separately trained with M gradient descent schemes. Algorithm describes the detailed procedure of our method.
4 EXPERIMENTS
Network structure of control agents. The neural network structure for each agent control consists of 2ayerss of fully colle-con layers, where each module has 128 latent dimensions. For the activation units, we used the specialized LidulleLippSin sh Chen et al. (2019); Kidger et al. (202T), to stabilize the FBSDEs during training. Please refer to Appendix A.6 for detailed information on the network architecture.
Datasets. For the evaluations, we used PhysioNet, Speech Commands, Beijing Air-Quality, and S&P- 500 Stock Market datasets. Refer Appendix [or data statistics and prepossessing procedures.
Aopenddx(A.5]forr *lease refer Appendix A.4 for the discussion theoretical optimality induced by the MBeond loss.

---

Published as a conference paper at ICLR 2022
4.1
TIME-SERIES DATA RECONSTRUCTION
Table 1: Evaluation of reconstruction tasks on the PhysioNet/Specch Commands datasets.
PhysioNet
Speech Commands
Methods
MSE (x10
NLL (x10²)
MSE (x10") 4 NLL (x10")
RNN- VAE
6.655
33.238
0.984
-4.18
ODE VAE
5.523
-24.195
0. 0.723
-3.763
Latent SDE (ODE-Eng
2.403
-11.978
0.841
-4.226
Latent SDE IRNN-Ene
2.418
-12.051
0.851
-4,250
Latent ODE (ODE-Enc)
2.395
-11.33
0.851
-4,251
Latent ODE (RNN-Ene -En)
2.375
11.840
0,880
-4.402
ENN-Decay
1.431
7.119
0.865
4.323
ODE-RNN
1.311
6.517
0.697
-3.479
GRU-D
1.424
7.083
0.838
-4.116
mTTAND
0.887
4.396
0.581
-29022
CSDE-TE
0.755
3. 3.667
0.435
-2.172
CSDE-TP-MV
0.728
-3.605
0.423
-2.109
In this experiment, we compared our model against baseline dynamic models: [Latent ODE, Chen
et al. 2, [attent SIDE, Li al. (2020), [ODE-RNN, Rubbanova al. (2019)) [GRU-D, Che
et al. 1. [mTAND, Shukla & Marlin (202 and DOEEVAE,[aaaaal Yıldiz et al.
(20 1019) J.
We used open-source codes provided by the authors for comparison. For the Latent ODE (SDE)
methods, RNN and ODE-RNN were used for the encoder structures, where the decoder structures were identically set to ODE (SDE). Table JableTshowe shows the performance of all baseline methods compared to the proposed CSDE-TP for the reconstruction tasks. As evaluation metrics, errors (MSE) and negative log-likelihood (NLL) with open-source code in RRubanova al. (2019). As shown in Table the proposed method consistently outperformed the baseline methods by large margin. this experiment, we observed that latent dynamics- based methods (e.g.. Latent ODE/SDE with RNN and ODE-RNN encoders) dimensions each control agent to 128 for both the reconstruction and prediction experiments. In the experiments both datasets, the Mckean- Vlasov (MV) type of the SDE model slightly improved the performance, where it subtracted the mean (i.e., mean-shifting) of the control agent outputs to normalize/reduce the intrinsic volatility in the inferred process X
used mean squared
models attained similar performances. We set the latent
4.2 TIME-SERIES DATA PREDICTION
Table 2: Evaluation of prediction tasks on the PhysioNet/Air Quality datasets.
PhysioNet
Air Quality
Methods
MSE (x10-l)
NLL (x10?)
MSE (x10-1)4
NLL (x10²)
V
5.369
-26.812
7.938
33.65
Latent ODE (RNN-Enc)
2.038
-10.153
3.403
-1.722
Latent ODE (ODE-Em) Enn))
1.977
-9.849
3.374
-1.633
Latent SDE IRNN-Enc)
2.002
-9.975
3.676
-1.831
Latent SDE (ODE-Enc)
1.995
-9.936
3.328
-L.660
mTAND
1.551
-7.719
1.867
-0.929
CSDE-TV TP CSDE-TP-MV
1.195
-5.937
1.277
-0.630
1.057
5.811
1.671
0.638
In this experiment, all the methods were evaluated predict the remaining 12 time sequences (e.g., 20%) on the test dataset given the first 36 -equeeeesssss (e.g., 80%), and we followed a procedure similar that suggested by Rubanova al. (2019) predict the future time series. Table 2 compares the performance in terms MSE and NLL., in which the proposed method considerably outperforms the baseline methods. In the experiment on the Air quality dataset, we set smaller latent dimension of 64 as the feature dimension compared to those of the other datasets. Interestingly. the CSDE-TP-MV model exhibited worse performance than the vanilla CSDE-TP model when the data dimension was very small. This result indicates that the mean-shifting in low-dimensional data space reduces the expressiveness of the CSDEs.
4.3 UNCERTAINTY ESTIMATION ON STOCK MARKET DATASET
When high volatility is observed over the temporal/spatial axes, conventional evaluation metrics such MSEs hardly capture the stochastic property of the time-series variations. Thus, to cap- ture the stochasticity, we evaluated the distance between the distributions the test data and the inferred/generateed data using the maximum mean discrepancy (MMD). We followed the protocol
8

---

Published as a conference paper at ICLR 2022
Table 3: Uncertainty estimation the Stock Market dataset.
Methods
RNN
RNN-VAE
ODE -OE--NNN
Latent ODE
Latent SDE
CSDE- TP
MMD (x 10 9)
526.12
427.05
201.48
194.75
190.11
94. 79
suggested by Li et al. (2177) to evaluate the MMD distance, where we used two Gaussian RBF kernels with bandwidths of [5.0, 10.0). Using this evaluation metric, we experimented on reconstruction tasks using the S&P-500 Stock Market dataset. Table 3 shows that the proposed CSDE-1 TP outperforms baselines and effectively recovers the distributional information of stock prices with the stochastic property of the SDE models and the proposed optimization framework. Interestingly, the latent SDE model attains better performance compared to the Latent ODE, as it utilizes additional Wiener process to model the data uncertainty. The performance improvement of the Latent SDE vanishes when we remove the diffusion term (o 0) of the latent SDE.
4.4
EMPIRICAL STUDY
0.30
MSt
MFcond
Training MSE
0.25
0.20
CSDE-TP-81
0.95
Naive
CSDE-TP-S2
Test MSE
CSDD
0.15
CSDE
0.85
0.0
2.5
5.0
10.0
Epochs
Epochs
(a) Efficiency of Markov-DP with CSDE-TP
(b) Efficiency of the MFcond loss Figure 2: Ablation study on the effectiveness of the proposed method.
Efficiency of the Markoy-DP-TP framework. To show the empirical advantages of our CSDE-TP model with Markov-DP learning schemes, evaluated our CSDE-TP according to different number of control agents on the prediction task using the Air Quality dataset. Figure (a) shows the training MSEs for several variants of the proposed model in the first 20 epochs, where CSDE-TP. Shallow1, -Shallow2, and -Deep (i.e., black, blue, and red lines) denote the proposed models with a different number of control agents, i.e., M 2, 8, and 48, respectively. The standard CSDE model (i.e., the black dashed line) utilized single agent, M = 1. For all models, the total number of training parameters was equivalently set R 40K, and the number of parameters was normalized. As shown in Figure (a), despite using the same number of parameters, employing multiple agents clearly outperforms the standard CSDE terms of the learning curve. From this fact, we can conclude that the Markov-DP-TP significantly increased the network efficiency compared to the standard CSDE, which indicates that our Markov-DP framework crucial for training controlled dynamics models. Efficiency of the MFcond loss. In this experiment, show the empirical advantages of the multi- conditioned CSDE in (8) against the naive partial-conditions experiments, the results were obtained for the prediction task with the Air Quality dataset. Figure (bb) shows the model confidence in testing MSES for the first 50 epochs, where shaded areas indicate the confidence regions (i.e., std). The proposed MFcond loss exhibits considerable performance improvement (.08 < .87) compared to the conventional native cost functional and reduces the variances loss landscapes with stable learning. With the theoretical discussion in Appendix (A.33 we conclude that the proposed CSDE actively exploits the information of the complex time series with multiple conditions to accurately generate complex time-series.
CSDE in (6). Similar to previous
5 CONCLUSION
In this paper, we introduce a novel Markov-type CSDE with the TP function that records the individual attention of each control agent at sub-intervals along the temporal axis. Using the properties of the CSDE and TP, we suggest Markov DP to efficiently train the control agents by decomposing the original problem into smaller sub-problems. To overcome the practical/theoretical issues, we propose two novel losses, namely, MFcond and MBcond losses. The MFcond loss captures the future time to estimate the running costs, while multiple conditions are actively provided to forward dynamics. The MBcond loss assures the theoretical optimality of the control agents and imposes regularization by providing additional information to backward dynamics. Experimental results demonstrate the efficiency of the proposed method for various tasks using real datasets.

---

Published as a conference paper at ICLR 2022
Acknowledgments. This work was supported by Institute of Information communications Technol- ogy Planning Evaluation (IITP) grant funded by the Korea government (MSIT) (No.2021-0-01341, Artificial Intelligence Graduate School Program (ChungAng university)).
10

---

Published as a conference paper at ICLR 2022
REFERENCES
Brian DO Anderson. Reverse-time diffusion equation models. Stochastic Processes and their
Applications, 12(3):313-325 1982.
René Carmona. Lectures on BSDEs, stochastic control, and stochastic differential games with financial applications. SIAM, 2016a.
René Carmona. Lectures on BSDEs, Stochastic Control, and Stochastic Differential Games with Financial Applications. Society for Industrial and Applied Mathematics, Philadelphia, PA, 201 doi: 10. 19744449, httht //pubs.. siammmma.mm olt...or//ool//a////.....o
Zhengping Che, Sanjay Purushotham, Kyunghyun Cho, David Sontag, and Yan Liu. Recurrent neural networks for multivariate time series with missing values. Scientific reports, 8(1):1-12, 2018.
Ricky T. Q. Chen, Yulia Rubanova, Jesse Bettencourt, and David Duvenaud. Neural ordinary differential equations. NeurIPS, 2018.
Ricky TQ Chen, Jens Behrmann, David K Duvenaud, and Joem-Heerri Jacobsen. Residual flows for invertible generative modeling. NeurIPS, 2019.
Edward De Brouwer, Jaak Simm, Adam Arany, and Yves Moreau. Gru-ode-bayes: Continuous modeling of sporadically-observed time series. NeurlPS, 2019.
Junteng Jia and Austin R Benson. Neural jump stochastic differential equations. NeurIPS, 2019.
Davar Khoshnevisan. Intersections of brownian motions. Expositiones Mathematicae, 21(2):97-114 2003.
Patrick Kidger, James Morrill, James Foster, and Terry Lyons. Neural controlled differential equations for irregular time series. In NeurlPS, 2020.
Patrick Kidger, James Foster, Xuechen Li, Harald Oberhauser, and Terry Lyons. Neural SDES as Mulinite-Dimnssee GANs. ICML, 2021.
Chun-Liang Li, Wei-Cheng Chang, Yu Cheng, Yiming Yang, and Barnabás Póezos. MMD GAN: Towards deeper understanding of moment matching network. In NIPS, 2017.
Xuechen Li, Ting-Kam Leonard Wong, Ricky TQ Chen, and David Duvenaud. Scalable gradients for stochastic diffcrential equations. In AISTATS, 2020.
Yingru Liu, Yucheng Xing, Xuewen Yang. Xin Wang, Jing Shi, Di Jin, and Zhaoyue Chen. Learning continuous-time dynamics by stochastic differential networks. arXiv preprint 20iv:2006.06144 2021.
Stefano Massaroli, Michael Poli, Jinkyoo Park, Atsushi Yamashita, and Hajime Asama. Dissecting neural odes. In NeurlPS, 2020.
James Morrill, Cristopher Salvi, Patrick Kidger, James Foster, and Terry Lyons. Neural rough differential equations for long time series. ICML, 2021.
Bernt Oksendal. Stochastic Differential Equations: : An Introduction with Applications. Springer- Verlag, Berlin, Heidelberg, 1992.
Bernt Karsten Øksendal and Agnes Sulem. Applied stochastic control jump diffusions, volume 498. Springer, 2007.
Etienne Pardoux and Shanjian Tang. Forward- Borward-backward stochastic differential equations and quasi- linear parabolic PDEs. Probability Theory and Related Fields, 123-2):23-1100 1999.
Sung Woo Park, Dong Wook Shu, and Junseok Kwon. Generative adversarial networks for markovian temporal dynamics: Stochastic continuous data generation. In ICMI., pp. 8413-8421, 2021.

---

Published as a conference paper at ICLR 2022
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang. Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, pigh-peeformmnce deep learning library. In NeurIPS. 2019.
Yulia Rubanova, Ricky TQ Chen, and David K Duvenaud. Latent ordinary differential equations for ireeularly-samplle time series, NeurIPS, 2019.
Satya Narayan Shukla and Benjamin Marlin. Multi-time attention networks for irregularly sampled time series. In ICLR, 2021.
Ikaro Silva, George Moody, Daniel Scott, Leo Celi, and Roger Mark. Predicting in-hospital mortality of icu patients: The physionet/comppuing in cardiology challenge 2021. 2012.
Yang Song. Jascha Sohl-Dickstein, Diederik Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In ICLR, 2020.
Yusuke Tashiro, Jiaming Song, Yang Song, and Stefano Ermon. Csdi: Conditional score-based diffusion models for probabilistic time series imputation. arXiv preprint arXiv:2107.03502, 2021.
Pete Warden. Speech commands: dataset for vocabua-ocaabulryy speech recognition. arXiv preprint arXiv:1804.03229, 2018.
Jiongmin Yong and Xun Yu Zhou. Stochastic controls: Hamiltonian systems and HJB equations, volume 43. Springer Science & Business Media, 1999.
Moshe Zakai. Some Classes of Poo-mmareeter Martingales. The Annals of Probability, 9(2):255 - 265, 1981.
Shuyi Zhang, Bin Guo, Anlan Dong. Jing He, Ziping Xu, and Song Chen. Cautionary tales on air-quality improvement in beijing. Proceedings the Royal Society A: Mathematical, Physical and Engineering Science, 473:20170457,, 09 2017.
Çagatay Yıldız, Markus Heinonen, and Harri Lähdesmäki. vae: Deep generative second order odes with bayesian neural networks, 2019.
12

---

Published as a conference paper at ICLR 2022
A APPENDIX
A.I DETAILED COMPARISON TO EXISTING METHODS
In this section, we investigate the relation between our method and existing methods.
Reverse SDE vs. Backward SDE. Song et al. 2020) suggested a novel SDE called reverse SDE, which shares semantically similar idea with BSDE: both reverse/backward SDEs enhance the forward SDE by providing additional information to drift/diffusion functions in forward dynamics.
The mathematical motivation of the reverse SDE in Andersooo 1982) is to pose the SDEs with Wiener processes W,, W. with respect to these minimal increasing/decrrastiin sigma algebras Ar, A. and define the relation between them:
dW,
(X,) (l)((X dW:.
(15)
where X₁ is a solution to the forward SDE and p: is the probability density of X,, Using the relation in (15). the reverse SDE transforms the prior distribution (e.g., Gaussian noise distribution) back into the data distribution (e.g.. 2D images) by gradually removing the noises and reconstruct the original data with the well-designed score function (i.e., Vp((z)) in backward dynamics. In contrast to the reverse SDE, the role of backward SDE in this paper consider the probabilistic reformulation to access the cost functional to provide the additional information in backward dynamics.
Stacked ODE vs. CSDE-TP, Massaroli et al. 2020) suggested the stacked Neural ODE that shares similar idea with the proposed CSDE-TP, where temporally piece-wise neural nets are considered to model the complex dynamics. However, the stacked ODE faces the aforementioned problem partial conditional information when generating complex data as their models only take initial values to propagate dynamics. As opposed to their models, the proposed model is trained with multiple observations and directly generates time series in data space without any latent embedding network. Furthermore, we generalize their optimal control problems to the stochastic version and propose the Markov-DP-TP framework that can systemically solve the problem.
DDPMs vs. CSDE-TP. Tashiro al. (02T) suggested denoising denoising diffusion models (DDPMs) that are conditioned on the set of the observed data, where the generated sequential data is assumed to be gradually transformed from an initial state in the forward direction and the backward process is parameterized by the neural network and trained to minimize specific ELBOs.
Specifically, the transition probability pa terized Gaussian distribution:
in the backward process defined as a parame-
po (x, - NXXx11;; Xt), do X.)),
(16)
where mean and covariance (A9.0r) are parameterized by the neural network 6. Similar to the pro- posed CSDE, their parameterized functions are closed-loop type processes and the whole probabilistic sequential model py is posed as the Markov chain: Po (Xo:r) = (x(r I Po(Xx_1(X,).
In contrast to the DDPM, the probability transition in the proposed CSDE is defined as the continuous generalization called controlled Folker-Planck equation (CFPE):
1 ① P6 (z. ll.. = - (,,,,, + Tr Va pr(x))
(17)
where > s [0, 7) and x. y E Rd, Pt X is the probability distribution of Xç. The CFPE in (17) one-to-one corresponds to the CSDE (i.e., X Compared to the discrete-time Gaussian transition model, this conditional probability can express complex continuous-time probability transitions while maintaining the Markov structure.
A.2 NOTATIONS AND BACKGROUND
We first define the basic definitions of probabilistic objects:
Definition 3. A filtration (Ft) is an increasing sequence of {Fi} of a-algebra such that Fo ... C F. F. The triplet (N,Fet IP) is called filtered probability space.
Definition 4. The filtration generated by Wiener process W, is defined as FW = o[Wo.... W.). In this case, W, is naturally FW -adapted by construction.
13

---

Published as a conference paper at ICLR 2022
Definition 5. The stochastic process {Xi} is called {Fi) -adapted if Xt is Fo measurable for every 0 TT
Throughout this paper, we work on the filtered probability space (9, (Fi P) with the d- dimensional Fi-Wiener process We and natural filtration J.l.. We assume that a" for all 1 S M is admissible Markov control, (i.e., a' is Fe-adapted and a² E A,, Xi has a unique solution).
Definition 6. (Markev Process) Let Xt, be Fr-adapted stochastic process. Then, Xt is the Markov process if the following equality holds:
[[X+|Fa]] = E[X,|Xa], Vs S.
(18)
Definition 7. (Controlled Stochastic Differential Equation)
x; =xx /. bu, X a) du + [ (u, Xu, a) dW.. for
(19)
The solution to the above CSDE is denoted as X.... If the initial states are specified (i.e., starting point X, = I)), we denote the solution as X.a.s.. By the definition of Markovian control agents, in all cases, the solution to the proposed CSDE in a Markov process.
Mathematical Assumptions. In this paper, we assume that functions b, are uniformly Lipschitz continuous along its spatial axis and bounded []b(t, 0: )ll (t, 0;-) at the entire interval [o, 7]. We assume that each functions b'(.. x. (', x. -) V ((., x)) are twice differentiable for all 1 1 i S M, (i.e., bi, a', V,I E (IR))) and both drift and diffusion functions are uniformly Lipschitz on its spatial axis, iee, b³, 0, 8eb,, of, 0ga', Hgo E Lip), and the trainable parameters of the control agents a', 8 are lying in the compact subset C of its ambient space (i.e., 6 E C c R.)..
A.3 ENLARGED INFORMATION BY COLLECTION OF OBSERVED DATA
In the proposed inference procedure, we define novel operator T in ( to consider the multi- conditioned dynamics with the Markov-type SDE model. Although this operator plays a central role in the paper, its mathematical properties have not been carefully dealt and investigated thoroughly. In this section, we discuss the relation between this operator and the enlarged information that is obtained by collecting past observations. In addition, we generalize the inference mechanism in (9) to mathematically rigorous form and discuss the effect of the proposed operator 'T by showing some probability inequality.
Suppose that we have two observed conditional states {X }. [X,,,} until the current time f, (nn, Im < < (g) and the objective is to predict/gennrrae the future value yt, using this infor- mation. We consider the deterministic time la by replacing random stopping time Tr,,. to simplify the discussion. First, we define the two-parameter stochastic process Y to model the proposed operator T in an alternative way:
Tim.th (tt t. (w) & 1
2 (XI,1 (w) + +22, (w))
(20)
where w E S2 takes a value in the probability space. The stochastic process Y is the (Fr,, F...). valued random variable for any fixed lm, In < / by the definition,, where F,,, VJ,,, A E(Fth UF is the composited smallest sigma algebra with two filtrations. In the definition, we assume that processes 1.12 . are derived from two independent Wiener processes We and W, Then, can define the two-parameter martingale Zakai 1981); Khoshnevisan (2003) in the following form: E [M(te,, Y(m,: F.., V.....
(m)
(21)
By the definition of M, it can be easily shown that MI is the reformulation of the MFcond loss for some fixed number of past observations. Note that M is truly a martingale because conditional
estimations are summed in the definition T. The control agents are trained to minimize the M
given the information induced by past observations (i.e., composited filtration (Ve..er J. }),
which indicates that the proposed inference procedure can infer the future value Xę, according to the enlarged information {V composited filtration, we obtain the following result using Doob's maximal inequality:
Sm. <t. FM J. By the fact that M is a martingale with respect to the
2n EE x
Mea + [|
We II] S
sup sup Bi o F.F V F...,. S
[, bn <t
(22)
14

---

Published as a conference paper at ICLR 2022
where the inequality shows that errors between the future value Mr, and the generated samples X at time fg are bounded by the maximal perturbation probability. As the control agents are trained to minimize the MFcond loss (i.e., M) in the right-hand side of inequality, it renders the probabilistic bound of La errors at future time lk-
A.4 DETAILED DISCUSSIONS ON THE MBCOND LOSS
In this section, we investigate the detailed theoretical structure of the MBcond loss and fundamental rationale for the optimality of control agents. For this, we rephrase the cost functional in the general form:
[.
J(t x) -B
1(8, X. + ((x)) X; =
(23)
The classical non-linear Feynman-Ka Kac theorem in Yong & Zhou (1999) states that given the cost functional J with the control agents a, one can obtain the second-ordes parabolic partial differential equation from (33
DJ 01
(VJ, + T loo (,,,
- 0,
(24)
where (.-)) denotes the inner product and the boundary condition is given as ((T,)) (xx) Subsequently, by applying ItO's formula to (23). we obtain the following probabilistic formulation:
[J 0t
=(,, +
Xx) + (x,, (b(s,, Xx.s a), VJJ. dt
2
+ S. (V)(, =- J. 1(s, ((s; J: Xx), dW.).
(25)
By rearranging each term above, the backward stochastic differential equation is induced directly.
= = + J. 1(s, X.)ds [/ (f. (1. X,), dW.).
(26)
Note that, in the main paper, we use the inverse sign convention, - (.).). to emphasize the backward direction. By using the formulations, the MBcond loss in 13 can be rewritten in full description as follows:
I Exe
| +
f." ('ssxx))ds - (6 a) aw.) xx
Li(a, Zi))
dl,
[0.7]
(27)
where yC-) (y...... yr) p(y1, -r) denotes the of observed data. The regularization effect comes from the expectation evaluation of the third term in (26). Specifically, one can obtain following equality by using the Itô's isometry:
E [. (o dt [t
E
(o (). aw.)
[X,
(28)
Because the MBcond loss is posed to minimize this additional martingale term in 28 in backward dynamics according the forward dynamics Xf. reduces the over-confidence generated time-
series. By the relation (f) > xt, - 7 - Lt(t, x:) for any (1, E [0. 7) x R+, the update
rule for the MBcond loss can be expressed as follows:
[c, (s, (1500)) ds
0% = 0 ae
(29)
where this formulation is similar to (5) and shows that gradient descent with respect 0" for the MBcond loss can be explicitly defined.
Admissible Control Set A. In previous discussions, show the relation between J with BSDE dynamics Z, and the well-defined gradient descent. The next step is to define the proper control set A to relate the gradient descent with optimality.
15

---

Published as a conference paper at ICLR 2022
Let us define the Hilbert space L - (f(t, 0); R-auued Fi- Fr-progressively measurable 4/ C) with the norm 1| = E I lott,, v; 2 < co. We assume that control agents a" is L
- for any 0% # 02 E R and any 1 S k - K. In all case, we assumee that any 0% lies in the compact subset C of R Each functions ('., x. )), 0' x. 1 I i M, (i.e,, o', V.l (²R.)) and both drift and diffusion functions are uniformly Lipschitz on its spatial axis, i.e., by, 0.b', ,.. "0', 0ga0,, ,80' Lip). As we defined v and I as usual Euclidean distance, regularity/uniform Lipschitzness for these functions are trivial.
Lipschitz on the parameter variable, i.e., (. (:.:;0,,, a..;;%% |
- = 0%,
(z)), 1(;,x) twice differentiable for all
For the fixed parameter f, we define the r-th control agent as l" + a. (. (')))0)))))))) - (0) E C. Truly, the image space of a" (0) is the closed subspace of the Hilbert space L due to the Lipschitzness with compactness of 0.
Let (k) N R. be the trajectory for the training parameters of the r-th control agent learning iteration k. Without loss of generality, Jr[a,] [t,, x(a"aa We define the Euclidean closed metric balls (Bay) JAEN centered at ((k) with the radius 5% < oo such that Bay = {0 E Rmm1100 (k)]l < 85. (k.) is local minimum of J,[0(K)]}- Let consider the sub-sequence {o(k)]keN S (0,(k)) which induces the ttricty-decreasinn cost functional (J,(0(k))
with the ordered index set N. Then, the admissible control set is defined as follows:
K (o))
a la a"
M M a.....aaM
X
C
c (30)
R 5-
where is the maximal element in N and the constant K E N indicates the last iteration index of training defined in Algorithm Intuitively, the control set A can be understood collection of local minimum obtained by M gradient descent schemes during training.
V v Jaa.] J[((((K)) = inf ]a(()]]
(31)
OEA
where V E (L, R2). By the definition of metric balls { (Ba,) } and strictly-decreasing proper- ties, the infimum in is attained when 0(K) = and the control agent (O)))) is optimal in this control set.
Relation to Stochastic Maximum Principle (SMP). We consider an arbitrary control in the convex set K E A with / E K and the optimal control ()(())) Let DJlp 4 (0((K) + e(3 a)) be the Gâteaux derivative (this can be defined while the control set is vector sub-space, A c L By the result of Pontryagin maximum principle, Theorem (4.12) in CCammona (2016a), one can obtain the inequality follows:
0 0a
a Da
H
(VM L.) (10(K) 011
H
(l,a(t,
B(t, Xx, 0)1 2 DJIs 2 (32)
for t E [0, 7] almost surely, where we define H A H(t, X,, Y,, az) for the Hamiltonian system H. with adjoint variables Y,, Ze and define the arbitrary control = a(.,,,0) E A with some The first inequality is satisfied due to the definition of Lipschitzian control agents. The optimality condition indicates converging upper-bound of DJ
g to 0.
In method, the optimality condition of the proposed learning framework is bounded by the Euclidean distance between ((K) and 0 in parameter space. Thus, the proposed framework poses a fundamentally different approach to interact with optimality conditions in SMP. As we define that the ((K) is a local minimum of J with the inequality [[0(K) 0| < 6k ,, the gradient descent scheme that induces the tight radius {5) TENN+ assures optimality by the relation 0 < DJ = 5K.
Relation to the HJB equation. We consider the infinitesimal generator L, of the non-homogeneous controlled Markov process X, as C = (V.. btt, a)) + .Tr [oo important relation between the proposed MBcond loss with the HJB equation as follows:
[t, (t, a) f(. We show the
16

---

Published as a conference paper at ICLR 2022
a (t, x²) 0*)
Time Inhomogeneow Embedding
Linear Layer
Linear Layer
Drift 8 (P
LipSwish
LipSwish
Diffusion o'(t,
X
t+!
Figure 3: The pipeline of the proposed CSDE-TP with neural control agents
Equivalence Relation
0J
Ot (t) + ((t, ((t, =
J(t,
0V (t, Ot
inf ((t,)) l(t,, x)]
(33)
agA
Non-lincar Feynman-Kac, MRoord loss
equation,
the left-hand side of 33). PDE formula is directly consequence of the non-linear Feynman- Kac theorem that we derive in (24). The distinct point is that control agents are obtained by the gradient
descent the MBeond loss with BSDE (i.e., Zi). Note that, as shown in (1).. p. ((K) is actually an
optimal control. This means that, without heavy calculations to solve the PDEs, the gradient descent algorithm also assures optimality of control agents the proposed control set A.
In contrast, the HJB equation in the right-hand side states that the optimal control agent be obtained by solving the second-ordes parabolic formula and the infimum is taken by considering algebraic properties of candidates for the exact solution. If the solution to HJBE exists the control set A. the PDE in the left hand side of (33) approximates the solution to the HJB. Overall, we argue that the MFBcond loss can provide novel deep learning-bass based paradigm adopt/solve the conventional stochastic optimal control problem in a feasible way (i.e., well-defined loss functions with the gradient descent scheme).
A.5 DATA PREPOSSESSING
PhysioNet dataset, Silva al. (2012), contains overall 8000 multivariate time series obtained for the first 48 hours a different patient's admission to intensive care unit (ICU). Each patient has a set of 35 various clinical features. We normalized features of all patients in the dataset to have zero mean and unit variance. We used half of time-series as the training dataset and the remaining parts as the test dataset.
Speech Commands dataset, Warden 2018), consists of one-second audio records of various spoken words such "Yes", "No", "Up" and "Down".. Since there were nearly 100,000 record samples, sub-sampled the dataset due to the dimensionality of training instances on two conflicting classes (i.e., "Right" and "Left"). Overall, 6950 time-series records were selected, where 80% were used as training dataset and the remaining parts as the test dataset. We pre-processed these time series by computing Mel-frequency cepstrum coefficients from the audio signal, so that each time series was spaced with 65 and 54 channels. Then, normalized each channel of all signals in the dataset to have zero mean and unit variance.
Beijing Air- Quality dataset, Zhang et al. (2017). consists of multi-year recordings of air quality data across different locations in Beijing. Each sample contains 6-dimensional time series features of PM2.s. PM SO,, NO,, CO, and O,. which recorded per hour. We segmented data to have the length of 48 and normalized each feature of data in the dataset to have zero mean and unit variance.
17

---

Published as a conference paper at ICLR 2022
S&P-500 Stock Market dataset consists of stock market data with 6-dimensional feature vectors (i.e., [High, Low, Open, Close, Volume, Adj Closel). For the complete data acquisitions, we excluded enterprises with incomplete recordings during sampling duration, thus total 381 enterprises are selected. The time-series are sampled every 30-min with T = 48 temporal states. Similar to Speech commands dataset, we used first 80% of temporal states to train the model and the remaining parts are used for prediction task.
A.6 EXPERIMENTS DETAILS
Different SDE candidates for CSDE. Owing to the abstract form of the proposed CSDE in ①.. various types of drift and diffusion functions (i.e., b and σ) can be selected according to different applications. In Table 4 we enumerate candidate functions. In the experiments, we adopted two models: Vanilla and Mckean- -lasov (MV) SDEs.
Hyperparameters. For the running and terminal costs (1 and V, respectively), we used the l2 distance, i.e., ((s, x)) = || 22112 and (z) || yrll In all experiments, is set to 0.95. To estimate the gradient of the MBcond loss, we estimated numerical gradients with the auto-grad library in Pytorch Paaszke et al. (2019)).
Network Architecture for Neural Control Agents.
Each control agent a' (t, Xt; Xx; ) has identical neu- ral network architecture, which consists of linear lay-
Table 4: Examples of Neural CSDEs
ers and non-linear units. Figure B shows the detailed network architecture. Each agent takes concatenation of temporal/spatial tensors Xy) as its input, where the temporal tensor is transformed into new form 1' by the time inhomogeneous embedding layer. We fol- lowed the setting suggested in Park et al. (202I) for this embedding. After time embedding, concatenated tensor (t', X,) is fed into two Linear layers with non- linearity units (i.e., LipSwish in Chen et al. (2019); Kidger et al. (022)) )) Finally, the transformed tensors split into the control terms for drift and diffusion functions. The diffusion functions are defined as non-degenerate types, where of (1, X,,a*)
SDE Type
b',0"
Vanilla
a a'(1,
Langevin
Va'(Xf), 20 v20 [1] (XFi) o² [Ea' (1, xf]]
Ornstein-Ulenbe
McKean- Vlasov
Diag(=:) and 2, is the output of the last linear layer. The latent dimension of each I. Linear layer the prediction task with the Air Quality dataset (= 64). Thus, a total number of training parameters for single control agent of is 1LK.
128 in all experiments except for
Simulation of CSDE and Temporal Privacy Function. Let T = E {tx J ISsN for the pre-fixed time interval A,. We apply the Marer--aruyama scheme approximately simulate the proposed
CSDE:
M
M
X = X² +
Σ w (t, (t:,;; + 1=1
Σ w w'(()))))))) (t) xf, (t, (f: 0') (34)
where 7 A Z(0, VA,a)) is d-dimensional Gaussian random variable with zero mean and covariance Adla.
Analysis of Instability at Contact Points. In every time stamps t, drift and diffusion functions are controlled over neural control agents a' where we assume that t_ and f. are adjacent points of contact point with infinitesimally small duration. The process A, indicates drift integral term and of denotes the diffusion term our forward CSDE dynamics. As shown in the inequality, the Markovian property is still preserved, and the magnitude of jumps are controlled by Lipschitzness of drif/ddifusion functions.
[["
[
E 2||x_ - Xx 11 S EE 1EIIA1PIX I
|اoاl ds
ds|xe X
+ E
|oo ds|
(35)
In probabilistic point of view, the set of contact points may be regarded as measure-zero, and the probabilistic evaluation not changed.
18

---

Published as a conference paper at ICLR 2022
a ('(0")
Agent a' (-.0")
Figure shows a particular example, where 14 temporal states (i.e., [T] = 14) with control agents are considered. In the figure, the black line indicates the trajectory of time series, blue dots denote the observed data points, and shaded grey dots denote the missing data points. Each control agent takes 5 data points, where 2 temporal states are shared to other agents. In the experiments, the total number of temporal privacy functions are maximally set to M /T//2. where each control agent shares 2 points for smooth transitions of stochastic dynamics.
Agent 2 (..0%)
4gent (+,0*)
Figure 4: The example of the temporal pri- vacy function.
A.7
ADDITIONAL EMPIRICAL STUDY
0.35
U.S5, MFcond MBcond
Algorithm 1
7- 1.0, MFoond
Random selection
0.30
0.0. MBcond
TeelMSF
0.25
Test MSE
0.20
0.15
0-
0.10
-2
10
20
Epochs
Epachs
(a) Effect of hyper-parameter y
(b) Effect of random stopping time (log-scaled)
Figure 5: Ablation study on the effectiveness of the proposed method.
Effect of hyper-parameter In Figure (a), (a), the effect of hyper-parameter is shown. Similar to Fig/2((b, the results were obtained for the prediction task with the Air Quality dataset. Each red, black, and blue line indicates the test MSE for different 7 E [0.0, 0.95, 1.0) over 50 epochs. If the MFcond deactivated during the training time i.e. 7 = 0.0, only MBcond loss is utilized to train the proposed TP, and the model produces poor results. As our inference procedure requires the model to train with multiple conditions, the obtained result seems obvious. the MBcond loss is deactivated during the training time i.e, 1.0, nulti-conditioned information in backward dynamics 2 are canceled, and the performance is decreased significantly i.e., 1.277 + 2.003. This clearly shows that MBcond loss boost the performance.
Effect of random stopping time. In Figure (b), the effect of strategy to select is shown. If we select threshold as uniform random variable U[s 7] which independent X,, then the network quickly falls into instability as shown in the red line of (lg This shows that the well-designed strategy for selecting threshold is crucial factor to stabilize the network learning landscape. Contrary to the random sampling strategy, our method defined in [ selecct half value of maximal MFcond loss in last learning steps as the threshold c for random stopping time (i.e., z max 4-1 ex). As the threshold is always bounded above the maximal loss in the last steps, random stopping time at iteration & is decided in the time set:
+ E (t: l(t, T:
2 max ltt, Te)
(36)
where 7 denotes the stopping time at learning iteration k. If the network trains the MFcond loss so 2
that la [(t, Tey, + training proceeds - co, then is clear that the stopping time vanishes k000 E 0. Thus, the strategy in (36 well-defined.
A.8 DETAILED EXPLANATIONS OF MARKOV DYNAMIC PROGRAMMING WITH TEMPORAL PRIVACY
For the clear explanation of proposed Markov-DD-TPP let us consider the detailed example. we decompose sub-problem (B') in (4) into another smaller sub-problems:
19

---

Published as a conference paper at ICLR 2022
J 1(8, )
inf inf [[ x al-))
+ inf R {[J(l
(37)
3
g(-r"
(B')
(C)
where we a--r) (y) Ius In this case, the problem (B) on interval [u. 7] is
now decomposed into smaller sub-problems (C), (C") on two intervals [u, w'] and [u', T]. Similarly to u in 44, another auxiliary time index v' is considered here for additional problem (C). The corresponding new temporal privacy function (s) lugagu is defined on the interval [u, w/).
By repeating temporal decomposition of original problem (A) M times, one can find the following hierarchical relations:
P1). original problem, == ... .T]. a, no temporal privacy
P2). Two sub-problems (B) (B") in
Time set, T [t, ...
T.
control agents, = [a"", (o""""("r)) (-
temporal privacy functions [w,}
P3). Three sub-problems (B) + (C) + (C'). Time set, = [t, ... u, ... u' .... ,T. control agents, a = |a", 3, 8 3
temporal privacy functions = [Wr, Wpr)
P4). M sub-problems, (A) + (B) + (C) +... Time set, =
T-t r T-1
T).
M
M
a = [a= a², a², aaa
temporal privacy functions = [Wi, W2
Wr
WM
The role of u in 33 and (4 is replaced to and u' in (P3), and replaced to (r T-t) in (P4) in the table the time interval is assumed to be regularly sampled. Similarly, the role of r in and is replaced to and r' in (P3).
A.9 TOY EXAMPLE ON SYNTHETIC DATA
In this section, we conduct the reconstruction experiment on synthetic data show the different behaviors and demonstrate the advantages of the proposed CSDE compared to previous methods.
Stochastic Trigonometrie Data. this experiment, we define the 100-dimensional stochastic process with composition of trigonometric functions (i.e., sin, cos) as follows:
Y 2 sin (5nt + Z,t) + 0.25 cos ( 13 nt Zzt + Z3
E RR100 100
(38)
where we assume i [0, 1.0] and the total number of temporal states are set to 48 (i.e., |T] 48). In the definition of synthetic process Y, both the period and amplitude are randomized with mean-zero Gaussian random variables (i.e., Z1 NO, 1.0), Zg N 2.0), Zg NNO0, (a))). With the effect of Gaussian random variables, the process contains high volatility in both the spatial/temporal axes. We compare our method to the auto-regressive ODE-RNN Rubanova et al. (2019) model using the open-source code implemented by the authors. To observe the fundamental difference between ODE-RNN and CSDE-TP, we stop the training procedure when the estimated MSEs both models attained the threshold (< .07). In Figures and 7 the first axis of trigonometric data visualized. The results of each model are indicated as the blue lines (i.e., X2)) and the synthetic trigonometric data is indicated the red lines (i.e., Y). The 95%--cnffidnce regions (i.e., CR-95) of both the test and predicted time-series are shown as red and blue shaded regions, respectively.
ODE-RNN. Figure 6 shows the results of the ODE-RNN model. Although the ODE-RNN model attains relatively similar MSEs compared to the proposed model, there are two main issues in their model be discussed.
20

---

Published as a conference paper at ICLR 2022
1) It hardly captures the vertical perturbation of test data induced by Zy and the obtained result produces small variance at every temporal states.
2) It hardly captures the horizontal perturbation of test data induced by Z1, Z2, and the obtained result produces the temporally unmatched trajectories.
These phenomenons occurred due the deterministic property of the ODE-RNN model, where the dynamical transition their model is posed as the ODEs that cannot express the stochastic variation.
E(Y]
E
Test Data, CR-95
Prediction, CR-95
1.25
1,00
0.50
0.25
-0.25
0.50
Figure 6: ODE-RNN.
E[X]
EXX,]
Test Data, CR-95
Prediction, CR-95
1.25
1.00
0.75
0.00
-025
0.50
Figure CSDE-TP.
CSDE-TP. Figure 7 shows the result of the proposed CSDE-TP model and shows the advantages of adopting the SDE in modelling stochastic dynamics, Compared to the results the ODE-RNN, the proposed method accurately captures both the vertical/horizontal perturbations and recover the 95% confidence region. It is clear that our CSDE-TP delicately expresses the complex volatility of stochastic trajectories.
Discussions. As aforementioned in Section 4.3 experimental results on synthetic stochastic data show that the MSE is not the best metric train/evaluate the time-series models if there exists the
21

---

Published as a conference paper at ICLR 2022
high volatility in the dataset. In this case, distributional metrics such MMD and Wasserstein distance can good substitutes for training/evaluating stochastic data.
A.10 FUTURE WORK
We plan extend the proposed CSDE model general controlled Markov Itó-Lévy jump diffusion model (Øksendal & Sulem (2007) to delicately express the complex time-series data. For example, the proposed CSDE can be generalized to the Markov Itô-Lévy jump diffusion of the following form:
dxp = b a(0)) + ot,, a())dd,, / r(t, ()N dZ),
(39)
where N(t, z) Eocost (n= 7h- ), and Poisson random measure Tj- As the previous work in Jia & Benson (2019) show the effectiveness of the jump process modelling complex discontinuous dynamics, we believe this generalization will produce the comparable results and broaden our understanding in modelling dynamical systems for time-series data.
22


Reviews:
[{"review_id": "onzWaP0rqXV", "reviewer": "Reviewer_NnV6", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper formulates a stochastic differential equation that is controlled by $M$ external agents over compact, non-overlapping time intervals. Due to the lack of overlap between the agents, there is only one agent controlling the dynamics at each sub-interval, a notion that the authors call \"temporal privacy\". The goal, as in works using the neural SDE framework, is to learn stochastic time-series data using this model. Using a stochastic optimal control formulation, the authors use this notion of temporal privacy to decompose the optimization into sub-problems which allows for separate optimization of each agent. \n\nHowever, carrying out the training of this model is nontrivial. To train, the authors first use a loss function that employs random stopping times to efficiently use the data $\\boldsymbol{y}$. The inference then proceeds similarly, propagating the sampled points according to the learned dynamics and subsequently averaging the results. A second loss that describes the evolution of the loss backward in time is used to augment this first loss. I think the connection of the second loss (MBcond) to the nonlinear Feynman-Kac theorem could be better explained in the main text. \n\nThe authors carry out several numerical experiments and achieve impressive results. In all cases the CSDE-TP significantly outperforms alternative strategies.  \n\nMinor comments:\n\nFigure 1 caption: \"It computes the ...\" what is \"it\"?\n\nTypo: \"is shoen in\"\n\n\"main idea is ... to minimize incoherence\" what does this mean? can this notion of incoherence be made more precise?\n\nThe phrase \"rich information\" comes up a few times. Perhaps it is better to simply say \"information\" \n\n\n\n", "summary_of_the_paper": "This paper presents a framework for learning a stochastic dynamics from observed trajectories. As opposed to conventional strategies based on recurrent neural networks, the procedure learns a model of the stochastic dynamics in real space. Secondly, in contrast to recent approaches based on neural ODEs / SDEs, the training is temporally localized through windowing functions that the authors refer to as temporal privacy. The main technical innovations involve the development of new loss functions that make learning with temporal privacy tractable. ", "summary_of_the_review": "The author introduce an original and highly effective optimization scheme for controlled SDEs for modeling stochastic dynamics. While there is not a thorough analysis of the algorithm (aside from a discussion of the theoretical optimality in the appendix) the experiments amply demonstrate that the approach is productive. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "nx-pcrVcEuM", "reviewer": "Reviewer_WGcS", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper looks pretty solid and novel in the theoretical part. The MBcond loss built on the forward-backward stochastic differential equations and implemented using deep neural nets are new to the area of time series analysis/prediction. The authors also explain/analyze connections between the proposed model and the HJB equation and the non-linear Feynman-Kac theorem. \n\n \n\nMajor issues: \n\n1. Motivation of temporal privacy is not clear. For each agent, it seems that it will only be valid for a specific time interval. The authors may want to explain when a control agent will kick in and when it will be invalid. At the time a control agent kicks in, will it cause jump of the stochastic process and is it still a Markov process in this case? \n\nIn addition, the 'temporal privacy' constrains the agents cannot simultaneously control the dynamics, which requires the equations (1) be a specific controlled stochastic differential equation. Therefore, the multiple control agents can be considered as one single agent.\n\n2. Eq. (4), why the expected terminal loss J given X_t^{\\alpha} is optimized using \\alpha^{-r}? Due to causality,  I think it should only depend on partial control agents which are valid after time u. \n\n3. Page 4, Definition 1. For each observation y, the dependent forward loss is calculated till time t as shown in the integration. So observations sampled at the time far away from t usually will incur larger losses due to drift and diffusion effects accumulated over longer time span. So observations will have different weight or impact on the training process, depends on time spans between observations and test data at time t. I understand that the authors want to evaluate prediction impact by starting CSDE from each observation on the test data. However, I am wondering whether it is the best way to define forward loss. Can we just calculate the forward loss between any two consecutive or adjacent observations y? By this way, we can still make full use of all observations to train control agents. \n\n4. Eq. (10), the authors jump to FBSDEs too quickly and it is hard for readers not familiar with this topic to understand. A diagram showing correspondence between forward/backward stochastic processes X/Z and related functions will help readers a lot. \n\n5. Time series data usually show characteristics like level, trend, cycles and holiday/promotion effects. Also in many real applications, time series data come with covariates which may be very predictive for target variables. State-of-the-art algorithms for time series prediction like DeepAR, NBeats, Prophet and Transformer, to name few, take some of the factors into account in the modeling process. It would be interesting to see discussions on how the proposed method handles these factors. Is it possible that different control agents are trained to be experts specialized on different patterns (e.g. trend and cycles) ? \n\n6. In the experiment, an ablation study on the MFcond loss and MBcond loss is encouraged, e.g., use only MFcond loss or MBcond loss to evaluate the algorithm.  It will help the readers to understand how important each loss is. \n\n7. On page 4, in the paragraph 2) Theoretical Optimality, the authors should point out the regularity conditions for the function $V(\\cdot)$ while using the Verification Theorem or specify the regularity conditions on the function  $l(\\cdot, \\cdot)$ and $$\\Psi(\\cdot), and I think most of the commonly used metric functions in machine learning theory would confirm enough regularity.\n \n8. In Section A.1, I cannot see the necessity to mention the reverse SDE, which has nothing to do with the stochastic control theory the authors referred to in the work. And I think the comparison between the reverse SDE and BSDE is not a related work, I cannot see the motivations or ideas they provide in the work.\n\nMinor issues: \n\n1. Sec. 3.1, define or list reference of \"controled F_t – adapted process\". \n\n2. Sec. 3.1, define variables d and m. \n Following equation (1), the definitions of the coefficient functions b,\\sigma,\\alpha^i have typos,\nshould be written as $b:[0,T]\\times \\mathbb{R}^d\\times\\mathbb{A}\\to \\mathbb{R}^{d}$,  $\\sigma:[0,T]\\times \\mathbb{R}^d\\times\\mathbb{A}\\to \\mathbb{R}^{d\\times d}$ and $\\alpha^i:[0,T]\\times \\mathbb{R}^d\\times\\mathbb{R}^m\\to \\mathbb{R}^n$. And here the admissible control set  $\\mathbb{A}$ is not clarified until I read the appendix.\n\n3. Page 3, first line, what is k? \n\n4. Eq. (3), T is replaced by u in the integration. Please explain the replacement. Also I think J(t, X_t^{\\alpha}) should be j(T, X_t^{\\alpha}) . \n\n5. Eq. (6), there is no y(.) shown in the equation. \n\n6. Eq. (9), what is s in the equation and what is the difference between s and t_s? \n\n7. In the last paragraph on page 2, the 'sub-intervals' of the ordered times $\\mathbb{T}$ is confusion. I guess the authors mean: the closed sub-intervals without intersection interiors, or just $[t_k,t_{k+1}], 1\\le k\\le N$.\n\n8. Another typos in line on page 3: in the expression $w_r(s) = \\mathbf{1}_{t\\le s\\le u}$,  $t, u $ should belong to $\\mathbb{T}$, and $1\\le k \\le N$ should be removed.\n\n9. In Definition 1, the stopping time $\\tau_s$ is not well-defined, and I guess the author means : $\\tau_s := \\inf_{t}\\{t>s: l(t,\\mathcal{T}_{s,t}^\\alpha)>\\epsilon\\}\\bigwedge t$, and the parameter $\\epsilon$ should be mentioned in the definition.\n\n10. On page 7, in Section 4, in Table 1, what does $$\\Sigma^i mean? Does the Vanilla case indicate that the diffusion term has the same form as the drift term and depends on the controls as well?", "summary_of_the_paper": "The authors propose a novel method to model stochastic dynamic of complex time series. They build connections between neural SDE and stochastic optimal control theory. By using the proposed MFcond and MBcond losses, they can train control agents to learn dynamics of time series more accurately than existing methods. ", "summary_of_the_review": "The authors propose  a new method for learning dynamics from continuous-time data using neural Markov controlled SDE. It connects stochastic optimal control theory and deep learning which is pretty novel in my opinion. So my recommendation is \"accept\". ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "cic0gAjAy8", "reviewer": "Reviewer_poqQ", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n+ The paper concerns a timely topic. The stochastic view of the continuous-time systems makes it even more interesting as ordinary differential equations based approaches dominate the recent literature for the time being.\n+ The empirical findings are strong. The model seems to excel among state-of-the-art continuous-time methods in standard benchmarks.\n+ We also empirically observe that the proposed MFcond significantly boosts the accuracy. Note that this comparison is made against the vanilla optimization objective (6). Since it is known that training continuous-time systems with long sequences suffer from optimization issues; thus, it would be more appropriate to consider an additional baseline in which the training is performed using subsequences (i.e., minibatches along the time dimension).\n\nWeaknesses:\n- Since the model addresses difficult sub-problems along the way, these sub-problems and appearing hyperparameters require a more in-depth investigation/explanation. In turn, this would allow us to appreciate the methodology and different choices the authors have made. This can be done, for instance, by simple empirical evaluations (as in Fig 4.4a) or by contrasting the proposed strategies against any simpler, alternative techniques. Unfortunately, as such, I'm not able to judge how much is gained by the proposed complicated and somewhat demanding procedures. Below are a few simple example questions:\n    - What happens when we discard the MFcond? More generally, what is the role of $\\gamma$ and how did you set it $\\gamma=0.95$?\n    - How do you pick $u$ in (3)? Does the optimization become more difficult/easy with $u$ approaching $t$?\n    - How does the value for $r$ affect the entire routine defined in (4)? Did you randomly pick $r$\n    - How does $\\epsilon$ affect the algorithm? Why is it set as in Alg.1?\n    - What if we simply optimize for shorter sequences instead of (6)?\n    - Setting the initial value to an observation (as in (6),(7)) may deteriorate the performance if the data is noisy. Is not this an issue after all? \n    - What is the justification for replacing $Z_t$ with $Z_t^\\alpha$? Since this directly violates the \"optimal agent\" requirement of (11), this particular choice needs to be carefully explained, possibly paired with a simple numerical illustration showing how it affects (10).\n- Writing and notation should be significantly improved. Below are concrete suggestions:\n    - Since (1) involves several terms, more verbal explanation would help the reader very much. Also, short explanations on \"Markov closed-loop feedback control\" and \"$\\mathcal{F}_t$-adapted process\" are needed.\n    - A cartoon drawing (or anything similar to Figure 1) of Sec 3.1 would be nice.\n    - What are the expectations with respect to in (3)?\n    - A verbal explanation of (7) would be very useful.\n    - The caption of Fig 1 should be improved. It is not clear what \"other trajectories\", \"it\", and \"empty parts\" refer to.\n    - Which distribution is the expectation in (8) with respect to? Does not $\\mathbf{y}(\\cdot)$ refer to data trajectories? \n    - It would be much easier to follow if the paragraph above \"Network inference\" is given much earlier than equations.\n    - The first paragraph in page 6 is very long and difficult to follow.\n    - Sec3.4 could start with a reminder of MBcond as it is not related to the methodology described in Sec 3.3.\n    - What exactly does \"cancels the effect of martingales in the diffusion term\" mean?\n    - Writing (12) in terms of differentials (as in (10)) instead of an integral would help the reader to contrast (10) and (12).\n    - What are the inputs to $\\mathcal{L}_f$? Just $\\alpha$'s as in (14) or $\\mathbf{y}(\\cdot)$ as well as in (8)? Also, why is the $\\alpha^i$ separated from the other $\\alpha$'s in (14)?\n    - Better titles appearing in Fig 4a legend (maybe reflecting the number of agents) would be nice.\n\nMore detailed comments and questions:\n- Why are the Q1 and Q2 important? Q1 needs further explanation and Q2 requires an explanation on the deficiencies of neural controlled differential equations (NCDE).\n- NCDE should be mentioned in A1 and the proposed method should be motivated in comparison with NCDE.\n- What prevents the agents from collapsing to a single mode?\n- Why is only the train MSE plotted in Fig 4a? Also, it would be nice to see the model trained until convergence.", "summary_of_the_paper": "This paper presents a new approach to train stochastic dynamical systems using a set of tools from stochastic optimal control theory. The methodology is built upon _controlled stochastic differential equations_ with multiple control agents that modulate the dynamical evolution. Each agent is deliberately chosen to be \"active\" in a certain time interval, which leads to so-called \"temporally private\" dynamics that allow for dynamic programming based optimization. Since the temporal segmentation of optimization objective requires arbitrary intermediate state $X_s$ with $0<s<T$ as initial values, the Markov forward condition (MFcond), which computes some sort of a mean value for future states, is proposed. To further stabilize the training and compensate for the methodology's lacking in theoretical optimality guarantees, the authors augment the original loss with the Markov backward conditional (MBcond) loss. The method is shown to outperform competing continuous-time methods on standard benchmarks. ", "summary_of_the_review": "The paper brings several interesting approaches to address the sub-problems that appear due to the construction of the SDE controlled by multiple agents. As much as I can follow, the equations are accurate and the methodology does not suffer from any theoretical deficiency. On the other hand, this makes it very difficult to evaluate how important different assumptions and building blocks are. As such, the model seems like a magical combination of a number of blocks but it would be much better to isolate each block individually and show how it contributes to the overall performance. Also, I believe the writing should be significantly improved (see my detailed notes in the \"main review\"). Although the method is interesting and timely, I recommend a reject and re-submission after the above-mentioned points are addressed.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "afPppetHc3", "reviewer": "Reviewer_hDHJ", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper seems to be well-structured and the math seems solid. The proposed loss function MFcond is a principled way to incorporate the intermediate observations into the dynamics. The experiments shows significant gains over several state-of-the-art baselines. I have a few comments and concerns:\n - The motivation behind modelling a time series such as stock price as controlled by multiple agent is not clear.\n - It is not clear to me whether multiple agents can be active at the same time. From the definition and the separation of B and B' in equation 4, it seems that the attention interval of two different agents are exclusive, but the example in figure 4 seems to show that agents control overlapped intervals.\n - This paper does not clearly define how the attention of each agent is set for empirical datasets. From the context it seems that each agent controls one interval between two empirical observations, but this need to be explicitly defined.\n - The figure 1-(b) shows a failure case where all simulated trajectories are far from the ground truth, yet the \"averaged\" prediction operator $\\mathcal{T}_{s,t}^{\\alpha}$ is close to the ground truth. Doesn't this mean that l2 distance between the ground truth and the mean predictor is a poor loss function?\n - In figure 1 the observation is plotted in a solid black line. I think scatter plot is more appropriate since the observations are at discrete time.", "summary_of_the_paper": "This paper proposes to model a time series directly with a neural controlled stochastic differential equation with multiple control agents. It introduces a concept of temporal privacy, which defines the attention of each control agent to be a certain time interval. Then the authors introduce Markov dynamic programming to efficiently minimize a loss function defined in terms of trajectory of the stochastic differential equations. In order to make the proposed temporal privacy work, the authors proposed two loss function: 1) MFcond minimizes the incoherence between the future estimates starting at any intermediate time stamp, 2) MBcond is to ensure the theoretical optimality of the control agents. ", "summary_of_the_review": "This paper proposes some novel ideas that works well in practice. I don't fully understand this paper, but I feel it deserve a broader audience.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors propose to combine ideas from SDEs and time series modeling with stochastic optimal control to present a framework for modeling continuous-time stochastic dynamics. The reviewers are in agreement that there are several good ideas presented here and that the interface of the perspectives the authors combine toward their proposed framework is an interesting one to explore. One referee mentions valid concerns in confusing points of the details in the presentation, and the positive reviewers echoed these concerns. In particular, more details and clearer exposition are needed for the decomposition into the subproblems and the problem of many hyper parameters. Nonetheless, my overall impression after a careful read of the paper and discussion is that these concerns are addressable and are to a degree ameliorated by the author response, and that they may be viewed as limitations outweighed by the merits of the novel ideas presented here. I emphasize that all reviewers were surprisingly consistent in their assessment of the shortcomings, and I encourage the authors to take these constructive criticisms seriously in preparing a final version of this paper.
Decision: Accept (Poster)