Neural Markov Controlled SDE: Stochastic Optimization for Continuous-Time Data

1 INTRODUCTION . Recently , there has been interest in using continuous dynamical systems to approximate complex time series . Neural ODE Chen et al . ( 2018 ) , which opened the way for continuous representation of neural networks , have been widely investigated and thoroughly analyzed by Massaroli et al . ( 2020 ) . As the stochastic generalization of ODE , Neural SDEs Li et al . ( 2020 ) have been proposed by regarding intrinsic stochasticity in data representations ( e.g. , stock market data ) . Since the conventional Neural ODE/SDEs only utilize the initial information of trajectories when propagating dynamics , modelling complex time-series with naive Neural ODE/SDEs has been regarded as inefficient and undesirable choices , as pointed out by Kidger et al . ( 2020 ) . To address these problems , Rubanova et al . ( 2019 ) presented an auto-regressive model to generalize recurrent neural networks ( RNNs ) to have continuous hidden dynamics with neural ODE . Furthermore , Chen et al . ( 2018 ) proposed an encoder-decoder structure with Neural ODE in the latent space to reconstruct/predict complex data representation . Although the aforementioned approaches produce remarkable results , they focus on suggesting additional probabilistic structures rather than improving the learnability of the Neural ODE model itself . Compared to aforementioned approaches , we focus on solving the fundamental issues of Neural ODE/SDEs . First , we raise two important questions . Q1 ) How can we construct an efficient network architecture for Neural ODE/SDE models that do not require additional recurrent networks to model complex time series ? Q2 ) How can we train Neural ODE/SDEs that can utilize richer information of observed sequences to accurately generate complex time series ? As SDEs can be posed as stochastic generalizations of ODEs , we focus on a stochastic framework and adopt the stochastic optimal control theory as our primary analysis tool for the rigorous and systematic analysis of the aforementioned problems . Keeping this in mind , the contributions of our paper are to answer the above two questions . A1 ) Novel probabilistic framework for stochastic dynamics . We propose a novel neural controlled stochastic differential equation ( CSDE ) to model the complex stochastic time series , where multiple control agents are defined to construct local dynamics in their own private temporal states . With this property , the proposed CSDE incorporates Markov dynamic programming , enables our model to directly infer the complex trajectory on data space rather than the latent space without any extra network ( e.g. , encoders/decoders ) , and shows remarkable efficiency compared to existing methods . A2 ) Novel conditional losses . We introduce a novel Markov forward conditional ( MFcond ) loss to utilize multi-conditioned dynamics instead of the conventional dynamics determined by partial initial conditions . The proposed MFcond loss makes our method to model the complex information of time-sequential data . To impose regularization and to ensure the optimality of control agents , we also suggest a novel Markov backward conditional ( MBcond ) loss . 2 RELATED WORK . ODE As a Latent Probabilistic Model . Rubanova et al . ( 2019 ) suggested an ODE-RNN by combining RNN with the latent dynamics induced by the Neural ODE . To deal with irregular time-stamps , exponential-decaying of the hidden states was also discussed by Che et al . ( 2018 ) . De Brouwer et al . ( 2019 ) assumed that the observations are sampled from the stochastic dynamics induced from SDEs and introduced GRU-ODE to approximate the observed stochastic time series . SDE As a Latent Probabilistic Model . Liu et al . ( 2021 ) incorporated Neural SDEs with recurrent models as a primary probabilistic dynamical model to generate stochastic continuous-time latent variables . While this SDE model could describe the stochastic dynamics on the latent space with recurrent structures ( e.g. , RNN encoder/decoder ) , it required a whole sequence of historical observations as inputs to the model . Unfortunately , this type of formulation leads to non-Markov types of SDEs , which makes it difficult to analyze the probabilistic characteristics of the dynamics . Unlike this model , we focus on the Markov SDEs while maintaining identical objectives . Neural CDE and RDE . Kidger et al . ( 2020 ) proposed a data-driven neural controlled differential equation called Neural CDE to incorporate a rough-path analysis theory and model complex time series . Morrill et al . ( 2021 ) extended the rough-path theory with a Neural RDE to deal with the continuous time series over long time . Generative SDE Models . Recently , Kidger et al . ( 2021 ) suggested SDE-based generative adversarial networks ( GANs ) . Park et al . ( 2021 ) utilized the temporal conditional Wasserstein distance to construct GANs for time-series generation . Please refer to Appendix A.1 for additional discussion on related works . 3 MARKOV NEURAL CONTROLLED SDE . In Section 3.1 , we introduce a novel SDE model that considers temporally private agents . In Section 3.2 , we propose the Markov-DP-TP framework to efficiently solve the stochastic optimal control problem with the proposed neural SDE model . Finally , we suggest novel Markov conditional forward and backward losses in Section 3.3 and 3.4 , respectively . In the Appendix , we provided the detailed technical definitions . 3.1 CONTROLLED STOCHASTIC DIFFERENTIAL EQUATIONS . The basic object of our interest is a controlled Ft-adapted process Xαt with multiple control agents α = { α1 , · · · , αM } ∈ A where A denotes the set of admissible control agents . In particular , the stochastic process Xαt is defined as a solution to the following CSDE : dXαt = M∑ i=1 wi ( t ) b i ( t , Xαt , α i ) dt+ M∑ i=1 wi ( t ) σ i ( t , Xαt , α i ) dWt , ( 1 ) where b and σ : [ 0 , T ] ×Rd×A→ Rd are the drift and diffusion functions , respectively . Each control agent αi : [ 0 , T ] × Rd × Rm , αi = αi ( t , Xt ; θi ) , ∀1 ≤ i ≤ M is defined as a Markov closed-loop feedback control , which is parameterized by the neural network θi . While every agent is defined as a closed-loop feedback-type Carmona ( 2016b ) , the solution to the CSDE above , Xαt , is the Markov process , which means that process Xαt is propagated using the information of the current state . Let T = { tk } 1≤k≤N be a set of ordered times1 such that 0 = t1 < · · · < tk < tl < · · · < tN = T . The set of functions { wi ( t ) } 1≤i≤M is defined as an indicator function on the intervals , wi ( t ) = 1tk≤t≤tl with predetermined starting/ending points tk , tl in T. We call this function temporal privacy ( TP ) because it represents each agent ’ s attention on different sub-intervals . Overall , in ( 1 ) , the stochastic process Xαt is propagated by summing M -number of individual agent ’ s weighted attentions { ∑M wib i ( · , · , αi ) , ∑M wiσ i ( · , · , αi ) } . To understand the behavior of the proposed CSDE more deeply , we consider the following detailed example : 1The time interval dt ≈ ∆t = |tk − tl| for any k , l can be set regularly/irregularly in our method . Role of Temporal Privacy . We define wr ( s ) = 1t≤s≤u , t , u ∈ T with r ≤ M . Then , Xαu in ( 1 ) given Xt at an interval [ t , u ] can be equivalently rewritten in the integration form : Xα= [ α 1 , ··· , αM ] u = X α t + ∫ u t M∑ i wi ( s ) b i ( s , Xαs , α i ) ds+ ∫ u t M∑ i wi ( s ) σ i ( s , Xαs , α i ) dWs = Xαt + ∫ wr ( s ) br ( s , Xαs , α r ) ds+ ∫ wr ( s ) σr ( s , Xαt , α r ) dWs = X αr u . ( 2 ) In ( 2 ) , the activated control agent to evaluate the stochastic process Xαu for the interval [ t , u ] is only αr ( i.e. , Xαu = X αr u ) owing to the definition of the weighting function w ( · ) ( t ) . This means that the remaining control agents { αj } j 6=r are not used for the evaluation of the stochastic process in the sub-interval [ t , u ] . While each agent αi is activated at its own private sub-interval , this leads our method to adopt dynamic programming ( DP ) to train Neural CSDEs in the form of ( 1 ) . In this paper , we aim to solve the optimal control problem via DP with multiple agents , where each agent specializes in solving a particular sub-problem in its private interval . 3.2 MARKOV DYNAMIC PROGRAMMING PRINCIPLES . The dynamic programming principle is one of the fundamental philosophies for dealing with stochastic optimal control problems . Its basic idea is to consider a family of sub-problems with different initial times/states and establish the relation among the sub-problems to systemically solve them . Using the mathematical property of the proposed CSDE with TP , we present an efficient learning strategy to solve stochastic optimal control problems via Markov dynamic programming ( Markov-DP ) . In this paper , we aim to solve the stochastic optimal control problem by training control agents α = [ α1 , · · · , αM ] and minimizing the cost functional J ( t , Xαt ) : [ 0 , T ] × Rd → R+ : J ( t , Xαt ) = E [ ∫ T t l ( s , Xαs ) ds+ Ψ ( X α T ) ∣∣∣∣∣Ft ] = E [ ∫ u t l ( s , Xαs ) ds+ J ( u , X α u ) ∣∣∣∣Xαt ] , ( 3 ) where l : [ 0 , T ] ×Rd → R+ is the running cost ( e.g. , L2 loss ) that computes the discrepancy between the propagated process Xαt and the observed data point yt at each time t , Ψ ( X α T ) : Rd → R+ is the terminal cost that estimates the discrepancy between the terminal state and the data yT . To evaluate the cost functional J ( t , Xαt ) at time t with control agents α , the running cost is integrated over the time interval [ t , T ] conditioned on filtration Ft . Note that the expectation conditioned on Ft in ( 3 ) can be substituted to the expectation conditioned on Xαt in light of the Markov property presented in Section A.2 , and the cost functional at time t only depends on the current state of the process Xαt . Markov-DP with Temporal Privacy . By combining the tower property of the conditional expectations with the dynamic programming principle and Itô ’ s formula ( Oksendal ( 1992 ) ) , one can show that a minimization problem can be recursively decomposed into sub-problems owing to the property of TP in our proposed CSDE : V ( t , Xαt ) , inf α J ( t , Xαt ) = inf α E [ ∫ u t l ( s , Xαs ) ds+ J ( u , X α u ) ∣∣∣∣Xαt ] ︸ ︷︷ ︸ ( A ) = inf αr E [ ∫ u t l ( s , Xαs ) ds ∣∣∣∣Xαt ] ︸ ︷︷ ︸ ( B ) + inf α ( −r ) E [ J ( u , Xαu ) |Xαt ] ︸ ︷︷ ︸ ( B ’ ) , ( 4 ) where V is an optimal cost functional ( i.e. , value function ) , αr denotes the r-th control agent , and α ( −r ) = [ α1 , · · · , 0 , · · · , αM ] indicates the set of remaining agents ( the r-th component is zero ) . In ( 4 ) , the minimization problem ( A ) over α is divided into two sub-problems using the dynamic programming principle , which are ( B ) and ( B ’ ) . Because the minimization problem ( B ) is only dependent on the control agent αr parameterized by the neural network θr , we compute the gradient descent of θr to solve the sub-problem ( B ) : θrk+1 = θ r k − ∂ ∂θr E [ ∫ { s : wr ( s ) =1 } l ( s , X αr ( · , · , θrk ) s ) ds ∣∣∣∣∣Xαt ] , ( 5 ) where wr ( s ) = 1t≤s≤u is the TP function at an interval [ t , u ] and k is the index for the learning iterations . In ( 5 ) , the r-th control agent αr minimizes the cost functional using the gradient descent scheme at its own temporal sub-interval . As the remaining sub-problem ( B ’ ) over agents α ( −r ) can also be recursively decomposed into smaller sub-problems using the dynamic programming principle , the original problem ( A ) is solved separately with M -number of control agents α = { α1 , · · · , αM } with the M -number of gradient descent schemes . This indicates that we can obtain the set of agents α ? = { αi ( · , · ; θi ? ) } by collecting individual optimal agents with sub-problems . In this paper , we combine the Markov-DP with M gradient descent schemes in ( 5 ) and CSDE with TP in ( 1 ) and introduce a novel Markov-DP-TP framework . In the numerical experiments in Section 4.4 , we show that the proposed Markov-DP-TP framework remarkably increases the model efficiency compared to conventional non-DP naive approaches , which makes our method directly model the complex time series in the data space . However , despite the improvements with our novel Markov-DP-TP framework , there exist remaining practical/theoretical issues that should be addressed to solve the optimal control problem with complex datasets . 1 ) Conditional Dependency . The main practical issue in implementing the Markov-DP-TP framework is that explicit conditional states are not given , e.g. , Xαt in ( 5 ) . As different initial/terminal conditions of SDE lead to totally different behaviors of induced dynamics , well-designed conditional information is a crucial factor in training the Neural CSDE for specific applications . In Section 3.3 , we introduce the Markov Forward conditional ( MFcond ) loss to train the Neural CSDE with well-posed conditional information that ensures accurate network predictions . 2 ) Theoretical Optimality . In the optimal control theory , there are well-known partial differential equations called Hamiltonian-Jacobi-Bellman ( HJB ) equations , which assure the theoretical optimality of control agents . If the control agents can solve the HJB equation , the proposed method attains the optimal state Vt ( Xαt ) = infα Jt ( X α t ) = Jt ( X α ? t ) . However , the optimal agents α ? of the proposed CSDE with gradient descent are not generally equivalent to the solution to the HJB equation . In Section 3.4 , we propose the Markov Backward conditional ( MBcond ) loss to assure the optimality of control agents and to provide information in backward dynamics for regularization .

Summary: This paper presents a new approach to train stochastic dynamical systems using a set of tools from stochastic optimal control theory. The methodology is built upon _controlled stochastic differential equations_ with multiple control agents that modulate the dynamical evolution. Each agent is deliberately chosen to be "active" in a certain time interval, which leads to so-called "temporally private" dynamics that allow for dynamic programming based optimization. Since the temporal segmentation of optimization objective requires arbitrary intermediate state $X_s$ with $0<s<T$ as initial values, the Markov forward condition (MFcond), which computes some sort of a mean value for future states, is proposed. To further stabilize the training and compensate for the methodology's lacking in theoretical optimality guarantees, the authors augment the original loss with the Markov backward conditional (MBcond) loss. The method is shown to outperform competing continuous-time methods on standard benchmarks.