Neural Markov Controlled SDE: Stochastic Optimization for Continuous-Time Data

1 INTRODUCTION . Recently , there has been interest in using continuous dynamical systems to approximate complex time series . Neural ODE Chen et al . ( 2018 ) , which opened the way for continuous representation of neural networks , have been widely investigated and thoroughly analyzed by Massaroli et al . ( 2020 ) . As the stochastic generalization of ODE , Neural SDEs Li et al . ( 2020 ) have been proposed by regarding intrinsic stochasticity in data representations ( e.g. , stock market data ) . Since the conventional Neural ODE/SDEs only utilize the initial information of trajectories when propagating dynamics , modelling complex time-series with naive Neural ODE/SDEs has been regarded as inefficient and undesirable choices , as pointed out by Kidger et al . ( 2020 ) . To address these problems , Rubanova et al . ( 2019 ) presented an auto-regressive model to generalize recurrent neural networks ( RNNs ) to have continuous hidden dynamics with neural ODE . Furthermore , Chen et al . ( 2018 ) proposed an encoder-decoder structure with Neural ODE in the latent space to reconstruct/predict complex data representation . Although the aforementioned approaches produce remarkable results , they focus on suggesting additional probabilistic structures rather than improving the learnability of the Neural ODE model itself . Compared to aforementioned approaches , we focus on solving the fundamental issues of Neural ODE/SDEs . First , we raise two important questions . Q1 ) How can we construct an efficient network architecture for Neural ODE/SDE models that do not require additional recurrent networks to model complex time series ? Q2 ) How can we train Neural ODE/SDEs that can utilize richer information of observed sequences to accurately generate complex time series ? As SDEs can be posed as stochastic generalizations of ODEs , we focus on a stochastic framework and adopt the stochastic optimal control theory as our primary analysis tool for the rigorous and systematic analysis of the aforementioned problems . Keeping this in mind , the contributions of our paper are to answer the above two questions . A1 ) Novel probabilistic framework for stochastic dynamics . We propose a novel neural controlled stochastic differential equation ( CSDE ) to model the complex stochastic time series , where multiple control agents are defined to construct local dynamics in their own private temporal states . With this property , the proposed CSDE incorporates Markov dynamic programming , enables our model to directly infer the complex trajectory on data space rather than the latent space without any extra network ( e.g. , encoders/decoders ) , and shows remarkable efficiency compared to existing methods . A2 ) Novel conditional losses . We introduce a novel Markov forward conditional ( MFcond ) loss to utilize multi-conditioned dynamics instead of the conventional dynamics determined by partial initial conditions . The proposed MFcond loss makes our method to model the complex information of time-sequential data . To impose regularization and to ensure the optimality of control agents , we also suggest a novel Markov backward conditional ( MBcond ) loss . 2 RELATED WORK . ODE As a Latent Probabilistic Model . Rubanova et al . ( 2019 ) suggested an ODE-RNN by combining RNN with the latent dynamics induced by the Neural ODE . To deal with irregular time-stamps , exponential-decaying of the hidden states was also discussed by Che et al . ( 2018 ) . De Brouwer et al . ( 2019 ) assumed that the observations are sampled from the stochastic dynamics induced from SDEs and introduced GRU-ODE to approximate the observed stochastic time series . SDE As a Latent Probabilistic Model . Liu et al . ( 2021 ) incorporated Neural SDEs with recurrent models as a primary probabilistic dynamical model to generate stochastic continuous-time latent variables . While this SDE model could describe the stochastic dynamics on the latent space with recurrent structures ( e.g. , RNN encoder/decoder ) , it required a whole sequence of historical observations as inputs to the model . Unfortunately , this type of formulation leads to non-Markov types of SDEs , which makes it difficult to analyze the probabilistic characteristics of the dynamics . Unlike this model , we focus on the Markov SDEs while maintaining identical objectives . Neural CDE and RDE . Kidger et al . ( 2020 ) proposed a data-driven neural controlled differential equation called Neural CDE to incorporate a rough-path analysis theory and model complex time series . Morrill et al . ( 2021 ) extended the rough-path theory with a Neural RDE to deal with the continuous time series over long time . Generative SDE Models . Recently , Kidger et al . ( 2021 ) suggested SDE-based generative adversarial networks ( GANs ) . Park et al . ( 2021 ) utilized the temporal conditional Wasserstein distance to construct GANs for time-series generation . Please refer to Appendix A.1 for additional discussion on related works . 3 MARKOV NEURAL CONTROLLED SDE . In Section 3.1 , we introduce a novel SDE model that considers temporally private agents . In Section 3.2 , we propose the Markov-DP-TP framework to efficiently solve the stochastic optimal control problem with the proposed neural SDE model . Finally , we suggest novel Markov conditional forward and backward losses in Section 3.3 and 3.4 , respectively . In the Appendix , we provided the detailed technical definitions . 3.1 CONTROLLED STOCHASTIC DIFFERENTIAL EQUATIONS . The basic object of our interest is a controlled Ft-adapted process Xαt with multiple control agents α = { α1 , · · · , αM } ∈ A where A denotes the set of admissible control agents . In particular , the stochastic process Xαt is defined as a solution to the following CSDE : dXαt = M∑ i=1 wi ( t ) b i ( t , Xαt , α i ) dt+ M∑ i=1 wi ( t ) σ i ( t , Xαt , α i ) dWt , ( 1 ) where b and σ : [ 0 , T ] ×Rd×A→ Rd are the drift and diffusion functions , respectively . Each control agent αi : [ 0 , T ] × Rd × Rm , αi = αi ( t , Xt ; θi ) , ∀1 ≤ i ≤ M is defined as a Markov closed-loop feedback control , which is parameterized by the neural network θi . While every agent is defined as a closed-loop feedback-type Carmona ( 2016b ) , the solution to the CSDE above , Xαt , is the Markov process , which means that process Xαt is propagated using the information of the current state . Let T = { tk } 1≤k≤N be a set of ordered times1 such that 0 = t1 < · · · < tk < tl < · · · < tN = T . The set of functions { wi ( t ) } 1≤i≤M is defined as an indicator function on the intervals , wi ( t ) = 1tk≤t≤tl with predetermined starting/ending points tk , tl in T. We call this function temporal privacy ( TP ) because it represents each agent ’ s attention on different sub-intervals . Overall , in ( 1 ) , the stochastic process Xαt is propagated by summing M -number of individual agent ’ s weighted attentions { ∑M wib i ( · , · , αi ) , ∑M wiσ i ( · , · , αi ) } . To understand the behavior of the proposed CSDE more deeply , we consider the following detailed example : 1The time interval dt ≈ ∆t = |tk − tl| for any k , l can be set regularly/irregularly in our method . Role of Temporal Privacy . We define wr ( s ) = 1t≤s≤u , t , u ∈ T with r ≤ M . Then , Xαu in ( 1 ) given Xt at an interval [ t , u ] can be equivalently rewritten in the integration form : Xα= [ α 1 , ··· , αM ] u = X α t + ∫ u t M∑ i wi ( s ) b i ( s , Xαs , α i ) ds+ ∫ u t M∑ i wi ( s ) σ i ( s , Xαs , α i ) dWs = Xαt + ∫ wr ( s ) br ( s , Xαs , α r ) ds+ ∫ wr ( s ) σr ( s , Xαt , α r ) dWs = X αr u . ( 2 ) In ( 2 ) , the activated control agent to evaluate the stochastic process Xαu for the interval [ t , u ] is only αr ( i.e. , Xαu = X αr u ) owing to the definition of the weighting function w ( · ) ( t ) . This means that the remaining control agents { αj } j 6=r are not used for the evaluation of the stochastic process in the sub-interval [ t , u ] . While each agent αi is activated at its own private sub-interval , this leads our method to adopt dynamic programming ( DP ) to train Neural CSDEs in the form of ( 1 ) . In this paper , we aim to solve the optimal control problem via DP with multiple agents , where each agent specializes in solving a particular sub-problem in its private interval . 3.2 MARKOV DYNAMIC PROGRAMMING PRINCIPLES . The dynamic programming principle is one of the fundamental philosophies for dealing with stochastic optimal control problems . Its basic idea is to consider a family of sub-problems with different initial times/states and establish the relation among the sub-problems to systemically solve them . Using the mathematical property of the proposed CSDE with TP , we present an efficient learning strategy to solve stochastic optimal control problems via Markov dynamic programming ( Markov-DP ) . In this paper , we aim to solve the stochastic optimal control problem by training control agents α = [ α1 , · · · , αM ] and minimizing the cost functional J ( t , Xαt ) : [ 0 , T ] × Rd → R+ : J ( t , Xαt ) = E [ ∫ T t l ( s , Xαs ) ds+ Ψ ( X α T ) ∣∣∣∣∣Ft ] = E [ ∫ u t l ( s , Xαs ) ds+ J ( u , X α u ) ∣∣∣∣Xαt ] , ( 3 ) where l : [ 0 , T ] ×Rd → R+ is the running cost ( e.g. , L2 loss ) that computes the discrepancy between the propagated process Xαt and the observed data point yt at each time t , Ψ ( X α T ) : Rd → R+ is the terminal cost that estimates the discrepancy between the terminal state and the data yT . To evaluate the cost functional J ( t , Xαt ) at time t with control agents α , the running cost is integrated over the time interval [ t , T ] conditioned on filtration Ft . Note that the expectation conditioned on Ft in ( 3 ) can be substituted to the expectation conditioned on Xαt in light of the Markov property presented in Section A.2 , and the cost functional at time t only depends on the current state of the process Xαt . Markov-DP with Temporal Privacy . By combining the tower property of the conditional expectations with the dynamic programming principle and Itô ’ s formula ( Oksendal ( 1992 ) ) , one can show that a minimization problem can be recursively decomposed into sub-problems owing to the property of TP in our proposed CSDE : V ( t , Xαt ) , inf α J ( t , Xαt ) = inf α E [ ∫ u t l ( s , Xαs ) ds+ J ( u , X α u ) ∣∣∣∣Xαt ] ︸ ︷︷ ︸ ( A ) = inf αr E [ ∫ u t l ( s , Xαs ) ds ∣∣∣∣Xαt ] ︸ ︷︷ ︸ ( B ) + inf α ( −r ) E [ J ( u , Xαu ) |Xαt ] ︸ ︷︷ ︸ ( B ’ ) , ( 4 ) where V is an optimal cost functional ( i.e. , value function ) , αr denotes the r-th control agent , and α ( −r ) = [ α1 , · · · , 0 , · · · , αM ] indicates the set of remaining agents ( the r-th component is zero ) . In ( 4 ) , the minimization problem ( A ) over α is divided into two sub-problems using the dynamic programming principle , which are ( B ) and ( B ’ ) . Because the minimization problem ( B ) is only dependent on the control agent αr parameterized by the neural network θr , we compute the gradient descent of θr to solve the sub-problem ( B ) : θrk+1 = θ r k − ∂ ∂θr E [ ∫ { s : wr ( s ) =1 } l ( s , X αr ( · , · , θrk ) s ) ds ∣∣∣∣∣Xαt ] , ( 5 ) where wr ( s ) = 1t≤s≤u is the TP function at an interval [ t , u ] and k is the index for the learning iterations . In ( 5 ) , the r-th control agent αr minimizes the cost functional using the gradient descent scheme at its own temporal sub-interval . As the remaining sub-problem ( B ’ ) over agents α ( −r ) can also be recursively decomposed into smaller sub-problems using the dynamic programming principle , the original problem ( A ) is solved separately with M -number of control agents α = { α1 , · · · , αM } with the M -number of gradient descent schemes . This indicates that we can obtain the set of agents α ? = { αi ( · , · ; θi ? ) } by collecting individual optimal agents with sub-problems . In this paper , we combine the Markov-DP with M gradient descent schemes in ( 5 ) and CSDE with TP in ( 1 ) and introduce a novel Markov-DP-TP framework . In the numerical experiments in Section 4.4 , we show that the proposed Markov-DP-TP framework remarkably increases the model efficiency compared to conventional non-DP naive approaches , which makes our method directly model the complex time series in the data space . However , despite the improvements with our novel Markov-DP-TP framework , there exist remaining practical/theoretical issues that should be addressed to solve the optimal control problem with complex datasets . 1 ) Conditional Dependency . The main practical issue in implementing the Markov-DP-TP framework is that explicit conditional states are not given , e.g. , Xαt in ( 5 ) . As different initial/terminal conditions of SDE lead to totally different behaviors of induced dynamics , well-designed conditional information is a crucial factor in training the Neural CSDE for specific applications . In Section 3.3 , we introduce the Markov Forward conditional ( MFcond ) loss to train the Neural CSDE with well-posed conditional information that ensures accurate network predictions . 2 ) Theoretical Optimality . In the optimal control theory , there are well-known partial differential equations called Hamiltonian-Jacobi-Bellman ( HJB ) equations , which assure the theoretical optimality of control agents . If the control agents can solve the HJB equation , the proposed method attains the optimal state Vt ( Xαt ) = infα Jt ( X α t ) = Jt ( X α ? t ) . However , the optimal agents α ? of the proposed CSDE with gradient descent are not generally equivalent to the solution to the HJB equation . In Section 3.4 , we propose the Markov Backward conditional ( MBcond ) loss to assure the optimality of control agents and to provide information in backward dynamics for regularization .

Summary: This paper proposes to model a time series directly with a neural controlled stochastic differential equation with multiple control agents. It introduces a concept of temporal privacy, which defines the attention of each control agent to be a certain time interval. Then the authors introduce Markov dynamic programming to efficiently minimize a loss function defined in terms of trajectory of the stochastic differential equations. In order to make the proposed temporal privacy work, the authors proposed two loss function: 1) MFcond minimizes the incoherence between the future estimates starting at any intermediate time stamp, 2) MBcond is to ensure the theoretical optimality of the control agents.