Learning Continuous-Time Dynamics by Stochastic Differential Networks

1 INTRODUCTION AND RELATED WORKS . Many real-world systems experience complicated stochastic dynamics over a continuous time period . The challenges on modeling the stochastic dynamics mainly come from two sources . First , the underlying state transitions of many systems are often uncertain , as they are placed in unpredictable environment with their states continuously affected by unknown disturbances . Second , the monitoring data collected may be sparse and at irregular intervals as a result of the sampling strategy or data corruption . The sporadic data sequence loses a large amount of information and system behaviors hidden behind the intervals of the observed data . In order to accurately model and analyze dynamics of these systems , it is important to reliably and efficiently represent the continuous-time stochastic process based on the discrete-time observations . In some domains , the derivation of the continuous-time stochastic model relies heavily on human knowledge and many studies focus on its inference problem ( Ryder et al. , 2018 ; Särkkä et al. , 2015 ) . But in more domains ( e.g. , video analysis ( Vondrick et al. , 2016 ) and human activity detection ( Rubanova et al. , 2019 ) ) , it is difficult and sometimes intractable to derive an accurate model to capture the underlying temporal evolution from the collected sequence of data . Although some studies have been made on approximating the stochastic process from the data collected , the majority of these methods define the system dynamics with a linear model ( Macke et al. , 2011 ; Yu et al. , 2009b ; a ) , which can not well represent high-dimensional data with nonlinear relationship . Recently , the Neural Ordinary Differential Equation ( ODE ) studies ( Chen et al. , 2018 ; Rubanova et al. , 2019 ; Jia & Benson , 2019 ; De Brouwer et al. , 2019 ; Yildiz et al. , 2019 ; Kidger et al. , 2020 ) introduce deep learning models to learn an ODE and apply it to approximate continuous-time dynamics . Nevertheless , these methods generally neglect the randomness of the latent state trajectories and posit simplified assumptions on the data distribution ( e.g . Gaussian ) , which strongly limits their capability of modeling complicated continuous-time stochastic processes . Compared to ODE , Stochastic Differential Equation ( SDE ) ( Jørgensen et al. , 2020 ) is a more practical solution in modeling the continuous-time stochastic process . Recently there have been some studies on bridging the gap between deep neural networks and SDEs ( Ha et al. , 2018 ) . In ( Hegde et al. , 2019 ; Liu et al. , 2020 ; Peluchetti & Favaro , 2020 ; Wang et al. , 2019 ; Kong et al. , 2020 ) , SDEs are introduced to define more robust and accurate deep learning architectures for supervised learning problems ( e.g . classification and regression ) . These studies focus on the design of neural network architectures , and are orthogonal to our work on the modeling of sporadic time series . In ( Tzen & Raginsky , 2019a ; b ) the authors studied the theoretical guarantees of the optimization and inference problems of Neural SDEs . In ( Li et al. , 2020 ) , a stochastic adjoint method is proposed to efficiently compute the gradients for neural SDEs . In this paper , we propose a new continuous-time stochastic recurrent network called Variational Stochastic Differential Network ( VSDN ) that incorporates SDEs into recurrent neural model to effectively model the continuous-time stochastic dynamics based only on sparse or irregular observations . Taking advantage of the capacity of deep neural networks , VSDN has higher flexibility and generalizability in modeling the nonlinear stochastic dependency from high-dimensional observations . Compared to Neural ODEs , VSDN incorporates the latent state trajectory to capture the underlying factors of the system dynamics . The trajectory helps to more flexibly model the data distribution and more accurately generate the output data than Neural ODEs . Parallel to the theoretical analysis ( Tzen & Raginsky , 2019a ; b ) and gradient computations ( Li et al. , 2020 ) , our study focuses more on exploring the feasible variational loss and flexible recurrent architecture for the Neural SDEs to model the sporadic data . The contributions of this paper are three-fold : 1 . We incorporate the continuous-time variants of VAE and IWAE losses into VSDN to train the continuous-time stochastic neural networks with latent state trajectories . 2 . We propose the efficient and flexible network architecture of VSDN which can model the complicated stochastic process under high-dimensional sporadic data sequences . 3 . We conduct comprehensive experiments to show that VSDN outperforms state-of-the-art deep learning methods on modeling the continuous-time dynamics and achieves remarkable performance in the prediction and interpolation of irregular or sporadic time series . The rest of this paper is organized as follows . In Section 2 , we first present the continuous-time variants of VAE loss , and then derive a continuous-time IWAE loss to train continuous-time statespace models with deep neural networks . In Section 3 , we propose the deep learning structures of VSDN . Comprehensive experiments are presented in section 4 and conclusion is given in section 5 . 2 CONTINUOUS-TIME VARIATIONAL BAYES . In this section , we first introduce the basic notations and formulate our problem . We then define the continuous-time variants of the Variational Auto-Encoding ( VAE ) and Importance-Weighted AutoEncoding ( IWAE ) lower bounds to enable the efficient training of our models . Due to the page limit , we present all deductions in Appendix A . 2.1 BASIC NOTATIONS AND PROBLEM FORMULATION . Throughout this paper , we define Xt ∈ Rd1 as the continuous-time latent state at time t and Yn ∈ Rd2 as the nth discrete-time observed data at time tn . d1 and d2 are the dimensions of the latent state and observation , respectively . X < t is the continuous trajectory before time t and X≤t is the path up to time t. Yn1 : n2 is the sequence of data points and Xtn1 : tn2 is the continuous-time state trajectory from tn1 to tn2 . Yt = { Yn|tn < t } is the historical observations before t and Yt = { Yn|tn ≥ t } is the current and future observations . For simplicity , we also assume that the initial value of the latent state is constant . The results in this paper can be easily extended to the situation that the initial states are also random variables . Given K data sequences { y ( i ) 1 : ni } , i = 1 , · · · , K , the target of our study is to learn an accurate continuous-time generative model G that maximizes the log-likelihood : G = arg max G 1 K K∑ i=1 logPG ( y ( i ) 1 : ni ) . ( 1 ) For Multivariate sequential data , there exists a complicated nonlinear relationship between the observed data and the unobservable latent state , which can be either the physical state of a dynamic system or the low-dimensional manifold of data . In our study , the latent state evolves in the continuous time domain and generates the observation through some transformation . 2.2 CONTINUOUS-TIME VARIATIONAL INFERENCE . In order to capture the underlying stochastic process from sporadic data , we design the generative model as a neural continuous-time state-space model , which consists of a latent Stochastic Differential Equation ( SDE ) and a conditional distribution of the observation . The latent SDE describes the stochastic process of the latent states and the conditional distribution depicts the probabilistic dependency of the current data with the latent states and historical observations : dXt =HG ( Xt , Yt ; t ) dt+RG ( Yt ; t ) dWt , ( 2 ) PG ( Yn|Y1 : n−1 , Xtn ) = Φ ( Yn|f ( Y1 : n−1 , Xtn ) ) , ( 3 ) where HG and RG are the drift and diffusion functions of the latent SDE . Wt denotes the a Wiener process , which is also called standard Brownian motion . To integrate the information of the observed data , HG is the function of the current state Xt and the historical observations Yt . However , RG only uses the historical data as input . It is not beneficial to include Xt as the input of the diffusion function , as it will inject more noise into gradients of the network parameters . A detailed example and analysis of the noise injection problem is given in Appendix B. Φ ( · ) is a parametric family of distributions over the data and f ( · ) is the function to compute the parameters of Φ . With the advance of deep learning methods , we parameterize HG , RG and f ( · ) by deep neural networks . Continuous-Time Auto-Encoding Variational Bayes : The exact log-likelihood of the generative model is given as logPG ( y1 : n ) = log ∫ PG ( X≤tn ) n∏ i=1 PG ( yi|y1 : n−1 , Xti ) dX≤tn ( 4 ) which does not have the closed-form solution in general . Therefore , G can not be directly trained by maximizing log-likelihood . To overcome this difficulty , an inference modelQ is introduced to depict the stochastic dependency of the latent state on observed data . Similar to the generative model , Q consists of a posterior SDE : dXt =HQ ( Xt , Yt , Yt ; t ) dt+RG ( Yt ; t ) dWt , ( 5 ) whereHQ is the posterior drift function . Different fromHG , HQ also uses the future observation Yt as the input and therefore the inference model Q induces the posterior distribution PQ ( X≤tn |y1 : n ) . Based on Auto-Encoding Variational Bayes ( Kingma & Welling , 2014 ) , it is straightforward to introduce a continuous-time variant of the VAE lower bound of the log-likelihood : LV AE ( y1 : n ) =− βKL ( PQ||PG ) + n∑ i=1 EPQ ( Xti ) logPG ( yi|y1 : n−1 , Xti ) , ( 6 ) KL ( PQ||PG ) = 1 2 ∫ tn 0 EPQ ( Xt ) ( ( HQ −HG ) T [ RGRTG ] −1 ( HQ −HG ) ) dt . ( 7 ) where PG ( X≤tn ) and PQ ( X≤tn ) are the probability density of the latent states induced by the prior SDE Eq . ( 2 ) and the posterior SDE Eq . ( 5 ) . KL ( ·||· ) denotes the KL divergence between two distributions and β is a hyper-parameter to weight the effect of the KL terms . In this paper , we fix β as 1.0 and LV AE is the original VAE objective ( Kingma & Welling , 2014 ) . In β-VAE ( Higgins et al. , 2017 ; Burgess et al. , 2018 ) , it is shown that a larger β can encourage the model to learn more efficient and disentangled representation from the data . Eq . ( 5 ) is restricted to having the same diffusion function as Eq . ( 2 ) . A feasible LV AE can not be defined to train VSDN-SDE without this restriction , as the KL divergence of two SDEs with different diffusions will be infinite ( Archambeau et al. , 2008 ) . The VAE objective has been widely used for discrete-time stochastic recurrent modals , such as LFADS ( Sussillo et al. , 2016 ) , VRNN ( Chung et al. , 2015 ) and SRNN ( Fraccaro et al. , 2016 ) . The major difference between these models and our work is that we incorporate a continuous-time latent state into our model while the latent states of the discrete-time models evolve only at distinct and separate time slots . Continuous-Time Importance Weighted Variational Bayes : LV AE ( y1 : n ) equals the exact loglikelihood when PQ ( X≤tn ) of the inference model is identical to the exact posterior distribution induced by the generative model . The errors of the inference model can result in the looseness of the VAE loss for the model training . Under the framework of Importance-Weighted Auto-Encoder ( IWAE ) ( Burda et al. , 2016 ; Cremer et al. , 2017 ) , we can define a tighter evidence lower bound : L̃KIWAE ( y1 : n ) =Ex1≤tn , ··· , xK≤tn∼PQ ( x≤tn ) ( log 1 K K∑ k=1 wk n∏ i=1 PG ( yi|y1 : n−1 , Xti ) ) , ( 8 ) where the importance weights satisfy the following SDE : d logwk =d log PG ( x k ≤tn ) PQ ( xk≤tn ) = −1 2 ( HQ −HG ) T [ RGRTG ] −1 ( HQ −HG ) dt − ( HQ −HG ) T [ RG ] −1dWt . ( 9 ) Given the variational auto-encoding lower bound LV AE ( · ) and the importance weighted autoencoding lower bound LKIWAE ( · ) for the continuous-time generative model , the tightness of the lower bounds are given by the following inequality : logPG ( y1 : n ) ≥ L̃K+1IWAE ( · ) ≥ L̃ K IWAE ( · ) ≥ LV AE ( · ) , ( 10 ) for any positive integer K. Consequently , L̃KIWAE ( · ) is infinite if the diffusions of Eq . ( 2 ) and Eq . ( 5 ) are different . In our implementation , we notice that the training of our models by L̃KIWAE is not stable , possibly due to the drawbacks of importance sampling and the Signal-To-Noise problem ( Rainforth et al. , 2018 ) . To alleviate the problem , we train our model by a convex combination of the VAE and IWAE losses : LKIWAE ( y1 : n ) = ( 1− α ) LV AE ( y1 : n ) + αL̃KIWAE ( y1 : n ) , α ∈ ( 0 , 1 ) . ( 11 ) With the use of reparameterization ( Kingma & Welling , 2014 ) , both LV AE ( y1 : n ) and LKIWAE ( y1 : n ) are differentiable with respect to the parameters of the generative and inference models . Therefore , they can be applied to train continuous-time stochastic models with deep learning components .

Summary: This paper introduces a latent variable model for high dimensional stochastic time-series. The model is akin to a VAE with RNNs that incorporate time-series data. The authors introduce two variants of the model, one which only contains a feedforward RNN (filtering) and another that contains feedforward and feedback RNNs (smoothing).  The authors use two inference procedures for the model, one the standard VAE, and the other importance weighted IWAE.  The work is reasonably clearly presented and the experiments are multiple data sets are a nice addition.