Continuous Latent Process Flows

1 Introduction . Sparse and irregular observations of continuous dynamics are common in many areas of science , including finance [ 15 , 36 ] , healthcare [ 16 ] , and physics [ 30 ] . Time-series models driven by stochastic differential equations ( SDEs ) provide an elegant framework for this challenging scenario and have recently gained popularity in the machine learning community [ 11 , 18 , 24 ] . The SDEs are typically implemented by neural networks with trainable parameters and the latent processes defined by the SDEs are then decoded into an observable space with complex structure . Due to the lack of closedform transition densities for most SDEs , dedicated variational approximations have been developed to maximize the observational log-likelihoods [ 2 , 18 , 24 ] . Despite the progress of existing works , challenges still remain for SDE-based models to be applied to irregular time-series data . One major challenge is the model ’ s representation power . The continuoustime flow process ( CTFP ; [ 11 ] ) uses a series of invertible mappings continuously indexed by time to transform a simple Wiener process to a more complex stochastic process . The use of a simple latent process and invertible transformations permits CTFP models to evaluate the exact likelihood of observations on any time grid efficiently , but they also limit the set of stochastic processes that CTFP can express to some specific form which can be obtained using Ito ’ s Lemma and excludes many commonly seen stochastic processes . Another constraint of representation power in practice is the Lipschitz property of transformations in the models . The latent SDE model proposed by Hasan et al . [ 18 ] and CTFP both transform a latent stochastic process with constant variance to an observable one using injective mappings . Due to the Lipschitz property existing in many invertible neural network architectures , some processes that can be written as a non-Lipschitz transformation of a simple process , like geometric Brownian motion , can not be expressed by these models unless specific choices of non-Lipschitz decoders are used . Apart from the model ’ s representation power , variational inference is another challenge in training SDE-based models . The latent SDE model in the work of Li et al . [ 24 ] ∗This work was done during an internship at Borealis AI . Correspondance to wsdmdeng @ gmail.com . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . uses a principled method of variational approximation based on importance weighting of trajectories between a variational posterior and a prior process . The variational posterior process is constructed using a single SDE conditioned on all observations and is therefore restricted to be a Markov process . This approach may lack the flexibility to approximate the true posterior process well enough in complex inference tasks , e.g. , in an online setting or with variable-length observation sequences . In this work we propose Continuous Latent Process Flows ( CLPFs ) 2 , a model that is governed by latent dynamics defined by an expressive generic stochastic differential equation . Inspired by [ 11 ] , we then use dynamic normalizing flows to decode each latent trajectory into a continuous observable process . Driven by different trajectories of the latent stochastic process continuously evolving with time , the dynamic normalizing flow can map a simple base process to a diverse class of observable processes . We illustrate this process in Fig . 1 . This decoding is critical for the model to generate continuous trajectories and be trained to fit observations on irregular time grids using a variational approximation . Good variational approximations and proper handling of complex inference tasks like online inference depend on a flexible variational posterior process . Therefore , we also propose a principled method of defining and sampling from a non-Markov variational posterior process that is based on a piecewise evaluation of SDEs and can adapt to new observations . The proposed model excels at fitting observations on irregular time grids , generalizing to observations on more dense time grids , and generating trajectories continuous in time . Contributions . In summary , we make the following contributions : ( 1 ) We propose a flow-based decoding of a generic SDE as a principled framework for continuous dynamics modeling of irregular time-series data . ( 2 ) We improve the variational approximation of the observational likelihood through a flexible non-Markov posterior process based on a piecewise evaluation of the underlying SDE ; ( 3 ) We validate the effectiveness of our contributions in a series of ablation studies and comparisons to state-of-the-art time-series models , both on synthetic and real-world datasets . 2 Preliminaries . 2.1 Stochastic Differential Equations . SDEs can be viewed as a stochastic analogue of ordinary differential equations ( ODEs ) in the sense that dZtdt = µ ( Zt , t ) + random noise · σ ( Zt , t ) . Let Zt be a variable which continuously evolves with time . An m-dimensional SDE describing the stochastic dynamics of Zt usually takes the form dZt = µ ( Zt , t ) dt+ σ ( Zt , t ) dWt , ( 1 ) where µ maps to an m-dimensional vector , σ is an m × k matrix , and Wt is a k-dimensional Wiener process . The solution of an SDE is a continuous-time stochastic process Zt that satisfies the integral equation Zt = Z0 + ∫ t 0 µ ( Zs , s ) ds+ ∫ t 0 σ ( Zs , s ) dWs with initial condition Z0 , where 2Code available at https : //github.com/BorealisAI/continuous-latent-process-flows the stochastic integral should be interpreted as a traditional Itô integral [ 27 , Chapter 3.1 ] . For each sample trajectory ω ∼ Wt , the stochastic process Zt maps ω to a different trajectory Zt ( ω ) . Latent Dynamics and Variational Bound . SDEs have been used as models of latent dynamics in a variety of contexts [ 2 , 18 , 24 ] . As closed-form finite-dimensional solutions to SDEs are rare , variational approximations are often used in practice . Li et al . [ 24 ] propose a principled way of re-weighting latent SDE trajectories for variational approximations using Girsanov ’ s theorem [ 27 , Chapter 8.6 ] . Specifically , consider a prior process and a variational posterior process in the interval [ 0 , T ] defined by two stochastic differential equations dZt = µ1 ( Zt , t ) dt + σ ( Zt , t ) dWt and dẐt = µ2 ( Ẑt , t ) dt+ σ ( Ẑt , t ) dWt , respectively . Furthermore , let p ( x|Zt ) denote the probability of observing x conditioned on the trajectory of the latent process Zt in the interval [ 0 , T ] . If there exists a mapping u : Rm × [ 0 , T ] → Rk such that σ ( z , t ) u ( z , t ) = µ2 ( z , t ) − µ1 ( z , t ) and u satisfies Novikov ’ s condition [ 27 , Chapter 8.6 ] , we obtain the variational lower bound log p ( x ) = logE [ p ( x|Zt ) ] = logE [ p ( x|Ẑt ) MT ] ≥ E [ log p ( x|Ẑt ) + logMT ] , ( 2 ) where MT = exp ( − ∫ T 0 1 2 ∣∣∣u ( Ẑt , t ) ∣∣∣2 dt− ∫ T0 u ( Ẑt , t ) T dWt ) . See [ 24 ] for a formal proof . 2.2 Normalizing Flows . Normalizing flows [ 3 , 8 , 12 , 13 , 21 , 22 , 23 , 28 , 29 , 31 ] employ a bijective mapping f : Rd → Rd to transform a random variable Y with a simple base distribution pY to a random variable X with a complex target distribution pX . We can sample from a normalizing flow by first sampling y ∼ pY and then transforming it to x = f ( y ) . As a result of invertibility , normalizing flows can also be used for density estimation . Using the change-of-variables formula , we have log pX ( x ) = log pY ( g ( x ) ) + log ∣∣∣det ( ∂g∂x ) ∣∣∣ , where g is the inverse of f . Continuous Indexing . More recently , normalizing flows have been augmented with a continuous index [ 6 , 10 , 11 ] . For instance , the continuous-time flow process ( CTFP ; [ 11 ] ) models irregular observations of a continuous-time stochastic process . Specifically , CTFP transforms a simple ddimensional Wiener process Wt to another continuous stochastic process Xt using the transformation Xt = f ( Wt , t ) , where f ( w , t ) is an invertible mapping for each t. Despite its benefits of exact log-likelihood computation of arbitrary finite-dimensional distributions , the expressive power of CTFP to model stochastic processes is limited in the following two aspects : ( 1 ) An application of Itô ’ s lemma [ 27 , Chapter 4.2 ] shows that CTFP can only represent stochastic processes of the form df ( Wt , t ) = { df dt ( Wt , t ) + 1 2 Tr ( Hwf ( Wt , t ) ) } dt+ ( ∇wfT ( Wt , t ) ) T dWt , ( 3 ) where Hwf is the Hessian matrix of f with respect to w and ∇wf is the derivative . A variety of stochastic processes , from simple processes like the Ornstein-Uhlenbeck ( OU ) process to more complex non-Markov processes , fall outside of this limited class and can not be learned using CTFP ( see Appendix A for formal proofs ) ; ( 2 ) Many normalizing flow architectures are compositions of Lipschitz-continuous transformations [ 7 , 8 , 17 ] . It is therefore challenging to model certain stochastic processes that are non-Lipschitz transformations of simple processes using CTFP without prior knowledge about the functional form of the observable processes and custom-tailored normalizing flows with non-Lipschitz transformations ( see Appendix B for an example ) . A latent variant of CTFP is further augmented with a static latent variable to introduce non-Markovian behavior . It models continuous stochastic processes as Xt = f ( Wt , t ; Z ) , where Z is a latent variable with standard Gaussian distribution and f ( · , · ; z ) is a CTFP model that decodes each sample z of Z into a stochastic process with continuous trajectories . Latent CTFP can be used to estimate finite-dimensional distributions using a variational approximation . However , it is unclear how much a latent variable Z with finite dimensions can improve CTFPs representation power . 3 Model . Equipped with these tools , we can now describe our problem setting and model architecture . Let { ( xti , ti ) } ni=1 denote a sequence of d-dimensional observations sampled at arbitrary points in time , where xti and ti denote the value and time stamp of the observation , respectively . The observations are assumed to be partial realizations of a continuous-time stochastic process Xt . Our training objective is the maximization of the observational log-likelihood induced by Xt on a given time grid , L = log pXt1 , ... , Xtn ( xt1 , . . . , xtn ) , ( 4 ) for an inhomogeneous collection of sequences with varying lengths and time stamps . At test-time , in addition to the maximization of log-likelihoods , we are also interested in sampling sparse , dense , or irregular trajectories with finite-dimensional distributions that conform with these log-likelihoods . We model this challenging scenario with Continuous Latent Process Flows ( CLPF ) . In Section 3.1 , we present our model in more detail . Training and inference methods will be discussed in Section 3.2 .

Summary: This paper introduces CLPF---novel parametrization of a latent SDE---that has higher expressivity than previous approaches. CLPF is obtained as a time-dependent invertible transformation of a base process. This transformation is implemented as a normalizing flow whose parameters are a function of another SDF. In addition, the authors introduce a novel non-Markov piecewise variational posterior process that elegantly circumvents the fact that SDE solutions are Markov. The method is evaluated on four well-designed syntethic datasets, whose purpose is to test different properties of CLPF. The efficacy to model more complicated data is tested on a real-world dataset, while an ablation study evaluates the relative importance of different model components.