Continuous Latent Process Flows

1 Introduction . Sparse and irregular observations of continuous dynamics are common in many areas of science , including finance [ 15 , 36 ] , healthcare [ 16 ] , and physics [ 30 ] . Time-series models driven by stochastic differential equations ( SDEs ) provide an elegant framework for this challenging scenario and have recently gained popularity in the machine learning community [ 11 , 18 , 24 ] . The SDEs are typically implemented by neural networks with trainable parameters and the latent processes defined by the SDEs are then decoded into an observable space with complex structure . Due to the lack of closedform transition densities for most SDEs , dedicated variational approximations have been developed to maximize the observational log-likelihoods [ 2 , 18 , 24 ] . Despite the progress of existing works , challenges still remain for SDE-based models to be applied to irregular time-series data . One major challenge is the model ’ s representation power . The continuoustime flow process ( CTFP ; [ 11 ] ) uses a series of invertible mappings continuously indexed by time to transform a simple Wiener process to a more complex stochastic process . The use of a simple latent process and invertible transformations permits CTFP models to evaluate the exact likelihood of observations on any time grid efficiently , but they also limit the set of stochastic processes that CTFP can express to some specific form which can be obtained using Ito ’ s Lemma and excludes many commonly seen stochastic processes . Another constraint of representation power in practice is the Lipschitz property of transformations in the models . The latent SDE model proposed by Hasan et al . [ 18 ] and CTFP both transform a latent stochastic process with constant variance to an observable one using injective mappings . Due to the Lipschitz property existing in many invertible neural network architectures , some processes that can be written as a non-Lipschitz transformation of a simple process , like geometric Brownian motion , can not be expressed by these models unless specific choices of non-Lipschitz decoders are used . Apart from the model ’ s representation power , variational inference is another challenge in training SDE-based models . The latent SDE model in the work of Li et al . [ 24 ] ∗This work was done during an internship at Borealis AI . Correspondance to wsdmdeng @ gmail.com . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . uses a principled method of variational approximation based on importance weighting of trajectories between a variational posterior and a prior process . The variational posterior process is constructed using a single SDE conditioned on all observations and is therefore restricted to be a Markov process . This approach may lack the flexibility to approximate the true posterior process well enough in complex inference tasks , e.g. , in an online setting or with variable-length observation sequences . In this work we propose Continuous Latent Process Flows ( CLPFs ) 2 , a model that is governed by latent dynamics defined by an expressive generic stochastic differential equation . Inspired by [ 11 ] , we then use dynamic normalizing flows to decode each latent trajectory into a continuous observable process . Driven by different trajectories of the latent stochastic process continuously evolving with time , the dynamic normalizing flow can map a simple base process to a diverse class of observable processes . We illustrate this process in Fig . 1 . This decoding is critical for the model to generate continuous trajectories and be trained to fit observations on irregular time grids using a variational approximation . Good variational approximations and proper handling of complex inference tasks like online inference depend on a flexible variational posterior process . Therefore , we also propose a principled method of defining and sampling from a non-Markov variational posterior process that is based on a piecewise evaluation of SDEs and can adapt to new observations . The proposed model excels at fitting observations on irregular time grids , generalizing to observations on more dense time grids , and generating trajectories continuous in time . Contributions . In summary , we make the following contributions : ( 1 ) We propose a flow-based decoding of a generic SDE as a principled framework for continuous dynamics modeling of irregular time-series data . ( 2 ) We improve the variational approximation of the observational likelihood through a flexible non-Markov posterior process based on a piecewise evaluation of the underlying SDE ; ( 3 ) We validate the effectiveness of our contributions in a series of ablation studies and comparisons to state-of-the-art time-series models , both on synthetic and real-world datasets . 2 Preliminaries . 2.1 Stochastic Differential Equations . SDEs can be viewed as a stochastic analogue of ordinary differential equations ( ODEs ) in the sense that dZtdt = µ ( Zt , t ) + random noise · σ ( Zt , t ) . Let Zt be a variable which continuously evolves with time . An m-dimensional SDE describing the stochastic dynamics of Zt usually takes the form dZt = µ ( Zt , t ) dt+ σ ( Zt , t ) dWt , ( 1 ) where µ maps to an m-dimensional vector , σ is an m × k matrix , and Wt is a k-dimensional Wiener process . The solution of an SDE is a continuous-time stochastic process Zt that satisfies the integral equation Zt = Z0 + ∫ t 0 µ ( Zs , s ) ds+ ∫ t 0 σ ( Zs , s ) dWs with initial condition Z0 , where 2Code available at https : //github.com/BorealisAI/continuous-latent-process-flows the stochastic integral should be interpreted as a traditional Itô integral [ 27 , Chapter 3.1 ] . For each sample trajectory ω ∼ Wt , the stochastic process Zt maps ω to a different trajectory Zt ( ω ) . Latent Dynamics and Variational Bound . SDEs have been used as models of latent dynamics in a variety of contexts [ 2 , 18 , 24 ] . As closed-form finite-dimensional solutions to SDEs are rare , variational approximations are often used in practice . Li et al . [ 24 ] propose a principled way of re-weighting latent SDE trajectories for variational approximations using Girsanov ’ s theorem [ 27 , Chapter 8.6 ] . Specifically , consider a prior process and a variational posterior process in the interval [ 0 , T ] defined by two stochastic differential equations dZt = µ1 ( Zt , t ) dt + σ ( Zt , t ) dWt and dẐt = µ2 ( Ẑt , t ) dt+ σ ( Ẑt , t ) dWt , respectively . Furthermore , let p ( x|Zt ) denote the probability of observing x conditioned on the trajectory of the latent process Zt in the interval [ 0 , T ] . If there exists a mapping u : Rm × [ 0 , T ] → Rk such that σ ( z , t ) u ( z , t ) = µ2 ( z , t ) − µ1 ( z , t ) and u satisfies Novikov ’ s condition [ 27 , Chapter 8.6 ] , we obtain the variational lower bound log p ( x ) = logE [ p ( x|Zt ) ] = logE [ p ( x|Ẑt ) MT ] ≥ E [ log p ( x|Ẑt ) + logMT ] , ( 2 ) where MT = exp ( − ∫ T 0 1 2 ∣∣∣u ( Ẑt , t ) ∣∣∣2 dt− ∫ T0 u ( Ẑt , t ) T dWt ) . See [ 24 ] for a formal proof . 2.2 Normalizing Flows . Normalizing flows [ 3 , 8 , 12 , 13 , 21 , 22 , 23 , 28 , 29 , 31 ] employ a bijective mapping f : Rd → Rd to transform a random variable Y with a simple base distribution pY to a random variable X with a complex target distribution pX . We can sample from a normalizing flow by first sampling y ∼ pY and then transforming it to x = f ( y ) . As a result of invertibility , normalizing flows can also be used for density estimation . Using the change-of-variables formula , we have log pX ( x ) = log pY ( g ( x ) ) + log ∣∣∣det ( ∂g∂x ) ∣∣∣ , where g is the inverse of f . Continuous Indexing . More recently , normalizing flows have been augmented with a continuous index [ 6 , 10 , 11 ] . For instance , the continuous-time flow process ( CTFP ; [ 11 ] ) models irregular observations of a continuous-time stochastic process . Specifically , CTFP transforms a simple ddimensional Wiener process Wt to another continuous stochastic process Xt using the transformation Xt = f ( Wt , t ) , where f ( w , t ) is an invertible mapping for each t. Despite its benefits of exact log-likelihood computation of arbitrary finite-dimensional distributions , the expressive power of CTFP to model stochastic processes is limited in the following two aspects : ( 1 ) An application of Itô ’ s lemma [ 27 , Chapter 4.2 ] shows that CTFP can only represent stochastic processes of the form df ( Wt , t ) = { df dt ( Wt , t ) + 1 2 Tr ( Hwf ( Wt , t ) ) } dt+ ( ∇wfT ( Wt , t ) ) T dWt , ( 3 ) where Hwf is the Hessian matrix of f with respect to w and ∇wf is the derivative . A variety of stochastic processes , from simple processes like the Ornstein-Uhlenbeck ( OU ) process to more complex non-Markov processes , fall outside of this limited class and can not be learned using CTFP ( see Appendix A for formal proofs ) ; ( 2 ) Many normalizing flow architectures are compositions of Lipschitz-continuous transformations [ 7 , 8 , 17 ] . It is therefore challenging to model certain stochastic processes that are non-Lipschitz transformations of simple processes using CTFP without prior knowledge about the functional form of the observable processes and custom-tailored normalizing flows with non-Lipschitz transformations ( see Appendix B for an example ) . A latent variant of CTFP is further augmented with a static latent variable to introduce non-Markovian behavior . It models continuous stochastic processes as Xt = f ( Wt , t ; Z ) , where Z is a latent variable with standard Gaussian distribution and f ( · , · ; z ) is a CTFP model that decodes each sample z of Z into a stochastic process with continuous trajectories . Latent CTFP can be used to estimate finite-dimensional distributions using a variational approximation . However , it is unclear how much a latent variable Z with finite dimensions can improve CTFPs representation power . 3 Model . Equipped with these tools , we can now describe our problem setting and model architecture . Let { ( xti , ti ) } ni=1 denote a sequence of d-dimensional observations sampled at arbitrary points in time , where xti and ti denote the value and time stamp of the observation , respectively . The observations are assumed to be partial realizations of a continuous-time stochastic process Xt . Our training objective is the maximization of the observational log-likelihood induced by Xt on a given time grid , L = log pXt1 , ... , Xtn ( xt1 , . . . , xtn ) , ( 4 ) for an inhomogeneous collection of sequences with varying lengths and time stamps . At test-time , in addition to the maximization of log-likelihoods , we are also interested in sampling sparse , dense , or irregular trajectories with finite-dimensional distributions that conform with these log-likelihoods . We model this challenging scenario with Continuous Latent Process Flows ( CLPF ) . In Section 3.1 , we present our model in more detail . Training and inference methods will be discussed in Section 3.2 .

Summary: The paper addresses probabilistic modeling of sparse or irregularly sampled time series. The proposed method captures the systems dynamics by a rather simple latent stochastic process, which is then warped into a more complex one using a time-indexed ANODE-like normalizing flow. The free parameters of the dynamical system are identified by maximizing the ELBO that lower bounds the likelihood of the observed sequence. The solution aims to overcome the limited expressive power of normalizing flows stemming for the necessity to use invertible transformations that also need to satisfy Lipschitz continuity restrictions.