Neural ODE Processes

1 INTRODUCTION . Many time-series that arise in the natural world , such as the state of a harmonic oscillator , the populations in an ecological network or the spread of a disease , are the product of some underlying dynamics . Sometimes , as in the case of a video of a swinging pendulum , these dynamics are latent and do not manifest directly in the observation space . Neural Ordinary Differential Equations ( NODEs ) ( Chen et al. , 2018 ) , which use a neural network to parametrise the derivative of an ODE , have become a natural choice for capturing the dynamics of such time-series ( Çağatay Yıldız et al. , 2019 ; Rubanova et al. , 2019 ; Norcliffe et al. , 2020 ; Kidger et al. , 2020 ; Morrill et al. , 2020 ) . However , despite their fundamental connection to dynamics-governed time-series , NODEs present certain limitations that hinder their adoption in these settings . Firstly , NODEs can not adjust predictions as more data is collected without retraining the model . This ability is particularly important for real-time applications , where it is desirable that models adapt to incoming data points as time passes and more data is collected . Secondly , without a larger number of regularly spaced measurements , there is usually a range of plausible underlying dynamics that can explain the data . However , NODEs do not capture this uncertainty in the dynamics . As many real-world time-series are comprised of sparse sets of measurements , often irregularly sampled , the model can fail to represent the diversity of suitable solutions . In contrast , the Neural Process ( Garnelo et al. , 2018a ; b ) family offers a class of ( neural ) stochastic processes designed for uncertainty estimation and fast adaptation to changes in the observed data . However , NPs modelling time-indexed random functions lack an explicit treatment of time . Designed for the general case of an arbitrary input domain , they treat time as an unordered set and do not explicitly consider the time-delay between different observations . To address these limitations , we introduce Neural ODE Processes ( NDPs ) , a new class of stochastic processes governed by stochastic data-adaptive dynamics . Our probabilistic Neural ODE formulation relies on and extends the framework provided by NPs , and runs parallel to other attempts to ∗Equal contribution . †Work done as an AI Resident at the University of Cambridge . incorporate application-specific inductive biases in this class of models such as Attentive NPs ( Kim et al. , 2019 ) , ConvCNPs ( Gordon et al. , 2019 ) , and MPNPs ( Day et al. , 2020 ) . We demonstrate that NDPs can adaptively capture many potential dynamics of low-dimensional systems when faced with limited amounts of data . Additionally , we show that our approach scales to high-dimensional time series with latent dynamics such as rotating MNIST digits ( Casale et al. , 2018 ) . Our code and datasets are available at https : //github.com/crisbodnar/ndp . 2 BACKGROUND AND FORMAL PROBLEM STATEMENT . Problem Statement We consider modelling random functions F : T → Y , where T = [ t0 , ∞ ) represents time and Y ⊂ Rd is a compact subset of Rd . We assume F has a distribution D , induced by another distribution D′ over some underlying dynamics that govern the time-series . Given a specific instantation F of F , let C = { ( tCi , yCi ) } i∈IC be a set of samples from F with some indexing set IC . We refer to C as the context points , as denoted by the superscript C. For a given context C , the task is to predict the values { yTj } j∈IT that F takes at a set of target times { tTj } j∈IT , where IT is another index set . We call T = { ( tTj , yTj ) } the target set . Additionally let tC = { ti|i ∈ IC } and similarly define yC , tT and yT . Conventionally , as in Garnelo et al . ( 2018b ) , the target set forms a superset of the context set and we have C ⊆ T. Optionally , it might also be natural to consider that the initial time and observation ( t0 , y0 ) are always included in C. During training , we let the model learn from a dataset of ( potentially irregular ) time-series sampled from F . We are interested in learning the underlying distribution over the dynamics as well as the induced distribution over functions . We note that when the dynamics are not latent and manifest directly in the observation space Y , the distribution over ODE trajectories and the distribution over functions coincide . Neural ODEs NODEs are a class of models that parametrize the velocity ż of a state z with the help of a neural network ż = fθ ( z , t ) . Given the initial time t0 and target time tTi , NODEs predict the corresponding state ŷTi by performing the following integration and decoding operations : z ( t0 ) = h1 ( y0 ) , z ( t T i ) = z ( t0 ) + ∫ tTi t0 fθ ( z ( t ) , t ) dt , ŷ T i = h2 ( z ( t T i ) ) , ( 1 ) where h1 and h2 can be neural networks . When the dimensionality of z is greater than that of y and h1 , h2 are linear , the resulting model is an Augmented Neural ODE ( Dupont et al. , 2019 ) with input layer augmentation ( Massaroli et al. , 2020 ) . The extra dimensions offer the model additional flexibility as well as the ability to learn higher-order dynamics ( Norcliffe et al. , 2020 ) . Neural Processes ( NPs ) NPs model a random function F : X → Y , where X ⊆ Rd1 and Y ⊆ Rd2 . The NP represents a given instantiation F of F through the global latent variable z , which parametrises the variation in F . Thus , we have F ( xi ) = g ( xi , z ) . For a given context set C = { ( xCi , yCi ) } and target set x1 : n , y1 : n , the generative process is given by : p ( y1 : n , z|x1 : n , C ) = p ( z|C ) n∏ i=1 N ( yi|g ( xi , z ) , σ2 ) , ( 2 ) where p ( z ) is chosen to be a multivariate standard normal distribution and y1 : n is a shorthand for the sequence ( y1 , . . . , yn ) . The model can be trained using an amortised variational inference procedure that naturally gives rise to a permutation-invariant encoder qθ ( z|C ) , which stores the information about the context points . Conditioned on this information , the decoder g ( x , z ) can make predictions at any input location x . We note that while the domain X of the random function F is arbitrary , in this work we are interested only in stochastic functions with domain on the real line ( time-series ) . Therefore , from here our notation will reflect that , using t as the input instead of x . The output y remains the same . 3 NEURAL ODE PROCESSES Model Overview We introduce Neural ODE Processes ( NDPs ) , a class of dynamics-based models that learn to approximate random functions defined over time . To that end , we consider an NP whose context is used to determine a distribution over ODEs . Concretely , the context infers a distribution over the initial position ( and optionally – the initial velocity ) and , at the same time , stochastically controls its derivative function . The positions given by the ODE trajectories at any time tTi are then decoded to give the predictions . In what follows , we offer a detailed description of each component of the model . A schematic of the model can be seen in Figure 1 . 3.1 GENERATIVE PROCESS . We first describe the generative process behind NDPs . A graphical model perspective of this process is also included in Figure 2 . Encoder and Aggregator Consider a given context set C = { ( tCi , yCi ) } i∈IC of observed points . We encode this context into two latent variables L ( t0 ) ∼ qL ( l ( t0 ) |C ) and D ∼ qD ( d|C ) , representing the initial state and the global control of an ODE , respectively . To parametrise the distribution of the latter variable , the NDP encoder produces a representation ri = fe ( ( tCi , y C i ) ) for each context pair ( tCi , y C i ) . The function fe is as a neural network , fully connected or convolutional , depending on the nature of y . An aggregator combines all the representations ri to form a global representation , r , that parametrises the distribution of the global latent context , D ∼ qD ( d|C ) = N ( z|µD ( r ) , diag ( σD ( r ) ) ) . As the aggregator must preserve order invariance , we choose to take the element-wise mean . The distribution of L0 might be parametrised identically as a function of the whole context by qL ( d|C ) , and , in particular , if the initial observation y0 is always known , then qL ( l ( 0 ) |C ) = qL ( l ( 0 ) |y0 ) = N ( l ( 0 ) |µL ( y0 ) , diag ( σL ( y0 ) ) ) . Latent ODE To obtain a distribution over functions , we are interested in capturing the dynamics that govern the time-series and exploiting the temporal nature of the data . To that end , we allow the latent context to evolve according to a Neural ODE ( Chen et al. , 2018 ) with initial position L ( 0 ) and controlled by D. These two random variables factorise the uncertainty in the underlying dynamics into an uncertainty over the initial conditions ( given by L ( t0 ) ) and an uncertainty over the ODE derivative , given by D. By using the target times , tT1 : n = ( t T 1 , ... , t T N ) , the latent state at a given time is found by evolving a Neural ODE : l ( tTi ) = l ( t0 ) + ∫ tTi t0 fθ ( l ( t ) , d , t ) dt , ( 3 ) where fθ is a neural network that models the derivative of l. As explained above , we allow d to modulate the derivative of this ODE by acting as a global control signal . Ultimately , for fixed initial conditions , this results in an uncertainty over the ODE trajectories . Decoder To obtain a prediction at a time tTi , we decode the random state of the ODE at time tTi , given by L ( tTi ) . Assuming that the outputs are noisy , for a given sample l ( t T i ) from this stochastic state , the decoder g produces a distribution over Y Tti ∼ p ( yTi |g ( l ( tTi ) , ti ) ) parametrised by the decoder output . Concretely , for regression tasks , we take the target output to be normally distributed with constant ( or optionally learned ) variance Y Tti ∼ N ( yi|g ( l ( ti ) , ti ) , σ2 ) . When Y Tti is a random vector formed of independent binary random variables ( e.g . a black and white image ) , we use a Bernoulli distribution Y Tti ∼ ∏dim ( Y ) j=1 Bernoulli ( g ( l ( ti ) , ti ) j ) . Putting everything together , for a set of observed context points C , the generative process of NDPs is given by the expression below , where we emphasise once again that l ( ti ) also implicitly depends on l ( 0 ) and d. p ( y1 : n , l ( 0 ) , d|t1 : n , C ) = p ( l ( 0 ) |C ) p ( d|C ) n∏ i=1 p ( yi|g ( l ( ti ) , ti ) ) , ( 4 ) We remark that NDPs generalise NPs defined over time . If the latent NODE learns the trivial velocity fθ ( l ( t ) , d , t ) = 0 , the random state L ( t ) = L ( t0 ) remains constant at all times t. In this case , the distribution over functions is directly determined by L ( t0 ) ∼ p ( l ( t0 ) |C ) , which substitutes the random variable Z from a regular NP . For greater flexibility , the control signal d can also be supplied to the decoder g ( l ( t ) , d , t ) . This shows that , in principle , NDPs are at least as expressive as NPs . Therefore , NDPs could be a sensible choice even in applications where the time-series are not solely determined by some underlying dynamics , but are also influenced by other generative factors .

Summary: The proposed NDP has two main advantages: 1- it has the capability to adapt the incoming data points in time-series (unlike NODE) without retraining, 2- it can provide a measure of uncertainty for the underlying dynamics of the time-series. NDP partitions the global latent context $z$ to a latent position $l$ and sub-context $z^\prime$. Then it lets $l$ follow an ODE, called latent ODE. This part is actually the innovation of the paper where by defining a latent ODE, the authors take advantages of ODEs to find the underlying hidden dynamics of the time-series. This assumption helps find better dynamics when the generating processes of time-series meet some ODEs. Then the authors define a stochastic process very like the idea from Neural Processes (NP) paper, that is, by defining a latent context $z$ (which here is a concatenation of $l$ and sub-context $z^\prime$) with a prior p(z) and integrating a Gaussian distribution of a function of $z$ (decoder $g(l,t,z^\prime)$ which is a neural network) over $z$.