Modeling Temporal Data as Continuous Functions with Process Diffusion
Authors: Marin Biloš
Kashif Rasul
Anderson Schneider
Yuriy Nevmyvaka
Stephan Günnemann
Venue: Submitted to ICLR 2023 (2023)
Keywords: time series
stochastic process
diffusion
probabilistic forecasting
score-based matching

Abstract: Temporal data like time series are often observed at irregular intervals which is a challenging setting for  the existing machine learning methods. To tackle this problem, we view such data as samples from some underlying continuous function. We then define a diffusion-based generative model that adds  noise from a predefined stochastic process while  preserving the  continuity of the resulting underlying function. A neural network is trained to reverse this process which allows us to sample new realizations from the learned distribution. We define suitable stochastic processes as noise sources and introduce novel denoising and score-matching models on processes. Further, we show how to apply this approach to the  multivariate probabilistic forecasting and imputation tasks. Through our extensive experiments, we demonstrate that our method outperforms previous models on synthetic and real-world datasets.

Under review as a conference paper at ICLR 2023
MODELING TEMPORAL DATA AS CONTINUOUS FUNCTIONS WITH PROCESS DIFFUSION
Anonymous authors Paper under double-blind review
ABSTRACT
Temporal data like time series are often observed at irregular intervals which is a challenging setting for existing machine learning methods. To tackle this prob- lem, view such data samples from We then define a diffusion-basee generative model that adds noise from a prede- fined stochastic process while preserving the continuity of the resulting underlying function. A neural network is trained to reverse this process which allows us to sample new realizations from the learned distribution. We define suitable stochas- tic processes as noise sources and introduce novel denoising and score-matching models on processes. Further, we show how to apply this approach the mul- tivariate probabilistic forecasting and imputation tasks. Through our extensive experiments, we demonstrate that our method outperforms previous models on synthetic and real-world datasets.
underlying continuous function.
INTRODUCTION
Time series data is collected from measurements of some real-world system that evolves via some complex unknown dynamics and the sampling rate is often arbitrary and non-constant. Thus, the assumption that time series follows some underlying continuous function is reasonable; consider, e.g., the temperature or load of a system over time. Although the values are observed as separate events, we know the temperature always exists and evolution over time smooth, not jittery. The continuity assumption remains when the intervals between the measurements vary. This kind of data can be found in many domains, from medical, industrial financial applications.
Different approaches to model irregular data have been proposed, including neural (ordinary or stochastic) differential equations (Chen et al., 2018; et al., 2020), neural processes (Garnelo al., 2018), normalizing flows (Deng et al., 2020) etc. As it turns out, capturing the true generative process proves to be difficult, especially with the inherent stochasticity the data.
We propose an alternative method, a generative model for continuous data that is based the diffusion framework (Ho et al., 2020) which simply adds noise to a data point until contains no information about the original input. At the same time, the generative part of these models learns to reverse this process so that we can sample new realizations once training is completed. In this paper, apply these ideas to the time series setting and address the unique challenges that arise.
Contributions. Contrary to the previous works on diffusion, we model continuous functions, not vectors (Fig. 1). To do so, first define a suitable noising process that will preserve continuity. Next, we derive the transition probabilities to perform the noising and specify the evidence bound on the likelihood well the new sampling procedure. Finally, we propose new models that take in the noisy input and produce the denoised output or, alternatively, the value of the score function.
Xy Pdsta
(0)
X \/ E)
History
Forerast
Conditional model
Xo(Xn, k)))
Noise
Figure I: (Left) We add noise from a stochastic process to the whole time series at once. The model Cy learns to reverse this process. (Right) We can use this approach to, e.g., forecast with uncertainty.

---

Under review as a conference paper at ICLR 2023
2 BACKGROUND
Given training data { where each a E RR, the goal of generative modeling is to learn the prob- ability density function p(z)) and be able to generate new samples from this learned distribution. Diffusion models (Ho et al., 2020; Song et al., 2021) achieve both of these goals by learning to reverse some fixed process that adds noise the data. In the following, we present brief overview of the two ways to define diffusion; in Section 2.1 the noise is added across N increasing scales, which is then taken to the limit in Section 2.2 by defining the diffusion using stochastic differential equation (SDE).
2.1 FIXED- STEP DIFFUSION
Sohl-Dickstein et al. (2015); Ho et al. (2020) propose the denoising diffusion probabilistic model (DDPM) which gradually adds fixed Gaussian noise to the observed data point zo via known scales 9n to define a sequence of progressively noisier values #1, #2, ................................ 4 The final noisy output æ ~ (0, I) carries no information about the original data point and thus the sequence of positive noise (variance) scales $1..... ..... AN has to be increasing such that the first noisy output æ1 is close to the original data æo and the final value N is pure noise. The goal is then to learn to reverse this process.
As diffusion forms a Markov chain, the transition between any two consecutive points is defined with conditional probability q< (xn1) N((II- Bn I). Since the transition kernel is Gaussian, the value at any step n can be sampled directly from æp- Given On, = 1 - 3, and
on I1k=1 O%k, we can write:
( (0o) NVVān (1 - -
(1)
Further, the probability of any intermediate value In-1 given its successor H, and initial zy is
q( 1 zo) = NN B,I1,
(2)
where: in 1.m xo
Van(1 - an-1)
an-11
En.
A,
Sn.
(3)
ān
1 1-ān
an
The generative model learns the reverse process p( 1ax)) Sohl-Dickstein et al. (2015) set p(æn-1122,) (z,) = (2u n), 3,, and parameterized peg with a neural network. The training objective is to maximize the evidence lower bound log p(xo) Z
E, (rulo((x//<))l((x))) + Σ DKL (n) #>1
- log log p( (x)))
(4)
In practice, however, the approach by Ho et al. (2020) is to reparameterize μ4 and predict the noise that was added Fg, using a neural network cp (Z,, n), and minimize the simplified loss function:
C(xo) Be-N((o,)) n..oo,...... [l|ee( (Veevv,, + vI R) <l12]
(5)
To generate new data, the first step is to sample a point from the final distribution xN ~ N and then iteratively denoise it using the above model (æN + ÆN-1 H from the data distribution. To summarize, the forward process adds the noise e to the input æp. at different scales, to produce æ,,, The model learns to invert this, i.e., predicts the noise from Cn.
2g)) to get sample
2.2 SDE DIFFUSION
Instead of taking a finite number of diffusion steps as in Section 2.1, Song et al. (2021) introduce a continuous diffusion of vector valued data, Zy ト F, where E [0. is now a continuous variable. The forward process can be elegantly defined with an SDE:
dæ s)ds + g(s)dW..
where W is a standard Wiener process. The variable s is the continuous analogue of the discrete steps implying that the input gets noisier during the SDE evolution. The final value øs ~ p(æs) will follow some predefined distribution, as in Section 2.1. For the forward SDE in Equation there exist a corresponding reverse SDE (Anderson, 1982):
dæ, [f s) (s)) V bogp(x,)) g(s)dW,
2

---

Under review as a conference paper at ICLR 2023
where Va. Va, log.o, is the score function. Solving the above SDE from S to O, given initial con- dition æy ~ p(æs) returns a sample from the data distribution. The generative model's goal is to learn the score function via a neural network (0 s), by minimizing the following loss:
((x)) = --D [ 8) V logg((>)))
(8)
Song et al. (2021) define the continuous equivalent to DDPM forward process as the following SDE:
1 (())x, + Caataads++BBBB080 VB())WW,, 2
dx
where ((s) and S are chosen in such way that ensures the final noise distribution unit normal, 2 (0, I). Given this specific parameterization, one can easily derive the transition probability (20, and calculate the exact score in closed-form (see Section 3.3 and Appendix A.3).
3 DIFFUSION FOR TIME SERIES DATA
In contrast the previous section which deals with data points that are represented by vectors, are interested in generative modeling for time series data. We represent time series as a set of points X {ax(t))... ...t.,....... ... (tMM))) -1]]. E [0, T'. observed across M timestamps. The observations can be equally spaced but this formulation encompasses iregularly-sampled data as well. We assume that each observed time series comes from its corresponding underlying continuous function xx-).
Our approach can be viewed as modeling the distribution "p(x(-))" over functions instead of vectors, which amounts to learning the stochastic process. To preserve continuity, we cannot apply the ideas from Section 2 directly, unless we assume measurements are independent of each other. The issue of adding an independent noise in the diffusion arises because it produces discontinuous samples, which is at odds with our assumption. In the following, we address this and propose a solution.
3.1 STOCHASTIC PROCESSES AS NOISE SOURCES FOR DIFFUSION
Instead of defining the diffusion by adding some scaled noise vector æ, we add a noise function (stochastic process) E(-) to the underlying data function xx-). The only restriction on c(-) is that it has to be continuous so that the output remains continuous well. which clearly rules out stochastic processes where time is indexed by a finite set e.g. <(.) ~ NN I). However, using a normal distribution proved be very convenient in Section 2 it allows for closed-form formulations of various terms, especially the loss. This is due to the fact that adding two Gaussian random variables leads another Gaussian variable, which, as aside, is a property shared by some other parametric distributions (Nachmani et al., 2021).
I) to data vector
Therefore, our goal is define E(-) which will satisfy the continuity property while giving us tractable training and sampling. Note that ł refers to the time of the observation and e(t) is the noise in contrast to the previous section where time-like variables and referred to the noise scale.
We could consider obtaining the noise from a standard Wiener process e(-) W(-). Although we will have all the terms closed-form: a clear disadvantage of this approach is that variance grows with time. Additionally, the distribution of W(O) is degenerate as we never add any noise. This can be solved in an ad hoc manner by shifting the whole time series similar to Deng et al. (2020).
Instead, in the following, we present two stationary stochastic processes that add the same amount of noise regardless of the time the observation. Note that the noise is correlated the time dimen- sion, hence the use of the stochastic process. An additional nice property of these processes is that they reduce to the diffusion from Section 2 in the trivial case of time series with only one element.
1. Gaussian process prior. Given a set of M time points t, propose sampling e(t), t E t from Gaussian process A (O, Σ). where each element of the covariance matrix is specified with a kernel cov(t, u) k(t, u). This produces smooth noise functions c(-) that can be evaluated at any t.
To define a stationary process, we have to use a stationary kernel; we will use a radial basis kernel kttt, u) exp(->(t 2) Adjusting the parameter y lets vary the flatness of the noise curves. Given a set time points t, we can easily sample from this process by first computing the covariance Σ k(t, t) and then sample from the multivariate normal distribution N (0, E).
3

---

Under review as a conference paper at ICLR 2023
Algorithm Loss (DSPD-GP diffusion)
Algorithm 2 Sampling (DSPD-GP diffusion)
I: Xo. L ~ pami (X,t)
1: input: t. = {fa...... fM-
2: Σ k(t,t)
2: Σ k(t, L = Cholesky(E)
3: Cholesky(X)
Xw (V 2)
4: é ~ I)O,
for n= N..... do
5: E = Le
5:
x~ *~,(0,
6: ~ U([I..... N))
6:
Xa-1= van X
1-an Lee Luo + Bo Vi-on
7: X, =van Xo + V1 a,e
7: end for
|E - cx (X,,
return Xo
2. Ornstein-Uhlenbed difflusion. The alternative noise distribution is a stationary OU process which is specified solution the following SDE:
de = - + aW,,
(10)
where We the standard Wiener process and we use the initial condition 6n ~ NOO,, I). We can obtain samples from OU process easily by sampling from time- changed and scaled Wiener process: ""a". The covariance can be calculated as cov 2) = exp(-7lt u/). The OU process is a special case of Gaussian process with Matérn kernel (v 0.5) (Rasmussen & Williams, 2005, 86). We discuss different sampling techniques and their trade-offs Appendix A.4.
In the end, both the GP and OU processes defined with multivariate normal distribution (O, Σ). where Σ is calculated using the times of the observations. As opposed to the diffusions from Section 2, we use correlated noise in the forward process. Our approach allows to produce continuous functions as samples and will prove to be a natural way to do forecasting and imputation.
3.2 DISCRETE STOCHASTIC PROCESS DIFFUSION (DSPD)
We apply the discrete diffusion framework to the time series setting. Reusing the notation from before, Xo denotes the input data and X,, = {a.,, (lo), F., (1ar is the noisy output after diffusion steps. In contrast to the classical DDPM (Section 2.1) where one adds independent Gaussian noise to data, we now add the noise from a stochastic process. In particular, given the times of the input observations, or OU process as defined in Section 3.1. We can write the transition kernel and the posterior as:
compute the covariance E and sample noise <(-) from GP
X X = NVVV Xo, (1 Cz )E)
(11)
q( X X N N B),,,
(12)
where the difference to Equations and 2 is the inclusion of Σ. Full derivation is in Appendix A.1.
The generative model defined with the reverse process pl xx, = (Nu n). 3, E), similar Ho al. (2020), but swap the identity matrix for Σ. Another key difference is that the model now takes the full time series and the time points in order to output the prediction which has the same size as X.. The architecture, therefore, has be type of time series encoder-decoder.
Since all the probabilities are still normal, the terms the ELBO (Equation 4) can be calculated in closed-form. Following Ho al. (2020), we also adopt the reparameterization which predicts the noise E(-) and use the simplified loss as in Equation 5:
((x)) (o, ( [I (vav, VI a. n) =|
(13)
More details can be found in Appendix A.2. The sampling is similar Ho al. (2020) but the noise comes from stochastic process instead of an independent normal distributions. In Algorithms and we show the training and the sampling procedure, respectively, when using the GP diffusion.
3.3 CONTINUOUS STOCHASTIC PROCESS DIFFUSION (CSPD)
Similarly to the previous section, we can extend the continuous diffusion framework to use the noise coming from Gaussian OU process. Now, the noise scales ((s) are continuous in the diffusion
4

---

Under review as a conference paper at ICLR 2023
time 8, Section 2.2. Given factorized covariance matrix Σ LL", modify the variance preserving diffusion SDE (Song et al., 2021):
dX, 8 2 (ss) V BB))
(14)
which gives us the following transition probability (see Appendix A.3 for details):
q (X, = (i, Σ) NV (Xoe So B(s)ds Σ -1-e- e 12 A(s)d)))
(15)
Since this probability is normal, the value of the function be computed in closed-form:
Vx, log g X, Xo) - jz),
(16)
which we can use to optimize the same objective as in Equation 8. Our neural network eø x., t, s) will take in the full time series, together with the observation times and the diffusion time s, and predict the values of the score function. As it turns out, we can again use the reparameterization in which we predict the noise, whilst the score only calculated when sampling new realizations.
3.4 IMPLEMENTATION DETAILS
In our work, we consider multivariate time series which means each observation at a certain time point is a d-dimensional vector. In the forward diffusion process we treat the data as individual univariate time series and add the noise to them independently. This is in line with the previous works where, e.g., independent noise is added to the individual pixels in an image. The model takes in complete noisy multivariate time series to learn the reverse process handles the correlations between the data dimensions and across the time dimension.
We note that the best results are obtained if the model is reparameterized always predict the independent Gaussian noise. In the discrete stochastic process diffusion, this means the noise is computed
LE, (O, I). where L Σ the covariance matrix of the stochastic process. The model then learns to predict č Similarly, in the continuous diffusion, we represent the score as Lē/a², where 6² 1 exp(- J (ss)ds) (Equation 16). In both cases, when we sample, the model will output the prediction of é, which we transform to get the sample from the stochastic process or to obtain the score function value. example of this can found Algorithm 2 for the discrete diffusion.
4 APPLICATIONS
To train a generative model, it must learn to reverse the forward diffusion process by predicting the noise that was added to the clean data. The input to the model is the time series (Xo, t) along with the diffusion step or diffusion time and the output is of the same size Xo- If additional inputs are available, we can also model the conditional distribution; for example, time series data often contains covariates for each time point of t. We can also condition the generation the past observations which essentially defines a probabilistic forecaster or condition only on the observed values which defines neural process or imputation model.
4.1 FORECASTING MULTIVARIATE TIME SERIES
Forecasting is answering what is going to happen, given what we have seen, and as such is the most prominent tasks in time series analysis. Probabilistic forecasting adds the layer of (aleatoric) uncer- tainty on top of that and returns the confidence intervals which is often a requirement for deploying models in real world settings. The neural forecasters are usually encoder-decoders, where the his- tory of observations (X" t") is represented with single vector and the decoder outputs the distribution the future values X given at time points t" Previous works relied on specify- ing the parameters of the output distribution, e.g., via diagonal covariance (Salinas et al., 2020) or some low-rank approximation (Salinas et al., 2019), relying on normalizing flows (de Bézenac et al., 2020: Rasul et al., 2021b), Generative Adversarial Networks (GANs) (Koochali al., 2021).
Recently, Rasul et al. (2021a) introduced a diffusion based forecasting model to learn the conditional probability p( X H). In particular, let X {ax(lo), ..., (0(MM-)) be history window of
5

---

Under review as a conference paper at ICLR 2023
Residual layer
X
Comv2d
Comv2d
π2
Piss cmb
Linoar
Conv2d-RoLU
ROLU
MLP
Linear
RoLU
Dilated Conv2d
Conv2d
MLP
Linear
Gated aclivation
Cn
Figure 2: Overview of the forecasting model. The inputs are time series Xn. diffusion step n, observation times and the history vector z. The output the predicted noise value 6n.
size M sampled randomly from the full training data. They specify the distribution p(m((tx) | using a conditional DDPM model. The forward process adds independent Gaussian noise to x (t) the same way as in DDPM. However, the reverse denoising model is conditioned on the history X which is represented with a fixed sized vector 2. After training is completed, the predictions are made in the following way: (1) X H is encoded with RNN obtain 2; (2) the initial noisy value is sampled x (lM) ~ (O, I); and (3) denoising is performed using the sampling algorithm from Ho al. (2020) but conditioned on obtain (las))). The final denoised value is the forecast and sampling multiple values allows computing empirical confidence intervals of interest.
In Rasul al. (2021a), the timestamps are always discrete and the prediction is autoregressive, i.e., the values are produced one by one. Our diffusion framework offers two key improvements: first, the predictions can be made at any future time point (in continuous time); and second, we can predict multiple values at the same time which scales better on modern hardware.
In our case, the prediction X will not be single vector but a set (æ(tt)...... =(M).,, } of size K. This type of data is naturally handled by process diffusion as defined in Section 3. The only thing left to do to design suitable denoising model Eg. Previous observations are again represented with an RNN to obtain and condition the reverse process on it. We propose an architecture similar to the TimeGrad model from Rasul et al. (202 (2021a) but which outputs multiple values simultancously. Figure 2 shows the architecture overview: we add t as an input and take the whole time series at once. Thus, we use 2D convolution where the extra dimension corresponds to the time dimension.
4.2 DIFFUSION PROCESS AS A NEURAL PROCESS
Neural processes (Garnelo et al., 2018) are a class of latent variable models that define a stochastic process with neural networks. Given set of data points (a dataset), the model outputs the probability distribution over the functions that generated this dataset. That is, for different datasets, the model will define different stochastic processes. Due this behavior, neural processes bear a resemblance to the Gaussian processes but can also be viewed as meta learning model (Hospedales et al., 2021).
Let X denote the observed data, in case, time series, and let X be the unobserved data at the time points t Garnelo et al. (2018) construct the mncoder-decoder model that uses the amortized variational inference for training (Kingma & Welling, 2014). The encoder takes in a set of observed points (xx,, tA) and outputs the distribution over the latent variable q(z). It is crucial that the encoder permutation invariant, i.e., the order in of the input points does not alter the result. This is easy to achieve using, e.g., deep sets (Zaheer et al., 2017). The decoder takes in the sampled latent vector z and the query time points & and predicts the values of the unobserved points X BB
Since our approach samples functions, we can condition the generation on an input dataset (X'A ( in order to create our version of neural process, based purely on the diffusion framework. The encoder will be deterministic neural network that outputs the latent vector z. contrary to (Garnelo et al., 2018) which outputs the distribution. Similar to Section 4.1, the diffusion is conditioned on and
output samples for any query t! Therefore, we capture the distribution X B X 4) directly. During training. we adopt the approach feeding in the data such that we learn p( X X B X A) which helps the model learn to output high certainty around td, see Garnelo et al. (2018).
In the end, our model sees many processes. The model learns to represent the observed points X such that the denoising process
pairs coming from different true underlying

---

Under review as a conference paper at ICLR 2023
corresponds to the correct distribution, given XA. After training is completed, take time series X A and output the samples at any set of query time points t". We can view such an approach as an interpolation or imputation model that fills-in the missing values across time. The main appeal is the ability to capture different stochastic processes within a single model.
4.3 PROBABILISTIC TIME SERIES IMPUTATION
Previous section considered interpolating points in time. Now, we look into filling-in the missing values across the observation dimensions, i.e., the imputation of the vectors. An element of the time series X is assigned a mask m of the same dimension that indicates whether the i-th value x; has been observed (m, = or is missing (m, = 0).
Given observed X and missing points XB_ Tashiro et al. (2021) propose a model that learns a conditional distribution p( X (x-). The model built upon a diffusion framework and the reverse process is conditioned on XA, similar to Section 4.2. We extend this by introducing noise from stochastic process, as presented above. The learnable model remains the same but we introduce the correlated noise in the loss and sampling. We posit that continuous noise process as an inductive bias for the irregular time series is a more natural choice.
RELATED WORK
Generative modeling with diffusion (Sohl-Dickstein al., 2015; Ho al., 2020; Song et al., 2021) recently gained traction as it provides good sampling quality on image generation (Dhariwal & Nichol, 2021; Ramesh et al., 2022; Rombach et al., 2021) and became the state-of-the-art method replacing GANs (Goodfellow et al., 2020). The modeling power translates to other tasks as well, so has been used in modeling, e.g., waveforms (Kong et al., 2021) and time series forecasting (Rasul et al., 2021a), but also discrete data such as text (Austin et al., 2021) and molecules (Anand & Achim, 2022; Lee & Kim, 2022).
The number of steps can be either fixed (Ho et al., 2020) or the steps can be continuously indexed (Song al., 2021; Huang et al., 2021). Based on this, the reverse process usually defined as denoising or score-based, respectively. We implement both of the approaches in our stochastic pro- cess diffusion. Many of the advances over the original approach focused on improving the sampling speed (Chung al., 2022; Jolicoeur-Martincau et al., 2021; Lyu et al., 2022), while others imple- ment the noise scheduling for better modeling capacity (Nichol Dhariwal, 2021b; Kingma et al., 2021). We can implement any of these techniques that improve general diffusion to make our meth- ods perform faster or have better sampling quality.
Neural ordinary differential equations (Chen et al., 2018) allow capturing the irregularly sampled time series as they can naturally handle the continuous time. As such, this work is a building block which can also be used alongside our method to devise more powerful denoising networks. Rubanova et al. (2019) construct an aucoder-decoded architecture based on neural ODEs which resembles the variational autoencoder (Kingma & Welling, 2014). The time series is, thus, modeled in a latent space by sampling random vector which is propagated with an ODE. Neural stochastic differential equations (Li et al., 2020) extend this by adding noise in every solver step. This still amounts to generating the time series in single pass, from to to tM 11-1. whereas we use the diffusion framework which refines the generated time series from pure noise XN to the final sample Xo.
Continuous-time flow process (CTFP) (Deng et al., 2020) uses normalizing flows (Kobyzev et al., 2020) to generate the time series by sampling the initial noise from the stochastic process and trans- form it with invertible function to obtain the sample from the target distribution. Although this allows exact likelihood training, the method cannot capture some processes (Deng et al., 2021) and is often augmented to include latent variable, i.e., it is trained
VAE.
Implicit neural representations (Sitzmann et al., 2020; Dupont et al., 2022) treat data points as func- tions, i.e., each data point is represented by its neural network which acts form of compres- sion. In our work, we treat time series as continuous functions but instead of fitting a different neural network to each point we model the probability over the functions using the diffusion framework.
7

---

Under review as a conference paper at ICLR 2023
CIR
Lorenz
OU
Predator- prey
Sine
Sink
DSPD
Gauss
-0.4830=0000766
1.1661:0.0987
0.7081+:00199
-3. 3.8666=0.0337
-1.3978:000108
-5.8355-0.0300
OU
-0.4694=0.0154
-733334-0.144p
0.5298+00.0229
-944932-00054
4.2337-+000057
-1.31145+0.2015
GP
-0.4512=0.0606
-8,4433=0.2435
0.5622+0.0050
-0.11138:0.4001
-..566810.0332
-1,400010000404
CSPD
Gauss
-0.4914=0.0073
1.9043:1.3224
0.5333+:0.0444
-3 2.5642=002014
-1.12042..05800
-5.6209 2057
OU
-0.4796=0.0100 -6.6638=0.0773
0.4959=0.011
-8.7748=0.141 -9.3409+0.199
-3.9496::0.1
-10.4739=0.248
GP
-0,4935 0.0052
-7.5565=0.4107
0.5251+0.0115
4.3591+-011222
0:22225
Table 1: Negative logeeeikioood on synthetic data (lower is better) shows OU/GP consistently better than independent noise.
Lorenz
OU
Predator-prey
Sine
Sink
CTFP Latent ODE
0.00051-0.0012 1.0+0.0 0.5115::0.0282
0.783:10.0756 0,955=000057
0.729=0.0227
0.981=-0.0104 1.0-0.0 0.5255:100009
0.7265 0.907+0.0394
0,998
0,512++0.0331 0.5055:10.0588
0.0213 0.5855=000219
DSPD-GP (Our)
0.5135+0002888
0.513=
Table 2: Accuracy of the discriminator trained to distinguish real data and model samples.
30
CR
0U
Predelcoprey
Sie
SEN
1.0
Figure 3: Real data and samples from our model based on an
process.
6 EXPERIMENTS
Probabilistic modeling. We test our DSPD and CSPD with independent Gaussian noise and noise from stochastic processes (GP and OU) on six synthetic datasets, coming from deterministic and stochastic dynamical systems. We also compare to the established baselines for irregular time series modeling, namely, latent ODEs (Rubanova et al., 2019) and continuous -imee flow process (CTFP) (Deng al., 2020). The detailed experimental setup along with further results is in Appendix B.
Table shows that using a stochastic process the noise source outperforms independent noise. The ablation in Table shows that using an independent denoising model, i.e., Eg that processes time series inputs individually, performs worse than model that processes the whole time series at once. Figure 3 demonstrates the quality of the samples. Finally, Table 2 compares our model with the baselines and demonstrates that our model produces samples that are indistinguishable to powerful transformer- based (Vaswani et al., 2017) discriminator model. The same does not hold for the competing methods.
Forecasting. We test our model as defined in Section 4.1 and Figure 2 against TimeGrad (Rasul al., 2021b) three established real-world datasets: Electricity, Exchange and Solar (Lai al., 2018). Due to the limitations of the CRPS-sum metric (Koochali et al., 2022), we report the NRMSE and the energy score (Gneiting & Raftery, 2007) averaged over five runs, but we note that the rank of the models' performance does not change when using other metrics as well. Table 3 shows that our method outperforms TimeGrad even though we predict over the complete forecast horizon at once, and Figure 4 demonstrates the prediction quality alongside the uncertainty estimate.
Neural process. We construct dataset where each time series X comes from different stochas- tic process, by sampling from Gaussian processes with varying kernel parameters and time series lengths. This a standard training setting in neural process literature (Garnelo et al., 2018). In our denoising network, we modify the attention- like layer to make it stationary (see Appendix B.2) and train as described in Section 4.2. Due to the use tanh activations in the final layers, combined with its stationary, our model extrapolates well, i.e., when tanh saturates the mean and variance do not vary far from observations. This is the same behaviour we see in the GP with an RBF kernel, for example. The quantile loss of the unobserved data under the true GP model is 0.845 while achieve 7.7 which indicates we capture the true process, which can also be seen in Figure 5. We remark that attentive neural process (Kim et al., 2019) does not produce the correct uncertainty.
8

---

Under review as a conference paper at ICLR 2023
TimeGrad
Ours
Dime iolon 20
O'monsioon 120
Dime nolon 250
Electricity
0.064=0.007
8425+611
0.045=0,002
7079++166 164
120
Vedan peddction
Dan
0.013:10.003
0.012:10.001
50% Hetnee
65
Exchange
0.057=0.002
0.031:10.022
0. 0.799=0.099
0.757=0,026
Solar
150:117
166:112
1000
(1:3
20
0U/00
2350
0000
Table 3: NRMSE (top) and energy score on real-world forecasting data.
Figure 4: Forecast and uncertainty intervals on Electricity.
Imputation. We compare to the CSDI model (Tashiro et al., 2021) introduced in Section 4.3 on an imputation task. To this end, seeds and model architecture, but change the noise source to Gaussian process. Following Tashiro al. (2021), we use Physionet dataset (Silva et al., 2012) which is a collection of medical time series collected at hourly rate. It already contains missing values but for testing purposes we choose varying degrees of missingness and report the results on the test set. We update the loss and sampling accordingly. as in Section 3. Table 4 shows that we outperform the original CSDI model even though we only changed the noise, and the dataset we used has regular time sampling.
exactly the same training setup, including the random
Truc Gaussion
Dillusion model
Missing
CSDI
DSPD-GP (Our)
2-
Contzar
10%
0.520=10.055
0.498:1.0336
50%
0.644:1.004
0.64411.029
90%
0.818110002
0.019
Table 4:
Imputation RMSE on Physionet
data with varying amounts of missingness.
0.9
0.0
0.2
0.5
Figure 5: Sampled curves given set of points.
DISCUSSION
In this paper, we introduced the stochastic process diffusion framework for time series modeling- We demonstrated that the improvements over the previous works come from (1) using the stochastic process as the noise source; and (2) using the model that takes the whole time series at once, instead of modeling points independently. We also show how one can condition the generation to obtain a forecasting model as well as interpolation and imputation models.
In experiments used mixture of synthetic and real-worke datasets which both have regular and irregular sampling. We outperform strong baseline models on all of the tasks which demon- strates practical utility of method.
7.1 FUTURE WORK
We used bare-bones diffusion without extensive tuning to demonstrate the modeling potential and make a fair comparison to other methods. However, it should be straightforward to improve upon our models by implementing recent advances diffusion models (e.g., Nichol & Dhariwal, 2021a).
In case we have large amount of points, can consider replacing the current sampling strategies with more scalable variants, such as switching to sparse GP (Quinonero-Cande) & Rasmussen, 2005). Additionally, time series encoder-decoder which might be helpful for high-dimensional data, such as those we encountered in the forecasting task.
can train the latent diffusion (Rombach et al., 2021) by first learning the
would be interesting to explore different architecture choices, e.g., implement improvements in also apply the
conditioning models via leamned activations (Ramos et al., 2022). Finally, presented methods to other areas outside time series, such as modeling point clouds or even images, have demonstrated that method is competitive regular grids.

---

Under review as a conference paper at ICLR 2023
ETHICS STATEMENT
We introduced new method for time series generation. As such, it has many applications, such as probabilistic forecasting and imputation, both of which are practical significance in the real- world settings. In particular, we would like to successful applications in the healthcare domain. As with any generative model, one has to pay attention to the privacy and fairness when collecting data and building model. We do not anticipate any negative outcomes applying our model on time series data. No personal information is contained in any the datasets.
REPRODUCIBILITY STATEMENT
Throughout the paper describe in detail how novel stochastic process diffusion framework works, including the algorithms for training and generation. We also describe the datasets and models used, both in the main text and the Appendix. All the datasets used publicly available and the code that reproduces the results is released to the reviewers.
REFERENCES
Namrata Anand and Tudor Achim. Protein structure and sequence generation with equivariant de- noising diffusion probabilistic models. arXiv preprint arXiv:2205.15019, 2022. 7
Brian D.O. Anderson. Reverse-time diffusion equation models. Stochastic Processes and their Applications, 12(3):313-326 1982.
Jacob Austin, Daniel D. Johnson, Jonathan Ho, Daniel Tarlow, and Rianne van den Berg. Struc- tured denoising diffusion models in discrete state-spaces. In Advances in Neural Information Processing Systems (NeurIPS), 2021.
Ricky Q. Chen, Yulia Rubanova, Jesse Bettencourt, and David Duvenaud. Neural ordinary differential equations. In Advances in Neural Information Processing Systems (NeurIPS), 2018. 1,7,11 17
Hyungjin Chung, Byeongsu Sim, and Jong Chul Ye. wwaae-llooee--ffseffesseeee Accelerating ditional diffusion models for inverse problems through stochastic contraction. In Proceedings of the IEEF/CVF Conference Computer Vision and Pattern Recognition, 2022. 7
Junyoung Chung, Çaglar Gülgehre, KyungHyun Cho, and Yoshua Bengio. Empirical evaluation of gated recurrent neural networks on sequence modeling. arXiv preprint 4rXhv:1412.3555, 2014.
17
William Jay Conover. Practical nonparametric statistics, volume 350. john wiley sons, 1999. 18
Emmanuel de Bézenac, Syama Sundar Rangapuram, Konstantinos Benidis, Michael Bohlke- Schneider, Richard Kurle, Lorenzo Stella, Hilaf Hasson, Patrick Gallinari, and Tim Januschowski. Normalizing kalman filters for multivariate time series analysis. Advances in Neural Informa- tion Processing Systems (NeurIPS), 2020.
Ruizhi Deng, Bo Chang, Marcus Brubaker, Greg Mori, and Andreas Lehrmann. Modeling con- tinuous stochastic processes with dynamic normalizing flows. In Advances in Neural Information Processing Systems (NeurIPS), 2020. 1, 17
Ruizhi Deng, Marcus Brubaker, Greg Mori, and Andreas Lehrmann. Continuous latent process flows. In Advances in Neural Information Processing Systems (NeurIPS), 2021. 7
Prafulla Dhariwal and Alexander Quinn Nichol. Diffusion models beat gans on image synthesis. In Advances in Neural Information Processing Systems (NeurIPS), 2021.
Laurent Dinh, Jascha Sohl-Dickstein, and Samy Bengio. Density estimation using real NVP. In International Conference Learning Representations, (ICLR), 2017. 17
10

---

Under review as a conference paper at ICLR 2023
Emilien Dupont, Hyunjik Kim, S. M. Ali Eslami, Danilo Jimenez Rezende, and Dan Rosenbaum. From data to functa: Your data point is a function and you can treat like one. In International Conference Machine Learning (ICML), 2022.
Marta Garnelo, Jonathan Schwarz, Dan Rosenbaum, Fabio Viola, Danilo J Rezende, SM Eslami, and Yee Whye Teh. Neural processes. In ICML 2018 workshop on Theoretical Foundations and Applications of Deep Generative Models, 2018. 6,
Tilmann Gneiting and Adrian E Raftery. Strictly proper scoring rules, prediction, and estimation. Journal of the American Statistical Association, 102(477):359-37) 2007. 8
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley. Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks. Communications of the ACM, 63(11) 634414413--14, 2020.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusson probabilistic models. In Advances in Neural Information Processing Systems (NeurIPS), 2020. 2, 4, 6, 7, 14, 15
Timothy Hospedales, Antreas Antoniou, Paul Micaelli, and Amos Storkey. Meta-learning in neural networks: A survey. IEEE transactions on pattern analysis and machine intelligence, 44(9): 5149-5169 2021. 6
Chin-Wei Huang, Jae Hyun Lim, and Aaron C. Courville. A variational perspective on diffusion- based generative models and score matching. In Advances in Neural Information Processing Systems (NeurIPS), 2021. 7
Alexia policoeur-Martinen Ke Li, Rémi Piché-Tailleferr Tal Kachman, and loannis Mitliagkas. Gotta go fast when generating data with score-based models. arXiv preprint 105i::2005.4088). 2021. 7
Hyunjik Kim, Andriy Mnih, Jonathan Schwarz, Marta Garnelo, S. M. Ali Eslami, Dan Rosenbaum, Oriol Vinyals, and Yee Whye Teh. Attentive neural processes. In 7th International Conference on Learning Representations, (ICLR), 2019.
Diederik Kingma, Tim Salimans, Ben Poole, and Jonathan Ho. Variational diffusion models. In Advances in Neural Information Processing Systems (NeurIPS), 2021. 7
Diederik P. Kingma and Max Welling. Auto-encoding variational bayes. In International Conference on Learning Representations, (ICLR), 2014. 6, 7
Ivan Kobyzev, Simon JD Prince, and Mareus Brubaker. Normalizing flows: An introduction and review of current methods. IEEE transactions pattern analysis and machine intelligence, 43 (11):3964-3979 2020. 7, 17
Zhifeng Kong, Wei Ping, Jiaji Huang, Kexin Zhao and Bryan Catanzaro. Diffwave: A versatile diffusion model for audio synthesis. In International Conference on Learning Representations (ICLR), 2021.
Alireza Koochali. Andreas Dengel, and Sheraz Ahmed. you like GAN it probabiiiliitic multi- variate times series forecast with GAN. Engineering Proceedings, 5(1), 2021. 5
Alireza Koochali, Peter Schichtel, Andreas Dengel, and Sheraz Ahmed. Random noise vs. state- of-the-art probabilistic forecasting methods: A case study on crps-sum discrimination ability. Applied Sciences, ((10))5104, 2022. 8
Guokun Lai, Wei-Cheng Chang, Yiming Yang, and Hanxiao Liu. Modeling long- and short-term temporal patterns with deep neural networks. In ACM SIGIR Conference on Research & Devel- opment in Information Retrieval, 2018. 8
Jin Sub Lee and Philip M Kim. Proteinsgm: Score-hased generative modeling for de novo protein design. bioRxiv, 2022.
Xuechen Li, Ting-Kat Kam Leonard Wong, Ricky T. Q. Chen, and David Duvenaud. Scalable gradients for stochastic differential equations. In International Conference on Artificial Intelligence and Statisties, 2020. 1,
11

---

Under review as a conference paper at ICLR 2023
Zhaoyang Lyu, Xudong Xu, Ceyuan Yang, Dahua Lin, and Bo Dai. Accelerating diffusion models via early stop of the diffusion process. arXiv preprint arXiv:2205.1252. 2022.
Eliya Nachmani, Robin San Roman, and Lior Wolf. Non gaussian denoising diffusion models. arXiv preprint arXiv:2 075822 2021. 3
Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. In International Conference on Machine Learning, (ICML), 2021a. a. 9
Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. In International Conference on Machine Learning (ICMI.) 2021b.
Joaquin Muuñonner--andden and Carl Edward Rasmussen. A unifying view of sparse approximate gaussian process regression. Journal of Machine Learning Research, 6(65):1939-199) 1959, 2005. 9
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text- conditional image generation with CLIP latents, arXiv preprint arXivv2204.06112,, 2022. 7
Alberto Gil Couto Pimentel Ramos, Abhinav Mehrotra, Nicholas Donald Lane, and Sourav Bhat- tacharya. Conditioning sequence-to-sequeee networks with learned activations. In International Conference on Learning Representations (ICLR), 2022. 9
Carl Edward Rasmussen and Christopher K. I. Williams. Gaussian Processes for Machine Learning-
The MIT Press, 2005. 4
Kashif Rasul, Calvin Seward, Ingmar Schuster, and Roland Vollgraf. Autoregressive denoising dif- fusion models for multivariate probabilistic time series forecasting. In International Conference on Machine Learning (ICML), 2021a. 5, 6,
Kashif Rasul, Abdul-Saboot Sheikh, Ingmar Schuster, Urs M Bergmann, and Roland Vollgraf. Mul- tivariate probabilistic time series forecasting via conditioned normalizing flows. In International Conference on Learning Representations (ICLR), 2021b. 5, 8
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. arXiv preprint arXiv:21 2021. 7.9
Yulia Rubanova, Ricky T. Q. Chen, and David K Duvenaud. Latent ordinary differential equa- tions for seggularly-samplee. time series. In Advances in Neural Information Processing Systems (NeurIPS), volume 32, 2019. 8,8
David Salinas, Michael Schnee-, Laurent Callot, Roberto Medico, and Jan Gasthaus. High-dimensional multivariate forecasting with low-rank Gaussian Copula Processes. In Ad- vances Neural Information Processing Systems (NeurIPS). 2019.
David Salinas, Valentin Flunkert, Jan Gasthaus, and Tim Januschowski. DeepAR: Probabilistic forecasting with autoregressive recurrent networks. International Journal of Forecasting, 36(3): 1181-1191, 2020. 5
Ikaro Silva, George Moody, Daniel Scott, Lco A Celi, and Roger G Mark. Predicting in- in-hospital mortality of ieu patients: The ahysionet/computing in cardiology challenge 2012. In 2012 Com- puting in Cardiology. IEEE, 2012. 9
Vincent Sitzmann, Julien Martel, Alexander Bergman, David Lindell, and Gordon Wetzstein. Im- plicit neural representations with periodic activation functions. In Advances Neural Information Processing Systems (NeurIPS), 2020. 7
Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep Unsuper- vised Learning using Nonequilibrium Thermodynamies. In International Conference on Machine Learning (ICML), 2015. 7.7
Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Scsee-based generative modeling through stochastic differential equations. In Interna- tional Conference on Learning Representations (ICLR), 2021. 2, 3, 15
12

---

Under review as a conference paper at ICLR 2023
Simo Särkkä and Arno Solin. Applied Stochastic Differential Equations. Institute of Mathematical Statistics Textbooks. Cambridge University Press, 2019. 15
Yusuke Tashiro, Jiaming Song, Yang Song, and Stefano Ermon. Csdi: Conditional score-based diffusion models for probabilistic time series imputation. In Advances Neural Information Processing Systems (NeurIPS), 2021. 7, 18
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, ukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances Neural Infor- mation Processing Systems (NeurIPS), 2017. 8, 17
Manzil Zaheer, Satwik Kottur, Siamak Ravanbakhsh, Barnabas Poczos, Russ R Salakhutdinov, and Alexander J Smola. Deep sets. In Advances in Neural Information Processing Systems (NeurIPS), 2017. 6
13

---

Under review as a conference paper at ICLR 2023
A DERIVATIONS
A.1 DISCRETE DIFFUSION POSTERIOR PROBABILITY
We extend Ho et al. (2020) by using full covariance E() to define the noise distribution across time If LL" and keeping the same definitions from Section 2.1 for Bn, On and on, We can write:
x, /1 - Bn + Le, X, vā + v1 or, Le,
(17)
(18)
with e E N/ I). This corresponds to the following transition distributions:
= (,
(19)
g( Xa) = (N(ā, an )).
(20)
We are interested in q Xo) Xo) x (Xxn|Xnn (nnni(XXxn(xxxo) Since both distributions on the right-hand side are normal, the result will be normal well. We can write the resulting distribution as N Ni, Σ), where:
R(X, Apd) + MI
$ Σ RAET
R = E, A (AE,, A + Σ2
with A= VT 1,I, 11]
Xo, E, ān- E, and E₂ = B,E. We can now write:
R R= (1 an E E 1 Bn / Bn 1n- Bn + ),
an an
ΣΣ
an (1 an-1) + an
1 an-l ann
an
and from there:
1 an-1
Van
Xn-V1- 1V-1ww
-uvan- -1----V--aa---
v
On
1 an-l
1 - -n-n On
an + vān-1
1
Xo
(21)
an
- an
1 On-1
VanXn + Vnn-1
B 80x
an
I an
and using the fact that Σ is a symmetric matrix:
Σ (1 an - On
an /a, -B - adn-)=)))
1 - an-l
Onl - an-1)) E
1 On
(22)
On
1 an-1
E.
an
Therefore, the only difference to the derivation in Ho et al. (2020) is the Σ(t) instead of the identity matrix in the covariance.
A.2 DISCRETE DIFFUSION LOSS
We use the evidence lower bound from Equation 4. The distribution (Xx,, (X,,, Xn) is defined (,,Xx) Miū C Σ),, where C₁ is some constant (Equations 21 and 22). Similar to Ho al. (2020),
14

---

Under review as a conference paper at ICLR 2023
choose the parameterization for the reverse process p(Xn-1(Xx)) t,)
where:
pg X t,
X,
Bn
(o
van
VI-
Then the KL-divergence is between two normal distributions so we can write the following, where C2 is term which does not depend on the parameters 0:
D Xo)ll ( x (X,))
( (x Xns n): 3r E)] 2 ( )
DRLlW CT
(fz 12) + C2.
If we follow the implementation from Section 3.4 we reparameterize the model to predict the noise from the unit normal distribution. Then, the matrix L, where Σ L L, can be factorized from the previous equation leaving us with the L -' term in the middle which evaluates to an identity. We found that simplifying the loss to computing the mean squared error between the true noise and the predicted noise co. in Ho et al. (2020), leads to faster evaluation and better results. Therefore, during training we use the loss as described Equation 13.
Note that in the above notation we have set of observations X for times t that we feed into the model Eg to predict a set of noise values E(f), t, whereas, previous works predicted the noise for each data point independently.
A.3 CONTINUOUS DIFFUSION TRANSITION PROBABILITY
Given an SDE in Equation 14 we want to compute the change in the variance Σ,, where s denotes the diffusion time. The derivation is similar to that in Song al. (2021). We start with the Equation 5.51 from Särkkä Solin (2019):
LΣC = E(J(X,, (X, 11) - [X, -a0,,,-)(((XX s)
ds where / is the drift, L is the SDE diffusion term and Q is the diffusion matrix. From here, the only difference to Song et al. (2021) is in the last term; they obtain B(s) while we have full covariance matrix from the stochastic process: B(s)E. Therefore, we only need to slightly modify the result:
dΣ, (() Σ.). -E),)
ds
which will gives us the covariance of the transition probability as in Equation 15. The derivation for the mean is unchanged as our drift term is the same as in Song et al. (2021).
A.4 SAMPLING FROM AN ORNSTEIN- BRNSTEIN-UHLENBBEC PROCESS
In the following, we discuss three different approaches to sampling noise E(-) from an OU process defined by y at time points lo,...,.o.. -M-
1. Modified Wiener. As already mentioned in Section 3.1,
time time-changed and scaled Wiener process: e->4W_2te. Sampling from a Wiener process is straightforward: given set of time increments Âln, ... A/M-1,
sample M points independently from [0,, Ata) and cumulatively sum all the samples. The time changed process first needs to reparameterize the time values. The issue arises when applying the exponential for large which leads to numerical instability. This can be mitigated by re-scaling t.
2. Discretized SDE. A numerically stable approach involves solving the OU SDE in fixed steps. The point at
c(0) is sampled from unit Gaussian. After that, each point is obtained based on the previous, i.c., i-th point e(t,) is calculated as e(t,) ce(t,-i) +
v1 c²2, where e exp(->(t - f--1) and 2 ~ ((0 1). This is an iterative procedure but is quite fast and stable.
3. Multivariate normal. Finally, we treat the process
multivariate normal distribu- tion with mean zero and covariance cov(t, u) = exp(- - -u)). Given a set of time points t it is easy obtain the covariance matrix Σ and its factorization L L. To sample, first
15

---

Under review as a conference paper at ICLR 2023
Dataset
Dim. d
Dom.
Freq.
Time steps
Pred. steps
Exchange
8
R
day
6, 071
30
Solar
137
R*
hour
7, 009
24
Electricity
370
R+
hour
5, 833
24
Table 5: Multivariate dimension, domain, frequency, total training time steps and prediction length properties the training datasets used in the forecasting experiments.
draw ē ~ (N.,,) I) and then Lč. Since our model performs best if it predicts é, we opted for this particular sampling approach. If is not changing, can computed once and the performance impact will be minimal. Also when sampling new realizations, L has to be computed only once, before the sampling loop (see Algorithm 2).
B EXPERIMENTAL DETAILS
B.1 PROBABILISTIC MODELING
B.1.1 DATASETS
The properties of the open datasets used in the forecasting experiment are detailed in Table 5. Ad- ditionally, we generate 6 synthetic datasets, each with 10000 samples, that involve stochastic pro- cesses, dynamical and chaotic systems.
1. CIR (Cox-Ingersoll-Ron SDE) is the stochastic differential equations defined by:
dr a(b a
where we set = 1, b = 1.2, = 0.2 and sample zy NNO,, 1) but only take the positive values, otherwise the /x term is undefined. We solve for I (I,..., 64).
2. Lorenz is chaotic system in three dimensions. It is governed by following equations:
f o x),
pr y 22,
xy 3z,
where p 28, 10, B 2.667, and is sampled 100 times, uniformly on [0, 2]. and
2, y, ~ (O 100Z).
3. Omsttein-Ullenbeck is defined as:
dr = (μt fr)dt + adW,
with = 0.02, 0 0.1 and a 0.4. We sample time the same way as for CIR.
4. Predator-prey a 2D dynamical system defined with an ODE:
4 = 2/3r - 2/3rr,,
ý = xz y/-
5. Sine dataset is generated mixture of 5 random sine waves a sin + c), where (3,, 1). N (0, 0.25), and e A0 1).
6. Sink is again a dynamical system, governed by:
da
4
10
dt
3
x,
with xo (),
16

---

Under review as a conference paper at ICLR 2023
B.1.2 CTFP
We implement continuous-time flow process (Deng et al., 2020) which is a normalizing flow model for stochastic processes. That is, there is predefined base distribution p(z) and series of invertible transformations such that we can generate samples a = f(z), and evaluate the density in closed- form by computing normalizing flows, see Kobyzev et al. (2020). The novel idea in CTFP is to change the base density to stochastic process, i.e., a Wiener process, obtain the distribution over the functions, similar to our work. In our case, we do not use invertible functions but learn to inverse the noising process, and additionally, we add noise at multiple levels instead only in the beginning. In the experiments we define a CTFP model as a 12-layer real NVP architecture (Dinh et al., 2017) with 2 hidden layers in each layer's 's MLP.
f- (ax) and using the change of variables formula. For more details on
B.1.3 LATENT ODE
Latent ODE is variational autoencoder architecture, with an encoder that represents the complete time series as a single vector following g(=), and decoder that produces the samples at observation times f4, (ft) = f(2). ~ q(z). The final step is projection to data space (f;) + ztt,). The key idea is to use the neural ordinary differential equation (Chen et al., 2018) to define the evolution of the latent variable z--), thus, have probabilistic model of the function. This is different to our approach as it models the function in a latent space, with a single source of randomness at the beginning of the time series. That is, the random value is sampled at t 0 and the time series is determined from there onward, whereas our method samples random values on the whole interval [0, 7) and does so multiple times (for N diffusion steps) until we get the new realization. In the experiments we use a two layer neural network for the neural ODE, and a another two layer network for projection to the data space.
B.1.4 OUR MODELS
We use two models, one is simple feedforward network, and the second is an RNN-based model. The model takes in the time series X, times of the observations and the diffusion step n or dif- fusion time s. The output is the same size as X. The feedforward model embeds the time and the diffusion step with positional encoding (Vaswani et al., 2017) and passes it together with X through the multilayer neural network. Here, there is no interaction between the points along the time dimension. The model, however, has the capacity to learn transformation based on time of ob- servation. The second model is RNN based, that is, we pass the same concatenated input as before to a 2-layer bidirectional GRU (Chung et al., 2014) and use a single linear layer to project to the output dimension. Table 6 shows that it is important to have interactions in the time dimension, regardless of the noise source, because otherwise we only learn the marginal distribution and the quality of the samples suffers.
CIR
Lorenz
OU
Predator-prey
Sinc
Sink
RNN-based model
CSPD DSPD
Gauss
0.5245=-0.0252
0.512=000012
0.568=0.051
0.5275=0.0383
0.5565=+0.0353
0.526=000085
GP
0.5115++0.082
0.5135+0.0288
0.5055+0.0458
0.5855-0.0219
0.5255+000099
0.513+0.0103
OU
0.514=0.0737
0.6095 0964
0.5605+000581
0.5865+:0.053
0.507++0.11
0.6255-10, 1672
Gauss
0.644=0.0373
0.5015
0..10510.0113
0.548:10.051
0.61110.0516
0.5495:10.0313
GP
0.57951:0.041
0.674_10.079
0.502510.0622
0.6071:00538
0.5575110.336
0.534510..020
OU
0.
0.715:10.0884
0.5255+00.011
0.5835=0.0723
0,556
0.5795
Feedforward model
CSPD DSPD
Gauss
0.624_0.0438
0.713-10.1798
0.527510.0371
1.0-0.0
0.7875:10.0585
0.96951100032
GP
0.558=0.0611
0.89410.212
0.553520.1152
0.7565=0.1362
0.73510.2146
0.784:0.2281
OU
1.0+::00
1.0+0.0
1.0=0.0
1.0=0.0
1.0+0.0
1.0-10.0
Gauss
0.537=0.0458
0.959=1008808
0.5155=0.0165
0.9995-10.001
0.6335:40.0765
0.9095 1306
GP
0.645=0.108
1.0++0.0
0.507+000264
0.894-0.212
0.894++0.212
0.88+00088
OU
0.984:10.032
1.0+0.0
0.9905:10.119
1.0=0.0
1.0:0.0
1.0+0..0
Table 6: Accuracy of the discriminator trained on samples from a diffusion model. Values around 0.5 indicate the discriminative model cannot distinguish the model samples and real data, Values closer to 1 indicate the generative model is not capturing the data distribution.
17

---

Under review as a conference paper at ICLR 2023
B.2
NEURAL PROCESS
B.2.1 DATASET
We sample points from Gaussian process to obtain single time series. In the end, we have 8000 time series and 2000 test time series. We sample the number of time points from Poisson distribution with À 10 but restrict the values to always be above 5 and below 50. The time points are sampled uniformly on [0, 1]. The observations are sampled from a multivariate normal distribution with mean zero and covariance obtained from an RBF kernel. The value in the kernel is uniformly sampled in [0.01, 0.05] for each time series independently. Half of the sampled points are treated as unobserved while the rest are used as context in the model.
B.2.2 MODEL
The denoising model takes in X (observed points) as conditioning variable and x (target points) as the noisy input. We first run a learnable RBF kernel k(tA,t* t to obtain a similarity matrix K between the observed and unobserved time points. We project X with neural network by transforming each point independently to obtain Z, and then obtain the latent variable of the same time dimension size as X B by multiplying K and Z. We then use Z as conditioning vector and add it to projected X B transform with a multilayer network, and obtain the output.
B.2.3 ADDITIONAL RESULTS
We test the hypothesis that using stochastic process with similar properties the data will lead to better performance. The difference to the neural process setup in Section 6 is that we fix the synthetic GP always have a 0.05. As can be seen from Figure 6, the marginal distribution will be equal regardless of which process and which keree parameter we use. On the other hand, when we look at path probability p(X), we notice better results when the noise process matches data properties (as was also shown in Table and 6). That means, while our model can reverse the process well, the qualitative properties of the sampled curves will be different. In particular, the curves will be rougher with increasing y in OU and smoother with increasing o in GP.
B.3 CSDI IMPUTATION
The imputation experiment presented in Sections 4.3 and uses the original CSDI model (Tashiro et al., 2021) and only changes the noise to include the stochastic process source. In this case, the time points at which we evaluate the stochastic process are regular which does not reflect the true nature of the Physionet dataset. Here, we change the setup such that the measurements keep the actual time that has passed instead of rounding to the nearest hour. This is still in favour of the original paper as only takes one measurement per hour and discards other they are present. The model from Tashiro et al. (2021) remains the same and we replace the independent normal noise with the GP noise with a E {0.005, 0.01, 0.02].
We run each experimental setup 10 times with different data maskings (see Tashiro et al. (2021) for more details) and report the results in Table 7. We perform the Wilcoxon one-sided signed- rank test (Conover, 1999) and reject the null hypothesis that the expected RMSE values are the same when p < 0.05. As we can see, higher values of a produce better results which makes sense since 0.005 is, informally, closer to independent Gaussian sampling than has stronger temporal dependency between the samples. We suspect 10%-missing case does not produce significant results due to noise. Using higher does not further improve the results.
0.02, which
Missingness:
10%
50%
90%
Metrics:
RMSE
p-value
RMSE
p-value
RMSE
p-value
CSDI (baseline)
0.603:10.274
0.658+0.060
0.839+00003
0.005
0.541=0..05
0.125
0.6471:0.09
0.116
0.824+0.032
0.188
0.01
0.575-10.15
0.125
0.640=10.050
0.001
0.823+0.028
0.032
0.02
0.515-40.039
0.326
0.636+10.050
0.001
0.811+0.032
0.001
Table 7: Imputation results averaged over 10 runs and p-value of Wilcoxon one-sided test.
18

---

Under review as a conference paper at ICLR 2023
0.002
0.02
0.2
2
GP
Context Target
-2
X
Sample
Sample #2
Sample #3
-2
0.002
0.02
0.2
2
OU Priors
Context Target
a
Sample
-2
X
Sample #2
Sample #3
-2
Figure (Top) Neural process with Gaussian process diffusion, fitted on GP synthetic data. Columns correspond to different values of the kernel parameter a. First shows samples from the GP prior. As we can see, the higher the value the smoother the process is. This also reflected in the samples from the model. (Bottom) Same but for Omsstein Uhlenbeck process, however, creasing the kernel parameter now decreases the smoothness. All of the models perfectly capture the marginal distribution.
Omstein-Uhlenbeck
19


Reviews:
[{"review_id": "xV1JqgBvvA", "reviewer": "Reviewer_8xdq", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes to model time series data as discretized observations from an underlying continuous stochastic process using a diffusion framework. The main contribution of the paper is to extend the method proposed in [1] and [2] to the case where a stochastic noise process is used for the forward process. Based on the proposed method, the paper also developed approaches for multivariate probabilistic forecasting and imputation tasks.  The effectiveness of the proposed method is demonstrated through synthetic and real-world datasets.\n\n\nReference:\n\n[1] Ho, Jonathan, Ajay Jain, and Pieter Abbeel. \"Denoising diffusion probabilistic models.\" Advances in Neural Information Processing Systems 33 (2020): 6840-6851.\n\n[2] Song, Yang, Jascha Sohl-Dickstein, Diederik P. Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. \"Score-Based Generative Modeling through Stochastic Differential Equations.\" In International Conference on Learning Representations. 2020.", "summary_of_the_review": "The paper is very well written and easy to follow. I am not sure about the level of novelty of the paper.", "strength_and_weaknesses": "Strength: the paper is well-written and easy to follow. The proposed methodology appears to be reasonable.\nWeakness: I am not so sure about the novelty of the proposed methodology.\n\n1. Although the authors repeatedly emphasized about the denoising approach is designed for stochastic processes, the loss function (13) and (8) with score function (16) are still vector-based loss functions. It would make more sense to generate the noise process from a stochastic process instead of a multivariate normal distribution. Even if one needs to discretize the stochastic process, the dimension of the noise process should be higher than the original observed data. Under the current setting, it is really difficult to tell how much difference is there between the proposed method and those in[1] and [2], from the implementation point of view.\n\n2. When considering the stochastic process, it is not as simple as \"Gaussian+Gaussian\" is still Gaussian. Even Gaussian processes with different covariance kernels have different properties such as smoothness and differentiability. This brings another question: what noise processes to use in practice? The paper proposes two candidates: a Gaussian process with a radial basis kernel and the Ornstein-Uhlenbeck process. Any guidelines on how to choose? Do parameters in the kernel function need to be estimated? Is it possible that a noise process with similar smoothness/differentiability/strength-of-dependence properties to the original data will yield better performance? These issues need to be investigated.\n\n\nReference:\n\n[1] Ho, Jonathan, Ajay Jain, and Pieter Abbeel. \"Denoising diffusion probabilistic models.\" Advances in Neural Information Processing Systems 33 (2020): 6840-6851.\n\n[2] Song, Yang, Jascha Sohl-Dickstein, Diederik P. Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. \"Score-Based Generative Modeling through Stochastic Differential Equations.\" In International Conference on Learning Representations. 2020.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "Uu1GUWSQpr", "reviewer": "Reviewer_c2e3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a diffusion model focusing on time series data where the diffusion function takes the predefined matrix from Cholesky decompositions of Gaussian processes. Experiments show some interesting results on time series predictions and imputations.\n", "summary_of_the_review": "Although the paper presents a nice idea, I feel the technical contribution of the paper is limited as stochastic process for noise part is not different than matrix-valued diffusion functions for diffusion models.", "strength_and_weaknesses": "Strength:\n- The paper presents an interesting idea as the covariance is introduced in diffusion processes to express some degree of correlation between temporal data points.\n- For prediction tasks, the method is able to produce estimations at multiple time points unlike the previous work [Rasul et al 2021a] which follows autoregressive approaches. The key contribution here is the temporal correlation defined by covariance functions..\n\nWeakness:\n- The paper explains the approach as replacing the noise vector as a stochastic process. In fact, in a much simpler view, this way is the same with the diffusion process that just has a diffusion function as a matrix $L$ from Cholesky decomposition. In light of these observations, the contribution of formulating discrete and continuous versions of diffusion models is not significant. \n- The justification for the loss function in Appendix A.2 is subtle without evidence to support the claim. In my opinion, $\\Sigma^{-1}$ plays an important role in the curvature of loss functions, containing information of temporal dependencies.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "Q4CawzzQrGt", "reviewer": "Reviewer_ienA", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper describes a novel diffusion model extending the concept to modelling irregularly sampled time series by utilizing correlated noise obtained from noise processes to produce continuous samples. The paper provide an elegant framework to combine forecasting, and imputation tasks.", "summary_of_the_review": "The paper is quite well written and the method is novel. A little bit more detailed comparison would benefit the paper, but in general I find the paper a valuable addition to state of the art.", "strength_and_weaknesses": "One of the main advantage of the method is the elegant framework to combine all kinds of inference tasks for possibly sporadically observed time series. Forecasting (prediction in temporal causal direction), smoothing (prediction in both direction,) and imputation tasks can be formalized as conditional generation tasks in the framework.\n\nOne of the weakness is that little guidance is provided on how to select the noise process, and that not many benchmark method is provided, see later.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "5Eyu86lZUN", "reviewer": "Reviewer_JDSC", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors proposes to learn a diffusion based generative model to model temporal data. The model is able to handle They test their model on various tasks such as forecasting and interpolation.", "summary_of_the_review": "Overall I think this is a good paper tackling a well motivated and realistic task with diffusion models in the setting where temporal data are not sampled regularly.", "strength_and_weaknesses": "Strength:\n- The abstract and introduction give a nice overview and motivation for the problem.\n- Overall the paper is clear.\n- The results are good and evaluation is somehow reasonable.\n\nWeaknesses:\n- I can't find the meaning of the bold format in your tables.\n- You should run a statistical test to compare the different methods.\n- I am not sure the editors' names are needed in the bibliography, removing them would ease the reading of it", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject