Title: Predict, Refine, Synthesize: Self-Guiding Diffusion Models for Probabilistic Time Series Forecasting
Venue: NeurIPS 2023
Accepted: yes
Keywords: diffusion models

time series forecasting

generative modeling

deep learning
Abstract: Diffusion models have achieved state-of-the-art performance in generative modeling tasks across various domains. Prior works on time series diffusion models have primarily focused on developing conditional models tailored to specific forecasting or imputation tasks. In this work, we explore the potential of task-agnostic, unconditional diffusion models for several time series applications. We propose TSDiff, an unconditionally-trained diffusion model for time series. Our proposed self-guidance mechanism enables conditioning TSDiff for downstream tasks during inference, without requiring auxiliary networks or altering the training procedure. We demonstrate the effectiveness of our method on three different time series tasks: forecasting, refinement, and synthetic data generation. First, we show that TSDiff is competitive with several task-specific conditional forecasting methods (*predict*). Second, we leverage the learned implicit probability density of TSDiff to iteratively refine the predictions of base forecasters with reduced computational overhead over reverse diffusion (*refine*). Notably, the generative performance of the model remains intact — downstream forecasters trained on synthetic samples from TSDiff outperform forecasters that are trained on samples from other state-of-the-art generative time series models, occasionally even outperforming models trained on real data (*synthesize*).

Our code is available at https://github.com/amazon-science/unconditional-time-series-diffusion

Predict, Refine, Synthesize: Self-Guiding Diffusion Models for Probabilistic Time Series Forecasting

Abstract

Diffusion models have achieved state-of-the-art performance in generative modeling tasks across various domains. Prior works on time series diffusion models have primarily focused on developing conditional models tailored to specific forecasting or imputation tasks. In this work, we explore the potential of taskagnostic, unconditional diffusion models for several time series applications. We propose TSDiff, an unconditionally-trained diffusion model for time series. Our proposed self-guidance mechanism enables conditioning TSDiff for downstream tasks during inference, without requiring auxiliary networks or altering the training procedure. We demonstrate the effectiveness of our method on three different time series tasks: forecasting, refinement, and synthetic data generation. First, we show that TSDiff is competitive with several task-specific conditional forecasting methods ( predict ). Second, we leverage the learned implicit probability density of TSDiff to iteratively refine the predictions of base forecasters with reduced computational overhead over reverse diffusion ( refine ). Notably, the generative performance of the model remains intact - downstream forecasters trained on synthetic samples from TSDiff outperform forecasters that are trained on samples from other state-of-the-art generative time series models, occasionally even outperforming models trained on real data ( synthesize ).

1 Introduction

Time series forecasting informs key business decisions [27], for example in finance [47], renewable energy [56], and healthcare [7]. Recently, deep learning-based models have been successfully applied to the problem of time series forecasting [46, 31, 5]. Instantiations of these methods use, among other techniques, autoregressive modeling [46, 42], sequence-to-sequence modeling [54], and normalizing flows [43, 8]. These techniques view forecasting as the problem of conditional generative modeling: generate the future, conditioned on the past.

Diffusion models [48, 20] have shown outstanding performance on generative tasks across various domains [10, 28, 30, 36, 44] and have quickly become the framework of choice for generative modeling. Recent studies used conditional diffusion models for time series forecasting and imputation tasks [44, 52, 1, 6]. However, these models are task specific, i.e., their applicability is limited to the specific imputation or forecasting task they have been trained on. Consequently, they also forego the desirable unconditional generative capabilities of diffusion models.

Predict

:

:

Train

Downstream

model

Synthesize

Predict

This raises a natural research question: Can we address multiple (even conditional) downstream tasks with an unconditional diffusion model? Specifically, we investigate the usability of task-agnostic unconditional diffusion models for forecasting tasks. We introduce TSDiff, an unconditional diffusion model for time series, and propose two inference schemes to utilize the model for forecasting. Building upon recent work on guided diffusion models [10, 19], we propose a self-guidance mechanism that enables conditioning the model during inference, without requiring auxiliary networks. This makes the unconditional model amenable to arbitrary forecasting (and imputation) tasks that are conditional in nature 3 . We conducted comprehensive experiments demonstrating that our self-guidance approach is competitive against task-specific models on several datasets and across multiple forecasting scenarios, without requiring conditional training. Additionally, we propose a method to iteratively refine predictions of base forecasters with reduced computational overhead compared to reverse diffusion by interpreting the implicit probability density learned by TSDiff as an energy-based prior. Finally, we show that the generative capabilities of TSDiff remain intact. We train multiple downstream forecasters on synthetic samples from TSDiff and show that forecasters trained on samples from TSDiff outperform those trained on samples from variational autoencoders [9] and generative adversarial networks [57], sometimes even outperforming models trained on real samples. To quantify the generative performance, we introduce the Linear Predictive Score (LPS) which we define as the test forecast performance of a linear ridge regression model trained on synthetic samples. TSDiff significantly outperforms competing generative models in terms of the LPS on several benchmark datasets. Fig. 1 highlights the three use cases of TSDiff: predict, refine, and synthesize.

In summary, our key contributions are:

TSDiff, an unconditionally trained diffusion model for time series and a mechanism to condition TSDiff during inference for arbitrary forecasting tasks (observation self-guidance);

An iterative scheme to refine predictions from base forecasters by leveraging the implicit probability density learned by TSDiff;

Experiments on multiple benchmark datasets and forecasting scenarios demonstrating that observation self-guidance is competitive against task-specific conditional baselines;

Linear Predictive Score, a metric to evaluate the predictive quality of synthetic samples, and experiments demonstrating that TSDiff generates realistic samples that outperform competing generative models in terms of their predictive quality.

The rest of the paper is organized as follows. Sec. 2 introduces the relevant background on denoising diffusion probabilistic models (DDPMs) and diffusion guidance. In Sec. 3, we present TSDiff, an unconditional diffusion model for time series, and propose two inference schemes to utilize the model for forecasting tasks. Sec. 5 discusses the related work on diffusion models for time series and diffusion guidance. Our empirical results are presented in Sec. 4. We conclude with a summary of our findings, the limitations of our proposals, and their potential resolutions in Sec. 6.

2 Background

2.1 Denoising Diffusion Probabilistic Models

Diffusion models [48, 20] provide a framework for modeling the data generative process as a discretetime diffusion process. They are latent variable models of the form p θ ( y ) = ∫ p θ ( y , x 1: T ) d x 1: T , where y ∼ q ( y ) is the true underlying distribution. The latent variables { x 1 , . . . , x T } are generated by a fixed Markov process with Gaussian transitions, often referred to as the forward process ,

where β t is the variance of the additive noise, y is the observed datapoint, and q ( x T ) ≈ N ( 0 , I ) . The fixed Gaussian forward process allows direct sampling from q ( x t | y ) ,

where α t = 1 -β t , ¯ α t = ∏ t i =1 α i , and ϵ ∼ N ( 0 , I ) . On the other hand, the reverse diffusion (generative) process is formulated as,

where p ( x T ) ∼ N ( 0 , I ) and σ t = 1 -¯ α t -1 1 -¯ α t β t . The model is trained to approximate the true reverse process q ( x t -1 | x t ) by maximizing an approximation of the evidence lower bound (ELBO) of the log-likelihood. Specifically, µ θ is parameterized using a denoising network, ϵ θ ,

which is trained to predict the sampled noise ( ϵ in Eq. 2) using the simplified objective function [20],

By suitably adjusting the denoising neural network to incorporate the conditioning input, this objective can be employed to train both unconditional and conditional models.

2.2 Diffusion Guidance

Classifier guidance repurposes unconditionally-trained image diffusion models for class-conditional image generation [10]. The key idea constitutes decomposing the class-conditional score function using the Bayes rule,

and employing an auxiliary classifier to estimate ∇ x t log p ( c | x t ) . Specifically, the following modified reverse diffusion process (Eq. 3) allows sampling from the class-conditional distribution,

where s is a scale parameter controlling the strength of the guidance.

3 TSDiff: an Unconditional Diffusion Model for Time Series

In this section, we present our main contributions: TSDiff, an unconditional diffusion model designed for time series, along with two inference schemes that leverage the model for downstream forecasting tasks. We begin by outlining the problem setup and providing a concise overview of our network architecture. Subsequently, we delve into our first scheme - observation self-guidance - which enables conditioning reverse diffusion on arbitrary observed timesteps during inference . Secondly, we present a technique to iteratively refine predictions of arbitrary base forecasters by utilizing the implicit probability density learned by TSDiff as a prior.

Problem Statement. Let y ∈ R L be a time series of length L . Denote obs ⊂ { 1 , . . . , L } as the set of observed timesteps and ta as its complement set of target timesteps. Our goal is to recover the complete time series y , given the observed subsequence y obs which may or may not be contiguous. Formally, this involves modeling the conditional distribution p θ ( y ta | y obs ) . This general setup subsumes forecasting tasks, with or without missing values, as special cases. We seek to train a single unconditional generative model, p θ ( y ) , and condition it during inference to draw samples from arbitrary distributions of interest, p θ ( y ta | y obs ) .

Generative Model Architecture.

We begin with modeling the marginal probability, p θ ( y ) , via a diffusion model, referred to as TSDiff, parameterized by θ . The architecture of TSDiff is depicted in Fig. 2 and is based on SSSD [1] which is a modification of DiffWave [28] employing S4 layers [18]. TSDiff is designed to handle univariate sequences of length L . To incorporate historical information beyond L timesteps without increasing L , we append lagged time series along the channel dimension. This results in a noisy input x t ∈ R L × C (see Eq. 2) to the diffusion model, where C -1 is the number of lags. The S4 layers oper-

Residual Blocks

TSDiff

Conv1x1

TSDiff Architecture

Conv1x1

Conv1x1

Conv1x1

Conv1x1

Conv1x1

Conv1x1

: Element-wise add.

: tanh

: Element-wise mult.

: ReLU

: Swish

Observation Self-Guidance

Conv1x1

ate on the time dimension whereas the Conv1x1 layers operate on the channel dimension, facilitating information flow along both dimensions. As typical for unconditional diffusion models, the output dimensions of TSDiff match the input dimensions. Note that while we focus on univariate time series in this work, TSDiff can be modified to handle multivariate time series by incorporating additional layers, e.g., a transformer layer, operating across the feature dimensions after the S4 layer.

In the following, we discuss two approaches to condition the generative model, p θ ( y ) , during inference, enabling us to draw samples from p θ ( y ta | y obs ) .

3.1 Observation Self-Guidance

Let t ≥ 0 be an arbitrary diffusion step. Applying Bayes' rule, we have,

which yields the following relation between the conditional and marginal score functions,

Given access to the guidance distribution, p θ ( y obs | x t ) , we can draw samples from p θ ( y ta | y obs ) using guided reverse diffusion, akin to Eq. (7),

The scale parameter s controls how strongly the observations, y obs , and the corresponding timesteps in the diffused time series, y , align. Unlike Dhariwal and Nichol [10], we do not have access to auxiliary guidance networks. In the following, we propose two variants of a self-guidance mechanism that utilizes the same diffusion model to parameterize the guidance distribution. The main intuition behind self-guidance is that a model designed for complete sequences should reasonably approximate partial sequences. A pseudo-code of the observation self-guidance is given in App. A.2.

Mean Square Self-Guidance. We model p θ ( y obs | x t ) as a multivariate Gaussian distribution,

where f θ is a function approximating y , given the noisy time series x t . We can reuse the denoising network ϵ θ to estimate y as

which follows by rearranging Eq. (2) with ϵ = ϵ θ ( x t , t ) , as shown by Song et al. [49]. This one-step denoising serves as a cost-effective approximation of the model for the observed time series and provides the requisite guidance term in the form of the score function, ∇ x t log p θ ( y obs | x t ) , which can be computed by automatic differentiation. Consequently, our self-guidance approach requires no auxiliary networks or changes to the training procedure. Applying the logarithm to Eq. (11) and dropping constant terms yields the mean squared error (MSE) loss on the observed part of the time series, hence we named this technique mean square self-guidance.

Quantile Self-Guidance. Probabilistic forecasts are often evaluated using quantile-based metrics such as the continuous ranked probability score (CRPS) [15]. While the MSE only quantifies the average quadratic deviation from the mean, the CRPS takes all quantiles of the distribution into account by integrating the quantile loss (also known as the pinball loss) from 0 to 1. This motivated us to substitute the Gaussian distribution with the asymmetric Laplace distribution that has been studied in the context of Bayesian quantile regression [58]. The probability density function of the asymmetric Laplace distribution is given by,

where Z is a normalization constant, b > 0 a scale parameter, and κ ∈ (0 , 1) an asymmetry parameter. Setting b = 1 , the log density yields the quantile loss with the score function,

with κ specifying the quantile level. By plugging Eq. (14) into Eq. (9), the reverse diffusion can be guided towards a specific quantile level κ . In practice, we use multiple evenly spaced quantile levels in (0 , 1) , based on the number of samples in the forecast. Intuitively, we expect quantile self-guidance to generate more diverse predictions by better representing the cumulative distribution function.

3.2 Prediction Refinement

In the previous section, we discussed a technique enabling the unconditional model to generate predictions by employing diffusion guidance. In this section, we will discuss another approach repurposing the model to refine predictions of base forecasters. Our approach is completely agnostic to the type of base forecaster and only assumes access to forecasts generated by them. The initial forecasts are iteratively refined using the implicit density learned by the diffusion model which serves as a prior. Unlike reverse diffusion which requires sequential sampling of all latent variables, refinement is performed directly in the data space. This provides a trade-off between quality and computational overhead, making it an economical alternative when the number of refinement iterations is less than the number of diffusion steps. Furthermore, in certain industrial forecasting scenarios, one has access to a complex production forecasting system of black-box nature. In these cases, refinement presents a cost-effective solution that enhances forecast accuracy post hoc, without modifying the core forecasting process - a change that could potentially be a lengthy procedure.

In the following, we present two interpretations of refinement as (a) sampling from an energy function, and (b) maximizing the likelihood to find the most likely sequence.

Energy-Based Sampling. Recall that our goal is to draw samples from the distribution p ( y ta | y obs ) . Let g be an arbitrary base forecaster and g ( y obs ) be a sample forecast from g which serves as an initial guess of a sample from p ( y ta | y obs ) . To improve this initial guess, we formulate refinement as the problem of sampling from the regularized energy-based model (EBM),

where ˜ y is the time series obtained upon combining y obs and g ( y obs ) , and R is a regularizer such as the MSE loss or the quantile loss. We designed the energy function such that low energy is assigned to samples that are likely under the diffusion model, p θ ( y ) , and also close to ˜ y , ensured by the diffusion log-likelihood and the regularizer in the energy function, respectively. The Lagrange

multiplier, λ , may be tuned to control the strength of regularization; however, we set it to 1 in our experiments for simplicity.

We use overdamped Langevin Monte Carlo (LMC) [50] to sample from this EBM. y (0) is initialized to ˜ y and iteratively refined as,

where η and γ are the step size and noise scale, respectively.

Note that in contrast to observation self-guidance, we directly refine the time series in the data space and require an initial forecast from a base forecaster. However, similar to observation self-guidance, this approach does not require any modifications to the training procedure and can be applied to any trained diffusion model. A pseudo-code of the energy-based refinement is provided in App. A.2.

Maximizing the Likelihood. The decomposition in Eq. (15) can also be interpreted as a regularized optimization problem with the goal of finding the most likely time series that satisfies certain constraints on the observed timesteps. Concretely, it translates into,

which can be optimized using gradient descent and is a special case of Eq. (16) with γ = 0 . Given the non-convex nature of this objective, convergence to the global optimum is not guaranteed. Therefore, we expect the initial value, y (0) , to influence the resulting time series, which can also be observed in the experiment results (see Table 3).

Approximation of log p θ ( y ) . To approximate the log-likelihood log p θ ( y ) we can utilize the objective used to train diffusion models,

which is a simplification of the ELBO [20]. However, a good approximation requires sampling several diffusion steps ( t ) incurring computational overhead and slowing down inference. To speed up inference, we propose to approximate Eq. (18) using only a single diffusion step. Instead of randomly sampling t , we use the representative step , τ , which improved refinement stability in our experiments. The representative step corresponds to the diffusion step that best approximates Eq. (18), i.e.,

The representative step is computed only once per dataset. It can be efficiently computed post training by computing the loss at every diffusion step on a randomly-sampled batch of training datapoints and then finding the diffusion step closest to the average loss. Alternatively, we can keep a running average loss for each t and compute the representative step using these running averages. This can be done efficiently because the loss used to obtain τ is the same as the training loss for the diffusion model.

4 Experiments

In this section, we present empirical results on several real-world datasets. Our goal is to investigate whether unconditional time series diffusion models can be employed for downstream tasks that typically require conditional models. Concretely, we tested if (a) the self-guidance mechanism in TSDiff can generate probabilistic forecasts (also in the presence of missing values), (b) the implicit probability density learned by TSDiff can be leveraged to refine the predictions of base forecasters, and (c) the synthetic samples generated by TSDiff are adequate for training downstream forecasters. 4

Datasets and Evaluation. We conducted experiments on eight univariate time series datasets from different domains, available in GluonTS [2] - Solar [29], Electricity [11], Traffic [11], Exchange [29], M4-Hourly [37], UberTLC-Hourly [13], KDDCup [16], and Wikipedia [14]. We evaluated the quality of probabilistic forecasts using the continuous ranked probability score (CRPS) [15]. We approximated the CRPS by the normalized average quantile loss using 100 sample paths, and report means and standard deviations over three independent runs (see App. B for details on the datasets and evaluation metric).

4.1 Forecasting using Observation Self-Guidance

We tested the forecasting performance of the two proposed variants of observation self-guidance, mean square guidance (TSDiff-MS) and quantile guidance (TSDiff-Q), against several forecasting baselines. We included Seasonal Naive, ARIMA, ETS, and a Linear (ridge) regression model from the statistical literature [21]. Additionally, we compared against deep learning models that represent various architectural paradigms such as the RNN-based DeepAR [46], the CNN-based MQ-CNN [54], the state space model-based DeepState [42], and the self-attentionbased Transformer [53] and TFT [32] models. We also compared against two conditional diffusion models, CSDI [52] and TSDiff-Cond, a conditional version of TSDiff closely related to SSSD [1] (see App. B.4 for a more in-depth discussion on the baselines). Note that we did not seek to obtain state-of-the-art forecasting results on the datasets studied but to demonstrate the efficacy of unconditional diffusion models against task-specific conditional models.

Table 1 shows that TSDiff-Q is competitive with state-of-the-art conditional models, but does not require task-specific training, achieving the lowest or second-lowest CRPS on 5/8 datasets. We also observe that the choice of guidance distribution is critical. While Gaussian (TSDiffMS) yields reasonable results, using the asymmetric Laplace distribution (TSDiff-Q) lowers the CRPS further. We hypothesize that taking the different quantile levels into account during guidance improves the results on the quantilebased evaluation metric (CRPS). Fig. 3 shows example forecasts from TSDiff-Q. Note that as self-guidance only imposes a soft-constraint on the observed timesteps, the resultant diffused values are not guaranteed to match the observations for these timesteps. The alignment between predictions and observations can be controlled via the scale parameter, s (see Eq. 9). In

01

Sep

02

27

28

29

30

31

100

150

200

Electricity

Ground Truth

Median Forecast

Forecast Horizon

90% Interval

25

Mar

26

27

28

29

30

31

0

100

200

KDDCup

Oct

Nov

Dec

Jan

2013

Feb

Mar

Apr

May

1.00

1.05

Exchange

practice we observed that the diffused values were close to the observations for our selected values of s , as shown in Fig. 5 (Appendix).

Forecasting with Missing Values. We evaluated the forecasting performance with missing values in the historical context during inference to demonstrate the flexibility of self-guidance. Specifically, we tested three scenarios: (a) random missing (RM), (b) blackout missing (i.e., a window with consecutive missing values) at the beginning of the context window (BM-B), and (c) blackout missing at the end of the context window (BM-E). We removed a segment equal to the context window from the end of the training time series to ensure that the model is not trained on this section. During inference, we masked 50% of the timesteps from this held-out context window for each scenario

while the forecast horizon remained unchanged. We trained the unconditional model on complete sequences only once per dataset, without making any modifications to the objective function. As a result, the three scenarios described above are inference only for TSDiff. For comparison, we trained conditional models (TSDiff-Cond) specifically on the target missingness scenarios. These models were trained to predict the missing values in the input time series, where some timesteps were masked according to the target missingness scenario (see App. B.3 for a detailed discussion on the missing values experiment setup).

The results in Table 2 show that TSDiff-Q performs competitively against task-specific conditional models, demonstrating its robustness with respect to missing values during inference. This makes TSDiff-Q particularly useful in real-world applications where missing values are common but the missingness scenario is not known in advance. It is noteworthy that we did not tune the guidance scale for each missingness scenario but used the same value as in the previous standard forecasting setup. We expect the results to further improve if the scale is carefully updated based on the missingness ratio and scenario.

4.2 Refining Predictions of Base Forecasters

We evaluated the quality of the implicit probability density learned by TSDiff by refining predictions from base forecasters, as described in Sec. 3.2. We considered two point forecasters (Seasonal Naive and Linear) and two probabilistic forecasters (DeepAR and Transformer) as base models. We refine the forecasts generated from these base models for 20 steps, as described in Algorithm 2 (see App. B.3 for a discussion on the number of refinement steps). Similar to self-guidance, we experimented with the Gaussian and asymmetric Laplace negative log-likelihoods as regularizers, R , for both energy (denoted by LMC) and maximum likelihood (denoted by ML)-based refinement.

The CRPS scores before (denoted as 'Base') and after refinement are reported in Table 3. At least one refinement method reduces the CRPS of the base forecast on every dataset. Energy-based refinement generally performs better than ML-based refinement for point forecasters. We hypothesize that this is

because the additive noise in LMC incentivizes exploration of the energy landscape which improves the 'spread' of the initial point forecast. In contrast, ML-Q refinement outperforms energy-based refinement on probabilistic base models suggesting that sampling might not be necessary for these models due to the probabilistic nature of the initial forecasts.

Refinement provides a trade-off; while it has a lower computational overhead than self-guidance, its performance strongly depends on the chosen base forecaster. It substantially improves the performance of simple point forecasters (e.g., Seasonal Naive) and yields improvements on stronger probabilistic forecasters (e.g., Transformer) as well.

4.3 Training Downstream Models using Synthetic Data

Finally, we evaluated the quality of generated samples through the forecasting performance of downstream models trained on these synthetic samples. Several metrics have previously been proposed to evaluate the quality of synthetic samples [12, 57, 26, 23]. Of these, we primarily focused on predictive metrics that involve training a downstream model on synthetic samples and evaluating its performance on real data (i.e., the train on synthetic-test on real setup). In the absence of a standard downstream model (such as the Inception network [51] in the case of images), prior works have proposed different network architectures for the downstream model. This makes the results sensitive to architecture choice, random initialization, and even the choice of deep learning library. Furthermore, training a downstream model introduces additional overhead per metric computation. We propose the Linear Predictive Score (LPS) to address these issues. We define the LPS as the test CRPS of a linear (ridge) regression model trained on the synthetic samples. The ridge regression model is a simple, standard model available in standard machine learning libraries (e.g., scikit-learn ) that can effectively gauge the predictive quality of synthetic samples. Moreover, a ridge regression model is cheap to fit - it can be solved in closed-form, eliminating variance introduced by initialization and training.

We compared samples generated by TSDiff against those from TimeGAN [57] and TimeVAE [9], two time series generative models from alternative frameworks. In addition to the linear model, we trained two strong downstream forecasters (DeepAR and Transformer) on synthetic samples from each generative model. The test CRPS of these forecasters is presented in Table 4. TSDiff's samples significantly outperform TimeVAE and TimeGAN in terms of the LPS showcasing their excellent predictive quality with respect to a simple (linear) forecaster. On the stronger DeepAR and Transformer forecasters, TSDiff outperforms the baselines on most of the datasets. Moreover, the scores obtained by TSDiff are reasonable when compared to those attained by downstream forecasters trained on real samples. Notably, these forecasters trained on real data had the advantage of accessing time features and lags that extend far beyond the sequence length modeled by TSDiff. These results serve as evidence that TSDiff effectively captures crucial patterns within time series datasets and generates realistic samples (see App. C for a qualitative comparison between real time series and those generated by TSDiff).

5 Related Work

Diffusion Models. Diffusion models were originally proposed for image synthesis [48, 20, 10], but have since been applied to other domains such as audio synthesis [28], protein modeling [3], and graph modeling [24]. Prior works have also studied image inpainting using diffusion models [35, 45, 25] which is a problem related to time series imputation. Similar to the maximum likelihood variant of our refinement approach, Graikos et al. [17] showed the utility of diffusion models as plug-and-play priors. The architecture of our proposed model, TSDiff, is based on a modification [1] of DiffWave [28] which was originally developed for audio synthesis.

Diffusion Models for Time Series. Conditional diffusion models have been applied to time series tasks such as imputation and forecasting. The first work is due to Rasul et al. [44], who proposed TimeGrad for multivariate time series forecasting featuring a conditional diffusion head. Biloš et al. [6] extended TimeGrad to continuous functions and made modifications to the architecture enabling simultaneous multi-horizon forecasts. CDSI [52] is a conditional diffusion model for imputation and forecasting which is trained by masking the observed time series with different strategies. SSSD [1] is a modification of CDSI that uses S4 layers [18] as the fundamental building block instead of transformers. The models discussed above [44, 52, 1] are all conditional models, i.e., they are trained on specific imputation or forecasting tasks. In contrast, TSDiff is trained unconditionally and conditioned during inference using diffusion guidance, making it applicable to various downstream tasks.

Diffusion Guidance. Dhariwal and Nichol [10] proposed classifier guidance to repurpose pretrained unconditional diffusion models for conditional image generation using auxiliary image classifiers. Ho and Salimans [19] eliminated the need for an additional classifier by jointly training a conditional and an unconditional diffusion model and combining their score functions for conditional generation. Diffusion guidance has also been employed for text-guided image generation and editing [39, 4] using CLIP embeddings [41]. In contrast to prior work, our proposed observation self-guidance does not require auxiliary networks or modifications to the training procedure. Furthermore, we apply diffusion guidance to the time series domain while previous works primarily focus on images.

6 Conclusion

In this work, we proposed TSDiff, an unconditional diffusion model for time series, and a selfguidance mechanism that enables conditioning TSDiff for probabilistic forecasting tasks during inference, without requiring auxiliary guidance networks or modifications to the training procedure. Wedemonstrated that our task-agnostic TSDiff, in combination with self-guidance, is competitive with strong task-specific baselines on multiple forecasting tasks. Additionally, we presented a refinement scheme to improve predictions of base forecasters by leveraging the implicit probability density learned by TSDiff. Finally, we validated its generative modeling capabilities by training multiple downstream forecasters on synthetic samples generated by TSDiff. Samples from TSDiff outperform alternative generative models in terms of their predictive quality. Our results indicate that TSDiff learns important characteristics of time series datasets, enabling conditioning during inference and high-quality sample generation, offering an attractive alternative to task-specific conditional models.

Limitations and Future Work. While observation self-guidance provides an alternative approach for state-of-the-art probabilistic time series forecasting, its key limitation is the high computational cost of the iterative denoising process. Faster diffusion solvers [49, 33] would provide a trade-off between predictive performance and computational overhead without altering the training procedure. Furthermore, solvers developed specifically for accelerated guided diffusion [34, 55] could be deployed to speed up sampling without sacrificing predictive performance. Forecast refinement could be further improved by better approximations of the probability density and the utilization of momentum-based MCMC techniques such as Hamiltonian Monte Carlo [38] and underdamped Langevin Monte Carlo [50]. In this work, we evaluated TSDiff in the context of probabilistic forecasting, however, it is neither limited nor tailored to it. It can be easily applied to any imputation task similar to our forecasting with missing values experiment where we only evaluated the forecasting performance. Moreover, the core idea behind observation self-guidance is not limited to time series and may be applicable to other domains. Tailoring the guidance distribution appropriately, enables the use of self-guidance to a variety of inverse problems.

References

Juan Lopez Alcaraz and Nils Strodthoff. Diffusion-based time series imputation and forecasting with structured state space models. Transactions on Machine Learning Research , 2023. ISSN 2835-8856. URL https://openreview.net/forum?id=hHiIbk7ApW . 1, 4, 7, 10, 22

Alexander Alexandrov, Konstantinos Benidis, Michael Bohlke-Schneider, Valentin Flunkert, Jan Gasthaus, Tim Januschowski, Danielle C. Maddix, Syama Rangapuram, David Salinas, Jasper Schulz, Lorenzo Stella, Ali Caner Türkmen, and Yuyang Wang. GluonTS: Probabilistic and Neural Time Series Modeling in Python. Journal of Machine Learning Research , 21(116): 1-6, 2020. URL http://jmlr.org/papers/v21/19-820.html . 6, 16, 17, 21

Namrata Anand and Tudor Achim. Protein structure and sequence generation with equivariant denoising diffusion probabilistic models. arXiv preprint arXiv:2205.15019 , 2022. 10

Omri Avrahami, Dani Lischinski, and Ohad Fried. Blended diffusion for text-driven editing of natural images. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 18208-18218, 2022. 10

Konstantinos Benidis, Syama Sundar Rangapuram, Valentin Flunkert, Yuyang (Bernie) Wang, Danielle Maddix Robinson, Caner Turkmen, Jan Gasthaus, Michael BohlkeSchneider, David Salinas, Lorenzo Stella, Francois-Xavier Aubet, Laurent Callot, and Tim Januschowski. Deep learning for time series forecasting: Tutorial and literature survey. ACMComputing Surveys , 2021. URL https://www.amazon.science/publications/ deep-learning-for-time-series-forecasting-tutorial-and-literature-survey . 1

Marin Biloš, Kashif Rasul, Anderson Schneider, Yuriy Nevmyvaka, and Stephan Günnemann. Modeling temporal data as continuous functions with process diffusion. arXiv preprint arXiv:2211.02590 , 2022. 1, 10

C Bui, N Pham, A Vo, A Tran, A Nguyen, and T Le. Time series forecasting for healthcare diagnosis and prognostics with the focus on cardiovascular diseases. In 6th International Conference on the Development of Biomedical Engineering in Vietnam (BME6) 6 , pages 809818. Springer, 2018. 1

Emmanuel de Bézenac, Syama Sundar Rangapuram, Konstantinos Benidis, Michael BohlkeSchneider, Richard Kurle, Lorenzo Stella, Hilaf Hasson, Patrick Gallinari, and Tim Januschowski. Normalizing kalman filters for multivariate time series analysis. Advances in Neural Information Processing Systems , 33:2995-3007, 2020. 1

Abhyuday Desai, Cynthia Freeman, Zuhui Wang, and Ian Beaver. Timevae: A variational auto-encoder for multivariate time series generation. arXiv preprint arXiv:2111.08095 , 2021. 2, 9

Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. Advances in Neural Information Processing Systems , 34:8780-8794, 2021. 1, 2, 3, 4, 10, 15

Dua Dheeru and E Karra Taniskidou. Uci machine learning repository. 2017. 6, 16

Cristóbal Esteban, Stephanie L Hyland, and Gunnar Rätsch. Real-valued (medical) time series generation with recurrent conditional gans. arXiv preprint arXiv:1706.02633 , 2017. 9

FiveThirtyEight. Uber tlc foil response, 2016. URL https://github.com/ fivethirtyeight/uber-tlc-foil-response . 6, 16

Jan Gasthaus, Konstantinos Benidis, Yuyang Wang, Syama Sundar Rangapuram, David Salinas, Valentin Flunkert, and Tim Januschowski. Probabilistic forecasting with spline quantile function rnns. In The 22nd international conference on artificial intelligence and statistics , pages 1901-1910. PMLR, 2019. 6, 16

Tilmann Gneiting and Adrian E Raftery. Strictly proper scoring rules, prediction, and estimation. Journal of the American statistical Association , 102(477):359-378, 2007. 5, 6, 17

Rakshitha Godahewa, Christoph Bergmeir, Geoffrey I. Webb, Rob J. Hyndman, and Pablo Montero-Manso. Monash time series forecasting archive. In Neural Information Processing Systems Track on Datasets and Benchmarks , 2021. 6, 16

Alexandros Graikos, Nikolay Malkin, Nebojsa Jojic, and Dimitris Samaras. Diffusion models as plug-and-play priors. arXiv preprint arXiv:2206.09012 , 2022. 10

Albert Gu, Karan Goel, and Christopher Ré. Efficiently modeling long sequences with structured state spaces. arXiv preprint arXiv:2111.00396 , 2021. 4, 10

Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. arXiv preprint arXiv:2207.12598 , 2022. 2, 10

Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems , 33:6840-6851, 2020. 1, 3, 6, 10

Rob J Hyndman and George Athanasopoulos. Forecasting: principles and practice . OTexts, 2018. 7, 20

Rob J Hyndman and Yeasmin Khandakar. Automatic time series forecasting: the forecast package for r. Journal of statistical software , 27:1-22, 2008. 20

Paul Jeha, Michael Bohlke-Schneider, Pedro Mercado, Shubham Kapoor, Rajbir Singh Nirwan, Valentin Flunkert, Jan Gasthaus, and Tim Januschowski. Psa-gan: Progressive self attention gans for synthetic time series. In International Conference on Learning Representations , 2021. 9

Jaehyeong Jo, Seul Lee, and Sung Ju Hwang. Score-based generative modeling of graphs via the system of stochastic differential equations. arXiv preprint arXiv:2202.02514 , 2022. 10

Bahjat Kawar, Michael Elad, Stefano Ermon, and Jiaming Song. Denoising diffusion restoration models. Advances in Neural Information Processing Systems , 35:23593-23606, 2022. 10

Patrick Kidger, James Foster, Xuechen Li, and Terry J Lyons. Neural sdes as infinite-dimensional gans. In International Conference on Machine Learning , pages 5453-5463. PMLR, 2021. 9

Stephan Kolassa and Tim Januschowski. A classification of business forecasting problems. 2018. 1

Zhifeng Kong, Wei Ping, Jiaji Huang, Kexin Zhao, and Bryan Catanzaro. Diffwave: A versatile diffusion model for audio synthesis. arXiv preprint arXiv:2009.09761 , 2020. 1, 4, 10, 21

Guokun Lai, Wei-Cheng Chang, Yiming Yang, and Hanxiao Liu. Modeling long-and short-term temporal patterns with deep neural networks. In The 41st international ACM SIGIR conference on research & development in information retrieval , pages 95-104, 2018. 6, 16

Marten Lienen, Jan Hansen-Palmus, David Lüdke, and Stephan Günnemann. Generative diffusion for 3d turbulent flows. arXiv preprint arXiv:2306.01776 , 2023. 1

Bryan Lim and Stefan Zohren. Time-series forecasting with deep learning: a survey. Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences , 379(2194):20200209, 2021. doi: 10.1098/rsta.2020.0209. URL https: //royalsocietypublishing.org/doi/abs/10.1098/rsta.2020.0209 . 1

Bryan Lim, Sercan Ö Arık, Nicolas Loeff, and Tomas Pfister. Temporal fusion transformers for interpretable multi-horizon time series forecasting. International Journal of Forecasting , 37(4): 1748-1764, 2021. 7, 21

Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. Dpm-solver: A fast ode solver for diffusion probabilistic model sampling in around 10 steps. arXiv preprint arXiv:2206.00927 , 2022. 10

Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. Dpmsolver++: Fast solver for guided sampling of diffusion probabilistic models. arXiv preprint arXiv:2211.01095 , 2022. 10

Andreas Lugmayr, Martin Danelljan, Andres Romero, Fisher Yu, Radu Timofte, and Luc Van Gool. Repaint: Inpainting using denoising diffusion probabilistic models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 11461-11471, 2022. 10

David Lüdke, Marin Biloš, Oleksandr Shchur, Marten Lienen, and Stephan Günnemann. Add and thin: Diffusion for temporal point processes. In Advances in Neural Information Processing Systems , volume 37, 2023. 1

Spyros Makridakis, Evangelos Spiliotis, and Vassilios Assimakopoulos. The m4 competition: 100,000 time series and 61 forecasting methods. International Journal of Forecasting , 36(1): 54-74, 2020. 6, 16

Radford M Neal et al. Mcmc using hamiltonian dynamics. Handbook of markov chain monte carlo , 2(11):2, 2011. 10

Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen. Glide: Towards photorealistic image generation and editing with text-guided diffusion models. arXiv preprint arXiv:2112.10741 , 2021. 10

F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research , 12:2825-2830, 2011. 21

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning , pages 8748-8763. PMLR, 2021. 10

Syama Sundar Rangapuram, Matthias W Seeger, Jan Gasthaus, Lorenzo Stella, Yuyang Wang, and Tim Januschowski. Deep state space models for time series forecasting. Advances in neural information processing systems , 31, 2018. 1, 7, 21

Kashif Rasul, Abdul-Saboor Sheikh, Ingmar Schuster, Urs Bergmann, and Roland Vollgraf. Multivariate probabilistic time series forecasting via conditioned normalizing flows. arXiv preprint arXiv:2002.06103 , 2020. 1

Kashif Rasul, Calvin Seward, Ingmar Schuster, and Roland Vollgraf. Autoregressive denoising diffusion models for multivariate probabilistic time series forecasting. In International Conference on Machine Learning , pages 8857-8868. PMLR, 2021. 1, 10

Chitwan Saharia, William Chan, Huiwen Chang, Chris Lee, Jonathan Ho, Tim Salimans, David Fleet, and Mohammad Norouzi. Palette: Image-to-image diffusion models. In ACMSIGGRAPH 2022 Conference Proceedings , pages 1-10, 2022. 10

David Salinas, Valentin Flunkert, Jan Gasthaus, and Tim Januschowski. Deepar: Probabilistic forecasting with autoregressive recurrent networks. International Journal of Forecasting , 36(3): 1181-1191, 2020. 1, 7, 21

Omer Berat Sezer, Mehmet Ugur Gudelek, and Ahmet Murat Ozbayoglu. Financial time series forecasting with deep learning: A systematic literature review: 2005-2019. Applied soft computing , 90:106181, 2020. 1

Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International Conference on Machine Learning , pages 2256-2265. PMLR, 2015. 1, 3, 10

Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502 , 2020. 5, 10

Gabriel Stoltz, Mathias Rousset, et al. Free energy computations: A mathematical perspective . World Scientific, 2010. 6, 10

Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna. Rethinking the inception architecture for computer vision. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 2818-2826, 2016. 9

Yusuke Tashiro, Jiaming Song, Yang Song, and Stefano Ermon. Csdi: Conditional score-based diffusion models for probabilistic time series imputation. Advances in Neural Information Processing Systems , 34:24804-24816, 2021. 1, 7, 10, 21

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems , 30, 2017. 7, 17, 21

Ruofeng Wen, Kari Torkkola, Balakrishnan Narayanaswamy, and Dhruv Madeka. A multihorizon quantile recurrent forecaster. arXiv preprint arXiv:1711.11053 , 2017. 1, 7

Suttisak Wizadwongsa and Supasorn Suwajanakorn. Accelerating guided diffusion sampling with splitting numerical methods. arXiv preprint arXiv:2301.11558 , 2023. 10

Changtian Ying, Weiqing Wang, Jiong Yu, Qi Li, Donghua Yu, and Jianhua Liu. Deep learning for renewable energy forecasting: A taxonomy, and systematic literature review. Journal of Cleaner Production , 384:135414, 2023. ISSN 0959-6526. doi: https://doi.org/10.1016/j. jclepro.2022.135414. URL https://www.sciencedirect.com/science/article/pii/ S0959652622049885 . 1

Jinsung Yoon, Daniel Jarrett, and Mihaela Van der Schaar. Time-series generative adversarial networks. Advances in neural information processing systems , 32, 2019. 2, 9

Keming Yu and Rana A Moyeed. Bayesian quantile regression. Statistics & Probability Letters , 54(4):437-447, 2001. 5, 15

A Technical Details

A.1 Asymmetric Laplace Distribution

The asymmetric Laplace distribution is a generalization of the (symmetric) Laplace distribution and can be viewed as two exponential distributions with unequal rates glued about a location parameter. An alternative parameterization of this distribution has been employed for Bayesian quantile regression [58],

where κ is the quantile level, m is the location parameter, and b is the scale parameter. Setting κ = 0 . 5 yields the symmetric Laplace distribution. The negative log-likelihood of the asymmetric Laplace distribution corresponds to the pinball loss. An illustration of the probability density function (PDF) and the unnormalized negative log-likelihood (NLL) for different quantile levels is shown in Fig. 4.

0.0

0.2

0.4

0.6

0.8

1.0

-10

-6

-2

2

6

10

z

0.0

0.2

0.4

0.6

0.8

1.0

0.2

0.4

0.6

0.8

1.0

0

2

4

6

8

10

(a) PDF

(b) NLL

A.2 Algorithms

Observation Self-Guidance. The observation self-guidance can be easily implemented using guided reverse diffusion [10]. The pseudo-code of observation self-guidance is provided in Alg. 1. Given an observation, y obs , and guidance scale, s , the algorithm starts from Gaussian noise, x T , which is iteratively denoised while being guided towards the observation.

Refinement. The energy-based refinement scheme requires an observation, y obs , a base forecaster, g , a step size, η , and a noise factor, γ . The pseudo-code of the energy-based refinement scheme is shown in Alg. 2. First, y obs and the prediction of the base forecaster, g ( y obs ) , are combined to ˜ y and used as initial guess, y (0) . This initial guess is then iteratively refined using the score function of the conditional distribution, p θ ( y ( i ) | ˜ y ) . Setting γ = 0 in Alg. 2 yields the maximum likelihood refinement scheme.

Algorithm 2 Energy-Based Refinement

Input: observation y obs , base forecaster g , step size η , noise factor γ , number of iterations N ˜ y ← Combine ( y obs , g ( y obs ))

B Experiment Details

B.1 Datasets

We used eight popular univariate datasets from different domains in our experiments. Preprocessed versions of these datasets are available in GluonTS [2] with associated frequencies (daily or hourly) and prediction lengths. Table 5 shows an overview of the datasets. We used sequence lengths ( L , Context Length + Prediction Length) of 360 (i.e., 15 days) for the hourly datasets and 390 (i.e., approximately 13 months) for the daily datasets. These sequences were generated by slicing the original time series at random timesteps. In the following, we briefly describe the individual datasets.

Solar [29] is a dataset of the photo-voltaic (i.e., solar) power production in the year 2006 from 137 solar power plants in the state of Alabama. 5

Electricity [29, 11] is a dataset of the electricity consumption of 370 customers. 6

Traffic [29, 11] is a dataset of the hourly occupancy rates on the San Francisco Bay area freeways from 2015 to 2016. 7

Exchange [29] consists of daily exchange rates of eight countries including Australia, British, Canada, Switzerland, China, Japan, New Zealand, and Singapore from 1990 to 2016. 8

M4-Hourly [37] is the hourly subset from the M4 competition. 9

KDDCup [16] is a dataset of the air quality indices (AQIs) of Beijing and London used in the KDD Cup 2018. 10

UberTLC-Hourly [13] is a dataset of Uber pickups from Jan to Jun 2015 obtained from the NYC Taxi & Limousine Commission (TLC). 11

Wikipedia [14] is a dataset of daily count of the number of hits on 2000 Wikipedia pages. 12

B.2 Metrics

Continuous Ranked Probability Score (CRPS). The CRPS is a popular metric used to evaluate the quality of probabilistic forecasts. It is a proper scoring rule [15] and is defined as the integral of the pinball loss from 0 to 1, i.e.,

where Λ κ ( q, y ) = ( κ -✶ { y<q } )( y -q ) is the pinball loss for a specific quantile level κ . The CRPS measures the compatibility between the predicted inverse cumulative distribution function (also known as the quantile function), F -1 , and the observation, y . In practice, the quantile function is not analytically available and the CRPS is approximated using discrete quantile levels derived from samples. We used the implementation in GluonTS [2], which defaults to nine quantile levels, { 0 . 1 , 0 . 2 , 0 . 3 , 0 . 4 , 0 . 5 , 0 . 6 , 0 . 7 , 0 . 8 , 0 . 9 } , estimated using 100 sample forecasts.

Linear Predictive Score (LPS). The LPS is our proposed metric to evaluate the predictive quality of synthetic time series samples from a generative model. We define the LPS as the test CRPS of a linear ridge regression model trained on the synthetic samples. We trained the ridge regression model using 10,000 synthetic samples and computed the CRPS on the corresponding real test set. Note that the CRPS of a point forecaster, such as the linear model, is equal to the 0 . 5 -quantile loss (also known as the Normalized Deviation).

B.3 Hyperparameters and Training Details

We trained TSDiff using the Adam optimizer for 1,000 epochs with a learning rate of 1e-3. Every epoch is comprised of 128 batches constructed with 64 sequences each. We used 3 residual layers with 64 channels in the backbone of TSDiff. We also added a skip connection from input to output for some datasets as we observed improvements in the validation performance. The number of timesteps in the diffusion process was set to T = 100 and we used a linear scheduler with β 1 = 0 . 0001 and β 100 = 0 . 1 . The diffusion timesteps were encoded into 128-dimensional positional embeddings [53]. Table 6 provides an overview of the hyperparameters.

Normalization. The magnitudes of the time series can vary significantly, even within the same dataset. Training on raw values of such time series can lead to optimization instabilities. Therefore, we normalized the time series before training. Specifically, we used the mean scaler in GluonTS [2] which divides each time series individually by the mean of the absolute values of its observed context,

where abs( · ) is the element-wise absolute function and mean( · ) denotes the mean aggregator. After inference, i.e., after computing y norm ta , we rescale the target back to the original magnitude,

Guidance Scale Selection. The guidance scale in observation self-guidance (see Alg. 1) controls the strength of guidance and was selected on the basis of the validation performance. We tuned the scale from the set { 2 / 32 , 3 / 32 , 4 / 32 , 5 / 32 } for mean square guidance and observed that it performs well with the scale 4 / 32 on every dataset. For quantile guidance, we tuned the scale from the set { 1 , 2 , 4 , 8 } . Table 7 reports the best performing scale for quantile guidance on each dataset. The guidance scale also controls the alignment between the diffused time series and the observations. Fig. 5 shows that diffused time series from TSDiff-Q align closely with the observations.

01

Sep

02

19

20

21

22

23

24

25

26

27

28

29

30

31

100

150

200

Electricity

Ground Truth

Median Prediction

Forecast Horizon

90% Interval

17

Mar

18

19

20

21

22

23

24

25

26

27

28

29

30

31

0

100

200

KDDCup

Jan

2012

Apr

Jul

Oct

Jan

2013

Apr

1.0

1.1

Exchange

Forecasting with Missing Values. As discussed in Sec. 4, we evaluated the forecasting performance of TSDiff under three missing value scenarios to demonstrate its flexibility. Specifically, we examined the ability of observation self-guidance to handle missing values during inference. For each dataset, we trained a single unconditional model for TSDiff-Q without any modifications to the training objective. During inference, we masked 50% of the timesteps in the context window during inference, e.g., 168 out of 336 timesteps on the Solar dataset. The selection of which timesteps to mask depended on the missing value scenario (see Fig. 6 for an illustration of the three scenarios). Observation self-guidance was then used to predict both the missing and the target values (i.e., the forecast). The CRPS was only computed for the forecast horizon. To ensure that TSDiff did not train on the timesteps that would be masked during inference, we excluded a window of length equal to L (context length + prediction length) from the end of the training time series. Consequently, the M4-Hourly and Wikipedia datasets were excluded from the missing value experiments due to the resulting time series being too short.

RM

BM-B

0

30

60

90

120

150

180

210

240

270

300

330

360

390

BM-E

Context

Ground Truth

Forecast Horizon

Missing Values

On the other hand, the conditional models (TSDiff-Cond) were trained individually for each scenario, i.e., the missing values were masked and the model was optimized to predict them correctly.

Computational Cost of Observation Self-Guidance. Table 8 reports the inference time of observation self-guidance against the conditional diffusion model (TSDiff-Cond) on the Exchange dataset while controlling for aspects such as batch size, number of samples and GPU. The additional overhead for observation self-guidance (i.e., TSDiff-MS and TSDiff-Q) comes from the computation of the gradient of a neural network (see the Eq. 10).

Refinement Representative Step. As discussed in Sec. 3.2, we use a representative diffusion step to approximate the ELBO instead of randomly sampling multiple diffusion steps. The representative step corresponds to the diffusion step that best approximates the average loss. Fig. 7 shows the loss per diffusion step, the average loss and the representative step, τ , for the eight datasets used in our experiments. We computed the loss per diffusion step on a randomly-sampled batch of 1024 datapoints, which took ≈ 13 seconds per dataset on a single Tesla T4 GPU.

Refinement Iterations Selection. To strike a balance between convergence and computational efficiency, our refinement scheme necessitates careful selection of the number of refinement iterations. The relationship between the relative CRPS and the number of iterations is visualized in Fig. 8 for the Solar dataset. Notably, immediate improvement is observed after just one iteration for point forecasters like Seasonal Naive and Linear. Conversely, DeepAR and Transformer exhibit gradual enhancements. This analysis was conducted solely on the Solar dataset, determining that 20 refinement iterations offer a favorable balance between performance and computational requirements. As a result, we set the number of refinement iterations to 20 for all other datasets and base forecasters.

B.4 Baselines

We compared against eleven baselines, both from statistical literature and deep learning-based models, for the forecasting experiments. In the following, we briefly describe the individual baselines.

0

20

40

60

80

100

Diffusion Step (

t

)

0.0

0.1

0.2

0.3

0.4

Loss

Solar

Average Loss

Loss Per Diffusion Step

Representative Step

(a) Solar

0

20

40

60

80

100

Diffusion Step (

t

)

0.0

0.1

0.2

0.3

0.4

0.5

Loss

Electricity

Average Loss

Loss Per Diffusion Step

Representative Step

(b) Electricity

0

20

40

60

80

100

Diffusion Step (

t

)

0.0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

Loss

Traffic

Average Loss

Loss Per Diffusion Step

Representative Step

(c) Traffic

0

20

40

60

80

100

Diffusion Step (

t

)

0.00

0.02

0.04

0.06

0.08

0.10

0.12

Loss

Exchange

Average Loss

Loss Per Diffusion Step

Representative Step

(d) Exchange

0

20

40

60

80

100

Diffusion Step (

t

)

0.0

0.1

0.2

0.3

0.4

0.5

Loss

M4

Average Loss

Loss Per Diffusion Step

Representative Step

(e) M4-Hourly

0

20

40

60

80

100

Diffusion Step (

t

)

0.0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

Loss

KDDCup

Average Loss

Loss Per Diffusion Step

Representative Step

(f) KDDCup

0

20

40

60

80

100

Diffusion Step (

t

)

0.0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

Loss

UberTLC

Average Loss

Loss Per Diffusion Step

Representative Step

(g) UberTLC-Hourly

0

20

40

60

80

100

Diffusion Step (

t

)

0.0

0.2

0.4

0.6

0.8

1.0

Loss

Wikipedia

Average Loss

Loss Per Diffusion Step

Representative Step

(h) Wikipedia

0

5

10

15

20

25

30

Refinement Iterations

0.92

0.94

0.96

0.98

1.00

Refined CRPS / Baseline CRPS

Seasonal Naive

Linear

Transformer

Seasonal Naive is a naive forecaster that returns the value from the last season as the prediction, e.g., the value from 24 hours (1 day) ago for time series with hourly frequency.

ARIMA [21] is a popular statistical model for analyzing and forecasting time series data, combining autoregressive (AR), differencing (I), and moving average (MA) components to capture underlying patterns and fluctuations. We used the implementation available in the forecast package [22] for R.

ETS [21] is a forecasting method that uses exponential smoothing to capture trend, seasonality, and error components in the time series. We used the implementation available in the forecast package [22] for R.

Residual Blocks

Conv1x1

TSDiff-Cond Architecture

Conv1x1

Conv1x1

Conv1x1

Conv1x1

Conv1x1

Conv1x1

: Element-wise add.

: tanh

: Element-wise mult.

: ReLU

: Swish

Conv1x1

Conv1x1

Linear is a linear ridge regression model across time, i.e., it takes the past context length timesteps as inputs and outputs the next prediction length timesteps. We used the implementation available in scikit-learn [40] with the regularization strength of 1 (its default value in scikit-learn ). The model was fit using 10,000 randomly sampled sequences with the same context length for each dataset as for TSDiff.

DeepAR [46] is an RNN-based autoregressive model that is conditioned on the history of the time series through lags and other relevant features, e.g., time features such as hourof-day and day-of-week. The model autoregressively outputs the parameters of the next distribution (e.g., Student's-t and Gaussian) and is trained to maximize the (conditional) log-likelihood. We used the implementation available in GluonTS [2] with the recommended hyperparameters.

MQ-CNN utilizes a convolutional neural network (CNN) architecture to capture patterns and dependencies within the time series data. The model is trained using the quantile loss to directly generate multi-horizon forecasts. We used the implementation available in GluonTS [2] with the recommended hyperparameters.

DeepState [42] combines RNNs with linear dynamical systems (LDS). The RNN takes additional covariates (e.g., time features and item ID) as input and outputs the (diagonal) noise covariance matrices of the LDS. Other parameters of the LDS such as the transition and emission matrices are manually designed to model different time series components such as level, trend, and seasonality. The LDS and RNN are jointly trained via maximum likelihood estimation. We used the implementation available in GluonTS [2] with the recommended hyperparameters.

Transformer is a sequence-to-sequence forecasting model based on the self-attention architecture [53]. The model takes time series lags and covariates as inputs and outputs the parameters of future distributions. We used the implementation available in GluonTS [2] with the recommended hyperparameters.

TFT [32] is another sequence-to-sequence forecasting model based on the self-attention architecture that includes a variable selection mechanism to minimize the contributions of irrelevant input features. The model is trained to minimize the quantile loss at the selected quantiles. We used the implementation available in GluonTS [2] with the recommended hyperparameters.

CSDI [52] is a conditional diffusion model for multivariate time series imputation and forecasting. Similar to TSDiff, CSDI is based on the DiffWave architecture [28], but utilizes transformer layers as fundamental building blocks. The original CSDI architecture was presented in the context of multivariate time series comprising temporal and feature transformer layers. We used the original implementation 13 in the univariate setting with the recommended hyperparameters and training setup in our experiments.

TSDiff-Cond is a conditional version of TSDiff. It integrates the observed context together with an observation mask using a Conv1x1 layer followed by an addition after the S4 layer

in each residual block as shown in Fig 9. TSDiff-Cond's architecture resembles that of SSSD [1] with three key differences:

-TSDiff-Cond uses S4 layers in the residual blocks before addition with the conditioning vector whereas SSSD uses them both before and after addition.

-TSDiff-Cond is a univariate model where feature dimensions indicate lagged time series values whereas SSSD is a multivariate model with feature dimensions indicating the different features in the multidimensional time series.

-TSDiff-Cond computes the loss on the entire time series, unlike SSSD which only generates the unobserved timesteps. This led to better performance of the conditional model in our experiments.

We used the same hyperparameters (e.g., context lengths and lags) for TSDiff-Cond as for TSDiff (i.e., the unconditional model) to isolate the effect of conditional training.

For the train on synthetic-test on real experiments, we compared against two popular time series generative models.

TimeVAE is based on variational autoencoders and includes network components specific to time series such as trend and seasonality blocks. We used the original implementation 14 and recommended hyperparameters in our experiments.

TimeGAN employs an autoencoder to train a generative adversarial network in the latent space. The network components are trained using a combination of supervised, unsupervised, and discriminative loss functions. We used the original implementation 15 and recommended hyperparameters in our experiments.

C Synthetic Samples

Figs. 10 to 17 illustrate synthetic time series created by TimeVAE, TimeGAN, and TSDiff in comparison to real time series. The generated samples from TSDiff closely resemble the real samples across all datasets, exhibiting higher quality compared to the baselines. For instance, in the case of the solar dataset (Fig. 10), TSDiff accurately captures periods of zero solar energy production during nighttime and generates a wide range of peaks within individual time series. In contrast, samples from TimeVAE and TimeGAN appear more homogeneous and fail to capture the zero values precisely. TimeGAN also experiences mode collapse issues in certain datasets like Electricity, M4-Hourly, and UberTLC-Hourly, while TSDiff consistently generates diverse and realistic samples.

Real

TSDiff

TimeGAN

TimeVAE

0

100

200

300

0

100

200

300

0

100

200

300

0

100

200

300

Real

TSDiff

TimeGAN

TimeVAE

0

100

200

300

0

100

200

300

0

100

200

300

0

100

200

300

Real

TSDiff

TimeGAN

TimeVAE

0

100

200

300

0

100

200

300

0

100

200

300

0

100

200

300

Real

TSDiff

TimeGAN

TimeVAE

0

100

200

300

400

0

100

200

300

400

0

100

200

300

400

0

100

200

300

400

Real

TSDiff

TimeGAN

TimeVAE

0

100

200

300

0

100

200

300

0

100

200

300

0

100

200

300

Real

TSDiff

TimeGAN

TimeVAE

0

100

200

300

0

100

200

300

0

100

200

300

0

100

200

300

Real

TSDiff

TimeGAN

TimeVAE

0

100

200

300

0

100

200

300

0

100

200

300

0

100

200

300

Real

TSDiff

TimeGAN

TimeVAE

0

100

200

300

400

0

100

200

300

400

0

100

200

300

400

0

100

200

300

400

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper focuses on generative diffusion modeling when some of the samples are known, which is advocated as the right setup for regression. Indeed, in a time-series context, those known samples may lie in the past (prediction), or provide boundary conditions (interpolation). \nIn this context, the classical diffusion equations can be worked out to yield a generation that is compatible with the known samples to some controllable extent. This is the key contribution for this paper, and I must say that traming regression as a constrained diffusion is an interesting  idea.\n\nFrom this starting point, the authors try to derive two variants: one where the model estimate is used as a prior, and one where some arbitrary other estimate is used.\n\nA set of forecasting experiments conclude the paper\n\nSTRENGTHS:\n- the paper is very readable and provides some background and references. It is quite clear\n- Framing regression as a constrained diffusion is definitely interesting and has some merits.\n- I would also say that I'm sure that the proposed method yields interesting results, compared to some other methods\n\nWEAKNESSES:\nNotwithstanding its qualities, I have several concerns with the paper that prevent me from giving it a high score.\n- I would say that the attempt at formulating the method in a Bayesian way is actually a bit awkward in my opinion, because it does not bring much more food for thought than just plainly presenting the guidance as a regularization over the (enegry-based) generating process. Indeed, the prior distribution that is picked in equation (5) just reads like picking some L2 regularization to me, and the quantile regularization that is advocated as working very well in practice just doesn't lend itself very well to a convenient Bayesian treatment. Actually, section 3.2 precisely drops this Bayesian vision to adopt an optimization-based point of view, which seems more natural to me.\n\n- I feel that there is actually a lot of redundancy between 3.1 and 3.2. As I see it, the two \"variants\" are actually the same, with the only difference that for self-guidance, the estimate is derived from the model while it is assumed arbitrary in 3.2.\n\nQUESTIONS:\n- L85: consists of decomposing\n- L119: \"see Eq 1\": It is not clear to me why you provide this reference to eq (1)\n- eq (5), again, the choice of a unit variance for the likelihood is weird to me. I would say this is only picked this way to motivate some L2 regularization term in a \"reverse mathemating\" sense\n- I don't understand how you get \\epsilon in equation (10) and further during inference time. \n- L249-250: when you write that the model does not require task-specific training, could you actually explicitly mention *what* is the model trained on ? I thought that there would be a separate training for each task ?\n\nLIMITATIONS:\n- will you provide some implementation ?", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes TSDiff, which is an unconditional diffusion model for time series generation. Besides, the authors propose self-guidance and prediction refinement. The empirical results showcase the superiority of TSDiff over existing baselines on forecasting, refinement, and generating synthetic samples.\n\nSTRENGTHS:\n1. The paper is joyful to read and the methodology is clearly described.\n2. The empirical results are more than enough. The authors conduct extensive experiments on various of tasks and datasets, making the proposed methodology very convincing.\n3. It is good to see that the authors proposed a new metric to make more reasonable comparison with other baselines in Section 5.3.\n\nWEAKNESSES:\n1. If y_obs is a subset of timesteps representing the observed timesteps, then is the conditional generation similar to an inpainting problem in 2D images? Then important baselines in solving inverse problems with pre-trained diffusion models, e.g., DDRM [1], should be included, or at least be discussed thoroughly.\n\n2. There are several equations that are not labeled with numbers. It would be the best if the authors can label them. I have several questions regarding the equation after Eq. (10): \n\n[1] Is tau hard to solve? It looks like solving tau is computationally expensive. It would help if the authors can provide some empirical results. Actually I did not quite understand \"tau can be computed efficiently by tracking the running mean for the losses at all diffusion steps during training\".\n\n[2] How does the optimum, tau, vary with different data point? In this equation, the expectation does not take x_t (or y) into account, does this mean we have to solve for tau for every y? Or do we jointly solve a tau using all the data points in the training set?\n\nQUESTIONS:\nSee Weaknesses.\n\nLIMITATIONS:\nThe limitations are well discussed in the paper.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper describes a diffusion model for time series problems. Contrary to the popular approach of using a conditional diffusion model, the authors proposed to use the unconditional diffusion model, which is supplemented by a self-guidance mechanism. The authors also proposed a prediction refinement algorithm to improve the prediction of any time series model by using the trained diffusion model.\n\nSTRENGTHS:\nTraining a conditional diffusion model is challenging due to the high dimensionality of the conditional problem. The authors proposed an interesting method of bypassing training a conditional diffusion model. The proposed methodology is simple and sound.\n\nWEAKNESSES:\nThe paper does not elaborate the details of their model, which makes it difficult to fully understand the methodology. For example, it does not clearly show the dimensions of input and output variables, how the missing inputs are treated, and so on.  There are also a few comments in Questions sections.\n\nQUESTIONS:\n1. In equation (6), as $t$ gets larger, the denominator $\\sqrt{\\overline{\\alpha}_t} \\rightarrow 0$, which is a natural consequence of the diffusion process. How do you handle this singularity issue? In practice, it will not be singular, but potentially cause a numerical instability.\n\n2. The target distribution, $p(y_{ta}|y_{obs})$, but it actually should be $p(y_{ta}|y_{obs}) = p(y_{unobs}|y_{obs})p(y_{obs}|y_{obs})$ and $p(y_{obs}|y_{obs}) = \\delta(y-y_{obs})$. So, upon the guidance denoising, we expect the observed $y$ should be converged to $y_{obs}$. I wonder if the model guarantees this.\n\n3. Provide the proof that equation (9) converges to the maximum likelihood solution when $\\gamma = 0$. Also, please provide a proof that (9) converges to $p(y)$.\n\n4. Carefully reading it, I can understand how the guided diffusion, eqn (5), is computed for missing data and prediction. But, it will be useful if the authors can elaborate it to improve the readability.\n\n5. It is claimed that the model can be used for imputation. Is the diffusion model trained with a missing data? Or, does it require a full data?\n\n*I am willing to upgrade my score once these questions are addressed.\n\nLIMITATIONS:\nOne of the questions of time series diffusion model is whether it memories the pattern or learns the dynamics. Based on the learning objectives, it is more likely to learn the patterns, which may limit the capability of diffusion-based models for the time series problems of chaotic dynamics and a stochastically forced system. Also, as the authors suggested, the computational cost is certainly of concern for a scalability. But these are beyond the scope of the study.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this paper, the authors proposed an unconditional diffusion model and a self-guidance mechanism for time series data that can be used for conditioning diffusion model for downstream tasks, e.g. time series forecasting and imputation. The effectiveness of proposed model is evaluated from three aspects: prediction, refinement of prediction from other base forecaster, and time series generation. The results show that unconditional diffusion models can achieve comparable forecasting accuracy to conditional ones. The proposed model can be potential useful for other downstream time series analysis tasks other than studied ones.\n\nSTRENGTHS:\n1. The author proposed a new self-guidance mechanism to approximate conditional probability p(y_obs|x^t) which enables time series forecasting with unconditional diffusion model\n2. Extensive experiments on multiple benchmarks demonstrate the effectiveness of the proposed unconditional diffusion model comparing to conditional ones.\n3. The paper is well written and easy to follow.\n\nWEAKNESSES:\n1. Refining the prediction of a weak forecaster is a good way to evaluate the model, but it has no real application scenario.\n\nQUESTIONS:\n1. The authors described the details of how to do self-guidance without explanations of why this approximation can work in practice. It would be helpful to explain the high level intuition behind it.\n2. The idea of self-guidance seems to be general, can it be used in other domain, e.g. image?\n3. It is a good way to evaluate the model with refining the prediction. But this is not a practical senario. If one has the diffusion model, why not just use it for prediction? There are a lot of spaces used for this in section 4. It would be better for me to discuss other more interesting things.\n4. The authors mentioned that the computational cost of the proposed model is high. It would be better to have such experiments comparing the forecasting speed with conditional diffusion models.\n5. The authors evaluate the time series synthesis using train on synthetic-test on real setup. It would also be interesting to see train on synthetic and real - test on real setup. Can synthetic time series helpful for forecasting?\n6. The experiments are conducted on univariate case, can the method be applied for multi-variate time series?\n7. Can other time series analysis task get benefit from proposed models other than forecasting and imputation? For instance, time series classification, clustering, similarity search.\n\nLIMITATIONS:\nYes, the author claimed the limitation of the paper, which is the computational cost.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-14"}