Transformer-Modulated Diffusion Models for Probabilistic Multivariate  Time Series Forecasting
Authors: Yuxin Li
Wenchao Chen
Xinyue Hu
Bo Chen
Baolin Sun
Mingyuan Zhou
Venue: ICLR 2024 poster (2024)
Keywords: time series forecasting
generative modeling
deep learning
diffusion model
Primary area: generative models

Abstract: Transformers have gained widespread usage in multivariate time series (MTS) forecasting, delivering impressive performance. Nonetheless, these existing transformer-based methods often neglect an essential aspect: the incorporation of uncertainty into the predicted series, which holds significant value in decision-making. In this paper, we introduce a Transformer-Modulated Diffusion Model (TMDM), uniting conditional diffusion generative process with transformers into a unified framework to enable precise distribution forecasting for MTS. TMDM harnesses the power of transformers to extract essential insights from historical time series data. This information is then utilized as prior knowledge, capturing covariate-dependence in both the forward and reverse processes within the diffusion model. Furthermore, we seamlessly integrate well-designed transformer-based forecasting methods into TMDM to enhance its overall performance. Additionally, we introduce two novel metrics for evaluating uncertainty estimation performance. Through extensive experiments on six datasets using four evaluation metrics, we establish the effectiveness of TMDM in probabilistic MTS forecasting.

Published as a conference paper at ICLR 2024
TRANSFORMER-MODUUATDD DIFFUSION MODELS FOR PROBABILISTIC MULTIVARIATE TIME SERIES FORECASTING
Yuxin Li', Wenchao Chen Xinyue Hu', Bo Chen " Baolin Sun', Mingyuan Zhou
National Key Laboratory of Radar Signal Processing, Xidian University, Xi'an, an, 710071, China. 2 Xidian University, Xi' an, 710071, China.
3 McCombs School of Business, The University of Texas at Austin, Austin, TX 78712
yuxinli20208@maal
com wccher.xiician@116. com xinyuehul 220 gmail. com
bchenêmail xidian. edu. en surbaolin0993@@man. mingyuan. zhou@mccombs utexas.
ABSTRACT
Transformers have gained widespread usage in multivariate time series (MTS) forecasting, delivering impressive performance. Nonetheless, these existing transformer-based methods often neglect essential aspect: the incorporation of uncertainty into the predicted series, which holds significant value in decision- Soansformer-Modullae Diffusion Model
making, In this paper, introduce (TMDM), uniting conditional diffusion generative process with transformers into unified framework to enable precise distribution forecasting for MTS. TMDM harnesses the power of transformers to extract essential insights from historical time series data. This information is then utilized prior knowledge, capturing covariate-depennencc in both the forward and reverse processes within the dif- fusion model. Furthermore, based forecasting methods into TMDM to enhance its overall performance. Ad- ditionally, we introduce novel metrics for evaluating uncertainty estimation performance. Through extensive experiments on six datasets using four evalua- tion metrics, we establish the effectiveness of TMDM in probabilistic MTS fore- casting.
seamlessly integrate well-designed transformer-
INTRODUCTION
Time series forecasting plays pivotal role in both the business and scientific domains of ma- chine learning, serving as vital tool for supporting decision-making in a wide array of down- stream applications. These applications encompass, but are not limited to, financial pricing analysis (Kim) 2003), transportation planning (Sapankevych & Sankar 2009). and weather pattern predic- tion (Chatfield 2000), among various other fields (Rasul et al.) 2022). The primary objective of time series forecasting is predict the response variable yo:M E R based on historical time series dataset represented as Co:N E dx This prediction process is characterized by the func- tion (W RR where is deterministic function that transforms the historical time series Co:N into the future time series Yo:M-
Existing time series forecasting methods commonly adopt additive noise model represent the future time series yo.M as the following: Yo:M = f( + no. where no follows a normal distribution N (000, a²). Consequently, we can calculate the expected value of yo.M given No. as E B[yo-M| [o:>] = f(xo.N) ). Classical time series forecasting methods (Liu et al. 2022 Wang et al., 2022 (Zhou al. (202I) that rely on this additive noise model typically provide univariate forecasts by accurately estimating the conditional mean Elyo:M [o:N). These models have shown significant advancements in recent years, particularly with the adoption of transformer-based architectures. Transformers leverage self-attention mechanisms and designs tailored for handling time series char- acteristics effectively. This enhancement empowers transformers to excel in modeling long-tern
"Corresponding author.

---

Published as a conference paper at ICLR 2024
dependencies within sequential data (Wu et al. 2021), enabling the development of more potent, large-scale models (Kenton& & Toutanova 2019).
However, the aforementioned methods pay less attention to whether the noise distribution can ac- curately capture the uncertainty of Yo:M given C0:N. In time series forecasting, modeling uncer- tainty holds paramount importance it directly affects our ability to assess the reliability of pre- dictions for downstream applications (Rasul et al. 2021b). This uncertainty significantly impacts decision-making accuracy. For instance, if point estimation model predicts that the conditional mean E|go.M [o:N]] for tomorrow's temperature is 12 "C, individuals still face a difficult deci- sion regarding whether to cultivate plants today, as the morning temperature might plummet to just 4 PC jeopardizing the plants' survival. Such models overlook the risks associated with uncer- tainty, which can be particularly crucial in certain contexts (Kim) 2003, Sapankevych & Sankar 2009). As another example, if we assign the predicted temperature for tomorrow as a Gaussian distribution, the corresponding uncertainty, represented by (121,, 2²) or (12,, 8²), could directly influence decision-making processes. The primary objective of this paper is to recover the full distribution of future time series yo.Ms conditioned on the representation captured by existing well- designed transformer- based models. To achieve this goal, we introduce a novel framework called the Transformer-Modulatee Diffusion Model (TMDM), which unifies the conditional diffusion gen- erative process (Ho et al. 2020: Sohl-Diiksttinn et al.) 2015 Song et al., 2020) with transformers, facilitating accurate distribution forecasting for time series.
Recently, diffusion- based generative models have garnered significant attention due to their capacity to generate high-dimensional data and provide training stability (Han et al. 2022). These models can be viewed from various perspectives, including score matching (Hyvärinen & Dayan 2005 Vincent 2011) and Langevin dynamics (Neal et al. 201T: Welling & Tch (011). However, there has been a recent development in our understanding of these models through the lens of diffusion probabilistic models (Graikos et al., 2022). These models initially employ forward process to transform data into noise and subsequently use a reverse process to regenerate the data from the noise (Ho et al.
2020
Current time-series diffusion models (Rasul al. 2021a Tashiro et al. 2021: Alcaraz Strodthoff]
2022 Shen & Kwok 2023) primarily concentrate on crafting effective conditional embeddings to be fed into the denoising network, which in turn guides the reverse process within the diffusion model. For instance, TimeGrad (Rasul al. 202 la) employs the hidden state from an RNN as the conditional embedding, while Diff (Shen & Kwok 2023) constructs this embedding based on two features explicitly designed for time series data. In contrast to prevailing approaches that solely utilize conditional embeddings during the reverse process, TMDM employs conditional information as prior knowledge for both the forward and reverse processes. We believe this approach be more efficient means of leveraging the representations captured by existing transformer-bae based time- series models (Liu et al. 2022 Wang al. 2022) as conditions, given their proficiency in estimating the conditional mean E Yo:M Zo:N |. Empowered by this potent prior knowledge, TMDM is geared
toward capturing the uncertainty of future time series yo.M, ultimately providing a comprehensive estimate of the entire distribution.
We summarize our contributions as follows: (1) the realm of probabilistic multivariate time series forecasting, we introduce TMDM, transformer- based diffusion generative framework. TMDM har- nesses the representations captured by well-designed transformer-basse time series models as priors. We consider the covariate-dependence across both the forward and reverse processes within the dif- fusion model, resulting in highly accurate distribution estimation for future time series. (2) TMDM integrates diffusion and transformer-base based models within cohesive Bayesian framework, employ- ing a hybrid optimization strategy, it serves as a bllgg-nd--il framework, seamlessly compatible with existing well-designed transformer-basee forecasting models, leveraging their strong capability to estimate the conditional mean of time series, facilitating the estimation of complete distributions. (3) In our experimental evaluation, we explore the application of Prediction Interval Coverage Prob- ability (PICP) (Yao et al. 2019) and Quantile Interval Coverage Error (QICE) (Han et al. (022) as metrics in the probabilistic multivariate time series forecasting task. These metries provide valuable insights into assessing the uncertainty estimation abilities of probabilistic multivariate time series forecasting models. Our study demonstrates TMDM's outstanding performance in four distribution metrics across six real-world datasets, emphasizing its effectiveness in probabilistic MTS forecast- ing.
2

---

Published as a conference paper at ICLR 2024
2 BACKGROUND
2.1 DIFFUSION MODEL
Diffusion probabilistic models (Sohl-Dickstein al., 2015 take the form pe pe
po (y Jdyb where yo Vo:M:.... Vo.M represent latent variables (Ho et al. 2020). One well- known diffusion model is the denoising diffusion probabilistic model (DDPM) (Ho et al. 2020). which consists two processes: the forward (diffusion) process and the reverse process. Following the Markov chain, the forward process gradually adds noise, transforming an input vector yo. into a Gaussian noise vector yo. over T steps:
1:T (yooMM B6.M) = IT= 9(36.M No.s)) g(36.r (6:1) == N(VI 3 Bo. 1 I'1)
(1)
where 3e represents small positive constant denoting the noise level. In practical applications, we directly sample Vo.M from y6:M as the following: 1(6..M yo = NN( V(Va'y (1 I),
where a' ;= 8 and a' := I a'. The reverse process involves denoising yb.r back your
and is defined as Markov chain with a learned Gaussian transition:
Po ( = M) IT. Po P0 y6:r), po (Voos A
16:M) == (Nuo0066 a (VbbM f))
(2)
In DDPM (Ho et al. 2020), the parameterization of pa pe( 36.M) is defined as:
1-1
μs (Bb () = (yb.s g¹ ae ( t) (B2) =ft= 1: 3 = clse 3 = 1-a"
Vi-o* cn ()))).
T-o! B²
where the Eg is denoising function and which can be trained by solving the following optimization problem:
min (()) = Egg ~a N. ||e co(yb.u, +))1?
Using the trained denoising function Eg. we can generate samples step by step from NO I) ran- domly. However, in the context of time series forecasting, the objective is generate the future time series yo:M conditioned on the historical time series Zo:N.. Several studies (Rasul et al., (2,2222aa) Tashiro et al. 2021 Alcaraz & Strodthoff] 2022 Shen & Kwok (022)) have explored adapting dif- fusion models for this task by injecting historical conditional information into the reverse process to guide the generative process.
2.2 PROBABILISTIC MULTIVARIATE TIME SERIES FORECASTING
Rd,, probabilistic multivariate time series forecasting tackles the problem of estimating the distribution of the subsequent future time series yo:As = {p(y1), p(12),.... p(yas) Ye RR.. However, it's essential to note that the exact distribution of p(y)) is computationally intractable, prompting the development various methods for its approximation. From the perspective of diffusion generation (Sohl-Dickstein al.) 2015), we establish a Markov chain (Geyer 1992) with learned Gaussian transitions, initiating from p(U6.M) (0, I), estimate the distribution of yo.M-
Given observed history MTS Vo.N {21, (2........ ZN
pe (BOT
D.T (O.N) := pluo.M IT Po Vo.ar To.N)
(5)
Pa (3b
yo. Do.N) = N (Nus (Huaytaa Do.N. 1), are (yó Co.N, f))
An essential premise probabilistic multivariate time series forecasting the continuity and inter- relation between the observed history Øo.N and the future time series yo:as However, two significant challenges arise: firstly, how to extract valuable time series information from Zo.N (Shen & Kwok 2023). and secondly. how to effectively employ this information to guide the generative process. Existing transformer-basee time series forecasting models (Liu et al. 2022 Wu et al. 202 Wang al. 2022, Zhou et al. 2021) tend to overlook the estimation of time series uncertainty. However, they still have specially designed structures to capture information in the time series, which can be used as conditional information directly. Regarding the second challenge, recent models (Ra- sul et al., [2211a] Alcaraz Strodthoffi (022)) attempt to inject conditional embeddings into the denoising network during the reverse process. In contrast these methods, proposed model,
3

---

Published as a conference paper at ICLR 2024
Fn, = Re.n,,
Musr (Bee. Brr...
Transformer
Viu -Va yhar + +(1 Va' Vune +--
$ N Nirrs D)
Diflusion process
Viffusion
Vou = =ou uu + Vonr YyBuw + + Yau (1/e Ma -1 Ve' Maee V1 cglal Mui))))
Condittion
yus
I
Reverseprooee
Yoar
Vis (11aa)) -0-/ -11-o Yhhr - f, M)
Figure 1: An illustration of the proposed TMDM. The left part is the condition generative model, containing existing well-designed transformer, used to generate the condition Yo.Ar. The right part the proposed conditional diffusion- based time series generative model that utilizes yo. as the prior and introduces covariate- derrrateeeeeeeeecee into both the forward and reverse diffusion chains.
TMDM, employs conditional information as prior knowledge for both the forward and reverse pro- cesses within the diffusion model. By integrating conditional information into the forward process, TMDM can consider a richer set of conditional information during the denoising process, This en- hancement enables TMDM better learn the inherent time series properties between ZO.N (captured by the condition) and yo:M (the generated target).
2.3 PICP AND QICE FOR ASSESSING UNCERTAINTY ESTIMATION
To enhance the assessment of uncertainty estimation capabilities in probabilistic multivariate time series forecasting tasks, we introduce novel metries: Prediction Interval Coverage Probability
(PICP) (aoo et al. 2019) and Quantile Interval Coverage Error (QICE) (Han et al. 2022 The computation of PICP P the following:
PICP = N M. 1 U.2
(6)
Un Sm
where glow and In predicted yo.M outputs given the same No.N input. In cases where the learned distribution accurately represents the true distribution, this measurement should closely align with the difference between the selected low and high percentiles (Han et al. (2022). QICE can be viewed as extension of PICP with higher granularity and without any uncovered quantile ranges. Its computation is as the following:
high represent the low and high percentiles, respectively, of our choice for the
1 where rm = +/4 Σ 2n 1 M I' M QICE := M7
bighu
With sufficient number of yo:M samples, the first step involves dividing them into M quantile intervals (QIs), each with approximately equal sizes. Subsequently, quantile values corresponding each boundary are determined. In contrast to PICP, QICE offers more detailed evaluation. In cases where fewer true instances fall within one QI, another QI may capture more instances, potentially leading increased absolute errors both QIs. Additionally, we utilize the Continuous Ranked Probability Score (CRPS) (Matheson & Winkler 1976, Gneiiing & Raftery] (207]) and CRPSsum for evaluation on each dimension of the time series. CRPSsum represents the CRPS computed for the sum of all time series dimensions.
PROPOSED METHOD
In this section, we present TMDM, a novel framework that combines the diffusion generative pro- cess (Ho et al. 2020 Sohl-Dickstein al. 2015) with well-designed transformer structures (Liu
4

---

Published as a conference paper at ICLR 2024
et al. 2022 Wang al. 2022). These transformer models excel at accurately estimating the con- ditional mean I Mo:M No while TMDM extends this capability to recover the full distribution of the future time series Yo.M. As depicted in Fig. I TMDM consists of two main components: a ceeesformer-pwweeed conditional distribution learning model (condition generative model) and a conditional diffusion-barte based time series generative model. These two models are integrated into unified Bayesian framework, leveraging hybrid optimization approach. From conceptual stand- point, TMDM can be viewed as a Bayesian generative model (Tran et al. 2019), where the generative process can be expressed as:
p(yaa)))
Ma:3r la P( (o:M) I[ p(ybrd Varse Do:M )p( | ())(()))dd )x(()dzdydbl
(8)
In this paper, we leverage well well-designed transformers, including the Non-stationary transformer (Liu et al., 2022), Autoformer (Wu et al. 2021). and Informer (Zhou et al, (021), to capture the information embedded within the historical time series æD:M. We utilize this information to model a latent variable 2, which in turn generates a conditional representation Go.M. This representation serves condition for the subsequent forward and reverse processes.
3.1 LEARNING TRANSFORMER POWERED CONDITIONS
Existing time-series diffusion models (Rasul et al. 202 la Tashiro et al. 2021 Alcaraz & Strodthoff] 2022 [hhn & Kwok 2023) have primarily focused on designing effective conditional embeddings to guide the reverse process, In contrast, our approach advocates the utilization of representations captured by well-established transformer-basee time series models. This shift offers several dis- tinct advantages. Firstly, significant advancements have been made in point estimation time series forecasting tasks recent years. Extensive research into time series properties has resulted in proposal of dedicated transformers tailored for this purpose (Liu et al. 2022 Wu et al. 2021 Wang ct al. 2022). We contend that employing conditions derived from such transformers is more effi- cient than relying on self-designed conditioning embeddings. Secondly, these specialized transform- ers exhibit a strong capability to estimate the conditional mean E El(oo Øo.N By employing this estimated mean as the condition, the diffusion model more effectively focus estimating certainty, simplifying the generative process. Conversely, using other specially designed conditions, such future mixup (Shen & Kwok 2023). may introduce new information but requires the dif- fusion model to simultancously estimate both the mean and uncertainty. rendering generation more complex. Finally, TMDM serves as a versatile point estimates and distribution estimates. If improved transformer structures for point estimation emerge, we can seamlessly integrate these advancements into the distribution estimation domain.
framework, bridging the gap between
Given the transformer structure 7 (.) and the historical time series Zo.N. we can capture the repre- sentation by T (Io.N). This representation serves as the guiding factor for approximating the true posterior distribution of z. This process is defined as the following:
q(= I (Iu:N) )) N (ū. (xu:)))) )), ã ( (zv:)))
Given a well-learned z, we can generate the conditional representation fo.M as the following:
z ~ N(O, and Uo:a ~ ~ (z),
(10)
Here, we model three nonlinearity functions, jıg. o., and μ.,. using neural networks. We initialize o. representing the covariance matrix, to the identity matrix In this manner, we define latent variable to summarize the information captured by well-designed transformers. This latent vari- able is then used to generate the conditional representation go:m for the subsequent forward and reverse processes in TMDM.
3.2 CONDITIONAL DIFFUSION-BASE BASED TIME SERIES GENERATIVE MODEL.
Different from vanilla diffusion models that assume the endpoint of the diffusion process, yo. adheres the standard normal distribution (O 1), incorporate the conditional representation M into p( yT to better account for the conditional information in Eq. 10 Drawing inspiration from Han et al. (2022), we model the endpoint of our diffusion process as the Tollowing:
p(yō.M M/ (o.M) A'(on.M I)
(11)
where yn.as. as defined in Eq. 10 incorporates the information captured by the transformer. In Eq. 11 To:m be viewed prior knowledge for estimating the conditional mean E(yoo:M
5

---

Published as a conference paper at ICLR 2024
Algorithm 1 Training
Algorithm 2 Inference
1:
Initialize the parameters; repeat
1: your ~ N (To.M.II
2: for t= to to do
Draw yo:M ~ g(36.M 2o.N)
3: Calculate reparameterize: Yu. (1/a')1/ (1
4: Draw t mv Uniform( (1,, 2, 7H)
Va') vI a'ea
5: Draw (0,,
4: if > 1: draw ((0,
Compute the loss in Eq. 16
Take numerical optimization step
5:
VorM To Yú.ss + nybus reduu V
on: VLLLO
6: else:
y6.M Yo.
8: until converged
7: end for
based on No.N- With diffusion schedule {38) (.1T (0, 1), the conditional distributions for the forward process at all other time steps can be defined as:
(yó.m M Bors )u.s)) ~ N (vô.M 1v1 11-B-uy Fou + - /I )0:M B'T)
(12)
In practical applications, we sample your directly from BoM with an arbitrary timestep t:
q (Bhas vo: )0:01 N (Bru.s Vatyl:M (1 Va yo:as (1 (1- vat)
(13)
1- Here, we define a' = 1- 3r and a² IT a² In Eq. 12 mean term, the diffusion process can be conceptualized as an interpolation between the true data y&.M and the conditional representa- tion Do:M- It commences with the true data yo and gradually transitions to Da.M- This approach effectively leverages the reliable conditional mean estimation Elyo. [ by transformers I (-). In the corresponding reverse process, initiated with yo containing information capable of accu- rately estimating Elvo: [..:N]. the generative process is significantly simplified. If the provided condition good enough, the model can then focus exclusively on uncertainty estimation.
Similar to many diffusion models designed for time series (Rasul et al. 202 la Shen & Kwok 2023), it is crucial for the reverse process to incorporate the conditional representation 90:M. Considering the forward process in Eq. 12 the corresponding manageable posterior for the forward process is:
q (yb.n Vo.M Vo. 9o.M) ~ N yo: 10080. + 71 71 + J20o: 3'1)
(14)
8'var-I I-a' (1-a"-1Wa ./2 + (Vat-)
3t 11-a"=12 Br
Yo
I-o'
The derivation can be find in Appendix D
3.3 HYBRID OPTIMIZATION
In this paper, we integrate the condition generative model and denoising model into a unified opti- mization objective. The condition generative model incorporates transformer T-)) structures and networks associated with the latent variable z. Within the diffusion model component, a denoising model is trained. As specified in Eq. TMDM's optimization objective is to maximize the evidence lower bound (ELBO) of the log marginal likelihood, formulated as:
log (y0.M ma.N) 2 log P Ww a Boiss (mo. ))
piu lo po (s)p(=) of
qfMa is Wa Mn.r--M.MM... (mo..))
O.T P(Woidd (o:ar)) al 1 90.x2) Eg(z (To [log p( (=)p(s)
= E Be In a [log
(
(15)
In Eq. [5] the first term, denoted Laimusion, guides the denoising model to predict uncertainty while subtly adjusting the condition generative model to offer more suitable conditional represen- tation. We view this as advantage of hybrid optimization. The second term, Leund, is introduced to maintain the capacity for accurate estimation of the conditional mean E(go: [ by the con- dition generative model. It also facilitates the generation of improved conditional representations by leveraging the capabilities of a well-designed transformer. Here, D (gllp) represents the Kull- back-Leibler (KI.) divergence from distribution distribution q. The aforementioned objective

---

Published as a conference paper at ICLR 2024
can be expressed as:
CELBO Ey(- log p(yuarr yours, (o.m)) Dec((( youus Do.ss )lp(( (o.ss)
+ Eو= Da (Vors BoLM, Vo.M Do.M) Ip(36:M M 16.M fo:m))
(16)
+ Eg(zl xol
=)] + (x I (xoxx) lip(z))
In Eq. 16 the first two rows originate from Lattiusion, while the last rows stem from Cooss As depicted the model parameters are optimized through stochastic gradient descent an end- to-end manner. The inference process is outlined in Algorithm
in Algorithm []]
EXPERIMENTS
4.1 EXPERIMENT SETUP
Dataset: Six real-world datasets with diverse spatiotemporal dynamics were chosen, compris- ing Electricity. ILI, ETT, Exchange, Traffic, and Weather. Table ! presents basic statistical infor- mation about these datasets. Further details can be found in Appendix
Table Summary of dataset statistics,
Dataset
Dimension
Freq.
Time sleps
Pred. steps
Exchange ILI
Day
7,588
192
Week
966
36
ETTm2
7
15 Min
69,680
192
Electricity
321
Houu
26,304
192
Traffic
862
Hour
17,544
192
Implementation details: In our experiments, we set the number of timesteps as T = 1000 and em- ployed a linear noise schedule with /² = 10-1 and ,r = 0.02, consistent with the setup in Ho et al. 22020). For the PICP, we selected the 2.5th and 97.5th percentiles. Therefore, an ideal PICP value for the learned model should be 95%. We employed 100 samples to approximate the estimated dis- tribution, and all experiments were repeated 10 times, with mean and standard deviation recorded.
Weather
21
10 Min
52,695
192
More details can be find in Appendix B
4.2
MAIN RESULT
4.2.1
BASELINES
We extensively compare our model with 14 baselines with different experiment settings. Including
diffusion-based time series models: TimeGrad (Rasul et al.
2021a), CSDI (Tashiro et al.
2021>
SSSD (Alcaraz & Strodthoff] 2022), VoEAE (L et al. 2022
and TimeDiff (Shen & Kwok
2023
Transformer- based models: Transformer-MAN (Rasul et al.
2021b). Transformer (Vaswani et al,
2017). Informer (Zhou et al. 2021), Autoformer [Wu al.
2021) and NSformer (Liu et al.)
2022);
VAE based models: VAE (Higgins et al.) 2016)., cST-ML (Zhang et al.) (0200) and DAC-ML (Zhang
al. 202T); ; and one additional well-designed method: GP-Copula (Salinas et al. 2019
4.2.2
QUALITATIVE ANALYSIS
To emphasize our distribution estimation capabilities, present the predicted median and visualize the 50% and 90% distribution intervals in Fig. 2 We compare TMDM with three other models: TMDM-min: A simplified version of TMDM that employs a basic transformer in the condition gen- erative model. TifeDiff: A recent time series prediction model that operates in non-autoregressive setting. However, it was primarily designed for point-to-point forecasting tasks, which may not pri- oritize probabilistic forecasting. TimeGrad: A well-known diffusion- autoregressive model.
Overall, TMDM demonstrates superior distribution estimation performance compared the other three models. While TMDM-min exhibits worse mean and uncertainty estimates than TMDM, we attribute this to the use of a different transformer the condition generative model. The NSformer employed in TMDM is more powerful for mean estimation, facilitating better overall distribution estimation. TifeDiff, designed for point-to-point forecasting tasks, generates distribution intervals with varying width. Consequently, sample points generated farther away from the true value are sparse. In challenging scenarios (columns 3, 4, and 5), the 50% distribution intervals abruptly expand due the absence of points in the middle section. This highlights the limitation of point- to to-point forecasting in capturing real multivariate time series data, making it less practical in real- world applications. TimeGrad, which relies on RNN to capture timing information, exhibits

---

Published as a conference paper at ICLR 2024
1.5
0.5
3
1.5
0.5
TMDM
1
-0.5
2
-0.5
0.5
0.5
-1.5
1
-1.5
0
200
230
0
200
200
200
1.5
0.5
3
1.5
0.5
-0.5
2
-0.5
0
0.5
0.5
-1.5
1
-1.5
200
200
200
200
200
ThoDiir
1.5
-0.5
0.5
3
2
1.5
-0.5
0.5
0
0.5
0.5
-1.5
1
-1.5
200
200
200
200
200
ThnColl
1.5
0.5
3
1.5
0.5
-0.5
2
-0.5
0
-1
0.5
-1.5
1
-1.5 0
200
200
0
200
200
200
Figure 2: Comparison of prediction intervals for the Exchange dataset. We display the predicted median and visualize the 50% and 90% distribution intervals, the black line representing the test set ground-truth.
Table 2: Performance comparisons six real-world datasets in terms of QICE and CRPS. The best results are boldfaced.
Duteet
Exchange
Electricity
Traftic
Weather
Melhad
OICE
QICE
CRPS
QICE
CRPS
QICE
CRPS
QICE
CRPS
CRPS
VAF.
8.28
1.00 0.112
9.3310.92
2:41.0
89910
0.79
199.000 0009
7.0400000
0.110 162
5.3710199 WY 0.67
10.319
9.07
0.099
157
799231996
0.94:6.244
9,0260519
19430.99
7,29H0
161
0:4300111
5.9900157
5:24301154
0,00:01155
8,29:0:00
0.119012
DAC-ML
7,3641100
0.55.0.193
8.71:0355
1.23300114
660111191
0.99...00
5.76004153
14330190]
43314416
0,50
0.91
0446001155
GIMCspuls
0.58
0:301
143100.0
6851000
0.50.0.87
5.85:0191
016101000
48200016
0.53
8.10
08410228
Truusomer MAP|
0.71
8.19500669
(19550199
50000.10
0.66
5:600.11
1326
3.55000000
0.45
7,40:00711
155
TineChul
5.2205226
7.8651125
5.7780187
3.34500199
170
0,39:00:166
04480011
SSSD
62000500
0,56-0.114
7,60-0.799
0.9450
M95
4,8840
166
0,57=8.79|
| 5.26600111
0.4640
26]
388500100
0,41=0.114
7.19:0674
0:4-201125
CSDE
5,9950500
7,7560389
1:1051144
517780152
0,3088012
4, 4,70
[1220004
3.50880:71
0,37:00321
5:34:00000
0.3750086
D'VAE
|13,11086 0:11:00099
15,8260.190
(19728.022
13,4820.11
0,44
| 13.41a.117
0.3630
120
12,00:0.113
0,31
14,6455000
0.39550.25
TineDair
13:34:00m
0.38080005
15:5011005
10531006
14:22:10154
0:40:00005
12.74300:0
0.381007
13.53
0:28:00115
13:18:00095
0.33:omm
OUES
4.38:4417
0,32:000066
6,74:0002
0.9250091
175
0.37=3000
3.81
0.3360085)
2.36501117
0.36=00.911
3.87:00681
0.36ee0m3
poor performance when estimating longer time series. For more detailed results, please refer to the Appendix D and F
4.2.3 QUANTITATIVE COMPARISON
Probabilistic multivariate time series forecasting: To assess the performance of TMDM in prob- abilistic multivariate time series forecasting, we applied our model to six datasets alongside several competitive baselines. Four metrics (QICE, PICP, CRPS, and CRPS.un) were employed as perfor- mance indicators for probabilistic forecasting. Additionally, we used two metrics (MSE and MAF) to evaluate other aspects of model performance. As shown in Table 2] the NSformer integrated with framework consistently achieved state-of-the-art (SOTA) performance across all bench- mark datasets. Notably, when compared to previous SOTA results, TMDM achieved a remarkable 17% reduction in QICE (from 5.32 to 4.38) for the Exchange dataset, 11% reduction (from 7.6 to 6.74) for ILI, 23% reduction (from 4.88 to 3.75) for ETTm2, 27 % reduction (from 5.26 to 3.81) for Electricity, 32% reduction (from 3.5 to 2.36) for Traffic, and 24% reduction (from 5.14 to 3.87) for Weather. It is worth noting that models like DDVAE (Li et al. 2022) and TimeDiff (Shen & Kwok, 2023] were originally designed for point-to-point forecasting tasks, where probabilistic forecasting was not their primary focus. Consequently, their performance on QICE and PICP was suboptimal. However, they still demonstrated competitive performance in CRPS, CRPS sum (see Appendix (). MSE, and MAE (refer to 3. As CRPS may not effectively evaluate the quality of distribution range, this underscores the importance of introducing new metrics (QICE and PICP) in the evalua- tion of probabilistic multivariate time series forecasting models. For additional results, please refer to the Appendix C
8

---

Published as a conference paper at ICLR 2024
Table 3: Performance comparisons on six real-world datasets regarding MSE and MAE. The best
results are boldfaced.
Datner
Exchange
bTTm2
Elearkyy
Tratfi
Weather
Method
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAB
MSE
MAE
MSE
MAE
TineGraa
223
0.000023 [255
1.150012
1.3550133
0.742022
0,74:0109
0.66501144
0.81:0141
0.9301199
0.57:50136
1.7703122
0.750000s
2.5500955
12130016
1.850094
0.6750004
0.5619212
0068
0.3631005
0.5650IMS
SSSI
100000011
0.6614127
2..52.0.11
1.8.00311
0871000
0.56
04718122
0,00.0.300
0,50.0.13
067..0119
D"VAE
0.76500118
24455.115
11100117
0,7050008
014600004
0.33550
0.9939119
0,79500122
0:43500199
0,34550131
TineDulf
0,48
0.43230100
22.46:0.145
1.09:0004
041300m4
0.42500033
0.27
0.32:9131
0.6350113
0.4780052
0.36230.16
0.37:00822
0,26
0.37.10015
1.99 ceusss
8J85 Lu0ze
0.271
0.35 100s
1019
0.27 2000
0.00
0.28:
0.25 (0000
Ablation study: component within our proposed framework, we conducted a comparative analysis of pre- diction results across three datasets using five models. Presented in Table 4 MLP-cond and Autoformer-cond serve as baseline mod- els, employing simple MLP or Autoformer in the condition generative model of TMDM. (0, -)-Prior leverages NSformer to generate conditional embeddings while assuming yo-u ~ (0, I) as a prior. When comparing our proposed model, TMDM, with MLP-cond and Auteformer-cond, we observed substantial improvement, achieving an average QICE reduction of 42% and 23% respectively. This emphasizes the effec- tiveness of utilizing representations captured by existing well-designed transformer-based models as conditions. Furthermore, it demonstrates our capability to seamlessly integrate advancements in transformer structures for point estimation into the domain of distribution estimation. Comparing (0,,) -Prrio with TMDM, we noted an average 19% reduction in QICE, highlighting the advan- tage of considering condition information as prior for both the forward and reverse processes. Finally, comparing No-hybrid with TMDM, we observed significant 16% reduction in QICE, un- derscoring the effectiveness of our proposed hybrid optimization method.
To assess the impact of each
Table 4: QICE and CRPS scores for different variants of the TMDM
Datasct Metric
Exchange QICE CRPS
ETTm2
Wether
QICE CRPS
QICE
CRPS
MLP-cond
7.86
0.64
9.61
1.14
5.06
0.50
Autoformer-cond
6.77
0.40
4.76
0.56
4.55
0.46
Mo P)-rrior
5.44
0.46
5.48
0.57
4.23
0.44
No-hybrid
5,96
0.38
4.47
0.46
4.18
0.41
TMDM
4.38
0.32
3.75
0.37
3,87
0.36
Table 5: Performance promotion by applying the proposed framework to transformer and its vari-
ants.
Datset
Exchange
(36)
ETTm2
Electneity
Tratfic
Weather
Method
MSE
MAE
MSE
MAE
MSE
MAE
MAE
MSE
MAE
MSB
MAE
TraasSormes
1.20::0:19
0.5400001
497
1.840000
1476612214
2.76:0226
0.2610016
0.36100.155
06750014
0:36:0016
0.577n045
0.53:0024
-Ours
116:90025
0,82190015
1.1890066
0,279000
0.3314.009
0.62:1009
0:33:00010
0.55:100S
0.4920015
Informer
1.31 0.8540021
573
1.59
4774400445
19
0.35
0.43+9.000
0.75-00177
0:12:00019
0:48:nu7
Ouus
L:1::0055
0,3320.014
4.22:0089
1.49:0.041
1:2500264
0:90:8.096
033410.010
0,4014.0133
0.69:00007
0,00:00011
0,41:0044
0.45:0027
Autofonmes
0.44:0 0,48:00082
1:20:::150
1:25
0047760.82
0.46::0.021
0.22:00009
0.334-0.015
0.05:0044
0:41:000
0.311:0015
+ Ours
8.43:10177
0.471000166
3003:001111
1:1819000
0.33319022
0.40::0018
0.20:4..00
0.29116.016
0:63:11012
10014
0.31:8001
0.34:0m9
NSlurner
0.25-00088
0.36100001
1993:0117
0.87:0000s
(0.53
0.48::00155]
0.18::0.012
0.28:.0011
0.62:00155
0.34:00133
0.25
0:20:00000
+ Ours
0:26 0.37190015
1999200085
0.851880026
(.277190023
0.35::0.155
0119198807
0.27:18.005
0.601-00008
0:35:00009
0.20000010
0:27:00009
Framework generality: Diffusion models have gained widespread attention owing to their capac- ity to generate high-dimensional data and their training stability (Han al.. 2022). In the context of point-to-point forecasting tasks, TMDM operates as a versatile framework that enhances train- ing stability when paired with various transformers. We apply our framework to four prominent transformers, showcasing the performance enhancements achieved by each model in Table Our method consistently reduces variance across multiple experiments and enhances the point-to-point forecasting ability of most transformers.
5 CONCLUSION
In this paper, we present TMDM, an innovative framework that merges diffusion generative process with existing well-designed transformer models. Our approach leverages the strengths of transform- ers, particularly their accuracy in estimating conditional means, and extends this capability as priors across both forward and reverse processes within the diffusion model. By employing this estimated mean as the condition, the diffusion model can more effectively focus on estimating uncertainty, simplifying the generative process. TMDM stands out as versatile plug-and aad-payy framework, ef- fectively closing the gap between point estimates and distribution estimates. It enables seamless integration with advanced transformer models for point estimation, promising even better forecast- ing accuracy. We introduce two novel evaluation metrics, enriching the toolbox for assessing uncer- tainty in probabilistic multivariate time series forecasting models. Our comprehensive experiments on six real-world datasets consistently demonstrate TMDM's superior performance, underscoring its effectiveness in enhancing probabilistic prediction quality.

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGMENTS
This work was supported in part by the National Natural Science Foundation of China under Grant U21B2006; in part by Shaanxi Youth Innovation Team Project; in part by the Fundamental Research Funds for the Central Universities QTZX23037 and QTZX22160) in part by the 111 Project under Grant B1 81833; The work of Wenchao Chen acknowiedges the support of the stabilization support of National Radar Signal Processing Laboratory under Grant (JKW202XOX) and National Natural Science Foundation of China (NSFC) (6220010437).
REFERENCES
Juan Lopez Alcaraz and Nils Strodthoff. Diffusion-based time series imputation and forecasting with structured state space models. Transactions Machine Learning Research, 2022.
Chris Chatfield. Time-series forecasting. 2000.
Charles J Geyer. Practical markov chain monte carlo. Statistical science, pp. 473-482 483,483,14 1992.
Tilmann Gneiting and Adrian E Raftery. Strietly proper scoring rules, prediction, and estimation. Journal of the American statistical Association, 102(477):359-375 2007.
Alexandros Graikos, Nikolay Malkin, Nebojsa Jojic, and Dimitris Samaras. Diffusion models as plug-and-play priors. Advances in Neural Information Processing Systems, 35:14715-14728, 2022.
Xizewen Han, Huangjie Zheng, and Mingyuan Zhou. Card: Classification and regression diffusion models. Advances in Neural Information Processing Systems, 35:18 1100-181111. 2022.
Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. beta-vae: Learning basic visual concepts with constrained variational framework. International conference on learning representations, 2016.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in
neural information processing systems, 33:6840-6855 2020.
Aapo Hyvärinen and Peter Dayan. Estimation of normmoomal statistical models by score match- ing. Journal of Machine Learning Research, 6(4), 2005,
Jacob Devlin Ming-Wes Wei Chang Kenton and Lee Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. Proceedings of NAACL-HILT, pp. 4171- 4186, 2019.
Kyoung-jac Kim. Financial time series forecasting using support vector machines. Neurocomputing, 55(1-2):307-319 2003.
Guokun Lai, Wei-Cheng Chang, Yiming Yang, and Hanxiao Liu. Modeling long-and short-term temporal patterns with deep neural networks. The 4/ international ACM SIGIR conference on research & development in information retrieval, pp. 95-104, 2018
Yan Li, Xinjiang Lu, Yaqing Wang, and Dejing Dou. Generative time series forecasting with dif- fusion, denoise, and disentanglement. Advances in Neural Information Processing Systems, 35: 23009-23022 2022.
Yong Liu, Haixu Wu, Jianmin Wang, and Mingsheng Long. Non-stationary transformers: Exploring the stationarity in time series forecasting. Advances in Neural Information Processing Systems, 35:9881-9893, 2022.
James E Matheson and Robert L Winkler. Scoring rules for continuous probability distributions. Management science, 22010):0077-1096 1976.
Radford M Neal et al. Meme using hamiltonian dynamics. Handbook of markov chain monte carlo, 2(1 1):2, 2011.
10

---

Published as a conference paper at ICLR 2024
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high- performance deep learning library. In NeurIPS, 32, 2019.
Kashif Rasul, Calvin Seward, Ingmar Schuster, and Roland Vollgraf. Autoregressive denoising diffusion models for multivariate probabilistic time series forecasting. International Conference on Machine Learning. pp. 8857-8868. 202
Kashif Rasul, Sadul-Saboor Sheikh, Ingmar Schuster, Urs M Bergmann, and Roland Vollgraf. Mul- tivariate probabilistic time series forecasting via conditioned normalizing flows. International Conference on Learning Representations, 2021b.
Kashif Rasul, Young-Jin Park, Max Nihlén Ramström, and Kyung-Min Kim. Vq-ar: Vector quan- tized autoregressive probabilistic time series forecasting. arXiv preprint arx5: 2022.
David Salinas, Michael Bohlke-Schneider, Laurent Callot, Roberto Medico, and Jan Gasthaus. High-dimensional multivariate forecasting with low-rank gaussian copula processes. Advances in neural information processing systems, 32, 2019.
Nicholas Sapankevych and Ravi Sankar. Time series prediction using support vector machines: survey. IEEE computational intelligence magazine, 4(2):24-35 38, 2009.
Lifeng Shen and James Kwok. Non-autoreegressive conditional diffusion models for time series prediction. International Conference on Machine Learning, pp. 6657-6668, 2023.
Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. International conference machine learning, pp. 2256-2265, 2015.
Yang Song, Jascha
Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. International Conference on Learning Representations, 2020.
Yusuke Tashiro, Jiaming Song, Yang Song, and Stefano Ermon. Csdi:
Conditional score-based diffusion models for probabilistic time series imputation. Advances in Neural Information Pro- cessing Systems, 34:24804-24811 2021.
Toan Tran, Thanh- Toan Do, Ian Reid, and Gustavo Carneiro. Bayesian generative active deep learn- ing. International Conference on Machine Learning, pp. 0295-6304, 2019.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in neural informa- tion processing systems, 30, 2017.
Pascal Vincent. A connection between score matching and denoising autoencoders. Neural compu- lution, 23(7):1661-1674, 2011.
Zhiyuan Wang, Xovee Xu, Weifeng Zhang, Goce Trajcevski, Ting Zhong, and Fan Zhou, Learning latent seasonal-trens representations for time series forecasting. Advances in Neural Information Processing Systems, 35:38775-38787, 2022.
Max Welling and Yee W Teh. Bayesian learning via stochastic gradient langevin dynamics. Proceed- ings of the 28th international conference on machine learning (ICML-11), pp. 681-688, 2011.
Haixu Wu, Jiehui Xu, Jianmin Wang, and Mingsheng Long. Autoformer: Decomposition trans-
formers with auto-correlation for long-term series forecasting. Advances in Neural Information Processing Systems, 01-22419-22433 2021.
Zhisheng Xiao, Karsten Kreis, and Arash Vahdat. Tackling the generative learning trilemma with denoising diffusion gans. International Conference on Learning Representattons,, 2021.
Jiyy Yao, Weiwei Pan, Soumya Ghosh, and Finale Doshi-Ved Velez. Quality of uncertainty quan- tification for bayesian neural network inference. CoRR, abs/1906.09636, 2019. URL http:
11

---

Published as a conference paper at ICLR 2024
Xin Zhang, Yanhua Li, Xun Zhou, Oren Mangoubi, Ziming Zhang, Vincent Filardi, and Jun Luo. Dac-ml: domain adaptable continuous meta- learning for urban dynamics prediction. 2021 IEEE International Conference Data Mining (ICDM), pp. 906-915, 2021.
Yingxue Zhang, Yanhua Li, Xun Zhou, and Jun Luo. cst-ml: Continuous spatial-temporal meta- learning for traffic dynamics prediction. In ICDM), pp. 1418-14135 1423, 2020.
Huangjie Zheng. Truncated diffusion probabilistic models and diffusion-based adversarial auto- encoders. Advances in Neural Information Processing Systems, 36(2022), 2022.
Haoyi Zhou, Shanghang Zhang. Jieqi Peng, Shuai Zhang, Jianxin Li, Hui Xiong, and Wancai Zhang. Informer: Beyond efficient transformer for long sequence time-series forecasting. Proceedings of 2021.
the AAAI conference artificial intelligence,
A DATASETS
Six real-world datasets with diverse spatiotemporal dynamics are selected, including: (1) Electricin
records the hourly electricity consumption of 321 clients from 2012 to 2014. (2) 11. collects the ratio of inftuenzallike illness (II.I) patients versus the total patients in one week, which is reported
weekly by the Centers for Disease Control and Prevention of the United States from 2002 to 2021.
2021) contains the data collected from electricity transformers, including (3) ETT (Zhou al.
load and oil temperature that are recorded every 15 minutes between July 2016 and July 2018. (4) Exchange al., 2018) records the daily exchange rates of eight different countries ranging from 1990 to 2016. (5) Traffid]- contains hourly road occupancy rates measured by 862 sensors on San Francisco Bay area freeways from January 2015 to December 2016. (6) Weather includes meteorological time series with 21 weather indicators collected every 10 minutes from the Weather Station of the Max Planck Biogeochemistry Institute in 2020. Table [ provides basic statistical information about these datasets.
B IMPLEMENTATION DETAILS
In our experiments, we set the number of timesteps as T = 1000 and employed a linear noise schedule with 3 = 10-4 and ST 0.02, consistent with the setup in Ho al. 22020). The latent states were configured to have dimension of 512 for all datasets. For the diffusion model, we adopted a simplified network architecture compared to prior work Xiao et al. (2021)) I): Zheng (2022). Initially, we replaced the transformer's sinusoidal position embedding with linear embedding for the timestep. We concatenated y0 A and zj0 : . and passed the resulting vector through three fully-comnected layers, each with an output dimension of 128. We performed Hadamard product between each output vector and the corresponding timestep embedding, followed by a Softplus non-linearity, before forwarding the resulting vector the next fully-connected layer. Finally, applied a fourth fully-connected layer to map the vector to a one-dimensional output for the forward diffusion noise prediction. We utilized the Adam optimizer with a learning rate of 0.0001 and batch size of 32. All experiments were implemented in PyTorch (Paszke et al. 2019) and conducted on an NVIDIA RTX 3090 24GB GPU. The prediction length can be found in Table The input sequence length for ILI is set to 36, while for other datasets, it is set to 96. For the Prediction Interval Coverage Probability (PICP), we selected the 2.5th and 97.5th percentiles. Therefore, an ideal PICP value for the learned model should be 95%. We employed 100 samples to approximate the estimated distribution, and all experiments were repeated 10 times, with mean and standard deviation recorded.
C MORE QUANTITATIVE RESULT
To assess the performance of TMDM in probabilistic multivariate time series forecasting, we applied our model to six datasets alongside several competitive baselines. Four metrics (QICE, PICP, CRPS,
hhttpss//rchivc..ic.m ics. uci htt.s//www....oooaoooooeoceeeaaa htt.://ccrriityoooooaaaaaaamaa1a 12014
htttp::/gis.cdc..gv/ www.yaay/hhiic/hhhpnasssssaaaaam
hhtp::/ppmms.dot.oom wow.:aam.ddoooooooooooooo gov/
hhtps://www.heg
12

---

Published as a conference paper at ICLR 2024
and CRPS)um) were employed performance indicators for probabilistic forecasting. Additionally, we used two metrics (MSE and MAE) to evaluate other aspects of model performance, As shown and Table 7 the NSformer integrated with framework consistently achieved state- of-the-art (SOTA) performance across all benchmark datasets. Notably, when compared to previous SOTA results, TMDM achieved remarkable 3.42 (from 71.12 to 74.54) improvement in PCIP for the Exchange dataset, 8.26 (from 79.57 to 87.83) improvement for ILI. 0.62 (from 72.58 to 73.2) improvement for ETTm2, 3.01 (from 79.34 to 82.35) improvement, b[i] for Electricity, 2.53 (from 84.3 to 86.83) improvement for Traffic, and 6.05 (from 66.92 to 72.97) improvement for Weather. It worth noting that models like VAE (Li et al. 2022) and TimeDiff (Shen & Kwok, 2023) were originally designed for point-to-por to-pont forecasting tasks, where probabilistic forecasting was not their primary focus. Consequently, their performance on QICE and PICP was suboptimal. However, they still demonstrated competitive performance in CRPS, CRPS (see Appendix ( MSE, and MAE (refer to Table B). This underscores the importance of introducing new metrics (QICE and PICP) into the evaluation of probabilistic multivariate time series forecasting models.
in Table and
Table 6: Performance comparisons on six real-world datasets in terms of CRPS-sum The best results are boldfaced.
Method
Exchange
ILI
ETTm2
Electricity
Truffic
Weather
VAB 7.19=0.178
9.84=02289
7,91
9.56+0.235
7.65=0.119
8.98+0.144
Gi-Coopuaa
4.78-=0.02
7.87-0.148
4.53440.277
8.36+-0.18
6.29-00112
6.98-+0.099
Transormer-MAF
10.162
6,28.10248
3.57.10330
6.27.10.177
1..110
6.19110281
TimeGrad
3.92:50233
6,06=02212
3.134:0227
5,46100111
4.67
6.01
CSDI
4.20=0.153
6.98=0.065
3.07=20130
4.5420.368
3.98=0.136
5.39=0.183
SSSD
3,31
6,30=0.127
2.28+00061
4,50=0:297
3,10=0.300 308
4.94=00238
D`VAE
2.05=00197
6.50=0.138
1.93:0.066
3.80=0.290
3.23=0.136
3.82:0.196
TimeDiff
2.46=0:342
7.01=0.126
1.9240.019
3.08#0.151
2,47=0.200
3. 16:2.777
1.87=00072
6.02=0.109
1.75400044
2887++0.18
2.10=00:885
1.79:+0.086
Table 7: Performance comparisons on six real-world datasets in terms of PICP. The best results are boldfaced.
Method
Exchange
ILI
ETTm2
Electricity
Traffic
Weather
VAE
64.62433.00
68.31:2.365
67.18111648
70.56:2.362
78.50a55538
58.92:33:15
GP-Copula
66.86:28.737
70.40:33223
70.9415.104
70.89 70.89:44881
80.3383.780
60.72:6.92
Transfommer MAF
10.766
73.68:66.853
71.23
13.8911.111
1.59
61.30.4..185
TimeGrad
69.16-10.254
74.2916.325
71.62155999
182
82.281111102
62.79117962
CSDI
69.2119.997
18.182249932
71.78122666
78. 78.9413.554
83.51226.983
62.711:4.832
SSSD
71,1247.330
79.5743.366
72.58447736
79. 19.344551166
84.30-44721
66.92455.22
D'VAE
21.38:15.034
5.203.0.723
13:20
33.96111411
7.12442.82
20.44:4:.575
TimeDDif
20.80:17.528
3.69110:620
13.16111.46
32.
9.11
21.6011.348
74.54:12016
87.83:772155
73.20
82.35
86.83:24.027
72.97.12.583
Evaluating model performance with time-series data at different granularities of significant impor- tance in real-world applications. Our proposed TMDM can excel in such scenarios for the following reasons:
1. Similar to most time series forecasting models Zhou et al. 2222 Wu et al. 12221 Liu et al. (2022), we incorporate the actual timestamps as learnable time embeddings for each data point. Leveraging well-designed Transformer, can effectively capture the temporal correlations within the data. This design ensures that TMDM can adapt to various granularities of time-series data.
2. Within the diffusion model component, also account for the time embedding in the data. This allows the model to generate multivariate time series with information from these embeddings, accommodating time series data at different granularities.
3. As shown in Table 1 in our paper. the selected datasets covered different granularities ranging from 10 minutes to 1 day, and TMDM demonstrated competitive performance across all databases.
13

---

Published as a conference paper at ICLR 2024
This confirms the model's ability to handle situations where time-series data is available different granularities.
4. To further evaluate TMDM's ability to handle varying time-series granularities, conducted an experiment where we randomly removed data points from a given time series Vo:M and used this modified dataset to test TMDM with the same settings described in the paper. This challenging experiment simulates scenarios where the time intervals in multivariate time series vary, making it rigorous test of the model's performance under changing granularities.
Table 8: Performance of TMDM on the time-series in different granularities.
Dataset Metric
Exchange QICE CRPS
ETTm2
Electricity QICE CRPS
Traffic
Wether
QICE CRPS
QICE CRPS QICE CRPS
TMDM-60
4.78
0.32
3.72
0.39
3.73
0.31
2.36
0.28
3.91
0.37
TMDM-30
4.14
0.30
3.93
0.38
3.85
0.35
2.34
0.25
3.79
0.34
TMDM
4.38
0.32
3.75
0.37
3.81
0.33
2.36
0.26
3.87
0.36
As depicted in the table above (Table (8. TMDM-60 denotes our model with prediction length of 192 + 60, where we randomly remove 60 samples from the data. In this setting, TMDM is challenged to forecast multivariate time series (MTS) with varying granularities based the time embedding. The results obtained from the 30 and 60 settings exhibit similar scores compared to the original setting, demonstrating the effectiveness of the proposed TMDM in accommodating time-series data with differing granularities.
The variation in the CRPSsum results of the baseline models compared the published results is primarily due to differences in the experimental settings, specifically concerning the history length and prediction length. We have included the experiments on the settings TimeGrand and CSDI in our paper, and the results can also be found as follows (Table and Table I0])
Table 9: TimeGrand setting result.
Dataset Metric
Exchange QICE CRPS
Electricity
Traffic
QICE CRPS
QICE CRPS
TimeGrand
3.63
0.006
2.51
0.0206
1.91
0.044
TMDM
2.48
0.004
0.016
1.07
0.013
Table 10: CSDI setting result.
Datasct Metric
Exchange QICE CRPS
Electricity
Traffic
QICE CRPS S.om
QICE CRPS
TimeGrand
2.59
0.021
2.02
0.044
CSDI
0.020
TMDM
1.36
0.014
1.23
0.015
CSDI occasionally matches slightly surpasses TimeGrand, yet TMDM consistently outperforms both models across various datasets and metrics. TMDM showcases superior performance in prob- abilistic forecasting, reflected in its lower QICE and CRPSSUM values. These results emphasize TMDM's 's significant advancements in predictive accuracy and distributional modeling compared to TimeGGand and CSDI across diverse datasets and evaluation metrics.
14

---

Published as a conference paper at ICLR 2024
D DERIVATION FOR FORWARD PROCESS POSTERIORS:
In this section, we derive the mean and variance of the forward process posteriors q (Vord BU.M Pi. Do:As) in Eq. 14
q (yond Vord M DO q (Vo.m
Voote Bo.M) q (youM yo.. Bo.M)
x expl 1/2 (ybrr (1 - v Vat)
yo:
gr
(whin - Vaa
1aa--yy0.rr - (1 Vaa ))00))))
yo:M
1 --a--1
ā' 1-1 -2Vā'(ziu - (1 vāl
a'(Yo:M)
yo )
α explp 3
2
BB
(2bi- - 2(vat=Iyh
yu.M + (1 vaa
y0. Mo.M
yo:M ))
a
= =xxp( 2 (A11866:1) -2A23o.M)
where
A a²(1 a² a111)+ +
1- a²
3'1- 1 aa
B¹ - a²-1)
Va--7 I
Va
Vā'(Va' -1)
V a!-1
A
a' U
+
8r yous +
Br
19
and we have the posterior variance:
3" = 1/A1 (1-a--1
ge
at
Mcanwhile, the following coefficients of the terms in the posterior mean through dividing each coefficient in A2 by A
Va-- I
8'-a''
70 1- a-a
/A1
1 a
V@
a1-a--1) Va
/A
71
a²
Vā'(Vā' -1) 1 1-va--1 Va--1
72 gr.
1- - )/A1
ā'' a¹ Vo'((1 - r-11) 1) + Br - -3' Va'
1 1-1
=1 (Val 1)(V vo + Va--11
1-
which together give us the posterior mean
pel plyb vh.u- 20.M) = 7o86-ar + 713.. + 7290.
15

---

Published as a conference paper at ICLR 2024
2
0
50
100
150
200
100
200
300
100
200
300
400
200
400
600
600
Prediction length: 96
Prediction length: 192
Prediction length: 336
Prediction length: 720
Figure 3: The prediction intervals in different predict lengths of the proposed TMDM
E DIFFERENT PREDICT LENGTHS COMPARISON OF PREDICTION INTERVALS
As shown in Fig. B] have presented prediction intervals generated by TMDM for various predic- tion lengths. In this experiment, we maintained a consistent history length, and it becomes evident that the prediction intervals widen the prediction length extends. This indicates TMDM's abil- ity to provide different levels of uncertainty when dealing with more challenge prediction tasks, valuable characteristic for real-world applications.
F MORE COMPARISON OF PREDICTION INTERVALS FOR THE EXCHANGE AND WEATHER DATASET
Fig. 4 Fig. 5 and Fig. 6 offer further comparisons of prediction intervals for the Exchange and Weather Datasets. These figures include examples of both successful and challenging prediction cases.
16

---

Published as a conference paper at ICLR 2024
3.5
1.5
IMDM
3
2
0.5
200
200
200
0
200
0
200
45
TinDE
1.5
3
2
0.5
200
200
0
200
0
200
0
200
2
4.5
4
ThnaCoal
1.5
2
2
0.5
1.5
2.5
0
200
200
200
0
200
0
200
1.5
4
0.5
TMDM
1
3M
-0.5
0.5
0
200
200
0
200
0
200
0
200
1.5
1
3M
0.5
2
-1.5
200
200
200
0
200
0
200
ThnQral
1.5
4
0,5
-0.5
1
1
3 w
-0.5
0.5
0
0
2
200
200
0
200
0
200
0
200
3.5
1.5
1.5
IMDM
2
25
0.5
200
0
200
0
200
0
200
200
3.5
1.5
25
1.5
0.5
200
200
200
0
200
200
3.5
TinaColl
3.5
1.5
1.5
2,
2.5
0.
1.5
0.5
-0.5
200
200
200
0
200
200
Figure 4: More comparison of prediction intervals for the Exchange Dataset. We display the pre- dicted median and visualize the 50% and 90% distribution intervals, the black line representing the test ground-truth.
17

---

Published as a conference paper at ICLR 2024
0.5
0.5
1.5
1
-0.5
0.5
0
200
200
200
200
200
0.5
0.5
1.5
1
-0.
0.5
0
0
200
200
200
0
200
200
0.5
0.5
1.5
1
-0.5
-0.5
1
0
0.5
-1
-1.5
0
0
200
0
200
0
200
0
200
0
200
0.5
1.5
1.5
1
0
-0.5
0.5
0
0.5
-1.5
-0.5
0
0
200
O
200
0
200
0
200
200
0.5
1.5
1.5
1
-0.
0.5
0
0.5
-0.5
0
0
200
0
200
0
200
0
200
200
0.5
1.5
1.5
1
-0.5
0.5
0
0.5
-0.5
-1
-1.5
0
0
200
0
200
0
200
0
200
200
1
3.5
2
2
0.5
1
ww
2.5
0
0
1.5
0.5
0
200
0
200
200
200
200
3.5
2
2
0.5
1
2.5
0
0
ww
1.5
-05
0
200
0
200
200
200
200
3.5
2
2
0.5
1
2,5
0
0
1.5
-1
-0.5
-1
-1
0
200
0
200
0
200
0
200
200
Figure 5: More comparison of prediction intervals for the Weather Dataset. We display the predicted median and visualize the 50% and 90% distribution intervals, the black line representing the test set ground-truth.
18

---

Published as a conference paper at ICLR 2024
2
1
2
0.5
1.5
0
0
0.5
-2
-1
-0.5
-0.5
0
200
0
200
200
200
0
200
2
1
0.5
1.5
0.5
-2
-1
-1
-0.5
-0.5
0
200
0
200
200
200
0
200
1.5
2
0.5
0
0
0
0.5
-2
-1
-1
-0.5
-0.5
0
200
0
200
200
200
0
200
0.5
0
-0.5
0.5
0
-1.
-055
-0.5
0
200
200
200
200
0
200
-0.
0.5
0
0.5
0
-055
-2
-0.5
0
200
0
200
200
200
0
200
0.5
0
1.
N
N
0.5
0
-1
-0.5
-2
-05
0
200
0
200
0
200
0
200
0
200
-0.5
0.5
0.5
0.5
-0.5
-1.5
0
-0.5
-1.5
-2.5
-0.5
-1.5
200
200
200
0
200
0
200
0.5
-0.5
0.5
0.5
-0.
0
-1
0
-0.5
-1.5
-2.5
-055
-1.5
200
200
0
200
0
200
0
200
0.5
-0.5
0.5
0.5
-0.
0
-1.5
0
-0.5
-.5
-2.5
-0.5
-1.5
0
200
0
200
0
200
0
200
0
200
Figure 6: More comparison of prediction intervals for the Weather Dataset. We display the predicted median and visualize the 50% and 90% distribution intervals, the black line representing the test set ground-truth.
19


Reviews:
[{"review_id": "InFgjxhoOH", "reviewer": "Reviewer_YJEP", "summary": "In this paper, the authors introduce a Transformer-Modulated Diffusion Model (TMDM), uniting conditional diffusion generative process with transformers into a unified framework to enable precise distribution forecasting for MTS. Extensive experiments are conducted", "questions": "Please address the questions above.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This paper is well-presented and well-organized.\n2. The paper introduces a Transformer-Modulated Diffusion Model (TMDM), uniting conditional diffusion generative process with transformers into a unified framework to enable precise distribution forecasting for MTS. \n3. Extensive experiments are conducted", "weaknesses": "1. This paper states many existing work did not consider the uncertainty of data, but more SOTA should be compared and considers like cST-ML which tries to capture traffic dynamics with VAE. Please provide detailed explanations or experiments accordingly.\n2. If the time-series data is in different granularities, does this model still work?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "hLrcsBCbSR", "reviewer": "Reviewer_SgLT", "summary": "The paper addresses the problem of probabilistic time series\nforecasting. The authors propose a conditional diffusion\nprocess that convex combines the point estimate from\na transformer model with the noise of the diffusion process.\nIn experiments on 6 datasets they show that their approach\noutperforms state-of-the-art baselines.", "questions": "The paper proposes a generic approach to wrap a diffusion\nmodel around any point estimate model for time series\nforecasting to make it probabilistic (s1). Most of the results\nshown in tab. 2 are pronounced improvements and almost\nall but the very last are consistently better than several\nstrong baselines (s2). Due to the ablation studies one can\nclearly see the impact of different modelling choices (s3).\n\nI only would like to discuss one point:\nw1. the results for CRPS_sum of the baselines (appendix, tab. 6)\n  varies from the published results.\n- e.g., tab. 6 reports CRPS_sum 3.92 on Exchange for TimeGrad,\n  but the TimeGrad paper reports 0.006.\n  your tab. 6 reports CRPS_sum 4.54 on Electricity for CSDI,\n  but the CSDI paper reports 0.017.\n- I think these differences need to be clearly explained, likely\n  due to different experimental conditions?\n  If so, it would be convincing to reproduce the experiments\n  of the strongest baseline papers and compare them on\n  the published settings, too.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "s1. generic approach that will work with any point estimate model.\ns2. consistently good results against several strong baselines.\ns3. ablation studies demonstrate the impact of different components\n  as well as the ability to wrap different point models.", "weaknesses": "w1. the results for CRPS_sum of the baselines (appendix, tab. 6)\n  varies from the published results.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "iicEvqcmJQ", "reviewer": "Reviewer_vL8K", "summary": "The manuscript proposes a method for multivariate time series forecasting. It utilises the (conditional) diffusion model, while starts from a \"condition\" which is from the output of a Transformer based model.  The advances of the proposed model is measured with (mainly) QICE and CRPS. MSE and MAE, as well as some qualitative results are also provided to show the effectiveness of the method.", "questions": "- can you provide more details on generating  $\\hat{y}_{0:M}$? What is the dimension of $z$? As in eq(10), is $z$ a scalar? How are the NNs for $\\tilde{\\mu}_z$, $\\tilde{\\sigma}_z$ and $\\mu_z$ formulated?\n\n- How $\\mu_z(z)$ differs from $\\mathcal{T}(x_{0:N})$? What happens if the NNs are omitted and $\\mathcal{N}(\\mathcal{T}(x_{0:N}), \\mathbf{I})$ is used instead of $\\mathcal{N}(\\mu_z(z), \\mathbf{I})$?\n\n - $f(x_{0:N})$ is in $\\mathbb{R}^C$ in introduction but $x_t$ and $y_t$ are in  $\\mathbb{R}^d$. Does $C=d$ hold?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Inspired by the conditional diffusion process for multivariate regression from Han et al. (2022), the manuscript proposes the conditional diffusion process for multivariate time series forecasting.\nTo the best of my understanding, the core contribution of the manuscript is to use the \"condition\" from a Transformer for the diffusion model, while both the Transformer and conditional diffusion model are from the literature.  \n\nThe manuscript is overall clearly written and easy to read. Some of the missing details are listed in Questions.\n\nThe proposed method for probabilistic multivariate time series forecasting has the potential to contribute to the community.", "weaknesses": "- The manuscript does not successfully highlight its contribution. \nSince both components of TMDM are from the literature, it is necessary to highlight why this is not trivial.\nIn my humble opinion, the 3rd contribution is quite weak, as both metrics are from existing works.\n\n - There are also other works that employs a Transformer and a probabilistic model for time series modelling. \ne.g., Transformer + Probabilistic Circuit is proposed in [1] for time series forecasting, and uncertainty estimation (similar to Fig. 2 in the manuscript) is provided. \nA discussion with such works might help to stress the novelty and contribution of the manuscript.\n\n - The generation of the conditional representation $\\hat{y}_{0:M}$ is not clear to me. Some details are omitted from eq(9) to eq(10).\n\n - The references are not up-to-date, many arXiv versions are already published.\n\n---\n[1] Yu, Zhongjie, et al. \"Predictive Whittle networks for time series.\" Uncertainty in Artificial Intelligence. PMLR, 2022.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)