MG-TSD: Multi-Granularity Time Series Diffusion Models with Guided Learning Process
Authors: Xinyao Fan
Yueying Wu
Chang Xu
Yuhao Huang
Weiqing Liu
Jiang Bian
Venue: ICLR 2024 poster (2024)
Keywords: denoising diffusion models
multi-granularity
time series forecasting
Primary area: general machine learning (i.e., none of the above)
TL;DR: We introduce the MG-TSD model, which leverages multiple granularity levels within the data to guide diffusion models for probabilistic time series forecasting.

Abstract: Recently, diffusion probabilistic models have attracted attention in generative time series forecasting due to their remarkable capacity to generate high-fidelity samples. However, the effective utilization of their strong modeling ability in the probabilistic time series forecasting task remains an open question, partially due to the challenge of instability arising from their stochastic nature. To address this challenge, we introduce a novel Multi-Granularity Time Series Diffusion (MG-TSD) model, which achieves state-of-the-art predictive performance by leveraging the inherent granularity levels within the data as given targets at intermediate diffusion steps to guide the learning process of diffusion models. The way to construct the targets is motivated by the observation that forward process of the diffusion model, which sequentially corrupts the data distribution to a standard normal distribution, intuitively aligns with the process of smoothing fine-grained data into a coarse-grained representation, both of which result in a gradual loss of fine distribution features. In the study, we derive a novel multi-granularity guidance diffusion loss function and propose a concise implementation method to effectively utilize coarse-grained data across various granularity levels.
More importantly, our approach does not rely on additional external data, making it versatile and applicable across various domains. Extensive experiments conducted on real-world datasets demonstrate that our MG-TSD model outperforms existing time series prediction methods.

Published as a conference paper at ICLR 2024
MG-TSD: MULTI-GRANULARITY TIME SERIES DIF- FUSION MODELS WITH GUIDED LEARNING PROCESS
Xinyao Fan**, 1* Yueying Wu?*, Chang Xu Yuhao Huang', Weiqing Liu", Bian.Jiang Bian
University of British Columbia', Peking University?, Nanjing University", Microsoft Research
xinyao. fanêstat www.uucc....... ca, wuyueyingestu.pkuu edu. huangyh@smail, nju.euu. cn, {chanx, weiging. 1iu, jiang, (atgg.iiijjemmiissoooo
ABSTRACT
Recently, diffusion probabilistic models have attracted attention in generative time series forecasting due to their remarkable capacity to generate high-fidelity sam- ples. However, the effective utilization of their strong modeling ability in the probabilistic time series forecasting task remains an open question, partially due to the challenge of instability arising from their stochastic nature. To address this challenge, we introduce novel Multi- Grani-Granularity Time Series Diffusion (MG- TSD) model, which achieves state-of-the-arr predictive performance by leverag- ing the inherent granularity levels within the data as given targets at intermedi- ate diffusion steps to guide the learning process of diffusion models. The way construct the targets is motivated by the observation that the forward pro- cess of the diffusion model, which sequentially corrupts the data distribution to standard normal distribution, intuitively aligns with the process of smoothing fine-grained data into a coarse-grained representation, both of which result in a gradual loss of fine distribution features. In the study, we derive a novel multi- granularity guidance diffusion loss function and propose a concise implemen- tation method effectively utilize coarse-grasd maase--raiied data across various granular- ity levels. More importantly, our approach does not rely on additional external data, making versatile and applicable across various domains. Extensive exper- iments conducted on real-world datasets demonstrate that our MG-TSD model outperforms existing time series prediction methods. Our code is available at attps://gitinee www.itttu...omuuuueue hetomm/Hundeddll MMG-MMMTTT TSD
INTRODUCTION
Time series prediction is critical task with applications in various domains such finance fore-
casting (Hou et al., 2021 Chen et al. 2018), energy planning (Koprinska et al., 2018 Wu et al.) 2021) climate modeling (Wu et al. 2023 202T), and biological sciences (Luo et al. 2020 Ra-
jpurkar et al. 2022). Considering that time series forecasting problems can be effectively addressed conditional generation task, many works leverage generative models for predictive purposes. For instance, Salinas et al. (2019) utilizes a low-rank plus diagonal covariance Gaussian copula: Ra- sul al. (2021) models the predictive distribution using normalizing flows. Recent advancements in diffusion probabilistic models (Ho et al. 2020) have sparked interest in utilizing them into prob- abilistic time series prediction. For example, Rasul al. (2020) auto-regressively generates data through iterative denoising diffusion models. Tashiro et al. (202T) uses a conditional score-based diffusion model explicitly trained for probabilistic time series imputation and prediction. These methods relying on diffusion models have exhibited remarkable predictive capabilities. However, there is still considerable scope for improvement. One challenge that diffusion models face time series forecasting tasks is the instability due to their stochastic nature when compared to determinis- tie models like RNNs and variants like LSTMs (Hochreiter & Schmidhuber 1997] ] Lai et al.) 2018),,
GRUs (Ballakur & Aryal 2020; Yamak et al.] 2019), and Transformers that rely self-attention mechanisms Vaswani et al. 2017 Zhou et al.
2021 2022
Wu et al. 202I). More specifically, the
These authors contributed equally this work.
Corresponding chanx @ @mierosoft.com

---

Published as a conference paper at ICLR 2024
diffusion models yield diverse samples from the conditional distributions, including possible low- fidelity samples from the low-density regions within the data manifold (Schwag et al., 2022). In the context of time series forecasting, where fixed observations exclusively serve as objectives, such variability would result in forecasting instability and inferior prediction performance.
To stabilize the output of diffusion model in time series prediction, one straightforward method is to constrain the intermediate states during the sam-
Dicfusion Process
pling process. Prior research in the realm of dif- fusion models has introduced the idea of classifier- guidance (Nichol et al.) (2021) and classifier-free guidance (Ho & Salimans] (022), where the pre- dicted posterior mean is shifted with the gradient of either explicit or implicit classifier. However, these methods require labels as the source of guidance while sampling, which are unavailable during out- of of-sample inference. We observe that the forward process of the diffusion model, which sequentially corrupts the data distribution to standard normal distribution, intuitively aligns with the process of smoothing fine-grained data into a coarser-graite grained representation, both of which result in a gradual loss of finer distribution features. This provides the in- sights that intrinsic features within data granularities may also serve as source of guidance.
Coassest- Eraained Data
Soo AAN
?
Coarse grraind
Gesothing
/ Observation
Prediction
Gridance
Figure 1:
The process of smoothing data from finest-grained coarsest- -raaienst--- nat- urally aligns with the diffusion process.
In this paper, we propose a novel Multi- Granularity Time Series Diffusion (MG-TSD) model that leverages multiple granularity levels within data to guide the learning process of diffusion models. The coarse-gre coarse-grained data at different granularity levels are utilized as targets to guide the learning of the denoising process. These targets serve as constraints for the intermediate latent states, ensuring regularized sampling path that preserves the trends and patterns within the coarse grrsseegrained data. They introduce inductive bias which promotes the generation of coarser features during intermediate steps and facilitates the recovery of finer features in subsequent diffusion steps. Consequently, this design reduces variability and results in high-quality predictions. Our key contributions can be summarized below:
1. We introduce novel MG-TSD model with innovatively designed multi-granularity guidance loss function that efficiently guides the diffusion learning process, resulting in reliable sampling paths and more precise forecasting results.
2. We provide a concise implementation that leverages coarse-grained data instances at var- ious granularity levels. Furthermore, we explore the optimal configuration for different granularity levels and propose a practical rule of thumb.
3. Extensive experiments conducted on real-world datasets demonstrate the superiority of the proposed model, achieving the best performance compared to the state-of- of-the-art methods.
2 BACKGROUND
2.1 DENOISING DIFFUSION PROBABILISTIC MODELS
Suppose xn ~ gx (2x) is multivariate vector from space X' = RD, Denoising diffusion probabilis- tic models aim to learn model distribution po (#ag that approximates the data distribution q(vu). Briefly, they are latent variable models of the form pa (zo) = Pe (Σo.N )dx where for n = 1,...,N N is a sequence of latent variables in the same sample space as To. The denoising diffusion models are composed of two processes: the forward process and the reverse process. During the for- ward process, small amount of Gaussian noise added gradually in steps to samples. is char- acterized by the following Markov chain: q( |xo) = II g(2,||xn-1) (zn-1); where q( (2 == N(VI- B, 3,, I). The step sizes are controlled by a variance schedule [3,, E (0, 1) 1) where represents diffusion step. A nice property of the above process is that can sample
2

---

Published as a conference paper at ICLR 2024
any arbitrary diffusion step in closed form, let on = 3n and an IT= Og. It has a.E. The reverse diffusion process is to recreate the real
been shown that In 20 + vī-ā,e. samples from Gaussian noise imput. It is defined as a Markov chain with learned Gaussian transi- tions starting with p(æN)) Nxxx; 0, I). The reverse process is characterized as pe (o(ED:N) p(ZrN) IT== Pe (z,)) where Po(Zn-1|zz)) = (i p4 (zn,.)). Eg (0),,,)))) jul RR x N RD and 2g RD N I. take the variable 2n E RD and the diffusion step E N as mppus, and share the parameters 0. The parameters in the model are optimized to mini- mize the negative log- likelihood ming [=)-)())) log (o via a variational bound. According to denoising diffusion probabilistic models (DDPM) in Ho et al. (2020), the parameterization of pa( ( chosen as:
Me (I 12)
Inn
1 On
EB (Va + vI (l),,,)))
(1)
Van
VI ān
where Eg is network which predicts
(0,1)) from In. We simplify the objective function into
La
simple
||en Eo co(vanoto + v1 -I-á, Gn n)]l
Once trained, we can iteratively sample from the reverse process pa (x En,) to reconstruct æo-
2.2 TIMEGRAD MODEL
We treat the time series forecasting task as a conditional generation task and utilize the diffusion models presented in Section 2.1 as the backbone generative model. TimeGrad model is a related work by |Rssll et al. (2020) which first explored the use of diffusion models for forecasting mul- tivariate time series. Consider a contiguous time series sampled from the complete history train- ing data, indexed from to T. This time series is partitioned into a context window of interval [1, to) and prediction interval [to. 7]. TimeGrad utilizes diffusion models from Ho al. (2020) to learn the conditional distribution of the future timesteps of the multivariate time series given their past. RNN is employed to capture the temporal dependencies, and the time series sequence up to timestep is encoded in the updated hidden state h, Mathematically, TimeGrad models qx ( [21
I (2x where æ, R denotes the time series timestep and h, RNN, (w,, h Each factor is learned via shared conditional denoising diffusion model. In contrast to Ho et al. (2020), the hidden states h are taken as an additional input in the denoising network eg (x h n). and the loss function for timestep and diffusion step is given by:
II= (x.
Fe |||e en(( (o + v1 Gn n, hr-1)l13),
where the first subscript in a represents the index of the diffusion step, while denotes the timestep within the time series.
2.3 PROBLEM FORMULATION
In the time series prediction task, let (1) represent the original observed data. The time series data is denoted as (1) [a|.... xt ........ ... x7/,, where represents the timestep t E [1, 7] and C E D. Specifically, our task is to model the conditional distribution of future timesteps of the time series [=!..... x given the fixed window of history context. Mathematically. the problem consider can be formulated as follows:
ax a a (
II ox a (= t=
(4)
METHOD
In this section, provide overview of the MG-TSD model architecture in Section 3.1 followed
by a detailed discussion of the novel guided diffusion process module in Section 3.2 inc including the derivation of the heuristic loss function and its implementation across various granularity levels.
3

---

Published as a conference paper at ICLR 2024
3.1 MG-TSD MODEL ARCHITECTURE
The proposed methodology consists of three key modules, as depicted in Figure
Multi-granularity Data Generator is responsible for generating multi-granularity data from ob- servations. this module, various coarse-grained time series are obtained by smoothing out the fine-grained data using historical sliding windows with different sizes. Suppose is a pre-defined smoothing (for example, average) function, and s is the pre-defined sliding window size for granu- larity level g. Then ( = f(x(",, s.). The sliding windows are non-overlapping and the obtained coarse-grained data for granularity y are replicated 5 times align over the timeline [1, T..
Temporal Process Module is designed to capture the temporal dynamics of the multi-granularinn time series data. We utilize RNN architecture each granularity level 9 separately to encode the time series sequence up to a specific timestep t and the encoded hidden states are denoted as h.. The RNN cell type is implemented GRU Chung et (2014).
Guided Diffusion Process Module is designed to generate stable time series predictions at each timestep E. We utilize multi- gulli-granularity data as given targets guide the diffusion learning process. A detailed discussion of the module can be found in Section 3.2
Timestog tap
Pines top to
DiEfusion
N
No
N
PM
Multi- Multi-granmtn granulart arity Data Generator
(HappoiI JUGCHES HORAlN)
f1
Cg
RNN
Sample
RNNN
ais
AA
RYN
M
#
G
Guided Diffusion Process Module
Eg
34
RNN
AP
Figure 2: Overview of the Multi-Granularity Time Series Diffusion (MG-TSD) model, consisting of three key modules: Multi-granuuarrty Data Generator, Temporal Process Module (TPM), and Guided Diffusion Process Module for time series forecasting at specific granularity level.
3.2 MULTI-GRANULARITY GUIDED DIFFUSION
In this section, we delve into the details of the Guided Diffusion Process Module, a key compo- nent in our model. Section I.2.I presents the derivation of a heuristic guidance loss for the two- granularity case. In Section 3.2.2 we generalize the loss to the multi-granularity case and provide concise implementation effectively utilize coarse-grained data across various granularity levels. Briefly, the optimization of the heuristic loss function can be simply achieved by training denois- ing diffusion models the multi-granularity data with shared denoising network parameters and partially shared variance schedule.
3.2.1 COARSE-GRAINGS GUIDANCE
Without loss of generality, consider two granularities: finest-grained data x:' (g1 1) from (()) and coarse- -rrrined data xy from ( at a fixed timestep t, where < t < T. We omit the subscript in the derivation for notation brevity. Suppose the denoising diffusion models presented in Section 2.1 are employed to approximate the distribution ((x:)) and let the variance schedule a (0, ) Suppose ag q(x0") ), where the subscript 0 denotes the index
be (8)
4

---

Published as a conference paper at ICLR 2024
of diffusion step. The diffusion models in Section 2.2 define a forward trajectory ((((in)) and 0 parameterized reverse trajectory pe po(2).N).
specific timestep, it does not account for the intrinsic structure of time series, such as trends, which are represented by coarse-graines time series. In this paper, guide the generation of samples by ensuring that the intermediate latent space retains the underlying time series structure. This is achieved by introducing coarse saurse-rrinedd targets x at intermediate diffusion step N! E N N 1]. Specifically, we establish the objective function as the log-likelihood of observed coarse-grained data x evaluated at the marginal distributions at diffusion step Nº. which can be expressed log po (z With an appropriate choice of diffusion step N2, the coarser features recovered from the denoising process could gain information from the realistic coarse-grained sample. Recall that the marginal distribution of latent variable at denoising step No determined by the 0 p-parameterized trajectory po (o( can be expressed as:
While Section 2.2 focuses predicting samples
Po(xNz) / (o
p((WN)
Pa (0 a., (N (5) Nº+1
II
where EN ~ NO, I). po po( = -i(nn-1; n), Eo n)),
To make the objective tractable, common technique involves optimizing variational lower bound on the likelihood in Equation 5 This can be achieved by specifying a latent variable sequence of length N Nº. such that the joint distribution of x and these latent variables is available. Conveniently, we employ diffusion process on x with total of N - N diffusion steps, defining a sequence of noisy samples x xx + 2 as realizations of the latent variable sequence. Then,
the guidance objective can be expressed as:
log (x = log
x po(x" No. N2+1
2 dxx
(6)
(N3+1):N'
Applying the same technique in Ho al. (2020). the guidance objective function in Equation 6 simplifies the loss function of the diffusion models (see the Appendix ( for proof details):
Ee m.lle c0(2%) )
where æg (TI= at )a= + 1 I[= aff and (O, I). When the variance schedule
chosen (a [ Ny= the loss function of the diffusion model in Ho al. 2020 is equivalent to
the guidance loss function presented in Equation 7
3.2.2 MULTI-GRANULARITY GUIDANCE
In general, for G granularity levels, data of different granularities generated by Multi-granularity Data Generator can be represented as X (² ..... (G), We expect these coarse-grained data can guide the learning process of the diffusion model at different steps, serving as constraints along the sampling trajectory. For coarse-grained data at granularity level g. where 9 E .......), we define the share ratio as Tg := - (No - 1)/N. It represents the shared percentage of variance schedule between the gth granularity data and the finest-grained data. For the finest-grained data, N! = 1 and nl = 1. Formally, the variance schedule for granularity is defined as
1 {a.
n. = 1...., No
(N =
(8)
if No + 1I...,,
N.
and (.8)) = (1 (1- ag) I- Accordingly, define ag (N?) Ih ag. and by (N2) ag (NT). We suppose N! N?... No <... NG.. which represents the diffusion index for starting sharing the variance schedule across granularity level g E (1,..., G). The starting index No is larger for coarser granularity level, aligning with the intuition that the coarser-grained data loses fine distribution features to a greater extent and is expected to resemble the samples from carlier sampling steps.
Furthermore, we use the temporal hidden states for granularity level g to timestep from the Temporal Process Module as conditional inputs for the model to generate time series corre- sponding granularity levels similar to Rasul al. (2020). Then the guidance loss function T.(0)(0)
5

---

Published as a conference paper at ICLR 2024
for gth ath-granularity xh), e at timestep t and diffusion step can be expressed as:
( (0) Be,ag ll(e vo(vohaa + Dic n, 11-1))
where hn = R 1 is the updated hidden states from the last step.
The guidance loss function with G granularity levels of data is L Eg w ( where w E [0, 1] is hyper-parameter controlling the scale of guidance from granularity g.
Training, The training algorithm is in Algorithm [ The final training objective is the weighted summation of loss for all granularities, including the finest granularity:
linal w (1) + guidannce (0)
w W my E0(2 n, 1)
(10)
9=1
where x = Vay + % E and Mg= w 1. The denoising network parameters are shared
across all granularitics during training.
Algorithm Training procedure
Context interval [L, to): prediction interval [fo. T]; number of diffusion step N; of share ratio for granularity (or equivalently {N2.9 9 E {L...., ... G}}); generated multi-granularij data tef............................. repeat
Input:
...... E (l...., G); initial hidden states h g (1.....G],
G). 2: Obtain h = RNN* 1) E (1....,C, G). t [1.....T]. T).
1: Sample the multi guatii-raanularity time series [xj I...., x9 {1....,.)}.
3: for t= to T do
4: Initialize n ~ Uniform(1...... NN and (O.11)
5: Reset the variance schedule (8% a (NT)) g {I...., G].
6: Compute loss L final according to Equation 10
7: Take the gradient L final
8: end for
until converged
Inference. Once the model is trained, our goal is to make predictions on the finest- finest-grained data, up certain number of future prediction steps. Suppose that the last context window ends timestep to we use Algorithm 2 to perform the sampling procedure and generate sample ø for the next timestep. This process repeated until reaching the desired forecast horizon. With different hidden states as conditional inputs, the model can sample time series at respective granularity levels.
Algorithm Inference procedure for each timestep [to. T]
Input: Noise N. ~N(0,, I) and hidden states h² (1..... G}
1: for n = to 1 do
2: if > then
3:
Sample z ~ 1).
4: else
5: =
6:
end if
7: for = to do
8:
1/-1-- Vog. 9
co(a so(an,, n, n,,b)) + Vollz, where og V1-0g
82
1-00 1-a"
89
n.
9: end for
end for
Return: ag.0. 1(finst--grained data); (Optional: 28.0- 9 E (,,...,G))
Selection of share ratio. We propose a heuristic approach to help select the appropriate share ratio 79, which is derived from N We determine the choice of No as the diffusion step which the distance between two distributions ((x3) and pe x is minimum, as shown below:
No == arg min D (gg((gg ((x"), po(x
(11)

---

Published as a conference paper at ICLR 2024
where D is measure for accessing discrepancy between two distributions, such KL divergence. In practice, we first pre- train a TimeGrad model and then compute the CRPS between the coarse- grained targets and the samples along the sampling path of finest-grained data during inference. The range of steps where the CRPS values can consistently maintain relatively small values suggests proper range of share ratios.
4 EXPERIMENTS
In this section, we conduct extensive experiments on six real-world datasets to evaluate the perfor- mance of the proposed MG-TSD model and compare with previous state-of-the-an baselines.
4.1 SETTINGS
Datasets. We consider six real-word datasets characterized by range of temporal dynamics, namely Solar, Electricity, Traffic, Taxi, KDD-cup cup and Wikipedia. The data is recorded at intervals of 30 minutes, hour, or I day frequencies. Refer to Appendix for details.
Evaluation Metrics. We assess our model and all baselines using CRPS (Continuous Ranked Probability Score), widely used metric for probabilistic time series forecasting, well as NMAE,um (Normalized Mean Absolute Error) and NRMSEuum (Normalized Root Mean Squared Error). For detailed descriptions, refer to Appendix D
Baselines. We assess the predictive performance of the proposed MG-TSD model in comparison (Salinas al., 2019). GP-scaling (Salinas et al. 2019). GP-Copr GP-Copula (Salinas et al.) 2019). . Transformer-MAN (Rasul al. 2020), LSTM-MAF (Rasul al. 2020 TimeGrad (Rasul al. 2021). and TACTiS (Drouin et al. 2022). The MG-Input ensemble model serves as the baseline with multi-granularity inputs. It
with multivariate time series forecasting models, including
combines two TimeGrad models trained on one coarse- coarse-grained and finest-grained data respectively, and generates the final predictions by a weighted average of their outputs.
Implementation details. We train our model for 30 epochs using the Adam optimizer with fixed learning rate of 10-5, We set the mini-batch size to 128 for solar and 32 for other datasets. The diffusion step is configured as 100. Additional prper-parameters, such as share ratios, granularity levels, and loss weights, are detailed in Appendix C.3 All models are trained and tested on single NVIDIA A100 S0GB GPU.
4.2
RESULTS
The CRPSaum values averaged over 10 independent runs reported in Table The results show our model achieves the lowest CRPS and outperforms the baseline models across all six datasets. The MG-Input model exhibits marginal improvement certain datasets when compared to the TimeGrad. This implies that while integrating multi-grant multi-granullaity information may result in some information gain, direct ensembling of coarse-grained outputs is inefficient in boosting performance.
4.3 ABLATION STUDY
Share ratio of variance schedule. To investigate the effect of share ratio, we evaluate the perfor- mance of MG-TSD using various share ratios across different coarse granularities. The experiment is conducted in two-granen -wo.nuaallarity setting, where one coarse granularity is utilized to guide the learn- ing process for the unnst-grained data. Table 2 CRPS.um values initially decrease to their lowest values and then ascend again as the share ratio gets larger. Furthermore, we observe for coarser granularities, the model performs better with a smaller share ratio. This suggests that the model achieves optimal performance when the share ratio is chosen at the step where the coarse gaatneegr samples most closely resemble intermediate states. guidance greatly enhances the model performance. How-
shows that for each coarse granularity level, the
Utilizing 4-hour 6-hour granularity ever, the improvement in performance diminishes as the granularity becomes coarser, such as 12 hours or 24 hours, possibly due to the greater loss of information local fluctuations.
In practice the selection of share ratio can follow the heuristic rule outlined Section 3.2.2 Fig- ure 3 provides illustrative plots for the share ratio selection curve of different granularities. The blue

---

Published as a conference paper at ICLR 2024
Table 1: Comparison of CRPS sum (smaller is better) of models on six real-world datasets. The
reported mean and standard error are obtained from 10 re-training and evaluation independent runs.
Method
Solar
Electricity
Traffic
KDD-cup
Taxi
Wikipedia
Vcc-LSTM ind-scaling
0.
0.004
0.0815 u107
0.35600gn.00
0. 0.125422.0178
GP-Scaling
0.3802-0..00
0.0499=0.001
0.0753 0112
0.298820.0444
0.2266
0. 1351
Copula
0.3612:0.mm otat
0.0287=a..00 2.0000
0.0618 0015
0.3157 2.0
0.1894 .. o007
0.0669 ooce
LSTM-MAF
0.3127 10.0002
0.0051 10:012200.001
0,0526 1c
0122919
0.222%5
0.0763 1, 4.06442v.uur
Transforer-MAF
0.3532 2ou0s
0.0272
0.0499 u0l1
0.29G1 10000o
0.1531 ao.nos
TimeGrad
0.303/4=9.0665
0.023220.0035
0.041 4.04144c.00 4012
0. 0.2502
0. 0.1265=a.0220
0.0556+0..00
TACTS
0.4200 osao
0.025P20.000
0.109344 0076
0.5400 1684
0,20770 ouGe
MGGIngut
0.52339 0.52899.o.orr or1
5.02288.0.00s
0,0658 Lo 0065
0,29977 L.o
0.1592 20.007
0.0567 1,0 0091
MG-TSD
0,3081 20.00s
0.0119.20.0017
0.0.23
0.1837 2o.0wes
0.1159.ju.0saz
0.0529 1000004
Table 2: Influence of share ratios for different granularities on Solar dataset. The reported mean and standard error are obtained from 10 re-training and evaluation independent runs.
Ratio
hour
6 hour
CRPS
NMAE
NRMSE
CRPS
NMA sum
NRMSE sum
20 %
0.3489 00.0100
0.3826
0.7177 200045
0.3378 20.0805
0.3703
0.6916 0536
40%
0.3405
0.3792
0.6870
0.3275
0.3608 a..
0.6650
60%
0.3268
0..604
2..519
0.3166
0.3-49 + 0368
0.6478=.o.n60
80
0.3172#t0.024
0..00000
0.6515=t.051
0.3221 =000425
0.3555+0.0445
0.6544++0.0745
100%
0.3178
0.3480 co566
0.6591g0.0503
0.3232g0.0000
0.3548220.011
0.6350 0000
Ratio
hour
24 hour
CRPS
NMAE.m
NRMSE
CRPS
NMAEuum
NRMSE
20%
0.3440=0.0000
0.3767 20.055
0.69992o.0772
0,3315 0,3315 2000066
0.3693
0 0,6801 0554
40%
0.337 0.3374200070
0.3713
0.6837
0.3276 192276200000 g0-0358
0.36122..0re
0.6722
60%
0.3240
0.3597
100000ss
0.669410.0z0
0.3382 0343
0.3737 10.0005
0.6878 0633
80%
0.3391 10.0000
0.3719
0.695320.000
0.3288 io. 04c0
0.3639
0.674120.092 20.0929
100%
0.3284ga.e823
0.3538
00.045
0.600920..001
0.3407
0.3407
=0.0248
0.369240.024
0.6933go.0525
curve in each plot represents CRPSsam values between coarse-grained targets and 1-hour samples come from 1-gran(finest-rran) model each intermediate denoising step: each point on the orange polylines represents the CRPS value of l-hour predictions by 2-gran MG-TSD models with dif- ferent share ratios ranging from [0.2, 0.4, 0.6, 0.8, 1.0], and the lowest point of the line segment can be used to characterize the most suitable share ratio for the corresponding granularity.
The diffusion steps that can achieve relatively small CRPS values are colored in grey, suggest- ing proper range for the share ratio at which the model can achieve satisfactory performance. From the plots, a strong correlation exists between the polyline of CRPS calculated during test time and the share ratio selection curve, which validates the effectiveness of the selection rule. In addition, as granularity transitions from fine to coarse (4h-6h-122) 12h-+24h), the diffusion steps at which the distribution most resembles the caarse-grained targets increase (approximately at steps 20->40-60--610 This comparison shows the similarity between the diffusion process and the smoothing process from the gessieg to coarse- -ranined data, both of which involve a gradual loss finer characteristics from the grriiee data through a smooth and convex transformation.
3.40
0.40
3.40
044
0.6
0.6
Goa
0.30
0.30
830
620
23 20
G.
20
200
denosing diffusion
13
40
30
500
26
43
66
step
80
103
20
9 stop
66
162
(a) CRPS values between 4- (b) CRPS values between 6- (C) CRPS valnes between 12- (d) CRPS values between 24h- targets and samples
hour targets samples
hour targets samples
gramularity targets samples
Figure Selection of share ratio for MG-TSD models
The number of granularities. We further explore the impact of the number of granularities on the MG-TSD model. As presented in Table 33] increasing the number of granularity levels gener- ally boosts the performance of the MG-TSD model, which demonstrates that the introduction of
8

---

Published as a conference paper at ICLR 2024
multi- multi-granularity information effectively guides the learning process of diffusion models. However, the marginal benefit diminishes with the increase in granularity amounts. The results suggest that utilizing four to five granularity levels generally suffices for achieving favorable performance.
Table 3: Influence of the number of granularities MG-TSD performance for Solar and Electricity Dataset.
Num of gran
Solar
Electricity
CRPS
NMAE
NRMSE
CRPS
NMA
NRMSE
0.31 72 0240
0.3510=o.024
0.6515=n.0671
0.0174e0.004
0.0226 0071
0. 0.0296 0066
0.31 care
0.3494 1o.057s
0.6452gn.oraz
0,0160 10-oom
0.0198 cons
0,0262 10 0o0
0.308 1
0.3445
0.6245
0.01-991..001
0.017840.0018
0.024110.0080
0 0.30933 0111
0.3430 L0.0553
0.6117 10.0745
0.0153
0.0181 10 0013
0.025410.0054
Num gran
Traffic
KDD-cup
CRPS
NMAE
NRMSE
CRPS
NMA
NRMSE
0.03-17-d 0020
0. 0396=0.0022
0.0393=n.0043
=02127=0.1167
0.3171 =0. 1557
0.37 1662
0.0334go 0004
0. 0.0382
0.0374gn.0066
0.2414go
0.3030 1789
0.3808 2108
0. 0..3326 0041
0. 0.03774 03371=0.0044
0.0373=n.00Ga
0. 0,2198
0. 0.2893 1564
0,3315
2o-012
0.0070.a.0110
0.0563go.crm
0.1837g0.056
0.2463_.o.o00
0.3001 1. 0097
4.4
CASE STUDY
To illustrate the guidance effect of coarse-graines data, visualize the ground truth and the pre- dicted mean for both I-hour and 4-hour granularity time series across four dimensions in the Solar dataset in Figure 4 For comparison, the prediction results for the 1-hour data from TimeGrad are also included. The results indicate that the TimeGrad model struggles to accurately capture the peaks in the series and tends to underestimate the peaks in solar energy. In the MG-TSD model, the coarse-grained samples display more robust capacity to capture the trends, subsequently guiding the generation of more precise fine-grained data.
Observations
MG-TSD: median prediction of 1h median prediction
TimeGrad: 50.0% pradiction interval
TimeGrad: median prediction
50.0% prediction interval
250
120
200
100
150
60-
100
50
50
40
20
20
0
00.00
06:00
12:00
18,00
0000
06:00
12.00
1800
00:00
00:
06.00
12:00
18:0
00:00
0600
12:00
18:00
00.00
60
06dn
12:00
18:00
00:00
06.00
12:00
10:00
00.00
00.00
08:00
12:00
16:00
00:00
06:00
12:00
18:00
00:00
Figure 4: Visualization of the ground-truth (Solar dataset), MG-TSD predicted mean for 4-hour
and --hour time series, and TimeGrad predicted mean for the -hour time series. Additionally, the 50% prediction intervals for the 1 h-hur data also included. These plots represent some illustrative dimensions out of 370 dimensions from the first 24-hour rolling-window.
CONCLUSION
In this paper, we introduce a novel Multi-Granularity Time Series Diffusion (MG-TSD) model, which leverages the inherent granularity levels within the data as given targets intermediate dif- fusion steps to guide the learning process of diffusion models. We derive a novel multi multt--raauularity guidance diffusion loss function and propose concise implementation method to effectively utilize coarse-graings data across various granularity levels. Extensive experiments conducted on real- world datasets demonstrate that MG-TSD outperforms existing time series prediction methods.

---

Published as a conference paper at ICLR 2024
REFERENCES
Alexander Alexandrov, Konstantinos Benidis, Michael Bohhke-Schneider, Valentin Flunkert, Jan Gasthaus, Tim Januschowski, Danielle C. Maddix, Syama Rangapuram, David Salinas, Jasper Schulz, Lorenzo Stella, Ali Caner Türkmen, and Yuyang Wang. GluonTS: Probabilistic and Neural Time Series Modeling Python. Journal of Machine Learning Research, 21(116):1-6, html
2020a. httt::/// ora/ http://cc//ppppe//////ee-e/
Alexander Alexandrov, Konstantinos Benidis, Michael Bohlke-Schneider, Valentin Flunkert, Jan Gasthaus. Tim Januschowski, Danielle C Maddix, Syama Rangapuram, David Salinas, Jasper Schulz, et al. Gluonts: Probabilistic and neural time series modeling in python. The Journal of Machine Learning Research, 21(1) 2162):4629-2 4664946444,, 2020b.
Amulya Arun Ballakur and Arti Arya. Empirical evaluation of gated recurrent neural network chitectures in aviation delay prediction. 2020 5th International Conference on Computing, Communication and Security (ICCCS), pp. 1-7, 2020. doi: 10.1 10.1100//CC499988822000000 010.4996677.2222276666558
Yingmei Chen, Zhongyu Wei, and Xuanjing Huang. Incorporating corporation relationship via graph convolutional neural networks for stock price prediction. In Proceedings of the 27th ACM International Conference Information and Knowledge Management, pp. 1655- 2018.
Junyoung Chung, Caglar Gulcehre, KyungHyun Cho, and Yoshua Bengio. Empirical evaluation of gated recurrent neural networks sequence modeling. arXiv preprint arXiv: 2014.
Emmanuel de Bézenac, Syama Sundar Rangapuram, Konstantinos Benidis, Michael Bohlke- Schneider, Richard Kurle, Lorenzo Stella, Hilaf Hasson, Patrick Gallinari, and Tim Januschowski. Normalizing kalman filters for multivariate time series analysis. Advances in Neural Information Processing Systems, 33:2995-3007, 2020.
Alexandre Drouin, Étienne Marcotte, and Nicolas Chapados. Tactis: Transformer-attentional cop- ulas for time series. In International Conference Machine Learning, pp. 5447-5493. PMLR, 2022.
Jonathan Ho and Tim Salimans. arXiv:2207. 12598, 2022.
Classifier-free diffusion guidance.
arXiv preprint
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, 33:6840-6855 2020.
Sepp Hochreiter and Jürgen Schmidhuber. Long Short- Memory. Neural Computation, (8):1735- 1780, 11 1997. ISSN 0869-7667. doi: 1916://ecc..999..8.... URL https: 7/doi. orddi.org/100 10.
1997 9. 8. 1735
Min Hou, Chang Xu, Yang Liu, Weiqing Liu, Jiang Bian, Le Wu, Zhi Li, Enhong Chen, and Tie- Yan Liu. Stock trend prediction with multi granuiigranllrrty data: A contrastive learning approach with adaptive fusion. In Proceedings of the 30th ACM International Conference on Information & Knowledge Management, pp. 700-709, 2021.
Irena Koprinska, Dengsong Wu, and Zheng Wang. Convolutional neural networks for energy time series forecasting. In 2018 international joint conference neural networks (IJCNN), pp. 1-8.
IEEE, 2018.
Guokun Lai, Wei-Cheng Chang, Yiming Yang, and Hanxiao Liu. Modeling Long- and Short-Term Temporal Patterns with Deep Neural Networks. In The 41st International ACM SIGIR Conference on Research & Development Information Retrieval, SIGIR '18, pp. 95-104, New York, NY, USA, June 2018. Association for Computing Machinery. ISBN 978- -5783---00--657222 doi: 10.1 5/3209978.32100008
Yan Li, Xinjiang Lu, Yaqing Wang, and Dejing Dou. Generative time series forecasting with dif- fusion, denoise, and disentanglement. Advances in Neural Information Processing Systems, 35: 23009-23022, 2022.
10

---

Published as a conference paper at ICLR 2024
Junyu Luo, Muchao Ye, Cao Xiao, and Fenglong Ma. Hitanet: Hierarchical time-awart atten- tion networks for risk prediction on electronic health records. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pp. 647-656, 2020.
Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen. Glide: Towards photorealistic image generation and editing with text-guided diffusion models. arXiv preprint arXiv: /2. 10741, 2021.
Yuqi Nic, Nam H Nguyen, Phanwadee Sinthong, and Jayant Kalagnanam. time series is worth 64 words: Long-term forecasting with transformers. In The Eleventh International Conference Learning Representations, 2022.
George Papamakarios. Theo Pavlakou, and lain Murray, Masked autoregressive flow for density estimation. Advances in neural information processing systems, 30, 2017.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high- performance deep learning library. Advances in neural information processing systems, 32, 2019.
Pranav Rajpurkar, Emma Chen, Oishi Banerjee, and Eric Topol. Al in health and medicine. Nature medicine, 28(1):31-38, 2022.
Kashif Rasul. PytorchTS, 2021. URL http:p///gghhho.ccoo///aaaaareer ch ch-ts
Kashif Rasul, Abdul-Saboor Sheikh, Ingmar Schuster, Urs Bergmann, and Roland Vollgraf. Mul- tivariate probabilistic time series forecasting via conditioned normalizing flows. arXiv preprint arXiv:2002.066 103, 2020.
Kashif Rasul, Calvin Seward, Ingmar Schuster, and Roland Vollgraf. Autoregressive denoising dif- fusion models for multivariate probabilistic time series forecasting. In International Conference on Machine Learning, pp. 8857-8868. PMLR, 2021.
David Salinas, Michael Schneie,, Laurent Callot, Roberto Medico, and Jan Gasthaus. High-dimensional multivariate forecasting with low-rank gaussian copula processes. Advances in neural information processing systems, 32, 2019.
Vikash Sehwag, Caner Hazirbas, Albert Gordo, Firat Ozgenel, and Cristian Canton. Generating high fidelity data from len-itysity regions using diffusion models. In Proceedings of the IEEÉ/CVF Conference on Computer Vision and Puttern Recognition, pp. 11492-11501, 2022.
Lifeng Shen and James Kwok. Non-autoregressive conditional diffusion models for time series prediction. In International Conference on Machine Learning, pp. 31016-31022 PMLR, 2023.
Yusuke Tashiro, Jiaming Song, Yang Song, and Stefano Ermon. Csdi: Conditional score-based diffusion models for probabilistic time series imputation. Advances in Neural Information Pro- cessing Systems, 34:24804 0444804-248666 16, 2021.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, E ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Ad- vances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017. URL httphhtts:/ppccceeeiietttt le/3 ler.afaee/333//eee11bba3aaaaeee. par
neur www.prpps..ccopppr...eee/e/eeeee
Haixu Wu, Jiehui Xu, Jianmin Wang, and Mingsheng Long. Autoformer: Decomposition trans-
formers with auto-correlation for long-term series forecasting. Advances neural information processing systems, 34:22419-22430 2021.
Haixu Wu, Hang Zhou, Mingsheng Long, and Jianmin Wang. Interpretable weather forecasting for worldwide stations with unified deep model. Nature Machine Intelligence, pp. 1-10, 2023.
11

---

Published as a conference paper at ICLR 2024
Peter T Yamak, Li Yujian, and Pius K Gadosey. A comparison between arima, Istm, and gru for time series forecasting. Proceedings of the 2019 2nd international conference on algorithms, computing and artificial intelligence, pp. 55-55, 2019.
Haoyi Zhou, Shanghang Zhang, Jieqi Peng, Shuai Zhang, Jianxin Li, Hui Xiong, and Wancai Zhang. Informer: Beyond Efficient Transformer for Long Sequence Time-Series Forecasting. Proceed- ings of the AAAI Conference on Artificial Intelligence, 2374-3468. doi: 00.16099aai..3551221733.2.
May 2021. ISSN
Tian Zhou, Ziqing Ma, Qingsong Wen, Xue Wang, Liang Sun, and Rong Jin. Fedformer: Frequency enhanced decomposed transformer for long-term series forecasting. International Conference Machine Learning, pp. 177268227285 PMLR, 2022.
12

---

Published as a conference paper at ICLR 2024
A APPENDIX: DERIVATION OF LOSS FUNCTION
Recall that we specify a sequence of noisy samples x N x by applying the forward process on æ², The superseript in Nº is suppressed for notation brevity. Suppose the coarse-grained data 2N ~ g(x"), where the subseript notation N, indicates that the observed x is treated as a sample from the distribution. (In the diffusion model, the subscript is typically denoted 0, but we start with N. to simplify the derivation).
log (2:%.) -log (o(%)) + DKL( (M )lppo (N +.NN (x%,)
- log + Eagg mn 12 lox log g(< (( +N: (2
po (20%)
-- --lopooaa + Ey log (l + logpo(2<
q( +N) 10% po(x
Bg log
(12)
Then, the training objective can be performed by optimizing the usual variational lower bound shown below:
(0 (V. po (o(
LVLB Eyla .N) log
gBg log po(
(13)
is obvious that the objective LVLB is equivalent to the that of diffusion model in Ho et al. (2020) when employing diffusion models on 29 with N N. steps. The forward process is defined as where (= == NV(1 3 3 r). The {,89 In=N, share values with the variance schedule { of the finest grained data from index N.. And, the reverse process is defined by the p-parameterized trajectory. Then following the same technique in Ho et al. (2020), the LVLA can reduce to the usual loss of diffusion models.
II 1(=g1=0)
N
B APPENDIX: EXPERIMENTS
B. BENCHIMAR EXPERIMENTS
The results of the benchmark experiments, evaluated based on the metrics NRMSE and NMAEsum= are presented in Table 4 and Table respectively. In the experiments. we include four extra baseline models for a more comprehensive comparison: TimeDiff (Shen & Kwok 2023),
DDVAE (Li et al. 2022). PStchTST (Nie et al. 2022). and AutoFormer (Wu al. 2021).
Table 4: Comparison of NRMSE (smaller is better) of models on six real-world datasets. The reported mean and standard error are obtained from 10 re-training and evaluation independent runs.
Method
Electricity
Traffic
KDD-cup
Wikipedia
Vec-LSTM ind-scaling
0.9952-0.0077
0.1439 0.1439aa..u22
0.1451 0.1451-..0a 10.024
0.4461 20.18ts
0.6388
0.1618
GP-Scaling
0.90011-0.000
0.0811 20,0002
0.1169
0.3115ee.0022
0.3558 =0.025
0. 0.1710= 1006
GP-Copula
0.8279
0.06 120.2.0
0. 12822
0.2606
0.312540.0118
0.0030
Autoformer
0.7016 0.mo0
0.0175 0.0175_n.wwo 0.0000
0.0851 0.om
0.8981 2008
0.34498 1o.000n
0. 0.1052 io. 0000
PatchTST
0.7270-n.oomm
0.0474
0897 1.1997a..oom
0.5137 20.0000
0.3650
0.091 10013aao
D²VAE
0.7172
0.16:10
0.4722 0.1317
0.5628
0.7621 1o.ssee
2 2,209-1 1st
TifeDiff
1.3985_a.0sae
0.37 2.37142n.00rg
0.3620=a..oee
0.4936
0.5479
20000s
0.14 2.14412.o.00
TimeGrad
0. 0.6953
0.03-18 0.0007
0.0653 n.a2ss
0. 0.1092 10122
0.2365
Loueset
0.0870 1o.00o
TACTIS
0.8632-0.0s51
0.0427 20.002
0.2270=0.0.s0
0.65 13
0.3357
MG-ISD
0.6178 20.148
1.02:1::g.oom
0.0563 20.0250
0,3001 10.0au7
0.23344g..0sx3
0.0810
13

---

Published as a conference paper at ICLR 2024
Table 5: Comparison of NMAEsum (smaller is better) of models on six real-world datasets. The
reported mean and standard error are obtained from 10 re-training and evaluation independent runs.
Method
Solar
Electricity
Traffic
KDD-cup
Taxi
Wikipedia
Vec-LSTM ind scaling
0.3081
-0.007 000007
0.1261 a0 0211
0,1042_a..00 0.025
0.4193go 1902
0.49744n.000 0331
0.1416 0180
GP-Scaling
0.4941 0.0065
0,06118 10.0048
0,0975_ _0.065
0,2892
0.2867 Loueas
0.1152 Lo.
GP.Copula
0.43022.0.0300 0.003
0,0312 0007
0.07699.0.000 0003
0.2140gg0.03
0.2330) 0038
0.0659 0061
Autoformer
0.6368
0.0388 co 0,3000
0.0684 0.0000
0.7658 0000
0.2652 1ou0000
0.1233 0000
PatcHTST
0.4351
0,0350 g0 2000
0,1219 0000
0.4499 0000
0.2885
20.000
0.0625 0000
DAVAE
0.4457=0.0377
0,1454ga 20 0502
0.3992-00.117
0.4874g0.002 000000
0.6050
2.0151 =2.0000 000G
TineDill
1.33443_..000
0,3519 10
0.47822_o.oo
0,3630gg
0.4521
0.11
TimeGrad
0.3604_-0.0400
0.0266 0o40
1.0000=000089
0.3614go.188
0.1368
0.0631 20.008
TACTSS
0..4438_0.0313
0,0310
0,13522_ 0.1352__0.013 0..52_0000050
0.6077
0.2244
MG-TSD
0.3347=0.0220
0.0008
0.0370ga..040
0.2463g0.
0.1300g0.0150
10.0007
B.2
MORE EXPERIMENT SETTINGS
B.2.1 PERFORMANCE FOR LONG-TERM FORECASTING
To evaluate the performance of MG-TSD for long-term forecasting, we maintain fixed context length of 24 and extend the prediction length to 24, 48, 96, and 144. The results of the datasets Solar and Eelectrity are displayed in Figure
The results in Figure 5 indicate that MG-TSD performs well for long-time forecasting. The results indicate that as the prediction length increases, the performance of our proposed method stays robust, exhibiting no sudden decline. Furthermore, our method consistently outperforms the competitive baseline. This performance advantage is anticipated to persist future trends, with no indication of convergence between the approaches.
RMSE
RMSE
timegrad
smagred
Urregred migttd
Ummggred
Urrggred rigisl
148
iggisd
rngiid
6:45
righs
L.M.
2
Ingtcd
E:00
1:30
302
2443
Lnnn
00.tolar: Pedforaae Erchuticn Acros Diffecet Prodictio Longhes
IN ewt.iiccy Pefformanze Dediaation Across Dificrea Production Lenghs
Figure 5: Performance evaluation across different prediction horizons for MG-TSD with TimeGrad as the baseline Model. The context length is fixed at 24h and the prediction length is tested at 24h, 48h, 96h, and 144h. The average CRPS, NRMSE, and NMAE metrics are computed for both MG-TSD and the baseline over 10 independent runs, with error bars indicating the corresponding standard deviations.
B.2.2 TIME AND MEMORY USAGE OF THE MG-TSD MODEL DURING TRAINING
Experiments have been conducted to evaluate the time and memory usage of the MG-TSD model during training across various granularities. These experiments were executed using a single A6000 card with 48G memory capacity. The Solar dataset was utilized in this context, with batch size of 128, an input size of 552, 100 diffusion steps, and 30 epochs.
As illustrated in Figure 6 there is a linear increase in memory consumption with an increase in granularity. A slight surge in training time is also observed. These findings are coherent with the architecture of our model. In particular, each additional granularity results in the introduction of an extra RNN in the Temporal Process Module and an increase in computation within the Guided Diffusion Process Module. As per thcoretical expectations, these resource consumptions should exhibit linear growth. The slight increase in training time can be ascribed to the design of the Multi-gramulann Data Generator Module which enables parallel forward processes across different granularities, thus promoting acceleration. Moreover, it is pertinent to mention that excessive increase in granularity may not notably boost the final prediction results, hence the granularity will be kept within certain range. Therefore, the consumption of memory will not rise indefinitely.
14

---

Published as a conference paper at ICLR 2024
J0
25
MemonyGBB
20
15
Memory Time
Granulartty
Figure 6: Comparison of Time and Memory Consumption at Different Granularity Levels in MG- TSD Model Training
B.2.3 VARIATIONS IN THE FREQUENCY DOMAIN OF TIME SERIES DATA: THE IMPACT OF GRANULARITY AND DENOISING STEPS
We sampled series from Solar dateset and we conducted a Fast Fourier Transform to extract the sea- sonality components of the series, as well as the samples of different granularities and corresponding noisy samples along the forward diffusion process.
3500
gr30:
3500
gran:
diffusion
3000
gran: 12h
3000
ditfuslon 50 diffuwenn im step: 100
gran: 24h
Amplitude
2500
Amplitude
2500
2000
2000
1500
1500
1000
1000
500
500
0-
50
125
150
50
100
125
150
Frequency
Frequency
(a) Different granularities
(b) Different diffusion steps
Figure 7: Variations in the frequency domain of time series data: the impact of granularity and denoising steps.
As depicted in Figure 7a), as granularity becomes coarser, the components of all outstanding fre- quencies get lower, while high-frequency peak (around 125 and 80) diminishes quicker than lower-frequency peak (around 45). Figure 77b) demonstrates the distribution of frequency compo- nents of the same noisy series with gradually ascending forward diffusion steps and the same pattern observable. This empirical study indicates the connection between the forward diffusion process and the smoothing process from fine-grained data to coarse-graine coarse-grained data, both of which result in losing finer informative features.
C APPENDIX: IMPLEMENTATION DETAILS
C.I BENCHMARK DATASETS
For our experiments, we use Solar, Electricity, Traffic, Taxi, KDD-cup and Wikipedia open-source datasets, with their properties listed in Table
The dataset can be obtained through the links below.
(i) Solar: https:/ / www. nrel www.//ggi///ee.oo wtwp:/rra//ollr-ppoeeeeee
wtw. html
15

---

Published as a conference paper at ICLR 2024
(ii Electricity:
http://///a..///...//.o////a//aa ylaaddiaagrams2222211
(iv) Taxi: http://tt//////aaacrrrr///////rr dataddatappppp
+
KDD-cup: ||Be:
(vi) http::/wh....o./......ooooooooa/
re release/datasets
Name
Frequency
Number of series
Context length
Prediction length
Multi-gramularin dictionary
hour
137
24
hour, hour, hour, 24hour. 48
Electricity
hour
370
24
24
[1 hour. hour, 12 hour, 24 hour, 48 hour]
Traffic
24
hour, hour, hour,
hour]
Tax1 30 min
1214
24
24
[30 min 2 hour, hour. 12 hour, 24 hour]
KDD cup
hour
270
48
[1 hour, hour, hour, 24hour, hour]
Nikipedia
day
2000
30
30
day, 4 day. 7 day, 14 day)
Table 6: Detailed information of the datasets used in our benchmark including data frequency and number of times series (dimension), including information about context length and prediction length and the multi-gm multi-granularity dictionary utilized in the multivariate time series forecasting task.
C.2 LIBRARIES USED
The MG-TSD code in this study is implemented using PyTorch (Paszke al. 2019). utilizes the PytorchTS library (Rasul 2021). which enables convenient integration of Py PyToorch models with the GluonTS library (Alexandrov al. 2020b) which heavily rely for data preprocessing, model training, and evaluation in our experiments.
The code for the baseline methods obtained from the following sources.
(i) Vec-LSTM-ind-scalingg models the dynamics via an RNN and outputs the parameters of an independent Gaussian distribution with mean-scaling.
Code: /ttps::/ ttw... com http:///oooooo.eeeeeeeeeeeeooooe
(ii) GP-scaling: model that unrolls L.TTM with scaling each individual time series before reconstructing the joint distribution via a low-rank Gaussian.
Code:
com www.mbohhheeeeee
(iii) GP-Copula: model that unrolls an LSTM on each individual time series. The joint emission distribution is then represented by low-rank plus diagonal covariance Gaussian copula. reereellase
Code: / com/
(iv) LSTM-MAF; model which utilizes LSTM for modeling the temporal conditioning and em-
ploys Masked Autoregressive Flow (Papamakarios et al. (017)) for the distribution emission. wtw.oorrrhttttteeeeeeeeeeeeeeeee www.rree/mmateereeeeeeeeee
Code: bttps://github. htt.p//.....c.oeaaaaaaeeeeeeeeee
//model/temffoow
(v) Transformer-MAFF: model which utilizes Transformer Vaswani al. 2017) for modeling the
temporal conditioning and employs Masked Autoregressive Flow (Papamakarios et al. 2017 for the distribution emission model.
Code:
s/ Tnodel/transformee httpannfoooeretett_eeoeeeee tempflow
(vi) TimeGrad: auto-regressive model designed for multivariate probabilistic time series fore- casting, assisted by an energy-based model.
Code: htt.:/hhw...cooaaaaaaaeeeeeee/ea
Code:
(vii) TACTiS: a paramtatttic copula model based on transformer architecture.
16

---

Published as a conference paper at ICLR 2024
(viii) VAE: a bidirectional variational V equue equipped with diffusion, denoise, and
disentaagglement
Code: http:////ohhaaa..ccm///mee//////
(ix) TimeDiff: a predictive framework trained by blending hidden contextual elements with future actual outcomes for sample conditioning.
Code: There no publicly available code; we obtained the code by emailing the author.
(x) Autoformer: redefines the Transformer with deep decomposition architecture, including quence decomposition units, self-correlation mechanisms, and encoder-decoders. http://tsss//tttt....o//ooo/etep httt://foomm/ttoooooooooooe
(xi) PSt an efficient design of Transformer-based models for multivariate time series fore- casting and self-supervised representation learning.
Code: https:// com http//ugiiniieee (ttp:/iiiegPPPach..e PPatccTTT
C.3 HYPER-PARAMETER SETTING FOR EACH MODEL.
Dataset
Num gran
Gran dict
Share ratio
Loss weight
[1,0.9]
[1h,4h] [1h,12h]
[1.0.8]
[0.9.0.1]
[1.0.6]
[1,0.9.0.8]
10.8, 0.1, 0.1] [0.9, 0.05, 0.05] [0.85, 0.10, 0.05]
[hh.4h.12h]
[1,0.8,0.]]
[lh,4h.24h]
Solar
[1,0.8,0.6]
Electricity Traffic KDD-cup
1.0.9.0.8.0.88
1hh,4h,12h,241
11.099.088.066
[0.8, 0.1, 0.05, 0.051 10.7,0.1,0.110.11
4
11h,4h,12h,48h,
[1,088.066.0.6]]
[1,088.066.0.44]
1.0.9.08...6660066
[1.0.9.08..06.004444]
10.8001.00005.000...000111
1h.,4h,...1h.2244
61.088,0.6..66..6666
(0.880005,000550000..0006
11h.4h.22h.44h.400
(1.0.8.0.6606660.4))
[0.660I1.00110.1..0.11
[1.0.8,066.044.0.4]]
[30m,2h]
[30m,6h]
[1,0.8]
Taxi
[0.9.0.]]
[30m,
[1.0.]]
[30m,24h]
[1d,4d]
[1,0.8]
Wikipedia
3
[1d,7d]
[0.9.0.1]
[1.0.6]
[1d,14d]
Table Tested hyper-pane parameee values for the MG-TSD Model. The reported results in the paper are based on a parameter search within these choices.
D APPENDIX: METRICS
More details about the metrics we adopt can be found in Gluonts documentation (Alexandrov al. 2020ab. We briefly summarize them as below:
CRPS From de Bézenac et al. (2020), CRPS is univariate strictly proper scoring rule which
measures the compatibility of a cumulative distribution function F' with an observation x E R as
CRPS(F,z) / (F(r) 1(2 S dy
where I fxr y} is the indicator function, which is if S and 0 otherwise. The CRPS attains the minimum value when the predictive distribution F same as the data distribution. CRPS extends CRPS to multivariate time series with simple modification.
CRPS
E, CCRSS(F-.. sam Σ xz)],
17

---

Published as a conference paper at ICLR 2024
where Fuum is calculated by summing samples across dimensions and then sorted to get quantiles. A smaller samS indicates better performance.
NMAE: NMAE is normalized version of the Mean Absolute Error (MAE) that takes into consid- cration the scale the target values. The formula for NMAE is as follows:
mean ((Y YYI)
NMAE
mean ((Y
Similarly, in this formula, represents the predicted time series, and Y represents the true target time series. NMAE calculates the average absolute difference between predictions and true values, normalized by the mean absolute magnitude of the target values. A smaller NMAE implies more accurate predictions.
NRMSE: NRMSE is normalized adaptation of the Root Mean Squared Error (RMSE) that factors in the scale of the target values. The formula for NRMSE follows:
mean ( - Y)
NRMSE
mean([Y
Here, ý represents the predicted time series, and Y represents the true target time series. NRMSE measures the average squared difference between predictions and true values, normalized by the mean absolute magnitude of the target values. A smaller NRMSE indicates more accurate predic- tions.
18

---

Published as a conference paper at ICLR 2024
E APPENDIX: MORE ILLUSTRATIVE PLOTS
Obsorvations
MG-T5D: median prediction 1h
TimoGrad: 50.0% prediction interval
median
prediction
Dim
Dim
Dim 2
Dim 3
250
125
200
100
40
100
50
50
50
25
00 100
12.00
18:00
00:30
06:00
12:00
18:00
00:00
12.00
18:00
00:00
05:00
12:00
18:00
00:00
06:00
12:00
18:00
0n
06:00
12:00
18:00
00:00
06:00
08:00
12,00
18:00
00000
06:00
12.00
18:00
00:00
08
00,
00
Dim
Dim
Dim 6
Dim 7
100
100
300
100
100
25
20
60:
08.
12.00
18:00
00:00
08:00
n0
18:0 18:00
00:00
00:00
12.00
18:00
00:00
06:00
12:00
18:00
00:00
00:00
06:00
12:00
18:00
00:00
06:00
12.00
18:00
00:00
00:00
08.
12:00
18:00
00:00
06
06:00
12.00
18.00
00:00
Dim
Dim
Dim 10
Dim 11
125
100
40
60
40
20
08:00
12:00
18:00
00:00
06:00
12:00
18:00
00.00
00:00
08:00
12:00
18:00
00:00
06:00
12:00
18:00
00:00
n0 00
08:00
12:00
1800
uo
06:00
12:00
18:00
00.00
06
08:00
12:00
18:00
00
06:00
12,00
18:00
00:00
60:0
Dim 12
Dim 13
Dim 14
Dim 15
60
60
100
40
40
50
20
25
00:00
08:00
12:00
18:00
00:00
06:00
12:00
18:00
00:00
00:00
1:00
12.00
18:00
00:00
06:00
12:00
18:00
00:00
00:00
12:00
18:00
00.00
06:00
12:00
18:00
00:00
00:00
06:00
12.00
18:00
00:00
06:00
12.00
18:00
00:00
Figure 8: MG-TSD and TimeGrad prediction intervals and test set ground-truth for Solar data of
some illustrative dimensions of 370 dimensions from first Dolling-windoog
19


Reviews:
[{"review_id": "vLrkBmOWU5", "reviewer": "Reviewer_Zq3E", "summary": "This paper employs the diffusion model for time series forecasting and introduces the Multi-Granularity Time Series Diffusion model, which comprises three key components: 1). The Multi-Granularity Data Generator, responsible for generating multi-granularity data. 2). The Temporal Process Module, which utilizes an RNN architecture to capture temporal dynamics. 3). The Guided Diffusion Process Module is aimed at generating stable time-series predictions. This model leverages various levels of granularity within the data to guide the forward process of the diffusion model. Additionally, the paper designs a multi-granularity guidance loss function and explores optimal configurations for different granularity levels, proposing a practical rule of thumb. Extensive experiments are conducted to showcase its precision and effectiveness.", "questions": "1.\tThere have been some similar works, such as TimeDiff[1] and D3VAE[2], which also applies the diffusion model. What are the technical advantages of these studies?\n2.\tThe paper designed MG-TSD based on the diffusion model. Why is it not compared with the diffusion-based models in the baseline? Besides, There are some newer Transformer-based models, such as PatchTST[3], and Autoformer[4] should be compared in your experiments.\n- [1]Shen L, Kwok J. Non-autoregressive Conditional Diffusion Models for Time Series Prediction[J]. arXiv preprint arXiv:2306.05043, 2023.\n- [2] Li Y, Lu X, Wang Y, et al. Generative time series forecasting with diffusion, denoise, and disentanglement[J]. Advances in Neural Information Processing Systems, 2022, 35: 23009-23022.\n- [3] Nie Y, Nguyen N H, Sinthong P, et al. A Time Series is Worth 64 Words: Long-term Forecasting with Transformers[C]//The Eleventh International Conference on Learning Representations. 2022.\n- [4] Wu H, Xu J, Wang J, et al. Autoformer: Decomposition transformers with auto-correlation for long-term series forecasting[J]. Advances in Neural Information Processing Systems, 2021, 34: 22419-22430.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The research problem addressed in this study is of paramount significance and holds great interest. Accurate time prediction has broad applications, including tasks like anomaly detection and energy consumption control.\n2. The paper introduces a novel and intriguing approach by linking various granularities in the time series with the forward process in the diffusion model.\n3. The paper is excellently written and presented in a clear and comprehensible manner.", "weaknesses": "1. Some related works can be further discussed.\n2. There is only one metric in the main experiment, which is not enough.\n3. Compared with the baseline, the performance improvement is not obvious.\n4. The use of the RNN architecture requires further explanation\n\n## After rebuttal\nmost of my concerns have been addressed.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "CTkQ6yHjXJ", "reviewer": "Reviewer_Zqn4", "summary": "Diffusion probabilistic models which can generate high-fidelity samples reserve stochastic nature. This characteristic makes it less effective in probabilistic time series forecasting tasks. To improve the efficiency of Diffusion probabilistic models, this paper introduces a novel MG-TSD model with an innovatively designed multi-granularity guidance loss function that efficiently guides the diffusion learning process. To effectively utilize coarse-grained data across various granularity levels, this paper propose a concise implementation method. What’s more, this approach does not rely on additional external data, making it versatile and applicable across various domains. Extensive experiments conducted on real-world datasets demonstrate the superiority of the proposed model, achieving the best performance compared to the state-of-the-art methods.", "questions": "1. In Equation 7 of Section 3.2.1, is the distribution of ∈ consistent with the distribution of x_N? Does the distribution of ∈ obey the normal distribution? The distribution and meaning of ∈ are not pointed out.\n2. Compared with the existing diffusion probabilistic models, does the MG-TSD model framework differs only in the Guided Diffusion Process Module ?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. In the context of the time series forecasting, where fixed observations exclusively serve as objectives, diffusion probabilistic models would result in forecasting instability and inferior prediction performance. Unlike constraining the intermediate states during the sampling process, this paper creatively leverages multiple granularity levels within data to guide the learning process of diffusion models.\n2. This paper provides a series of ablation experiments to test the effect of share ratio and the number of granularities, it evaluate the performance of MG-TSD using various share ratios across different coarse granularities and the number of granularities.\n3. Clarity:  The paper offers a clear presentation to the model architecture with a good explanation of the methodology.", "weaknesses": "1. This paper lacks an evaluation of the time complexity of the model. It may be more sufficient to add experiments that consume memory and time.\n2. MG-TSD is consisting of Multi-granularity Data Generator, Temporal Process Module (TPM), and Guided Diffusion Process Module. However, the ablation experiment part of this paper lacks performance testing of each module, especially Multi-granularity", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5xrmqQHlQT", "reviewer": "Reviewer_FBCv", "summary": "This work proposes a Multi-Granularity Time Series Diffusion (MG-TSD) model for time series prediction. In general, MG-TSD controls the learning process of the diffusion model by leveraging the temporal signals in time series data at different granularity levels. In particular, the authors link the forward process of the diffusion model to the data smoothing process. Motivated by this, this work develops a multi-granularity guidance diffusion loss function, such that the inherent features within data can be preserved and a regularized sampling path can be achieved. Experiments on real-world time series datasets demonstrate the effectiveness of the proposed approach.", "questions": "According to the inference procedure, the proposed time series forecasting approach can predict one future horizon time step at a time. How to apply this predictive approach for the long-term time series forecasting task effectively?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. In the context of time series forecasting, it is a good idea to stabilize the diffusion model with the help of coarse-grained temporal signals in time series data. \n2. The derivation of the multi-granularity guidance loss function is solid. Besides, the learning procedure devised in this work is applicable.", "weaknesses": "1. The assumption that \"... forward process of the diffusion model ... intuitively aligns with the process of smoothing fine-grained data into a coarser-grained representation ...\" is not verified through theoretical analysis or empirical study. \n\n2. Experimental settings for the time series forecasting task are unclear. For example, how the context interval and prediction interval are constructed in a time series dataset? Do you utilize a sliding window to roll the time series to build the context and predict intervals? Do the consecutive context/predict intervals overlap or not?\n\n3. More experimental results are expected. For instance, the authors only evaluate the time series forecasting methods under one length setting in each dataset, like the setting of context-24-predict-24 is utilized in Solar, Electricity, Traffic, Taxi, and KDD-Cup datasets.\n\n4. The reproductivity of this work is a concern.\n\n\n\\\\======================\\\\\n\nAfter rebuttal.\n\nMost of my concerns are clarified.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)