Diffusion-TS: Interpretable Diffusion for General Time Series Generation
Authors: Xinyu Yuan
Yan Qiao
Venue: ICLR 2024 poster (2024)
Keywords: Diffusion models
Synthetic Time series
Imputation
Forecasting
Primary area: generative models
TL;DR: We propose an interpretable diffusion model for generating time series (un)conditionally.

Abstract: Denoising diffusion probabilistic models (DDPMs) are becoming the leading paradigm for generative models. It has recently shown breakthroughs in audio synthesis, time series imputation and forecasting. In this paper, we propose Diffusion-TS, a novel diffusion-based framework that generates multivariate time series samples of high quality by using an encoder-decoder transformer with disentangled temporal representations, in which the decomposition technique guides Diffusion-TS to capture the semantic meaning of time series while transformers mine detailed sequential information from the noisy model input. Different from existing diffusion-based approaches, we train the model to directly reconstruct the sample instead of the noise in each diffusion step, combining a Fourier-based loss term. Diffusion-TS is expected to generate time series satisfying both interpretablity and realness. In addition, it is shown that the proposed Diffusion-TS can be easily extended to conditional generation tasks, such as forecasting and imputation, without any model changes. This also motivates us to further explore the performance of Diffusion-TS under irregular settings. Finally, through qualitative and quantitative experiments, results show that Diffusion-TS achieves the state-of-the-art results on various realistic analyses of time series.

Published as a conference paper at ICLR 2024
DIFFUSION-TS: INTERPRETABLE DIFFUSION FOR GENERAL TIME SERIES GENERATION
Xinyu Yuan, Yan Qiao
Hefei University of Technology 1ry53150gmal comm, giiaoya@@h.ut.edb.coo
ABSTRACT
Denoising diffusion probabilistic models (DDPMs) are becoming the leading paradigm for generative models. It has recently shown breakthroughs in au- dio synthesis, time series imputation and forecasting. In this paper, we propose Diffusion-TS, novel diffusion-basee framework that generates multivariate time series samples of high quality by using an encoder-decoder transformer with dis- entangled temporal representations, in which the decomposition technique guides Diffusion-TS to capture the semantic meaning of time series while transformers mine detailed sequential information from the noisy model input. Different from existing diffusion- based approaches, we train the model to directly reconstruct the sample instead of the noise in each diffusion step, combining Fourier-based loss term. Diffusion-TS is expected to generate time series satisfying both inter- pretablity and realness. In addition, is shown that the proposed Diffusion-TS can be easily extended to conditional generation tasks, such as forecasting and imputation, without any model changes. This also motivates us to further explore the performance of Diffusion-TS under irregular settings. Finally, through qual- itative and quantitative experiments, results show that Diffusion-TS -S achieves the tate-of-theearr art results on various realistic analyses of time series.
INTRODUCTION
Time series is ubiquitous in real-world problems, playing a crucial component in a wide variety of domains such finance, medicine, biology, retail, and climate modeling (Lim & Zohren (021).
However, lack of access to these dynamical data is a key hindrance to the development of machine learning solutions in some cases where data sharing may lead privacy breaches (Alaa al. 202T).
Synthesizing realistic time series data is viewed as a promising solution and has received increasing
attention driven by advances in deep learning. With perceptual qualities superior GANs while avoiding the optimization challenges of adversarial training, score- based diffusion models (Song
al.) [22TT: 2020). especially denoising diffusion probabilistic models (DDPMs) (Ho et al. 2020).
have taken the world of image, video, and text generation (Ho et al. 2022 Li et al. 2022a Dhariwal
& Nichol 2021
Harvey al. 2022) by storm than ever before.
It is hopeful that the diffusion models can be extended to the time-series area to tackle the chal- lenging problem of high-quality time series generation. Although some recent works pioneered the extension of diffusion models to time-series-related applications, almost all of them are designed for lask-agnostic generation (e.g., imputation (Tahiro et al., 2021 Alcaraz Strodthofi] 2022) and forecasting (Li et al. 2022b Shen & Kwok 2023)) that trai and sample with additional informa- tion. Meanwhile, the rare work on unconditional rmletee synthesis with diffusion models focus on synthesizing univariate (Kong et al., 2021| Kollovieh et al. 2023) short time series Lim et al. 2023). But first, these diffusion- methods (Lim et al.) 2023 Das et al. (023) typically employ Recurrent Neural Networks (RNNs) as the backbone to jointly model temporal dynamies and com- plex correlations. These autoregressive methods have limited long-range performance due to error accumulation and slow inference speed. The second challenge lies in that the plentiful combinations of independent components like trend, seasonality, and local idiosyncrasy of a ries are usually destroyed by gradually adding the noise to data in the diffusion process. As a result,
time se-
"Corresoonding author. The code is available http:://ghhh..cm////cccgyyy//nnn

---

Published as a conference paper at ICLR 2024
they can hardly recover the lost temporal dynamics since the temporal properties have not been in- tentionally preserved. It becomes exacerbated when time series has apparent seasonal oscillations since existing solutions lack the inductive bias to initiatively capture periodicity (LIU et al. 2022) Furthermore, it is also difficult for them to provide expert knowledge to explain both conditional and unconditional generation, and therefore they often lack interpretabilityy
To better tackle the aforementioned challenges, in this paper, we aim to develop Diffusion-TS, a non-autoregressive diffusion model for synthesizing time series of high quality in various scenarios, in which we explicitly model the temporal dynamics of highly complicated (e.g., multivariate and long-term) time series by introducing a transformer-basse architecture for the underlying model that learns disentangled scasonal-trent constitution of time series. This is achieved by imposing different forms of constraints on different representations. These disentangled representations not only offer Diffusion-TS an interpretable perspective on general synthetic tasks, they also plays a role in guiding the capture of complicated periodic dependencies beyond much simplified assumptions. Additionally, we design Fourier-based loss to reconstruct the samples rather than the noises in each diffusion step, which leads to a more accurate generation of the time series. Another notable design in Diffusion-TS is conditional generation method called reconstruction-batt makes the Diffusion-TS versatile for various conditional applications, such as time series imputation and forecasting, leading greater flexibility without requiring any parametrically updating.
sampling, which
In summary, our major contributions are as follows:
We propose a time series generation framework named Diffusion-TS. which combines seasonal-trend decomposition techniques with denoising diffusion models. This is achieved by a Fourier-based training objective, and the embedding of a deep decomposition archi- tecture. The framework allows the model to learn meaningful temporal properties from the data, making a highly efficient and interpretable solution for general time series genera- tion.
For conditional generation, adopt an instance-awane guidance strategy built target metric (e.g. reconstruction), which enables Diffusion-TS to adapt different controllable generative tasks in plug-and-play way.
Our experiments demonstrate that Diffusion-TS can generate realistic time series while maintaining high degree of diversity and novelty under challenging settings, and is com- petitive with existing diffusion-based methods designed for downstream applications. We also present the explanability of the model with several case studies.
PROBLEM STATEMENT
We denote X1:r = (#1,.... 27)= E rxd as a time series covering a period of r time steps, where d denotes the dimension of observed signals. Given the dataset DA { {Xi 1.r
N ] i=1 of N samples of time-series signals, our unconditional goal is to use a diffusion based generator to approach the
function of X = G(Z,) which maps Gaussian vectors Z,- ( ) E =xxx to the signals that are most similar with the signals in DA, where T' denotes the total diffusion step. In our method, we consider the following time series model with trend and multiple seasonality
zj = S Σ
81.j + Bj; j= 0,
where æ represents the observed time series, Sj denotes the trend component, $t,j is the i-th sea- sonal component and ej denotes the remainder part which contains the noise and some outliers at time j. And the goal of controllable generation is to generate samples from a conditional distribution p(-ly), where y is a control variable can be any real-world signal that will dictate the synthesis.
3 DIFFUSION- TS: INTERPRETABLE DIFFUSION FOR TIME SERIES
As aforementioned, time series usually exhibit complex patterns in many real-world scenarios. In- spired by the effectiveness of seasonal-trend decomposition analysis in time series modeling, the core idea of Diffusion-TS TS is to introduce an interpretable decomposition architecture to the under- lying network based on a transformer. We take such formulation for three main reasons: (i) the use of disentangled patterns in the diffusion model has yet to be explored; (ii) our method is highly
2

---

Published as a conference paper at ICLR 2024
My MYNUMAMN
q(x (xx1)
wwwhhpphhho Jo MuMPRAAAii
X.
X
(x (x,,)
Interpnetbble syathesis
Reverse Process
The forward and reverse diffusion on time series data in Diffusion-TS The denoising network learns predict the clean time series dp from Da based on interpretable decomposition. In the reverse pass, the generator network gradually injects the expert knowledge to make the synthetic series target move toward the real ones.
Figure 1:
interpretable owing to the specific designs of architecture and objective; (iii) temporal information isolated into several parts, and it enables us to capture the potentially complex dynamics from diffused data with explainable disentangled representations. The learned disentanglement also tends to make the imputation forecasting process more reliable (LIU et al. 2022). Thus finally, will show how to conduct time series imputation and forecasting by the trained off-the-shelf diffusion model during the sampling process.
3.1 DIFFUSION FRAMEWORK
As shown in Figure 1 start by introducing the diffusion model that typically contains two pro- cesses: forward process and reverse process. In this setting, a sample from the data distribution To ~ ((x) is gradually noised into standard Gaussian noise Øт No0, I) by the forward process, where the transition is parameterized by ((=)(x,-1) = W(;;; v1 B 11 with 3r E (0, 1) as the amount of noise added diffusion step Then a neural network learns the reverse process of gradually denoising the sample via reverse transition pe (xt-1//zz) = N(xx-11 (1 t), Ce(r,, t)). Learning to clean FT through the reversed diffusion process can be reduced to learning to build surrogate approximator to parameterize Ho (xt, t) for all t. Ho et al.] (2020) trained this denoising model μo ( f) using a weighted mean squared error loss which we will refer to as
T
L(xo) Σ
[1 mo) μs (xx, 1)]1
where (0)) is the mean of the posterior g( 2t). This objective can be justified as optimizing a weighted variational lower bound on the data log likelihood. Also note that the original parameterization μa (Z, t) can be modified in favour of eo 0) or eg(x,, t). Please refer to Appendix B for details.
3.2 DECOMPOSITION MODEL ARCHITECTURE
Here on high level, we choose an incoder-decoder transformer that enhances the models' ability capture global correlation and patterns of time series. This way, information of the entire noisy sequence is encoded before decoding. We renovate the decoder as deep decomposition architecture as shown in Figure 2 The decoder adopts multilayer structure in which each decoder block contains transformer block, feed forward network block and interpretable layers (Trend and Fourier synthetic layer). Detailed descriptions of the whole model can be found Appendix E we now ready to elaborate on the details of the disentangled representation.
We achieve the disentanglement by enforcing distinct forms constraints on different components, which introduce distinct inductive biases into these components and make them more liable to learn specific semantic knowledge. Trend representation captures the intrinsic trend which changes gradu- ally and smoothly, and seasonality representation illustrates the periodic patterns of the signal. Error
3

---

Published as a conference paper at ICLR 2024
representation characterizes the remaining parts after removing trend and periodicity. Before the start, we define the imput of interpretable layers as (t) where i L...., denotes the index of the corresponding decoder block at diffusion step t.
Trend Synthesis. The trend component describes the smooth underlying mean of the data, which aims to model slow-varying behavior. To produce reasonable trend components, We use the poly- nomial regressor (Oreshkin al. 2020, Desai et al. (222)) to model the trend Vi as follows:
1> Σ (C (inearr(w())) + X;x") C [1, i=1
Vt.
where X is the mean value of the output of the 7th decoder block, and '.' denotes tensor multi- plication, Here slow-varying poly space C' the matrix of powers of vector e = [0, 1, 2,,,................ ......T............. 2, 1/"/T, /T, and p is small degree (e.g.
3) to model low frequency behavior.
Seasonality & Error Synthesis. In this part, we will try to recover components other than trends in the model input. This includes the periodic components (Seasonality) and the non-periodic ones (Error). The main challenge is automatically identify seasonal patterns from the noisy input Et. Inspired by the trigonometric representation of seasonal components based on Fourier series
(De Livera al. 2011 Woo al.) 2022), we capture the seasonal component of the time series in
Fourier synthetic layers using Fourier bases:
F(wseas))
Pit = (F(w)eas)))
(4)
(1)
(K)
arg Topk (4())
(5)
ke/1,=,1t/2/+11
S A co TC i ) cos ay 2
where arg TopK is to get the top K amplitudes and K is hyperparameter. A!) 2 are the phase, amplitude of the k-th frequency after the discrete Fourier transform F respectively. fú represents the Fourier frequency of the corresponding index k. and (-) denotes (-) of the corresponding conjugates. In fact, the Fourier synthetic layer selects bases with the most significant amplitudes in the frequency domain, and then returns to the time domain through an inverse transform to model the seasonality. Finally, we can obtain the original signal by the following equations:
6o t, 0) = V Σ St. + R.
where R is the output of the last decoder block, which can be regarded as the sum of residual periodicity and other noise.
3.3 FOURIER-BASED TRAINING OBJECTIVE
To enforce the model unsupervised uncover these time-series components, we train the neural net- work to predict estimate fo(#,, 1, directly. Then the reverse process shown in Figure can be approximated via
Vo
Var(1 ai-1) Dt
1 āi 3,
#t-1
do Ft, t,
(8)
1 1-
āi
where z, N' I), an
1 Br and I Os.- We take the following reweighting strategy:
A1x (1
Esimpte Er Es we || (o 4, 12 W.
3
where À is a constant, i.c. 0.01. Similar to Ho et al. (2020). these loss terms are down-weighted at small t force the network focus on larger diffusion step.
In addition, we propose to guide the interpretable diffusion training by applying it to the frequency domain with the Fourier transform, a mathematical operation that converts a finite-length time do- main signal to its frequency domain representation (Bracewell & Bracewell] 1986). Fons et al. (2022) shows that the Fourier-based loss term beneficial for the accurate reconstruction of time series signals. Formally, we have
La = Er,0
Wt Ax - Lo 1, 12 >a])FFT((n) - (8T(
(10)
4

---

Published as a conference paper at ICLR 2024
Tensor Multtplication
wete Substraction
Element wise Addition
Fostionall Encoding
Connect to noet blok
Input of each block
X
t
Encoder Output
Sesoneliiy & Enor
Denze Lager
Reshape
Transformer Block
C
Trend Synthetic Layer
Feed Forword
Cosy Lxt
T Trend Taper
Founier Synthetic Layer
Fourler Tinnsform
Top & Selecruna
Meen
Decodlr
Inverso Fourler
Decaker Flock2
Uransioo..
Decoder Blokk
Taund
Fourier Systhelic Layer
Figure 2: The network architecture of the decoder in proposed 2g approximator.
where FFT denotes the Fast Fourier Transformation Elliott Rao 1982, and Y1, 22 are weights to balance two losses.
3.4 CONDITIONAL GENERATION FOR TIME SERIES APPLICATIONS
The above has described the details of the unconditional time series generation. In this section, will describe conditional extensions of the Diffusion-TS, in which the modeled 2o is conditioned targets y. The goal is utilize pre-trained diffusion model and the gradients of classifier to approximately sample from the posterior p(Zo.T lb) II p(z y), where p(=;-1 y) x
p(z+-1(xz) p(y] 2.). Here using Bayes' theorem, we run gradient update on Iz-1 to control generation via the following score function:
V -イ- log p(r,,1lZze y) VaL. logp(z;-11z2) + V 2 log ((,
(11)
where logp(z; (rt) is defined by diffusion models, while log x, - parametrized by an classifier, which can be approximated by V log p(w) proved in Chung et al. (2022
The method can be explained as a way to guide samples towards areas where the classifier has high likelihood. Then given conditional part xa and generative part Ib, proposed method to approximate conditional sampling for imputation and forecasting can be defined as follows:
6n t, f) = (x,,, t, 0) + V.. (I 2, (x,, t, 8) 2 + log ((=;,1lax)),
(12)
where 7 is hyperparameter that trades off functions (the first for conditional consistency and the second for better fluency). The gradient term can be interpreted as a reconstruction-basee
guidance, with 77 controlling the strength. Following Li al. (2022a). we repeat multiple steps of the gradient update for each diffusion step to improve the control quality. Then by replacing i (v,, 1, Vogzat / 04C, the sample xt-1 will be generated using the new To. Algorithms in Appendix lay out how such sampling scheme is used for conditional generation.
EMPIRICAL EVALUATION
In this section, we first study the interpretable outputs of the proposed model. Then we evaluate our method in two modes: unconditional and conditional generation, to verify the quality of the generated signals. For time series generation, select four previous models to compare with: TimeVAD (Desai et al.) 2021). Diffwave (Kong et al. 2021), TimeGAN (Yoon et al., 2019), Cot- GAN Xu et al.] 2020) and DiffTime (Coletta al. 2023) which be viewed
unconditional CSDI (Tashiro et al. 2021). We also compare to the CSDI and SSSD (Alcaraz & Strodthoff 2022)
on conditional tasks. Finally, we conduct experiments to validate the performance of Diffusion- TS when clean data is not sufficient. Implementation details and ablation study can be found in Appendix G aadd(C.] respectively.
5

---

Published as a conference paper at ICLR 2024
4.1 DATASETS
We use real-world datasets and 2 simulated datasets in Table 11 to evaluate our method. Stocks the Google stock price data from 2004 to 2019. Each observation represents one day and has features. ETTTh dataset contains the data collected from electricity transformers, including load and oil temperature that are recorded every 15 minutes between July 2016 and July 2018. Energy is a UCI appliance energy prediction dataset with 28 values. IMRI is benchmark for causal discov- ery, which consists of realistic simulations of www.eeoleeee--eeeeeeeeee Here, select a simulation from the original dataset which has 50 features. Sines has features where each feature created with different frequencies and phases independently. MuJoCo is multivariate physics simulation time series data with 14 features.
(BOLD) time series.
4.2 METRICS
For quantitative evaluation of synthesized data, we consider three main criteria (1) the distribu- tion similarity of time series; (2) the temporal and feature dependency; (3) the usefulness for the predictive purposes. We adopt the following evaluation metrics (see Appendix [G.3 for the detailed descriptions): 1) Discriminative score (Yoon et al. 2019) measures the similarity using classifica- tion model to distinguish between the original and synthetic data as a supervised task; 2) Predictive score Yoon et al.] 2019) measures the usefulness of the synthesized data by training post-hoc sequence model to predict next-step temporal vectors using the train-sy train-synthesis- and-test-real (TSTR) method: 3) Context-Fréchhe Fréchet Inception Distance (Context-FID) score (Paul et al. 2022) quantifies the quality of the synthetic time series samples by computing the difference between representations of time series that fit into the local context; 4) Correlational score (Ni et al. 2020) uses the abso- lute eπrоr between cross correlation matrices by real data and synthetic data to assess the temporal dependency.
4.3 INTERPRETABILITY RESULTS
1.5
0.
0,5
000
-1.0
-10
-325
-00
-2.
1.20
-0.7
20
0.0m
1.0
2.4
-26
1.0
-10
100
-36
20
300
(a) Input
(b) Origin
(c) Output
(d) Trend
(c) Scason & Error
Figure 3: The reconstruction performance of Diffusion-TS with the interpretable configurations. Each row a time series example per dataset, top to bottom (Sines, Stocks and ETTh).
We start by analyzing the reconstruction performance of our model for multivariate time series. Fig- ure3 illlustrates demos of the original and reconstructed samples by Diffusion-TS in three datasets. The model takes the corrupted samples (shown in (a)) with 50 steps of noise added as input, and outputs the signals (shown in (c)) that try to restore the ground truth (shown in (b)), with the aid of the decomposition of temporal trend (shown in (d)) and season & error (shown in (e)). As would be expected, the trend curve follows the overall shape of the signal, while the season & error os- cillates around zero. Fusing the two temporal properties, the reconstructed samples can be a great agreement with the ground truth. Our conclusion here is that by introducing interpretable architecture, the time series generated by the Diffusion-TS can exhibit explainability of disentangle- ment with almost no accuracy loss. Additionally, we also conduct case study for further validating interpretability on synthetic data in Appendix

---

Published as a conference paper at ICLR 2024
4.4 UNCONDITIONAL TIME SERIES GENERATION
In this part, we first replicate the experimental setup in Yoon et al. (2019) to report the performance of the model with respect to the aforementioned benchmarks, Afterwards, we investigate the ability of the models to synthesize longer time series data.
Table Results on Multiple Time-Series Datasets (Bold indicates best performance).
Metric
Methods
Sines
Stocks
ETTh
MuJoCo
Energy
fMRI
Diffusion-TS
0,006=.000
0,147# 025
0.116*.010
0.013#.001
0.089:.024
0.105+.006
TimeGAN
0.1
0,103#.013
0.300s.013
0,563=0052
0.767#.100 103
1.2924.218
Context-FID
TimoVAE
0.307=.060
0.215+.035
0.8054.188
0.251=0015
1.631 142
14.449=9969
Score
Diffwave
0.0142.002
0.232+.032
0.8734.061
0.393=.041
1.0314.131
0.2441.018
DiffTime
0.006+.001
0.2364.072 074
0.299+.044
0.188 028
0.2794.045
0.340+.015
(Lower the Better)
Cot-GAN
1.337=.068
0.408+1086
0.980:.071
1.094.0799
1.039+.028
1.813+.550
Diffusion-TS
0.015#.004
0.004a.001
0.049
0.193a.027
0.8562.147
1.411
TimeGAN
0.045=.010
0.063+ 005
0.210#0006
0.886*.039
4.010# 104
23.502= 039
Correlational
TimeVAE
0.131=0010
0.095#.000 008
0.111=020
0.388=001
1.688+.228
17.296+.55 526
Score
Diffwave
0,030+.020 0.022#0005
0.175
0.5
5.001 154
3.9274.049
DiffTime
0.017+0004
0.006+ 002
0.067+.005
0.218+001
1,158+0095
1.501+.045 048
(Lower the Better)
Cot-GAN
0.049=0010
0.087#.004
0.249*.009
1.042+0007
3.164+0061
26.824 449
Diffusion-TS TS
0.0062.007
0.067+.015
0.061#.009
0,008=t002
0.122 003
0.1672.023
TimeGAN
0.011+0008
0,102+.021
0.1142.055
0.238:.068
0.236 012
0.484:1.42
Discriminative
TimeVAE
0.041#.044
0.145 120
0.209+.058
0.230s1002
0.499+.000 000
0.4764.044
Score
Diffwave
0.017+0008
0.232+ 061
0.190#0008
0,203+.096
0,4933.00 004
0.402+. 029
DiffTime
0,013=t006
0.097+.016
0. 0.100+0007
0.154
0.445#t004
0.245+.051
(Lower the Better)
Cot-GAN
0.254
0,230+.016
0.325+.099
0.426+.023
0.498 002
0.4924.018
Diffusion-TS
0.093#.000
0.036+.000
0.119:.002
0.007=.000
0,250=
0.099:.000
TimeGAN
0.093=0019
0.038 001
0.124:.001
0.025=0003
0.273+. 004
0.1
Predictive Score
TimeVAE
0.093+0000
0.039#.000
0.1262.000 004
0.012=0002
0.292+.000
0.113+0003
Diffwave
0,093+.000
0,047#0000
0.130+0001
0.013=0000
0.251+0000
0,101+0000
DiffTime
0.093=0000 000
0.038+ 001
0.121#.004
0,010=000
0.252 000
0,100+0000
(Lower the Better)
Cot-GAN
0.100=0000
0.047+0001
0. 0.1292.000
0.068=0009
0.259 000
0.1
Original
0.094#.001
0.036+.00 001
0.121#.005
0.007=0001
0.250+.00 003
0.090+0001
In Table T we list the results of the 24-1ength time series generation used in most of the existing related works. It shows that Diffusion-TS consistently produces higher-quality synthetic samples over other baselines in terms of almost all metrics. For example, Diffuision-TS can remarkably improve the discriminative score by an average of 50% in all six datasets. It is also notable that for high-dimensional datasets (i.e., MuloCo, Energy and fMRI), the leading performance of Diffusion- TS is even significant. That demonstrates that Diffusion-TS complex time series synthesis. Table the results of long-term time series generation on ETTh and Energy datasets. We randomly generate 3000 sequences with various lengths (64, 128, 256), then use aforementioned metrics to assess the generation quality of different methods. From the results, Diffusion-TSS achieves the best overall performance, implying the efficacy of interpretable decomposition for long-term time-series modelling. Notably, different from other baselines, the performance of Diffusion-TS changes quite steadily as the length of sequence increases. means Diffusion- TS retains better long-term robustness, which is meaningful for real-world applications.
well tackle the challenges of
Dr Original
10
TimeGAN
-10
Diffusion-Ts
Onip nal
Origina nal TimeGaM
-10-
Original Difusier-Ts
-10-
-10
30
-10
10
-10
-5
10
-10
10
D3ta Denety Extimate
Data Denstty Entimate
Or Orignnal TimeGAN
Data Denaby Edtimate
Original Diffusion
Densty Extimate
Original TimeGAN
Original Diffusion-T
+
8.6
0.8
0.8
(a) ETTh
(b) Energy
Figure 4: Visualizations the time series synthesized by Diffusion-TSS and TimeGAN.
7

---

Published as a conference paper at ICLR 2024
To visualize the performance of time series synthesis, we adopt two visualization methods. One is to project original and synthetic data in 2-dimensional space using t-SNE (Van der Maaten & Hinton 2008). The other is to draw data distributions using kernel density estimation. As shown in the 1st row in Figure E and pictures Appendix C.2 Diffusion-TS overlaps original data areas markedly
better than TimeGAN. The 2nd row in Figure 4 shows that the synthetic data's 's distributions from Diffusion-TSs are more similar to those of the original data than TimeGAN, For more visualizations and distributions, please refer Appendix H
0.7
CD
1000
0.6
Volue
70
40
0.4
4.0
20
40
0.4
20
40
Time
Time
Time
Time
1.00
1,00
0.75
0.75
0.66
0.50
0.50
0.2
15
0.00
0.00
Time
Time
(a) ETTh
(b) Energy
Figure 5: Examples of time series imputation (1*t row) and forecasting (2nd row) for ETTh and Energy datasets. Green and gray colors correspond to Diffusion-T and CSDI, respectively.
4.5 CONDITIONAL TIME SERIES GENERATION
Here, we present the conditional generation on time series imputation and forecasting. For impu- tation tasks, we apply masking strategy following a geometric distribution used in Zerveas et al. (2021), which controls both the lengths of the missing sequences and the missing ratio r, instead of selecting the missing points randomly, since single missing time point may often be easily pre- dicted from the immediately preceding and succeeding points. For both tasks, the length of time series is set as 48 time steps, for which given the first w continuous time points, we forecast on the remaining 48 w time points. We provide imputation and forecasting examples ETTh and En- ergy datasets in and the blue circles show the ground-truth imputation targets. The median values imputations are shown as the line and 5% and 95% quantiles are shown as the shade. It indicates that Diffusion- - gives more reasonable imputations and predictions with high confidence against CSDI.
(more examples in Appendix D The red crosses show the observed values
0.12
001000
Soaarge Error
0.08
0.10
0.06
Sqarrg
0.04.
0.06
30.025
0.02
10%
25%
50% Missing ratio
75%
24 Forecasting Window
10%
25%
50%
75%
Missing ratic
Forecasting Window
(a) ETTh
(b) Energy
Figure 6: Performance of diffusion methods for time-series imputation and forecasting. Mean and confidence intervals are obtained by running each method 5 times.
We also run detailed experiments to evaluate our conditional extension. Diffusion-TS (Diffusion- TS-G) based on the aforementioned Langevin sampler is compared with CSDI, the original Diffwave and Diffusion-TS-R using replace-based conditional sampling (see Appendix [Dr detailed descrip- tions for the conditional adaptation). In Figure we show the empirical results, where Diffusion-TS- G outperformed all baselines even at high levels of missing ratio. We can observe that Diffusion- TS-R achieves close accuracy the low missingness ratio, but for the high missing ratio 75% and 90%, there is a clear improvement with the reconstruction reconstruction-guided sampling strategy. It means that ceasonal-trend decomposition naturally facilitate infilling restrictions increase, however,
Appen
8

---

Published as a conference paper at ICLR 2024
when the missing ratio is high and there are additional constraints, it may also make the entire sequence deviate from the ground truth. For completeness, we conduct additional experiments and
include other time series baselines such TLAE (Nguyen & Quanz 202T) in Appendix C. We
can see our model still performs well, as it has the best performance against the haselines.
Cold Starts and Irregular Settings. In this experiment, we explore whether the Diffusion-TS can keep performance from degrading under regular data deficit condition. Cold start refers to training model when little no data is available. We use only 10/25/50/75% of the time series data in each dataset as training data respectively. Then we remove time series values from the rest dataset, leading to 10% ~ 20% missing values for data point. After cold starts the regular part of the dataset, we leverage the model to impute these irregular time series then continue training by incorporating them into the training set. We call this process irregular training and Figure TT shows the results. We observed that even at the 10% training threshold, Diffusion-Te TS still has superior results on the multiple time-series datasets compared to baselines in Table I In addition, Diffusion-T3 shows better discriminative and predictive scores in all cases after restoration. The performance of the model always at the same level as if there were no missing data, which shows the efficacy of the Diffusion-TYs TS under insufficient and irregular settings.
4.6 ABLATION STUDY
To evaluate the effectiveness of the Diffusion-TSS, we compare the full versioned Diffusion-TS with its three variants: (1) w/o FFT, i.c. Diffusion-TS without Fourier-based loss term during training, (2) w/o Interpretability, i.e. Diffusion- -S without seasonal- -trend design in the network, (3) w/o Transformer, i.e. Diffusion-TS based on a Convolutional network without encoder and self-attention mechanism. (4) e-prediction, Diffusion-TS using traditional noise prediction parameterization for training and sampling. Results are presented in Table 2 (see detailed evaluation in Appendix
Table 2: Ablation study for model architecture and options. (Bold indicates best performance).
Metrie
Method
Sines
Stocks
ETTh
Mujoco
Energy
fMRI
Diffusion-TT TS
0,8162.007
0.067=.015
0.0612.009
0.0081.002
0.122:.003
0.167s.023
Discriminative Score
w/o FFT w/o Interpretubility
0,007+.006
0.127=011
0.096#:007
0.010#.002
0.135+0004
0.177+0013
0.009+0006
0.101
0.071+0010
0.021+.014
0.125+.003
0.267# 034
w/o Transforer
0.0101.007
0.104:.024
0.0824.006
0.0391.014
0.3241.015
0.123:..064
(Lower the Better)
e-prediction 0.0401.011
0.131=0014
0.099::010
0.023+.006
0.197=0001
0.1682.030
Diffusion-TS w/o H+T
0,093+.000
0.036:.000
0.119:.002
0,007+.000
0.250=.000
0,099
Predictive
0.093+.000
0.03
0.121#.004
0.008+.001
0,250=0000
0.100+0001
Score
w/o Interpretability
0.0931.000
0.037=0000
0.119
0.0081.001
0.2511.000
0.1011.000
w/o Transformer
0,093+.000
0.036:.000
0.126*#.04
0.008+0000
0.319#0006
0,099:.000
(Lower the Better)
c-prediction
0.097+.000
0.039=0000
0.120t.000 002
0.008+0001
0,00
0.100= 000
Original
0.094+.001
0.0362.001
0.121::005
0.007+.001
0.250 003 0.090+.001
We can find that the best result on each dataset is usually obtained with Diffusion-TS. Removing the attention and residuals often causes big performance drops. However, when the dataset, i.c. fMRI, has high frequency and dimension, network with only an interpretable design achieves the most significant performance improvement. In addition, the FFT-loss term and signal prediction parameterization show a crucial role in general. The conclusion is that our design is relatively steady across all experiment settings. And the decomposition not only adds interpretable capability without losing accuracy, but also does boost the unconditional generation of the diffusion model when there obvious frequency fluctuations in the data set.
CONCLUSIONS
In this paper, we have proposed Diffusion-TSS a DDPM-based method for general time series gen- eration. In addition to the generative capability of DDPMs, the Diffusion-TS TS is powered by the TS-specific loss design and transformer-basee deep decomposition architecture. Besides, our model trained for unconditional generation could be readily extended for conditional generation by com- bining gradients into the sampling. The experiments have shown our model is capable of a wide range of time-series generative tasks and can achieve competitive performance. Shown in Table g, one notable limitation of DDPMs is the high cost of inference, which is demanding for more computing resources to generate sample compared with GAN-based approaches, although our derlying model is lightweight anyway. To improve Diffusion-T TS for which both conditional and unconditional inference procedures converge in shorter time would be a potential future work.

---

Published as a conference paper at ICLR 2024
REFERENCES
Ahmed Alaa, Alex James Chan, and Mihacla van der Schaar. Generative time-series modeling with fourier flows. International Conference on Learning Representations, 2021.
Juan Miguel Lopez Alcaraz and Nils Strodthoff. Diffusion-based time series imputation and fore- casting with structured state space models. arXiv preprint arXiv:2208.099: 2022.
Alexander Alexandrov, Konstantinos Benidis, Michael Bohlke-Schneider, Valentin Flunkert, Jan Gasthaus, Tim Januschowski, Danielle C Maddix, Syama Rangapuram, David Salinas, Jasper Schulz, et al. Gluonts: Probabilistic and neural time series modeling in python. The Journal of Machhne Learning Research,
4634, 2020.
Ronald Newbold Bracewell and Ronald Bracewell. The Fourier transform and its applications, volume 31999. McGraw-Hill New York, 1986.
Hyungjin Chung, Jeongsol Kim, Michael T Mccann, Mare Klasky, and Jong Chul Ye. Diffusion posterior sampling for general noisy inverse problems. arXiv preprint arXiv:2209.1468) 2022.
Robert B Cleveland, William S Cleveland, Jean E McRae, and Irma Terpenning. Stl: A seasonal- trend decomposition. J. Off. Stat, 6(1 6-13,3773, 1990.
Andrea Coletta, Sriram Gopalakrishan, Daniel Borrajo, and Svitlana Vyetrenko. On the constrained time-series generation problem. arXiv preprint urXiv:330.001717, 2023.
Ayan Das, Yongxin Yang, Timothy Hospedales, Tao Xiang, and Yi-Zhe Song. Chirodiff: Modelling chirographic data with diffusion models. arXiv preprint arXiv:2304.0378. 2023.
Alysha M De Livera, Rob Hyndman, and Ralph D Snyder. Forecasting time series with complex seasonal patterns using exponential smoothing. Journal of the American statistical association, 106(496):1513-1527 2011.
Abhyuday Desai, Cynthia Freeman, Zuhui Wang, and Ian Beaver. Timevae: A variational auto- encoder for multivariate time series generation, 2021.
Prafulla Dhariwal and Alex Nichol. Diffusion models beat gans on image synthesis, 2021.
Alexander Dokumentov and Rob J. Hyndman. Str: Seasonal-trend decomposition using regression, 2021.
DF Elliott and KR Rao. Fast fourier transform and convolution algorithms, 1982.
Cristóbal Esteban, Stephanie L Hyland, and Gunnar Rätsch. Real-valued (medical) time series generation with recurrent conditional gans. arXiv preprint arXiv: 1tt.11766002633. 2017.
Elizabeth Fons, Alejandro Sztrajman, Yousef El-laham, Alexandros Iosifidis, and Svitlana Vyetrenko. Hypertime: Implicit neural representation for time series, 2022.
Shuyang Gu, Dong Chen, Jianmin Bao, Fang Wen, Bo Zhang, Dongdong Chen, Lu Yuan, and Baining Guo. Vector quantized diffusion model for text-to-image synthesis, 2022.
William Harvey, Saeid Naderiparizi, Vaden Masrani, Christian Weilbach, and Frank Wood. Flexible diffusion modeling of long videos, 2022.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models, 2020.
Jonathan Ho, Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David J. Fleet. Video diffusion models, 2022.
S. Rao Jammalamadaka, Jinwen Qiu, and Ning Ning. Multivariate bayesian structural time series model, 2018.
Daniel Jarrett, Ioana Bica, and Mihaela van der Schaar. Time-series generation by contrastive imi- tation. Advances in Neural Information Processing Systems, 34:28968-28999 2021.
10

---

Published as a conference paper at ICLR 2024
Jinsung Jeon, Jeonghak Kim, Haryong Song. Seunghyeon Cho, and Noseong Park. Gt-gan: General purpose time series synthesis with generative adversarial networks. Advances in Neural Informa- fion Processing Systems, 35:36999-37011 2022.
Diederik Kingma and Max Welling.
Auto-encoding variational bayes. arXiv preprint
arXiv:1312.61 14, 2013.
Marcel Kollovieh, Abdul Fatir Ansari, Michael Bohlke-Schneidee Jasper Zschiegner, Hao Wang, and Yuyang Wang. Predict, refine, synthesize: Self-guiding diffusion models for probabilistic time series forecasting, arXiv preprini wrXi::2307.1449: 2023.
Zhifeng Kong, Wei Ping, Jiaji Huang, Kexin Zhao, and Bryan Catanzaro. Diffwave: A versatile diffusion model for audio synthesis, 2021.
Xiang Lisa Li, John Thickstun, Ishaan Guirajani, Perey Liang, and Tatsunori B. Hashimoto. Diffusion-Im improves controllable text generation, 2022a.
Yan Li, Xinjiang Lu, Yaqing Wang, and Dejing Dou. Generative time series forecasting with dif- fusion, denoise, and disentanglement. Advances in Neural Information Processing Systems, 35: 23009-23022, 2022h.
Bryan Lim and Stefan Zohren. Time-series forecasting with deep learning: a survey. Philosophical Transactions of the Royal Society A, 379(219)):20200099, 2021.
Haksoo Lim, Minjung Kim, Sewon Park, and Noseong Park. Regular time-series generation using sgm. arXiv preprint arXiv:2301.085// 2023.
SHUAI LIU, Xiucheng Li, Gao Cong, Yile Chen, and YUE JINGG. Multivariate time-series impu- tation with disentangled temporal representations. In The Eleventh International Conference on Learning Representations, 2022.
Zhengxiong Luo, Dayou Chen, Yingya Zhang, Yan Huang, Liang Wang, Yujun Shen, Deli Zhao, Jingren Zhou, and Tieniu Tan. Videofusion: Decomposed diffusion models for high-quality video generation, 2023.
Olof Mogren.
C-rnn-gan: Continuous recurrent neural networks with adversarial training, 2016.
Nam Nguyen and Brian Quanz. Temporal latent auto-encoder: A method for probabilistic multi- variate time series forecasting. In Proceedings of the AAAI Conference Artificial Intelligence, volume 35, pp. 9117-9125, 2021.
Hao Ni, Lukasz Szpruch, Magnus Wiese, Shujian Liao, and Baoren Xiao. Conditional sig- wasserstein gans for time series generation. arXiv preprint arXiv:2006.0544 05422, 2020.
Boris N. Oreshkin, Dmitri Carpov, Nicolas Chapados, and Yoshua Bengio. N-beats: Neural basis expansion analysis for interpretable time series forecasting, 2020.
Jeha Paul, Bohlke-Schneider Michael, Mercado Pedro, Kapoor Shubham, Singh Nirwan Rajbir, Flunkert Valentin, Gasthaus Jan, and Januschowski Tim. Psa-gan: Progressive self attention gans for synthetic time series, 2022.
Hengzhi Pei, Kan Ren, Yuqing Yang, Chang Liu, Tao Qin, and Dongsheng Li. Towards generating real-world time series data. In 2021 IEEE International Conference Data Mining (ICDM), pp. 478. IEEE, 2021.
Kashif Rasul, Calvin Seward, Ingmar Schuster, and Roland Vollgraf. Autoregressive denoising dif- fusion models for multivariate probabilistic time series forecasting. International Conference on Machine Learning, pp. 8857-8868. PMLR, 2021.
Steven I. Scott and Hal R Varian. Bayesian variable selection for nowcasting economic time series. In Economic analysis of the digital economy, pp. 119-135. University of Chicago Press, 2015.
Lifeng Shen and James Kwok. Nut-uutoreeeessive conditional diffusion models for time series prediction. arXiv preprint arXiv:2306.05000 055043 2023.

---

Published as a conference paper at ICLR 2024
Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, et al. Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:2011.13456 2020.
Yang Song, Conor Durkan, lain Murray, al. Maximum likelihood training of score-based diffusion models. Advances in Neural Information Processing Systems,
2021.
Robin Strudel, Corentin Tallec, Florent Altché, Yilun Du, Yaroslav Ganin, Arthur Mensch, Will Grathwohl, Nikolay Savinov, Sander Dieleman, Laurent Sifre, and Rémi Leblond. Self- conditioned embedding diffusion for text generation, 2022.
Yusuke Tashiro, Jiaming Song, Yang Song, and Stefano Ermon. Csdi: Conditional score-based diffusion models for probabilistic time series imputation, 2021.
Marina Theodosiou. Forecasting monthly and quarterly time series using stl decomposition. Inter- national Journal of Forecasting, 27:1 17:1178-1195, 10 2011. doi: 10. 1f.0016..jforreast2.00..011
Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, 9(11), 2008.
Vikram Voleti, Alexia Jolicoeur- Marttnnaa,,, and Chris Pal. Mevd-masked conditional video diffusion for prediction, generation, and interpolation. Advances in Neural Information Processing Systems, 35:23371 -23385, 2022.
Qingsong Wen, Jingkun Gao, Xiaomin Song, Liang Sun, Huan Xu, and Shenghuo Zhu. Robuststl: A robust seasonal-trend decomposition algorithm for long time series, 2018.
Qingsong Wen, Zhe Zhang, Yan Li, and Liang Sun. Fast robuststl: Efficient and robust seasonal- trend decomposition for time series with complex patterns. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery Data Mining, pp. 2203-2213, 2020.
Gerald Woo, Chenghao Liu, Doyen Sahoo, Akshat Kumar, and Steven Hoi. Etsformer: Exponential smoothing transformers for time-series forecasting. arXiv preprint arXiv:2202.013881 2022.
Haixu Wu, Jiehui Xu, Jianmin Wang, and Mingsheng Long. Autoformer: Decomposition trans- formers with auto-correlation for long-ten term series forecasting. Advances in Neural Information Processing Systems, 34:22419-22430, 2021.
Tianlin Xu, Li Kevin Wenliang, Michael Munn, and Beatrice Acciaio. Cot-gan: Generating sequen- tial data via causal optimal transport. Advances in neural information processing systems, 33: 8798-8809, 2020.
Dazhi Yang, Vishal Sharma, Zhen Ye. Li Hong Idris Lim, Lu Zhao, and Aloysius Aryaputera. Fore- casting of global horizontal irradiance by exponential smoothing, using decompositions. Energy, 81:111-119, 03 2015. doi: 10. 00.1016/j.enery.// 201.. 14.11.082.
Jinsung Yoon, Daniel Jarrett, and Mihaela Van der Schaar. works. Advances in neural information processing systems, 32, 2019.
generative adversarial net-
Peiyu Yu, Sirui Xie, Xiaojian Ma, Baoxiong Jia, Bo Pang, Ruiqi Gao, Yixin Zhu, Song- Song-Chun Zhu, and Ying Nian Wu. Latent diffusion energy-based based model for interpretable text modeling. 2022.
Zhihan Yue, Yujing Wang, Juanyong Duan, Tianmeng Yang. Congrui Huang, Yunhai Tong, and Bixiong Xu. Ts2vec: Towards universal representation of time series. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pp. 8980-8987, 2022.
Luca Zancato, Alessandro Achille, Giovanni Paolini, Alessandro Chiuso, and Stefano Soatto. Strie: Stacked residuals of interpretable components for time series anomaly detection. 2022.
George Zerveas, Srideepika Jayaraman, Dhaval Patel, Anuradha Bhamidipaty, and Carsten Eick- hoff. A transformer- based framework for multivariate time series representation learning. In Proceedings of the 27th ACM SIGKDD conference on knowledge discovery & data mining, pp- 2124, 2021.
Tian Zhou, Ziqing Ma, Qingsong Wen, Xue Wang, Liang Sun, and Rong Jin. Fedformer: Frequency enhanced decomposed transformer for long-term series forecasting, 2022.
12

---

Published as a conference paper at ICLR 2024
A RELATED WORK
Time Series Generation. Deep generative models of all kinds have exhibited high-quality samples in a wide variety of domains, where time series generation is one of the most challenging tasks in the generative research field. Existing literature for time-series synthesis is based predominantly on generative adversarial networks (GANs), and many GAN-based architectures use recurrent networks
to model temporal dynamics (Mogren 2016 Esteban et al. 2017; Yoon et al. 2019 Pei et al. 202T] Jeon et al. (2022). Among these based works.[|ogrent (2016) first introduced a GAN structure with LSTM called C-RNN-GAN. Then a Recurrent Conditional GAN (RCGAN) (Esteban et al.
2017) was proposed which uses a basic RNN as generator and discriminator and auxiliary label information as a condition to generate medical time series. Yoon et al. developed TimeGAN (Yonn et al.] (2199) by adding an embedding function and supervised loss to the original GAN for capturing the temporal dynamics of data throughout time. addition, Pei et al. (2021) proposed RTSGAN, Paul andPaal et al. (2022) designed PSA-GAN that generates long univariate time series samples of high quality using progressive growing of GANs along with self-attention.
Due to the instabilities typical of adversarial objectives, another line of work in the time series field recently focuses on other deep generative methods. [arrett et al. (202I) imitates the sequential be- energy model reinforcement. Fourier Flows (Alaa et al. 2021) was proposed as a method based on normalizing flows followed by a chain of spectral filters leading to exact likelihood optimization. DDsaai al. (2021) introduced TimeVAD which implements an interpretable temporal structure, and achieves reasonable results on time series synthesis with Variational Autoencoder (VAE). Most recently, Implicit neural representations (INRs) have also been used for time series generation. Fons et al. (2022) leverages the latent embeddings learned by the hypernetwork for the synthesis of new time scries by interpolation.
havior of time series using a drepwise-decompsse
Interpretable Time Series Decomposition. Seasonal-trend decomposition is an effective method used to decompose time series into seasonal, trend components and remainder components to attain interpretability time series analysis (Cleveland et al.] 1990 De Livera et al. 2001 I. [Soott& & Varian
2013
Hammalamadake et al. 2018 Dokumentov & Hyndman [202 T 2011]] Yang et al.,
Wen et al. 2018:2020) The classic approach for performing the decomposition is the widely used STL algorithm (Cleveland et al. 1990). To handle multiple seasonality and seasonal shifts, STR (Dokumentov & Hyndman] (022T) has been proposed by jointly extracting trend, seasonal, and remainder components without iterations. [D Livera et al. (201T) introduced framework based on the state space model for complex seasonal time series. More recently, a novel seasonal-tres de- composition called RobustSTL and its extended work (Wen et al. 2018 2020) have been proposed. The ability to break time series into interpretable components has been topic of recent interest in the field of anomaly detection (Zancato et al., 2022). forecasting (Zhou et al., [2022] Wu et al.] 2021], and generation (Desai et al.) 2001). This work follows N-BEATS (Oreshkin et al. 2020), univariate time series forecasting architecture that explicitly encodes seasonal-trend decomposition into the network by defining two blocks respectively.
2013
Denoising Diffusion Probabilistic Models. Denoising diffusion probabilistic models (DDPMs) are a new class of generative models with nice properties (Ho et al,, 2020). They have demonstrated
great success in both continuous and discrete data domains, producing images (Ho al.) 2020
Dhariwal & Nichol| 2021 Gu et al. 2022). text (Li et al. 2022a Yu et al. 2022; Strudel et al.
2022), audio (Kong et al. 2021), and video (Luo et al. [023]] Harvey et al. 2022 Ho et al. 2022)
that have state-of-the-art sample quality. Very recently. diffusion models have also been developed
for time series data. TimeGrad (Rasul et al., 2021) is a conditional diffusion model which predicts in an autoregressive manner. with the denoising process guided by the hidden state of a recurrent neural network. CSDI (Tashiro et al.) 2021 and SSSD (Alcaraz Strodthoff, 2022) use a self- supervised masking to guide the denoising process like image inpainting. To alleviate the problem of boundary disharmony, the non-autoregressivee diffusion model TimeDiff (Shen & Kwok| (023) uses future mixup and autoregressive initialization for conditioning. Regarding unconditional generation,
Lim et al.] (2023) employs recurrent neural networks as the backbone to product regular 24-time- series using SGM. And most recently, based the structured state space model used in SSSD, Kollovich et al. 2023) conducts univariate time series generation and forecasting with self-guiding strategy. Meanwhile, Coletta al. (2023) approximated the diffusion function based on CSDI where
13

---

Published as a conference paper at ICLR 2024
they remove the side information provided embedding. They also introduced GuideDDiffTime to handle new constraints such as trend or fixed-value, without re- re-training.
B DENOISING DIFFUSION PROBABILISTIC MODELS (DDPMS)
In this section, we provide brief overview of DDPMs. On high level, diffusion models sample from distribution by reversing a gradual noising process. In particular, the forward process gradually corrupts original data Tg E R2 via a fixed Markov chain ;0,...;27 ET with each variable in R as follows:
g( (axs /I 3, ((F17/2))
II -(
(13)
where & (0, 1) is variance at diffusion step t. The increasingly noisy variables 21..I have the same dimensionality as zu. and 2y is an isotropic Gaussian.
notable property of the forward process is that using notation at := 3r and ā == I ag. we
can sample 2t any arbitrary time step / in closed form: g( (mo) = N( Va,io (1 (1- a.)
(14)
Thus using reparameterization trick Kingma & Welling (213)) and defining ~ N(0, T), we have It V0,20 + ū.C.
(15)
Starting from the noise IT, we can run the reverse process parametrized by the model po( == N(xx=11 1o0 ( 1), (a ()) to get zo. The diffusion model is trained to maxi- mize the marginal likelihood of the data Ezo (z)]) and we can write the variational lower bound (VLB) as follows:
T
Cou == log pe (p
+
Σ D 20) po(2x-1(=z)) + DxI(g(ZT(Z)) |l p(oT))
L.-1
(16)
The main objective is sum of independent terms L1-1. There are many different ways to parame- terize pa (x 1), and the most obvious option is to predict ss (x 1) directly:
Lorigin == E Et
|1ā(z,, 2o) ne (x t)]?
(17)
2 (Zi, t)
where (a(,,, za) is the mean of the posterior ( 1z- Fo) which defined follows:
q( 11,, = (z - to + Voul xz-1) Tt. 1 02 at Finally, Ho et al. (200)) found predicting e worked best, especially when combined with reweighted loss function:
- B,T).
(18)
a
[|| -Ilc-co(tt co( e)1]
Lsimple Be,00..
(19)
But they also admitted that there is the possibility of predicting directly. Note that estimating Equation] 19 is equivalent to estimating scaled version of the score function, the gradient of the log density of the noisy data:
1 at (z, Varzo)
V., log (z (20)
(20)
-T- Cra
Thus, data generation through denoising depends on the score-function and can be seen as noise conditional score-based generation Voleti et al. (2022). Then the reverse process po (x in DDPM can be written as:
a (x. VI - &i where is hyperparameter, and standard Gaussian noise.
Tt
cx( 1)) a
(21)
0z
14

---

Published as a conference paper at ICLR 2024
C ADDTIONAL EXPERIMENTAL RESULTS
In this section present additional experiments which we omitted in the main body of the paper due to limited space.
C.I DETAILED EXPERIMENTS FOR LONG-TERM TIME SERIES GENERATION
In order to further verify its performance stability in producting long multivariate time-series data, we add discriminative score and predictive score:
Table 3: Detailed Results of Long-term Time-series Generation. (Bold indicates best performance).
Dataset
Length
Diffusion-TS
TimeGAN
TimeVAE
Diffwave
DiffTime
Cot-GAN
Context-FID
64
0.6312.058
1.130 102
0,827s 146
1.543..153
1.279#.083
3.008
Score
128
0.787=.062
1.553#1669
1,062= 134
2.354
2.584
4.639=427
(Lower Better)
256
0.423+.038
5.872+:208
0.826#.093
2.899a.289
3.5242.830
8.0
Correlational
0.0824.005
0.483+.019
0.067a.006
0.186*.008
0.0944.010
0,271+0007
Score
128
0,088+.005
0.188:0006
0.05420007
0,203+.006
0.113:.012
0.17620006
ETTh
(Lower the Better)
256
0.064:.007
0.522+.013
0.0462.007
0.199#.003
0.
0,222+.010
Discriminative
64
0.106:.048
0,227+.078
0.171 142
0.254
0.150+.003
0.296*
Score
0.144:.060
0.188+.074
0.154=0087
0,274+.047
0.176+.015
0.4
(Lawer Hetter)
0.060:.030
0.442+.056
0.178=006 076
0.304#.068
0.243+.005
0.461+0010
Predictive
0.116:.000
0,132+.008
0.118
0.133+.000
0.118+.004
0,135 003
Score
128
0.110:.003
0.153+.014
0.113+.00
0.129
0.1200.008
(Lower the Better)
256
0.109#.013
0,220#0008
0.1 1.110=000 027
0.132+.001
0.118+0003
0. 0.129 000
Context-FID
64
0.135=.017
1,230=.070
2.662= 087
2,697+.418
0.7624 157
1.824 144
Score
128
0.087+.019
2.535.372
321 106
5.552+.528
1.344+. 131
1.822
(Lower Better)
256
0.126:.024
5.032+.831
3,768 998
5.572+.584
4.735 729
2.533+.467
Correlational
0.672+.035
3.668#. 106
1.653= 208
6.8474.083
1.81+.211
3.319
Score
0.4512.079
4.790 116
1.820= 329
6.6632.112
1.376t.20 201
3.7132.055
Energy
(Lower the Better)
256
0.361
4.487#.214
1,279 114
5.690s.102
1.800t. 138
3.739+0089
Discriminative
64
0.078:.021
0.498#.001
0.499=0000
0.497#.004
0.328+.031
0.499#.001
Score
128
0.143t.075
0.499#.001
0.499
0.499
0.396+ 024
0.499#0001
(Lower the Better)
256
0.290s.123
0.499+0000
0.499*000
0.499
0.4
0.498+
Predictive
0.249:.000
0,291#.003
0.302+.001
0.252+.001
0.252+ 000
0.262+
Score
0.2474.001
0.303+.002
0.318=000
0.252+.000
0.251.+ 000
0.269+ 002
(Lower the Better)
256
0,245+.001
0.351
0.353+000 003
0,251
0.251+:000
0.275
1.0
10
-10
Original
Diffinian-15
-10
Diffusian-TS
-10
Diflwir-
-10
Original Diffusiar-IS
-10
10
-10
10
-10
10
-10
5
10
10
Orio mal
Original TimeCAN
TimeGAN
5
-5
5
-10
Original
TimeGAN
-10
-10
Original
TimeGAN
10
(a) ETTh-24
(b) ETTh-64
(c) ETTh-128
(d) ETTh-256
Figure 7: t-SNE plots of the time series of length 24, 64, 128 and 256 synthesized by Diffusion-
TS and TImeGAN, Red dots represent real data instances, and blue dots represent generated data samples in all plots.
15

---

Published as a conference paper at ICLR 2024
0.2
0.5-
0,5
0.0
0.0
-0.5
0.4-
-0.5
Difusion-TS
Original Diffusion:
Original Difuskon-S
-101
-1.0
-0.5
0.5
1.0
Original TImeGAN
Orig ina TimeGAN
1.0
Original TimeGAN
0.5
0.9
-0.2
TimeGAN
Original
0.5
0.5
+0.5
1.0
-0.6
-0.5
1.0
(a) 2T
(b) ETTh-64
(c) ETTh-128
(d) 2TTh-256
Figure 8: PCA plots of the time series of length 24, 64, 128 and 256 synthesized by Diffusion-
TS and TimeGAN. Red dots represent real data instances, and blue dots represent generated data samples in all plots.
C.2 ADDITIONAL 2- 2-DIMENSIONAL PLOTS ON ETTH DATASET
In order to further compare the diversity of the generated time-series using Diffusion-TS, and TimeGAN with the ETTh dataset, we applied PCA and t-SNE analyses to visualize how well the generated time-series distributions cover the real data distributions. According to the figures, syn- thetic samples generated by Diffusion-TS have significantly more overlap with the original data than the SOTA method.
C.3 ADDITIONAL EXPERIMENTS FOR IMPUTATION AND FORECASTING PERFORMANCE COMPARISON
To demonstrate that the excellent performance of Diffusion-TS extends to further baselines, we now repeat imputation and forecasting experiments in Alcaraz & Strodthofi (2222) All baselines results were collected from the original publications. We first conduct imputation experiments on the Mu- JoCo sequences of length 100. We report an averaged MSE for single imputation per sample on the test set over 5 trials. Table[[shows the empirical results on the MuJoCo data set, where Diffusion-TT outperforms all baselines except for CSDI on the 70% missing ratio. We then demonstrate Diffusion- TS's forecasting capabilities on the Solar data set collected from GluonTS (Alexandrov et al. 2020), where the conditional values and forecast horizon are 168 and time steps respectively. The aceu- racy of all models on the long time series is shown in Table 5 Similar to the observation made in imputation tasks, our proposed method can achieve significantly better performance. These results further support the effectiveness of our proposed model, well as its ability in processing long time-series.
Table 4: Imputation MSE results for the MuJoCo data set. Here, we use a concise error notation where the values in brackets affect the least significant digits e.g. 0.572(12) signifies 0.572 0.012. Similarly, all MSE results are in the order of le-3.
Model
70% Missing
80% Missing
90% Missing
RNN GRU-D
11.34
14.21
19.68
ODE-RNN
9.86
12.09
16.47
NeuralCDE
8.35
10.71
13.52
Latent-ODE
3
2.95
3.6
NAOMI
1.46
2.32
4.42
NRTSI
0.63
1.22
4.06
CSDI
0.24(3)
0.61(10)
4.84(2)
SSSD
0.59(8)
T.00(5)
1.90(3)
Diffusion-TS
0.37(3)
0.43(3)
0.73(12)
16

---

Published as a conference paper at ICLR 2024
Table 5: Time series forecasting results for the solar data
Model
MSE
GP-copula
9.8c2+5.2e
TransMAF
9.30E+02
TLAE
6.8e247.5el
CSDI
9.0e2+6. lel
SSSD
5.03e2+1
Diffusion-TS
3.75e243.6e1
Figure and 10 shows example prediction results on solar and stock data set respectively. As can be seen, Diffusion- TS produces high-quality predictions close to the ground truth.
1.00
Findictionn
25
175
200
100
Hittey Grund
50
Prediction
25
50
75
100
125
150
175
200
150
Himoryy Ground Tith
Predicion
0-
25
50
200
125
100
Hittyy Findittion
25
25
175
200
(a)
Figure 9: Visualizations of time series forecasting on Solar dataset by the proposed Diffusion-TS..
C.4 IRREGULAR TRAINING
In this subsection, we provide the irregular training results on four real-world datasets mentioned in the main paper in Figure 11
C.5 DISENTANGLEMENT VALIDATION SYNTHETIC DATASET
In order to validate Diffusion-TS' interpretability shown in Equation 2 we conduct experiments on synthetic time-series where time-series composed of seasonal part and trend-cyclical part. We generate 2000 time series, each consisting of 160 time steps for training. As shown in Figure [2 we visualize the ground truth trend and seasonality patterns, together with the learned ones. We can clearly find that the learned disentangled components very similar the ground truth, and resid- ual part is close to zero. This verifies the interpretability of our proposed progressive decomposition architecture.
17

---

Published as a conference paper at ICLR 2024
2500
Hastyy Gmunt
10
20
30
40
2500
Ground Tuth
Fnderton
2300
Hatory
3-
1-
10
20
30
Hittery
Gmuni
Fredetion
2700
2600
10
20
30
40
105
Ground frndition
2-
(a)
Figure 10: Visualizations of time series forecasting on Stock dataset by the proposed Diffusion- TS.
Energy
1MRI
ETTh
Stocks
Discriminative score
0.25
0.14
0.35
0.30
0.25
0.20
0.30
0.10
0.20
0.25
0.15
0.20
0.06
0.10
0.10
0.05
0.1
0.25
0.5
0.75
0.1
0.25
0.5
0.75
0.1
0.25
0.5 0.75
0.1
0.25
0.5
0.75
0.255
Cold start Irregular training
0.042
Predictive score
0.254
Original
0.104
0.125
0.040
0.102
0.120
0.252
0.039
0.100
0.038
0.250
0.099
0.110
0.037
0.1
0.5
0.5
0.75
0.1
0.25
0.5
0.75
0.1
0.25
0.5
0.75
0.1
0.25
0.5
0.75
Regular Ratio
Figure 11: Performance of using different regular ratios the cold start and irregular training ex- periments with 10% 20% missing. Cold start only clean data, while the other restores and incorporates irregular time series during training. Overall, Diffusion- TS can mostly achieve the quality of missing values via irregular training.
18

---

Published as a conference paper at ICLR 2024
Dirty Input
0-
1-
-2
20
40
60
80
100
120
140
160
Original Trend
1.0
0.5
0.0
-1.0
100
160
Learnt Trend
1.0
0.5-
0.5
20
40
60
100
120
140
160
Original Season
0.0-
Learnt Season
1.0
0.5
-0.5
-1.0
20
40
60
80
100
120
140
160
Learnt Residual
1.0
0.0-
-1.0
Original Data
1.0
0.5
-0.
-1.0
40
60
100
120
140
160
Reconstruction
1.0
0.0
-0.5
0
20
40
60
80
100
120
140
160
(a)
Figure 12: Disentanglement validation on synthetic dataset.
19

---

list the Published as a conference paper at ICLR 2024
C.6 HYPERPARAMETER TUNING AND SENSITIVITY
We did limited hyperparameter tuning in this study to find default hyperparemters that perform well across datasets. The range considered for each hyper-parameter is: batch size [32; 64; 128), the number of attention heads: [4; 8], the number of basic dimension: [32. 64. 96. 128], the diffusion steps: [50. 200, 500. 1000] and the guidance strength: [1., le-1, 5e-2, 1e-2, 1e-3]. In table we evaluate the impact of the different guidance strength in the model according the mean squared errors on the MuJoCo data set. Also in this table, we notice that the increasing or reducing the strength do not improve the results of conditional generation. A single Nvidia 3090 GPU is used for model training. all of our experiments, we use cosine noise scheduling and optimize our network using Adam with (31, 52) (0.9, 0.96). And linearly decay learning rate starts 0.0008 after 500 iterations warmup. For conditional generation, we set the inference steps, " to be 200, 0.05 respectively. We use 90% of the dataset for training and the rest for testing. In Table 7 report the model size of our method and baselines. Diffusion models have generally more parameters than other methods, but method has fewer parameters than Diffwave. And in Table [] detailed hyperparameter settings.
Table 6: Diffusion- TS with different guidance strengths E [1., 1c-1, 5c-2, le-2, lc-3].
70% Missing
80% Missing
90% Missing
1.
2.8(13)
4.1(10)
6.8(17)
le-1
0.37(4)
0.45(0)
0.82(9)
5e-2
0.37(3)
0.43(3)
0.73(12)
le-2
0.60(5)
0.70(10)
1.07(14)
le-3
3.1(8)
7.2(20)
19.6(22)
Table 7: Comparison of model size.
Model
Sines
Stocks
Energy
VAE
97,525
104,412
677, $77.418
TimeGAN
34,026
48,775
1,043,179
Cot-GAN
40,133
52,675
601,539
Diffwave
533,592
599,448
1,337,752
Diffusion-TS
232.177
291,318
1,135,144
Table 8: Hyperparameters, training details, and compute resources used for each model
Parameter
Sines
Stocks
ETTh
MuJoCo
Energy
[MRI
attention heads
4
4
4
4
attention head dimension
16
16
16
16
24
24
encoder layers
1
4
4
decoder layers
2
2
2
2
3
4
batch size
128
128
128
64
64
timesteps/ / sampling steps
500
500
500
1000
1000
1000
training steps
12000
10000
18000
14000
25000
15000
model size
232,177
291,318
350,459
357,214
1,135,144
1,382,290
training time
17min
15min
31min
25min
60min
44min
sampling time (every 2000)
23s
26s
31s
50s
65s
72s
C.7 ABLATION STUDY
To evaluate the effectiveness of the Diffusion-TS, compare the full versioned Diffusion-TS with its three variants: (1) w/o FFT, ie. Diffusson-TT TS without Fourier- based loss term during training, (2) w/o Interpretability, i.e. Diffusion-TS without seasonal-trend design in the network, (3) w/o
20

---

Published as a conference paper at ICLR 2024
Transformer, i.e. Diffusion- -S based on Convolutional network without encoder and self self-attention mechanism. (4) e-prediction, Diffusion-TD using traditional noise prediction parameterization for training and sampling. Here shown as Table 9 we provide ablation study 24-length time series in detailed version.
Table 9: Ablation study for model architecture and options. (Bold indicates best performance).
Metric
Method
Sines
Stocks
ETTh
Mujocn
Energy
FMRI
Diffusion-TS
0.0061.000
0.147
0.1162.010
0.0132.001
0.0891.024
0.105s.006
Context-FID Score
w/o FFT
0.008+.001
0.216=0031
0.219:0211
0.017+.002
014
0.189t.003
wfo Interpretability
0.008+0000
0.1
0.121=0004
0.018 002
0,105+ 020
0.149=.010
w/o Transformer
0.0074.000
0.254a.024
0.229t.024
0.040+.003
0.4964.088
0.100s.010
(Lower the Better)
e-prediction
0.0191.004
0.2152.033
0,265#.008
0.0174.001
0.189#.054
0.1901.003
Diffusion-TS
0.0152.004
0.004:.001
0.0492.008
0.193:.027
0.8562.147
1.112.042
Correlational
w/o FFT
0.016+:005
0.005+0005 005
0.063+.012
0,204+0017
0.3
1.625+.00 021
Score
w/o Interpretability
0.0161.003
007
0.0674.009
0.199s.026
1,013:.088
1.340s.050
w/o Transformer
0.016t.005
0.042+0007
0.055#0005
0,2542.060
2.164:.057
1.01+.041
(Lower the Better)
e-prediction
0.024+0006
0.007=0006
0.065+0008
0.208=0016
1.111=:114
1.166x.00 031
Diffusion-TS
0.006:.007
0.067
0.061s.099
0.008s.002
0.122+.003
0.167+.022
Discriminative
w/o FFT
0.0071.006
0.127::019
0.0961:007
0.0101.002
0.13511004
0.1774.013
Score
w/o Interpretability
0.009+0006
0.101
0.071=:010
0.021=.014
0.125=0003
0.267#.034
w/o Transformer
0.010+:007
0.104=0024
0.082+:006
0.039+:014
0.324+0015
0.1234.064
(Lower the Better)
e-prediction
0.0401.011
0.131
0.0991.010
0.0234.006
0.197:0001
0.1681.030
Diffusion-TS
0,0931.000
0.0362.000
0.119:.002
0,0074.000
0.250:.000
0.0992.000
Predictive
w/o FFT 0.093:.000
0.038=0000
0.121=0004
0.008
0,250=.000
0,100=0001
Score
w/o Interpretability
0,093+.000
0.037=0000
0.119:.008
0.008+0001
0.251+0000
0.101+0000
w/o Transformer
0,0931.000
0.0362.0000
0.126E:004
0.0081.000
0.3191:006
0.0992.000
(Lower the Better)
e-prediction 0.0974.000
0.039=0000
0.120#:002
0.008+.001
0.25 100
0.10020000
Original
0.0941.001
0.0362.001
0.121
0.007+.001
0,250+.003
0.090s.001
Table 10: Ablation analysis of the decomposition on MuloCo dataset.
Model
70% Missing
80% Missing
90% Missing
Residual
0.51(1)
0.59(7)
0.85(10)
Residual+Season
0.45(5)
0.52(3)
0.77(9)
Treedual+TTeen
0.46(2)
0.50(5)
0.80(7)
Season+Trend
0.63(3)
1.05(6)
1.42(10)
Diffusion-TS
0.37(3)
0.43(3)
0.73(12)
We also conduct fine-grained ablation study to verify the effectiveness of our proposed disentan- gled framework. The ablative results are shown in Table [0] We find that the model performance drops no matter which part is removed, which validates that all of our proposed disentangled rep- resentations play an important role in generative tasks and jointly enhance the performance of the final model.
D REPLACE-BASER IMPUTATION WITH DIFFUSION MODEL
We now introduce the imputation method with Diffwave and Diffusion-TS-R used for the experiments in Section 4.5 Given an irregular sample
oh x with zbb denotes all observed values and denotes all missing values, [Song al. (2020) proposed gen- eral method for conditional sampling from the jointly trained diffusion model po (2r). By defining the known and unknown dimensions of x, as (xx) and Ω (Ix) respectively, they have Pt (f) (zx)
= =)) = Pt ([ : (xx)))) where ② (x²) denotes samples from (Ω (zx) [ (zo = v)), and [1))) (2:)] represents the concatenation of two sets of dimen- sions. More concretely, in their approach, the samples for 22t are replaced by exact samples from the forward process g [b= in Equation 14 at each iteration, while the sampling procedure for updating z! is still sampling from po .t+1). The samples zhh then have the correct marginal dis- tribution, and 21 will conform with 21th through the denoising process. Using this strategy, we can generate intact sample that follows the correct conditional distribution in addition the correct
21

---

Published as a conference paper at ICLR 2024
Diffusion-Ti TS
41 X.)
Cre
Biemant bebstraccior
Enshelding
LayeeWane
Protiooall Excoding
Encoder
sact layer
Irput
Deep Decomposition Decoder
Decoder
liput
Daquul
Seccnditt & Ence
Riffacce Firhelling
Lapethien:
F.S.
Slock
Milthead Albntice
Lepettion:
Frevord
Sread Speihelk:
Bacaiing
Matthoul Artediie
Meen
Decoddr
Docoda Block
Toand
Figure 13: Overview figure of the Diffusion-TS using Transfomer inspired architecture with the decomposition design.
marginal. We will refer to the infilling method for conditional sampling as replace- imputation with diffusion model.
E MODEL DETAILS
We now describe the details about our model architecture. The overview of Diffusion-TS is shown in Figure As deseribed in main paper, the framework contains two parts: a sequence encoder and an interpretable decoder. Each encoder block contains a full attention and feed forward network block, while each transformer block in decoder consists of a full attention and a cross attention to combine the encoding information. For the diffusion embedding, we follow previous works (Ho) et al. 2020: Gu et al. 2022) with transformer sinusoidal positional embedding to encode the diffu- sion step. The diffusion step t is injected into the network using the Adaptive Layer Normalization operator, which can be written as: Layoooor (w) + b, where w is the intermediate activations, a, and b, are obtained from linear projection of the diffusion embedding.
22

---

Published as a conference paper at ICLR 2024
F ALGORITHMS
The full reconstruction-guiieed sampling algorithm is illustrated on the left of where Replace (-) denotes replace-basee based imputation algorithm introduced in D As aforementioned, we take multiple gradient steps for each diffusion step improve the generative quality. Note that the reverse process of diffusion model can be viewed as two periods: creating in the early stages and smoothing in the rest stages, thus more gradient updates large diffusion step / guide the generation then reduce the number of gradient steps in stages to accelerate sampling. The algorithm in the right of presents this optimization in detail.
Algorithm 2 Optimized Conditional Sampling
Algorithm 1 Reconstruction guided Sampling
Require: gradient scale 1), conditional data Ta. trade-off coefficient 7, times set K
Require: gradient scale 7), conditional data Ta. trade-off coefficient
1: NO, I);
2: for all from T to 1 do for all from [[t] to do
1: 2r ~ ~N. I);
3:
2: for all from T to 1 do
3:
[Sa: .bb] 2n (20,
4:
[aa ăg. + ( 0);;
L1 = | 12
5:
4== = I - - 3.
4:
5:
It-1 +N (fo 1, 0), x4) E);
6:
#t-1 t NN (,, x1). E);
7:
L2=
6:
L2 | - 11 (i F4, 0) 3z4) [ <);
1 1 (0 t, x) /E; It 2t + V (Li 722)
7:
in 80(0,, 1, 8) + V (L1 + 722)
8:
8:
I + Ni4 (To, x() :))
9:
end for
9: It-1 + Replace Laa te:
10: It-1 < N (Rg (ze, t, 0) m4) E):
end for
11: It-1 t Replace TTo æz-1.
12: end for
G EXPERIMENT DETAILS
G.1 DATASETS
Table 11 shows the statisties of the datasets and all datasets available online via the link.
Table 11: Dataset Details.
Dataset
# of Samples
dim
Link
Sines
10000
http:///tthbb.ommmsyyooo/////Cnf
Stocks
3773
https///ihnnneeeaaooooooo//oooop
ETTh(1)
17420
https///sttbbbboom///ootat////sr
MuJoCo
10000
14
http://pttubboomm/cccpmm/////nno
Energy
19711
28
http://archh....ss..e.ccuu/ttlt
fMRI
10000
50
https//www..miiboo...ukkkaaaaas
G.2 BASELINES
We apply and optimize the following accessible source codes for generative experiments.
TimeGAN: hutp:://gthbbbbommmyyyymyyyy/7nf
: http://gthbbbccmmaaaueeea///ee
:ot-GAN: h http:///fttibbbcmm/iaaaaiiii///p
Diffwave: https//uffiiaav-ddmoo.itbbbbbbbb
CSDI DiffTime: http:///wwwggg/////oo/ooooooo/mn
23

---

Published as a conference paper at ICLR 2024
For GAN-based methods, use the 3-layer GRU-based neural network architecture with hidden size that is 4 times larger than the feature size. We modify settings of TimeVAE and Diffwave so that they have roughly the same trainable parameter size residual layers as 4, residual channels as 64, and attention heads as 8. We also change the kernel-size of the convolutional layers following the settings of DiffTimm. Finally, both Diffwave and CSDI use the same diffusion settings (e.g. diffusion steps) as ours if necessary.
For CSDI, set the number of
G.3 EVALUATION METRICS
Discriminative & Predictive score. The discriminative score is calculated as |accuracy 0.5), while the predictive score is the mean absolute error (MAF) evaluated between the predicted values and the ground-truth values in test data. For fair comparison, of TimeGAN Yoon et al. (019) for the discriminative and predictive score. Both the classifier and sequence-prediction model use 2-layer GRU-based neural network architecture.
the experimental settings
Context-FID score. A lower FID score means the synthetic sequences are distributed closer the original data. Paul et al. 2022)) proposed a Frechet Inception distance (FID)-like score, Context-FID FCrnhext-Frechet Inception distance) score by replacing the Inception model of the original FID with time series representation learning method called Ve2 (Yue et al. 2022). They have shown that the lowest scoring models correspond to the best- rest-performing models in downstream tasks and that the Context-FID score correlates with the downstream forecasting performance of the generative model. Specifically, we first sample synthetic time series and real-time series respectively, Then we compute the FID score of the representation after encoding them with pre-trained TS2Vec model.
Correlational score. Following Ni al. 2020 estimate the covariance of the ith and jth feature time series as follows:
cove.
x
Σ
(22)
t=1
r=1
r=1
Then the metric on the correlation between the real data and synthetic data is computed by
1
covT,
cov!,
(23)
10
/cov ", cOV
V cov. /cov,,cov/
35
H ADDITIONAL VISUALIZATIONS
We introduce additional visualization and distribution outcomes in Figure 14 and Figure 15
I ADDITIONAL EXAMPLES
We present additional examples illustrate various imputation examples show the characteristic of time-series imputation and forecasting on the Energy dataset in Figures to 19
24

---

Published as a conference paper at ICLR 2024
Sirees
Slocks
Th
MuloCa
Einergy
IMRI
(a) Ours
(b) TimeGAN
(c) Time
(d) Diffwave
(e) DifiTime
(I) Cot-GAN
Figure 14: t-SNE visualizations of all methods multivariate datasets. Red is for original data, and blue for synthetic data.
Sines
Slocks
ET
MuloCo
Energy
10-
166
(a) Ours
(b) TimeGAN
(c) Time VAE
(d) Diffwave
(e) DiffTime
(f) Cot-GAN
Distributions of all methods multivariate datasets. blue solid line is for original data, and yellow dotted line for synthetic data.
Figure 15:
25

---

Published as a conference paper at ICLR 2024
0.2
0.1
0.3
0.250
20
40
20
40
20
40
20
40
0.60
0.42
value
0.55
0.10
0.38
0.20
0.36
20
0.4
0.6
0.30
0.4
0.25
0.20
0.2
20
20
40
20
40
20
40
0.40
value
0.6
0.30
0.35
0.30
0.25
0.30
20
40
20
40
20
40
20
40
0.35
0.4
0.30
0.30
0.25
0.25
0.25
0.20
20
40
0.4
0.45
W/
0.6
0.6
0.35
20
40
20
40
20
40
20
40
0.35
0.6
0.30
0.25
20
20
40
20
40
20
40
Figure 16: Comparison of imputation for the Energy dataset (90% missing). The result is for time series sample with all 28 features. The red crosses show observed values and the blue circles show ground-truth imputation targets. Green and gray colors correspond Diffusion-TS and Diffwave, respectively. For each method, median values of imputations are shown as the line and 5% and 95% quantiles are shown as the shade.
26

---

Published as a conference paper at ICLR 2024
0.15
0,9
0.10
0.55
0.4
0.50
0.00
0.70
0.70
0.8
0.55
0.55
20
40
20
40
20
40
20
40
0.900
0.74
0.34
value
0.875
0.75
0.32
0.850
0.70
0.30
0.825
0.95
0.075
0.75
0.050
0.80
0.70
20
40
20
40
20
40
20
40
0.75
0.88
0.725
0.75
0.700
0.84
20
0.4
0.9
0.7
0.3
0.62
0.6
0.7
0.60
0.1
20
1.0
1.0
0.8
0.90
0.85
0 20
20
40
20
40
Figure 17: Comparison of imputation for the Energy dataset (50% missing). The result is for time series sample with all 35 features. The red crosses show observed values and the blue circles show ground-truth imputation targets. Green and gray colors correspond Diffusion-TS and Diffwave, respectively. For each method, median values of imputations are shown as the line and 5% and 95% quantiles are shown as the shade.
27

---

Published as a conference paper at ICLR 2024
0.4
0.4
0.6
0.2
0.4
20
20
40
20
40
20
40
0.35
value
0.5
0.4
0.35
0.25
0.4
0.20
20
20
20
40
0.5
0.4
0.3
value
0.3
0.3
0.40
0
20
40
20
40
20
40
20
40
0.8
value
0.40
0.4
0.6
0.35
0.2
0.2
0.30
20
20
40
20
40
20
40
0.4
0.5
0.5
0.3
0.4
0.2
0.4
0.3
20
40
0.4
0.8
0.3
0.75
0.6
0.2
0.70
0.4
0.1
0
20
20
40
20
40
20
40
1.0
0.4
0.4
0.3
0.0
0.0
0
20
20
40
20
40
20
40
Figure 18: Comparison of forecasting for the Energy dataset (36 forecasting window). The result for a time series sample with all 28 features. The red crosses show observed values and the blue circles show ground-truth imputation targets. Green and gray colors correspond to Diffusion-TS and Diffwave, respectively. For each method, median values of imputations are shown as the line and 5% and 95% quantiles are shown as the shade.
28

---

Published as a conference paper at ICLR 2024
0.3
0.375
0.4
0.3
0.325
0.300
0.0
40
20
40
0.60
0.55
0.20
0.55
0.15
0.50
0.50
0.10
20
40
20
40
20
40
20
40
0.45
value
0.30
0.40
0.40
0.35
0.15
0.35
0.25
0.30
0.30
40
0.3
0.9
0.5
0.2
0.8
0.1
0.7
0.2
0.3
0.0
20
40
0
20
40
20
40
20
40
0.6
0.35
0.25
value
0.30
0.3
0.20
0.25
0.4
0.2
20
20
40
20
40
20
40
0.3
0.8
value
0.70
0.2
0.1
20
40
20
40
1.00
0.50
0.25
20
0
20
40
20
40
20
40
Figure 19: Comparison of forecasting for the Energy dataset (24 forecasting window). The result for a time series sample with all 28 features. The red crosses show observed values and the blue circles show ground-truth imputation targets. Green and gray colors correspond to Diffusion-TS and Diffwave, respectively. For each method, median values of imputations are shown as the line and 5% and 95% quantiles are shown as the shade.
29


Reviews:
[{"review_id": "sBP16wrtlQ", "reviewer": "Reviewer_bNeL", "summary": "A transformer based denoising diffusion probabilistic model, Diffusion-TS, is proposed for generating multivariate time series. \nRepresentations from Oreshkin 2020, Desai 2021, De Livera 2011 and Woo 2022 are adopted for trend and seasonality. Under time series model (1) and Fourier-based loss term following similar work as Ho 2020 and Fons 2022, the \\hat{x}_{0} is estimated directly for unconditional time series. \nThe proposed method Diffusion-TS is compared with four other models TimeGAN 2019, TimeVAE 2021, Diffwave 2021, and Cot-Gan 2020 under 6 datasets (stock price, electricity transformer, etc. The proposed method works better than the other four methods most frequently under the evaluation rules suggested by Yoon 2019, Paul 2022 and Ni 2020 especially under high-dimensional datasets.\nExtension to conditional model by adding gradients, as well as simulation results, are presented.", "questions": "P4, Please define similarly as in Woo (2022) your \\omega_seas or at least add definition by English. \nP4, Equation (4), please either split the A and \\Phi definition in two rows or write as a vector, so would not be confused as compound operators. \nP5, Could you clarify the Figure 2, ‘Decoder Block 1’, ..., ‘Decoder Block N’ each refers to under the N samples (i=1, ... ,N) of time-series signals on P2 or something else considering the squares/arrows in that figure. \nP17, Can the dataset names or links be more specific? There’re multiple datasets available under some links.  \nP6 and P18, Not questioning the vitality of your result, just want to have a better understanding of the performance to be expected under your method for other potential new runs: \nHow different Table 1 results can be if you did try other hyper-parameters?    \nPrediction under jump model can be hard. How well the performance of your model can be when there’s big jump in observed data? (The Google stock data used is from 2004 to 2019, the big changes in 2020 and 2022 not included. The electricity transformer temperature and blood oxygen related data might be typical stable time series. I’m not familiar with MujoCo data.)", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This idea and algorithm of the proposed method are generally well presented. It’s compared with multiple recent time series generation methods and simulation results outperform these in most cases.", "weaknesses": "Potentially the proposed method works well or even better than other existing methods under typical seasonal time series data. Pending further exploration/confirmation how well the performance the proposed method can be under times series of real data with different unique features like big jump in stock price. Also wondering the hyper-parameter selection impact on the result and convergence speed.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "J7GX7qryNo", "reviewer": "Reviewer_Prro", "summary": "This paper proposes a diffusion model for time series generation. The model involves a custom architecture with components that help synthesizing the trend and seasonal components of the time series. The model is trained to directly estimate the observation from arbitrary diffusion steps. The objective function of the diffusion model is also modified to incorporate a loss based on fourier transform of the time series. Experiments show improved performance over existing generative models in terms of different generative metrics.", "questions": "- What is the empirical significance of the trend & seasonality synthesis blocks and the fourier regularization? Did the authors conduct ablations for these components? (Check the results in [1] which doesn't use any of these components)\n- Why does Diffwave perform so much worse in the case of long time series generation? The sequence lengths considered in this work are smaller than those used in audio synthesis. Is it possible that there is a bug in the experiments setup? \n    - Why did not authors not compare against a CSDI-like model for unconditional generation? That would probably be a better comparison and ablation for the components proposed in this work. \n\n[1] Lim, Haksoo, et al. \"Regular Time-series Generation using SGM.\" arXiv preprint arXiv:2301.08518 (2023).", "limitations": "", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- The model explores both unconditional and conditional generation using a single model.\n- The model performs better than baseline models in terms of time series generation.", "weaknesses": "- The claim \"little attention has been given to leveraging the powerful generative ability for general time series production\" is not entirely correct. See [1] and [2], for example. Furthermore, in the light of [2], the claim \"... **first** DDPM-based framework for both unconditional and conditional time series synthesis\" is not correct either. [2] proposes a self-guidance mechanism to use unconditional diffusion models for conditional time series tasks and also studies the unconditional generative properties of the model. This limits the novelty of the paper and Sec. 3.4, in particular.\n- The paper is poorly written and does not tell a coherent story. Sec. 3.2 reads like an arbitrary combination of ideas. The individual components are also not analyzed later via ablations. The manuscript also has many typographical errors (synthetic instead of synthesis, DSDI instead of CSDI).\n- The evaluation on the conditional tasks is limited. The model is only compared against Diffwave and CSDI (which is fairly close to Diffwave) and baselines from time series forecasting literature are missing. It is also unclear how these CSDI and Diffwave baselines were trained. \n\n\n\n[1] Lim, Haksoo, et al. \"Regular Time-series Generation using SGM.\" arXiv preprint arXiv:2301.08518 (2023).      \n[2] Kollovieh, Marcel, et al. \"Predict, refine, synthesize: Self-guiding diffusion models for probabilistic time series forecasting.\" arXiv preprint arXiv:2307.11494 (2023).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "oIYe60v7f0", "reviewer": "Reviewer_iyrP", "summary": "The authors propose Diffusion-TS, a diffusion method for multivariate time series generation. The method involves an interpretable decomposition into trend and seasonality components. The diffusion model is trained using a joint L2 and Fourier loss, and the authors apply conditional generation techniques to apply the method to forecasting and imputation directly without changes in the model. Experiments show Diffusion-TS achieves state-of-the-art performance on time series datasets, including in an irregularly sampled setting.", "questions": "Main:\n- In Appendix C.2, what is the \"w/o interpretability\" model? Is this equivalent to $x_0(x_t, t, \\theta) = R$ in Eqn 7, setting the $V_{tr}^t$ and $S_{i, t}$ terms to $0$? How crucial are the polynomial regressor and Fourier synthetic layer techniques?\n- How well does the trend-seasonality decomposition method of Diffusion-TS achieve what's intended for the toy model (Eqn 1), for example on synthetic data?\n- How do discriminative and predictive scores compare on the long-term time series generation task (Table 2)?\n- For the long-term generation task (Table 2), why is it that Diffusion-TS seems to perform generally better with longer time series? It seems like longer time series generation should be a more difficult task.\n- In Figure 6, is MAE computed only over missing data (imputation targets) or over the full time series including existing data? For Diffusion-TS-G, since a soft constraint is used to enforce the conditional generation, how closely do the generated time series for imputation/forecasting match the existing data?\n- How does the number of parameters compare between Diffusion-TS and its competitors?\n\nClarifications:\n- How is the data in Figure 3 generated? Is column b the ground truth and column a the result of adding noise?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "Originality:\n- As far as the reviewer is aware, the authors are the first to combine a trend-seasonality decomposition technique with diffusion for time series generation. The authors mention this choice leads to improved performance across the various evaluation metrics.\n\nQuality:\n- The experiments are generally well-constructed. A variety of datasets and evaluation metrics are used for standard unconditional generation, and several ablation studies are included in the appendix.\n- The authors show qualitative results showing the trend and seasonality components predicted by the model generally behave as intended.\n\nClarity:\n- The paper is generally well-written, and the experiments are described clearly.\n\nSignificance:\n- The experimental results are quite compelling. In the unconditional time series generation settings (Table 1), Diffusion-TS outperforms existing methods. The performance gap is especially clear for higher-dimensional datasets and long-term generation (Table 2).", "weaknesses": "- Generally, it's a bit unclear to the reviewer which design choices are crucial to the improvement in performance, especially the trend-seasonality decomposition. Two detailed techniques are involved in implementing the decomposition, the polynomial regressor and the Fourier synthetic layers, but it's unclear to what extent these techniques contribute to the model performance.\n- It's a bit unclear how compelling the qualitative results are. For example, in Figure 3, the trend components seem quite uninformative compared to the season/error components. In Figure 4, it's unclear whether the t-SNE visualizations support the claim that the Diffusion-TS distribution better aligns with the data distribution than TimeGAN distribution.\n- Experimental metrics reported are inconsistent. For example, correlational and context-FID scores are reported in Table 2, and discriminative and predictive scores are reported in Figure 7 and Appendix C.1/C.2.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)