A Non-isotropic Time Series Diffusion Model with Moving Average Transitions
Authors: Chenxi Wang
Linxiao Yang
Zhixian Wang
Liang Sun
Yi Wang
Venue: ICML 2025 poster (2025)
Keywords: Diffusion models
Time series forecasting
Time series super-resolution
Primary area: applications->time_series
TL;DR: We proposed a novel time series diffusion model with moving average transitions.

Abstract: Diffusion models, known for their generative ability, have recently been adapted to time series analysis. Most pioneering works rely on the standard isotropic diffusion, treating each time step and the entire frequency spectrum identically. However, it may not be suitable for time series, which often have more informative low-frequency components. We empirically found that direct application of standard diffusion to time series may cause gradient contradiction during training, due to the rapid decrease of low-frequency information in the diffusion process. To this end, we proposed a novel time series diffusion model, MA-TSD, which utilizes the moving average, a natural low-frequency filter, as the forward transition. Its backward process is accelerable like DDIM and can be further considered a time series super-resolution. Our experiments on various datasets demonstrated MA-TSD's superior performance in time series forecasting and super-resolution tasks.

A Non-isotropic Time Series Diffusion Model with Moving Average Transitions
Chenxi Wang 12
Linxiao Yang?
Zhixian Wang" 12 Liang Sun 2 Yi Wang'
Abstract
time series analysis (Yang al., 2024). So far, pioneer works have accommodated the diffusion model for time se- ries forecasting (Rasul et al., 2021; Shen & Kwok, 2023; Li et al., 2022), missing value imputation (Tashiro et al., 2021; Alcaraz. & Strodthoff, 2023), uncertainty quantification (Li et al., 2024) and so on.
Diffusion models, known for their generative abil- ity, have recently been adapted time series anal- ysis. Most pioneering works rely the standard isotropic diffusion, treating each time step and the entire frequency spectrum identically, How- ever, it may not be suitable for time series, which often have more informative low-frequency com- ponents. We empirically found that direct applica- tion of standard diffusion time series may cause gradient contradiction during training, due to the rapid decrease of low-frequency information in the diffusion process. To this end, we proposed a novel time series diffusion model, MA-TSD,, which utilizes the moving average, a natural low- frequency filter, as the forward transition. Its back- ward process is accelerable like DDIM and can be further considered a time series super-resoltton.. Our experiments on various datasets demonstrated MA-TSD's 's superior performance in time series forecasting and super-resolution tasks.
Despite the initial success of these works, most of them still relied on the classical standard isotropic diffusion model, namely the Denosing Diffusion Probabilistic Model (DDPM) (Ho et al., 2020). It treats each time step indepen- dently and applies the same diffusion schedule. In the fre- quency domain, both low and high-frequency components are also degraded identically (see Figure 1). However, low- frequency components are usually more informative than high-frequency ones in time series analysis (Xu al., 2024). We found that decreasing the low-frequency components identically to the high-frequency ones during the diffusion process may cause drastic reduction of the essential time series information. It may further lead to contradictions on the gradient directions of DDPM at different diffusion steps, impeding the training convergence (see Section 3). Therefore, it's inappropriate to handle all the frequencies with the same diffusion process, and the classical design of the DDPM doesn't fully fit the inductive bias of time series data.
1. Introduction
Time series data is widely adopted in the real world. Ex- tensive examples include electricity consumption in power systems, stock prices in financial markets, traffic flows in transportation systems, etc. Over the past decade, remark- able time series models have been developed with versatile deep neural networks to perform various time series analy- ses (Wang et al., 2024).
To tackle such inequality in the frequency domain of time series, we utilized moving average operation, a natural low- pass filter, to build non-isotropic time series diffusion model, Moving Average Time Series Diffusion (MA-TSD). In the forward process. moving averages are set by small- to-large kernel sizes, gradually coarsening the time series until zero zero-frequeee free-freyency components. The corresponding transi- tion matrices are no longer diagonal like standard diffusion models. A corresponding dataset-based noise schedule is provided alongside. Similar to Denosing Diffusion Implicit Models (DDIM) (Song al., 2021a), we give an accelerable backward process with customized strategy to select back- ward steps. Naturally, with the coarse-to-fine philosophy, the backward process of MA-TSD can also be viewed as time series super-resolution. Empirically, we show on di- verse datasets that MA-TSD has outstanding performances over time series forecasting and super-resolution tasks.
In recent years, the diffusion model (Ho et al., 2020) has risen shining generative model, showing remarkable per- formance for image and video synthesis. Such supercity in modeling complex data distributions also drives the commu- nity to seek how to adapt it to time series, and thus empower
'Department of Electrical and Electronic Engineering, The University of Hong Kong, Hong Kong SAR "DAMO Academy, Alibaba Group, Hangzhou, China. Correspondence to: Yi Wang @yiwang@cee.hku.hh... eoc.hku.hk>.
Proceedings of the 42nd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).
Contributions:
1) We empirically disclosed the training

---

A Non-isotropic Time Series Diffusion Model with Moving Average Transitions
Degraded Data (Kara)
Added Noise (8,e))
Mm
DDPM
M
Magnizudi
Diffusion step
-10
-00
Proposed
MMM
Magnitrade (II))
-25
Froegemny ful fu/
Figure 1. DDPM versus our proposed moving average diffusion process. Visualization relies reparamcterization, i.c. K BBEE. Left: Comparison in time domain. Right: Comparison in the frequency domain. For illustration, e, is fixed.
issue when directly applying DDPM to time series data, and explored the relationship between the gradient similarity at different diffusion steps and the change of frequency infor- mation. We accordingly proposed novel time series diffusion model with moving average transition. The backward process also be naturally considered as time series super-resolution. 3) We conducted extensive experi- ments to demonstrate our salient performances over existing DDPM- based diffusion models on time series-related tasks like time series forecasting and time series super-resolution.
whole process is modeled as a Markovian chain:
II [<<x (x)))
where the one-step transition is given by: ( = N (Vara,-1; : (1 (1- at) The coefficient ae E [0, 1] monotonically decreases with t. Through the property of Gaussian distribution, the transition from 2n æ; can be derived:
g<>;)=0)) = N -N:==NVVVaae (Varzo, (1 - (1-āt) at)
(3)
2. Background
with the transition coefficient a; = I1 ag Through Bayes rule and the property of Markovian chain. Equa- tion (2) can be reformulated as:
Given samples from data distribution q(æo), diffusion models are latent variable models in the form of (æa)
f Po (XD:7) dæi.T. trying to approximate the unknown q(xo). The joint distribution is usually modeled as a Marko- vian chain: p(Zor) = p(ær) I ( xz). The la- tent variables of the diffusion models lie in the same space of the original data, i.e. x, E L Vt E [0, 10.......7 T). In our context, Zg is time series with time steps.
T
q (zo) = 9(01 xx)) II g(x1-11211, t=2
(4)
where the close form of g( vo) can be analyt- ically derived by Bayes rule. With the factorization of the forward po (* and the backward q( (2n)) the optimization target (Equation (I)) can be simpli- fied. large part of the simplified optimization tar- get ming [ Therefore, the backward one-step transition po (xt-1l is modeled to approximate the true posterior:
The trainable parameters 9 are optimized minimize the negative variational lower bound of the log-likelihood the data distribution q(xo)::
(z 20))|Pe (x))]
min L By( [log q (2)) log po (woo)) )] (1)
where the conditional joint ( (x)) is the core of diffu- sion design. The designs of classical DDPM and DDIM will be introduced to pave the way for our proposed method.
Pe (2t-1/2;)) (x. 1 fo( t)):
(5)
where fo f) is a trainable denosing neural network to estimate the æo. The loss function can be consequently simplified as:
2.1. Denosing diff'usion probabilistic model
In DDPM, the forward process degrades an original data Zn by gradually compressing the data and adding Gaussian noises until T E N+ diffusion steps so that all the structures of original data are lost, i.e. q(2TT (m)) = N/O0 I). The
C = E
[15o (æx, t) x x00l
(6)
with x, = Vajaa + --- e NO I) via reparame- terization. This means that the network fo takes in the noisy
2

---

A Non-isotropic Time Series Diffusion Model with Moving Average Transitions
data 2y and the current diffusion step I, and outputs the prediction of the clean data øо- Alternatively, the network fo can also be optimized to estimate the added noise e,, and then apply Zn = (x:-- - vI - ā.fe t))/ vā, to obtain the prediction of clean data. After training, one can simply generate a synthetic data sample by iteratively denoising a ar ~ p(ær)) with Equation (5).
types of time series analysis tasks can also be modeled in this way, for example super-resolution (conditional on low- resolution time series).
3. Empirical Findings on Applying DDPM to Time Series
In this section, we discovered a training issue when directly applying DDPM on time series (see Appendix A for de- tailed settings). As depicted in the left of Figure 2, the training process of DDPM on a typical time series dataset, Electrin ricity, experienced large variations, even in the beginning stage. As we introduced above, the DDPM is op- timized denoise the corrupted time series at all diffusion steps (see Equation (6)), but we observed that the direc- tions of model gradients at different diffusion steps could be contradicted during training. When we specifically probed the 100th training step (see the left down of Figure 2), the gradients of about the first 25% diffusion steps show greater similarity, while they could be opposite with the rest 75% diffusion steps, and vice versa. Since each data sample is corrupted to varying degrees during DDPM training, the averaged gradient of each mini-batch can accordingly show considerable variation with polarized per-sample gradients. Consequently, it may lead to the unstable optimization of DDPM on time series.
2.2. Denosing diffusion implicit model
Based on DDPM, DDIM generalized the foward process to be non-Markovian and derived a new backward process, First, DDIM bypasses the DDPM design of ((1z/(2(-1) and directly considers the conditional joint (( (mo) in the form of Equation (4), where g( (r, 2o) is specially designed to ensure that for all t, the transition g always matches Equation (3), In other words, the marginal is satisfied, I q q(<+-1 _ To ( a do) = q( (2))-
Since the conditional joint g( (a)) is defined as the same form as DDPM's, the optimization target of DDIM can be identically factorized, and the backward one-step transition is also chosen as Equation (5). In this way, DDIM shares the identical loss function of DDPM.
For backward process, DDIM offers an accelerable infer- ence option. Specifically, given an ascending subset of [1,... .T). denoted as {T Vt, E [0, with r S T,
DDIM allows the following sampling scheme:
To further analyze the reason for this phenomenon, we first explored the change of the frequency information during the whole diffusion process (see the right up of Figure 2). We computed the spectral energy ratio between low- oowffrquuecyy components and high-frequency ones of the time series at each diffusion step. During the whole diffusion process, low-frequency energy was dominant at the early stage, and then steeply decreased until turning point after which the cornupted time series barely had salient low-frequency information, i.c., they became almost noises. If we compare this energy ratio change with the gradient similarity with different diffusion steps (see the right of Figure 2). We found that the shift of gradient directions is highly aligned with the turning point of energy ratios. It implies that when DDPM is directly applied on time series, low-frequency information decays so steeply that there is large discrepancy in the model's perception of the input data, with few steps in the early diffusion being informative, whereas, in the middle and late diffusion, they are nearly noises. Therefore, to prevent the drastic decline of the energy ratio, we expect a time series diffusion model to have a gradual diffusion process that can keep more low-frequency information.
Tt.-, = @/uu-fo(110 +;)+
Dt. Von. fo t.) + TT, E,
م nh.
where } are hyperparameters. DDIM can generate rea- sonable synthetic data within 50 steps for images. Besides, when Wr = 0, also known deterministic sampling, such scheme is considered as the numerical solution of proba- bility flow ordinary differential equation (Son) et al., 2021b).
2.3. Conditional diffusion for time series forecasting
Time series forecasting can be viewed as a conditional gen- eration task. Given a look-back window c E RH with H time steps, we are aimed to predict the next L. steps, i.e. the target window Vo. In other words, we are interested in the conditional distribution ( To include the guidance of the look-back window into the diffusion model, we can add the condition at each transition (Shen& & Kwok, 2023):
T
In the following sections, we will introduce our method, Based the moving average, natural low-pass filter, it keeps more essential time series information during diffu- sion, alleviates the gradient contradiction during training, and thus obtains a less fluctuate training process (see the
p(ZorT/c) = p(æT) II
1 c). [=1
(7)
Accordingly, the condition is handled as another feature input to the denosing network, fo(21, 1, c) R an- Other
3

---

A Non-isotropic Time Series Diffusion Model with Moving Average Transitions
Energy ratio
D0PP Cluz
10
Train
16
200
100
250
500
step
diffuusio stop
diffusion step
diffusion stop
Gradient Similarity
DDPM
11.0
2010
400
500
240 step=0
300
-11
T=100
T=500
T=1000
train stsp=100
Figure 2. Gradient analysis on Electricity dataset. Left up: Training loss curves of DDPM and ours. Left down: Cosine similarity
matrices of gradients w.r.t. denosing network with different diffusion steps (total diffusion steps 7 1000). Right up: Energy ratio different diffusion steps between the low-frequency components and the high-frequency ones. Right down: Cosine similarity matrices with different T' at 1h0 training step.
where pa-), a-)) denotes the functions obtaining the mean and standard deviation of a time series, respectively. There- fore, we will build on the diffusion framework on the nor- malized time series. The denormalization strategies will be investigated in Section 4.4.
pelBo-a ai)
Norua
Demoem
NY
MMN
4.2. Non-isotropic forward process
Our proposed MA-TSD. Dashed lines denote the forward process, while solid lines represent the backward process.
Figure 3.
We now consider the following transition process for normal- ized time series, g(zr|=o) := - K,z0 1)., which can also be reparameterized to:
left of Figure 2).
2: = K,z0 + B.E,, ((0,
(9)
with the fransition matrix K, E LRLLL, and the noise sched- ule 3g R. In the standard DDPM, the transition matrix diagonal with identical entries, K = diag(Vā.). and the noise schedule is set accordingly maintain the variance, 8 v1 Cz..
4. Moving Average Time Series Diffusion
In this section, we present our proposed MA-TSD, depicted by Figure 3. Time series will first be normalized to be zero- mean and unit-variance. Then, we utilize moving average to build the diffusion model. Finally, the generated time series will be denormalized for downstream tasks.
Transition matrix. In our design, we expect to utilize moving average to build our non-isotropic transition. First, let us consider non-overlapping moving average filters. The kernel sizes {ki) are naturally chosen as the factors of the length of our target time series, given by:
4.1. Instance normalization
Since moving average operation doesn't 't modify the zero- frequency component of time series, the final p(ær) will no longer be easy-to-sample standard Gaussian distribu- tion if we directly construct moving average diffusion on Therefore, we apply instance normalization to each target time series sample Zg E R and obtain the normalized 2n E L with zero mean and unit variance, specifically:
(1:) (ke N k. < L, mod k; = 0). (10)
Here, we sort the {ke) in ascending order, and use the index instead of t to tell from diffusion step indices for now.
The corresponding moving average kernels are denoted as [K and for each kernel, it convolves the normalized time series, i.c. K; * 2g. Such convolution can unrolled and reformulated as matrix multiplication for generality,
-((0))
(8)
o(w))

---

A Non-isotropic Time Series Diffusion Model with Moving Average Transitions
namely K; =g K 120 with K; (nroll(()) E RLL/k))xL We can further interpolate K; along the time step axis to make it square to keep the shape of 0 un- changed during transitions, and thus the transition matrix for nth moving average kernel is given as:
The mean is chosen in order guarantee that the mod- elling of the joint matches the marginal for all t, i.e. J g( (z)g(==|=0))
g(zr-1/z)). In other words, the choice of Equation (14) ensures ((=)|z0) = N (z,; , T) for all t. The proof of such choice is included in Appendix B.3.
K = Interp (UnroLI(K.)) E RLXL
(11)
4.3. Accelerable backward process
Though the moving average transition matrices are defined well, we find that such transition could have large jumps between adjacent kernel sizes, since the factors of the time series length can be con-consecutive integers. To extend such design to the continuous case (i.e. arbitrary diffusion steps), can further interpolate {K] along side the diffusion steps to have unlimited 7-step transition matrices H
Analogous to DDIM, we now define the backward process on the normalized time series as follows: p9(2x-1(zz) = q (Zz-1/z;, fo(Zz, i, c)), where the denosing network fo ( c) tries to predict the clean normalized time se- ries. For generality, we include the possible condition as input of the denoising network for the conditional gener- ation task, like time series forecasting. For unconditional tasks, we can simply set = Ø.
[KT = Interp ((K))),
(12)
For the normalized time series, we modeled q(z1:1 and po (Zt-1 (2+) the same as DDIM, so it's natural to derive the similar optimization target as Equation (6) but in the normalized space:
where K; is no longer a diagonal matrix like DDPM. A sim- ple example of how we obtain the moving average transition is included in Appendix B.I.
Noise schedule. For the noise schedule, we follow the vari- ance preserving principle in (Ho et al., 2020; Song et al., 2021b). The noise schedules of the standard diffusion can be analytically designed to keep variances according to the transition coefficient vä., i.e. 3: vI Cg, while the decrease of time series variance caused by moving average varies by datasets. Therefore, we provide a dataset-based noise schedule to complement our forward process. Specifi- cally, we can first compute the averaged decrease ratio " over the whole time series dataset:
C2 = Es co0.,cr(((00.) [1L6 (2), zoll] (15)
A After training, we can also accelerate the backward process as we introduced in Section 2.2, given as:
21:11 = K t. fo ti c)+
B7 B n (z - -u.KKuit (Z1,, ty. c)) + mi;E,
(16)
Br.
where {t:]T is an ascending sub-sequene of [1, [1,.... .T. The detailed proof that the accelerated backward process doesn't 't essentially change the training objective can be found in the (Song et al., 2021a).
(K o (xx)
T = Eeg ((23))
(13)
Then, we accordingly set 8t = VI- 7p as our noise sched- ule to compensate for variance decrease. the last diffu- sion step, the kernel size of the moving average equals the time series length, and the corresponding "T 0 = 1, (0, I). It should be noted that
Acceleration strategy. Though the backward process can be fastened by selecting subset of total diffusion steps, how this subset is chosen may cause performance differences. Here, we offer a reasonable sampling strategy based on our moving average forward process. Specifically, we recall that the diffusion transition matrices } are obtained by interpolating the original moving average transition matrices { [K]). We consider shortening the backward process by finding those diffusion steps whose transition matrices the closest to the original {K]). For the 2t original matrix Ky, we search by:
which ensures ( though we conduct the diffusion model on the normalized time series, it makes no difference to compute (7x), { over the normalized or original time series, The proof can be found in the Appendix B.3.
Conditional joint distribution. Similar to DDIM, we directly define the following family of joint distribu- tion: g 20) == ((=T|=0) 11. g(zz- 1/z,, zo), where ((z)|zo) = N (xr: K KT I) and for t 2:
f = arg min K, 1/ s.t. K. (KH) (17)
(zT:
Therefore, we can collect n diffusion steps (47T corre- sponding to the original non-overlapping moving average kernels, as our accelerated backward steps. We call such a strategy as factor-only backward since the selected back- ward steps are only related to the factors of the length of
0(2111/21,20 20)
1
18 n/?
(14)
W K:-120
(zi
K
Kizo),
Br
5

---

A Non-isotropic Time Series Diffusion Model with Moving Average Transitions
the time series. When 7/t = 0, Vi E [1,... .T. the back- ward process can also be viewed numerical solution to an ODE with Euler discretization. Further, if the function Interp(-) in Equation (12) interpolates evenly between K", 1, "",,K,11 1 Vi 2 2, the factor-only backward essentially solves that ODE with larger steps. Refer to Appendix B.3 for details.
prediction network for á we optimize both the denosing model fo and the statistics prediction model with parameters w. 99 = { ure 7 shows how these two networks work together for time series forecasting. Specifically, we modify the loss function on the normalized data (Equation (15)) into a hybrid case:
- J. in a hybrid manner. In the appendix, Fig-
Lnybrid = >x2Zz + Xu + No
(18)
Backward as super-resolution. As the forward pro- cess is designed as gradually coarsening the time series, the backward process can be spontaneously utilized for super-resolution. Here, we don't 't have to include the low- resolution time series as the condition input into MA-TSD, since the framework itself demonstrates the multi-resolution property.
where we denote Cu = -Beogg(o)) |l (c) ((
and Lo = Exo-~((o)) |198 (c) -(0))) The coeffi- cients λu. Au: )o are hyperparameters. We interpret that C= aims to learn the shape of the target time series, while Cu and C. are set for the statistics. Therefore, we can set j g"(c), of time series forecasting. Besides, when the coefficients Xe Ans Xo are set properly, we can prove that the hybrid loss (Equation (18)) is essentially the upper bound of the loss of conditional diffusion models without instance normalization. The detailed proof can be found in Appendix B.3.
[ to denormalize zu in the context
To be specific, let us consider coarse time series whose downscale rate is one of the factors of the original time series length. Utilizing the moving average transition ma- trix, we denote such coarse time series as æ; K/ and denote the normalized one as 2;. The super-resolution scale we expected is then exactly kG. With Equation (17), we can locate such scale in the diffusion process, and accordingly choose a subset of total diffusion steps [1,.... : [.]. Then, can start with _;;- = 2s. and iteratively apply Equa- tion (16) to obtain a super-resolution result of z,. Therefore, the backward process of MA-TSD can be viewed as super- resolution.
Time series super-resolution For super-resolution, since the coarse time series æ; obtained by moving average shares the same mean of the target time series, we can use j = u(æ;). On the other hand, though the standard devia- tions are not shared, we can utilize the dataset-basse noise schedule to re-seale (xx as: â = ( where "/e is the decrease ratio of the standard deviation at the diffusion step & mentioned before.
Compared with the diffusion-basee models which take low- resolution inputs as conditions, the time complexity is de- creased from (n.M) to ((M), where n is the number of scales and (M) is the complexity of training.
5. Experiments
In this section, we mainly focus on two important time series analysis tasks, forecasting and super-resolution. The stan- dard time series synthesis task is included in Appendix C.I. An ablation study of our framework design also included.
4.4. Denormalization
Given the denoised 20 from Equation (16), we need to denormalize it to generate the final time series æo. i.c. to 20 ã + f.. In this section, the choice of üt â is considered different, depending on the downstream application of MA- TSD.
5.1. Time series forecasting
Datasets. We consider six real-world datasets with di- verse temporal dynamics, commonly used by the commu- nity (Wang al., 2024), namely El Electricity, ETTh2, ET Tm2, exchange, traffic, weather.
Time series synthesis. For the unconditional generation, we usually expect unlimited time series synthesis. There- fore, we can simply sample from the empirical distribu- tion of time series means and standard deviations, á ~ Temp (H(o)), â ~ gemp (omp(a (Zz)). Given a time series dataset, we can easily obtain the empirical distributions and denor- malize them.
Evaluation metrics. We assess time series forecasting us- ing MSE (Mean Squared Error) for deterministic accuracy and CRPS (Continuous Ranked Probability Score) for prob- abilistic accuracy.
Time series forecasting. For time series forecasting, the mean and standard deviation of the target window is vital for the prediction accuracy (Kim et al., 2021; Qin et al., 2024). It's improper to randomly sample from the empir- ical distribution to conduct denormalization. We consider to utilize the look-back window e to produce the jt, ā for the target window. To prevent training a separate statistics
Benchmarks. We compare our proposed MA-TSD with other diffusion- brasdd time series forecasting models, includ- ing CSDI (Tashiro et al., 2021), SSSD (Alcaraz& & Strodthoff, 2023), D3VAE (Li et al., 2022), TMDM (Li et al., 2024) and mr-diff (Shen et al., 2024). Details about the implemen- tation and comparison to non non-diffusion models are included in the Appendix C.2.
6

---

A Non-isotropic Time Series Diffusion Model with Moving Average Transitions
Table Average MSEs over prediction lengths
(96, 192, 336, 720).
The best bold and the second best underlined.
METHOD
EnRCTRTCCTE
ETTH2
ETTH2
EXCEANGE
TRAFFTC
NEATHER
RANK
CSDI
0,4581
0.2571
2.1230
1.2557
0.4991
0.1938
3.50
SSSD
1.0257
0.7201
0.8936
2.9004
1.9662
0.6905
5.17
D3VAE
0.8450
1.3961
3.3449
2.1086
6.3583
1.5461
5.67
TMDM
0.4071
0.2508
0.1789
0.7885
0.1805
0.2209
2.83
MR-DIFF 0.5287
0.2172
0.1700
0.4801
0.2471
0.2078
2.67
MA-TSD 0.3404
0.2121
0.1241
0.3718
0.1660
0.2074
1.17
Table Average CRPSs over prediction lengths L = (96, 192,336, 720).
The best bold and the second underlined.
METHOD
ELECTRICITY
ETTh2
ETTx2
EXCEANGE
TRAFFIC
NEATHER
RANK
CSDI 0,1939
0.
0,4720
0.3028
0.1883
0.1261
3.33
SSSD
0.3216
0.3108
0.3565
0.6743
0.4579
0.3129
5.17
D3VAE
0,3111
0.4173
0,6497
0.5380
0.8314
0.4497
5.67
TMDM
0.1881
0.1591
0,1253
0,2959
0.1076
0.1380
2.00
MR-DIFF
0.2357
0.1647
0,1293
0.2117
0.1453
0.1506
3.17
MA-TSD 0.1747
0.1567
0.1135
0.2178
0.1156
0.1501
1.67
Results. As depicted in Table I and Table 2, the proposed MA-TSD generally outperforms the benchmark time series diffusion models, achieving the best or the second best posi- tion 6/6 and 5/6 datasets, respectively. The improvement on the weather dataset is marginal, possibly because it is recorded in higher resolution (Table 6). Both informa- tive high-frequency components and stochastic noises are revealed, which could be simultaneously suppressed by the moving average process, and thus lead to the difficulty to accurately forecast. Refer to Appendix C.2 for visualization and Table for the full results on each prediction length.
benchmark models are re-trained for each super-resolution scale, since the condition inputs change. Details about the implementation are all included in the Appendix C.3.
Results. Table records the results of time series super- resolution. The proposed MA-TSD generally exceeds the conditional DDPM and FM-VP in terms of both consistency and quality. Despite the slight inferiority in Context- on the So aalar dataset at the large scale, ours still outperformed the FM-VP model significantly in terms of consistency. No- tably, MA-TSD performs SR naturally through its backward process instead of retraining individually on each scale. Be- sides, our SR backward starts in the middle of the whole process, resulting in even fewer backward steps compared to benchmarks (Figure 12). Therefore, we believe that our method provides better trade-off of computational over- heads, SR quality, and consistency to the low-resolution input. Refer Appendix C.3 for more visualization.
5.2. Time series super-resolution
Datasets, We consider three high-resolution real-world datasets with 5-minute resolution, MERED, wind, Solar. For each dataset, tasks, i.e. Amin-to-155mn (3x), Smin-to-30mi (6x), and 5min-to-60min (12x).
the models for the following 3
5.3. Ablation study
Evaluation metrics. We assess time series super-resolution by Consistency and Context-FID. The former one measures MSE between the low-resolution inputs and the down-scaled super-resolution outputs (Saharia et al., 2022), while the latter one examines the quality of the super-resolution results compared to the real high-resolution time series (Jeha et al., 2022).
In this section, we evaluate the effectiveness impor- tant components and designs of MA-TSD through unlim- ited time series synthesis on the mentioned MFRED, Wind, Sol datasets.
Key modules. Two key different designs from the standard time series diffusion model are the moving average diffusion schedule and the instance normalization in our proposed MA-TSD. We compared the possible design with/without these two components, as shown in Table Without MA and IN, namely directly applying DDIM on time series data, exhibited the worst performance. Equipped with either IN or
Benchmarks. We compare ours with two diffusion models directly conditioned on low-resolution inputs. One trained under DDPM framework (Saharia et al., 2022), and the other trained by flow matching with a variance-preserviee path (Lipman et al., 2023), which denote FM-VP. The

---

A Non-isotropic Time Series Diffusion Model with Moving Average Transitions
Table 3. Comparison on time series super-resolution. The best hold and the second hest underlined.
MFRED
WTND
SOTAR
SCALE
METHOD
CONSIST.
CONTEXT-
CONSIST.
CONTEXT-
CONSIST.
CONTEXT-FIL
MA-TSD
0.0032
0.1047
0.0067
0.2863
0.0106
0.3491
3
DDPM
0.0291
3.1028
0.0382
7.4843
0.0367
1.7197
FM-VP
0.0328
1.3481
0.0636
4.2311
0.0523
0.7950
MA-TSD 0.0037
0.1235
0.0098
1.0241
0.0115
0.6972
6
DDPM 0.0214
3.0740
0.0343
7.9524
0.0260
2.2293
FM-VP
0.0238
1.3381
0.0505
4.4683
0.0361
0.6846
MA-TSD 0.0047
0.4358
0.0136
3.0567
0.0129
1,4135
12
DDPM
0.0157
3,4044
0.0318
9.2504
0.0429
6.2596
FM-VP
0.0156
1.5222
0.0350
4.8737
0.0249
0.8616
random
uniform
factor-only
Table 4. Context-FID. of MA-TSDs with different components.
MA: Moving Average schedule. IN: Instance Normalization
Wind
MODULES
DATASETS
Contoor .TD
MA
IN
MERED
WIND
SOLAR
29.7093
52.3499
43.6236
V
13.1704
44.6908
35.3704
000
1/T
T/T
1/7
4.9403
21.3606
8.0744
V
v
2.6742
16.9026
8.0297
Figure 4. Comparison among accelerated backward strategies.
6. Related Works
MA, the model obtained considerable improvements, with MA functioning more significantly than IN, which implies the importance of MA in our framework design. Combining both modules achieved the best scores over all three datasets.
Diffusion models have been embraced by the time series analysis community for their advanced probability modeling ability. (Rasul et al., 2021) first combined autoregressive modeling of recurrent neural networks and the diffusion process for time series forecasting. Then (Shen & Kwok, 2023) proposed non-autoregressive diffusion strategy for foreasting, improving on both efficiency and accuracy. In addition, there are also several works (Kollovieh et al., 2024; Alcaraz & Strodthoff, 2023; Tashiro et al., 2021) that link time series forecasting and time series imputation, modeling them with conditional generation design, proposing unified frameworks for these tasks with diffusion models.
Accelerated backward. For given backward step budget T TS T T===10 =07=10 100, we compare our factor-only backward strat- egy with 1) randomly selecting subsets from [1.... ,T' and 2) uniformly choosing a subset, i.e. 1 Linspace(1 (1,,a (, Notably, given budget т, uniformly sampling renders a fixed subset [ta]T. resulting deterministic result.
As shown in Figure 4, given fixed ratio - T/T, randomly selecting the backward steps may cause large variances in model performances. Uniform sampling did offer a better result than random sampling, but the marginal gain on the performance decreases by T r/T. Compared to both random and uniform strategies, our factor-only strategy consistently offers fair and effective results given the same 7/2 budget, optimizing the resources while maintaining the quality of the results. Especially MFRED and Solar, whose periodical patterns are more significant, the factor-only accelerating strategy could even achieve competitive performance with the full-step backward, indicating that the strategy captures the key transition steps and can utilized for fastening MA-TSD.
Recently, the community began to fuse the unique time series property into diffusion models. (Fan et al., 2024) leveraged coarse time series data as guidance during the diffusion process, and added regularization terms into loss function to constrain the backward process coarse-to- fine. (Shen et al., 2024) set several diffusion stages, where the previous diffusion stage generates coarse time series as condition input for the latter stage to refine. (Liu et al., 2024) leveraged the historical windows to retrieve the k nearest samples as references to guide diffusion model to generate more accurate forecasts. Despite the recent special design on time series data, they still rely on the DDPM process and
8

---

A Non-isotropic Time Series Diffusion Model with Moving Average Transitions
hardly improve on the typical isotropic design to meet the characteristics of time series.
Impact Statement
This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here.
Beyond time series diffusion models, some works simi- larly investigated non-isotropic diffusion models for images. (Hoogeboom & Salimans, 2023; Rissanen et al., 2023) de- signed frequency-domain diffusion with Gaussian blurring as transition. (Daras al.. 2023) tried to generalize the transition to linear corruptions, and gave examples of blur- ring and masking while (Bansal et al., 2024) proposed diffusion model with arbitrary degradation functions, for example snowification and animorphosis, but without noise. However, few of them obtained tremendous improvements, let alone being further explored by our time series commu- nity.
References
Alcaraz, J. L. and Strodthoff, N. diffusion-based time series imputation and forecasting with structured state space models. Transactions on Machine Learning Re- search, 2023. ISSN 2835-8856. URL https:// openreview net wwrumroruur/omm ?idmhHiddiHiii Ibk 7ApW.
Bansal, A., Borgnia, E., Chu, H.-M., Li, J., Kazemi, H. Huang, F., Goldblum, M., Geiping, J., and Goldstein, T. Cold diffusion: Inverting arbitrary image transforms with- out noise. Advances in Neural Information Processing Systems, 36, 2024.
Regarding diffusion design, probably the most related work to ours is (Hoogeboom & Salimans, 2023), which shared similar high-level idea of building the degradation pro- cess with low-pass filters (blurring in theirs, MA in ours). However, they still tried to fit low low-pass filters into the tradi- tional DDPM's Markovian process in the frequency domain, while we reformulated a Markovann design with new backward process. Besides, our noise schedule is specially designed to be dataset-based regarding the variance de- crease caused by the filters on different time series data. Please refer to Appendix B.4 for more detailed information.
Daras, G., Delbracio, M., Talebi, H., Dimakis, A., and Milanfar, Soft diffusion: Score matching with gen- eral corruptions. Transactions on Machine Learning Research, 2023, ISSN 2835-8856. URL https:// openreview net
Fan, X., Wu, Y., Xu, C., Huang, Y., Liu, W., and Bian, J. MG-TSD: Multi-granularity time series diffusion models with guided learning process. In The Twelfth International Conference on Learning Representations, 2024. URL https: wptp//vww.eeeeeeeeeeeeeeeeeeeee. net. forum? id=c2iY60Lktd.
7. Conclusion
In this paper. we first revealed that direct application of standard DDPM to time series data may cause gradient con- tradiction, because of rapid degradation of low-frequency information. A novel time series diffusion model, MA-TSD, is accordingly proposed, equipped with moving average for- ward transitions to keep more low-frequency information. The backward process can be accelerated in a DDIM style and further act as super-resolution. The experiments show that MA-TSD has superior performances over the state-of- the-art time series diffusion models in terms of forecasting and super-resolution.
Ho, J., Jain, A., and Abbeel, Denoising diffusion proba- bilistic models. Advances in neural information process- ing systems, 33:6840-6851, 2020.
Hoogeboom. E. and Salimans, T. Blurring diffusion models. In The Eleventh International Conference on Learning Representations, 2023. URL https: /ttp:s/////eeeeieee net /
Software and Data
Jeha, P., Bohlke-Schneider, M., Mercado, P., Kapoor, S., Nir- wan, R. S., Flunkert, V., Gasthaus, J., and Januschowski, T. Psa-gan: Progressive self attention gans for synthetic time series. In The Tenth International Conference on Learning Representations, 2022.
Our codes can be found in https://github. com/ Wi willWangl113/Mor www///ooovvvveeaoaaeee Average- fifusiion.
Acknowledgements
Kim, T., Kim, J., Tae, Y., Park, C., Choi, J.-H., and Choo, J. Reversible instance normalization for accurate time-series forecasting against distribution shift. In International Conference on Learning Representations, 2021.
The work was supported in part by the Research Grants Council of the Hong Kong SAR (HKU 17200224), and in part by the Alibaba Group through Alibaba Research Intern Program.
Kollovich, M., Ansari, A. F., Bohlke-Schneine M., Zschiegner, J., Wang, H., and Wang, Y. B. Predict, refine,

---

A Non-isotropic Time Series Diffusion Model with Moving Average Transitions
synthesize: Self-guiding diffusion models for probabilis- tic time series forecasting. Advances in Neural Informa- tion Processing Systems, 36, 2024.
Peebles, W. and Xie, S. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF Interna- tional Conference on Computer Vision, pp. 4195-4205, 2023.
Li, Y., Lu, X., Wang. Y., and Dou, D. Generative time series forecasting with diffusion, denoise, and disentanglement. Advances in Neural Information Processing Systems, 35: 55009-23022, 2022.
Qin, D., Li, Y., Chen, W., Zhu, Z., Wen, Q. Sun, L., Pinson, P., and Wang. Y. Evolving multi- scale normalization for time series forecasting under distribution shifts, arXiv preprint arXiv:2409.19715 2024.
Li, Y., Chen, W. Hu, X., Chen, B., Zhou, M., et al. Transformer- Transformer-modulatee modulated diffusion models for probabilis- tic multivariate time series forecasting. In The Twelfth International Conference on Learning Representations, 2024.
Rasul, K., Seward, C.. Schuster, I., and Vollgraf. R. Au- toregressive denoising diffusion models for multivariate probabilistic time series forecasting. International Con- ference on Machine Learning, pp. 8857-8868. PMLR, 2021.
Lipman, Y., Chen, R. T. Q.. Ben--amu,, H., Nickel, M., and Le, M. Flow matching for generative modeling. In The Eleventh International Conference on Learning Represen- tations, 2023. URL https: / /tte:://opppee /pwpreveeeee...eoeeeeeep net// forum? Pwd.ddddPP...........
Rissanen, S., Heinonen, M., and Solin, A. Generative mod- elling with inverse heat dissipation. In The Eleventh International Conference on Learning Representations, 2023. URL https::///// /openreview net forum? id=4PJUBT9F201
/
Liu, J.. Yang, L., Li, H., and Hong. S. detrieval-augmentee diffusion models for time series forecasting, In The Thirty- eighth Annual Conference on Neural Information Pro- cessing Systems, 2024. https://: net/ net/forune www.eoooommmmmmeeeeeee httttpp:://ppreevveeew openreview.
Saharia, C., Ho, J., Chan, W., Salimans, T., Fleet, D. J., and Norouzi, M. Image super-resolution via iterative refinement. IEEE transactions on paltern analysis and machine intelligence, (5 45(4):4713-4726, 2022.
Liu, Y., Wu, H., Wang, J., and Long, M. Non-stationary transformers: Exploring the stationarity in time series forecasting. Advances in neural information processing systems, 35:9881 -9893, 2022.
Shen, L. and Kwok, J. Non-autoregreessive conditional diffusion models for time series prediction. In Inter- national Conference Machine Learning, pp. 31016- 31029. PMLR, 2023.
Meinrenken, C. J., Rauschkolb, N., Abrol, S., Chakrabarty, T., Decalf, V. C., Hidey, C., McKeown, K., Mehmani, A., Modi, V., and Culligan, P. J. Mfred, 10 second interval real and reactive power for groups of 390 us apartments of varying size and vintage. Scientific Data, 7(1):375, 2020.
Shen, L., Chen, W., and Kwok, J. Multi-rcsolution diffu- sion models for time series forecasting. In The Twelfth International Conference on Learning Representations, 2024.
Song, J., Meng. C., and Ermon, S. Denoising diffu- sion implicit models. In International Conference on Learning Representations, 2021a. URL https:// openreview net forum? dieiit=tt1ggi (d=StggiarrCH..
Naiman, I., Berman, N., Pemper, I., Arbiv, I., Fadlon, G., and Azencot, O. Utilizing image transforms and diffu- sion models for generative modeling of short and long time series. Advances in Neural Information Processing Systems, 37:121699 -119999-1113000 21700, 2024a.
Song, Y., Sohl-Dickstein, J., Kingma, D. P., Kumar, A., Ermon, S., and Poole, B. Score-based generative mod- eling through stochastic differential equations. In In- ternational Conference on Learning Representations, 2021b. htth:: U net /orum? id=PxTIG12RRHS
Naiman, I. Erichson, N. B., Ren, P., Mahoney, M. W. and Azencot, O. Generative modeling of regular and irregular time series data via koopman VAEs. In The Twelfth International Conference on Learning Representations, 2024b. URL https: http:///pp//eeeeeeeee wwwp:/rwwiewwweeeeeeeee net/ forum? id=eY7sLboavE.
Tashiro, Y., Song, J., Song, Y., and Ermon, Csdi: Con- ditional basee diffusion models for probabilistic time series imputation. Advances in Neural Information Processing Systems, 34:24804-24816. 2021.
Nie, Y., Nguyen, N. H., Sinthong, P., and Kalagnanam, J. A time series is worth 64 words: Long-term forecasting with transformers. In The Eleventh International Conference on Learning Representations, 2023. URL https:// openrevi www.:eeet/o/ttee dot.
Wang. Y., Wu, H., Dong, J., Liu, Y., Long, M., and Wang. J. Deep time series models: A comprehensive survey and benchmark, arXiv preprint arXiv:2407. 07.4007//227 2024.
10

---

A Non-isotropic Time Series Diffusion Model with Moving Average Transitions
Wu, H., Xu, J., Wang, J., and Long, M. Autoformer: Decom- position transformers with auto-correlation for long-term series forecasting. Advances in neural information pro- cessing systems, 34:22419 -24300, 2021.
Xu, Z., Zeng, A., and Xu, Q. FITS: Modeling time series with $10k$ parameters. In The Twelfth In- ternational Conference on Learning Representations, 2024. URL
forum?
id=bWenvZ3gMb.
Yang, Y., Jin, M., Wen, H., Zhang, C., Liang, Y., Ma, L., Wang, Y., Liu, C., Yang. B., Xu, Z., et al. A survey diffusion models for time series and spatio-temporal data. arXiv preprint arXiv:2404. 18886, 2024.

---

A Non-isotropic Time Series Diffusion Model with Moving Average Transitions
A. Details of directly applying DDPM on time series
As an example of illustrating the potential glitch when DDPM is directly applied on time series data, we used Electricity dataset, typical time series dataset with complex patterns. We consider L 400 and the spectral energy ratio is calculated follows:
Σ 1i=1
(19)
Nxw + |ug
where we denote the real fast Fourier transform (rFFT) of the time series 2g as RAo =[u)] ug: ... , 240 Nom and uz E C is the hth frequency component. The parameter is the split ratio. We set zound(0.2 NNyg)) in this example, since the frequency components are a relative low level after then (see Figure 5).
00
Mageitai
0.3
022
Prequcoc (divided by Nste))
Figure 5. Frequency energy of data sample in Electricity dataset. The x-axis is devided by Naya
During training, we fixed the random seed to guarantee the same initialization of models and the order of data samples. In this simple example, for each epoch, we did 20 times mini-batch training with batch size set as 64, thus in total 2000 training steps.
B. Details of MA-TSD
B.1. Transition matrix example
As depicted in Figure 6, we consider the situation of L = 6 as an naive example to illustrate how we get the square transition matrix K, with moving average.
B.2. Combination of denosing networks and conditioning networks
The combination of denosing networks and conditioning networks are shown in Figure 7. The Encoder embeds the noisy 2; and fuse with the position embedding of In the conditional generation, the condition e is also encoded and fused together. Then, the Decoder will output the prediction 2g, and the conditional decoder will output the o, P for denormalization, needed. During inference, 2;-1 will be obtained by 2, and ép.
12

---

A Non-isotropic Time Series Diffusion Model with Moving Average Transitions
Unroll
Interp. timee steps
Ritl
Kit1*
di diffiiiio strepe
Interpi
[K
Unroll
Interp.
K
Figure 6. Example at kn = 2 and k2 = 3. The convolution kernels are first unrolled to matrix, and then interpolate along side the time steps to be square, Across the diffusion steps, the transition matrices also interpolated.
B.3. Proofs and Derivations
B.3.1. DECREASE RATIO N/ CALCULATION
We denote the decrease ratio calculated over the 7/:
Kr u(to aaaono
a (K,z))
T (K, (2:0 p(x))))) a(z0 m(xx0)
1/- = Exgrg(Zz)
Emoong(ne))
E Exorg)
o(z0)
onn afamn
a 100 ((xO))
= -Bag~g(ar))
E ((oo)
%t
o(x0- u(z)))
where the first equal the second line is because moving average doesn't change the mean value of æp, i.e. Ke(wo- - (((o)) = K - p(xo), and the second equal in the second line is because a + const.) = (x0)) Therefore, it's the same to calculate 7 over 2n and zu.
B.3.2. THE CHOICE OF g(=1-11=1, 20)
Given the defined q(21.T| 20)= = g(=r (2)) II= (2z--1/ 20), the defined q(2,-1/2e 20) in Equation (14) and (<z](20) == A (zT: 2o IT.) we can have (z) N (ze Kizo, I) for The proof is similar that of (Song et al., 2021a), with the transition generalized K,
Proof. We can prove the above statement through induction argument. Assume that for t < T, q(<e|=0) N (z: K2 20, I) stands, and if q( = N (z--1: - 82_ I) also stands, then we can prove it from i = T to = 1 with the initial q(TT|=0) = . (zr; Kr 83.1).
First, via the marginalization, we have:
q( g(z(-1/20) = g(z1-1|Zz, 20))g() zo)dZz.
(20)
13

---

A Non-isotropic Time Series Diffusion Model with Moving Average Transitions
Encoder
Decoder
20
Z0
20
Pos. Emb.
Cond. Enc.
Cond. Dec.
Figure The denoising and statistics prediction models time series forecasting. include unconditional generation c Ø, we make the dashed line for illustration. Pos. Emb.-Position Embedder, Cond. Enc.-Condition Encoder, Cond. Dec.-Condition Decoder.
BBL T (z)
With the given d==x/z0)= = - (z1; Ki00, 1) and g(zz- 20) = N (K-
K,z0), we
can deduce that g(zt-1|z0) is also Gaussian, with the mean Ps--: and the variance D
V Bi-1 mi
Na-1 K
(Kı20
K(zu) = K(-1
Br.
(21)
B
nh I P7-11.
Ez
1/- 18
B
Therefore, ((zt-1/20) -N(K, (( holds and the inductive argument can be processed.
B.3.3. BACKWARD ODE
Let us first consider our backward process without acceleration, given by:
B Vi
= -.1=(_--((3,
B. (z+ Kefo (zt,t,c)) + m.c
(22)
When 7/2 = 0, we can have:
B--1
2₁-1 = Ki-
(zt
Kifo (Zi,tt))
(23)
Br
After reformulation, we further have:
21-1
21
K
fo(zztt,))
(24)
3,-1
B,
B
3e
which can be viewed as the numerical solution with Euler method to the following ODE with the discrete step At 1:
2t d
fo (21, (2 t)d
KJ
(25)
B,
Br
Now, we consider the even interpolation between {K'} J where we anchor the original K", and lincarly inject m intermediate
matrices between K' and K Then the collected diffusion step subset { are essentially {i(m + 1) = {m + 1, 2m + 1), ... nmm + 1)}. Therefore, backward with {tz } in this situation is basically Euler method with the discrete step At m
14

---

A Non-isotropic Time Series Diffusion Model with Moving Average Transitions
B.3.4. HYBRID OPTIMIZATION
The conditional diffusion loss function without instance normalization over æ can be written by:
Lx Exo.c,t (ar, i, 20
(26)
where we he distinguish from So and Iu mentioned previously.
Further, express the into: 2o axxo) + (ıxo), and then parameterize the denosing network accordingly: he(2z, E, = fo(zi,t, c) (c) + g
For simplicity, we denote l = ( 1, - zo 12. and have:
1 = (a,, c) - mol1
(27)
(1fo(zr,t,c) (c) + (c) 20 o(0)) - ((
(28)
I (llfo(ze,t,)) (c) 2n a(z))ll2 (c) - m(xx))ll2)
(Triangle inequality)
(29)
(Ilfolz,ttc)) (((()) 2o o(x)) + = [c) - ( + 12)
(30)
(l|zo (9 (c) a (2o)) ) + (C( (fo(z),4,c) 24,)
(31)
I
- o( | c. =20ll2 + 1,)
(Triangle inequality)
(32)
(11 (10 1,)
(33)
- (11z + (c) + (1 + + 12)
(Cauchy inequality).
(34)
Further, we can scale | to the maximum absolute value of the all the 2a over the dataset. Then, for a (c). tries to approximate the real standard deviation, (c) s o(xo), and the maximum oxx)) over the whole training dataset ((x))) can be obtained. We can further limit the output of the 9 the largest o(z)). Therefore, we can scale
(11=oll3 + 1 (c) 1) (Zmax + m 1) =
Therefore, we have:
Cx ox
[+]
(35)
< E.o
[11=00|:: + [b)(())] +1) (1 + +12 +(++111
(36)
< E Fop..et
(12 + +1 12)1
(37)
We compare the hybrid loss funetion:
Chybrid Aghx + AnLa AbCo
Exo.c, fo c) -20l + An (ull - p(xo)1 (o [x0)
(38)
-ErO.C [det2 + Aul + dale]
We can see that if the A2 = Ap = >o X, then La S Luybrid-
B.4. Differences from Blurring Diffusson Models (Hoogeboom & Salimans, 2023)
From a high-level perspective, BDM and ours shared similar idea, i.c., building the degradation process with low-pass filters (blurring in BDM, MA in ours). However, there exist clear distinctions.
Filtering space: filtered the data in the time space by convolution (matrix multiplication), g(x,, mo) = N(K 3 while BDM blurs the images in the frequency domain and transform back the pixel domain (using convolution theorem), i... q 2u) = (VVa,, V Zo, I) where V V are Discrete Cosine Transform (DCT) and Inverse DCT (IDCT), and (r, represent the frequency response of Gaussian blurring kernel, diagonal matrix, whose each entry of E (0, 1] is coefficient of ęth frequency component. For low pass filters, Vt, of decreases by i until (nearly) zero to suppress high frequency components.
Markovian or not: Since a is diagonal, BDM proposed that for each frequency component, a standard Markovian DDPM can be constructed. The one-step transition is accordingly defined, i.e. q(u, u = =N(a Ut- x2 / I),
15

---

A Non-isotropic Time Series Diffusion Model with Moving Average Transitions
where u4 V To is the frequency representation and 0:jt-1 / However, dividing 04-1 could be numerically unstable in practice. As mentioned above, of could become to be (nearly) zero for larger dividing 0t-1 is unstable for all diffusion steps. Though some epsilons can be added to ensure stable division, tiny errors in the high frequency components will still be amplified a lot through the iterative backward process. Chances are that the generated data are dominated by the improperly amplified high frequency components. Therefore, when it comes to designing a diffusion process with non-reversible low-pass filters, we believe it's improper to follow the Markovian assumption and define the necessary g(#, (.-1)). Instead, in our framework, faced with similar non-reverible MA, we bypassed the definition of q(x;, (x-1). assumed q(T1.T z0) -on-Markovi (in the DDIM-style) and then delicately defined g( Tt. z6) to satisfy (2x; ano) for all Thus, whether it's Markovian not is another distinct difference from ours and BDM.
Noise schedule: BDM designed az = a, where d, is the frequency response of blurring kernel and E [0, 1] is an extra scaler decreasing by t, and the noise schedule = 1 - a² In our framework, the noise schedule 3 is dataset-based, chosen regarding the variance decrease caused by MA (Equation (13)) on different datasets.
In summary, despite the similar bigh-level idea, there exist clear differences between BDM and ours. BDM tried to fit the standard Markovian DDPM framework, while we reformulated a framework with special adaptation on moving average filters and time series.
C. Experiment details
We launch our experiments on a single NVIDIA GeForce RTX 4090 24GB GPU.
C.1. Synthesis
We follow the setting of KoVAE (Naiman et al., 2024b) and ImagenTime (Naiman et al., 2024a) on three datasets, ETTh2, Exchange, and ECG (medical time series)', i.e. L = 24. KoVAE is SOTA VAE-based time series generative model while ImagenTime a diffuison-basee one.
We also include discriminative score (Dise. Score) and predictive score (Pred. Score) as additional metrics for evaluating the fidelity and usefulness of synthetic time series, the following table shows.
Table Results on standard time series synthesis
DATASET
MODEL
DISC. SCORE
PRED. SCORE
CONTEXT-FID
KOVAE 0.069
0.034
0.258
ETTH2
IMAGENTIME
0.053
0.054
0.118
OURS
0.044
0.026
0.075
KOVAE
0.137
0.038
1.520
EXCHANGE
IMAGENTIME
0.129
0.067
1.112
OURS
0.030
0.027
0.083
KoVAE
0.459
0.081
1.206
ECG
IMAGENTIME
0.400
0.079
1.223
OURS
0.345
0.076
0.979
We can see that compared to the SOTA time series generative models, our method still shows salient improvements in discriminative score and predictive score, illustrating the capability of generating high-fidelity and useful synthetic time series samples.
C.2. Forecasting
The length of the lock-back window is 96, and the target time series length is set L E {96, 192, 336, 720). Such a setting is aligned with (Wang et al., 2024). Table recorded the information of the forecasting datasets, Electricity?, ETT
'nttps: http:://wwwwwwglleee/// haangg/ee.co/atttsttseeeeeeeee htt.a//www.oaaaaaeee///////attta
hittps://archieee i ics. usi http:///haaaaaaeeeeeeeeeettttaaa
1201
hhttps://githut thub. com/ http///ooooooooeeo http:/hoyyy///tttaaeeee
16

---

A Non-isotropic Time Series Diffusion Model with Moving Average Transitions
Table Forecasting datasets
Dataset
Resolution
Time steps
Description
Electricity hour
26304
Electricity consumptions
ETTh
hour
17420
Oil temperature of power transformers
ETTm 15 min
69680
Oil temperature of power transformers
Exchange
1 day
7588
Panel data of exchange rates
Traffic
hour 17544
Traffic loads
Weather 10 min
52695
Meteorological indicators
Exchange", Traffic² and Weather
We set the batch size as 64, the learning rate as 2 x 10-4, 4 the training epoch as 100 with early stopping, and the diffusion step T = 100. To include the condition input, the condition encoder and decoder are all Multi-Layer Perceptrons (MLP). The hyperparameters of hybrid optimization are chosen X₂ Ap X The specific network architectures can be referred as our source code.
The diffusion models inferred for 100 times to calculate the metrics. For MSE, averaged over 100 times to have deterministic forecasts, while for CRPS, we first calculated nine quantiles at (0.1, 0.2, ... 0.9), and then approximated
CRPS.
For each target length, benchmarks and our proposed model are trained with 5 different random initialization seeds, and the full results on each target length are reported in Table 7.
Non-diffusion time series forecasting benchmarks. Although our paper focuses on how improve the time series diffusion model, also believe that it's necessary to include SOTA non-diffusion time series forecasting methods as reference. Therefore, we included Autoformer (Wu et al., 2021), Non-stationary Transformer (Liu et al., 2022), and PatchTST (Nie et al., 2023) to compare deterministic forecasting performance. We run all models in the same setting mentioned above, i.e., L = (96, 192, 336, 720), and the averaged MSEs over all reported in Table 8.
Regarding overall performance, our model still ranks first among these benchmarks, though is slightly inferior to ETTh2 and weather compared PatchTST.
It should be noted that these SOTA architectures are particularly tailored for time series forecasting and well adapted to the benchmark datasets, while forecasting is one of the downstream applications of our proposed MA-TSD framework. Therefore, we think there exists great potential to accommodate the SOTA architectures into our MA-TSD framework to have a better forecasting performance in our future work.
C.3. Super-resolution
For time series super-resolution, we set the length of the time series as L = 576, and also train with the batch size as 64, the learning rate as 2 x 10-1, the training epoch as 100 with early stopping, the diffusion step T = 100.
The information the datasets, MFRED (Meinrenken et al., 2020), Wind" and Solar are listed Table 9.
Table Super-resolution datasets
Dataset Resolution
Time steps
Description
MFRED 5 min
25908
Household electricity load in Manhattan
Wind Solar
5 min
26496
Power generation from Australian wind farms Power generation from Australian PV panels
5 min 52992
thttps://github.co cwtp ftcttp//pems.deee
series-data
hhttps://ww.bbbc--een. mpg. de//wtteeeee/eeeeeeeeeeeee
htttps////eendd..org/rroooodd 8554888
hatpe://renndd.or//eerodd//
17

---

A Non-isotropic Time Series Diffusion Model with Moving Average Transitions
clectricity
etth2
-1.3
-1.11
50
100
130
200
190
exchange_ree rate
traffic
weather
2.78
roal
1475P1
Toal
Test
S00EP
Ieal
Test
2000
0.1
-0.2
1.50
Pigure 8. Forecasting samples on L. 96
electricity
etth2
etlm2
roal
Exst
50S6PP
roal
Test
SUSAPI
real
Iist
50X2P
0.01
0.5
00-
-1.5
130
200
250
300
exchhnne_ratt
SUSGPI
0.77
0.50
0.25
0100
1.5
-1
-11.0
100
1.50
-300
250
Figure 9, Forecasting samples on L 192
cloctricity
ctth2
ctim2
real
fost
roal
feet
SUSSPP
7oal
905PPT
exchange rate
traffic
weather
1.73
tcal
Scst
D0
zcal
Test
9055P1
1.0
1.25
-122
0.75
0.50
100
100
200
300
430
Figure 10. Forecasting samples on L = 336
18

---

A Non-isotropic Time Series Diffusion Model with Moving Average Transitions
Table 7. Forecasting performance measured in MSE and CRPS.
Daraset
MA-TSD
medill
TMDM
SeSD
DAVAR
CSDI
MSF
CRPS
MSF
CRPS
MSF
CRPS
MSE
CRPS
MSF
CRPS
MSE
CRPS
0.225.0.004
0.976
0.309
0.810gomm
0.506
0-358
Electeiit
192
0.297 .0.0
10000
0.218
0.152 20.00
1,000 20
g.319
0.885
0,314
0.415go
356
0.347
177
2.496
0.224as.ass
0.125
0.192
1,05
0.52Ma
0
0.508
0.307
0.150aam
20:04
0.2755
0.478anirs
203g0 oos
1072
0.335
0.871
0.318
0.601.0002
0.228
95
0.136
0. 0.121
a.
0.154
0 20
g.14
0 0.5000
0. 4.271
1.77.0
0.4544
0.212.0
2.140
FTT12
109
2712gn
0.30
1665
6:236
2.
336
0.220 0.
0. 10540/383
00236 mn
0. 1.6
0:277 p0
0.170 1.170
0.708
0.351 aaitar
1. 1.522g0 tee
0.414grass
0.962 non
D. 168 nee
0.284,00017
0.191
0,289
0.278
0.330
95
0.0
1805..000
0.12
2000
0.080-0:004
0.204
0.719 a.r
1.00 at7
2.027
ETTm2
192
0.105
0m
0.
0.129an
0.132
0. 0.142
0.114
0.782
2.90440 0.3322.001s
11224
1. 1.762.0
0.136
0.04
1211,.002
L038 194
a.30a
a.67e a.400
2.751
720
0.185gamms
143
235
157 29.99
0 207
106 183
1.20 oss
0.457
3.212 can
0.585
2 2:369
0. 0.
0,202 vocs
B.196
1.82
0.674 tat
0.290
0.170l ness
Exchange
192
0.
15.000002
0.251 gnnn
0.170ee.003
0.600gue
0.257 20:33
3.672 201
0. 0.775
2 2.008g00000 sou
0.801
0.200
0.1 goa
336
0.3
0 20:
0 12002a
1.902a0
0.353
3,131
2 2.851 0.710
0.674gnass
0,00
0.902220.
0.868 1a0
1.163 sm0
0.358
g.226 14
2.301
0.587a
3,80 ans
95
0.166
11.2
0.269
0.1532
0 200
0.114
1993
0.453a
4 4.714
0.729
0 0777
0.145
Traffie
0.157
0.228 om1
0.136ge..e
1,900 000
7.60igo onn
0.918
0:276 ona
336
0.157 gass
0.137 20.907
0. 0.101 on
0.102
1.965
0.458
5.155
0.754
0.310
0.153
720
184
0,267
0.160go.0m
0. 1.009
0.065am0094
8.212
0,064atos
1322a
0.312an.es
go
1.447 uas
0.089
Weather
192
0.1 0.14
1.126
0. 0.15220
0.130:a.062
0. 161
0.120 1::20:
0.0001
1.901 2.300
0,508
0.138
0. 0.10910.002
2
2.161
0:221 unn
2.750
1600
0.457aness
0.204
1.33310m
720
0 0.36 o07
0.21 g0:
g.208ge:363
0 0.3
0. 0.183-2 384
0.639gmmm
0.35 g0:004
1.237
0.396
0.338gooz
D. 0.177ga0m
Table 8. Average MSEs over prediction lengths L {96, 192, 336, 720].
method
Electricity
ETTh2
ETTm2
exchange
traffic
weather
rank
Autoformer Nonstationary Transformer
0.594
0.218
0.168
0.601
0.267
0.293
3.83
0.367
0.230
0.146
0.440
0.229
0.278
2.83
PatchTST
0.412
0.202
0.122
0.500
0.179
0.189
1.83
MA-TSD
0.340
0.212
0.124
0.372
0.166
0.207
1.50
For MFRED and Solar, the original resolution is 10 seconds and I minute respectively, we resampled them to minutes for alignment.
We unify the denoising networks of both our proposed MA-TSD and the DDPM as the DiT(Peebles & Xie, 2023). For consistency, we calculated the MSE between the low-resolution inputs and the down-scaled super-resolution outputs. For Context-FID, we trained Autoencoders for each training dataset individually, obtained the time series embeddings of super-resolution outputs and the real high-resolution data, and calculated the Fréchet distance over these two embeddings.
The comparison of inference steps on each scale is shown in Figure 12. Here, though we conduct the experiments only on {3. 6, 12] scales, we can theocratically calculate the expected inference steps on other scales according to Equation (17).
Visualization of super-resolution can be found in Figure 13, Figure 14 and Figure 15.
19

---

A Non-isotropic Time Series Diffusion Model with Moving Average Transitions
electricity
etth2
ettm2
90RP1
90RPT
0.0
-10-
-0.-
-1.0
200
400
600
800
400
000
200
400
500
excebange rate
traflic
weather
200l
Jest
DOSRET
EOAL
Sost
DOPGE
2.0
0.00
Figure 11. Forecasting samples L 720
10
inference steps
10
Propoced DDPM
10
12
14
scale
Figure 12. Comparison of inference steps under different super-resolution scales.
20

---

A Non-isotropic Time Series Diffusion Model with Moving Average Transitions
3
6
12
Input
0-
me
DDPM
MA-TSD
0
mmw
Real
m
200
400
600
200
400
600
200
400
600
Figure Super-resolution on MFRED dataset
21

---

A Non-isotropic Time Series Diffusion Model with Moving Average Transitions
3
6
12
Input
2
DDPM
2
MA-TSD
Real
2
0-
200
400
600
200
400
600
200
400
600
Figure 14. Super-resolution on Wind dataset
22

---

A Non-isotropic Time Series Diffusion Model with Moving Average Transitions
3
6
12
Input
2-
3-
MA-TSD DDPM
3-
2-
Real
2
0-
200
400
600
200
400
600
200
400
600
Figure Super-resolution on Solar dataset
23


Reviews:
[{"review_id": "8L3QjnqvQz", "reviewer": "Reviewer_gWYs", "summary": "This paper presents a non-isotropic time series diffusion model (MA-TSD) for time series analysis. The key idea is to use a moving average in the forward process to better preserve low-frequency information, thereby avoiding gradient conflicts during training. The model also features an accelerable backward process, similar to DDIM, which can be viewed as time series super-resolution, and employs instance normalization to standardize the data. Experiments on real-world datasets (such as Electricity, ETTh2, and Traffic) show that MA-TSD outperforms existing diffusion-based models in both forecasting and super-resolution tasks.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your time and effort in reviewing, as well as your constructive feedback, which helps strengthen our work.  We would like to address your questions as follows.\n\n**1. Lack of non-diffusion benchmarks for comparison**\n\nWe agree that non-diffusion benchmarks are also necessary for general comparison, even if our work focuses on the improvement of time series diffusion models. We've added related experiments with Autoformer, Non-stationary transformer, and PatchTST for time series forecasting.\n\n\n|            | Electricity | ETTh2 | ETTm2 | Exchange | traffic | weather |     rank |\n| :--------- | ----------: | ----: | ----: | -------: | ------: | ------: | -------: |\n| Autoformer |       0.594 | 0.218 | 0.168 |    0.601 |   0.267 |   0.293 |     3.83 |\n| NSformer   |       0.367 | 0.230 | 0.146 |    0.440 |   0.229 |   0.278 |     2.83 |\n| PatchTST   |       0.412 | 0.202 | 0.122 |    0.500 |   0.179 |   0.189 |     1.83 |\n| **ours**   |       0.340 | 0.212 | 0.124 |    0.372 |   0.166 |   0.207 | **1.50** |\n\nRegarding overall performance, our model still ranks first among these benchmarks, though it is slightly inferior to ETTh2 and weather compared to PatchTST. \n\nIt should be noted that these SOTA architectures are particularly tailored for time series forecasting and well adapted to the benchmark datasets, while forecasting is one of the downstream applications of our proposed MA-TSD framework. Therefore, we think there exists great potential to accommodate the SOTA architectures into our MA-TSD framework to have a better forecasting performance in our future work.\n\n**2. Computational demands**\n\nWe'd like to analyze the computation burden of our method from two perspectives, i.e., spatially and temporally.\n\nSpatially: It should be noted again that the training process of our framework is almost identical to the standard diffusion, i.e., sample a batched time series window, degrade it, and use a denoising network to learn. The main difference lies in the forward (degrading) process. In our framework, we need to store a series of kernel matrices $\\boldsymbol{K}_t$ in advance for degrading at different diffusion time steps, which in practice is a tensor with shape $T \\times L \\times L$ (T=diffusion steps, L=sequence length). Thus, if we try to model an extremely long sequence, it could increase the storage demand in quadratic growth. Fortunately, $\\boldsymbol{K}_t$ is sparse because only unconvolved time steps remain zeros, and such sparsity decreases by t. For example, when $T=100, L=576$, the sparsity ($1 - oc / L^2$, $oc$ is the number of non-zero entries) at $t=20,40,60,80$ is [0.979 0.945 0.879 0.694]. Therefore, though we need to store a series of kernel matrices, they are mostly highly sparse, thus not imposing much storage burden.\n\nTemporally: The most time-consuming part of diffusion models is sampling, but we can utilize the accelerated backward process to speed up, especially in the time series super-resolution task. As we illustrated in the paper, we conduct SR naturally through our backward process without retraining the whole model like benchmarks that rely on conditional inputs. \n\nIn practice, we used only one NVIDIA 4090 24GB to conduct all the experiments. Even if we run a $L=720$ task from scratch on the largest dataset (ettm2), it only takes $\\textbf{0.72}$ GPU hours to train and sample, thus no significant computational demands in our method.\n\n**3. Dataset diversity**\n\nWe highly agree that dataset diversity is important for evaluation. For now, our used datasets have included Traffic (traffic system), Electricity/MFRED (power system), Exchange_rate (financial market), ETT (IoT sensors), Weather (meteorological system), and Solar/Wind (renewable energy). These are widely used benchmark datasets in the field of time series [1,2].\n\nBeyond these, we also added an ECG dataset (medical system) during the rebuttal period for a more complete evaluation. We launched a standard time series generation experiment on the ECG dataset and included two more benchmarks, i.e., KoVAE (RNN-based) and ImagenTime (Diffusion-based). The results are shown below.\n\n| Dataset  |   Model    | Disc. Score | Pred. Score | Context-FID |\n| :------: | :--------: | :---------: | :---------: | :---------: |\n| ECG |   KoVAE    |    0.459    |    0.081    |    1.206    |\n| | ImagenTime |    0.400    |    0.079    |    1.223    |\n| |  **ours**  |  **0.345**  |  **0.076**  |  **0.979**  |\n\nDisc. Score evaluates the fidelity by training a post-classifier to tell whether the data is real or not. Pred. Score evaluates the usefulness by training an RNN forecasting model with the generated data and testing it on the real [3]. Thus, we can see the superiority of our method on both fidelity and usefulness over benchmarks.\n\n[1] Deep Time Series Models: A Comprehensive Survey and Benchmark\n\n[2] A Survey on Diffusion Models for Time Series and Spatio-Temporal Data\n\n[3] TSGBench: Time Series Generation Benchmark", "claims_and_evidence": "The paper argues that using standard DDPM on time series causes gradient conflicts because low-frequency information decays too quickly, leading to inconsistent data perception across diffusion steps. It claims that incorporating a moving average in the forward process helps preserve low-frequency components, resulting in more stable training and higher-quality generation. These claims are backed by both theoretical analyses (such as gradient similarity matrices and spectral energy ratio comparisons) and experimental evidence on datasets like Electricity. However, the comparisons are mainly limited to diffusion models and do not include mainstream non-diffusion methods like Transformers or RNNs.", "methods_and_evaluation_criteria": "The method combines a moving average forward process, instance normalization, and an accelerable backward process to better handle time series frequencies and preserve low-frequency information. It is evaluated on forecasting and super-resolution tasks using metrics like MSE, CRPS, and Context-FID. Overall, these techniques and metrics are suitable for the problem, though the model is complex and might be computationally demanding for large datasets. (Note: My background in time series is limited, so I may not fully grasp all the details of these evaluation metrics.)", "theoretical_claims": "The paper presents clear and rigorous derivations for the forward and backward processes, noise scheduling, and the conditional joint distribution. These proofs are based on fundamental principles of probability and mathematics. Although the derivations can be dense for non-experts, no major errors were found.", "experimental_designs_or_analyses": "The experiments are well-designed. The authors tested their model on six real-world datasets with different time patterns, using metrics such as MSE, CRPS, and Context-FID, and they performed ablation studies to evaluate the key components of MA-TSD. The results clearly demonstrate the model’s advantages in training stability, low-frequency information retention, and generation efficiency. Including comparisons with non-diffusion models could further improve the evaluation.\nPlease note that my understanding of time series' research is limited, so some of my comments might not be completely accurate.", "relation_to_broader_scientific_literature": "The paper builds on established diffusion models like DDPM and DDIM, adapting them for time series by using a moving average to preserve low-frequency information and improve training stability. This approach extends ideas from image and video synthesis to sequential data tasks such as forecasting and super-resolution. Please note that I am not a time series expert, so my understanding of the broader literature in this area is limited.", "essential_references_not_discussed": "No.", "other_strengths_and_weaknesses": "Strengths:\n- The model demonstrates superior performance in both time series forecasting and super-resolution tasks.\n- The work combines rigorous theoretical analysis with comprehensive experimental validation, lending strong support to its claims.\n\nWeaknesses:\n- Despite the accelerated backward process, the overall model structure is complex, and the computational demands may be high when processing large-scale time series data.\n- The experiments are primarily conducted on specific datasets, so the model’s performance on other types or domains of time series data remains to be further verified.\n\nAgain, I am not an expert in time series.", "other_comments_or_suggestions": "Nothing for this part.", "questions_for_authors": "No, just reply the weakness.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Ed8ppXYBnL", "reviewer": "Reviewer_mCdS", "summary": "When training a standard diffusion model on time series dataset, (Contribution 1) the authors identified that gradients conflict between small t and large t values, which hinders training. To address this issue, they propose (Contribution 2) a heuristic solution by adding \"moving average\" as an additional corruption to the forward equation, which is widely used in time series data. Additionally, since mean and variance play important roles in time series data, they trained the model for post-denormalization from z to x separately. They also propose an accelerated sampling method similar to DDIM.\n\nExperiments were conducted on six time series datasets, (Contribution 3) showing the highest performance compared to baseline methods in forecasting and super-resolution tasks. Ablation studies demonstrate that the proposed methods significantly contributed to these performance improvements.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We greatly appreciate your acknowledgement of our work. We would like to address your questions as follows.\n\n**Q1: The difference between Blurring Diffusion Model (BDM) and ours**\n\nFrom a high-level perspective, BDM and ours shared a similar idea, i.e. building the degradation process with low-pass filters (blurring in BDM, MA in ours). However, there exists clear distinctions. \n\n1. **Filtering space**: Though we think it's minor due to convolution theoram, we still point out at first to pave the way for the following parts. Specifically, we filtered the data in the time space by convolution (matrix multiplication),$q(\\mathbf{x}_t | \\mathbf{x}_0) = \\mathcal{N} (\\mathbf{K}_t\\mathbf{x}_0,\\beta_t^2 \\mathbf{I})$ while BDM blurs the images in the frequency domain and transform back to the pixel domain, i.e. $q(\\mathbf{x}_t|\\mathbf{x}_0) = \\mathcal{N} ( \\mathbf{V} \\boldsymbol{\\alpha}_t \\mathbf{V}^\\top \\mathbf{x}_0, \\sigma_t^2 \\mathbf{I})$, where $\\mathbf{V}^\\top,\\mathbf{V}$ are  DCT and IDCT, and $\\boldsymbol{\\alpha}_t$ is the frequency response of Gaussian blurring kernel, a diagnoal matrix, whose each entry $\\alpha_t^i \\in (0, 1]$ is a coefficient of i th frequency component. For low pass filters, $\\alpha_t^i$ decreases by i until (nearly) zero to suppress high frequency components.\n\n2. **Markovian or not (Major)**: Since $\\boldsymbol{\\alpha_t}$ is diagnoal, BDM proposed that for each frequency component, a standard Markovian DDPM can be constructed. The one-step transition is accordingly defined, i.e. $q(u_t | u_s) = \\mathcal{N}(\\alpha_{t|s}u_s, \\sigma_{t | s}^2)$, where s=t-1, $u_t=V^\\top x_0$ is the frequency representation and $\\boldsymbol{\\alpha}_{t | s}=\\boldsymbol{\\alpha}_t / \\boldsymbol{\\alpha}_s$. However, dividing  $\\boldsymbol{\\alpha}_s$ could be problematic in practice, because $\\alpha_t^i$ could become to be (nearly) zero for large i, so dividing $\\boldsymbol{\\alpha}_s$ is numerically unstable for all diffusion steps. Therefore, though BDM claimed to have Markov transition under the DDPM framework, we think it's improper to define $q(x_t | x_s)$ when the transition operation from $x_0$ to $x_t$ is non-invertible. In our framework, faced with similar invertible MA, we bypassed the definition of $q(x_t|x_s)$, assumed $q(x_1:T | x_0)$ non-Markovian (in the DDIM-style) and then delicately defined $q(x_s|x_t, x_0)$ to satisfy $q(x_t|x_0)$ for all t. Thus, whether it's Markovian or not is another distinct difference from ours and BDM. \n\n3. **Noise schedule**: BDM designed $\\boldsymbol{\\alpha}_t = a_t \\mathbf{d}_t$, where $\\mathbf{d}_t$ is the frequency response of blurring kernel and $a_t \\in [0,1]$ is an extra scaler decreasing by t, and the noise schedule $\\sigma_t = 1-a_t^2$. In our framework, the noise schedule $\\beta_t$ is dataset-based, chosen regarding the variance decrease caused by MA (Eq.13) on different datasets. To compare these two, we mimic BDM's noise schedule, i.e. multiplying an $a_t$ to $\\mathbf{K}_t$ and set $\\beta_t = 1-a_t^2$, and ran the experiments on forecasting task (L=96). We can see that test errors indeed increased, which might be because of BDM's $\\sigma_t$ schedule ignored the extra effects of blurring ($\\mathbf{d}_t$) and only consider the $a_t$.\n\n|   Dataset   |  Model   |  MSE  | CRPS  |\n| :---------: | :------: | :---: | :---: |\n| electricity |  ours  | 0.288 | 0.158 |\n| | ours+a | 0.303 | 0.158 |\n|    ETTh2    |  ours  | 0.136 | 0.121 |\n| | ours+a | 0.166 | 0.130 |\n|  Exchange   |  ours  | 0.098 | 0.110 |\n| | ours+a | 0.193 | 0.136 |\n\nIn summary, despite the similar high-level idea, there exist clear differences between BDM and ours. BDM tried to fit in the standard DDPM framework, while we reformulated a framework with special adaptation on moving average filters and time series. Unfortunately, BDM didn't release their official codes, and we are unable to reproduce it on time series data for more quantitative comparison.\n\n**Q2: Moving average aligns gradient signals**\n\nBy intuition, a mild data degradation in diffusion should benefit the learning of denoising networks, since the inputs could be mostly noise if degradation is too fast. For example T=500 in Fig.2, the gradients at t> ~190 (right down block) are similar because x_t are almost noise and the model can hardly learn anything. For t < ~190 (left up block), they are more informative and contribute to denoising learning more. Thus, we'd like to \"prolong\" the left up block to help the learning. \n\nMeanwhile, to reconstruct a time series, low-frequency (LF) information is usually more helpful. Thus, we're motivated to use MA, a widely used TS low-pass filter, to design a mild degradation where LF is kept more and serves as common information for all diffusion steps. In this way, the inputs for denoising models are more structurally similar so that gradients conflicts at different steps could be less.\n\n**Others**: Forecasting examples are in https://anonymous.4open.science/r/ICML2025_rebuttal-134C/forecast_plot.png", "claims_and_evidence": "There are two claims in this paper:\n\n(1) When training diffusion models on time series data, gradient conflicts occur between small timesteps and large timesteps. \n(2) This issue can be mitigated through moving average data degradation.\n\nAnd Claims 1 and 2 are supported phenomenologically through Figure 2. \n\nHowever, the explanation of why MA-TSD prevents gradient conflict, i.e., Claim 2, still seems insufficient. Regarding the methodology of corruption through moving average, it would be much better if they could provide a more developed theoretical or intuitive motivation through toy examples beyond \"time series data has more important low-frequency signals compared to other data.\"\n\n(Minor Q) To my knowledge, gradient conflict phenomena and instability issues based on timestep regimes also occur in image datasets [1]. However, blurring diffusion models [2], which is similar method from this paper, considered do not effective as demonstrated in this paper. Could the reasons be: (1) moving average is more effective in time series data? (2) learning in latent space through VAE solves this problem? I'm curious about the inconsistency where similar methods weren't effective for images but achieved significant performance improvements in time series data. (3) Or is it because the noise schedule is determined by Eq. (13)? I'm curious about your opinion or intuition. Experimental evidence is plus. \n\n[1] Truncated Consistency Models, https://arxiv.org/abs/2410.14895, ICLR25\n\n[2] Blurring Diffusion Models, https://arxiv.org/abs/2209.05557", "methods_and_evaluation_criteria": "MA-TSD has methodological similarities to blurring diffusion models, but has (1) a distinct motivation regarding \"gradient conflict\" and (2) shows notable performance improvements in time series data, thus having distinguishable contributions.\n\nEvaluation was conducted on six datasets using MSE and CRPS for time series forecasting, and Consistency and Context-FID for super-resolution evaluation. Performance improvements were shown across almost all datasets and metrics. Though I'm not an expert in this field, they conducted sufficiently extensive evaluations and demonstrated impressive performance improvements.", "theoretical_claims": "There are no theoretical claims in this paper that require verification.", "experimental_designs_or_analyses": "There are no special issues with the experimental settings in this paper.\n\nA notable point is that Table 4's ablation study accurately shows the performance contribution size of moving average and instance normalization. I personally find the performance contribution of MA impressive.", "relation_to_broader_scientific_literature": "Research that achieves performance improvements through changes to the forward equation has actually become an almost abandoned research direction in images, so it's impressive that it has been rediscovered in time series data. This shows that even if it wasn't effective for images, there is room for more research on forward equations for video or audio that have time series characteristics. Some might evaluate this research's novelty as low, but I'd like to assign high novelty in terms of rediscovery.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "I think the writing and the presentation is quite clear. \n\nAs I've mentioned several times, the greatest value of this paper is showing that a method I thought had been abandoned can still be valid in other domains.", "other_comments_or_suggestions": "I hope to see qualitative results from various methods, M3VAE, TMDM, MR-DIFF, to understand the qualitative difference between various methods.", "questions_for_authors": "Q1: What exactly is the difference from Blurring Diffusion Models [1]? I'm curious about the points where the contribution is distinguished. If I can clearly understand where the contribution is distinct, I would like to raise my score to accept.\n\nQ2: The fact that moving average aligns gradient signals by timestep seems like a very surprising discovery to me. Do you have any theory or intuition that can explain this? The reason that \"time series data often have more informative low-frequency component\" doesn't seem sufficient to me.\n\n[1] Blurring Diffusion Models, https://arxiv.org/abs/2209.05557", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {"ethical_review_flag": "Flag this paper for an ethics review."}}, {"review_id": "kDwdX4Ki97", "reviewer": "Reviewer_EJkG", "summary": "This paper proposes MA-TSD (Moving Average Time Series Diffusion), a novel time series diffusion model that replaces the standard isotropic diffusion process with a moving average transition. The key motivation is that existing isotropic diffusion models degrade low and high-frequency components identically, which is problematic for time series data where low-frequency components often carry more essential information.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the valuable feedback and the opportunity to strengthen our experiments. Below, we address each of the weaknesses:\n\n**1. Standard Time Series Generation**\n\nWe agreed that the standard generation task is necessary for evaluating our model. Thus, we follow the setting of your mentioned benchmarks and experiment on three datasets, ETTh2, Exchange, and ECG (newly added, medical time series). We also include discriminative score and predictive score as additional metrics for evaluating the fidelity and usefulness, as the following table shows:\n\n| Dataset  |   Model    | Disc. Score | Pred. Score | Context-FID |\n| :------: | :--------: | :---------: | :---------: | :---------: |\n|  ETTh2   |   KoVAE    |    0.069    |    0.034    |    0.258    |\n|    | ImagenTime |    0.053    |    0.054    |    0.118    |\n|    |  **ours**  |  **0.044**  |  **0.026**  |  **0.075**  |\n| Exchange |   KoVAE    |    0.137    |    0.038    |    1.520    |\n| | ImagenTime |    0.129    |    0.067    |    1.112    |\n| |  **ours**  |  **0.030**  |  **0.027**  |  **0.083**  |\n| ECG |   KoVAE    |    0.459    |    0.081    |    1.206    |\n| | ImagenTime |    0.400    |    0.079    |    1.223    |\n| |  **ours**  |  **0.345**  |  **0.076**  |  **0.979**  |\n\n\nWe can see that compared to the SOTA time series generation models, our method still shows salient improvements in discriminative score and predictive score, illustrating the capability of generating high-fidelity and useful synthetic time series samples.\n\n**2. Super-resolution(SR)**\n\nWe included flow matching with Variance Preserving path (FM-VP) as our additional SR benchmark. The backbone used in FM-VP is kept the same as our original benchmark and model. The following table records the comparison:\n\nTable.1 Consistency error comparison\n| Scale | Model |   MFRED   |   Wind    |   Solar   |\n| :---: | :---: | :-------: | :-------: | :-------: |\n|   3   | ours  | **0.003** | **0.007** | **0.011** |\n|   3   | FM-VP |   0.033   |   0.064   |   0.052   |\n|   6   | ours  | **0.004** | **0.010** | **0.012** |\n|   6   | FM-VP |   0.024   |   0.050   |   0.036   |\n|  12   | ours  | **0.005** | **0.014** | **0.013** |\n|  12   | FM-VP |   0.016   |   0.035   |   0.024   |\n\nTable.2 Context-FID comparison\n| Scale | Model |   MFRED   |   Wind    |   Solar   |\n| :---: | :---: | :-------: | :-------: | :-------: |\n|   3   | ours  | **0.105** | **0.286** | **0.349** |\n|   3   | FM-VP |   1.348   |   4.231   |  0.795   |\n|   6   | ours  | **0.124** | **1.024** |   0.697   |\n|   6   | FM-VP |   1.338   |   4.468   | **0.685** |\n|  12   | ours  | **0.436** | **3.057** |   1.413   |\n|  12   | FM-VP |   1.522   |   4.874   | **0.862** |\n \nDespite the slight inferiority in Context-FID on the Solar dataset when the SR scale goes up, our proposed method still outperformed the FM-VP model generally, especially in terms of consistency. It should be noted again that we perform SR naturally through our backward process instead of retraining the whole conditional model like benchmarks. Therefore, we think our method provides a trade-off of (1) training overheads, (2) SR quality, and (3) consistency to the low-resolution input.\n\n**3. Forecasting**\n\nAlthough our paper focuses on how to improve the time series diffusion model, we also agree that it's necessary to include SOTA time series forecasting methods as a reference. Therefore, we included Autoformer[1], Non-stationary Transformer (NSformer)[2], and PatchTST[3] to compare deterministic forecasting performance. We run all models in the same setting in our paper, i.e., L  = {96, 192, 336, 720}.\n\n|            | Electricity | ETTh2 | ETTm2 | Exchange | traffic | weather |     rank |\n| :--------- | ----------: | ----: | ----: | -------: | ------: | ------: | -------: |\n| Autoformer |       0.594 | 0.218 | 0.168 |    0.601 |   0.267 |   0.293 |     3.83 |\n| NSformer   |       0.367 | 0.230 | 0.146 |    0.440 |   0.229 |   0.278 |     2.83 |\n| PatchTST   |       0.412 | 0.202 | 0.122 |    0.500 |   0.179 |   0.189 |     1.83 |\n| **ours**   |       0.340 | 0.212 | 0.124 |    0.372 |   0.166 |   0.207 | **1.50** |\n\nRegarding overall performance, our model still ranks first among these benchmarks, though it is slightly inferior to ETTh2 and weather compared to PatchTST. \n\nIt should be noted that these SOTA architectures are particularly tailored for time series forecasting and well adapted to the benchmark datasets, while forecasting is one of the downstream applications of our proposed MA-TSD framework. Therefore, we think there exists great potential to accommodate the SOTA architectures into our MA-TSD framework to have a better forecasting performance in our future work.\n\n\n[1] Autoformer: Decomposition Transformers with Auto-Correlation for Long-Term Series Forecasting\n\n[2] Non-stationary Transformers: Exploring the Stationarity in Time Series Forecasting\n\n[3] A Time Series is Worth 64 Words: Long-term Forecasting with Transformers", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "Yes, but a generative evaluation is missing, as described in the weaknesses response.", "theoretical_claims": "No", "experimental_designs_or_analyses": "No", "relation_to_broader_scientific_literature": "Time series generative models", "essential_references_not_discussed": "Yes, the paper is about a generative model, yet it does not address time series generation literature.", "other_strengths_and_weaknesses": "**Strengths:**  \n- The use of different diffusion processes for different frequency levels is a novel and insightful approach.  \n- The gradient analysis provides valuable insights and strengthens the motivation behind the method.  \n\n**Weaknesses:**  \n- Despite proposing a new generative approach, the paper does not evaluate it on standard generative modeling benchmarks [1, 2, 3].  \n- In the super-resolution section, the improvement is demonstrated against only one diffusion model, limiting the assessment of its adaptability to other diffusion frameworks like EDM [4], FM [5], or alternative architectures.  \n- The forecasting benchmark lacks comparisons with state-of-the-art forecasting models. While I acknowledge the distinction between deterministic and probabilistic forecasting, a direct comparison remains essential.\n\n[1] Interpretable Diffusion for General Time Series Generation\n\n[2] Utilizing Image Transforms and Diffusion Models for\nGenerative Modeling of Short and Long Time Series\n\n[3] Generative Modeling of Regular and Irregular Time Series Data via Koopman VAEs\n\n[4] Elucidating the Design Space of Diffusion-Based Generative Models (EDM)\n\n[5] Flow Matching for Generative Modeling", "other_comments_or_suggestions": "No", "questions_for_authors": "See weaknesses", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "DkqKvbInUM", "reviewer": "Reviewer_7aos", "summary": "This paper proposes a non-isotropic time series diffusion model with moving average transitions (MA-TSD). First, the authors empirically found that, when directly applying DDPM to time series data, the directions of model gradients at different diffusion steps are contradicted during training, which leads to unstable optimization. Second, they propose the moving average transition, which functions like a low-pass filter and keeps the informative low-frequency components. Meanwhile, they introduce the Non-isotropic forward process and the corresponding loss function, as well as the acceleration strategy (like DDIM) in inference. Extensive experiments are conducted, which demonstrate the salient performances of MA-TSD over existing DDPM-based diffusion models on time series-related tasks.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are grateful for your valuable suggestions and insightful comments. We'd like to reply as follows:\n\n**1. Uniform sampling strategy**\n\nWe've added the default DDIM sampling strategy to our experiments. Specifically, for a given sampling budget, we start with t=T and uniformly select the sampling steps. The results can be seen at (https://anonymous.4open.science/r/ICML2025_rebuttal-134C/sample_strategy.png). We can see that uniform sampling did offer a better result than random sampling, but the marginal gain on the performance decreases by $\\tau/T$. Compared to both random and uniform strategies, our factor-only strategy also shows superior performance.\n\n\n**2. Ablation on $L$**\n\nWe did an ablation study on the sequence length $L={48, 96, 192}$ and recorded the gradient similarity in this link (https://anonymous.4open.science/r/ICML2025_rebuttal-134C/ablation_L.png). We observe that for all L, gradient conflicts still exist when directly applying DDPM on time series data, and our method can keep alleviating. It should be noted that the severity of such conflicts is not changed with L varying. Instead, we think the gradient conflicts mainly come from the degradation design, and in our framework, MA always keeps more low-frequency information across different diffusion steps. These low-frequency components could serve as common information during diffusion training. Thus, the inputs for denoising models are more structurally similar, and consequently, the gradients across different steps are similar.\n\n\n**3. Backbone architecture**\n\nFor the denoising backbone, we mainly rely on the DiT, with an adaptation to 1D time series. For the condition encoder in the forecasting task, we adopt an MLP with RevIN[1], which is widely used in TS forecasting. In our paper, we don't focus much on the neural architecture design, and we believe that the backbones can be replaced with other reasonable networks. Combining the SOTA time series architecture with our framework could also be one of our future works.\n\n**4. Applicable to video generation**\n\nWe appreciate your insightful comments for inspiring us to explore more of our framework. Though we are not experts in the field of video generation, we still try to evaluate our framework on a simple \"video-like\" time series. \n\nSpecifically, we believe that the transition between frames in a video should be smooth and mild because even the most common frame rate (like 24 FPS) is higher than the sampling rate of time series data. Thus, we manually create a pure 2-D signal: $x(t)=\\sqrt{0.5 t} + 0.7 \\cos(t) + 0.3 \\sin(4 t), y(t)=\\sqrt{0.2 t} + 0.8 \\cos(t) + 0.2 \\sin(3 t)$ with $t\\in[0, 4\\pi]$ recorded with 1280 time steps (mimicking a 53s video in 24FPS), and concatenate them as a bivariate time series. Then, we slide temporal windows with $L={48, 96, 192}$ and record the gradient similarity of directly using DDPM and our framework. As shown in (https://anonymous.4open.science/r/ICML2025_rebuttal-134C/simple_syn.png), we found a similar phenomenon, and our framework also works to ease the gradient conflicts to some extent. However, we think more framework adaptation is needed to bridge the large modality gap for video generation.\n\nIn summary, our framework shows similar effects on our synthetic \"video-like\" time series, and we reckon that our high-level idea could also be applied to video generation with some necessary adaptation. We sincerely hope that our framework could also be enlightening and supportive for other fields beyond time series.\n\n\n[1] Reversible Instance Normalization for Accurate TSF Against Distribution Shift", "claims_and_evidence": "Yes, they are.\n\n1. **Claim:** When directly applying DDPM to time series data, the directions of model gradients at different diffusion steps are contradicted during training. **Evidence:** Supported by Figure 2\n2. **Claim:** The moving average schedule and the instance normalization are useful.  **Evidence:** Tables 1,2,4.\n3. **Claim:** The backward process of MA-TSD can be naturally considered as time series super-resolution. **Evidence:** Table 3", "methods_and_evaluation_criteria": "Yes. They make sense.", "theoretical_claims": "This paper does not have any theoretical claims.", "experimental_designs_or_analyses": "- In Figure 4, the factor-only accelerating strategy is only compared against randomly select subset strategy. To better validate its effectiveness, the authors should further include a comparison with uniformly sample strategy (e.g., the default DDIM approach, which uses uniformly spaced timesteps for acceleration).\n\nother experimental designs and analyses are sound.", "relation_to_broader_scientific_literature": "The empirical findings that directly applying DDPM on time series data will cause contradicted gradients might be helpful for the long video generation task.", "essential_references_not_discussed": "No", "other_strengths_and_weaknesses": "Strength:\n\n- The empirical findings that direct application of standard diffusion to time series may cause gradient contradiction during training (due to the\n  rapid decrease of low-frequency information in the diffusion process) is valuable，​and is insightful for the video generation community.\n- The proposed ​moving average schedule and ​instance normalization demonstrate effectiveness.\n- The ​factor-only acceleration strategy is effective.\n- The paper provides ​thorough experimental validation and ablation studies.\n\nWeakness\n\n- The paper should include an ​ablation study on the parameter L， and analyze how varying L impacts the severity of gradient contradiction when directly applying DDPM to time series.\n- The paper does not specify the ​model architecture (e.g., is it a 1D U-Net?)", "other_comments_or_suggestions": "A suggestion: The authors are suppose to ​cite relevant references in the tables (e.g., in table 1, table 2) to allow readers to easily trace the sources of compared methods.", "questions_for_authors": "Beyond time-series forecasting, how applicable is this method to other scenarios, such as video generation/video prediction (treating frames as sequences)? \n\nAccording to Figure 2, existing video diffusion models may also suffer from gradient contradiction. However, considering that existing video diffusion models typically predict shorter video clips, to what extent is this issue critically impactful? For example, How does the similarity maps in Figure 2 change when the $L$ varies?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper receives two weak accept and two accept. It proposes to use moving average to maintain more informative low-frequency components which are important for time series. Even though all the reviewers consider acceptance, I encourage the authors to update the final version according to the review suggestions.
Decision: Accept (poster)