Non-stationary Diffusion For Probabilistic Time Series Forecasting
Authors: Weiwei Ye
Zhuopeng Xu
Ning Gui
Venue: ICML 2025 spotlightposter (2025)
Keywords: Probabilistic Time Series Forecasting
Denoising Diffusion Process Model
Non-stationary Forecasting
Primary area: deep_learning->sequential_models_time_series

Abstract: Due to the dynamics of underlying physics and external influences, the uncertainty of time series often varies over time. However, existing Denoising Diffusion Probabilistic Models (DDPMs) often fail to capture this non-stationary nature, constrained by their constant variance assumption from the additive noise model (ANM). In this paper, we innovatively utilize the Location-Scale Noise Model (LSNM) to relax the fixed uncertainty assumption of ANM. A diffusion-based probabilistic forecasting framework, termed Non-stationary Diffusion (NsDiff), is designed based on LSNM that is capable of modeling the changing pattern of uncertainty. Specifically, NsDiff combines a denoising diffusion-based conditional generative model with a pre-trained conditional mean and variance estimator, enabling adaptive endpoint distribution modeling. Furthermore, we propose an uncertainty-aware noise schedule, which dynamically adjusts the noise levels to accurately reflect the data uncertainty at each step and integrates the time-varying variances into the diffusion process. Extensive experiments conducted on nine real-world and synthetic datasets demonstrate the superior performance of NsDiff compared to existing approaches. Code is available at https://github.com/wwy155/NsDiff.

Non-stationary Diffusion For Probabilistic Time Series Forecasting
Weiwei Ye Zhuopeng Xu
Ning Gui'
Abstract
model conditional expectations (Zhou al., 2021; Wu al., 2020), effective
Due to the dynamics of underlying physics and external influences, the uncertainty of time se- ries varies over time. However, existing De- noising Diffusion Probabilistic Models (DDPMs) fail to capture this non-stationary nature, con- strained by their constant variance assumption from the additive noise model (ANM). In this pa- per, innovatively utilize the Location-Scale Noise Model (LSNMM) to relax the fixed uncer- tainty assumption of ANM. A diffusion-based probabilistic forecasting framework, termed Non- stationary Diffusion (NsDiff), is designed based on LSNM that capable of modeling the chang- ing pattern of uncertainty. Specifically, NsDiff combines denoising diffusion-baseee conditional generative model with conditional mean and variance estimator, enabling adaptive endpoint dis- tribution modeling. Furthermore, we propose an uncertainty-awar noise schedule, which dynami- cally adjusts the noise levels accurately reflect the data uncertainty each step and integrates the time-varying variances into the diffusion pro- cess. Extensive experiments conducted nine real-world and synthetic datasets demonstrate the superior performance of NsDiff compared to ex- isting approaches. Code is available at https: / //github. www/wwywllwwcowo/o/oo httm//ww155/// //NDifff
particularly in high-stakes fields like healthcare (Bertozzi et al., 2020) and finance (Li & Bastos, 2020), often requires accurately estimating the uncertainty underlying the data (Kendall & Gal, 2017; Xu et al., 2024). To address this problem, many recent studies have focused on probabílistic time series forecasting (Rasul et al., 2021: Chen et al., 2024: Li et al., 2024b), where the goal estimate distribution of possible future outcomes along with their associated uncertainties.
Endpoint Distribution
Prediction Interval
1)
Ground Truth
Denoise
.NOXX), II
Ground Truth
Denoise
.V100)900)))
Denoise
Figure I. A figure illustrates DDPMs with different endpoints trained to estimate the number of influenza-lic discase patients weekly. We plot the endpoint distributions and prediction inter- vals (0, I) (Top). (f))) (Middle), and ((f(((((X (()) (Bottom) on the left and right, respectively. The red dashed line indicates the division the training and test dataset.
1. Introduction
Time series forecasting plays a key role in various fields such as traffic prediction (Ermagun & Levinson, 2018) and supply chain management (Chopra Meindl, 2021). Given historical multivariate series X, general forecasting meth- ods involve training an f(X) to predict a future series Y, which can be viewed modeling the E|Y|X). []] Although recent research has demonstrated promising capabilities to
The Denoising Diffusion Probabilistic Models (DDPMs) have recently gained significant attention for probabilistic time series forecasting due to their powerful ability to gener- ate high-dimensional data (Rasul et al., 2021; Tashiro et al., 2021: Li et al., 2024a). Existing DDPMs typically rely on the Additive Noise Model (ANM) (Spirtes et al., 2001), which assumes Y f(X) c, where ~ (N a)) rep- resents stationary Gaussian noise. The primary objective of these models is not only to estimate the conditional ex- pectation Y X via fX), but also to accurately capture uncertainty by modeling the noise distribution e. While DDPMs with stationary Gaussian noise have achieved sub-
Equual contribution 'School of Computer Science and Engineer- ing, Central South University, Changsha, China. Correspondence to: Ning Gui <ninggui gmail.com>.
Proceedings of the nd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Non-stationary Diffusion For Probabilistic Time Series Forecasting
stantial success in domains such as computer vision and natural language generation (Ho et al., 2020; Dhariwal & Nichol, 2021; Gu et al., 2022), they are less effective for modeling non-stationary time series data, where patterns of uncertainty vary contextually (Lee et al., 2024).
2. Related Works
2.1. DDPM for Probabilistic Forecasting
Denoising Diffusion Probabilistic Models (DDPMs) have shown promising results in the probabilistic forecasting area (Tyralis & Papacharalampous, 2022). Rasul al. (2021) introduce TimeGrad, an autoregressive diffusion model guided by a recurrent neural network hidden state. Tashiro et al. (2021) propose masking strategy for training diffusion models, applicable to tasks like imputation and forecasting. Alcaraz & Strodthoff (2022) extend DDPMs with a structured space model to capture long-term depen- dencies. Time (Shen & Kwok, 2023) utilized future mixup and autoregressive initialization. Li et al. (2022b) integrate multiscale denoising score matching to guide the diffusion process, ensuring generated series align with the target. DiffusionTS (Yuan & Qiao, 2024) trains the model to reconstruct the sample rather than noise, using a Fourier- based loss term. Kollovieh et al. (2024) propose a self- guiding strategy for time series generation and forecast- ing based on structured state-space models. By leveraging the bridge-based based model introduced by Shi et al. (2023), Chen et al. (2023) present a convergence analysis of the Schrödinger bridge algorithm and propose improvements to the diffusion process. However, these methods generally assume fixed endpoint variance, which is hard to model non-stationary time series.
Figure 1 illustrates an example from the ILI (influenza- like illness) dataset, with different endpoint distributions (Left) and estimated uncertainty (Right) on different models: TimeGrad (Rasul et al., 2021), TMDM (Li et al., 2024b), and NsDiff (ours). In the upper part of Figure 1, TimeGrad (Ra- sul et al., 2021) employs the endpoint /o I), which fails to capture non-stationary characteristics. TMDM (Li et al., 2024a) uses N (X), I) as endpoint, representing changing averages. On the test dataset (shown to the right of the red dashed line), where both the number of patients and the cor- responding deviation increase, the performance differences are evident. TimeGrad fails to model both the underlying trends and deviations. In contrast, TMDM effectively cap- tures the trends through its f(X), but its stationary covari- ance I limits its ability to accurately estimate uncertainty, which is critical for the probabilistic time series forecasting.
To better address non-stationarity with changing uncertainty, introduce Location-Seaal Noise Model (LSNM) into DDPMs, which relaxes the traditional Additive Noise Model (ANM) by incorporating a contextually changing variance: = + Vg(()e, je, where x(X) is an X-dependent variance model and is a standard gaussian noise. LSNM is capable of modeling both the contextual mean through f(X) and the contextual uncertainty through v (X). In the special case where (X) = 1, this simplifies the standard ANM. Building upon this more flexible and expres- sive assumption, we propose the Non-stationary Diffusion Model (NsDiff) framework, which provides an uncertainty- aware noise schedule for diffusion process. In summary, our contributions are:
2.2. Non-stationary Time Series Forecasting
To address non-stationarity, Li et al. (2022a) employ a approach to predict data distributions, while Du al. (2021) propose an adaptive RNN for dis- tribution matching to mitigate non-stationary effects. Liu et al. (2022) introduce a non-stationary Transformer with de- stationary attention to account for non-stationary factors in self-attention. Wang et al. (2022) use global and local Koop- man operators to capture patterns at different scales, and Liu et al. (2024a) apply Koopman operators to components iden- tified via Fourier transforms. Other approaches decompose stationary and non-stationary parts, such as Ogasawara et al. (2010) with local normalization, and Passalis et al. (2019) with a learnable, instance-wise normalization. RevIN (Kim et al., 2021) addresses the distribution shift using reversible normalization, and recent works (Fan et al., 2023; Liu et al., 2024b) explore finer-grained trend modeling. Jiang et al. (2023) addresses non- -satstitoonaitt in chaotic systems by pre- serving invariant measures to stabilize dynamical systems over time without relying on domain-speciftt priors. Fourier transforms, closely linked with non- non-stationarity, are also been applied to tackle these issues (Fan et al., 2024; Ye et al., 2024). Despite these advances in time series forecasting, the non-stationary uncertainty in probabilistic forecasting remains largely unexplored.
We observe that the ANM inadequate for capturing the varying uncertainty and propose novel framework that integrates LSNM to allow for explict uncertainty modeling. This work is the first attempt to introduce LSNM into probabilistic time series forecasting.
To fundamentally elevate the noise modeling capabili- ties of DDPM, we seamlessly integrate time-varying variances into the core diffusion process through an uncertainty-awar noise schedule that dynamically adapts the noise variance at each step.
Experimental results indicate that NsDiff achieves su- perior performance in capturing uncertainty. Specifi- cally, in comparison to the second-bes recent baseline TMDM, NsDiff improves up to 66.3% on real-world datasets and 88.3% on synthetic datasets.
2

---

Non-stationary Diffusion For Probabilistic Time Series Forecasting
3. Preliminary
where na is the estimated noise by a denoising model, which optimizes the following objective:
3.1. Problem Formulation
Given a historical multivariate time series X E R where N is the historical window size and D denotes the number of feature dimensions. The probabilistic forecasting task is predict the distribution of the future multivari- ate time series Y = {p(y1), ((y)) p(y)))...,pp( ....,,(yy))) wy e R2), where M is the future window size. While previous works model the future series with ANM: Y = (X) + 6, we model it based on LSNM with a more generalized data model:
EY (x)-u(X))).--wwooa..ll 7o
(4)
Following this basic forward and reverse process, many diffusion- based methods improve the reverse process (Rasul et al., 2021; Shen & Kwok, 2023) or prior distribution (Li et al., 2024a) with the historical time series information. However, they fix the variance of the prior distribution and focus on the expectation matching. The prior setup and training of uncertainty are largely ignored.
Y= fo() + (X)c
(1)
4. Methodology
where the e ~~ ) is Gaussian noise. The fX) and (x) can be viewed as prior knowledge with pre-trained parameters & and v,, where the (X) is modeling the con- ditional expectation X and (x) is modeling the varying uncertainty. In this paper, we incorporate this two prior knowledge into the diffusion model to tackle the non- stationary challenge in probabilistic time series forecasting.
In this section, we introduce the proposed NsDiff, including the design of forward and reverse process distributions, as well as the training and inference procedures of NsDiff. Furthermore, we discuss two simplified versions of NsDiff. The outline of NsDiff is given in Figure 2.
4.1. Forward and Reverse Process
3.2. Denoising Diffusion Probabilistic Models
In previous diffusion-basee methods, the uncertainty prior was missing, and they tended to set the endpoint of diffusion
DDPMs (Ho et al., 2020) is a popular generative model to estimate the uncertainty for future time series. In the origi- nal DDPM, the future series distribution can be represented as Po ()) := Po(( (P odY 1.T, where Y YI...,, YT are la- tent variables. The joint distribution is defined as Markov chain pa( Yo.T :- Y(T)) IIL pa (Y,-i)xe (Y),, where the endpoint of diffusion is set to p( (YY)) := I). To gener- ate the distribution po Po o)) DDPM designs two processes: a forward process to gradually add noise and a reverse pro- cess to denoise. In the forward process, the future series Y, gradually diffuses to the given prior endpoint Yr without any trainable parameters.
Y to (0, I) or N( (X), I). To improve this, we use different noise model LSNM to form the endpoint:
p P(Yr)fo(#) go (z)) = (ffo
(5)
where (X) models the conditional expectation E|Y|X] which can be parameterized by any forecasting model, e.g., Dlinear (Zeng et al., 2023) or PatchTST (Nie et al., 2022). We follow previous works (Kim et al., 2021; Liu al.. 2024b) train the prior scale of uncertainty gy We use the input variance to predict the output variance.
The forward process incrementally modifies the noise at each step to approach the endpoint distribution. To seam- lessly integrate vare-ng variances into the diffusion process, we propose an uncertainty-awart aware noise schedule, and incorporate data variance into the forward process dis- tribution q(Y, Yr-1, (x(),, Ir (X), dY,). Specifically, given well-pretrained models for Pv. and a prior state Y,-1. we control the scaled variance to transition from the actual variance Ty, at the starting point to the endpoint gø (X). The resulting distribution is normally distributed as:
o(Y Yo) II Y, Y,-1)
(2)
where B E (0, 1) is diffusion schedule for controlling the endpoint Yr ~ I). This forward sampling can be simplified by q( ( YY) (Y Vā Yo, (1 a) in practice, where == 1 Be and 0r == I 0g.
The reverse process parameterizes pa (Y-II and com- pares it against forward process posteriors g(,,,//Y, Yo). DDPM has shown that matching these two posteriors is equivalent to estimating the added noise 17 in the forward process. Thus, the parameterization of po (Y-1 is:
N(Y:: Va,Y,
Y (1 (X))
(8.0 - a13ixx.))
(6)
0+-1
where the shared coefficient Br is a noise scaling con- stant. As the noise step t increases, the term 8.go (X)
Po Y.) = N(Y 1-: 1:: (Y,,), 1 ō (I)
(3)
3r nn))
grows and Q decreases. At = T, where a = 0, the variance converges to the assumed endpoint go (X.
(Y,,t) == vat
YY:-
VI α
3

---

Non-stationary Diffusion For Probabilistic Time Series Forecasting
Training
Uncertainty-Aware E
0% Ground Truth Variance
Pretrained
( Endpoint Varlance
Forward Process
LSNM Endpoint
wwmmmnl
Yo
V
V2
Yt
Y
Yt
Reverse Process
Estimating QYv-1 Y, Yo, (x). gx)) using na. og ča (Y., (8), (X).)
Loss
noll
Σ
Σ
log
6 dg
FaP
noise match
uncertainty optimization
Inference
Parameter Estimation de
2. (-2, 2 12
M
3.
%
V. 11-(1-v (x))) (B. (x) x + T, A.M0 ro fx5 T
Predicted Distribution
Pa Y,, ) mb) Denoise (Sample)
Follow Step 1,2,34
Va
( PoCKs )2i=)
Sample
Yo
Y
Y
V
Y7
Figure The outline NsDiff. integrates LSSM--aased endpoint an uncertainty aware noise schedule. During training phase, a noise and variance estimator, Çe, is optimized to approximate the reverse process distribution. During inference, it samples from LSNM endpoint use estimated reverse distribution to iteratively denoise and generate the final prediction.
This enables DDPM to adaptively adjust the noise lev- els at each step to capture the data uncertainty. The for- ward distribution admits a closed-form sampling distribution q( (Y.\Yo. lo X), ge ()), o..) with an arbitrary timestep
In the reverse process, the posteriors of Y Yi-1 are tractable when conditioned on Yo. which can be restated as:
YY Yo. (x :N
(9)
N(Y. 45 04 Yo (1 (X)),
where
(3, - ( X) 3,ox.)
(7)
i = 70 Y, + m 12/a
(10)
δ 0 t-1
(11)
0 +
where we define the following coefficients:
and 70,1,2 in j given as:
v0,100
"1 / 0
II
A:==1 1 Ct.
7o 0 to
Og
010
(12)
(8)
72 varlar 110 + 10-110
à :
Σ
0i o
B. ōe dr
2:0:-++0
We leave the derivation in Appendix A.2. We follow the basic step of DDPM to parameterize a denoise model po (Y(Y-1/ fo go (X)) to match the forward process posteriors q ( Y fo (X) g ().
we leave the detailed derivation o; to Appendix A. and all these coefficients are positive numbers. Notably, under perfect estimator (assuming go (X) oY.), or simplifies to (X). and with the additional assumption of σY, = I, degenerates to the earlier constant variance settings (8,I). More detailed discussions and derivations can be found in Section 4.6 and Appendix A.5.
4.2. Loss Function
We approximate the denoising transition step Po (Y Y,, (X) (X)) the ground-truth de-

---

Non-stationary Diffusion For Probabilistic Time Series Forecasting
Dy X), Xv)) by optimizing the KL. divergence (Hershey & Olsen, 2007) between the posterior distribution and the parametrized distribution pg. Like classic DDPM, we optimize only the diagonal variance term, denoted as o and de respectively. The loss defined as the KL divergence of the noise matching term:
noising transition step (
Algorithm Training
Input: Data X, target Y, model fa. noise and variance
estimation model Ço. total timesteps T
Pre-train fx(X) to predict TE(Y[X)
Pre-train Pe (X) to predict Var(Y|X)
repeat
Draw Yo (Yo X)
Draw t ~ Uniform( (I,,...,7)
L - E DRL (Næ; jı, GIW (y: no. oo))
Draw v ~ (0,1)
n nol 1 + Σ σ: x E
Σ
log
σ Tni
Compute Y;:
Y. vā Yo + (L vasffo))
B
(13)
+ +B Br) - + 3,oY, 77 D using Eq. 7 Compute estimated noise and variance:
where 7g is the estimated noise and n is the ground truth noise. The first term ensures the estimation of the posterior mean, while the rest terms guarantee the estimation of the variance. We provide the proof in Appendix A.3.
No Fe = (Y, fo (X),
Compute loss C Numerical optimization step on VoL until Convergence
D using Eq. 13
4.3. Pretraining fo and 9
To train fe. follow prior work (Li et al., 2024a) and utilize the Non-stationary Transformer (Liu et al., 2022) as the backbone model. The training process is identical to that of standard supervised time series models (Zhou et al., 2021). For the training of gx(X), we use a sliding window approach to extract the estimated ground truth variance, similar to references (Kim et al., 2021; Liu et al., 2024b: Ye et al., 2024). Specifically, given time series label Yo the estimated ground truth variance is defined as:
process in Algorithm At the inference phase, according to Eq. 10 and 11, the calculation of the parameters for the reverse distribution requires estimating both Y, and ø For the estimation of Yo. we follow prior work (Han et al., 2022) and utilize the relationship between ; and Yo as defined in Eq. 7. However, the estimation of oYo lacks a direct correspondence with Y,. To estimate dY,, one straightforward approach is to directly use Iy (X). However, demands perfect predictor and does not incorporate the reverse process into parameter estimation. Actually, Eq. 11 can be expanded a quadratic equation with respect to aY.- Thus, we utilize the quadratic expansion of Eq. 11 to approximate oYo, we leave the detailed derivation at Appendix A.4. Specifically, expanding Eq. 11 gives the following solvable equation:
oY = VarrSSidingWindoor (Y)))
(14)
thus, the training of gu (X) is formulated supervised task. In our implementation, we utilize sliding stride of and window size of 96. The function Je is implemented as a three-layer MLP, with outputs passed through the softplus activation (Zheng et al., 2015) to ensure positivity. Further implementation details can be found in Appendix C.2 and we examine the necessity of pretraining Appendix B.2.
AOo + AOOYY + +2
(15)
where the coefficients are
Ap == 0,3,3,-1
4.4. Training NsDiff
A == 37 + 01 - -81--11---119-22 (x))
The target of NsDiff training is to match posterior distri- bution q by parameterizing Pe- Like traditional DDPM, NsDiff can be trained end-to-end by sampling random and noise 17 from uniform and Gaussian distributions spectively. According to Eq. 13, we build an estimation (X), t) during the training process to match the noise and variance. The overall procedure is presented in Algorithm
(0 + ac,:))
(16)
A2 = 9e (X) )2 3² - Br-1)
2 (a) -1 c, B3)
Ao is positive value, and according Vieta's theo- rem (Lang, 2012), when d2 < 0, the equation has exactly one positive root. The constraint for A < 0 is equivalent to:
model Ee( (Y,,,, f fx)),, gu (X)),,
at 3
4.5. Inference
(x) < o
(17)
3-1
The target the inference phase is to recur- sively sample from the parameterized distribution po (Y,- (x), we provide a detailed
Therefore, the solvability of the equation is governed by the noise level parameter 3,. Under the typical DDPM param- eterization (Ho al., 2020), where 8, ranges from 0.0001
5

---

Non-stationary Diffusion For Probabilistic Time Series Forecasting
Algorithm 2 Inference
Further derivations show that this is simply a constant mul- tiplication of the variance term from prior works (Han et al., 2022), and the training of the variance is not necessary. How- ever, this estimation of uncertainty has two main drawbacks. First, assuming a perfect estimator inherently introduces bias. In addition, this approach estimates the variance with- out leveraging the denoising process, as the variance is fully determined by pretrained gu (x.)..
Input: data X, models for Iv. and &a Initialize Yr ~ (X), go (X))
for 1. = T to 1 do
if' >1 1 then Draw z (),
end if
Compute na, 0o = Eo i, S0 (X),9x()))
Compute oY,- d1 V2F /Ap- -1
using Eq. 18
Unit Variance (w/o LSNM): Assuming known unit vari- ance, i.e., 9 (X) oY = I, Eq. 6 becomes:
23o
Compute Yo =
Y, - (1 (-- fx()
Var
N(Y: Va, Y-1 + (1 vf (X), at) I). (20)
(B, Be)go (X) + To
D using Eq.
which is consistent with previous work (Han et al., 2022). TMDM (Li al., 2024a) is a typical probabilistic fore- casting model built under this assumption. The results for TMDM are presented in Section 5.2 and 5.3, where we con- duct experiments on real and synthetic datasets, respectively. We provide detailed derivations and more discussions in Appendix A.5.
if / 1 then
Set Y 70 Yo + +2fo Vooz
else
Set Y Yo
end if
end for Output: Yo
5. Experiments
5.1. Experiment Setup
0.02, the coefficient on the right-hand side of Eq. 17 be- comes sufficiently large, thereby ensuring the equation's solvability. Hence, by solving the quadratic equation in Eq. 15, we can estimate the value of dy, during inference stage, the specific formula is given by Eq. 18:
Nine popular real-world datasets with diverse characteristics are selected, including Electricity (ECL), ILI, [T, h2, ml, m2). ExchangeRage (EXG), Traffic, and SolarEnergy (Solar). Table summarizes basic statistics for these datasets. To estimate uncertainty variation between the train and test datasets, we use the ratio of test variance to train variance, selecting the highest value across dimensions to capture non-stationary uncertainty. A detailed notebook this calculation is available in our repository. For dataset splits, we follow previous time series prediction works (Wu et al., 2022; Li et al., 2024b): the ETT datasets are split 12/4/4 months for train/ vainn/valtest, while others are split 7:1:2, Details can be found Appendix C.I.
Datasets:
\1 + X 4And2
oY,
(18)
2λo
Experimentally, the approach exhibits consistent solvabil- ity across all datasets. We provide more discussions in Appendix A.4.
4.6. Simplified Variants of NsDiff
In this section, we discuss two simplified versions of NsDiff by simplifying the variance terms in Eq. We summarize these two variants Table 1, and provide the ablation results in Section 5.3.
Table 2. Dataset properties, including total dimentions, total tim- steps, prediction steps, evaluated uncertainty variation.
Dataset
Dim.
Steps
Pred,steps
Uncert, Var.
ETTml
7
69,680
192
2.53
Table 1. NsDiff Variants.
ETTm2
7
69,680
192
1.27
Variants
Endpoint
Forward Noise
E
7
17420
192
2.50
w/o LSNM w/o UANS
N (x))) (Xfo(X) (X))
I.I 8.go
ETTh2
7
17,420
192
1.29
EXG
8
7,588
192
0.85
NsDiff
(Xfo(X)) (X))
(X) B.O...
ILI
7
966
36
8.26
ECL
321
26,304
192
3.94
Traffic
862
17,544
192
181.83
Perfect Estimator (w/o UANS): Assuming a perfect vari- ance estimator g6 (X)= o Eq. 6 becomes the following:
Solar
137
52,560
192
0.92
We selected five strong probabilistic forecast- ing baselines for comparison, including TimeGrad (Rasul et al., 2021), CSDI (Tashiro al., 2021), TimeDiff (Shen
Baselines:
N 15 o0 Y Y (1- (o-var).o(X))
(19)
(1- az)g. B (X))
6

---

Non-stationary Diffusion For Probabilistic Time Series Forecasting
Table 3. Experiment result on nine real-world datasets. bold face indicate best result.
Models
Datasets
ETThl
ETTh2
ETTml
ETTm2
ECL
EXG
ILI
Solar
Traffic
TimeGrad (2021)
CRPS
0.606
1.212
0.647
0.775
0.397
0,826
1.140
0.293
0.407
QICE
6.731
9.488
6.693
6,962
7.118
9,464
6,519
7.378
4.581
CSDI
CRPS
0.492
0.647
0.524
0.817
0,577
0.855
1.244
0.432
1.418
(2022)
QICE
3,107
5,331
2.828
8.106
7.506
7,864
7.693
9.957
13.613
TimeDiff (2023)
CRPS
0.465
0.471
0.464
0.316
0.750
0.433
1.153
0.700
0.771
QICE
14.931
14.813
14.795
13.385
15,466
14.556
14.942
14.914
15,439
DiffusionTS (2024)
CRPS
0.603
1.168
0.574
1.035
0.633
1,251
1.612
0.470
0.668
QICE
6.423
9.577
5,605
9.959
8.205
10.411
10.090
6.627
5,958
TMDM (2024)
CRPS
0.452
0,383
0.375
0,289
0.461
0.336
0,967
0.350
0,557
QICE
2.821
4,471
2.567
2.610
10.562
6,393
6.217
9.342
10.676
NsDiff
CRPS
0.392
0.358
0.346
0.256
0.290
0.324
0.806
0.300
0.378
(ours)
QICE
1.470
2.074
2.041
2.030
6.685
5.930
5.598
6.820
3.601
& Kwok, 2023), TMDM (Li et al., 2024a) and Diffu- sionTS (Yuan& & Qiao, 2024). Specifically, TMDM denoises from (X)((X.II) while others denoise from IO.
metric (QICE). For example, QICE is reduced by 47. 47.9% on ETTh1, 53.6% on ETTh2, 20.5% on ETTml, and 66.3% Traffic. Notably, on the Traffic dataset, which has the highest uncertainty variation (181.83), NsDiff achieves the most significant improvement, underscoring its strength in handling high-uncertainty scenarios.
Experiment Settings: Experiments are conducted under popular long-term multivariate forecasting settings, using an input length of 168 in all experiments. All experiments are run with seeds 3] for 10 epochs. We use the best result from the validation set to evaluate the model on the test set. The learning rate is set to 0.001, batch size of 32 and the number of timesteps T = 20, consistent with prior work (Rasul et al., 2021). We employ linear noise schedule with 8¹ = 10 -4 and BJ = 0.02, in line with the setup used in conventional DDPM (Ho et al., 2020). At inference, we generate 100 samples to estimate the distribution. For the baseline models, we utilize their default parameters.
Sample Showcases To provide a clearer understanding of NsDiff's performance. we visualize sample from the ETTh1 dataset in Figure 3. As shown, NsDiff' effectively captures the uncertainty, even under the distribution shift be- tween input and output. In contrast, TMDM, while capable of detecting mean variations, fails to adequately model the uncertainty due to its assumption of uncertainty invariance. Other models, such as TimeGrad, CSDI, and TimeDiff, which begin denoising from (O, I), struggle capture both the mean and variance. For example, as seen on the right side of the figure, TimeGrad predicts stable trend instead of the observed downward shift. This highlights the limitations of these models in handling non non-stationary behavior. In contrast, NsDiff excels at modeling such non- stationary dynamics while providing accurate uncertainty estimation, demonstrating its robustness and effectiveness in challenging forecasting scenarios. We provide other show- cases in Appendix D.
Following prior work (Li al., 2024a), we two probabilistic forecasting metrics: Quantile Interval Cover- age Error (QICE) (Han et al., 2022) and Continuous Ranked Probability Score (CRPS) (Matheson & Winkler, 1976). For both metrics, smaller values indicate better performance. Detailed formula is provided Appendix C.3. We provide point forecast results at Appendix B.I.
Metries:
5.2. Main Experiments
To evaluate the performance of NsDiff in probabilistic mul- tivariate time series forecasting, we tested it on nine real- world datasets and compared five competitive baselines. The results, summarized in Table 3, show that NsDiff consis- tently achieves state-of-the-art (SOTA) performance, with superior uncertainty estimation capabilities, except on the Solar dataset, which exhibits low uncertainty variation (0.92 shown in Table 2). Compared to the second-ber best and pre- vious SOTA TMDM, which uses an endpoint distribution of (X), I), NsDiff demonstrates significant improve- ments, particularly in the uncertainty interval estimation
5.3. Experiments On Synthetic Data
To accurately evaluate NsDiff's performance under time- varying conditions, we designed two synthetic datasets using the LSNM. Specifically, the formula used is Y m[t] + v(t]e, where m and defines the level of trend and uncertainty variation. In the linear setting, m increases linearly from 1 to 10, and v follows the same pattern. In con- trast, for the quadratic setting, v grows quadratically from 100. The total length of the generated dataset is 7588, and we predict univariate features. The results of these ex- periments are summarized in Table 4. Further details about
7

---

Non-stationary Diffusion For Probabilistic Time Series Forecasting
NsDiff
TMDM
TimeDiff
CSDI
TimeGrad rad
MA M M
Figure 3. The 95% prediction intervals of a ETThl sample, the black line is the true values, the red area represents the prediction interval.
the dataset construction can be found in Appendix C.1.2.
5.4. Ablation Experiments
This section compares two simplified variants of NsDiff discussed in Section 4.6, the ablation experiments are con- ducted on ETTh1 dataset. The abaltion variants are : (1) w/o LSNM: without LSNM assumption, which assumes condi- tional unit constant variance (Øy, = I) (2) w/o UANS: without uncertainty-awart noise schedule, which assumes a perfect noise estimator (o (x))... The results, presented in Table 5, show that NsDiff achieves the best performance, not only in overall metrics but also in the sta- bility of results (lower variance). Notably, while assuming a perfect uncertainty estimator (w/o UANS) improves CRPS by introducing variable uncertainty. it remains suboptimal in QICE compared to w/o LSNM and exhibits higher variance. This is likely due to potential overfitting of the variance esti- mator, as it fully relies on ge (X). These findings highlight the importance of a controllable noise schedule, rather than solely relying on a perfect gy (X).
Table Performace comparision for synthetic datasets
Variance
Linear
Quadratic
Models
CRPS
QICE
CRPS
QICE
TimeGrad
1.129
3.669
2.204
10.740
CSDI
1.100
3.332
1.866
5.050
TimeDiff
1.274
10.314
2.495
14.670
DiffusionTS
1.454
9.290
2.123
11.273
TMDM
1.111
4.542
2.217
11.404
NsDiff
1.057
0.987
1.777
1.336
As shown in Table 4, NsDiff achieves remarkable perfor- mance under conditions with varying variance. Compared to the previous model, TMDM, in terms of QICE, NsD- iff improves performance by 78.3% on the linear-growing variance dataset, and this improvement increases to 88.3% on the quadratic- grooiingggginggggg variance dataset. These results demonstrate the superior performance of NsDiff in captur- ing uncertainty shifts.
Table S. Variants information and ablation experiment results.
Synthetic Dataset Showcases. To visually illustrate whether NsDiff can capture the uncertainty shift between the training and test datasets, we provide an example of a lin- ear synthetic dataset in Figure 4, where the estimations for training and extended testing samples are plotted. As shown in the figure, both TMDM and NsDiff effectively capture the uncertainty within the training set. However, in the testing area (to the right of the red dashed line), TMDM assumes invariant uncertainty, while NsDiff successfully captures the uncertainty shift. This clearly demonstrates that NsDiff ef- fectively captures the distribution shift between the training and test datasets, whereas previous methods under ANM fail to do so.
Metrics
Forward Noise
QICE
CRPS
w/o LSNM w/o UANS
II
10.21
0.452+0.027
(X)
3.18420.787
0.413+0.015
NsDiff
G.90 + BIa,or,
1.47020.207
0.3922h.009
6. Conclusion
In this paper, we present Non-stationary Diffusion (NsDiff), novel class of conditional Denoising Diffusion Proba- bilistic Models (DDPMs) specifically designed to advance probabilistic forecasting. NsDiff represents the first attempt to integrate the Location-Scale Noise Model (LSNM) into probabilistic forecasting, providing a more flexible and ex- pressive framework for uncertainty representation in the data. We introduce an uncertainty-aware noise schedule, which enhances the noise modeling capabilities of DDPMs by incorporating time-varying variances directly into the diffusion process. NsDiff provides a generalized framework that extends the flexibility of existing models; by incorpo- rating a pretrained mean and variance estimator along with the designed noise schedule, NsDiff enables accurate un- certainty estimation, thereby opening new opportunities for advancing research in probabilistic forecasting.
Gperr Tul
taan
CRPS: 1.111
1.057
4.542
0.98:
200
1010
30000
1200
Figure 4. The estimated variance and ground truth in lincar vari- ance dataset, the variance is estimated using 100 samples. The red dashed line indicates the split of training and extended test sets.
8

---

Non-stationary Diffusion For Probabilistic Time Series Forecasting
Acknowledgements
Fan, W., Wang, Wang, D., Wang, D., Zhou, Y., and Fu, Y. Dish-ts: general paradigm for alleviating distribution shift in time series forecasting. In Proceedings of the AAAI conference on artificial intelligence, volume 37, pp. 7522-7529, 2023.
This work supported in part by the National Natural Sci- ence Foundation of China (Grant No. 6247075381) and par- tially funded by Huaneng Headquarters Technology Projects with No. HNKJ23-HP97, We would also like to thank the anonymous reviewers for their constructive feedback and suggestions.
Fan, W. Yi, K., Ye, H., Ning, Z., Zhang, Q. and An, N. Deep frequency derivative learning for non-stationary time series forecasting, arXiv preprint arXiv:2407.00500 2024.
Impact Statement
This paper presents work whose goal advance the field of Machine Learning. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here.
Gu, S., Chen, D., Bao, J., Wen, F, Zhang, B., Chen, D., Yuan, L., and Guo, B. Vector quantized diffusion model for text-to-image synthesis. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 10696-10706, 2022.
References
Han, X., Zheng, H., and Zhou, M. Card: Classification and regression diffusion models. Advances in Neural Information Processing Systems, 15118000-11155 2022.
Alcaraz, J. M. L. and Strodthoff,, N. Diffusion- based time series imputation and forecasting with structured state space models. arXiv preprini rrXiv:2208.09399 2022.
Hershey, J. R. and Olsen, P. A. Approximating the kull- back leibler divergence between gaussian mixture mod- els. In 2007 IEEE International Conference on Acoustics, Speech and Signal Processing-CAASSPP volume 4, pp. IV-317. IEEE, 2007.
Bertozzi, A. L., Franco, E., Mohler, G., Short, M. B., and Sledge, D. The challenges of modeling and forecasting the spread of covid-19. Proceedings of the National Academy of Sciences, 117(29):17332-1677
2020.
Chen, Y., Deng, W., Fang, S., Li, F., Yang, N. T., Zhang, Y., Rasul, K., Zhe, S., Schneider, A., and Nevmyvaka, Y. Provably convergent schrödinger bridge with applica- tions to probabilistic time series imputation. In Interna- tional Conference on Machine Learning, pp. 4485 4513. PMLR, 2023.
Ho, J., Jain, A., and Abbeel, Denoising diffusion proba- bilistic models. Advances in neural information process- ing systems, 33:6840-6851 2020.
Jiang, R., Lu, P. Y., Orlova, E., and Willett, R. Training neural operators preserve invariant measures of chaotic attractors. Advances in Neural Information Processing Systems, (6-27645-27669 2023.
Chen, Y., Goldstein, M., Hua, M., Albergo, M. S., Boffi, N. M., and Vanden-Eijnden, E. Probabilistic forecast- ing with stochastic interpolants and f." ollmer processes. arXiv preprint arXiv:2403. 13724, 2024.
Kendall, A. and Gal, Y. What uncertainties do we need in bayesian deep learning for computer vision? Advances in neural information processing systems, 30, 2017.
Chopra, and Meindl, P. Supply Chain Management: Strategy. Planning, and Operation. Pearson, 8 edition, 2021.
Kim, T., Kim, J., Tae, Y., Park, C., Choi, J.-H., and Choo, J. Reversible instance normalization for accurate time-series forecasting against distribution shift. International Conference on Learning Representations, 2021.
Dhariwal, and Nichol, A. Diffusion models beat gans image synthesis. Advances in neural information processing systems,
2021.
Kollovieh, M., Ansari, A. F., Bohlke-Schneider M., Zschiegner, J., Wang, H., and Wang, Y. B. Predict, refine, synthesize: Self-guiding diffusion models for probabilis- tic time series forecasting, Advances in Neural Informa- tion Processing Systems, 36, 2024.
Du, Y., Wang, J., Feng, W., Pan, S., Qin, T., Xu, R., and Wang, C. Adarnn: Adaptive learning and forecasting of time series. In Proceedings of the 30th ACM international conference on information & knowledge management, pp-
402-411, 2021.
Lai, G., Chang, W.-C., Yang, Y., and Liu, H. Modeling long-and short-term temporal patterns with deep neural networks. In The 41st international ACM SIGIR confer- ence on research development in information retrieval, pp. 95-104, 2018
Ermagun, A. and Levinson, D. Spatiotemporal traffic fore- casting: review and proposed directions. Transport Re- views, 38(6):786 814, 2018.

---

Non-stationary Diffusion For Probabilistic Time Series Forecasting
Lang, S. Algebra, volume 211. Springer Science & Business Media, 2012.
Ogasawara, E., Martinez, L. C., De Oliveira, D., Zimbrão, G., Pappa, G. L.., and Mattoso, M. Adaptive normal- ization: A novel data normalization approach for non- stationary time series. In The 2010 International Joint Conference on Neural Networks (IJCNN), pp. 1-8. IEEE, 2010.
Lee, S., Lee, K., and Park, T. Ant: Adaptive noise sched- ule for time series diffusion models. arXiv preprint arXiv:2410.14488, 2024.
Li, A. W. and Bastos, G. S. Stock market forecasting using deep learning and technical analysis: a systematic review. IEEE access, 8: 1882222, 2020.
Passalis, N., Tefas, A., Kanniainen, J., Gabbouj, M., and Iosifidis, A. Deep adaptive input normalization for time series forecasting. IEEE transactions neural networks and learning systems, 31(9):3760-37755 2019.
Li, W., Yang, X., Liu, W., Xia, Y., and Bian, J. Ddg-da: Data distribution generation for predictable concept drift adaptation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pp. 4092-4100, 2022a.
Rasul, K., Seward, C., Schuster, I., and Vollgraf, R. Au- toregressive denoising diffusion models for multivariate probabilistic time series forecasting. International Con- ference on Machine Learning, pp. 8857-8868. PMLR, 2021.
Li, Y., Lu, X., Wang, Y., and Dou, D. Generative time series forecasting with diffusion, denoise, and disentanglement. Advances in Neural Information Processing Systems, 35: 33009-23022 2022b.
Shen, L. and Kwok, J. Non-autoregressive conditional diffusion models for time series prediction. In Inter- national Conference on Machine Learning, pp. 31016- 31029. PMLR, 2023.
Li, Y., Chen, W., Hu, X., Chen, B., Zhou, M., et al. cuinsformer-moullttee diffusion models for probabilis- tic multivariate time series forecasting. In The Twelfth International Conference Learning Representations, 2024a.
Shi, Y., De Bortoli, V., Campbell, A., and Doucet, A. Diffu- sion schrödinger bridge matching. Advances in Neural Information Processing Systems, 36:62183-622 02-22222,22222 2023.
Spirtes, P., Glymour, C., and Scheines, R. Causation, pre- diction, and search. MIT press, 2001.
Li, Y., Chen, W., Hu, X., Chen, B., Zhou, M., al. Transformer-moculltte diffusion models for probabilis- tic multivariate time series forecasting. In The Twelfth International Conference on Learning Representations, 2024b.
Tashiro, Y., Song, J., Song, Y., and Ermon, S. Csdi: Con- ditional score-based diffusion models for probabilistic time series imputation. Advances in Neural Information Processing Systems, 34:24804-24816, 2021.
Liu, Y., Wu, H., Wang, J., and Long, M. Non-stationary transformers: Exploring the stationarity in time series forecasting. Advances in Neural Information Processing Systems, 35:9881 19893, 2022.
Tyralis, H. and Papacharalampous, G. A review of proba- bilistic forecasting and prediction with machine learning, arXiv preprint (wXiv:2209.08300) 2022.
Liu, Y., Li, C., Wang, J., and Long, M. Koopa: Learning non- stationary time series dynamics with koopman predictors. Advances in Neural Information Processing Systems, 36, 2024a.
Wang, R., Dong, Y., Arik, S. Ö., and Yu, R. Koopman neural forecaster for time series with temporal distribution shifts. arXiv preprint arXiv: 10.03675 2022.
Wu, H., Hu, T., Liu, Y.. Zhou, H., Wang, J., and Long, M. Timesnet: Temporal 2d-varittion modeling for gen- eral time series analysis. In The eleventh international conference on learning representations, 2022.
L Liu, Z.,, Cheng, M., Li, 7.,, Huang, 7., Liu, Q., Xie, Y., and Chen, E. Adaptive normalization for non-stationary time series forecasting: A temporal slice perspective. Ad- vances in Neural Information Processing Systems, 36, 2024b.
Wu, Z., Pan, S., Long, G., Jiang, J., Chang, X., and Zhang, Connecting the dots: Multivariate time series forecasting with graph neural networks. In Proceedings of the 26th ACM SIGKDD international conference knowledge discovery & data mining. pp. 753-763, 2020.
Matheson, J. E. and Winkler, R. L. Scoring rules for contin- uous probability distributions. Management science, 22 (10):1087-1096, 1976.
Nie, Y., Nguyen, N. H., Sinthong, P., and Kalagnanam, J. A
Xu, Z., Li, Y., Liu, C., and Gui, N. Ordering- based causal discovery for linear and nonlincar relations. arXiy preprint arXiv:2410.05800 2024.
time series is worth 64 words: Long-tern forecasting with transformers. arXiv preprint arXiv:2211. 14770, 2022.
10

---

Non-stationary Diffusion For Probabilistic Time Series Forecasting
Ye, W., Deng. S., Zou, Q. and Gui, N. Frequency adaptive normalization for non-stationary time series forecasting. arXiv preprint arXiv:2409. 2024.
Yuan, X. and Qiao, Diffusion-ts: Interpretable diffu-
sion for general time series generation. arXiv preprint arXiv:2403.01 000.03.017440. 022.0011222224 2024.
Zeng, A., Chen, M., Zhang, L., and Xu, Q. Are transformers effective for time series forecasting? In Proceedings of the AAAI conference on artificial intelligence, volume 37, pp. 11121-11128 2023.
Zheng, H., Yang. Z., Liu, W., Liang, J., and Li, Y. Im- proving deep neural networks using softplus units. 2015 International joint conference on neural networks (JJCNN), pp. 1-4. IEEE, 2015.
Zhou, H., Zhang, S., Peng, J., Zhang, S., J., Xiong, H., and Zhang, W. Informer: Beyond efficient transformer for long sequence time-series forecasting, Proceedings of the AAAI Conference on Artificial Intelligence, 35(12):
11106-11115, 2021.

---

Non-stationary Diffusion For Probabilistic Time Series Forecasting
A. Deriviation
A.1. Closed-form for Forward Process Distribution Y
Following the original DDPM, we first at := 1 - 3r and ag := II: 0, where 3 E (0, is a diffusion schedule. To simplify the deriviation, we further define 3r := 1 Or and o, (1 -a) De (X) + (1 - oz) 0,0Y,. With the forward distribution in Eq. 6, expanding the forward process starting from t to 0 gives the following deriviation:
Y, = varY (1 fa + Vom..
Vai (1 (1- Va-11 (x(X) + -o: + (1 vai) fo(X) + Voute
V0101-1 yoi - vot-1) [x( (1 /o sx(X) vou 01- Vori
(0 + (1 vaya4-1) fol (X) 0 0
Varar-i 1 (1-
Vo. 2)] (1 / fo(X) va,o,-1 + 1
Va10l-101-2 1-3 (1-Var0(-101=1) (X) - 0,0 0
Y, = a. + [1- va ... a1 fx(X)= +
Σ
II
ag
Tc
k+1
(21)
Eq. 21 describes the relationship between Y, and Yo. To simplify the notation, we further give the following definition:
= (a + (1 - a;))g (X) (at - (a--atooy o,
(22)
IMI M
aj
a = a1) + a - av-1) + (a:a)-((1- +... = 1
II
(23)
1+1
1=1
Σ II
aj
ar = 0 + +
0
(24)
k+1
k=iittk
t-1
aj
az = ot + tnaa + a100-100
Σ
as)ar-k
(25)
km0 kmt-
where we define above Eq. 23, and 25 as drg ag ā respectively. Eq. 21 becomes the following form:
Y. +1= vor + V(ar ā. + āz)go (X) + (ā. áo)o., no
(26)
Vo.
= -oo Yo + (1 (o-vo.)f(()) + (3. - ()) + B,oY no
(27)
Jo.
where we define 3. ãة d₁ and o; - 3) (X) + B. Eq. 27 gives the relationship between Y, and Yo. which admits the closed-form sampling distribution given Yo with an arbitrary timestep
A.2. Reverse Posterior Distribution
To simplify the notation of the following derivation, we first give the following definition:
A Y, (1 vor) (X)
(28)
B vā.- Yo + Vār-1) 1) (X)
(29)
12

---

Non-stationary Diffusion For Probabilistic Time Series Forecasting
With above definition, the conditional distribution of reverse process is:
9 (
Y,, Yo, X) x (Y
I
Yo. 9o.
A vaxYo Y (Y,-1-B))
exp
01-1
A 2/0 AY,A (Y,12)
(Y111)2
2BY, 1-1
B²
exp
(30)
Tt
01-1
1
2/aA
1
2B
exp
(Y (--1
1-++
(Y--1)
Y
r-11
0x
0t-1
0+1
VaiAA
B
exp
(Y,-1)2
+
Y--1
0:-1
at
0t-1
then, the parameter ji in the posteriors of Y is equivalent to:
V
+
B
- Bo,
j
0
de-1
0,01-1
d:
0t-1
v oz
(Y, - (1-
Vor) sa(X)) 0 + (vot-1 Yo - vau-1) So (X)
0101-1+01
(31)
10:
va,ō.-1
Vailar - 1)0 1 +1 (1-vo--00)
Yo+
Y,+
fo(X)
a,0
0 + oe
70
7
72
where, the specific form be written as:
|
C2 (S. X) + Y,B,oYo
0,01-1 0:
(a) - at 31 + B7) + σY (cn3e-1 a(
v V (3: - 31)g X) + 3,0Y
0,0,-1 00
01 IN (a) - aq a1 + B7) + σY (00 a(
(32)
722 Voi(0n 1)0,-11 + (1 va--1lor
0,, t
- - - 3, - 31- ))g X) + 0n - / V..
9o (x) a1 + 33 + oY (a, + a1
Similarly, for the parameter ō in the posteriors of Y,-1, we have:
0,01-1
o6-1 a+01-1+
(33)
(Bgg (X) + +. ()) ((3r)))((1111)))))))) ---1 Do (X) 3r 10
31 Iv (X), 3 (87) + TY Co. (or +.,
A.3. Loss Function
Firstly, KI. divergence between two gaussians can be given as:
[ = E [DKL (q (Y.-1 Y.,
po (Y.-1 (xx
E [DKL (N(Y A,, IW (o-1:Mo.00))
E
(Mg (1) Σg (11 ( + Tr 1
log
del. del(E))
C
(34)
2
det (Eo)
||pl iil + Σ
á log
x E
Tg,
13

---

Non-stationary Diffusion For Probabilistic Time Series Forecasting
where Σ is diagonal matrix representing the covariance matrix and represents its diagonal vector. Let 14 be in the form similar equation 10, and replace Yo with Y, using Eq. 27, it gives:
Mg 70
Y,, (1 vārlf(() (BL11 31-1)9u(X) + x
+nY, + (2)
(35)
Replacing ji with equation 10 in Hg jr, defined as:
Mg jı = / - (e-1) + -x. 77)
(Bt-1
(36)
x (n 770
Using Eq. 36, the loss function can be derived as:
σ. 00.i
σ o
E n
nall3
Σ
Σ log
(37)
where the first term || - mall2 the uncertainty. Specifically, assume that de which degenerates to the general version of the DDPM: E [|n ne n0
2 is matching the noise in each step and the remainder >,
a_
E;
log a
is optimizing
A.4. Estimating oY, Through ore
At inference time, the calculation of 10.1.2 requires estimating oY.. One straightforward approach is to directly use g(z); however, this method assumes perfect predictor and does not involve the reverse process in parameter estimation. Since the predictor has already provided an estimate of the reverse noise, we utilize the quadratic expansion of equation 33 to approximate o
By substituting the predicted og into equation 33 and rearranging terms, we obtain:
ar,r1Br-11 o%. + + - Br- 1))0 (X) 00(an3: ar oYo
Ao
A
yot 1))23((BB -
oag (X) -aa - -
=0
(38)
when d2 0, the quadratic equation has exactly positive root using the Vieta theorem
d2
0. σy, is given by:
Ao
A1
X
42
(39)
2
The constraint Xg 0 can be described by the following inequality:
Ot 3
(X) σe
+
(40)
B--1- - 3r-1)
where the coefficient of right-hand side is very large value; specifically, under our default beta settings with beta start from 0.0001 to 0.02, this value sufficiently large to ensure the sovlability of Eq. 39; experimentally, the results can be obtained in all datasets and samples.
14

---

Non-stationary Diffusion For Probabilistic Time Series Forecasting
A.5. Simplified Versions of NsDiff
A.5.1. PERFECT ESTIMATOR
Given perfect estimator go (X). substituting this into Eq. obtain the variance or = (X). Substituting this into
Eq. 22, Eq. 27 becomes:
Y, = v ai o + 10 (1-var) (x(X) + v a+) go (X)mo
(41)
hence, we have 5N (11 a.) gr (X). Further follows the deriviation of Appendix A.2 by replacing A, B, and o we obtain following results:
Vit
1 - 04
Y (v aa +1+ (Vār 1) (Var + +oootva.-0 vah-1) fx(
Be Yo
(42)
at
1 at
1 at
σ = - (1-ā)) (X)
(43)
for the loss function, the posterior variance is known in inference time; hence the training for the variance ő is not necessary. This variant can be simply trained using L = E [[|> no n0 Furthermore, in this result, the variance is simply a constant scaling of the variance in previous work (Ho et al., 2020; Han et al., 2022; Li al., 2024a).
A.5.2. UNIT VARIANCE
Assuming a unit endpoint variance (ge (X)=1) gives identical posterior mean in Eq. 42 and and variance with only the coefficient:
[1 - i
1 - 0t-1
Y Livas + +1+ (Vā, 1) ()) / (x(
Be Yo
+
(44)
1 āe
1- ât
ò= = (1 -1-68 (1-ag)
(45)
this is also trained in L = |||-
no
Note that TMDM (Li et al., 2024a) is a typical work under this setting.
B. Other Results and Discussions
B.1. Point Forecast Results
In time series forecasting tasks, mean square (MSE) and mean average error (MAR) reflect the point estimation accuracy. We provide the results evaluated on these two metrics in Table 6 and Table 7 of the real and syncthetic datasets respectively:
Table 6. MAE/MSE result on nine real-world datasets, bold face indicate best result.
Models
Datasets
ETTh1
ETTh2
ETTml
ETTm2
ECL
EXG
ILI
Solar
Traffic
TimeGrad
MSE
0.813
1.496
0.831
0.967
0.504
1.058
1.414
0.446
0.535
(2021)
MAE
1.062
3.462
1.218
1.690
0.505
1.567
4.197
0.475
0.983
CSDI (2022)
MSE
0.708
0.900
0.752
1.069
0.822
1.081
1.481
0.675
0.925
MAE
0.949
1.226
1.002
1.723
1.007
1.701
4.515
0.763
1.731
TimeDiff (2023)
MSE
0.479
0.485
0.477
0.333
0.764
0.446
1.169
0.713
0.784
MAE
0.517
0.456
0.537
0.268
0.879
0.402
3.958
0.821
1.350
DiffusionTS (2024)
MSE
0.774
1,411
0.744
1.232
0.856
1.564
1.788
0.740
0.815
MAE
1.089
3.273
1.030
2.372
1.072
3.628
6.053
0.749
1.473
TMDM (2024)
MSE
0.607
0.490
0.455
0.395
0.359
0.430
1.175
0.316
0.425
MAE
0.696
0.512
0.494
0.315
0.257
0.334
3.636
0.250
0.679
NsDiff (ours)
MSE
0.523
0.490
0.455
0.352
0.306
0,412
0.985
0.307
0.373
MAE
0.594
0.514
0.488
0.281
0.209
0.300
2.846
0.242
0.637
15

---

Non-stationary Diffusion For Probabilistic Time Series Forecasting
Note that TifeDiff is model speciffically designed for long-term point forecasting. As in this Table, in datasets with high non-stationarity, NsDiff still achieves SOTA, attributed to the dynamic mean and variance endpoint and the uncertainty-aware noise schedule.
Table 7. MAE/MSE result two synthetic datasets, bold face indicate best result.
Variance
Linear
Quadratic
Models
MSE
MAE
MSE
MAE
TimeGrad
1.546
3.726
2.629
10.677
CSDI
1,516
3.641
2.553
10.204
TifeDiff
1.537
3.776
2.649
11.275
DiffusionTS
1.738
4.766
2.504
9.620
TMDM
1.514
3.639
2.582
10.403
NsDiff
1.512
3.616
2.490
9.543
On the synthetic dataset, the advantages of NsDiff are more pronounced, due to the high variation of both mean and variance in the dataset. This proves NsDiff's performance under non-stationary environment.
B.2. Effects of Pretraining
Although NsDiff follows the pretraining paradigm to stabilize training and achieve optimal performance, end-to-end training remains a viable alternative. Our experiments demonstrate that the networks can be trained jointly without significant performance degradation. For instance, when evaluating on the ETThi dataset with and without pretraining, we observe comparable results in the CRPS metric. We report the results in Table 8.
Table 8, The comparison between pretraining and and-to-end training, bold face indicate best result.
epoch
pretrain
end-to-end
0.4181
0.4407
2
0,4041
0.4227
3
0.3977
0.4045
4
0.3926
0.4004
5
0.3889
0.3868
6
0.3795
0.3873
As can be seen, although joint train experiences a slight performance degradation (1.86%), it still outperforms the previous state-of-the-arr TMDM (0.452). However, compared to pretraining, co-training is slightly harder converge.
B.3. Computation Efficiency
To analyse the computational efficiency of NsDiff, compare the training and inference memory and time cost and performance, and report the results in Table
Table 9, Computation efficiency comparison, bold face indicate best result.
Model
Mem. TrBin(MB) Mem. Mee.lnnfrrece(MMB)
Tim. (im.Train(ms)
(im.Inference(ms)
CRPS
QICE
TimeGrad
27.47
8.61
47.89
8319.29
0.606
6.731
CSDI
109.81
22.61
60.50
446.70
0.492
3.107
TimeDiff
15.66
3.40
33.93
238.78
0.465
14.931
DiffusionTS
65.03
79.23
94.51
8214.53
0.603
6,423
TMDM
221.58
213.46
33.26
237.37
0.452
2.821
NsDiff
68.20
57.75
32.13
208.07
0.392
1.470
As shown in Table 9, compared to previous SOTA TMDM, NsDiff achieves SOTA and has smaller memory costs and higher
16

---

Non-stationary Diffusion For Probabilistic Time Series Forecasting
efficiency. This is because NsDiff does not introduce additional hidden variables and only adds small number of basic operations. should be noted that while NsDiff does not achieve the lowest memory cost, primarily due to its default use of the relatively heavy Non-stationary Transformer mean estimator (Liu et al., 2022), the NsDiff framework itself is model-agmostic. As such, the mean estimator can be replaced with lighter conditional expectation forecasting model e.g. DLinear (Zeng et al., 2023) to optimize memory efficiency.
C. Reproducibility
We provide all relevant data, code, and notebooks httpp:://github..mmmm
C.1. Datasets
C.1.1. REAL DATASET
Nine real-world datasets with varying levels of uncertainty were chosen, including: (1) Electricity' which documents the hourly electricity usage of 321 customers from 2012 2014. (2) ILF which tracks the weekly proportion of influenza-like illness (ILI) patients relative to the total number of patients, reported by the U.S. Centers for Disease Control and Prevention from 2002 to 2021. (3) ETT (Zhou et al., 2021) which includes data from electricity transformers, such as load and oil temperature, recorded every 15 minutes between July 2016 and July 2018. (4) Exchang (Lai et al., 2018) which logs the daily exchange rates of eight countries from 1990 to 2016. (5) Traffic* which provides hourly road occupancy rates measured by 862 sensors on San Francisco Bay area freeways from January 2015 to December 2016. (6) SolarEnergy" a
dataset from the National Renewable Energy Laboratory containing solar power output data collected from 137 photovoltaic plants in Alabama in 2007.
C. 1.2. SYNTHETIC DATASETS
To accurately evaluate the performance of NsDiff under time-varying conditions, we design two synthetic datasets using the LSNM. Specifically, the data generation follows the formula:
Y m(t] + v(t)e,
where m and define the trend level and the uncertainty variation, respectively. The results of these experiments are summarized in Table 4. We provide the generation codes below:
Linear Synthetic Dataset:
import numpy np
def renerate_synthetii..
(length):
means = np. np. (1, 10, length)
# means from to 10
stadev ap. linspace (1, 19, length)
# standard deviations from to 10
data ze. (Length)
for t in (angg(1entth)) data[t] np. www.rando..oo...m (loc-means [t], scale=stddev [t])
return data
Quadratic Synthetic Dataset:
import numpy as np
def
(1ength):
means np. inssacc,,, 10, 1ength) # means from to 10
h http://........oooos//.s//s//sst
"http://pems.dot http://pemmmddcc.aaaaa.. ca.
hhttp://ww.nrll.go www..re....oggg.///eoo ppwp html
17

---

Non-stationary Diffusion For Probabilistic Time Series Forecasting
stddev np. linspace (1, 10, length)
standard deviations from to 10
data - np. 2p.zeross
for t in zange (length):
data[t] np. www.raandd.omot. normal (loc-means [t], scale=stdddev(t) sttdeev return data
In the linear setting, m increases linearly from to 10, and v follows the same pattern. In contrast, for the quadratic setting, grows quadratically from to 100. The total length of the generated dataset is 7588, and predict univariate feature.
C.2. (X) implementation
C.2.1. COMPUTE oYo
The ground truth variance can be estimated in various ways, such as using specific dates or sliding window. Given proven success of employing sliding windows time series analysis to predict variance (Liu al., 2024b), we adopt the sliding window approach to extract the ground truth variance. The Python code for computing ov, is provided below.
def s_sigma (x, y, window_size-96)
Compute variance using sliding window.
Args:
(torch. : Input tensor (B, T, N).
y (torch. Tensor) : Output tensor (B, N) window_size (int): sliding window size (default: 96).
Returns:
torch. Tensor: Variance tenser (B, 0, N).
***
all data terch. (1x, yl, dim=1) Combine input and output window_size, 1) $ Create sliding windows
windows all_data.uufoloe
sigma = windows. var (dim=3, unbiased-False) & Compute variance
return sigmal;, shape [1];, :] Extract output step variance
C.2.2. ARCHETECUTURE
The architecture of pretrained variance estimator can take various forms. Without loss of generality, we employ simple 3-layer Multi-Layer Perceptron (MLP) as the variance estimator. The MLP is configured with a hidden size of 512 and utilizes ReLU activations between the layers. The PyTorch implementation this architecture is provided below:
nn. Sequeneiiiiaae
nn. Lineir (seq len, hidden hidden_size) size),
nn.ReLU (),,
Ln.ein (hidden_siee) size, hidden sizee__size), Rn.ReLU (),
Lineir (hidden_size, lend
C.3. Metrics
CRPS: The continuous ranked probability score (CRPS) (Matheson & Winkler, 1976) measures the compatibility of cumulative distribution function (CDF) F with an observation & as
CRPS(F,x) = | (F(2)) 11x s 2 dz
(46)
where Iz is an indicator function. Employing the empirical CDF of F, i.e. F(z) = E. [2x0 S 2} with samples 0.. ~ as natural approximation of the predictive CDF, CRPS can directly computed by samples from DDPM. We generated 100 samples to approximate the distribution F.
18

---

Non-stationary Diffusion For Probabilistic Time Series Forecasting
The quantile interval calibration error (QICE) (Han al., 2022) quantifies the deviation between the proportion of true data contained within each quantile interval (QI) and the optimal proportion, which is 1 1/M for all intervals. To compute QICE, we divide the generated y-samples into M quantile intervals with roughly equal sizes, corresponding to the boundaries of the estimated quantiles. Under the optimal scenario, when the learned distribution matches the true distribution, each QI should contain approximately 1/M of the true data, QICE is formally defined as the mean absolute error between the observed and optimal proportions, and can be expressed as:
QICE:
M Σ m==1
QICE = M
Fm
(47)
M
where I'm 1/ E yo 20
,high... denote the Un S... me Here, condition is indicator function. The terms flow and y/n lower and upper boundaries of the mn-th quantile interval, respectively. Intuitively, under ideal conditions with sufficient samples, QICE should approach O, indicating that each contains the expected proportion of data. Following Li et al. (2024a), calculate the QICE by partitioning the probability range into ten equal decile-based intervals.
D. ShowCases
We present additional results in Figure 5, which clearly demonstrate that NsDiff effectively captures the inherent uncertainty in the data, even in the presence of significant variations. Notably, in the ILI dataset, NsDiff accurately identifies the reduced variance, a feature that other methods fail detect. Moreover, on the ExchangeRate dataset- a highly volatile financial datffet-NsDiff successfully identifies both the substantial variance and the overall trend, providing precise interval estimates. In contrast, other methods exhibit notable shortcomings: for instance, TimeGrad predicts excessively large downward trend, CSDI produces overly wide intervals, and TMDM fails to adequately cover the data range. These results underscore the robustness and accuracy of NsDiff in handling diverse and challenging datasets.
NsDiff
TMDM
TimeDiff
CSDI
TimeGrad
ETTRR
A//
10
ExcallyePeta
Figure 5. The 95% prediction intervals comparison with other models.
19


Reviews:
[{"review_id": "ghZLddOXpv", "reviewer": "Reviewer_PZBU", "summary": "The paper introduces a novel diffusion-based probabilistic forecasting framework, called NsDiff, which is designed to address the non-stationary nature of uncertainty in time series data. Traditional Denoising Diffusion Probabilistic Models (DDPMs) typically rely on an Additive Noise Model (ANM) with fixed variance, limiting their ability to capture the dynamic uncertainty observed in many real-world applications. To overcome this limitation, the authors propose incorporating a Location-Scale Noise Model (LSNM) that allows the noise variance to vary with the input context. The authors also try to validate their results in a wide range of datasets with comparisons to some existing methods.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1**: additional key references [1-2]. \n\n**A1**: Thanks for your recognition of our work. We agree that the references [1–2] provide meaningful context and will help strengthen our discussion. In particular, we find several aspects of these works especially relevant to our setting:\n\n\nThe work by Chen et al. [1] proposes a novel probabilistic forecasting framework based on Föllmer processes and stochastic interpolants. Their approach to learning noise schedules through tailored loss functions resonates with our motivation to adapt diffusion endpoints for better uncertainty modeling, especially under non-stationary conditions.\n\nJiang et al. [2] address the challenge of long-term forecasting in chaotic systems by preserving invariant measures. Their use of contrastive learning to stabilize dynamics over time without requiring domain-specific priors is an inspiring direction that aligns with our interest in modeling non-stationary behavior robustly.\n\n\nWe will cite and briefly discuss these works in the revised version to better contextualize our contributions within the broader landscape of diffusion-based and non-stationary forecasting techniques.\n\n\n[1] Chen, Y., Goldstein, M., Hua, M., Albergo, M. S., Boffi, N. M., & Vanden-Eijnden, E. (2024). Probabilistic Forecasting with Stochastic Interpolants and Föllmer Processes. arXiv preprint arXiv:2403.13724. \n\n[2] Jiang, R., Lu, P. Y., Orlova, E., & Willett, R. (2023). Training Neural Operators to Preserve Invariant Measures of Chaotic Attractors. Advances in Neural Information Processing Systems, 36, 27645–27669.", "claims_and_evidence": "I think the claims made in the submission are clear and well-supported by the evidence detailed in the main text and the supplementary material. It is nice that the authors open-sourced their code.", "methods_and_evaluation_criteria": "I think the proposed methods make sense for the applications of probabilistic forecasting of non-stationary systems.", "theoretical_claims": "I looked at the proof in Appendix A, which is straightforward and easy to follow.", "experimental_designs_or_analyses": "I think the experiments done by the authors are impressive and comprehensive as they test the proposed method on many real-world datasets of different nature and dimensionalities.", "relation_to_broader_scientific_literature": "I think the proposed method has a potential of being applied to many scientific applications, including weather forecasting, forecasting of physical systems (e.g. fluid dynamics), and epidemics.", "essential_references_not_discussed": "I believe the paper would benefit from incorporating several additional essential references that provide valuable context and complementary perspectives. For example:\n\n[1] Chen, Y., Goldstein, M., Hua, M., Albergo, M. S., Boffi, N. M., & Vanden-Eijnden, E. (2024). Probabilistic Forecasting with Stochastic Interpolants and Föllmer Processes. arXiv preprint arXiv:2403.13724.\nThis work is relevant because it also leverages diffusion models for probabilistic predictions, proposes innovative loss functions for optimizing noise schedules, and offers flexible alternatives for the base measure within diffusion models.\n\n[2] Jiang, R., Lu, P. Y., Orlova, E., & Willett, R. (2023). Training Neural Operators to Preserve Invariant Measures of Chaotic Attractors. Advances in Neural Information Processing Systems, 36, 27645–27669.\nThis reference is pertinent as it addresses similar applications where noise induces non-stationarity in observations and presents methods to enhance long-term prediction accuracy.\n\nIncorporating these references would strengthen the discussion by situating the proposed work within the broader context of recent advances in diffusion models and long-term forecasting under non-stationary conditions.", "other_strengths_and_weaknesses": "N/A", "other_comments_or_suggestions": "N/A", "questions_for_authors": "I don't have any questions for authors.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "dHebxGz0sZ", "reviewer": "Reviewer_iEvE", "summary": "In this paper, the authors considered modeling the uncertainty quantification when applying diffusion models to time-series forecasting tasks. In the beginning, the authors first demonstrated a toy case study that the DDPM may not perform well on uncertainty prediction tasks due to the traditional Additive Noise Model (ANM) scheme. After that, the authors designed the location scale noise model to alleviate this issue, and proposed the Non-stationary Diffusion Model (NsDiff) framework. In the NsDiff, the authors redesigned the forward noise process and reformulated the backward generation process rigorously. Finally, the authors conducted various experiments to demonstrate the efficacy of the proposed approach.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1**: should Eq. 1. be given as the ODE/SDE integral form?\n\n**A1**: Thanks for the insight. However, we believe the reviewer may be referring to Eq. 7 instead of Eq. 1: Eq. 1 describes the LSNM and is not a stochastic process, so it cannot be written as SDE; Eq. 7 defines the data perturbation process and can be written as SDE. We give a theoretical discussion as follows. \nFirst, the Euler-Maruyama discrete form of NsDiff is\n\n\n$$\n\\mathbf{Y}(t+\\triangle t) = Y(t) -\\frac{1}{2}\\beta(t)(\\mathbf{Y}(t) - f_\\phi(\\mathbf{X}))\\triangle t + \\sqrt{\\sigma_{Y_0} - \\beta(t)(\\sigma_{Y_0} -g_\\psi(\\mathbf{X}))\\triangle t} \\sqrt{\\beta(t)\\triangle t}\\mathbf{z}(t)\n$$\n\nThe diffusion coefficient is related to the $\\triangle t$, which is non-Itô-integrable, making it intractable to define a clean continuous-time reverse SDE for the process. To give more theoretical insight, we resort to the perfect estimator assumption (i.e., $\\sigma_{Y_0}=g_\\psi(\\mathbf{X})$) and give the forward/reverse SDE as follows:\n\n$$\nd\\mathbf{Y} = -\\frac{1}{2}\\beta(t)(\\mathbf{Y} - f_\\phi(\\mathbf{X}))dt + \\sqrt{g_\\psi(\\mathbf{X})\\beta(t)}d\\mathbf{w} \n$$\n\n$$\nd\\mathbf{Y} = [-\\frac{1}{2}\\beta(t)(\\mathbf{Y} - f_\\phi(\\mathbf{X}))-g_\\psi(\\mathbf{X})\\beta(t)\\nabla_\\mathbf{Y}\\log p_t(\\mathbf{Y})]dt + \\sqrt{g_\\psi(\\mathbf{X})\\beta(t)}d\\mathbf{\\bar w}\n$$\n\nWe will include more detail in the latest version.\n\n**Q2**: improve Figure 2 and the description of Eq. 21.\n\n**A2**: Thanks for this suggestion, we will revise the paper according to your advice. \n\n**Q3**: SB problem and include SB-based baselines.\n\n**A3**: Indeed, the problem can be viewed as a SB problem. Taking CSBI as an example, we can introduce an uncertainty-aware prior for it by replacing its $p_\\text{prior}$ with the prior of NsDiff $\\mathcal{N}(f_\\phi(X),g_\\psi(X))$, to combine the advantages of NsDiff. Compared to CSBI, NsDiff can estimate the true variance $\\sigma_{Y_0}$ by $\\sigma_\\theta$ (see App. A.4)  to learn uncertainty more accurately (as reflected in the results of Sec. 5.4). We include a SB-based baseline CSBI in Table 3 and 4, see **Revewer ZzLi Q3**.\n\n**Q4**: MSE/MAE, computational time.\n\n**Q4**: NsDiff still achieves SOTA on MSE/MAE. Furthermore, compared to previous SOTA, NsDiff has smaller memory costs and higher efficiency. This is because NsDiff does not introduce additional hidden variables and only adds a small number of basic operations.\n\n**Q5**: Figure 5, the TMDM model seems to perform better, why?\n\n**A5**: In Figure 5, **TMDM actually performs worse than NsDiff**. Of course, the difference is less significant compared to Traffic as ETT datasets have relatively small uncertainty variation as in Table 2 (e.g. 1.2 ETTm1, 1.3 ETTm2). Specifically, Figure 5 shows TMDM produces a less accurate mean prediction, leading to larger MAE and MSE than Nsdiff; and,  TMDM's predictions do not sufficiently cover the true values, resulting in poorer CRPS and QICE scores. \n\n**Q6**: Is it possible to apply the diffusion model solvers during inference stage?\n\n**A6**: Yes, following A1, NsDiff has the following ODE form:\n$$\nd\\mathbf{Y} = [-\\frac{1}{2}\\beta(t)(\\mathbf{Y} - f_\\phi(\\mathbf{X}))- \\frac{1}{2}g_\\psi(\\mathbf{X})\\beta(t)\\nabla_\\mathbf{Y}\\log p_t(\\mathbf{Y})]dt\n$$\nwhere the ODE follows a semi-linear structure, remaining compatible with DPM-Solver. However, since time series tasks typically require only a few steps (less than 100) for effective performance, the inference efficiency is already sufficient. Therefore, we do not recommend applying DPM-Solver at the cost of prediction accuracy, given an additional assumption in A1.\n\n**Q7**: Bridge-based Models [1-3]  and noise editing [4]  should be discussed.\n\n**A7**: Thanks for these references. we have discussed SB-based methods in Q3. The noising editing method is interesting and relevant. While NsDiff gives a distribution-wise improvement on the initial noise, the given paper consider a sample-wise perspective to optimize the initial noise to a more reasonable space. We will provide more discussion in the latest version.\n\n**Q8**: What would happen if the pre-trained model does not predict well (stiffness). Include various random seeds to demonstrate the robustness.\n\n**A8**: NsDiff incorporates the true variance $\\sigma_{Y_0}$ into the learning process to alleviate this stiffness problem. Although the pre-trained model $g_\\psi(X)$ may not predict well, NsDiff use Eq. 18 to estimate the true variance. As shown in Table 5 of the ablation experiments (we report results mean and std on various seeds), incorporating this variance estimation improves both the performance and the robustness of our method.\n\n**Q9**: The convergence of NsDiff has not been discussed.\n\n**A9**: We acknowledge that the convergence analysis is a critical problem, which is missing in basically all previous works, e.g. TimeGrad, TMDM etc. We will explore this more in future work.", "claims_and_evidence": "The claims made in this submission are supported by rigorous and convincing evidence. However the reviewer has the following two issues:\n1. On page 3, Eq. 1. Given that diffusion models' inference is solving the SDE/ODE, should this part be given as the integral form?\n2. In Figure 2, the proposed paper, to the reviewer's understanding mainly focuses on the uncertainty prediction of time series. However, the picture during the inference stage, does not include the uncertainty interval.", "methods_and_evaluation_criteria": "Yes, the methods and evaluation criteria make sense for demonstrating the derivation of convergence analysis. Nevertheless, the reviewer still has the following concerns:\n1. To the best of the reviewer's knowledge, the authors attempt to transform some predicted value (distribution 1) to another similar distribution, which includes the uncertainty information. Based on this, to the reviewer's understanding, this problem can be treated as a kind of Schrodinger bridge problem. In addition, related works have applied the Schrodinger bridge [1] in the imputation procedure, which is similar to the CSDI model.\n2. Regarding the evaluation criteria, to the best of the reviewer's knowledge, time-series forecasting mainly focuses on prediction accuracy. The authors have not included related evaluation metrics like mean square error or mean absolute error [2]. It would be better to clarify this issue.  \n\n---  \nReferences: \n[1]. Provably Convergent Schrodinger Bridge with Applications to Probabilistic Time Series Imputation, ICML-2023 \n[2]. Transformers in Time Series: A Survey", "theoretical_claims": "The reviewer attempts to check the derivation of the theorem and it seems that nearly all the derivations look good to the best of the reviewer's knowledge.", "experimental_designs_or_analyses": "1. In the supplementary material, Figure 5, the TMDM model can perform well under ETT datasets compared to the proposed NsDiff, what causes this result? \n2. The computational time has not been proposed.\n3. Is it possible to apply the diffusion model solvers like DPM solver during the model inference stage?\n4. As mentioned above, the baseline comparison lacks related baseline models like Schrodinger bridge imputation models.  \n---  \nReferences:  \n[1]. DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 Steps, NeurIPS 2022", "relation_to_broader_scientific_literature": "The diffusion model is of great importance and applying the diffusion model to model the uncertainty information in the context of time-series forecasting is of great necessity. Thus, the key contributions of the paper related to the broader scientific literature are of great important.", "essential_references_not_discussed": "As mentioned above, there remain the related works from the following two aspects have not been discussed:\n1. Bridge-based Models: The diffusion models designed by diffusion bridges [1,2] have not been well discussed. The DDPM can be treated by a special kind of Ornstein–Uhlenbeck bridge [3]. \n2. The noise editing: It seems that the noise was modified to delineate the predicted time-series result. Thus, related works on noise selection [4] could be considered to some extent.\n\n--- \nReferences:  \n[1]. Provably Convergent Schrodinger Bridge with Applications to Probabilistic Time Series Imputation, ICML-2023  \n[2]. Flow Matching for Generative Modeling, ICLR 2023  \n[3]. Image Restoration Through Generalized Ornstein-Uhlenbeck Bridge, ICML 2024  \n[4] Xiefan Guo etal, Initno: Boosting text-to-image diffusion models via initial noise optimization. CVPR, 2024", "other_strengths_and_weaknesses": "### Strengths\n1. The topic is related to the ICML conferences.\n2. The proposed approach is interesting.\n3. The derivation is rigorous.\n\n### Weaknesses\n1. Major weaknesses have been listed in the abovementioned items.\n2. To the reviewer's knowledge, the initial value is of great importance for solving ODE. What would happen if the pre-trained model does not predict well (in the context of ODE, we call it stiffness [1])? It would be better to demonstrate the results under various random seeds to demonstrate the robustness of the proposed approach.\n3. The convergence of the proposed approach has not been discussed.\n---  \nReferences:  \n[1]. Numerical Methods for Ordinary Differential Equations", "other_comments_or_suggestions": "See the abovementioned chat window.", "questions_for_authors": "See the abovementioned chat window.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "XsLB1FgZPI", "reviewer": "Reviewer_gSNF", "summary": "This paper introduces a new probabilistic time series forecasting method based on non-stationary diffusion by estimation the step-wise means and variances. The proposed method is validated on different real-world datasets.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1**: The differences from TMDM.\n\n**A1**: We believe there are some misunderstandings regarding the relationship between our method and TMDM, particularly in the contributions and derivation aspects. We believe the differences from TMDM are clearly presented throughout the paper. To aid clarity, we list some key differences between NsDiff and TMDM along with parts of where they are discussed in the paper. We hope the following table helps the reviewer quickly identify and understand these distinctions with TMDM.\n\n| differences| mentioned in|\n| -- | -- |\n| TMDM uses ANM assumption, while NsDiff uses LSNM assumption. | Figure 1, right to line 23-27.|\n| TMDM uses $\\mathcal{N}(f_\\phi(x),\\mathbf{I})$ endpoint while NsDiff uses $\\mathcal{N}(f_\\phi(x),g_\\psi(x))$ endpoint | Figure 1, left to line 57-74. |\n| TMDM uses traditional noise schedule, while NsDiff introduces uncertainty-aware noise schedule.| Section 4.6 gives how NsDiff can degenerated to TMDM. Table 5 further gives exp. results of different noise schedule.|\n| TMDM does not optimize variance in loss function | Eq. 13 where TMDM has only the left term. |\n| TMDM does not estimate reverse variance. | line 284 in Algorithms 2. |\n| TMDM could not handle non-stationary variance | Table 3/Table 4 gives the exp. results on real/synthetic dataset, Figure 3/4 gives a visualized illustration.|\n\nWe believe that in the derivation presented in Appendix A, all the relevant parts listed in the table above are different from those in TMDM. For example, NsDiff infer the reverse distribution variance by solving Eq. 38, while TMDM does not introduce this step.\n\n**Q2**: the method is based on fine-tuning, which requires path sampling during training. Is this computationally expensive for DDPM in time series?\n\n**A2**: The fact is **no path sampling is required during training**. The training and inference procedures are the same as those of a standard DDPM, except for using a different endpoint and noise schedule. We only introduce some basic operations during training and inference, so the overall computational complexity remains. See **Q5** for experimental results.\n\n**Q3**: What if the variance predicted via MLE is too large (when some spiky data points appear)? Using sliding windows might not address this issue perfectly.\n\n**A3**: We do introduce a design to address this issue by using an uncertainty-aware noise schedule, which incorporates the true variance into the diffusion training process. This design reduces reliance on the variance predicted by the estimator (e.g., via MLE), which can be overly large in the presence of spiky or noisy data. See Table 5 for the experiments results, where it can be seen that by explicitly learning from the true variance, NsDiff becomes more robust to such cases, rather than relying solely on sliding window heuristics or estimator outputs. \n \n\n**Q4**: The numerically stability of NsDiff.\n\n**A4**: In NsDiff, the only potential numerical issue is the solvability of the equation in Eq. 15. We have already provided the conditions for solvability in Eq. 17. As stated in the paper (lines 269-272), this equation always has a solution. Therefore, theoretically, **NsDiff does not introduce additional numerical instability issues. In practice, no numerical instability has been observed in our experiments.** We kindly invite reviewers to check and run our code to verify this. \n\n**Q5**: The efficiency and memory costs in training and inference phases vs. performance improvements.\n\n**A5**: As shown in the following table, compared to TMDM, NsDiff achieves SOTA and has smaller memory costs and higher efficiency. This is because NsDiff does not introduce additional hidden variables and only adds a small number of basic operations.\n\n\n|Model|Mem.Train(MB)|Mem.Inference(MB)|Tim.Train(ms)|Tim.Inference(ms)|CRPS|QICE|\n|----|---|--|---|--|--|--|\n|TimeGrad|27.47|8.61|47.89|8319.29|0.606|6.731|\n|CSDI|109.81|22.61|60.50|446.70|0.492|3.107|\n|TimeDiff|**15.66**|**3.40**|33.93|238.78|0.465|14.931|\n|DiffusionTS|65.03|79.23|94.51|8214.53|0.603|6.423|\n|TMDM|221.58|213.46|33.26|237.37|0.452|2.821|\n|NsDiff|68.20|57.75|**32.13**|**208.07**|**0.392**|**1.470**|\n\nThe results are tested on ETTh1, with 100 diffusion steps.\n\n**Q6**: Some metrics are missing, e.g., MAE.\n\n**A6**: NsDiff still achieves SOTA on these metrics, see **Reviewer ZzLi Q3**.", "claims_and_evidence": "No. Below are some of my concerns.\n1)Estimating the variance of time series is numerically tricky. Please clarify how to predict the variances in a numerically stable way using pretraining in Sec.4.3. And what if the variance predicted via MLE is too large (when some spiky data points appear)? Using sliding windows might not address this issue perfectly.\n\n2)From Algorithm 1, it seems that the proposed training method is in fact based on fine-tuning, which requires path sampling during training. Is it too computationally expensive for diffusion-based time series forecasting models?", "methods_and_evaluation_criteria": "No.\nApart from the results reported in Table 3, it would be more convincing if the authors could report the relevant RMSEs and MAEs to show the proposed method is an unbiased point-wise predictor. In my opinion, PICP and QICE may be more suitable for evaluating uncertainty quantification tasks than forecasting tasks.", "theoretical_claims": "Yes.", "experimental_designs_or_analyses": "Yes.\n\n1)The CRPS and QICE may not be sufficient to evaluate the model’s forecasting performance.\n\n2)The authors may also need to report the time efficiency and memory costs of the proposed training and inference procedures.", "relation_to_broader_scientific_literature": "The proposed method is largely built on the following paper.\n\nReference:\n[1] Li, Y., Chen, W., Hu, X., Chen, B., Zhou, M., et al. Transformer-modulated diffusion models for probabilistic multivariate time series forecasting. In The Twelfth International Conference on Learning Representations, 2024.\n\nThe derivations in Appendix A in this paper (which is the core of the proposed method) is directly adapted from Appendix D in [1].", "essential_references_not_discussed": "No.", "other_strengths_and_weaknesses": "Strengths:\n1) The writing is clear and easy to follow.\n\n2) Including uncertainty estimation for time series diffusion model is reasonable.\n\n3) The relevant arithmetic prof regarding the computation of the step-wise variances is given.\n\nWeaknesses:\n1) The novelty of the proposed method is limited as it heavily builds upon the existing method [1].\n\n2) The numerically stability of the proposed method is somewhat questionable.\n\n3) The computational cost of proposed method is high compared to existing diffusion methods, e.g., CSDI.\n\n4)Some important evaluation metrics for forecasting are missing, e.g., MAE and RMSE.", "other_comments_or_suggestions": "Please include a detailed account of how the proposed method builds upon and differs from prior works, e.g., [1]., with appropriate citations; in the Appendix and main text.", "questions_for_authors": "Given the substantial computational cost of training this proposed method compared to other diffusion-based models, is the performance improvement significant enough to justify the expense?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "BUU7BGGS4V", "reviewer": "Reviewer_ZzLi", "summary": "This paper introduces NsDiff, a novel diffusion-based framework for probabilistic time series forecasting that explicitly addresses non-stationary uncertainty. Recognizing that conventional DDPMs rely on a fixed variance assumption from the additive noise model (ANM), the authors propose the integration of a Location-Scale Noise Model (LSNM) to allow the variance to vary with the input data. NsDiff combines a pre-trained conditional mean and variance estimator with an uncertainty-aware noise schedule that dynamically adapts noise levels at each diffusion step. Extensive experiments on nine real-world and synthetic datasets demonstrate that NsDiff significantly outperforms existing methods, such as TimeGrad and TMDM, especially in capturing changing uncertainty patterns. Although the paper provides thorough theoretical derivations and promising empirical results, some aspects—such as the integration of the pre-trained estimators and the robustness of the noise schedule—could benefit from further clarification.\n\n1. Why does NsDiff not adopt a fully end-to-end joint optimization approach? Is it necessary to pre-train the two networks separately?\n\n2. Figure 2 contains numerous curved lines. While I understand that this may reflect the authors’ intentional design style, it appears rather unappealing and should be modified.\n\n3. Although the task focuses on probabilistic MTS forecasting, I recommend additionally reporting metrics such as MSE and MAE, since the mean is an important characteristic of the distribution.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1**: Why does NsDiff not adopt a fully end-to-end joint optimization approach? Is it necessary to pre-train the two networks separately?\n\n**A1**: yes, the networks can be trained jointly without large performance loss. Below is an example to train on ETTh1 with and without pretraining, where we report CRPS metric. \n\n|epoch|pretrain|jointtrain|\n|-------|----------|----------|\n|1|0.4181|0.4407|\n|2|0.4041|0.4227|\n|3|0.3977|0.4045|\n|4|0.3926|0.4004|\n|5|0.3889|**0.3868**|\n|6|**0.3795**|0.3873|\n|MSE earlystop|||\n\nAs can be seen, although joint train experiences a slight performance degradation (1.86%), it still outperforms the previous state-of-the-art TMDM (0.452). However, compared to pretraining, co-training is slightly harder to converge. We will clarify this in the updated version.\n\n**Q2**: Figure 2 should be modified, e.g. remove curved lines.\n\n**A2**: Thanks for this suggestion, we will modify Figure 2 according to your advice in the lastest version.\n\n**Q3**: Report additional metrics such as MSE/MAE.\n\n**A3**: Thanks for your insight. We give the results on real and synthetic datasets at the following tables with an addtional baseline CSBI (as required by Reviewer gSNF), the settings are consistent with the main context. The repo is updated accordingly to include CSBI code.\n\nWe present the results of MSE/MAE of synthetic datasets at follows:\n|Variance|Linear||Quadratic||\n|-----------|------|---------|---------|---------|\n|Models|MSE|MAE|MSE|MAE|\n|TimeGrad|1.546|3.726|1.626|4.173|\n|CSDI|1.516|3.641|1.546|3.768|\n|TimeDiff|1.537|3.776|1.559|3.762|\n|DiffusionTS|1.738|4.766|1.689|4.823|\n|TMDM|1.514|3.639|1.493|3.568|\n|NsDiff|**1.512**|**3.616**|**1.479**|**3.448**|\n\nAs seen in this Table, NsDiff still achieves SOTA in uncertainty variation conditions. \n\nWe present the results of MSE/MAE of real datasets at follows:\n\n|Models|Datasets|ETTh1|ETTh2|ETTm1|ETTm2|ECL|EXG|ILI|Solar|Traffic|\n|-------------|----------|--------|--------|--------|--------|--------|--------|--------|--------|---------|\n|TimeGrad|MSE|0.813|1.496|0.831|0.967|0.504|1.058|1.414|0.446|0.535|\n|(2021)|MAE|1.062|3.462|1.218|1.690|0.505|1.567|4.197|0.475|0.983|\n|CSDI|MSE|0.708|0.900|0.752|1.069|0.822|1.081|1.481|0.675|0.925|\n|(2022)|MAE|0.949|1.226|1.002|1.723|1.007|1.701|4.515|0.763|1.731|\n|CSBI|MSE|0.634|0.820|0.757|0.636|0.783|0.897|1.438|0.651|0.848|\n|(2023)|MAE|0.762|0.659|0.526|0.841|0.923|0.746|4.344|0.748|1.527|\n|TimeDiff|MSE|**0.479**|**0.485**|0.477|**0.333**|0.764|0.446|1.169|0.713|0.784|\n|(2023)|MAE|**0.517**|**0.456**|0.537|**0.268**|0.879|0.402|3.958|0.821|1.350|\n|DiffusionTS|MSE|0.774|1.411|0.744|1.232|0.856|1.564|1.788|0.740|0.815|\n|(2024)|MAE|1.089|3.273|1.030|2.372|1.072|3.628|6.053|0.749|1.473|\n|TMDM|MSE|0.607|0.490|**0.455**|0.395|0.359|0.430|1.175|0.316|0.425|\n|(2024)|MAE|0.696|0.512|0.494|0.315|0.257|0.334|3.636|0.250|0.679|\n|NsDiff|MSE|0.523|0.490|**0.455**|0.352|**0.306**|**0.412**|**0.985**|**0.307**|**0.373**|\n|(ours)|MAE|0.594|0.514|**0.488**|0.281|**0.209**|**0.300**|**2.846**|**0.242**|**0.637**|\n\nNote: TimeDiff is a model speciffically designed for long-term point forecasting.\n\nAs in this Table, in datasets with high non-stationarity, NsDiff still achieves SOTA, attributed to the dynamic mean and variance endpoint and the uncertainty-aware noise schedule.   \n\n\nHere we give the results of an addtional baseline CSBI, where NsDiff still remains SOTA. \n|Models|Datasets|ETTh1|ETTh2|ETTm1|ETTm2|ECL|EXG|ILI|Solar|Traffic|\n|-------------|----------|---------|---------|---------|---------|---------|---------|---------|---------|---------|\n|CSBI|CRPS|0.552|0.571|0.502|0.491|0.585|0.659|1.109|0.498|0.875|\n|(2023)|QICE|6.141|5.230|3.471|8.918|7.982|6.870|7.175|10.830|11.382|", "claims_and_evidence": "See the summary.", "methods_and_evaluation_criteria": "See the summary.", "theoretical_claims": "See the summary.", "experimental_designs_or_analyses": "See the summary.", "relation_to_broader_scientific_literature": "See the summary.", "essential_references_not_discussed": "See the summary.", "other_strengths_and_weaknesses": "See the summary.", "other_comments_or_suggestions": "See the summary.", "questions_for_authors": "See the summary.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The work under review proposes to explicitly model the time-dependent uncertainty inherent to time series forecasting in diffusion models. A location-scale noise model is used to flexibilize the usual simple additive noise model.

All reviewers agree about the soundness and value of this contribution. The method is theoretically sound, widely applicable, and extensively tested empirically. Given the relevance and prominence of diffusion-based methods, this work has potential to impact many others.

Please do include the point-based results (MSE, MAE) requested by multiple reviewers in the final version.
Decision: Accept (spotlight poster)