MMPD: Diverse Time Series Forecasting via Multi-Mode Patch Diffusion Loss
Authors: Yunhao Zhang
Wenyao Hu
Jiale Zheng
Lujia Pan
Junchi Yan
Venue: ICLR 2026 Poster (2026)
Keywords: time series forecasting
loss function
Primary area: learning on time series and dynamical systems
TL;DR: We propose the MMPD loss for patch-based time series forecasting backbones to model complex future distributions, enabling them to generate multiple diverse predictions with corresponding probabilities.

Abstract: Despite the flourishing in time series (TS) forecasting backbones, the training mostly relies on regression losses like Mean Square Error (MSE). However, MSE assumes a one-mode Gaussian distribution, which struggles to capture complex patterns, especially for real-world scenarios where multiple diverse outcomes are possible. We propose the Multi-Mode Patch Diffusion (MMPD) loss, which can be applied to any patch-based backbone that outputs latent tokens for the future. Models trained with MMPD loss generate diverse predictions (modes) with the corresponding probabilities. Technically, MMPD loss models the future distribution with a diffusion model conditioned on latent tokens from the backbone. A lightweight Patch Consistent MLP is introduced as the denoising network to ensure consistency across denoised patches. Multi-mode predictions are generated by a multi-mode inference algorithm that fits an evolving variational Gaussian Mixture Model (GMM) during diffusion. Experiments on eight datasets show its superiority in diverse forecasting. Its deterministic and probabilistic capabilities also match the strong competitor losses, MSE and Student-T, respectively. The source code is publicly available at: https://github.com/Thinklab-SJTU/MMPD.

Published as a conference paper at ICLR 2026
MMPD: DIVERSE TIME SERIES FORECASTING VIA MULTI-MODE PATCH DIFFUSION LOss
Yunhao Zhang', Wenyao Hu', J Zheng², Lujia Pan? & Junchi Yan 'Shanghai Jiao Tong University, "Huawei Noah's 's Ark Lab (zhangyunhac, hwy123455 yanjunchije jttu edu. ithengjiale2,
panlu panlu Shwawwee..iiiiiii...... com
Prob: 38.46%
Prob: 24.18%
23.08%
Input
Prod 50% Confidence Interval
710h
31th
Thath
Figure 1: MSE loss (Left) vs. our MMPD loss (Right) using the same decoder-only Transformer backbone on dataset ETTml, imput 336-predict 192 task. MSE results in single, ambiguous prediction with a symmetric, constant confidence interval, failing to capture sudden changes in the future. In contrast, our MMPD generates multiple sharp predictions with associated probabilities (only Top-3 predictions are shown), and the confidence intervals are asymmetric and vary over time. More visualizations are shown in Fig. of Appendix
ABSTRACT
Despite the flourishing in time series (TS) forecasting backbones, the training mostly relies on regression losses like Mean Square Error (MSE). However, MSE assumes one-mode Gaussian distribution, which struggles to capture complex patterns, especially for real-world scenarios where multiple diverse outcomes are possible. We propose the Multi-Modk Patch Diffusion (MMPD) loss, which be applied to any patch-bac based backbone that outputs latent tokens for the future. Models trained with MMPD loss generate diverse predictions (modes) with the corresponding probabilities. Technically, MMPD loss models the future distribu- tion with diffusion model conditioned latent tokens from the backbone. A lightweight Patch Consistent MLP is introduced as the denoising network to ensure consistency across denoised patches. Multi-mode predictions are generated by a multi-mode inference algorithm that fits an evolving variational Gaussian Mixture Model (GMM) during diffusion. Experiments on eight datasets show its superiority in diverse forecasting. Its deterministic and probabilistic capabilities also match the strong competitor losses, MSE and Student-T, respectively. The source code publicly available at: htttps://ge hub. com Th i nk al- SJTU/MMM DMMP
INTRODUCTION
Time series (TS) forecasting have made fast progress. Plenty of backbones have been proposed, incorporating various techniques like sparse attention et al. 2019 /hou et al.. (021), season
decomposition (Wu et al. 2021 Zeng et al. 2023). frequency enhancement (Zhou et al. (022).
patchify (Nie al. 2023 Zhang Yan
2023 and cross-channel dependency (Liu et al.] 2023).
Despite the rich works on backbone design, most works rely on regression losses like Mean Square Error (MSE) for training. However, using MSE essentially assumes that the future follows Gaussian distribution with fixed variance (details in Sec. 3.1. Such parametric distribution has several limitations, including its symmetric formulation and independent, constant uncertainty.
"Correspondence to: Junchi Yan. This work was partly supported hy Scientific Research Innovation Capability Support Project for Young Faculty (U40) of the Ministry of Education of China, SRICSPYY-ZV20250019,

---

Published as a conference paper at ICLR 2026
Most importantly, the single-mode Gaussian cannot support diverse forecasting, where the same past may lead to multiple possible futures. Diverse forecasting is necessary in the real world. On the data side, multi-mode pattern is fundamental property: identical inputs can diverge into different futures due to unobserved background contexts (Bergmeir) 2024). On the application side, it is a natural requirement in downstream tasks: in domains like trading, averaged forecast offers little actionable insight, whereas multi-mode predictions enable risk-aware decision (Tsay) 2005).
Therefore, even with carefully designed backbones, the model's capacity remains limited if the training loss cannot capture complex distributions. Several works have attempted to improve loss functions: Le Guen & Thome 2019 2020) explore the Dynamic Time Warping (DTW). Due high complexity, DTW-based losses are hard to scale to long-term forecasting. Salinas et al. (2020); Rasul et al. 2023) model the future with negative binomial and Student-T distributions, Woo et (2024) uses mixture of parametric distributions. Although better than MSE, their formulations are still manually predefined, limiting the ability to model complex distributions.
To fill the gap, we propose Multi-Mode Patch Diffusion (MMPD) loss to model complex future distributions. MMPD is generally applicable to any patch-based backbone that divides input series into patches and outputs latent tokens for the future, now one of the most important categories of backbones in both supervised and foundation models. Given one input, models trained with MMPD loss predict multiple diverse futures (modes), each with an associated probability, illustrated in the right of Fig. [ Meanwhile, MMPD) loss also integrates with traditional deterministic forecasting, similar MSE
Technically, inspired by recent efforts of diffusion on visual tokens (Li et al. 2024a). MMPD constructs diffusion process for future series conditioned on tokens from upstream forecasting backbones. In Sec. 3.2, we propose a lightweight Patch Consistent MLP as the denoising network MMPD loss. When denoising a patch, it not only takes the corresponding token as the condition but also considers adjacent noisy patches, ensuring consistency across patches. The integration with deterministic forecasting achieved by optimizing the diffusion objective at special anchor inputs. As diffusion samples exhibit multi-mode patterns, multi-mode inference algorithm is devised in Sec. 3.3 It fits an evolving variational Gaussian Mixture Model (GMM) at each diffusion step alongside the reverse process. Priors from the forward process are injected via variational inference to guide the update of GMM. At the end of the reverse diffusion, the GMM outputs multi-mode predictions with corresponding probabilities. The highlights are:
1) Beyond the dominant MSE loss that assumes a simple Gaussian distribution, we propose the MMPD loss, leveraging the diffusion process to capture complex distributions. MMPD loss is backbone-agnostic and readily applicable to any patch-baste backbone.
2) Observing multi-mode patterns in predictions, devise multi-mode inference algorithm that out- puts diverse predictions with associated probabilities. Unlike pre-defined mixture distributions (Woo) et al. 2024), the number and structure of modes are adaptively inferred, offering greater flexibility.
3) Experiments on eight datasets show the superiority of MMPD loss in diverse forecasting. Its deterministic and probabilistic capabilities also match the best-performing competitor losses, MSE and Student-T, respectively. Its generality is validated on four different backhones.
2 PRELIMINARIES ABOUT DIFFUSION MODELS
We leave the related works to Appendix A and briefly overview the preliminaries about diffusion models. Given training samples and corresponding conditions (e.g., images and captions): y", q(y", c), Diffusion models define a forward Markov process that gradually adds noise to the sample:
K
q(yl:K ly c))
q
.c)
=N VI Bay k- 1.3I)
(1)
k=1
where {.& E (0, 1)}K is the variance schedule to control the added noise, With the forward process, reverse Markov process for denoising is modeled by neural network:
K
(y 0: c) II
po
(xt,c) Po(y (x),) -N (y*-'1:me c, c, (2)
2

---

Published as a conference paper at ICLR 2026
where o% is step-dependent constant and js represents the neural network that parameterizes the reverse process. The parameters of As are learned by minimizing the negative log- ligeeiklioood Ruxxx .e) log Ps [)]. Through parameterization and simplification, the final objective is:
[|le ,ck.c cy C, 1)) -"=van =a=ya + v Ogg e~ O,e
En k.k.
where o = 1 32 and 0₂ = II. Ct.. Eg is a network to parameterize μ4 in Eq. It takes noisy sample y condition с and diffusion step k as input and ouiputs the estimated noise in y". Once well-trained, new samples can be generated from the reverse process, i.c. Eq.
METHODOLOGY
3.1
BROADENING THE DEFINITION OF LOSS FROM PROBABILISTIC VIEW
Given the past series x E IRT TS forecasting aims to predict values of the same series at the desired future horizon y RT. From probabilistic view, the task is model the conditional distribution of the future given its past: pxy]x). Assuming an independent Gaussian with predicted mean and fixed variance: po (ylx) (y: fo (x), I)), the objective of maximum likelihood estimation yields:
max E (xxy) Pl pa (y = max E (x.y)
y fo(x))1 Const
0
2
min E (xxy)
MSE(ff(x).
20
where So(-) I + IR is neural network parameterized by 0 to predict the mean. or E R is the constant standard deviation. y) y) the training dataset distribution.
Using gradient-based optimizers like Adam for training, the constant coefficient - will be absorbed by the step size. Therefore, the following relationship holds (Bishop & Nasrabadi 2006
Using MSE loss implicitly assumes the future follows an independent Gaussian distribu- tion, with predicted mean and fixed constant variance.
This assumption is restrictive with limitations: 1) single-mode Gaussian is unsuitable when multiple distinct futures are possible: 2) predicted steps are assumed independent, yet real-world steps are often correlated; 3) variance is fixed, whereas uncertainty typically evolves over time; 4) the Gaussian is symmetric, but real-world predictions may be asymmetric, e.g., rainfall is nonnegative.
More generally, regardless of how refined the network is, its expressiveness will be limited because MSE assumees simple, parametric form for the future distribution. The same limitations apply to MAE loss, which assumes a Laplace distribution with predicted location and fixed, independent scale.
To move beyond restricted assumptions and model more complex distributions, we decouple the forecasting network into a backbone and projector:
fx(x) 9e (H), H = h (x), = {6, }
(5)
1) The backbone (-) extracts latent representations and contains the majority of the parameters. Plenty of backbones with various techniques have been proposed in recent years (Wen et al. 2023).
2) The projector (-(-) maps the representations to the output space, typically via lightweight MLP with few parameters. Its design highly depends on the chosen distribution-for instance, using a Gaussian with predicted variance requires the projector output both mean and variance.
Since the backbone is the core and dominates the parameter count, from the perspective of backbone optimization, the projector can be viewed as part of the loss, forming composite, trainable loss:
min Loss (H, y), H = h (x)
In this broader definition, the projector Io acts as an auxiliary module that guides backbone optimiza- tion. This is conceptually related the adversarial loss (Goodfellow et al. 2014). where a learnable
We focus univariate forecasting: for multivariate data, the loss is computed per channel and averaged.
3

---

Published as a conference paper at ICLR 2026
discriminator is introduced to guide generator training. Adopting this view, design flexible losses that capture richer distributions beyond Gaussian forms. Moreover, traditional losses naturally fall into this framework: MSE can be expressed as: MSE y) = lly - Io (
3.2 DIFFUSION TRAINING WITH PATCH CONSISTENT MLP
Unlocking the capacity of backbones requires losses corresponding more flexible distribution fami- lies. Leveraging the strong ability to capture complex distributions, we propose diffusiio-basse loss,
MMPD* (H, y). Unlike standalone TS diffusion models relying on specialized architectures (Tashiro
al. 202T), MMPD serves as a plug-and-play loss applicable across various backbones.
In this work, we focus on patch-based backbones, which divide past series into patches as input and output latent tokens for the future] Specifically, in these backbones, past series is divided into patches of length P and then embedded into T/P latent tokens. Backbones capture dependency among tokens and output latent tokens H {b, -- T/P, each corresponding to future patch. As discussed in Appendix A many recent supervised and pre-training models fall into this category.
The core idea of our MMPD loss is enable flexible distribution modeling through condi- tional diffusion, where future latent tokens serve as the condition. To achieve this, denoising net- work <y (h,) ) k:) required. As an aux- iliary module guide backbone optimization, this denoiser should be lightweight. A straight- forward strategy is split y* into patches p and use an MLP to denoise each patch p conditioned on token hy, as done in Li et al. (2024a) for visual tokens. However, such independent MLP models the marginal distribu- tion of each patch p(p, x), 1 S j S 1 rather than the joint distribution of all future patches P(PI,.... pi|x). This can lead inconsistent samples during inference, resulting in discontin- uous jumps between patches shown in Fig. Bla).
MWM
step' k next,
prey
AdaLN-MLP
tokany
Figure 2: Patch Consistent MLP for diffusion. To predict the noise ej patch pý. besides the cor- responding token h, and diffusion step k, adjacent noisy patches centered around j (pf-r.... P and / pi+1...... . p colored in red) are also input to A-ML as conditions. This ensures con- sistency across denoised patches.
To maintain consistency among patches while keeping denoiser lightweight, we extend Adap- tive LayerNorm MLP (ALP (Peebles & Xie (023) to construct Patch Consistent MLP:
cy(y*, (hy [ k) = 16 éd)] éi] ê AdaL.N-MPP(ff.. c/))
cjj token, + step" + prev + next
token, w(tnken) h, step" k Emb (mb(step) (k)
prev ((prev) . 1 next = (newt) =
[p)+1* .... pr
As illustrated in Fig. the predicted noise ē TR. the concatenation of predicted noise in each patch, i.e., éj 2023), with details in Appendix B To predict noise in patch pý: the conditioning vector integrates four components: latent tokens tokeny, diffusion timestamp step*, previous and next noisy patches prevž and next Here, (token) E R.dmudee Xdmoousi W(prer), (prew) E Rddmede XP are learnable matrices. Emb (step)()) is the positional encoding, is constant haper-parameter that controls the adjacent range around patch j that the MLP can access. Padding is used when j S r or j > -
AdaL.NMMLP is the denoising MLP from DiT block (Peebles & Xie
The key distinction between Patch Consistent MLP and independent MLPs lies in its of adjacent patches as conditions, i.c., the last line of Eq. | Fig. b) shows that this ensures consistency across patches. Moreover, the additional parameters introduced (i.e., W W(nezt) are minimal.
Integration with deterministic forecasting. MMPD loss provides a flexible distribution. But in many applications, deterministic forecasting is still needed, which is the role typically served by MSE.
MMPD can also be adapted non-patch-basee hackbones with minor modification, as shown in Appendix
4

---

1. Fig. 9 of Appendix also shows that prediction accuracy is robust w.r.t A* across broad range. After trainig the determmistic prediction is directly obtained as -I-/=== Vag. ea (hy) k*), bypassing costly diffusion iterations. This integration introduces no new architectures, as it reuses the denoiser eg. Importantly, the deterministic forecasting term does Published as a conference paper at ICLR 2026
U.SK
k= 03K
0.1
AMP JA
p(2*) n(w")
p(At)
Priors at k: (at""
Figure 3: (a) Samples predicted by an independent denoising MLP on dataset Dynamic, showing the inconsistency between patches. (b) Samples predicted by our Patch Consistent MLP, displaying clear multi-mode patterns that are challenging to represent using simple statistics (black line: median, red area: 50% confidence interval). (c) The evolution of our multi-mode inference algorithm: at each step k. posteriors and estimations p /Z*) ):P(("), (opw"),((A p(A*), )), fr are updated via variational EM steps, based
on newly generated samples {y- n=1- The updates are also guided by priors each step
A naive solution is to generate multiple samples and take the mean or median. However, diffusion iterations are costly. For efficiency, we integrate deterministic forecasting within the diffusion framework. Considering the diffusion objective (Eq. 3 . if the noise cancels the sample at step ke
yiu yo. Thus, such that y" = the target reduces scaled negative ground truth vi-aa- treat (0, (h k/=
k²) as anchor input for deterministic forecasting and define the joint objective:
C > |e co (h, (b:))=1,))11 (1 ~)
Vāk- y ++ (6,0. V1- - a...
(8)
where A 0.99 by default balances the probabilistic and deterministic objectives. k* is make vig
āgg- close to 0.5 such that VI-ai-
not conflict with the diffusion term it is merely special case of the diffusion objective at the anchor.
3.3 MULTI-MODE INFERENCE THROUGH EVOLVING VARIATIONAL. GMM
Once trained, we get a flexible distribution pa x)).. However, unlike MSE loss that corresponds to closed-form Gaussian distribution, pe (y( modeled by diffusion process an implicit distribution that lacks an explicit analytic form. To summarize this distribution and extract interpretable informa- tion for downstream tasks, prior works typically draw samples from it and then compute statistics such as the median and confidence interval (Rasul et al.) 2021a Shen & Kwok (023). However, as shown Fig. b)) the samples exhibit multi-mode patterns, indicating that the same past can lead to multiple possible futures. Such diversity cannot be adequately captured by simple summary statistics.
To address this, we propose multi-mode inference algorithm that explicitly summarizes distinet outcomes. Suppose the true distribution takes the following multi-mode form:
M Σ
<(y%)
wm 8(y
m
where { 1== denote the . possible predictions and the probability to predict y is Wm. To estimate { [w y 1.M m==1, we combine this multi-mode prior with the forward diffusion process (i.c.,
Eq. [] yielding the forward distribution at step k:
q(y" x
Σ
WnNN : Varyin (1 T
(10)
This is Gaussian mixture distribution governed by two priors:
5

---

Published as a conference paper at ICLR 2026
Algorithm 1 Multi-Mode Inference Algorithm
Input:
Future tokens { number of samples N draw via diffusion, maximum number of modes M, prior hyperparameters p. in Eq. [T
Initialize: Estimation [um)m=) and parameters in posteriors {ūK m "m Jm=1
2 LM
Ouput: Statistics and probability of each mode
KNN Generate initial samples at step K: {y fa=1 ~ N(y5
for k K - 1x.....,0 do
# Generate samples at step k; [ynJä= Diffusion (Lyhtt) (ha
# E-step: update posterior )
N
M
p(Z*)
II
(Frm
yrhm;
Ium
Eee nit
== m=1
In rhm
lIvt
p 1 + (n(27)) + [wtttt
In( (
- Σ ))
# M-step: update estimation p., posterior pl p.p..((A))))) pmm No.
Ex Tim
Nk. - 7/
p(w*) Dirichlet (w ;
#
Ni u
N t +
p(A*) 1 Gamma(Af, Ah. The ohh ; wit
+ 2 me
Nk Th
2/ mm Σ
1 112
end for Mode [y arg max The = m
P(Modem. Modem] /N and get statistics of each mode
1) Mixture weights. The weights wm remain constant across steps k. In practice, the number of effective modes should be limited, i.e., W'n Rs 0 for most modes, to avoid frag-ffrgeented predictions. 2) Covariance matrix. The covariance matrix evolves with it and should be (1 ag. I at step k.
Consequently, when drawing N samples via reverse diffusion, the collection at step k; Y (y") W should follow the Gaussian mixture distribution in Eq. 10 This observation leads to the use of GMM over other clustering models. By fitting GMM alongside the reverse process each 1M step, we can recover [wmm Yn "nJm= m= from the final GMM at step O. To better leverage the priors on mixture weights and covariance, we employ a variational GMM rather than the standard GMM, which enables explicit prior injection. The prior at step k is set as:
N
M
N M
a(Y*Iz² 4 A+)
II
II
N Ny poas (A "I)=am q(Z"
II
II
(w
)
am
R=1 m==
n=1 m==1
#Prior for mixture weights: q(w") Dirichlet(w": T). Jm = pm=
(1))
#Prior for variance: (A*))
Gamma : ums en)) un u, Dy (1 a
m=1
We use q(-) ) to denote prior distributions and {m u u m= without tilde for parameters in priors. z {z/ (n=) where each z a one-hot latent variable indicating the mode yt belongs to. Line I defines the GMM with means 4 {u J-M inverse variances A = {A JiM and mixture weights w* f m= Line assigns a constant Dirichlet prior w.r.t diffusion step A for w Its parameter decays by when mode index m increases, encouraging nigher-indexee modes to vanish. Line 3 assigns Gamma prior on A such that (E[A*,
1M
āg, consistent with the covariance prior. No prior is set for ju it is related to the unknown y The only hyperparameters are / and u, with controlling the decay of the Dirichlet prior over mixture weights and u setting the shape of the Gamma prior variances. Their effects are evaluated in Appenidx F
/ /
With these priors, we get the multi-mode inference Algorithm| [ with detailed derivations Ap- pendix C p(-) denote posterior distributions and {* U,, mm-=1 with tilde are for parameters in posteriors. As shown in Fig. 3 c), rather than using standard GMM as a post-processing method only applied to [ 1= the knowledge from Eq. [0via via evolving prior distributions, which mitigates the difficulty of GMM initialization and makes more informed choice of the number of active modes.
2 1.
distribute GMM iterations across diffusion steps. This allows to utilize

---

Published as a conference paper at ICLR 2026
Table 1: Top-3 MSE/MAE, MSE, CRPS evaluations of different losses. The forecasting horizon T is {96, 192, 336, 720} for the first seven datasets and {60, 120, 180, 300} for Dynamic. Results
are averaged over 4 horizons. Rank: average rank on 8 datasets. Bold/underline: best/second. Inf: infinity problem caused by outliers. See Table 7 in Appendix E Eoor full results.
Dunse: ETThi ETTal ETT22 ETTm2
WIH BCL
Tralice Dynunic Rank |E7TI ETTml
ETT22 ETTu2
WTH
ECL
Traflie Dynamic
Rank
Messie
Top-3 MSE
Top-3 MAF
MSF
0.450
0.348
0.364
0.264
0.234
0.176
0.453
0.336
3.5
0.440
0.381
0.508
0.320
0.262
0.278
0.326
0.311
4.875
MAE
0.441
0.364
0.365
0.276
0.235
0.179
0.440
0.425
5.25
0.437
0.375
0.592
0.321
0.263
0.272
0.310
0.295
4.125
Camaton
0.439
0.361
0.379
0.280
0.255
0.165
0.419
0.343
4.75
0.457
0.386
0:411
0.357
0.293
0.256
0.282
0.309
5.125
SundentT
0.430
0.352
0:375
0.286
0.241
0.165
0.416
0.300
4.25
0.428
0.371
0.598
0:383
0.261
0:250
0.761
0.292
3,375
Mix
0.425
0.259
0.343
01245
0.319
0.147
0.412
0.323
1.875
0.426
0.333
0.387
0.308
0.243
0.240
0.361
0.346
MMPD
0.396
0,369
0.299
0.214
0,195
0.147
0,389
0.301
0,412
0.331
0.357
0.285
0.231
0.238
0.254
0.207
Mettc
MSB
CRPS
MSE
0.425
0.350
0.376
0.270
0.227
0.160
0.399
0.345
1,675
0.337
0.307
0.308
0.247
0.218
0.270
1.345
0.255
4.25
MAE
0.432
0.355
0.366
0.214
0.235
0.164
0.417
0.425
3.5
0.346
0.313
0.299
0.447
0.220
0.288
0.362
0..75
4.625
Gamsiia
0.434
0.357
0.382
0.251
0.255
0.163
0.413
0.349
4
0.317
0.282
0.515
0.256
0.228
0.190
0.217
0.233
4,375
Studes-T
0.426
0.349
0.372
0.281
0.241
0.164
0.418
0.392
3.5
0.310
0.271
0.500
0.250
0.201
0.187
0.304
0.224
2.25
Mix
0.446
0.358
0.390
0.285
0.259
0.167
0.426
0.482
6
0.316
0.269
0,510
0.247
0.209
lnf
0.305
0.224
3
MMPD
0.412
0.337
0.354
0.264
0.229
0.164
0.400
0.353
1.75
11318
0.270
0,301
0.345
0.199
0.391
0.202
0.203
Table 2: MSE, Mix and MMPD losses over backbones: Crossformer, SegRNN and MaskAE. Horizons consistent with those in Table I. #1st: number of first ranks across 8 datasets. Inf: infinity problem caused by outliers. See Table
& in Appendix B for the full results.
Duneet
JTTDi
ETT2
WTH ECL Trutic Dynankc Bist ETTLI
ETTml
ETTh
ETTa2 WTH
BCL
Traffle
Dynamic
Music
Top-3 MSE
Tip-3 MAE
MSE
0,443
0.375
1,372
0,266
0,225
0.156
0.451
0.331
0
14552
01399
0.412
0,325
0.258
0,258
0.342
0,305
0
Crosstoemer
Mix
0.433
0.330
1.359
012330
0,200
0.150
0.624
0.330
0-
11435
013358
02400
0.3477
0.215
0.255
0.279
0.235
0
MBPP
0.381
0.310
0.315
0.223
0,197
0.152
0.484
0.285
0.410
0.353
0.373
0.285
0.226
0.245
0.261
0.194
MSE
0.480
0.345
0.305
0.273
0.222
0.185
0.664
0.333
0
16511
0.417
0.408
0.337
0.209
0.286
0.515
0.307
0
SoRRNN
0.43
0.341
0.365
0.314
0155
0.430
0.330
D.32
0.378
0.389
0.311
0.346
0.345
0.533
0.247
MMPP
0.402
0.321
0,201
0.150
0:415
0.285
0.426
0.375
0.377
0.365
0.334
0.242
01260
0,210
0.436
0.354
0.355
0.226
0.436
0.330
D.447
0.392
0.387
0.263
0.290
0.320
0.312
MasLAE
0.415
0.340
0.425
0.384
0.241
0.265
0.244
MMPD
0,999
0.280
0.311
0.197
0.144
0,387
0.296
06816
0.362
0,367
0,504
0.235
0.3354
0.255
0,205
Music
MSS
CRPS
MSE
0.440
0.383
0.388
0.271
0.228
0.170
0:418
0.339
0.344
0.316
0.319
0.249
0.221
0.274
0.348
0,253
0
Groefoemcr
0.400
0.999
0.40%
0.271
0.341
0152
0.490
0.455
0.305
0.281
0.321
6:348
0.282
Inf
0.217
Inf
MMPD
0.416
0.374
0.270
0L170
0.314
0.267
0.316
0.249
01204
0,119
01200
6.194
0.376
0.158
0.425
0,341
0,314
0,259
0.221
0,273
0,345
0,255
SaJRNN
0.452
MMPD
0.434
0.328
0.257
0.286
0,204
MSF
0.437
0.387
0.3 356
0.787
0,230
0.162
0.413
0.349
D.541
0.310
0.303
(1255
(1220)
0.271
0.344
0.259
MackAF
Mix
0.456
0.371
0.394
0.291
0.270
01770
0.441
0.476
0.309
0.277
Inf
0,250
Inf
0.194
0
0,222
MMPD
0.421
0.342
0.362
0,281
0
0.161
0.404
0.350
0.319
0.277
0,205
0.255
0
0,388
0.291
0,302
4 EXPERIMENTS
We conduct experiments on: ETTh1, ETTml, ETTh2, ETTm2, WTH, ECL., Traffic, Dynamic. The first seven are widely used datasets from previous works (Nie et al. 2023). The new Dynamic consists of 17 signals from a complex dynamical system without obvious periodic patterns. For each dataset, we fix the look-back window T and make predictions different horizons T.
Following Top-K accuracy for image classification (He et al. 2016), we use Top-K MSE/MAE (K 3 in our setting) to evaluate multi-mode prediction: Top-K K modes with the highest probabilities are selected and the minimum MSE/MAE among K modes is reported. Using small K and guided by the probability of each mode, Top-K K. MSE is more applicable than Best MSE (Le Guen & Thome 2020), which computes the MSE of all N samples and reports the best. We also report traditional metrics such MSE and Continuous Ranked Probability Score (CRPS) evaluate deterministic and probabilistic accuracy. Detailed setups, including datasets and metrics, are shown in Appendix [
4.1 MAIN RESULTS
MMPD ps. Baseline Losses. We compare MMPD with the following losses: 1) deterministic losses
MSE (Nie et al. 2023) and MAE (Liu et al., 2022a); 2) distribution-based losses Gaussian (Salinas
ct al. 2020) and Student-T (Rasul et al. 2023); 3)Mix (Woo et al., 2024) that mixes multiple
parametric distributions for flexible modeling. We maintain the main backbone as a patch-based
decoder-only Transformer (Goswami al. 2024
Lin et al. 2024b) and change the losses.
Top-3 MSE and Top-3 MAE in Table T show that only Mix and MMPD can capture multi- mode patterns. Among them, our MMPD loss consistently outperforms Mix, as the number and form of mixture components in Mix are predefined, while in MMPD, they are learned directly from the data. Regarding deterministic forecasting performance measured by MSE, our MMPD loss is
7

---

Published as a conference paper at ICLR 2026
Tip3 KG4
To-3 NCC
Te
C20
(a)
(b)
(c)
Figure 4: (a) Metrics for varying adjacent range r Patch Consistent MLP (r O: independent MLP) Dynamic (prediction horizon 180). (b) Metrics for varying probabilsticcdetemmiissi objective balancing weight A, with all other settings identical to (a). (c) Top-3 MSE/MAE comparison between multi-mode inference and post-processing methods.
comparable to the best competitor, MSE loss, and even outperforms on some datasets, showing MMPD effectivelyy integrates deterministic forecasting. Similarly, MMPD performs par with the best best-performing baseline, Student-T T, in terms of probabilistic forecasting measured by CRPS.
Generality of MMPD Loss across Backbones. Besides the decoder-only Transformer used in Table 1 we also compare MMPD loss with MSE and Mix across the following three backbones: 1)
channel-mixing Transformer Crossformer (Zhang & Yan 2023), 2) patch-based RNN SegRNN (Lin) iet al., 2223), 3) Masked AutoEncoder using pure Transformers MaskAE (Zhang et al. 2024b). Results in Table demonstrate that the diverse forecasting capability of MMPD significantly outper- forms MSE and Mix across all three backbones. The deterministic forecasting ability measured by MSE is comparable MSE loss, which is consistent with Table [ is worth noting that, due the log-normat component, Mix loss is likely to generate outliers, leading to the infinity problem in CRPS. This issue particularly severe for the RNN-base based SegRNN. contrast, the CRPS of the MMPD loss remains stable, regardless of whether the upstream backbone is an RNN or a Transformer.
4.2 MODEL ANALYSIS
Ablation of Patch Consistent MLP. In Fig. 4a), the independent MLP, which does not incorporate adjacent patches (r 0), performs poorly multi-mode prediction, with the Top-3 MSE even exceeding MSE. This occurs because the independent MLP only models the marginal distribution of each patch pe (x) rather than the joint distribution of all patches, leading inconsistent samples
as shown in Fig. 3 [). In contrast, the Patch Consistent MIP, even with 1, significantly reduces both Top-3 MSE and Top-3 MAE Further increasing slightly improves performance.
Effect of Balancing Weight À in Eq. 8 Fig. ()
shows that when training with only the diffusion objec- tive (A 1.0), the deterministic prediction capability measured by MSE poor. Decreasing it to 0.999, MSE is greatly reduced, while other metrics are not harmed. As gradually decreases, MSE improves and stabilizes. An interesting observation is that when À decreases from 1.0 to 0.9, besides MSE, other multi- mode and probabilistic metrics also get better. We suspect that this is the collaborative effect brought by joint training of the two objectives.
MSE Loss
MMPD Loss
Lingar Projection
Flatten
Decoder (same as Encoder)
Encoder
pest petches petches to predict
A blation of Multi-Mode Inference Algorithm. Fig. (c) compares our multi-mode inference algo- rithm with various post-processing methods. Random assignment performs the worst, highlighting the neces- sity of multi-mode extraction. Our multi-mode infer- ence algorithm significantly outperforms KMeans and spectral clustering. Furthermore, MMPD surpasses Dost,GMM, which uses the same GMM formulation MMPD but is applied directly to final samples (y )= without the evolving priors. This stems from fitting evolving GMM with dynamic priors gradually denoised samples, which mitigates the
Figure Approach for adapting encoder- only backbones to use MMPD loss: learn- able tokens, indicating the patches to predict, are appended to the end of the past patch sequence. The padded sequence is fed to the same network as encoder, transforming the backbone into a decoder-only model. Only the output tokens corresponding to the future series are used for MMPD loss computation.
8

---

Efficiency Analysis. As shown in Table the Patch Consistent MLP for MMPD incurs marginally higher FLOPs than the conventional MLP for MSE loss. During training. MSE loss requires one MLP forward pass, while MMPD loss requires two: one for the diffusion objec- Published as a conference paper at ICLR 2026
MiT Dive AMB-T0 any pickeenru 1o11
woll (ber MarT
CMR Deurebrr
(a)
(h)
(c)
(d)
Figure 6: Comparison of adapted decoder-only PatchTST (Nie et al.
2023) with MMPD loss
(Dec+MMPD loss) against the original encoder-only version with MSE loss (Enc+MSE loss). (a), (b): number of parameters and metrics (MSE, Top-3 MSE and CRPS) for varying input length 7'' on ETTml, the prediction horizon is set to т = 192. (c), (d): number of parameters and metrics for varying prediction horizon on ETTml, the input length is set to T 768.
difficulty of parameter initialization in GMM and automatically selects a more appropriate number of activated modes. Due to page limit, other hyperparameter evaluations (c.g., noise schedule, diffusion steps, hyperparameters in Algorithm D are provided in Appendix FF
Adapting Encoder-Only Backbones for MMPD Loss. In the left of Fig. 5 besides backbones
we have evaluated, there are encoder-only backbones that flatten the encoder outputs and linearly
project them to predict the series for MSE loss (Nie et al. 2023 Lno & Wangg 2024). They do not generate future latent tokens, meaning MMPD loss cannot be directly applied. In the right of Fig. [ we transform them into decoder-only ones for our MMPD loss by appending Iearnable tokens the end of the input sequence.
We adapt the encoder-only PatchTST (Nie al., 2023) into decoder-only backbone enable MMPD. In Fig. (o)) the scale of the projection layer in encoder-only version increases with input/output lengths, while remaining constant with our adaptation. Decoder-only PStchTST with MMPD gets lower MSE, indicating better scalability. Also, our adaptation enables mode and probabilistic forecasting, evidenced by improved Top-3 MSE and CRPS in Fig. d))
MMPD Loss vs. Standalone TS Diffusion Models. We also compare MMPD loss with standalone TS diffusion models 1)CSDI (Tashiro et al. 2021). 2)T (Kollovieh et al.] 2023). 3)MG-
TSD (Fan et al. 2024). 4)Diffusion-TS (Yuan & Qiao 2024). 5)D*U (Li et al. 2025). These models involve complex denoising networks, making long-term experiments conducted in Sec. 4.]
challenging. Following Tashiro al. (2021). forecast the next 24 steps using the past 96
ETTh1. Results in Table 3 show that our MMPD significantly outperforms standalone diffusion models. This is because the decoupling from the backbone allows the MMPD loss fully leverage the advanced backbone. Moreover, the lightweight denoising MI. in MMPD ensures faster inference compared to diffusion models with heavier networks and complex diffusion processes.
Table 3: Evaluation of MMPD loss against TS dif- fusion models on ETThI, 1 T = 96, 7 = 24. "Time" refers to average inference time per instance.
Tep-3 MSE
Top-3 MAE
MSE
CRPS Timets)
CSDI
0.225
0.304
0.339
0.265
1.014
tive and another for the deterministic in Eq. 8 However, since the backbone dominates train-
TSDiff
0.275
0.336
0.345
0.292
0.419
MG-TSD>
0,287
0.331
0.340
0.306
0.217
ing cost (i.e., FiRS >> F FMPP. FPC-MLP). the total training FLOPs of MMPD remain nearly identical those of MSE. This equivalence also holds for deterministic inference, where both losses require a single MLP pass. For probabilistic and multi-mode inference, which MSE cannot perform, MMPD's overhead is significantly lower than that of standalone TS Diffusion models, as MMPD only requires a single heavy backbone pass followed by multiple lightweight MLP passes. The theoretical analysis is consistently supported by the experimental memory occupancy and speed measurements in Table
Diffusio-TT
DRU
0.244
0.344
0.351
0.338
0.285
0.453
Mussder-MMPP
0.186
0.280
0.298
0.254
0,075

---

Published as a conference paper at ICLR 2026
Table FLOPs across structures/stages for MSE Loss, TS Diffussonn Models and MMPD Loss. S:
number of blocks in MLPs, d (short for dmmodet): hidden state dimension. FOLD. FMLP, PLC FLOPs of backbone, conventional MLP and Patch Consistent MLP. GMM FLOPs in MMPD are
omitted they are negligible vs. neural networks. See derivation in Appendix G
Structuro/Siage
MSE Loss
Diffusion Models
MMPD Loss
Projector
PILL
O 1)d Pd)
FPC-MPP O(71(55 3)6 + (2r 2)Pd)
Triining (forward only) Deteministic Inference
Fass FMEP
Flas NK
Figs 2F2C
Faus PMEP
Flau + FPC-MLP
BaooMMuli-MMdd Inference
NIA
Fis + N NKFPO-MEP
Table 5: Memory, training (per batch) and inference time (per instance) of MSE Loss, Diffusion-TSS and MMPD Loss on dataset WTH, T 336, T 192, batch 32, . 100, K 20.
MSE Loss
Diffusion-TS
MMPD Loss
Stage
Memory (GB)
Time (ms)
Memory (GB)
Time (ms)
Memory (GB)
Time (ms)
Training
2.599
89.9
4.358
676.4
2.930
106.3
Deterministic Infer
0.031
2.3
N/A
0.034
3.1
Prob/Multi-Me Mode Infer
N/A
11.245
28,495.1
0.505
415.8
FURTHER DISCUSSIONS AND CONCLUSION
In Appendix H we extend MMPD beyond its basic setting to patch-based backbones by inserting Transformer decoder layer between the backbone and MMPD loss. In Appendix [1 beyond the single-dataset paradigm, we further apply MMPD to a multi-task model, UNITS (Gao al., 2024). to perform multi-task, few-shot and zero-shot forecasting. In both cases, MMPD functions auggay loss that can be incorporated with minimal changes. Compared with original models trained with MSE loss, MMPD preserves deterministic forecasting performance while enabling richer distribution modeling, supporting multi-mode and probabilistic forecasting.
In conclusion, have proposed MMPD, a diffusion- loss that goes beyond the dominant single-mode MSE loss in TS forecasting. By modeling complex distributions, MMPD equips models with multi-mode forecasting capabilities-an essential feature for many real-world applications, par- ticularly those involving risk-aware decision making. Extensive benchmark experiments demonstrate the effectiveness and broad applicability of our approach.
REFERENCES
Juan Miguel Lopez Alcaraz and Nils Strodthoff. Diffusion-based time series imputation and fore- casting with structured state space models. Transactions on Machine Learning Research (TMLR), 2022.
Christoph Bergmeir. Fundamental limitations of foundational forecasting models: The need for multimodality and rigorous evaluation. In Workshop at NeurIPS, 2024.
Christopher M Bishop and Nasser M Nasrabadi. Pattern recognition and machine learning. 2006.
Si-An Chen, Chun-Liang Li, Sercan O Arik, Nathanael Christian Yoder, and Tomas Pfister. Tsmixer: An all-mlp architecture for time series forecasting. Transactions on Machine Learning Research (TMLRR) 2023.
Marco Cuturi and Mathieu Blondel. Soft-dtw: a differentiable loss function for time-series. In International Conference on Machine Learning (ICML), 2017.
Xinyao Fan, Yueying Wu, Chang Xu, Yuhao Huang, Weiqing Liu, and Jiang Bian. Mg-tsd: Multi- granularity time series diffusion models with guided learning process. In International Conference on Learning Representations (ICLR), 2024.
Valentin Flunkert, David Salinas, and Jan Gasthaus. Decpar: Probabilistic forecasting with autore- gressive recurrent networks. International Journal of Forecasting, 2017.
Shanghua Gao, Teddy Koker, Owen Queen, Tom Hartvigsen, Theodoros Tsiligkaridis. and Marinka Zimik. Units: A unified multi-task time series model. In Conference on Neural Information Processing Systems (NeurIPS), 2024.
10

---

Published as a conference paper at ICLR 2026
Tilmann Gneiting and Adrian E Raftery. Strietly proper scoring rules, prediction, and estimation. Journal of the American statistical Association, 2007.
Ian J Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Conference Neural Information Processing Systems (NeurIPS), 2014.
Mononito Goswami, Konrad Szafer, Arjun Choudhry, Yifu Cai, Shuo Li, and Artur Dubrawski. Moment: A family of open time-series foundation models. In International Conference on Machine Learning (ICML), 2024.
Kaiming He. Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In IEEE / CVF Computer Vision and Pattern Recognition Conference (CVPR), 2016.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In Conference on Neural Information Processing Systems (NeurIPS), 2020.
Qihe Huang, Lei Shen, Ruixin Zhang. Shouhong Ding, Binwu Wang. Zhengyang Zhou, and Yang Wang. Crossgnn: Confronting noisy multivariate time series via cross interaction refinement, In Conference on Neural Information Processing Systems (NeurIPS), 2023.
Ming Jin, Shiyu Wang, Lintao Ma, Zhixuan Chu, James Y Zhang, Xiaoming Shi, Pin-Yu Chen, Yux- uan Liang, Yuan-Fang Li, Shirui Pan, et al. Time-llm: Time series forecasting by reprogramming large language models. In International Conference on Learning Representations (ICLR). 2024.
Taesung Kim, Jinhee Kim, Yunwon Tae, Cheonbok Park, Jang-Ho Ho Choi, and Jaegul Choo. Re- versible instance normalization for accurate time-series forecasting against distribution shift. In International Conference on Learning Representations (ICLR), 2021.
Marcel Kollovieh, Abdul Fatir Ansari, Michael Schnee-S Jasper Zschiegner, Hao Wang, and Yuyang Bernie Wang. Predict, refine, synthesize: Self-guiding diffusion models for probabilistic time series forecasting. In Conference on Neural Information Processing Systems (NeurIPS), 2023.
Zhifeng Kong and Wei Ping. On fast sampling of diffusion probabilistic models. arXiv preprint arXiv:2106.0013 2021.
Vincent Le Guen and Nicolas Thome. Shape and time distortion loss for training deep time series forecasting models. In Conference Neural Information Processing Systems (NeurIPS), 2019.
Vincent Le Guen and Nicolas Thome. Probabilistic time series forecasting with shape and temporal diversity. In Conference on Neural Information Processing Systems (NeurIPS), 2020.
Colin S. Lea, Michael D. Flynn, René Vidal, Austin Reiter, and Gregory Hager. Temporal convo- lutional networks for action segmentation and detection. In IEEE / CVF Computer Vision and Pattern Recognition Conference (CVPR), 2017.
L. Li, J. Yao, L. K. Wenliang, T. He, T. Xiao, and J. yan. Grin: Generative relation and intention network for multi-agent trajectory prediction. In Conference on Neural Information Processing Systems (NeurIPS). 2021.
Qi Li, Zhenyu Zhang, Lei Yao, Zhaoxia Li, Tianyi Zhong, and Yong Zhang. Diffusion-based decou- pled deterministic and uncertain framework for probabilistic multivariate time series forecasting. In International Conference on Learning Representations (ICLR), 2025.
Shiyang Li, Xiaoyong Jin, Yao Xuan, Xiyou Zhou, Wenhu Chen, Yu-Xiang Wang, and Xifeng Yan. Enhancing the locality and breaking the memory bottleneck of transformer on time series forecasting, In Conference on Neural Information Processing Systems (NeurIPS), 2019.
Tianhong Li, Yonglong Tian, He Li, Mingyang Deng, and Kaiming He. Autoregressive image generation without vector quantization. In Conference on Neural Information Processing Systems (NeurIPS), 2024a.
Yuxin Li. Wenchao Chen, Xinyue Hu. Bo Chen, Mingyuan Zhou, et al. meaesforme-mmoddlltee diffusion models for probabilistic multivariate time series forecasting. In International Conference Learning Representations (ICIR), 2024b.
11

---

Published as a conference paper at ICLR 2026
Shengsheng Lin, Weiwei Lin, Wentai Wu, Feiyu Zhao, Ruichao Mo, and Haotong Zhang. Seg- rnn: Segment recurrent neural network for long-term time series forecasting. arXiv preprint 11X1::23008.1100, 2023.
Shengsheng Lin, Weiwei Lin, Wentai Wu, Haojun Chen, and Junjie Yang. Sparsetsf: Modeling long-term time series forecasting with 1k parameters. In International Conference Machine Learning (ICML), 2024a.
Shengsheng Lin, Weiwei Lin, Wentai Wu, Songbo Wang, and Yongxiang Wang. Petformer: Long- term time series forecasting via plachholder-enhhneed transformer. IEEE Transactions on Emerging Topics in Computational Intelligence, 2024b.
Minhao Liu, Ailing Zeng, Muxi Chen, Zhijian Xu, Qiuxia Lai, Lingna Ma, and Qiang Xu. Scinet: Time series modeling and forecasting with sample convolution and interaction, In Conference on Neural Information Processing Systems (NeurIPS), 2022a.
Shizhan Liu, Hang Yu, Cong Liao, Jianguo Li, Weiyao Lin, Alex X. Liu, and Schahram Dust- dar. Pyraformer: Low-complexity pyramidal attention for long-range time series modeling and forecasting. In International Conference on Learning Representations (ICLR), 2022b.
Xu Liu, Junfeng Hu, Yuan Li, Shizhe Diao, Yuxuan Liang, Bryan Hooi, and Roger Zimmermann. Unitime: A language-empowere unified model for cross-domain time series forecasting. ACM Web Conference (WWW), 2024a.
Yong Liu, Tengge Hu, Haoran Zhang, Haixu Wu, Shiyu Wang, Lintao Ma, and Mingsheng Long. itransformer: Inverted transformers are effective for time series forecasting. In International Conference on Learning Representations (ICLR), 2023.
Yong Liu, Haoran Zhang, Chenyu Li, Xiangdong Huang, Jianmin Wang, and Mingsheng Long, Timer: Generative pre- me-itrined transformers are large time series models. International Conference on Machine Learning (ICML), 2024b.
Donghao Luo and Xue Wang. Modernten: A modern pure convolution structure for general time series analysis. In International Conference Learning Representations (ICLR), 2024.
Meinard Müller. Dynamic time warping. Information retrieval for music and motion, 2007.
Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. In International Conference on Machine Learning (ICML), 2021.
Yuqi Nie, Nam H. Nguyen, Phanwadee Sinthong, and Jayant Kalagnanam. A time series is worth 64 words: Long-term forecasting with transformers. International Conference on Learning Representations (ICLR), 2023.
William Peebles and Saining Xie. Scalable diffusion models with transformers. In IEEE / CVF Computer Vision and Pattern Recognition Conference (CVPR), 2023.
Kashif Rasul, Calvin Seward, Ingmar Schuster, and Roland Vollgraf. Autoregressive denoising diffusion models for multivariate probabilistic time series forecasting. In International Conference on Machine Learning (ICML), 202
Kashif Rasul, Abdul-Saboor Sheikh, Ingmar Schuster, Urs Bergmann, and Roland Vollgraf. Multi- variate probabilistic time series forecasting via conditioned normalizing flows. In International Conference Learning Representations (ICLR), 2021b.
Kashif Rasul, Arjun Ashok, Andrew Robert Williams, Arian Khorasani, George Adamopoulos, Rishika Bhagwatkar, Marin Biloš. Hena Ghonia, Nadhir Hassen, Anderson Schneider, et al. Lag-llama: Towards foundation models for time series forecasting. In RO-FoMo: Workshop on Robustness of Few-shot and Zero-shot Learning in Foundation Models NeurIPS, 2023.
David Salinas, Valentin Flunkert, Jan Gasthaus, and Tim Januschowski. Deepar: Probabilistic forecasting with autoregressive recurrent networks. International journal of forecasting, 2020.
12

---

Published as a conference paper at ICLR 2026
Lifeng Shen and James Kwok. Non-autoregreesive conditional diffusion models for time series prediction. In International Conference on Machine Learning (ICML). 2023.
Yusuke Tashiro, Jiaming Song, Yang Song, and Stefano Ermon. Csdi: Conditional score-based diffusion models for probabilisticc time series imputation. In Conference on Neural Information Processing Systems (NeurIPS), 2021.
Ruey S Tsay. Analysis of financial time series. John wiley & sons, 2005.
E Vijay, Arindam Jati, Nam Nguyen, Gift Sinthong, and Jayant Kalagnanam. Tsmixer: Lightweight mlp-mixer model for multivariate time series forecasting. In ACM SIGKDD International Confer- ence on Knowledge Discovery & Data Mining (KDD), 2023.
Qingsong Wen, Tian Zhou, Chaoli Zhang, Weiqi Chen, Ziqing Ma, Junchi Yan, and Liang Sun. Transformers in time series: A survey. 2023.
Gerald Woo, Chenghao Liu, Akshat Kumar, Caiming Xiong, Silvio Savarese, and Doyen Sahoo. Unified training of universal time series forecasting transformers. In International Conference on Machine Learning (ICML), 2024.
Haixu Wu, Jiehui Xu, Jianmin Wang, and Mingsheng Long. Autoformer: Decomposition transformers with auto-correlation for long-term series forecasting. In Conference Neural Information Processing Systems (NeurIPS), 2021.
Haixu Wu, Tengge Hu, Yong Liu, Hang Zhou, Jianmin Wang, and Mingsheng Long. Timesnet: Temporal 2d-variation modeling for general time series analysis. In International Conference Learning Representations (ICLR), 2023.
Jinsung Yoon, Daniel Jarrett, and Mihaela Van der Schaar. Time-series generative adversarial networks. In Conference on Neural Information Processing Systems (NeurIPS), 2019.
Chengqing Yu, Fei Wang, Zezhi Shao, Tao Sun, Lin Wu, and Yongjun Xu. Dsformer: double sam- pling transformer for multivariate time series long-term prediction. In Conference Information and Knowledge Management (CIKM), 2023.
Xinyu Yuan and Yan Qiao. Diffusion-ts: Interpretable diffusion for general time series generation. In International Conference on Learning Representations (ICLR), 2024.
Ailing Zeng, Muxi Chen, Lei Zhang, and Qiang Xu. Are transformers effective for time series forecasting? In AAAI Conference on Artificial Intelligence (AAAI), 2023.
Yitian Zhang, Liheng Ma. Soumyasundar Pal, Yingxue Zhang, and Mark Coates. Multi-resolution time-series transformer for long-term forecasting. In International Conference on Artificial Intelligence and Statistics (AISTATS), 2024a.
Yunhao Zhang and Junchi Yan. Crossformer: Transformer utilizing cross-dimension dependency for multivariate time series forecasting. In International Conference on Learning Representations (ICLR), 2023.
Yunhao Zhang, Minghao Liu, Shengyang Zhou, and Junchi Yan. UP2ME: Univariate pre-training to multivariate tine-tuning as general-purpose. purpose framework for multivariate time series analysis. In International Conference on Machine Learning (ICML), 2024b.
Zhenwei Zhang, Xin Wang, and Yuantao Gu. Sageformer: Series-aware graph-enhanced transformers for multivariate time series forecasting. arXiv preprint arXiv:2307.01666 2023.
Haoyi Zhou, Shanghang Zhang, Jieqi Peng, Shuai Zhang, Jianxin Li, Hui Xiong, and Wan Zhang. Informer: Beyond efficient transformer for long sequence time-series forecasting. In AAA/ Conference on Artificial Intelligence (AAAI), 2021.
Tian Zhou, Ziqing Ma, Qingsong Wen, Xue Wang, Liang Sun, and Rong Jin. Fedformer: Frequency enhanced decomposed transformer for long-term series forecasting. In International Conference on Machine Learning (ICMI.,, 2022.
Tian Zhou, Peisong Niu, Liang Sun, Rong Jin, et al. One fits all: Power general time series analysis by pretrained Im. In Conference on Neural Information Processing Systems (NeurIPS), 2023.
13

---

Published as a conference paper at ICLR 2026
A RELATED WORKS
Backbones for TS Forecasting. Early works employ RNNs Fllunkert et al. 2017) and CNNs Leaa
et al. 2017) as backbones. Transformers for TS were introduced later, incorporating techniques such sparse attention (Li et al. 2019: Zhou et al. 2021), trend-season decomposition (Wu et al. 2021), frequency enhancement (hou et al. ) and hierarchical structure (Liu et al., 2022b). Nie
al. 20h3):ZZhag & Yan (2023) proposed patch-based Transformers that divide time series into
patches. This approach was later adopted widely, resulting in the development of various patch-based
models (Vijay al., [2223 Lin et al. 2023; Luo Wang 2024; Zhang et al. [023] Yu al. 2023
Zhang ct al. 2024a). Furthermore, many recent pre-training models (Zhang et al. 2024b Woo
at.l. 2022| Goswami et al.
2024 Liu al. 2024b) and cross-modality models (Liu et al.) 2024a
Zhou et al.
2023 Jin et al. 2024) also adopt patch-based backbones. Other studies have explored lightweight designs (Zeng et al. 2023 Lin et al., (0224)) and cross-channel dependency capture (Liu
et al. 2023; Huang et al. 2023). Despite such advances in backbones, they mostly use regression loss functions. This greatly limits the backbones' capability, especially in diverse forecasting.
Loss Functions for TS Forecasting, Despite extensive research on backbones, limited focus was paid to forecasting specific losses. A line of works uses Dynamic Time Warping (DTW) (Müller 2007), which computes the similarity between two series with dynamic programming. Cuturi & Blondel (2017) makes DTW differentiable. Le Guen & Thome (019 2020) extend DTW to evaluate shape and temporal distortions. However, is hard to scale non- paraa lelizable DTW-based losses to long-term tasks. Another line predicts future distributions by estimating their parameters. Common distributions include Student-T (Rasul et al.) 2023). Gaussian and negative binomial (Salinas et al.) 2020). Additionally, Woo et al. (2024) mix multiple parametric distributions (e.g., Gaussian, Log- normal, etc.) via a Softmax layer. However, these methods rely on predefined formulations, failing to capture complex patterns.
Deep Generative Models for TS. Pioncering efforts on deep generative models for TS include GANs Yoon et al. 2019), normalizing flows (Rasul et al. 2021b) and VAEs (Li et al. 2021). With the success of diffusion models (Ho et al., 2020; Peebles & Xiel 2023). many diffusion models for TS have also been proposed. |Rasul et al. (2021a) propose an RNN- based diffusion model. [Tashiro
al. (202T); Alcaraz& Strodthoff (2022) condition diffusion on observed data for imputation, [Shen & Kwok (2023) and Li al. (2024b) respectively
the prediction of autoregressive models and Transformers as the prior knowledge to guide diffusion. Yuan & Qiao (2024) introduces trend-season decomposition to enhance diffusion. Li al. (2025) propose a two-stage framework that predicts the deterministic component via point forecasting and models the probabilistic residual with diffusion. These efforts primarily focus on refining denoising networks optimizing the diffusion process. In contrast, our approach leverages diffusion models to develop a backbone-agnostic loss function.
B DETAILS OF ADALN-MLP
The AdaLN-MLP is an MLP that takes a noisy patch p and the condition vector e as input and predicts the noise in p. It is originally component of Diffusion Transformer (Peebles & Xie 2026). The Diffusion Transformer was designed to replace the U-Net backbone in diffusion models, and introduced Adaptive LayerNorm (AdaLN) blocks to inject conditions into diffusion models. One AdaLN block consists of an A-ttt-Att block and an AdaLN-MLP block. For efficiency, we only use the AdaLN-ML.P in our MMPD loss. The computation process of one AdaL.N-MPP block is as follows:
(+)) (0-1) a gate
(s)
MLP (z(a / s/(inh)
AdaLN 7, = (1 +) LayerNorm(z) + B
(12)
(( gute (cc)
(5) scale
w(s) scale (C), (1())
w())
gale
(+)) Rddmade is the output of s-th block (with z (0) being the linearly embedded p) and dmdmtttt
is the condition. o denotes the element-wiss product and MLP(-) is the standard multilayer perceptron.
agiic (8) ((s) pddmoott are parameters to adjust the layer norm and they are obtained by
projecting the condition ((c) with W(s) W(s) ((
shin E Rddmedel Xdmadet respectively, where Φ(-)
gale:
scaler
14

---

Published as a conference paper at ICLR 2026
is the activation function. Passing through S AdaLN-MLP blocks, z is used to make the final prediction by:
é Wilinal) AdaLN(z(s)
, sscale 3 3
(13)
(out) scale
W
scale (c))
G W
shift ((c)
where Wiiiiaa) R Pxtmmode and W scale?
Wi E Rdmoded Xdmaaei.
shill
C DERIVATION OF MULTI-MODE INFERENCE ALGORITHM
To leverage the prior knowledge from forward diffusion, we set the following prior at step l:
"Y 1r A
II
II
Ny
p
! I)=mm
m==1
N M
7(" (w*))
II
(w
n=1 m== =Dirichlet( w TT, Te = pm
(14)
<(w")
((Aa)
II
Gamma (A u/: mmm)
mmm
Um
U1 um
(1 (ax)
Note that we use q(-) to denote prior distributions and { u wh ] Jmm without tilde for parameters in prior distributions. p(-) are for posterior distributions and {T) um: vk I for parameters in posterior distributions.
m==
With the above priors, we get the joint distribution q Zt w AA (1 <Y**Z** jut; A (Z) ( (g
Obtaining samples at step k. Y {y In=l our goal is to maximize the marginal log probability max,* In (nn(("+ (1=) and get the posterior distribution (Z², A A popt). where f denotes optimal parameters. It hard to directly optimize the marginal distribution and obtain the posterior as they both contain complex integral terms. Therefore, we introduce variational distribution p w A+) to approximate the posterior through variational inference (Bishop & Nasrabadi 2006). To maximize max,* In gYY* f²*) and approximate posterior qZZ w". A Y 4x") with pZZ w.. A²), we get the following objective:
In KL , (l w w*, A u*1]
max 4* .k
p(Z w A I p(zz w² ar,,At) (00]n1191 [ ( - In g(Z², w A Y ux))
dz #7 dA
max
μ* .
(15)
I p(zz, w AR) ,*, (Yk zt, wh AA(p(t) dz" dw dA* p(Zz, o(Z,,wk.e AAk)
max
ph.ph
C(p², p(z", w², A)))
max p
where 2² is short for pZZ w A+) and C(x², p(zz, w w²))))) is often called Evidence Lower Bound (ELBO) variational inference. Using mean field approximation, we decompose the varia- tional distribution into p(Z w A²) p(Z )p(w")p(A'
Given newly generated samples at step k. Y we maximize C(u², , p(Z*),p(w') p((*), www."(()'"'""""" ).p(A))) w.r. p(Z*), k p( (x), p(A*) one by one. An advantage of fitting an evolving variational GMM alongside the diffusion process is that we can use well-fitted posteriors, ie., p(z++1) p+* p(w ++1 p(A 1+1) at step k + 1 as the initialization of step k. This is because in diffusion, the difference between samples generated in two adjacent steps y and Y is small, and our setting, the difference between the prior distributions of two adjacent steps is also minimal. As result, initialize
15

---

Published as a conference paper at ICLR 2026
p( "",pp(")) p((*), using optimized parameters from last step l as the following:
N M
p(z*) = II
Gk+1y 1mm
Fm
n=1 m=1
1¹th 1
(16)
p(w*) Dirichlet(w (w #h+1)
M p(A)) = II m=1
Gamma (A xk+1 o
E-step: Update p(Z*). Fixing 4º p(w')(())))) p(A)), the objective becomes:
L(u², p(Z)) p(w"),
p(Z²)
[ ZA
p(Z (P(A)) p(Z*)
In ( μ. A Jaz dA
max p(Z*)
(17)
p(Z )p (p(l"((")))) In (Z w)dZ dw / 2 wk KL [(
+ Const
min p(2*)
where Const denotes constant terms w.r.t pZZ),, p(Z²) is a new distribution with: (z*) (Ix 1¹th [] + Boews) q(Z w + Const
In
(18)
Therefore, KL divergence minimized when p(Z*) = p(Zz)):
In ln p
M
E)(A))
Σ
Σ
zm (y t ( Ep(w
Σ
Σ
mum
#
In w Const
(19)
N M
Fmm In Prm + Const
n=1 m=1
where
k+1
In Tno
1ly
+ + In(T)
+ (w - In(
2 Um
(20)
+ (x 20
m
Normalizing it, we get the formulation of p(Z*):
.V.
M
p
p(
Trom
7
I M Σ
(21)
===
m=1
which means that the posterior probability of y belonging to mode m is p(z = 1) Tim-
M-step: Update μ². Note that p(Z*)) has already be updated. Fixing p(Z²), p(w" (AA the objective becomes:
C(p², p(z))(()))))))))))))) ) p(Z))))(w))))(())) ((AA))))))))))
[ A
p(Z )p(A Ing fr A dA + Const
max
(22)
max
Ep(Z) (AF))
zmm I 1 (A -11)
μ²
M
+k+1 m
N
min
Ily
< Unn
m=
1=
16

---

Published as a conference paper at ICLR 2026
Setting the gradient w.r.t p zero, get the updated μ²:
be Tmm n
2 7um
(23)
m mam
M-step: Update p(w*). Note that p p(Z)), nx have already be updated. Similar to E-step, fixing p(Z*), , p(A the objective becomes:
C(H², p(zz) ). p(w")(())))) p(A'))
max p(w*)
J p( (p(w)) In q(2 w )dZ" dw
max p(w*)
(24)
1 p(wh)) g( dow p(w*)) In p(w*) dw + Const
And the log probability of the optimal p(w*) should be:
Inp(w*)
2 [In (Z (Z*1 + In ((w") + Const
M
(25)
mm + Noh 1) In w Const
Therefore, the updated p(w*) is the following Dirichlet distribution:
p( ((w") Dirichlet( w²; *Tt,,, * + Ni
(26)
Update p(A*). Note that p(Z*), 4rth p(w*)) have already be updated. Fixing p(Z²), ), 4h p(w w ), the objective becomes:
M-step:
C(H² p(W")
max P(A))
[ p(Z*))(A In " ) dA
max p(A
(27)
S Aa
((A*) )dA* / (nA*)lnp(A))a )dA + Const
The log probability of the optimal p(AA) should be:
In p(A)
= ) [In q( Y 1 11t AA + In ((A)) Const
(28)
y Ni 1) In A
Fram lly
1 112 A
Const
which means the updated p(A*) k) is the following Gamma distribution:
P(A*)
II
Gamma CaamaaAk wht o
(29)
Nik m
From ly
U
Eq. 26 are the resulting four steps in Algorithm As each of the four steps raises "[Lu"p(Z" p(w")
". p(AA)) the objective gets greater after one iteration. is also possible to perform the four-step iteration multiple times at each step l: to ensure convergence.
17

---

Published as a conference paper at ICLR 2026
D DETAILED SETUP OF EXPERIMENTS
D.1 DATASETS
We conduct experiments on eight datasets following Nie et al. (203); Wu et al. (0023). These
datasets include:
1ETTHl and 2)ETTm1. These two datasets record 7 key indicators of electricity transformer, such as load and oil temperature. ETTh records data points every hour, and ETTml records every 15 minutes. The whole datasets use data from the first 20 months and split it into train/vlidation/test sets with a ratio of 0.6:0.2:0.2.
period of two years and
3)ETTh2 and 4)ET The contents, formats and data split of these two datasets sim- ilar to those of ETThl and ETTmI, but the records are from another electricity transformer.
5T. This dataset contains weather indicators in Beutenberg, including air tempera- ture and dewpoint, with data points recorded every 10 minutes throughout the year 2020. The train/validation/eest sets are split with ratio of 0.7:0.1:0.2.
6)EC.. This dataset records the hourly electricity consumption (in kW) 321 clients from 2012 2014. The train/validation/eess sets are split with ratio of 0:7:0.1:0.2.
7)Traffic This dataset includes road occupancy rates measured by 862 sensors freeways in the San Francisco Bay area from July 2016 to June 2018, with data points recorded every hour. The train/validationnttss sets split with ratio of 0.2.0.
D)nnnii.. This dataset consists of 17 sensors reading and control signals of a sim- ulated complex dynamical system, with data points recorded every second. The orig- inal dataset is from https: www. kaggle. com wttpe////ttttttseptteattaeieeeee calmmicallae
5,000,000 timestamps. For training time concerns, foreaast and contains the first 10% data, which corresponds to 500,000 timestamps, and split it into train/validationntess sets with ratio of 0:7.0.1:0.2. It is worth noting that, despite using only 10% of the original data, the selected timestamps still far exceed those in the previous seven datasets.
The first seven datasets are widely used datasets for TS forecasting (Nie et al. 2023 Wu al. 2023). In line with standard protocol,
the last T 336 steps to predict the next (96, 192, 336, 720] steps. To evaluate model performance in more complex scenarios, we introduce new dataset, Dynamic, which has more complex patterns with obvious periodicity. For Dynamic, we use the last T = 600 steps (10 minutes) to predict the next r (60, 120, 180, 300) steps (corresponding {1, 2, 5) minutes). Additionally, due to the large scale of Dynamic, we divide its test set into non-overlapping windows by setting the sliding window step equal to r. In contrast, for the other datasets, we generate overlapping windows with sliding step of following the common protocol. Detailed statistical characteristics of the used datasets are shown in Table
Table 6: Statistical characteristics of datasets
Dataset
#Channels
#Timestamps
Split
T
Field
ETThl
14,400
0.6:0.2:0.2
336
[96, 192, 336, 720}
Electricity Transformer
ETTml
7
57,600
0.6:0.2:0.2
336
[96. 192, 336, 720
Electricity Transformer
ETTh2
7
14,400
0.6:0.2:0.2
336
(96. 192, 336, 720
Electricity Transformer
ETTm2
7
57,600
0.6:0.2:0.2
336
(96, 192, 336, 720
Electricity Transformer
WTH
21
52,696
0.7:0.1:0.2
336
(96. 192, 720)
Meteorological Indicators
ECL
321
26,304
0.7:0.1:0.2
336
(96, 192, 336, 720)
Electricity Consumption
Traffic
862
17,544
0.7:0.1:0.2
336
(96, 192, 336. 720)
Road Occupancy
Dynamic
17
500,000
0.7:0.100.2
600
(60, 120, 180, 300}
Complex Dynamical System
18

---

Published as a conference paper at ICLR 2026
D.2 METRICS
Top-K MSE. The ground truth target is y E RR, Multi-mode predictions are {ym Jo 1 Vm R
and corresponding probabilities are {wm 1M
mm= Em= Wm 1. Top-K MSE is computed as:
M = Top-K ((um 1M 1ME)
Vm E M, MSE.
11 m yll
(30)
7
Top-K MSE
min MSE
MEM
We first pick the Top-K modes with the highest probabilities into M. Then MSE of each mode in M computed. Finally, the minimum MSE among the selected mode reported as Top-K MSE.
Top-K MAE. The computation of Top-K K MAE is similar Top-K MSE, with MSE changing to MAE:
M Top-K( ((um)) (((m))m))))))
1 Vm E M,MAE -11ym y/11
(31)
Top-K MAE= min MAE
MSE. Given ground truth target E IR and deterministic prediction y E R², MSE is computed as:
MSE
yl y11
(32)
CRPS. CRPS is a frequently used metric for probabilistic prediction and it is originally defined for the scaler variable. Given the ground truth target y E and the cumulative distribution function (CDF) of the predicted distribution Fly), the CRPS is defined as:
+00
CRPS(F.v) v)
(F
1 2 v))
dÿ
(33)
Gneiting & Raftery (2007) show that CRPS can also be computed by:
CRPS(F,0) E. |iū
vll
(34)
where J."" are random variables following the predicted distribution, i..., distribution corresponding to F(y).
Following this formulation, given the ground truth target y RT and samples draw from the probabilistic model ( Y, E RT, we compute the CRPS by:
CRPS,
Σ
Mi. Vel
Σ
T
Jy.l
N
2N
(35)
1 7
CRPS CRPS,
where Vi. E R is the predicted value of timestamp t in sample y;- Ve IR is the target at timestamp t. We approximate the CRPS each step and use the average across steps as the CRPS the whole series.
Remark. CRPS also has the following connection with quantile loss:
CRPS(F 2 22/ Paly Fa'(a)
(36)
(a(u) = w(a - 1(u < 0))
19

---

Published as a conference paper at ICLR 2026
As result, some works (Woo et al. 2024
approximate CRPS, by:
2 K
CRPS;
Pan (yt Q(( (x))
(37)
where (( 0&) computes the oug-quantile at step I. {o=) are some pre-defined quantiles for integral approximation. We do not use this approximation because it will be affected by the selection of {au}k= and the quantile function ((-).
We have described the metrics for a univariate instance. Since all the datasets used are multivariate, we compute the metrics for each channel and then average them across channels to obtain the metrics for multivariate instance. Finally, the metries for dataset are calculated by averaging the metrics across all instances in the test set.
D.3 IMPLEMENTATIONS
The default patch size is set to = 12. For the sake of saving runtime memory, we use 24 at {336, 720} when the dataset is ECL Traffic. As for the Patch Consistent MLP, we use one-block MLP with the dimension hidden states dmndde = 256, adjacent range hyper-parameter 3 on all datasets.
Regarding the training process, we use a linear noise schedule with Ktrain = 1, 000 diffusion steps (Ho al. 2020). The default diffussion-deeemiiistie balancing weight, i.e., À in Eq. 8 is set to 0.99. When using SegRNN as the backbone, it is set to 0.9. Adam optimizer with a learning rate of le-4 is used for optimization. The maximum number of training epochs is
20, and if the validation loss does not decrease over 5 consecutive validations, the training process is terminated early. The commonly used instance normalization (Kim et al. 202I) is also applied reduce the distribution shift.
As for inference, for each input, we generate N 100 samples and set the maximum number of modes to M 10. In our multi-mode inference algorithm, we resample the inference-time diffusion steps to K in 20 and perform 10 EM iterations each step. The syper-parameten are set to 0.5, = 100. For other baselines, we draw N samples from their corresponding distributions and post-process them with GMM the same formulation as in MMPD to obtain multi-mode predictions.
All models are implemented in Pytorch and run on NVIDIA GeForce RTX 3090 GPUs with 24GB memory.
E FULL RESULTS
Due to space limitations in the main text, present the full results from Sec. 4.1 here. Table 7
shows the Top-3 MSE, Top-3 MAE, MSE and CRPS evaluations of different loss functions across various forecasting horizons Mix, and MMPD across three different backbones at different forecasting horizons. Finally, Table 9 presents the MSE and CRPS evaluations across three different backbones different forecasting horizons.
TableeBJdisplaa the Top-3 MSE and Top-3 MAE evaluations for MSE,
20

---

Published as a conference paper at ICLR 2026
Table 7: Full Top-3 MSE, Top-3 MAE, MSE and CRPS evaluations of different loss functions on different forecasting horizons т, which is set to {96, 192, 336, 720} for the first seven datasets and {60, 120, 180, 300} for Dynamic. "Gauss" " is short for "Gaussian". "T" is short for "Student-T" Bold/underline indicates the best/second. Our method is marked in gray. Inf indicates the infinity problem caused by outliers.
Mesic
Tige5 MSE
Tops5
MSE
CRPS
1oss
MSS
MAT:
Gooo
[MMPP]
MSI
MAR
Cars
MSE
MAP
Gees
[MNPD]
MSI
MAR
Mix
|MMP
0.355
500
0:381
0.555
0.571
8:309
0.5MM
01335
0.400
0.351
0.552
0:311
0.374
0,385
0,383
0.377
0.359
0336
0.555
0.283
0.285
0:209
336
0.445
0.450
0:47
04456
6.10
2.115
0.689
0.144
0.444
0.442
0.419
0.633
0.836
0,499
0.490
0:554
0.445
0.433
0.43
0.349
0.300
0.522
6314
0.355
730
0.456
0.501
0.501
0.270
0.510
9.455
0.088
0.486
0.476
0,853
0.456
0.453
0.851
0.489
0,888
0,092
0.511
0.36
0.591
0.365
0.388
0,35
0.390
0.411
0.119
0.130
0:176
0.425
16412
0.333
0,566
0,300
0.11A
0:318
(0255
0.251
0.264
0.15
8.180
0.355
0.336
0.344
0.333
0.202
0:13
0.500
0.90
0,297
0.57
0.303
1.258
0.28
0.291
6:399
0:239
11358
0.331
0,310
0.364
0.315
11303
0356
0.34
1390
1L352
11:33
0.316
0.276
6.27
03291
750
0.441
0.435
0.444
0.436
0.922
0.367
04455
0.815
0.432
0..08
0.50
0.993
0,27
0440
0.500
0.433
0.364
0.338
0.550
0.317
0
0.011
0.313
0.553
0.309
0.315
0.771
0,30
0.700
(0213
0
0.385
33566
0.30
8:241
0.54
0.336
0331
01343
0.328
0310
0.294
0.26
0.905
0.34
0.22
01293
0.364
0.363
0.371
8.361
0.211
0.211
11393
03419
3.40
0.301
0-14
11.11
11413
16435
0305
0.56%
133
0.41
5:001
1L366
11318
0.315
0.320
6316
0.314
6.312
750
0.448
0.111
0.450
0.40
6.30
8.39
0.663
0.133
0.459
0.445
0.55
0.467
0344
0,912
0.400
0,23
0.43
0.350
0.336
0.365
0.582
0.33
0.322
16:
[1:35
0.114
0.1
1100
1:150
B135
0.64
11:53
11.10
16338
0342
0124
0,00
01134
B.19
5190
1.179
0:395
0:31
0:11
6.19
0.91
8,192
ITTT
750
113333
0.353
0
1.30
0.306
0.405
0.365
0.332
1100
6.234
0.113
03912
0.59
11.33
0.397
11385
0.417
16354
0.407
0310
0.367
0.394
1.50
1356
1061
0.396
0.993
5311
0.300
0.399
01318
03377
1L369
0.364
1126:
0.301
0.226
0.310
0 5.00
6.20
0.00
0,255
9.2M
9,20
16
14342
0.155
0.176
0.155
81211
5.100
11:03
11330
14303
010
6315
1180
0181
5,10
510
11000
10133
11:58
0.171
0.172
8.88
0133
WTH
730
14344
11325
0.353
0.226
0.315
0.235
0.356
0.261
0.366
5:541
03564
0:591
11.383
11337
11358
16.98
11291
16335
0.30
0.361
0.554
1.31
1354
0.512
0.591
0.211
0.307
0355
1329
13:41
11313
0.375
11:33
0.233
0.276
0.340
0.26
0.51
6314
10577
1221
1.25
6.11
16
14.40
0.364
0.130
0.133
6119
8:119
0:377
11342
11:34
11230
0.221
0113
6.100
01122
61131
81155
11119
11:33
1.259
0.275
0.167
0.365
100
51136
0:34
0:36
01.30
11241
14331
0.47
0181
01144
8151
16:21
0.380
0.17
8.74
01181
ECL
730
18239
0.222
0.217
0.207
03115
11330
11393
16397
0.71
833
0114
1.30
0.30
0/312
1.100
10358
0.333
0.304
6.11
0:14
0.211
10.7%
96
10.93
0.406
0.382
0.300
0.003
8:300
0.205
0.383
0.666
0.366
0.765
0.37
0.00
0,309
0.388
0.90
0.396
0.335
0.364
3.306
0 100
0131
6.100
8-774
0:39)
01.277
11.14
1.877
0410
41415
11.35
Teall
730
11425
0.216
0:431
0.408
0.462
8.441
03572
11355
11307
16281
035
0.300
1,833
DA5]
D.477
DANS
11465
11444
11335
0.376
0.34
0.21
(1221
6.311
0.51
10.20
0.365
0.334
0.277
0.11
8.179
0.214
0.190
0.355
0.133
0.152
0.160
0.340
0.300
0.355
0.285
0.348
0.355
0.303
0.320
1.165
0.10
0115
6133
0.30
0.503
0.275
02265
01.33
11284
14334
1.14
0.911
5307
11343
0.299
0.17
0336
0.27
Dynark
300
0.452
0.561
0:455
0.500
0410
8.415
0.409
0.394
0.439
0.388
0.308
0.361
0.554
0.554
0,452
0.995
0.615
0.45
0315
0.358
0.266
0 0.0
0,570
AAg
21

---

Published as a conference paper at ICLR 2026
Table 8: Full T MSE and Top-3 MAE evaluations of MSE, Mix and MMPD losses across three different backbones on different forecasting horizons which is set to (96. 192, 336, 720} for the first seven datasets and f60, 120, 180, 300} for Dynamic. Bold indicates the best among three losses. Our method is marked in gray.
Metric
Top 3 MSE
Tup 3 MAE
Backhome
Crosaformer
SeeRNN
MaskAE
Crossformer
SegRNN
MaskAE
Losa
MSE
Mix
MMPD
MSE
Mix
[MMPD]
MSE
Mix
[MMPD]
MSF.
Mix
[MPP]]
MSF
Mix
[MPD
MSE
Mis
MMPD
96
0,365
0.329
0.336
0.575
0.354
0.334
0.314
0.351
0.341
3.406
0.372
0.377
0.413
0.389
0.385
0.407
0.290
0.376
192
0.417
014014
0.421
00.466
0.416
0.503
01437
0.413
0.441
0.448
0.415
0.481
0,,05
336
0.462
0.666
0:664
0.661
0.464
0.459
0.458
720
0.525
0.533
0.500
00.977
0.503
01.99
0.457
0.499
0,452
0.447
0.490
0.464
0.475
0.492
0.471
Avg
0,443
0.433
0.381
0:440
0.435
0.402
0.438
0.415
0.399
0.452
0.433
0.410
00445
0.432
0.426
0.447
0.425
0.416
96
0.2774
0.202
0.248
0.311
0.254
0.250
0.266
0.234
0.207
0.341
0.293
0.307
0.375
0.332
0.331
0.354
0.314
0.299)
192
0.347
0.284
0.269
0.364
0.313
0.301
0.326
0.278
1:240
0.384
0.335
0.333
0.406
0.366
0.365
0.374
0.334
0.314
ETThl
336
730
0,492
0.599
0.337
0.497
0.326
0.345
0.410
0.454
0.347
07425
0.401
0.329
0.442
0.381
0.329
0.417
0.375
0.297
0.455
0.410
0.366
0.437
0.365
0.407
0.455
0.432
0.391
0.424
0.424
0.379
0.405
0.443
0.359
0.406
01401
0.354
Aog
0,378
0.350
0.385
0.335
0.321
00.354
0.514
0.280
0.2997
0.358
0.353
00:117
0.378
0.375
0.389
0.355
0.34
0.301
0.294
0.258
0:210
0.279
0.359
0.352
0.341
0.351
192
0.368
0.355
0.359
00356
0,211
0.341
0-531
0.218
0402
0.394
0.401
0.359
0,279
0.378
0.377
0,271
ET
336
720
0.433
0.386
0.362
0.427
0.327
0.377
0.353
0.429
0.355
0,350
0,14
0.376
0.387
0.425
0.358
0.390
0,222
0.368
0.420
0.468
0.401
0.452
0,377
0.422
0.419
0.457
0.387
0.419
0.733
0.417
0:414
0.448
0.391
0.426
0.374
0.414
Avg
0,372
0.359
0.315
0.365
0.341
0.321
0.335
0,540
0,311
0.412
0:400
0.372
00:408
0.389
0.377
0.392
0.384
0.367
96
0.155
0.150
0.141
0.173
0,161
0.142
0.171
0.191
0.148
0.250
0.247
0.231
0.267
0.252
0.239
0.263
0.250
0.233
192
0.220
0.200
0.196
0.236
0.208
0.235
0.214
0.291
0.314
0.288
0.258
0.308
0.285
0.385
FTTmm
336
0.295
0,391
0.338
0.255
0.334
0.587
0.297
0.267
0.355
0.235
0.306
0.409
0.251
0.542
0.261
0.406
0.346
0.321
0.295
0.413
0.354
0.326
0.399
0.420
0.358
0.321
0.311
Avg
0,266
0.236
0:273
0.248
0.250
0.247
0.325
0.307
00337
0.337
0.308
96
0.134
0.130
0.120
0.140
0.140
0.125
0.140
0.130
0.118
0.189
0.174
0,157
0.198
0.179
0.165
0.190
0.171
0.153
192
0.186
0.168
0.164
0:182
0.189
0.165
0.185
0.364
0.162
0.241
0.223
0.202
0.139
0.234
0.210
0.136
0.207
0.197
weather
720
336
0.327
0.247
0.284
0.217
0.297
0.210
0.327
0:240
0.295
0.233
0.217
0.296
0.246
0.332
0,257
0.200
0,300
0.210
0.347
0.296
0.300
0.252
0,304
0.241
0.788
0.352
0.304
0.766
0,309
0.252
0.284
0.364
0.309
0.764
0.369
0.241
Avg
0.223
0.200
0.197
0.232
0.214
0.201
0.2266
0.201
0.197
0..668
0.235
0.226
0.266
0.246
0.234
0.763
0.238
0.225
0.146
0.150
0.353
0.352
0.347
0.209
192
0.169
0.148
0.168
0.143
0.164
0.139
0.75
0.242
0.73
0.267
ECI.
720
336
0.192
0.229
0.169
0.193
0.186
0:158
0:230
0:100
0.190
0.190
0.184
0.224
0.186
0:156
0.182
0.205
0.327
0.288
0.366
0.279
0.326
0.292
0.280
0.244
0.278
0.319
0.286
0.345
0.275
0.235
0.269
Avg.
0.184
0.160
0.152
0.183
0.155
0.150
0.17%
0.150
0.144
002288
0.255
0.245
0.286
0.245
0.242
0,280
0.241
0.234
96
0.397
0.375
0.367
0.409
0.388
0.365
0.384
0.372
0.345
0.310
0.259
0.248
0.305
0.232
0.242
0.299
0.245
0.235
192
0.436
0.409
0.389
0.440
0.419
0.410
0.418
11400
0.373
(.326
01272
0.253
0.118
0.245
0.251
0.316
0.257
0.244
Traflic
720
336
0.453
0.518
0.483
0.431
0.415
0.463
0.543
0.450
0.516
0.428
0.481
0.509
0.432
0.418
0.476
0.392
0.291
0.341
0.304
0.280
0.260
0.283
0.380
0.378
0.251
0.280
0.255
0.263
0.381
0.321
0.265
0.291
0.253
0.378
Avg
0.451
0.424
0.464
0,430
0.476
0,416
0.342
0.279
0.253
0.329
0:227
0.194
0:278
0.348
0.176
0:230
0.205
0.172
0.207
0.131
0.089
0.212
0
0.213
0.149
0.105
120
0,300
0.297
0.258
0.301
0.332
0,266
0.301
0.301
0,263
0.281
0.218
0.163
0.285
0.252
0,185
0.286
0.224
0.177
Dynami
180
300
0,440
0.357
0.401
0.355
0.429
0.332
P:356
0:445
0.395
033444
0,728
0. 1000
0:365
0.458
01:443
0.437
0.754
0,414
01329
0.399
0.263
0.319
0.224
0.300
0.329
0.403
0. 0.209
0.258
11239
0.308
0.35%
0.410
0.267
0.338
0.299
(1229)
AArg
0,331
0,307
0.295
0,333
0,350
0.245
0.339
11321
0.29
0.304
11,233
0.194
0.307
0.247
0.210
0.312
0.244
0.203
22

---

Published as a conference paper at ICLR 2026
Table 9: Full MSE and CRPS evaluations of MSE, Mix and MMPD losses across three different
backbones on different forecasting horizons which is set to (96, 192, 336, 720} for the first seven datasets and f60. 120, 180, 300) for Dynamic. Bold indicates the best among three losses. Our method is marked in gray. Inf indicates the infinity problem caused by outliers.
Mesric
MSE
CRPS
Backhome
Crosaformer
SeeRNN
MaskAE
Crossformer
SegRNN
MaskAB
Losa
MSE
Mix
IMPPD|
MSE
Mix
NMPP
MSE
Mix
[MPPD]
MSR.
Mix
[MPP
MSF
Mis
[MPP
MSE
Mis
MMPD
96
0,573
0.418
0.381
0.577
0.396
0.374
0.386
0.397
0.383
0.220
0.294
0.293
0.320
0.294
0.296
0.320
0.293
0.291
192
0:419
0.459
0,419
0.422
0.453
0.337
0.315
0.305
0.336
0.330
0.316
0.354
0.312
336
0.455
0,468
0.454
0.454
0.349
0.332
0.346
0.347
720
0.512
0.495
0.456
0.454
00.400
0.487
01.08
0.457
0.370
0.352
0.363
0.363
0.345
Avg
0,440
0,460
0.416
0.433
0,453
0:444
0.437
0.456
0.421
0.344
0.303
0.314
0.334
0.328
0.341
0.319
0.319
96
0.293
03.319
0.345
0.324
0.317
0.332
0.294
01.223
0.255
0.234
0.244
0.266
0.297
0.261
0.273
0.282
0.25%
0.255
192
0.360
0.351
0.355
0.365
0.359
0.3 189
0.385
0.559
0.316
0.307
0.260
0.272
0.315
Inf
0.292
0.500
0.262
0.259
ETTmll
336
730
0.478
0,398
0.375
0.559
0.445
0.409
0.401
10449
0.390
01400
0.380
0.445
0.428
0.315
01448
0.373
0.359
0,405
0.351
0.323
0.250
0.540
0.316
0.295
0.346
0.328
0.321
Ind
0.328
0.301
0.389
0.317
0.276
0.312
0,310
0.285
Avg
0.382
0.399
0.388
0.583
0,381
0.386
0.335
0.571
0,342
0.316
0.261
0.267
0.322
Tnf
0.299
0.310
0.277
0.277
96
0.332
0.326
0.329
0.312
0.320
0.319
0.288
0.338
0.296
0.285
0.286
0.258
0.278
0.275
0.25
0.260
0.281
0.265
192
0.399
0.394
0.579
0.388
0.364
0.394
0.316
0.520
0.313
0.312
0.2%6
0.306
ETTh2
720
0.425
0,472
0.425
0.417
0.430
0.303
0.418
10435
0.410
0.324
0.351
0.356
0.330
0.346
0.345
0.321
0.329
0.340
0.338
0.317
0,331
Avg
0.365
0.403
0.374
0.376
0.375
0.376
0.366
0.504
0.362
00319
0.303
0.316
00314
Tnf
0.313
0.303
0.305
96
0.166
0.177
0.174
0.185
0.182
0.175
0.184
0.182
0.183
0.197
0.300
0.195
0.211
0.197
01200
0.217
0.196
0.200
192
0.234
0.226
0.241
0:247
0.241
0.265
0.248
0:257
03256
0.228
0.225
0.231
0.243
Tef
0.244
0.259
0.231
0.344
ETT12
336
730
0.387
0.299
0.297
0.391
0.391
0.272
0.300
0.383
0.324
0.391
0.307
0.397
0.405
0.310
0.339
01406
0,363
0,382
0.309
0.264
0.200
0,308
0.251
0.518
0.270
0.313
0.303
Tar
0.517
0.261
0.273
0.318
0.309
0.262
0.318
0.270
Avg
0.271
0.273
0.270
0.279
0.285
0.285
0.287
0.291
0,281
0.249
0.248
0.249
0.259
Ind
0.257
0.259
0.250
0.258
0.144
0.156
0.170
0.149
0.172
Taf
0.168
0.148
0.196
0.209
0,188
0.249
0.194
0.211
0.002
0.186
0.198
0,192
0.199
WTH
730
336
0.250
0.322
032663
0.336
0.376
0.241
0.323
00:05
0.368
0, 340
0.328
0.347
0.530
0.377
0,333
(1238
0:379
0.255
0-219
0.263
0.281
0332
0.240
Tnf
0,227
0.267
0.279
0.233
0.251
0.376
0,216
0.261
Avg
0.228
0..441
0.232
0.236
0,273
0.232
0.230
01.70
0.251
0.222
0.202
0.204
0.222
lnc
0.210
0.220
lnt
0.201
96
0.135
0.147
0.138
0.134
0,152
0.135
0.130
0.137
0.130
[.2661
[nf
0.170
0.260
0.170
0.258
0.166
0.166
192
0.156
0.165
01544
13155
0,171
0154
0.149
0.156
01447
(1288
Ind
0.186
0.267
0.83
0.265
0.179
0.178
ECL
730
336
0.176
0.210
0.195
0.222
0.187
0.208
0.37
0.210
0.251
0.104
0172
0.206
0.204
0.166
0.173
0.214
0.164
0.202
(.291
0.376
Inf
Tnf
0.209
0.224
0.375
0.291
Tef
0.196
0.222
0.273
0.789
0.192
0.238
0.216
0.191
Avg
0,170
0.182
0.170
0.168
0.187
00112
0.170
0.274
End
0.197
0.275
Lnc
0.193
0.271
0,194
0.188
0.386
00:27
0.425
0.359
0.205
0.317
0.935
0.180
0.412
0.443
0:412
0.646
0.385
0.427
0.344
0.341
0.199
0.341
Traffic
730
336
0:423
0.454
0.460
0.507
0.431
0:47
0.463
0.536
0.402
0.466
13442
0.488
0.440
0.34%
0.362
0.216
0.234
0.207
0.223
0.358
0.344
Tnf
lsc
0,207
0.222
0.358
0.342
012017
0.226
0.201
0.218
Arg
0.418
03459
0.420
0.428
0,468
0.421
0.403
01441
0,404
0.344
0.217
0.208
0.345
lac
0.205
0.344
0.208
01201
60
0.238
0.326
0.237
0.238
0.367
0.238
0.241
0.335
0.245
0.200
Emf
0.119
0.202
0.172
0.130
0.204
0.149
0.132
120
0.311
0.432
0.308
0.313
0.454
0.311
0.316
1.441
0.316
01237
Ind
0.171
0.239
Inc
0.187
0,242
0.204
0.183
Dynomic
300
180
0.366
0.441
0.493
0.570
0.172
0.360
0.446
0.300
0.565
0.474
0.368
0.446
0.462
0.376
0.511
0.635
0.458
0.570
0.307
0.366
Tnf
Ind
0.271
0.215
0.310
0.368
Tef
Inc
0.225
0.274
0.273
0.317
0.341
0.293
0.222
0.270
Avg
0.339
0.455
0.542
0,465
0.349
0.476
0.253
Inf
0.255
0.259
23

---

Published as a conference paper at ICLR 2026
F ADDITIONAL EXPERIMENTS ON HYPER-PARAMETERS
Noise Schedule. In our main experiments, use the linear schedule as default. Ta- ble TO] further evaluates two advanced sched- ules: Quadratic (Kong & Ping 2021) and Co- sine (Nichol & Dhariwal 202 ). Results show that MMPD benefits from these advanced sched- ules. This indicates that developing time-series- specific schedules is promising direction, since time series and image data have fundamentally different characteristics.
Table 10: Metrics comparison of different noise schedules on ETTh1 (T 336, r 96). The de- fault schedule used in main experiments is marked in gray.
Top-3 MSE
Top-3 MAE
MSE
CRPS
Quadratic 0.314
0.362
0.379
0.286
Cosine
0.317
0.364
0.382
0.289
Linear 0.329
0.371
0.375
0.289
Table 11: Top-3 MSE/MAE evaluations versus varying maximum number of modes M in Algo- r on ETTh1 (7 336, T = 96). The default setting in main experiments is marked in gray.
Diffusion Steps in Training. Fig. demon- strates that increasing the number training diffusion steps Krain significantly improves Top-3 MSE, Top-3 MAE and MSE. Notably, this enhancement comes without computational overhead during inference, as we employ resam- pling with fixed inference steps fer- For our experiments, we adopt Ktrain = 1, 000 as the default setting.
5
15
25
T MSE 0.3200 0.3197
0.3200 0.3199
0.3203
Top-3 MAF 0.3667 0.3667
0.3668 0.3667 0.3668
Table 12: T MSE/MAE evaluations versus varying mixture weights prior hyperparameter in Algorithm T on ETThl (T' 336, T 96).
Diffusion Steps in Inference. Fig. g shows that increasing the number of inference diffu- sion steps Kinfer consistently improves predic- tion accuracy. However, as revealed by our ef- ficiency analysis in Table this improvement comes with increased computational overhead. To achieve a balance between accuracy and ef- ficiency, we set Kinfer 20 as our default. configuration.
0.1
0.3
0.5
0.7
0.9
Top-3 MSE 0.329
0.322
0.320
0.314
0.307
T MAE 0.377
0.378
0.367
0.358
0.354
Table 13: Top-3 MSE/MAE evaluations versus varying variance prior hyperparameter in Algo- rithm 1 on ETTh1 (T = 336, 7 = 96).
Anchor step k- in Eq. 8 Fig. 9 shows that overly large .ة. harms multi-mode and prob- abilistic prediction, while small ones degrade deterministic prediction. The performance is ro- bust across a broad range around og- = 0.5. To maintain a simple formulation of Eq. g we choose k' to make dgg- close to 0.5.
0.001
0.01
0.1 1 10.
100
1000
Top-3 MSE 0.322 0,321 0.323 0,323 0,318
0.320
0.314
Top MAE 0.372 0.372
0,372 0.361
0.367
0.364
Maximum number of modes M in Algorithm T. Table shows that the multi-mode accuracy remains robust w.r.t M. This arises from using variational GMM rather than a standard one. In this formulation, only the maximum number of modes needs to be specified, while the inference algorithm automatically determines the appropriate number of active modes. This behavior is illustrated in Fig. 3 c), where we set M = 10, but only 3 modes are activated after inference.
Mixture weights prior hyperparameter p in Algorithm 1 P controls the prior over the number of activated modes-a larger P encourages utilizing more modes. Table 12 shows that larger leads to slightly lower Top-3 MSE/MAE, but activating too many modes may confuse downstream users.
Variance prior hyperparameter u in Algorithm I u influences the prior over the variance, where a larger value reflects greater confidence in the variance estimated from forward diffusion. As shown in the Table 13 below, performance remains robust.
G DETAILED FLOPS ANALYSIS
FLOPs of MSE Loss. MSE loss uses conventional MLP to project latent tokens into future series. A conventional S-block MLP consists of three components:
One linear input layer with ((d) FLOPs;
24

---

Published as a conference paper at ICLR 2026
1330
4370
337
Triding New
Tuaarn Oron
Trandin Kpa
Traddin
(h)
(d)
Figure 7: (a) Top-3 MSE, (b) Top-3 MAE, (c) MSE, (d) CRPS evaluations versus varying diffusion steps in training (Kerain) ETThl (I 336, 96). The Linear noise schedule is used and the diffusion steps in inference are set to Kinfer = 20.
0311
0270
030
(a)
(c)
Figure 8: (a) Top-3 MSE, (b) Top-3 MAE, (c) CRPS evaluations versus varying diffusion steps in inference (nrr).). Kerain = 1, 000 and other settings are same as Fig. MSE omitted as Kinfer does not affect deterministic prediction.
0.310
Tap-1
Top-3
CRPS
0.300
0.000
0.67
0.22
0:43
0.300
0.33
0.25
0.00
05
0.7
0.5
0.509
0100
035
0.999
(a)
(b)
Figure 9: (a) Top-3 MSE & Top-3 MAE, (b) MSE & CRPS evaluations versus varying .g.. in Eq. on ETThl (T' 336, т = 96).
S hidden blocks, each has two linear layers and one activation layer between them. The FLOPs are (22?);
One linear output layer with O(Pd) FLOPs;
Therefore, the FLOPs for a conventional MLP applied to one token is O ((2S + 1)d2 + Pd). To predict a series of length т, this MLP is simultancously applied to 7/P tokens, making the total FLOPs FMLP = O (2S 1)d² Pd]).
For both training and deterministic inference, MSE loss requires one backbone forward pass and one MLP forward pass, the FLOPs are FosS FMLP. It should be noted that MSE loss cannot perform probabilistic or multi-mode prediction, marked with "N/A" Tables
FLOPs of MMPD Loss. MMPD loss uses Patch Consistent MLP as the denoising network, which is slightly more complex than the conventional MLP above. A S-block Patch Consistent MLP consists of three components:
Input operations in Eq. requiring O + (2r 1)Pd) FLOPs;
S AdaLN-MLP blocks in Eq. 12 each requires ((5d²) FLOPs;
One AdaLN output block in Eq.
13 requiring O(2d² + Pd) FLOPPs:
Same as MSE, the Patch Consistent MLP is simultaneously applied to T tokens, so the total FLOPs FPC-MIA MIP O [ 3)d (2r Pd)).
25

---

Published as a conference paper at ICLR 2026
As for EM steps in Algorithm|! [ three vector operations contribute most FLOPs and other scalar operations are negligible. The three vector operations are: 1) Computation of In Ymm requires M(3 2) p. requires O(2TMN); 3) eih requires O(3rMN) So one EM step has the FL.OPs of FEM = MN).
For training, MMPD Loss requires one backbone forward pass and two Patch Consistent MLP passes: one for the diffusion objective and another for the deterministic objective in Eq. So the training FLOPs are FoRs + 2FPC-MP.. MLP. Considering the backbone dominates training cost (i.e., FiNs >> FML.,, FPC- MLP), training FLOPs of MMPD remain nearly identical to those of MSE.
Similar to MSE, deterministic inference of MMPD loss requires one backbone forward pass and one Patch Consistent MLP forward pass, so the FLOPs are Fixs + FPc- PLP, differences only lic in MLP architectures. For probabilistic and multi-mode predictions, MMPD needs generate N samples, each through K diffusion-EM EM iterations. So the FLOPs are FOX. + KNN FNFFrC-M Frc-MLI MLP + FEM) As FEM << FPC-MLPP we omit it in Table 4 for simplicity. To generate one instance, MMPD loss only requires a single backbone pass, followed by K lightweight MLP passes, making its cost significantly lower than that of TS Diffusion models requiring backbone passes.
H FURTHER DISCUSSION: EXTENDING MMPD TO NON-PATCH-BASEE BACKBONES
The latent representations H extracted by non-patch- based backbones are not naturally expressed as (h,) thus MMPD loss cannot be directly applied. As illustrated in Fig. [0] we address this by inserting a single Transformer decoder layer: H serves as key and value, while learnable tokens indicating prediction patches act as query. This decoder layer transforms H into tokens suitable for MMPD loss.
MMPD Loss
H
Non-patch-based Backbone
Transformer Decoder Layer
potches to predict
Using this adaptation, we apply MMPD to following non-patch-basee backbones: 1) TSMixer (Chen et al.) 2023). fully MIP model without patching: 2)iTrans- former (Li et al., 2024b), non-patch- patch-based Trans- former designed to model inter-channel dependencies. Table 14 shows that our adaptation with MMPD loss performs on par with the original architectures using MSE loss in deterministic forecasting, while signifi- cantly outperforming them in multi-mod and proba- bilistic forecasting. This demonstrates the effective- ness of our adaptation and highlights the potential of applying MMPD to a broader range of backbones.
Figure 10: Adapting non-patch-basee back- bones use MMPI) loss: a single Trans- former decoder layer is inserted between the backbone and MMPD. In this layer, back- bone output H serves as key and value, while learnable tokens indicating patches to predict act as the queries. The decoder output is then used for MMPD loss computation.
Table 14: Comparison of non-patch-based backbones (TSMixer and iTransformer) trained using MSE loss and adapted MMPD loss on datasets CTThh/ETTm//WTT T 336, T 192.
Dataset
ETTh1
ETTml
WTH
ETThl
ETTml
WTH
ETTh
ETTml
WTH
ETTml WTH
Metric
Top-3 MSE
Top-3 MAE
MSE
CRPS
TSMixer
MSE
0.384
0.285
0.140
0.415
0.353
0.198
0.390
0.306
0.149
0.323
0.289
0.169
MMPD
0.367
0,230
0.118
0.396
0.309
0.161
0.378
0.306
0.148
0.304
0,260
0.150
iTranformer
MSE
0.390
0.298
0.147
0.414
0.361
0.200
0.400
0.317
0.158
0.324
0.292
0.173
MMPD
0,361
0.279
0.132
0.395
0,326
0.180
0.386
0.334
0.170
0.304
0.278
0.165
I FURTHER DISCUSSION: EXTENDING MMPD TO MULTI-TASK LEARNING
Beyond the traditional single-dataset paradigm, we further extend the MMPD loss to multi-task learning, where unifted model is trained across multiple datasets and settings. Such model can directly perform multi-task forecasting and be adapted to few-shot or zero-shot forecasting on new datasets.
26

---

Published as a conference paper at ICLR 2026
Table 15: Multi-task forecasting comparison of UNITS pretrained with MSE and MMPD loss on 20
forecasting tasks.
Metric
MSF
MAE
Top-3 MSR
Top-3 MAR
CRPS
Pretraining Loss
MSE
MMPD]
MSE
MMPD
MSE
MMPD
MSE
MMPD
MSE
MMPD
NNS
0.618
0.595
0.551
0.530
0.623
0.572
0.559
0.512
0.453
0.390
ECL-9
0.168
0.161
0.270
0.260
0.178
0.146
0.285
0.241
0.271
0.189
EC-192
0.184
0.175
0.283
0.272
0.195
0.163
0.301
0.256
0.280
0.201
ECL-336
0.203
0.192
0.300
0.289
0.203
0.174
0.313
0.270
0.285
0.211
ECL-720
0.242
0.229
0.331
0.318
0.234
0.215
0.338
0.302
0.300
0.234
ETTh1-96
0.397
0.367
0.420
0.4(0)
0.377
0.325
0.413
0.367
0.334
0.286
ETTh1-192
0.438
0,408
0.448
0.427
0.433
0.386
0.443
0.404
0.353
0.312
ETTh1-336
0.465
0,442
0.465
0.448
0.456
0.436
0.458
0.425
0.365
0.325
ETTh1-720
0.507
0.472
0.500
0.478
0.513
0.474
0.496
0.464
0.394
0.347
Exchange-192 192
0.261
0.225
0.364
0,343
0.223
0.160
0.338
0.279
0.318
0.274
Exchange-336
0.464
0,415
0.494
0.469
0.349
0.306
0,429
0.393
0.439
0.372
ILI
2.073
2.345
0.895
0.967
2.045
1.981
0.894
0,855
0.739
0.785
Traffic-96
0.475
0,446
0.314
0.288
0,483
0.414
0.343
0.265
0.352
0.211
Traffic-192
0,484
0,460
0.314
0.292
0.477
0.426
0.336
0.268
0,350
0.211
Traffic-336
0.498
0.477
0.319
0.299
0.519
0.465
0.349
0.280
0.357
0.219
Traffic-720
0.532
0.510
0.336
0.315
0,545
0.507
0.359
0.299
0.361
0.233
Weather-96
0.163
0.166
0.214
0.213
0,149
0.136
0.203
0.173
0.176
0.166
Weather-199 192
0.212
0.213
0.257
0.254
0.178
0.180
0.236
0.220
0.212
0.202
Weather-336
0.267
0.270
0.297
0.294
0,269
0.245
0.268
0.267
0.251
0.241
Weather-720
0.344
0.353
0.347
0.345
0.323
0.328
0.317
0.327
0.300
0.287
Winning Counts
5/20
15/20
1/20
19/20
2/20
18/20
1/20
19/20
1/20
19/20
Table 16: Few-shot forecasting comparison of UNITS tuned with MSE and MMPD loss on new datasets. For each setting, only 5% of the training set is used for prompt-based tuning.
Metric
MSE
MAE
Top-3 MSE
Top-3 MAE
CRPS
Prompt Tuning Loss
MSE
MMPD
MSE
MMPD
MSE
MMPD
MSE
MMPD
MSE
MMPD
ETTh2-99
0.409
0.377
0.415
0.403
0.382
0.341
0.409
0.374
0.326
0.292
ETTh2-192
0.380
0.381
0.398
0.403
0.365
0.335
0.389
0.373
0.343
0.318
ETTh2-336
0.436
0.447
0.437
0.445
0.415
0.397
0.431
0.413
0,361
0.347
ETTh2-720
0.449
0.445
0.454
0.453
0.431
0.420
0.458
0.447
0.382
0.361
SaugeenRiverloow
1,270
1.248
0.576
0.569
1.128
1.088
0.510
0.485
0.543
0.480
Winning Counts
2/5
3/5
2/5
3/5
0/5
5/5
0/5
5/5
0/5
5/5
Table 17: Zero-shot forecasting comparison on new datasets of pretrained UNITS using MSE and
MMPD.
Metric
MSE
MAE
Top-3 MSE
Top-3 MAE
CRPS
Pretraining Loss
MSE
MMPD
MSE
MMPD
MSE
MMPD
MSE
MMPD
MSE
MMPD
Solar
0.202
0.169
0.320
0.299
0.207
0.095
0.331
0.193
0,272
0.187
River
2.336
2.294
0.735
0.746
2.298
1.722
0,733
0.722
0.643
0.624
Hospital
1.115
1.175
0.818
0.834
0.998
0.860
0.772
0.705
0.690
0.622
Winning Counts
1/3
2/3
2/3
1/3
0/3
3/3
0/3
3/3
013
3/3
We evaluate this idea using UNITS (Gao et al. 2024), unified multi-task model that integrates multiple tasks within a single framework. Since our focus is solely on forecasting, we adopt the supervised variant (UNITS-SUP without incorporating classification techniques. As patch-based model originally trained and tuned with MSE loss, MMPD loss can be integrated into it with minimal changes to enhance its ability to capture complex distributions.
Multi-task Forecasting, Following Gao et al. (2024), we first conduct multi-task supervised pretraining on 20 forecasting tasks using either MSE or MMPD loss. Results in Table 15 show that for deterministic forecasting measured by MSE/MAE, MMPD loss matches or improves performance over MSE. A possible reason is that MMPD provides a more challenging objective that encourages richer representations, which are especially beneficial in multi-task scenarios. For multi-mode and probabilistic forecasting measured by Top-3 MSE/MAE and CRPS, UNITS trained with MMPD significantly outperforms its MSE-trained counterpart. This demonstrates that MMPD integrates well with UNITS and effectively enables the modeling of complex distributions.
27

---

Published as a conference paper at ICLR 2026
Few-shot Forecasting. We then tune the pretrained models with MSE and MMPD loss to perform few-shot prediction on new datasets. each setting, only 5% of the training data is utilized and the crompt-basete tuning from Gao et al. (2024) is used. As shown in Table T6 MMPD matches MSE in deterministic accuracy while consistently outperforming it in probabilistic and multi-mode forecasting.
Zero-shot Forecasting, Finally, Table 17 evaluates zero-shot forecasting capabilities of models pretrained with MSE and MMPD, Results consistent with the few-shot setting: MMPD maintains comparable deterministic accuracy while offering superior probabilistic and multi-mode performance.
28

---

Published as a conference paper at ICLR 2026
J
VISUALIZATIONS
MSE Loss
MMPD Loss
57.45%
55%%
324
0.00
56.25%
43.75%
$-1
-1.5
-.75
-2.75-
-1.0
-1.00-
-122
-1.25
-150
-1.00
V0.11
243
200
204
6:07
497
306
417
Prob: 611 61.22%
Prob: 36 38.70%
1.8-
ECL #91
1.0
Dynamic #12
-TPrat: 23.T1
9Ext
ha
EEN Cevieeene TOOTE
Dynamic #6
Plito 73, 0000
Prot: 27 0050
Figure 11: Forecasting cases of MSE loss MMPD loss. Each represents instance, with the dataset name and channel number indicated on the left. Predictions from MSE and MMPD losses separated by thick vertical dashed line. On the left, one-mode predictions generated by the MSE loss are shown, along with the corresponding input series. On the right, multi-mode predictions generated by the MMPD loss are displayed, with their corresponding probabilities shown the top.
K DECLARATION OF LARGE LANGUAGE MODELS USAGE
In this work, large language models were used solely for word choice and language polishing. They did not contribute to research ideation, methodology or analysis.
29


Reviews:
[{"review_id": "sLAXZagObJ", "reviewer": "Reviewer_7ixF", "summary": "This paper addresses the challenge of diverse forecasting in time series analysis, where multiple plausible future outcomes exist rather than a single deterministic forecast. It introduces MMPD loss, a diffusion model-based probabilistic loss function that can be integrated with various patch-based forecasting backbones. The experiments on eight benchmark datasets and comparisons with several baselines demonstrate MMPD’s good performance in diverse, deterministic, and probabilistic forecasting.", "questions": "See above", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Introduces a diffusion-based loss capturing multi-modal future distributions, addressing fundamental limitations of traditional single-mode losses.\n2. Can be applied to any patch-based forecasting backbone, making it flexible for different architectures.\n3. Effectively generates multiple sharp predictions with corresponding probabilities, enabling risk-aware decision-making.\n4. The proposed AdaLN-MLP accounts for adjacent patches, reducing inconsistency and unrealistic discontinuities in forecasts.", "weaknesses": "1. Multi-mode inference requires multiple samples and iterative GMM fitting, which may introduce latency for real-time applications.\n2. The diffusion-based loss and multi-mode inference algorithm add complexity to the training and prediction pipeline.\n3. How robust is MMPD to different patch sizes and ranges of adjacent patch conditions in the Patch Consistent MLP?\n4. Can MMPD be extended to explicitly handle multivariate forecasting jointly rather than per-channel averaging?\n5. How does MMPD perform under non-stationary or concept drift conditions in time series data?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ktugXFrlte", "reviewer": "Reviewer_XDPi", "summary": "This paper addresses the limitation of MSE-based training in time series forecasting, which assumes unimodal Gaussian distributions and fails to capture multiple plausible future outcomes. The authors propose to use a diffusion model conditioned on backbone latent tokens to model multi-modal future distributions. A Patch Consistent MLP serves as the denoising network, and a variational GMM-based inference algorithm generates diverse predictions with probabilities.", "questions": "1.  During training, you optimize diffusion loss on random (y^k, k) pairs. During inference, you fit GMM by assuming samples follow Eq. 10. Is there a theoretical or empirical guarantee that the learned diffusion model produces samples consistent with this GMM structure?\n\n2. The diffusion variance schedule is critical. Do you use standard schedules from image diffusion, or is adaptation needed for time series?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The paper tackles the restrictive unimodal Gaussian assumption inherent in MSE loss, which is a genuine problem for real-world time series where identical inputs can lead to multiple plausible futures due to unobserved contextual factors.\n\n2. The work provides clear motivation from both data perspective and application perspective, making the contribution practically valuable.\n\n3. The proposed MMPD loss can be applied to any patch-based forecasting backbone without architectural changes, and the generality is validated across four different backbones, demonstrating broad applicability.", "weaknesses": "1. While the paper compares against MSE and Student-T, it lacks comparison with more recent and sophisticated probabilistic forecasting methods (e.g., recent normalizing flows, autoregressive diffusion models, or transformer-based probabilistic forecasters), making it difficult to assess where this stands in the current landscape.\n\n2. Despite claims of being \"lightweight,\" the method requires iterative diffusion steps K during training, generating N samples and fitting GMM at each reverse step during inference, and storing and processing adjacent patches. No concrete analysis of training time overhead, inference latency, or memory usage compared to MSE baseline is provided.\n\n3. The used GMM assumptions may be too restrictive. While GMM is more flexible than single Gaussian, it still assumes modes follow Gaussian distributions with spherical covariance. Real multi-modal distributions may have asymmetric modes, correlated dimensions, or non-Gaussian shapes, limiting the method's ability to capture truly complex distributions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jYGEW3NUeX", "reviewer": "Reviewer_hVy7", "summary": "This paper reframes time-series forecasting by making the training objective itself a conditional diffusion model. A patchified forecaster produces future latent tokens, and a lightweight, cross-patch Patch-Consistent MLP learns to denoise them, capturing inherently multi-modal futures. At inference, the method does not rely on ad-hoc sampling; instead, it tracks modes through the reverse process with a stepwise variational GMM, yielding a small set of trajectories with calibrated probabilities. The approach is plug-and-play for patch-based backbones and includes a direct deterministic path without sacrificing efficiency. Across eight benchmarks, it consistently improves Top-K diversity metrics while remaining competitive—or better—on standard MSE (point forecasting) and CRPS (probabilistic accuracy).", "questions": "See W1 to W3.", "limitations": "", "rating": 8, "confidence": 2, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "- **S1**: The paper distills a clear limitation of point-estimation training and turns the objective into a flexible distribution model, which is highly instructive for time-series forecasting research. Importantly, the authors make an effort to generalize beyond patch-based designs, discussing (and partially validating) how the idea can extend to non-patch backbones—thus increasing the method’s relevance to a wider set of architectures.\n\n- **S2**: The empirical study spans diverse datasets, multiple backbones, and both deterministic and probabilistic metrics, complemented by ablations and efficiency analyses. This breadth and depth make the conclusions credible.", "weaknesses": "- **W1**: Please consider releasing the training/evaluation code and experiment scripts (including configs, seeds, data preprocessing, and checkpointing details). This would enable exact reproduction of tables/figures and facilitate fair downstream comparisons.\n\n- **W2**: Recent LLM-style time-series methods also employ patching. Adding representative LLM-based patch baselines would strengthen the empirical positioning and clarify where MMPD stands relative to these emerging approaches.\n\n- **W3**: The paper states “We focus on univariate forecasting; for multivariate data, the loss is computed per channel and averaged.” Can author discuss how MMPD handles inter-channel correlations.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4SCbsbDRJ8", "reviewer": "Reviewer_kPh3", "summary": "The paper proposes MMPD as a training objective: it turns the loss into a conditional diffusion process. Given future latent tokens produced by an upstream patch-based backbone, a lightweight Patch-Consistent MLP performs denoising to model complex, multi-modal future distributions; at inference, a progressively-evolving variational GMM extracts several plausible futures along with their probabilities.", "questions": "See W1 and W2. If the author can address my concerns, I will consider increasing my score.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "- **S1** Solid work with complete design and experiments: from motivation and method (loss = diffusion + Patch-Consistent MLP) to inference (evolving variational GMM) and ablations/efficiency analysis, the pipeline is very clear and well executed.\n- **S2** Motivation is important and well argued: the paper clearly articulates that MSE-centric training implicitly assumes a unimodal Gaussian with fixed variance, which struggles to cover real-world multiple possible futures and time-varying/asymmetric uncertainty. The authors cleverly interpret MSE through the lens of a distributional assumption and then extend it to a higher-level, more expressive formulation.", "weaknesses": "- **W1** Baseline coverage is incomplete: while comparisons include point-estimation and certain probabilistic/mixed-distribution schemes, the work does not compare against D3U[1], which follows a related paradigm of decoupling deterministic and uncertain components.\n- **W2** Patch-based pipelines typically introduces padding. For these non-real-value patches, it is unclear whether the Patch-Consistent MLP applies an explicit distinction been made. Moreover, padding patches lack a valid $\\text{next}_{k_j}$ to model. The paper should provide further implementation details and experiments clarifying how padding patches are treated.\n\n[1]. Li, Q., Zhang, Z., Yao, L., Li, Z., Zhong, T., & Zhang, Y. (2025). Diffusion-based decoupled deterministic and uncertain framework for probabilistic multivariate time series forecasting. In The Thirteenth International Conference on Learning Representations.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)