Dynamical Diffusion: Learning Temporal Dynamics with Diffusion Models
Authors: Xingzhuo Guo
Yu Zhang
Baixu Chen
Haoran Xu
Jianmin Wang
Mingsheng Long
Venue: ICLR 2025 Poster (2025)
Keywords: Diffusion Model
Generative Model
Prediction Learning
Dynamics
Primary area: generative models
TL;DR: We propose Dynamical Diffusion, which incorporates temporally aware forward and reverse processes and improves performance in general temporal predictive tasks.

Abstract: Diffusion models have emerged as powerful generative frameworks by progressively adding noise to data through a forward process and then reversing this process to generate realistic samples. While these models have achieved strong performance across various tasks and modalities, their application to temporal predictive learning remains underexplored. Existing approaches treat predictive learning as a conditional generation problem, but often fail to fully exploit the temporal dynamics inherent in the data, leading to challenges in generating temporally coherent sequences. To address this, we introduce Dynamical Diffusion (DyDiff), a theoretically sound framework that incorporates temporally aware forward and reverse processes. Dynamical Diffusion explicitly models temporal transitions at each diffusion step, establishing dependencies on preceding states to better capture temporal dynamics. Through the reparameterization trick, Dynamical Diffusion achieves efficient training and inference similar to any standard diffusion model. Extensive experiments across scientific spatiotemporal forecasting, video prediction, and time series forecasting demonstrate that Dynamical Diffusion consistently improves performance in temporal predictive tasks, filling a crucial gap in existing methodologies. Code is available at this repository: https://github.com/thuml/dynamical-diffusion.

Published as a conference paper at ICLR 2025
DYNAMICAL DIFFUSION: LEARNING TEMPORAL DYNAMICS WITH DIFFUSION MODELS
Xingzhuo Guo; Yu Zhang; Baixu Haoran Xu, Jianmin Wang, Mingsheng Long School of Software, BNRist, Tsinghua University, Beijing 100084, China {gx223, zhangyu24, cbx22, xu-hr22 (emails. werllss.sinnghnn tsinghua. edu.cn [ liwwang, mi mingsheng] @t s nuuuaa edu.cm cn
ABSTRACT
Diffusion models have emerged as powerful generative frameworks by progres- sively adding noise to data through a forward process and then reversing this process to generate realistic samples. While these models have achieved strong performance across various tasks and modalities, their application to temporal predictive learning remains underexplored. Existing approaches treat predictive learning as a conditional generation problem, but often fail to fully exploit the temporal dynamics inherent in the data, leading to challenges in generating tem- porally coherent sequences. To address this, we introduce Dynamical Diffusion (DyDiff), theoretically sound framework that incorporates temporally aware for- ward and reverse processes. Dynamical Diffusion explicitly models temporal transitions each diffusion step, establishing dependencies on preceding states to better capture temporal dynamics. Through the reparameterization trick, Dy- namical Diffusion achieves efficient training and inference similar to any standard diffusion model. Extensive experiments across scientific spatiotemporal forecast- ing. video prediction, and time series forecasting demonstrate that Dynamical Diffusion consistently improves performance in temporal predictive tasks, fill- ing a crucial gap in existing methodologies. Code available at this repository: https: /ttps:///gttuut www.tommmmc...cee /dynamiiii ca d-di fuusion
INTRODUCTION
Diffusion models (Sohl-Dickstein et al., 2015)] Song & Ermon 2019 Ho et al. 2020: Song et al, 2020) refer to a class of generative models that progressively corrupt data by adding noise through a "forward process" and then iteratively denoise a random input during inference to generate highly realistic samples via the "reverse process", This unique approach has positioned them powerful alternative to traditional generative methods. To date, diffusion models have demonstrated strong performance across a wide range of tasks (Kingma et al. 202T Saharia al. 2022a Dhariwal &
Nichol 202T) and data modalities (Kong et al. Chen et al. 2021 Yang et al. 2023 Ho et al.]
2022 Blattmann et al., 2023a).
Due to their strong capability to model data distributions, diffusion models are gaining attention in the field of temporal predictive learning. Several recent studies (Voleti et al., 2022; Gao et al., 2023) have explored the application of diffusion models to predictive learning tasks by reinterpreting these tasks as conditional generation problems. In this approach, the model trained to predict the future conditioned on historical data, such as predicting the next video frame based preceding frames. Despite yielding promising results, these methods did not explicitly leverage the temporal nature of the data, which may pose challenges for generating temporally coherent sequences (Blattmann al., 2023abb b). While increasing the capacity of deep models can alleviate this issue, the fundamental challenge of integrating temporal dynamics into diffusion processes remains underexplored.
To this end, we propose Dynamical Diffusion (DyDiff), a framework that defines temporally aware forward and reverse diffusion processes. Specifically, in the forward process, each latent is not only modified through the conventional noise addition procedure but is also derived from its temporally
"qqual contribution.

---

Published as a conference paper at ICLR 2025
preceding latent. In this way, Dynamical Diffusion explicitly captures temporal transitions each diffusion step. Through a theoretical derivation, we establish the existence the corresponding reverse process and extend it to generate multi-step predictions simultaneously. By leveraging the reparameterization trick, the learning of Dynamical Diffusion is formulated into feasible optimiza- tion objectives. This enables efficient training with no additional computational cost compared to standard diffusion models and facilitates efficient sampling similar to the standard DDPM (Ho et al. 2020) and its variants (Song al. 2021).
Our contributions can be summarized as follows:
We investigate temporal predictive learning using diffusion models and highlight the un- derexplored challenge of integrating temporal dynamics into the diffusion process.
We introduce Dynamical Diffusion, theoretically guaranteed framework that explicitly models temporal transitions at each diffusion step. We outline key design choices that enable efficient training and inference of Dynamical Diffusion.
We conduct experiments various tasks across different modalities, including scien- tific spatiotemporal forecasting, video prediction, and time series forecasting. The results demonstrate that the proposed Dynamical Diffusion framework consistently enhances per- formance in predictive learning.
2 PRELIMINARIES
Diffusion models. Diffusion models (Sohl-Dickstein et al. 2015: Ho et al. 2020) and their vari- ants (Song& & Ermon, 2019 Song al. 2020) have shown outstanding capabilities in capturing complex data distributions. The core design of diffusion models involves dual forward and reverse processes. Formally, the forward process gradually corrupts real data Xy ~ q(xo) according to noise schedule (āt) At timestep the corrupted data can be sampled as
X Va v1 arEt,
where C (O, I) denotes random Gaussian noise. Subsequently, in the reverse process, nerual network Eg is trained to invert forward process corruptions with po (Xt-1 with the objective of minimizing the variational lower bound
L(0)= E...x..e..N((o.. [||E= 1 En (Varxo + VI- 1)]]]
Once trained, sampling diffusion models is performed by iterative denoising from X ~ (0 I) to Xp- Similar to other types of generative models, diffusion models are in principle capable of modeling conditional distributions. This can be achieved by modifying the reverse process to learn p c). where represents the condition.
Predictive learning with diffusion models. The goal of predictive learning is to predict future states 1:9 based on observations P::0 By substituting the condition с with observations the predictive learning task can be naturally interpreted as conditional generation task, making well-suited for diffusion models solve. This approach requires minimal modifications to the original diffusion process and has been adopted by several recent works (Voleti et al. 2022 Gao al. 2023).
3 METHOD
We observe that, when integrating diffusion models into predictive learning, there are two notable along which the model must learn simultaneously. The first axis, referred to the "prediction axis", requires the model to learn the temporal dynamics of the data. The second axis, termed the "denoising axis". necessitates that the model distinguishes noise from corrupted states.
From this perspective, we identify mismatch in previous methodologies. As shown in Figure Dh. the forward process in standard diffusion models progresses solely along the denoising axis. In P:0 serve only conditions for denoising networks, with no particular, historical observations x
2

---

Published as a conference paper at ICLR 2025
condition
x
Xg
Kg
X
X
x
Forward Process
Forward Process
x
la
denoising
:
prediction
Forward Process
(a) DPM
(b) DY DYfuusion
(c) Dynamical Diffusion
Figure I: Comparison of diffusion modeling approaches in predictive learning.
temporal dependency considered between temporally adjacent latents x; and x;- This modeling strategy isolates the predictive task along the denoising axis, while overlooking the internal continu- ity and forecasting capabilities of the dynamics that could potentially enhance the diffusion process.
In contrast, an alternative process in Rühling Cachay al.] 2023), ) as depicted in Figure p. progres- sively generates intermediates between two states. The forward and reverse processes are modeled as temporal interpolation and extrapolation, respectively. In this approach, the diffusion process is applied on the prediction axis. While this addresses the mismatch issue, it requires the predictability of the intermediate states. Furthermore, the ability to generate high-quality samples has not been fully validated, as the core mechanism of adding and removing noise in DPMs has been eliminated.
Building on the aforementioned considerations, we propose Dynamical Diffusion (DyDiff), de- signed to concurrently model both the denoising and prediction axes. As illustrated in Figure D Dynamical Diffusion explicitly introduces the mixture historical states in the diffusion process. By controlling different mixing manners with respect to the timestep Dynamical Diffusion ables temporal-aware forward and reverse processes, which we present in Subsections 3.1 and 3.2 respectively.
3.1 FORWARD PROCESS
In the standard forward process, the corrupted latent diffusion step constructed as X, y0,x0 + vI E, Inspired by recurrent neural networks (Hochreiter) 1997 Chung et al. 2014) and state-space models (Gu al., 2022) that capture temporal transitions through iterative structures, in Dynamical Diffusion, we define each latent x² by the combination with its previous latent x formalized as follows:
V (Vaxx) + v1 - a.e + V
Tt x
where { are the newly introduced timestep-aware schedule hyperparameters to control the dependence of the previous latent. By expanding the above equation along the prediction axis, we obtain
x Vā.- Dynamics (x, (x:5)))))) + v1 ā. C.
(4)
where
Dynamics (xr -Ps
Pss 7c) = V x0 + v1 71 Dynamics (x -PPss-1 (7))
(5)
and
Vn.
NO..
(6)
represents non-independent random Gaussian noise (proof in Appendix A. The definition of Dynamics refers to a timestep-award awware aware mixture of all historical states, and further achieves temporal dynamics by adequately controlling the factor 5.- Notably, the noise factor vI aj remains un- changed regardless of the choice of 52 and identical to that in the standard diffusion process. As result, the signal-to-noise ratio (SNR) in the diffusion model is preserved.
3

---

Published as a conference paper at ICLR 2025
Discussion Based Equation (5). when 9/ all prior information is ignored, and the forward process approximates the one in standard diffusion models. Conversely, as " decreases, earlier historical observations given greater weight. In designing the schedule for {5 advisable for 7/2 to be a non-increasing function, ensuring that larger values of f correspond to stronger emphasis
historical states. Additionally, setting t = guarantees that xi x", preserving the initial state. Notably, unlike âT 2 in standard diffusion models, it is not necessary for "r Rs O. If "T approaches zero, the reverse process would start from Synamica(X)) (*a) x for all 8, which might be less relevant than utilizing recent states. In practice, we adopt the schedule Tt 'ja + (1 17), where E [0, 1] is time-independent factor. By default, we set = 0.5. We
will further analyze the effect of 77 Section 4.4
3.2 REVERSE PROCESS
The forward process defines marginal distribution g(x] |xp P::) that is composed of both random noises and historical states. Next, we discuss the posterior distribution and the reverse process for Dynamical Diffusion.
3.2.1 SINGLE- STEP PREDICTION CASE
We begin with the single-step prediction case, i.e. S = 1. In this scenario, all the previous states Xg P.s-1 involved in the diffusion process are fully known. We formulate the following theorems, with proof attached Appendix A.2
Theorem 1. In a manner akin to DDIM (Song et al. 202I), there exists a non-Markovian forward process with the following marginal distribution
q (x (Vā Dynamics(x,) P.1, :7), (1 T.)
Furthermore, learning of the reverse process can be reparameterized into the following denoising objective
((0) E, Exx .. [|les - En (Vor Dynamicc(xx))) PP:1 ;77) + v1 an Ee. 111 (8)
with DDIM-like sampler
po po(x Xx, xy p::0 = N N ((VOL) -Dnamiicsxa P:O xpredi 7--1) + V1 δ,-1 - 02 oi
where
x -
vI (x) P.0,
/1 % Dynamics (x, P:O 9r) / (10)
Xpred
Vor
refers to the predicted ground truth.
Theorem 2. (Informal) There exists a DDPM-like (Ho et al. 2020> Markovian forward process which shares the same marginal distribution Equation (7), and the reverse process can be learned using the same objective function as Equation (8) and inferred using a DDPM-like sampler
Po |x] x P:0)
( xpred= xo P.0), o
(11)
with Ar, referring the posterior mean derived by the forward process.
Remarks. Theorems 2 indicate that when 5 = 1, it is feasible to learn a denoiser network similar to standard diffusion models. This denoiser serves reparameterization of the reverse process and minimizes the variational lower bound on the posterior distribution. The main differ- is that the denoiser in Dynamical Diffusion aims distinguish from the noisy disturbance of Dynamics( cxmiiss(x) P:1) instead of xf_
4

---

Published as a conference paper at ICLR 2025
3.2.2 EXTENTION TO SULPI-STEP PREDICTION
We now extend the proposed reverse process to the multi-step prediction scenario, i.e., S 1. Compared with the case when 5 1, the forward process additionally introduces dependencies multiple latents, and the reverse process must consider the absence previous ground truth 1:s-1 for given The following theorem presents the reparameterized objective, with detailed proof Appendix A.3
X
Theorem 3. (Informal) There exists a DDIM-like and a DDPM-like forward process satisfying the marginal distribution
(x x P:S) = [Vva Dynamics(xx) P.9 ; -.)),))))
(x
7t).
(12)
where J, is non-identity covariance matrix with (Ji) (v5) k Additionally, the reverse
process can be reparameterized into the following denoising objective
(()) 1x (.
|| сœ (V Dynamics P:S + vI - a; Ec, t) [|]
(13)
with DDIM/DDPM like samplers which are extensions of Equations (9) and 11
Remarks. Equation naturally generalizes the case when S = 1. Specifically, for each state
8, the marginal distribution ([x; -**) retains exactly the same form as Equation (7). When con- sidering all latents x² collectively as joint distribution, Dynamical Diffusion differs from standard diffusion models in both the forward and reverse processes.
In the forward process, as discussed Equation the latents are dependently defined. This dependency leads non-identity covariance matrix when combining all states into a joint distribution. Consequently, the denoiser must learn from non-independent sampled noises E, accommodating the correlations introduced by the dependent states.
In the reverse process, when reconstructing xpied from the current latents x! and the predicted noises 2t.5, the addition of noises to the dynamics necessitates computing the inverse dynamics function (see Appendix [B.. The sampler must then reapply noises the recalculated dynamics to accurately recover the predicted states. Similar algorithms are employed on 21:5 to obtain e1:8, ensuring consistency in the reverse diffusion steps.
Algorithm. The pseudocode for the training and inference processes of Dynamical Diffusion is provided in Algorithms [nd respectively. Compared with standard diffusion models, Dynamical Diffusion differs only in its preparation of inputs and outputs for the denoiser Eg, without introducing any additional forward or backward passes. Consequently, the computational cost remains similar that of standard approaches.
EXPERIMENTS
In this section, we evaluate Dynamical Diffusion (DyDiff) in three different settings and compare its performance against the standard diffusion model (DPM). We demonstrate that DyDiff is ver- satile to provide competitive performance across a range of tasks (Section 4. conduct in-depth analysis understand the prediction process of DyDiff (Section specifically mentioned, we use the framework of Stable Video Diffusion (Blattmann et al. 2023a), which achieves the of-t-oof-the-rr art performance on video generation tasks. We provide experimental details in Appendix C. along with additional comparisons presented in Appendix D
4.2 and 4.3 and
Unless
4.1 SCIENTIFIC SPATIOTEMPORAL FORECASTING
Setup. We begin by evaluating the models' performance in scientific spatiotemporal forecasting using the Turbulence Flow dataset (Wang et al. 2020) and the SEVIR dataset (Veillette et al. 2020 This scenario requires the model to learn the underlying physical dynamics. Turbulence Flow is simulated dataset governed by partial differential equations (PDEs), capturing spatiotemporal dy- namics of turbulent fluid flows, specifically the velocity fields. Each frame contains two channels
5

---

Published as a conference paper at ICLR 2025
Algorithm Training of Dynamical Diffusion Algorithm 2 Inference of Dynamical Diffusion
1: procedure Dynamics(x .7)
Require: procedure InverseDynamics (Algorithm
2: Xdyn < X6i
3: for in [I. + 1, R2) do
4: Xdyn
v1 xyn + Vxx0
Sample x1:5 1:55 pred ~ (0,
5: end for
2: x +Dynamics(xx, 1:5
6: return Xyy
3:
7:
end procedure
4: for t in [T, 1] do
8:
E + E (x 1:
x P:0
9: while not converged do
x 1: f x
v
1:9)
/Va
10:
Sample x PPS
dyn
P:0 Dynamics(x,) P.O
7)) 11: Sample e 1:5 ~ NO I) T[
dyn
x1:8 Dyn
8:
x pred
P:S
+ Inversebynamics(x P:S
dyn
Tt)
13:
Cayn
1:5
---Damiiic(s 1:8 7t)
InverseDynamicate 1:5
((0)
[|| dy celvaxx dyn
10:
1:8
Dynamics( pred ?t-1)
+V1 -1- C dyn x6
P:0
11:
Xt-1
I.S
v V -Drnami ca(x prod デt- -t-1)
P-S
Backprop with L(f) and update 16: end while
+ +V1 16
1:8
end for 13: return x
12:
17:
return
representing turbulent flow velocity along the æ and y directions. The task is predict future ve- locity fields based prior observations. Following the configuration of Wang al. generate sequences 15 frames at spatial resolution 64 x 64 grids, using input frames to predict the sub- sequent 11 frames. SEVIR is large-scale dataset curated specifically for metcorology and weather forecasting research. Each sample in SEVIR represents 384km x 384km observation sequences over 4 hours. Following (Gao et al. 2023), select the task of predicting Vertically Integrated Liquid (VIL), where the model learns to forecast future precipitation levels. For this dataset, 7 input frames used to predict the 6 frames, with each frame having resolution of 128 x 128 grids.
For evaluation, we report the neighborhood-basee Continuous Ranked Probability Score (CRPS) (Gneiting & Raftery 2007) and Critical Success Index (CSI) (Schaefer 1990, Jolliffe & Stephenson 2012), following (Ravuri et al., 2021: Zhang et al.) (202)). The CRPS metric empha- sizes the model's 's ensemble forecasting capabilities, while the CSI metric evaluates the accuracy of the model's predictions in peak regions. Lower CRPS values and higher CSI scores indicate better performance.
Table 1: Scientific spatiotemporal forecasting results on the SEVIR and Turbulence Flow datasets. w, arg, and max represent hyteeseenthyyyheeeee hyperparameters in evaluation metrics (see Appendix C
SEVIR
Turbulence
Method
CRPS
CSI 1
CRPS L
CSI f
(w5)
(w5)
(w8, avg)
(w8, maz)
(w8, avg)
(w8, maz)
DPM
8.67 7.62
15.41 13.56
0.285
0.0313 0.0275
0.0364 0.0315
0.8960
DyDiff (ours)
0.319
0.8998
Results. Table I presents the numerical results. On both datasets, DyDiff consistently outperforms the standard DPM. achieving 12% reduction in CRPS on the Turbulence dataset. Further, fig- ane illustrate qualitative analyses. evident that Dy outputs more accurate predictions than standard DPM, particularly over longer time horizons.

---

Published as a conference paper at ICLR 2025
DPM Predictions
565
DPM Residuals
0.25
-0.5
DyDiff Predictions
DyDif Residuals
Figure 2:
Visualization of predicted velocity fields on the Turbulence dataset. The top row dis- plays the ground truth values. Residuals highlight the discrepancies between predictions and ground truths. Standard DPM predictions, characterized by two distinct positive regions (colored red), do not align with physical laws. In contrast, Dynamical Diffusion yields more accurate predictions.
Observations
2
6
Ground Truth
DPM Predictions
DyDiff Predictions
Figure 3: Visualization of predictions on the SEVIR dataset. The first row displays observational states, while the second row shows the corresponding ground truth. For longer prediction times, such as 8 4, standard diffusion models struggle to capture heavy-precipitation regions, particularly noticeable in the top right corner. In contrast, Dynamical Diffusion consistently provides more accurate predictions for these critical areas.
4.2 VIDEO PREDICTION
Setup. Next, we evaluate the performance of different methods on the BAIR (Ebert et al. 2017) and RoboNet (Dasari al., 2019) datasets, which serve as benchmarks for assessing the model's ability to predict object movements in real-world scenarios. The BAIR robot pushing dataset consists of 43k training videos and 256 test videos. Each video records the motion of robot it pushes objects in tabletop setting. The goal is to predict 15 future frames based on single initial frame. The RoboNet dataset consists of 162k videos captured across 7 different robotic arms interacting with hundreds of objects in diverse environments and viewpoints. Following previous work (Yu) et al. 2023). we use 256 videos for testing and predict 10 future frames based on input frames. For both datasets, each frame has resolution of 64 64 pixels. We report performance using four commonly adopted metrics: FVD (Unterthiner et al., 2018), PSNR (Huynh- Thu & Ghanbari. 2008), SSIM (Wang al. 2004), and I.PPPS (Zhang al. 2018). Among these metrics, FVD

---

Published as a conference paper at ICLR 2025
measures video-level consistency, while the other three computed per image and reflect the average prediction accuracy.
Table 2:
Video prediction results on the BAIR robot pushing and RoboNet dataset. LPIPS and SSIM scores are scaled by 100 for convenient display.
BAIR
FVD4 PSNR SSIM LPIPS!
RoboNet
FVDJ PSNR1 SSIM LPIPS)
action-free & 64x64 resolution
action-free & 64x64 resolution
DPM
72.0
21.0
83.8
9.2
DPM
92.9
24.9
83.9
8.2
DyDiff (ours)
67.4
21.0
84.0
9.0
DyDiff (ours)
81.7
25.1
84.2
7.9
action-conditooned 64x64 resolation
action-conditioned & 64x64 resolution
DPM 48.5
25.9
92.0
4.5
DPM
77.0
26.4
87.3
6.0
DyDiff (ours)
45.0
26.2
92.4
4.2
DyDiff (ours)
67.7
26.5
87.5
5.9
Results. We present the experimental results in Table 2 covering both action-free and action- conditioned scenarios. Dynamical Diffusion consistently surpasses the standard DPM in all evalu- ated metrics, showing greater improvements in FVD, verifying its ability to make temporally con- sistent predictions. Qualitative outputs in Figure I illustrates that Dynamical Diffusion effectively addresses the artifact issues present in DPM for both background and foreground objects.
BAIR
2-2
5-3
s-d
5-5
5-6
S-7
5
2
10
s-11
12
s
s-14
5-15
DPM Predictions
DyDiff Prodictions
Figure 4: Visualization of The positions of robot arms under Dynamical Diffusion are more precise than standard DPM.
predictions the BAIR dataset. Zoom in for details.
NoboNet
5-
6-6
10
10
Ground
Truth
(Osservations)
(Observations)
DPV Prodictions
DyDiff Prodictions
Figure 5: Visualization action aeniion-oonitiooeed predictions the RoboNet dataset. Zoom for details. For standard diffusion models, (left) the pink shovel is missing, and (right) the red bottle distorted. This indicates the potential temporal inconsistency of standard diffusion models. On the contrary, Dynamical Diffusion can generate consistent frames, especially for the background.
4.3
TIME SERIES FORECASTING
Setup. We further evaluate the model on six multivariate time series datasets: Exchange. Solar, Electricity, Traffic, Taxi, and Wikipedia. These datasets encompass time series with varying dimen- sionalities, domains, and sampling frequencies. We benchmark Dynamical Diffusion against the diffusion-based method TimeGrad (Rasul et al.) 2202la). using TimeGrad's backbone and experi- mental setup. For evaluation, we employ the Summed CRPS (Matheson & Winkler 1976).
Results.
We present the experimental results in Table Dynamical Diffusion significantly out- CRPS on four out of six datasets (Solar, Traffic,
performs standard diffusion models terms of Taxi, Wikipedia). For the remaining two datasets (Exchange, Electricity),, the performance aligns
8

---

Published as a conference paper at ICLR 2025
Table 3:
Time series forecasting results on six benchmark datasets. CRPS is measured for its mean and standard deviation across five runs trained with different seeds.
CRPS +
Method
Exchange
Solar
Electricity
Traffic
Taxi
Wikipedia
DPM 0.007 0.000
0.372
+0.002
0.042 0.000
0.122 10.012
0.070.0..00r
w/ DyDiff 0.007
0.31 1.316+0.010
0.023
0.040
0.120+00.00
0.066+0.015
with the variance range of the baseline models. Overall, these results highlight the effectiveness of Dynamical Diffusion as versatile predictive model across diverse datasets.
4.4 ANALYSIS
Analysis latents. Dynamical Diffusion introduces novel forward and reverse processes, which affect the latents during inference stages. In Figure [6] we visualize and compare the latents of Dynamical Diffusion and the standard DPM. We also calculate the error between latents and final frames and plot the curve in Figure 7a It is observed that DyDiff generates less noisy samples in an earlier denosing steps compared with standard diffusion model, especially for larger
Timestep
1-
50
40
35
32
30
28
26
25
24
22
20
18
15
10
1-0
DPM, s=
DPM, s= 15
DyDiff, s-
DyDiff. s- 15
Figure 6: Visualization of latents during the inference process of the BAIR dataset, with timestep t divided by 20. At the same timestep (such as t = 22), the backgrounds frames generated by consistent with the final predictions, while standard diffusion models hold
Dynamical Diffusion noisier latents. Similar comparisons (such as f = 28) on Dynamical Diffusion show that frames with 15 are less noisy than 1 single timestep.
Effeet of dependent noises. When using Dynamical Diffusion to predict multiple steps simulta- neously, the forward process use non-independent noises ē V + +V1 - - in Theorem and Algorithm To further explore its necessity, independent noises et instead of ट.. Results are shown in Figure 7b It is observed that when using independent noises e², the performance gets worse and even underperforms the baseline. Therefore, using non-independent noises necessary for Dynamical Diffusion.
as illustrated design ablation study that uses
Different gamma schedules. For simplicity, Dynamical Diffusion adopt = 0.5 as the default setting for the gamma schedule 7t = rjū; + (1 7)) To further explore the sensitivity to hyper- parameters, we conduct experiments using various with values in the set {0, 0.1, 0.9, 1). and report the results on the Turbulence dataset. Notably, = corresponds to the baseline of standard diffusion. Results are shown in Figure It is observed that v E 10.1, 0.5, 0.9} demonstrate similar performance and all significantly surpass the standard diffusion model, indicating the robustness of hyperparameter design in Dynamical Diffusion. Yet at 7 1.0, the model performance signifi- cantly drops and even underperforms the baseline, indicating that a schedule with Fr ze 0 may not be effective, discussed in Section 3.1

---

Published as a conference paper at ICLR 2025
DPM, s-1
0.06
0.040
DPM,
0.08
DyDiff,
0.025
M8B
0.06
DyDif s
CRPS
0.04
CRPS
0.030-
0.0
0.02
0.025
0.00
40
10
DPM
DyDif
w/ w/i.d. noise
(a) Error intermediate latents.
(b) Ablation study.
Values of
Figure 7: Analysis experiments of Dynamical Diffusion.
5 RELATED WORK
This paper studies around the diffusion equations, a topic ex- tensively explored in the literature. Previous research has primarily concentrated noise sched- ules (Nichol Dhariwal 2021 Karras et al., 2022, Liu et al. 2023), training objectives (Salimans & Ho 2021 Karras et al. (022), and efficient sampling techniques (Song et al.) [021] Lu et al., 2022). These methods aim to enhance the modeling general data distributions including non- temporal modalities such as images. In contrast, our work presents novel approach that explicitly incorporates temporal dynamics by modifying the diffusion equations.
Studies on diffusion framework.
Deep predictive learning methods. Predictive learning aims to forecast a system's future be- havior by learning the underlying dynamics that drive its evolution. One common approach is to train the model to predict one step at time and then unroll it autoregressively to make multi-step predictions. However, this method can be prone to compounding errors as the forecast horizon increases (de Bezenac et al., 2018 Scher & Messori] 2019 Chattopadhyay et al.) 2020; Keisler
2022 Bi et al. 2023). To address this. existing approaches focus on improving model architec- 2022: Lam et al. [022]] Yu et al. (2023). unrolling models during
tures Yan et al. 2021 |Wang et al.)
training (Brandstetter et al., 2022 Pathak et al. [022]] HAN al.] 2022| Bi et al. 0023), or incor-
porating domain-specific knowledge (de Bezenac et al. 2018 Kochkov et al.) 2021 ( Mamakoukas ct al. 2023). Despite these efforts, performance in Long-horizon prediction remains limited (Pathak al. 2022). Alternativly.. some methods forecast multiple steps simultancously (Weyn al. [019
Brandstetter et al. 2022; Ravuri et al. 2021 Zhang et al. (202)), showing advantages over autore- gressive methods in several contexts (Vleti et al., 2022 Gao et al.) (023). In both lines of work, generative models, such as generative adversarial networks (GANs) and diffusion models, have been leveraged for their superior ability model distributions, enhancing prediction quality (Rasul al.,
2021a la Zhang et al., 20233] Mardani el al.) 2023; Gao et al., 2023; Pathak et al. (024)). Our work specifically focuses on generating multi-step predictions simultaneously with diffusion models,
topic that has gained increasing attention in the research community.
Predictive learning with diffusion models. To enhance predictive learning with diffusion models, Ho al. (2022): BBlattaann al.) (2023a); Voleti al. (2022); Gao al. (2023); Rasul al. (2021a) design specific predictive model architectures for different modalities. Wu et al. (2023): Ruhe et al. (2024); Chen et al. (2024a) propose Hate-wise timestep schedules. Notably, these methods, both the forward and reverse processes remain consistent with standard formulations. Therefore, our work serves as a complement to existing approaches.
CONCLUSION
In this paper, we investigate temporal predictive learning using diffusion models and highlight the underexplored challenge of integrating temporal dynamics into the diffusion process. For this pur- pose, we introduce Dynamical Diffusion, a theoretically guaranteed framework that explicitly mod- els temporal transitions at each diffusion step. Dynamical Diffusion introduces simple yet efficient design that adds noises to the combination of the current state and historical states, which is further learned by denoising process. Experiments on various tasks, including scientifie spatiotemporal forecasting, video prediction, and time series forecasting, demonstrate that Dynamical Diffusion consistently enhances performance in general predictive learning.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGEMENT
This work was supported by the National Natural Science Foundation of China (U2342217 and 62021002), the BNRist Project, and the National Engineering Research Center for Big Data Soft- ware. We also gratefully acknowledge the contributions of Yuchen Zhang for helpful discussion and Jincheng Zhong for the Overleaf premium plan.
REFERENCES
Alexander Alexandrov, Konstantinos Benidis, Michael Bohlke-Schneider, Valentin Flunkert, Jan Gasthaus, Tim Januschowski, Danielle C Maddix, Syama Rangapuram, David Salinas, Jasper Schulz, et al. Gluonts: Probabilistic time series models in python. arXiv preprint arXiv: 05264, 2019.
Mohammad Babaeizadeh, Mohammad Taghi Saffar, Suraj Nair, Sergey Levine, Chelsea Finn, and Dumitru Erhan. Fitvid: Overfitting in pixel-level video prediction. arXiv preprint WrXiv:2106.33193 2021.
Kaifeng Bi, Lingxi Xie, Hengheng Zhang, Xin Chen, Xiaotao Gu, and Qi Tian. Accurate medium- range global weather forecasting with 3d neural networks. Nature, 2023.
Christopher M Bishop. Pattern recognition and machine learning. Springer google schola, 2:1122- 1128, 2006.
Andreas Blattmann, Tim Dockhorn, Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, Dominik Lorenz, Yam Levi, Zion English, Vikram Voleti, Adam Letts, et al. Stable video diffusion: Scaling latent video diffusion models to large datasets. arXiv preprint 2rXiv:231..15127, 2023a.
Andreas Blattmann, Robin Rombach, Huan Ling, Tim Dockhorn, Seung Wook Kim, Sanja Fidler, and Karsten Kreis. Align your latents: High-resolution video synthesis with latent diffusion models. In CVPR, 2023b.
Johannes Brandstetter, Daniel E Worrall, and Max Welling, Message passing neural pde solvers, In ICLR, 2022.
Ashesh Chattopadhyyay Mustafa Mustafa, Pedram Hassanzadeh, and Karthik Kashinath. Deep spa- tial transformers for autoregressive data-driven forecasting of geophysical turbulence. In Pro- ceedings of the 10th international conference climate informatics, pp. 106-112, 2020.
Boyuan Chen, Diego Marti Monso, Yilun Du, Max Simchowitz, Russ Tedrake, and Vincent Sitz- mann. Diffusion forcing: Next-token prediction meets full-sequence diffusion. arXiv preprint arXiv:2407.01392, 2024a.
Kaiyuan Chen, Xingzhuo Guo, Yu Zhang, Jianmin Wang, and Mingsheng Long. Cogdpm: Diffusion probabilistic models via cognitive predictive coding. In JCML, 2024b.
Nanxin Chen, Yu Zhang, Heiga Zen, Ron Weiss, Mohammad Norouzi, and William Chan. Wave- grad: Estimating gradients for waveform generation. In ICLR, 2021.
Junyoung Chung, Caglar Gulcehre, KyungHyun Cho, and Yoshua Bengio. Empirical evaluation of gated recurrent neural networks on sequence modeling. arXiv preprint a41. 2014.
Sudeep Dasari, Frederik Ebert, Stephen Tian, Suraj Nair, Bernadette Bucher, Karl Schmeckpeper, Siddharth Singh, Sergey Levine, and Chelsea Finn. Robonet: Large-scale multi-robot learning- In CoRL, 2019.
Emmanuel de Bezenac, Arthur Pajot, and Patrick Gallinari. Deep learning for physical processes:
Incorporating prior scientific knowledge. In ICLR, 2018.
Emmanuel de Bézenac, Syama Sundar Rangapuram, Konstantinos Benidis, Michael Bohlke- Schneider, Richard Kurle, Lorenzo Stella, Hilaf Hasson, Patrick Gallinari, and Tim Januschowski. Normalizing kalman filters for multivariate time series analysis. In NeurIPS, 2020.
11

---

Published as a conference paper at ICLR 2025
Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans image synthesis. In NeurIPS, 2021.
Frederik Ebert, Chelsea Finn, Alex X Lee, and Sergey Levine. Self-supervised visual planning with temporal skip comnections. In CoRL, 2017.
Zhihan Gao, Xingjian Shi, Boran Han, Hao Wang, Xiaoyong Jin, Danielle Maddix, Yi Zhu, Mu Li, and Yuyang Bernie Wang. Prediff: Precipitation nowcasting with latent diffusion models. In NeurlPS, 2023.
Tilmann Gneiting and Adrian E Raftery. Strictly proper scoring rules, prediction, and estimation. Journal of the American statistical Association, 2007.
Albert Gu, Karan Goel, and Christopher Re. Efficiently modeling long sequences with structured state spaces. In ICLR, 2022.
Agrim Gupta, Stephen Tian, Yunzhi Zhang. Jiajun Wu, Roberto Martin-Martin, and Fei-Fei. Maskvit: Masked visual pre-training for video prediction. In ICLR, 2023.
XU HAN, Han Gao, Tobias Pfaff, Jian-Xun Wang. and Liping Liu. Predicting physics in mesh- reduced space with temporal attention. In ICLR, 2022.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In NeurlPS, 2020.
Jonathan Ho, Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David Fleet. Video diffusion models. NeurlPS, 2022.
Hochreiter. Long short-term memory. Neural Computation MIT-Press, 1997.
Quan Bluynh-Thu and Mohammed Ghanbari. Scope of validity of psmr in image/video quality as- sessment. Electronics letters, 2008
Rob Hyndman, Anne B. Kochler, J. Keith Ord, and Ralph D. Snyder. Forecasting with Exponential Smoothing: The State Space Approach. Springer Science & Business Media, 2008
Ian T Jolliffe and David B Stephenson. Forecast verification: a practittoner's guide in atmospheric science. John Wiley & Sons, 2012.
Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion- based generative models. NeurlPS, 2022.
Ryan Keisler. Forecasting global weather with graph neural networks. arXiv:2202.07575, 2022.
arXiv preprint
Diederik Kingma, Tim Salimans, Ben Poole, and Jonathan Ho. Variational diffusion models. In NeurlPS, 2021.
Dmitrii Kochkov, Jamie A Smith, Ayya Alieva, Qing Wang, Michael P Brenner, and Stephan Hoyer. Machine leamlingacceelertteed computational fluid dynamics. Proceedings of the National Academy of Sciences, 118021):e1101784118, 2021.
Zhifeng Kong, Wei Ping, Jiaji Huang, Kexin Zhao, and Bryan Catanzaro. Diffwwave: A versatile diffusion model for audio synthesis. ICLR, 2021.
Rahul G Krishnan, Uri Shalit, and David Sontag. Structured inference networks for nonlinear state space models. In AAAI, 2017.
Remi Lam, Alvaro Sanchez-Gonziie Matthew Willson, Peter Wirnsberger, Meire Fortunato, Fer- ran Alet, Suman Ravuri, Timo Ewalds, Zach Eaton-Rosen Weihua Hu, et al. Graphcast: Learning skillful medium-range global weather forecasting. arXiv preprint arXiv:2212. /2794, 12794
2022.
Xingchao Liu, Chengyue Gong, and Qiang Liu. Flow straight and fast: Learning to generate and transfer data with rectified flow. In ICLR, 2023.
12

---

Published as a conference paper at ICLR 2025
Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. Dpm-solver: A fast ode solver for diffusion probabilistic model sampling in around 10 steps. In NewrIPS, 2022.
H. Lütkepohl. New Introduction to Multiple Time Series Analysis. Springer Science & Business Media, 2005.
Giorgos Mamakoukas, lan Abraham, and Todd Murphey, Learning stable models for prediction and control. IEEE Transactions on Robotics, 39(3):2255-223 2275, 2023.
Morteza Mardani, Noah Brenowitz, Yair Cohen, Jaideep Pathak, Chich- Yu Chen, Cheng- Cheng-Chin Liu, Arash Vahdat, Mohammad Amin Nabian, Tao Ge, Akshay Subramaniam, et al. Residual corrective diffusion modeling for km-scale atmospheric downscaling, 2024. URL https://arxiv. org/abs/2309. 15214, 2023.
James E Matheson and Robert L Winkler. Scoring rules for continuous probability distributions. Management science, 1976.
Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. In ICML, 2021.
Jaideep Pathak, Shashank Subramanian, Peter Harrington, Sanjeev Raja, Ashesh Chattopadhyay. Morteza Mardani, Thorsten Kurth, David Hall, Zongyi Li, Kamyar Azizzadenesheli, et al. Four- castnet: A global data-driven higl-resouution weather model using adaptive fourier neural opera- tors. arXiv preprint 2rXiv:2202..12/11 2022.
Jaideep Pathak, Yair Cohen, Piyush Garg, Peter Harrington, Noah Brenowitz, Dale Durran, Morteza Mardani, Arash Vahdat, Shaoming Xu, Karthik Kashinath, et Kilometer-scale convection allowing model emulation using generative diffusion modeling. arXiv preprint 440:: 2024.
William Peebles and Saining Xie. Scalable diffusion models with transformers. In ICCV, 2023.
Kashif Rasul, Calvin Seward, Ingmar Schuster, and Roland Vollgraf. Autoregressive denoising diffusion models for multivariate probabilistic time series forecasting. In ICML. 2021a.
Kashif Rasul, tivariate probabilistic time series forecasting via conditioned normalizing flows. In ICLR, 2021b.
Sheikh, Ingmar Schuster, Urs M Bergmann, and Roland Vollgraf. Mul-
Suman Ravuri, Karel Lenc, Matthew Willson, Dmitry Kangin, Remi Lam, Piotr Mirowski, Megan Fitzsimons, Maria Athanassiadou, Sheleem Kashem, Sam Madge, et al. Skilful precipitation nowcasting using deep generative models of radar. Nature, 2021.
David Ruhe, Jonathan Heek, Tim Salimans, and Emiel Hoogeboom. Rolling diffusion models. In ICML, 2024.
Salva Rühling Cachay, Bo Zhao, Hailey Joren, and Rose Yu. Dyffusion: A dynamics-informed diffusion model for spatiotemporal forecasting. In NewrIPS, 2023.
Chitwan Saharia, William Chan, Huiwen Chang, Chris Lee, Jonathan Ho, Tim Salimans, David Fleet, and Mohammad Norouzi. Palette: Image-to-imal diffusion models. SIGGRAPH, 2022a.
Chitwan Saharia, Jonathan Ho, William Chan, Tim Salimans, David J Fleet, and Mohammad Norouzi. Image super-resolution via iterative refinement. In TPAMI, 2022b.
Tim Salimans and Jonathan Ho. Progressive distillation for fast sampling of diffusion models. In ICLR, 2021.
David Salinas, Michael Bohlke-Schneider, Laurent Callot, Roberto Medico, Jan Gasthaus, and Roberto Medico. #igh-dimensional multivariate forecasting with low-rank gaussian copula pro- cesses. In NewrIPS, 2019.
David Salinas, Valentin Flunkert, Jan Gasthaus, and Tim Januschowski. Deepar: Probabilistic fore- casting with autoregressive recurrent networks. International Journal of Forecasting, 2020.
13

---

Published as a conference paper at ICLR 2025
Joseph T Schaefer. The critical success index indicator of warning skill. Weather and fore- casting, 1990.
Sebastian Scher and Gabriele Messori. Generalization properties of feed-forward neural networks trained on lorenz systems. Nonlinear processes in geophysics, 26(4):381-39) 2019.
Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In 20M1,,0015,
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In ICLR, 2021.
Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. In NeurlPS, 2019.
Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. ICLR, 2020.
Thomas Unterthiner, Sjoerd Van Steenkiste, Karol Kurach, Raphael Marinier, Marcin Michalski, and Sylvain Gelly. Towards accurate generative models of video: A new metric & challenges. arXiv preprint 18/2.01// 2018.
Roy van der Weide. Go-garch: A multivariate generalized orthogonal garch model. Journal of Applied Econometrics, 2002.
Mark Veillette, Siddharth Samsi, and Chris Mattioli. Sevir: A storm event imagery dataset for deep learning applications radar and satellite meteorology. In NeurIPS, 2020.
Ruben Villegas, Arkanath Pathak, Harini Kannan, Dumitru Erhan, Quoc Le, and Honglak Lec. High fidelity video prediction with large stochastic recurrent neural networks. In NeurlPS, 2019.
Vikram Voleti, Alexia for prediction, generation, and interpolation. In NeurlPS, 2022.
and Chris Pal. Mevd-masked conditional video diffusion
Rui Wang, Karthik Kashinath, Mustafa Mustafa, Adrian Albert, and Rose Yu. Towards physics- informed deep learning for turbulent flow prediction. In SIGKDD, 2020.
Yunbo Wang, Haixu Wu, Jianjin Zhang, Zhifeng Gao, Jianmin Wang, S Yu Philip, and Mingsheng Long. Predrnn: A recurrent neural network for spatiotemporal predictive learning. In TPAMI, 2022.
Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero Simoncelli. Image quality assessment: from error visibility to structural similarity. In IEEE transactions on image processing, 2004.
Jonathan A Weyn, Dale R Durran, and Rich Caruana. Can machines learn to predict weather? using deep learning to predict gridded 500-hpa geopotential height from historical weather data. Journal of Advances in Modeling Earth Systems, (88) 2693, 2019.
Bohan Wu, Suraj Nair, Roberto Martin-Martin, Li Fei-Fei, and Chelsea Finn. Greedy hierarchical variational autoencoders for sureesscle video prediction. CVPR, 2021.
Jialong Wu, Shaofeng Yin, Ningya Feng. Xu He, Dong Li, Jianye Hao, and Mingsheng Long- ivideogpt: Interactive videogpts are scalable world models. arXiv preprint arXiv:2405.1522 2024.
Tong Wu, Zhihao Fan, Xiao Liu, Hai-Tao Zheng, Yeyun Gong, Jian Jiao, Juntao Li, Jian Guo, Nan Duan, Weizhu Chen, et al. Ar-diffusion: Auto-regressive diffusion model for text generation. In NeurIPS, 2023.
Wilson Yan, Yunzhi Zhang, Picter Abbeel, and Aravind Srinivas. Vidcogpt: Video generation using vq-vac and transformers. arXiv preprint 3XXi:21044.10177 2021.
Ling Yang, Zhilong Zhang, Yang Song. Shenda Hong, Runsheng Xu, Yue Zhao, Wentao Zhang, Bin Cui, and Mina-Hsuan Yang. Diffusion models: comprehensive survey of methods and applications. In ACM Computing Surveys, 2023.
14

---

Published as a conference paper at ICLR 2025
Lijun Yu, Yong Cheng, Kihyuk Sohn, José Lezama, Han Zhang, Huiwen Chang, Alexander G Hauptmann, Ming-Hsuan Hsuan Yang, Yuan Hao, Irfan Essa, et al. Magvit: Masked generative video transformer. In CVPR, 2023.
Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as perceptual metric. In CVPR, 2018.
Yuchen Zhang, Mingsheng Long, Kaiyuan Chen, Lanxiang Xing, Ronghua Jin, Michael Jordan, and Jianmin Wang. Skilful nowcasting of extreme precipitation with nowcastnet. Nature, 2023.
15

---

Published as a conference paper at ICLR 2025
A MATHEMATICAL PROOF
A.1 DERIVATION OF FORWARD PROCESS
In this subsection give the proof of the form of x in Equation 4
Proof. By mathematical induction on s. With definitions in Equation
x = Vtr (vāxs + vI - ãGE,) + v 9t X;
Equation
Dynamics cs (x, P:s p)),7)) 71)
v x + v1 7t Dynamics (xo P ,7t).
and the boundary condition as the start state,
X vāpxo" vI a:e.
1 P
holds
x = (Vā VI- atEe) v + v1 - 91
= V5: (Vāxxū + vI v anEi)
VI- (Vār Dynamics (xp 170 + v1 ai 2
Vot (V7e x v1 - Dynamics (xo p:o0-1,;11)
+ v1-a (Vi+ E vI - 7t-- E 1
Vā Dynamics (x ;e) + vI ai E7.
Since ē and ef are independent normal noise, it satisfies
E FiE VI 7E (O.1).
which completes the proof.
A.2 CASE WHEN
DDIM-like sampler. We follow the proof in DDIM (Song et al. 2021
Proof. Define the following non-Markovian forward process:
qxx,_1x,, q( x Pol) II | xx P.l)
1=2
with q(x|/1xp) p:1) = N (Vār Dynamics(x)) P:1 (T), (1-))11ār)) ãr I))
x. =N Var-i Dynamics(x, P:1 1)+1)++
v 11 of Xt Vār Dynamics (x) 7t)
V1- ---0
then it suffies to prove <xx|1x, P:0) = N (Var DinnDiimmmsx,,, P:1 (.)) (1 - -x))). By mathemat- ical induction from T - 1 to 1, we have
xx P:1) N (Voz Dynamics(xx P:1 70); (1 - -
x(x|_1|\,, x PPI1) .V
Vo--i Sor-i-Dynac Dynamicss P:1 --+1)
1 O.-1 -o.. Vor Dynamics(x)) P:1 (1)) of
04
16

---

Published as a conference paper at ICLR 2025
then by letting x b x and
X0 P:1 as global condition, according to Eq.
(2.115) (Bishop] 2006).
g( 1x P:1)
((x:1x0 P:1 (x(=))x((111/// P.))) )
xx
(b(b) /. (bla))
also a Gaussian with
Pt-1 = Va--1 Dynamics Xx P.l -
+ V ā o7. vā: Dynamics(xx) P:l ,5) - Vāa Dynamics (x, -PP PP: ,5) VI- 04
vā--i Dynamics(x, P:l -)-1), E - â of (1- (1- (1-
which completes the proof.
DDPM-like sampler. We follow the proof in DDPM (Ho et al. 2020
Proof. The proof is structured two steps. First, define the following Markovian forward process:
x x X
P:I) ~ N( E,)
q
f4 Vorti X + Vor (V1 Dynami cs (x) -P:0, :t)) Vtr 7r Dynamics(x, PP:0, ; 92-1))
Σ. = (1 - a - ā₂ - 7:)) I
where ā₁ = IIi 04 9t = I1 %. It suffies to prove the marginal distribution
qxx ) A (Vo, Dynamics(x,) P-1 7d); (1 ā,)
By mathematical induction on / from to T 1, we have ((x |x] Xo P:0) and gf q(x}|x; P:1) Gaussian distributions, and thus by letting < x! x|+1 and + x according to Eq. (2.115) (Bishop) 2006).
P:1 global condition,
o (x p:1)-
xxx]1x] P-1 x xi -P- )xx)
x
(b) (c(a)a(blaadda
is also a Gaussian distribution with
p++1 Vou+i7 Dynamics(x: P:1 7))
+ Vaii (VI 7+1 Dynamics(x,, P:0
m441 V1+1 J641 Dynamics(x
-P.. :1))
or Dynamica(x, P:1 7t)
Σ = (1 0:4114111 - a 741)) a4+17++11111 -2)) I (1 ā.+1
Next, we consider the posterior distribution for the reverse process. Since q(x; [x]-1 x P.1) and q(x|_, -r-)) ) are both Gaussians, by letting a + x!_1,b b + x], and + xo P.1 as global
condition, according to Eq. (2.116) (Bishop) 2006).
P:U [x{ Xn P-1 g(x} x 11 X
qxx x], P:l)
{x+ x P:I)
(a(a ge (a)
(b)
also a Gaussian distribution. Therefore. the existence of the reverse process is proved, where the mean and variance could be derived accordingly.
17

---

Published as a conference paper at ICLR 2025
A.3 CASE WHEN $
Proof. For S >> 1, by definition in Equation 3 let
yi X2
y X
AA v
then y? V + vT-a C.e. satisfying the same Guassian distribution as standard diffusion models and independent with y Vs' # s. Thus
qly xp PiS) = q(x; x P.1) II (())
By defining the distribution of x Appendix A.2 and ył, S S as DDPM/DDIM for standard diffusion models, the reverse process gets proved by combining these independent latents together.
Remark. Following the proof, seemingly there is no need to add noise on dynamics for 8 > in theoretical view. In practice, the manner that remains dynamics could potentially help model generalization.
B ALGORITHM FOR INVERSE DYNAMICS
In this section we discuss the calculation of the inverse dynamics used in the inference process, i.e., calculate xi when given Dynamics(x,, P.k, ()) for -P s S s. From Equation (5), we have
Dynamics (xp Pte; 7) VI 7 Dynamics P:s-1;
The pseudocode of calculating inverse dynamics is shown in Algorithm
Algorithm Inverse Dynamics
1: procedure InverseDynamicas cs dyn
L:R 77)
2: x -x
for in [L 1, R, do
4: x5 Xxjyn VI 720 //R
5: end for
6: return xf:R
7:
end procedure
C IMPLEMENTATION DETAILS
C.I SPATIOTEMPORAL FORECASTING AND VIDEO PREDICTION
Training details. For the benchmark datasets, including BAIR, RoboNet, Turbulence, and SE- VIR, we utilize the state-of-the-arr architecture of Stable Video Diffusion (Blattaann et al. 2023a) Specificallyy, we first train frame-wist wise VAEs from scratch. In line with Blattmann et al. we also incorporate adversarial discriminator during VAE training to enhance reconstruction quality. The spatial downsampling ratio for the VAE set to x across all datasets. Once trained, the VAE encodes the original data into latent space with channel size of 3, and all diffusion processes carried out in this latent space. We adopt a 3D UNet as the diffusion model. All diffusion models also trained from scratch. 4 the detailed hyperparameters on these datasets.
18

---

Published as a conference paper at ICLR 2025
Table 4: Hyperparameters of DyDiff training.
Low-resolution
High-resolution (128
DyDiff
BAIR
RoboNet
Turbulence
SEVIR
Input channel
3
3
Prediction length
15
10
11
6
Observation length
2
7
Training steps
5 10s
5 10a
3 x 10a
4 10a
VAE channels
[128, 256, 512]
VAE downsampling ratio
4 x 4
VAE weighting
1 x 10-6 3
Latent channel
SVD channels
[64, 128, 256]
Batch size
16
Learning rate
1 x 10-4
LR Schedule
Constant Adam
Optimizer
Evaluation metrics. below:
We evaluate each method using commonly employed metrics, as outlined
Critical Success Index (CSI) (Schaefer 1990) quantifies the accuracy of binary predictive decisions. Following (Chen et al. 2024b). we apply spatial window around each grid for neighborhood-baxee evaluation (Jolliffe & Stephenson 2012) evaluate the "closeness"
of the forecasts. The window size (w) is set to 5, and average pooling (avg) is used within the window.
Continuous Ranked Probability Score (CRPS) (Gneiting Raftery, 2007) measures the alignment between probabilistic forecasts and ground truth data. To compute CRPS, the model generates multiple forecasts, allowing the score to capture the entire probability distribution. Following (Chen et al. (2024b), we calculate the neighborhood-basee CRPS using a window size of 8, and report results for two pooling modes: average pooling (ang) and max pooling (maz).
The Peak Signal-to-Noise Ratio (PSNR) (Huynh-Thu & Ghanbari 2008 measures the ratio between the maximum possible signal power (in this case, an image or video) and the power of the noise or distortion affecting it. higher PSNR indicates less distortion and closer match to the original image.
The Structural Similarity Index Measure (SSIM) (Wang et al. 2004 is widely used metric for evaluating the quality of images and video frames by assessing their perceived structural similarity. The SSIM score ranges from -1 to I, with 1 indicating perfect struc- tural similarity and lower values indicating greater dissimilarity. To better present the re- sults, scale the SSIM score by factor of 100.
The Learned Perceptual Image Patch Similarity (LPIPS) (Zhang al. 2018 evaluates the similarity between two images by passing them through a pretrained neural network. The network extracts features from both images, and the LPIPS score is calculated based on the distance between these feature representations. A smaller LPIPS score indicates higher similarity. Similar to SSIM, we also scale the LPIPS score by 100.
The Fréchet Video Distance (FVD) (Unterthiner et al. 2018 is based on the Fréchet dis- tance, mathematical measure that computes the distance between two distributions. For FVD, these distributions represent the feature space of real and generated videos extracted by neural network. Unlike the metrics mentioned above, FVD incorporates the temporal dimension of videos, making it more suitable for evaluating video generation models.
Sampling protocals. During inference, we use DDIM sampler with 50 steps for both the stan- dard DPM and our proposed Dynamical Diffusion. For video prediction benchmarks, including
19

---

Published as a conference paper at ICLR 2025
BAIR and RoboNet, following prior works (Gupta al. 2023 Wu et al. stochastic nature of video prediction by sampling 100 future trajectories per test video and selecting the best one for the final PSNR, SSIM, and LPIPS scores. For FVD, entific spatiotemporal forecasting tasks, including Turbulence and SEVIR, we generate predictions for each test sample compute CRPS and CSI, in line with prior work (Chen al. 2024b).
2024). account for the
all 100 samples. For sci-
C.2 TIME SERIES FORECASTING
Training details. For time series forecasting tasks, we follow the benchmark of TimeGrad RRa- sul al. 2021a), which framework apply diffusion models with the next-token prediction paradiam in time series forecasting. For implementation of Dynamical Diffusion, we set P = 0, ie., apply dynamics on only the latest state match the Markovian properties in the RNN used in TimeGrad. Since time series have greater volatility, we set > = 0.3(1 - 04)) for training and inference stability. We use exactly the same model architecture as TimeGrad. Since TimeGrad does not provide publically available reproducible setups, carefully tune the baselines and Dy- namical Diffusion for the best performance on each dataset. All datasets are available through GluonTS (Alexandrov et al. 2019), with detailed information shown in Table
Table 5: Properties of time series forecasting datasets.
Dataset
Dimension Domain
Frequency
Steps
Prediction length
Exchange
8
R+
BUSINESS DAY
6,071
30
Solar
137
RR+
HOUR
7,009
24
Electricity
370
R++
HOUR
5,833
24
Traffic
963
(0.1)
HOUR
4,001
24
Taxi
1,214
N
30-MIN
1,488
24
Wikipedia
2,000
N
DAY
792
30
Evaluation metrics. Following TimeGrad (Rasul et al.) 202 we employ the Summed CRPS (Matheson & Winkler 1976) to capture the joint effect, where score evaluated based on the sum of predicted distribution.
Sampling protocals. We DDIM sampler with 50 steps for the standard DPM and Dynamical Diffusion. For calculating the Summed CRPS, we generate 100 predictions for each test sample.
D MORE EXPERIMENT RESULTS
D.1 SCIENTIFIC SPATIOTEMPORAL FORECASTING
In this section, we further experiment with Diffusion Transformers (DiT) (Peebles & Xie 2023 Table presents the results of the Turbulence Flow dataset. We follow the same evaluation protocols outlined in Appendix C for these experiments. The results demonstrate that DyDiff significantly outperforms standard diffusion models, confirming its general applicability.
Table 6: Scientific spatiotemporal forecasting results on the Turbulence Flow dataset.
CRPS
CSI 1 (w5)
Backbone
Method
(w8, ang)
(wr8, max)
DPM
0.0313
0.0364
0.8960
SVD
DyDiff (ours)
0.0275
0.0315
0.8998
DPM
0.0434
0.0480
0,8403
DIT
DyDiff (ours)
0,0358
0.0395
0.8548
20

---

Published as a conference paper at ICLR 2025
D.2 VIDEO PREDICTION
In this section, we provide additonal comparisons with state-of-the-art deterministic models, in- cluding VideoGPT (Yan al. 2021), MaskViT (Gupta al. 2023), FirVid (Babaeizadeh et al. 2021]. MAGVIT (Yu et al. 2023). SVG (Villegas et al. 2019). GHVAE (Wu et al., 2021). and iVideoGPT (Wu et al.) 2024) on video prediction benchmarks. Table 7 and present the results on BAIR and RoboNet datasets, respectively. On the BAIR dataset, Dy demonstrates compa- rable performance the action-free scenario and significantly outperforms previous deterministic methods in the action-conditioned scenario. However, the RoboNet dataset, characterized by its di- verse object motion trajectories, poses substantial challenge for both DPM and DyDiff, with both methods falling short in performance. Notably, these methods employ networks with significantly more parameters than ours for instance, iVideoGPT contains 114M parameters, and Mask ViT con- tains 189M, compared pretraining process (Wu et al. 2024).
model's 63M parameters. Besides, some methods involve addition
Table Addition comparison with deterministic methods on BAIR dataset. "-" marks that the value is not reported in the original papers. LPIPS and SSIM scores are scaled by 100 for convenient display.
BAIR
FVD4
PSNR1
SSIM
LPIPS.
action-free & 64x64 resolution
VideoGPT 2021
103.3
Mask 2023
93.7
FitVid 2021
93.6
MCVD 2022
89.5
16.9
78.0
MAGVIT 2023
62.0
19.3
78.7
12.3
iVideoGP
2024
75.0
20.4
82.3
9.5
DPM
72.0
21.0
83.8
9.2
DyDiff (ours)
67.4
21.0
84.0
9.0
action-conditioned & 64x64 resolution
Mis 2023
70.5
iVideoGPT 2024
60.8
24.5
90.2
5.0
DPM
48.5
25.9
92.0
4.5
DyDiff (ours)
45.0
26.2
92.4
4.2
Table 8: Addition comparison with deterministic methods on RoboNet dataset. LPIPS and SSIM scores are scaled by 100 for convenient display.
RoboNet
FVD1
PSNR
SSIMt LPIPSJ
action-conditioned & 64x64 resolution
MaskViT 2023
133.5
23.2
80.5
4.2
SVG
123.2
23.9
87.8
6.0
GHVAE 2021
95.2
24.7
89.1
3.6
FitVid 2021
62.5
28.2
2.4
iVideoGPT 2024
63.2
27.8
90.6
4.9
DPM
92.9
24.9
83.9
8.2
Dyfiff (ours)
81.7
25.1
84.2
7.9
D.3 TIME SERIES FORECASTING
This section shows additonal comparisons with the standard diffusion model baseline (TimeGrad (Rasul et al. 202 and the state-of-the-art time series forecasting models, includ-
ing VES (Hyndman et al. 2008 VAR (Lütkepohl] (-asso- GARCH (van der Weide 2002) DeepAR (Salinas et al. 2020 STP/GP-Copull (Salinas et al., 2019), K KVAE (Krishnan et al. 2017). NKF (de Bézenac et al. 2020) and Transfomerr MAR (Rasul et al. 2021b). As demonstrated
in Table diffusion- based forecasting models fundamentally achieve similar better performance
21

---

Published as a conference paper at ICLR 2025
compared with deterministic models. Furthermore, Dynamical Diffusion generally outperforms standard diffusion baselines.
Table 9: Addition comparison with deterministic methods on Time Series dataset. CRPS (lower indicates better) is measured for its mean and standard deviation across five runs trained with differ- ent seeds. marks that the value is not reported in the original papers.
Method
Exchange
Solar
Electricity
Traftic
Taxi
Wikipedia
VES 2008
0.005 60.05-00.000
0.900+0.0or
0.880
0.350-0.002
VAR
0.005 2.000
0.830 nos
0.039 ou1
0.290.2o.00:
VAR AAR-Lass 1as0 2005
0.012
0.510
0.025
0.150
3. 3.100
0.0233+..000 000
0.880+.0.000 002
0.190 001
0.370=66.00
DeepAR
2020
0.3366+0.00
0.025+
0.055-0.00
0.12T
LSTM Copula 2019
0.007-+0.000
0.319 611
0.064-2..o0 0.08
0.103=o.00e
0.326 20.007
0.241 00.033
GP-Copala 2019
0.007 +0.0+00.000 40,000
0.337 +0337+000024
0.025 .002
0.078
0.208
0.086+0.000
KVAE 2017
0.01440.000
0.3-10
0.051:0.011
0.100==0.00b
0.095
2020
0.320 020
0.016go. 2016.00.011 001
0.100_2o.002
Trunslorter MAF22221 2021b
0,005
0.30111.00
0.021
0.056
0.179
0.063
TimeGrad DPM
00007
0.372+00.06 064
0.021 002
0.042-6.008
0.122
+0.012
0.070+0.007
TimeGrad w/ DiDif
0.007
0.316+0.010
0.023=0.000
0.040+0.000
0.120+0.00
0.066+0.013
22


Reviews:
[{"review_id": "SZtKJ8bFAo", "reviewer": "Reviewer_o9zW", "summary": "The authors proposed a new method to approach the problem of predictive learning in the context of diffusion models. Instead of utilizing conditional generation, in which the generation is conditioned on the history, their method devises a forward process that gradually incorporates the history while simultaneously adds noise to the data. In this way, the process mixes the \"diffusion time\" and the \"data time\". Training can be done using the usual denoising score matching method.", "questions": "1. The author(s) stressed that the method is \"theoretically sound\" (repeated twice) and \"theoretically guaranteed\". It'd be good if the author(s) can clarify a. in which sense is it theoretically sound? b. What precisely is \"guaranteed\" theoretically? \n2. I'm confused by the sentence \"Notably, the noise factor $\\sqrt{1-\\alpha_t}$ remains consistent with theoriginal diffusion process along the denoising axis. Hence, the new forward process preserves the same signal-to-noise ratio for any diffusion step $t$\". Surely the authors do not mean $\\sqrt{1-\\alpha_t}$ is independent of $t$, as opposed to what the notation suggests?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "Mixing temporal transitions with diffusion process is a novel idea. The experiments seem extensively done and the proposed method does deliver better performance when compared to DPM in many cases, though not as powerful as heavy models such as iVideoGPT", "weaknesses": "1. The writing can at various places uses some improvements. For instance, section 1, from the third paragraph onwards, appear to have large overlap with the abstract. In section 5, why the papers mentioned as \"related work\" described in the 1st paragraph are related seem to be obscure.  \n2. The paper does not contain much insight about why iterative scheme like the one described in (4)-(5) is a helpful one, or more precisely, why it is a good structure to impose. Even adding some simple explanations on the basic features of the scheme, such as \" larger values of $t$ correspond to a stronger emphasis on historical states\", will help bringing some insights.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "N1MQUTuZIA", "reviewer": "Reviewer_YRPt", "summary": "Dynamical Diffusion (DyDiff) is a novel framework that enhances diffusion models for temporal predictive learning tasks. It introduces temporally aware forward and reverse processes that explicitly model temporal transitions at each diffusion step, addressing the challenge of integrating temporal dynamics into diffusion processes. DyDiff achieves this by incorporating a mixture of historical states controlled by a new schedule parameter γ, allowing it to capture temporal dependencies more effectively. The framework is theoretically sound and can be efficiently trained and sampled similar to standard diffusion models. Experiments across scientific spatiotemporal forecasting, video prediction, and time series forecasting demonstrate that DyDiff consistently outperforms standard diffusion models, particularly in generating temporally coherent sequences and improving performance over longer time horizons.", "questions": "See weakness", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The paper introduces a novel framework called Dynamical Diffusion (DyDiff) that incorporates temporal dynamics into diffusion models for predictive learning tasks. This represents an original approach to addressing limitations of existing diffusion-based methods for temporal data. The authors creatively combine ideas from standard diffusion models with explicit modeling of temporal transitions.\n\nThe paper is generally well-written and structured logically. Key concepts and the proposed method are explained clearly with helpful illustrations. The authors provide pseudocode for the algorithms, which aids in understanding the implementation. Some technical details are relegated to appendices to maintain flow in the main text.", "weaknesses": "1. The paper does not adequately address why temporal dynamics need to be explicitly modeled, given that neural network architectures like transformers can inherently learn such relationships. Specifically:\n\t-\tLack of justification: The authors do not provide a clear explanation or empirical evidence for why explicitly modeling temporal dynamics is necessary, given that modern architectures like transformers are theoretically capable of learning temporal relationships on their own.\n\t-\tLimited comparison: There is no direct comparison between the proposed Dynamical Diffusion method and transformer-based approaches that implicitly learn temporal dynamics. This makes it difficult to assess the true value added by the explicit modeling of temporal relationships.\n\t-\tInsufficient analysis: The paper does not thoroughly analyze the limitations of existing architectures in capturing temporal dynamics, which would help justify the need for the proposed approach.\n\t-\tOverlooked alternatives: The authors do not discuss potential alternatives for improving temporal modeling within existing frameworks, such as modifications to transformer architectures or attention mechanisms.\n\t-\tUnclear efficiency trade-offs: The paper does not address whether the explicit modeling of temporal dynamics introduces additional computational overhead compared to letting a neural network learn these relationships implicitly.\n\n2. The paper primarily compares Dynamical Diffusion with standard diffusion models. To better establish its significance, the authors should:\n- Include comparisons with other state-of-the-art methods in temporal predictive learning, not just diffusion-based approaches.\n- Provide a more comprehensive literature review to contextualize their work within the broader field of temporal predictive learning.\n\n3. The paper could benefit from more extensive ablation studies to understand the contribution of different components of the proposed method. Additionally, a thorough analysis of hyperparameter sensitivity would strengthen the work. The authors could:\n- Conduct ablation studies on the key components of Dynamical Diffusion.\n- Analyze the sensitivity of the method to different hyperparameters, especially the newly introduced γ̄_t schedule.\n- Provide guidelines for selecting optimal hyperparameters for different types of temporal data.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "GCpEG9EH4K", "reviewer": "Reviewer_vb3D", "summary": "The authors address temporal learning dynamics - an underexplored area in diffusion models. They introduce Dynamical Diffusion, a simple (essentially one term) modification of existing diffusion dynamics to directly incorporate this temporal dependency in both the forward and backward processes. They show that tracking the temporal similarities results in higher quality samples across multiple modalities.", "questions": "N/A", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "I think this is a good paper. In my view, the motivation is more akin to how momentum methods developed in optimization - in both the cases, tracking the temporal dynamics was important. The idea, in that regard, isn't too novel, but I am glad to see the large number of experiments the authors have performed. I think this will be a paper whose method will be used as benchmark by future papers.", "weaknesses": "N/A", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "OlrovQN8yw", "reviewer": "Reviewer_hYkb", "summary": "The authors propose a new diffusion model for time series data incoporating the temporal dependencies between states in the forward and backward processes. Extensive empirical evidence shows that the model indeed outperforms regular denoising diffusion most notably on the SEVIR and Turbulence Flow datasets.", "questions": "- What are the advantages of using diffusion-like models for dynamical data compared to previous predictive approaches?\n- Are there any features of diffusion models that previous methods cannot achieve? can you demonstrare these empirically?\n- Can you extend your derivations to show how $q(x_{t-1}|x_0^{-P:s})$ is obtained from the forward process for both the ddpm and ddim sampling methods?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- **A clear problem and a sound solution**: The paper identifies a significant gap in how current diffusion models handle temporal data - they treat predictive learning primarily as conditional generation without fully leveraging temporal dependencies in the data. The authors address this limitation by modelling explicitly temporal transitions at each diffusion step (in both the forward and backward processes) through a mixture process.\n- **Thorough evaluation**: The authors conduct a strong empirical evaluation using datasets from various domains: scientific spatiotemporal forecasting, video prediction, and time series forecasting. The experiments also include ablations and analyses of key components like dependent noise and the $η$ parameter. In addition to the reported metrics, qualitative visualizations illustrate the strengths of the model espcially compared to regular diffusion.\n- **Good presentation**: The paper is well-written, with a logical flow from motivation to results.", "weaknesses": "- **Why diffusion models?**: I understand that the main contribution of the paper is extending the capability of diffusion models, and as such it makes sense to focus the evaluation of DyDiff vs DPM. However, I think a general motivation for diffusion/generative models for dynamical data is needed (see questions). Additionally, the authors could discuss how their approach in general relates to previous methods (see questions).\n\n- **Derivations for the reverse process and conditional reverse process are incomplete**: I have a hard time following the derivations in appendix A2 for $q(x_{t-1}^1|x^1_t)$ and $q(x^1_{t-1}|x^1_t, x_0^{-P:1})$, which I find to be too brief. Can the authors explain how $q(x_{t-1}|x_t, x_0^{-P:1})$ is obtained from the forward process?\n \n- **Dataset descriptions lack context**: The Turbulence Flow dataset is introduced without context about what it represents (including what each frame represents physically) or why it's relevant for evaluation. For SEVIR, while mentioned as a \"spatiotemporal Earth observation dataset\", the significance of Vertically Integrated Liquid (VIL) prediction is not explained. The same goes for the RoboNET, BAIR, and time-series datasets. A few sentences introducing the datasets used are needed in each experimental subsection. \n\n- **The metrics used are not defined**: Critical evaluation metrics (CRPS, CSI, FVD, PSNR, SSIM, LPIPS) are used without proper definition or motivation. This makes interpreting the results difficult. A formal (brief, if needed) definition of these metrics is crucial for the assessment of the empirical evaluation of the paper. \n\n- **Important baselines are missing**: The evaluation is missing important baselines like DYffusion [1], which is mentioned when motivating the present work, but not compared to it empirically. Other more recent relevant work is Rolling Diffusion [2], which the authors may choose to not discuss given how recently it was published.\n\n- **The code is not provided as part of the submission**: while this is optional, I believe it is a useful practice to ensure reproducibility. \n\n**nitpicks**\n- could you bold the best numbers in table 8 appendix D2?\n- 'Dyffusion' is misspelled in page 2, last paragraph\n\n[1] \"DYffusion: A Dynamics-informed Diffusion Model for Spatiotemporal Forecasting\", Salva Rühling Cachay, Bo Zhao, Hailey Joren, Rose Yu, NeurIPS 2023\n\n[2] \"Rolling Diffusion Models\", David Ruhe, Jonathan Heek, Tim Salimans, Emiel Hoogeboom, ICML 2024", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)