Improved Sampling Of Diffusion Models In Fluid Dynamics With Tweedie's Formula
Authors: Youssef Shehata
Benjamin Holzschuh
Nils Thuerey
Venue: ICLR 2025 Poster (2025)
Keywords: physics-based simulations
diffusion models
improved sampling
Primary area: applications to physical sciences (physics, chemistry, biology, etc.)
TL;DR: We propose an early truncation algorithm and an iterative refinement method to let diffusion models produce highly accurate results with low NFEs.

Abstract: State-of-the-art Denoising Diffusion Probabilistic Models (DDPMs) rely on an expensive sampling process with a large Number of Function Evaluations (NFEs) to provide high-fidelity predictions. This computational bottleneck renders diffusion models less appealing as surrogates for the spatio-temporal prediction of physics-based problems with long rollout horizons. We propose Truncated Sampling Models, enabling single-step and few-step sampling with elevated fidelity by simple truncation of the diffusion process, reducing the gap between DDPMs and deterministic single-step approaches. We also introduce a novel approach, Iterative Refinement, to sample pre-trained DDPMs by reformulating the generative process as a refinement process with few sampling steps. Both proposed methods enable significant improvements in accuracy compared to DDPMs, DDIMs, and EDMs with NFEs $\leq$ 10 on a diverse set of experiments, including incompressible and compressible turbulent flow and airfoil flow uncertainty simulations. Our proposed methods provide stable predictions for long rollout horizons in time-dependent problems and are able to learn all modes of the data distribution in steady-state problems with high uncertainty.

Published as a conference paper at ICLR 2025
IMPROVED SAMPLING OF DIFFUSION MODELS IN FLUID DYNAMICS WITH TWEEDIE'S S FORMULA
Youssef Shehata"
Benjamin Holzschuh
Nils Thuerey
Technical University of Munich 85748 Garching, Germany
ABSTRACT
State-of-the-art Denoising Diffusion Probabilistic Models (DDPMs) rely on an expensive sampling process with a large Number of Function Evaluations (NFEs) to provide high-fidelity predictions. This computational bottleneck renders dif- fusion models less appealing as surrogates for the spatio-temporal prediction of physics-based problems with long rollout horizons. We propose Truncated Sam- pling Models, enabling single-step and few-step sampling with elevated fidelity by simple truncation of the diffusion process, reducing the gap between DDPMs and deterministic single-step approaches. We also introduce novel approach, Iterative Refinement, to sample pre-trained DDPMs by reformulatnng the generative process refinement process with few sampling steps. Both proposed methods enable significant improvements in accuracy compared to DDPMs, DDIMs, and FDMs with NFEs S 10 on a diverse set of experiments, including incompressible and compressible turbulent flow and airfoil flow uncertainty simulations, Our proposed methods provide stable predictions for long rollout horizons in time-dependent problems and are able to learn all modes of the data distribution in steady-stat state problems with high uncertainty.
INTRODUCTION
Turbulent flow is prevalent in everyday phenomena ranging from natural occurrences (Sullivan and
McWilliams 2024 Pyakurel et al. 2017) to engineering applications (Tulapurkara) 1997 (
Cheah et al.
2007] Volpe et al. 2014). Computational Fluid Dynamics (CFD) is essential for understanding these flows, with direct numerical simulations being the gold standard. However, it requires high-resolution grids resolve the full spectrum of turbulent spatial and temporal scales, resulting in computationally intensive simulations (Popel (012).. This limitation has propelled the recent surge in data-driven approaches. Leveraging the abundance of high- and low-fidelity data, machine learning algorithms offer various opportunities to enhance the accuracy and efficiency of turbulence simulations (Vinuesa) and Brunton, (002),, particularly for phenomena like turbulent flows, which are challenging for traditional CFD methods.
Diffusion models (DMs) as surrogates. DMs (Hyvärinen) 2005 Soohi-Dickstein al. 2015 have demonstrated great potential in various domains (Dhariwal and Nichol 2021 Wang et a Lugmayr al.; Ho al. Li et al., 2023); however, their application
Quid-based problems remains an underexplored area of research. To date, applications involved inverse problems (Holzschuh et al. 2023), high-fidelity reconstruction (Shu al. (023), autoregressive sampling in two-dimensional Yang and Sommer 2023 Lippe et al., 2023; Kohl et al. 2024) and three-dimensional (Lienen et al. 2024) settings, and an uncertainty-an aware surrogate for airfoil simulations (Liu and Thuerey 2024).
Motivation for using DMs. DMs have been shown to autoregressively generate videos or simulation trajectories, which unconditionally stable over very long time horizons (Kohl et al. 2024). Temporal stability is difficult to accomplish using supervised loss terms, requiring memory techniques like multi-step unrolling (Um al. 22020). Additionally, the probabilistic nature of DMs can deal very well with measurement noise or missing data (Huang et al. 2024), making them highly
"Correspondence to: ly ww.shhata@@umm.de
'The source code is available at http::://gttbbbb.cm///m../
of

---

Published as a conference paper at ICLR 2025
robust and versatile. Since prediction from the DM samples from the posterior, this allows small variations due to uncertainty in the imput to naturally evolve and amplify over time, creating diverse predictions over many steps, while individual trajectories always remain physically accurate.
Need for faster and more accurate sampling. The main drawback of DMs, especially evident in fluid problems, is the long inference time due to the large Number of Function Evaluations (NFEs) required, and the limited accuracy compared to deterministic baselines (Cachay et al. 2023). Therefore, our objective in this study is to reduce the inference time disparity between DMs and single-step deterministic baselines while concurrently enhancing the accuracy of their predictions. This is achieved through our proposed straightforward training and sampling procedures.
The main contributions of this work can be summarized as follows:
1. We re-formulate the autoregressive problem to facilitate flexible sampling with extended parallelization and enable single networks to predict not only the next timestep but also intermediate states.
2. We introduce Truncated Sampling Models (TSMs) to enable single-step and few-step sam- pling while preserving or even augmenting sampling fidelity. We describe how truncation of the diffusion process, typically employed to reduce NFEs, can improve inference accuracy. Additionally, we distinguish our proposed TSMs from related approaches and highlight their efficiency and ease of implementation.
3. We introduce an inherently stochastic Iterative Refinement (IR) approach to enable flexible sampling of conditional diffusion models, allowing reduced NFEs with improved accuracy compared to ancestral sampling. We explain the intuition behind the approach and provide comparative analysis against existing methods throughout our experiments.
4. We empirically demonstrate the efficacy of the proposed methods in reducing inference steps and improving the accuracy of diffusion models for fluid dynamics simulations through diverse set of experiments, including compressible and incompressible turbulent flows in both time-dependent and steady-state settings.
2 BACKGROUND
2.1 DENOISING DIFFUSION PROBABILISTIC MODELS
Denoising Diffusion Probabilistic Models (DDPMs) (Ho et al. 2020 class of generative DMs, convert a data distribution g(zo) to prior distribution ((T)) ~ NO. I) over T steps through Markovian forward diffusion process g(x; x+-1) by gradually adding Gaussian noise with noise schedule 3, We can sample the state x, directly from zo through the parameterized closed form:
a(x xo) = Vauto + V
where ā. I og and or imated using neural network through a parameterized Gaussian distribution pe (#t-1 mz) = N (#1-1i μ4 (r,, Ee t)). where μe and Eg are the network predicted mean and variance, respectively. However, in this study, the variance is kept constant according to the noise schedule 3 and is chosen to be Ex t) = I with of Br.
3. The forward process posterior g( xx) is approx-
During inference, the reverse process (i.e., ancestral sampling) begins from TT NO I) and iteratively samples zt-1 ~ Pa ( xx)) for T steps until reaching a fully denoised state zg. The network is trained to estimate the forward process posterior by minimizing the Kullback-Leibler (KL) divergence (L(g(x;-1 xx)]lpo (#+-1 xz)), which reduces to the loss function (Ho et al. 2020
1-1 := Ex...
e es ev(aayo vI- a,e,
where the network only learns to predict the noise each noise step to perform partial denoising.
The denoising step is related to Tweedie's ' formula (Efron) 2011). which can be used to estimate the posterior mean E[co (14; 0) from a noisy sample xt via
E[ 0) (#+ ví aneo Vor-
2

---

Published as a conference paper at ICLR 2025
2.2 PROBLEM DEFINITION
The Navier-Stokes (NS) Partial Differential Equations (PDEs) represent a class of problems that epitomize the complex physics encountered in engineering and scientific disciplines. For an arbitrary domain S2, fluid motion is governed in space and time T by the NS PDE, defined as:
du θT + (u- vju= = -p+
f. 1 Au+ + Re
Op θT + + u) =
(4)
where f is external forcing, and u, p. and p are the velocity, pressure, and density, respectively. Re is the non-dimensional Reynolds number, controlling the severity of diffusive to convective transport.
Reformulated autoregressive sampling. For time-dependent problems, to reach the target state x(Tr), where x uu, p. p}. and with the initial condition Xg, our reformulation of autoregressive sampling is defined as (notice that the notations x(...) and x... are used interchangeably))
x(T,) = Pb (TT, P0 (... P (FT, xo.
(5)
where the prediction stride j denotes how far we sample in the future hased on the physical timestep
ōr. The shortened notation p denotes the DDPM iterative sampling, P/ (xr, Xo. = x(j-8r)
(p( (e Xo. .).) j). In essence, P3 maps any fluid state x(7) x(T+j--7)) without the need to estimate the intermediate states x(T + i or) Vi < j. as typically required by classical numerical solvers to satisfy the Courant-Friedrichs-LLeyy (CFL. convergence conditions (de Moura and Kubrusly, 2013). Although conditioning on has been previously explored in the context of multi- partmee conditioning (Gupta and Brandstetter] 2023). our contribution emphasizes the use of to facilitate flexible, parallclizable, and potentially more accurate diffusion sampling instead of next-step predictions.
Flexibility in predicting future states. By conditioning a surrogate model on j for j E (O..... T.. achieve two major benefits. First, the model can predict all possible intermediate states between r = 0 and 7 = T Ör. In comparison to methods such as DYffusion (Cachay et al.) (023) that require independent forecaster and temporal interpolator networks to achieve this task, our formulation enables a single network to predict the next timestep in addition to intermediate ones. Second, we are able balance between the accuracy of the first-step prediction and error accumulation (Lienen et al. 2024). For instance, smaller values would lead to first-step predictions with high accuracy; however, they would require longer rollout steps, leading to more error accumulation and vice versa. Thus, we will demonstrate in our experiments how an optimal value for j can lead to better accuracy than stxp sampling.
3 RELATED WORK
DMs as flow surrogates In inverse problems involving temporal evolution, Holzschuh et al. (2023) utilized DMs to predict a system's state backward in time, integrating an approximate inverse physics simulator into the sampling process. For super-resolution, [hhu et al. (2023) applied DDPMs to turbulent flows by reconstructing high-fidelity flow fields from low-fidelity inputs, achieving remarkable results even with sparse input data. Furthermore, regarding time-dependent autoregressive predictions, Yang and Sommer (2023) attempted to predict nonlinear fluid fields at specific points in time based initial conditions. Kohl et al. (2024) introduced an autoregressive conditional diffusion model (ACDM) capable of predicting fluid states over extended time horizons while maintaining sample quality and temporal stability, and provided benchmark results on various datasets and against multiple baselines. [ienen et al. (2024) explored spatio-tempore predictions in three-dimensional turbulent flows and achieved faster processing times than conventional solvers. Additionally, PDE- refiner facilitates the enhancement of sampling precision across all frequency components inherent in PDE solutions through a multistep refinement procedure (Lippe al. (023). Finally, an uncertainty- aware surrogate for steady-state airfoil turbulence was presented by Liu and Thuerey (2024) to predict the uncertainty of the simulations and provide samples from the learned ground truth distribution.
Expedited sampling. The slow sampling time of DMs is a major drawback, prompting extensive research to reduce the computational cost without compromising quality. Early endeavors included
3

---

Published as a conference paper at ICLR 2025
learning the variances of the reverse process and optimizing the noise schedule (Nichol and Dhariwal 2021). In tandem, (Song et al. (202I) presented Denoising Diffusion Implicit Models (DDIMs) that generalize DDPMs via class of non-Markovian diffusion processes instead of the Markovian diffusion process of DDPMs, leading to shorter deterministic generative processes. By expressing
DMs in common framework known elucidated DMs (EDMs), Karras al. (2022) introduce design space featuring separable design choices that can be optimized to attain expedited sampling with sfate-of-the-ar accuracy. Further, introducing Bespoke solvers represents novel framework for crafting custom ordinary differential equation solvers tailored to pre-trained models, yielding parameter-efficien solvers with negligible training overhead (Shaul et al. 2024). Moreover, distilla-
tion techniques (Luhman and Luhman 2021: [Salimans and Ho 2022 Meng et al. 2022 Sauer et al.) offer promising avenues for achieving high-fidelity image synthesis with diminished computation overhead through few steps and one-step inference, albeit at the cost of accuracy (Salimans and Ho 2022 Meng et al., 2022). Analogously, Song et al. (2023) propose consistency models, trained via distillation in isolation, to directly map noise to data and facilitate both single-step and few-step sampling. Finally, other endeavors include latent diffusion models (LDMs) (Rombach et al.) (0022 and Truncated Diffusion Probabilistic Models (TDPMs) (Zheng et al. 2023). LDMs compress imput states into latent spaces of reduced degrees of freedom through an additional encoder-decoder network, consequently reducing computational costs associated with DDPM sampling for the same/reduced NFEs. TDPMs truncate the last steps of the forward diffusion process, leading to shorter generative process starting from hidden noisy distribution by leveraging an additional generative adversarial network (GAN)- based implicit generator to match the prior to the aggregated posterior.
These diverse approaches collectively contribute to augmenting the generative process of DMs, addressing computational challenges, and enhancing sample quality. The two approaches that we introduce offer complementary solutions, particularly in the domain of physics- based simulations. and provide insights for the development of DMs in general.
4 NOVEL TRAINING AND SAMPLING APPROACHES
4.1 TSM: TRUNCATED SAMPLING MODEL
Motivation. An interesting phenomenon of DDPMs, particularly when trained with a linear 3r, is the ability to skip a small percentage (i.e., S 20%) of the reverse diffusion process while maintaining the sampling quality (Nichol and Dhariwal 2021 Furthermore, approaches have been devised to target a relevant range of noise levels during training by prioritizing intermediate noise levels (Karras) et al. 2022 Choi et al. 2022), thereby enhancing the loss per noise level.
We re-visit these approaches with a new perspective: We truncate significant part from the last steps of the reverse Markov chain with high skip percentage& s to reduce NFEs and focus the training on noise steps preceding the truncation. We refer a model trained for a limited part of the diffusion process and sampled with truncated ancestral sampling as Truncated Sampling Model (TSM). Focused training by restricting the sampling window for t parallels approaches by Karras et al. (2022) and Choi et al. (2022) to improve the loss per noise level (see Fq. 2. with our main objective to achieve enhanced sampling accuracy. Hence, we expect >> 0 to lead to better accuracy and reduced NFEs.
Sampling. Algorithm I summarizes the sampling procedure for conditional TSMs, with differences from ancestral sampling highlighted in blue. TSM sampling follows ancestral sampling until 21,, i.e., pe (T), := p(ZT) I Pe( 1xx) where t, = [s T] and E (0, 1]. At t = t.. instead of sampling -1 ~ (o ),,, we estimate it using the posterior mean B[ 0.. see Eq. 3
Training. The TSM training procedure involves the choice of the hyperparameter s (skip percentage) and its use as a lower bound for sampling diffusion steps. Hence, the sole adjustment to the DDPM training algorithm from Ho et al. (2020) is defined as ~ Uniform([ts. ., ...,..,Ty T)), Based on the skip percentage 8, typically > 0.2 for more pronounced outcomes, a balance between NFEs, sampling accuracy, and stochasticity can be achieved. Extreme skip percentages (e.g.. z 1) are feasible, resulting in unprecedented high-accuracy single-step diffusion sampling, albeit at the cost of reduced stochasticity. Hence, relatively lower s values are optimal for problems of stochastic nature to enable learning all modes of the data distribution.
4

---

Published as a conference paper at ICLR 2025
Algorithm 1 Truncated Ancestral Sampling for conditional TSMs
Require: (TSM), s (skip percentage), (condition)
1: 2T ~ _V(0, I)
2: for [- = T.
1. do 3: 2 ~~ 1=,N(0,1) ift 1, clse = =0
4: Tt-1 vor
[t VI-ui col co((ee,,e)) 002
DDPM sampling po( xx).
5: end for
6: if >> then 80 = (21, - vI co c))/ v))
7:
Tweedie's formula E[ 0| (see Eq. 3p.
8: return 20
TSMs vs TDPMs. We highlight the main differences between TDPMs (Zheng et al. 2023) and our proposed TSMs. First, TDPMs apply the truncation to the last steps of the diffusion process, whereas our approach focuses on these last steps and conversely truncates the first steps. Second, TDPMs rely on an additional implicit generator network (GAN) to match the prior with the aggregated posterior. This implicit generator requires joint training with the DDPM, thus adding extra complexity to the training process. On the other hand, our training procedure closely resembles the straightforward DDPM training procedure and doesn't It add complexity, thus rendering TSMs an appealing approach for enhancing sampling efficiency. Finally, while the sample quality of TDPMs is adversely impacted by large truncations, TSMs enable one-step inference with the same or improved sampling quality, relying on characteristics specific to fluid dynamics datasets as we explain in Section 5
4.2 IR: ITERATIVE REFINEMENT
Motivation. A fundamental characteristic of DMs is their iterative sampling process, which sys- tematically reduces noise from a pure Gaussian noise Zy to a noise-free state Ty over successive iterations. This ancestral sampling procedure is performed by gradually estimating all intermediate states Iy-1:1 until reaching a fully denoised state To. matching the training data distribution ((0)). Nevertheless, during training. DDPM network learns only predict the noise field at each noise level of a predetermined 3, independent of any adjacent states (see Eq. 2 Therefore, for a pre-trained DDPM model, various sampling methods can be employed without re-training, as long as they pertain to 3r originally used for training.
We leverage this property of DMs introduce an intuitive refinement algorithm novel sampling method for DDPMs, which we refer to as Iterative Refinement (IR). In IR sampling, we consider much shorter noise schedule y = {tr..... ,t,] [T..... 1} and interpret the different noise levels to essentially correspond to different levels of detail for a given state. We believe that, for any provided initial state Tinit, there exists a sequence y that defines the minimal number of noise levels (or levels
of detail) sufficient to augment the accuracy of Timi Hence, we optimize to ensure that the accuracy of the final prediction closely matches all levels of detail present in the ground truth state Zo.
Sampling. Algorithm summarizess the gen- erative process for IR. Given an initial state Timi (assumed to be noise-free. free, low-order ap- proximation of 20) and a refinement schedule 7. we iteratively apply forward diffusion followed by an estimation of the posterior mean to pre- dict a series of gradually enhanced approxima- tions of the noise-free state H, Vi = .....,,, where N = I71, at distinct noise levels. Con- versely, ancestral sampling po ( gradu- ally removes noise for the same schedule (as-
Algorithm 2 IR sampling procedure
Require: Cg (DDPM model), Zini (initial state),
7 (!r....,ee) (refinement schedule)
1: 40 f- Finit
2:
for Each in 7 do
3:
~ (0,
4:
It Varuo + /1 Oee
5:
D Forward diffusion g using Eq.
6:
ig = ,o, (a, - vI - d L c))
7: Tweedie's 's formula B[Go| 0] from Eq.
suming 7 {T ........ 1}) leading to intermediate states that are partially noisy. IR allows for flexible choice of the length and distribution of y to fit the problem under consideration and ensures higher accuracy of predictions using fewer NFEs
8: end for
9: return 20
compared ancestral sampling. We also argue that Zini be (partially) noisy state obtained
5

---

Published as a conference paper at ICLR 2025
through truncation of ancestral sampling, for example, even sampled from the prior distribution (ær), further relaxing the computational overhead before IR sampling.
Optimizing the refinement schedule The most critical component of IR sampling is y, balancing between inference quality and NFEs. Hence, an optimized 7y should consider the initial state Finit the model's accuracy Lę-1 at each noise step (defined as in Eq. 2). and the nature of the problem. All our proposed y schedules are optimized using greedy algorithm, which we found to easily lead highly satisfactory results with minimal effort and computational cost. More details regarding " optimization are presented in Appendix B A direct consequence of using this greeding algorithm that the final output átg provides a better approximation of zo than Finit and all preceding approximations:
[l =ol|a] < E|I =5olla] <... < E Billod xo < [[[] - =ool2]-
(6)
We believe that better results could be achieved through more sophisticated optimization of although this will incur additional training overhead.
Method novelty. While IR makes use of Tweedie's 's formula to obtain an intermediate approximation of the posterior mean ž.. it differs from the various resampling methods for posterior sampling (Song) jet al. 2024] Zhang et al., 2024) in the following aspects. (1) Freedom to choose Vinit and optimize refinement schedule
While all sampling methods begin the inference procedure by sampling Iy ~ (O, I). IR is more flexible as it enables the starting point zinst to be ØT. a low-fidee fiddeiility prediction, a partially noisy output. (2) IR efficiently samples fluid states using pre- pre-trained model without the need for an auxiliary neural network or an optimization task to enforce data consistency to .g. This reduces the computational cost during sampling and introduces flexibility in choosing Y without restrictions on requiring the estimates of do to be evaluated at noise steps close to 0 to ensure meaningful prediction (Yu et al. 2023). Further, while PDE-Refiner (Lippe al.) 2023) shares similarities with DDPMs and IR, our method differs in key aspects. Unlike training from scratch with fixed schedule. IR is a sampling algorithm for pre-trained DDPMs. This enables flexibility in choosing an optimal combination of y and Tinit, making it both a standalone sampling algorithm (when Finit (0, I)) and a refinement strategy for noisy or low-fidelity inputs.
Relation to DDIMs.
The general recursive sampling formula is defined by Song et al. 2021
Ft
Cre (1+a)
/ ش ( +o
c6 (x))
Ft- 1
0
0
direction pointing to mn
predicting
with a; = 0 resulting in a deterministic forward process. One can see that Eq. 7 consists of two main components, prediction of Ty and a direction pointing to Ig, exactly matching IR sampling from Algorithm 2 with a reversed order. For deterministic DDIMs, the predicted noise eo in Eq. 7 7is used to define the forward process instead of a randomly sampled Gaussian noise. This directly results from their choice of the mean function for the inference distribution q (xt-1 It, xo) (refer to Song et al. (202T) for more details). Despite the similarity, our results demonstrate that IR consistently outperforms deterministic DDIM sampling. We hypothesize that the stochastic nature of IR aids in rectifying errors incurred in earlier sampling steps. However, the dynamics of stochastic sampling complicated in practice and might introduce additional errors (Karras et al. 2022). Consequently, this observation may not generalize to other datasets and domains.
5
EXPERIMENTS
5.1 TEST CASES
We consider two-dimensional (2D) fluid flow test scenarios. including compressible transonic flow (Tra), incompressible forced turbulence (Fturb), and steady-state airfoil turbulence uncertainty (Air), as shown in Fig. [ Details regarding all datasets can be found in Appendix A These cases were selected to ensure diversity (similar to PI)EBench (Takamoto et al. (022)) and facilitate the assessment of various method aspects, such as temporal stability and stochasiticity. For the transient (i.e. time-dependent) cases, the prediction stride is defined as j E {0, 1...., 10). i.c.,
T= 10, and is provided as an additional input channel to the network. Details regarding training and Inffusion-relater hyperparameters for all test cases can be found in Appendix C

---

Published as a conference paper at ICLR 2025
Flub -lul
AV
Figure I: Showcasing diverse flow fields from our test cases. p is the dimensionless pressure.
Transonic flow (Tra). Benchmark dataset by Kohl et al. (2024) for compressible transonic flow over cylinder on a 128 x 64 grid including uz, My. P. and flow fields. The case becomes particularly challenging high Mach Numbers Ma due to the presence of shock waves, Interpolation int and extrapolation ext datasets for evaluation involve R = 60 timesteps with Mu E (0.66, 0.67, 0.68) and Ma E {0.50, 0.51, 0.52), respectively. Temporal stability is tested on Traiong with 240.
Forced turbulence (Fturb). We generate the dataset by solving the incompressible NS PDE with sinusoidal forcing term to obtain forced turbulence, i.e., Kolmogorov flow. We consider range of Re values with Re {200, 1000, 2500, 4000} for training, while testing is split into interpolation (int: Re = {1750)) and extrapolation (ext: Re = (100, 5000)) regions. The dataset includes Ue and My flow fields with 64 64 resolution. Autoregressive sampling employed for R = 30 timesteps for two sequences per Re. An additional dataset F Fturbiosg with R 120 is considered to assess temporal stability of the models (see Appendix A for more details). We train deterministic single-step baseline for comparison, using the same architecture as DDPMs without conditioning on Ie while using identical training parameters (see Appendix C).
Airfoil turbulence uncertainty (Air). The benchmark dataset by Liu and Thuerey (2024) models the inherent uncertainty of steady-state airfoil flow simulations with various airfoil profiles, Re values, and angles of attack. For each combination of parameters, 25 solutions are obtained through different solver settings (e.g., number of iterations), representing the data uncertainty. Two main studies considered, (I) Airone considers the uncertainty due to Re only while keeping other parameters fixed (2) Ai IMuiti considers the uncertainty arising from all parameters. Air AirMii limited to 1250 different configurations and both studies are restricted to a 32 x 32 resolution.
5.2 RESULTS
Evaluation metries. We evaluate the accuracy of DMs against the ground truth data through various metrics. For the Fturb and Tra cases, consider the temporal-average MSE (TA-MSE), turbulent kinetic energy spectrum (TKE), domain-wide wide kinetic energy (DWKE), temporal correlation p for the absolute velocity |ul, and temporal stability for significantly long rollout horizons. For results obtained with j > 1, we restrict the analysis this section to sampling with the stride j without sampling the intermediate states (i.e., we only make [R/j] predictions reach the target timestep). Details regarding the sampling of the intermediate states are presented Appendix [D] In the Air case, the focus is on evaluating how accurately the data distribution, characterized by the mean μ and standard deviation a for each channel y. is captured by the surrogates by evaluating (MSE, )a and (MSE, a. where (-)a denotes the average of field.
We focus our evaluation on the speedup obtained via a reduction in NFEs, relying on the fact that we utilize the same architectures with almost the same number of parameters for all models. Training and sampling for all test cases were carried out using NVIDIA GeForce RTX 2080 Ti GPU. This section is solely dedicated to quantitative analysis of the top- perfoornnnn models. We provide our comprehensive set of results as well as other baselines (such as ResNet, Fourier neural operators (FNO), PDE-Refiner, latent-space transformers (TF), and Heteroscedastic models) in Appendix E. Qualitative samples provided in Appendix E.2 Moreover, compare methods against DDPMs, DDIMs, and EDMs and exclude other expedited sampling methods outlined in Section 3 based on the following considerations. Distillation techniques require extensive retraining without yielding significant accuracy improvements over the teacher model. Bespoke solvers introduce excessive complexity and resource demands to manage varying strides j in our autoregressive formulation.
7

---

Published as a conference paper at ICLR 2025
Top performing models for transient cases. We present TA-MSE results for the Tra and Fturb cases in Table Using j = 1, our approaches significantly surpass DDPMs and DDIMs accuracy while requiring only fraction of the NFEs, as low single-step inference, whereas EDM exhibits variable performance. In Tra (see Table left), our approaches transcend the benchmark ACDM (Kohl al. 2024) with x4 and x20 (i.e., single-step inference) speedup for IR and TSM, respectively. Additionally, the single-step TSM outperforms the best baseline from Kohl et al. (2024), while IR yields marginally lower accuracy. The most accurate EDM reported here, with comprehensive tests for various samplers detailed in Fig. (Appendix E.I. demonstrating marginal increase in accuracy compared to TSM, albeit requiring 4x the NFEs.
Regarding the Fturb case (see right of Table Db. while single-step TSMs demonstrate sufficient ac- curacy compared to expensive DDPMs and DDIMs, they prove suboptimal to the baseline. Although the model "TSM T80 =0.75" with 20 NFEs (see T Appendix F.1) marginally outperforms the baseline, its increased computational cost renders less favorable for our objective. In contrast to Tra, IR sampling with 10 NFEs provides the most accurate results, superseding the baseline, 80-step DDPM, 40-step DDIM, 10-step EDM, and 20-step TSM. Also, EDM found to be on par with the single-step TSM in terms of accuracy but is ten times slower.
Optimal j. For optimal j values greater than 1 in both test cases (see Appendix E.1 all models, except EDMs, reduce the TA-MSE significantly compared to values obtained by -1. While this improvement is seen only when sampling with the main stride j. we demonstrate in Appendix [ how
sampling the intermediate states could further reduce the TA-MSE, especially for large j.
Temporal stability and physics-based metrics. In addition to TA-MSE, we examine the correlation between the predicted |u| to the ground truth over time for both Tra and Fturb cases. As shown in Fig. 8 top TSM and IR models with the lowest TA-MSE consistently exhibit high correlation to the ground truth across all timesteps, further confirming the superiority of our approaches in improving over DDPMs Also, in Fig. 9 we showcase the temporal stability of the models over extended rollouts as they maintain physically-coniiste from the ground truth. Furthermore, shown in Fig. 2 the TKE and DWKE plots indicate that both physics- metrics are consistent with the TA-MSE results presented in Table T demonstrating agreement in models accuracy, except for the top performance by EDM for the DWKE metric.
results even after the predictions have decorrelated
Foundations of method's performance. Our methods deliver superior results due to the effective use of Tweedie's s formula in conjunction with the characteristics of the data distribution. While Tweedie's 's formula introduces biases for multimodal distributions, we demonstrated their applicability to predominantly unimodal distributions in fluid dynamics (Shu et al. 2023 Yang and Sommer 2023). As result, its application in our work does not lead to any bias, leading to single- and step sampling with the same or improved accuracy. Moreover, since typical fluid datasets are limited to relatively coarse resolutions, they inherently lack high- high-frequency details, allowing the truncation of the last steps in the reverse process without loss of information, see Fig, 2a
Air top performing models. Table 2 summarizes the top results for (MSE,,,)s and (MSE, )o in both One and Multi cases. In the One case, our enhancement over the benchmark from Liu and Thuerey 2024) is achieved by using a DDPM with a linear 3r and normalizing the dataset.
Table 1: TA-MSE values for the top performing models in the Tra (left) and Fturb (right) cases. The table shows how our TSMs and IR sampling can yield highly accurate results with NFEs 5 for Tra and NFES < 10 for Fturb. UNet,, refers to UNet with unrolled training. In IR, T100/T80 is the base model, is Gaussian noise used as Tiits and the absence of y implies linear sampling steps. Standard deviation values are estimated over all timesteps and multiple samples.
Tea (10-9)
Pturo (10-2)
Model
NFEs
ext
int
Model
NFEs
ext
int
ACDM T20 Kohl at. al. 2024
20
2.3 1.4
2.7
2.1
UNet. (Kohi 2014
1
1.6
1.5
1.5
Baseline
3.95 + 3.84
482 + 5.23
DDPM T100
100
3.7
4.1 3.7
DDPM T80
80
6.16
2 6.54
5.27
5.55
EDM Deterministic Euler
1.3 1.3
1.0
EDM Stochastic Euler
4.75
4.553
4.78
4.26
DDIM T20
10
3.2
4. 3.9
DDIM T80
40
4.31
+ 4.62
6.97
6.84
IR T100- N' (Ours)
1.6
2.0 1.7
TSO NN (Ours)
2.93
3.34
1.70
166
TSM T100 =1 (Ours)
1.2
1.5
TSM T100 =1 (Ours)
1
5.00
5.52
4.39
8

---

Published as a conference paper at ICLR 2025
Ptubb
Tra
Fturb
10
10
0.6
0.4
67
DOIN
10
GF
DBM 10urst
0
DBAM
(ouns)
DDPN EDM
10-4
EDM
TSM 10un)
TSM (Oun)
10-1
10a
10-:
10
0.0
DDPM
EDM
DDIM
(Ourd
TSM (Ours)
Wavenumber
Wavenumber
(a) Scaled turbulent kinetic energy spectrum (TKE)
(b) DWKE
Figure 2: (a) TKE= = [()) //(,) where N is the total number of nodes in the domain and ū = F(u), i.e., the Fourier transform of the velocity u. (b) Correlation coefficient p for the time evolution of the DWKE, which deseribes the total kinetic energy of the system DWKE(u)
* + Ei=1 2212/2. High p means the predicted states highly-correlate with the ground truth with respect to the temporal evolution of the DWKE. Both plots are based on the entire test dataset.
N
Table 2: Air test case results for the top-performing models. Our proposed models (TSM and IR) for Air (left) provide highly accurate estimation of the data distribution for both μ and with up to x20 speedup. In A Twiles (right), only the TSM supersedes the benchmark model in both parameters. In IR, 006 refers to Pinit obtained by truncated sampling. MModel uses cosine fa.
kl.. (10-1)
(10--)
Midel
NFEs
(MSE,,,
(MSE,,. lo
Model
NPEs
(MSE,,
(MSE,,
DDPM T20IC"
Liu and Thmerey 2024
200
3.79
a: 0.27
8,40 a 0.69
DDPM T2HOC" (Lu and Thursey 2014
200
1.76
= 0.23
0,99 e 0.12
DDPM 1200
200
2.88 + 0.26
7,05 + 0.22
DIPM TIDO
100
2 1.31
0.57 0,83
EDM Diteminiticc Heun
20
8.13 2: 1.28
10.1 1.67
EDM Stochastic Enler
40
1.13 = 0,39
0.67 = 0.11
DDEM T220
100
3.68
à 0.44
7.34 0.22
DDIM T100
50
2112
= 0.99
0.64 2 0.25
T -0.5 (Ours)
41
2.87
+ 0.32
6.76 0.20
T100- -0.6- (mi))
41
207
- 0:59
0.59 +004
TSM T100 -0.9 (Ours)
10
3.30
0.30
5,89 + 0.33
TSM T100 -0.75 (Ouzs)
25
1666
0.53
0.64 0.04
Comparable accuracy is obtained using 10 NFES (i.e., x20 speedup) by our TSM, whereas the best results achieved by IR with 41 NFHs followed by suboptimal results from DDIM with 100 NFEs. Furthermore, Fig. depectcss the data distribution over increasing Re, backing up our claims regarding the accuracy of the low-NFEs models. EDMs perform poorly on this dataset, and exhibit difficulties converging to smooth target states. However, for Mu Multi, they achieve the lowest errors for My and are only competitive for dg with NFEs 2 40, see Fig. [O (right). Notably, TSM requires at least 25 NFEs ( (x2.5 more than in One) to learn the full data distribution with high accuracy, while DDPM and DDIM models yield optimal results with the NFES used One. We believe further hyperparameter tuning could potentially improve the TSM results with less NFEs.
Further analysis of TSMs. We analyze the effect of on the accuracy of TSMs compared to ancestral sampling without truncation for all test cases. As shown in Fig. 3] TSMs unconditionally outperform or at least match the accuracy of DDPMs with much fewer NFEs, depending on 8. Substantial enhancement in accuracy with few-step sampling (i.e., high s) is made possible in the Tra and Fturb cases; however, in the Air rir test case, for the top performing linear 3e-based models with 0.5, we only able to reduce NFEs for the same accuracy of DDPM. Noteworthy, for 1 in the Air case, the output is slightly noisy regardless of the satisfactory accuracy. For the cosine Be-based model, we evidently see significant improvement to DDPM, which performs quite poorly and, accordingly, even the best TSM results for this case exhibit inadequate accuracy.
Further analysis of IR. In Fig. 4b we compare IR against DDIMs with varying NFEs for identical sampling schedules in transient cases. Besides IR's consistency in transcending DDIMs across various base DDPMs, whereas IR does not manifest a noticeable trend. The stochastic nature of IR makes it difficult to find clear relations across different datasets and NFES as it is by-e-b tuned (Karras al. 2022).
notice a discernible correlation between NFEs and accuracy for DDIMs,
Limitations. In TSMs, since the training was focused on a limited part of the diffusion process, they exhibit inflexible sampling; neither DDIMs nor IR could be applied unless the sampling steps t = (taans.... fend} C (T....,,t) which is not the case for high s values. Also, while IR and TSMs enable single- and few-step sampling with increased accuracy compared ancestral sampling,

---

Published as a conference paper at ICLR 2025
1.50
1.50
1.50
T-300
T-
1,00
3.00
1.00
0.75
0.75
0.75
Nomm
025
0.25
0.25
Mar. UNo-UN
0.00
0.00
0.00
Figure 3: Effect of on TSMs performance for all test cases using 1 (for transient problems). Evaluation metrics are normalized by the s = value; hence, values lower than demonstrate higher accuracy of TSM compared to DDPM with no truncation. The shaded regions represent the standard deviation from both ext and int regions, and multiple samples.
Afrise
Flurb
0.10
1.0
DOMM -7000
COIN -20
2.5
0:3 T50
02IN 750
0.08
Moan TAMMSE MGMEE
11.
P.
120 (0
T202 10um)
2.0
(R- (R-750 chun))
(R- (Dus)
3 0.06
Ground Tuch
2.5
0.04
DOPM
EDM
0.4
Nove
1.0
0.02
0.5
TSM (Ours)
0.0
0.2
0.4
0.8
1.0
0.0
0.4
0.2
0.3
Ra
X107
NFEST
NFEES/T
(a) yy. plot for Ai Tone case.
IR DDIM for transient test cases.
Figure 4: (a) Comparing the average By of the models presented in Table 2 for the Airane case with increasing Re. Ground truth and Benchmark plots are obtained from Liu and Thuerey (2024). (b) Comparing the sampling accuracy of IR against DDIM for the Tra (left) and Fturb (right) cases using j = and the same linear sampling schedule. The plot shows consistency by IR to provide better estimations than DDIM for both base DDPMs. The shaded regions represent the standard deviation from both ext and int regions, and multiple samples.
they are not guaranteed to supersede the accuracy of deterministic baselines. However, we have empirically shown that methods improve over DDPMs, DDIMs, and EDMs in terms of speed and/or accuracy across our diverse datasets.
CONCLUSION
We have introduced two novel training and sampling approaches to enable single- and few-step sampling DDPMs without compromising inference quality. Our first contribution is a Truncated Sampling Model (TSM), capable of achieving single-step inference while maintaining enhancing accuracy through early truncation of the reverse process. Additionally, our second contribution, Iterative Refinement (IR), targets pre-trained DDPMs by formulating the sampling process as a refinement endeavor to facilitate high-fidelity inference with reduced NFEs compared to existing sampling methods, such DDIMs. We have showcased the efficacy of TSMs and IR in minimizing the disparity between DDPMs and deterministic baselines across a diverse set of experiments, assessing various facets of our approaches. Our proposed methods significantly enhance sampling speed and quality in fluid dynamics simulations and we posit their potential applicability in other domains for which diffusion models are considered sate-of-the--ar
Future work. For TSMs, transfer learning is desirable alternative to aid in reducing the computa- tional burden of training from scratch. Additionally, integrating TSMs and IR together and with other enhancement methods holds promise further improving the accuracy of DMs, relying on their flexibility for seamless integration with various techniques such as guidance (Dhariwal and Nichol
2021 Ho and Salimans 2021 Hong et al. 2023), ) distillation (Luhman and Luhman, 2021] Salimans and Ho 2022). and latent diffusion models (Rombach et al.] 2022). Finally, through proposed formulation of the spatio-tempon prediction task, flexible sampling with novel sampling schemes
(similar to Harvey et al. (2022) is another interesting venue of future research.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
The authors would like to thank Björn List for his valuable comments and the insightful discussions during the early stages of this work. We also express our gratitude Georg Kohl and Qiang Liu for their support and detailed clarifications regarding the Tra and Air test cases.
REFERENCES
Salva Rühling Cachay, Bo Zhao, Hailey James, and Rose Yu. Dyffusion: A dynamics-informed diffu- sion model for spatiotemporal forecasting. Thirty-seventh Conference on Neural Information Processing Systems, 2023.
K. W. Cheah, T. S. Lee, S. H. Winoto, and Z. M. Zhao. Numerical flow simulation in centrifugal pump at design and off-design conditions. International Journal of Rotating Machinery, 2007:1 2007. ISSN 1023-621X. doi: 10.1155/2007/83444
J. Choi, J. Lee, C. Shin, S. Kim, H. Kim, and S. Yoon. Perception prioritized training of diffusion models. In Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
Carlos A. de Moura and Carlos S. Kubrusly. The Frienrrchsriiiiics----- Lewy (CFL) Condition. Birkhääser Boston, Boston, 2013. ISBN 978-0-8176-83931 doi: 10.1007/978-0-88 17777-00-817--8944
Prafulla Dhariwal and Alexander Quinn Nichol. Diffusion models beat gans on image synthesis. In Advances in Neural Information Processing Systems, 2021.
Bradley Efron. Tweedie's formula and selection bias. Journal of the American Statistical Association 1662496) 1614, 2011. ISSN 01621459. URL http: /www http://wwwwssooooro/./.aaaaaaaa 23239562
Jayesh Gupta and Johannes Brandstetter. Towards multi-spatioteempraalscco generalized pde modeling. Transactions on Machine Learning Research, 2023.
William Harvey, Saeid Naderiparizi, Vaden Masrani, Christian Dietrich Weilbach, and Frank Wood. Flexible diffusion modeling of long videos. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems, 2022.
Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. In NeurIPS 2021 Workshop Deep Generative Models and Downstream Applications, 2021.
Jonathan Ho, William Chan, Chitwan Saharia, Jay Whang, Ruiqi Gao, Alexey Gritsenko, Diederik P. Kingma, Ben Poole, Mohammad Norouzi, David J. Fleet, and Tim Salimans. Imagen video: High definition video generation with diffusion models.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In Advances in Neural Information Processing Systems, 2020.
Jonathan Ho, Tim Salimans, Alexey A. Gritsenko, William Chan, Mohammad Norouzi, and David J. Fleet. Video diffusion models. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances Neural Information Processing Systems, 2022.
Philipp Holl, Vladlen Koltun, Kiwon Um, and Nils Thuerey. phiflow: A differentiable pde solving framework for deep learning via physical simulations. In Neural Information Processing Systems (NeurIPS) DifCCVGP workshop, 2020.
Benjamin Holzschuh, Simona Vegetti, and Nils Thuerey. Solving inverse physics problems with score matching. Thirty-sevnnthh Conference on Neural Information Processing Systems, 2023.
Susung Hong, Gyuseong Lee, Wooseok Jang, and Seungryong Kim. Improving sample quality of diffusion models using self-attention guidance. In IEEE/CVF International Conference Computer Vision (ICCV), 2023.
Jiahe Huang, Guandao Yang, Zichen Wang, and Jeong Joon Park. Diffusionpde: Generative pde- solving under partial observation, 2024.
11

---

Published as a conference paper at ICLR 2025
Aapo Hyvärinen. Estimation of non-normalized statistical models by score matching, Journal of Machine Learning Research, 6(24):695 709, 2005. URL http: /tttp:://mmmr...///e///// wwwp://ggpppppppppppeo
lyyariinen0a..amm
Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion- based generative models. In Advances in Neural Information Processing Systems, 2022.
Dmitrii Kochkov, Jamie A. Smith, Ayya Alieva, Qing Wang, Michael P. Brenner, and Stephan Hoyer. Machine carning-accelerated computational fluid dynamics. Proceedings of the National Academy of Sciences of the United States of America, 1 118(1)) 2021. doi: 10. 10.1073/pnas.22 21017841 18.
Georg Kohl, Li-Wei Chen, and Nils Thuerey. Benchmarking autoregressive conditional diffusion models for turbulent flow simulation. 2024.
Yifan Li, Kun Zhou, Wayne Xin Zhao, and Ji-Rong Wen. Diffusion models for non-autoregressive text generation: A survey. In Peter Stone and Edith Elkind, editors, Proceedings of the Thirly-Second International Joint Conference on Artificial Intelligence, pages 0992-6701, California, 2023. International Joint Conferences on Artificial Intelligence Organization. ISBN 978- 978-1-956792-00-4 doi: 00224996/iccai222237750
Marten Lienen, David Lüdke, Jan MainsenPPaluus, and Stephan Günnemann. From zero to turbulence: Generative modeling for 3d flow simulation. In The Twelfth International Conference Learning Representations, 2024.
Phillip Lippe, Bastiaan S. Veeling, Paris Perdikaris, Richard E Turner, and Johannes Brandstetter. Pde-refiner: Achieving accurate long rollouts with neural pde solvers. In Thirty-seventh Conference on Neural Information Processing Systems, 2023.
Qiang Liu and Nils Thuerey. Uncertainty-aware surrogate models for airfoil flow simulations with denoising diffusion probabilistic models. AIAA Journal, pages 1-22, 2024. ISSN 0001-1452. doi: J063440..
Andreas Lugmayr, Martin Danelljan, Andres Romero, Fisher Yu, Radu Timofte, and Lue van Gool. Repaint: Inpainting using denoising diffusion probabilistic models. URL URLphttpp//arxiv.orgg pdf / 2201. 09865v4
Eric Luhman and Troy Luhman. Knowledge distillation in iterative generative models for improved sampling speed, 2021.
Chenlin Meng, Ruiqi Gao, Diederik P Kingma, Stefano Ermon, Jonathan Ho, and Tim Salimans. On distillation of guided diffusion models. In NeurIPS 2022 Workshop -asee Methods, 2022.
Haomiao Ni, Changhao Shi, Kai Li, Sharon X. Huang, and Martin Renqiang Min. Conditional image-to-video generation with latent flow diffusion models. In 2023 IEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 18444-18455. IEFF, 2023. ISBN 179-8-3503--112-8. doi: 10. 1LL109/CVVPR222222222..0.00
Alex Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. 2021.
Stephen B. Pope. Turbulent Flows. Cambridge University Press, 2012. ISBN 9780521591256 doi:
10.1017/CB09780511 1E111188033..
Parakram Pyakurel, James H. VanZwieten, Manhar Dhanak, and Nikolaos I. Xiros. Numerical modeling of turbulence and its effect on ocean current turbines. International Journal of Marine
Energy, 97, 2017. ISSN 1214-1669.
doi: 10. 1010016/.jjmme22007.11.000 URL https
www.sciennedirret.tc rect. ctm/ htt.a//www.....cee.e//.e//aaaaaa
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition (CVPR), 2022.
Tim Salimans and Jonathan Ho. Progressive distillation for fast sampling of diffusion models. In International Conference Learning Representations, 2022.
12

---

Published as a conference paper at ICLR 2025
Axel Sauer, Dominik Lorenz, Andreas Blattmann, and Robin Rombach. Adversarial diffusion distillation. URL
170422
Neta Shaul, Juan Perez, Ricky T. Q. Chen, Ali Thabet, Albert Pumarola, and Yaron Lipman. Be- spoke solvers for generative flow models. In The Twelfth International Conference on Learning Representations, 2024.
Dule Shu, Zijie Li, and Amir Barati Farimani. A physics-informed diffusion model for high-fidelity flow field reconstruction. Journal of Computational Physics, 478:111972, 2023, ISSN 0021-9991. doi: J0. 2tt.:ppp2222.11199999 11972. URL Ttw.iiceepp1/S02219992222
com/ science/
Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsuper- vised learning using nonequilibrium thermodynamics. In Francis Bach and David Blei, editors, Proceedings of the 32nd International Conference on Machine Learning, volume 37 of Pro- ceedings of Machine Learning Research, pages 2265, Lille, France, 2015. PMLR. URL mlr wtt.ll...eess..../.. /ttp:///ooooooann....prees/e//// witt:/ooooddistsiiiii.......
Bowen Song, Soo Min Kwon, Zecheng Zhang, Xinyu Hu, Qing Qu, and Liyue Shen. Solving inverse problems with latent diffusion models via hard data consistency, 2024.
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In Interna- tional Conference on Learning Representations, 2021.
Yang Song, Prafulla Dhariwal, Mark Chen, and Ilya Sutskever. Consistency models. In Proceedings of the 40th International Conference on Machine Learning, ICML'23. JMLR.org, 2023.
Peter Sullivan and James C. McWilliams. Oceanic frontal turbulence. Journal of Physical Oceanography, 54(2):333-358, 2024. ISSN 0022-3670. doi: 10.1175/P0-D-23-003..1
Makoto Takamoto, Timothy Praditia, Raphael Leiteritz, Dan MacKinlay, Francesco Alesiani, Dirk
Pflüger, and Mathias Niepert. Pdebench: An extensive benchmark for scientific machine learning. In Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2022.
E. G. Tulapurkara.
Turbulence models for the computation of flow past airplanes. Progress in Aerospace Sciences, 33(1-2):71-165, 1997. ISSN 03760421. doi: 10.1016650366-0421996) 00002-4.
Kiwon Um. Robert Brand, Yun Fei, Philipp Holl, and Nils Thuerey. Solver-in-the-Lopp Learning from Differentiable Physies to Interact with Iterative PDE-Solvers. Advances in Neural Information Processing Systems, 2020.
Ricardo Vinuesa and Steven L. Brunton. Enhancing computational fluid dynamics with machine learning. Nature computational science, 266):358-366, 2022. doi: 10.1038443588-02--0222
Raffaele Volpe, Valérie Ferrand, Arthur Da Silva, and Luis Le Moyne. Forces and flow structures evolution on a car body in a sudden crosswind. Journal of Wind Engineering and Industrial Aerodynamics, 128: 125, 2014. ISSN 01676105. doi: j.0aa./.wweia.224.......
Weilun Wang, Jianmin Bao, Wengang Zhou, Dongdong Chen, Dong Chen, Lu Yuan, and Houqiang Li. Semantic image synthesis via diffusion models.
Jay Zhangjie WU, Yixiao Ge, Xintao Wang, Stan Weixian Lei, Yuchao Gu, Yufei Shi, Wynne Hsu, Ying Shan, Xiaohu Qie, and Mike Zheng Shou. Tune-a-video: One-shot tuning of image diffusion models for text-to-video generation. In 2023 IEEE/CVF International Conference on Computer Vision (ICCV), pages 7589-7599, IEEE, 2023. ISBN 979-8-3503-0718-4, doi: 10. 11110//CC511000020000 200.0002300000
Gefan Yang and Stefan Sommer. A denoising diffusion model for fluid field prediction. 2023.
Jiwen Yu, Yinhuai Wang, Chen Zhao, Bernard Ghanem, and Jian Zhang. Freedom: Training- free mnergy-guided conditional diffusion model. Proceedings of the IEEE/CVF International Conference Computer Vision (ICCV), 2023.
13

---

Published as a conference paper at ICLR 2025
Bingliang Zhang. Wenda Chu, Julius Berner, Chenlin Meng. Anima Anandkumar, and Yang Song. Improving diffusion inverse problem solving with decoupled noise annealing, 2024.
Huangjie Zheng, Pengcheng He, Weizhu Chen, and Mingyuan Zhou. Truncated diffusion probabilistic models and diffusion-based adversarial auto-encoders. In The Eleventh International Conference on Learning Representations, 2023.
14

---

Published as a conference paper at ICLR 2025
A EXPERIMENTAL SETUP
We summarize the details for all our datasets in Table 3 For detailed information regarding the generation of these datasets, please refer to the corresponding papers for Tra (Kohl et al.] 2024) and Air (Liu and Thuerey 2024). For the Tra and Fturb cases, only a single parameter is varied, namely Re, and is provided
input channel to all models. While in Tra we consider velocity (u, and uy), pressure (p), and density (p) fields as input and output fields, the velocity fields are only considered in Fturb. Additionally, for both datasets, the models are conditioned on the prediction stride j. provided as an additional input channel to the network.
In the Air case, there are three main parameters being changed: airfoil shape S, angle of attack (AoA), and Re. These parameters are provided as three input fields/channels to the network and the model outputs the velocity and pressure dimensionless fields. The number of outputs obtained from the different solver settings (referred to as "Sequences per Param" in Table are fixed to 25 solutions for every parameter configuration. For the Fire case, S is fixed to the raf30 airfoil shape and an AoA of 20", while in Ai Airmitt, the three parameters are varied.
Generating the Fturb dataset. We solve the incompressible form of the NS PDE (Eq. with Op/0r= = 0) using a spatially varying sinusoidal forcing term f = sin (4y)² 0.1u (similar to Kochkov al. (20II) for a range of Re values for training and testing. Using Row (Holl al.. 2020), we discretize a square domain ! = [0, 2T] x [0, 2T] on 128 x 128 cartesian grid. Both spatial derivative terms (i.e., the advection and diffusion terms) are discretized using a 6th-rr implicit finite difference method and are solved iteratively using the Conjugate Gradient (CG) solver. The time advancement scheme uses a -dderr Runge- term with -order accuracy using the CG solver. In addition, this step ensures divergence- flow velocity. Each RK-4 step consists of 25 iterations to ensure convergence at each timestep. The outputs are then downsampled to 64 x 64 resolution. To augment the difficulty of the problem, record each snapshot after 25 timesteps from the solver (i.e., we consider a temporal stride of 25 frames) with CFL 0.7 and variable år. Details regarding the number of sequences generated per Re and the total number of frames used for training and testing can be found in Table [3 The only difference between the sequences for each Re the initial condition, defined a random noise based on a distinct random seed. To ensure fully convergent flow fields, each simulation is run for 20 timesteps (including the temporal stride) as a warmup before the outputs are recorded.
scheme, which also handles the pressure gradient
Table 3: Parameter values for all datasets. *Using 1387 airfoil shapes. ** **Uiing 30 airfoil shapes.
Dataset
Param Purpose
Values
Sequences per Param
Total Sequences
Total Frames
Tra (128 64)
Ma
training
{0.53, 0.54..................... 0.66) J
0.7.9,0.70......
33
501
16533
test
ext: (0.50, 0.5).
int: [0.66, 0.67.0668)
6
60
360
long: [0.6410.65) 0.65)
4
240
960
Fturb
Re
training
(200, 1000, 2000)
240
960
48960
(64 x 144)
test
ext: [100) 5000)
2
4
30
120
int: {1750)
60
Long: [100, 5000)
6
120
720
Aix
Ri
training
(1.5x10f_ 3.5 3.5x110 5.5x11077.55100
7.5x119,..5x1100 905x
25
125
(32%) 32)
test
ext: (0* 10.5x 10")
25
50
int: (2.5x 10", 4.5x 100.10...... 6.5x10' 8.5x10")
25
100
Re &
training*
Re: (10', 10"), (-2.5- 22.5)
25
31250
(32 x 32)
AoA S
test
Re: 10", 10%) Li (10", 101)
25
3250
AoA: (-25, - 22. (22.5,
B REFINEMENT SCHEDULES FOR IR
As mentioned in Section optimization of the IR schedules y is limited to straightforward greedy optimization, presented in Algorithm 3 We start the optimization by choosing the max possible length of (N) and a value for the first refinement step (K). If K is not provided, the default starting value is T (i.e., the number of noise steps of the pre-trained DDPM). For the first step, we consider K possible options for refinement, i.e., {a E Z 1 S < K). As long as the validation loss L is decreasing, we keep looping over all K values. However, if the loss does not improve after tol steps (i.e., a tolerance set by the user to achieve early stopping for non-promising optimizations),
15

---

Published as a conference paper at ICLR 2025
continue to the next refinement step. The starting point for the next timestep is defined to be step smaller the current optimized step to further reduce the computational cost. for the current step no value found to reduce the current best loss Lueat the entire optimization is terminated, leading to the loss in Eq. 6 even if | < N. These restrictions are imposed to reduce the computational cost of optimization, though they might lead to non-optimal results. More expensive gradient-based and gradient-free optimization algorithms offer the potential to yield better results; however, we believe that the accuracy gain would not compensate for the concomitant computational cost.
Algorithm 3 Greedy optimization of y with early stopping
Require: N (Max length of 7), K (starting value)
Require: tol (tolerance value), Tini (Initial state, 69 (pre-trained DDPM)
7 1- 0
2: for = to N do
3:
Theet <- 00
4: topt -1
5: counter f- 0
6:
for j K to do Thmm +- append(y, ()
Go over all possible values. D Update "hemp by appending step j to ty (unchanged). & Evaluate Thep the validation dataset. b Update the current step if accuracy is higher.
7:
8:
L recursive satpaissmmmlllniiii. Themp co)
9:
if <<< Liess then
Lieett 4- L topt f- counter
11:
13:
else
counter f- counter +1 if counter > tol then break
15:
Terminate current step: failed to improve Lbest after tol trials.
17:
end if end
18:
19: end for
20: if lop =-1 then
break 22: end if
21:
Terminate optimization: improvement found.
23:
append(r, lopt)
24: K + topt -
D Consider only smaller values for the next refinement step.
end for
26:
return
Using the greedy algorithm, the optimization schedules utilized in all our experiments are summarized as follows (and visualized in Figure 5):
71, = 0.805 - 0.2i, =i= 0,1,2,3,4
72,1 0.805 0.1i,
Vi =0,1,2, ... 8, Vi =,1,, 13,
(8)
73.1 0.655 - 0.05i,
74,1 0.905 0.05i, Vi 0,1,,, 18.
Also, 75 = (1/T). When no y schedule is provided, we use linear sampling schedule by default:
Nineur (N)
fir E 0 for N e
(9)
73
7
0.2
0.4
01.6
0..8
4/T
Figure 5: Illustration for the refinement schedules defined in Equation 8 Each node represents noise step. All schedules start from the node with the largest t/T.
16

---

Published as a conference paper at ICLR 2025
C SIMULATION PARAMETERS
The training hyperparameters used for all test cases are presented in Table Our training is limited to the L2 loss for DDPMs, TSMs, EDMs and any baselines. We employ an overlap by one frame (see Appendix[Fjoo details regarding dataset overlapping) for the Fturb case, except for the case of = 1 in TSMs where we overlap 6 frames, since the training procedure was unstable for the first few epochs, leading to suboptimal results. However, we stop the training at half the number of epochs reported in to maintain the same number of training iterations for fair comparison. In Tra, all models were trained with max overlap (i.e., 10 frames overlapping) but with early stop once the validation loss stabilizes, typically much earlier than the number of epochs presented in Table
Table 4: Summary of the training hyperparameters employed in all test cases.
Parameter
Fturb
Tra
Airone
Data size (frames)
48960
16533
150
31250
Resolution
64x64
128x64
32x32
32x32
Batch size
64
32
25
128
Epochs
300
3000
12000
3000
Learning rate (start, end)
10 10-4, 4, 10
10
10 10-4. 10-5
10 10-4, 10-5
Learning rate schedule
Cosine
None
Cosine
Cosine
Optimizer
AdamW
AdamW
Adam
Adam
Weight decay
10 2
10- 2
10 2
10T
EMA decay
0.999
0.999
0.999
0.999
Farly stop?
No
Yes
Yes
No
The utilized network architectures and the diffusion-relate hyperparameters summarized in Table 5) We use the exact same network architectures as provided in benchmark papers for fair comparison and to demonstrate the applicability of our approaches on diverse settings and architectures. For the baseline method trained in the Fturb case, we use Cin = 4 instead of 6 as the state variable x is ignored and the network is not conditioned on the noise step t. Also, the baseline is trained to directly predict the fluid state 2z+jör instead of the noise eg as in DDPMs. Apart from these modifications, the training of the baseline and DDPMs identical.
EDM training and sampling. Our implementation of EDMs based on the work by Karras et al. (2022). We implement their Algorithm (deterministic sampler) and Algorithm 2 (stochastic sampler) using orrorr Euler and 2nd Heun's 's methods with design choices from the last column of Table I (Karras et al. 2022). Additionally, we consider preconditioning and weighted loss function using the parameters recommended by the authors. For each case, we ran the model using different combination of determmiistic/stoohastie samplers and Euler's s/Heun's 's methods. For the stochastic sampler, the parameters (Scturn, Stmin: Stmary Snoise] were non-comprehensively tuned to attain the best possible results. We found the values 10, 0, co, and for Schurn, Sumin, Stmaar and Spoises respectively, to generally yield satisfactory results across datasets. Noteworthy, EDMs introduce more hyperparameters for tuning compared DDPMs, DDIMs, and our approaches.
Table 5: Network architecture and diffusion-relatee hyperparameters used in all test cases. Cin and Cont refer to the network's 's number of input and output channels, respectively. Bsturt and Bend of the noise schedule 3, are similar to Ho et al. (2020) but are scaled by factor depending on the chosen noise steps T. as defined in Nichol and Dhariwal (222T) and Kohl et al. (2024).
Parameter
Fturb
Tra
A Fone
A1 Fauies
Architecture
Kohl et al.
2024
Kohl et al.
2024
Liu and Thuerey 2024
Liu and Thuerey 2024
Cin
6
Cout
2
3
Saan
10-4 (400/T)
10 4 (500/T)
10-4. 4, (1000/T)
10-4. (1000/T)
Send
0.02 (400/T)
0.02 - (500/T)
0.02 (1000/T)
0.02 (1000/T)
Schedule
Linear
Linear
Linear / Cosine
Linear
17

---

Published as a conference paper at ICLR 2025
D PARALLEL RECURSIVE SAMPLING
Speedup through parallel sampling. One the benefits of conditioning surrogate model on the prediction stride is enable parallel sampling of transient test cases. To date, related work concerned with physics-based simulations of transient nature focus on autoregressive sampling only (refer to Section B for more details). Nonetheless, we posit that our formulation of the problem definition (see Section 2.2), combined with our proposed improvement approaches (i.e., TSMs and IR), would enable parallel sampling with reduced inference time for the same computational budget.
Inspired by the progress in video diffusion models (Ho et al., 2022 Ni et al. 2023, Wu et al. 2023
Harvey et al. 2022), we present two examples of sampling schemes that allow for parallel sampling of diffusion models in transient test cases. As demonstrated in Fig. G for sample problem with R
10 states, while autoregressive sampling (see Fig. (6a) requires 10 successive predictions to sample the entire simulation, it only requires steps for parallel sampling with a batch size = (cf. Fig. 6b). With n = 5, max parallelism can be achieved to sample the entire simulation
steps only speedup factor n, assuming that T 2 For the two parallel sampling schemes, we assume that sampling with j = will lead to lower TA-MSE compared to sampling with j 1 for the entire simulation.
(cf. Fig. (6). This corresponds
5
(a) Autoregressive sampling (n 1). (b) Parallel sampling with 2. (c) Max parallelism with 5.
Figure 6: Parallel sampling of a time-dependent simulation with R = 10, enabled through models conditioned on the prediction stride j. the batch size used for sampling. Each row represents the state of the entire simulation, where the states evolve in time going downwards through the rows. Numbers within blue cells (cells predict) correspond the value of j used to sample from the nearest initial condition (green cell). Gray cells are states already sampled and are not needed for current or subsequent predictions. For (b) and (c), we assume that j = leads to highest prediction accuracy compared to other values.
Higher accuracy when sampling intermediate timesteps. For models that allow sampling using j > we not only enable flexible parallel sampling to reduce inference time as discussed, but also we improve the overall prediction accuracy. Typically when intermediate states are sampled with stride Juee smaller than the primary sampling stride j. we gain (marginally) higher overall accuracy compared to ignoring intermediate states for sampling (using j only). In Table 6 we demonstrate this by sampling the best IR and TSM models for the Tra and Fturb test cases. In both test cases, we see marginal improvement in the mean TA-MSE for small j; however, the improvement is more pronounced for higher j values. This is primarily attributed to the argument discussed by Lienen et al.) (2024) pertaining higher accuracy of predicting with small timestep sizes for short rollout horizon (R = for sampling the intermediate states) before error accumulation predominates, leading to lower accuracy for longer rollout horizons. This greatly motivates future research in flexible sampling of time-dependent physics- based simulations similar to Harvey et al. (202)).
18

---

Published as a conference paper at ICLR 2025
Table Mean TA-MSE results for transient cases for predictions with j 1, and with and without sampling the intermediate states. For the former, the model names include "(all)". TA-MSE is averaged over ext and int regions. Standard deviation values are calculated over all timesteps. both regions, and multiple samples. The results show consistent improvement when sampling the intermediate states with Jses over sampling with the primary prediction stride j only.
Model
8
Tza (10 3)
IR 1T00--M
2.06 + 199
2.73 2.66
3.15
4 2,93
4.18
3.39
3.54 = 2.98
1100-NM (all)
1994 + 182
233
2.75
2.83
3.06 = 2.82
2.72
= 3.39
TSM T100 a
1,46
1.55
1.78 = 1.82
2.17
2.11
2.35 = 2.07
2.32
= 2.12
TSM T100 =1 (all)
1.36 1.47
1.59 1.68
1.93
2.22
1.74 1.74
2.09 = 3.22
Eturb (10
IR 188-NV
2.18
0.88
2.97 = 0.95
2.28 1: 0.81
4.18 1 1.91
2.97 1: 0.55
T80 NV. 74 (all)
1.89
0.77
2.09 0.75
2.58 + 3,40
3,36
+: 4,23
3.47 : 6.47
TSM T8O =0.75
3.63
1.95
5,46
2.57
6.61
+ 2.84
10.14
5,48 + 0.62
TSM T80 =0.75 (all)
3.22
2 1.77
4.07
= 2.17
5.10
6,19 = 4.62
4.79 = 7.15
E FULL RESULTS OF EXPERIMENTS
In this section, we present our full results from all cases in addition the top performing models presented in Section The results are split into quantitative and qualitative sections.
E.1 QUANTITATIVE RESULTS
Tra test case The results for using j = and optimal j > are presented in Table 7 values obtained with j 1, several insights can be observed. First, in comparison ACDM (Kohl jt al. 2024), we are able to provide more accurate results through IR and TSMs with stegle-teep and few-step sampling (NFEs < 5), even though problem formulation (defined in Section 2 is much harder compared to ACDM for the same utilized architecture. Noteworthy, the ACDM results obtained by conditioning over two previous timesteps, whereas all our models are conditioned on the previous timestep only. Second, various TSMs are capable of improving over all considered baselines with and without advanced training mechanisms, with the most accurate model being single-step solver. IR only improves over the base DDPMs and surpasses DDIMs by a significant margin for both similar and optimized sampling schedules. In our experiments, we focused on using random Gaussian noise (00 I) for the initial state Zmm to reduce the computational overhead before starting the refinement process and also because we found early experiments that this setup yields slightly more accurate results than starting from a prediction obtained through truncated sampling of the base DDPM. Third, IR and DDIM were able to provide noise-free results even when the base DDPM produces noisy output (c.g. DDPM T20). Fourth, in comparison to EDMs, we observe that although using a deterministic Euler solver yields the best results for this case, with negligible margin over TSM, it requires 4 NFEs. In contrast, TSM is the only model that enables single-step, high-fideiity inference. Note that we evaluate several solvers for EDM (see Fig. 7 left), but we only report the optimal settings for each solver in Table 7 Finally, we note that PDE-Refiner performs suboptimally with a significant gap compared to other baselines. The model was found to be highly sensitive its key hyperparameters (Kohl et al. 2024), rendering the hyperparameter tuning for this model challenging and ultimately resulting in suboptimal performance.
For TA-MSE
T7e
2.5
Play
Determiristic Euler
2.0-
Deteministicc - Heun
Stochastic- Euler
33
Storhastic Heun
TA-N22
40
100
20
ATC
Figure 7: Evaluating different combinations of EDM samplers for the Tra (left) and Fturb (right)
test cases. Standard deviation regions omitted for clarity.
19

---

Published as a conference paper at ICLR 2025
For j > observe substantial improvements in accuracy, for low-fidelity models, except for EDMs. Suboptimal few-steps DDIMs are able to achieve comparable accuracy to the best baselines, which is also true for IR and TSMs. Noteworthy, the numbers reported for j are limited to sampling with j only without considering the intermediate states. We show in Appendix [ that sampling intermediate states could further enhance the overall accuracy, partieularly for large J-
Table 7: TA-MSE (10-3) values of the Tra test case for 1 (left) and the best results using the optimum j (right). UNeta and UNet, refer to a UNet with training noise and unrolled training, respectively. The reader is referred to Kohl et al. (2024) for a full description of the baselines reported. Lowest TA-MSE values for each group of models are highlighted in bold. Standard deviation values obtained over all timesteps and multiple samples. "Model produces noisy output.
j
optimum j
Model
NFEs
int
int
Kohl et 2024)
ACDM
2.3 +
2.7
2.1
UNeta
1.4 + 0.8
1.8
1.1
UNeta
1.6 0.7
1,5
1.5
ResNetai.
1.7 1.0
1.7
-
1.4
1NO
+
5.5
2.6
TF
3.3 + 1.2
6.2
4.2
PDE-Refine Refiner
4
5.4 2.1
7.1
2.1
DDPM
T20*
20
3.5 3.1
2.6 2.1
1.6 + 1.3
2,1 2.0
T100
100
3.0 + 2.7
4.1 3.7
1.9 +
1.8
3.1 + 2.7
EDM
Buler Deterministic
1.1
1.0
1.3 +
1.1
Stochastic
4
1.7
1.1
1.0
1.7 +
1.7
1.1
1.0
Heun Deterministic
2.7 2.7
2.7
1.6
1.4
2.2
+
1.1
Stochastic
2.0 + 2.2
2.0
18
1.8 +
2.0
1.8
1.6
DDIM
T20
5
4.4 3.3
6.7 1.7=55 5.4
1.9 +
1.7
2.5 +
2.2
10
3.2 + 2.7
4.2 02-+39 3.9
1.4 +
1.3
1.8
1.7
T100
5.2
6.1
5.5
6.7 4.3
3.3
7.8 5.2
6.1
4.1
11144 4.7
6.0 4.4
3.2 1
3.3
2.3
IR (Ours)
T20
5
2.8 2.8
1,7 1.7=10 1.7
4
1.311 1.4
1,5 1.5
10
2.9
1.6 1.5
1.2 12111 1.1
1.6 1.6
T100
5
1.7 1.4
2.1
1.7
2.1 1.8
10
3,8 3.2
3.3
2.3
2.7 2.4
3.1
3.0
1.9
3.0 2.8
4.4 4.0
3.7
1.7
4.1 3.8
T20, m
5
2.3 2.3
2.0
1.3114
1.7 1.8
9
3.1
1.6 1.6
1.3
1.7 1.7
14
1.7118
1.7
1.7
1.5 1.6
19
2.4
1
1.3 1.3
1.4 1.4
T100, 711
5
1.6 1.3
2.0
1.6
2.0 1.7
9
3.0
2.7 2.7224
2.7 2.4
2.7
2.4
14
3.2 3.0
2.9 - 2.8
2.5 + 2.8
2.7
2.5
19
2.6 2.6
3.1
2.9
2.2
2.6
2.3
TSM (Ours)
T15, 8 0.25
18818
1.7 1.5
1.2 +
1.2
1.5
1.6
8 0.5
7
2.1 2.2
1.2 1
111
1.2 -
1.1
1.6
1.4
8 0.75
3
1.4
2.4
1.4
1.6
1.4
8
1.3
1.6 1.5
1.2
1.6
T100.
1
1.2
1.5
1.2 2 1.1
1.5
1.5
20

---

Published as a conference paper at ICLR 2025
Tra
Fturb
1.0
1.0
Bascline
0.9
DDPN
0.9
201M
EDM
EDM
0.8
DON
0.8
DD
(Dun))
50
1.5
25
Timestep r
Timestep
Figure 8: Tra (left) and Fturb (right) absolute velocity |u] correlation to ground truth using j = 1 for the models reported in Table I Values are averaged over ext and i nt regions and the shaded area represents the standard deviation from both regions and multiple samples.
Temporal stability Trauv
Temporal stability
0,025
0.20
CDM
(0ur))
[0k- xx 3lbrri
Xx 20belh
0.15
0.015
0.30
DOM
DDPM
[Oun)
UNita
EDR
TSM (Ours)
POE Refiner
0.005
250
0.05
1000
120
Figure 9: Time evolution for the temporal stability parameter between consecutive timesteps for nop-performing models (see Table T compared to the ground truth simulation (GT). Dashed lines obtained from Kohl et al. (2024). Standard deviation regions are omitted for clarity.
Temporal analysis. In Fig. 8 (left), we see how the |u| correlation to the ground truth evolves over time for the most accurate models, demonstrating consistency with the TA-MSE metric. In addition, we evaluate the temporal stability of these top- perforrming models on the Traiong dataset, which has much longer sequence length (i.e., It = 240 instead of 60), to ensure that the models' predictions remain physically consistent even after the predicted trajectories have significantly deviated from the ground truth. In Fig. (left), we compare the time evolution of the stability parameter, defined Compared to the
the rate of change of each flow field x. by evaluating [x / ground truth results, we see that stability parameter remains bounded within the range (0015, 0.02) for both the ground truth and all models, except for low-fidelity PDE-Refiner model that produces non-physical predictions beyond z 200. While DDIM, DDPM, and IR models slightly exceed the lower bound, they maintain temporal stability.
Fturb test case The results for using j = 1 and j >> 1 are presented in Table 8 Similar to the previous test case with j =1, we observe comparable trends in accuracy for the different models. IR sampling with DDPM T80 base model provides the most accurate results with NFEs as low as steps that surpass the baseline with a significant margin. Best result is achieved by a 19-step IR model with optimized schedule "4. However, with much less NFEs, highly satisfactory results possible, such as using f for the DDPM T80 base model. Single-step TSMs also provide competitive accuracy values which marginally below the accuracy of the baseline. By using optimum values, single-step TSMs are able to surpass the haseline and provide comparable results to IR, which provides further better results than with j 1. The most accurate EDM is achieved through stochastic Euler solver with 10 NFEs, delivering much better results than all 2nd-order Heun solvers with a fraction of NFEs. The model has an optimum j = 1; hence, DDPMs, DDIMs, IR, and TSMs at lower NFEs when the optimum > 1. Although the most accurate EDM solver on par with the single-step TSM in terms of accuracy, is 10x slower and doesn't provide better results when j 1.
outperformed by most
Temporal analysis. The time evolution of p([u)) is shown in Fig. (right), while the stability parameter for long rollouts evaluated on Fturbisag with R 120 instead of 30 is presented in Fig. 9 (right). Both figures show consistent results to the TA-MSE values and demonstrate the accuracy of the different models over time. Noteworthy, the stability parameter (Fig. g right) of the
21

---

Published as a conference paper at ICLR 2025
ground truth and all models remains constrained within [0.1, 0.15], with the exception of minor deviation observed for DDIM at 70, which doesn't lead to non-physical results for the remaining timesteps.
Table 8: TA-MSE (10-2) values of the Fturb test case for j (left) and the best results using the optimum j (right). Lowest TA-MSE values for each group of models are highlighted in bold. Standard deviation values are obtained over all timesteps and multiple samples. "Model produces noisy output.
optimum j
Model
NFEs
ext
int
ext
int
Baseline
3.95 +: 3.84
4.82 1 5.23
3
3.22 1: 2.88
3,73 4: 3,55
DDPM
T20'
20
20.66
16.71
24.01
+ 20.58
10
7.80
= 5.22
4,51 + 3.66
TS0
50
10.60
11.47
9.85
8.76
7.31
7.28
5.54 + 5.07
T80
6.54
5.27
5.55
4.68
4.52
2.71
2.22
EDM
Fuler Deterministic
4
5.65
5.58
5.67 5.77
5.65
5.58
5.67
+ 5.77
Stochastic
10
4.75 :: 4,53
4.78
:: 4.26
4,75
4: 4.53
4,78
+ 4.26
Heun- Deterministic
40
9.22 = 8,93
12.06
+ 11.43
5
9.23 1 9.10
7.48
+ 6.24
Stochastic
9.60 10.32
9.35
8.05
8.27
8.16 - 7.73
DDIM
TS0
5
14.21 12.09
26.65 +: 20.68
7.64 6.62
13.31 12.80
10
8.49 = 8.74
14.79 = 14.07
4
6,53 = 5.99
8.26 = 7.74
25
7.27 = 7.93
11.84 = 11.47
5.99 = 6.17
7.22 = 7.46
T80
5
9.77 + 8.56
19.91 + 16.50
6
4.78 + 4.68
5.92 + 5.37
10
5.61 5.79
10.47 + 9.90
4.19 4.06
3,99 - 3.67
4.89 4.31 4.62
7.72
4.10
3,59
3.27
40
6.97 6.84
3.99 4.13
3.53
3.20
IR (Ours)
T50
5
4,36 = 4.65
4.05 + 3.86
3.44 : 3.15
2.89 + 2,80
10
4.44 :: 4.99
3.24 + 3.06
3.85 1: 3.79
2.64 :: 2.59
4.24 = 4.69
3.52 : 3.41
4.27
2.56
2.34
T80
5
3.21 3.37
2.77 2.67
6
2,89
2.41
1,86 + 1.58
10
2.93 3.34
1.70
1.63
3.36
3.68
1.19
1.18
20
2.61 = 2.73
2.27
= 2.43
6
2,88
= 2.51
1.54 = 1.22
40
3.19 = 3.41
2.58
2.59
2
3.63
= 3.71
1,36 = 1.17
T50,
5
4,36 4.65
4,05 =:: 3.86
3.44 1: 3.15
2.89 + 2,80
72
9
= 6.09
2.92 = 2.66
4.16 + 4.15
2,46 = 2.36
73
5.29 5.84
4,07
4,08
3,32 3.25
19
3.59
3.89
4.08 4.22
3.50
3.51
3.62 + 3.77
T80,
5
3.21
= 3.37
2.77 + 2.67
6
2.89
2.41
1.86 = 1.58
9
3.21
+ 3.59
2.24 + 2.13
3.42
+ 3.07
1.75 + 1.52
14
2.93
3.31
2.27 1: 2.08
6
2.93 4: 2,50
1.87 + 1.66
74
2.82
1,95
2,90
1.45
TSM (Ours)
T20,
0.5
10
6.55
7.30
6.64 6.54
2
5,81
6.30
4.57 1: 4.42
0.75 =
3
7.54 = 7.88
8.54 = 7.38
5.85
5.20
6.09 + 5.13
9.09 = 8.92
10.96 = 9.01
14.19 = 9.87
1.36 = 1.05
T50, & 0.5
25
6.74 7.61
5.48 : 5.39
3
4.43 1 4.45
5.06 ::: 5.29
0.75
100:4000 4.90
5.94 =
3.90 - 3.67
4.73 = 4.56
1
6.25
5.94
3.53
3.25
2.40 2.37
T80, 0.75
5.46
5.80
3.68 3.33
3.89
3.61
3.58 :: 3.63
20
3.84
3,85
4.35 3.96
2
3,43
3.03
3.83 # 3,59
1
5.29
+ 5.33
6.32 6.69
5
4.44
+ 4.13
1,93 + 1.86
T100, =
1
5.00 = 5.52
4,39 - 4.76
5
3.91
+ 3.59
1.20 = 1.04
22

---

Published as a conference paper at ICLR 2025
1.2
Afto
+20
2.0
Alows
2.35
2.30
Sta 1oe
1.06
1.00
1.0
40
60
03
200
25
30
75
100
325
150
175
200
23
50
102
325
150
375
202
Figure 10: Evaluating different combinations of EDM samplers for the Air Aone (left) and A Aimiti (middle and right) test cases. Standard deviation regions are omitted for clarity.
Air test case The results for the Airane and Ai Axilit test cases are presented in Table 2 For the Ai Fens test case, as discussed in Section 5 we were able to provide better results than the benchmark by Liu and Thuerey (2024) with the same number of NFEs by using linear 3e, alongside dataset normalization. Similar to the two previous test cases, DDIMs reduce NFEs by compromising accuracy, while our IR approach leads to the most accurate results for this study. We found that using Tinit (O I) similar to the two previous test cases lead to suboptimal results that are not report here. However, relying on the accurate DDPM base models, we provide Tmit through truncated sampling of the base model. This truncation of the base DDPM yields adequately accurate results but dominated by noise in the output. In this case, IR simply refines the input (which is conceptually the essence of the approach) remove the noise in the output while maintaining slightly improving the accuracy. Here, only a single step of refinement was sufficient to provide the highest accuracy results. Regarding TSMs, we observe competitive accuracy for both linear and cosine 3, with moderate NFEs (< 20), which already reduces the computational cost compared to the benchmark and the base DDPM by a significant margin. While for linear 83 single step inference possible learn the entire data distribution, the output from these models were found to be slightly noisy. For the cosine B,, single-step TSM fails to learn the data distribution (as demonstrated by the extremely high MSE,) and only learns the mean (as shown by the low MSE,). Hence, this leads to the conclusion that with the proper choice of Bg and the associated diffusion-relatee parameters, single-step inference through TSMs is made possible even in test cases with high uncertainty. Additionally, we evaluate several EDM samplers for this test case and conclude that there is little variability between the pop-performing solvers in terms of accuracy. The most accurate EDM is deterministic order Heun solver that fails to deliver competitive results. Even at higher NFES, up to 200, we observe only minor improvement in accuracy (see Fig. [O] left). This outcome is likely attributable suboptimal hyperparameter configurations for both training and sampling.
Based on these insights. we restrict our experiments for the Airmuiti case few models only, especially that training and sampling are considerably expensive for this study. We, therefore, only train a DDPM with since it was unsuccessful in the previous study. We see equivalent results in this study: IR sampling achieves the best results by refining a noisy estimation from the base DDPM, followed by TSM with NFEs 25 and 10. DDIMs are the least accurate even with high NFEs. The results of evaluating multiple EDM solvers are presented in Fig. (middle and right). For (MSE,,, )a. all samplers, even those with NFEs < 25, outperform all other models. However, for (MSE, )a. achieving satisfactory results compared to the top DDPM, TSM, and IR models requires NFEs 2 40. Nonetheless, the most accurate sampler exhibits lower accuracy than these models.
100 and TSMs with different s values, excluding single-step inference
Note that heteroscedastic models only predict the data uncertainty by assuming a distribution for the data and estimating its parameters. Consequently, samples drawn from this learned distribution often non-physical (e.g., see Fig. 9 Liu and Thuerey[(224) distribution of solutions, which is then used to estimate the uncertainty. Therefore, heteroscedastic models cannot be directly compared to DDPMs in terms of speed. Nonetheless, they are an established method for predicting the data uncertainty in this area and represent a strong baseline for comparison.
In contrast, DMs reconstruct the target
23

---

Published as a conference paper at ICLR 2025
Table Airone (left) and Air MMilti (right) test case results. Best results in each group of models are highlighted in bold. Standard deviation values are obtained from multiple samples. 'Model uses cosine Sg. # Model only predicts My and dy- **Model produces slightly noisy output. ** MMdel produces too noisy output.
Airore (10-1)
Aisnatei (10-3)
Model
NFEs
(MSEp,
(MSE,,).
Model
NFES
(MSE,,,)e
(MSE,,))
Liu and Thuerey
2024
Liu and Thuerey
2024
DDPM T200C"
200
3.24
1: 0.55
7.66
0.74
DDPM T200C
200
1.76 1: 0.23
0.89 1 0.12
leteroscedastic*
3.35
1 0.60
4.47
0.37
leeeeossedasticc*
1.50
0.12
1.55 1. 0.03
DDPM
DDPM
T100
100
3.73
1 0.34
7.73
0.24
T100
100
2.34
1.11
0.57 1: 0.03
T200
2.88
1 0.26
0.22
EDM
T200C
200
23.0
0.39
6.28
0.35
Buler Determ.
40
1.25 + 0.41
0.89 + 0.22
EDM
Stoch.
40
1.13
0.39
0.67 = 0.11
Fuler Determ.
20
8.28 + 1.85
10.8 1.51
Heun Determ.
80
1.28
0.42
0.90 0.23
Stoch.
9.66 + 0.95
11.3 + 1.94
Stoch.
80
1.10
0.38
0.71 + 0.13
Houn Detern.
20
8.13 = 1.28
10.1 + 1.67
DDIM T100
Stoch.
20
10.4 + 0.75
11.8 + 0.27
2.33
1.06
0.83 1 0.16
DDIM
25
2.16
1.01
0.67 = 0.07
T100
10
8.42 + 0.27
10.80 + 0.51
50
2.12
0.99
0.64 = 0.05
25
4.46 +: 0.31
8.09 0.23
IR (Ours)
50
3.81
+ 0.27
0.22 7.71
T100, =06, 75
41
2.07 1: 0.99
0.59 : 0.04
T200
10
9.12 1 0.33
10.40
0.54
TSM (Ours)
25
4.67
+: 0.40
7.65
0.29
T100, 8 0.5
50
2.58
+ 1.32
0.79 + 0.16
50
3.98
0.44
7.33
0.25
8 0.75
25
1.66
0.53
0.64 1 0.04
100
3.68
0.44
7.24
1 0.25
= 0.9
1.91
0.71
0.70 + 0.06
TR (Ours)
T100, 0.6, 73
41
2.87
1. 0.32
6.76
0.20
T200, 0.6,
81
2.07
1 0.21
6.07
0.26
TSM (Ours)
T100, =0.5
50
3,26
+ 0.42
7.71 + 0.20
=0775
25
3.25 + 0.38
6.98
+ 0.18
=09
10
3.30
+ 0.39
5.89
0.33
=0.55
4.98
0.60
+ 0.42
=
4.84
5.96 0.31
x =055
100
3.59
0.31
0.24
=0.75
50
3,02
0.39
6.99 0.34
=0.9
20
3,92 + 0.40
7.08 + 0.42
=095
10
4.24 +: 0.23
9.08 + 0.15
==
4.58 + 0.22
8.62 + 0.40
T200C*, =0.25
150
5.32 + 0.58
8.73 + 0.28 0.28
=05
100
3,44
+ 0.30
8.03 0.38
=075
50
3.22
+ 0.16
+ 0.18
=0.9
20
4.21
0.30
0.18
..
3.69
0.03
90.6 0.04
E.2 QUALITATIVE RESULTS
We provide sample results for all test cases in this section. Results for all flow fields of the Fturb study are presented in Fig. [T] while results for the Tra test case are presented Fig. [2] Finally, sample results for Air Kone and Ai Ewutts test cases are presented in Fig. 13 and Fig. 14 respectively.
24

---

Published as a conference paper at ICLR 2025
GT
Baseline
DDPM
EDM
1.5
1.0
DDIM
0.5
IR (Ours)
TSM (Ours)
10
15
r= 19
24
f= 29
Figure 11: Sample predictions of |u| from the Fturbext test case with Re 5000. Results are based on the top performing models presented in Table (right).
25

---

Published as a conference paper at ICLR 2025
08
DDPM
00
EDW
DDIW
0.4
00
TSW
(a) [x|
DOPN
EDW
05
DOW
IR (Ours)
022
F= 30
44
F= 59
0.0
(b)
EDM
DOWN
(Ours)
r= 15
7=
(c) p
Figure 12: Sample predictions of the different flow fields from the Tra.xt test case with Ma 0.52. Results are based on the top performing models presented in Table (left..
26

---

Published as a conference paper at ICLR 2025
DDPM
EDM
DDIM
IR (Ours)
TSM (Ours)
Pa;
0
0
Figure 13: Sample predictions for the expectation μ and standard deviation o of all flow fields for the A Fone case with 10% Results are based on the top performing models in Table (left).
-022 0.00
13550300-02500
02
22
DDPM
DDIM
IR (Ours)
TSM (Ours)
Figure 14: Sample predictions for the expectation μ and standard deviation of all flow fields from the Airiai test case for the fx75141 airfoil with -18.94 angle of attack and Re x Results are based on the top performing models presented in Table (right).
27

---

Published as a conference paper at ICLR 2025
F DATASET OVERLAPPING FOR TRANSIENT CASES
In time-dependent problems, the proposed formulation requires training the DDPM to predict T + different steps with prediction stride E [0, Ti. For benchmark datasets (e.g., Tra Kohl al. (2024) that are sufficient to train for predicting next timesteps only, our formulation leads to overfitting and sub-optimal results due to the limited data, especially when the dataset iterated as demonstrated in Figure 15a This naive iteration over the dataset frames means that require dataset that T bigger to train without overfitting. The casiest approach to increase the dataset without the need for generating more training data is implement overlapping, depicted in Figure 15b Although the demonstrated overlapping leads to more iterations per epoch depending on the degree overlap, we found in our tests that training could be terminated much earlier, leading to almost constant total number of training iterations.
In the Fturb case, enough data was present to require signifieant overlapping: however, apply max overlap (i.e., overlapping 10 frames, since T = 10) for the Tra test case to circumvent overfitting issues and provide accurate results. As discussed, we usually terminated the training procedure much earlier than the number of epochs reported in Table as soon as the validation loss reaches plateau.
17
(a) Dataset without overlap.
18
19
25
26
27 28
34
35
(b) Dataset with max overlapping frames.
Figure 15: Illustration of dataset overlapping circumvent limited data for training. Numbers inside cells indicate the index of the frame in the dataset, whereas the colored cells indicate one "simulation" based the horizon T. Each column represents single frame but with different assigned indices for dataset iteration. The figure shows for a horizon of 8 steps (i.e., j E [0, 7] with T = 8) how max overlapping (max overlap T 8 frames) can lead increase in the considered dataset from 18 to 90 frames each with unique label within "simulation" (i.c., color). Frames with bold border are provided more than one label.
G HYPERPARAMETER TUNING
Our methods rely on hyperparameters that play a crucial role in determining their performance terms of accuracy and speed. Unlike DDIMs, which provide trade-off between speed and accuracy, our methods are tuned using heuristics-based approach to efficiently identify near-optimal hyperparameter configurations with minimal evaluations. This approach depends on the specific case under consideration, whether it is spatio-temporal deterministic dataset or stochastic one with complex distribution. The hyperparameters can be determined conveniently based on the following considerations, and are kept constant for each case during inference runs thereafter.
Deterministic test cases
TSM:
The search for the optimum value typically begins within the range [0.5, 1), espe- cially a large number of diffusion steps T chosen, to attain higher speedup. Figure provides several insights for the optimal combination of and T for both transient and steady-state cases. First, we evaluate both extremes within the specified range and then employ standard line search approach arrive at optimal value of requiring most
28

---

Published as a conference paper at ICLR 2025
or three additional evaluations. Further, we also give priority for models with low T to minimize the NFEs required for inference.
IR: we usually start with Vinit (o 1) and run our greedy optimization algorithm (Algorithm 3) with low N to obtain an efficient 7 with as low NFEs as possible. As often yields highly accurate results for our transient cases. Then, N can be gradually increased to explore longer schedules that has the potential to enhance the overall accuracy with little increase in NFEs.
demonstrated in Figure 4b)
Stochastic test cases
TSM: We follow the same procedure as before; however, we refrain from extreme values as they often lead to low- sow-fidelity, noisy outputs (see Table Therefore, the search region for is restricted to [0.5, 0.9], with the lower bound further reduced for smaller T values.
IR: From early tests, it is found that Timit is optimal when obtained through truncated sampling (see Algorithm| T of a pre-trained DDPM with 0.5. While this approach typically produces highly accurate initial outputs, the results often exhibit noticeable noise. Consequently, a refinement schedule y with N < steps utilized to climinate the noise, while simultancously improving preserving the accuracy of the noisy initial input.
Empirical results across all datasets, as presented in Tables 8 and demonstrate that our methods require relatively few evaluations to arrive at hyperparameter configurations with highly competitive results. Compared to competitive EDMs, the reduced number of associated hyperparameters in addition to our heuristic approach provide our methods with a distinct competitive advantage, alongside the other advantages outlined previously.
29


Reviews:
[{"review_id": "GrScursuRA", "reviewer": "Reviewer_TSeh", "summary": "The paper introduces a truncation approach and an iterative refinement process in the sampling procedure of the Denoising Diffusion Probabilistic Model (DDPM) that enable to reduce the number of function evaluations without decreasing the accuracy. The first method proposes to stop the sampling process at an earlier time point and to estimate the denoised sample using Tweedie's formula. The second method uses the forward diffusion for a given shorter noise schedule and the denoised sample is approximated using Tweedie's formula. The authors show the efficiency of the approach on the simulations of airflow field.", "questions": "- Does s=1 mean that the model is trained as a single variational autoencoder? Or is this truncation only used for sampling? (paragraph about the training is not clear to me)\n\n- naive question for my understanding: At the beginning of Section 5.1, the authors say that they consider time series of flow field from j= 1 to T. It is not clear how the time series are handled here. Could the authors clarify this point?\n\n- How do the authors handle the high fluctuations areas of the domain? It seems that some region of the domain have a highly turbulence flow field (low pressure vortex) and this would require a more flexible model in this specific area. \n\n- Did the authors try to change the value of the initial input $x_{init}$? \n\nI look forward to reading the answers of the authors and I can change my score depending on their answers.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The paper reads very well and provides some good contributions to the development of diffusion models. \n\n   - Although the presented approach is based on some known results (Tweetie's formula, ancestral sampling), the proposed solution seems very efficient in practice and leads to lower computational costs. Both idea's exploit the denoising Tweedie's formula in forward and backward sampling.\n\n   - Including the truncation in the training is smart trick that eases the training. \n\n   - The numerical experiments show that the presented approaches achieve better performances than traditional DDPM sampling and reduce the computational costs.\n\n- The approach seem to be very efficient in physic based simulation of flow field. I think this is a very good contribution in this specific domain.", "weaknesses": "Although the approach seems efficient in the presented numerical experiments, there is a number of points that need to be clarified. \n\n- Tweedie's formula is well known. It seems that this has been already used in some previous works such of [Delbracio et Milanfar, 2024]. In  their work, the authors provide some intermediate reconstructions through this formula. They show that by adding some stochastic steps \nthey can get better performances than state-of-the-art. I know the model is not the same but it would have been interesting to highlight the links with this work because they seem very closely related. May the authors comment on that point?\n\n- The results show that TS outperforms traditional DDPM by using s=1. This means that on this specific problem, there is no need to sample intermediate diffusion steps. I wonder whether this aspect is problem specific or this happens for a wider range of problems. Is this result expected? \n\n- This would have been interesting to see how this method perform against traditional DDPM sampling on image reconstruction. Indeed, DDPMs usually perform well on such problems.\n\n- The training data are deterministic sequences of flow field data. It would have been interesting to observe how the model performs on noisy dataset. \n\n- Overall, I like the idea of truncation and iterative refinement, but since there is no major theoretical contribution in this work, I would have liked to see more numerical results. The claim \"Truncation is all you need\" would have been justified if the authors had included numerical results on different applications. So far, the paper makes very good contribution in this specific domain, and proposes an interesting approach to reduce computational costs.\n\nDelbracio et Milanfar, 2024, Inversion by Direct Iteration: An Alternative to Denoising Diffusion for Image Restoration. TMLR.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "olgtZ03TQc", "reviewer": "Reviewer_PMEd", "summary": "This article first analyzes DDPM and finds that after an appropriate truncation (stop diffusion), the model has high fidelity and high-efficiency sampling performance. On this basis, an iterative refinement method is introduced to further improve accuracy and long-term stability.", "questions": "The experimental results are very good, and I hope to add more theoretical analysis.  I will be happy to improve my score in subsequent discussions.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "Advantages:\n\n1. As we all know, if we consider an infinite boundary heat equation (diffusion process), or consider long-term diffusion, then the usefulness of a long part of the noise addition/noise reduction behavior is not that great. The article fully considers and utilizes the diffusion behavior in a finite time (truncation), thus achieving a balance between accuracy and running time, which is a very good point.\n\n2. Truncated sampling reduces the uncertainty of the sample to a certain extent, or increases the accuracy of the sample response distribution.\n\n3. Based on the content of the appendix(in particular, part D), the experimental effect is very significant. In other words, iterative refinement even makes up for the truncated sampling to a certain extent (the useful information and samples that are truncated).", "weaknesses": "Disadvantages:\n\n1. I want to know whether there is a mathematical inference for the truncation sampling standard, or whether it is completely based on experience, that is, truncation and retention (importance) interpretability.\n\n2. The purpose of refinement iteration and truncation sampling is to improve efficiency while ensuring a certain degree of accuracy. I think this requires a game. How to achieve such a balance? Is there a more rigorous mathematical explanation?\n\n3. I think experiments can increase the breadth. One is to compare with a more general SDE instead of just with DDPM (and Kohl's 2024 work). In addition, for experiments, do you consider more general PDE solutions?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "pr34H4eepY", "reviewer": "Reviewer_8zxM", "summary": "This paper studies the application of diffusion models to physics simulations. Over the past years, neural networks have emerged as surrogate modeling approach for physics simulations, with a key use-case being computationally efficient inference. However, for this purpose, diffusion models have the drawback of requiring many function evaluations due to their iterative ancestral sampling procedure. To this end, the authors propose two contributions: (1) truncation of the last steps of the reverse diffusion process, and (2) iterative refinement, which considers a much shorter noise schedule at inference time. Both methods reduce the number of function evaluations and thereby increase sampling speeds. Moreover, the empirical results demonstrate that accuracy is generally maintained and sometimes even improved compared to standard expensive sampling procedures.", "questions": "**Q1:** Given that PDE-refiner is conceptually relatively similar to iterative refinement, I am surprised to see a quite large performance difference between the two methods in Appendix D.1. Can the authors explain the reasons behind this large performance gap?\n\n**Q2:** The truncation of the last steps of the reverse diffusion process seems to be equivalent to a modification of the noising schedule: we can choose the noise schedule $\\beta_t$ such that the first step in the forward diffusion process has already a quite low signal-to-noise ratio (in line with the level corresponding to the last step before truncation in the reverse process), and afterwards noise is added gradually as per usual, while reducing the total amount of steps in the forward process in line with the skip percentage. Can the authors provide their thoughts on this perspective and whether or not they agree? If they agree, can they comment on why the noise schedule that is equivalent to the truncated process is a good choice for this problem setting relative to other problem settings, and in this way place their contribution in the broader context of noise schedules?\n\n**Q3:** Please comment on W1-W3.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "**S1:** The experimental setup is rigorous, comparing methods in both pointwise metrics and relevant physics-based metrics to provide complementary perspectives on their performance for three relevant datasets. Moreover, many additional results and baselines can be found in the appendix, making an extensive empirical evaluation overall.\n\n**S2:** Two methodological contributions are evaluated: truncation of the last steps of the reverse diffusion process, and iterative refinement, which optimizes the inference sampling schedule such that less denoising steps are required. Both contributions are aimed at reducing the number of function evaluations to improve computational complexity of diffusion models for physics simulations. This is a relevant research direction, since reducing computational complexity of computational procedures is one of the primary use-cases of neural simulation models, for which diffusion is emerging as a promising modeling approach. \n\n**S3:** The paper is well-written, and the explanations of the proposed algorithms are intuitive and easy to follow and understand. The clear structure of the text helps the reader to efficiently navigate the paper.", "weaknesses": "**W1:** While reading the text, I found it difficult to distill what the key differences between iterative refinement and PDE refiner are (Lippe et al., 2023). Does it have to do with the greedy optimization method of the refinement schedule (to my knowledge PDE refiner uses a fixed schedule), or details in the formulation of the diffusion process (a nonzero vs zero drift term in IR and PDE refiner respectively), or something else? Since both IR and PDE refiner are quite similar, it would be good if the ‘method novelty’ paragraph explicitly contrasts the two approaches and highlights their differences. Additionally, if the greedy optimization of $\\gamma$ is a core novelty relative to PDE refiner, then it would be beneficial to explain this more elaborately in the main text rather than the appendix, since it would be a key aspect of one of the contributions in this case.\n\n**W2:** One of the goals of the paper is to show that the proposed methods close the gap between the diffusion models and deterministic baselines. However, most of the results in the main text (both tables and plots) focus only on diffusion models. It would be relatively straightforward to also show the results of one or two deterministic methods that are considered by the authors in part of the plots and tables, for example in Figure 2 and Table 2. This would help the reader to get a better understanding of the tradeoffs of existing diffusion-based approaches, deterministic approaches, and the proposed methods without taking additional space in the paper.\n\n**W3:** The conditioning on the autoregressive step size (j in Sec. 2 of the paper) is already introduced in Gupta et al (2022), and as such cannot be claimed as a contribution of the paper (currently point 1 of the contributions listed in the introduction). Since this is not a core point in the rest of the paper, it seems that this can straightforwardly be removed from the list of contributions in the introduction without affecting the rest of the work and the core contributions significantly.\n\n**References:**\n\nGupta, J. K., & Brandstetter, J. (2022). Towards multi-spatiotemporal-scale generalized pde modeling. arXiv preprint arXiv:2209.15616.\n\nLippe, P., Veeling, B., Perdikaris, P., Turner, R., & Brandstetter, J. (2023). Pde-refiner: Achieving accurate long rollouts with neural pde solvers. Advances in Neural Information Processing Systems, 36.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "373Tm3RqJy", "reviewer": "Reviewer_fSso", "summary": "This paper introduces Truncated Sampling Models (TSMs) and Iterative Refinement (IR) to improve the efficiency and fidelity of Denoising Diffusion Probabilistic Models (DDPMs) for fluid simulations by enabling reduced steps sampling through truncation of the diffusion process. These methods significantly reduce inference time and improve stability over long rollout horizons for turbulent flow and airfoil simulations.", "questions": "- Could you theoretically or intuitively clarify the effectiveness of the pre-trained diffusion model in the IR sampling procedure at noise level $t = \\gamma $? The distribution $x_{init}$ at noise level $t = \\gamma $ appears to differ from the distribution $x_0$ at the same level $t = \\gamma $. Additionally, in Equation 6, is it ensured that the error between the final output and the $x_0$ remains sufficiently small, such that $E[‖x _0^N - x_0 ‖_2 ]<\\epsilon $?\n\n- Could you also clarify (experimental support would be helpful) line 288 (2), which states that IR sampling does not require data consistency to $\\hat{x}_0$? I mean why the proposed method does not require the data-consistency? Enforcing data consistency to $\\hat{x}_0$ could be plugged in after line 7 in Algorithm 2 to improve accuracy in a single iteration, without compromising the number of iterations needed. Also, related to the above question, data consistency could help reduce the error towards zero, such as [1] and [2]. \n\n[1] A physics-informed diffusion model for high-fidelity flow field reconstruction, 2023.\n\n[2] Diffusionpde: Generative pde-solving under partial observation, 2024.", "limitations": "", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- Interesting topics – Generative diffusion models as surrogate model for fluid simulations.\n- Clear motivation – Reduce computation costs in the generative process using a novel reverse sampling approach.", "weaknesses": "- The paper lacks clarification of superiority of the proposed methods over other surrogate models, such as neural operators which are currently the most widely used ML-based surrogate models in fluid dynamics for speed and accuracy. Given that a primary contribution of this paper is reducing time costs, a more thorough comparison with advanced neural operators – either highlighting the proposed method’s improved accuracy at similar time costs or its time efficiency at comparable accuracy – would strengthen the argument. However, there are few comparisons with neural operators in the experiments in main text; although Unet is included, more advanced neural operators should be included. Additionally, the proposed method does not appear to clearly outperform Unet.\n- The title may mislead some readers, as “physics-based simulations” implies a broad range of applications, while the paper is mostly on fluid dynamics. To improve clarity, I recommend replacing “physics-based simulations” to “Fluid Dynamics Simulations”. Alternatively, the authors could clarify whether they intend to generalize their approach to other physics-based simulations beyond fluid dynamics or provide examples of how the method could be applicable to other physics domains.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "LBCrEWWXLs", "reviewer": "Reviewer_6cX6", "summary": "The paper addresses the high computational requirements of existing diffusion-based techniques for modelling dynamical systems. They propose two sampling techniques that lead to good sample quality with only a few NFEs. The first one requires modifications to the training process and is performed by truncating the diffusion process close to the clean data. The second one is compatible with pre-trained DDPMs and proposes an iterative refinement based on Tweedie’s formula. The paper provides extensive experimental evidence on three datasets: incompressible and compressible turbulent flow (2D) and airfoil flow simulation (3D).", "questions": "1. **Regarding W1** - I’d be curious to see how the methods perform on dynamical systems with more complicated nonlinear dynamics, for example the Kuramoto-Sivashinsky equation used in PDE-Refiner (Lippe et al. [3]). That is a fourth-order non-linear equation where correctly capturing the high frequencies seems to be more important than in other settings, so it would be interesting to see how you compare there with benchmarks such as PDE-Refiner [3] (which can also be interpreted as a refinement of an MSE-trained one-step prediction). \n2. **Regarding W2** - Do you have any intuition why these methods outperform DDPM and whether the DDPM baseline could be improved?\n3. **Regarding W4** - Could you provide some experiments that test the long rollout performance of these sampling methods vs. traditional ones? Including frequency spectra of generated states would also help.\n4. **Regarding Minor 2** - It seems that in general the optimum $j$ for these methods is between [2, 4], but have you noticed any significant patterns? Were there differences between interpolation and extrapolation tasks?\n5. **Eq (5).** I am not sure I understand this equation. I will omit the $\\theta$ in the $p_{\\theta}$ subscript because the equations do not render correctly. If $p^T(x_T, \\mathbf{x}_0, j) = \\mathbf{x}(j \\cdot \\delta t)$, then wouldn’t $\\mathbf{x}(2j \\cdot \\delta t) = p^T(x_T, \\mathbf{x}(j \\cdot \\delta t), j) = p^T(x_T, p^T(x_T, \\mathbf{x}_0, j), j)$ (i.e., we still start from white noise $x_T$, but we now condition on the output of the previous step)? In my mind $\\mathbf{x}(\\tau_f) = p^T(x_T, p^T(...p^T(x_T, \\mathbf{x}_0, j)...), j)$, but maybe I didn’t interpret the equation correctly.\n6. When comparing the results in Table 5 vs. Tables 6 and 7 - Shouldn’t the metrics corresponding to, for example, Fturb TSM T80 s =0.75 ($j=2$) from Table 5 ($3.63 \\pm 1.95$) be the same as in TSM T80, s=0.75 with optimal $j=2$ ($3.43 \\pm 3.03$) Table 7? What’s the difference between these settings?\n\n**----Update after rebuttal-----**\n\nThe majority of my concerns have been addressed and I believe the paper offers a useful empirical investigation into how to speed up sampling with diffusion models for physics-based simulations. As such, I increased my score to 6.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. **Relevant topic.** Lately, there has been increasing interest in modelling dynamical systems with diffusion models due to their probabilistic nature. However, many works (Kohl et al. [1], Shysheya et al. [2], Lippe et al. [3]) acknowledge that diffusion models tend to be computationally costly, and techniques that reduce this cost would be greatly beneficial. This is exactly the problem this paper aims to address.\n2. **Clear distinction from other works.** The paper clearly delineates its contributions from the already existing work, and how the proposed sampling techniques differ from other approaches.\n3. **Good experimental evidence.** The paper provides good empirical evidence, with experiments on three diverse datasets, and using a wide range of metrics.\n4. **Well-structured, clear writing.** Overall, I found the structure and writing clear.\n\n[1] Kohl, G., Chen, L., & Thuerey, N. (2023). Benchmarking Autoregressive Conditional Diffusion Models for Turbulent Flow Simulation.\n\n[2] Shysheya, A., Diaconu, C., Bergamin, F., Perdikaris, P., Hern'andez-Lobato, J.M., Turner, R.E., & Mathieu, E. (2024). On conditional diffusion models for PDE simulations.\n\n[3] Lippe, P., Veeling, B.S., Perdikaris, P., Turner, R.E., & Brandstetter, J. (2023). PDE-Refiner: Achieving Accurate Long Rollouts with Neural PDE Solvers. ArXiv, abs/2308.05732.", "weaknesses": "1. **Unclear how the method generalises to other settings.** I acknowledge that the aim of the paper is to study the efficacy of the proposed sampling methods in the context of dynamical systems. As the authors mention, the fact that they lead to good results is probably due to:\n- The characteristics of the data distribution—states with fairly coarse resolutions, where the high frequency information has been lost during downsampling.\n- The task considered—predicting next step (forecasting) distribution, which is predominantly unimodal.\n  \n  I’d be curious to know whether the sampling methods remain applicable \n  - for other tasks that might require modelling multimodal distributions: e.g. some sort of inverse problem (reconstruct field based on sparse, partial observations, compatible with multiple solutions), which is also a task of interest in dynamical system modelling, \n  - or for other datasets: potentially still PDEs but not as coarsened and with more complicated nonlinear behaviour, or images. \n\n  One could argue that Air_multi requires sampling from a more complicated distribution, and there the improvements are not as pronounced as in other settings. Perhaps you could include a brief discussion on this, or a preliminary experiment on a more complex PDE dataset, such as Kuramoto-Sivashinsky (see **Q1**)?\n2. **Counter-intuitive comparison to DDPM.** Maybe this is not necessarily a weakness, but I find it counter-intuitive that these sampling methods outperform DDPM, given that DDPM (or at least the continuous-time frame formulation) has stronger theoretical foundations (Similarly to how DDIM is posed as a framework where you can trade off computation for sample quality, not gain in both). I would have expected the same accuracy with significantly fewer NFEs to be possible, but not both. Is it possible that the DDPM model could be further tuned to achieve similar performance and maybe the DDPM baselines are not optimal? Maybe this is where a comparison with EDM would have been beneficial, as it provides a more principled approach of setting up diffusion models.\n3. **Lack of theoretical guarantees.** While the methods seem effective in practice, the paper does not provide any theoretical guarantees. I realise this might be hard to derive, but this makes the applicability of the methods more ad-hoc. This also reflects in the empirical investigation, where there is no clear recipe for what works best and how one should choose the hyperparameters optimally.\nAlso when making statements such as “However, we assert that\nour methods will consistently improve over ancestral sampling, as demonstrated in our experiments.” - This was only shown in three experiments, and the paper does not contain guarantees that this would always hold, so I would avoid over-generalising.\n4. **Lack of analysis of the stability of longer rollouts.** The paper provides several metrics to analyse the performance of the sampling methods, but none provides intuition/results about how the metrics evolve in time for the transient datasets (Tra and Fturb) (e.g. per-time-step MSE or per-time-step correlation). In particular, I’d be interested to compare the performance of the benchmarks to your methods on rollouts longer than what the model has been trained on. (This could, for instance, be tested on Tra_long from Kohl et al. [1]).\n  \n    I think that one potential weakness of these sampling schemes is that they lose more of the high frequency information than the traditional sampling schemes. Maybe this doesn’t affect the short rollouts significantly but it might negatively impact the accuracy of longer term rollouts (see Lippe et al. [3]). It would also be good to check that if extended significantly beyond the training range, the proposed sampling methods still generate physically plausible states, and outperform the baselines.\n5. **Lack of a comparison to EDM.** While I agree that a comparison to, for example, distillation techniques is outside the scope of the paper, I think EDM is relevant as a baseline. The fact that it is “designed to handle more complex stochastic data with multimodal distribution” does not mean it is not relevant for cases that do not exhibit much stochasticity. And it shouldn’t incur a different computational cost at training time. I think a comparison to EDM would be very useful to the community to figure out what the fastest and most accurate way to set up diffusion models for dynamical systems is. If the methods proposed here outperform EDM, this makes the paper stronger. If they don’t, I think this would also be a valuable result, potentially implying that EDM is a robust technique (regardless of data distribution) that should be used as a “first thing to try” as opposed to spending time and resources on hyperparameter tuning of more ad-hoc techniques.\n   Maybe the authors could include a comparison to EDM on one or two experiments?\n6. **Unclear experimental setup.** I found it hard to figure out some experimental details in certain places. I am aware these details exist in other papers (Kohl et al. [1]), but for ease of interpreting the results, it would help to include some more details, potentially as a brief table in the appendix which summarises the most important dataset characteristics. For example: \n   - Tra - What is the Mach number range of the training trajectories? Is it also Ma $\\in [0.53, 0.63] \\cup [0.69, 0.90]$ as in Kohl et al.?\n   - Tra - How many trajectories are there per Mach number?\n   - Fturb - I am slightly confused about the number of states within each trajectory for this dataset. You mention that each simulation contains 51 temporal states, but do you just consider 30 out of these for the test results? And what do you mean by “AR sampling is employed for $R = 30$ timesteps … for two sequences per $Re$?”\n   - Fturb - you are feeding in the $Re$ number as conditioning information to the model as in Kohl et al., right?\n\n**Minor**\n\n7. **Comparison to DDIM (L306)** - You say that “IR should supersede the deterministic DDIM sampling regarding accuracy and NFEs” due to its stochastic nature. While I agree that stochastic sampling is beneficial because it can correct previous errors in sampling, Karras et al. [4] mention that, in practice, the situation is more complex because approximating the extra Langevin term introduces error in itself (see Section 4 Stochastic sampling). Thus, I would not say there is any guarantee that IR would supersede DDIM in all scenarios.\n8. **Lack of discussion on $j$.** I find it interesting that the optimum $j$ value varied so much between methods, as shown in Tables 6 and 7, but you do not include any discussion about this. Do you have any insight about why this might be the case?\n9. **Small typos**, such as L291 “to be evaluated”, L776 “presnted”, L772 “where” missing at the beginning of the line, L819 “allow” rather than “allows”, L840 “are not needed” etc.\n\nOverall, I think the paper is clearly written and structured, and generally presents convincing empirical evidence. However, I think its quality could be significantly improved by including experiments on longer rollouts, a comparison to EDM, and potentially clarifying the regimes in which the techniques are effective (with the inclusion of negative results if necessary).\n\n[4] Karras, T., Aittala, M., Aila, T., & Laine, S. (2022). Elucidating the Design Space of Diffusion-Based Generative Models. ArXiv, abs/2206.00364.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)