Physics-Informed Diffusion Models
Authors: Jan-Hendrik Bastek
WaiChing Sun
Dennis Kochmann
Venue: ICLR 2025 Poster (2025)
Keywords: physics-informed
scientific machine learning
denoising diffusion
inverse problems
generative modeling
Primary area: generative models
TL;DR: We introduce a framework that unifies diffusion models with physics-informed neural networks and significantly improves the adherence of generated samples to physical laws.

Abstract: Generative models such as denoising diffusion models are quickly advancing their ability to approximate highly complex data distributions. They are also increasingly leveraged in scientific machine learning, where samples from the implied data distribution are expected to adhere to specific governing equations. We present a framework that unifies generative modeling and partial differential equation fulfillment by introducing a first-principle-based loss term that enforces generated samples to fulfill the underlying physical constraints. Our approach reduces the residual error by up to two orders of magnitudes compared to previous work in a fluid flow case study and outperforms task-specific frameworks in relevant metrics for structural topology optimization. We also present numerical evidence that our extended training objective acts as a natural regularization mechanism against overfitting. Our framework is simple to implement and versatile in its applicability for imposing equality and inequality constraints as well as auxiliary optimization objectives. Code is available at https://github.com/jhbastek/PhysicsInformedDiffusionModels.

Published as a conference paper at ICLR 2025
PHYSICS-INFORMED DIFFUSION MODELS
Jan-Hendrik Bastek
WaiChing Sun
Dept. of Mechanical and Process Eng-
Dept. of Civil Eng. and Eng. Mechanics Columbia University
ETH Zurich
Zurich, Switzerland
New York, NY, USA
jbastekBethz.ch ch
wsun@columbia. edu
Dennis M. Kochmann
Dept. Mechanical and Process Eng. ETH Zurich
Zurich, Switzerland
dmk 0ethz. ch
ABSTRACT
Generative models such denoising diffusion models are quickly advancing their ability to approximate highly complex data distributions. They are also increasingly leveraged in scientific machine learning, where samples from the implied data distribution are expected to adhere to specific governing equations. We present a framework that unifies generative modeling and partial differential equation fulfillment by introducing first-principle-bsse loss term that enforces generated samples to fulfill the underlying physical constraints. Our approach reduces the residual error by up to two orders of magnitude compared to previous work in a fluid flow case study and outperforms task-specific frameworks relevant metrics for structural topology optimization. We also present numerical evidence that our extended training objective acts as a natural regularization mechanism against overfitting. Our framework is simple implement and versatile in its applicability for imposing equality and inequality constraints as well as auxiliary optimization objectives.
INTRODUCTION
Denoising diffusion models (Sohl-Dickstein et al. 2015) are generative models that have gained
popularity due to their success in learning intricate data distributions across various modalities,
including images (Ho et al. 22000; Nichol et al.) 2021 Rombach et al. (021). l). videos (Ho et al.
2022a b), graphs (Niu et al. 2020 Hoogeboom al. 2022). text (Li et al. 2022 Wu al. 2023), and
audio waveforms (Kong et al. 2021). Originally popularized within the image generation community,
their outstanding representation abilities are also increasingly leveraged in the context of scientific machine learning. Diffusion models have been used, among others, to upscale
ww-fiidlity data to reduce computational costs (Shu et al., 2023), design new molecules (Xu et al. (2022) materials (Xie et al., 2022: Düreth et al. 2023) with desired properties, or generate metamaterial unit cells tailored to given stress strss--traannao response in complex mechanical settings (Bastek Kochmann 2023). For such inverse problems, key strength is their ability to efficiently capture the full distribution of feasible solutions and designs rather than focusing on a single deterministic outcome.
A common thread of all those applications is the explicit knowledge of the underlying governing equations, which such implied distribution must obey. Often, the training data does not stem from experimental observations but is rather generated by numerical simulations, which ensure that those equations are fulfilled. Nevertheless, diffusion models are traditionally trained on a purely data- driven objective (Xie et al.
2022 Buehler 2022 Bastek & Kochmann) 2023) Vlassis & Sun 2023 Sardar et al. 2023 Li et al.
2023 Lienen et al. 2024) and hence do not strictly enforce the intrinsic
constraints during model training. Consequently, samples generated from such models may be
Code is available
thtb:///tttuuu.ooo. http://hcc....//aatt/////////eee Informedds ffusi onModels

---

Published as a conference paper at ICLR 2025
statistically aligned with the training data but may not meet the required precision in scientific applications, where adherence to the underlying physics is crucial for deployment (Jacobsen et al. 2024).
More recently, efforts have been made to ensure that samples generated from the learned distribution conform to the known constraints. Examples include imitating human motion (Yuan al. (022))
ensuring manufacturability of proposed designs (Mazé & Ahmed, 2023), or, most importantly for physical systems, satisfying the underlying physical laws (Shu al. [023] Jacobsen al. (024).
typically given by a set of partial differential equations (PDEs). Yet, a fundamental framework that rigorously addresses the incorporation of PDE constraints in such generative model settings, along with robust mechanism to enforce these constraints akin to the well-established
neural networks (PINNs) (Raissi al. 2019) has remained elusive. We close this gap by proposing new framework that unifies both settings and integrates PDE constraints meaningfully into the training process. Unlike previous approaches that primarily rely on some form of post-processing of generated samples and lack derivation from first principles, and directly embed constraints into the proven representation strength of diffusion models. This approach yields state-of-the-are results in terms of PDE fulfillment. By drawing synergies between the domains of PINNs (Raissi et al. (2019) and generative modeling, we introduce physics -iffocs- diffusion models (PIDMs).
provide this theoretical foundation
Contributions. We make the following key contributions: (i) We present a novel and rigorous approach that unifies denoising diffusion models with PINNs and informs the model of PDE con- straints during training, and we demonstrate via rigorous numerical experiments that our framework significantly reduces the PDE residual compared state-of-the-ar methods. (ii) We provide evidence that the additional training objective does not necessarily compromise the data likelihood; instead, we observe that it acts as an effective regularization against overfitting. (iii) Our approach is simple to implement into the training protocol of existing diffusion model architectures, and inference is unaffected. (iv) While we here focus on PDEs as a sophisticated type of equality constraint, our framework is equally applicable to other equality and inequality constraints as well as auxiliary optimization objectives (potentially also provided via a differentiable surrogate model).
BACKGROUND
2.1 DENOISING DIFFUSION MODELS
Denoising diffusion models are state-of-the-art generative models (Ho et al., 2020 Dhariwal & Nichol 2021 Yang et 2024) that learn to gradually convert sample of a simple prior, typically unit Gaussian, to a sample from a generally unknown data distribution g(zp). The idea is to introduce a fixed forward diffusion process that incrementally adds Gaussian noise to a given data sample
zo ~ g(x)), following variance schedule {Be E (0, (0,1)
over steps, defined as
((11r (mo) = II (
g(<,(xx--)) N(a;;; v 1,
(1)
1
To generate new samples, we consider the reverse process,
T
g( = p((x))))
II ((==-1)<1) 1
q( = N((0-11:1 t). E(xr, t)),
(2)
in which we approximate the unknown true inverse conditional distribution g( lxx)) with neural network Py (xx-1112;) parameterized [ by It aims to estimate the mean μ4o. while we fix the covariance to Σ (x,, t) - trained by maximizing the variational lower bound of the log-likelihood, which can be simplified to several loss terms that mainly consist of KL-divergences between two Gaussians (and are hence computable in closed form) (Sohl-Dickstein et al., (2055). While the obvious choice is to estimate μlg. alternative parameterizations possible. We can obtain the mean μ4 timestep via combination of the reparameterization trick and Bayes' theorem (Ho et al. 2020 Kingma & Welling 2013):
I-ar 3 I, with ā₂ = I 04, a = 1 -- Br. The network is
1
31
01
x4-1)
E. + va--18
(1 Et) =
Zt
E1
Z0.
(3)
VI
04
ae
1 āz
2

---

Published as a conference paper at ICLR 2025
where e represents Gaussian noise to diffuse Zo to z₁. Since d; is known during training, predicting Hy fixes the Gaussian noise E or the clean signal zy and vice versa, and we can equivalently train the model to predict these quantities. While Ho et al. (2020) simplified the training to minimize an unweighted noise mismatch, we here consider the loss
L(0) == Er,...e A | To (Zo. e). t)]l
(4)
where To is the model estimate of the clean signal (omitting the parametric dependence 0 for conciseness), and Å is set to Min-SNR-5 weighting (Hang et al., 2023). Note that equation f is equivalent to the mean squared error of the e fe mismatch up time-dependent weighting factor (Ho et al. 2020; Salimans & Ho 2022).
2.2 ASSEMBLY OF GOVERNING EQUATIONS
Physical laws are typically formulated as set of PDEs over domain Q C IR expressed as Flu(E)] = 0, (21,5,, :Ea' S (u1(E), uz uz(E), "*", Uh (e())) IR,
(5)
with boundary conditions
Biu(E)]= 0, & a2,
(6)
where F is differential operator, B is boundary condition operator, 0n is the boundary of domain Q and u is the sought solution field that satisfies the set of PDEs for all & E St and boundary conditions ön. We emphasize that reserve for spatial coordinates defined domain, while x refers to data, which here typically contains the solution field on a discretized set of spatial coordinates, elaborated below.
physical
We assume that samples generated from the model æn (o(x)) must satisfy equation and equation [6] In this study, mainly consider image-based architectures common in diffusion models. More formally, we consider a discretized pixel grid 82hh C x Z (which may serve as an approximation of the continuous domain R²2,, where 0 consists of the boundary pixels. For X n pixels, the model's output is thus defined over 2n E _x For instance, in the context of Darcy flow problems in porous media (Jacobsen et al., 2024) 2. describes the pressure field, or in the context of solid mechanics the displacement field (Gao et al. 2022). While PINNs (Raissi et al. 2019) establish an explicit map .V E + u between the input and solution field, which enables the computation of required derivatives for F (and potentially B) via automatic differentiation, we here approximate those derivatives via finite differences or comparable methods that directly operate on Zp. The residual (zo) of the (discretized) solution field æ is then established as a measure of the discrepancy between the generated sample Ty and the governing equations it expected to satisfy. is defined by the corresponding PDEs and the respective boundary conditions. More precisely, stack both contributions into a vector as
F w 13 æ
R(
For evaluation, we introduce the mean absolute error RMAE(ZD) defined in equation 28 Ap-
pendix A.6
3 PHYSICS-INFORMED DIFFUSION MODELS
We explore a scenario in which our generative diffusion model must learn a distribution whose samples are to comply with set of governing equations, i.c.. R(x,) = 0. In the usual data-driven setting, this is only indirectly ensured through the training data, typically collected from a forward simulator that produces data points which inherently follow the governing equations. These fully describe the physical system and provide us with, in principle, an infinite source of information (Rixner & Koutsourelakis 2021), whereas solved instances { .... 28 posing as training data, represent only finite set of evaluations in terms of the sought solution fields [ Thus, our strategy is to first state our optimization objective based on the more fundamental governing equations and only subsequently incorporate training data. it typically imposes constraints on higher-order derivatives (comparable to Sobolev training (Czamecki et al.) 2017)).
Even if we assume access to infinite training samples, the residual information may still be beneficial, since
3

---

The goal is to Published as a conference paper at ICLR 2025
3.1 CONSIDERATION OF PDE CONSTRAINTS
We maintain the probabilistic perspective of generative models and introduce the residuals as virtual observables (Rixner& & Koutsourelakis 2021 distribution
which we consider to be sampled from the
qR (f) = (ñ; (xo),
(8)
In the limit o² + O, this recovers the deterministic setting of strictly enforeing the residual equations. as all probability mass is concentrated in (xx)). This can be used to compute the virtual likelihood pf)),, which we expand in terms of drawn samples 2o as
po( = f (f, ao) = ( zo) (200) dæz = Ewo-pe (ce)) QR (F (0))
This factorization of po (f, zo) is reasonable as the residual follows via equation 8 maximize the usual log- liliiiihhoodd over the virtual observable i.e.,
arg max Er [log po (f)] arg max Eeg [log qu (f 0[
(10)
Thus, any samples from pe are evaluated on their log ligeliillhood via equation 8 which can also be understood as probabilistic reinterpretation of the loss used to train PINNs (Raissi et al. 2019). Alternatively, equation 10 can also be understood as sampling from an unnormalized target distribution assembled by eva evaluating log 9R over Ig, as is often considered in data-free settings (Zhang & Chen 2022; Vargas et al. 2023) [Brrner et al. 2024p. Our setting, however, differs as we consider the more common scenario of applying diffusion models to learn data distribution, as elaborated as follows.
3.2
CONSIDERATION OF OBSERVED DATA
We emphasize our focus on generative model class in which training data is typically available. This crucial for two main reasons: first, identifying solutions 2g that satisfy the governing equations is non-trivial, and we may understand the obtained training data as guidance for the model towards feasible solutions, which may accelerate the training. Alternatively, the constraints may be far from "well-posed" problem and admit large set of solutions, while we only interested in small subset of those reflected by the data. Second, equation 10 contains any distribution that produces samples with zero residuals, and the model may simply collapse to single solution instance (such as the classical PINN setup), which is not the use case for a generative scenario. Rather, we aim leverage the proven capabilities of generative models to learn complex distributions, such as optimal mechanical designs or fluid flows based on some conditioning, while ensuring adherence to the physical laws. Even if no data is available, we may introduce an uninformative prior such as unit Gaussian to promote the exploration of different solutions, which we briefly investigate in Appendix A.6.I
We thus extend the objective equation 10 by including the usual data likelihood term, which is taken as the standard optimization objective in data-driver diffusion models as
arg max Exz [log pa (wo)] + (xp [log q 0 [x)]]
(1))
0
We show in Appendix A.I that, if diffusion models interpreted score-based models. straight- forward inclusion of the virtual (residual) likelihood in the loss function recovers a consistent training objective in the sense that the optimal score model also maximizes the introduced virtual (residual) likelihood and recovers samples from g(
3.3 SIMPLIFICATION THE TRAINING OBJECTIVE
The joint loss equation I| requires sampling not just from ((wo) but also po (=(o), which is costly for diffusion models due to their iterative nature. We explore two ideas to mitigate this increased complexity by considering (i) straightforward evaluation of the residual based the readily available Eg and (ii) an accelerated sampling strategy via denoising diffusion implicit models (DDIMs) (Song et al. 2021a).
4

---

Published as a conference paper at ICLR 2025
Mean estimation. As seen in equation 4 the diffusion model objective be understood as minimizing a weiehweig mean-squared distance between the predicted and true sample, given a noisy input. Hence, &o is estimate of [[ (Song et al. 2021b). Evaluating the residual this estimate is therefore in general not fully consistent, as ( Ztn x. * R (zool (2 [x:)] (also referred as the Jensen gap (Gia et al. 2017)). Only the last sampling step will this estimate align with generated sample To (i.e., at f 1, assuming 31 0) and otherwise introduce a conflicting objective between the data and residual loss with inereasing To mitigate this, we propose increase the variance of the introduced residual likelihood with increasing t, while enforcing tighter adherence to the constraint as +
Sample estimation. Alternatively, we may aim to evaluate the residual of an actual sample 20 ~ (o(o)), motivated by the implied consistency shown in Appendix A.. (though in idcalized settings). This comes at the expense of increased computational complexity due to the many forward passes required the denoising process. We hence consider (deterministic) DDIM (Song al.. 2021a) to accelerate sampling while maintaining high sample quality, with an inherent trade-off between sample quality and the number of DDIM timesteps. from any l and again increase the variance reduced sample quality (sce Appendix |A.4 for further details). While we focus on diffusion models, note that single-step generation models such consistency models (Song et al. (2023) are promising alternative, offering direct access to a sample.
consider a simple two-step sampling of wo + T, where timesteps are coarser, which leads to
Since the remaining derivations hold true for both the mean and sample estimation, distinguish between these estimates and denote both of them with z (i.e., 2/ == c = E [2 æ or z DDIM 2.)), based which the residual can efficiently be evaluated. As æτ is obtainable any timestep 4, we adapt equation 11 to
longer
arg max (= [log (x0)) + (mprr-po( [log ( (x))())))
(12)
Estimates at noisier inputs may be less accurate for aforementioned reasons, thus we penalize constraint violation less as + T by modeling qu (f]z)(<1:1) (#1:r) for simplicity based on scaled version of the variance scheduler of the reverse process
T
9Re Tie(flan (f(x)(((11))) = II Qr (F(
where (x((fa](( +)) = (f; (xx)). E1/cI),
(13)
where Σ, is, as introduced before, the fixed variance of the denoising process. This idea of adjusting the temperature of the target distribution shares some similarity with the annealed noise distribution introduced in concurrent work by Sanokowski et al. (2024) in the context of combinatorial opti- mization. The scale factor 0 introduced in equation 13] which is the only hyperparameter in our setup, effectively dictates the penalty for deviating from R( O. Figure shows graphical illustration of this process.
N (f 0; R( E,)
E 2 or DDIM
0
po (o(x (x1-1 x1)
TT
Et
F0 R (z)) = 0
Figure 1: An approximation 2Ú of the clean signal for residual evaluation can be obtained at any denoising timestep by directly considering the estimated expectation E[ |as]] or by actual (accelerated DDIM (Song et al. 2021a)) sampling. We tighten the variance of the virtual likelihood as 0.
As equation 12 still requires sampling over æ1.7 ~ (x we simplify this by instead sampling from the available q(F1.r), effectively ignoring the likelihood ratio (see Appendix A.3 details).
5

---

Published as a conference paper at ICLR 2025
As the expectation, the classical data loss equation 4 can straightforwardly extended to include a physics- informed loss, resulting in the loss of our proposed PIDM:
original (Sohl-Dickstein et al. 2015) and thysics-drived objectives then under the same
LPID( = 11,-111,7] :00 >e| - Eo t) + 2
[ t>]1
(14)
where Σn = E is the rescaled variance. Since En 0 for deterministic last step, we set Eg -- Σn. Algorithm displays disppay the updated training objective, requiring only minor modifications to the standard training setup (Ho al. 2020).
Algorithm 1 Physics-infonme diffusion model training
1: Set Σ, E//c
2: repeat
3: Zo ~ g(20o)
4: t~ Uniform (1,.... T)
5: C~N(O,T)
6:
3² P==vārat VI ā,e
7:
Estimate æg via @n- = E| [e.] or DDIM sampling (Song et al. 2021a | ( x1. t) 1
8: Take gradient descent step on Vo >ell=z - 20 t)1 + 25.
9:
until converged
We point out that the model can also be trained to match the mean or noise as in Ho et al. (020), from which
equally assemble E[20o] [z,] if considering mean estimation via equation (see Appendix A.2 for estimation with score models) but we empirically found that this leads to larger residual errors. Also, we emphasize that PDEs can be understood constraints and we could generally consider any differentiable forward surrogate model Ro that estimates some property classification of the predicted samples to be matched, i.e., Rg (arg) i Farget- For a discussion on how to incorporate inequality constraints and auxiliary optimization objectives see Appendix Lastly, we note that the optimal scaling c generally depends on the considered scenario, and it is here estimated by simple parameter sweep. As usual in multi-objective optimization, trade-offs between the different loss contributions are expected, and e must be selected so that the model is meaningfully informed by the residual loss but does not ignore the data likelihood.
specific instance of equality
4 EXPERIMENTS
We here present two benchmarks for distributions from which samples are implied to adhere to specific governing PDEs and constraints, recently studied in contemporary work (Jacobsen et al., 2024, Giannone et al., 2023). To obtain intuition for the effects of the proposed loss, see the toy problem presented in Appendix A.6.1 1.661 as an instructive example.
4.1 DARCY FLOW
Setup. We first study the 2D Darcy flow equations, which describe the steady-state solution for fluid
flow through porous medium, here square domain f2 [0, 1]2 Generally, follow the setup
of previous studies (Jacobsen et al. 2024 Zhu & Zabaras 2018) by sampling permeability fields
(({) from Gaussian random field, which solved for their (unique) pressure distribution p(E). Similarly, we create a training and a validation dataset of 10,000 and 1,000 datapoints, respectively. by solving the governing equations (see equation 29) for sampled permeability field on 64 x 64 grid. Second-orded central finite differences are used to assemble and solve linear system (see Jacobsen
al. (2024)), giving pairs (K,P) P) with K, K,P& E Raxx. We consider a U-Net (Ronneberger et 2015) architecture with 64 x 64 pixels as in- and output dimensions that align with the considered grid and allow for the same residual evaluation via finite differences also used to create the dataset. We increase the variance of the virtual residual likelihood by setting 10-3 for the mean estimation and c = 10 for the sample estimation. Throughout the experiments, we observed that the mean estimation performs best with variance of the residual likelihood around two orders of magnitude smaller than the best performance for the sample estimation. Further details of the considered setup and implementation are given in Appendices A.6.2 and A.7

---

Published as a conference paper at ICLR 2025
(a) Residual error
(b) Test data loss
104
MYAE (270) 200 ~ Po (20)
10a
CoCoGien
Diffusion
PIDM-ME (ours)
PG-Diffusion
PIDM-ME (ours)
Diftusion
PG-Dituusion
PIDM-SE SE (ours)
(ours)
Zoll
100
104
200
10-1
10
10-1
10-2
E
10-2
10
2
3
Training iteration
-105
Training iteration
-10
Figure 2: Evaluation of the residual error (a) and test data loss (b) during training. In (a), we generate 16 samples every 10k training iterations and plot the average (solid lines) and individual (dots) residual errors for the standard diffusion model ("Diffusion" ) the physics physics-guided guided model IPG-Diffusion (Shu al. 2023), CoCoGen (Jacobsen et al. 2024). and the proposed PIDM using either mean or sample estimation ("PIDM-MF" and "PIDM-SE respectively). Note that for CoCoGen not all samples converged, so that we excluded the non-converged data from the indicated average. In (b), plot the data loss evaluated test set for the proposed PIDM variants and those frameworks that differ from ours during training.
(a)
Permeability
Pressure p
Residual RMAE RMAE(K,
10a
101
Diffusion
100
5
0.0
S
2
0.2
(b)
Permcability K
Pressure
Residual RAMAE K,p)
0.10
10
PIDM
5
10-2
0.00
00.55
0
10-3
E1
EL
E1
Figure 3: Generated permeability and pressure fields well the corresponding residual error from diffusion models trained on the Darcy flow dataset, where (a) is sampled from a standard diffusion model and (b) from our proposed PIDM with mean estimation. Additional samples are shown in Appendix A.8.I
Results. To evaluate the performance of our proposed PIDM, we benchmark it against three relevant
setups: (i) a model trained on the standard (purely data-driven) objective equation 4 (ii) a "physics- guided" model trained on the standard objective but using residual information guidance, as proposed by (hhu et al. (2023), and (iii) model similar to (i) but with first-order residual corrections during inference, described in CoCoGen (Jacobsen et al. 2024 (see Appendix A.6.2 for further

---

Published as a conference paper at ICLR 2025
details). We examine the performance of the different diffusion model variants by tracking the evolution of the residual eтror of generated samples from the learned distributions zo ~ po(xx) alongside the test data loss throughout the training process in Figure 2 The PIDM showcases a remarkable improvement, reducing the residual error by around two orders of magnitude in comparison to the standard diffusion model. This holds for both the mean and sample (DDIM) estimation, where the former outperforms the latter variant in terms of the residual error. The mean estimation takes around 23% longer to train compared the standard diffusion model due to the residual evaluation, whereas the sample estimation took around 69% longer due to the additional forward pass (see Appendix(A.44, We could not observe a significant reduction of the residual error for the physics-guided model (Shu et al., 2023). We hypothesize that the mere inclusion of gradient information into the model seems insufficient because does not truly enforce residual minimization. CoCoGen (Jacobsen et al. 2024) shows a moderate improvement but does not bring the residual close to values where the samples could be considered physically consistent. We display generated samples in Figure and Appendix A. for both the standard diffusion model and PIDM, confirming a drastically reduced residual error over the whole domain. Figures 2p. Pp and confirm that the PIDMs maintain the generative diversity with both estimation techniques and do not collapse to single solution instance. It successfully generates permeability (and pressure) fields that adhere to the data distribution (besides respecting the PDE).
For any multi-objective loss as given in equation 14 a trade-off between the data and residual minimization is generally expected, depending on their relative weighting dictated by c (Wang et al., 2021). Interestingly, Figure 2p shows that the PIDM with mean estimation eventually recovers a similar test data loss as the vanilla diffusion model and is significantly less prone to overfitting. Even more notable is that the PIDM with sample estimation initially has similar test data loss trajectory as the vanilla model-despite despite the additional loss term and also remains more robust to overfitting. This is strong evidence of the consistent data and physics loss (see Appendix (. Thus, sample estimation drastically improves the alignment of the two losses with only a single additional forward pass. The increased robustness to overfitting in both mean and sample estimation is clear indicator that the model Iearns a more robust internal representation of the data distribution, as it is forced to generate samples that adhere the true distribution or, equivalently, the underlying data generation mechanism- viz. the known constraints that govern the system. Thus, adding the physical (residual) loss may benefit the generative performance by enhancing its generalization capability.
4.2 TOPOLOGY OPTIMIZATION
Setup. We consider 2D structural topology optimization as a second example. In this setting, the goal is to find the optimal material distribution (E) that maximizes the mechanical stiffness, or equivalently, minimizes the compliance of a structure under a set of constraints that typically consist of mechanical equilibrium, boundary conditions, and a volume constraint. This is classically solved via the SIMP method based on a finite element (FE) discretization (Bendsøe & Sigmund 2004).. We again consider a square domain Ω = 00, 1]2 and benchmark our proposed PIDM to state-of-thh-an frameworks (Mazé & Ahmed) [023 Giannone et al., 2023) that also provide dataset consisting of 30,000 optimized structures with various boundary conditions and volume constraints and two proposed test scenarios with in- and out-of-distributiio boundary conditions. We train U-Net architecture similar to the one in Section 4.1 but with a larger latent dimension and additional in- and output channels this dataset. To ensure consistent evaluation of the residuals, we do not use finite differences but interpret the pixels as nodes of the underlying FE mesh to assemble a consistent stiffness matrix equívalent the one used to generate the data. We scale the variance of the residual likelihood with e = 0.01 and introduce the volume constraint as an additional equality constraint with 0.1 (as the optimal topology will contain the maximum allowed material). We also introduce a slight bias to minimize compliance (setting À = 10-6 for the optimization objective, see Appendix A. Further details of the setup, model architecture, residual evaluation, and implementation are presented in Appendices A.6.3 and A.7
Results. We evaluate the performance of our proposed PIDM in comparison to the standard diffusion model, PG-Diffusion (Shu et al. 2023) and CoCoGen (Jacobsen et al. 2024). as well as two recent variants specifically tailored to topology optimization. These propose to modify the sampling process by either using additional guidance models to reduce compliance and improve manufacturability (Mazé & Ahmed 2023) or enforeing the denoising trajectory to be closer an iterative optimization
8

---

Published as a conference paper at ICLR 2025
trajectory (Giannone et al. 2023). Both methods require auxiliary datasets which complicate model training in contrast, our method is a much simpler and well rell-motivated extension of the standard training without requiring any additional data models. While training of the 'main' diffusion model takes longer for the PIDM due to the additional computational complexity and memory requirements, note that inference is identical the standard diffusion model and does not require additional surrogate models as, e.g., in Mazé & Ahmed (2023). Results are shown in Table I for the PIDM with sample estimation, which we observed outperform the mean estimation slightly. We evaluate relevant performance metrics on the two test sets (with seen and unseen boundary conditions, respectively). with generated samples presented in Figure and Appendix A.8.2
Table Performance comparison of diffusion model variants topology optimization. We consider in- and aut-of-distrribue
boundary conditions described in Mazé & Ahmed (2023) and provide the median RMAE of the predicted solution fields (where applicable), the median compliance error (MDN % CE) and the mean volume fraction error (% VFE). "Giannone al. (2023) further improve their model by running additional SIMP post-processing steps, but for consistent comparison, we consider unprocessed samples from the model.
in-distribution
out-of-distribution
RMAR
VFB
RMAR
CE 4
Diffusion
136M
1.86e
-0.2
2.93
1.97e-3
2.80
PG-Diffusion (Shu 2023
136M
1.82c-3
0.09
3.59
1.92e-3
0.81
3.23
CoCoGen (Jcobsen el al. 2024
136M
1.51c
0.14
4.00
1.56c 3
0.58
3.64
TopoDiff-G (Maze & Ahmed
2023
239M
0.83
1.49
1.82
1.80
DOM Giannone el all
121M
1.59
PIDM (OuTs)
136M
1.24e-3 3
0.06
2.25
1.29e
0.56
1.91
(a)
Design
Residual RMAE(P, ug U2)
SIMP Design
(CF- 01.6%% 0.49)
(C = 3.89, Vaax = 0.48)
10
0.6
0.6
0.4
10-2
0.4
Design
Residual RMAE(P, U. U22)
SIMP Design p (C 6.05. Vaux 0.32)
(CE 3.06%. 5 0.32)
(b)
1.0
10-1
0.8
10-2
0.2
0.2
10-3
0
41
Figure 4: Generated designs, including the compliance error CE and volume P. and the residual error (based on the displacement fields, not shown) from diffusion models trained on the SIMP dataset and the corresponding SIMP design, including the compliance C and volume Vmax- We plot sample (a) from a standard diffusion model and (b) from our proposed PIDM. All samples are conditioned on the out-of-distribution the given boundary conditions in red.
In the SIMP design, we indicate the applied load by a blue dot, and
Importantly, the PIDM provides not only the optimized designs but also the displacement fields, which are of significant interest for mechanical analysis, e.g., to estimate the stress distribution in the

---

Published as a conference paper at ICLR 2025
structure. We observe residual reduced by 33% and 35% compared to the standard diffusion model on the in- and out-of-distribution set, respectively, highlighting closer adherence to the mechanical equilibrium. Somewhat surprisingly, observe that the standard model performs very well in terms of the optimization objective, but also emphasize that it has an increased volume fraction error-thus more volume will allow for stiffer structures. Contrary, the PIDM only has a slight increase in the volume fraction error but significantly outperforms previous frameworks (Mazé & Ahmed] 2023 Giannone et al. 2023 terms of compliance minimization, despite the absence of auxiliary data and surrogate models. Lastly, adaptations of PG-Diffusion (Shu al., 2023) and CoCoGen (Jacobsen et al., 2024) also cannot match the PIDM in terms of the residual (on both test sets), though CoCoGen can reduce it slightly. Additionally, the median compliance and especially the volume fraction error perform significantly worse.
results are not fully comparable,
RELATED WORK
Conceptually closest to our work are two recent contributions by [Shu et (2023) and Jacobsen et al.) (2024), to which we compare the proposed PIDM extensively in Section Our studies indicate that the PIDM significantly reduces the residual error of generated samples compared to both variants, especially for the Darcy flow study. Focusing on topology optimization, two recent contributions (Mazé & Ahmed 2023; Giannone et al. 2023) propose improvements that require auxiliary data and/or surrogate models, while we here show that the PIDM, solely by being
significantly outperforms both methods in terms of the optimal stiffness. In a broader context, Wang al. (2023) optimized the conditioning variable create an online dataset of soft robot designs, which includes physical performance and may generate samples with improved physical utility. Yet, the focus was leveraging pre-trained models create diverse 3D shapes. Yuan al. (2022) leveraged diffusion models for human motion synthesis, where inference is altered by projecting the intermediate steps to a physically plausible motion that is verified via a reinforcement learning approach. Similarly. Christopher et al. (2024) projected intermediate steps to the closest point in feasible set, which is generally unknown for more complicated constraints such as PDEs. general, such processcns methods may indeed mitigate some of the mismatches of the generated samples (or fulfill them exactly if the constraints are sufficiently simple). Yet, they are fundamentally limited, as they do not address the underlying distribution learned by the model.
6 CONCLUSION
We have unified the data-driven perspective of diffusion models with a physics-informedt paradigm, enabling the models to internalize the constraints that generated samples must adhere to. Our framework significantly outperforms purely data-driven models and prior work, verified by two highly relevant case studies, and numerical evidence hints that the PIDM obtains a more robust representation of the data distribution that is less prone overfitting. We hope this work stimulates others to extend their generative model training objective when, besides data, further information-Ba it in the form of PDEs other constraints- on the generated samples is available, as is often the case within the realm of scientific machine learning. Future work may explore more sophisticated virtual likelihood variance schedulers, which we here simply coupled to the standard denoising process with scaling estimated by a parameter sweep. More generally, architectures with consistent residual evaluation that do not operate on fixed regular grid should be considered, currently restricting the studies to geometrically simple domains. Remedy can be found in graph-based architectures that can handle arbitrary meshes (Gao et al., 2022) or by employing implicit encodings of coordinates (Liu et al.) 2023). Additionally, coordinate-based representations allow for exact calculation of the gradients required to evaluate the imposed PDEs via automatic differentiation, though at increased computational cost.
ACKNOWLEDGMENTS
The authors thank François Mazé for providing the code to generate the additional datasets for the topology optimization case study and Matheus Inguaggiato Nora Rosa for the helpful discussions about the corresponding implementation of the residual evaluation.
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey Hinton. Layer Normalization. 2016. URL
http:://rat www.//gg//////o/.///.o//..//..//
abs .06450
Matthias Baer. (findiff) Software Package. 2018. URL https:///gttubb.cmm/////
Findiff
Jan-Hendrik Bastek and Dennis M. Kochmann. Inverse design of nonlinear mechanical metamaterials via video denoising diffusion models. Nature Machine Intelligence, 5(12): ISSN 2522-5839. doi: 10. 000000064222--003-000022222 URL https: www.natuurecom
2023. 2023.
Setticlesss22555 020-5-0230077-00
Martin P. Bendsøe and Ole Sigmund. Topology Optimization. Springer Berlin Heidelberg, Berlin, Heidelberg, 2004. ISBN 978-3-642-07698-55 doi: 00.10079978-3-66-0506888 URL http:
//1ink.springes. com 10. 11.110111191111 10w.0000---762--00000000--
Julius Berner, Lorenz Richter, and Karen Ullrich. An optimal control perspective on diffusion-based
generative modeling. Transactions on Machine Learning Research, 2024. ISSN 2835-8856. URL
htttps::/openreview net wwt.tteuutt////eet Forum?id=oYI
Markus J. Buehler. Modeling Atomistic Dynamic Fracture Mechanisms Using Progressive Transformer Diffusion Model.
Journal of Applied Mechanics. 89
(12), 2022. ISSN 0021-8936.
doi:
URL https://
Asmedigitalcs Ilection. iem asme org/appe 221009/1146377/Moeee
Atomistic wwwpnastii-DDamii--Peeeeo - Mechanisms
Jacob K Christopher, Stephen Baek, and Ferdinando Fioretto. Constrained synthesis with projected diffusion models. In The Thirty-eighth Annual Conference Neural Information Processing
Systems, 2024.
Wojciech Marian Czarnecki, Simon Osindero, Max Jaderberg. Grzegorz Šwirszez, and Razvan Pascanu. Sobolev Training for Neural Networks. 2017. URL http://arxiv. abs/
170600483000000 04859
Prafulla Dhariwal and Alexander Quinn Nichol. Diffusion models beat GANs on image synthesis. In Advances in Neural Information Processing Systems, 2021. URL httppss///opeereeeee
net. pert/formm? d=AAWuCvz.VV.
Christian Düreth, Paul Seibert, Dennis Rücker, Stephanie Handford, Markus Kästner, and Maik Gude. Conditional diffusion-based microstructure reconstruction. Materials Today Communications,
35:105608, 2023. ISSN 23524928. doi: 10.10166j.mtcomm.2222. 2023. 105608. URL https:/7
linkinghub.c sevier. com/ http://ww.....eeeeeee/e/////////
Stefan Elfwing. Eiji Uchibe, and Kenji Doya. Sigmoid-weighted linear units for neural network func- tion approximation in reinforcement learning. Neural Networks, 107:3-11, 2018. ISSN 0893-6080. doi: http:://dbbaagy//l...o0ooeeeeees
URL https: www. sci http://eieeccccccerrreeeeeee
com/ http://iiiooeeerreeee//////////1
Han Gao, Matthew J. Zahr, and Jian Xun Wang. Physics-informed graph neural Galerkin networks: A unified framework for solving PDE-governed forward and inverse problems. Computer Methods in Applied Mechanics and Engineering, 390: 14502, 2022. ISSN 00457825. doi: 10.1016//..mm
2021. 14502. URL htthttps:: /oi. org 10160....111111 cma 2021 114502
Xiang Gao, Meera Sitharam, and Adrian E. Roitberg. Bounds on the Jensen Gap, and Implications for Cran-Concentrated Distributions. 2017. URJjttee//aax....orr/abb//
Giorgio Giannone, Akash Srivastava, Ole Winther, and Faez Ahmed. Aligning optimization trajec- tories with diffusion models for constrained design generation. In Thirty-sevnnth Conference on Neural Information Processing Systems, 2023. URLhttpp. ptttps://openrevi.ee net. formm?
1d=KTR33hMmMX
Juan Gómez and Nicolás Guarín-Zapata SolidsPy: 2D-Finite Element Analysis with Python, 2018. RAFTT/So1 idsPy
URL | httpt:t::///ot. // thub. com
11

---

Published as a conference paper at ICLR 2025
Tiankai Hang, Shuyang Gu, Chen Li, Jianmin Bao, Dong Chen, Han Hu, Xin Geng, and Baining Guo. Efficient Diffusion Training via Min-SNR Weighting Strategy. In Proceedings of the IEEE/CVF International Conference Computer Vision (ICCV), pp. 7441-7451, 2023.
Jonathan Ho and Tim Salimans. Classifier- Fessifier-FFre Diffusion Guidance. 2022. URL
org/ haee/ab//// 2207 12598
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models, 2020. URL
Sittps://axxiv.oo http://..rrraa////////////////
abs/ 2002.
Jonathan Ho, William Chan, Chitwan Saharia, Jay Whang, Ruiqi Gao, Alexey Gritsenko, Diederik P. Kingma, Ben Poole. Mohammad Norouzi, David J. Fleet, and Tim Salimans. Imagen Video: High Definition Video Generation with Diffusion Models, 2022a. PtLltttps///axii//
abs / 021022210.. 02303
Jonathan Ho, Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David J. Fleet. Video Diffusion Models, 2022b. URL
Emiel Hoogeboom, Victor Garcia Satorras, Clément Vignac, and Max Welling. Equivariant Diffusion for Molecule Generation 3D. 2022. URL httl:tttp///rrxi..rr/////// http://rr..orrraa/a//.....0.....
Christian Jacobsen, Yilin Zhuang, and Karthik Duraisamy. CoCoGen: AyssicallyCoonsistett and Conditioned Score-based Generative Models for Forward and Inverse Problems, 2024. URL
htttp:://arxv..er wwtp//ii.rr//oo/ooooo/o/// 2312. 10527
Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, and François Fleuret. Transformers are
RNNs: Fast Autoregressive Transformers with Linear Attention. 2020. URL http://arxiv
org/ abcc/bbs/2066.. 16236
Diederik P. Kingma and Jimmy Ba. Adam: A Method for Stochastic Optimization. 3rd International Conference on Learning Representations, ICLR 2015 Conference Track Proceedings, pp. 1-15, 2014. URLpttp://arxiin wwtp:/wwwwwwwwpww///./oaa// 6980
Diederik P Kingma and Max Welling. Auto-Encoding Variational Bayes. 2013. URL http:
7/arxiv. org/ httpg//b//131.... 6114
Diederik P Kingma, Tim Salimans, Ben Poole, and Jonathan Ho. Variational Diffusion Models. In Advances in Neural Information Processing Systems, 2021. URL http:tts////peeeeii... net/ 2d=22
Zhifeng Kong, Wei Ping, Jiaji Huang, Kexin Zhao, and Bryan Catanzaro. Diffwave: A versatile diffusion model for audio synthesis. In International Conference on Learning Representations, 2021. URL
Tianyi Li, Alessandra S. Lanotte, Michele Buzzicotti, Fabio Bonaccorso, and Luca Biferale. Multi- Scale Reconstruction of Turbulent Rotating Flows with Generative Diffusion Models. Atmosphere. 15(1):60, 2023. ISSN 2073-4433. htl.tttts:// ://www.w/wwwwwwww http://wwwww.........////. wwt.o//////433/1111012
Xiang Lisa Li, John Thickstun, Ishaan Gulrajani, Percy Liang, and Tatsunori Hashimoto. Diffusion- LM improves controllable text generation. In Advances in Neural Information Processing Systems, 2022. /wtp:ttpp///ppeeeeeeee...ec net. wtw.toouu//ttiooo
LLyk]
Marten Lienen, David Lüdke, Jan Hansen-Palmus, and Stephan Günnemann. From Zero to Tur- bulence: Generative Modeling for 3D Flow Simulation. In The Twelfth International Confer- ence on Learning Representations, 2024. URL https: http://pppeeeeeeeeeeee net forum?ide
ShlwoC1XaM
Hongying Liu, Zekun Li, Fanhua Shang, Yuanyuan Liu, Liang Wan, Wei Feng, and Radu Timofte. Arbitrary scale super-resolution via deep learning: A comprehensive survey. Information Fusion, pp. 102015, 2023. ISSN 15662535. URL 102015
1016////...0000..1.111.101011.11 w.fff////............... 2023
12

---

Published as a conference paper at ICLR 2025
François Mazé and Faez Ahmed. Diffusion Models Beat GANs Topology Optimization. Pro- ceedings of the 37th AAAI Conference on Artificial Intelligence, AAAI 2023, 37:9108 17:9108-9116, 9116, 2023. ISSN 5399. doi: v0.1609/aaii..37.8.2220. 26093.
Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen. GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models. 2021. URL BtLhtt:t//aatt
Zhenguo Nie, Tong Lin, Haoliang Jiang. and Levent Burak Kara. TopologyGAN: Topol- ogy Optimization Using Generative Adversarial Networks Based on Physical Fields Over the Initial Domain. Journal of Mechanical Design, 143(3), 2021. ISSN 1050-
0472. doi: 10.1115/1 4049533. 40.1533. org
http::pp:e/aammeecttaaaooolleeee URL http::::ss
asme
Topo
Chenhao Niu, Yang Song, Jiaming Song, Shengjia Zhao, Aditya Grover, and Stefano Ermon. Per- mutation invariant graph generation via score-based generative modeling. In Proceedings the Twenty Third International Conference on Artificial Intelligence and Statistics, volume 108 of Proceedings of Machine Learning Research, pp. 4474 4484. PMLR, 26-28 Aug 2020. URL www.rreeeeee......o....o.
ngs. w. html
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köpf, Edward Yang. Zach Veto,, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang. Junjie Bai, and Soumith Chintala. PyTorch: An imperative style, nigh-performance deep learning library. Advances Neural Information Processing Systems, 2019. ISSN 23318422. URL
http: /ttp:///axiv..o.o www.orgg///o... abs 11t//1112200110
M. Raissi, Perdikaris, and G.E. Karniadakis. Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential
equations. Journal of Computational Physics, 378:686-700 2019. ISSN 00219991. doi: 10. 2pp.6//... S0021 9991 8 8307125
URL httts:://iikkingggegssss
el sevier. wtt.:/oo//tt/eeeeee com/retrrivee/pii...
Maximilian Rixner and Phaedon Stelios Koutsourelakis. probabilistic generative model for semi- supervised training of coarse coarse-grained surrogates and enforcing physical constraints through virtual observables. Journal of Computational Physics, 434, 2021. ISSN 10902716. doi: 10.1016/jjjep. 2021.110218.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- Resolution Image Synthesis with Latent Diffusion Models. 2021. URL
abs/2112. 10752
Olaf Ronneberger, Philipp Fischer, and Thomas Brox U-Net: Convolutional Networs Image Segmentation. 2015. URL htt.::pp//aart...or//a///////// 04597
Networks for Biomedical
Tim Salimans and Jonathan Ho. Dioirrlli for fast sampling of diffusion models, 2022.
URL hww.appps//arrv..orr////////22..
Sebastian Sanokowski, Sepp Hochreiter, and Sebastian Lehner. A diffusion model framework for unsupervised neural combinatorial optimization, 2024. URL https://arxiv..rt
2406. 01661
Mohammed Sardar, Alex Skillen, Malgorzata J. Zimoñ, Samuel Draycott, and Alistair Revell. Spectrally Decomposed Diffusion Models for Generative Turbulence Recovery. 2023. URL
http:://rxiv wro.org//aaao www.or/g////////..//.o../.aaeeee
aww./bss...... 2212. 231.2150299
Dule Shu, Zijie Li, and Amir Barati Farimani. A fidelity flow field reconstruction. Journal of Computational Physics, 11972, 2023. ISSN 10902716. doi: 2023.11197 111972
diffusion model for high-
13

---

Published as a conference paper at ICLR 2025
Jascha Johl-Dicksteiis Eric A. Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep Unsupervised Learning using Nonequilibrium Thermodynamics. ICML, 2015. URL
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising Diffusion Implicit Models. In
International Conference on Learning Representations, 2021a. URL httt:tt://ppeereeiiiie
net/ 20ne/Eorrmm11o-s1lgiirree11
Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and
Ben Poole. Score-Based Generative Modeling through Stochastic Differential Equations. In International Conference on Learning Representations, 2021b. URL httts::/opennrev////
Yang Song, Prafulla Dhariwal, Mark Chen, and Ilya Sutskever. Consistency Models. 2023. URL http://arxiv org/ abs//ab3././3311/
Francisco Vargas, Will Sussman Grathwohl. and Arnad Doucet. Denoising Diffusion Samplers.
In The Eleventh International Conference on Learning Representations, 2023. URL https
7 //openreview.o net. wtt.:mmmm--pp/
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention Is All You Need. 2017. URL http::/arxiv..rr
abs/1708 03762
Pascal Vincent. A Connection Between Score Matching and Denoising Autoencoders. Neural Computation, 23(7)1661-1674 2011. ISSN 0899-7667. doi: 10. 10.1162/NFCO a a_1122 URL https://di
E. edu http:t/eeweeeeeeaaaa..///////6/.
Pauli Virtanen, Ralf Gommers. Travis E. Oliphant, Matt Haberland. Tyler Reddy, David Couraaeeau, Evgeni Burovski, Pearu Peterson, Warren Weckesser, Jonathan Bright, Stéfan J. van der Walt, Matthew Brett, Joshua Wilson, K. Jarrod Millman, Nikolay Mayorov, Andrew R. J. Nelson, Eric Jones, Robert Kern, Eric Larson, C Carey, ilhan Polat, Yu Feng, Eric W. Moore, Jake VanderPlas, Denis Laxalde, Josef Perktold, Robert Cimrman, Ian Henriksen, E. Quintero, Charles R. Harris, Anne M. Archibald, António H. Ribeiro, Fabian Pedregosa, Paul van Mulbregt, Aditya Vijaykumar, Alessandro Pietro Bardelli, Alex Rothberg, Andreas Hilboll, Andreas Kloeckner, Anthony Scopatz, Antony Lee, Ariel Rokem, C. Nathan Woods, Chad Fulton, Charles Masson, Christian Häggström, Clark Fitzgerald, David A. Nicholson, David R. Hagen, Dmitrii V. Pasechnik, Emanuele Olivetti, Eric Martin, Eric Wieser, Fabrice Silva, Felix Lenders, Florian Wilhelm, G. Young, Gavin A. Price, Gert-Ludwig Ingold, Gregory F. Allen, Gregory R. Lee, Hervé Audren, Irvin Probst, Jörg P. Dietrich. Jacob Silterra, James T Webber, Janko Slavič, Joel Nothman, Johannes Buchner, Johannes Kulick, Johannes L. Schönberger, José Vinícius de Miranda Cardoso, Joscha Reimer, Joseph Harrington, Juan Luis Cano Rodríguez, Juan Nunez-Iglesias, Justin Kuczynski, Kevin Tritz, Martin Thoma, Matthew Newville, Matthias Kümmerer, Maximilian Bolingbroke, Michael Tartre, Mikhail Pak, Nathaniel J. Smith, Nikolai Nowaczyk, Nikolay Shebanov, Oleksandr Pavlyk, Per A. Brodtkorb, Perry Lee, Robert T. McGibbon, Roman Feldbauer, Sam Lewis, Sam Tygier, Scott Sievert, Sebastiano Vigna, Stefan Peterson, Surhud More, Tadeusz Pudlik, Takuya Oshima, Thomas J. Pingel, Thomas P. Robitaille, Thomas Spura. Thouis R. Jones, Tim Cera. Tim Leslie, Tiziano Zito, Tom Krauss, Utkarsh Upadhyay, Yaroslav O. Halchenko, and Yoshiki Vázquez- Baeza. SciPy 1.0: fundamental algorithms for scientific computing in Python. Nature Methods, doi: 10. 00.10//44199220019 016-----2.. URL https
17(3):261-27)) 2020. ISSN 1548-7091.
7/www. nature www.e///ettttttteceoeecoooeeeeee
Nikolaos N. Vlassis and WaiChing Sun. Denoising diffusion algorithm for inverse design of mi- crostructures with fine-tuned nonlinear material properties. Computer Methods in Applied Mechan- ics and Engineering. 413:1 16126, 2023. ISSN 00457825. doi: 10. 601 10..0006j.cmmaa2233.... 2023.....2222222222 URL seever...... com httpp///eeeeeeee//////0055533330
https:://inkingget
Phil Wang. Implementation of Imagen. Google's Text-to- Neural Network that beats DALL-E2, in Pytorch, 2022. URL /ttttts:///gtt/uiit com/ http/lucidiriin wwwgeiaanni... pyttrch
Sifan Wang, Yujun Teng. and Paris Perdikaris. Understanding and Mitigating Gradient Flow Pathologies in Physics-Informer Neural Networks. SIAM Journal on Scientific Computing,
14

---

Published as a conference paper at ICLR 2025
43(5):43055-A30881 2021. ISSN 1064-8275. doi: 10.1137/20M1318443.. URL https
//epubs. siam.
Tsun-Hsuan Wang, Juntian Zheng, Pingchuan Ma, Yilun Du, Byungchul Kim, Andrew Everett Spielberg, Joshua B. Tenenbaum, Chuang Gan, and Daniela Rus. Diffusebot: Breeding soft robots with physics-augmented generative diffusion models. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL https: http:s////ppereview.eeeet id id-1zo411oUEs
Tong Wu, Zhihao Fan, Xiao Liu, Hai-Tao Zheng, Yeyun Gong, yelong shen, Jian Jiao, Juntao Li, zhongyu wei, Jian Guo, Nan Duan, and Weizhu Chen. AR-diffusion: Auto-regressive diffusion model for text generation. In Shirty-eevetth Conference on Neural Information Processing Systems, 2023. URL
Yuxin Wu and Kaiming He. Group Normalization. 2018. URL http://arxiv.orr/
1803.08497
Tian Xie, Xiang Fu, Octavian-Eugen Ganea, Regina Barzilay, and Tommi Jaakkola. Crystal diffu- sion variational autoencoder for periodic material generation. In International Conference on Learn- ing Representations, 2022.
Minkai Xu, Lantao Yu, Yang Song, Chence Shi, Stefano Ermon, and Jian Tang. GeoDiff: A geometric
diffusion model for molecular conformation generation. In International Conference Learning Representations, 2022. URL /Rtppttss///ppeereee//
net/ det.rummm2d-2222EMM22.
Ling Yang. Zhilong Zhang, Yang Song, Shenda Hong, Runsheng Xu, Yue Zhao, Wentao Zhang, Bin Cui, and Ming-Hsuan Yang. Diffusion Models: A Comprchensive Survey of Methods and Applications. ACM Computing Surveys, 5-(4): 2024. ISSN 0360-0300. doi: 10.1 10.1645/3626235. URL htt.::/::/...aa...ggg///////////
acm. http::////1.111455536666665
Ye Yuan, Jiaming Song, Umar Igbal, Arash Vahdat, and Jan Kautz. PhysDiff: Physics-Guides Human Motion Diffusion Model. 2022. URL /tt::/t/aarr 22112. 02500
Qinsheng Zhang and Yongxin Chen. Path Integral Sampler: Stochastic Control Approach For
Sampling. In International Conference on Learning Representations, 2022. URL https //openreview.o net. www.:nnnmttoorttttt...... uCb2ynRubbyyyyy
Qinsheng Zhang and Yongxin Chen. Fast sampling of diffusion models with exponential integrator. In The Fleventh International Conference on Learning Representations, 2023. URL [
/ooenreviee net./ wowummidd=ooooooooooo fb
Yinhao Zhu and Nicholas Zabaras. Bayesian deep convolutional encoder- networks for surrogate modeling and uncertainty quantification. Journal of Computational Physics, 366:415- 447, 2018. ISSN 19911 doi: 10. 10011888..0....008.....0..000 URL lsevi com ret wtw.:/eetriivveeeeee /ttp://www////110000099999911119
15

---

Published as a conference paper at ICLR 2025
A APPENDIX
A.1 CONSISTENCY
We here give a theoretical argument for the fact that, in an ideal setting, diffusion models trained via score-matee matching with an additional virtual (residual) likelihood term continue to recover the true distribution. Adopting the perspective of score-based models (Song et al., 2021b), we consider a continuous time variable / [0, T) instead of the discrete setting. The forward diffusion process of some data æ E IR_ can then generally be described by the stochastic differential equation (SDE)
dæ
F, + d,
(15)
where w is a Wiener process, and the drift and diffusion coefficients F₂ E RDxD and G. E DDxD respectively, are chosen so that the transition kernel is a simple Gaussian that can be computed in closed form. Here, the popular diffusion variants proposed by |Ho et al. (2020) and DDIM (Song
et al. b02 correspond to their continuous counterparts F. 2
d log and G. = /
d log I,
where Ot is continuously decreasing from og aT 0. There is no unique reverse diffusion process, but common choices of the corresponding reverse SDE are included in the parameterization proposed by [Zhang Chen 2023),
X2 G.G. (og)ia)))) di + dG,
dæ = F5x: 2
(16)
where tn is a Wiener process that runs, as dī, backwards in time, and À 2 O. If the score V log (a) is known, we can generate new samples by sampling from the known prior distribution p (T) and applying equation 16 DDPM (Ho et al. 2020) and (deterministic) DDIM (Song et al. 202 then and À = O, respectively) and discretizations of equation 16 (Song al. 2021b] Zhang & Chen 2023). We can show that straightforward extension of the usual score-matching objective with the virtual residual likelihood is consistent as it continues recover the data distribution:
correspond to certain parameterizations (A
Proposition 1. (Consistency) Let p(10)) be a distribution with samples 3o ~ p(ar) satisfying some constraint R (#r)) = O. Consider
Sopt = arg min Unif ppteo) (wa) [A(t) IV logg(zz/ s((,, t)ll
log or (x(2
(17)
where A(t) > is a time-dependent weight and vū is obtained by solving the reverse SDE equation T6 initiated at with score s(x,, t). Then, solving the reverse SDE equation from ar ~ p(ET)) with Supt as the score for zn corresponds to sampling from p(æo)-
Proof. It is well-known that score matching V log (ax (z)) is equivalent to matching V log p(x:) (Vincent 2011). Assuming perfect recovery of the score, i.e., s(x, t) V log p(æ,) for all æ,, t. the marginal distribution p" (ax) of equation 16 matches the forward diffusion p(æ:) for all 0 S ( < T independent of when p(xT) = p" (xT) (Zhang & Chen 2023, Prop. 1). To generate new samples, may start from any latent 2; ~ p'(x;) which can hence equivalently be obtained via the forward marginal De ~ p(æz), and solve equation for vy ~ p(æo). As we consider virtual observables e introduced via qR (f) (flazi) (r; R R(x)), o² T) we have
- -log gr (f(=g)
2 13(
(18)
where C' is constant that does not depend on zi (and hence s). Since x p(xg) satisfies R(xj))
O by assumption, the optimal score model is also minimizer of the (negative) virtual ligeliieldhood
equation 18 and Sopt recovers the optimal score. Hence, solving equation 16 from oT ~ p(æT) for Zo with Sopt as the score generates samples from the true distribution 2o p(xo).
Remark. In practice, equation 16 has to be discretized, since no closed-form solution is available. Hence, even for perfect score model, this numerical approximation will introduce bias (Zhang & Chen 2023). Also, note that we are free to choose DDPM (Ho et al., 2020) or DDIM (Song et al. 202 discretize equation since the optimal score is unaffected by this choice.
16

---

Published as a conference paper at ICLR 2025
Lastly, emphasize that minimizing the residual error some intermediate latent variables introduces inconsistencies. Assume some general forward conditional marginal of the form p(a, kn) (xx; agZp, of from which we sample via ay ofe, where 6 (0, r). But general, Ro + ofc) * 0; thus enforcing R(x)) = does not align with the underlying marginal distribution.
A.2 MEAN ESTIMATION FOR SCORE- BBSEDD MODELS
As summarized in Appendix A. score-based models are an alternative perspective on diffusion models that may be understood their continuous-tine generalization (Song et al. 2021b). Here, the model learns the data score (zt, t) z V log Pe (e with which we can solve the reverse SDE equation I6 to generate new samples. For forward conditional marginal of form p(2;( N (xx;; Tg o I), we can obtain Eg = E( æ as (see e.g., Kingma et al. (2021))
1 at
@ + of (o),
(19)
based on which the virtual residual likelihood can be estimated.
A.3 DETAILS ON THE SIMPLIFIED TRAINING OBJECTIVE
As shown in Section 3] we arrive at an optimization objective equation 12 that requires sampling over latents from the learned distribution #1:7 po(21:r) We simplify this by instead sampling from the available (()))), which can be understood as ignoring the likelihood ratio, as we aim to minimize
E (((
log po (xx)) + E (r)pp((r:))) [- log qr (+
0 و (x
log g(21:1 (00)
+ E B (m
| log (r = 0/20 (x1r)))
Pe (wo:r)
log 2( 1 (m0)
po(
+ E (
log ( = 0|2
Po (wo:r)
((x1:T)
log (xx1rr (20))
~ Bx((morr)
+ E P
log qre (f (xx)())))))
Po (Xo:r)
log (001rr (oo)
= =Bq(morr)
log qR (f 0fz7((1))))
Po (Woor)
We justify this simplification by the assumption that optimizing the variational bound will bring po( close ( and thus reduce this bias with ongoing model training. Evaluating the final expression gives us the presented physics-informed diffusion model loss equation [4
A.4 DDIM SAMPLING
For completeness, we here provide the deterministic DDIM sampling scheme derived by Song et al. (222aa) in terms of do:
1 ar. ār:
FT.-1 = fo 20
(I.,, - Va, to (II,,
(20)
Generally, T is a sub-sequence of the full denoising sequence [1....,]] T). During training, æ, is available and we aim to estimate in a given number of reduced timesteps, considering sequence As we empirically found no notable improvement by providing multiple intermediate
7 = .I....,f,, f). timesteps, but this comes with the cost of the additional forward passes, we set T [1, f]. Hence, we sample I1 from I; in one step via equation 20 and then obtain 2g by 20(211 t = 1). resulting two forward passes of the model. Note that reduce the impact of worse estimates at noisier timesteps by the increased variance the virtual residual likelihood, penalizing deviations from R (zn) less.
17

---

Published as a conference paper at ICLR 2025
A.5
OTHER CONSTRAINT TYPES AND AUXILIARY OPTIMIZATION OBJECTIVES
A.5.1
INEQUALITY CONSTRAINTS
Inequality constraints also common in physics (e.g., the second law of thermodynamics). Consid- eration these constraints of type
h((ar)) S hmux
(21)
can be introduced via Rine = ReLU(h(zo)) mmmaa) = max (0, 1 (())) /mmaa))). which we analogously introduce into the variational loss the Gaussian (Riny l'ie 2njl (I2, 1)) (fineq; Rimey (z ///c) with Finey O.
A.5.2 OPTIMIZATION OBJECTIVES
Optimization objectives of the form
min T
(22)
can also be considered by treating the optimum as a pseudo-observable jupi and similar strategy as before. In general, however, this optimum js-unlike F and fimeq typically unknown. As a remedy.
we may extend the mismatch of the actual and optimal objective by a pseudo-observable foo - and pose this as sample from the exponential distribution, as shown by Rixner & Koutsourelakis, (202))
I (xx)) Jopt + Paps Expon (A),
(23)
where
Expon (z; Ae-
(24)
<<0 0
By similar reasoning as before, we introduce
4 (f
II au (Pop (z)(
(25)
[=1
where (fopl ()) Ae (()
and observe that the log-likelihood
log 9. (fop (fo (2) (21:T))
logg() XJ (2)( + Aap
(26)
decouples Jope from zn due to the properties of the exponential distribution. Therefore, knowledge of Jopt is not required for training. The loss function hence extended
LPIDM-opi ( = E, we [Aollzo - Lo t)
1 25,
[ t))ll + AJ AT (27)
Note that we could also couple the parameter À of the exponential distribution to the denoising timestep (similar to E,), but choose to keep it constant here. As usual in multi-objective optimization, trade-offs between the different loss contributions expected, depending the relative weighting (Wang et al. 2021).
A.6 NUMERICAL STUDIES
For the presented evaluations, we introduce the mean absolute residual error as follows
M
N
RMAE (xo)
Σ
[x)]
+ Σ
By
[zo]]
(28)
M + N
where and j denote the i-th PDE and j-th boundary condition constraint, and we consider a total of M and N such constraints, respectively.
18

---

Published as a conference paper at ICLR 2025
A.6.1 TOY PROBLEM
Setup. In simple, instructive example, we demonstrate the implications of the proposed physics- informed loss function equation both considering the mean and sample estimation for the residual evaluation. The objective
learn a distribution 9(x0) that samples points uniformly on the unit circle S = { E R || = J. so that samples should obey a simple algebraic equality constraint. Thus, generated samples of the model 2g directly correspond to the two spatial coordinates &2). 0.1 and sample estimation, respectively, and thus increased the variance of the implied virtual likelihood.
We here identified
0.005 via simple hyperparameter sweep for the mean and
We choose simple 3-layer MLP of latent dimension 128 with 2D vector (E1, {2) as in- and output. Information about the diffusion timestep f added by transforming to an embedding which is element-wist multiplied by the output of the linear layer, generally followed by the Softplus activation except for the last layer. We train the model for 400 epochs on 10,000 randomly sampled points of the unit circle, using the Adam optimizer (Kingma & Ba 2014) with learning rate of 5 x 10 We use batch size of 128, and 100 diffusion timesteps with a cosine scheduler (Dhariwal & Nichol (20T). The training time of each considered model variant takes around 12 minutes on an Nvidia Quadro RTX 6000 GPU (equipped with 24GB GDDR6 memory).
Results. To understand the impact of the physics-infee informed loss, we train the proposed PIDM either via mean or sample estimation in four different scenarios: (i) via the classical setup of the variational bound on the data likelihood equation (ii) via the proposed PIDM loss equation 14] (iii) by only considering the residual loss (and neglecting the variational bound), and (iv) by again considering the PIDM loss equation 14 but with data sampled from an uninformative prior Zo ~ (0, I). We present the average residual error of 100 samples generated over 400 training epochs and plot 100 generated samples after training in Figure 5 and Figure 6 for the mean and sample estimation, respectively. More specifically, provide the mean absolute residual error evaluated as RMAE (æx) |||E|1 1/. which is averaged over all samples.
(b) Generated samples an ~ po(w))
(a) Average residual error
RMA (wo)), ao ~v Po (220)
10
(ii)
(iv)
10-1
0
10-3
100
200
300
400
Training iteration
E1
Figure 5: (Mean estimation) Evaluation of the average residual error of 100 generated samples during training (a, averaged over 10 training runs with 25/75% -uuanillss using different seeds) and 100 generated samples after training (b, from representative models) in four different settings. We consider a diffusion model trained with the standard (data-driven) objective (i), our proposed PIDM (ii), model trained solely on the residual loss term (iii), and again our proposed PIDM but with data sampled from an uninformative Gaussian prior (iv). The residual during training is evaluated via mean estimation, i.e., 2 || In (b), we indicate the unit circle to which all samples should be constrained. Colors (b) match those in (a).
We observe that the studies for the mean and sample estimation present similar picture, and we may summarize the findings independently of the considered estimation mechanism as follows. As indicated in Figure 5h and 64, we observe that the proposed physics-ing physics-informed loss (ii) indeed outperforms the standard setup (1) in terms of the residual error after approximately 100- 120 training epochs. This is also visually confirmed in Figure where the samples from the "constraint-infomee
19

---

Published as a conference paper at ICLR 2025
(b) Generated samples 2o po(
(a)
Average residual error
RMAE (wo) ao ~v po (2o)
10
1
(ii)
(iv)
10-1
0
10-3
100
200
300
400
Training iteration
&
Figure 6: (Sample estimation) The setting is identical to the one provided in Figure 5 but we here
estimate zi via DDIM (Song et al.
2021a
e 0 MRMAE 0.080)
C 0.005 (RMAE = 0.061)
c= 0.02 (RMAE = 0.048)
0
S
S
0
1
{
= 0.04 (RMAE = 0.013)
C == (RMAE 0.0037)
= 5 (RMAE = 0.0024)
S
0
0
-1
E1
E1
Figure 7: Evaluation of 400 generated samples after training proposed PIDM with sample estimation the same setting as Figure but with varying scale factors c. We also indicate the mean absolute residual error RMAE, averaged over all samples and the unit circle which all samples should be constrained.
model align more accurately on the unit circle (see also the top inset). When training the model solely on the constraints (iii,, the model efficiently reduces the residual. However, it converges to single point randomly located somewhere on S, shown in the bottom inset. This is indeed expected: the model finds no penalty for collapsing to any points on S₁, and similar results are generally observable if the residual penalty vastly exceeds the data loss term. Interestingly, the model can also approximate the target distribution even in the absence of training data using only an uninformative prior (iv), though here at reduced accuracy. We do not explore this idea further, but
20

---

Published as a conference paper at ICLR 2025
this suggests the of uninformative prior as regularization that prompts the model to explore a wider range of solutions within the constraint space. Although the simple setting of this study prevents straightforward analogies to more complex setups, it confirms the validity of our proposed framework and its ability to enforce constraints.
Study on the influence of relative weighting. To systematically examine the effect of the scale factor e that dictates the importance the model assigns to the residual loss, we provide results for six different values Figure The experimental setup remains consistent with the previous configuration for the sample estimation, except for the variation in c. As standard (data-driven) diffusion model. For increasing c, the residual error is increasingly reduced, but the model eventually generates samples increasingly concentrated at point randomly located on the constraint manifold.
O, recover the
As 1, the distribution fully collapses the model ignores the data loss, and the behavior resembles that of model trained exclusively on the residual loss (similar to scenario (iii) in Figure For moderate values, such 0.005, the distribution achieves optimal balance: maintaining full diversity while effectively minimizing the residual error.
A.6.2 DETAILS OF THE DARCY FLOW STUDY
Background. The Darcy flow equations describe the steady-state solution of fluid flow through porous medium. Given a permeability field K(E), the pressure distribution p({) and velocity field (E() are governed by
u(E) = - K(E)) E
V u(E) S(E), EEΩ
(E) in(E) = 0, EE
(29)
J P(E) O,
where ñ denotes the outward unit vector normal the boundary. Similar contemporary work Zhu & Zabaras 2018 Jacobsen et al. 2024), we consider a 2D square domain and set the source function
if E 1 < 2 w for i = 1,2
T -r,
f(E) =
if otherwise,
f 1 1+ _ 2 for 1,2
(30)
0,
with r = 10 and w = 0.125. We sample ((E) from a Gaussian random field (GRF), i.e.
K()) exp()(E)),
~ (N))~((O k(-)))
(31)
with covariance
1: (E, exp (- E E /l), with 1= 0.1.
(32)
Instead of directly considering the GRF, we reduce its dimensionality by considering its Karhunen- Loève expansion up to s = 64 terms,
G(E) (E)
(33)
with >; and (5(E) as, respectively, the eigenvalues and eigenfunctions of equation 32 sorted by decreasing X and 2 (O, 1). Discretization of equation 29 is achieved via second-order central finite differences, and we refer to Jacobsen et al. (2024) for details. To incorporate the boundary conditions and the integral constraint, we extend the linear system of type Ap = by additionally imposing the above constraints, so that A IR 4n +)xn" p R" and f E =m==+4n+1 (where = 64). We solve for the over-determined pressure field using the scipy www.iniinnns....... 1st [Virtanen al. 2020) solver with default settings.
Model architecture and residual computation. The considered U-Net (Ronneberger al. 2015)
architecture is based on Wang (2022), given its demonstrated success in learning the denoising process (Dhariwal & Nichol 2021). Our configuration uses image input resolution of 64 x 64
21

---

Published as a conference paper at ICLR 2025
pixels, aligning with the grid resolution of the linear system under consideration. Importantly, require the residual evaluation of the predicted images R(x)), as required in equation [4] to be consistent with the data, as otherwise the optimal data likelihood is in partial conflict with the (virtual) residual likelihood. This is ensured by using the same finite difference stencils as in the dataset creation, essentially reassembling f. except that remove the integral constraint, since it can be trivially fulfilled by shifting the predicted pressure field (Jacobsen et al.) (2024). Finite difference stencils are implemented via t torch. nn Conv2D Passze al. 2019) with a custom kernel, which we can precompute for stencils up to arbitrary order via findi ff (Baer 2018).
We here restrict ourselves to an unconditional model, though extensions to conditional generation (Ho & Salimans 2022) are straightforward. The U-Net has two in- and output channels and is trained to predict the clean signal based on a noisy input, as stated in Section 2.1 Thus, the model trained to generate pairs (K, 2) where K is sampled similar to equation 31 and is the corresponding (unique) pressure field that satisfies the Darcy flow equations equation
29 The residual then
assembled by considering F F(za] == V (KVp) + f O. Further details of the model architecture
and training hyperparameters are given in Appendix A.7 and the code.
Comparison with other frameworks. Note that our architecture of the chysics-guides diffusion model (ii) is not exact replication of the one given by (Shu al. (2023) due to the different setting, but the proposed conditioning mechanism is closely mimicked, as detailed in the code. Jacobsen al. (2024) presented a strategy iteratively "correct" the latent variables S, and samples wo during inference by upplying, gradient gradient descent based on the PDE residuals (iii). We first followed the optimal setting proposed in Jacobsen et al. (2024) and applied gradient- based descent Va. | (x(x)) (xx)]13 with = 2 10 +/ max V z. R(x,) for the last N steps of the sampling iterations and M additional iterations. We set M 25 and N = 50 according to the best-reported results (equal to starting corrections halfway through the sampling). However, we encountered stability issues for the above value of e, likely due to differences the sampling scheme and fewer considered timesteps compared JaJobsennn et al. (2024), who adopted a score-based perspective. We therefore conducted parameter sweep to identify the converging results with the best performance, which we identified at e = x 10- 6/ max Va, (xx). While additional sweeps might yield some further incremental improvement, we observed that different values of e yielded similar results, as long as the updates converged.
A.6.3 DETAILS OF THE TOPOLOGY OPTIMIZATION STUDY
Background. Topology optimization aims to identify structure with optimal mechanical prop- erties, typically optimal mechanical stiffness. This can be formalized as the minimization of the mechanical compliance C under of equality constraints (given by mechanical equilibrium and boundary conditions) and inequality constraints (typically a volume constraint). Under the assumptions of linear elasticity, the problem reads
min p(E)
(() dΩ, & e Ω 2
C P(E) E [0, 1].
subject to:
(34)
/ ((E) du < Vmux:
9
V (()) f(E) = 0.
The last equation implies quasistatic mechanical cquilibrium. Here, and or denote the strain and stress tensor fields, respectively, f is distributed body force and Vmnx a given (maximum) volume constraint. We consider a linear elastic material, which couples e and or via Hooke's law, a CC= where C' is the fourth-order stiffness tensor, and the colon denotes double tensor contraction. For simplicity, we may assume isotropic material, C' is characterized by two material constants (e.g.. Young's 's modulus E and Poisson's 's ratio 1/). The solution field is given in terms of the displacements (E), from which the strain tensor follows are applied as u = à on the boundary 02, C Ω and traction boundary conditions or á = on an, C 22 where vi denotes the outward unit vector normal to the boundary C Ω
Vu T]. Dirichlet boundary conditions
22

---

Published as a conference paper at ICLR 2025
In practice, equation 34 is usually discretized via finite elements. We here consider regular finite element mesh, based on a 65 x 65 grid of nodes with (64 x 64) four-node quadrilateral elements under plane-stress assumptions (with E' 1, 0.3). This turns the mechanical equilibrium equation into a linear system of type KU = F, where K E I22n is the global stiffness matrix and U E IR2" are the global nodal displacement and the external force vectors, respectively, with 65. Dirichlet boundary conditions are imposed by appropriately modifying K and F. Optimized topologies can subsequently be obtained via the Solid Isotropic Material with Penalization (SIMP) method (we refer to Bendsøe & Sigmund (2004) for details). SIMP considers continuous densities in equation 34 to allow for gradient- based optimization, defining C(E) = P(E)²CO, where > 1 promotes binary entries (corresponding to material placement, P 1. or void. p - 0) and C denotes the material properties of the given isotropic base material. As SIMP often requires many costly finite element analyses iteratively refine the solution and may get stuck in local minima, deep learning frameworks including generative adversarial networks (Nie et al. (021) and diffusion models (Mazé Ahmed 2023 Giannone et al., 2023) have been explored as alternatives to mitigate some of these challenges.
Model architecture and residual computation. We consider the same U-Net architecture as in Section [4. exxept for the following differences. Similar to Nie et al. (202T). we also provide the von Mises stress and strain energy fields of the unoptimized domain, as well as the boundary conditions (including the loads) and volume fraction allow for conditioning in addition to the noisy signal of the solution fields (consisting of the two displacement and density fields) to the model. The model aims to reconstruct the clean signal of the two displacement fields 112 and the optimal density p. Besides, we increase the latent dimension of the U-Net to 128 to have similar number of overall parameters compared to previous work (Mazé Ahmed 2023; Giannone et al. (023). Lastly, we
apply sigmoid activation after the density field channel output to ensure E [0, 1I. The predicted p(E) then enters the governing equations via CE) = C())
For the residual and compliance computation, using finite differences as in Section 4.1 to assemble equation 34 introduces inconsistencies with the FEM solution (i.e., training data with non-zero residuals), likely due to the sharp transitions in the density field and point-wise introduction of loads. This is problematic, as the minimization of the residual then does not fully correspond to samples from the data distribution, leading to optimization conflict. We hence treat the pixels direct representations of the FE grid and apply the stiffness matrix K to evaluate the residual. As we consider U-Net with in- and output pixel dimensions of 64 x 64, we apply bilinear interpolation to down- or upscale all nodal quantities to a 65 x 65 grid where necessary. Note that we can obtain the global stiffness matrix efficiently by precomputing the local stiffness matrix (up to factor depending on the corresponding density p(E) at the element level) given, e.g., by Soli colidsPy (Gómez & Guarín Zapata 2018) and vectorizing the global assembly. Further details of the model architecture and training hyperparameters are given in Appendix A.7 and the code.
Evaluation. We present metrics consistent with Mazé & Ahmed 2023): Giannone et al. (003)), namely, the compliance error is introduced CE = (C(zo) C( 2o (IMM) / C(Z0, and the volume fraction error as VFE = p ( - Ptarget| / /Paargat where f is the (binarized) density averaged over the domain. All models use 100 denoising steps generate a sample. We note that the models provided by Mazé & Ahmed (2023): Giannone et al. (003) are trained for only 200k iterations but with a 16 times larger batch size (64). Besides, these frameworks require additional overhead due to the generation of auxiliary data and training of potential surrogate models (Mazé & Ahmed 2023). For the comparison with PG-Diffusion (Shu et al. 2023) and CoCoGen (Jacobsen et al. [2024) we included both the mechanical equilibrium equations and the volume constraint in the residual. For CoCoGen, computing the full Jacobian Vz, (x,) (here, R(x,) is vector-valued, which the authors use to scale the residual exceeds the VRAM of our GPU (even with a batch size of I and efficient implementation via torch. func. . bacfwd) We thus considered a constant scaling factor, conducted extensive hyperparameter study, and report the best results (in terms of the median residual on the in-distribution test set).
23

---

Published as a conference paper at ICLR 2025
A.7 U-NET ARCHITECTURE AND TRAINING DETAILS
As described in Section 4.1 and 4.2 we consider U-Net-based architecture (Ronneberger et al.
2015). The main model and training hyperparameters are summarized in Tables 2 respectively. The model is implemented and trained using Py PyTorch (Paszke et al. 2019). Further details can be found in the code.
For the Darey flow study, training for 300k iterations took approximately 13 hours for the standard diffusion setup, 16 hours for the PIDM with mean estimation, and 22 hours for the PIDM with sample estimation. For the topology optimization study, training took approximately 48 hours for the standard diffusion setup and 54 hours for the PIDM (with sample estimation). All models were trained on single Nvidía Quadro RTX Quadro RTX 6000 GPU equipped with 24GB GDDR6 memory.
Table 2: Denoising diffusion architecture hyperparameters.
Hyperparameter
Value
In-, output channels (Darcy flow)
2, 10, 3
output channels (Topology optimization)
ResNet blocks per down- and upsampling pass
ResNet block normalization
Group Normalization (W Hc 2018
ResNet block activation function
SiLU (Elfwing Layer Normalization [Ba at. 2016 64 x 64 + 32 x 32
2018
Attention block normalization
Feature map resolutions (downsampling pass)
16
48 x
Latent dimensions feature maps, Darcy flow)
32 + + 128 + 256
Latent dimensions feature maps, Top, opt.)
128 + 512 1024 32
Attention /Vaswani et 2017 Katharopoulos
2020
head dimension
Number of attention heads
Table 3: Denoising diffusion process and training hyperparameters.
Hyperparameter
Value
Number of diffusion timesteps
100
3 B--scheduler
Cosine schedule Dhariwal & Nichol 2021
Batch size (Darcy flow), mean estimation
64
Batch size (Darcy flow), sample estimation
16
Batch size (Topology optimization, sample estimation)
4
Iterations (Darcy flow)
300k
Iterations (Topology optimization)
600k
Learning rate
10 4
Optimization algorithm
Adam Kingma & Ba. 2014
EMA start (iteration)
1,000
Exponential Moving Average (EMA) decay
0.99
Dropout
none
24

---

Published as a conference paper at ICLR 2025
A.8
ADDITIONAL SAMPLES
A.8.1
DARCY FLOW
(a)
Permeability K
Pressure
Residual RAAAE (K (Kp)
0.10
10-
5
10-2
0.00
-0.05
0
10-3
E1
51
(b)
Permeability K
Pressure
Residual RAAR (Kpp) p)
10
0.0
10-2
0.1
10-3
E1
(c)
Permeability K
Pressure
Residual RMAE K p)
109
0.05
10-1
S
0.00
10-2
-0.05
10-3
(d)
Permeability
Pressure
Residual MAEE (, p)
109
0.03
0.02
0.01
10
-0.01
0
10
E1
E1
Figure 8: Additional samples of permeability and pressure fields well as the corresponding residual error from the proposed PIDM trained on the Darey flow dataset. Sample (a) and (b) are sampled from PIDM with mean estimation, while (c) and (d) sampled from a PIDM with sample (DDIM) estimation.
25

---

Published as a conference paper at ICLR 2025
A.8.2
TOPOLOGY OPTIMIZATION
(a)
Design
Residual RMAF (p, u. U2)
SIMP Design
(CE
6.05%, p 0.36)
13.17, Vmnx 0.36)
10
0.8
0.8
0.0
0.6
10-2
0
10-3
(b)
Design
Residual RM ti U22
SIMP Design P
(CE 34.84% 0.44)
(C 9.82, Vina 0.44)
1.0
100
0.8
0.8
10-1
4
10-2
0.2
0.2
0
10-3
(c)
Design 0.12%.
Residual REAEE uz Uzg
SIMP Design
(CE
0.38)
(C 4.55, V
0.38)
100
0.8
0.6
10-1
0.6
0.4
10-2
10-3
0
(d)
Design
Residual RMAE(p, ii u2)
SIMP Design P
(CE
01.3%% 0 = 0.48)
(C = 3.89, Vaux 0.48)
1.0
10
0.6
0.4
0.1
0.2
0.2
10-3
0
E1
Figure 9: Additional generated designs, including the compliance error CE and volume P. and the residual error (based on the displacement fields, not shown) from the PIDM trained on the SIMP dataset and the corresponding SIMP design, including the compliance C and volume V All samples are conditioned on the out-of-distribution test set. In the SIMP design, we indicate the applied load by blue dot, and the Dirichlet boundary conditions in red.
26


Reviews:
[{"review_id": "O5dMAPezqk", "reviewer": "Reviewer_szYR", "summary": "The paper proposes a method which introduces a virtual observable (Rixner & Koutsourelakis, 2021) to the score-matching denoising diffusion training objective (Song et al., 2021). \nThe authors provide a theoretical argument (see Appendix A.1) showing that, in ideal settings, this enables the diffusion model to recover the true distribution while enforcing physical constraints. \nAdditionally, the authors theoretically motivate a simplification of the augmented objective (see Appendix A.3), ignoring the likelihood ratio, and justify this by assuming that ongoing optimization will reduce the introduced bias. \nThe authors present experiments on fluid flow (Darcy flow) and structural optimization (topology optimization). \nThese experiments indicate improved reduction in PDE residual compared to state-of-the-art methods and provide evidence that the additional training objective does not compromise data likelihood and may instead serve as an effective regularizer against overfitting.", "questions": "The augmentation of the denoising diffusion model objective to incorporate virtual residuals represents a noteworthy contribution to the field. However, it would enhance clarity and rigor if the authors explicitly positioned this adaptation as their primary theoretical contribution.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. Originality: The paper incorporates previous work on virtual observables (Rixner & Koutsourelakis, 2021) to train denoising diffusion models with embedded physical constraints. In the realm of physics-informed diffusion models, this approach contrasts with methods like PG-Diffusion (Shu et al., 2023) and CoCoGen (Jacobsen et al., 2023), which incorporate physics via auxiliary models or adjustments post-training.\n2. Quality: The paper provides coherent theoretical arguments for the application of virtual observables in diffusion models (see Appendix A.1 and A.3) and implements an experimental setup on benchmark tasks, following configurations from established related work (see Appendix A.6.2 and A.6.3).\n3. Clarity: The paper is clearly structured and well-written.\n4. Significance: The method could enhance generative modeling for scientific applications by producing physically consistent samples, as shown in fluid dynamics and structural optimization benchmarks.", "weaknesses": "1. Clarity: The approach is substantially grounded in prior work on virtual observables (Rixner & Koutsourelakis, 2021), making the contribution more incremental than suggested. \nGreater clarity would be achieved by explicitly framing the contribution as an adaptation of this framework tailored for denoising diffusion models.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "3MbXDQDbEw", "reviewer": "Reviewer_fEo6", "summary": "The paper aims to model random variables using diffusion-based models on data spaces, whose samples follow certain physical constraints described by partial differential equations (PDEs) and boundary conditions. Specifically, these PDE constraints are expressed in the form $\\mathcal{F}(x) = 0$ for $x$ in the domain $\\Omega$, where $\\mathcal{F}$ is a differential operator governing the physical constraints. For example, consider a data space where each data point represents a uniform discretization over a fixed time interval of projectile motion; in this scenario, each data point must comply with the physical laws related to gravity. The paper proposes a regularization objective applied to the generated samples of the diffusion-based models., which does not alter the common training objectives of the models.\n\nTo incorporate the physical constraints, the paper introduces a residual term defined as $R(x) = \\mathcal{F}(x)$. They model the likelihood of the residual given $x$ as $p(r \\mid x) = N(r; R(x), \\sigma^2 I)$, where $N$ denotes the normal distribution and $\\sigma > 0$ is a positive constant. The training process aims to maximize the likelihood $p(r = 0 \\mid x)$, effectively encouraging the residuals to be zero and thus satisfying the physical constraints. With this likelihood function, the paper proposes maximizing the expected log-likelihood of the residuals with respect to the model, in addition to maximizing the evidence lower bound (ELBO) of the diffusion-based models.\n\nSince computing the expected log-likelihood of the residuals with respect to the model is computationally expensive—requiring sampling from the diffusion-based models—the paper proposes sampling $x_0$ for a given $x_t$ instead of generating samples from the prior.\n\nThe paper demonstrates the effectiveness of the proposed method through various experiments.\n\n----\nUpdated the soundness from 3 to 4 and the presentation & contribution from 2 to 3 after the authors' rebuttal.", "questions": "N/A", "limitations": "", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "The paper introduces an interesting approach by modeling random variables using diffusion-based models that adhere to physical constraints defined by PDEs and boundary conditions. This integration of physical laws into the modeling process through a regularization objective is particularly compelling. This novel framework has some potential to positively influence the machine-learning community, especially in domains where physical consistency is essential. \n\nThe demonstrated effectiveness across various experiments underscores the practicality and impact of the proposed method.", "weaknesses": "The paper would benefit from significant improvements in writing clarity. \n\nTo fully understand and summarize the contributions, I had to reference several previous studies, which suggests that the paper is not entirely self-contained. For instance, lines 127-129 appear to cover a key concept, yet there is no explanation provided, nor is the term $R$ properly defined. \n\nAdditionally, there are ambiguities in the notation—for example, it’s unclear if $x$ in Section 2.1 is intended to represent the same variable as $\\xi$ in Equations (5) and (6), raising questions about consistency. \n\nFurthermore, the abstract is overly vague, making it difficult to grasp the paper’s main content and contributions at a glance.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "S157riPjSg", "reviewer": "Reviewer_VMKn", "summary": "The authors proposed a new strategy to generate physical data under the physical constraints by directly combing the training target of diffusion models and the physical PDEs in the loss function. The data used in PDEs are estimated via one-step score-based mean estimation or two-step DDIM sampling. The authors compared their method to other physical-guided machine learning models in the darcy flow generation and topology optimization tasks, which shows their models outperformed others in conforming to the physical constraints but underperformed in other criteria.", "questions": "1.\tWhat advantages does the proposed method have in topological optimization task compared to the traditional methods (SIMP)? For optimization problems, it does not seem to be necessary to generate a distribution. The authors mentioned that their method can also provide the displacement fields. Is it a feature that exclusively can be obtained with their method? Could the authors explain in detail?\n2.\tHow does the sampling step in the sample estimation (DDIM) affect the performance? Will the performance improve with more steps?\n3.\tHow much will the scaling c in the loss function affect the performance? Could the authors show the experiment results of different scaling c?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Compared to previous methods [1][2] that integrates physical constraints by projecting the PDE residuals to the latent space, the authors introduced the PDE residuals as virtual observables and explicitly optimized on the physical constraints.\n\n[1] Christian Jacobsen, Yilin Zhuang, and Karthik Duraisamy. CoCoGen: Physically-Consistent and Conditioned Score-based Generative Models for Forward and Inverse Problems. pp. 1–25, 2023.\n\n[2] Dule Shu, Zijie Li, and Amir Barati Farimani. A physics-informed diffusion model for high-fidelity flow field reconstruction. Journal of Computational Physics, 478:111972, 2023.", "weaknesses": "In their experiment, although their proposed method performs best in satisfying physical constraints, it shows suboptimal performance on other metrics. Moreover, the approach of adding a physical residual term to the loss function is straightforward and offers limited innovation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Nsw0qP2DYs", "reviewer": "Reviewer_oHnR", "summary": "This paper employs Diffusion Models to satisfy constraints, with applications to Partial Differential Equations (PDEs) and Topology Optimization. The diffusion model is trained using a combination of a data-free loss term and a data-driven diffusion loss.", "questions": "1. **Clarification Needed:**  \n   The paper states:  \n   > \"Since the remaining derivations hold true for both estimation techniques, we introduce \\(x_0^*\\) to denote either the mean or the sample estimate on which the residual is evaluated.\"\n\n   The rationale for introducing \\(x_0^*\\) is unclear. What advantage does this provide? Why not directly sample \\(X_0\\) from the model? How is \\(x_0^*\\) computed?\n\nCan you make an ablation on this choice?\n\n2. How does your method perform when the data-based loss term is removed?", "limitations": "", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "The setting of combining a data-based diffusion loss with a data-free diffusion loss in general is an interesting question.", "weaknesses": "1. The core contribution of this paper is the use of Diffusion Models to sample from an unnormalized target distribution. However, there is extensive prior work on this topic (see [1], [2], [3], [4]), which the authors have not cited. The paper's approach includes an additional Physics-Informed Neural Network (PINN) regularization loss, which can be considered a special case of Eq. 6 in Sanokowski et al. [1], where the temperature is set to $0$. The authors of [1] emphasize that temperature regularization, combined with annealing, is crucial for avoiding local minima (see Fig. 2, middle panel, in their paper). Thus, the current approach may perform poorly in the absence of a supervised loss term.\n\n   Additionally, the \"Noise Distribution\" described in Eq. 12 of the paper is essentially the \"Annealed Noise Distribution\" introduced in Sec. 3.2 of [1]. The authors should also cite [5], which discusses training neural networks with PINN losses in a generative manner without using any data.\n\n   The authors could address these concerns by properly citing these works in the related literature. Furthermore, the proposed method might benefit from adopting the loss function used in [1] and incorporating annealing during training. It would be interesting to study the combination of a data-driven loss with a data-free annealing-based loss.\n\n2. Experimental Evaluation:\nThe experimental evaluation is limited and should be expanded to cover a broader range of settings.\n\n### References\n[1] Zhang, Qinsheng, and Yongxin Chen. \"Path Integral Sampler: A Stochastic Control Approach For Sampling.\" International Conference on Learning Representations.\n\n[2] Berner, Julius, Lorenz Richter, and Karen Ullrich. \"An Optimal Control Perspective on Diffusion-Based Generative Modeling.\" Transactions on Machine Learning Research.\n\n[3] Vargas, Francisco, Will Sussman Grathwohl, and Arnaud Doucet. \"Denoising Diffusion Samplers.\" Eleventh International Conference on Learning Representations.\n\n[4] Sanokowski, Sebastian, Sepp Hochreiter, and Sebastian Lehner. \"A Diffusion Model Framework for Unsupervised Neural Combinatorial Optimization.\" Forty-First International Conference on Machine Learning.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)