Generating Physical Dynamics under Priors
Authors: Zihan Zhou
Xiaoxue Wang
Tianshu Yu
Venue: ICLR 2025 Poster (2025)
Keywords: diffusion models
generative models
physical dynamics
priors
Primary area: applications to physical sciences (physics, chemistry, biology, etc.)

Abstract: Generating physically feasible dynamics in a data-driven context is challenging, especially when adhering to physical priors expressed in specific equations or formulas. Existing methodologies often overlook the integration of ''physical priors'', resulting in violation of basic physical laws and suboptimal performance. In this paper, we introduce a novel framework that seamlessly incorporates physical priors into diffusion-based generative models to address this limitation. Our approach leverages two categories of priors: 1) distributional priors, such as roto-translational invariance, and 2) physical feasibility priors, including energy and momentum conservation laws and PDE constraints. By embedding these priors into the generative process, our method can efficiently generate physically realistic dynamics, encompassing trajectories and flows. Empirical evaluations demonstrate that our method produces high-quality dynamics across a diverse array of physical phenomena with remarkable robustness, underscoring its potential to advance data-driven studies in AI4Physics. Our contributions signify a substantial advancement in the field of generative modeling, offering a robust solution to generate accurate and physically consistent dynamics.

Published as a conference paper at ICLR 2025
GENERATING PHYSICAL DYNAMICS UNDER PRIORS
Zihan Zhou', Xiaoxue Wang', Tianshu Yu-+
School of Data Science, The Chinese University of Hong Kong *ChemLex Technology Co., Ltd.
sihanzhoul@link cuhk. ouhk.edu. edu.cn, wxx@chemlex. tech yutianshu@curs cuhk. edu.co cn
ABSTRACT
Generating physically feasible dynamics in a data-driven context is challenging. especially when adhering to physical priors expressed in specific equations formulas. Existing methodologies often overlook the integration of "physical pri- ors", resulting in violation of basic physical laws and suboptimal performance. In this paper, we introduce a novel framework that seamlessly incorporates physi- cal priors into diffusion-baase generative models to address this limitation. Our approach leverages two categories of priors: 1) distributional priors, such as roto- translational invariance, and 2) physical feasibility priors, including energy and momentum conservation laws and PDE constraints. By embedding these priors into the generative process, our method can efficiently generate physically realis- tic dynamics, encompassing trajectories and flows. Empirical evaluations demon- strate that our method produces high-quality dynamies across diverse array of physical phenomena with remarkable robustness, underscoring its potential to ad- vance data-driven studies in AI4Physics. Our contributions signify substantial advancement in the field of generative modeling, offering robust solution to gen- erate accurate and physically consistent dynamics.
INTRODUCTION
The generation of physically feasible dynamics is fundamental challenge in the realm of data- driven modeling and A14Physics. These dynamics, driven by Partial Differential Equations (PDEs), are ubiquitous various scientific and engineering domains, including fluid dynamics (Kutz) 2017).
climate modeling (Rasp et al. 2018), and materials science (Choudhary et al. 2022) Accurately generating such dynamics is crucial for advancing our understanding and predictive capabilities in
these fields (Bzdok & loannidis 2019). Recently, generative models have revolutionized the study of physics by providing powerful tools to simulate and predict complex systems.
Generative v.s. discriminative models. Even when nigh-peeforming discriminative models for dynamics are available such as finite elements (Zhang et al. [201]] Uriirte et al.) (202),, finite differ- ence (Lu et al. 20221: Salman al. 2022), finite volume (Ranade al., [221] physies-informed neural networks (PINNs) (Raissi et al. 2019), generative models are crucial in machine learning for their ability capture the full data distribution, enabling more effective data synthesis (de Oliveira et al. (017). anomaly detection (Finke et al., (021). and semi-supervised learning (Ma et al. (019)). They enhance robustness and interpretability by modeling the joint distribution of data and labels, offering insights into unseen scenarios (Takeishi & Kalousis 2021). Generative models are also piv- otal in creative domains, such as drug discovery (Lavecchia, 2019). where they enable the creation of novel data samples.
Challenge. However, the intrinsic complexity and high-dimensional nature of physical dynamics pose significant challenges for traditional learning systems. Recent advancements in generative modeling, particularly diffusion-based generative models (Song et al. 2020)) have shown promise in capturing complex data distributions. These models iteratively refine noisy samples to match the target distribution, making them well-suited for high-dimensional data generation. Despite their success, existing approaches often overlook the incorporation of "physical priors" expressed in spe-
"corresponding author

---

Published as a conference paper at ICLR 2025
eific equations formulas, which are essential for ensuring that the generated dynamies adhere to fundamental physical laws.
Solution. In this work, propose novel framework that integrates priors into diffusion- generative models to generate physically feasible dynamics. Our approach leverages two types of priors: Distributional priors, including roto-transe noto-translational invariance and equivariance, ensure that models capture the intrinsic properties of the data rather than their specific representations: Physical feasibility priors, including energy and momentum conservation laws and PDE constraints, enforce the adherence to fundamental physical principles, thus improving the quality of generated dynamics.
The integration of priors into the generative process is complex task that necessitates a deep understanding of the relevant mathematical and physical principles. Unlike pre- dictive tasks, where the objective is to estimate a specific ground-truth value zg, diffusion generative models aim to characterize a full ground-truth distribution v log (æ;) or E/wro ær] (notation difference complicates the direct application of priors based on ground-truth values to the output of generative models. In this work, we propose framework to address this chal- lenge by effectively embedding priors within the generative model's output distribution. By incorporating these priors into a diffusion-based generation framework, our approach can efficiently produce physically plausible dynamics. This capability is particularly useful for studying physical phe- nomena where the governing equations are too complex to be learned purely from data.
0
Equation []. This fundamental
Figure 1: Animated visualization of generated samples of shallow water dynamics, showcasing the variations over time. Use the latest version of Adobe Acrobat Reader view.
Results. Empirical evaluations of our method demonstrate its effectiveness in producing high- quality dynamics across a range of physical phenomena. Our approach exhibits high robustness and generalizability, making it promising tool for the data-driven study of A14Physics. In Fig. [ provide a generated sample of the shallow water dataset Mhartinez-Arndaa et al. 2018). The gen- erated dynamics not only capture the intricate details of the physical processes but also adhere to the fundamental physical laws, offering an accurate and reliable representation of underlying systems.
Contribution. In conclusion, our work presents a significant advancement in the field of data-driven generative modeling by introducing novel framework that integrates physical priors into diffusion- based generative models. In all, our method 1) improves the feasibility of generated dynamics, mak- ing them more aligned with physical principles compared to baseline methods; 2) poses the solution to the longstanding challenge of generating physically feasible dynamics; 3) paves the way for more accurate and reliable data-driven studies in various scientific and engineering domains, highlighting the potential of A14Physics in advancing our understanding of complex physical systems.
PRELIMINARIES
In Appendix A we present a comprehensive review of Related Work, specifically focusing on three key areas: generative methods for physics, score-based diffusion models, and physics-informed neural networks. This section aims to provide foundational knowledge for readers who may not be familiar with these topics. We recommend that those seeking to deepen their understanding of these areas consult this appendix.
2.1 DIFFUSION MODELS
Diffusion models generate samples following an underlying distribution. Consider a random vari- able Zn E IR drawn from an unknown distribution 40- Denoising diffusion probabilistic mod-
els (Song & Ermon 2019 Song et al., 20200 Ho al. 2020) describe a forward process 2,, E [0. governed by an Ito stochastic differential equation (SDE)
d log 04 (t)) dt dt
do? d ai
dx; = 2t) + (t)ddw, Vo ~ qu. f(t = dt
0 (1)
2

---

Published as a conference paper at ICLR 2025
where W; E IR" denotes the standard Brownian motion, and aq and dr are predetermined functions of t. This forward process has a closed-for formed-fomm solution of q (x, œD) = W (x) ay o/ and has corresponding reverse process of the probability flow ordinary differential equation (ODE), running from time 7 to O, defined as (Song et al. 2020)
dæ
= f())x, 1 (g log (x.), dt
IT ~ QT( xo) = qr (r(T))
The marginal probability densities {mx)) of the forward SDE align with the reverse ODE (Song et al. 2020). This indicates that if we can sample from qr (xT) and solve Equation 2 then the resulting zo will follow the distribution go. By choosing a, + 0 and + 1, the dis- tribution 9r (xr) can be approximated as a normal distribution. The score Va log qr (zx) can be approximated by deep learning model. The quality of the generated samples is contingent upon the models' ability to accurately approximate the score functions (Kwon et al.) 2022 Gao & Zhu 2024). A more precise approximation results in distribution that closely aligns with the distribu- tion of the training set. To enhance model fit, incorporating priors of the distributions and physical feasibility into the models advisable. Section will elaborate on our methods for integrating distributional priors and physical feasibility priors, as well as the objectives for score matching.
2.2 INVARIANT DISTRIBUTIONS
An invariant distribution refers to a probability distribution that remains unchanged under the action of specified group of transformations. These transformations include operations such as trans- lations, rotations, or other symmetries, depending on the problem domain. Formally, let 9 be group of transformations. A distribution q is said be G-invariant under the group if for all transfor- mations G E G. we have g( (G())) = g(æ).. Invariance under group transformations is particularly significant in modeling distributions that exhibit symmetries. For instance, in the case of 3D coordi- nates, invariance under rigid transformations -ucch as translations and rotations (SE(3) group) is essential for spatial understanding (Zhou et al. 2024). Equivariant models are usually required to embed invariance. A function (or model) f T - Rn is said to be (g. -qquiivrrri where G is the group actions and L is function operator, if for any G E 9, (()())) (f()))
METHOD
In this study, we aim to investigate methodologies for enhancing the capability of diffusion models to approximate the targeted score functions. We have two primary objectives: 1) To incorporate distributional priors, such as translational and rotational invariance, which will aid in selecting the appropriate model for training objective functions; 2) To impose physical feasibility priors the diffusion model, necessitating injection of priors to model's output of a distribution related to the ground-truth samples (specifically, V.. log (a(a)) or E(zo a.)). In this section, consider the forward diffusion process given by Equation where 2, = a, + ore, with e ~ 10,,
3.1 ANCORPORATING DISTRIBUTIONAL PRIORS
In this section, we study the score function V log (2x) for G-invariant distributions. Understand- ing its corresponding properties
guide the selection of models with the desired equivariance, facilitating sampling from the G-invariant distribution. In the following, we will assume that the sufficient conditions of Theorem [ hold so that the marginal distributions qe G-invariant. The definitions of the terminologies and proof of the theorem can be found in Appendix F.I
Theorem (Sufficient conditions for the invariance of qo to imply the invariance of qe). Let go be G- G-ivvariant distribution. If for all G G, G is volume--reserving diffeomorphism and isometry, and for all 0 < a < 1, there exists H E 9 such that ((xx)) = (G(),, then qe is also G-invariant.
Property of score functions. Let q: be a G-invariant distribution. By the chain rule, we have Va log qe (xx) = V log qe (G(())) = aG(#.) Va(m) log qe (G(z))), for all G E G. Hence,
de
Va(=e log le( (G(x)) (aG(z:)
Va (xt))
(3)
Ox
3

---

Published as a conference paper at ICLR 2025
This implies that the score function of G-invariant distribution is (G. V equivvaraan.. We should use a (g, V '-qquivaria model to predict the score function. The loss objective is given by
Jscore
0
Ex, mma.c (t)) ||so (mt, () - Ve log (=<))))))))
(4)
where wt) is a positive weight function and so is (g, equivariant model. We will discuss the handling of the intractable score function V log q (z:) subsequently in Equation
In the context of simulating physical dynamics, two distributional priors are commonly considered: (F((n)-invarian) and cotnuutaiooniivariann They ensure that the learned representations are con- sistent with the fundamental symmetries of physical laws, including rigid body transformations and indistinguishabiliyy of particles, thereby enhancing the model's ability generalize across different physical scenarios. The derivations for the following examples can be found in Appendix F.2
Example 1. (SE(n) -invarantt distribution) If qo is an 3E(n)-inarrian distribution, then qe is also SE(n) invariant. The score function of an SE(n) (invaratt distribution is (0 and translational-ivvaiinn.
Example 2. (Permutation-ivvaiantt distribution) If qo is a permutation-invelt then qe is also inniaaiaiionnnnnrant.. The score Sunction of a permulation-inaariaa distribution is permulation- equivariant.
In the following, we will show that using such (G, V --quuvvvrrrii model, we are essentially training a model that focuses on the intrinsic structure of data instead of their representation form.
Equivalence class manifold for invariant distributions. An equivalence class manifold (ECM) refers to the minimum subset of samples where all the rest elements are considered equivalent to one of the samples in this manifold (informal). For example. in threc-dimensional space, coordi- nates that have undergone rotation and translation maintain their pairwise distances, which allows the use of set of coordinates to represent all other coordinates with the same distance matrices, thereby forming an equivalence class manifold (see Appendix B for the formal definition and ex- amples). By incorporating the invariance prior to the training set, we can construct ECM from the training set or a mini-batch of samples. The utilization of ECM enables the models to concentrate on the intrinsic structure of the data, thereby enhancing generalization and robustness to irrelevant variations. We assume that the distribution of x follows an G-invariant distribution q.- Let yp map a; to the corresponding point having the same intrinsic structure in ECM. Then there exists G g such that (((()))) = Dt Since gi is G-invariant, we have ge (xx)) = OFCM (f=(z))) (((a:))))PPUifommmlt (PUlform(() (G), where FECM is defined on the domain of ECM. Taking the logarithm and derivative, we have
V (())) log q (xx) V (()) log FEM (s((zx)). Note that Va log qe (xx) @e((ee) V ((x) log 4: (z.). Hence,
( p(a)) log 10gg (v(x)))
V log qi (xx))
(5)
Ox
This implies that the score function of the G-invariant distribution is closely related to the score function of the distribution in ECM. Such result indicates that if we have (g, -equivaain model that can predict the score functions in ECM, then, this model predicts the score functions for all other points closed under the group operation. We summarize this result in the following theorem whose proofs can be found in Appendix F.3
Theorem (Equivalence class manifold representation). If we have a (G, --quiivarra model such that so s0( () Va log 4ECM (2+)) almost surely on æ. ECM, then we have so ba( 1)
V x log qe (2;)) almost surely.
Objective for fitting the score function. The score function Va log (xx) is generally intractable and we consider the objective for noise matching and data matching (Vincent 201T Song et al.
2020 Zheng et al. 2023), where objectives and optimal values are given by
T noise (0) E E
è (2,,1) o. V log qi (ax) (()) |leo (xr,, 1) <ll²]
(6a)
1
xc + of va log qe (xx).
T data (0) = Er, (()) ||000 (xt, t) - xo
ag (2 t)
(6b)
at
ar
4

---

Published as a conference paper at ICLR 2025
The diffusion objectives for both the noise predictor Eø and the data predictor Ty intrinsically linked to the score function, thereby inheriting its characteristics and properties. However, the data predictor incorporates term, - 22z,, whose numerical range exhibits instability. This instability dz complicates the predictor's ability to inherit the straightforward properties of the score function. Therefore, to incorporate G-invariance, it advisable to employ noise matching, which is given by Equation and Eo is (G, V - -equivariant, which is the property of the score function.
A specific instance of a distributional prior is defined by samples that conform to the constraints imposed by PDEs. In this context, the dynamics any given spatial location depend solely on the characteristics of the system within its local vicinity, rather than on absolute spatial coordinates. Un- der these conditions, is appropriate employ cunsllatio-invariinn models for both noise matching and data matching. Nevertheless, the samples in question exhibit significant smoothness, As a re- sult, utilizing the noise matching objective necessitates that the model's 's output be accurate every individual pixel. In contrast, applying the data matching objective only requires the model to pro- duce smooth output values. Therefore, it is recommended to adopt the data matching objective for this purpose. The selection between data matching and noise matching plays critical role in determining the quality of the generated samples. For detailed experimental results, refer Sec.
Remark this section, we primarily explore the principle for incorporating distributional priors by selecting models with particular characteristics. Specifically:
1. When the distribution exhibits G-invariance, V- eqquvvaraa model should be em- ployed alongside the noise matching objective (Equation (6a).
2. For samples that are subject to PDE constraints and exhibit high smoothness, the data matching objective (Equation 6b recommended.
3.2 INCORPORATING PHYSICAL FEASIBILITY PRIORS
In this section, we explore how to incorporate physical feasibility priors such as physics laws and explicit PDE constraints into noise and data matching objectives in diffusson models. By Tweedie's formula (Efron 2011 Kim & Ye 2021 Chung et al., 2022), we have E[z, æy] = (e + V.
a log qt (x:)). Hence,
1 az
B[xo|
(xt o (z,, t)),
E[zg xx] x (xc, t).
For both noise and data matching objectives, we are essentially training model to approximate E|zu 21). A purely data-driven approach is often insufficient to capture the underlying physical constraints accurately. Therefore, similar to PINNs (Leiteritz & Pflüger] (202]), we incorporate an additional penalty loss JR into the objective function to enforce physical feasibility priors R. (æx) = 0 and set the loss objective to be J(0)- Jscore (0) (O), where Jacore is the data matching or noise matching objectives and feasibility loss. We consider the data matching objective where Do (I,, t) approximates B[E, æz]- For noise matching models, we can transform the model's output by Equation 7 For general cases, cannot directly add the constraints R (mx) due to the presence of Jensen's ' gap (Bastek et al. 2024), i.e., R (E(æco æx)) + E(R (xo) æ.. = O. However, in some special cases, can avoid dealing with this gap.
a hyperparameter to balance the diffusion loss and physical
the output of the diffusion model E[xg xx,
Linear cases. When the constraints are linear/affine functions, Jensen's gap equals O. Hence, we can directly add the constraints to (xt, f). We have JR. (0) Er, (t) | (we (æx, t)
Multilinear cases. A function is called multilinear if it is linear in several arguments when the [uo]
other arguments are fixed. Denote xo =
E pm+t" up E 1 E R". When the constraints
vo
function is multilinear w.r.t up we can write the constraints in the form of R (aa) = Woug+br = 0, where Wo o and by are functions of vo- In this case, we Er. [w])t] W Wo (z,, t) + bn [?]. Such a design is supported by the following theorem whose
the penalty loss JR (0)
proof be found in Appendx F4
5

---

Published as a conference paper at ICLR 2025
Theorem 3 (Multilinear Jensen's gap). The optimizer for Bi. ( (xx,,) Up [|]] is the
reweighted optimizer of ai, [w[t]] (æt, + bool?]] with reweighted variable W Wo-
Convex cases. If the constraints R is convex, by Jensen's 's inequality, R (Bla)
xx)) < E (zo)
2z] = O. Hence, = [[E[R (mr) æ;]ll² s |R (E(ær) æx)]) 1| When a data matching model approximately optimized, directly applying constraints to the model's output minimizes the up- per bound of the constraints on zo- The upper bound the Jensen's 's gap related the absolute centered moment op - < 2/ - 1 -1| where μ = E[
the constraints function R approach R(u) slower than Iy 14/ and grow Zo - =oo faster than =|Øo|" for n 2 7/. then the Jensen's gap E R (xo) xı] - R (E(zo æx]) approaches to 0 no slower than o on (Gao al. 2017). Usually, in the reverse diffusion process, On as / since the generated noisy samples converge to a clean one. this case, we use the penalty loss of
JR. Em.... (I) | (rr (11)
In the aforementioned three scenarios, the implementation level, the model's output may be directly considered as the ground-truth sample 2p itself, rather than the conditional expectation E| xt.. These scenarios are referred to "clementary cases". In the following, will discuss how to deal with nonlinear cases using the above elementary cases.
Reducible nonlinear cases. For nonlinear constraints, mathematically speaking, we cannot di- rectly apply the constraints to E[zro dz.]. However, we may recursively use multilinear functions to decompose the nonlinear constraints into elementary ones as: R (xx) = gı o.. gm (xo) = O, where all gi clementary. Using elementary functions for decomposition, linear constraints into clementary ones by treating terms causing nonlinearity as constants, and 2) reduce the complex constraints into several simpler ones. In this case, the penalty loss is set to
may 1) reduce non-
JR (0) = Er, ((t) |g o ....g g.. (vo (I,,t)) } See Sec. 4.2 for concrete examples of nonlin- ear formulas for the conservation of energy.
General nonlinear cases. For general nonlinear cases, if it is not feasible to decompose the non- linear constraints into their elementary components, it may be necessary to consider alternative approaches where we may reparameterize the constraints variable into elementary cases. Given the nonlinear constraints, we reparameterize it as R (xp) g (h(#a)) 0, where g is elemen- tary and is non-necessarily elementary functions. Subsequently, another diffusion model, de- noted as 2e (æ,, t), is trained to predict h(æo), utilizing the same hidden states as model ze (æ,, t). This training process employs the methods applicable to elementary cases. The objective is for model žo to learn the underlying physical constraints and encode these constraints into its hid- den states. Consequently, when model ve predicts, it inherently incorporates the learned phys- ical constraints g parameterized by h(xo)) To train model Øo. set the penalty loss be JR. (0) E,, w(() @e (æx, t) h( See Appendix E.I for implementation details.
Notably, our proposed methods for integrating constraints, the explicit form of prior knowledge, such the physics constants required for energy calculations, is not necessary. Instead, it suffices to determine whether the model's output parameters are elementary w.r.t. the constraints. This approach enhances the applicability of our methods to broader spectrum of constraints.
In conclusion, incorporating the physics constraints depending on their complexity. For elementary constraints, one can directly omit Jensen's 's gap and impose the penalty loss the model's output. In the case of nonlinear constraints, decomposition or reparameterization techniques are utilized to transform constraints into elementary ones.
Remark 2.
be achieved in different ways
4 EXPERIMENTS
In this section, we assess the enhancement achieved by incorporating physics constraints into the fundamental diffusion model across various synthetic physics datasets. We conduct a grid search to identify equivalent set of suitable hyperparameters for the network to perform the data/noise matching, ensuring a fair comparison between the baseline method (diffusion objectives without penalty loss) and our proposed approach of incorporating physics constraints. Appendix provides

---

Published as a conference paper at ICLR 2025
detailed account of the selection of backbones and the training strategies employed for each dataset. We also provide ablation studies in Sec. |.3 of 1) data matching and noise matching techniques for different datasets, revealing that incorporating distributional prior enhances model performance; 2) the effect of omitting Jensen's gap, finding that nonlinear constraints can hinder performance not properly handled. However, appropriately managing these priors using our proposed methods can lead to significant performance improvements.
4.1 PDE DATASETS
PDE datasets, including advection (Zang 1991). Darcy flow (Li et al. 2024), Burgers (Rudy et al,
2017). and shallow water (Klöwer et al. 2018), are fundamental resources for studying and modeling various physical phenomena. These datasets enable the simulation of complex systems, demonstrat- ing the capability of models for broader application across a wide range of PDE datasets. Through
this, they facilitate advances in understanding diverse natural and engineered processes.
Experiment settings.
The PDE constraints for the above datasets are given by:
Advection:
a, x) + (0, x)
0,
(8a)
Darcy flow:
0,u(z, t) V(a(x) Vu(x,)) = =f(x),
(8b)
Burger:
0,((,, + u(x, (),,t(,t)) =0,
(8c)
- Shallow water: -,hh or, = -hy, O,h = -2 (0.u + 0,0).
(8d)
A detailed introduction and visualization of the datasets can be found in Appendix C.I In this study, we investigate the predictive capabilities of generative models applied to advection and Darcy flow datasets. Our experiments focus on evaluating the models' accuracy in forecasting future states given initial conditions. Additionally, we examine the models' ability to generate physically feasible samples that align with the distribution of the training set on advection, Burger, and shallow water datasets. The evaluation metrics are designed to assess to what extent the solutions adhere to the physical feasibility constraints imposed by the corresponding PDEs.
Injecting physical feasibility priors. We train the models that apply the data matching objective as suggested in Remark [ We employ finite difference methods to approximate the differential equations. This approach renders the PDE constraints linear for the advection, Darcy flow, and shallow water datasets. However, PDE constraints become multilinear for the Burgers' equation dataset (see Appendix C.2 for the proof). Thus, the first set of datasets: advection, Darcy flow, and shallow water correspond to the linear case, while the Burgers' equation dataset corresponds the multilinear case. We can directly apply the physical feasibility constraints on the model's output.
Experimental results. Results can be seen Tab. of diffusion models in predicting physical dynamics, given initial conditions, within generative framework that produces a Dirac distribution. The accuracy of these models is evaluated using the RMSE metric. The observed loss magnitude is comparable to the prediction loss using with FNO, U-Net, and PINN models (Takamoto et al.) 2022) (refer to Appendix E.4 for further details). Our results indicate that the incorporation of constraints consistently enhances the accuracy of the prediction. In Tab. 2 the feasibility of the generated samples is evaluated by calculating the RMSE of the PDE constraints, which determine the impact of incorporating physical feasibility priors on diffusion models. We also provide visualization of the generated samples in Fig.
In Tab. I we analyze the performance
13
Performance comparison of diffu- sion models with/without priors for predict- ing physical dynamics. The models' accu- racy measured using the RMSE metric, highlighting the impact of incorporating con- straints on improving prediction accuracy.
Table I:
Table 2: Comparative analysis of diffusson models, assessing the feasibility of generated samples with/without physical feasibility pri-
ors. We evaluate the RMSE of PDE con- straints, demonstrating the effect of physical fcasibility priors on the adherence to PDEs.
Method
Advection (x100 2) Darcy flow (x10-* 2)
Method weo prior
Advection
Burger
Shallow water
w/o prior
17263=0.0491
2,0648 1078800000 0.0600
00.024
10,0550
w/ prior
1.636=0.0667
1.9678 0.0651
w/ prior
2.305510.001
6.610:20012
7.7618
7

---

Published as a conference paper at ICLR 2025
4.2 PARTICLE DYNAMICS DATASETS
We train diffusion models to simulate the dynamics of chaotic three-body systems in 3D (Zhou & Yu 2023) and five-spring systems in 2D (Kuramoto) T975: Kipt et al. 2018) (see Appenidx. D.1 for visualizations of datasets). In the case of the three- body, we unconditionally generate the posi- tions and velocities of three particles, where gravitational interactions govern their dynamics. The stochastic nature of this dataset arises from the random distribution of the initial positions and ve- locities. In five-spring systems, each pair of particles has a probability 50% of being connected by spring. The movements of the particles are influenced by the spring forces, which cause stretch- ing or compression interactions. We conditionally generate the positions and velocities the five particles based on their spring connectivity.
Notations. The features of the datasets represented as ( [C(O) Vi0j E R LxXX22 where (( V(0) Ix Here, the matrix ( encapsulates the coordinate features, while ( encapsulates the velocity features. The superscript denotes the time for the diffusion process and the subscripts denote the matrix index. represents the temporal length of the physical dynam- ics, K denotes the number of particles, and D corresponds to the spatial dimensionality. We use the subscript to indicate time, while the subscripts i, j. and are used to denote the indices of particles. The subscript d represents the index corresponding the spatial axis. We also the subscript of 0 denote the corresponding values of the model's prediction of E E (0) X with inputs X()) and t, and X = [Ca Vel.
Injecting -invariiierreeneee and permutation invariance. Two physical dynamic systems are governed by the interactions between each pair of particles, resulting in a distribution that is SE(n) and permutation invariant. Our objective is develop models that SO(n) equivarrant, translation invariant, and permutation equívariant. We intend to apply noise matching objective to achieve the desired invariant distribution. However, to the best of our knowledge, no such architecture satisfying the above properties has been established within the context of diffusion generative models. There- fore, we opt to utilize a data augmentation method to ensure the model's equivariance and invariance properties (Chen et al. 2019 Botev et al. 2022). i.e. apply these group operations in the training process, which enforces models to be equivariant and invariant.
Conservation of momentum. For both datasets, the momentum conservation is given by:
K m& k, constanty,
VI L,, d= I,...,D. D.
(9)
Here, represents the mass of k-th particle, and V lkk denotes the velocity along axis d of the k- th particle at time (. The total momentum in each axis remains constant, as indicated by the equality.
This constraint is linear w.r.t. Vik.d corresponding to the linear case. Let : RRLKXD XR xR RD calculate the mean of the total momentum over time and set the penalty loss
L
JR (0) = Er.
w
Σ m& (Vo) fa(V {ma 1
(10)
I=
d=1
Conservation of energy for the three-body dataset. The total of gravitational potential energy and kinetic energy remains constant over time. The energy conservation equation is given by:
K Gm,my
K D
ΣΣ k=1d=1
ml
(V) ) constant,
(.ka)
AI 1....,,
(11)
R
where G denotes the gravitational constant. R R || c denotes the Euclidean distance between the i-th and j-th particle at time L. This constraint is nonlinear with (() but can be de- composed into elementary cases. Note that the constraint multilinear w.r.t. 1/R (0) and (V()) vik,a) Hence, from the results of the general nonlinear cases, we train another model sharing the same
8

---

Published as a conference paper at ICLR 2025
Figure 2: Visualization of generated samples from the three-body (first row) and five-spring (second row) datasets. The lefumost figures in each row represent methods without priors, the middle figures correspond to our proposed methods, and the rightmost figures illustrate the physical properties as they evolve over time. Both total momentum and total energy should remain conserved. The samples generated by our methods demonstrate stronger adherence physical feasibility.
hidden size as the model for noise matching to predict these variables related to the conservation of energy. Furthermore, since these variables are convex w.r.t. (0)), by the results of the convex case, we can directly apply the penalty loss as:
(V L.k.d) $11 (12)
JR. (0) Beg
(t)
1
w2(()
(Vo)i L.K.d
Ro )..s
R.
R,j
(0)
Co (x(1), t),, ||. i.c. model's prediction of the Euclidean distance between two particles calculated from its prediction of coordinates. This penalty loss can also be derived from the reducible case. The detailed derivation can be found in Appendix D.2
where (Ro),, ||oo (x(*),t) ()),
Conservation of energy for the five-spring dataset. The combined elastic potential energy and the kinetic energy are conserved throughout time. The equation for the conservation of energy is represented by:
K
D
Σ (i,))E
2n
R
2
2
m&
(v)
2 l.k.d) = constant,
VI = 1,..,,, L.
(13)
where x denotes the clastic constant, R2 | and j-th particle time and & denotes the edge of springs connecting particles. m& represents the mass of the k-th particle. Analogue to the conservation of energy for the three-body dataset, we can reduce the nonlinear constraints into elementary cases.
(0)
C denotes the distance between the i-th
Experimental results. The results can be seen in Tab. 3 and Fig. to Appendix E.2 for a detailed account the experimental settings, as well as a more extensive comparison of the effects of hyperparameters and various methods for injecting constraints across the discussed cases. Our analysis indicates that for the three-body dataset, the incorporation of the conservation of energy prior, via the reducible performance across evaluated metrics. Similarly, applying the conservation of momentum prior to the five-spring dataset significantly reduces the momentum error in the generated samples. This also
15 and we refer readers
method, substantially enhances the model's

---

Published as a conference paper at ICLR 2025
Table 3: Sample quality of the three-body and five-spring datasets. For both datasets, we simulate the ground-truth future motion based on the current states of the generated samples and report the MSE error between the ground-truth motion and the generated ones. We also calculate the error of physical feasibility such as conservation of the energy and momentum, which should remain unchanged along the evolution of the systems.
Three-body
Five eive-spring
Method
Traj error
Vei crror
Energy error Dynamic etror
Momentum error
Energy error
w/o prior
2.4132
2.5745 0.00990
4.329210.7235
3.3699-0.0463 0.0462 5.0731 0.38981100011
5.1754+0.022
1.0618 10.0243 0.7418 0.0129
prior
19880-10.3418
0.8328 0.1042
0.5465
contributes to a reduction the errors associated with dynamics and energies. Fig. demonstrates that the total momentum and energy of samples generated with the incorporation of priors exhibit greater stability compared to those without priors. We also provide sampling results using the DPM- solvers (Lu et al. 2022) in Appendix E.7 which significantly lower computational expenses.
4.3 ABLATION STUDIES
Table 4: Results of an ablation study comparing the ef- feets data matching and noise matching techniques. The findings show that incorporating distributional prior improves model performance. We use the mean of trajectory error and velocity error as the metric for the three-body dataset.
Distributional priors through match- ing objective. We employ data match- ing and noise matching techniques for the PDE and particle dynamics datasets, re- spectively. An ablation study is conducted to investigate the effects of applying the alternative matching objective on the par- ticle dynamics and PDE datasets, both without physical feasibility priors. The results, presented in Tab. [ demonstrate that incorporating a distributional prior can significantly improve the model's per- formance.
Meshod
Threa-hay
Pive-speing 5.1929
Darey flow
Shallow water
distributonal alternative
26084
8,150
4.7241
5.3120
7.268
27.40
Table 5: Results show the impact of enforcing energy conservation constraints on the three-body dataset.
Direct application of nonlinear constraints (prior by PINN) can degrade performance, while proper han- dling (prior by ours) improves accuracy.
Omitting Jensen's gap. In the three- body dataset, we employ a multilinear function simplify constraints into con-
Method
Traj error
eeror
Energy
w/o prior
2.5613
2.6555
3.8941
vex scenarios. We now conduct an abla- tion study in which the output of diffu- sion model is considered the ground-truth,
prior by PINN
2.6048
2.6437
4.2219
prior by ours
1.6072
0.7307
0.5062
and the constraint of energy conservation is imposed similarly to the injection of constraints by penalty loss in the prediction tasks of PINNs. This configuration is referred to as "prior by PINN". We define a penalty loss based on the variation of energy over time, analogous to the penalty loss used to enforce momentum conservation con- straints. However, unlike the conservation of momentum, which is governed by a linear constraint and can thus be applied directly, the conservation of energy involves nonlinear constraint. This introduces Jensen's gap, preventing the direct application of the constraint. The results, presented in Tab. 5 indicate that directly applying nonlinear constraints can degrade the model's performance. However, appropriately handling these constraints can significantly improve the sample quality.
CONCLUSION
In conclusion, this paper presents a novel method for generating physically feasible dynamics using diffusion models by integrating distributional and physical feasibility priors. We inject distribu- tional priors through equivariant models and noising matching, while incorporating physical fea- sibility priors through constraint decomposition. Empirical results demonstrate the robustness of method across various physical phenomena, highlighting its promise for data-driven A14Physics research. This work emphasizes the importance of embedding domain knowledge into learning systems, bridging physics and machine learning through innovative use of physical priors.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
This work was supported by the National Science and Technology Major Project of China under Grant 2022ZD01 16408. This work is also supported by a grant from ChemLex Technology Co., Ltd..
REFERENCES
Rucha Apte, Sheel Nidhan, Rishikesh Ranade, and Jay Pathak. Diffusion model based data genera- tion for partial differential equations. arXiv preprini arXiv:2306. 11075, 2023.
Jan-Hendrik Bastek, WaiChing Sun, and Dennis M Kochmann. Physics-informed diffusion models. arXiv preprint arXiv:2403. 14404, 2024.
Mathis Bode, Michael Gauding, Zeyu Lian, Dominik Denker, Marco Davidovic, Konstantin Klein- heinz, Jenia Jitsev, and Heinz Pitsch. Using physics-informed enhanced super-resolution gener- ative adversarial networks for subfilter modeling in turbulent reactive flows. Proceedings of the Combustion Institute, 38(2):2617 2-222 2021.
Aleksander Botev, Matthias Bauer, and Soham De. Regularising for invariance to data augmentation improves supervised learning. arXiv preprint arXiv:2203.00 033344 2022.
Danilo Bzdok and John PA loannidis. Exploration, inference, and prediction in neuroscience and biomedicine. Trends in neurosciences, 42((4:255 2019.
Shengze Cai, Zhiping Mao, Zhicheng Wang, Minglang Yin, and George Em Karniadakis. Physics- informed neural networks (pinns) for fluid mechanics: review. Acta Mechanica Sinica, 37(12): 1727-1738, 2021.
Ruijin Cang, Hechao Li, Hope Yao, Yang Jiao, and Yi Ren. Improving direct physical properties prediction of heterogeneous materials from imaging data via convolutional neural network and morphology-awan generative model. Computational Materials Science, 150:212-221, 2018.
Weicong Chen, Lu Tian, Liwen Fan, and Yu Wang. Augmentation invariant traiiign In Proceedings of the IEEF/CVE International Conference on Computer Vision Workshops, pp. 0-0, 2019.
Kamal Choudhary, Brian DeCost, Chi Chen, Anubhav Jain, Francesca Tavazza, Ryan Cohn, Cheol Woo Park, Alok Choudhary, Ankit Agrawal, Simon JL Billinge, et al. Recent advances and applications of deep learning methods materials science. npj Computational Materials, (1):59, 2022.
Hyungjin Chung, Jeongsol Kim, Michael T Mccann, Mare L Klasky, and Jong Chul Ye. Diffusion posterior sampling for general noisy inverse problems. arXiv preprint arXiv:2209. 14687, 2022.
Junyoung Chung, Caglar Gulcehre, KyungHyun Cho. and Yoshua Bengio. Empirical evaluation of gated recurrent neural networks on sequence modeling. arXiv preprint
2014.
Salvatore Cuomo, Vincenzo Schiano Di Cola, Fabio Giampaolo, Gianluigi Rozza, Maziar Raissi, and Francesco Piccialli. Scientific machine learning through physics-informet neural networks: Where we and what's next. Journal of Scientific Computing, 92(3):88, 2022.
Luke de Oliveira, Michela Paganini, and Benjamin Nachman. Learning particle physics by example: location-aware generative adversarial networks for physics synthesis. Computing and Software for Big Science, 1(1):4, 2017.
Ivan Dokmanie, Reza Parhizkar, Juri Ranieri, and Martin Vetterli. Euclidean distance matrices: essential theory, algorithms, and applications. IEEE Signal Processing Magazine, 32(6):12-30,
2015.
Bradley Efron. Tweedie's formula and selection bias. Journal of the American Statistical Associa- tion, 102-496) 2011.
Amir Barati Farimani. Joseph Gomes, and Vijay S Pande. Deep learning the physics of transport phenomena. arXiv preprint 17Xiv:1709.0244 02432, 2017.

---

Published as a conference paper at ICLR 2025
Thorben Finke, Michael Krämer, Alessandro Morandini, Alexander Mück, and Ivan Oleksiyuk. Autoencoders for unsupervised anomaly detection in high energy physics. Journal of High Energy Physics. 2021())11322 2021.
Xiang Gao, Meera Sitharam, and Adrian E Roitberg. Bounds on the jensen gap, and implications for mean-concentratee distributions. arXiv preprint arXiv: 17X2.0:/112.0266: 2017.
Xuefeng Gao and Lingjiong Zhu. Convergence analysis for general probability flow odes of diffu- sion models in wasserstein distances. arXiv preprini arXiv:2401.17956 2024.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 2020.
Moritz Hoffmann and Frank Noé. Generating valid euclidean distance matrices. arXiv preprint arXiv:1910.03131. 2019.
Jeebyun Hwang, Jeongwhan Choi, Hwangyong Choi, Kookjin Lee, Dongeun Lee, and Noseong Park. Climate modeling with neural diffusion equations. In 2021 IEEE International Conference on Data Mining (ICDM), pp. 230-239. IEEE, 2021.
Yayati Jadhav, Joseph Berthel, Chunshan Hu, Rahul Panat, Jack Beuth, and Amir Barati Farimani. Stressd: 2d stress estimation using denoising diffusion model. Computer Methods in Applied Mechanics and Engineering. 416:1 16343, 2023.
Arbaaz Khan and David A Lowther. Physics informed neural networks for electromagetti analysis. IEEE Transactions on Magnetics, 58(9):1-4, 2022.
Kwanyoung Kim and Jong Chul Ye. Noise2score: tweedic's approach to self-supervised image denoising without clean images. Advances in Neural Information Processing Systems, 34:864- 874, 2021.
Thomas Kipf, Ethan Fetaya, Kuan-Chich Wang, Max Welling, and Richard Zemel. Neural relational inference for interacting systems. In International conference machine learning, pp. 2688- 2697. PMLR, 2018.
M. Klöwer, M. F. Jansen, M. Claus, R. J. Greatbatch, and S. Thomsen. Energy budget-based backscatter in a shallow water model of a double gyre basin. Ocean Modelling, 132, 2018. doi: 10.101 20010.0..001.09999
Yoshiki Kuramoto. Self-entrainment of a population of coupled non-linear oscillators. In Inter- national Symposium on Mathematical Problems in Theoretical Physics: January 23-29, 1975, Kyoto University, Kyoto/Japan, pp. 420 422. Springer, 1975.
J Nathan Kutz. Deep learning in fluid dynamics. Journal of Fluid Mechanics, 814:1 2017.
Dohyun Kwon, Ying Fan, and Kangwook Lee. Score-based generative modeling secretly minimizes the wasserstein distance. Advances in Neural Information Processing Systems, 35:20205-200 202117 2022.
Antonio Lavecchia. Deep learning in drug discovery: opportunities, challenges and future prospects. Drug discovery loday, 24110):2017-2032, 2019.
Zaharaddeen Karami Lawal, Hayati Yassin, Daphne Teck Ching Lai, and Azam Che Idris. Physics- informed neural network (pinn) evolution and beyond: A systematic literature review and biblio- metric analysis. Big Data and Cognitive Computing, 6(4):140, 2022.
Raphael Leiteritz and Dirk Pfiüger. How to avoid trivial solutions in physics-infonee informed neural net- works. arXiv preprint arXiv:21 2021.
Zongyi Li, Nikola Kovachki, Kamyar Azizzadenesheli, Burigede Liu, Kaushik Bhattacharya, An- drew Stuart, and Anima Anandkumar. Fourier neural operator for parametric partial differential equations. arXiv preprint 100.0:2010008885, 2020.
12

---

Published as a conference paper at ICLR 2025
Zongyi Li, Hongkai Zheng, Nikola Kovachki, David Jin, Haoxuan Chen, Burigede Liu, Kamyar Azizzadenesheli, and Anima Anandkumar. Physics-inie informed neural operator for learning partial differential equations. ACM/JMS Journal of Data Science, 1(3):1-27, 2024.
Marten Lienen, David Lüdke, Jan Hansen- Palsuu, and Stephan Günnemann. From zero to turbu- lence: Generative modeling for 3d flow simulation. arXiv preprint arXhv:2306.01766, 2023.
Phillip Lippe, Bas Veeling, Paris Perdikaris, Richard Turner, and Johannes Brandstetter. Pde-refiner: Achieving accurate long rollouts with neural pde solvers. Advances in Neural Information Pro- cessing Systems, 36, 2024.
Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. Dpm-solver: A fast ode solver for diffusion probabilistic model sampling in around 10 steps. Advances in Neural Information Processing Systems, 35:5775-5787, 2022.
Lu Lu, Xuhui Meng, Zhiping Mao, and George Em Karniadakis. Deepxde: deep learning library for solving differential equations. SIAM review, 63(1):208-22) 228, 2021.
Wei Ma, Feng Cheng, Yihao Xu, Qinlong Wen, and Yongmin Liu. Probabilistic representation and inverse design of metamaterials based on a deep generative model with semi-supervised learning strategy. Advanced Materials, (15))) 1901111, 2019.
S Martinez-Aranda:
Javier Fernández-Pear Pato, Daniel Caviedes- Ignacio Carcía-Palacín and Pilar García-Navarro. Towards transient experimental water surfaces: A new benchmark dataset for 2d shallow water solvers. Advances in water resources, 121: 30-149, 2018.
Emile Mathieu, Vincent Dutordoir, Michael Hutchinson, Valentin De Bortoli, Yee Whye Teh, and Richard Turner. Geometric neural diffusion processes. Advances in Neural Information Process- ing Systems, 36, 2024.
Maziar Raissi, Paris Perdikaris, and George Karniadakis. Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations. Journal of Computational physics, 378:686-707, 2019.
Rishikesh Ranade, Chris Hill, and Jay Pathak. Discretizationnet: A machine-learning based solver for navier-stokes equations using finite volume discretization. Computer Methods in Applied Mechanics and Engineering, 378: 1 13722, 2021.
Stephan Rasp, Michael Pritchard, and Pierre Gentine. Deep learning to represent subgrid processes in climate models. Proceedings of the national academy of sciences, 115(39):9884-988) 2018.
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomed- ical image segmentation. In Medical image computing and computer-assisted intervention- MICCAI 2015: 18th international conference, Munich, Germany, October 5-9, 2015, proceed- ings. part III 18, pp. 234-241. Springer, 2015.
Samuel H Rudy, Steven L Brunton, Joshua Proctor, and Nathan Kutz. Data-driven discovery of partial differential equations. Science advances, 3(4); 16026144 14, 2017.
Chitwan Saharia, Jonathan Ho, William Chan, Tim Salimans. David J Fleet, and Mohammad Norouzi. Image super-resolution via iterative refinement. IEEE transactions pattern anal- ysis and machine intelligence, 45(4):4771 47263344,62200 2022.
Ahmed Khan Salman, Arman Pouyaei, Yunsoo Choi, Yannic Lops, and Alqamah Sayeed. Deep learning solver for solving udvection-diffusee equation in comparison to finite difference meth- ods. Communications in Nonlinear Science and Numerical Simulation, 115:106780, 2022.
Victor Garcia Satorras, Emiel Hoogeboom, and Max Welling. E (n) equivariant graph neural net- works. In International conference on machine learning, pp. 9323-9332, PMLR, 2021.
Dule Shu, Zijie Li, and Amir Barati Farimani. A inysics-informed diffusion model for high-fidelity flow field reconstruction. Journal of Computational Physics, 478: 11972, 2023.
13

---

Published as a conference paper at ICLR 2025
Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. Advances in neural information processing systems, 32, 2019.
Yang Song, Jascha Sohl- Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. arXiv preprint WrXiv:2011.13456, 2020.
Yang Song, Conor Durkan, lain Murray, and Stefano Ermon. Maximum likelihood training of score-based diffusion models. Advances in neural information processing systems, 34:1415- 1428. 2021.
Makoto Takamoto, Timothy Praditia, Raphael Leiteritz, Daniel MacKinlay, Francesco Alesiani, Dirk Pflüiger, and Mathias Niepert. Pdebench: An extensive benchmark for scientific machine learning. Advances in Neural Information Processing Systems, 35:1596-1615 1611, 2022.
Naoya Takeishi and Alexandros Kalousis. Physics-integrated variational autoencoders for robust and interpretable generative modeling. Advances in Neural Information Processing Systems, 34: 14801, 2021.
Carlos Uriarte, David Pardo, and Ángel Javier Omella. A finite element based deep learning solver for parametric pdes. Computer Methods in Applied Mechanics and Engineering, 14562, 2022.
Pascal Vincent. connection between score matching and denoising autoencoders. Neural compu- tation, 23(7):1661-166) 2011.
Xintao Wang, Ke Yu, Shixiang Wu, Jinjin Gu, Yihao Liu, Chao Dong, Yu Qiao, and Chen Change Loy. Esrgan: Enhanced super-resolution generative adversarial networks. In Proceedings of the European conference on computer vision (ECCV) workshops, pp. 0-0, 2018.
Jin-Long Wu, Karthik Kashinath, Adrian Albert, Dragos Chirila, Heng Xiao, et al. Enforcing sta- tistical constraints in generative adversarial networks for modeling chaotic dynamical systems. Journal of Computational Physics, 406:109209 2020.
Minkai Xu, Lantao Yu, Yang Song, Chence Shi, Stefano Ermon, and Jian Tang. Geodiff: A geo- metric diffusion model for molecular conformation generation. arXiv preprint arXiv:2003.02923, 2022.
Qi Yan, Zhengyang Liang, Yang Song, Renjie Liao, and Lele Wang. Swingnn: Rethinking per- mutation invariance in diffusion models for graph generation. arXiv preprint arXiv:2307.0166 2023.
Gefan Yang and Stefan Sommer. A denoising diffusion model for fluid field prediction. arXiv preprint arXiv22301.11661, 2023.
Liu Yang. Dongkun Zhang, and George Em Karniadakis. Physics-informed generative adversarial networks for stochastic differential equations. SIAM Journal on Scientific Computing, 42(1): A292-A317, 2020.
Jason Yim, Brian I. Trippe, Valentin De Bortoli, Emile Mathieu, Amaud Doucet, Regina Barzilay, and Tommi Jaakkola. Se (3) diffusion model with application to protein backbone generation. arXiv preprint arXiv:2302.02277 2023.
Thomas A Zang. On the rotation and skew-symmetric forms for incompressible flow simulations. Applied Numerical Mathematics, 7(1):27-40, 1991.
Lei Zhang, Lin Cheng, Hengyang Li, Jiaying Gao, Cheng Yu, Reno Domel, Yang Yang. Shaoqiang Tang, and Wing Kam Liu. Hierarchical deep-learnigg neural networks: finite elements and be- yond. Computational Mechanics, 67:207-230, 2021.
Kaiwen Zheng, Cheng Lu, Jianfei Chen, and Jun Zhu. Improved techniques for maximum likelihood estimation for diffusion odes. In International Conference on Machine Learning, pp. 42363- 42389. PMLR, 2023.
14

---

Published as a conference paper at ICLR 2025
Zihan Zhou and Tianshu Yu. Learning to decouple complex systems. In International Conference on Machine Learning, pp.
PMLR, 2023.
Zihan Zhou, Ruiying Liu, Jiachen Zheng, Xiaoxue Wang, and Tianshu Yu. On diffusion process in se (3)-invariant space, arXiv preprint arXiv:2403.01450 2024.
15

---

Published as a conference paper at ICLR 2025
A RELATED WORK
A.1 GENERATIVE METHONSS FOR PHYSSCC
Numerous studies have been conducted on the development of surrogate models to supplant nu-
merical solvers for physics dynamics with GANs (Farimani et al. 2017, de Oliveira et al. 2017]
Wu et al. 2020] Yang et al. 2020; [ode et al. (021) and VAES (Cang et al. 2018). Neverthe-
less, to generate realistic physics dynamics, one must accurately learn the data distribution or inject
physics prior (Cuomo et al. 2022). Recent advancements in diffusion models (Song et al., 2020)
have sparked increased interest in their direct application to the generation and prediction of phys-
ical dynamics, circumventing the need for specific chysics-based formulations (Shu et al., 2023
Lienen et al., [2023 Yang & Sommer [023] Apte et al.] 2023) Jadhav et al., [023] Bastek et al.
2024). However, these approaches, which do not incorporate prior physical knowledge, may exhibit
limited performance, potentially leading to suboptimal results.
A.2 SCORE-BASED BASED DIFFUSION MODELS
Score-based diffusion models a class of generative models that create high-quality data samples by progressively refining noise into detailed data through a series of steps (Song et al.] (020). These models estimate the score function, the gradient of the log-probability density of the data, using neural network (Song & Ermon 2019). By applying this score function iteratively to noisy samples, the model reverses the diffusion process, effectively denoising the data incrementally, and generates samples following the same distribution as the training set. Although numerous studies on diffusion models have focused on generating SE(3)-invariant distributions (Xu et al. 2022, Yim et al. [023] Zhou et al., 2024), there remains lack of comprehensive research on the generation of general in- variant distributions under group operations. Meanwhile, in contrast to GANs, the outputs produced by diffusion models represent the distributional properties of the data samples. This fundamental difference means that physical feasibility priors cannot be added directly to the model output due to the presence of Jensen gap (Chung et al. 2022), i.e. R. (E|æz | æz.)) * [R[ (æo) æz)] A poten- tial solution to this problem involves iterating and drawing samples during the training process and subsequently incorporating the loss of physics feasibility on the generated samples (Bastek et al.) 2024). However, this approach necessitates numerous iterations, often in the hundreds, rendering the training process inefficient.
A.3 PHYSSIS-INFORREDD NEURAL NETWORKS
Physics-Informen Neural Networks (PINNs) are class of deep learning models that incorporate physical laws and constraints into their training process (Lawal et al.. 2022). Unlike traditional training processes, which learn patterns solely from data, PINNs leverage priors including PDEs that describe physical phenomena to guide the learning process. By incorporating these physical feasibility equations as part of the penalty loss, alongside the data prediction loss, PINNs enhance their ability to model complex systems. This integration allows PINNs to be applied across various fields, including fluid dynamics (Cai et al. 22221), electromagnetism (Khan & Lowther 2022), and climate modeling (Hwang et al. 2021). Their ability to integrate domain knowledge with data-driven learning makes them powerful tool for tackling complex scientific and engineering challenges.
B EXTENSION ON EQUIVALENCE CLASS MANIFOLD
B.1 FORMAL DEFINITION
Let X be set and ~ be an equivalence relation on X. The equivalence class manifold M is defined as the set of equivalence classes under the relation ~. Formally,
M = fo x E X, y E
(14)
where M is Riemannian manifold and [zz] denotes the equivalence class of x, defined as:
[x] {u E X 1y~x))
(15)
16

---

Published as a conference paper at ICLR 2025
B.2 EQUIVALENCE CLASS MANIFOLD SE(3) -INVRRANTA DISTRIBUTION
The following theorem provides a method to use set of coordinates to represent all other coordi- nates having the same pairwise distances.
Theorem 4 (Equivalence class manifold of SE(3)-invariant distribution (Dokmanic al. 2015
Hoffmann & Noć 2019 Zhou et al. 2024)). Given any pairwise distance matrix D E R4 there
exists a corresponding Gram matrix E RR*x defined by
1 (D1s D:1 Dij)
Mú 2
(16)
and conversely
Di Ma + M5j -2M
(17)
By performing the singular value decomposition (SVD) on the Gram matrix M E R=xx (associated with D), we obtain exactly three positive eigenvalues λ1, X2, xs and their respective eigenvectors
11 22, V3, where X
2 X2 Z X3 0. The set of coordinates
V11
0
C = V2, vsl
0
X2
(18)
0
0
Vā
satisfies has the same pairwise distance matrix as D.
Define the above mapping from the pairwise distances to coordinates as f. Then, the equivalence class manifold of SE(3)-invariant distribution can be given by
M = (J(D) D is pairwise distance matrix).
(19)
M satisfies the property of being a Riemannian manifold (Zhou et al. 2024).
C PDE DATASETS
A summary of the important properties of datasets can be found in Tab.
Table 6: Comparative summary of datasets. The table highlights key aspects such the type of generation (conditional or unconditional), the matching objective, the distributional priors, and the nature of the constraint cases.
Datasets
Cond/Uncond generarion
Matching objective
Distriburional prices
Constraint cases
advection
both
linear
PDE
Darcy
Hurger
conditional unconditinal
data (Eiquation 6b
PDE constraints
multilineur
water
particle dynamics
Uhree: body
unconditonal
moise (Fiuaaton 6a
SEG33 permulathon invariant
all cases
live-spring
permutation invariant
C.I DATASET SETTINGS
Advection. The advection equation is fundamental model in fluid dynamics, representing the transport of a scalar quantity by a velocity field. The dataset presented herein consists of numerical solutions to the linear advection equation, characterized
a @,t(t,x)) + (t,,(t,))) = 0. x (), (0,
(20)
where denotes the scalar field and / = 0.1 is constant advection speed. The visualization of training samples can be seen in Fig. Based on the initial conditions provided for the advection equation, our model utilizes generative framework to predict the subsequent dynamics, with the specific aim of forecasting the next 40 frames. We then compare these predictions with the ground- truth to assess performance. Additionally, evaluate the model's capability to generate samples unconditionally, without initial conditions, and measure performance using the physical feasibility implied by the PDE constraints.
17

---

Published as a conference paper at ICLR 2025
Samples from the advection dataset with varying initial conditions. The horizontal axis represents the spatial coordinate v, while the vertical axis represents the parameter I.
Figure 3:
Darcy flow. Darcy's 's law describes the flow of fluid through a porous medium, which is fun- damental principle in hydrogeology, petroleum engineering. and other fields involving subsurface flow. The mathematical formulation of the Darcy flow PDE is given by:
Öeu(x, 1) V((())Vu(z; ()) = f(x), x E (0.
(21)
where u(x, represents the fluid pressure at location x and time 6, a(x) the permeability or hydraulic conductivity, and (xx) denotes sources sinks within the medium. Given the initial state
= O, we use the generative scheme forecast the state / 1. Fig. provides visualization
of training samples. The accuracy of these predictions is evaluated by comparing them with the ground-truth values.
Figure 4: The figure illustrates representative training samples from the Darcy flow dataset. The first row displays the values for the function alz), while the second row shows the values of u(x, t) at time
Burger. The Burgers' equation is fundamental PDE that appears in various fields such as fluid mechanics, nonlinear acoustics, and traffic flow. is a simplified model that captures essential features of convection and diffusion processes. The equation is given by:
Óuu(2, t) + u(æ,, ()0,u(x, () 0,
(22)
where u(x, t) represents the velocity field, x and t denote spatial and temporal coordinates, re- spectively. We unconditionally generate samples following the distribution of the training set and evaluate feasibility within the realm of physics as dictated by the constraints of PDE.
Shallow water. The linearized 2D shallow water equations describe the dynamics of fluid flows under the assumption that the horizontal scale is significantly larger than the vertical depth. These equations instrumental in fields such oceanography, meteorology, and hydrology for modeling wave and current phenomena in shallow water regions. Let u and v denote the components of the velocity field in the æ- and y-directions, respectively. The variable represents the perturbation in the free surface height of the fluid from a mean reference level. The parameter e denotes the phase speed of shallow water waves, which function of the gravitational acceleration and the mean water depth. The equations are expressed as follows:
Ou.
0h
Ov
ah
0h
JJu
c
(23)
0t
Dr
0t
Oy
Dt
Fr
Öy
18

---

Published as a conference paper at ICLR 2025
Figure 5: The figure depicts representative training samples from the Burger dataset. The samples within this dataset exhibit minimal variability. The horizontal axis denotes the spatial coordinate x, whereas the vertical axis represents the parameter t.
We conditionally generate the dynamics of shallow water expressed by h, conditioned on the given c. We provide a visualization of one sample in Fig.
Figure 6: In the figure, the sequence from left to right represents a sample of the dynamies of shallow water. Each sample consists of 50 frames, from which 10 frames have been uniformly selected for visualization.
C.2 CONVERTING TO ELEMENTARY CASES BY FINITE DIFFERENCE APPROXIMATION
vevectoon and shallow water.
The original constraint of the advection equation is given by
at x)) + 30,u(tt x) = 0.
(24)
If the finite difference method approximate the derivative, assume a grid with time steps in and spatial points ze. Let un denote the approximation to u.),, For the time derivative,
use forward difference approximation: d.u difference approximation: d,u have
~
- For the spatial derivative, use a central
Af
Substituting these approximations into the PDE, we
Aw
"
wf 2+1 ur = 0. M
n+1
(25)
At Rearrange to obtain an equation that involves only u values and constants:
Ar
Al Ar
Al
w " - (1
Ju,
u
++1
0.
(26)
Ar
In this form, the constraint linear equation involving + u u shallow water constraints can be performed in an analogous manner.
The linearization of the
Darcy flow. The given Darcy flow equation is:
0ru(x, t) V(a(z)Vu(z,()) = f(x),
(27)
Using the finite difference method, we discretize the domain into a grid with grid spacing Ar Ay and At. Let úsj represent u(x;, yj, l.n and u² represent u(r;, (y). The finite difference
19

---

Published as a conference paper at ICLR 2025
approximations for the gradients and divergence are:
u²+11 un
du =
(28a)
2At
Uitly 1,-1,5
O, ~
(28b)
2
Ui,i+1
Wi.j-1
Oyu ~
(28c)
2Dy
The Hessian matrix of V2u(I, is also linear w.r.t u(x; t) when approximated by the finite differ- ence method. Hence, the left-hand side of the Darcy flow equation is the sum of terms linear w.r.t u(x t) and thus the constraint is linear.
Burger. The partial differential equation
0, t) + u(x, ())
(29)
can be approximated using finite differences as follows: time derivative (forward differ- u(z,(+Ai)-u(aa spatial derivative (central difference): 0 0_u(x,t) s
ence): 0,u(I, t) 2
u(an+ An.t) u(a An.1) 2Ar
Substituting these into the PDE gives:
u(x, + At) u(x,t) + uu(x t). u(x + Az, t) u(x Ar,
(30)
At
2Ar
Hence, the constraint is multilinear w.r.t. u(x, 1) if we consider values of other points stants.
D PARTICLE DYNAMICS DATASETS
D.1 DATASET INTRODUCTION
The dataset features are structured as X (C(0) V(0] where ((0) and ( are both elements of x... x In this context, L refers to the temporal length of the physical dynamics, K represents the number of particles, and D denotes the spatial dimensionalityy Specifically, ((O) captures the coordinate features, and (0) captures the velocity features. For the three- body dataset, the param- eters set as L 10, K 3. and D 3. indicating temporal length of 10, with 3 particles in a 3-dimensional space. Similarly, for the five-spring dataset, L = 50, K = 5, and D = 2, corre- sponding temporal length of 50, particles, and 2-dimensional space. For both datasets, generated 50k samples for training. We aim to generate samples following the same distribution as the training dataset.
Fig. 7 provides visual representations of two samples from the particle dynamics dataset, showcasing the behavior of systems within the three-body and five-spring datasets.
D.2 DETAILS OF INJECTING THE CONSERVATION OF ENERGY FOR THE TIREE-BODY DATASET
The total of gravitational potential energy (GPE) and kinetic energy (KE) remains constant over time. The formula of the energy conservation equation is given by:
K
Gmamy
ΣΣ
5 my (V)
V.k.a)2 constant,
VI I,..,,
(31)
R
(0)
where denotes the gravitational constant, and all three bodics have the same mass my. R (
||c[0] I,a c( denotes the Euclidean distance between the i-th and j-th mass at time This con- straint is nonlinear with (0) but be decomposed into elementary cases. Note that the constraint
20

---

Published as a conference paper at ICLR 2025
Figure 7: The presented figures illustrate samples from the particle dynamics dataset. The first two depict sample from the three-body dataset, while the subsequent two rows represent sample from the five-spring dataset.
is multilinear w.r.t. 1/R t.iy and (V(.)a)
(.k.a) ). Hence, from the results of the general nonlinear cases, we can apply the penalty loss JR (0) JAPE (0) TKE (0), and
I GPF (0) = Er, (t) Σ
Re
(32a)
1,60
R1
JKL (0) = Er, W2(()
Σ L,k,d
Vo
(v i.k,dd
(32b)
1,k,d
where Ro and Ve share same hidden state as the model Xe and X. predicts [[ X X
The setting details of these models are introduced in Appendix E. We refer such a setting as "noise matching conservation of energy (general nonlinear)". Meanwhile, note that 1 1/R and (v(0) a2 are convex w.r.t. model's output. From the results in the convex case, we can directly apply the penalty loss output of X and set the penalty loss to
T JApE (0) = Ex,
W1 (t)
Σ
(33a)
(Ro (Ro).,5
R
JTT (0) Er,
W2 (1)
Σ
(Vo)i.... -
(v(00) 1.R.d
(33b)
L.k.d
where (Ro),.) ICCo (X(9) ()),, Co (X("), 1), |l. i.e. model's prediction of the Euclidean dis- tance between two masses. We refer such a setting as "noise matching + conservation of energy (reducible nonlinear)", since this penalty loss function can be derived using a multilinear function composed with convex functions. In comparison to the penalty loss described
Co
Equation 32 the penalty loss presented in Equation 33 is applied directly to the output of X This direct applica- tion imposes stronger constraint, thereby more effectively ensuring that the model adheres to the specified physical constraints.
21

---

Published as a conference paper at ICLR 2025
E EXPERIMENTS DETAILS
Configuration. We conduct experiments of advection, Darcy flow, three-body, and five-spring datasets NVIDIA GeForce RTX 3090 GPUs and Intel(R) Xeon(R) Silver 4210R CPU @ 2.40GHz CPU. For the rest of the datasets, we conduct experiments on NVIDIA A1100-XMM-800G GPUs and Intel(R) (e) Platinum 8358P CPU @ 2.60GHz CPU.
Training details. We use the Adam optimizer for training, with a maximum of 1000 epochs. We set the learning rate le-3 and betas to 0.95 and 0.999. The learning rate scheduler ReduceL-- ROnPlateau with factor=0.6 and patience=10. When the learning rate is less than 5e-7, we stop the training.
Diffusion details. As for diffusion configuration, set the steps of the forward diffusion process 1000, the noise scheduler o, sigmoid (inssacce( 5, 1000)) and at / /1-00 o7. The loss weight w(t) is set (t) do? 2 log of (Song al. 2021 We generate samples using the DPM-Solver-1 (Lu et al. 2022).
dt
Experiment summary. We summarize the choice for backbones and properties of datasets in Tab. 6 We conducted an equivalent search for the hyperparameters of both the baseline methods and the proposed methods. The specific search ranges for each dataset and the corresponding hyper- parameters are summarized in Tab.
Table 7: Summary of the model hyperparameters.
Disasels
Backhone
Model hyperpanaetees
Bos3h siRe
312), layers:
Dancy Now
Karms Cikl
[128]
Burger
Karms Uner Bo &
200
dina: [32]
shallow water
3D Karns Und (10 &
2020
dim: [16]
64
RNN hidden size: (64, 128, 256, 312, 10224)
particle dynamics
livespriig
three-body
(Sttoras et
NN+GRU Chang
3121 HORL
2014
ata 2014
[3.4,5)
[256, 312, TUZA]
Error bars. Error bars are computed by varying both training and sampling seeds (3-5 runs),
except for Darcy flow, where only the sampling seed is varied due the long training time.
E. TRAINING GENERAL NONLINEAR CASES
Three-body dataset. To reduce the nonlinear conservation of the energy by general nonlinear cases, we apply the penalty loss JR. (0) = TGPE (0) + +Tk (0), and
T GPE (0) = Er,0,.e
wI( Σ
Re 1.65
(34a)
R
R
(0)
2112 ||
KF (0) = -EE,
w2(()
Σ L.k.d
(vo)
(vil
(34b)
1.k,d
The models Rg and V share the same hidden state the model Xe. where Xe is tasked with predicting ()) x The GRU architecture serves as the backbone for Xe. Consequently, we have designed the outputs the models RS and V, to generated by an additional linear layer that takes the hidden state of the GRU within X input.
22

---

Published as a conference paper at ICLR 2025
Five-spring dataset. JKE (0), and
For the five-spring dataset, apply the penalty loss JR (0)
Jpp (0) +
PE (0) ,00,
wi (t)
Σ
[Ro)
R
(35a)
Lis
(iJJEE,
8112 (V.keed) (kk,d)
J (0) Rr,
w2((1)
Σ
(Vo)
(vi())
(35b)
L,k,d
L.k.d
The models Ro and Vo utilize the same hidden state the model Xo. with Xo responsible for predicting () x( The underlying structure of X is based on EGNN for extracting node and edge features and GRU network for dealing with time series. As a result, the outputs of R produced by an additional linear layer that processes the edge features generated by EGNN within Xg, and the outputs of Vo are produced by an additional linear layer that takes the hidden state of the GRU within X. input.
E.2 DETAILS OF EXPERIMENT RESULTS
Tab. S and Tab. present the outcomes of the grid search conducted on both the three-body and five- spring datasets. For the three-body datasets, the top three combinations of hyperparameters- sidedaddeeesss-ddeenn size and the number of layers-are highlighted for each training method. For the five-spring datasets, the top three hidden size hyperparameters identified for each training method are provided.
Table 8: The outcomes of the grid search conducted on the three-body datasets are summarized. For each training method, we highlight the top three combinations of hyperparameters, focusing on hidden size and the number of layers. "linear" refers to the settings in [0 and "reducible nonlinear" and "general nonlinear" refer to the settings in Equation 33 Equation 34 respectively. We first perform grid search to find the optimal hyperparameters. Then, conduct experiments with the optimal parameters, running each experiment five times using the seeds 42, 0, 1, 2, and 3. However, observe that using seed 2 for noise matching results in poor performance, from all methods.
exclude this trial
Method
Hyperparameter
Trajectory error
Velocity
Energy
256, 4
5.2455
4,2028
12.758
data matching
512, S
5.7765
3,8985
13.636
236,
3.309%
4,4144
11.643
256,
24132+0.122
2.5745 +0.0900
4.3292-40.7235
noise matching
256,
2.5695
2.6713
3,3944
512,
2.6368
2.7192
3.5427
noise matching
512.
2.126110.0533
2.2815=10.0344
3.8698.10 2486
1024. 4
2.4179
2.5261
3,9003
conservation momentum (linear)
512,
2.4188
2.5264
6.8971
noise matching
128, 3
1.9880=0.3418
0.8328=0.1008
0.5465=00.0700
128,
1.6659
0.7605
0.5198
conservation of energy (reducible nonlincar)
128.
1.7821
0.8030
0.4532
noise matching
512,4
2.2160=50.0669
2.0091
4.59415:0.3961
512.
2.5335
2.6234
3,8091
conservation of energy (general nonlinear)
1024.
2.5068
2.6737
5,2131
E.3 SENSITIVITY OF PENALTY LOSS WRIGHT
Regarding the hyperparameters of the penalty loss weight, experimental results show that the model performance is not highly sensitive to variations in the loss weight. Specifically, we conducted search for the loss weight across logarithmic scale, testing approximately five times. To further illustrate this, we provide an example of how the loss weight affects model performance on the three-body dataset using conservation of momentum in Fig.
23

---

Published as a conference paper at ICLR 2025
Table Grid search results for the five-spring datasets are provided. We present the top two hyper- parameter hidden size identified for each training method. The results of our experiment indicate that the output of the GRU model inadequate accurately predict the distance between particles. This limitation renders the methods designed for reducible nonlinear cases inapplicable, and conse- quently, their results have been excluded from our study. In contrast, the convoluted edge features generated by the EGNN model are sufficiently informative for predicting particle distances. More- over, the application methods suitable for general nonlinear cases improves performance. "linear" refers to the settings in Equation 10 and "general nonlinear" refers to the settings in Equation 35
Method
Hyperparameter
Dynamic error (x10-2)
Momentum eroor
Energy emoo
1024
5.3120
5.2320
1.1204
data matching
5.3872
5.1448
1.1030
3.8054:10.0286
5.3699=0.0462
0:0.0233
noise matching
256
5,1950
5.3468
1,0805
noise matching
256
5,0731 -i 0.0406
0.3898 0.0118
0.7418 0.0129
conservation of momentum (linear)
512
5,0990
0.4335
0.7652
noise matching
256
5. 5.16433+0.0528
5.3359=0.081
1.0500--0.0473
conservation of energy (general nonlinear)
1024
5.1809
5.3902
1.0879
5.0
errors
4.0
vel error
3.5
energy
3.0
0.0001
0.000s
0.001
0.005
0.01
0.05
loss weight
Figure 8: Sensitivity of model performance to variations in the penalty loss weight on the threc-body dataset. The table shows the results of testing different loss weights logarithmic scale, with performance metrics recorded at five distinet points. The results indicate that model performance is relatively stable across range of loss weight values.
E.4 COMPARSION PREDICTION METHODS
We conduct a comparison between the performance of the generation and prediction methods. In Tab. 10 we present comparative analysis of generative models and prediction models for predicting physical dynamics, specifically advection and Darcy flow. The results of the prediction methods are taken from Takamoto et (2022), which performs a comprehensive comparison of FNO (Li al. Unet (Ronneberger et al. 2015), and PINN (Raissi et al.) 2019) (using DeepXDE (Lu et al. Generative models that diffusion techniques, both with and without prior information.
2020
2021
exhibit comparable performance in both tasks. The diffusion model with priors shows an improve- ment over the one without priors. this work, we do not conduct the procedure super- or denoising (Wang et al., 2018 Saharia et al. (022). which are critical in practical applications to produce high-quality, clean, and detailed images from diffusion models. Hence, the performance of diffusion models can be further enhanced by the introduction of super-resolution and denoising.
Despite the inherent minor noise in the data, the model successfully captures the underlying pat- terns of the system, identifying low-frequency components and representing broad trends over time. However, challenges arise in accurately modeling high-frequency details, especially in highly non- linear systems (Darcy flow dataset). This limitation becomes apparent in multi-step forecasting, where errors in resolving finer details can accumulate, leading to significant deviations from the true system behavior the prediction horizon extends (Lippe et al. 2024).
24

---

Published as a conference paper at ICLR 2025
Table 10: Performance comparison of diffusion generative models with prediction models. The
results of the prediction methods are brought from Takamoto et al. 2022).
Method
Backbone
Advection
Darey flow
diffusion w/o prior
1.716 10
2.016 10
Generation
Karras Unet
diffusion w/ prior
1.621 x 10
1.954 x 10
forward propagator approximation
FNO
4.9 10
1.2 x 10
Prediction
autoregressive method
Unet
3.8 10-2
6.4 10-3
PINN
DeepXDE
7.8 x 10-1
We also test how predictive models perform on the five-spring dataset. We test the performance of NRI decoder in Kipf et al. (2018), which specially designed for the dynamics and interacting systems. The results can be seen in Tab.
Table 11: Performance comparison of diffusion generative models with prediction models on the five-spring dataset. We use the model (decoder only) in Kipf et al. (2018) to predict the dynamics and use EGNN (Satorras et al. 2021) and GRU (Chung et al. 2014) for diffusion methods. The metric of dynamic error different from those in other tables. For other tables, we use the mean of the error between the generated dynamics and the ground truth dynamics solved by finite element methods in the following 8 steps, while in this table, we use only step. The definition the momentum error and energy eπror the same. Note that diffuuson methods cannot surpass predictive methods without priors. After the incorporation of the priors, diffusion models generate high-quality samples compared with the predictive methods.
Method
Dynamic error (x10
Momentum error
Energy
noise matching
2.5178
5.3511
1.0891
noise matching + priors NRI predict
2.4329
0.3687
0.7448
2.4471
5.2234
1.4577
E.5 WIIY NOT TRANSFORMER?
We attempted to implement a transformer architecture as the backbone for sequential data in particle dynamics datasets. However, our results indicate that the transformer-based model does not achieve performance comparable to that of recurrent structure backbones. This discrepancy is likely due to nature of physical dynamics, where the next state is strongly dependent on the current state. The attention mechanism employed by transformers may reduce performance in this context, it does not inherently account for the temporal evolution of states.
E.6 ABLATION STUDIES ON DISTRIBUTIONAL PRIOR THROUGH DATA AUGMENTATION
Some existing works argue that considering the invariance property of the distribution may sacrifice the empirical performance Yan et al. 2023). We perform further ablation studies to examine the sig- nificance of the distributional priors. Specifically, we evaluate two scenarios: 1) with permutational data augmentation, and 2) without permutational augmentation. To perform the permutational data augmentation, we randomly permute particles along with other properties such as their edge con- nectivity. The results are presented in Tab. 12 and 13 below.
Table 12: Ablation study on data augmentation for the equivalence the model on the three- body dataset.
Traj error
Velocity error
Energy error
Method
w/o aug
w/ aug
w/o aug
w/ aug
w/o aug
w/ aug
momentum conservation
2.1953
2.1409
2.2974
2.2529
3,4364
4.1116
reducible energy conservation
1.8170
1.6072
1.9239
0.7307
3.2382
0.5062
We also test the equivalence the trained models, one trained with data augmentation and the other without. Results show that, even without the introduction of data augmentation for equivalence, the
25

---

Published as a conference paper at ICLR 2025
Table 13: Ablation study on data augmentation for the equivalence of the model on the five-spring dataset.
Dynamic error
Momentum error
Energy error
Method
w/o aug
w/ aug
w/o aug
w/ aug
w/o aug
aug
momentum conservation
6.3972
5,0919
0.3564
0.3687
1,5536
0.7448
reducible energy conservation
6.6211
5.1615
7.0716
5.3032
2.4910
1.0548
model still learns to exhibit the desired equivalence, which aligns with our mathematical analysis. Furthermore, when equivalence data augmentation is applied, the model achieves significant re- duction in equivalence error, decreasing from 1.6e-3 to 3.5e-4. This further supports the correctness of our analysis, demonstrating that, when using two models with the same capacity, (G, V V- cquivariant model should be employed for noise matching.
E.7 SAMPLING IN FEWER STEPS USING DPM-SOLVERS
We conduct experiments of using the DPM-solvers (Lu et al. 2022 sample in fewer steps. By reducing the number of diffusion steps required. DPM solvers significantly lower computational expenses in generating physics dynamics. This efficiency achieved with minimal degradation in performance, ensuring that the resulting dynamics remain closely aligned with the underlying physical principles. We apply the DPM-Solver-3 (Algorithm Lu al. (2022)) and the results
can be seen in Fig. and 10
calaer ASSOR OU gaenay
aynamics arrors
3.2-
2,07
10
50
50
1.00
150
260
20
100
150
total sampling steps
total sampling steps
Figure 9: Results of sampling through DPM- Solver-3 on the three-body dataset. The x- axis denotes the values for M in Algorithm 2
Figure 10: Results of sampling through DPM-Solver-3 on the five-spring dataset. The x-axis denotes the values for M in Al- gorithm 2 in Lu et al. 2022)
in Lu et al. 2022).
F PROOFS
F.1 SUFFICIENT CONDITIONS FOR THE INVARIANCE OF MARGINAL DISTRIBUTION
Definition 5 gellue-presservnng) A function whose derivative has a determinant equal to I is known as a volume-preserving function.
Definition 6 (isomorphism), An isomorphism is peructure-presenvve mapping between two struc- tures of the same type that can be reversed by an inverse mapping.
Definition (diffeomorphissm, A diffeomorphism an isomorphism of differentiable manifolds. is an invertible function that mups one differentiable manifold to another such that both the function and its inverse are continuously difserentiable.
Definition (isometry), Let X and Y be metric spaces with metrics (e.g., distances) dx and dy. A map? f: X + Y is called an isometry for any a, b E X, dx(a, b) = dy (f(a), f(b)).
Definition 9 (homothety). If for all E G and for all sealar < a < 1, there exists H E g such that H(ax) = aG(x). H would be homothety (a transformation that scales distances
26

---

Published as a conference paper at ICLR 2025
by constant factor but does necessarity preserve angles). The group 9 formed by all such transformations is called the homothety group.
Theorem (Sufficient conditions for the invariance of qu to imply the invariance of qe) Let qo be G-invariant distribution. If for all G E g, G is volume-preserrvnn diffeomorphism and isometry, and for all < a 1, there exists H E 9 such that H((ar) = (G()), then qt also G-invariant.
Proof. For VE diffusion (defined in Sec. 3.4 in Song et al. 2020 ), di (zt, wo) = N (are Wo, o
For any G-invariant distribution yo and G E G, we have
qe G (xxe) = f qe (G(+) To go))gol (z dxo
probability chain rule
(36a)
= =ʃ (G(())
(G
change of variables
(36b)
= f fN((G()))
(G(az) (xo)), Tf go (
volume-preserving diffeomorphism
(36c)
= =f N (zx xo, oi (o
isotropic Gaussian N and isometry G
(36d)
= - ( (o)))u(z;)da)
(36e)
(xx)
(36f)
Hence, the marginal distribution @t at any time I is an G-invariant distribution.
For VP diffusion (defined in Sec. 3.4 in [Song et al. (200)). assume 04 0 any time t. le (zt 2p) = N (xx ayp0, (1 I).. Define qe
qe( xz). Note that Ø2 To + AT (O I), is random variable generated by some VE diffusion process. Hence, its marginal distribution at any time / is G-invariant, For any G-invariant distribution we have
qe (x:))
(a G(x:))
by definition by sufficient conditions G-invariance
(37a)
a (H((a)))
(37b)
gr (a)
(37c)
(xxz
(37d)
Discussion of related theorems.
Theorem 10 (Proposition in Xu et al. 2022 ). Let q (FT) be an SE(3)-invariant density func-
tion, i.e., q (xT) q (G (xT) for G E SE(3). If Markov transitions q (x;-1 xx) are SE(3)- equivariant, i.e., (x:-1 xx) q (G (1)-1) G (xx)). then we have that the density (xo) = -ivvariatt
J (xT) qo (mo:T-1 ar) dænrr is also
Xu et al. (2022) explore the integration of invariance during the sampling process while disregarding it during the forward process. Xu et al. (022)) propose that sampling through equivalent translational kernel results in invariant distributions. In contrast,
Theorem demonstrates that when the transition probabilities of the Markov chain are not distribution can still be SE(n) -invariinaiiint.. This result offers stronger conclusion than that presented by Xu et al. (202)).
the resulting composed
Theorem (Proposition 3.6 Yim et al.) (2023), Proposition 3.1 Mathieu et al. (024)). Let be Lie group and H. a subgroup of G. Let (0) go for an H. invariant distribution qo- If (x = b (t)) de+ 1//2 ( for bounded, H-equivariant coefficients b and E satisfying bo Ln dL (b) and E-La()) left-invariant metric. Then the distribution qi of (()) is H-invariant.
(E.), and where B(t) is Brownian motion associated with
27

---

Published as a conference paper at ICLR 2025
In contrast our Theorem Theorem in Yim et al. (2023) and Mathieu et al. (2024) imposes specific conditions on the relationship between the forward diffusion scheduler and the group oper- ators, whereas our theorem does not. However, Yim et al. (2023) impose fewer constraints on the properties the group operations.
F.2 INVARIANT DISTRIBUTION EXAMPLES
F.2.1 (E(n)-INVARIAN DISTRIBUTION
qo is an invarrian distribution, then qi is also -nnnarinnrriant.
Proof. Given any E SE(n), let Gax:) Ra b, where R (0(n),
volume-preservingg det (dG(a) det (R))=
diffeomorphism: smoothness: The transformation G(x)) Ræ smooth because it involves linear operations (rotation and translation) that are smooth. Specifically, the rotation R and the translation are smooth functions of their parameters; bijectivity: The function (2) is bijective. For any The inverse function is given by: G (g) = R (y b). where E R Since is rotation matrix, it is invertible, and its inverse R is also smooth. Therefore, the inverse function smooth.
RR. the function G(æ) is one-to-one and onto.
isometry: for all x, E R" G(x) G(v)ll2 = |Re + - (Ry + b) I| = (xc y) R R (xc y) || yl Hence, [[G(z) G | yll-
homothety:
Given any G SE and (aax) R (ax) + ab = a (Rx b) = (G(z).
1, let H(x) Ra ab E SE(n). Then,
Hence, sufficient conditions are satisfied and 4 is also (EEnn--nnvarian.
Let q be an SE(n)-invariant distribution. Given set of m points C E Raxm, write it in the vector form vec(C) mm For any G(C) RC + b, R SO(n), let R Rmmxm be block matrix with R on its diagonal block and b |b TT__ bJj E Rm We have G(z)) R. b. Then, Gv) = (R ) R. Hence, V (()) log (G(x)) RV, - log g(x). which imples Vacc log g (G( RV. log gCC). Thus, the score function of an SE(n)-ine iinvariant distribution is eequivarraa and iiliaatiiona-ivvriannn
F.2.2 PERMUTATION-IIVARIAN DISTRIBUTION
We first list some useful properties of the Kronecker product:
vcc(AXB) T A) (X)
det (A 8 B) = det (A)m det (B)", where A E Rmxm B E Raxn
(A & B T A & T
(A & (C D) (AC) & (BD).
For square nonsingular matrices and B: (A & B 8-B
If go is permutation-innaatie then g: also permutation-invariant.
Proof. Given any G E 9 with G(X) = PXPT, we consider its vector form of vec (G(X)) =
vec (PXP (P& & P) vec (X).
volume-preserving: det ( (dvec(G(X)) = det ( (P&P) = det (P (P@ P)
vec(
det (P)" det (P)" det (P)2n (+1)2n
28

---

Published as a conference paper at ICLR 2025
diffeomorphism: smoothness: G is smooth because it involves matrix multiplication, which is a smooth operation in X. Since P and PT are constant matrices (not func- tions of X). G inherits the smoothness from the matrix operations: bijectivity: Suppose = ( = PXPT. To recover X from Y, we compute: X = = YP, which is
valid operation because P is invertible.
isometry: note that (P & P) P) (P
X P
P
(P
(P (
PP
I & I = I. For all X, Y E Ruxn
vec (X))) vec (G(Y) 1?
(38a)
(P @ P) vec (X) (P @ P) vec (Y) ||
(38b)
(vec (X) vec (Y)) (P P) P P) (vec (X) vec (Y))
(38c)
(vec (X) (Y)) T [ (X) vec (Y)
(38d)
vec (X) (Y)
(38e)
Hence, ||G(X) G(Y) I|{ X 1
homothety: Given any 9 and 0 < a < 1 = G E G. Then, H(aX)
(aX) aPXP xX(
Hence, sufficient conditions are satisfied and q: is also permutation-invariant
Let be a permutation invariant distribution of feature X E R"** such as affinity/conmecciittyy matrices representing relationships connections between pairs of entities (e.g., nodes in graph) or Gram matrices in kernel methods representing similarities between set of vec- (X) for any permutational matrix P Consider the vectoriza- tion of X. Let ((vec(X)) == g(X). Note that vec (PXP)) = (P @ P) vec(X). Hence, Vvec(PXPT) log ((vee(PXPP) ))
tors. Let (PXPTT
(P & P)Vvor(X) log ğ( (X))) This implies that PXPT log l = PVx log (()PT. Thus, the score function of
(P
2 P) -V (x) log ((vec(X))
invariant distribution is permutation-equivartt
F.3 ECM EQUIVALENCE
Theorem (Equivalence class manifold representation). If we have a (G. --)ivvrrrriiian model such that so (o(t,)) v log (ECM)((.)) almost surely on a, ECM, then we have so (r:,, 1) Va log qe (xx) almost surely.
Proof. Suppose we have a (G, V-1 ( equivariant model sø and se(v(z)) V(tan log JECM (f(x))
almost surely. Then, we have
(a()) = ae(xa)
Ox. so (x())
by equivariance of the model
(39a)
0p(xx)
V ((m)) log FECM ( (p(x))
(39b)
dæ
Va log (z)
by Equation 5
(39c)
F.4 MULTILINEAR JENSEN'S GAP
The following lemma is directly from the results of the optimal values of noise matching, which will be used in proving Theorem. 3
Lemma 12. The gradient of E Eu. (tt)]l (æ,, t) c| w.r.t. O at (ze, t) = -o Ve log (xx) equals O.
Theorem 3 (Multilinear Jensen's gap). The optimizer for Br. uoo (we, 1) un [|]] is the
reweighted optimizer of Bi, [w][[]]]oo (æt, + boll with reweighted variable W. W..
29

---

Published as a conference paper at ICLR 2025
Proof. Without loss of generality, suppose that the optimizer of us is given by uo: + use- The loss optimizer of data matching is given by uoi (xx,,) 1 into the PDE loss term, we have
at (u, +o/V log q (æx)). Substituting
Bi [w(t)] Woue, (x t) + boll
(40a)
= a. (t))
Wo (u, + V log 2e (are) + Wouso + boll3]
(40b)
1
= E [wt(t)
Wo (a:n0 + ore + of log (zi) + Wouse + b0ll
(40c)
= (ut))) + bo + at Woe+ o Wo Vu log a (2:)) + Wousell
(40d)
at
Er (t))ll at Woc+ + of WoV log ai (xt) Wousoll7]
(40e)
a1
ag
mg (()() W E ( log at (xx)
uso
[
(40f)
az
Dropping the reweighting term of /az does not change the optimal solution. When uso = 0. observing the above objective and the noise matching objective, the above objective is the reweighted objective noise matching by replacing with W Wo-
G VISUALIZATION OF GENERATED SAMPLES
In this study, we refrain from applying super-resolution and denoising procedures, which are essen- tial in practical applications for generating high-quality, clear, and detailed images from diffusion models. Consequently, the generated samples contain noise. For the thrce-body dataset, since we only generate 10 frames, apply the cubic spline to visualize smooth trajectory and this is not applied when evaluating the quality samples.
30

---

Published as a conference paper at ICLR 2025
PDE matching
data matching PDE
PDE
matching PDE constraints
PDE
matching
matching
matching
Figure II: Examples of Darcy flow samples generated by models trained with/without PDE con- straints.
31

---

Published as a conference paper at ICLR 2025
data matching
PDE loss
data matching
PDE loss
data matching POL constraints
PDE
matching PDL constraints
PDE loss
PDE
matching
PDE
data matching POL constraints
PDE
matching POL constraints
PDE loss
Figure 12: Examples of Burger samples generated by models trained with/without PDE constraints.
32

---

Published as a conference paper at ICLR 2025
Figure 13: Examples of shallow water samples generated by models trained with PDE constraints.
Each sample contains 50 frames and we uniformly visualize of them.
33

---

Published as a conference paper at ICLR 2025
50000
Figure 14: The figure illustrates examples of faree-body samples produced by models trained with and without conservation constraints. The left panel represents the baseline model, the middle panel corresponds to our model, and the right panel depicts the energy as a function of time.
34

---

Published as a conference paper at ICLR 2025
song
AVI
Figure 15: The figure illustrates examples of five-spring samples produced by models trained with and without conservation constraints. The left panel represents the baseline model, the middle panel corresponds to model, and the right panel depicts the momentum a function of time.
35


Reviews:
[{"review_id": "g8vJYXMeqk", "reviewer": "Reviewer_QDux", "summary": "This work proposes a framework that incorporates physical priors, including distributional priors and physical feasibility priors, to generate physically realistic dynamics with diffusion-based models.", "questions": "- The three contributions listed in Section 1 appear similar and convey the same meaning. Accurately summarizing the manuscript's contributions will be more helpful for the readers.\n\n- Why distribution priors (such as translational rotational invariance) are important for generating physical dynamics? As stated in [1], the invariant sampling and invariant loss functions by restricting architecture designs often sacrifice empirical performances. Are simple data augmentations that approximate probability equivariance also available in physical dynamics generation? Experiments with data augmentations that approximate probability equivariance might help analyze the effectiveness of equivariant models.\n\n- How are the physical priors selected in this work? It appears that the roto-translational invariance and the priors based on energy and momentum conservation laws are chosen arbitrarily. There seems to be no systematic analysis that explores all possible categories of physical priors or how to incorporate each type of prior into diffusion models. I believe there are many more physical priors that generative models should be constrained by. Categorizing them and discussing each category systematically would provide more insight.\n\n- In Line 498-508, the paragraph title is \"Data matching vs noise matching\". However, the ablation study is conducted for incorporating a distributional prior or not, which is expected to investigate the training objective instead.\n\n---\n[1] SwinGNN: Rethinking Permutation Invariance in Diffusion Models for Graph Generation. TMLR 2024.", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The manuscript is well-written and easy to follow.\n2. Thorough derivations are provided in the main content and appendix.\n3. Sufficient experiments and ablation studies are conducted to validate the effectiveness of the proposed method.", "weaknesses": "Please see the Question part.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qdQZ90y6hm", "reviewer": "Reviewer_RSW6", "summary": "The paper proposes a novel framework to generate physically feasible dynamics by incorporating physical priors into diffusion-based generative models. Unlike traditional generative approaches, which often disregard physical constraints, this model integrates two types of priors: distributional priors (such as invariance to roto-translation) and physical feasibility priors (e.g., conservation laws for energy and momentum, and partial differential equation constraints). By embedding these priors into the generative process, the method produces realistic dynamics across a variety of physical systems.", "questions": "- The model includes various hyperparameters, especially in the weighting of physical feasibility losses. How sensitive is the model to these settings? Could the authors provide guidance on tuning these parameters, or propose default values based on their findings?\n\n- What does \"intrinsic structures of pairwise distance\" refer to?\n\n- Does the resolution matter for \"reducible nonlinear cases\"?\n\n- As I noticed, all of the datasets in this work seem to be with multi-dimensional features. Yet in biology, we usually just have single-dimensional features possibly collected at a time-step. Will this affect the performance of the method?\n\n- I think the five springs dataset is probably more like an ODE dataset instead of a PDE. The transition functions are second-order ODEs.\n\n- Following last question, how about reporting the errors of NRI (Kipf et al., 2018) on the experiments in Section 4.2 as well? \n\n- I have a tiny suggestion on the setup of the paper. If it is hard to fit a whole section of related works in the main body, it would be a good practice to merge a part of the related works to the section of preliminaries. Readers need the clear message in the related works to recall the research gap.\n\n- Please check the score function in line 146, it does not consistent with the one in line 115. Please be careful about the suffices.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The work is innovative in its focus on embedding physical feasibility directly into diffusion-based generative models, specifically through the use of both distributional and physical priors. By designing a process that enforces these constraints, the model is capable of generating realistic dynamics, distinguishing it from more traditional generative approaches that may ignore or only partially enforce physical laws.\n\n- Moreover, the paper is well-written and is thorough with both the theoretical part and the empirical results for the proposed model. \n\n- The paper addresses a challenge in generating accurate physical dynamics with AI methods. It enables more reliable and scientifically sound simulations in areas like environmental, material science, and possibly the fluid simulations.", "weaknesses": "- The experiments focus on synthetic datasets or physics-inspired datasets, and while this is valuable, additional testing on real-world, noisy datasets would definitely help.\n\n- If possible, comparing the model's performance with more baselines will make the empirical results more convincing. Yet it might be difficult to find the methods from the exactly same field, but some methods from, e.g. time-series forecasting, can be taken into consideration.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "WJtkjAxfPt", "reviewer": "Reviewer_JiqC", "summary": "This work explores ways of incorporating physics-informed priors in diffusion models for dynamical systems. In particular, they investigate **distributional priors**, aiming to incorporate appropriate symmetries (such as roto-translational invariance, permutation invariance, etc.), and **physical feasibility priors** as a way to encourage adherence to physical laws (e.g. momentum and energy conservation, PDE constraints, etc.). The manner in which the latter are imposed depends on the nature of the constraints, and the authors analyse several such cases separately (e.g. linear, multilinear, nonlinear, etc.).\n\nThe work provides empirical evidence on i) four PDE datasets, where PDE constraints are used, and ii) two particle dynamics datasets, where the diffusion model leverages the relevant physical laws, and training involves data augmentation to incorporate appropriate symmetries.", "questions": "1. **Related to W1** - What is the element of novelty in your exposition of incorporating distributional priors? If there is not any, this is not a problem, but then the relevant related works should be cited and the paper should indicate this clearly.\n2. **Related to W3** - Can you also provide error bars on all results?\n3. **Related to W6** - Could you comment on the extra cost of introducing the physical constraints using finite difference methods?\n4. **Alternative objective** - For the distributional prior, have you ever tried a different objective that is a mix between the noise and data objective (for example, v-prediction [Salimans et al. [4]])? \n5. For the PDE datasets, why don’t you provide the RMSE of the PDE constraints on all datasets?\n6. From my understanding, you only impose either the momentum conservation constrain or the energy conservation constraint. What if in the particle dynamics datasets you imposed both a conservation of momentum and a conservation of energy penalty? Would this lead to instabilities?\n7. When employing finite difference methods to approximate the differential equations, how do you make sure that the spatio-temporal discretisation of the datasets is fine enough to yield significant results? Is the discretisation the reason why the RMSE of the PDE constraints in Table 2 is not closer to 0?\n8. Could you please include in the captions of Table 4 and 5 the quantity that you are reporting (I am assuming Traj error)?\n9. Isn’t the result in Table 4 for Shallow Water with distributional prior the same as the result in Table 2 Shallow water? If so, shouldn’t it be 8.151 instead of 8.150? And for the three-body dataset, shouldn’t it be 2.5613 as in Table 3 instead of 2.6084, or is there a difference between those settings?\n\n[4] Salimans, T., & Ho, J. (2022). Progressive distillation for fast sampling of diffusion models. arXiv preprint arXiv:2202.00512.", "limitations": "", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. **Relevant topic.** There is increased interest in the dynamical system modelling community to incorporate relevant priors into models. Not only would this help with more efficient training, but perhaps more importantly, it should lead to better stability and capabilities to generalise out-of-distribution (OOD - when the test data comes from a different distribution than the training data).\n2. **Wide range of datasets.** The experiments are performed on a wide range of datasets - four PDE ones, and two particle dynamics datasets. This helps at showcasing how the techniques would be employed in various scenarios, studying both linear and multilinear PDE constraints in the PDE datasets, as well as (reducible and general) nonlinear ones in the particle dynamics datasets.", "weaknesses": "1. **Contribution in distributional prior incorporation.** I believe there is some related work that is not being mentioned in section 3.1, as there are now plenty of works studying the incorporation of relevant priors in diffusion models. For example, Mathieu et al. [1] incorporate a series of geometric priors in infinite-dimensional modelling, also focusing on conditions for the diffusion process to be G-invariant assuming that the score network is G-equivariant.\n2. **Lack of empirical evidence from the first methods subsection.** I do not think that the theory behind the first Methods subsection (Incorporating distributional priors) is sufficiently evidenced empirically.\n- In the PDE datasets, the only result used empirically is that the data matching objective should be used, but this is rather weakly supported theoretically with the smoothness argument (although evidenced empirically in Table 4).\n- In the particle dynamics datasets, I understand that finding the right architecture that incorporates all the relevant inductive biases can be hard, but if in the end data augmentation was used, how is the theory supported empirically in this case? I think you should include a discussion on this, as well as more details on how data augmentation was performed. \n- Also related to the particle dynamics dataset, it seems that for the five-spring dataset you are using the EGNN architecture, (but you do not cite it - I am assuming you are referring to Satorras et al. [2]). I think this should be stressed in the main text with a brief discussion on the choice of architecture (referring to the inductive biases it incorporates). It's an important component given the chosen topic and I do not understand why it wasn't more clearly emphasised. And why are you using it in the five-spring dataset, but not in the three-body one?\n- Finally, the section on ECM from Methods is not referenced later on in the paper. From my understanding, its purpose is to justify the data augmentation approach (in which case this connects to the previous point). I think a clearer connection between methods and how they are applied should be made.\n3. **Lack of error bars.** The results do not contain any sort of error bars, which actually makes it very difficult to assess the effectiveness of the proposed methods. Especially in the PDE datasets results, the differences between the mean RMSEs are fairly small, and, depending on the errors, the distinction between the method w/o and w/ prior might actually turn out not to be significant. Could you please include them?\n4. **Inadequate sample quality.** For the Darcy flow samples, the quality is very poor. I understand that no super-resolution or denoising procedures have been employed, but it seems like there is too much noise left in the samples. Do you take a final denoising step after the end of the diffusion process (by applying Tweedie’s formula)? Do you normalise the data? I can see that the data values for Darcy flow (with poor quality) are much lower (0.025-0.150) vs. Burgers (0.0-1.0), but the last sigma value is probably the same, so the proportional influence of the noise is higher in the Darcy flow dataset, potentially leading to poorer quality. \n5. **Effectiveness of general nonlinear constraints.** The results on general nonlinear constraints are not that promising. In the three-body dataset, the results for noise matching + conservation of energy (general nonlinear) are very much comparable (especially for certain hyperparameters) to plain noise matching (although this is hard to assess because of the lack of error bars) - see Table 8. In the five-spring dataset (Table 9), the improvement is once again marginal for general nonlinear. Therefore, although I liked the idea of utilising the same hidden states as the model, I would argue that it did not prove to be effective in these experiments. If the authors agree, I think this limitation should be highlighted properly in the paper.\n6. **No discussion on extra computational cost.** The paper does not discuss anything about the extra computational costs of the proposed method.\n\n**Minor**\n\n7. **Lack of thorough details/references on architecture.** In Table 7, I believe the Karras Unet should be referenced (from the EDM [3] paper). The paper also mentions that in the five-spring dataset the EGNN architecture is used but no reference is given (Satorras et al. [2]) and maybe the acronym should be defined for completeness.\n8. **Normalised metrics.** For easier comparison between datasets, it might be better to use normalised metrics rather than absolute ones.\n9. **Small legend in graphs and typos.** The font of the legends in the plots is too small (see Figure 2, Figure 8, Figure 9, etc.). There are also several typos throughout the paper (L107 extra “of”, L114 should be $\\alpha_T$ and $\\sigma_T$?, L271 related to the absolute, etc.). \n10. **Inconsistent notation for score.** Sometimes you specify the score as $\\nabla_{\\mathbf{x}} \\log q_t(\\mathbf{x})$, sometimes as $\\nabla_{\\mathbf{x}} \\log q_t(\\mathbf{x}_t)$.\n\nOverall, I think that the paper contains some nice ideas, but fails to clearly connect them to the empirical evidence, the experiments are poorly presented (with lack of error bars) and the effectiveness of some cases seems overstated.\n\n[1] Mathieu, E., Dutordoir, V., Hutchinson, M.J., Bortoli, V.D., Teh, Y.W., & Turner, R.E. (2023). Geometric Neural Diffusion Processes. ArXiv, abs/2307.05431.\n\n[2] Satorras, V.G., Hoogeboom, E., & Welling, M. (2021). E(n) Equivariant Graph Neural Networks. ArXiv, abs/2102.09844.\n\n[3] Karras, T., Aittala, M., Aila, T., & Laine, S. (2022). Elucidating the Design Space of Diffusion-Based Generative Models. ArXiv, abs/2206.00364.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "61hGXB72jA", "reviewer": "Reviewer_2sjz", "summary": "This paper proposes two ways to incorporate physical priors into generative models: (1) they inject distributional priors by choosing the proper equivariant models and (2) they incorporate physical feasibility priors by decomposing nonlinear constraints into elementary cases. The key is to identify elementary cases where Jensen's gap can be omitted and decompose complicated constraints into elementary cases. In general, this paper is a good contribution to the AI4Science community, where many scientific priors are available and should be incorporated into learning systems.", "questions": "* Why does performance improve only incrementally after adding priors?\n* In Equation (4) and line 205, are there typos? \\nabla x -> \\nabla_x?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* The motivation is clear - this paper is a clear contribution to the AI4Science (esp AI4Physics) community where data-driven models should be combined with scientific inductive biases. Although physics-informed learning has been mainstream in scientific machine learning, physics-informed learning in the context of generative modeling is relatively new. \n* This paper is both theoretically (supported by theorems) and practically sound (supported by experiments). The idea of decomposing a complex constraint into elementary cases is a good strategy.", "weaknesses": "* Empirical improvement is marginal (Table 1, 2, 3).\n* This framework is useful when (1) we don't understand the system fully but (2) we know some partial information (existence of energy etc). However, all the examples in the paper are synthetic, so we know both the underlying equation and energy but \"pretend\" that we don't know about the equation. I understand this is only a proof of concept, but a more realistic example would greatly strengthen the paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)