Text2PDE: Latent Diffusion Models for Accessible Physics Simulation
Authors: Anthony Zhou
Zijie Li
Michael Schneier
John R Buchanan Jr
Amir Barati Farimani
Venue: ICLR 2025 Poster (2025)
Keywords: AI4Science
PDE
Neural Operator
Latent Diffusion
Text2PDE
Primary area: applications to physical sciences (physics, chemistry, biology, etc.)
TL;DR: We develop novel methods for text-conditioned physics simulation that are accurate and efficient.

Abstract: Recent advances in deep learning have inspired numerous works on data-driven solutions to partial differential equation (PDE) problems. These neural PDE solvers can often be much faster than their numerical counterparts; however, each presents its unique limitations and generally balances training cost, numerical accuracy, and ease of applicability to different problem setups. To address these limitations, we introduce several methods to apply latent diffusion models to physics simulation. Firstly, we introduce a mesh autoencoder to compress arbitrarily discretized PDE data, allowing for efficient diffusion training across various physics. Furthermore, we investigate full spatiotemporal solution generation to mitigate autoregressive error accumulation. Lastly, we investigate conditioning on initial physical quantities, as well as conditioning solely on a text prompt to introduce text2PDE generation. We show that language can be a compact, interpretable, and accurate modality for generating physics simulations, paving the way for more usable and accessible PDE solvers. Through experiments on both uniform and structured grids, we show that the proposed approach is competitive with current neural PDE solvers in both accuracy and efficiency, with promising scaling behavior up to $\sim$3 billion parameters. By introducing a scalable, accurate, and usable physics simulator, we hope to bring neural PDE solvers closer to practical use.

Published as a conference paper at ICLR 2025
TEXT2PDE: LATENT DIFFUSION MODELS FOR ACCeSSIBlE PHYSICS SIMULATION
Anthony Zhou, Zijie Li & Amir Barati Farimani Carnegie Mellon University (ayz2, zi zijieli, afariman wandrewnwaadrew.mmm edu
Michael Schneier & John R. Buchanan, Jr. Naval Nuclear Laboratory (michase schneier, wtw.:aaaa.aaaaaa... www.aaaaaaannnnppp.coo
ABSTRACT
Recent advances in deep learning have inspired numerous works on data-driven solutions to partial differential equation (PDE) problems. These neural PDE solvers can often be much faster than their numerical counterparts; however, each presents its unique limitations and generally balances training cost, numerical ac- curacy, and ease of applicability to different problem setups. To address these lim- itations, we introduce several methods to apply latent diffusion models to physics simulation. Firstly,
introduce a mesh autoencoder to compress arbitrarily dis- cretized PDE data, allowing for efficient diffusion training across various physics. Furthermore, we investigate full spatio-temporal solution generation to mitigate autoregressive error accumulation. Lastly, we investigate conditioning on initial physical quantities, well as conditioning solely on a text prompt to introduce text2PDE generation. We show that language can be compact, interpretable, and accurate modality for generating physics simulations, paving the way for more usable and accessible PDE solvers. Through experiments on both uniform and structured grids, we show that the proposed approach is competitive with current neural PDE solvers in both accuracy and efficiency, with promising scaling behav- ior up to ~3 billion parameters. By introducing scalable, accurate, and usable physics simulator, we hope to bring neural PDE solvers closer to practical
INTRODUCTION
Neural PDE solvers are an exciting new class of physics solvers that have the potential to improve many aspects of conventional numerical solvers. Initial works have proposed various architectures to accomplish this, such as graph-based (Li & Farimani, 2022: Battaglia et al., 2016). physics- informed (Raissi et al., 2019), or convolutional approaches (Thuerey et al., 2020). Subsequent work on developing neural operators (Li et al., 2021; Lu et al., 2021; Kovachki et al., 2023) established them as powerful physics approximators that can quickly and accurately predict PDEs, and recent work has focused on improving many of their differrnnt aspects. These advances have established different models that can adapt to irregular grids (Brandstetter et al., 2023; Li et al., 2023a;c; Wu et al., 2024), speed up model training (Li et al., 2024a; Alkin et al., 2024; Tran et al., 2023; Li et al., 2024b; 2023b), achieve more accurate solutions (Lippe et al., 2023), or generalize to a wide variety of parameters (MeCabe et al., 2023; Hao et al., 2024; Herde al., 2024; Zhou et al., 2024).
Despite these advances, there are still many factors that limit the practical adoption of neural PDE solvers. Great progress has been made in addressing the accuracy and generalizability of neural solvers, however, deploying these advances will require further work in developing new engineering tools. Although neural solvers have become highly capable, they have also become highly spe- cialized; increasingly, the practical use of neural solvers is limited by their inaccessibility and the complexity of the field. Deep learning has already enabled vision and language tools designed with productivity and usability in mind; is there a way we can create similar tools for engineers?
"Corresponding Author.

---

Published as a conference paper at ICLR 2025
passes over cylinder position: 6.14, enters velocity of Reynolds
"Plume 1: Hook-like shape bulbous top tapored botom. Large, upper corner, extending top center-left. Plume Elongated oval pill-shape, slightly curver. Large, right side, excending from bottom Plome 3: Hook-like shape, miroring first plume but inverted. Large, lower lell curner. extending from botiom to center-ief"
"Air Bows over an obstucle resembling a low step at the bettom balf. spanning the width of the demaai and over quarter the beight of the domain..
Figure We introduce latent diffusion models for physics simulation, with the remarkable ability of generating an entire PDE rollout from a text prompt. Three generated solutions are displayed with their model inputs.
Text2PDE Models In this work, we take step towards creating more accessible physics simula- tors by introducing language as novel modality to interact with physics solvers, creating tex12PDE models. This has both benefits and drawbacks. One potential benefit is that language extremely general: while different physical phenomena usually require problem-specifh, meshing and solvers, humans constantly observe and interact with physics and, as result, build a common understanding and vocabulary for describing these phenomena. This could reduce the barrier to entry for adopting neural solvers by allowing practitioners to express ideas in language to prompt a physics model. In- deed, the popularity of text2image models is partly due to the ease of expressing visual ideas in lan- guage, rather than, for example, needing the computer vision expertise to query a class-conditional image generation model with ImageNet label. Beyond improving accessibility, language have technical benefits. As long as the physical phenomena can be described in text, its domain specialties can be abstracted away, such the discretization, geometries, boundaries; this allows models to work with just a single modality. In addition, language can be compact: rather than need- ing raw physical data to specify simulation, this information can be distilled into dense textual descriptions.
also
However, text2PDE models present unique challenges. Since text can describe a broad set of physics, the text2PDE model must generate arbitrarily discretized results. Furthermore, produc- ing high-dimensional spatio-temporal physics simulations from a low-dimensional text prompt can be challenging. Lastly, language certain cases, specifying exact initial conditions may be preferred. Indeed, we envision tex12PDE models not replacement for current methods, but rather leverage. We hope to empower engineers with a diverse toolkit, where different neural solvers and numerical methods can be used together throughout the design eycle to better ideate, simulate, and realize engineering outcomes.
be imprecise in describing many physics problems, and in
additional tool for engineers to
Contributions To work towards this goal, we introduce methods to use latent diffusion models to generate PDE solutions. Within this framework, we generate an entire solution trajectory at once to avoid autoregressive eтrоr propagation and improve accuracy. We also develop novel autocncoder strategy to efficiently train/query models and to adapt latent diffusion arbitrary discretizations. We also investigate conditioning on either text or physics modalities, such as the initial frame of a solution, to switch between an interpretable interface and more precise one. Lastly, we seale latent diffusion models up to nearly 3 billion parameters to show their ability to accurately generate complex outputs from compact inputs. We believe that this scalability, accuracy, and usability can unlock generative models that can bring neural PDE solvers closer practical adoption. To further this objective, all code, datasets, and pretrained models are released at: https: / /ggitub. com/ anthonyzhou-11 pdee
2

---

Published as a conference paper at ICLR 2025
2
BACKGROUND
2.1
PROBLEM SETUP
We consider modeling solutions to time-dependent PDEs in time dimension t [0, 7 and F2,, T2, P2,........ IDI" E Ω. Following Brandstetter et al. (2023), we
multiple spatial dimensions x = [x1,
express these PDEs in the form:
a,u F(I,,, u, U, U,..
E [o,T]
Rd is a quantity that is solved for with physical dimension dz. Furthermore, the PDE can be subject to initial conditions u (x) and boundary conditions (t,) 0:
Where [0, x2 >
u(0.x) (x) (t, x) = 0 EE (0, ON
XE
(2)
(3)
In general, the solution u is discretized timestep to obtain xm)) collocated M mesh points Xm, C Ω for m E current timestep predict the solution at future timestep u(t + At, xm)
M). Neural solvers aim to solve time-dependent PDEs by
using the solution fo(u((, Xm)), Many variants of this setup exist, whether it is to use multiple input timesteps or predict multiple output timesteps. Despite these variants, the dominant approach still follows numerical solvers by autoregressively advancing the solution in time, with certain notable exceptions (Lienen al., 2024; Du et al., 2024).
Autoregressive solvers are natural choice for neural solvers; they follow numerical solver prece- dents, as well as reflect the Markovian nature of time-dependent PDEs. However, this paradigm has many potential drawbacks, namely the stability of autoregressive solvers (Lippe et al., 2023). Al- though many approaches have been developed to mitigate autoregressive error accumulation (Brand- stetter et al., 2023), the fundamental problem still exists: complex PDEs require small timescales to accurately resolve physical phenomena, yet this requires longer rollouts which increase error aceu- mulation (Lozano-Da Durán & Jiménez, 2014; 14; Lienen et al., 2024). To address this issue, we leverage spatio-temporal diffusion to directly predict an entire solution trajectory. This becomes a generative, rather than autoregressive, problem conditioned information about the desired PDE solution.
2.2 SPATIO-TEMPORAL DIFFUSION FOR PDES
We consider modeling the conditional probability distribution of the discretized solution u e RT xx at all timesteps i [0, and spatial coordinates Xm. Ω given the initial condition u and boundary condition B[u]:
Pe ( uu/u" B(u]) ss p(ulu", Blu))
(4)
The distribution is approximated using a denoising diffusion probabilistic model (DDPM) Po (Ho al., 2020; Sohl-Dickstein et al., 2015). DDPMs act reverse noising process q(x, [x-1) that transforms a PDE roll-out = Xo over N steps into Gaussian noise (i.e. 9(xx Xo) sJ N/ f)). New PDE solutions can then be generated by sampling noise vector x ~ ~N(0, I) and iteratively approximating the denoising distribution pa (x,-1 (x,). While this generates a random sample from the distribution of all PDE samples (Xo ~ p(u),, the trajectory of single solution u is highly depen- dent its initial condition u and boundary condition B(u]. Therefore, the conditional denoising process Po (x,-1 xx, u", B (u) is used to model the conditional distribution Equation 4.
Latent Diffusion Since u can be extremely high-dimensional, we perform the forward and re- verse diffusion process in learned latent space. More specifically, given a solution u E 7' xd,, where T represents the temporal resolution, M represents the spatial resolution, and dp is the phys- ical dimension, an encoder E can embed u into a latent vector z = Eu)). Importantly, z is in a compressed latent space I lxxd, where L is latent and d, is the latent dimension. To project latent vectors back to the physical space, a decoder D is used to reconstruct sample D(z).
resolution smaller than T x M,
3

---

Published as a conference paper at ICLR 2025
q
ZT
negriegate
Forward Noising
Conditioning
E
Denoisingx
Abc
X
q
Zrr
interpolate
D
Croe-Art
Sel-
To
Patchify
wlsLN
Figure 2: The proposed architecture, Samples are mapped to a grid through a learned aggregation before being encoded to latent vector and noised. A denoising process is learned, with conditioning from text or physics- based modalities. Denoised latents decoded grid and mapped mesh through learned interpolation.
METHODS
In this section, we describe the main components of using latent diffusion models for physical simulation. An overview of the methods can also be seen in Figure
3.1 AUTOENCODERS FOR PDE DATA
Practical physics applications often discretize solutions on an unstructured mesh, which helps im- prove the accuracy regions of interest. This be difficult for conventional autoencoders to accommodate, and indeed much of the previous work is focused on compressing uniformly struc- tured images, videos, PDE solutions (He et al., 2021; Feichtenhofer et al., 2022; Zhou & Farimani, 2024). Therefore, we propose an autoencoder that can compress data given at arbitrary collocation points and reconstruct them arbitrary query points to extend diffusion models unstructured data.
Mesh Encoder Following the problem setup, at every timestep [0, we can consider M coordinates C Ω for m E {0, ,. M). defined. This framework captures solutions defined at any point in space and time. We map input timesteps 4, coordinates Xm. and solutions u(4, xm) onto Ti uniform timesteps 4 E [0. Ti). Mi uniform grid coordinates X; C l for f0, 10. Mr]. and latent vectors q xi) E Rd at each uniform spatio-temporal coordinate. This is done with the following kernel integral:
where a solution vector u(t, xm) E Rd is also
a(4,, xI) = 10.77x0 k(l,, X1, x)u(t, x)dtdx
(5)
In practice, following Li al. (2020) and et al. (2023c), the domain of integration truncated to spatio-temporal ball with radius centered at & and defined as (t,, which enforces constraint that latent vectors can only depend on local solutions and coordinates. Furthermore, the kernel is parameterized with network that computes kernel value given physical coordinate y = concat(t, x) and latent coordinate y: = concat (f,, x)). Lastly, the integral is approximated by Riemann sum the M, M physical coordinates that lie within the ball B, (y:), denoted as yb C [0, 7] x Ω for b E {0, 1, 1,..,,M,6 These approximations result in the following expression:
Mr
q(yi) = - k(y),))u(yddy Σ
6
B.(y)
where p(y)) is the Riemann sum weight for every point yb in the ball By)y.). This kernel integral essentially aggregates neighboring physical solutions for every latent coordinate and has the nice property approximating an operator mapping from the input function u(y) to the latent function q(y:). Additional details and visualizations of this kernel integral are provided in Appendix B.3.
4

---

Published as a conference paper at ICLR 2025
leverage con- ventional convolutional neural network (CNN) autoencoders to compress and reconstruct the data, which consists of various convolutional. upsampling, downsampling, and attention layers. Specif- ically, given a latent grid q E IR xM,,,, the CNN encoder will downsample the resolution by a factor of f to produce a latent vector z E IgT:/fxM./xxd where di is the latent dimension. The CNN decoder reverses this process decode a latent grid qa from the latent vector z.
Convolutional Backbone Once uniform latent grid q(y:) is calculated,
Using this method allows our autoencoder to benefit from architecture unification, where best prac- tices from separate domains can benefit PDE applications. In particular, can leverage variational or vector quantized latent spaces (Kingma & Welling, 2022; Esser et al., 2021) to avoid arbitrarily high-variance latent spaces during training. Furthermore, improve reconstruction quality, the au- toencoder can benefit from incorporating generative adversarial network (GAN) or perceptual loss metric (Rombach et al., 2022; Zhang et al., 2018). We leave further details and ablation studies of these modifications to Appendix B.I. As a summary, we find that GANs and perceptual guidance can in certain cases improve reconstruction performance: however for simplicity and to ensure the broad upplicability of our framework, we omit them in our main results since they require extensive optimization and are highly woolemmddpeedeer Furthermore, we find that highly regularized latent spaces reduce downstream diffusion performance due to lower reconstruction accuracy, and that dif- fusion backbones are sufficiently parameterized to model higher-variance variance latent spaces. As a result, main results use small Kullback-Leibe (KL) penalty during autoencoder training.
Mesh Decoder To move from the latent grid to the physical mesh, the kernel integral can be reversed. Specifically, given a decoded latent grid qa(y:). the decoded solution can be calculated by the Riemann sum: uyl = Eb Mr s(y. (v)))()))))((s)) Note that in this case, the sum is evaluated over the ball (y), and the coordinates ys are defined as the latent grid points yr that lie within (y). This reverse process essentially aggregates neighboring latent vectors for every physical coordinate, and also has the property of being discretivationninvariin latent solution can be decoded onto arbitrary mesh by evaluating the Riemann sum at arbitrary query points y. shown in Figure 8. Lastly, for PDE problems that are uniformly discretized, the kernel integration can be omitted and CNN autoencoder can be directly used.
In particular, the
Comparison to GNNs and Neural Fields To demonstrate the utility of our method, we draw comparison with previous work on processing irregular PDE data, namely using GNN or neural field approaches (Pfaff et al., 2021; Cao et al., 2023; Immordino et al., 2024; Yin et al., 2023; Serrano et al., 2024). Although these works do not explicitly train autoencoders, we extend these works construct GNN- and neural field-based autoencoders to benchmark our proposed method. We provide additional details on these alternative methods as well as present auxiliary results in Appendix B.2. As a summary, we believe that the lack of inductive bias in GNNs and neural fields makes it challenging to aggressively compress and reconstruct unstructured data, which already have very little bias. The proposed approach benefits from the locality constraints of the kernel integral as well as the translation invariance, compositionality, and local connectivity of CNN kernels.
3.2 LATENT DIFFUSION
Once latent space has been learned, we consider freezing the encoder and decoder to learn the latent distribution p(z). Specifically, we train a denoising model eg n)), where E {0, I,..., ,N} is an intermediate step within the Gaussian noising process of length N, and z, is the noised latent at step n This model is trained by minimizing a reweighted variational lower bound on p(z):
L = E A |le - ce (Z,, 112- Since the latent zo is encoded from uniform latent grid, we can leverage a Unet to parameterize eœ (Ronneberger et al., 2015). However, recent work by Peebles & Xie (2023) suggests that this inductive bias is not necessary and we evaluate the use of diffusion transformer (DiT). We find that the DiT backbone outperforms the Unet backbone; the main results reported with a DiT implementation, but comparison can be found in Appendix C.2.
Additionally, recent advances in diffusion frameworks have proposed many modifications to the de- noising process, the most straightforward of which is scaling the latent space (Rombach et al., 2022).
We deviate from the usual notation of t representing the noising timestep to avoid overloading it with t which is previously defined as physical time of the PDE solution unff, x).
5

---

Published as a conference paper at ICLR 2025
Furthermore, Nichol & Dhariwal (2021) has shown the potential benefits of using cosine noise scheduler or learning the variance _g(zn: n) of the reverse process. Lastly, Salimans & Ho (2022) introduced reparameterizing the reverse process to predict the velocity with on defined as in Ho et al. (2020). We implement and evaluate the effects of these modifica- tions; for interested readers, the findings can he found in Appendix C.I. In general, we find that these modifications can have varying effects on the performance of our model; however, we leave optimizing the diffusion process to future work. For our main results, we scale the latent space, use a linear noise schedule, do not learn the variance, and use an parameterization.
va,, v1 Xn,
3.3 CONDITIONING MECHANISMS
To sample from the conditional distribution p(ulu", Blul), we add the conditional information to the denoising model eø &n, n,, u", B[ul]. For current PDE benchmarks and in many practical applications, the boundary conditions do not change between solutions; as such, we use u(0, xm.) as a proxy for both u" and B Biu]. As such, conditioning information can be added to the denoising backbone by defining a domain-specinf encoder тæ that maps the first frame of the solution to conditioning sequence length, and de is the conditioning dimension. Conveniently, the proposed encoder & is agnostic to the temporal and spatial discretization of PDEs; therefore, the same architecture can be used for t by flattening the downsampled latent dimension (Ti/f x Mi/ = No).
First Frame Conditioning
Tp (u(0, xm)) E N.x,, where No is the sequence
Text Conditioning Another interesting conditioning domain is natural language. Although im- precise, humans describe and understand physics through language, and in principle, the initial and boundary conditions of a physics simulation could also be described by text. An advantage of this modality is that it is interpretable and compact; instead of needing to specify the discretization scheme and the physical values at every mesh point, only the important behavior driving the phys- ical phenomena need to be described. However, clear drawback is the underdetermined nature of the PDE problem: for given prompt describing the initial and boundary conditions, there may exist many plausible solutions. Furthermore, this approach introduces the problem of captioning physics simulations; we develop additional methods to achieve this and interested readers are directed to Appendix D for details on PDE captioning.
Regardless, text-conditioned PDE simulators are a promising direction and to achieve this, we pro- pose text-specific encoder To based on pretrained transformer encoders (Vaswani al., 2023; Devlin et al., 2019). Specifically, given a tokenized and embedded prompt p E IR Ned, with se- quence length &
and dimension de transformer encoder can produce conditioning sequence (p))) c E R. N.d., This has the additional benefit of leveraging large, pretrained transformer encoders: in particular, we fine-tune RoBERTa (Liu et al., 2019) to accelerate the learning of rg.
Implementation After defining two conditioning modalities and their corresponding encoders To we consider mechanisms for incorporating the conditioning sequence с into the denoising back- bones. For the Unet backbone, pross-attention layer can be used between flattened, hidden Unet representation vz(z,) and the conditioning sequence e (Rombach et al., 2022). For the DiT back- bone, we apply mean pooling to produce غ mean(e) E dd in order to apply the adaLN-Zero conditioning proposed by Peebles & Xie (2023). Since mean pooling provides only global infor- mation, also insert cross-attention layer after self-attention DiT layers to allow hidden DiT representations (z,... to attend to the conditioning sequence c. (Chen et al., 2024).
A final consideration is the use of classifier- free guidance to yield tradeoff between sample diver- sity and quality (Ho & Salimans, 2022). For PDEs, there is only one solution for a given initial and boundary condition; as result, the desired behavior is enhance the sample quality at the cost of diversity. We study this effect for the conditional generation of PDE samples and report findings in Appendix D.4. We find that different weights w only noticeably affect generated samples at their extrema (w 2 0, w > 10); as result, classifier-free guidance is omitted from our main results.
4 EXPERIMENTS
We explore the proposed latent diffusion model (LDM) for three PDE datasets: 2D flows around a cylinder (Pfaff et al., 2021), 2D buoyancy-driven smoke flows (Gupta & Brandstetter, 2022), and 3D>

---

Published as a conference paper at ICLR 2025
Text
Reference
Model
Params
Tílops
L2 Loss
GINO
72M
0.73
0.2445
MGN
101M
32.16
0.2617
OFormer
131M
17.34
0.3386
LD
198M
0.81
0.1522
FF
667M
1.16
0.1309
LDMs- 313M
0.83
0. 0.1796
LDMM-Text
804M
1.20
0. 0.1476
Table 1: Cylinder Flow, Left: Model parameters. flops, and validation losses across different baselines. Right: Sample rollouts for first frame and text conditioned models compared to the reference solution.
turbulence around geometric objects (Lienen et al., 2024). The cylinder flow dataset is discretized on a mesh to smoothly model the cylindrical geometry and its wake; however, the smoke buoyancy and 3D turbulence datasets are uniformly discretized. Additional dataset details be found in Appendix For each model, we report its parameter count and relative L.2 loss. Since parameter count can be poor measure of training cost and model complexity (Peebles & Xie, 2023), additionally report the floating point operations per second (flops) needed for single forward pass during training with batch size of calculated using DeepSpeed (Rasley al., 2020). Additional details on training and model hyperparameters can be found in Appendix F.
4.1 CYLINDER FLOW
Dataset Following Pfaff et al. (2021), we use 1000 samples for training and 100 samples for validation. Each sample contains approximately 2000 mesh points and is downsampled to evolve over 25 timesteps, which is compressed to a latent size of 16 x 16 x 16. Each sample is varied the position and radius of the cylinder, well in the inlet velocity of the fluid. This results in samples with Reynolds numbers of 100-1500 and includes Karman vortex streets (Stringer et al., 2014b). In addition, the physical variables velocity and pressure. Lastly, caption this dataset, extract the cylinder position, radius, and Reynolds number and procedurally generate prompts based on a template; we include further details and ablation studies in Appendix D.I.
Results We consider benchmarking against GINO (Li (Li al., 2023c), MeshGraphNet (MGN) (Pfaff al., 2021), and OFormer (Li et al., 2023a), which represent a variety of operator-, graph-, and attention-based neural solvers. Since these are autoregressive models, we provide the initial frame of the solution and autoregressively predict the next 24 frames. We compare this to our latent diffusion model conditioned on both the first frame (-FF) and a text prompt (-Text), which trained to generate a full solution trajectory at once. Furthermore, we consider two model sizes: Small (S) and Medium (M); the results shown in Table with relative L2 loss.
We observe that the latent diffusion model can outperform current models on this benchmark and pressure errors are around the same velocity errors. Notably the inclusion of flops allows for additional insight into model efficiency. We reproduce results showing that GINO models are very efficient (Li et al., 2023c), yet LDMs still match this by using latent space and diffusing entire rollout at once. Furthermore, our model can achieve lower errors than baselines and more efficiently than graph- attention-based neural surrogates. Lastly, conditioning on the first frame outperforms text conditioning, presumably because more information is provided in the initial ve- locity and pressure fields. Text remains an accurate conditioning modality; moreover, the latent diffusion model shows good scaling behavior, with increased performance at larger model sizes.
4.2 BUOYANCY-DRIVER FLOW
Dataset We additionally evaluate our model on regular-gnn grid benchmark of conditional buoyancy- driven smoke rising in square domain (Gupta & Brandstetter, 2022). We use 2496 training samples and 608 validation samples, with varying initial conditions and buoyancy factors, and at a spatial resolution of 128 128 and 48 timesteps, which is compressed latent dimension of 6 x 16 x

---

Published as a conference paper at ICLR 2025
FF
Reference
Text
Re-Salved
Model
Params
Tflops
L.2 Loss
FNO
510M
0.85
0.5126
Unet
580M
21.48
0.3050
Dil-Resnet
33.2M
58.75
0,4466
JULL
ACDM
404M
51.23
0.4766
LDMs-FF
243M
1.41
0.3459
LDMM -FF
725M
1.68
0.3177
LDML -FF
1.55B
2.19
0.2728
LDMs- Text
334M
1.39
0.4290
LDMM-Text Text
825M
1.65
0.3158
LDM2 TTxt
2.69B
2.53
0.2944
Table 2:
Buoyancy-Drives Flow, Left: Model parameters, flops, and validation losses. Text- conditondi losses" are evaluated after re-solving the ground truth. Right: Sample rollout; variations in generated initial states of text models can accumulate large deviations, such, they evaluated against a re-solved trajectory.
16. The smoke density is modeled by a passive transport equation, and as result, the physical variables velocity and density. Lastly, for the smoke buoyancy problem, is more challenging to describe an initial condition with text. In particular, plumes may exhibit different shapes, locations, and sizes, in addition to the initial velocity field being even less visually distinct. Nevertheless, we investigate captioning the initial condition using a multimodal LLM. We design prompt and provide the density field an image; additionally, we outline individual smoke plumes in the initial density field using a Canny edge detector (Canny, 1986) and provide this segmented image in the LLM prompt as well. For additional details and examples, we refer readers to Appendix D.2.
Results We consider benchmarking against the FNO, Unet, and Dilated Resnet architectures (Li et al., 2021; Ronneberger et al., 2015; Gupta & Brandstetter, 2022; Stachenfeld et al., 2022). Ad- ditionally, we benchmark against an autoregressive diffusion model (ACDM) (Kohl et al., 2024), to evaluate the need for latent space and full spatio-temporal sample generation. To ensure fair comparison, we provide these baselines with the initial solution and autoregressively predict the next 47 frames. We compare this to our LDM model, conditioned on the first frame (-FF) or the text prompt (-Text), and in different model sizes: Small (S), Medium (M), and Large (L), in Table
Due to the underdetermined nature of text conditioning, propose additional evaluation metric. This is motivated by the observation that text-conditioned diffusion models often generate a variety of initial conditions that semantically satisfy a given prompt, yet this affect the temporal rollout significantly. Although the sampled solution may not match specific solution in the validation set, it still fulfills the semantic goal of text conditioning. Therefore, sample solution for each text prompt in the validation set, then use the sampled initial condition to produce a ground-truth using PhiFlow (Holl & Thuerey, 2024), which would evaluate the physical consistency of generated solutions without restricting the diversity of the LDM simulator. The losses using this re-solved trajectory reported in Table 2, with additional visualizations in Appendix A.
We observe that LDM models conditioned on the first frame of PDE solution can outperform current models and are an efficient class of neural solvers. Additionally. density errors are around 1.5 times the velocity errors. Furthermore, text-conditioned diffusion models are able to model a set of phenomena that both semantically satisfy the prompt and are physically consistent, although the samples may not match the validation set. In addition, many text-conditioned samples are not contained in the training or validation sets, indicating that the generative model has learned the un- derlying physics to generate new samples based on text prompt. Furthermore, for this benchmark, the initial conditions are generated from a Gaussian distribution, which is challenging to describe with prompt and real-world problems often contain more structure that natural language can better describe. Lastly, the LDM model displays good scaling behavior, with efficient training at large parameter counts.
8

---

Published as a conference paper at ICLR 2025
Reference
Model
Params
Tílops
L2 Loss
DTKE
FNO
1.02B
1.62
0.862
6.524
FactFormer
41.4M
53.8
0.795
6.022
PF
Dil-Resnet
24.8M
249
0.707
6.153
LDM-FF
2.72B
9.78
0.602
5,630
LDM-Text
2.73B
9.43
0.693
5.653
Text
Table 3D Turbulence. Left: Model parameters, flops, validation losses and log-TKE distance. Right: Sample rollouts for first frame and text conditioncd models compared to the reference solution.
4.3 3D TURBULENCE
Dataset To investigate scaling behavior to larger, more complex systems, we consider evaluating the LDM framework on 3D turbulence problem of air flows over various geometric objects such as squares, elbows, U-shapes, etc. (Lienen et al., 2024). We use 36 training samples and 9 validation samples, downsampled to a resolution of 96 x 24 x 24 and 1000 timesteps; however, we set the prediction horizon to 48 timesteps during evaluation. The average Reynolds number is 2e5 and the physical variables are velocity and pressure. Additionally, the simulations are manually captioned based on the geometry of the initial condition; we include additional details in Appendix D.3.
Results We consider benchmarking against FNO3D (Li et al., 2021) as well as FactFormer (Li et al., 2023b), a scalable attention-based neural solver, and Dil-Resnet (Stachenfeld al., 2022), an accurate yet computationally expensive CNN solver. In addition to relative L2 loss, we report the L2 distance between the log turbulent kinetic energy DIKE between predicted and true samples, since chaotic and small-scale fluctuations in turbulence make it unlikely for similar samples to be close under Euclidean distance when evolved over time (Lienen et al., 2024). is currently challenging for any neural surrogate to accurately model 3D turbulence, however the proposed model is able to do so more accurately than baselines and remains efficient. Although first-frame conditioning is more accurate, text conditioning is still an accurate conditioning modality. Lastly, there are significant smoothing effects which seems to be the cost for temporal stability and efficient training in 3D.
4.4 DISCUSSION
Cylinder Flow I.I Loss over Time
We show that latent diffusion models can be viable approach to generating spatio-temporal physics sim- ulations. Specifically, by sampling spatio-temporal noise and leveraging the ability of diffusion models to approximate complex distributions, jectory can be generated from an initial condition. This circumvents the traditional paradigm whereby solvers need to evolve solutions based on a current timestep, and aids in increasing accuracy by prevent- ing error accumulation. We demonstrate this by plot- ting the time-dependent prediction error for various models in Figure 3. Additionally, the use of la- tent space allows this computation to be performed efficiently and with unstructured data, and the intro- duction of mesh autoencoder allows latents to be decoded where solutions need to be refined (i.e. wake regions) and onto arbitrary geometries. Lastly, the scaling behavior of the proposed model is promising and reflects similar observations of scaling diffusion and iitention-based models in other domains (Peebles & Xie, 2023; Kaplan al., 2020). The benchmarked datasets relatively small (<5000) samples; with a larger, more diverse dataset, this scaling ability could unlock unified physics surrogates across wide variety of phenomena.
0.12
OWomer
entire tra-
0.00
0.04
LDM Text LDM ET
0.00
4
12 19
Figure 3: Losses al each timestep evaluated for 10 samples. Average losses each timestep are bolded, individual sample losses are opaque.
Additionally, the introduction of text-conditioned physics simulators is an interesting and novel re- search direction. In constrained PDE formulations, text is compact and accurate conditioning

---

Published as a conference paper at ICLR 2025
modality, nearly matching first-frame conditioning in cylinder flow problems despite conditioning on around 1000x less data (~1Mb vs. ~1Kb), and with the added advantage of being more inter- pretable and usable. The diversity and case of text-conditioned physies simulators could be useful tool for engineers and designers to quickly test different ideas without needing to define geometries, discretizations, or generate initial solution. After deciding on design, the same architecture can then be used with first-frame conditioning to precisely model the resulting physical phenomena.
5 RELATED WORKS
5.1 VIDEO DIFFUSION MODELS
Many prior works have considered using diffusion models to generate videos. Initial works have focused on adapting image diffusion models to the video domain, proposing mechanisms to en- sure temporal consistency or upsample ppatioltempora resolutions (Singer et al., 2022; Wu et al., 2023; Blattmann ct al., 2023b). Many approaches have investigated directly generating videos by modifying the diffusion backbone (Ho et al., 2022b), with subsequent works improving the resolu- tion, quality, and consistency of generated videos (Ho al., 2022a; Bar-Tal et al., 2024). Within these two paradigms, an abundance of research has examined the scaling of video diffusion mod- els (Blattmann et al., 2023a), improved conditioning methods (Girdhar et al., 2023), training-free adaptation (Khachatryan et al., 2023; Zhang et al., 2023), and video editing (Esser et al., 2023).
5.2 DIFFUSION MODELS FOR PDES
Previous works on adapting diffusion models for PDEs have worked with generating individual timesteps and without the use of a latent space (Kohl et al., 2024; Lienen et al., 2024; Huang et al., 2024; Yang Sommer, 2023). To improve temporal resolution and consistency, the use of a tem- poral interpolator has also been investigated (Cachay et al., 2023). Although not directly used to generate PDE solutions, diffusion has also been investigated to refine PDE solutions (Lippe et al., 2023; Serrano et al., 2024). Previous work has also considered using diffusion to accomplish al- ternate objectives, such as unconditional generation, upsampling PDE solutions, in-painting partial PDE solutions, or recovering solutions from sparse observations (Du et al., 2024; Shu et al., 2023)
CONCLUSION
Summary We have demonstrated that latent diffusion models represent a powerful class of physics simulators. Specifically, we have developed framework for encoding and decoding arbitrary PDE data into a latent space to generate unstructured physics solutions. Furthermore, we have shown that generating full spatio-temporal solutions can mitigate error accumulation to improve accuracy. Lastly, we introduce conditioning on physics data as well as language, which is a novel modality that can be more compact, interpretable, and accessible.
Limitations A limitation of diffusion models is the increased time and computation required for inference. We seek address this using a DDIM sampler, and show that by using fewer denoising steps, diffusion models can be faster than conventional, deterministic neural solvers while main- taining accuracy in Appendix C.3. For transparency, we also compare the inference times of using DDPM sampling on the Cylinder Flow and Smoke Buoyancy problems in Appendix G. Future work could expand on this to further accelerate sampling and improve sample quality, such as distillation
techniques (Salimans & Ho, 2022; Luhman & Luhman, 2021; Meng et al., 2023) and consistency models (Song et al., 2023). An additional limitation is that the LLM captioning can sometimes hallucinate, which degrades the quality of the smoke buoyancy captions and the resulting tex12PDE model. Lastly, diffusing an entire rollout at once fixes the temporal resolution of the generated solution, whereas autoregressive models can generate arbitrarily long sequence. We present pre- liminary work to address this limitation by using the proposed LDM model autoregressively. to alleviate the need to scale the model when predicting longer time horizons, shown in Appendix C.4.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
This work was funded by Fluor Marine Propulsion, LLC under purchase order number 142474. Additionally, this research used resources of the National Energy Research Scientific Computing Center (NERSC), Department Energy Office Science User Facility using a Generative Al for Science NERSC award DDR-ERCAP-M4755 for 2024.
REFERENCES
Benedikt Alkin, Andreas Fürst, Simon Schmid, Lukas Gruber, Markus Holzleitner, and Johannes Brandstetter. Universal physics transformers: A framework for efficiently scaling neural opera- tors, 2024. URL httpp:::1.. wttpttts//aaiiit.t.pp.pp http:////gggabbb...abbbee 12365.
Omer Bar-Tal, Hila Chefer, Omer Tov, Charles Herrmann, Roni Paiss, Shiran Zada, Ariel Ephrat, Junhwa Hur, Guanghui Liu, Amit Raj. Yuanzhen Li, Michael Rubinstein, Tomer Michaeli, Oliver Wang, Deqing Sun, Tali Dekel, and Inbar Mosseri. Lumiere: A space-time diffusion model for video generation, 2024. URL ht https: /ttp///aavvv.///.///aa//eeeeaaaa .rgg wwtp://ggg/bb.aa..ee..oaae
12945.
Peter W. Battaglia, Razvan Pascanu, Matthew Lai, Danilo Rezende, and Koray Kavukcuoglu. Interaction networks for learning about objects, relations and physics, 2016. URL https:
00222.
Andreas Blattmann, Tim Dockhorn, Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, Dominik Lorenz, Yam Levi, Zion English, Vikram Voleti, Adam Letts, Varun Jampani, and Robin Rom- bach. Stable video diffusion: Scaling latent video diffusion models to large datasets, 2023a. URL
https://arxivw www. 22.1.. 15127.
Andreas Blattmann, Robin Rombach, Huan Ling, Tim Dockhorn, Seung Wook Kim, Sanja Fidler, and Karsten Kreis. Align your latents: High-resolution video synthesis with latent diffusion models, 2023b. URL htttps://arrtt org abs
D Blevins. Flow-induced vibration. 1 1990. URL https:ttp///t /ttp:///wwwwww//. www. osti htt.:otttt..oobbbbib.oo/o/ 68070
Anton Bovier, Michael Eckhoff, Véronique Gayrard, and Markus Klein. Metastability in reversible difffusion processes Sharp asymptoties for capacities and exit times. Journal of the European Mathematical Society, 006(():399 424, 2004. URL http: eedmll.......//mm org/ doc 20o.//77000 2776
Johannes Brandstetter, Daniel Worrall, and Max Welling. Message passing neural pde solvers, 2023. URL https::/aaxiv org/ /220 03376.
Salva Rühling Cachay, Bo Zhao, Hailey Joren, and Rose Yu. Dyffusion: A dynamics-informed dif- fusion model for spatiotemporal forecasting, 2023. URL https://aaxix// http///awiiv.rrraa/////aa/ 01984.
John Canny. A computational approach edge detection. IEEE Transactions Pattern Analysis and Machine Intelligence, PAMII-866:669-698, 1986. doi: 1221199/TAAMII119844777777
Yadi Cao, Menglei Chai, Minchen Li, and Chenfanfu Jiang. Efficient learning of mesh-based phys-
ical simulation with bsms-gnn -nn, 2023. URL https: /tt//aiivvv//./////// org/ob/gg///.ooo abs/ 22100. 02573.
Shoei-Sheng Chen. Flow-induced vibration of eircular cylindrical structures, 6 1985. URL https: / //www.... isti gtv/biillliii 6331788.
Shoufa Chen, Mengmeng Xu, Jiawei Ren, Yuren Cong, Sen He, Yanping Xie, Animesh Sinha, Ping Luo, Tao Xiang, and Juan-Manuel Perez-Rua. Gentron: Diffusion transformers for image and video generation. 2024. URL https: / arxiv. www.a.oogw..a///...........a
04557.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding, 2019. URL https: / /arxiv.orgg abs/181 0. 04805.
11

---

Published as a conference paper at ICLR 2025
Pan Du, Meet Hemant Parikh, Xiantao Fan, Xin- Yang Liu, and Jian-Xun Wang. Confil:: Con- ditional neural field latent diffusion model generating spatiotemporal turbulence, 2024. URL
https: /tte:///axxii org/ org/abs/////oooo. 2000s/2400 2203. 05940.
Patrick Esser, Robin Rombach, and Bjöm Ommer. Taming transformers for high-resolution image orgo/b/g////. 22122.222... 09841.
synthesis. 2021. URL
Patrick Esser, Johnathan Chiu, Parmida Atighehchian. Jonathan Granskog, and Anastasis Ger- manidis. Structure and content-guided video synthesis with diffusion models, 2023. URL /taxiv//axxiv... www.///gg//////oooo/ooooooo.ee// 22tp//3022.00.1....
Christoph Feichtenhofer, Haoqi Fan, Yanghao Li, and Kaiming He. Masked autoencoders spa- tiotemporal learners, 2022. URL https::////. /ttpaaawwgg//aaaaaa/////..to../t abs 2atbs/200 22005 09113.
R.D. Gabbai and H. Benaroya. An overview of modeling and experiments of vortex-induced vibration circular cylinders. Journal of Sound and Vibration, 282(3):575-616, 2005. ISSN 0022-460X. doi:
URL https: /www.
sciencedirect htt.://c.............///aa/////a
Rohit Girdhar, Mannat Singh, Andrew Brown, Quentin Duval, Samanch Azadi, Sai Saketh Ramb- batla, Akbar Shah, Xi Yin, Devi Parikh, and Ishan Misra. Emu video: Factorizing text-to-video generation by explicit image conditioning, 11 2023. URL http:/// /ttp///axvvvv////////////////.aa org/ abs/211. 10709.
Jayesh K. Gupta and Johannes Brandstetter. Towards multi-spattooeeppraa--scatt generalized pde org. /tds/22000 152.9..16666.. 15616.
modeling, 2022. URL
Zhongkai Hao, Chang Su, Songming Liu, Julius Berner, Chengyang Ying, Hang Su, Anima Anand- kumar, Jian Song, and Jun Zhu. Dpot: Auto-regressive denoising operator transformer for large- scale pde pre-trainin,, 2024. URL pestps://arkee wwg. abt/////////// 03542.
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners, 2021. URL 06377.
or//1///111.
Maximilian Herde, Bogdan Raonić, Tobias Rohner, Roger Käppeli, Roberto Molinaro, Emmanuel de Bézenac, and Siddhartha Mishra. Poseidon: Efficient foundation models for pdes, 2024. URL https://arxis org/oogg///ooooooooooo httg//ab//2400 19101.
Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance, 2022. URL https:////aarxi
org/aba /bs/ 0.207.12590
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models, 2020. URL 11239.
Jonathan Ho, William Chan, Chitwan Saharia, Jay Whang, Ruiqi Gao, Alexey Gritsenko, Diederik P. Kingma, Ben Poole, Mohammad Norouzi, J. Fleet, and Tim Salimans. Imagen video: High definition video generation with diffusion models, 2022a. URL https://azxi. org/ abs/ 2210. 02303.
Jonathan Ho, Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David J. Fleet. Video diffusion models, 2022b. URL
org/ abs /aabs//220 /2204.22222 22044.3458.
Philipp Holl and Nils Thuerey. Pnew (PhiFlow): Differentiable simulations for pytorch, tensorflow and jax. In International Conference on Machine Learning. PMLR, 2024.
Jiahe Huang, Guandao Yang, Zichen Wang, and Jeong Joon Park. Diffusionpde: Generative pde- solving under partial observation, 2024. URL https: http://wwwwwwwpaa/////o// wwtpo//gg//g..ooo/////////
arxiv.
Gabriele Immordino, Andrea Vaiuso, Andrea Da Ronch, and Marcello Righi. Predicting transonic flowfields in non-homogeneous unstructured grids using autoencoder graph convolutional net- works. 5 2024. URL http: //tp///aavv.////aa///a//eaeaaaaa waw. 22405 04396.
12

---

Published as a conference paper at ICLR 2025
Andrew Jaegle, Felix Gimeno, Andrew Brock, Andrew Zisserman, Oriol Vinyals, and Joao Carreira. Perceiver: General perception with iterative attention, 2021. URL abs/21033 03206.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language 08361.
models, 2020. URL https:://///tc htt///axxiii arxiv. org/
Levon Khachatryan, Andranik Movsisyan, Vahram Tadevosyan, Roberto Henschel, Zhangyang Wang, Shant Navasardyan, and Humphrey Shi. Text2vidco-zero. Text-to-image diffusion models /2303. 13439.
zero-shot video generators, 2023. URL
Diederik P Kingma and Max Welling. Auto-encoding variational bayes, 2022. URL https:
//arxiv. org/ abs 1312. 6114.
Felix Kochler, Simon Niedermayr, Rüdiger Westermann, and Nils Thuerey. Apebench: bench- mark for autoregressive neural emulators of pdes, 2024. URL 2411.00 00180.
Georg Kohl, Li-Wei Chen, and Nils Thuerey. Benchmarking autoregressive conditional diffu- sion models for turbulent flow simulation, 2024. URL httttpp://axxxv.orree wtt.ooggg/bb//.///e////// 01745.
Nikola Kovachki, Zongyi Li, Burigede Liu, Kamyar Azizzadenesheli, Kaushik Bhattacharya, An- drew Stuart, and Anima Anandkumar. Neural operator: Learning maps between function spaces with applications to pdes. Journal of Machine Learning Research, 24(89): 2023. URL http:/ jmmm..m........ orgg papers/ hetps///////222// .html.
Zijie Li and Amir Barati Farimani. Graph neural network-accelerated lagrangian fluid simulation. Computers & Graphics, 103:201-211, 2022. ISSN 0097-8493. doi: https://doi.or//00.100 o/j6//..a.. 2022.02.004 URL https: pettps://www. /www//wwwwoomm.o www.://cieneeireee..o rect. com/ cctp//ceeeeeee/attiii////aa
Zijie Li, Kazem Meidani, and Amir Barati Farimani. Transformer for partial differential equations' operator learning, 2023a. URL https: http:::///axvvvvo/ http:////wwx...ooo////////////// wtw.o/////./.......////// 13671.
Zijie Li, Dule Shu, and Amir Barati Farimani. Scalable transformer for pde surrogate modeling, 2023b. URL
org. abs/2305 17560.
Zijie Li. Saurabh Patil, Francis Ogoke, Dule Shu, Wilson Zhen, Michael Schneier, John R. Buchanan Jr., and Amir Barati Farimani. Latent neural pde solver: reduced-order modelling framework for partial differential equations, 2024a. URL https: : /tap:s:///aaee //tp//aavvvvv///v///a//e.e.aaavl org abs /tdlb///400 2402.17852
Zijie Li, Anthony Zhou, Saurabh Patil, and Amir Barati Farimani. Cafa: Global weather forecasting with factorized attention on sphere, 2024b. URL htt.ttttt//aaaete. org.orgg/aao..o htt//bb//2ee 240.. 07395.
Zongyi Li, Nikola Kovachki, Kamyar Azizzadenesheli. Burigede Liu, Kaushik Bhattacharya, An- drew Stuart, and Anima Anandkumar. Neural operator: Graph kernel network for partial differ- ential equations, 2020. URL https:://axxx:: org/ abs/ 0000000333000
Zongyi Li, Nikola Kovachki, Kamyar Azizzadenesheli, Burigede Liu, Kaushik Bhattacharya, An- drew Stuart, and Anima Anandkumar. Fourier neural operator for parametric partial differential equations, 2021. URL https:///aax org abs 20100.00000000 08895.
Zongyi Li, Nikola Borislavov Kovachki, Chris Choy, Boyi Li, Jean Kossaifi, Shourya Prakash Otta, Mohammad Amin Nabian, Maximilian Stadler, Christian Hundt, Kamyar Azizzadeneshelli and Anima Anandkumar. ccometry-informee neural operator for large-scale 3d pdes, 2023c. URL 2309.00583.
https: /arxiiii.. http:/awwww.gg/eee/ea////a
algg//////ae 2309. 00583.
Marten Lienen, David Lüdke, Jan Hansen- Palmus, and Stephan Günnemann. From zero turbu- lence: Generative modeling for 3d flow simulation, 2024. URL https: /arxivv....///////////aa///aavt/ org/ abs/ 2306. 0166.17766
13

---

Published as a conference paper at ICLR 2025
Shanchuan Lin, Bingchen Liu, Jiashi Li, and Xiao Yang. Common diffusion noise schedules and sample steps are flawed, 2024. URL https: //tp///aavvvvv/.av///a//e..eaaaa http//aawivv...aaaaaa./././ htt.s/ogggabb/......./b 08891.
Phillip Lippe, Bastiaan S. Veeling, Paris Perdikaris, Richard E. Turner, and Johannes Brandstetter. Pde-refiner: Achieving accurate long rollouts with neural pde solvers, 2023. URL https:
//aaxiv..rrg//// atw./////..2 05732.
Yinhan Liu, Myle Out, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach, 2019. URL https: /ttp///axivv.v/..a//aa/eeaaaawaa www.xiv.org//bb... /9077/900... 192...
Adrián Lozano- Durán and Javier Jiménez. Teseevesolved evolution of coherent structures in turbu- lent channels: characterization of eddies and cascades. Journal of Fluid Mechanics, 759:432-4711 2014. doi: 10.101 1tji7..211.245555
Lu Lu, Pengzhan Jin, Guofei Pang, Zhongqiang Zhang, and George Em Karniadakis. Learning nonlinear operators via deeponet based on the universal approximation theorem of operators.
Nature Machine Intelligence, 3(3):218-229, March 2021. ISSN 2522-5839. doi: 10. 1031038/
0222566021-00002-- URL http:ttt/////// doi org/ /togg//... 1038/100000 / 442256 021 00300-5.
Eric Luhman and Troy Luhman. Knowledge distillation in iterative generative models for improved sampling speed, 2021. URL https:://artiii http:////aaxii...a/aababs///e /t///b///23 /21 01 02388.
Michael McCabe, Bruno Régaldo-Saint Blancard, Liam Holden Parker, Ruben Ohana, Miles Cran- mer, Alberto Bietti. Michael Eickenberg, Siavash Golkar, Geraud Krawezik, Francois Lanusse, Mariel Pettee, Tiberiu Tesileanu. Kyunghyun Cho, and Shirley Ho. Multiple physics pretraining for physical surrogate models, 2023. URL https: /ttp//rrxvvv//// org/ abs 2231.. 02994.
Nick McGreivy and Ammar Hakim. Weak baselines and reporting biases lead to overoptimism in machine learning for fluid-related partial differential equations, 2024. URL https: /arxiv. org/ / 2407 07218.
Chenlin Meng, Robin Rombach, Ruiqi Gao, Diederik P. Kingma, Stefano Ermon, Jonathan Ho, and Tim Salimans. On distillation of guided diffusson models, 2023, URL htttt:t/ssaaxi....o. abs/ abs/2210. 03142.
Alex Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models, 2021. URL https://arxe org/ wttgg////////////////// 2212 09672.
William Peebles and Saining Xie. Scalable diffusion models with transformers, 2023. https:::
//arxiv. hrgp/ivvorrg/aoo abs////////// httaaaa/aaaaaee 221.22222..992
Tobias Pfaff, Meire Fortunato, Alvaro Gonzaez-Gozzalez, and Peter W. Battaglia. Learning mesh- based simulation with graph networks, 2021. URL 03409.
M. Raissi, Perdikaris, and G.F. Karniadakis. Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equa- tions. Journal of Computational Physics, 378:686-707, 2019, ISSN 0021-9991. doi: https://doi. 10.045. URL https: /wwww//www/..// wew.enieieeeeeee.eeeeeeeeeeeeeee com/ science///////ee/ art atticcle/s11/5002199918833
Jeff Rasley, Samyam Rajbhandari, Olatunji Ruwase, and Yuxiong He. Deepspeed: System opti- mizations enable training deep learning models with over 100 billion parameters. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery Data Mining, KDD '20, pp. 3505 3506, New York, NY, USA, 2020. Association for Computing Machin- ery. ISBN 978 450379994,, 1145/339 4486 .. 3406703.
doi: 10.1145/3394486.44066 URL https: /tap:p:://do... http//d..ooo/ org/10.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models, 2022. URL https: htthhtts////xit /arxiv.orgg fbs/2112 10752.
14

---

Published as a conference paper at ICLR 2025
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedi- cal image segmentation, 2015. URL https:tpp...m :ttpppii////iv..////aa/aaaaaamaa org/ abs/1505.00555
Tim Salimans and Jonathan Ho. Progressive distillation for fast sampling of diffusion models, 2022.
URL https:://aatttt.e org/ abs/ 2202. 00512.
Louis Serrano, Thomas X Wang. Etienne Le Naour, Jean-Noö Noël Vittaut, and Patrick Gallinari. Aroma: Preserving spatial structure for latent pde modeling with local neural fields, 2024. URL https: //arxiv. org/ abs/ 24 2406. 02176.
Dule Shu, Zijie Li, and Amir Barati Farimani. A physics-informed diffusion model for high-fidelin flow field reconstruction. Journal of Computational Physics, 478:111972, April 2023, ISSN 0021-9991. doi: 10.
URL http::/dx..do doi. ottp:/// 1016/].
jep. 2023. 111972.
Uriel Singer, Adam Polyak, Thomas Hayes, Xi Yin, Jie An, Songyang Zhang, Qiyuan Hu, Harry Yang, Oron Ashual, Oran Gafni, Devi Parikh, Sonal Gupta, and Yaniv Taigman. Make-a-video: Text-to-video generation without text-video data, 2022. URL https: /arxivvv. org/ abs/ 2209. 14792.
Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsuper- vised learning using nonequilibrium thermodynamics, 2015. URL https:: httppss//axxvv.rgg/// bbbs/1503 03585.
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models, 2022. URL /ettss:/arxiv..r org/ abs//aas/// 220000000... 02502.
Yang Song, Prafulla Dhariwal, Mark Chen, and lya Sutskever. Consistency models, 2023. URL
https: //ariii////ivv////////aa org/abss 2303 01469.
Kimberly Stachenfeld, Drummond B. Fielding, Dmitrii Kochkov, Miles Cranmer, Tobias Pfaff, Jonathan Godwin, Can Cui, Shirley Ho, Peter Battaglia, and Alvaro Sanzhez- Learned coarse models for efficient turbulence simulation, 2022. URL https::/arxxiv.rrrg 2112.15275.
R.M. Stringer, J. Zang, and A.J. Hillis. Unsteady rans computations of flow around cir- cular cylinder for wide range of reynolds numbers. Ocean Engineering, 87:1-9, 2014a. ISSN 0029-8018. doi: http:///bouuugooo...oooooooeeeee 2014..0... URL https: /www.
sciencedirect. com/ htt.a//cwwc..ceceeeeet.e..//////
Robert Stringer, Jun Zang, and Andrew J. Hillis. Unsteady rans computations of flow around
circular cylinder for wide range of reynolds numbers. Ocean Engineering, 87:1-9, 2014b. URL
https: api. http:/ppp..eenniiscicciiccacaeee icscholast org/ Corpps 1227786644
Nils Thuerey, Konstantin Weißenow, Lukas Prantl, and Xiangyu Hu. Deep learning methods for soynolds-averagee navier-stokes simulations of airfoil flows. AIAA Journal, 58(1)255-36, January 2020. ISSN 1533-385X. doi: 10.2514/1 2..51//..008291..1 URL http: dx doi. dot..ccc.orroo..o org 11.2514, 1. 1.J058291.
Alasdair Tran, Alexander Mathews, Lexing Xie, and Cheng Soon Ong. Factorized fourier neural operators, 2023. URL http:::s///xxw..r/////a//
13802.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need, 2023. URL https::///// //arxiv. org/abs// 1006 0777666666..... 03762.
Sifan Wang, Jacob H Seidman, Shyam Sankaran, Hanwen Wang, George J. Pappas, and Paris Perdikaris. Bridging operator learning and conditioned neural fields: unifying perspective, 2024. URL https: htt///axvv.// .rgg abs /240440... .. 13998.
Haixu Wu. Huakun Luo, Haowen Wang, Jianmin Wang, and Mingsheng Long. Transolver: A fast transformer solver for pdes on general geometries, 2024. URL https: / /arxivi. org/ abs/ 2 2402 02366.
15

---

Published as a conference paper at ICLR 2025
Jay Zhangjie Wu, Yixiao Ge, Xintao Wang, Stan Weixian Lei, Yuchao Gu, Yufei Shi Wynne Hsu, Ying Shan, Xiaohu Qie, and Mike Zheng Shou. Tune-a-video: One-shot tuning of image diffusion models for text-to-video generation. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 7623-7633, Octoher 2023.
Gefan Yang and Stefan Sommer. A denoising diffusion model for fluid field prediction, 2023, URL attps://arxe owg. atg/abs/////oooooooo.oooo//// 2301. 11661.
Yuan Yin, Matthieu Kirchmeyer, Jean-Yv Yves Franceschi, Alain Rakotomamonjy, and Patrick Gal- linari. Continuous pde dynamies forecasting with implicit neural representations, 2023. URL
https://arxiv.orge alos/ 2209. 14855.
M.M. Zdravkovich. Conceptual overview of laminar and turbulent flows past smooth and rough circular cylinders. Journal Wind Engineering and Industrial Aerodynamics, 33(1):53 62, 1990.
ISSN 0167-6105. doi: http:://wooogggooooooooo/o///oo0 URL https: ntttps://ww.
sciencedi htt.://eerret..ccccoccceeeeereer htt.a//c......act..acaaaaaaaaaan
Richard Zhang, Phillip Isola, Alexei Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as perceptual metric, 2018. BtLhhtss:///xxivv org/bb// 1801. 03924.
Yabo Zhang, Yuxiang Wei, Dongsheng Jiang, Xiaopeng Zhang, Wangmeng Zuo, and Qi Tian. Con- trolvideo: Training-free controllable text-to-video generation, 2023. URL https: : /arriiv. org/abs/ 2305. 13077.
Anthony Zhou and Amir Barati Farimani. Masked autoencoders are pde learners, 2024. URL nttps://arxiv. orgg wwt.:bsg///aooooooooooooooo..ost abs/2403 117728.
Anthony Zhou, Cooper Lorsung, AmirPouya Hemmasian, and Amir Barati Farimani. Strategies for pretraining neural operators, 2024. URL htt.t:tts////ttti/ http:////wwwwwo/oa////////////// /206 08473.
16

---

Published as a conference paper at ICLR 2025
A
ADDITIONAL RESULTS
"Air flows over obstacle resembling a low step the bottom half, spanning the width of the domain and over quarter the bottom domatn, starring from
height of The stop units units high and and 0 The flow is turbulent.""
'Air flows over an obstucle resembling nurrow cruss shupe that consists of narrow vertical that is joined logether with harizontal pillars the center left and center right. The har units wide and units high and located at at the bottom center of the domaln, starting from y = to 23 and to 14, The first pillar 10 unies wide and high located second pillar is units wide and 5 units high and is located at the center right of the domain, starting from 10 turbulent"
center left
domain, starting from y
and
= and
"Air flows obstucle resembliing wide rectangles that each span chird of the height of the dumain. The rectangles are nearly equal size each other and are vertically opposite. The first rectangle is to units wide and & units high and located rectangle 10 units wide and 10 units high and is located at the center of the domain, starting from 14 to
of domaln, starting from to and
The second
Figure 4: Additional examples of text-conditioned generation of 3D turbulence samples with the velocity magnitude rendered. The true solution shown top, followed by the sampled solution the bottom. While the generated solutions smoothen high-frequency features, the solutions remain stable and are broadly accurate. Rendered with vAPE4D (Koehler et al., 2024).
17

---

Published as a conference paper at ICLR 2025
1555
Pleme drscippiina:
Sloppe Carred, clmpptted Solapee Sioc: larps, seacning verricaly. Location: Central, atrabching from bertom
Rounded, monabooome
Size. Mediun,, intage. Location: left
slightly Modium
Locution:
555
Plune descripiions
Slape: Curvod look or Jalape, Sacc Lazae (occupic about 1/3 of the imnge), Locatioe Ocnter-ád, cxtcading from top to butlioan of image
2. Sloppe Trisagusz wwigeslaged,, Sioe Medicm (abort 1/4 the imags soo)) Locatioe Uppsr right teetee
3. Shope Fartially staible careed regnent, Stze: Small (steitie portion about 1/8 of the tmage) Location: Lewer right comer, partinily cut off tnag?
boode:
Fime dostiptkose
Stapes Cureed, elougped plene widaning end. Larps, guanning most the image
Uyper portion the image, curving
froms right.
2. Stage: Trianguiar cotioal plune with cartow lose expanding outward. Size: Medium to largo, occapying the cxntiall arco of image, Location:
Ocater Lie image, copanling 2oo 36 right.
3. Shagec Cureed, alangeed plens similee in the find, but inverted Soze Larg,, xpaming maee of the image witch Easation: Lower portion of the imays. curving from left togiga.
Plume dverriptione:
Curved,
bullous cod.
Locotion: extedding
Mishrssen--aeeee originncing
Mediun, smallr
Shape: Elongate, cussod stroak,
modiun,
planes, Estending
Lottons curving
specrds tomeetlle Light
Figure Additional examples text conditiodd generation of smoke buoyancy examples. The true solution shown on top, followed by the sampled solution on the bottom. The model is only given a text description of the initial frame, which is also displayed for each example with the additional observations omitted.
18

---

Published as a conference paper at ICLR 2025
Firid porre ryfrde wrtk ardincaf33 jut joetio:: 8-4, 835, Wirld minm with oricdity affd). (J), The Regpiik-rrbee la (3). The Frip is Iemmar
pasea clladrr
paata::
calen
vxladly
The Sepaoli
tutrokan.
This poose sorr cyfasder with sradiee 6.9 and portie:: 3J1 932 Fisd with vylidity of Th Segaltermbe's 176, The foom biihsleer.
pauss calzalez podlisu:
Beynndd
Fully
Figure Additional examples of text-conditioned generation of cylinder flow examples. The true solution shown on top, followed by the sampled solution on the bottom.
B AUTOENCODER DETAILS
Model
LI Loss
Model
LI Loss
AKL
KL Loss
LI Loss
Base
.0340
Base
.00535
5e-6
1.8e4
0.378
+GAN
.0311
+GAN
.00916
2e-7
4.8e4
0.239
+L.PIPS .0208
+L.PIPS
.00938
(c) KL Weight (Axr.) Ablation Smoke Buoyaney
(a) GAN/LPIPS Ablation for
(b) GAN/LPIPS Ablation for
Cylinder Flow
Smoke Buoyancy
Table Validation losses between reconstructed and true samples with various CNN backbone parameters.
B. ABLATION STUDIES
We present additional results investigating the use of adversarial training and perceptual loss metrics (LPIPS) autoencoder performance. An important consideration is that current pretrained models needed for LPIPS operate only on regularly spaced data; therefore, to maintain fair comparison, the cylinder flow data are resampled onto 128x32 grid be able to evaluate an LPIPS loss. Since the benefits of using these additional methods are not clear, they are omitted from autoencoder training the main results; however, this could be a direction for future work improve the accuracy of LDMs. Results for the cylinder flow and smoke buoyancy problems given in Tables 4a and 4b.
Adversarial Training A common technique to improve the reconstruction accuracy of autoen- coder to introduce discriminator to assess whether the reconstructed input is fake real (Esser et al., 2021). Since our proposed mesh encoder is agnostic to the input discretization. it can also be used as discriminator by encoding given PDE sample, flattening the latent space, and averaging the logits. In practice, the autoencoder is trained in two stages: the first stage minimizes the recon- struction error and maximizes the probability of an erroneous discriminator classification, and the second stage optimizes the discriminator when given real and reconstructed sample.
On the cylinder flow dataset, the use a discriminator improves autoencoder performance, however, for the smoke buoyancy dataset does not. We hypothesize that complex PDEs more
19

---

Published as a conference paper at ICLR 2025
challenging to learn if they are fake real, since deviations from real samples could be reasonably interpreted as eddies or nigh-frequency features in more turbulent regimes. We anticipate this could be improved with additional tuning, but due to its difficulty we omit the use of adversarial training for the main results, and indeed lower losses can also be obtained but simply scaling the autoencoder and training it for longer.
Perceptual Loss Common L2 and L1 losses capture global statistics the similarity of recon- structed and true samples, however, they can misrepresent the semantic similarity of different sam- ples. To improve these analytical losses, Zhang et al. (2018) propose use differences in hidden activations of pretrained models as a loss metric. This is commonly implemented using a pretrained VGGI6 model on ImageNet, however, this hardly appropriate for PDE applications. We leverage a pretrained operator transformer from Hao et al. (2024), which has seen fluid dynamics data to evaluate an LPIPS loss. final consideration is the fixed time window of pretrained PDE models; to adapt this to arbitrarily long spatio-temporal samples we autoregressively apply DPOT and average its hidden activations along a temporal rollout of the true and reconstructed samples. Again, we find that LPIPS is more beneficial for simpler physics.
KL Penalty We evaluate the effect of different KL penalties on the variance of the autoencoder latent space as well as the prediction L1 loss after downstream training of latent diffusion model in Table 4c. We find that larger KI. penalties lead to lower KL losses, indicating that the latent space is closer to a Gaussian. However, this reduces the reconstruction aceuracy of the autoencoder, which degrades the performance of the latent diffusion model, Alternatively, lower KI. penalties result in higher-varianee latent spaces, but this is outweighed by the better reconstruction capabilities.
Ball Radius
LI Loss
Model
LI Loss
0.045
.0138
GNN
.382
0.033
.0141
Neural Field
.171
0.020
.0148
Ours
.011
(a) Ball Radius Ablation for Cylinder Flow
(b) Architecture Ablation for Cylinder Flow
Table 5: Validation losses between reconstructed and true samples with various mesh learning parameters.
Ball Radius We consider training the mesh encoder/decoder with different ball radii to measure the effect of the size of receptive field of the learnable aggregation and interpolation. Using the same hyperparameters for the CNN backbone, we train the mesh autoencoder different ball radii on the cylinder flow benchmark and report the validation LI reconstruction losses in Table 5a. We find that increasing the radius improves the reconstruction accuracy, likely since there more points within the ball used to construct the latent grid, as well as more latent grid points used to query output points. However, this comes at an additional computational cost due to the need integrate over more points when approximating the kernel integral.
B.2 GNN/NEURAL FIELD COMPARISONS
We consider using GNN- and neural field-based approaches to encode and decode mesh data. The results for the proposed architectures on the cylinder flow problem are given in Table 5b. In general, the GNN and neural field methods do not seem to perform well. We hypothesize that this could because compressing large spatio-temporal samples is very challenging without proper inductive biases. Specifically, GNNs and attention-basse neural field architectures very general in their pooling and unpooling operations, which makes it challenging to extract compressed features and reconstruct dense outputs.
Graph Neural Networks To compress and expand mesh-based data, can apply graph pooling and unpooling methods. In particular, we consider a method of pooling and unpooling nodes based on bi-stride pooling (Cao et al., 2023). Nodes are pooled every other BFS (Breadth-Firss Search) frontier in a manner that preserves connections between pooled and unpooled nodes, and doesn't t require learnable parameters. Pooled and unpooled nodes different compression levels can be
20

---

Published as a conference paper at ICLR 2025
3D Kernel Aggregation
Define Latent Grid
Kernel Aggregation
Nodal Values
Figure 7: Kernel Aggregation. Neighbors to latent grid points are used to compute operator mapping hetween the input mesh and latent grid. This can also be extended to arbitrary dimensions, such including time dimension or extra space dimension. Additionally, this can he reversed compute an operator mapping between the decoded latent grid and output mesh.
cached and used for the expansion or contraction of graphs. At the encoder, pooled nodes use message passing between unpooled nodes in a defined neighborhood to aggregate information and downsample the graph. At the lowest resolution, we apply cross-attention between the node list and fixed latent vector, similar to Perceiver pooling (Jaegle et al., 2021). To interpolate fixed latent back onto a graph, we use cross-attention between fixed latent and a learnable latent node list, which is truncated to the correct dimension. The low-resolution graph can be used supernodes in a cached upsampled graph to pass messages to their unpooled neighbors until the original graph is reconstructed.
Neural Fields Neural fields represent a mapping between set of coordinates and their functional values. While this is restricted to modeling single sample, the addition of condition in conditional neural fields allows this framework to be adapted to different PDE samples. To compress an arbi- trary spatio-temporal vector, the spatial and temporal axes separately aggregated using Perceiver pooling. Specifically, the architecture alternates between computing the cross-attentic on spatial or temporal points and fixed vector. This then be downsampled upsampled using conventional CNN methods. To interpolate a fixed latent back onto an output mesh, the fixed latent can be seen condition for neural field queried at the output mesh. We implement this based the CViT architecture (Wang et al., 2024). A transformer encoder computes self-attention to encode the latent vector into a condition, while the query coordinates are interpolated from learned latent grid. A transformer decoder then calculates the cross- atteettotn between the query points and the condition, which is decoded to the reconstruction.
B.3 KERNEL ACGREGATTON//NTTRPOIIIN
Given input that is arbitrarily discretized u(t, x,,), it can be seen the of values in which the true function u is given at. We seek to map this function u(y) to another function q(r), where can control the discretization be uniform. From Li al. (2020), this mapping can be approx- imated by a learnable kernel integral, specifically qιx) = fp ko(z, ()u(y)dy, where is given on given on coordinates y. Importantly, we can choose the query coordinates
coordinates æ and and evaluate this kernel integral at these arbitrary queries; however, choosing uniform query points is convenient for the CNN backbone. In practice, this is implemented as in Figure Nodal values consisting of pressure and velocity are given on mesh and uniform latent grid is defined. To make the kernel integral tractable, the integration domain is truncated to a local ball, where neigh- bors within a certain radius of latent grid points contribute to the operator mapping. Again, from Li et al. (2020), this local Monte Carlo approximation can still be accurate as the error scales with m 1/2 where m is the number of points sampled.
We can reverse this process at the decoder to interpolate the decoded latent grid onto the output mesh Specifically, the latent function qu is now mapped to the output function us by the kernel integral u,(y) = So Ke x z=)x(())dr. Importantly, the operator mapping ensures that the output mesh can be queried at arbitrary locations y,, rather than fixing the decoded solution (both for the autoencoder and LDM) to single discretization. This discretization invariance using the LDM to generate a solution based on text prompt, but decoding it onto arbitrary query points, such as in Figure 8, and is empirically reproduced by Li et al. (2023c) as well.
be when
21

---

Published as a conference paper at ICLR 2025
Proupt/IC
Query
LDM
D
Figure 8: Discretization Invariance. Given a sampled latent solution from the conditional LDM, it can be decoded onto an arbitrary set of query points.
C DIFFUSION DETAILS
C.I DENOISING ABLATIONS
Model Isimple
Model
Eiimpte
Model
Iisimpte
Model Iismppte
Noise 0,056
Fixed E
0.056
Scale=1.0
0,034
Linear 0.034
Velocity 0.072
Learned En
0.059
Scale=0.2
0.015
Cosine 0.069
(a) Parameterization
(b) Learned Eg
(c) Latent Space Scaling
(d) Noise Scheduler
Table Validation reweighted VLB losses evaluated different settings of the denoising process
Denoising Parameterization We investigate parameterizing the reverse process by predicting the velocity Vā, vI- -,, rather than the noise (Salimans Ho, 2022). This is moti- vated by the idea that the prediction of pure noise becomes increasingly unstable as the number of timesteps decreases. We find that v-prediction slightly increases the VLB loss, however, this effect not significant.
Learning Eg Following Nichol Dhariwal (2021) we investigate parameterizing the reverse pro- cess variance as Eg(2,, t) = exp(v log 3, + (1-e) log B,), where is learned vector that modulates the transition between 3, and B. Furthermore, the loss is modified to minimize the log-likelihood. We find that learning Mg does not significantly effect the reverse process.
Latent Space Scaling Following Rombach et al. (2022), scale the latent space approximate unit variance across latent encodings of the training set. When the latents are too large, the addi- tion of Gaussian noise with unit variance does not sufficiently noise the latents (2T // I)), which harms the denoising process during inference. While the KL-llos could be increased during autoencoder training limit the variance of the learned latent space, this comes at the cost of duced reconstruction accuracy. As such, scaling the latent space is an important consideration, and a proper scaling factor is needed achieve good results.
Noise Scheduler Following Nichol & Dhariwal (2021), we evaluate using cosine linear noise scheduler. We find that empirically, a linear noise scheduler performs better in our case; however, optimizing the noise schedule is still an area for future work.
C.2 DIFFUSION BACKBONE ARCHITECTURES
Model
Val LI Loss
We evaluate parameterizing the reverse process using 3D Unet or DiT in Table 8. In both cases, the models need predict spatio-temporal noise, either by using 3D convolutions or patchi- fying across space and time. Models are trained with approxi- mately the same parameter count and conditioning on either the first frame or a text prompt for the cylinder flow problem. We report LI losses between generated samples and the validation set. We find that a DiT backbone slightly outperforms a Unet backbone.
Unet-FF
0.0416
Unet-Text DiT-FF
0.0492
0.0385
DiT-Text
0.0404
Table 8: Different model architec- tures for the Cylinder Flow problem
22

---

Published as a conference paper at ICLR 2025
Model
S
Inf. Time
Rel. L2 Loss
Stepe, Cylinder
GINO
0.112
0.2445
019
MGN
1.683
0.2617
OFormer
1.336
0.3386
015
10
0,4572
0.1595
017
20
0.6119
0.1583
016
LDM
50
1.0844
0.1579
100
1.8716
0.1595
010
1000
15.753
0.1522
10
0.5672
0.1313
20
0.8654
0.1267
LDMM-FF
50
1.7188
0.1306
CSMM ngg
100
3.1561
0.1281
Interuae Time vs, DUIVE Slepo, Cyttader libre
1000
27.895
0.1309
0,4711
0.1941
20
0,6405
0.1980
LDex-Text
50
1.1522
0.1949
100
1.9936
0.1903
1000
16.251
0.1796
10
0.5770
0.1474
20
0.8836
0.1536
LDMx-Text
50
1.7635
0.1513
100
3.2456
0.1495
1000
29.096
0.1476
Table Cylinder Flow Sampling. Comparison inference times (8) and relative errors different baselines and LDM models under varying numbers of denoising steps S using DDIM or DDPM sampling.
C.3 DDIM SAMPLING
We evaluate the use of DDIM to accelerate sampling after training an LDM with the denoising ob- jective. We implement DDIM sampler (Song al., 2022) and evaluate the inference time and relative 1.2 loss of sampled solutions on the Cylinder Flow problem at different denoising steps S. These results are presented Table 7 on a single NVIDIA A100 GPU, We find that DDIM can recover most of the solution even at small number of denoising steps, and in certain cases produce higher quality samples. Through the use DDIM, combination with using a latent space and spatio-temporal sampling, diffusion models ministic neural solvers in addition to being more accurate. We hypothesize that DDIM is effective due to fundamental differences in the modeled distributions within the image and PDE domains.
produce solutions faster than conventional, deter-
We consider the underlying conditional distribution (o(u)c that is approximated by conditional diffusion. Within image domains, this distribution has many modes in order to cover a diverse set of sampled images for given prompt, however, in PDE settings this distribution ideally only has single mode since when conditioned on a past timestep, there should only be a single sampled solution. Therefore, in PDE settings, we can expect Langevin sampling to converge quickly, since the distribution that is sampled from has only single mode. Furthermore, since score estimates are still trained using the original denoising objective, they are still accurate at the initial sample Xo: therefore, although Langevin sampling can start far from po (u|c), the score estimates are still accurate enough quickly approach the desired mode.
One method empirically evaluating this could be to calculate the Jacobian the estimated score function V Vzsa(z) for diffusion models trained in PDE and image settings. The Jacobian of the score function would correspond to the Hessian of the estimated underlying potential function Fo),,, which is related the mean mixing time of the Langevin sampler. The mean mixing time is cor- related with the inverse of the Hessian determinant (Bovier et al., 2004), which could measure the complexity of the estimated distribution (x)) & Fo (x:) and its effect sampling times.
23

---

Published as a conference paper at ICLR 2025
Model
Rel. L2 Loss
GINO
Unstable
Conditioning
Abc
0.0 0,
MGN
0.171
Abc
or
LDM
OFormer
0.088
{0. ... 0.
X
LDMAR-FF 0.068
LDMAR-Ten Text
0.085
Table 9: Left: Autoregressive LDMM results on Cylinder Flow, using identical hyperparameters to Table 1. Right: Framework for autoregressively generating text-conditioned PDE samples.
C.4 AUTOREGRESSIVE LDMS
Although sampling complete PDE trajectories can result in lower error accumulation, this fixes the temporal horizon and would require scaling the model in order to achieve longer rollouts. To address this, we present results on using the proposed LDM framework autoregressively. albeit with large prediction window. For LDM models conditioned on physical field values, this can be easily done by passing the last frame of the predicted solution back to the LDM model. However, for text- conditioned LDM models, the prompt is only relevant for the first prediction. To address this, we investigate a conditioning strategy which the model can dynamically switch between text- and physics-based conditioning during inference by padding the unused modality with zeros, shown in Table 9. From a user perspective, querying the model with a text prompt remains the same; however, the model internalyy uses different conditioning modalities at different steps.
To validate this, we train the LDM model on the Cylinder Flow problem to predict 100 timesteps, in four windows of 25 timesteps each. We also compare this approach with neural solver baselines re-trained with longer prediction horizon. To maintain a comparison to the original experiment, keep the hyperparameters and model sizes the same and present the results Table 9 for the medium-sized LDM model. Predictably, errors grow across all models in the longer time horizon setting, with GINO becoming unstable due to error propagation. Notably, the rext-conditioned LDM model incurs performance loss presumably due to switching modalities during inference.
D CONDITIONING DETAILS
D.1 CYLINDER FLOW CAPTIONING
For the cylinder flow problem, we extract the cylinder radius, position, inlet velocity, and Reynolds number from the first frame. Since the cylinder radius much smaller than the domain. is given in centimeters while the position is given in meters. Additionally, we determine the flow regime based on the Reynolds number: Re<200 laminar, Re<350 = transition, Re>350 = turbulent. One limitation is that this is overly simplistic; flow regimes are generally problem-specific and can vary even with the same Reynolds number. In addition, there are varying standards to characterize these flow regimes from Reynolds numbers (Blevins, 1990; Zdravkovich, 1990; Stringer al., 201 2014a). However, given the lack resources to manually label this dataset, we adopt this simple strategy since these semantic differences do not greatly affect performance. Values are inserted in the prompt:
*Fluid passes over cylinder with a redius of inylineer rediuss.2t1 and position: inylinder, pos 101 (cylinder_posite
Fluid with volocity saiist_vlooctty.22 The Reynolds number (xegnolds_nomoore The flow (eLov_reginee_"
Since the text prompt is procedurally generated. we ablate the full prompt against two baselines. The first baseline considers generating text prompt solely with the extracted values, without any of the surrounding text, and uses the same RoBERTa model to encode the condition (Text-Ablate). The second baseline embeds the extracted values, normalized between -1 and 1, into a vector and uses MLP to encode the condition (Vector-Ablate). We train these two baselines and a model using the full prompt and report the validation L.1 loss in Table 10. We find that the extra
Model
Val L1 Loss
Full Prompt Text-Ablate
0.0404
0.2168
Vector-Ablate
0.1029
Table 10: Different model architec- tures for the Cylinder Flow problem
24

---

Published as a conference paper at ICLR 2025
context is beneficial for language models in understanding the meaning of different extracted values. Furthermore, using a language model can be a more expressive way of encoding low-dimensional physical values into meaningful condition.
D.2 SMOKE BUOYANCY CAPTIONING
The captioning procedure for the smoke buoyancy dataset is more challenging, since the initial density and velocity fields do not contain features that are easily extracted and described by text. To address this, we prompt an L.M (Claude 3.5 Sonnet) to describe an image of the initial density field. To improve the quality of generated captions, we also use a Canny edge detector to outline shapes in the image and provide this segmented image in the LLM prompt. The prompt is given below.
You xill be aaciyzing an imagz et a simulation of the Nevicr-Stokes equations to identity smoke pluncs, count then, and describe theis cheractoristion You will also be provided an image with the houndariea the snoke plunes
the simatttion in
Follow these instructions carafully:
Pical, exanine the provided inage: <image>
Second, exanine the provided segmented image: <swgnentee__nagee
To identiff, plunes:
1. Look for segiona of beight colus, which contraat againat the darker background.
2. Pay attention to color variations and changes in the shape of the plumas. the grewn lines the segnemted image help ify boundaries plunes.
2o count the aurber of plunes:
1. Scen the wntire inage eea
Use the
linss the segnenced inage help separate differenz plunes.
3. Pay attent fon to darker regions that separste fffrertt plunes.
To deseribe each plane's shape, size, and lccation:
(e.g., colum-like, mushroon nushroon-shaped, dispersed oud, -tianullt,, carved) and changes in shage the plune.
Shapes Characterive
2. Sise: Eatinate the relative sise compared to other elementa in the inage (e.g., small, mediun, large)
and dexcribe how the plune changex in sixe.
3. locationz Describe the pasilian using general
(e.g., upper laft curnes, pentes, lower right)
or in relation to other snoke plunes.
Remeniie Lbe segnented image Vo help deace each plune's shape, aixe, and location.
Present your nindings the loving format:
<ansxer>>
plunes ident ified [Tnsert nunber]
Plune descripLiom::
1. (Shapel, [Sizel, [tocation]
2. [Shapel, [Sixe], [Lorstiaan]
iContinue for eact plume isentities)
(Only incllde any relevent detaill sbout the overall patterns or shapes observed, Pay abtent ion to symnesry and how the plunes intersos with each other.] <ansver>
Aeditioonal observet observations: lons:
Recemnbee be as descrigt ivw and accurate as possible in your analysia.
We find that using segmented image improves quantitative and qualitative metries in labeling physics simulations. In particular, the correct number of smoke plumes are identified more often, and the resulting descriptions richer and more detailed. We give example in Figure 9. Addi- tionally, to inform models of the forcing term, the buoyancy factor is prepended to the L.M answer by inserting it into the template: "The buoyancy factor is ().". We also compare the responses of Claude 3.5 Sonnet, Gemini 1.5 Pro, and GPT-4o in Figure 9.
D.3 3D TURBULENCE CAPTIONING
Due to the small number of samples, we manually caption this dataset. To maintain consistency across human-geneen generateeee captions, we follow template that asks for pre-specified set of features. In particular, we look to describe qualitative features, the position, and the size of the shape, as well as how many shapes are present. An example template is provided below; however, the templates can vary based on the quantity and composition of the shapes:
over obstacle resenbl ing Aquel itetive Jabel <anape) (posit it on labell. The flav turbulent."
[ehape_ labell
at
25

---

Published as a conference paper at ICLR 2025
Rurbes of pluse 1dmniicio::
Claude 3.5 Sonnet
Stabez of 2tures -dantified:: desoriprtens:
* Claude
3.5 Sonnet
1. Corved crescets sape, large, certered in the Inage
Deceoent-shigee
center
3. Trienpilet et flane iil,, suclin, ippes right ucctme
Trianquiar, small, apper left comnee
I:latquiar sile survedt onge, nestur Lowee lelt cotnee 5. Curved aiaggated shaps, mediary Lower right COEDRE
Lrage shaped
staped plans desinsting the conposition. abaps and aice, czeating balenced aypeazazie.
contrai sreadens
The image disp lays oiriking syemetey. with tae sarge reallex plares are sool sooeed In each corter of the Lsage, creatinng styllsed ompoxition, The cverall pesters smn-1ite
Rurbes of Pluas Lacnticiv:: 1
Genii11
depceapli9e2
with the contzel plae touning the coce and the wosser planes
1. Bound, Seape, Centez
Makker of plase idont Pium descripticn sens:
Gemini 1.5 Pro
accitione: cosecvatioon The plane is symeetiical and bas as. vtange pecaesies and alan u puayue besogtvard.
1. Curved, Kedion, Upper latt
kumber of plased Ldentifleds 2
5, Cuevell Kallish J0p4s night
4P146
Corond, Lange, Denter plams han cootral latation,
Addisional cbonrvations: piunes sce asmtlesly staped,
that To cost pros neat featine the teacs, ccmpytng oiddle
bow.
graduel pluncs appear bo be rotating in cloctvise dircetion Plone notably larger.
astending deas lefo the
curoed, Mediun, cus-left and Bottom-sich bottoeecight comners. these havn sintiar cirred eirraiing each other di stagenally by acroen the image.
two
the top-lefe are austier compared to
the
Nanber or siuess sout:itied: FLew descriplions:
Andtblane roserratione
trage wchlbiit clear two sna plares sppearing axia foom
Conter acts 1. Curvod: feall, Uppez Richt Seall Cireslity Boalis Somer-tait
cerlectiors of each other alonn
bettos Light, The cestral peoeinant may indicate the hain flo the simiaated fiild dynanion. The intoraction betwenn the a reaots treanisien batween regiena beighs and dark
Aaticional observations: The symeerry in the image suppests wasrcred attucture, par Lioutar- asound the cunter. itsteso
chearred the smage.
Lais comollay, rith actiua appooile
piures the
the inage and two maller lepor-iight and Lowce-loft cemeero.
Figure Comparison of LLM outputs after prompting with the original image, or both the original and a segmented image. We also compare responses from Claude 3.5 Sonnet, Gemini 1.5 Pro, and GPT-4o.
D.4 CLASSIIFER-FREE GUIDANCE
Classifier-free guidance can be a method to modulate the strength of the conditioning information. This is implemented by using conditional dropout during training and combining conditional and unconditional noise estimates during inference: ?e (Zn, n, c) = (1 + w)< (z,, n, c) weg vocg(2,,) n)- The weight modulates the tradeoff between sample quality and diversity, with larger values in- creasing the guidance strength and decreasing sample diversity. Following Lin et al. (2024), we also rescale the conditional portion of the sample generation process with the recommended value of o = 0.7. Empirically, we find that only at very large or small weights does the generated sample change appreciably. We plot the trends Figure 10 for first-frame conditioning. We find that at large values of much of the generated solution is saturated and many features are washed out. At low values of w, many features are not present and the generated solution displays a lack of phenom- ena. We hypothesize that Rassificr-frre guidance is not necessary for physics applications where the primary concern is the numerical accuracy of solutions, rather than diversity or visual quality and indeed the loss of models trained without classifier-fre guidance (w 1) is often better than using other values of w.
E DATASET DETAILS
E CYLINDER FL.Ow
We use data from Pfaff et al. (2021), which is generated from water flowing over a cylinder con- strained in channel, modeled by the incompressible Navier-Stokes equations:
dv
vV - vptt,
VV = +
v.v=0
(7)
0
26

---

Published as a conference paper at ICLR 2025
Refermsce
(oo CFT))
K=116
Irraler
Iendf:
Traderd
Figure 10: Comparison of different weights used for classifier-fie guidance. In general, using vanilla condi- tional generation results the most accurate solutions.
Reynolds number distibutiin, Cylinder Flow Datasal
The original data are generated using COMSOL, with 600 timesteps from = 0 to f on a spatial do- main (x: [0, 1.6] y [0, 0.4]), with varying inlet locities and cylinder sizes/positions. To speed up our experiments, we downsample along the temporal dimen- sion to 25 timesteps. We use 1000 samples for training and 100 samples for validation. Both datasets contain samples with Reynolds numbers ranging from 100-1500. Additionally, we identify Karman vortex streets in data samples by calculating the norm of the temporal veloc- ity derivative || de, di ll2 and comparing it to a cutoff; if this quantity remains large or increases over time then periodic vortex shedding continues quantity decreases or goes to zero over time then the so- lution has reached laminar steady state. The Reynolds number of training data samples, as well as Karman vortex street behavior is plotted in Figure 1.. The observed Karman vortex streets within this dataset are consistent with the literature describing its occurrence at Reynolds numbers between 300-2e5 (Chen, 1985; Gabbai & Benaroya, 2005).
camus serta derected
Vetes Street
207
200
402
405
2002
1220
-.00
11.00
Figure 11: Distribution of Reynolds num- bers within the training Samples with Karman Vortex streets also identified.
occur and if this
E.2
SMOKE BUOYANCY
We data from Gupta & Brandstetter (2022), which is generated from smoke driven by buoyant force constrained box, modeled by the incompressible Navier-Stokee equations coupled with an advection equation representing the smoke density d:
dv Øt VV = vV -vptr,
v.v=0
(8)
Od
vd=0 Øt
The original data are generated using PhiFlow (Holl & Thuerey, 2024), with 56 timesteps from t 18 to 102, which results in At 1.5s. Furthermore, the simulation is solved on spatial domain (xr = [0.32], y = [0, 32)) with a resolution 128 x 128. To ensure compatibility with downsampling operations, which
with factor of 2, truncate the simulation horizon to t = 48. We use 2496 samples for training and 608 samples for validation.
E.3
3D TURBULENCE
We use data from Lienen et al. (2024), which is generated from LES simulation in OpenFOAMM with viscosity of 10-5 The original data is given resolution of 192 x 48 48 and at 5000
27

---

Published as a conference paper at ICLR 2025
timesteps. A variety of shapes (steps, corners, pillars, teeth, bars, elbows, donut, U-shape, T-shape, H-shape, squares, crosses, platforms, etc.) are used to model flows around different objects, and for additional visualizations refer readers page 18 of the original paper (Lienen et al., 2024). Details on the calculation the log-TKEE distance can also be found the original paper.
F TRAINING DETAILS
To maintain a consistent setup in all experiments, the only ground truth information provided to each model is from the first frame of the simulation. Additionally, maintain fair FLOPs com- parison, the compute required for autoregressive models during training is multiplied by the number of timesteps. This is because the LDM model considers all timesteps once during training, while autoregressive models only consider a single timestep.
F.1 CYLINDER FLOW
All baselines were trained on single NVIDIA RTX 6000 Ada GPU with a learning rate of 10-5 until convergence. LDM models were trained on a single NVIDIA A100 40GB GPU. For each model, we describe the key hyperparameters.
We train GINO model with latent grid size of 64 and GNO radius of 0.05, Furthermore, the FNO backbone uses a hidden dimension of 128 and 32 modes.
GINO
MeshGraphNet We train a MGN model with layers and a hidden size of 1024.
OFormer We train an OFormer model with 6 layers and a hidden size 512.
LDM
We train an autoencoder with a latent grid size of 64, GNO radius of 0.0425, hidden di- mension of 64, and downsampling layers, resulting in latent size of 16 x 16 16, which is compression ratio of around 48. Additionally, the DiT backbone uses 1000 denoising steps, a patch size of 2, hidden size of (512/1024), and depth of (24/28) depending model size.
F.2
SMOKE BUOYANCY
All baselines were trained on single NVIDIA RTX 6000 Ada GPU with a learning rate of 10 until convergence. LDM models were trained four NVIDIA A 100 40GB GPUs. For large models, four NVIDIA A100 80GB GPUs were used to handle memory requirements. For each model, we describe the key hyperparameters.
FNO
We train FNO model with hidden dimension of 192, 24 modes, and 6 layers.
Unet
We train Unet model with downsampling layers and hidden size of 128, with architec- ture modifications (Unetmod) proposed in Gupta & Brandstetter (2022).
Dil-Resnet We train Resnet with a hidden dimension of 256, 4 layers, and a dilation up to 8.
ACDM We train an ACDM model with a single conditioning frame and with a Unet backbone with hidden dimension of 256 and 4 downsampling layers.
LDM We train autoencoder with hidden dimension of 64 and 4 downsampling layers, sulting in a latent size of x 16 x 16, which a compression ratio of 512. Additionally, the DiT backbone uses 1000 denoising steps, spatial patch size of and temporal patch size of 1, hidden size of 111/1024/1536//3344) and a depth of (24/28) depending on model size.
F.3 3D TURBULENCE
baselines were trained on a single NVIDIA A100 GPU with a learning rate of 10-5 until conver- gence. LDM models were trained on four NVIDIA A100 80GB GPUs were used to handle memory requirements. For each model, we describe the key hyperparameters.
FNO We train FNO model with hidden dimension of 192, 16 modes, and 6 layers.
28

---

Published as a conference paper at ICLR 2025
Factformer
We train Factformer model with hidden dimension of 256, 8 heads, and 18 layers.
Dil-Resnet
We train Resnet with a hidden dimension of 128, 4 layers, and a dilation of up to 8.
LDM We train an autoencoder with hidden dimension of 64 and 4 downsampling layers, result- ing latent size of 6 12 3 x 3, which a compression ratio of 4096. The DiT backbone uses 1000 denoising steps, patch size of (I, 1, 2, 2), hidden size of 2048, and depth of 28.
G INFERENCE TIME COMPARISON
Model
Params
Time
Model
Params
Time
FNO
510M
1.52
Unet
580M
2.33
Model
Params
Time
GINO
72M
0.11
Dil-Resnet
33.2M
5.83
MGN
101M
1.68
FNO
1.02B
1.57
ACDM
404M
317.8
OFormer
131M
1.34
FactFormer
41.4M
5.15
LDM3-FF
243M
16.86
Dil-Resnet
24.8M
38.8
LDMs-FF
198M
15.75
LDMv-FF
725M
25.23
LDMM-FF
667M
27.89
LDM, -FF
2.72B
75.2
LDM,-F -FF
1.55B
57.44
LDMs Text
-Txt
313M
16.25
LDM -Txt
2.73B
75.7
LDMs -DMs-Text
334M
16.30
LDMM-
804M
29.10
LDMM-TTet
825M
26.49
(c) Inference Time for 3D Turbu-
(a) Inference Time for Cylinder Flow
LDM,- -Text
2.69B
84.46
lence
(b) Inference
Time for
Smoke
Buoyancy
Table 11: Inference time to predict single temporal rollout (batch size 1), evaluated single NVIDIA RTX 6000 Ada GPU NVIDIA A100 GPU (3D Turbulence). Reported times are seconds and averaged over nine validation samples.
For transparency. we report inference times of all models on the Cylinder Flow, Smoke Buoyancy, and 3D Turbulence benchmarks in Table 11. We reproduce previous results that GINO an efficient architecture for unstructured problems (Li et al., 2023c). Furthermore, we reproduce previous results demonstrating the increased compute needed for dilated Resnets (Li et al., 2023b). Lastly, we find that latent diffusion framework is much faster than prior work autoregressive diffusion or learning directly in the pixel space.
However, recognize that the increased latent diffusion inference times may be potential limita- tion. The current implementation uses 1000 denoising steps, which leaves ample room for accelerat- ing inference. Indeed, using DDIM sampler, shown in Appendix C.3, is able to vastly accelerate inference and outperform conventional, deterministic neural solvers in both speed and accuracy.
To compare the proposed method numerical baseline, the numerical solver needs be coars- ened to match the accuracy of the neural solver, as well as a reasonable effort needs to be made to compare against state of the art numerical solver (McGreivy & Hakim, 2024). The solvers used for the Cylinder Flow, Smoke Buoyancy, and 3D Turbulence benchmarks are COMSOL, PhiFlow and OOA, which are all high-performance numerical packages. We do not have access the COMSOL code to solve the cylinder flow problem with reduced accuracy. However, the original authors report a speedup of 11-290x (Pfaff al., 2021), which may be enough margin overcome coarsening the simulation to 90% of its original accuracy (see Table 1, model is reported with a 13% relative 1.2 error). Additionally, we also do not have access to the OpenFOAM simulation, but the original authors report simulation time of around 10-20 minutes 16 CPU cores (Lienen et al., 2024).
For the smoke buoyancy benchmark, we have access Phiflow and, in fact, use it to re-solve text- conditioned samples. We compare the accuracy and speed of using PhiFlow at lower resolutions with latent diffusion models in Table 12. PhiFlow Threadripper PRO 5975WX 32-Cores) than on our GPU (NVIDIA RTX 6000 Ada), so numerical solver times are reported using the CPU. When running lower-resolution PhiFlow simulations, the
solved faster
CPU (AMD Ryzen
29

---

Published as a conference paper at ICLR 2025
original initial condition downsampled, evolved with the solver, and upsampled using bicubic interpolation. There are tangible accuracy and speed gains from using a latent diffusion model; how- faster inference techniques such as DDIM needed to vastly outperform numerical solvers. Also of note is that tex12PDE generation something that numerical solvers cannot do.
Model
Inference Time (s)
Val L.1 Loss
LDMg- -FF
16.863
0.154
LDMM-FF
25.233
0.139
LDMp-FF
57.442
0.118
LDMs-TeXI
16.294
0.174
LDMM-Text
26.494
0.132
PhiFlow-32
69.266
0.172
PhiFlow-64
70.569
0.116
PhiFlow-122 128
75.468
0 (Ground-Trunte
Table 12: Comparison of inference speed and accuracy. PhiFlow-X denotes the spatial resolution used for numerical simulation. PhiFlow speed and accuracy values averaged five validation samples.
30


Reviews:
[{"review_id": "Dem5d9FJky", "reviewer": "Reviewer_tcKA", "summary": "this paper introduce framework for text to simulation use latent diffusion models, enables condition on language, initial conditions. this work generate a entire trajectories for improved accuracy without error accumulation.", "questions": "- what is re-solved trajectory referred to in line 394?\n- why doing text-conditioned simulation generation, is the pde type pre-defined for each model? How important is the training data text caption?\n- how does text-conditioning help the simulation results or they are dis-entangled.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- apply latent diffusion models to physics simulation with mesh encoder/decoder provide a scalable solution. \n- a new text conditioned PDE simulation framework. \n- supporting both structured and unstructured grids, and shows promising results on different types of PDE problems", "weaknesses": "-  it's unclear the benefits and why doing text-conditioned simulation generation\n- I find paper's contribution is more on using latent diffusion for pde simulation, not the text to PDE generation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "TACH2mvqiG", "reviewer": "Reviewer_9pTM", "summary": "The paper introduces a framework called TEXT2PDE, which employs text-conditioned latent diffusion models (LDMs) to model spatio-temporal PDE trajectories. The authors introduce a mesh autoencoder that can encode and decode physical fields on arbitrary meshes. Spatio-temporal trajectories are generated using a conditional denoising process of an LDM with the initial conditions and boundary conditions of the PDE imposed into the conditioning sequence of the denoising backbone. A main novelty is the introduction of text conditioning, by employing a pre-trained transformer encoder. The authors argue that using natural language as a conditioning modality can improve the interpretability and usability of neural PDE solvers.", "questions": "1. As the text prompts appear to accurately capture the first frame of the reference (as shown in Table 2 and Figure 4 in Appendix A), I wonder if the model is memorizing and retrieving the smoke plume from the training dataset that most closely matches the input. For the existing prompts, would it be possible to display the most similar first frame from the training set?", "limitations": "", "rating": 5, "confidence": 3, "soundness": 1, "presentation": 3, "contribution": 2, "strengths": "1. Thanks to the encoding of arbitrary meshes, the authors are able to apply their method to complex geometries – an important use case for the application of PDEs.\n1. The main strength of the paper is the introduction of text conditioning for generation of PDE trajectories. The idea is novel and opens up a new research direction.\nHowever, it would be still interesting to see more specific motivating examples of where text conditioning could be useful in real-world use cases of PDE solvers.", "weaknesses": "1. The evaluations seem lacking when assessing the text conditioning capability of the Text2PDE model. Firstly, the initial and boundary conditions of the cylinder dataset can be entirely encoded using just three real numbers, making the expressiveness of natural language excessive for this case. Secondly, the smoke dataset appears to be a poor fit for text-based description, as the smoke plumes are too unconstrained and unstructured to be accurately captured by language. (See also the question 1.)\n\n2. In my opinion, evaluating text conditioning would require a new dataset with more regular shapes placed in general positions. The configurations should be rich enough to necessitate natural language for description, but constrained enough that a short text can accurately describe the image. Currently, it seems that the TEXT2PDE system is limited to using text conditioning to generate only the initial condition. This could be achieved more directly by combining text-to-image techniques and then running a standard (neural) PDE solver. The added benefit of using a combined system is not clear. It could be more interesting (and useful) if the PDE dynamics were also informed by natural language—perhaps through a text-encoded equation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "n41RGvNoPu", "reviewer": "Reviewer_7eso", "summary": "The authors introduce a new approach to physics simulations using latent diffusion models. Their method is anchored by (i) the use of mesh autoencoders, (ii) full spatio-temporal modeling to avoid autoregressive error accumulation, and (iii) conditioning on text prompts.", "questions": "- Autoregressive vs entire solutions. It is not clear whether producing an entire solution trajectory is really beneficial. First of all it doesn’t scale to 3D problems or larger 2D problems. Secondly, it might be unphysical. Do the authors have insights on this?", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- Latent space modeling for physics simulations. By using latent diffusion models, the authors effectively compress and encode high-dimensional physics data into a lower-dimensional latent space. Such latent space approaches contribute to more adaptable and faster simulations, making it a powerful alternative to traditional numerical solvers. There is a surge of latent space PDE models, this paper fits well into this paradigm. The full spatio-temporal diffusion modeling conditioned on text prompts and the subsequent decoding stands out as unique modeling approach.\n\n- Text prompts for simulation control. Text2pde is an interesting direction since it allows users to define simulation parameters or initial conditions using simple, interpretable text commands. The potential for language-based input makes physics simulations more accessible and generalizable. Yet it is not clear what is the difference to simple parameter modulation, the nice capturing of the initial condition is cool.\n\n- Application to mesh-based dynamics.", "weaknesses": "- Text2pde seems to be hard to scale to larger problems. To avoid confusion: scalability here refers to input representations. The mesh autoencoder is a good direction, yet the latent space is a latent grid, which if going to 3D is fundamentally limiting. Also for larger 2D grids, a tokenized representation might be preferable. Secondly, the generation of the full trajectory is storage intensive, and for 3D problems would result in a 4D tensor, which is computationally infeasible. Most turbulent problems, where neural surrogates might shine are 3D, presenting a drawback of the applicability of the method.\n- Text conditioning in this framework seems to be a slightly sophisticated parameter conditioning / parameter modulation. The nice feature so far is the generation of the initial mesh, initial condition. It would greatly help to make these differences clearer. Further the text prompt part needs to be adapted for each problem individually. It should be clarified how the text part can maybe be used across settings? Is this even possible?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "EQB4oYhoxU", "reviewer": "Reviewer_jPnk", "summary": "Very new paper. I support this because it actually works, I appreciate the authors working! Just like language model for catalyst. But I just do not see why people need this as they need for catalyst-gpt. The authors know fluid, they do not need this even they can make this fluid-gpt. For people do not know fluid, why they need it?", "questions": "The author should write down more about when, who will need this method to do what. And the flow past cylinder data seems under-resolved. The mesh is too coarse, can we make it finer?", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "It is first time people to get-like fluid sequence generation, to combine llm with diffusion is good.", "weaknesses": "The motivation is lacking. The flow simulation is not physical.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4Tbt1QO89x", "reviewer": "Reviewer_Somp", "summary": "This paper proposes a surrogate neural network model that solves PDE problems. The authors claim innovations in three parts: mesh en/decoder, latent physics solver, and conditioning. The mesh en/decoder unifies differently discretized solution data achieved from various numerical solvers. Unlike an autoregressive model, the latent physics solver utilizes a diffusion model and predicts a full spatiotemporal solution. For the conditioning, two mechanisms, i.e., state-based and text-based methods, are proposed.", "questions": "The example of Cylinder Flow describes that the physical variables are velocity and pressure. Do the reported L1 Loss values take into account both variables? If so, the errors of both variables should be discussed more thoroughly.\n\nSimilarly, it would be worth having further clarification of the error for two variables (velocity and density) in the Buoyancy-driven Flow example.\n\nIn Buoyancy-driven Flow, the samples vary with not only initial conditions but also buoyancy factors. Does the model get conditioned with the buoyancy factor? I guess not. Then, the trajectories (evolved with different buoyancy factors) represent somewhat different physics. In this case, I am not sure how the model has learned the underlying physics.\n\nIt would be valuable to see how the radius of a spatio-temporal ball affects the model performance.\n\nIt would be helpful to clarify how to define/train the kernel $\\kappa(\\cdot)$.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "Under the aim of developing a surrogate neural PDE solver, the paper claims three major novelties (also with their strengths) of the proposed model.\n\n1. En/decoding PDE data via a uniform spatiotemporal latent grid: Regardless of discretization, the PDE data is first encoded into the uniform grid via a weighted sum using a learned parameterized kernel. Its architecture unification is beneficial, and thanks to the benefit, the conventional CNN can outperform GNN and Neural Fields in training an autoencoder for irregular data.\n\n2. Latent diffusion model for entire solution trajectory prediction: Instead of a state prediction of the autoregressive model, the proposed neural PDE solver aims to predict the trajectory given initial and boundary conditions. This generative model can mitigate error accumulation, and the work compares with other models.\n\n3. Text conditioning: In addition to the traditional physics-state initial and boundary conditions, the paper proposes to use natural language as a means of conditioning. The authors claim that this can be a promising direction for PDE simulators.\n\nOverall, with the combination of these novelties, the paper claims their surrogate PDE solver outperforms the other models such as GINO, OFormer, FNO, ACDM, etc.\n\nAdditionally, thanks to the accompanying codes, the results should be reproducible (although I did not run the codes myself).\n\n---\n\n***During the discussion,*** the authors clarified my concerns; thus I raised my score to 6 from 5.", "weaknesses": "As the title \"text2pde\" stands for, the paper proposes a text-based conditioning for PDE. Although it is an interesting direction to investigate, the practical benefit of the current model is not fully convincing to me. If I am not wrong, the text-based conditioning of the proposed method eventually relies on a set of extracted values that are necessary to set up the initial condition, and this is only for conditioning a single initial frame. In my view, this can be easily and more precisely defined with a script-based (or template-based) initialization. Arguably, the text-based approach may rather be an ambiguous means to use in this context. Moreover, this part looks independent of the solver; it tackles how to generate an image (i.e., a PDE state in this context) from a text, which looks closer to text2image rather than text2pde.\n\nThe validation of the proposed solver could be further improved. Only two examples of N.-S. equations were investigated. The proposed entire-trajectory prediction was only evaluated for 25 steps in the cylinder flow case with low turbulence, which is relatively less complex, and 48 steps in the buoyancy-driven flow. Further experiments for the multi-step trajectory prediction would improve the validation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "gXI89kutQm", "reviewer": "Reviewer_zsUD", "summary": "This work presents a more accessible physics simulation by introducing the language model so that the physics solver can be interacted by using the test, called text2PDE. The proposed method is verified cylinder flow and buoyancy-driven flow although they cannot demonstrate enough the claimed effect.", "questions": "1. The reviewer has some doubts about the selected generative model. For such prediction and simulation tasks, diffusion models are not always effective due to their SDE features. Flow matching, which is based on ODEs, appears more suitable for PDE solvers. Could the authors discuss the application of both models for PDE solvers and explain why they chose diffusion models over flow matching?\n2. In line 229, \"we find that GANs and perceptual guidance can in certain cases improve reconstruction performance, but for simplicity, we omit them in our main results.\" The author said GANs can improve the performance, but why you do not apply GANs?\n3. In Figure 5, the middle two figures do not show turbulent flow, making this characterization inaccurate. Specifically, the second figure clearly shows laminar wake flow, and while the third figure shows vortices, it has not yet reached turbulent conditions.", "limitations": "", "rating": 3, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. The developed framework encodes and decodes arbitrary PDE data into a latent space to generate unstructured physics solutions.\n2. The language model is applied in the framework to make the PDE solver more accessible.\n3. The paper applies two experiments to demonstrate the superior performance of the proposed method.", "weaknesses": "1. The reviewer raises doubts about the novelty and contribution of the work, seemingly only using the encoding of the language model as a condition to generate solutions of PDEs.\n2. Why do the two experiments adopt different baselines and lack baselines such as FNO for cylinder flow?\n3. From line 047 to 050, while this statement may seem obvious, it's not a scientific issue. It is necessary to have physics and numerical knowledge to build PDE solvers. At the same time, approaches based on FNO, DeepOnet and similar numerically-driven methods can learn accurate PDE solvers without requiring physics knowledge.\n4. From line 053, this work deviates from the challenges described above, and there are concerns about potential overclaiming. Based on the author's two experiments, the proposed model only works in specific PDEs, spatiotemporal dimensions, and simple scenarios where the required physics knowledge is quite basic. In other words, the authors failed to address a crucial question: for complex PDE systems, how to represent the system using language, when mathematical formulas are clearly more suitable, while it is the main challenge the reviewer understand from the previous paragraph.\n5. Spatial and temporal resolution is crucial for PDE solvers. How does this work handle different spatial resolutions? Specifically, how does this method distinguish between initial conditions with different resolutions within the same system?\n6. Could you provide more details about the autoencoder in Sec. 3.1? This part is quite abstract and difficult for readers to understand intuitively. Please briefly explain the autoencoder's architecture, input/output structure, and training methodology. Additionally, during the diffusion model training, is the encoder trained simultaneously or are its parameters fixed?\n7. In line 235, the decoder appears to be CNN-based, which imposes constraints on the spatial dimensions of the output solutions. Furthermore, interpolation operations can lead to inaccuracies in the results, even when solutions on regular grids are accurate.\n8. In the method section, the authors have placed too many crucial details in the appendix, like Appendix C.2, B.1, B.2, D, making it impossible to understand the method without referring to the appendix.\n9. The selected parameters for cylinder flow are overly simplistic, only covering laminar flow cases. Generally, the Kármán vortex street phenomenon is more common in cylinder flows, yet the authors did not include this scenario.\n10. All data used is 2D, and it's unclear whether this model can handle 1D and 3D experiments. The lack of such experiments significantly reduces the persuasiveness of the work.\n11. Without knowing the scale of the data, reporting only L1 loss makes evaluation difficult. Could authors additionally report the relative L2 loss?\n12. As the authors acknowledge, the inference time of diffusion models limits their application, which is confirmed by the statistical results. The authors should consider using inference acceleration methods such as DDIM and verify that the method remains effective under DDIM sampling.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)