Text2PDE: Latent Diffusion Models for Accessible Physics Simulation

Authors: Anthony Zhou, Zijie Li, Michael Schneier, John R Buchanan Jr, Amir Barati Farimani
Primary area: applications to physical sciences (physics, chemistry, biology, etc.)
Keywords: AI4Science, PDE, Neural Operator, Latent Diffusion, Text2PDE

Abstract:
Recent advances in deep learning have inspired numerous works on data-driven solutions to partial differential equation (PDE) problems. These neural PDE solvers can often be much faster than their numerical counterparts; however, each presents its unique limitations and generally balances training cost, numerical accuracy, and ease of applicability to different problem setups. To address these limitations, we introduce several methods to apply latent diffusion models to physics simulation. Firstly, we introduce a mesh autoencoder to compress arbitrarily discretized PDE data, allowing for efficient diffusion training across various physics. Furthermore, we investigate full spatiotemporal solution generation to mitigate autoregressive error accumulation. Lastly, we investigate conditioning on initial physical quantities, as well as conditioning solely on a text prompt to introduce text2PDE generation. We show that language can be a compact, interpretable, and accurate modality for generating physics simulations, paving the way for more usable and accessible PDE solvers. Through experiments on both uniform and structured grids, we show that the proposed approach is competitive with current neural PDE solvers in both accuracy and efficiency, with promising scaling behavior up to $\sim$3 billion parameters. By introducing a scalable, accurate, and usable physics simulator, we hope to bring neural PDE solvers closer to practical use.

Decision: accept (poster)
Reviewer scores: 6, 5, 6, 6, 6, 3

Reviews and discussion:
<thread>
creation date:
1730271483697

thread_type:
official_review

<thread_object>
creator: reviewer_zsud

<content>
summary:
This work presents a more accessible physics simulation by introducing the language model so that the physics solver can be interacted by using the test, called text2PDE. The proposed method is verified cylinder flow and buoyancy-driven flow although they cannot demonstrate enough the claimed effect.

strengths:
1. The developed framework encodes and decodes arbitrary PDE data into a latent space to generate unstructured physics solutions.
2. The language model is applied in the framework to make the PDE solver more accessible.
3. The paper applies two experiments to demonstrate the superior performance of the proposed method.

weaknesses:
1. The reviewer raises doubts about the novelty and contribution of the work, seemingly only using the encoding of the language model as a condition to generate solutions of PDEs.
2. Why do the two experiments adopt different baselines and lack baselines such as FNO for cylinder flow?
3. From line 047 to 050, while this statement may seem obvious, it's not a scientific issue. It is necessary to have physics and numerical knowledge to build PDE solvers. At the same time, approaches based on FNO, DeepOnet and similar numerically-driven methods can learn accurate PDE solvers without requiring physics knowledge.
4. From line 053, this work deviates from the challenges described above, and there are concerns about potential overclaiming. Based on the author's two experiments, the proposed model only works in specific PDEs, spatiotemporal dimensions, and simple scenarios where the required physics knowledge is quite basic. In other words, the authors failed to address a crucial question: for complex PDE systems, how to represent the system using language, when mathematical formulas are clearly more suitable, while it is the main challenge the reviewer understand from the previous paragraph.
5. Spatial and temporal resolution is crucial for PDE solvers. How does this work handle different spatial resolutions? Specifically, how does this method distinguish between initial conditions with different resolutions within the same system?
6. Could you provide more details about the autoencoder in Sec. 3.1? This part is quite abstract and difficult for readers to understand intuitively. Please briefly explain the autoencoder's architecture, input/output structure, and training methodology. Additionally, during the diffusion model training, is the encoder trained simultaneously or are its parameters fixed?
7. In line 235, the decoder appears to be CNN-based, which imposes constraints on the spatial dimensions of the output solutions. Furthermore, interpolation operations can lead to inaccuracies in the results, even when solutions on regular grids are accurate.
8. In the method section, the authors have placed too many crucial details in the appendix, like Appendix C.2, B.1, B.2, D, making it impossible to understand the method without referring to the appendix.
9. The selected parameters for cylinder flow are overly simplistic, only covering laminar flow cases. Generally, the Kármán vortex street phenomenon is more common in cylinder flows, yet the authors did not include this scenario.
10. All data used is 2D, and it's unclear whether this model can handle 1D and 3D experiments. The lack of such experiments significantly reduces the persuasiveness of the work.
11. Without knowing the scale of the data, reporting only L1 loss makes evaluation difficult. Could authors additionally report the relative L2 loss?
12. As the authors acknowledge, the inference time of diffusion models limits their application, which is confirmed by the statistical results. The authors should consider using inference acceleration methods such as DDIM and verify that the method remains effective under DDIM sampling.

questions:
1. The reviewer has some doubts about the selected generative model. For such prediction and simulation tasks, diffusion models are not always effective due to their SDE features. Flow matching, which is based on ODEs, appears more suitable for PDE solvers. Could the authors discuss the application of both models for PDE solvers and explain why they chose diffusion models over flow matching?
2. In line 229, "we find that GANs and perceptual guidance can in certain cases improve reconstruction performance, but for simplicity, we omit them in our main results." The author said GANs can improve the performance, but why you do not apply GANs?
3. In Figure 5, the middle two figures do not show turbulent flow, making this characterization inaccurate. Specifically, the second figure clearly shows laminar wake flow, and while the third figure shows vortices, it has not yet reached turbulent conditions.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730288238420

thread_type:
official_review

<thread_object>
creator: reviewer_somp

<content>
summary:
This paper proposes a surrogate neural network model that solves PDE problems. The authors claim innovations in three parts: mesh en/decoder, latent physics solver, and conditioning. The mesh en/decoder unifies differently discretized solution data achieved from various numerical solvers. Unlike an autoregressive model, the latent physics solver utilizes a diffusion model and predicts a full spatiotemporal solution. For the conditioning, two mechanisms, i.e., state-based and text-based methods, are proposed.

strengths:
Under the aim of developing a surrogate neural PDE solver, the paper claims three major novelties (also with their strengths) of the proposed model.

1. En/decoding PDE data via a uniform spatiotemporal latent grid: Regardless of discretization, the PDE data is first encoded into the uniform grid via a weighted sum using a learned parameterized kernel. Its architecture unification is beneficial, and thanks to the benefit, the conventional CNN can outperform GNN and Neural Fields in training an autoencoder for irregular data.

2. Latent diffusion model for entire solution trajectory prediction: Instead of a state prediction of the autoregressive model, the proposed neural PDE solver aims to predict the trajectory given initial and boundary conditions. This generative model can mitigate error accumulation, and the work compares with other models.

3. Text conditioning: In addition to the traditional physics-state initial and boundary conditions, the paper proposes to use natural language as a means of conditioning. The authors claim that this can be a promising direction for PDE simulators.

Overall, with the combination of these novelties, the paper claims their surrogate PDE solver outperforms the other models such as GINO, OFormer, FNO, ACDM, etc.

Additionally, thanks to the accompanying codes, the results should be reproducible (although I did not run the codes myself).

---

***During the discussion,*** the authors clarified my concerns; thus I raised my score to 6 from 5.

weaknesses:
As the title "text2pde" stands for, the paper proposes a text-based conditioning for PDE. Although it is an interesting direction to investigate, the practical benefit of the current model is not fully convincing to me. If I am not wrong, the text-based conditioning of the proposed method eventually relies on a set of extracted values that are necessary to set up the initial condition, and this is only for conditioning a single initial frame. In my view, this can be easily and more precisely defined with a script-based (or template-based) initialization. Arguably, the text-based approach may rather be an ambiguous means to use in this context. Moreover, this part looks independent of the solver; it tackles how to generate an image (i.e., a PDE state in this context) from a text, which looks closer to text2image rather than text2pde.

The validation of the proposed solver could be further improved. Only two examples of N.-S. equations were investigated. The proposed entire-trajectory prediction was only evaluated for 25 steps in the cylinder flow case with low turbulence, which is relatively less complex, and 48 steps in the buoyancy-driven flow. Further experiments for the multi-step trajectory prediction would improve the validation.

questions:
The example of Cylinder Flow describes that the physical variables are velocity and pressure. Do the reported L1 Loss values take into account both variables? If so, the errors of both variables should be discussed more thoroughly.

Similarly, it would be worth having further clarification of the error for two variables (velocity and density) in the Buoyancy-driven Flow example.

In Buoyancy-driven Flow, the samples vary with not only initial conditions but also buoyancy factors. Does the model get conditioned with the buoyancy factor? I guess not. Then, the trajectories (evolved with different buoyancy factors) represent somewhat different physics. In this case, I am not sure how the model has learned the underlying physics.

It would be valuable to see how the radius of a spatio-temporal ball affects the model performance.

It would be helpful to clarify how to define/train the kernel $\kappa(\cdot)$.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730476072133

thread_type:
official_review

<thread_object>
creator: reviewer_jpnk

<content>
summary:
Very new paper. I support this because it actually works, I appreciate the authors working! Just like language model for catalyst. But I just do not see why people need this as they need for catalyst-gpt. The authors know fluid, they do not need this even they can make this fluid-gpt. For people do not know fluid, why they need it?

strengths:
It is first time people to get-like fluid sequence generation, to combine llm with diffusion is good.

weaknesses:
The motivation is lacking. The flow simulation is not physical.

questions:
The author should write down more about when, who will need this method to do what. And the flow past cylinder data seems under-resolved. The mesh is too coarse, can we make it finer?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730650764348

thread_type:
official_review

<thread_object>
creator: reviewer_7eso

<content>
summary:
The authors introduce a new approach to physics simulations using latent diffusion models. Their method is anchored by (i) the use of mesh autoencoders, (ii) full spatio-temporal modeling to avoid autoregressive error accumulation, and (iii) conditioning on text prompts.

strengths:
- Latent space modeling for physics simulations. By using latent diffusion models, the authors effectively compress and encode high-dimensional physics data into a lower-dimensional latent space. Such latent space approaches contribute to more adaptable and faster simulations, making it a powerful alternative to traditional numerical solvers. There is a surge of latent space PDE models, this paper fits well into this paradigm. The full spatio-temporal diffusion modeling conditioned on text prompts and the subsequent decoding stands out as unique modeling approach.

- Text prompts for simulation control. Text2pde is an interesting direction since it allows users to define simulation parameters or initial conditions using simple, interpretable text commands. The potential for language-based input makes physics simulations more accessible and generalizable. Yet it is not clear what is the difference to simple parameter modulation, the nice capturing of the initial condition is cool.

- Application to mesh-based dynamics.

weaknesses:
- Text2pde seems to be hard to scale to larger problems. To avoid confusion: scalability here refers to input representations. The mesh autoencoder is a good direction, yet the latent space is a latent grid, which if going to 3D is fundamentally limiting. Also for larger 2D grids, a tokenized representation might be preferable. Secondly, the generation of the full trajectory is storage intensive, and for 3D problems would result in a 4D tensor, which is computationally infeasible. Most turbulent problems, where neural surrogates might shine are 3D, presenting a drawback of the applicability of the method.
- Text conditioning in this framework seems to be a slightly sophisticated parameter conditioning / parameter modulation. The nice feature so far is the generation of the initial mesh, initial condition. It would greatly help to make these differences clearer. Further the text prompt part needs to be adapted for each problem individually. It should be clarified how the text part can maybe be used across settings? Is this even possible?

questions:
- Autoregressive vs entire solutions. It is not clear whether producing an entire solution trajectory is really beneficial. First of all it doesn’t scale to 3D problems or larger 2D problems. Secondly, it might be unphysical. Do the authors have insights on this?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730696127864

thread_type:
official_review

<thread_object>
creator: reviewer_9ptm

<content>
summary:
The paper introduces a framework called TEXT2PDE, which employs text-conditioned latent diffusion models (LDMs) to model spatio-temporal PDE trajectories. The authors introduce a mesh autoencoder that can encode and decode physical fields on arbitrary meshes. Spatio-temporal trajectories are generated using a conditional denoising process of an LDM with the initial conditions and boundary conditions of the PDE imposed into the conditioning sequence of the denoising backbone. A main novelty is the introduction of text conditioning, by employing a pre-trained transformer encoder. The authors argue that using natural language as a conditioning modality can improve the interpretability and usability of neural PDE solvers.

strengths:
1. Thanks to the encoding of arbitrary meshes, the authors are able to apply their method to complex geometries – an important use case for the application of PDEs.
1. The main strength of the paper is the introduction of text conditioning for generation of PDE trajectories. The idea is novel and opens up a new research direction.
However, it would be still interesting to see more specific motivating examples of where text conditioning could be useful in real-world use cases of PDE solvers.

weaknesses:
1. The evaluations seem lacking when assessing the text conditioning capability of the Text2PDE model. Firstly, the initial and boundary conditions of the cylinder dataset can be entirely encoded using just three real numbers, making the expressiveness of natural language excessive for this case. Secondly, the smoke dataset appears to be a poor fit for text-based description, as the smoke plumes are too unconstrained and unstructured to be accurately captured by language. (See also the question 1.)

2. In my opinion, evaluating text conditioning would require a new dataset with more regular shapes placed in general positions. The configurations should be rich enough to necessitate natural language for description, but constrained enough that a short text can accurately describe the image. Currently, it seems that the TEXT2PDE system is limited to using text conditioning to generate only the initial condition. This could be achieved more directly by combining text-to-image techniques and then running a standard (neural) PDE solver. The added benefit of using a combined system is not clear. It could be more interesting (and useful) if the PDE dynamics were also informed by natural language—perhaps through a text-encoded equation.

questions:
1. As the text prompts appear to accurately capture the first frame of the reference (as shown in Table 2 and Figure 4 in Appendix A), I wonder if the model is memorizing and retrieving the smoke plume from the training dataset that most closely matches the input. For the existing prompts, would it be possible to display the most similar first frame from the training set?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730879975156

thread_type:
official_review

<thread_object>
creator: reviewer_tcka

<content>
summary:
this paper introduce framework for text to simulation use latent diffusion models, enables condition on language, initial conditions. this work generate a entire trajectories for improved accuracy without error accumulation.

strengths:
- apply latent diffusion models to physics simulation with mesh encoder/decoder provide a scalable solution. 
- a new text conditioned PDE simulation framework. 
- supporting both structured and unstructured grids, and shows promising results on different types of PDE problems

weaknesses:
-  it's unclear the benefits and why doing text-conditioned simulation generation
- I find paper's contribution is more on using latent diffusion for pde simulation, not the text to PDE generation.

questions:
- what is re-solved trajectory referred to in line 394?
- why doing text-conditioned simulation generation, is the pde type pre-defined for each model? How important is the training data text caption?
- how does text-conditioning help the simulation results or they are dis-entangled.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732501006927

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Thank you to all Reviewers

comment:
Thank you to reviewers for writing thoughtful reviews. We have individually replied to each review and updated the manuscript, however for easy reference we have summarized the changes here:

## Major Changes
- Added new 3D Turbulence experiments to investigate scalability + complex systems
    - Section 4.3: Main results + errors
    - Appendix A: New visualizations
    - Appendix E.3: Dataset description
    - Appendix D.3: Captioning description
    - Appendix G: Added inference times
- Added new experiments to allow the LDM framework to also be used auto-regressively to enable scaling to longer time-horizons and multi-step prediction
    - Appendix C.4: Main results + method explanation
- Added new experiments using DDIM to accelerate LDM sampling  
    - Appendix C.3: Main results + rationale/explanation

## Minor Changes
- Added more details on the kernel aggregation/interpolation to explain discretization invariance.
    - Appendix B.3: Improved explanation and new visualizations
- Added more details about the Cylinder Flow dataset
    - Appendix E.1: Improved explanation and new visualizations
- Changed L1 Loss to Relative L2 Loss in main text tables
    - Tables 1 and 2: Changed to L2 loss
- Added ablation study on ball radius
    - Appendix B.1: New table and discussion
- Clarified Cylinder Flow captioning
    - Appendix D.1: minor clarifications

We encourage reviewers to browse the updated manuscript due to the large number of changes. Additionally, we apologize for the late reply, however, we are still interested if there are any additional comments or thoughts. Thank you!
</content>
</thread_object>

</thread>
<thread>
creation date:
1732606705043

thread_type:
official_comment

<thread_object>
creator: area_chair_xkq5

<content>
comment:
Dear all,

The deadline for the authors-reviewers phase is approaching (December 2).

@For reviewers, please read, acknowledge and possibly further discuss the authors' responses to your comments. While decisions do not need to be made at this stage, please make sure to reevaluate your score in light of the authors' responses and of the discussion.

- You can increase your score if you feel that the authors have addressed your concerns and the paper is now stronger.
- You can decrease your score if you have new concerns that have not been addressed by the authors.
- You can keep your score if you feel that the authors have not addressed your concerns or that remaining concerns are critical.

Importantly, you are not expected to update your score. Nevertheless, to reach fair and informed decisions, you should make sure that your score reflects the quality of the paper as you see it now. Your review (either positive or negative) should be based on factual arguments rather than opinions. In particular, if the authors have successfully answered most of your initial concerns, your score should reflect this, as it otherwise means that your initial score was not entirely grounded by the arguments you provided in your review. Ponder whether the paper makes valuable scientific contributions from which the ICLR community could benefit, over subjective preferences or unreasonable expectations.

@For authors, please respond to remaining concerns and questions raised by the reviewers. Make sure to provide short and clear answers. If needed, you can also update the PDF of the paper to reflect changes in the text. Please note however that reviewers are not expected to re-review the paper, so your response should ideally be self-contained.

The AC.
</content>
</thread_object>

</thread>


Metareview:
metareview:
The reviewers are divided (6-5-6-6-6-3) about the paper, but they overall lean towards acceptance. The paper introduces a new approach to physics simulations using latent diffusion models that can be conditioned on text prompts. Along the way, the paper introduces full 4d spatiotemporal generation and diffusion on arbitrary (non-regular) meshes. A number of concerns have been raised by the reviewers, and most of them have been addressed during the author-reviewer discussion period. The proposed changes include clarifications and new experiments that better investigate Text2PDE's capabilities.

The main concern that remains is with regards to the text conditioning capability of Text2PDE. Conditioning on text prompts is an interesting direction but makes the evaluation more challenging, as text descriptions are inherently fuzzy. To address this issue, a new dataset with 3d shapes has been proposed by the authors.

Overall, while the usefulness and validation of text conditioning may still be debated, the paper presents valuable contributions in the context of physics simulations using latent diffusion models. For these reasons, and given the overall positive reviews, I recommend acceptance. I encourage the authors to address the remaining concerns and to implement the modifications discussed with the reviewers in the final version of the paper.