Learning Distributions of Complex Fluid Simulations with Diffusion Graph Networks

Authors: Mario Lino Valencia, Tobias Pfaff, Nils Thuerey
Primary area: applications to physical sciences (physics, chemistry, biology, etc.)
Keywords: graph neural networks, diffusion models, physics simulations

Abstract:
Physical systems with complex unsteady dynamics, such as fluid flows, are often poorly represented by a single mean solution. For many practical applications, it is crucial to access the full distribution of possible states, from which relevant statistics (e.g., RMS and two-point correlations) can be derived. Here, we propose a graph-based latent diffusion model that enables direct sampling of states from their equilibrium distribution, given a mesh discretization of the system and its physical parameters. This allows for the efficient computation of flow statistics without running long and expensive numerical simulations. The graph-based structure enables operations on unstructured meshes, which is critical for representing complex geometries with spatially localized high gradients, while latent-space diffusion modeling with a multi-scale GNN allows for efficient learning and inference of entire distributions of solutions. A key finding of our work is that the proposed networks can accurately learn full distributions even when trained on incomplete data from relatively short simulations. We apply this method to a range of fluid dynamics tasks, such as predicting pressure distributions on 3D wing models in turbulent flow, demonstrating both accuracy and computational efficiency in challenging scenarios. The ability to directly sample accurate solutions, and capturing their diversity from short ground-truth simulations, is highly promising for complex scientific modeling tasks.

Decision: accept (oral)
Reviewer scores: 8, 8, 8, 6, 8

Reviews and discussion:
<thread>
creation date:
1730213766509

thread_type:
official_review

<thread_object>
creator: reviewer_uxmj

<content>
summary:
This paper proposes using graph-based diffusion methods to sample steady states in fluid dynamics problems.  Impressively, the training does not need simulations run all the way to convergence.  Behind the scenes, the proposed model uses a multiscale graph network, downsampled using strategies that are typical in numerical PDE.

strengths:
The results in this paper are impressive, and the algorithm makes sense.  I am particularly glad to see relatively realistic test cases as well as reasonable engineering justification for the design behind the proposed architecture.

Also, thank you for preparing a polished, easy-to-read paper submission.  Relative to the other ICLR papers I had to read, this one had a thoughtful and clear discussion.

weaknesses:
From an algorithmic perspective, the building blocks of the proposed method are unsurprising and reflective of larger trends in the fields (graph networks with a multiscale aspect, diffusion sampling, and so on).  So, I would not consider this work to be too exciting from a methodological perspective, although perhaps a reader with more experience specifically in numerical fluid dynamics would better be able to evaluate this aspect.  I am willing to overlook this, however, as the method seems solidly engineered (why invent a new approach to ML just to claim novelty?) and the results look great.

questions:
Line 107, “defined their discretization mesh” (typo?)

Can the authors provide intuition on why the problem described on line 142 is well posed?  In particular, if the learning method exclusively sees short trajectories that end before reaching equilibrium, why should this information be sufficient to train a method that directly predicts converged states it hasn’t seen in the training data?  The results seem to indicate this is possible, but it seems counterintuitive to me that this task is reasonable.

Line 169, “interpolates between the bounds of the process’ entropy” --- I wasn’t sure what this means.  Also, (Ho et al., 2020) isn’t written for graph-based models --- does this make a difference?

The receptive field of a “vanilla” DGN (line 180) is relatively small --- typically a vertex only “talks to” its k-ring, where k is the number of layers.  Does this affect performance of the proposed model, even after incorporating the latent diffusion method?

Line 207:  What does “perceptually equivariant” mean?  How do you verify?

Line 249:  no need for a hyphen in “bottleneck”

Line 384:  Do statistical limitations get in the way of estimating W_2^graph accurately?  It seems like this would be a high-dimensional optimal transport problem, which itself can be difficult to estimate reliably from samples.  How many samples are used here?

Line 451:  If I understand properly, the “3x faster” claim here would be substantially dampened if you had to include training time of the diffusion model.  Is this right?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730330214223

thread_type:
official_review

<thread_object>
creator: reviewer_84s9

<content>
summary:
The paper aims to predict possible steady states of complex unsteady dynamical systems given their mesh discretization and physical parameters. With a novel graph-based (latent) diffusion model to predict equilibrium states of dynamical systems, the proposed model can handle unstructured mesh.

strengths:
- Integrating GNN with a diffusion model is an interesting idea, as it allows for better generality and great flexibility in unstructured mesh.
- The paper claims to predict a full distribution of possible equilibrium states, although trained on states of a short trajectory.

weaknesses:
Some critical information is missing. 
- The underlying physics problem needs to be explained more. I would expect some rigorous mathematical definitions such as PDEs to formulate the problem.  
- The related work should cover more classical approaches to the problem of predicting equilibrium states of dynamical systems. 
- The learning problem should also be better formulated.

questions:
- What is a statistical equilibrium?
- What is the generality of the model? Does it generalize to different trajectories, different physical parameters, or even different domain shapes or sizes?
- Generally, how complex a dynamical system is the model able to handle? Some dynamical systems such as the Lorenz system are chaotic, so are they still predictable with the proposed model?

---
**Update**

Thank you for your detailed response. I think I have a better understanding of the concepts, and I believe it is meaningful work. Thus, I decided to raise my score.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730392504624

thread_type:
official_review

<thread_object>
creator: reviewer_j9he

<content>
summary:
This paper uses Denoising Diffusion to sample states from the equilibrium distribution of fluid flow tasks. Instead of operating on a 2D grid, it employs a graph-based approach combined with a multi-scale Graph Neural Network (GNN) to process information. Using only short target trajectories, the model can learn the full distribution of solutions. Additionally, by combining the diffusion approach with a variational graph auto-encoder, a latent version that operates on a reduced latent graph with fewer nodes can, in certain scenarios, improve results.

strengths:
- Novelty and Innovation: The paper presents a novel and compelling approach by combining multi-scale Graph Neural Networks (GNNs), Denoising Diffusion Probabilistic Models (DDPM), and Variational Graph Auto-Encoders (VGAEs), pushing the boundaries in fluid flow modeling.

- Clear and Well-Structured Presentation: The authors provide a thorough and well-organized presentation, clearly explaining the approach and the motivation behind it, making it evident why this method is both practical and impactful.

- Relevant and Thoughtful Application: The paper demonstrates a meaningful application of the algorithm, showcasing how the method can effectively address specific challenges in fluid flow tasks.

- Strong Results and Evaluation: The results are impressive, backed by a robust evaluation that validates the approach's effectiveness and potential advantages over existing methods.

weaknesses:
- Lack of Code Submission: The absence of a code submission limits the reproducibility and accessibility of the results for further research and practical implementation.

- Minor Formatting Issues: Acronyms are not consistently introduced upon first appearance, which could impact readability. For instance, "DDPM" is used without prior definition (e.g., on line 76).

- Missing Citation: The original GAN paper by Goodfellow (2014) is not cited (e.g., on line 74).

questions:
- Do you plan to release the code in the future, or are there specific reasons it hasn’t been shared yet? If not, could you provide further details on any proprietary components or dependencies that may impact reproducibility? In the paper, the authors only mentioned the release of the  datasets.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730426668854

thread_type:
official_review

<thread_object>
creator: reviewer_prac

<content>
summary:
This paper presents two graph-based diffusion models for generating equilibrium states of fluid dynamics systems: one employing diffusion in physical space and another in the latent space generated by graph encoders. The presentation is clear and the methodological choices appear well-reasoned, particularly the use of multi-scale GNN as an encoder to support the latent diffusion model. The approach shows promise for generalization to broader applications where steady state prediction is needed.

While technically sound, the primary limitation lies in the methodology's originality. The work largely assembles existing methods rather than introducing novel techniques, though this integration is done throughoutly.

I assumed two important references should also be included and addressed to help with the literature review of this paper.

1. "Direct Prediction of Steady-State Flow Fields in Meshed Domain with Graph Networks" should be included as a baseline:
    - it directly addresses steady-state flow prediction using GNNs. This leads me to question when we should prefer deterministic versus generative approaches.
    - I can see their generative method working well when "steady states" show variations (like the periodic wakes in von Kármán vortices), but for cases with strictly stationary steady states - like this paper's ellipse and airfoil examples - wouldn't a deterministic approach make more sense?
    - I'd strongly encourage the authors to include this baseline and run comparative experiments and discuss these trade-offs based on results

2. "Efficient Learning of Mesh-Based Physical Simulation with Bi-Stride Multi-Scale Graph Neural Network," which shares similar ideas about multi-scale GNN architecture and more importantly an automatic graph hierarchy construction.
    - Frankly, compared to a refered paper in the current manuscript: (Fortunato 2022) which requires manual drawing of coarsen layer, this paper's automatic graph hierarchy construction is closer pipeline-wise. Hence I think it should be acknowledged in the related work, and the related method paragraphs.
    - While I don't think they need to conduct ablation studies here, as it's not the main focus, mentioning it would strengthen their literature review.

I recommend a borderline accept for this paper now. I would consider improving my score if the authors address these missing references, and if they add the suggested baseline and the comparative analysis between deterministic and generative approaches, and thoroughly discuss when each method works best.

strengths:
- Clear writing
- The components of the whole pipeline proposed are rather reasonable
- The experiment details, the network input/output structure, and the architectures are clear and hence the reproductbility should be easy

weaknesses:
- A bit lack of innovation of the components of the whole method

questions:
- NA
</content>
</thread_object>

</thread>
<thread>
creation date:
1730677909943

thread_type:
official_review

<thread_object>
creator: reviewer_5xrz

<content>
summary:
For many fluid dynamics problems, the exact trajectory dynamics are not relevant; instead, domain scientists are more concerned with quantities of interest that arise from steady state distributions. The paper proposes a multi-scale GNN architecture contextualized within the DDPM framework. Similar to prior and current state-of-the-art diffusion models, the work uses a latent representation, instead of performing diffusion over the original data space. In order to differentiate between both contributions, experiments are performed on both DGNs (Diffusion Graph Networks) and LDGNs (Latent Diffusion Graph Networks).

Experiments are performed on 3 experimental settings: 1) pressure on the wall of an ellipse in 2D laminar flow 2) velocity and pressure around the ellipse 3) pressure on a 3D wing. The baselines include a Bayesian GNN, a Gaussian mixture model, a Gaussian regression model, and a VAE.

strengths:
- The studied problem is of particular interest to the broader physics-inspired ML community and domain scientists. 
- Both DGN and LDGN clearly outperform the tested baselines, and show promising results.
- The mathematical and architectural details are well-written.
- To my knowledge, this is one of the first works using diffusion models + GNNs in the space of ML4PDEs.

weaknesses:
- The compared baselines are not widely used or state-of-the-art methods. At the same time, the three datasets, while representative or real-world tasks, are newly generated. This makes the results within the paper hard to contextualize within the broader literature.
- The related works section on DDPMs for graph structure needs to further differentiate the proposed method from existing works.

questions:
- How does the proposed method differentiate itself from [1] (cited in the related works) which also uses a UNet style GNN? Similarly, how is the proposed GNN different / improved upon from [2]?
- Are performance improvements over the baselines a result of the diffusion model formulation or the GNN design? Have the authors considered using a latent DiT or different sampling approach (e.g., DDIM)?

[1] Haomin Wen, Youfang Lin, Yutong Xia, Huaiyu Wan, Qingsong Wen, Roger Zimmermann, & Yuxuan Liang. (2024). DiffSTG: Probabilistic Spatio-Temporal Graph Forecasting with Denoising Diffusion Models.

[2] Lino, M., Fotiadis, S., Bharath, A., & Cantwell, C. (2022). Multi-scale rotation-equivariant graph neural networks for unsteady Eulerian fluid dynamics. Physics of Fluids, 34(8), 087110.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732026355302

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General reply to all reviewers

comment:
We thank the reviewers for their thoughtful feedback, which we are incorporating to improve the completeness and clarity of our manuscript. These modifications and additions are detailed in our replies to each reviewer. Below, we highlight the most significant updates:

- We have clarified the differences between our method and prior work in the Related Work section and Appendix B.
- We have clarified the concept of statistical equilibrium and provided a precise definition of the learning problem in the manuscript.
- To further demonstrate the generality of our method, we include performance results for a new test dataset for the Ellipse and EllipseFlow tasks. This dataset features an ellipse with a 10-degree angle relative to the free stream, whereas the training data assumes a 0-degree angle.
- We justify the choice of our sampling approach and introduce Flow Matching as an alternative for faster inference. We will add an appendix with preliminary comparisons between DGN, LDGN, Flow Matching (FM), and Latent FM (LFM) models on the EllipseInDist dataset. Our findings show that the proposed method naturally extends to Flow Matching, and yields accurate samples with fewer sampling steps (10 for this scenario). 
- Reviewer praC raised concerns about the lack of a deterministic baseline trained specifically on the mean flow. To address this, we will include an additional baseline and compare its performance with the DGN and the “vanilla GNN”. The latter already represents a deterministic GNN trained to match the expectation of states observed during training.
- In the original Results section, we hypothesized that the success of diffusion-based models in learning the full distribution from short trajectories stems from their ability to identify shared physical patterns across systems. To strengthen this claim, we conducted a new experiment to assess model performance when trained on datasets with varying numbers of short trajectories. Results show that DGN and LDGN outperform baseline models, suggesting that these models effectively interpolate among trajectories, whereas baseline models rely more heavily on having seen specific data.
- We have updated the GPU runtime for PDF estimation in Table 10. The revised measurement was taken with nearly 100% GPU utilization, unlike the previous measurement, which did not fully utilize available GPU power. CPU and single-sample-generation runtime remain unchanged.

We will post an update once the revised manuscript has been uploaded.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper on computational fluid dynamics proposes a multi-scale GNN architecture combined with diffusion models working both in physical and in latent space. Five expert reviewers provided very favorable reviews and appreciated novelty, technical soundness, excellent performance, generalization capabilities, an well-written paper, strong evaluation, good discussions. The authors could answer the questions by the reviewers, and there the assessments were consistent and positive. The AC concurs and recommends acceptance.