Physics-enhanced Neural Operator: An Application in Simulating Turbulent Transport

Authors: Shengyu Chen, Peyman Givi, Can Zheng, Xiaowei Jia
Primary area: applications to physical sciences (physics, chemistry, biology, etc.)
Keywords: turbulent flow, neural operator, knowledge-guided machine learning, sequential simulation

Abstract:
The precise simulation of turbulent flows is of immense importance in a variety of scientific and engineering fields, including climate science, freshwater science, and the development of energy-efficient manufacturing processes. Within the realm of turbulent flow simulation, direct numerical simulation (DNS) is widely considered to be the most reliable approach, but it is prohibitively expensive for long-term simulation at fine spatial scales. Given the pressing need for efficient simulation, there is an increasing interest in building machine learning models for turbulence, either by reconstructing DNS from alternative low-fidelity simulations or by predicting DNS based on the patterns learned from historical data. However, standard machine learning techniques remain limited in capturing complex spatio-temporal characteristics of turbulent flows, resulting in limited performance and generalizability. This paper presents a novel physics-enhanced neural operator (PENO) that incorporates physical knowledge of partial differential equations (PDEs) to accurately model flow dynamics. The model is further refined by a self-augmentation mechanism to reduce the accumulated error in long-term simulations. The proposed method is evaluated through its performance on two distinct sets of 3D turbulent flow data, showcasing the model's capability to reconstruct high-resolution DNS data, maintain the inherent physical properties of flow transport, and generate flow simulations across various resolutions. Additionally, experimental results on multiple 2D vorticity flow series, generated by different PDEs, highlight the transferability and generalizability of the proposed method. This confirms its applicability to a wide range of real-world scenarios in which extensive simulations are needed under diverse settings.

Decision: reject
Reviewer scores: 6, 5, 5, 6, 5, 3

Reviews and discussion:
<thread>
creation date:
1730191712726

thread_type:
official_review

<thread_object>
creator: reviewer_gwq8

<content>
summary:
This paper proposes a physics enhanced neural operator (PENO) for regressing Direct Numerical Simulation (DNS) states to accurately  model turbulent flows. The proposed pipeline consists of a next-step neural predictor that takes an input state of the solver, consisting of a combination of DNS and Large Eddy Simulation (LES) variables (velocity/vorticity/pressure) at a given time step $t$. Then the PENO method predicts the next time step; this can be done recursively to predict a series of steps, effectively advancing the simulation without requiring a numerical solver. The proposed approach is based on Fourier Neural Operators (FNOs), which approximate the Navier-Stokes Partial Differential Equation (PDE) solution through a transformation of the input variables to Fourier space. However, trained FNOs with mean squared error losses have two shortcomings: limited generalization and reduced accuracy due ignorance of the underlying PDE, and impaired ability to capture high-frequency information of the learned dataset. The authors tackle these issues by estimating temporal gradients through direct assessment of the underlying PDE and through augmenting the regression with an additional network branch that performs super-resolution to capture high-frequency details.  The method is trained and tested in two datasets: the Forced Isotropic Turbulent flow (FIT) and the Taylor-Green vortex (TGV) flow, and the authors present some quantitative and qualitative analysis comparing the proposed approach against previous methods.

strengths:
The authors correctly identify issues with the known  spectra bias in networks trained with MSE losses. These approaches tend to fail capturing crucial high-frequency information of the learned dataset. The proposed architecture is compared against previous baselines, and it shows interesting qualitative evaluations for next time step DNS prediction. Lastly, using the PDE information during training is an effective way to improve accuracy and generalization.

weaknesses:
My biggest concern with papers that propose to replace solvers is their usual lack of fair comparison to a well implemented physics solver running on similar hardware (GPU) (for a thorough analysis check "Weak baselines and reporting biases lead to overoptimism in
machine learning for fluid-related partial differential equations"). A solver is a computational graph that can accurately predict the next state without the usual issues of data-driven approaches (lack of generalization, reduced accuracy). Without a clear intuition why a neural approach is able to make such a computational graph more efficient (e.g., reducing the dimensionality of the input through model reduction), it’s hard to justify why such a method is useful in the first place. This is the case for this paper, as the authors didn’t provide a fair comparison between their method and a DNS solver.  The authors could include a comparison of computational efficiency and accuracy between their method and a state-of-the-art DNS solver implemented on similar hardware for better assessing the efficiency of such proposed approach. Additionally, the authors could also more explicitly discuss the potential advantages of their neural approach over traditional solvers.

The paper also could have done a better job of evaluating errors in time, which are one of the common issues of regressing simulations with neural approaches, as this issue is not present in traditional solvers. Neural approaches that integrate in time tend to highly diverge relative to a ground truth solver, check “Stability analysis of chaotic systems from data” and “How Temporal Unrolling Supports Neural Physics Simulators” for further references. Furthermore, the authors could include a more detailed analysis of how prediction errors accumulate over time, showing more precise error metrics (apart from the ones presented in Figure 5) for different prediction horizons. You could also suggest they discuss how their method addresses (or doesn't address) the issue of divergence over time compared to traditional solvers.

Moreover, the chosen network architecture/approach for the regression of the next time step is outdated, since currently many other authors are now relying on the power of diffusion models to properly capture the complex behavior of fluid simulations. The authors even add a bit of noise to “improve generalization”, but it’s stated in the paper that a careful evaluation of such an approach is needed. I would suggest that just implementing a straightforward conditioned diffusion model would be a better solution. Otherwise, the authors could also compare their method against a state-of-the-art diffusion model approach, explaining why they chose their current architecture over more recent alternatives. The two-stage network solution also seems a bit cumbersome, and I suspect one could do the regression in a single-step. Perhaps the authors could do a better job at justifying why this approach was chosen over a single-step regression.

The manuscript, in its current version, also contains several exposition issues. Several typos (e.g, L025: “. we further”, L030: “results confirms”, L082: “training data are scarce”, L144: “denotes”, L174: “filtration” -> filtering, L223 (Figure 3): “Naiver Stoke”, L241: “appendix”, L275: “do not require”, L278: “, We create output”, L341: “5, 024”, L478: “. the simulated”, L494: “faces increased” are present. These errors reduce readability and can be confusing to the reader. 

Lastly, the authors miss out several important references that are relevant to this work: The following papers are of direct relevance to the current submission: “Benchmarking Autoregressive Conditional Diffusion Models for Turbulent Flow Simulation”, “Uncertainty-aware Surrogate Models for Airfoil Flow Simulations with Denoising Diffusion Probabilistic Models”,  “Unsteady Cylinder Wakes from Arbitrary Bodies with Differentiable Physics-Assisted Neural Network” and “How Temporal Unrolling Supports Neural Physics Simulators”.

The aforementioned weaknesses and questions present in the next section justify my score for this paper.

questions:
- At L141, it is included a transformation from the spatial domain to the Fourier domain for $\mathbf{Q}(t)$ at time $t$. Is the integration domain for the Fourier transformation to be a for a fixed time step, or is it for multiple time steps? The Fourier transform is written through integrating on $dt$, but I believe that the authors meant integration on the Fourier domain. Consider writing these variables differently for better exposition.   

- At L209 the authors parametrize the non-linear function that represents the temporal derivative of $\mathbf{Q}$ through a parameter $\theta$. Why is this relevant? I could not find where this parameter was referenced in other places of the manuscript. 

- Equation 1 denotes the evolution of an arbitrary variable $\mathbf{Q}$ when $\mathbf{Q} = \vec{v}$, where $\vec{v}$ is the fluid velocity. Why not write it only for velocity? If ones considers the Navier Stokes equation with regards to the vorticity, different formulations have to be considered (e.g., vorticity-streamfunction vs vorticity-velocity formulations), alongside with potential complications on the definition of boundary conditions.

- Why the authors didn’t include measuring the MSE in Tables 1 and 2? There seems to have space to do so, and it would benefit the quantitative analysis of the paper.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730262151501

thread_type:
official_review

<thread_object>
creator: reviewer_p9ec

<content>
summary:
This paper proposes to embed PDE-based constraints into FNOs for simulating fluids. Specifically, the paper targets turbulent transport, and long-term unrolling, i.e., many repeated invocations of the trained prediction network. FNOs have inherent known limitations in terms of the frequency spectrum that they can process. Here, the FNO blocks are augmented with “PDE branches” that predict the evolution the flow with a regular numerical method. 

To prevent loosing smaller frequencies, the authors propose to predict two versions in parallel (low and high resolution), that are merged to produce an output that should better reflect the frequency distribution of the targets.

The method is evaluated on an isotropic turbulence case, and a taylor-green vortex. The authors also share their implementation at submission time, which is neat to see. Unfortunately, only very short rollouts of 10 to 20 steps are shown. For “real” applications in turbulence this seems very short.

Two recommendations regarding the presentation: 

Fig. 4, does not really show the important part, the high frequencies. As often done in fluids papers, I can recommend rescaling by wavenumber (optionally with an exponent >1), or to provide some metrics on how much the frequency content improves.

Why give separate values for each channel in Table 1? You could simply mention that SRGAN has difficulties with the z-component, but it’s usually taken for granted that a “proper” architecture can handle multiple output channels.

strengths:
The paper has several interesting aspects:
* It targets important and challenging scenarios
* Improvements over regular FNOs are demonstrated
* The submission comes with three-dimensional test cases
* A nice range of “pure prediction” baselines are included

weaknesses:
One the negative side
* Effectively only 2 test scenarios are evaluated
* The metrics for evaluating the quality of the turbulent outputs are “unusual”, why not employ common metrics, starting with MSE, over energy spectra, TKE etc.?
* Section 3.2 is fairly unclear, Q-tilde and Q-hat are not used in any equations
* The length of the rollouts is extremely short. Long-term stability is not evaluated.

Maybe the largest problem I see with this submission is the omission of previous solver-in-the-loop approaches. These have a very similar goal to improve a coarse simulated baseline (here the evolution of eq. 1) with a learned model. The paper neither cites or discusses any of the related works, Um’2020, Kochkov’2021 etc. As far as I understand, the method in 3.1 corresponds exactly to a learned correction setup with an FNO as architecture. I still see merit in the subtler differences, and 3.2 seems to be non-standard (albeit a bit unclear), but these parallels should be made clear from the beginning. Similarly, the discussion of new aspects should put the work in context to those previous papers.

As many point were left open after the rebuttal and discussion, I don't recommend to accept the paper to ICLR in its current form. I think there are quite a few important open questions to address before publication.

questions:
Please comment on similarities with solver-in-the-loop approaches, and clarify 3.2, as outlined above.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730374524002

thread_type:
official_review

<thread_object>
creator: reviewer_ikbq

<content>
summary:
Direct numerical simulations of turbulent flows can be prohibitively expensive to carry out. Fully data-driven or hybrid physics-based machine learning models can be quite promising in reducing the turnaround times for reconstructing fine scale data from coarse grained simulations or long time prediction of flow-field given historical DNS data. Motivated by the potential benefits of fourier neural operators in handling complex spatio-temporal data, the authors propose a physics enhanced neural operator method to model complex flow-field dynamics.
While FNOs work in a purely data driven fashion, the PENO in addition to data, also leverages the physics knowledge in the form of the underlying governing PDEs of turbulent flows. The authors also introduce a self-augmentation technique to enable long-time simulations/roll-outs. They demonstrate the model's capability on  different turbulent flow datasets and test across different resolutions as well.

strengths:
The problem definition is clear with regard to the PENO being trained under a forecasting objective satisfying the physics constraints.
Instead of having a single network satisfy both data-driven and physics-based constraints, PENO has two branches, one FNO and the other physics based PDE branch. The final prediction / forecast is a weighted combination of the outputs of both branches.  Instead of using continuous derivatives, the authors use temporal and spatial discretisation to estimate the gradient terms in the governing equations. This is beneficial as it reduces the load on the neural network to strongly learn the continuous derivatives in the presence of only sparse data.
The authors clearly describe their methodology, datasets used, training and testing protocol.
They provide results validating their method across different benchmarks and other data-driven models. 
The objective function is a simple forecasting MSE based loss function.This makes the learning easy and could prevent the competing objectives problem otherwise encountered in PINNs. Moreover, PENO allows multiple data sources to be combined as well such as DNS and LES through a weighted combination.

weaknesses:
Although the authors claim novelty in the physics enhanced operator, the authors have seemed to ignore previous works on physics informed/enhanced operator learning in this domain. Physics informed neural operator: https://arxiv.org/pdf/2111.03794v3, Physics informed DeepONet https://ar5iv.labs.arxiv.org/html/2207.05748 and its derivatives. Instead of comparing their model with other operator learning frameworks, they infact compare with some of the super-resolution models which in some way seems out of context. It would have been better if the authors could have provided comparison with the other previously proposed Physics based operator learning frameworks which came out much earlier than this work. While the contours look decent in the results, comparison of spatial and temporal spectrum would be worthwhile in showing if the model is capable of overcoming spectral bias that otherwise plagues neural networks in general. It is not clear if PENO can operate in a purely physics based training regime without the data-driven component as in PINO.

questions:
1. Could you elaborate on the key differences between PENO and Physics informed neural Operator?
2. Is there any reason to not compare your model with other operator learning based frameworks and not include them in the survey or in this study?
3. Could you explain why temporal or spatial spectrum of the flow-fields have not been included in the evaluation? Any justification for why dissipation different has been considered?
4. Can PENO be used in a fully data-free regime as in the PINO paper? Where given only the initial and boundary conditions, can the model be trained to simulate turbulent flow?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730573352520

thread_type:
official_review

<thread_object>
creator: reviewer_yhqa

<content>
summary:
The authors provide an interesting approach to Physics-informed ML. However I cannot recommend this paper for acceptance unless (i) the paper can clarify some missing information  and (ii) additional information is provded on the model comparison with other approaches (see questions). This is nonetheless promising work -- hopefully the authors can address these shortcomings in paper presentation in the rebuttal.

strengths:
1. Good motivation for neural operator work -- surrogates for DNS are a major research focus.
2. Good test cases GIT, FIT and Taylor-Green are well accepted configs. in the physics community.
3. Potentially promising results.

weaknesses:
1. Quantitative evaluation in table 2 is missing model parameters/FLOPs to discount scaling effects on architecture comparison.
2. More information on training data could be provided in appendix. See question 4.

questions:
1. Line 76 -- Are Neural Operators really that efficient compared to other deep learning approaches? Chung et al (NeurIPS 2023) showed that Fourier neural operators in turbulent flow applications can have large matrix dimensionalities that result in poor scaling behavior when compared to other NN architectures. 
2. Figure 2 -- How does it make sense to have both DNS and LES data at time t as inputs? After an initial condition, LES and DNS behavior would drift from each other since the missing physics in LES would change the trajectory of the simulation. There is an ablation study to show minor improves from LES inputs. But some clarification on fundamental mechanism would be useful for readers.
3. Table 2 -- what's the number of parameters/FLOPs of the different architectures. Model scales can affect predictive performance -- see Chung et al (NeurIPS 2023).
4. What numerical solver for data? What is the spatial and temporal differencing scheme? Are flow compressible or incompressible? What is the resolution of DNS w.r.t Kolmogorov lengthscales?
5. What are the exact numbers of training and test samples used for each case? A table in Appendix could be useful for clarity.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730602384040

thread_type:
official_review

<thread_object>
creator: reviewer_cppw

<content>
summary:
The paper presents a Physics-Enhanced Neural Operator (PENO) model designed to improve the simulation of turbulent flows. PENO integrates physical knowledge of partial differential equations (PDEs) with a Fourier Neural Operator (FNO) framework, aiming to capture complex flow dynamics accurately. The model introduces a self-augmentation mechanism to maintain high-frequency information, which is often lost in traditional FNO models. Evaluations on multiple turbulent flow datasets show that PENO effectively reconstructs high-resolution DNS data, generalizes across resolutions, and mitigates error accumulation in long-term simulations. This demonstrates PENO's applicability to various scientific and engineering fields where extensive, high-resolution simulations are essential but computationally expensive.

strengths:
+ Integrates physics-based knowledge directly into the neural operator framework, enhancing the ability to capture complex flow patterns that purely data-driven models often miss.
+ Demonstrates high generalizability across various datasets and spatial resolutions, showing adaptability to different turbulent flow conditions.
+ Introduces a self-augmentation mechanism that preserves high-frequency flow patterns over time, essential for maintaining accuracy in simulations that run for extended periods.
+ Outperforms existing models in structural similarity index measure (SSIM) and dissipation difference metrics, indicating superior performance in both spatial accuracy and gradient capturing ability.

weaknesses:
- Relies on large eddy simulation (LES) data for optimal results, which, while cheaper than DNS data, still adds to the data requirements and may not always be available.
- Risk of overfitting in long-term simulations, particularly when adjusting random Gaussian perturbation parameters, as extensive tuning is needed to balance accuracy and robustness.
- Challenges with maintaining accuracy in extremely high-dimensional or fine-grained simulations, where capturing all necessary dynamics requires considerable computational power and data.
- Not sure if the pressure predictions also are of high quality.

questions:
1. What about the error in the predicted pressures? It has been known that it is more dificult to predict the pressures using neural operators.
2. How easy would it be to extend the approach to non rectilinear domains. One of the main limitations of FNO is that they are suitable only for rectilinear domains.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730685218975

thread_type:
official_review

<thread_object>
creator: reviewer_c3wa

<content>
summary:
This paper is focused on improving the Fourier Neural Operator, incorporating physical knowledge and making some architectural improvements in order to better simulate turbulent flow.  By adding physics knowledge, adding an extra network branch, and adding high-frequency signals at each time step, the method is designed to outperform existing techniques for this application.  The authors' PENO method is evaluated on well-known datasets / test problems.

strengths:
1. The paper is written well, and the figure quality is high.

2. The paper makes several novel (from what I can tell) modifications to the existing FNO algorithm, and it appears that these all turn out to be improvements.

3. Results seem quite positive: it looks like the PENO method outperforms the target baselines exactly as hoped by the authors.

weaknesses:
1. It is not clear how much each of the various improvements contributes to the overall success of the method.  For example, if you just use FNO with the additional self-augmentation mechanism or just with the additional physics data, how well do things perform compared to what is ultimately the PENO method?

2. Fig. 6 is somewhat helpful, but it might be more informative to instead show the DNS result, and then just plot the errors of |method - DNS| for each method?  In particular, since you're using SSIM, readers may naturally want to look at the spatial distributions of errors of the various methods, not just the magnitudes, so that could be helpful.

3. I think there are some missing details around training: what optimization algorithm was used, what learning rates, how long did training take (and ideally how does this compare to other methods like FNO), what do the training and validation loss curves look like (so we can see how much overfitting may be happening), etc.

Minor:

"In the field of computational fluid dynamics (CFD)," just say "In CFD," here

questions:
1. Table 1: Why were SSIM and dissipation difference the chosen metrics here?  This could be informative for readers to understand.

1. Could the authors speak at all to the generalizability of the PENO technique to problems besides turbulent flow?  How do the authors imagine other researchers building upon and extending the present work?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732508472149

thread_type:
official_comment

<thread_object>
creator: area_chair_khze

<content>
title:
Reviewers' Response

comment:
Dear Reviewers,

As the author-reviewer discussion period is approaching its end, I would strongly encourage you to read the authors' responses and acknowledge them, while also checking if your questions/concerns have been appropriately addressed.

This is a crucial step, as it ensures that both reviewers and authors are on the same page, and it also helps us to put your recommendation in perspective.

Thank you again for your time and expertise.

Best,

AC
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper introduces the Physics-Enhanced Neural Operator (PENO), a novel approach that builds upon Fourier Neural Operators (FNOs) to emulate turbulent flows. PENO aims to overcome the limitations of traditional FNOs, including limited generalization, reduced accuracy from neglecting underlying physics, and the inability to capture high-frequency details. PENO attempts to improve upon standard FNOs by incorporating physical knowledge of the underlying PDEs. This is done through a self-augmentation mechanism that the authors claim it retains high-frequency information by integrating "PDE-enhancement branches" to predict the flow evolution using numerical methods plus a  super-resolution which is then weighted with a regular FNO branch. The method is evaluated on two datasets: Forced Isotropic Turbulence (FIT) and Taylor-Green Vortex (TGV) flow.

Reviewers found the approach promising but raised several concerns. These included the limited length of the demonstrated rollouts, the need for clarification on certain aspects of the method, and comparisons with other approaches, as some important related literature seemed to be absent. Concerns were also raised regarding the rationale for using both LES and DNS inputs, which can diverge due to the chaotic nature of the problem. Additionally, the metrics used are not standard in CFD, making it difficult to gauge the method's behavior and advantages. Several ablation studies are missing, making the relative importance of each innovation to the overall performance unclear. The authors provided only a brief response during the rebuttal period, failing to address the reviewers' concerns or provide an updated manuscript. As such, I recommend rejection.