Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper proposes to use MPNNs to reconstruct the (time-averaged) pressure and velocity fields around an airfoil based on measurements of the pressure distribution at the airfoil's surface. Some global parameters of the flow (farfield velocity, angle of attack, turbulence intensity) are also predicted. The authors use Grouped Reversible GNNs with three different layer types and with Feature Propagation. The models are trained in a supervised manner using a set of over 1k FVM simulations, and in evaluations, the GIN and GAT layers perform better than the GEN layer.

strength_and_weaknesses: Strengths of the paper include the use of varying experimental conditions (airfoil shapes, angles of attack, Re) and high Reynolds numbers. The paper also contributes the dataset of simulations used to train the model, and the code used for the experiments is available.

A limitation of the experiments is that only 2d, steady state cases were evaluated. Real settings of interest where pressure sensors are mounted on a physical airfoil would generally be 3d and time-dependent. The experimental exploration done is also limited to testing the 3 layers -- there is no report of testing of the different hyperparameters of the model (L, N, C). The physics-based divergence-free loss component is mentioned, but then not discussed further, so it's unclear what role it plays.

clarity,_quality,_novelty_and_reproducibility: The paper is organized well and easy to understand. The authors combine existing ML techniques to solve the proposed problem, but the paper itself does not present any general ML innovations. The used setup is more general than prior work in that the model is trained in conditions of high Re, and with different airfoil shapes at varying angles of attack. The released code and data should make it possible to reproduce the results.

Questions/suggestions:
- Isn't the absolute value of measured pressure potentially important to identify the physical situation one is in? You state that you normalize pressure features to avoid "data leakage". Could you please explain a bit more what you meant here?
- In addition to the absolute error values in Tab. 1 and 2, it would be interesting to see what are the relative errors that the model makes, and whether they vary with parameters such as the Reynolds number (it's conceivable that more turbulent flows are harder to correctly predict).
- In all figures plotting the pressure/velocity fields generated by the models and the ground truth values, it would be helpful to also plot the difference between the ground truth and the model predictions.

summary_of_the_review: This is an interesting application paper. By tackling a wider range of conditions in a single model, this paper moves towards more realistic scenarios than prior work. However, with all simulations restricted to 2d steady-state conditions, this still feels quite far from the practical applications with MEMS sensors mentioned in the abstract as motivation. With no ML innovations, and lack of clarity whether the relative performance of the different GNN layers in this context has any wider applications, I think that this paper could be better suited to a more specialized venue.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The manuscript is describing how to estimate the flow of air around an airfoil only from the pressure and velocity distribution on the airfoil for high Reynolds number flows. Graph based structure is used to define a "mesh" has some intelligence, i.e. features at each vertex. The paper combines far-field information with contextual qualities of the incoming field. A digital ensemble of training data is created with OpenFoam and RANS turbulence models.

The far-field conditions are initialised with Bernoulli flow and the pressure with a random sample with a mean and standard deviation of the values on the airflow. The pressure values are propagated with about 20 feature propagation steps. Different architectures are studied.

The trained Graph model do give quite good velocity profiles, although the turbulence and flow detachment point seem to diverge. From global features, especially the far field velocity, is estimated very well.

. 
strength_and_weaknesses: Strengths: Original problem, and a solution that provides quite good inference. 

Weaknesses:  The update scheme in the graph could be seen in terms of the Lattice Boltzmann models, except the flow update is defined in an a way that does not automatically lead to zero divergence in summing up the velocities to each vertex to zero. Perhaps relaxation has to be done for a faster convergence. What happens, if the solution is just propagated with from the airfoil surface to the outer edge with the Boltzmann lattice update rules? Also, I think more work should be done on calculating the training data with a LES solver that goes to sub graph network "mesh" sizes, so that the complexity of the features could be used to find a closure for the turbulence and give more reliable solutions for the turbulence, as this is the fundamental problem at high Reynolds numbers.
clarity,_quality,_novelty_and_reproducibility: The paper is clear, consistently written. The degree of novelty is a little vague. Good information is provided for reproducibility.
summary_of_the_review: The paper reproduces the state of the art of standard CFD codes, with the addition that the inflow velocity is kept open. The benefits of this solution should be brought more clearly forward - like how much faster this compared to a normal CFD analysis of the problem - especially as the turbulence prediction seems to be off.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper considers the problem of extrapolating and reconstructing the flow around 2-D airfoils using measurements on the surface. A deep reversible graph neural network is trained to perform this reconstruction 
strength_and_weaknesses: Strengths

- A rich dataset is generated with graph representations of fluid around the airfoil taking 60,000 nodes, different inflow conditions (flow velocity, turbulence, and angle of attack) and different airfoil shapes from the UUIC database.

- The study not only explores different shapes but does a decent sweep of inflow conditions ranging across a large interval of Reynolds number (2 * 10^5 to 6.5 * 10^6). Non-laminar flow (beyond 5 * 10^5) makes flow reconstruction difficult. 

- The idea of using global context features in addition to node features is interesting. Incorporating the length of the boundary between cells as an edge feature is also neat. But could authors discuss why the analysis was limited to 1 chord circle centered on the manifold? Could the scalability challenge be elaborated with some quantitative analysis? 

Weaknesses

- In absence of access to code, it is not feasible to determine if the meshing by Gmsh and using OpenFOAM has yielded trustworthy simulation data. There is some mention of details such as sizing field but the reviewer is unsure about fidelity of the dataset. Typically, tools such as Ansys Fluent and Comsol are used because of their ability to provide guidance in running CFD on different shapes. Even then, human expertise is critical in CFD analysis. Since the paper mentions dataset as a critical component, the absence of code and dataset for review makes it difficult to determine the quality and hence significant of this part of the paper.

- The experimental results appear to be mixed. The proposed approach fails on farfield turbulence intensity prediction and struggles with prediction of AoA.  It appears the training set was 896 large and the testing was done on 112 airfoils. It is important to make sure that the training and test set had airfoils that were sufficiently different from each other. It will be useful to describe how this split was done. Considering many different splits will also be important. Using a parametric airfoil shape generator can enable scaling the experiments further. Also, is this method restricted to 2D? Why not test the method also on 3D examples? 


clarity,_quality,_novelty_and_reproducibility: The paper is easy to read. The novelty is limited to application of ML for CFD analysis. The experimental evaluation is a bit shallow and there are not enough details available to determine reproducibility of the work. The code showing the mesh construction and openFOAM configuration would be helpful. 
summary_of_the_review: Overall, the paper takes a standard GNN approach and uses that for CFD analysis. Some of the doubts of the reviewer are attempting to ascertain the value of this work in the CFD domain, since the novelty from the point of view of ML is limited to application. The paper appears to be a work in progress.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper introduces a new dataset simulating the steady airflow on an airfoil. The dataset contains a large number of different geometries and initial conditions. The authors are then interested in estimating the velocity field and the global context of the simulation from the sole knowledge of sparse measurements of the pressure field on the surface of the airfoil. The proposed method rely on feature propagation and grouped reversible GNN.
strength_and_weaknesses: The paper focuses on a difficult but very interesting problem, with direct applications in aviation. Reconstruction of flows from sparse measurements has already been studied in Erichson et al. but in a very simple scenario. The task introduced by the authors is much more challenging, and the proposed solution seems reasonable. Following the structure of the paper, I organized my questions and remarks in two sections dealing with the dataset first, then the model. These are sorted in ascending order of importance.

About the dataset

- I would have appreciated more statistics about the dataset, including its total weight, average number of nodes and edges, etc.

- It seems that the simulations were carried out under the assumption of incompressibility. It seems to me that this assumption does not hold for flow on an airfoil (Sanchez-Gonzales et al. 2020). Can the authors discuss the relevance of this assumption?

- However, I am quite skeptical about the novelty of the dataset. The authors simulate the evolution of the velocity and pressure field over a large number of different wing profiles and under a large set of initial conditions. However,  this type of dataset already exists in other works. In particular, [1] seems to me to be the closest, but other variants exist [2], [3] (non-exhaustive list). Would it be possible to complete the related works section of the article by comparing the new dataset with the existing works?

About the model

- In appendix B, the models are limited to 4.5k parameters. It seems very low. In comparison, MeshGraphNet (Sanchez-Gonzalez et al. 2020) contains 1.5M parameters for L=15 layers of GNN (but the model is designed for dynamic forecasting). Is this a typo? Are GNN weights shared across layers?

- Section 4.2 presents an auxiliary loss enforcing the divergence of the velocity field to be close to zero. This is perfectly justified in the case of an incompressible simulation. Nevertheless, it is not always respected by the simulator. Have the authors verified that the null divergence hypothesis is verified on their dataset? Also, what is the effect of ablating this term on model performances?

- In general, I deplore the absence of more baselines to compare with the proposed model, which novelty is rather limited. I wonder in particular what is the effect of the grouped reversible GNN relatively to a more traditional solution such as chaining layers of GAT/GIN/GEN?

- The model seems to use a simulation bias to obtain a first approximation of the magnitude of the velocity farfield (the fact that the simulation uses a null pressure field at the boundary). So I have three questions: (1) To what extent could this approximation be performed on real data? (2) How far is this first approximation from the true velocity farfield (this could be checked by plotting figure 5a for this approximation)? and (3) what is the contribution of this first estimation in the performance of the models? in other words, does the performance deteriorate noticeably without this approximation?

I nevertheless point out the relevance of the experiments carried out, in particular table 2 and figure 5 which correctly analyze the important points of the model. I noted several citation errors in the references, and 11 pre-prints among the 33 citations.

[1] Thuerey, N., Weißenow, K., Prantl, L., & Hu, X. (2020). Deep learning methods for Reynolds-averaged Navier–Stokes simulations of airfoil flows. AIAA Journal, 58(1), 25-36.

[2] Ahmed, S., Kamal, K., Ratlamwala, T. A. H., Mathavan, S., Hussain, G., Alkahtani, M., & Alsultan, M. B. M. (2022). Aerodynamic Analyzes of Airfoils Using Machine Learning as an Alternative to RANS Simulation. Applied Sciences, 12(10), 5194.

[3] Schillaci, A., Quadrio, M., Pipolo, C., Restelli, M., & Boracchi, G. (2021, January). Inferring functional properties from fluid dynamics features. In 2020 25th International Conference on Pattern Recognition (ICPR) (pp. 4091-4098). IEEE.
clarity,_quality,_novelty_and_reproducibility: The paper is well written and clear. Novelty is limited for both the dataset (many existing works released what I think are similar datasets), and the model (the proposed approach seems to be the concatenation of two existing methods).
Reproducibility seems possible, yet code release would be very much appreciated.
summary_of_the_review: Despite the good quality of the work presented, my first impressions lead me to believe that the novelty is limited. Beyond that, some design choices seem questionable to me, and I hope the authors can provide us with convincing justifications. In that case, I may increase my grade.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper on estimating fluid dynamics has received four expert reviews, three of which were highly critical. The main issues raised were a lack of innovation on the ML side (even agreed on by the one very slightly favorable reviewer), lack of reproducability, missing comparisons, and some technical issues were raised wrt to the models. While the authors could address some technical points, the issues on novelty and evaluation remained.

The AC assesses, that the paper is not yet ready for publication and recommends rejection.

justification_for_why_not_higher_score: This is a clear case
justification_for_why_not_lower_score: This is a clear case
Decision: Reject