Venue: NeurIPS 2022 Track Datasets and Benchmarks

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: summary and contributions: Authors and the team present AirfRANS, high-fidelity CFD datasets for the RANS problems. Relevant DL baselines and benchmarks are also provided to demonstrate the effectiveness of the datasets. Particular problems of interest are generalization tasks (big and scarce data regime, Reynolds number, and angle of attack extrapolation). 
strengths: The numerical solution of Navier-Stokes equations for fluid dynamics is one of the most important PDEs in science and engineering. Current numerical solvers can be computationally expensive and might take thousands of CPU hours.  Having an accurate surrogate model becomes critical. The major contribution of this work is to provide a dataset that would allow a deep learning-based surrogate model to become possible. The dataset is created by RANS equations and shared in openly accessible storage space. 
weaknesses: 1.	From my understanding, a data paper should tell end users, what “knowledge” would you gain out of the data and what you can use the data for his/her own benefits. The last thing that you would like to see is to simply assemble a bunch of data with some benchmark. From that perspective, the authors and his team are doing a good job providing useful knowledge on top of the data and the benchmarks are interesting as well. However, I think they can do better. One example would be that you can design datasets in various challenging levels: easy, medium, and hard. Test baseline models on different challenging levels can not only give us a sense of how the baseline models would perform, but also give future users an idea of how to choose among different datasets. 


2.	One of the major purposes of the dataset is for the surrogate model. From that perspective, it would be very interesting to study the performance of the DL models by using different sizes of the samples as training sets, and how the performance of the surrogate model will be impacted. 

3.     How computationally expensive to generate those data? Will you and your team further improve the datasets based on those limitations that you provided? 


4.     All the data provided in the current repo are simulations, are you planning to use the DL model for real applications? How realistic  your simulations are by comparing with real data? Knowledge gap there? 


5.	The GitHub link is not working, please fix it.  

correctness: In general, the submission seems correct to me, but the authors and the team need to address the weakness before publishing this work. 
clarity: There are some typos and errors that could be improved. The appendix of the manuscript is not complete (missing sections J to P)!  
relation to prior work: Yes. The differences are clearly stated in the manuscript. 
additional feedback: N/A.
review_title: An interesting manuscript, but more edits would be needed.

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: The dataset contains the description and data from a classical aerodynamic problem, the airfoil simulation. 

The authors use consolidated procedure to simulate the airfoil using the two of the NACA airfoil family descriptions. 

The authors also describe the underlying CFD equation used in simulation, in particular the RANS model and provide details on the variables and parameters.

The authors provide dataset pre-processed to be use for machine learning task and also the original data from the numerical simulation tool. 

The authors use the openFOAM simulation tool, a opensource simulator. 

EDIT: I increased my score 6 -> 7 
strengths: 1. the paper presents dataset for a very concrete use case and relevant use case
2. the authors uses openFOAM, which is public available and widely used simulation software
3 the data cover various scenarios 
4. the authors provide solid baselines 
5. accurate description of the underly equation, the generation of the mesh and preparation
6. analysis of the performance of the models 
7. the paper is well written

weaknesses: While I really like the application and the provided dataset and baseline, I see currently the following limitations:

1. the code of the baselines is not currently available
2. it is not clear it the authors will provide the set up of the openFOAM simulation in order to extend the dataset; this is a critical aspect
3. the dataset itself is limited in number of samples. 1000 sample for 3 and 4 dimensional parameters seems rather limited
4. the maintenance of the data is not clear
5. the authors describe the Langley NASA data, but they seems not to be used neither to validate the basic simulation, neither the trained models
6. the authors provides simulation for the RANS model, but do not consider and provide data with not averaged pde, since the authors do not provide the simulation data with these alternative models a better motivated explanation is needed or some sample simulation would be interesting to have;
7. the simulation have been done by testing outside the training interval, would be nice to have also testing inside the range of the training set (but not overlapping)
8. it would be appreciated the discussion on the "derived" variables
9. it is not clear the values used in the sampling. the values seem arbitrary
10. the authors describe that the jacobian of the velocity is used; I am not sure I understand: a) how are they computed and b) how are they used
11. some references in the introduction are missing (first three sentences)
12. The authors should improve the analysis of the low performance on the drag component, currently it is merely stated that the models do not performe well
13. it would be appreciated an analysis of why MLP works so well; is it because the mesh voxels are located mostly in the same locations for all simulations?
14. since the authors used a open source simulation, more effort could be implemented in generate richer data
15. how have the authors validated that the generate simulation are physically realistic?

I am happy to change my evaluation in the previous points
correctness: The experiments set up and their execution is sound, but the sample size is probably too small (1000 samples in total). I am aware of the space required is a critical aspect. 
clarity: Yes, the paper is in general well written (a part from missing references in the introduction) and the problem well presented along with the procedure and mesh generation.

I would thought like more detail on the physical equations and alternative to the average model approach. 
relation to prior work: yes
additional feedback: The application is very intersting and I hope the paper meets the standards to be published.
review_title: Very intersting dataset, but with some shortcomings

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: This work presents a dataset to study the 2D RANS equations over airfoils with different airfoils, Reynold numbers, and angles of attacks regarding four different tasks.

The authors already address all my concerns. The supplementary material is very detailed. More importantly, the Github is working now and another Github for simulation is provided. This is good for the reproductivity of this work. I would raise my score to 7.
strengths: 1) The authors have designed four different challenges with this dataset, including full data regime, scarce data regime, Reynolds extrapolation, and angle of attack extrapolation. The baselines with MLP, GraphSAGE, PointNet and Graph U-Net are also provided, which make it easy for other researchers to build new ML models and compare them with the results in this paper.
2) The experiment consists of both quantitative and qualitative evaluation metrics. The MSE and Spearman's rank correlation are calculated for velocity, pressure field and force coefficients to quantify the ability of ML models.
3) The paper has a detailed description of the dataset construction including the RANS equations, airfoil design, mesh generation, and dataset generation. It is easy for readers to understand the whole procedure. The computational challenges in the data generation due to the number of meshes are clearly clarified.
weaknesses: 1) The discussion about the characteristics of this dataset is needed. Most of the paper is about the generation of the data, but the information about the feature of the dataset itself is missing. I think the authors need to explain why this dataset can be the typical benchmark for the design of ML models in this area. 
2) Several sections about the networks and the dataset are not finished in the appendix. Only one baseline about the full data regime is provided in the paper while the other three are still unavailable. It is hard for me to give a fair evaluation of the benchmarking without these contents.

correctness: The claims are correct. The evaluation methods are designed in a good way.
clarity: Yes. It is well written and easy to follow.
relation to prior work: The difference between this work and the authors' previous work in ICRL workshop 2022 is not discussed clearly.
additional feedback: 1) Page 5 line 174 "the four field $\bar{u}$, $\bar{p}$ and ${v}_{t}$" Are the velocity fields $\bar{u}$ with different spatial component treated as two fields here.
2) What is the difference between the dataset with the dataset you present in the ICLR workshop？ Why call this dataset a high-fidelity version?
3) In Figures 1a and 2, the numbers in the images are too small to see. 
4) The dataset is generated with random airfoil, Reynolds number, and angle of attack. The range of the Reynold number and angle of attacks are quite large, 2 to 6 million for Reynold number and -2.5 to 15 for the angle of attack. But only 800 samples are in the training set. Can this small amount of dataset cover the whole data domain?
5) In Figure 3, the lift coefficients are evenly distributed in the whole domain but most of the drag coefficients are distributed at low values. Is this distribution good for the study of this problem? Could this be a reason that the models can not perform well in the drag coeffiecnt?
6) The GitHub link in the paper is broken.
review_title: A dataset for studying the 2D RANS equations over airfoils but several sections in the paper are unfinished.

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: summary and contributions: This paper proposes a dataset of fluid dynamics simulations of two-dimensional airfoils to serve as a benchmark of ML surrogate models for fluid dynamics. The dataset is constructed to cover a range of airfoil shapes and fluid flow conditions, and multiple physically relevant dataset splits are proposed. The benchmark results of several popular ML models are included. 

The main contribution of this paper is the fidelity of the dataset itself, as well as design process for constructing the dataset. 
strengths: 1. The dataset proposed seems sensible and well designed. From what I can see, care is put into construction and running of the simulations, as well as selection of the design space. The proposed dataset splits are also sensible. Overall, I think this is a very solid contribution and the dataset will be a useful benchmark for the field.
weaknesses: 1. In the paper, four different dataset splits are described yet only the benchmark results for two of them (the interpolation regimes with "full data" and "scarce data") are provided. The extrapolation performance of ML models are key measurements and the results should be provided. Failures in extrapolation can also reveal unintentional flaws in the design space of the dataset so I would like to see those benchmarks to assess the construction of the dataset.

correctness: The main claim of the paper is correct. The dataset is well-designed and the benchmarking done rigorously, however only results for two of the proposed four dataset splits are provided.


clarity: The paper is overall well-written.
relation to prior work: The authors clearly discuss how the dataset in this work is at a higher fidelity and covers a wider design space than previous contributions.
additional feedback: 1. For the 4-digit NACA airfoils, it is not clear to me what geometrical issues resulting from P < 1.5 are such that setting P to 0 would solve them. Is it an issue with mesh generation?

2. Could you explain what do you mean by "relative error" in the benchmarking results, and why these relative errors are lower in the scarce data regime than in the full data regime? eg 2.95 vs 3.92 for the MLP model.
review_title: Computational fluid dynamics airfoil dataset & benchmark for ML surrogate models

final_score: rating: 6: Marginally above acceptance threshold
review_content: summary and contributions: The authors provide high resolution dataset of the 2D Reynolds-Averaged-Navier-Stokes solution over airfoils, and compare 4 different deep learning models as baseline surrogates. The authors also test the generalization ability by extrapolating the simulation parameters for the test dataset. While this is an important and appealing topic in the computational fluid dynamics research field, the paper lacks clarity mainly on the motivation and objectives, which I think is caused by poor writing.

*EDIT: I improved my score from 4 to 6 after the rebuttal.
strengths: Procedures for the simulations are well-documented, sufficient references and literature reviews of state-of-the-art, and detailed descriptions of the simulation setups and theoretical background are provided as supplementary materials.
weaknesses: Models chosen as baselines only include purely ML models. There are many new hybrid ML models that were recently proposed to tackle turbulent flow problems (see e.g. TF-Net, Wang et. al., 2020), which I think could be more useful and give more insights.
The writing in general could be improved significantly. This also causes lack of clarity in most of the content.
Link to Github repository doesn't work.
The authors also provide baseline for full and scarce data regime, also for extrapolation, but the comparison is not made very clear.
correctness: All the statements claimed in the paper seem to be consistent and sound.
My question is: is all the simulation setup used to generate the dataset representative enough to cover various scenarios of fluid flow around airfoils? More concretely, are the boundary conditions and parameter values used are wide enough?
Specifically about boundary condition: would a new model needs to be trained if a boundary condition is changed?
clarity: Quality of writing is poor. There are many sentences that are not understandable and confusing, because of grammatical errors.
relation to prior work: The related work section provides a sufficient literature review of state-of-the-art methods and models.
However, I'm wondering why the authors use only pure ML methods as baselines and not include physics-informed ML such as PINN, TF-Net (specifically developed for modelling turbulent flow) or operator based model such as FNO, DeepONet.
These models might be more useful as baselines compared to MLP or U-Net.
additional feedback: I would be happy to change my review if the quality of writing is significantly improved.
review_title: Good content but lack of clarity

Meta-review: metareview: This paper provides high-fidelity datasets for computational fluid dynamics, based on Reynolds-Averaged-Navier–Stokes simulations over airfoils. While it is possible for anyone to generate such a dataset, doing so is expensive and hence, having a reference dataset can be helpful for lowering the bar for entry for researchers. The paper also provides graph neural net baselines on 4 different dataset regimes, which can help in diverse benchmarking for future works. The dataset and code is open-sourced for further use. There were some important concerns raised by the reviewers regarding the missing details on dataset construction and use, which were appropriately addressed by the authors.
confidence: 3: The area chair is somewhat confident
Decision: Accept