Venue: NeurIPS 2022 Track Datasets and Benchmarks

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: The paper presents a benchmark for using machine learning to solve PDEs. The PDEs considered vary from relatively simple to complex. In addition to the data, the authors provide a framework for data generation that can be extended with different ranges of parameters and different equations.

strengths: * The benchmark considers a large number number of PDEs for flow problems, time dependent and independent, in 1,2,3D. Some of these are typically very complicated and time consuming to implement for someone with or without the background and time, so the breadth of examples presented here is very useful. Furthermore, the interface seems very nice as it directly works with PyTorch. This standardization also means that extensibility should be very good, which is a big benefit for both people who are interested in phrasing problem and those interested in just solving them.
* The benchmark considers performance at various regimes (parameter values which may give rise to new phenomena), which is very important for generalization and getting these ML methods to the level of being black box solvers.
* The benchmark sets forth new comparison metrics which give a better view at physical consistency than traditional ML error metrics. An interesting discussion of how some methods perform on these is also given.

weaknesses: * The benchmark does not include a comparison of the machine learning approaches with traditional marching schemes, either implicit or explicit. For a task where there exist very strong methods that already solve this problem with convergence guarantees, it is important to compare with traditional baselines to assess true progress in this task. Although generation was done with these methods (hopefully to high enough accuracy to produce valid ground truth), comparisons can be done with larger time steps.
* The benchmark does not compare the time to get a simulation result, which is even more complicated in this setting given the training + eval time required for machine learning.
* The benchmark does not attempt to do prediction in regions outside of training distribution. This can be important for generalization depending on the setting, and in any case a good point of discussion. 
* Arguably, non-rectangular domains are pretty important for flow problems, although this was a stated limitation.

correctness: Data generation seems correct, at least given the details in the supplementary info about the finite difference schemes and deference to external solvers. The authors do a good job of deliberating on physical phenomena and difficulties of numerical solutions in various regimes of parameter values. It would be good for the authors to clarify how they ensure that the data generated has sufficient accuracy to be a valid ground truth.

clarity: The paper is clearly written.
relation to prior work: There is a good discussion of prior work, although this work does not stand as a strict superset of previous efforts to create a benchmark for learning physical simulation. Otness et al 2021, while more limited in the number of PDEs examined, attempts a comparison with traditional simulators through coarsening time steps and trying to match runtime (which is one of the weaknesses here).

additional feedback: NA
review_title: Benchmarking learning performance on PDEs

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: The authors introduce a benchmark suite for surrogate modeling, with a focus on deep network models. The benchmark includes problems in one to three dimensions, problems based on Navier-Stokes equations and shallow water equations and several common 1D benchmarks from scientific computing. Simulators for generating new data, pre-trained network models, and pre-computed training data are provided.
strengths: - Very timely work: Scientific machine learning is a quickly growing area with little standardized benchmarks. Thus, works like this one are urgently needed to assess new methods in this area.
- The benchmarks cover a wide range of physics (compressible and incompressible flows, waves, 1D-3D problems) that are relevant for a wide range of science and engineering problems.
- The authors provide benchmarks that are otherwise time-consuming to set up because they require knowledge about the underlying physics (boundary conditions etc), the numerical methods to use, and familiarity with software that implements the numerical methods. Thus, the provided benchmarks can save lots of time.
- The authors emphasize error metrics that are more specific to science and engineering problems than standard metrics considered in machine learning (for example, cMSE and bMSE).
- The data sets and repository are well organized and documented.
weaknesses: - The authors consider a very specific problem of scientific machine learning (see Section 3.1), namely surrogate modeling. Scientific machine learning also investigates techniques for numerically solving PDEs without having training data available from ground-truth solvers, among many other things. I therefore don't think that the title "... for Scientific Machine Learning" is appropriate and instead should be changed to something that includes surrogate modeling rather than scientific machine learning. In that respect, I also don't agree with the statement "The objective of Scientific ML is to find some ML-based surrogate" on page 3, because scientific machine learning is much broader than surrogate modeling.

- Because the focus is on surrogate modeling, there should be at lease one traditional, widely used baseline such as non-intrusive POD models (for example: Guo and Hesthaven, Data-driven reduced order modeling for time-dependent problems, https://doi.org/10.1016/j.cma.2018.10.029). Otherwise the benchmarks don't help to assess if the machine learning methods can provide improved results compared to traditional surrogate modeling techniques.

- The authors compare methods based on accuracy. However, one major purpose of surrogate modeling is to achieve a low runtime of evaluating a model but no runtime results (training, evaluation runtimes) are provided. Having runtime results would help to assess the speedup that can be obtained with various surrogate models.
correctness: The supplemental material is extensive and provides detailed problem descriptions. The GitHub repository is well structured and all training data are available. The evaluation metrics are appropriate and go beyond classical evaluation techniques and are motivated by science problems. The selected problems cover a wide range of physics (parabolic and hyperbolic problems, Navier-Stokes)
clarity: The paper is well written. Some minor changes could further improve the clarity:
- The function v defined at the beginning of Section 3.1 has no range.
- In Section 3.1, the notation of v(., ., \theta) changes to v_{\theta}() without explanation
- The author should consider introducing a time step size delta t at the beginning of Section 3.1 instead of using t + 1.
relation to prior work: The present work looks like a valuable extension of the benchmark suite introduced in [36] by Otness et al. The present work is very similar in style and covers similar physics (fluid dynamics, wave) and similar baselines (U-Net) but provides more benchmark examples and more baseline surrogates than [36], except that no runtimes results are provided as in [36].
additional feedback: See comments under Weaknesses and Clarity.
review_title: An extensive benchmark collection for surrogate modeling of physics problems

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: 
The authors introduce a dataset to benchmark surrogates that approximate the solution of partial differential equations (PDE). The surrogates can be used for solving related problems, such as the inverse problem, i.e. to find the initial conditions (t=0) of dynamic systems, given information from t>0 about the system. The authors present an ablation of various models using different neural networks, such as U-Net, Frourer Neural Operator, etc. They also study different metrics to evaluate the precision of the prediction of the surrogates. The benchmark is easy to access through an API.

strengths: - Novel dataset for benchmarking Machine Learning model applied to physics.
- Easy to access API. 
- Interesting application of the surrogates for solving the inverse problem
- Important ablation for different types of surrogates and metrics to approximate PDE.
weaknesses: - Diversity: My main concern is about the diversity of the benchmark. There are few datasets (11) and the application domain is mainly hydrodynamics. 
- Weak benchmark motivation: In order to motivate further the use of surrogates for scientific ML, and therefore the benchmark, I think it is essential to provide an additional problem setting beside the inverse problem.
- Writing: The authors should consider that they submit to a conference related to Machine Learning, rather than physics. Therefore I think that a nicer introduction to PDE and its importance in modeling physical systems would be great.
- Weak discussion of prior work.


correctness: The benchmark construction is clearly explained by presenting the PDE that originated the data. The authors present explicit formulation and motivation for the equations and the associated system.
clarity: The paper is overall clear, but I had to jump back and forth to understand some concepts: e.g. inverse problem (which is defined a couple of pages after its mention). 
relation to prior work: The related work is short and could be enriched by listing, for instance, how much data/PDEs/surrogates or which metrics previous work used? LIke that it is easier to understand the position of this work. However, I understand that given the novelty of the settings, related work can be probably scarce.
additional feedback: 
I understand that surrogates allow the users to speed up the evaluation of the dynamic systems. Therefore, it is important to have accurate surrogates that model well the underlying system. However, for training the surrogate, you need to spend some time as well, is this not counter-productive? I would like that the authors elaborate more on:
- The advantage of using surrogates. How much is the time gained, for instance: time for simulation vs time for the surrogate training vs time for other solvers, compute resources for simulation vs compute resources for training?
- The sensitivity of the tasks using the surrogates (e.g. solving the inverse problem/setting) to the accuracy of the surrogate. How much additional accuracy do I need to get better solutions for the “downstream” task? Could we have some information about this?
review_title: Dataset for benchmarking surrogates that approximate PDE

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: This paper proposes a benchmark for learning the solutions to PDEs.
Section 3 describes the basic setup of solution mappings of PDEs
and proposes ~10 classes of PDEs to solve that are representative
of many of the most challenging problems in the community.
strengths: + This paper is well-positioned to be a standard in this new field of
  learning the solutions to PDEs.
  It is important to have established families
  of PDEs to have a standardized way of comparing methods.
  Otherwise, many new papers jointly propose new classes of problems
  along with new methods and it is difficult to understand
  the main contribution.
+ The baseline methods considered in this paper
  (U-Net, FNE, PINNs, and a gradient-based inverse method)
  are reasonable and seemingly perform well on all of the PDEs
+ Ablations such as in Figure 3 provide nice insights into
  how methods change as the underlying family of PDEs changes.
weaknesses: While I am optimistic about the general direction of the paper,
I feel there are a few shortcomings that need to be addressed
to make the paper influential. I am very open to reconsidering
my evaluation of the paper with a discussion on these:

+ It is not clear how impactful better-solving the tasks in
  this paper will be and the paper does not convey how poor or
  well the baselines perform on the problems beyond the error metrics.
  I would find it insightful to visualize the solution quality.
+ While the paper focuses on error metrics, I'm also very interested
  in understanding how the computational time compares between methods.
+ I would also find it useful to add as baselines standard ("classical")
  PDE solvers for all of the tasks and report their error metrics
  and runtimes.
+ It's not clear how varied the PDEs are within each family and
  I would again find it useful for intuition to visualize samples
  from the distributions.
+ The paper focuses on the setting where the PDE solution operator
  is regressed onto a ground-truth solution obtained by a high-accuracy
  solver. However there are other interesting classes of methods such
  as the
  [Physics-Informed Neural Operator](https://arxiv.org/pdf/2111.03794.pdf)
  which also work by optimizing the PDE residuals directly that
  would be worthwhile incorporating into a benchmark as well.
+ It's important to connect some of the results back to
  established results published in this space. For example,
  how do the results compare to Figure 3 of the related PDEs in the
  [Fourier Neural Operator](https://arxiv.org/pdf/2010.08895.pdf) paper?
  It would be extremely strong to reproduce their results on
  exactly their settings and then expand beyond that.
correctness: Yes
clarity: Yes
relation to prior work: Yes, see also last comment in weaknesses on connecting the experimental results to related work
additional feedback: n/a
review_title: Well-needed benchmark with some room to improve

Meta-review: metareview: The paper proposes a new benchmark consisting of a variety of PDE problems. Almost all the reviewers appreciated the extensiveness of the problems included: time-(in)dependent, 1-3D etc. There is code for generating data, training baseline models, recording variety of metrics -- all the qualities that make for a complete and useful benchmark. The reviewers wanted to see some evaluations comparing ML models with classic approaches, which the authors have duly incorporated during the discussion period. Considering all these aspects, coupled with the significant opportunity for Scientific ML to solve challenging real-world problems in the near and long term, I am optimistic this benchmark will aid ML and PDE communities immensely and lower the bar for entry for researchers and practitioners alike.
confidence: 4: The area chair is confident but not absolutely certain
Decision: Accept