Venue: NeurIPS 2021 Track Datasets_and_Benchmarks Round1

Reviews:

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: summary and contributions: The authors introduce a physical systems evaluation dataset framework that focuses on evaluating machine learning algorithms for simulation problems. They provide four systems of increasing difficulty to evaluate baseline methods (spring, wave, Naive-Strokes and spring mesh), explore the trade-offs between derivative-based prediction and step prediction and advocate the usage of K-nearest-neighbors to better understand the complexity of different simulation tasks.
strengths: The paper is very concise, easy-to-follow and well-illustrated.

The authors do a great job motivating the four representative benchmark physical systems, and provide a comprehensive array of baseline data-driven methods. While neither of these are exhaustive, the flexibility of their framework allows for the integration of other learning tasks or machine learning methods. 

All in all, their contribution promisingly lays the groundwork for future research in the field of scientific computing
weaknesses: While the graphs are typically easy-to-follow,  I was slightly confused by repeated colors in figures 3 and 4. My understanding is that same-color marks are different architectures of the baseline methods. However, different architectures could have significantly different strengths and weaknesses. For instance, shallow MLPs are typically more robust to noisy datasets than deep MLPs. 

Do you believe that readers would benefit from having more fine-grained labels for the methods (e.g. shallow vs deep MLP/CNN) in figures 3 and 4?

Further, due to the non-deterministic nature of some NN-based approaches, it would make sense to average results over multiple runs. In the main paper there doesn't seem to be any indication of this. Were the results presented as averages over multiple runs?
correctness: The 4 benchmark systems are motivated and thoroughly detailed. The increasing level of complexity of the different datasets seems like a reasonable choice for the project.

The data-driven methods discussed represent standard approaches in the field of physical simulations.
clarity: The paper is very well written and clear. 
relation to prior work: While previous work is discussed in section 2, there is no explicit description of any physical system benchmark. Is this because because all prior work has had a major lack of standardization? One or two examples of how prior work has been more unstructured could strengthen your motivation.
additional feedback: MSE seems like a reasonable metric to evaluate the trajectories of simulations. However, my understanding is that predictions close to $t=0$ are more likely to be correct, and that errors compound by letting the system run for longer.  If this is correct, a weighted squared error (e.g. decaying squared-error) could be a more suitable metric for longer trajectories. However, I understand that this would've added an extra dimension of complexity to the overall paper.
review_title: Concise and Clear Benchmark for Physical Simulations

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: The present work is motivated by the need for a) thorough evaluation of data-driven approaches in scientific computing pipelines and b) the lack of standardized benchmarks in the literature.   The authors focus on physical simulation benchmarks that a) map a high-dimensional state space into another high-dimensional space (as in temporal integration schemes, mapping the state of the system at one time step to the next, or b) from a high-dimensional input space to a lower-dimensional output (as in surrogate models, mapping the initial conditions to a functional of the solution.  In addition the paper addresses the narrow data regime, where initial conditions are sampled from a low-dimensional manifold (even within a high-dimensional state space), and the wide regime, where initial conditions span a truly high-dimensional space. 

The main contribution of the paper is the presentation of a suite of simple, representative physics problems along with reference numerical solutions for traditional time integration schemes to benchmark data-driven methods (MLPS, CNNs, kernel machines, Nearest neighbors).  The key conclusion of the paper is that, even in the simplest physical models, current
data-driven pipelines, while providing qualitatively acceptable solutions, are quantitatively far from
directly numerically integrating physical models, and this performance gap appears unfeasible to
close by merely scaling up the models and/or the dataset size.   Another finding is that a simple L2-based nearest neighbor regressor outperforms most deep learning models in the narrow regime for complex systems such as incompressible Navier-Stokes systems. 
strengths: -- Provides a standard benchmark for contrasting traditional solutions for numerical computation with physics based models against data-driven methods.
-- I like the progression of complexity of the benchmarks and the exploration of narrow and wide data regimes. Such a standard benchmark allows for systematic evaluation and tradeoff analysis.
-- The experimental results provide sufficient insights 
-- The code and data is made accessible and is extensible
-- No ethical/social implications exist
weaknesses: -- Apart from highlighting the deviations in results between data driven and traditional methods, It is not absolutely clear how the benchmark results points to ML algorithm improvements.  (The authors state this as a point of strength of the paper, but it is not obvious what they imply by this).
-- While I like the organization of the benchmark in the paper, I would have assumed that a more simpler differential system with well known closed form solution for the result may be an interesting option as a baseline.  
-- The paper will be strengthened significantly if other aspects not covered by the authors (described in the limitation sections) are included including the missing timing analysis.  
correctness: The claims appear to be correct and the experiments are conducted in a sound way. The experimental design and evaluation is fine.
clarity: The paper is very well written and clear. 
relation to prior work: The references to prior work is adequate.   I have seen a number of papers in the field with each of them doing their own evaluation and the current paper is attempting to have a coherent benchmark covering a range of problems.  I am absolutely not sure if there are other relevant benchmark papers in the field that are not cited.
additional feedback: Can you comment about the points in the weakness section?
review_title: An attempt to provide unified benchmarks for neural net / ml models in scientific computation involving physics based model simulations. 

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: This paper introduced a benchmark for the methods of data-driven-based physical system simulation. They provided a dataset of four physical systems (spring, wave, spring mesh, navier-stokes) including data, data generation methods, codes, and documents. Then, they conducted a variety of experiments using knn, nn kernel, MLPs, and CNNs. This work is interesting, and important for learning physical systems. The descriptions are also detailed. However, the value of this benchmark needs to be more explained, please see Weaknesses.
strengths: - This is hard work (850h on GPU + 2180h on CPU). It is really difficult to process such a large amount of data and keep it unified and correct.

- The data, source codes, and documents are all well organized.

- The generation methods, documents, and experiments are described in detail.
weaknesses: For a paper of dataset and benchmark, the contributions should be novel/originality data, comparison and analysis of state-of-the-art methods, and the impact on the future of related fields. I hope the authors could give more explanation about these points.

- The paper claimed that the proposed benchmark takes a step towards unified evaluation protocols and metrics, but it is unclear.

- Data. The complex system is more important for simulation problems. The value of simple system data needs to be explained. The data of four systems are also generated in the existing paper, such as spring in [7], Navier-Stokes in [48]. What is the difference between the provided data and the previous data?

- Benchmarks. Only simple baseline models are tested (except for u-net on naiver-stokes), and the paper claim that these models could not solve the problem well. However, advanced models have been proposed and compared with the baseline models in the existing papers. SOTA models should be tested and analyzed.
correctness: Yes.
clarity: Yes.
relation to prior work: No.
additional feedback: - more references should be cited in the Introduction to clarify the current problem.
- difference with the previous works should be described.
review_title: Interesting work, but the value of data and benchmarks is unclear.

Meta-review: This paper introduces a physical evaluation dataset framework for scientific computing pipelines that map one high-dim state space into another high- or low-dim one, providing a suite of simple representative physics problems.  Reviewers appreciated for motivation, clarity, comprehensiveness, and overall contribution to the space.
Decision: Accept