Venue: NeurIPS 2022 Track Datasets and Benchmarks

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: summary and contributions: The authors engineered a platform for users to perform simulations of spcefic kinds, power grid and the pneumatic. ML models are used to fit some controlling parameters used in the simulation engines.
strengths: Physical simulations are becoming an important interface marrying ML and Science. Fluid dynamics, molecular dynamics, quantum dynamics and neucler power simulators, to name only a few, all fall into the broad concept of "physical simulations". It is desired that more platforms naturally supporting both AI framework and physical simulators could come into use.
weaknesses: 1. The name of "LIPS" seems ambitious, but the current work is limited to pneumatic and power-grid. It is not clear how LIPS can be extended to other physical simulations, and to what kind/domain of physical simulations would be accounted in the future plan for LIPS. 
2. The central work is more engineer-related, and there lacks a theoretical breakthrough or any essential connection with ML or AI. So the reviewer would be concerned with its readership in the context of NeurIPS. 
correctness: The reviewer is not an expert in PDE simulations and cannot judge the correctness.
clarity: The overall writing of the paper seems fine to the reviewer.
relation to prior work: The authors properly discussed previous work related to some PDE simulations, and some existing industrial applications to power grids.
additional feedback: Currently no more feedback.
review_title: Good efforts but does not fit with NeurIPS

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: This paper presents a benchmark for applying machine learning techniques to speed up industrial physical simulation. In addition to laying out two real world problems where speedups are needed, the paper categorizes important metrics for application of solution techniques and presents these results qualitatively. The authors provide an API for access to the data and scripts to reproduce results.

strengths: * For researchers in physical simulation, it is difficult to find problem settings that are interesting/complex to explore and are guided by industrial need. This benchmark is a very useful and distinct addition to the existing set of canonical PDEs that are often used to benchmark learning methods.
* The evaluation criteria effectively target metrics to compare against existing state of the art methods (traditional simulation), not just among other learning methods. These include how much faster, how well do they generalize and are they physically consistent.
* The results are very well presented (Table 2) and while it is not always possible to threshold performance, I would like to see something like this adopted more widely.
* The datasets and reproducible results are released using a simple API. The authors also set up the benchmark on an automated platform, making comparisons with future work more transparent.

weaknesses: It would be nice to have a comparison with differently tuned classical simulations. For example, with FEM in the pneumatic tire case, you can vary the mesh coarseness as well as the time step. This would be interesting to study as a trade off between speed and accuracy.

correctness: Yes, see strengths about the evaluation methods.
clarity: The paper is clearly written and the results are presented well.

relation to prior work: I think there is good discussion of prior work. Similarities exist, but this paper effectively sets itself apart from other learning benchmarks on dynamical systems and other treatments of the subtasks in its stated evaluation criteria and goals.

additional feedback: N/A
review_title: Benchmark driven by industry need captures important metrics for method application

final_score: rating: 4: Ok but not good enough - rejection
review_content: summary and contributions: The paper introduces a set of datasets for learning the industrial physical simulation. The proposed datasets are targeted at two real-world applications: power grid and pneumatic simulation of tires. The power-grid dataset consists of two benchmark tasks: Risk assessment through contingency screening & Remedial action search. The pneumatic-simulation dataset also consists of two benchmark tasks: Static vertical stiffness simulation & Design testing during a rolling cycle. To evaluate the performance on these benchmark tasks, the paper also proposes metrics: ML-related performance, Industrial Readiness, Application-based out-of-distribution Generalization, and Physics compliance. The paper finally presents several heuristic baselines to show the unsatisfactory	performance of current methods, which calls for a family of new methods. 

Overall, this paper presents a set of promising benchmark tasks for industrial simulation but lacks sufficient details and baseline methods to support the claims. 
strengths: **Significance**: 
The paper considers an interesting industrial simulation learning problem, and presents two valuable datasets for learning the simulation for power grid and the pneumatic types. The datasets might be useful in a broad AI4science community and could facilitate the relevant research. 


**Accessibility**: 
The paper provides the link to the code and also offers a nice way to compare different methods via Codabench. 
weaknesses: **Significance**: [the challenges posed by this benchmark is unclear] 

Although the paper offers a good discussion of the need for fast physical simulation and presents the two datasets for learning such physical simulation, it is still unclear what the challenges this benchmark is targeted at. Specifically, the datasets themselves have not been described to reveal the underlying research questions. For example, for Risk assessment through contingency screening benchmark, it seems that the major challenge is the out-of-distribution (OOD) generalisation. But it is not quite clear what “OOD topology test set” (line 227) really is and how it differs from the training set (in terms of OOD). It would be nice to provide more analysis about the dataset difference and the OOD issue. For other benchmark tasks, e.g., Remedial action search and  Static vertical stiffness, it is not even clear what the tasks are from the machine learning perspective, let alone the challenges posed by the dataset.


**Relevance**: [the relevance to the machine learning research is unclear] 

The baseline methods proposed in the paper do not sufficiently showcase the difficulties of the benchmark tasks. It instead poses another question as to how relevant the benchmark is to the machine learning research. For example, a fully-connected (FC) architecture is used for the power-grid benchmark, whereas a convolution architecture is used for pneumatic. It looks like different neural network architectures (an engineering choices, not a research question) may play a crucial role. Would it be possible to try more heuristic methods (e.g., different architectures) on both datasets? Moreover, it is unclear how the proposed evaluation metrics can be quantified, e.g., “Scalability: the computational complexity of a surrogate method should scale well depending on the problem size” (line 198 - 199), "extrapolate over minimal variations of the problem geometry” (line 204), and “the expected level of criticality of the benchmark” (line 208)


**Accountability**: The current version of the paper raises some questions about the dataset details: from the machine learning perspective, what are the input and output for each dataset? How large is the dataset size? How the training and testing dataset is split? Why U-net is used as a baseline for pneumatic dataset? 


The current version of this paper may not be qualified as an accept. If the paper could present the dataset and the benchmark with clearly defined research questions and more convincing baselines, it would be a good paper. Given the time constraint in the rebuttal period, it might be hard to address all above issues. So I tend to reject this paper for its current version. 
correctness: The claims are correct. However, it is unclear how the dataset is constructed (e.g., input and output, data structure etc.). the evaluation methods are a bit weak and the evaluation metrics are not quite clear. 
clarity: The paper is written well in general. But I have some questions: 

* Table 2: Circles in the table title and in the table are in different colors. 

* Table 2: it is unclear what it means by “two problem scales reported”

* Table 2: what is the meaning of these numbers? e.g., 19(7). 

* Section 4.1 Comprehensive evaluation criteria: how should the Industrial Readiness, particularly “Scalability: the computational complexity of a surrogate method should scale well depending on the problem size”, be measured? 
relation to prior work: yes
additional feedback: n/a
review_title: Reviews for industrial simulation learning paper

final_score: rating: 6: Marginally above acceptance threshold
review_content: summary and contributions: This paper proposes a simulation for industry use cases (e.g., Power System Operators) as opposed to general-purpose simulation tasks and 4 categories of evaluation metrics. They also provide an open-source benchmarking suite and a public leaderboard for user submission and a fully automated and comparable evaluation.
strengths: This paper focuses on specific industrial tasks rather than general physical simulations, which benefits the development of ML algorithms in industrial applications.
weaknesses: No clear comparison between the proposed simulator and previous ones. According to Section 3, there are some similar simulators such as SimBench and Power Grid Lib. A table that lists all features and advantages of these simulators would make the comparison clearer.


correctness: This paper proposes 4 sets of metrics, including ML-related performance, Industrial Readiness, Application-based out-of-distribution Generalization, and Physics compliance. According to the description of these metrics, the proposed benchmark is appropriately designed.
clarity: Yes.
relation to prior work: A table that lists all features and advantages of these simulators would make the comparison clearer.
additional feedback: None.
review_title: Paper Review

Meta-review: metareview: This work introduce a new benchmark to evaluate machine learning techniques for physical simulation. Based on two real-world applications (power grid and pneumatic simulation), the authors clarified the research questions and define evaluation metrics. Existing ML-based methods do not perform well on the proposed datasets. Overall all reviewer agreed that this work proposes a promising benchmark and I think the proposed benchmark can be useful in real-world application. Even though one reviewer has some concerns on details and setups, the authors provided detailed response to address them. For this reason, I'd like to recommend accept.
confidence: 4: The area chair is confident but not absolutely certain
Decision: Accept