Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper presents a dataset containing synthetic models of power grids combined with the statistical results of dynamic simulations (quantifying single-node basin stability and survivability - abbreviated SNBS and SURV). The authors train a baseline GNN to show initial efficacy of the task of predicting SNBS and SURV, as well as of transferring the learned model to larger power grids.
strength_and_weaknesses: Strengths:
* The proposed dataset characterizes an important problem. Specifically, the problem of dynamic stability is indeed increasingly important to address as power grids integrate larger proportions of time-varying renewable energy.

Weaknesses:
* The dataset structure is not clearly described, and the benchmark task (what are the inputs, what are the outputs, and what is the metric of success) is not cleanly defined. This information is implicit within the text and citations, but needs to be laid out much more clearly and explicitly to enable understanding of the specific benchmark presented - e.g., I would have expected to see some of the relevant equations laid out in the paper.
* The setup of the baseline GNNs is also not clearly enough described. For instance, the text describes that the GNNs are given "an adjacency matrix representing the topology of power grids and a binary feature vector representing sources and sinks," without additional detail - however, for a dataset paper, those inputs should be clearly spelled out.
* Given that this is a machine learning audience as opposed to a power systems audience, the introduction needs to be made much more accessible. For instance, terms/phrases such as "have less inertia," "contingencies," and "self-organized synchronization mechanism" may not be immediately accessible to an ML audience. In addition, the word "transformers" should likely be clarified as "power transformers," due to the different default meaning of this term in the deep learning literature.
* Since all the data is synthetically generated, it is not clear to me why this is presented as only a static dataset, rather than the data generation code also being shared. See, e.g., [1] for an example of a dataset paper that does it this way.

[1] Joswig-Jones, Trager, Kyri Baker, and Ahmed S. Zamzam. "OPF-Learn: An open-source framework for creating representative AC optimal power flow datasets." 2022 IEEE Power & Energy Society Innovative Smart Grid Technologies Conference (ISGT). IEEE, 2022.
clarity,_quality,_novelty_and_reproducibility: While there is great potential in this paper, I think there is unfortunately great room for improvement in terms of clarity and reproducibility. The novelty is in that dynamic stability datasets of the presented size do not exist in the literature. The quality of the dataset is likely good, but it is difficult to tell from the writeup.
summary_of_the_review: The proposed dataset characterizes an important problem. However, the writeup does not clearly define the dataset/benchmark task or the baseline method. In addition, it would likely make more sense to present this as a software package for generating data, _alongside_ a fixed dataset, rather than just as a fixed dataset. 

If the authors are able to make significant revisions to the writeup during the revision period, I would potentially be willing to significantly increase my score, given that my issues are (likely) with the presentation rather than the content of the work (at least, based on what I'm able to assess from the current writeup). In general, I think dataset papers are extremely valuable, particularly those dealing with climate/energy problems, but unfortunately the current writeup is not of high enough quality for this reviewer to recommend acceptance without such significant revisions.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The authors introduce new datasets of dynamic stability of synthetic power grids. 
They show that large Graph Neural Networks GNNs outperform GNNs from previous work at predicting single-node basin stability.They demonstrate that GNNs can be used to identify trouble maker-nodes in the power grids and show that GNNs trained on small grids can perform accurately on a large synthetic Texan power grid model.
strength_and_weaknesses: I am unfortunately not super knowledgeable about power-grids but enjoyed reading this interesting paper. 

I believe that the strength of this paper lies in the introduction of a new dataset of higher complexity than previous existing datasets. I think this is very important for evaluating and training models.
The other strength of the proposed model is the computational efficiency and scalability (1,800 times faster for grids of size 100 and 4,6 ×107 times faster for the synthetic Texan power grid). 

The fact that GNNs can generalize from a small dataset to a bigger one is very compelling, but the datasets are very similar, and generated using the same modeling strategy. I am not surprised that a GNN can generalize well in this setting, since what it learns from dataset20 will hold in dataset100. I think that this result is overstated by the authors. I might not understand the differences between dataset20 and dataset100 well enough to understand the depth of this statement. I'd appreciate if the authors could explain this to me in more details. 

I'm also not sure that it makes sense to flag a node as trouble maker if small disturbances are amplified by a factor of at least 6. This seems a bit arbitrary. Is there any reason coming from analysis of real datasets for this? 
This is then used for a classification task whereas I think a regression task would be more appropriate. For example, predicting the amplification factor rather than predicting if the factor is above or below 6. Also, reporting the uncertainty should be interesting. Is the model more confident on nodes with higher amplification factor? To use in a real setting, we would want the model to have a good uncertainty representation. 

One thing I didn't understand is why, in figure 3, the texan grid has 97% of stable nodes although the distribution of SNBS seems to give much less probability for high SNBS (around 1). I'd appreciate some clarification here.

In figure 5, the model seems to work very well for low SNBS but have skewed predictions for high SNBS. Would it be possible to fix this using some non-linearity? 

Overall, there are lots of typos that make the paper a bit hard to read. For example in the abstract: 
We show that large GNNs outperform GNNs from previous work as well as as handcrafted graph features and semi-analytic approximations.




clarity,_quality,_novelty_and_reproducibility: This work is novel as it introduces a new synthetic dataset and extends previous models. 
I think the clarity and quality of the writing could be improved by carefully checking the typos (a few entire sentences do not make sense, even in the abstract)
summary_of_the_review: I am not very confident, and overall enjoyed the paper. 
The new dataset is a big contribution, but I would like some clarification on the tasks that are used to evaluate the model (trouble node classification, SNBS prediction) as I'm not sure they fully support the claims made in the paper.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper introduces a new dataset of probabilistic dynamic stability of synthetic power grids which is much larger than the existing datasets. Moreover, it also includes a Texan power grid model alongside with the dataset. Second, this paper also proposes a new method to identify the nodes which could be the one leading to the unstability. Finally, the papers evaluate several GNN baselines on the proposed benchmark and made a complete analysis. In the experimental results, the paper shows the potential to use the GNN model trained with the offline data to conduct the prediction on the real-world power grid.
strength_and_weaknesses: ## Strength:
1. This paper is generally well written and the background is sufficient to let the people who is not an expert in power systems understand the task.
2. The details of generating data are provided.
3. The figures are concise to show the ideas.
4. The anlysis of experimental results are sufficient and solid.

## Weaknesses:
1. No significant novelty appeared in this paper.
2. The method to identify the trouble-maker nodes is not clearly described at least from the main part of paper. I suggest the authors can merge the related contents in the appendix and rewrite it using compact words.
3. The authors claim that the trouble-maker identification is a classification task. However, in my view it is just a criterion rather than a classification task that need the ML model to conduct.

clarity,_quality,_novelty_and_reproducibility: The originality of this paper is good, since it provides a new dataset about a significant real-world problem with detailed description of the generation of data so that it is trustful.

The novelty of this paper is lacking, but it is just because of the output of this paper. It provides a larger dataset than the previous ones, rather than propose a new research problem.

The clarity of this paper is good enough, except the description of identification of trouble-maker nodes. 

The reproducibility of this paper is good, since it provides the experimental details to reproduce the results and the authors promise that they will open-source the dataset and codes afterwards.

The general quality of this paper is above the average in my view.


summary_of_the_review: This paper provides a new benchmark dataset about a significant real-world problem and it has shown the evaluation with several baselines which could provide some lessons to the following people. The background of the problem is sufficient. The description of the identification of trouble-maker nodes needs to be clarified, which may change my decision.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The reviewers were split about this paper and did not come to a consensus: on one hand they appreciated the importance of releasing datasets/benchmarks to the ICLR community (especially in the climate/energy space), on the other hand they were confused why the data was presented as static given that the authors have designed a simulator and argued that the description was not very accessible to an ML audience and is missing important details. After going through the paper and the discussion I have decided to vote to reject for the following reason: because this a dataset paper, presentation and empirical analysis should be held to a higher standard than a paper that also proposes new methodology. Even with the changes made by the authors in response to reviewer comments, the paper is not easy to access for a general ML reader. For example, the first five paragraphs of the introduction, while interesting, are overly detailed to an ML audience that is developing new models for the dynamic stability prediction problem. Much of it is historical background which can be condensed, or highly technical information which requires very specialized knowledge to parse. Mathematical notation is also often confusing, e.g., the authors use $p(y | x \sim \rho)$ to mean $p(y|x)$ integrated over distribution $\rho(x)$, instead of the standard $\mathbb{E}_{\rho(x)}[p(y|x)]$. In another example $\phi(\infty)$ is used, which I believe means $\phi$ at the final timestep, but prior to this $\phi_i$ is defined as a term indexed only by nodes $i$, its extension to a time-dependent function is never stated, but is assumed to be known in equation (3). Further the analysis is limited: e.g., Figure 5 is used to argue that increasing the number of grids increases model performance, but only two grid sizes are used, and it is not clear why the authors choose 7000 grids as the larger size, instead of regular increments increasing from 800 (a size used by Nauck et al., 2022). For these reasons, I vote to reject.
However, these things do not mean the paper is a bad fit for all venues. In fact, I bet this paper would be much better received at an power-generation conference/journal. I’d recommend submitting this work there and then creating a competition for NeurIPS, which has worked out well for past power grid datasets: https://l2rpn.chalearn.org/home
justification_for_why_not_higher_score: See above metareview.
Decision: Reject