PGLearn - An Open-Source Learning Toolkit for Optimal Power Flow

Authors: Michael Klamkin, Mathieu Tanneau, Pascal Van Hentenryck
Primary area: datasets and benchmarks
Keywords: optimal power flow, machine learning, dataset

Abstract:
Machine learning techniques for Optimal Power Flow (OPF) problems have recently garnered significant attention, reflecting a broader trend of leveraging machine learning to approximate and/or accelerate the resolution of complex optimization problems. These developments are necessitated by the increased volatility and scale in energy production for modern and future grids. However, progress in ML for OPF is hindered by the lack of standardized datasets and evaluation metrics, from generating and solving OPF instances, to training and benchmarking machine learning models. To address this challenge, this paper introduces PGLearn, a comprehensive suite of standardized datasets and evaluation tools for ML and OPF. PGLearn implements realistic data generation procedures that capture both global and local variability, ensuring that datasets are representative of real-world conditions. In addition, it supports multiple OPF formulations, including AC, DC, and second-order cone formulations. Standardized datasets are made publicly available to democratize access to this field, reduce the burden of data generation, and enable the fair comparison of various methodologies. PGLearn also includes a robust toolkit for training, evaluating, and benchmarking machine learning models for OPF, with the goal of standardizing performance evaluation across the field. By promoting open, standardized datasets and evaluation metrics, PGLearn aims at democratizing and accelerating research and innovation in machine learning applications for optimal power flow problems.

Decision: reject
Reviewer scores: 8, 3, 3, 6, 6, 8

Reviews and discussion:
<thread>
creation date:
1729172255040

thread_type:
official_review

<thread_object>
creator: reviewer_tmt6

<content>
summary:
The paper tackles an important issue in the field of machine learning for Optimal Power Flow (OPF) problems by introducing PGLearn, a standardized suite of datasets and evaluation tools. This addresses the lack of uniformity in data and metrics that has hindered progress in this area.

strengths:
•	The paper presents robust procedures for generating realistic datasets.
•	It offers support for multiple OPF formulations, including AC, DC, and second-order cone models.
•	The dataset is structured in the N-1 contingency format which can be used to enhance power systems reliability.
•	Standardized datasets are made publicly accessible for broader use.
•	A comprehensive toolkit is provided for training, evaluating, and benchmarking machine learning models for OPF.
•	The limitations of the dataset are clearly documented and reported for transparency.

weaknesses:
•  The input data is generated using a uniform sampling procedure, but in the future to extend the work for Unit Commitment or multi-period OPF, time series data should also be incorporated. For instance, the recent paper "PowerGraph: A Power Grid Benchmark Dataset for Graph Neural Networks" provides an OPF dataset that uses time series data to model input demand.

questions:
• Could you provide comments on the results of benchmarking the datasets using the provided metrics across different machine-learning methods? A leaderboard would be the easiest way to access and interpret these results.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730525540055

thread_type:
official_review

<thread_object>
creator: reviewer_uide

<content>
summary:
This paper presents PGLearn, a set of datasets and evaluation tools for optimal power flow (OPF), which builds upon the learnings of existing open-source initiatives. Their main contribution relies on including a different data generation process that considers local and global variability, better representing the actual use cases. It supports different formulations of the OPF problem, namely AC, DC, and second-order cone formulations. The motivation behind this resource is to standardize the performance evaluation among the OPF community, which seems to be fragmented and lacking reproducibility between contributions.

strengths:
- The paper revisited the OPF problem and identified a gap in the existing approaches, such as the sampling strategy. Then, they proposed a better method that considers the correlation between loads and better represents real-life scenarios.
- Including the primal and dual solutions is another novelty, as few existing datasets include the dual, which is something specific research directions need.
- The authors propose a complete pipeline to interact with PGLearn containing standardized datasets, existing optimizers, data augmentation schemes, distinct OPF formulation, and metric calculation.
- The paper is well structured and clear and introduces their contribution smoothly.
- The authors stated clearly the limitations of their contribution

weaknesses:
- Algorithm 1, the notation of what you return could be misleading as you used both $(p^d,q^d)$ for referenced demand and sampled demand. It looks as if you are returning the same reference value at the end of the algorithm.
- The sampling strategy needs tuning the $\epsilon$ and global range. You could add more detail on how to pick the values, giving a case or detail why did you choose the quantities you set.
- The supplementary material is not self-explanatory for someone outside of the OPF domain; I can understand each file from what I read in your paper's appendix, but you could have added basic instructions. Without instructions, it is just a folder with h5 files.
- For a work like yours, you should've added an anonymized version of your code to assess how you implemented what you described only with words in the main paper
- If putting the code is restrictive at this stage, you could also give an idea of the workflow of PGLearn so readers can get an idea of what kind of interaction they can have with your dataset.

questions:
- When you mention having over 1000000 OPF samples, are they all precomputed with your dataset, or do you provide a method to access the results (h5 files)?
- Can you please provide me with step-by-step instructions on empirically using the supplementary material to check your contribution?
- How can you ensure that your sampling strategy stays within normal ranges? Why did you choose the values you picked?
- In section 5.1, what is the main difference between the Optimality gap and the Distance to optimal solution?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730539441051

thread_type:
official_review

<thread_object>
creator: reviewer_acra

<content>
summary:
The paper introduces PGLearn, a suite of standardized datasets and evaluation tools designed to advance ML applications in OPF problems. It tries to provide the standard genration of varied data and evaluation metrics by providing realistic datasets that reflect real-world conditions and support multiple OPF formulations.

strengths:
1. The dataset is timely and important given the current active research in ML for OPF
2. The presentation is clear and easy to follow

weaknesses:
1. For my best understanding, the paper mainly focus on the dataset introduction, it does not deliver the technical advances in the search field
2. While the datasets aim to capture real-world conditions, unforeseen variability in actual grids may still limit the applicability of the models trained on these datasets.
3. It only focus on OPF problems, which may not be fit for the general audience of the ML community

questions:
1. The authors are suggested to include a general review on the ML for OPF papers, e.g., how ML are generally applied to OPF problems. There have been several literature review papers on it.

2. The authors are suggested to state clearly whether the kit contains the function to train/test different ML models? If so, please clarify how to integrate the numerious DNN models and approaches; if not, how should the mentioned Metrics be calculated? Is it still done based on the user side to mannually do so?

3. It is also useful to add a set of adversial samples in the dataset for robustness testing under the worst case scenario

4. Can the suite allow users to self-define the problem and use e.g., API calls, to load the generated dataset? It can be helpful if user have some uncommon reformulated OPF formulations and other general optimization formulation
</content>
</thread_object>

</thread>
<thread>
creation date:
1730605764157

thread_type:
official_review

<thread_object>
creator: reviewer_19cy

<content>
summary:
This work describes a pipeline for sampling training data of Optimal Power Flow-machine learning tasks. It proposes a unified framework for sampling training data samples for OPF problem based on pre-defined grid models.The proposed  PGLearn implements realistic data generation procedures that capture load-side variability, and a set of OPF formulations are implemented.

strengths:
* The idea of performing standard sampling of OPF data is constructive and shall help the power systems and machine learning community. 
* The paper calls for a standard implementation and testing of Machine learning for OPF problems, which are important.
* The paper brings our descriptions on a set of metrics on algorithm evaluation.

weaknesses:
* The proposed method is still limited to user-defined synthetic settings, while the scalability to other grid models/data distributions, other power system tasks, or settings other than supervised learning are not discussed. 
* The contribution is limited to dataset creation and standard testing. No algorithm insights nor new findings of learning to solve optimal power flow problems are reported.
* Data augmentation is treated as the sole solution for tackling the non-standardized implementation problem for ML-OPF. Are there any other choices, such as sim-to-real transfer, multi-task or meta learning?
* Though the paper claimed considering the correlation between samples, only uniform noises are added in a two-step approach. Uniform sampling is still far away from the real-world load data correlations. And the paper omits the discussion on how the proposed data augmentation approach resembles realistic settings.
* The design of standard grid testbed models were rooted in abstraction and representation of real-world transmission grids. However, the paper falls short in explaining in details how the real-world data distribution can be applied to construct dataset for PGLearn.
* In the demand sampling part, " The power factor at each node is kept constant, i.e. the active and reactive power at each load is scaled by the same amount." This is not a very realistic setting, as power factor is having variations, and has a big impact on the OPF solutions.
* There is no inclusion nor analysis of contingencies in OPF problems. Essentially the introduction mentions renewable uncertainties multiple times, while the proposed method does not address such uncertainty issues.
* The PGLearn module is limited to implementations of few methods by similar group of authors. A more thorough literature review and inclusion of methods are recommended:
Zhang, Ling, Yize Chen, and Baosen Zhang. "A convex neural network solver for DCOPF with generalization guarantees." IEEE Transactions on Control of Network Systems 9, no. 2 (2021): 719-730.
Zhou, Min, Minghua Chen, and Steven H. Low. "DeepOPF-FT: One deep neural network for multiple AC-OPF problems with flexible topology." IEEE Transactions on Power Systems 38, no. 1 (2022): 964-967.
Owerko, Damian, Fernando Gama, and Alejandro Ribeiro. "Unsupervised optimal power flow using graph neural networks." In ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 6885-6889. IEEE, 2024.
* It seems the results on the four accuracy metrics and computation metrics are not reported for the implemented methods.

questions:
I have the following questions regarding the current paper:

* Could the authors point the link to the anonymousrepo? And in these two repos, are the proposed four accuracy metrics and computation metrics realized?
* PGLearn generates a bunch of data in terms of primal and dual solutions. But some more interactive representations shall be considered, e.g., generator/line value w.r.t capacity, prediction error, visualization of solution on grid, correlation between nodes and etc.

Some minor issues and questions:
* in Introduction, "The increased volatility and scale of energy generation in modern and future grids..." Shall be power generation in the scope of OPF. Similarly in the second paragraph: " account for uncertainties in renewable energy generation and/or demand..."
* In Introduction, "previously intractable applications such as real-time risk analysis", yet in this work, there is no discussion on how ML can help with OPF risk.
* "inherent volatility of wind and solar generation creates OPF problems that are, or will be, orders of magnitude larger that today’s instances." Why the increase of volatility will make the OPF instances larger? 
* The PGLib-OPF seems to be an important reference. Could the author explain more on "PGLib-OPF only provides a single snapshot per grid"? Does that mean only one data sample is provided for each power grid model?
* In Appendix A.2.2, the authors mentioned the bus-pair variables were changed to branch on each variable. Could the authors explain more why this has marginal impact on the relaxation? And does this mean the ML model is learning the newly defined variables instead?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730756405072

thread_type:
official_review

<thread_object>
creator: reviewer_eirm

<content>
summary:
The authors argue that the numerous datasets used in the benchmarking of ACOPF codes, incl. the Grid Optimization Competition with $9M+ in prize money (https://gocompetition.energy.gov/) and Google DeepMind OPFData (https://arxiv.org/pdf/2406.07234), have minor issues, and need to be replaced by their benchmark. The authors consider 65,536 of samples generated for instances on up to 9241 buses (the Pegase project instance), which are smaller than the OPFData (Lovett et al., 2024) with instances on up to 13,659 buses, but bigger than those of PowerModels (Coffrin et al., 2018) on up to 118 buses with 10,000 samples. The authors consider correlated demand perturbations, but do not consider diurnal variations, or similar.

strengths:
The writeup is reasonably clear.

weaknesses:
The instances on up to 9241 buses (the Pegase project instance) are smaller than those of the Deepmind OPFData (Lovett et al., 2024) with instances on up to 13,659 buses. 

While considering correlated demand perturbations may improve the realism somewhat, the authors should ideally consider real-world time-series capturing diurnal variations, or similar. Likewise, they could consider time-varying production limits, which are readily available to all market participants in Europe via the ENTSO-E Transparency Platform (https://transparency.entsoe.eu/). 

The supplementary material contains no code, so the code could not checked for reproducibility or usefulness.

questions:
Have you considered the use of the ENTSO-E Transparency Platform (https://transparency.entsoe.eu/)?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730760687209

thread_type:
official_review

<thread_object>
creator: reviewer_m2kx

<content>
summary:
The authors proposed a library that generates datasets to standardize ML for OPF research. While each study in the literature shows improvement over earlier models, it is hard to fairly compare the model performance across different studies. 

The authors proposed a library that constructs input output pairs to different formulations of the OPF problem (AC-OPF, DC-OPF and second order cone relaxation of AC-OPF) on large middle to large sized test cases. They also address data augmentation issues in the literature that results in a narrow demand range and implement a procedure that captures global larger demand characteristics. The library also contains solvers implemented in efficient libraries to find the corresponding optimal solutions and evaluation metrics to measure model accuracy and computational cost.

strengths:
1. A typical ML for OPF study goes through the steps of generation of input parameters (e.g. demand), finding corresponding solutions, model training and evaluation. The he authors contribute the literature with a toolkit that standardizes all these processes.
2. I think main contribution of the study is that the proposed library improves  comparability and fair assessment of existing models. Many studies in the literature show improvement over some baselines but are not able to compare against the state of the art. The library enables researchers to compare their approaches with previously reported benchmarks without implementing each specific model.
3. Another contribution is the data augmentation procedure that potentially generates different load characteristics.
4. The library also implements widely solved formulations of OPF using efficiently written solvers.

weaknesses:
1. There are not many systems implemented in the library and most of them are large systems. Researchers may want to work on smaller systems for proof of concept or visualizing the model behaviour. It would be nice to add them.
2. While the proposed data augmentation method generates wider range of demand dynamics than sampling random factor and multiplying the base load, I am not sure this approach captures the global dynamics. The demand characteristics can differ widely within the hours of a day or among different seasons, which can result in different constraints to bind at the solution. This is mentioned as a limitation, but I think more detailed demand scenarios should be considered to train more reliable NN models.

questions:
See weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732553983557

thread_type:
official_comment

<thread_object>
creator: authors

<content>
comment:
We thank the reviewers for evaluating our paper and for their constructive feedback which has helped us improve the paper.
In what follows, we review the most common feedback and summarize the changes they inspired.

## Paper positioning

The core contribution of our paper is twofold: the PGLearn collection of datasets, and standard evaluation metrics to benchmark the performance of ML methods for OPF problems. 
Both are accompanied by open-source code for data-generation and model evaluation.
We believe that releasing these large standardized datasets is a meaningful step forward for the community, as it will enable multiple research teams to contribute to this field without requiring extensive computational resources, and will streamline the comparison of various methodologies.

Given the large breadth of work in this field, we do not have the resources to implement, train and evaluate, by ourselves, all existing methods for ML and OPF.
Rather, we provide a standard evaluation framework for researchers to measure and report performance in a principled fashion.
We thank reviewer `tMt6` for suggesting a community leaderboard; we hope to host one in the future to which the community can contribute.

## Additional cases

Several reviewers (`M2KX`, `EiRM`) suggested including additional cases as part of PGLearn, also noting that the selection focuses too much on the medium/large benchmarks.
While our focus on large systems is motivated by the gap between existing academic works and the scale of real-life systems, we recognize the need for including smaller systems for testing / development.
Therefore, we have extended our collection with four small cases from the PGLib collection (`14_ieee`, `30_ieee`, `118_ieee`, `300_ieee`), and two larger cases `13659_pegase` and `midwest24k`.
The updated version of Table 1, showing the full case statistics, is included below for reference:

| Case name    | Nodes | Loads | Generators | Edges | Total PD | Total PG | Global range |
|--------------|-------|-------|------------|-------|----------|----------|--------------|
| 14_ieee      | 14    | 11    | 5          | 20    | 0.3  GW  | 0.4  GW  | 70% -- 110%  |
| 30_ieee      | 30    | 21    | 6          | 41    | 0.3  GW  | 0.4  GW  | 60% -- 100%  |
| 89_pegase    | 89    | 35    | 12         | 210   |   6  GW  |  10  GW  | 60% -- 100%  |
| 118_ieee     | 118   | 99    | 54         | 186   |   4  GW  |   7  GW  | 80% -- 120%  |
| 300_ieee     | 300   | 201   | 69         | 411   |  24  GW  |  36  GW  | 60% -- 100%  |
| 1354_pegase  | 1354  | 673   | 260        | 1991  |  73  GW  | 129  GW  | 70% -- 110%  |
| nyiso_2030   | 1576  | 1446  | 323        | 2427  |  33  GW  |  42  GW  | 70% -- 110%  |
| 1888_rte     | 1888  | 1000  | 290        | 2531  |  59  GW  |  89  GW  | 70% -- 110%  |
| 2869_pegase  | 2869  | 1491  | 510        | 4582  | 132  GW  | 231  GW  | 60% -- 100%  |
| 6470_rte     | 6470  | 3670  | 1330       | 9005  |  97  GW  | 118  GW  | 60% -- 100%  |
| Texas7k      | 6717  | 4541  | 637        | 9140  |  75  GW  |  97  GW  | 80% -- 120%  |
| 9241_pegase  | 9241  | 4895  | 1445       | 16049 | 312  GW  | 530  GW  | 60% -- 100%  |
| 13659_pegase | 13659 | 5544  | 4092       | 20467 | 381  GW  | 981  GW  | 60% -- 100%  |
| midwest24k   | 23643 | 11727 | 5646       | 33739 | 104  GW  | 318  GW  | 90% -- 130%  |

Moreover, we now group the 14 cases into four categories based on the number of buses:

**Small ($<$1k buses)**:  14_ieee, 30_ieee, 89_pegase, 118_ieee, 300_ieee

**Medium ($<$5k buses)**: 1354_pegase, nyiso_2030, 1888_rte, 2869_pegase

**Large ($<$10k buses)**: 6470_rte, Texas7k, 9241_pegase

**Extra-Large ($>$10k buses)**: 13659_pegase, midwest24k

Finally, we would like to point out that, compared to the recent OPFData dataset, we provide 1) our data-generation code, 2) additional test cases, 3) multiple formulations (DC, SOC and AC optimal power flow), and 4) both primal _and_ dual solutions.


## Demand Sampling Procedure

Reviewers `M2Kx`, `EiRM`, `19cY`, and `tMt6` remarked that PGLearn does not incorporate time series information, as we noted in our Limitations section.
We agree that capturing temporal information is an important aspect for impactful ML research in this field.
Nevertheless, we believe that two more pressing needs are 1) increasing the scale of standard test cases used by the community and 2) capturing a broader range of variation in total demand (as evidence by Figure 2 in the paper).
We are currently working on integrating time series information in our data-generation and evaluation procedures, which will be part of a future version of PGLearn.

Following the feedback of reviewer `19cY`, we have updated our demand sampling procedure to vary both active and reactive power demand components, thus introducing additional variability in power factors (see updated Algorithm 1).
</content>
</thread_object>

</thread>
<thread>
creation date:
1733290188613

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Summary of discussion period

comment:
Dear reviewers and area chairs,

We would like to thank you for your time reviewing our manuscript and for the chance to discuss your feedback.
Please find below a summary of our overall discussion.

## Paper positioning and contribution

Our goal is to broaden access to and improve research on ML and Optimal Power Flow problems, by releasing open standardized datasets and evaluation tools. In particular, we provide
* public datasets that comprise
  * large-scale OPF instances (up to 22,000 buses) with a broad range of total demand
  * multiple OPF formulations (DC, SOC, AC)
  * complete primal and dual solutions and solver metadata such as computing times 
* cross-platform, array-based data formats that allow researchers to use our dataset without requiring deep power systems knowledge
* open-source data-generation and evaluation code for researchers to benchmark their methods in a principled fashion

This addresses several of the most pressing concerns raised in previous work, e.g., the survey by Khaloie et al, as well as some crucial limitations in existing datasets (namely, that uncorrelated load perturbations result in a very narrow range of total demand). Therefore, we believe the paper fits well within the _dataset and benchmark_ ICLR track, and provides meaningful benefits to the community.

## Main points raised by the reviewers

The main points raised by reviewers are the following
* the initial datasets should include larger and smaller instances
* the distribution of OPF instances assumes constant power factor and does not include time series information
* the absence of code in the supplementary material
* the absence of new ML methodology / architecture for learning OPF solutions


## Changes we have made in response to the reviewers' comments

In order to address the reviewers' feedback, we made the following changes:
* we included smaller (down to 14 buses) and larger (up to 22,000 buses) OPF instances
* we updated our data-generation procedure to allow for non-constant power factors

Furthermore, we emphasized the following aspects of our work:
* we pointed out that very few public datasets exist that combine grid topology information and load and generation time series. We agree that capturing temporal structure is an important next step, which we will include in future releases of PGLearn.
* we noted that our paper is part of the _dataset & benchmark_ track, and therefore focuses on data generation and evaluation metrics.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper introduces PGLearn, a toolkit with open-source datasets and evaluation tools for machine learning (ML) in Optimal Power Flow (OPF) problems. The authors aim to solve the problem of inconsistency in datasets and evaluation metrics in this field. The paper has some strengths, like providing datasets with different OPF formulations (AC, DC, and SOC), adding variability in data generation, and offering a framework for standardized evaluation. These tools are useful for the research community to compare different ML methods more easily. The authors also discuss the limitations of current datasets and provide code to help other researchers in this area.
Despite that, there are few key weaknesses. The proposed method mostly uses synthetic data and does not include realistic time-series data or real-world grid dynamics. Reviewers pointed out that more comprehensive data, like from the ENTSO-E Transparency Platform, could make this work stronger. Also, the paper does not offer any new ML methods or technical insights for solving OPF problems, so the contribution is limited to dataset creation. There is little comparison with other benchmarks, and no detailed testing of ML models on the proposed datasets.