Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: In this work, the authors studied applying multi-agent reinforcement learning (MARL) to active voltage control in the power distribution networks, and targeting for bridging the gap between the power community and MARL community. Main contributions include: 
1) develop an open-source environment of active voltage control for MARL and formally define this problem as a
65 Dec-POMDP;
2) conduct large scale experimentation with 8 state-of-the-art MARL algorithms on different scenarios of voltage control problem;
3) found valuable observations, e.g.,  notice the importance of converting voltage constraints to auxiliary reward functions and verify the sensitivity of MARL performances to the auxiliary reward functions, with a new bowl-shape auxiliary reward proposed which stands out in the test results;
4) by analyzing the experimental results, the authors imply the possible difficulties of applying MARL and the further research directions. The authors also conducted the comparison with traditional control methods, and empirically show the rationality and hope to use MARL as a solution for active voltage control.
main_review: This paper is clearly written and well-organized. The main objective is to present/formulate a problem in power networks as an exciting and yet challenging real-world scenario for application of multi-agent reinforcement learning (MARL), developing an open-source environment of active voltage control for MARL, conducting some initial empirical study, and summarizing the main observations (including some key observed challenges/limitations of existing MARL techniques in such problem and some potential future research directions). It mainly targets for bridging the gap between the power community and MARL community in this research topic.

I think the problem is interesting and the paper contains valuable results. Bridging the gap between MARL community and power community is important, and the research work here might potentially motivate more future follow-up research in this direction and would also be helpful for real world large scale application of MARL in power networks.

However, for publication of this result on a top MARL/ML conference like NIPS, I'm still not fully convinced by the novelty and contribution of this study. The main contribution of this this paper is more about formulating the active voltage control problem in power network as MARL problem, conducted some initial study of existing MARL techniques in such problem, and identified some key limitations/difficulties there. If the authors could have some results where they addressed some of these identified limitations and made some improvements (which can be somewhat customized to such power network domain) to the existing MARL techniques, then I'd be more convinced that its contribution is sufficient for publication as a regular paper on such a top tier MARL/ML conference like NIPS.

On the other hand, to be clear, I do think this paper indeed studies an interesting problem and it contains valuable research results that might be good for publication at some good conference/journals, and it will be beneficial for future research, especially for interaction between power community and MARL community. But I just think the contribution and novelty in current manuscript might not be strong enough for publication as a regular paper on such a top tier MARL/ML conference like NIPS.

Update: I've read all the review discussions here and checked out all the detailed responses and clarifications from the authors. I'm slightly more convinced about the contribution and novelty of this paper, and I decided to upgrade my rating from 5 to 6. Though I do agree with several other reviewers that, even after the responses and clarifications from the authors, this paper still seems to be quite at the border line.
limitations_and_societal_impact: Yes
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper proposes a new environment for multi-agent reinforcement learning that addresses the problem of active voltage control on electric power distribution networks. The authors formulate the voltage control problem as a Dec-POMDP, and implement a number of different MARL baselines to assess their performance on the environment. They also assess the effect of different voltage loss functions on this environment.
main_review: I am very much on the borderline regarding this paper. On the one hand, I think it addresses a very important problem, and that it would be extremely beneficial to both the ML/MARL and power systems communities to have this kind of distribution system control environment readily available. On the other hand, I think one of the key assumptions underpinning the current framework needs to be better justified, that there should be more details about the actual usability of the environment, and that some of the technical notations and explanations need to be cleaned up.

Key assumption: One of my main concerns is regarding the assumption made in the paper that multiple inverters are controlled by one company that has full information within its operational region. I do not believe this model is common in present distribution networks -- that is, where smart inverter control exists, it is generally the case that each inverter operates separately based on local information. The authors should provide much more context around this assumption (e.g., how does this grid operation mode relate to existing grid operation modes?), both to avoid inadvertently misleading the ML community about the state of current grid operations, and to better justify the technical assumptions on which the rest of the work is based.

Usage instructions: Since the proposed contribution is an environment with baselines, I was surprised to see no details relating to the practical usability of the environment. For instance, would this environment be integrated within existing MARL frameworks, and/or does it have a user interface that is similar? Is the plan to provide a quickstart guide to help users easily get started? These kinds of details are important to understanding whether this kind of environment would actually be able to be widely adopted, or whether ML researchers/practitioners would find it difficult to use in practice.

Terminology and explanations: As one of the goals of the paper is to make the voltage control setting accessible to ML practitioners, the authors should be careful to go through and make sure all power systems terminology and assumptions are properly defined, e.g.,:
* Line 92: Define "power curtailment"
* Line 127--129: It would be important to clarify why and when the distribution network can be modeled as a tree graph (especially since the 33-bus system evaluated is actually not a tree, though the authors mention in the appendix that they modify it to remove loops).
* Line 140: Define "per unit"

Other questions and comments:
* Lines 41--42: Voltage and power are indeed related, through the power flow equations, which means that the constraint and control action are indeed linked. Would it be possible to clarify what is meant by this statement?
* Line 140: "When the load is heavy during the nighttime" -- Why is load heavier at night, especially during seasons with no HVAC? Or is it just meant that *net* load is heavy due to low availability of PV power at night?
* Line 184--186: The current notation for $\mathcal{R}$ implies that it is a set of sets, rather than a union of sets.
* Line 191: What is meant by "at the last step"?
* Line 205, "State Transition Probability Function": The notation here should be cleaned up, as $\delta$ outputs a state but $Pr(s_{t+1}|s_{t})$ outputs a probability, so it's not immediately clear to me how $\mathcal{T}$ is actually defined.
* Line 339--340: More details should be given on how and why MARL beats droop control methods.

Minor:
* Line 193: "radius" --> "radians"
* Lines 194 and 195: "measures" --> "measurements"
* Line 218: Since "loss" can have multiple meanings in this context, it may be worth clarify that "loss" is used in the ML context here.
limitations_and_societal_impact: Yes, the authors have adequately addressed these points.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper introduces a new open-source environment for training RL agents on controlling the voltage of a power distribution network. The problem is formulated as a multi-agent Dec-POMDP. Agents control PV inverters using only local measurements coming from sensors situated in the region they operate. Their task is to maintain the voltage within safety range. Several MARL algorithms are evaluated in three different power networks. The paper also investigates different reward functions.
main_review: The main contribution of this work is the introduction of a new benchmark domain for testing MARL algorithms. This consists of a simulator of a power network. The environment is built using existent network topologies and real data. I definitely agree with the authors' view on the importance of having realistic scenarios where RL methods can be tested appropriately before using them in real-world applications. It is very unfortunate that the code is not provided in the supplementary material. It is hard to assess how useful this work will be for the community without being able to test the environment.

I miss some intuitive explanation of the dynamics of the problem. Some technical details are given in the Appendix but these are hard to follow for someone lacking in knowledge of the field. In particular, I am looking for details like, how much past information about the network state is useful for making decisions. How do the actions taken in the past affect the network in the long run. I.e. what's the horizon of the problem? All this could be illustrated using some sample trajectories coming from the standard operation of the network. To what extent do the changes in the policies of the agents in one region affect other regions? I.e how strongly coupled are the different regions in the network? This information could shed some light on why some MARL algorithms seem to perform better than others.

It is not clear whether an agent controls one or multiple PV inverters. Why are there multiple agents for each region? As far as I can see, all agents in a region receive the same information. If that’s the case, wouldn’t it be more effective to have all PV inverters within a region be controlled by a single policy? 

The environment is partially observable, yet according to Section 5: MARL Algorithm Settings (line 283-287) the policies are modeled with MLPs receiving only the last two observations as input. Using only the last two observations doesn’t seem sufficient to satisfy the Markov assumption, especially considering the intrinsic non-stationarity of the problem (power generation and consumption fluctuate during the day).

Minor details:

h is first defined as a function that maps nodes to the corresponding subset of measures for that node (line 195) and then also used as the history length (line 228).


limitations_and_societal_impact: N/A
review_title: null

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: The main contribution of the article is mention in the abstract "... establishes an open-source environment. It aims to bridge the gap between the power community and the MARL community". and In section 1: "There is no commonly accepted benchmark to provide the basis for fair comparison of different solutions. To facilitate further research on this topic, and to bridge the gap between the power community and MARL community", The authors seem to forget about L2RPN.
main_review: - The authors say in section 1 "we present a comprehensive test-bench and open-source environment for MARL based active voltage control". The effort for standardization is not good enough because they do not seem to know what measures should be taken to this. If there has to be a standard framework for RL based learning of smart grids then it must include the following specifications
    - Definition of actions and their domains
    - Definition of states and effect of grid-users on the states
    - Definition of environment and its standard response to actions
    - Define standards for randomness and noise in states, observations, actions, and their effects. 
- The algorithms ànd networks need to be fully customizable. 


- Formulation is sketchy. The actions are only dependent on the reactive power, while other researchers have used many different types of actions. Restricting the algorithm to limited actions will definitely affect the results, but the authors do not seem to realize that. In section 5.3 they describe the difference of results between their algorithms and traditional solutions, but they seem to forget that they are not using the complete range of actions here. 

- The authors seem to forget that the network layers matter a lot with respect to the algorithm, see (https://openreview.net/forum?id=LmUJqB1Cz8)

- The authors also seem to have forgotten the effect of replay selection on the results. I conclude this by their choice reward function which they name as bowl function, which could also be a parabolic function too. Probably this is why they have doubts about their statements e.g. line 302-303, and section 5.3.

- The implementation details mentioned in the appendix do not add much to the article. 


limitations_and_societal_impact: They need to be more elaborate about the limitations of the proposed technique/framework, secondly they should clearly mention the limitations of their solution within the proposed framework.
review_title: null

Meta-review: This paper presents an multi-agent RL environment for active voltage control on power distribution networks. This could be a new interesting and practicality-oriented benchmark environment for multi-agent RL research. Most of the reviewers' concerns were (1) lack of access to the implementation of the simulator, which authors explicitly promised to make the code public if the paper gets accepted, and (2) lack of reference to another power network environment L2RPN / GridOp, which is not exactly active voltage control but should have been referenced. AC is recommending borderline accept, and if the paper gets accepted, please address (1) and provide detailed comparison to L2RPN / GridOp in the related work section, so that readers understand what exact power network problem the paper is tackling.
Decision: Accept (Poster)