Multi-Agent Reinforcement Learning for Active Voltage Control on Power Distribution Networks
Authors: Jianhong Wang
Wangkun Xu
Yunjie Gu
Wenbin Song
Tim C Green
Venue: NeurIPS 2021 Poster (2021)
Keywords: multi agent reinforcement learning
power distribution network
active voltage control

Abstract: This paper presents a problem in power networks that creates an exciting and yet challenging real-world scenario for application of multi-agent reinforcement learning (MARL). The emerging trend of decarbonisation is placing excessive stress on power distribution networks. Active voltage control is seen as a promising solution to relieve power congestion and improve voltage quality without extra hardware investment, taking advantage of the controllable apparatuses in the network, such as roof-top photovoltaics (PVs) and static var compensators (SVCs). These controllable apparatuses appear in a vast number and are distributed in a wide geographic area, making MARL a natural candidate. This paper formulates the active voltage control problem in the framework of Dec-POMDP and establishes an open-source environment. It aims to bridge the gap between the power community and the MARL community and be a drive force towards real-world applications of MARL algorithms. Finally, we analyse the special characteristics of the active voltage control problems that cause challenges (e.g. interpretability) for state-of-the-art MARL approaches, and summarise the potential directions.

Multi-Agent Reinforcement Learning for Active Voltage Control on Power Distribution Networks
Jianhong Wang"
Wangkun Xu" Imperial College London wangkun www000mperi111....m.m ac.
Imperial College London
j jiahhogg. wang160imperial. ac.
Yunjie Gu
Wenbin Song Shanghaitech University
Tim C. Green
University of Bath
Imperial College London t. greennimperial ac
yg9340bath. ac. uk
s songabeshanghaitecch edu. en
Abstract
This paper presents a problem in power networks that creates an exciting and yet challenging real-world scenario for application of multi- multi-agent reinforcement learn- ing (MARL). The emerging trend of decarbonisation is placing excessive stress on power distribution networks. Active voltage control is to relieve power congestion and improve voltage quality without extra hardware investment, taking advantage of the controllable apparatuses in the network, such as roof-top photovoltaics (PVs) and static var compensators (SVCs). These control- lable apparatuses appear in vast number and are distributed wide geographic area, making MARL a natural candidate. This paper formulates the active voltage control problem in the framework of Dec-POMDP and establishes an open-source environment. It aims to bridge the gap between the power community and the MARL community and be drive force towards real-world applications of MARL algorithms. Finally, we analyse the special characteristics of the active voltage control problems that cause challenges (e-g. interpretability) for state-of-the-ard MARL approaches, and summarise the potential directions.
promising solution
1 Introduction
Multi-agent reinforcement learning (MARL) has demonstrated impressive performances on games (e.g., Chess M1. StarCraft 3), and etc.) and robotics [4. There are now increasing interests and thrusts apply MARL in real-world problems. Power network is a natural test field for MARL algorithms. There are many problems in power networks involving the collaborative or competitive actions of a vast number of agents, such as market bidding [5]. voltage control, frequency control, and emergency handling (6) One of the obstacles for applying MARL in power networks is the lack of transparency and guarantee, which unacceptable considering the importance of reliable power supply for the whole society. Therefore, it is sensible to start with problems that are less sensitive to reliability but hard to be solved by conventional methods.
Active voltage control in power distribution networks is such a candidate problem. The voltage control problem has been studied for years but only comes under spot light recently due to the increasing penetration of distributed resources, e.g. roof-top photovoltaics (PVs) [7]. The excessive active power injection may cause voltage fluctuations beyond the threshold of grid standards [ The voltage fluctuations can be relieved by exploiting the control flexibility of PV inverters themselves
'Equal contributions. I *orresponndecce to Yunjie Gu who is also an honorary lecturer at Imperial College London. Tim C. Green is also the di-direttor of the Energy Futures Laboratory (EFL).
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

along with other controllable apparatuses, such as static var compensators (SVCs) and on load tap changers (OLTCs). An elaborate scheme is needed to coordinate these multiple apparatuses at scale to regulate voltage throughout the network with limited local information, which is called active voltage control [- The active voltage control problem has many interesting properties. (1) is combination of local and global problem, i.e., the voltage of each node is influenced by the powers (real and reactive) of all other nodes but the impact recedes with increasing distance between nodes. (2) It is a constrained optimisation problem where the constraint is the voltage threshold and the objective is the total power loss, but there is no explicit relationship between the constraint and the control action. (3) A distribution network has radial topology involving rich structure that can be taken as prior knowledge for control. but the node-branch parameters of the topology may be significantly uncertain.
(4) Voltage control has relatively large tolerance and less severe consequences if the control fails to meet standard requirements.
There have been several attempts to apply MARL in active voltage control 6 13 Each work on particular case showed promising performances of state-of-the-art MARL approaches with modifications adapting to the active voltage control problem. However, it is not clear if these methods scale well to a larger network, and the robustness against different scenarios, such as penetration levels and load profiles, are yet to be investigated. There is no commonly accepted benchmark to provide the basis for fair comparison of different solutions.
To facilitate further research on this topic, and to bridge the gap between the power community and MARL community, we present comprehensive test-bench and open-source environment for MARL based active voltage control. We formally define the active voltage control problem as a Dec-POMDP 17 that widely acknowledged in the MARL community. We present an environment in Python and construct 3 scenarios with real public data that span from small scale (i.e. 6 agents) to large scale (i.e. 38 agents).
The contributions of this paper are summarised as follows: (1) We formally define the active voltage control problem as a Dec-POMDP and develop an open-source environment (2) We conduct large scale experimentation with 7 state-of- of-te-of-the-at MARL algorithms on different scenarios of the active voltage control problem. (3) We convert voltage constraints to barrier functions and observe the importance of designing an appropriate voltage barrier function from the experimental results. (4) By analysing the experimental results, we imply the possible challenges (e.g. interpretability) for MARL. to solve the active voltage control problem and suggest potential directions for future works.
2 Related Work
Traditional Methods for Active Voltage Control. Voltage rising and fluctuation problem in distribution networks has been studied for 20 years [88. The traditional voltage regulation devices such as OLTC and capacitor banks [10] are often installed at substations and therefore may not be effective in regulating voltages at the far end of the line 19). The emergence of distributed generation, such as root-top PVs, introduces new approaches for voltage regulation by the active reactive power control of grid-connected inverters (II.. The state-of-the-an active voltage control strategies can be roughly classified into two categories: (1) reactive power dispatch based on optimal power flow (OPF) 22 211: and (2) droop control based on local voltage and power measurements 2 23)
Specifically, centralised OPF [2 26) minimises the power loss while fulfilling voltage constraints (e.g. power flow equation defined in Eq I distributed OPF 27 28 used distributed optimization techniques, such as alternating direction method of multipliers (ADMM), to replace the centralised solver. The primary limitation of OPF is the need of exact system model [2]]. Besides, solving constrained optimisation problem is time-consuming. so is difficult to react to the rapid change of
load profile [30]. On the other hand, droop control only depends performance relies on the manually-designe global information [30]. It possible to enhance droop control by distributed algorithms, but extra communications are needed [9]. In this paper we investigate the possibility of applying MARL techniques on the active voltage control problem. Compared with the previous works on traditional methods, (1) MARL is model-free, so no exact system model is needed; and (2) the response of MARL is fast so as to handle rapid changes of environments (e.g. the intermittency of renewable energy).
its local measurements, but its parameters and is often sub-optimal due to the lack of
2

---

Aulti-Agent Reinforcement Learning for Active Voltage Control. In this section, we discuss the works that applied MARL active voltage control problem in the power system community. [4 [6 applied MADDPG with the reactive power of inverters or static var compensators (SVCs) as control actions.
[22 applied MADDPG with manually designed voltage inner loop, so that agents set reference voltage instead of reactive power as their control actions. [1] applied MATD3 also with reactive power as control actions. (15] applied MASAC, where both reactive power and the curtailment of active power are used control actions. In the above works, distribution networks are divided into regions, with each region controlled by a single agent [33]. It is not clear these MARL approaches scales well for increasing number of agents. In particular, it is not clear if each single inverter a distribution network can behave as an independent agent. this work, we model the active voltage control problem Dec-POMDP [77, where each inverter is controlled by an agent. We propose Bowl-shape as a barrier function to represent voltage constraint as part of the reward. We build up open-source environment for this specific problem that can be easily deployed with the existing MARL algorithms.
Concurrent Works. L2RPN 34 is environment mainly for the centralised control over power transmission networks with node-splittnng and lines switch-off for topology management. Gym-ANM 35 is an environment for centralised power distribution network management. Compared with these 2 works, our environment mainly focuses on solving the active voltage control problem in power distribution networks, with the decentralised/ddisrrbbte manner. Moreover, provide more complicated scenarios with the real-world data. Finally, our environment can be easily and flexibly extended with more network topologies and data, thanks to the supports of PandaPower [36] and SimBench 37
Background
3.1 Power Distribution Network
Equivalent Elettric Circuit
Pi-01
Genoraation Transmooion
1%
Local Energy Power System
Distributkin
Figure 1: Illustration on distribution network (block a-b-c) under PV penetration. The solid and dotted lines represent the power and information flows respectively. Block d is the detailed version of distribution network and block is the circuit model of block d.
An electric power distribution network is illustrated in Figure stage a to c. The electricity is generated from power plant and transmitted through transmission lines. Muti-stage transformers are applied to reduce the voltage levels while the electricity is being delivered to the distribution network. The electricity is then consumed by residential and industrial clients. typical PV unit consists of PV panels and voltage-source inverters which can be installed either on roof-top or in the solar farm. Conventionally. there exist management entities such as distributed system operator (DSO) monitoring and operating the PV resources through the local communication channels. With emergent PV penetration, distribution network gradually grows to be an active participant in power networks that can deliver power and service to its users and the main grid (see the bidirectional power flows in Figure stage-d).
System Model and Voltage Deviation. In this paper, we consider medium (10-24kV) and low (k23)1kV) voltage distribution networks where PVs are highly penetrated. We model the distribution network in Figure [ as tree graph G = (V. E), where V = (0, 1, 1..... .N. and E 2........ N} represent the set of nodes (buses) and edges (branches) respectively 224. Bus the connection to the main grid, balancing the active and reactive power in the distribution network. For each bus i E V. let is and 0 be the magnitude and phase angle of the complex voltage and s, P. + jq: be the complex power injection. Then the active and reactive power injection can be
considered as
3

---

defined as follows:
p." 12 1/2
Σ
Pij
V/ (gy cos 0zj + biz sin 0iy);
Vi E V \ (0)
JEV:
JEV
(I)
q. o//
0
bij + D2
Σ EEV.
U/ (gig sin Biy + big cos Cos
AA V {0)
%EV.
= {il (i.j) E E} is the index set of buses connected to bus i. By and by are the conductance and susceptance on branch (i.j). Ogg = 8 Bg is the phase difference between bus i and j. pf and 9f" are the active power and reactive power of the PV on the bus i (that are zeros if there is no PV on the bus i). pf and qe are the active power and reactive power of the loads on the bus i (that are zeros if there is no loads on the bus i). Eq[T can represent the power system dynamics which is essential for solving the power flow problem and active voltage control problem [8] (details in Appendix A. A.1-A.2). For safe and optimal operation, 5% voltage deviation is usually allowed, i.e., vo = 1.0 per unit (p.u.) and 0.95 p.u. S 0; < 1.05 p.u., Vi E \ {0}. When the load is heavy during the nighttime, the end-user voltage could be smaller than 0.95 p.u. [39]. In contrast, to export its power, large penetration of pf" leads to reverse current flow that would increase v out of the nominal range (Figure d [1].
where ==
Optimal Power Flow. In this paper, OPF is regarded as an optimization problem minimizing the total power loss subject to the power balance constraints defined in [q PV reactive power limits, and bus voltage limits [1]. As the centralized OPF has full access the system topology, measurements, and PV resources, it provides the optimal active voltage control performance and can be used benchmark method (details in Appendix A.3). However, the performance of the OPF depends on the accuracy of the grid model and the optimisation is time-consuming which makes it difficult to be deployed online.
Droop Control. To regulate local voltage deviation, the standard droop control defines piece-wise linear relationship between PV reactive power generation and voltage deviation at bus equipped with inverter-based PVs [30 [2] (details in Appendix A.4). It is fully decentralised control and ignore both the total voltage divisions and the power loss.
3.2 Dec-POMDP
Multi Multi-agent reinforcement learning (MARL) an extension from the reinforcement learning problem, with multiple agents in the same environment. The problem of MARL for cooperation among agents conventionally formulated as Dec-POMDP [17]. It is usually formulated as tuple such that (I. O, T, S2, p, 7). ) I is an agent set; S is a state set; A X.ezA; is the joint action set, where Ar is each agent's s action set; O = 2,
the joint observation set, where O2 is each agent's 's observation set; T S x A x S - [0, 1] is transition probability function that describes the dynamics of an environment; r S x A + IR is a global reward function that describes the award to the whole agents given their decisions; Ω S x A x O + [0, 1] describes the perturbation of the observers (or sensors) for agents' joint observations over the states after decisions; p S + [o, 1] is a probability function of initial states; and y (0, 1) is discount factor. The objective of Dec-POMDP is finding an optimal joint policy æ T that solves max, E, E >rr.].
4 Distributed Active Voltage Control Problem
4.1 Problem Formulation
For the ease of operations, large-scale power network divided into multiple regions and there are several PVs installed into each region that is managed by the responsible distribution network owner. Each PV is with an inverter that generates reactive power control the voltage around a stationary value denoted as Uar In our problem, we assume that the PV ownership (e.g. individual homeowners or enterprise) is independent and separated from the distribution network ownership (DNO) [33] Specifically, a PV owner is responsible for operating a PV infrastructure. In other words, each PV able to be controlled under a distributed manner so that it is natural to be considered as an agent. To enforce the safety of distribution power networks, all agents (i.e. PVs) within region share the

---

observation of this region Since each agent can only observe partial information of the whole gird and maintaining the safety of the power network is a common goal among agents, it reasonable to model the problem as a Dec-POMDD [17] that can be mathematically described as 10-tuple such that SI,S, A. R, O, Trr Q2 P. 7), where p is the probability distribution for drawing the initial state and y is the discount factor.
Agent Set. There is set of agents controlling set of PV inverters denoted as Z. Each agent is located at some node in G (i.e. a graph representing the power network defined as before). We define a function g g:
V to indicate the node where an agent located.
Region Set. The whole power network is separated into M regions, whose union is denoted as R {Rx C V k- < M, k N), where UR,ER Rx S V and RR, n Rx₂ = 9 if 1 # k2- We define a function f V R that maps node to the region where is involved.
State and Observation Set. The state set is defined S C x P x Q x V, where = ((p' q²) p", -" E (0, " a set of active powers generated by PVs; Q = {q : - E (0. 20)
} is a set of (active and reactive) powers of loads; P = {p" pp" E (0, x0 x0){
is set of reactive powers (0, 2o) (MM A [-,, ||| is set of voltage wherein v is vector of voltage magnitudes and is a vector of voltage phases measured in radius. Us pf. of, Pf"V and qf" are denoted as the components of the vectors V. pd, q pPY and qa respectively. We define a function h : P(V) - P(S) that maps subset of V to its correlated measures, where IP(.) denotes the power set. The observation set is defined as O = X O; where O,= (ho fo (i) indicates the measures within the region where agent is located.
generated by PV inverters at the preceding step: V {(v, 0)
Action Set. Each agent i E I is equipped with a continuous action set A, = {a; : -c S S C. 0}. The continuous action represents the ratio of maximum reactive power it generates, i.e., the reactive power generated from the kth PV inverter is of aş v(sg (pf")², where sy is the maximum apparent power of the ktt node that is dependent on the physical capacity of the PV inverter
' og > O, it means penetrating reactive powers to the distribution network. If ag < O, means absorbing reactive powers from the distribution network. The value of e is usually selected as per the loading capacity of a distribution network, which is for the safety of operations. The joint action set is denoted as A X AizrAi
State Transition Probability Function. Since the state includes the last action and the change of loads is random (that theoretically can be modelled as any probabilistic distribution), we can naturally define the state transition probability function as T S x A x S + [0. 1] that follows Markov decision process. Specifically. [st+ a,) = Pr(s 10 at), where a E A and sg, S1+1 E S. ö(, a) + St+r denotes the solution of the power flow, whereas Pr(s s+++>) describes the change of loads (i.e. highly correlated to the user behaviours). < At is an extremely short interval much less than the time interval between two controls (i.e. a time step) and Àt in this paper.
Observation Probability Function. We now define the observation probability function. In the context of electric power network, it describes the measurement errors that may occur in sensors. Mathematically, we can define it as Ω : S x A x O + [0, 1]. Specifically, 2(0 184- az) = S + (0, Σ), where (0, Σ) is isotropic multi-variable Gaussian distribution and Σ is dependent on the physical properties of sensors (e.g. smart meters).
Reward Function. The reward function is defined as follows:
1
M Σ (vx) - a aa1(4""))
r
where (u)- is voltage barrier function and (,(4")) = bेे ||q the reactive power generation loss (i.e. a type of power loss approximation easy for computation). The objective is to control the voltage within a safety range around Gers while the reactive power generation is as less as possible, i.e.,
*Sharing observation in this problem is reasonable, since only the sensor measurements (e.g. voltage, active power, etc.) are shared, which not directly related the commercial profits [3] The observation of each PV is collected by the distribution network owner and then the full information within the region is sent to each agent.
'Note that the reactive power range actually dynamically changes at each time step.
'Yielding (9 (V) ) at each time step t is equivalent to yielding A Ae(gf") ) (i.e. the change of reactive power
PV ). For easily satisfying the safety condition, generation at each time step). since (g." )2 = (gh PV 1 + Ar(gh
we directly yield "h
eV each time step in this work.
5

---

(4("") < é and > 0. Similar to the mathematical tricks used in 8-VAE [3], by KKT conditions we can transform constrained reward to unconstrained reward with Lagrangian multiplier E (0, 1) shown in Eq Since (-) is not easy to define in practice (i.e., it affects (g(q")) )), we aim to study for good choice in this paper.
Objective Function. The objective function of this problem is max, Ex DE 7 rrr,, where X Tei n. O2 x A; + [0, 1] and O, = (OT) is a history of observations with the length h. Literally, we need to find an optimal joint policy maximize the discounted cumulative rewards.
4.2
Voltage Barrier Function
0.30
0.30
0.08
0.00
Renalty
Penalty
Penalty
0.00
0.00
0.22
0.00
135
1.50 1055
1.0
632
0.55
1.05
635
1:00
125
Lio
Voltage (p.u.)
Voltage (p.u.)
Voltage (p.u.)
(a) L--spape.
(b) 1.2-shape.
(c) Bowl-shape.
Figure 2: This figure shows 3 voltage barrier functions, where LI-shape and L.-shaae are baselines while Bowl-shape is proposed in this paper.
We define Uie = 1 p.u. in this paper, and the voltage needs be controlled within the safety range from 0.95 p.u. 1.05 p.u., which sets the constraint of control. The voltage constraint is difficult to be handled in MARL, so we use a barrier function to represent the constraint. LI-shape (see Figure 2a was most frequently used in the previous work 1345,, however, this may lead to wastefull
reactive power generations since |At.l > 1 within the safety range of voltage. Although L2-shape Ar, (see Figure 2b) may alleviate this problem, may be slow to guide the policy outside the safety range. To address these problems, propose barrier function called Bowl-shape that combines the advantages LI-shape and L2-shape. gives steep gradient outside the safety range, while provides a slighter gradient as voltage tends to the that enables
Adel + 0 as ->
5
Experiments
5.1
Experimental Settings
Power Network Topology. Two MV networks, IEEE 33-bus [4 and 141-bus 445 are modified as systems under test 2 To show the fiexibility on network with multi-voltage levels, we construct 1 110KV-20VV 20-00k--0044V (high-medium-low voltage) 322-bus network using benchmark topology from SimBench 37 For each network, main branch firstly determined and the control regions partitioned by the shortest path between the terminal bus and the coupling point on the main branch. Each region consists of 1-4 PVs dependent on various regional sizes. The specific network descriptions and partitions are shown in Appendix D.I. To give picture of the tasks, we demonstrate the 33-bus network in Figure 3
Data Descriptions. The load profile of each network is modified based on the real-time Portuguese electricity consumption accounting for 232 consumers of 3 years ? To highlight the differences between residential and industrial users, we randomly perturb +5% on the default power factors defined in the case files and accordingly generate real-time reactive power consumption. The solar data is collected from Elia group then interpolated with 3-min resolution that consistent with the real-time control period in the grid. To distinguish among different solar radiation levels in various regions, the 3-year PV generations
R j.e. a Belgiums power network operator. The load and PV data are
'The original topologies and parameters can be found in MATPOWER 46 description file such as https //github. wwwp:hhiiiie...............

---

ZONE
27
ZONE
ZONE
ZONE.2
19
20
21
22
10
13
15
16
17
18
Illustration on 33-bus network. Each bus is indexed by circle with number. control regions are partitioned by the smallest path from the terminal to the main branch (bus 1-6). We control the voltages on bus 2-33 whereas bus 0-1 represent the substation or main grid with the constant voltage and infinite active and reactive power capacity. G represents an external generator; small Ls represent loads; and the sun emoji represents the location where PV is installed.
Figure 3:
from 10 cites/regions are collected and PVs in the same control region possess the same generation profiles. We define the PV penetration rate (PR) as the ratio between rated PV generation and rated load consumption. In this paper, we set PR E {2.5, 4, 2.5} as the default PR for different topologies. We oversize each PV inverter by 20% of its maximum active power generation to satisfy the IEEE grid code [2]. Besides, each PV inverter is considered to be able to generate reactive power in the STATCOM mode during night 47 The median and 25% 75%% quantile shadings of PV generations, and the mean and minima- manimma shading of the loads are illustrated in Figure The details can be found in Appendix D.2.
Botto Powee (M9N)
PV Generation
Rctive Promeer (H00))
Load Consumption
2.0
2.5
Load
Generation
2.5
12 Hour
14
16
20
22
12
0.0
10
16
Hour
Hour
(a) 33-bus network.
(b) 141-bus network.
(c) 322-bus network.
Figure 4: Active PV generations and load consumption.
MARL Simulation Settings. We now describe the simulation settings standing by the view of MARL.. In 33-bus network, there are regions with 6 agents. In 141-bus network, there are regions with 22 agents. In 322-bus network, there are 22 regions with 38 agents. The discount factor 7 is set to 0.99. in 2 is set to 0.1. To guarantee the safety of distribution networks, we manually set the range of actions for each scenario, with [-0.8, 0.8] for 33-bus network, [-0.6, 0.6] for 141-bus network, and [-0.8, 0.8] for 322-bus network. During training, we randomly sample the initial state for an episode and each episode lasts for 240 time steps (i.e. a half day). Every experiment is run with 5 random seeds and the test results during training are given by the median and the 25%-75% quartile shading. Each test is conducted every 20 episodes with 10 randomly selected episodes for evaluation.
Evaluation Metrics. In experiments, we two metries evaluate the performance of algorithms.
Controllable rate (CR): It calculates the ratio of time steps where all buses' voltages being under control during each episode.
Power loss (PL): It calculates the average of the total power loss over all buses per time step during each episode.
We aim to find algorithms and reward functions with high CR and low PL.
MARL Algorithm Settings. We evaluate the performances of state-of-the-art MARL algorithms, i.e. IDDPG 48), MADDPG [49], COMA (50), IPPO 2), MAPPO B. SQDDPG 48) and MATD3 51 on this real-world problem with continuous actions. Since the original COMA can only work
7

---

for discrete actions, we conduct some modifications so that it can work for continuous actions (see Appendix B). The details of settings are shown in Appendix C.
5.2
Main Results
CCP4
Episcde
Cplecde
Eplcade
Epleace
Esleade
Eslesce
(a) CR-L1-33.
CR-L2-33.
(c) CR-BL-33.
(d) PL-LI-33.
PL.-L2233.
PL-BL-33.
280
100
Eptacda
200
130
Eplaode
200
116 500
CR-L1-141.
(h) CR-12-141.
(i) 14-
i PL-LI-141.
(k) PL-L2-141.
(i) 14-
40122
40/21
N000
L10 253 A30
25 202
230
2.0
11E 230 20 80
(m) CR-LI-322. (n) CR-L2-322.
(0) BL-32,
(p) PL-L1-322.
(q) PL-L2-322.
(n) PL-BL-322,
Figure 5: Median CR and PL of algorithms with different voltage barrier functions. The sub-caption and BL is the contraction of Bowl.
indicates
Algorithm Performances. We first show the main results of all algorithms all scenarios in Figure MADDPG and MATD3 generally perform well on all scenarios with different voltage barier functions. COMA performs well over CR on 33-bus networks and the performances fall on the large scale scenarios, but its PL is high. Similarly, SQDDPG performs generally well over CR on 33-bus and 141 [41-bus networks, but performs the worst on 322-bus networks and its PL on 141-bus networks is high. This reveals the limitations of COMA and SQDDPG on the scaling to many agents. Although MAPPO and IPPO performed well on games 2 their performances on the real-world power network problems are poor. The most probable reason could be that the variations of dynamics and uncertainties distribution networks are far faster and more complicated than games and their conservative policy updates cannot fast respond. IDDPG performs at the middle place in all scenarios, which may be due to non- stationttony dynamics caused by multiple agents [49]
RL
3mml
Erw
3mk
Ep/e0de
1pdecde
Epleade
Eptedde
Ealeace
Ezledde
(a) CR-33.
(b) CRL
(c) 322.
(d) PL-33.
(c) 141
(f
PL-322.
Figure 6: Median performance of overall algorithms with different voltage barrier functions. The sub-caption indicates metric-scenario.
Voltage Barrier Function Comparisons. To study the effects of different voltage barrier functions, we show the median performance of overall 7 MARL algorithms in Figure (6) It can be observed that Bowl-shape can preserve the high CR, while maintain the low PL on 33-bus and 141-bus networks. Although Ll-shape can achieve the best CR on 33-bus and 141 4bus networks, PL on the 141-bus network is the highest. L2-shape performs the worst on 33-bus and 141-bus networks, but performs the best on the 322-bus network with the highest CR and the lowest PI.. The reason could that its slighter gradients is more suitable for the adaption among many agents. From the results, we can
8

---

conclude that L1-shape, shwpeshape and L2-shape are the best choices for the 33-bus network, the 141-bus network and the 322-bus network respectively.
3.04
RAdo
0.00
0.5
3.
16
0.3
Reward
Reward
Reward
Algurahin
Algoranim
Algurttun
(a) CR-33.
(b) CR-141.
(c) CR-322.
33355
-3399
LInss
02
Reward
Reaaad
Reward
Algorihm
Algortim
Algoritim
(d) PL-33.
(c) PL-141.
PL-322.
Figure 7: Median performances of overall algorithms trained with various rewards consist of distinct voltage barrier functions shown in 3D surfaces. The sub-caption indicates metric-scenario.
Diverse Algorithm Performances under Distinct Rewards. To clearly show the relationship between algorithms and reward functions, we also plot 3D surfaces over CR and PL w.r.t. algorithm types and reward types (consisting of distinct voltage barrier functions) in Figure 7 is obvious that the performances of algorithms are highly correlated with the reward types. In other words, the same algorithm could perform diversely even trained by different reward functions with the same objective but different shapes.
5.3 Comparison between MARL and Traditional Control Methods
To compare MARL algorithms with the traditional control methods, we conduct a series of tests on various network topologies (i.e. 33-bus, 141-bus, and 322-bus networks). MADDPG trained by Bowl-shape is selected as the candidate for MARL. The traditional control methods that we select only demonstrate the voltages and powers on a typical bus with a PV installed (i.e. one of the most difficult buses to control) during a day (i.e. 480 consecutive time steps) in summer and winter respectively. The results for the 33-bus network is presented here and the results for other typologies are given the Appendix E.2. From Figure 8 can be seen that all methods control the voltage within the safety range in both summer and winter. Additionally, the power losses of MARL are lower than droop control but higher than OPF. This phenomenon possibly due to the fact that droop control is fully distributed algorithm which cannot explicitly reduce the power loss and OPF is a centralised algorithm with the known system model, while MARL lies between these types algorithms. This implies that MARL may outperform droop control for particular cases but it needs to be kept in mind that the droop gain used here may not be optimal. is also worth noting that the control actions of MARL has similar feature to droop control in the 33-bus case, and this feature is also observed in the 141-bus and 322-bus cases (see Figure 14-15 in the Appendix E.2 for details). However, this droop-like behaviour is missing in the winter case of 322-bus network, where the MARL control action is opposite to the desired direction, resulting in higher power losses and voltage deviation. This might be due to the over-generation as the MARL share the same policy for different seasons, so that it leads to the problem of relative overgeneralisation [52]. In summary, MARI, seems able learn the features of droop control but fails to outperform droop control significantly, and is even worse than droop control for certain cases.
OPF [24] and droop control 222) For conciseness,

---

As a result, there is still significant headroom of improvement for MARL. in performance, robustness, and interpretability.
1.10
1.50
0.40
P
MASE
1.05
MARI OFF
Power (4900400011
2.20
O O-Dmap
1996
0.30
0.50
0.90
17
14
16
18
32 24
1.00
10
13
20
24
0.00
10 13
14
16
38
20
Hou
M1U1T
HOUT
0.05
None
MAIL
1.05
MUIL 0 OFF
0.04
Droup
0.20
Power
0.00
2.00
0.30
0.02
0.50
0.01
-3.10
Hour
HOUT
Hour
(a) Voltage.
(b) Power.
(c) Power loss.
Figure 8: Compare MARL with traditional control methods bus 18 during a day for 33-bus network. 1st row: results for a summer day. 2nd row: results for a winter day. None and limit in (a) represent the voltage with control and the safety voltage range respectively. P and Q in (b) indicate the PV active power and the reactive power by various methods.
5.4 Discussion
We now discuss the phenomenons that we observe from the experimental results.
It is obvious that the voltage barrier function may impact the performance of an algorithm (even with tiny changes for the common goal). This may of general importance as many real-world problems may contain constraints that are not indirect in the objective function. An overall methodology is needed for designing barrier functions in state-constraint MARL.
MARL may scale well for the number of agents and the complexity of networks, and only requires a very low control rate for active voltage control.
The results above show the evidence that MARL may behave diversely from the traditional control methods. Some of the learnt behaviours may induce better performance, but some others deteriorate the performance. This shows the promising benefits of MARL in industrial applications but also highlights the drawbacks in interpretability and robustness.
The combination of learning algorithms with domain knowledge a potential roadmap towards interpretable MARL. For the active voltage control problem, the domain knowledge may present as network topology, inner control loops (say droop control), and load pattern. The exploitation of such domain knowledge reduces the dimensions of MARL exploration space and may offer lower bound of performance as a guarantee. Encoding domain knowledge such as the network topology priori for model-bate based MARL is also potential direction.
6 Conclusion
This paper investigates the potential of applying multi-agent reinforcement learning (MARL) the active voltage control in power distribution networks. We firstly formulate this problem as a Dec- POMDP and then study the behaviours of MARL with various voltage barrier functions (reflecting voltage constraints as barrier penalties). Moreover, we compare the behaviours of MARL with the traditional control methods (í.e. droop control and OPF), and observe that MARL is possible to generate inexplicable behaviours, so an interpretable and trustable algorithm is highly desired for industrial applications. Finally, the environment used in this work open-sourced and easy to follow so that the machine learning community is able to contribute to this challenging real-world problem.
10

---

Acknowledgement
This work is supported by the Engineering and Physical Sciences Research Council of UK (EPSRC) under awards EP//000909//.
References
[1] David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, al. Mastering the game of go without human knowledge. nature, 550(7676):354 359, 2017.
[2] Christian Schroeder de Witt, Tarun Gupta, Denys Makoviichuk, Viktor Makoviychuk, Philip HS Torr, Mingfei Sun, and Shimon Whiteson, Is independent learning all you need in the starcraft multi-agent challenge? arXiv preprint arXiv:201 2020.
[3] Chao Yu, Akash Velu, Eugene Vinitsky, Yu Wang, Alexandre Bayen, and Yi Wu. The surprising effectiveness of mappo in cooperative, multi-agent games. arXiv preprint arXiv:2103.01955 2021.
[4] Christian Schroeder de Witt, Bei Peng, Pierre-Alexandre Kamienny, Philip Torr. Wendelin Böhmer, and Shimon Whiteson. Deep multi-agent reinforcement learning for decentralized continuous cooperative control. arXiv preprint arXiv:2003.06709 2020.
[5] Dawei Qiu, Jianhong Wang, Junkai Wang, and Goran Strbac. Multi agent reinforcement learning for automated peer-to-peer energy trading in double-side auction market. In Zhi-Hua Zhou, editor, Proceedings of the Thirtieth International Joint Conference Artificial Intelligence, IJCAI 2021, Virtual Event Montreal, Canada, 19-27 August 2021, pages 2913 2913-2920. ijeai.org. 2021.
[6] Di Cao, Weihao Hu, Junbo Zhao, Guozhou Zhang, Bin Zhang, Zhou Liu, Zhe Chen, and Frede Blaabjerg. Reinforcement learning and its applications in modern power and energy systems: A review. Journal Modern Power Systems and Clean Energy, 8(6): 2020. doi: ...5833/MPCE.2220000000
[7] Efstratios Batzelis, Zakir Hussain Rather, John Barton, Bonu Ramesh Naidu, Billy Wu, Firdous UI Nazir, ONYEMA SUNDAY NDUKA, Wei He, Jerome Nsengiyaremye, and Bandopant Pawar. Solar integration in the uk and india: technical barriers and future directions. Report of the UK-India Joint Virtual Clean Energy Centre (JVCEC), 2021.
[8] Reinaldo Tonkoski, Dave Turcotte, and Tarek H. M. EL-Fouly. Impact of high pv penetration on voltage profiles in residential neighborhoods. JEEE Transactions on Sustainable Energy, (3):518-527, 2012. doi: 10.1 10.1099/SSE..20222199944
[9] Pedro MS Carvalho, Pedro F Correia, and Luis AFM Ferreira. Distributed reactive power generation control for voltage rise mitigation in distribution networks. IEEE transactions on Power Systems, 23(2):666-77,, 2008
[10] Tomonobu Senjyu, Yoshitaka Miyazato, Atsushi Yona, Naomitsu Urasaki, and Toshihisa Fun- abashi. Optimal distribution voltage control and coordination with distributed generation. IEEE Transactions on Power Delivery, 23(2): 1336):1236-2222, 2008. doi: 10.1109/TWRRD2200 2007. 1908816...1..
[11] Konstantin Turitsyn, Petr Sule, Scott Backhaus, and Michael Chertkov. Options for control of reactive power by distributed photovoltaic generators. Proceedings of the IEEE, 99(6): 1063-1073, 2011. doi: 10.110//PR00.200..11660000
[12] Johanna Barr and Ritwik Majumder. Integration of distributed generation the volt/var management system for active distribution networks. IEEE Transactions Smart Grid, 6(2): 576-586, 2015. doi: 00.1109/TSG.201.226663
[13] Di Cao, Junbo Zhao, Weihao Hu, Fei Ding, Qi Huang, and Zhe Chen. Distributed voltage regulation of active distribution system based on enhanced multi-agent deep reinforcement learning, arXiv preprint 200.:0000000050666 2020.

---

[14] Di Cao, Weihao Hu, Junbo Zhao. Qi Huang, Zhe Chen, and Frede Blaabjerg. A multi-agent deep reinforcement learning based voltage regulation using coordinated pv inverters. IEEE Transactions on Power Systems, (5(5)4110-4125 2020.
[15] Di Cao, Junbo Zhao, Weihao Hu, Fei Ding, Qi Huang, Zhe Chen, and Frede Blaabjerg. Data- driven multi-agent deep reinforcement learning for distribution system decentralized voltage control with high penetration of pvs. IEEE Transactions on Smart Grid, 2021.
[16] Haotian Liu and Wenchuan Wu. Online multi-agent reinforcement learning for decentralized inverter-based volt-var control. IEEE Transactions on Smart Grid, 2021.
[17] Frans A Oliehoek and Christopher Amato. A concise introduction to decentralized POMDPs. Springer, 2016.
[18] C.L. Masters. Voltage rise: the big issue when connecting embedded generation long 11 kv overhead lines. Power Engineering Journal, 16(1):5-12, 2002. doi: 10. 10.10499pe:2002110..
[19) Giuseppe Fusco and Mario Russo. A decentralized approach for voltage control by multiple distributed energy resources. IEEF Transactions on Smart Grid, pages 1-1, 2021, doi: 10.1 109/ TSG.2021.3057544
[20] Yashodhan P. Agalgaonkar, Bikash C. Pal, and Rabih A. Jabr. Distribution voltage control considering the impact of pv generation tap changers and autonomous regulators. IEEE Transactions on Power Systems, 29(1): 2014. doi: 10.110//PWRRS.201322227
[21] Panagis N. Vovos, Aristides E. Kiprakis, A. Robin Wallace, and Gareth P. Harrison. Centralized and distributed voltage control: Impact distributed generation penetration. IEEE Transactions on Power Systems, 22(1):476-483, 2007. doi: 10.1 1..1109/TPRSS2200
[22] Pedram Jahangiri and Dionysios C. Aliprantis. Distributed volt/var control by pv inverters. IEEE Transactions on Power Systems, 28(3):3429-3439 2013. doi: 10.1 10.1109/ WWW.WRS..01..22222 201..225335..
[23] Johannes Schiffer, Thomas Seel, Jörg Raisch, and Tevfik Sezi. Voltage stability and reactive power sharing in inverter-based microgrids with consensus-based distributed voltage control. IEEE Transactions on Control Systems Technology, 24(1):96-109, 2016. doi: 10.1 109/TCTT. 2015.244 20152242
[24] Lingwen Gan, Na Li, Ufuk Topcu, and Steven H Low. Optimal power flow in tree networks. In 52nd IEEE Conference on Decision and Control, pages 2318-2318. IEEE, 2013.
[25] Yan Xu, Zhao Yang Dong, Rui Zhang, and David Hill. Multi-timescale coordinated voltage/var control of high hwwpwaab--penetrtte.. distribution systems. IEEE Transactions on Power Systems, 32(6):4398-4408 2017.
[26] Emiliano DallAnese, Sairaj V. Dhople, and Georgios B. Giannakis. Optimal dispatch of photovoltaic inverters in residential distribution systems. IEEE Transactions Sustainable Energy, 5(2):487-477, 2014. doi: 10. 109/TSTTF.201332229922.
[27] Weiye Zheng, Wenchuan Wu, Boming Zhang, Hongbin Sun, and Yibing Liu. fully dis- tributed reactive power optimization and control method for active distribution networks. IEEE Transactions on Smart Grid, 72)):1021-103, 2016. doi: 2...110//SG..01..23000.
[28) Zhiyuan Tang, David J Hill, and Tao Liu. Distributed coordinated reactive power control for voltage regulation in distribution networks. IEEE Transactions on Smart Grid, 12(1):312-323, 2020.
[29] Hongbin Sun, Qinglai Guo, Junjian Qi, Venkataramana Ajjarapu, Richard Bravo, Joe Chow, Zhengshuo Li, Rohit Moghe, Ehsan Nasr-Azadami Ujjwol Tamrakar, Glauco N. Taranto, Reinaldo Tonkoski, Gustavo Valverde, Qiuwei Wu, and Guangya Yang. Review of challenges and research opportunities for voltage control in smart grids. IEEE Transactions on Power Systems, (4(4))2790-280) 2019. doi: 10. (t1.////SS.201..299999
[30] Ankit Singhal, Venkataramana Ajjarapu, Jason Fuller, and Jacob Hansen. Real-time local volt/var control under external disturbances with high pv penetration. IEFE Transactions on Smart Grid, 10(4):3849-388 3859, 2019. doi: 10.1109/TSG..218.2225
12

---

[31] Mehdi Zeraati, Mohamad Esmail Hamedani Golshan, and Josep M Guerrero. Voltage quality improvement in low voltage distribution networks using reactive power capability of single- phase pv inverters. IEEF Iransactions on smart grid, 1065):5057-5065, 2018.
[32] Shengyi Wang, Jiajun Duan, Di Shi, Chunlei Xu, Haifeng Li, Ruisheng Diao, and Zhiwei Wang. data-driven multi-agent autonomous voltage control framework using deep reinforcement learning. IEEE Transactions Power Systems, 35(6):46444463 4654, 2020.
[33] Scott Burger, Jesse Jenkins, Carlos Batlle López, and José Ignacio Pérez Arriaga, Restructur- ing revisited: competition and coordination in electricity distribution systems. 2018.
[34] Marvin Lerousseau. Design and implementation of an environment for learning to a power network (12rpn). arXiv preprint arXiv:2
2021.
[35] Robin Henry and Damien Ernst. Gym-anm: Reinforcement learning environments for active network management tasks in electricity distribution systems. Energy and AI, page 100092, 2021.
[36] Leon Thurner, Alexander Scheidler, Florian Schäfer, Jan-Hendrik Menke, Julian Dollichon, Friederike Meier, Steffen Meinecke, and Martin Braun. Pandapoweran open-source python tool for convenient modeling, analysis, and optimization of electric power systems. IEEE Transactions on Power Systems, 33(6):6510-6522 2018. doi: 10.1 2..1009/PWRS.201882288
[37] Steffen Meinecke, Danan Sarajli, Simon Ruben Drauz, Annika Klettke, Lars-Peter Lauven, Christian Rehtanz, Albert Moser, and Martin Braun. Simbencha benchmark dataset of electric power systems to compare innovative solutions based power flow analysis. Energies, 13(12), 2020. ISSN 1996-1073. URL htttps://wwtt
[38] Hadi Saadat et al. Power system analysis, volume 2. McGraw-hill, 1999.
[39] Yashodhan P Agalgaonkar, Bikash C Pal, and Rabih A Jabr. Distribution voltage control considering the impact of pv generation on tap changers and autonomous regulators. IEEE Transactions on Power Systems, 29(1):182-192, 2013.
[40] Guangya Yang, Francesco Marra, Miguel Juamperez. Søren Bækhho Kjær, Seyedmostafa Hashemi, Jacob Østergaard, Hans Henrik Ipsen, and Kenn H. B. Frederiksen. Voltage rise mitigation for solar pv integration Iv grids studies from pvnet. dk. Journal of Modern Power Systems and Clean Energy, 3(3):111 2015. doi: 10. 00.10077/40555-01500022..
[41] Antonio Simecz-Epósitoo, Antonio J Conejo, and Claudio Cañizares. Electric energy systems: analysis and operation. CRC press, 2018
[42] Ieee standard for interconnection and interoperability of distributed energy resources with associated electric power systems interfaces. IEEE Std 1547-2018 (Revision of IEEE Std 1547-2003), pages 1-138, 2018. doi: 10.1109/EEESTD 2018.223111.8.1.
[43] Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. beta-vae: Learning basic visual concepts with a constrained variational framework. 2016.
[44] M.E. Baran and F.F. Wu. Network reconfiguration in distribution systems for loss reduction and load balancing. IEEE Transactions on Power Delivery, 4(2):1401-1407, 1989. doi: 10.1 1090/961.... 226277.
[45] H.M. Khodr, F.G. Olsina, P.M. De Oliveira-De Jesus, and J.M. Yusta. Maximum savings approach for location and sizing of capacitors in distribution systems. Electric Power Sys- tems Research, 78(7):1 192-1 2008 ISSN 0378-7796. doi: https///doi.or//00.uu 1016/ kepss:2007.10.002 URL. $03787796070021443
[46] Ray Daniel Zimmerman, Carlos Edmundo Murillo-Sánchez, and Robert John Thomas. Mat- power: Steady-state operations, planning, and analysis tools for power systems research and education. IEEE Transactions power systems, 26(1):12-19, 2010.
13

---

[47] Rajiv K. Varma and Ehsan M. Siavashi. Pv-statcom: A new smart inverter for voltage control in distribution systems. IEEE Transactions on Sustainable Energy, -14):1681-169,, 2018, doi: 2..11099TSTE.211.88888
[48] Jianhong Wang, Yuan Zhang, Tae-Kyun Kim, and Yunjie Gu. Shapley q-value: A local reward approach to solve global reward games. Proceedings of the AAAI Conference on Artificial Intelligence, 34(05):72857292, Apr 2020.
[49] Ryan Lowe, Yi Wu, Aviv Tamar, Jean Harb, Pieter Abbeel, and Igor Mordatch. Multi-agent actor-critic for mixed cooperaive-coompettive environments. In Isabelle Guyon, Ulrike von Luxburg. Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett, editors, Advances in Neural Information Processing Systems 30; Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pages 2379-6390, 2017.
[50] Jakob N Foerster, Gregory Farquhar, Triantafyllos Afouras, Nantas Nardelli, and Shimon Whiteson. Counterfactual multi-agent policy gradients. In Thirty-Second AAAI Conference on Artificial Intelligence, 2018.
[51] Johannes Ackermann, Volker Gabler, Takayuki Osa, and Masashi Sugiyama. Reducing overestimation bias in multi-agent domains using double centralized critics. arXiv preprint arXiv:1910001465, 2019.
[52] Ermo Wei and Sean Luke. Lenient learning in independent-learmer stochastic cooperative games. The Journal of Machine Learning Research, 17(1):2914-22 14-2955, 2016.
14


Reviews:
[{"review_id": "o2PZ6q9HrMP", "reviewer": "Reviewer_YRiZ", "summary": "In this work, the authors studied applying multi-agent reinforcement learning (MARL) to active voltage control in the power distribution networks, and targeting for bridging the gap between the power community and MARL community. Main contributions include: \n1) develop an open-source environment of active voltage control for MARL and formally define this problem as a\n65 Dec-POMDP;\n2) conduct large scale experimentation with 8 state-of-the-art MARL algorithms on different scenarios of voltage control problem;\n3) found valuable observations, e.g.,  notice the importance of converting voltage constraints to auxiliary reward functions and verify the sensitivity of MARL performances to the auxiliary reward functions, with a new bowl-shape auxiliary reward proposed which stands out in the test results;\n4) by analyzing the experimental results, the authors imply the possible difficulties of applying MARL and the further research directions. The authors also conducted the comparison with traditional control methods, and empirically show the rationality and hope to use MARL as a solution for active voltage control.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper is clearly written and well-organized. The main objective is to present/formulate a problem in power networks as an exciting and yet challenging real-world scenario for application of multi-agent reinforcement learning (MARL), developing an open-source environment of active voltage control for MARL, conducting some initial empirical study, and summarizing the main observations (including some key observed challenges/limitations of existing MARL techniques in such problem and some potential future research directions). It mainly targets for bridging the gap between the power community and MARL community in this research topic.\n\nI think the problem is interesting and the paper contains valuable results. Bridging the gap between MARL community and power community is important, and the research work here might potentially motivate more future follow-up research in this direction and would also be helpful for real world large scale application of MARL in power networks.\n\nHowever, for publication of this result on a top MARL/ML conference like NIPS, I'm still not fully convinced by the novelty and contribution of this study. The main contribution of this this paper is more about formulating the active voltage control problem in power network as MARL problem, conducted some initial study of existing MARL techniques in such problem, and identified some key limitations/difficulties there. If the authors could have some results where they addressed some of these identified limitations and made some improvements (which can be somewhat customized to such power network domain) to the existing MARL techniques, then I'd be more convinced that its contribution is sufficient for publication as a regular paper on such a top tier MARL/ML conference like NIPS.\n\nOn the other hand, to be clear, I do think this paper indeed studies an interesting problem and it contains valuable research results that might be good for publication at some good conference/journals, and it will be beneficial for future research, especially for interaction between power community and MARL community. But I just think the contribution and novelty in current manuscript might not be strong enough for publication as a regular paper on such a top tier MARL/ML conference like NIPS.\n\nUpdate: I've read all the review discussions here and checked out all the detailed responses and clarifications from the authors. I'm slightly more convinced about the contribution and novelty of this paper, and I decided to upgrade my rating from 5 to 6. Though I do agree with several other reviewers that, even after the responses and clarifications from the authors, this paper still seems to be quite at the border line.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes", "review_text": "", "extra_scores": {"ethical_concerns": "None", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2.5 hr"}, "extra_text": {}}, {"review_id": "Jg19OB6X2ZK", "reviewer": "Reviewer_S8R6", "summary": "This paper proposes a new environment for multi-agent reinforcement learning that addresses the problem of active voltage control on electric power distribution networks. The authors formulate the voltage control problem as a Dec-POMDP, and implement a number of different MARL baselines to assess their performance on the environment. They also assess the effect of different voltage loss functions on this environment.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I am very much on the borderline regarding this paper. On the one hand, I think it addresses a very important problem, and that it would be extremely beneficial to both the ML/MARL and power systems communities to have this kind of distribution system control environment readily available. On the other hand, I think one of the key assumptions underpinning the current framework needs to be better justified, that there should be more details about the actual usability of the environment, and that some of the technical notations and explanations need to be cleaned up.\n\nKey assumption: One of my main concerns is regarding the assumption made in the paper that multiple inverters are controlled by one company that has full information within its operational region. I do not believe this model is common in present distribution networks -- that is, where smart inverter control exists, it is generally the case that each inverter operates separately based on local information. The authors should provide much more context around this assumption (e.g., how does this grid operation mode relate to existing grid operation modes?), both to avoid inadvertently misleading the ML community about the state of current grid operations, and to better justify the technical assumptions on which the rest of the work is based.\n\nUsage instructions: Since the proposed contribution is an environment with baselines, I was surprised to see no details relating to the practical usability of the environment. For instance, would this environment be integrated within existing MARL frameworks, and/or does it have a user interface that is similar? Is the plan to provide a quickstart guide to help users easily get started? These kinds of details are important to understanding whether this kind of environment would actually be able to be widely adopted, or whether ML researchers/practitioners would find it difficult to use in practice.\n\nTerminology and explanations: As one of the goals of the paper is to make the voltage control setting accessible to ML practitioners, the authors should be careful to go through and make sure all power systems terminology and assumptions are properly defined, e.g.,:\n* Line 92: Define \"power curtailment\"\n* Line 127--129: It would be important to clarify why and when the distribution network can be modeled as a tree graph (especially since the 33-bus system evaluated is actually not a tree, though the authors mention in the appendix that they modify it to remove loops).\n* Line 140: Define \"per unit\"\n\nOther questions and comments:\n* Lines 41--42: Voltage and power are indeed related, through the power flow equations, which means that the constraint and control action are indeed linked. Would it be possible to clarify what is meant by this statement?\n* Line 140: \"When the load is heavy during the nighttime\" -- Why is load heavier at night, especially during seasons with no HVAC? Or is it just meant that *net* load is heavy due to low availability of PV power at night?\n* Line 184--186: The current notation for $\\mathcal{R}$ implies that it is a set of sets, rather than a union of sets.\n* Line 191: What is meant by \"at the last step\"?\n* Line 205, \"State Transition Probability Function\": The notation here should be cleaned up, as $\\delta$ outputs a state but $Pr(s_{t+1}|s_{t})$ outputs a probability, so it's not immediately clear to me how $\\mathcal{T}$ is actually defined.\n* Line 339--340: More details should be given on how and why MARL beats droop control methods.\n\nMinor:\n* Line 193: \"radius\" --> \"radians\"\n* Lines 194 and 195: \"measures\" --> \"measurements\"\n* Line 218: Since \"loss\" can have multiple meanings in this context, it may be worth clarify that \"loss\" is used in the ML context here.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes, the authors have adequately addressed these points.", "review_text": "", "extra_scores": {"ethical_concerns": "N/A", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}, {"review_id": "BEp7BrxC4AB", "reviewer": "Reviewer_6BFo", "summary": "The main contribution of the article is mention in the abstract \"... establishes an open-source environment. It aims to bridge the gap between the power community and the MARL community\". and In section 1: \"There is no commonly accepted benchmark to provide the basis for fair comparison of different solutions. To facilitate further research on this topic, and to bridge the gap between the power community and MARL community\", The authors seem to forget about L2RPN.", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "- The authors say in section 1 \"we present a comprehensive test-bench and open-source environment for MARL based active voltage control\". The effort for standardization is not good enough because they do not seem to know what measures should be taken to this. If there has to be a standard framework for RL based learning of smart grids then it must include the following specifications\n    - Definition of actions and their domains\n    - Definition of states and effect of grid-users on the states\n    - Definition of environment and its standard response to actions\n    - Define standards for randomness and noise in states, observations, actions, and their effects. \n- The algorithms ànd networks need to be fully customizable. \n\n\n- Formulation is sketchy. The actions are only dependent on the reactive power, while other researchers have used many different types of actions. Restricting the algorithm to limited actions will definitely affect the results, but the authors do not seem to realize that. In section 5.3 they describe the difference of results between their algorithms and traditional solutions, but they seem to forget that they are not using the complete range of actions here. \n\n- The authors seem to forget that the network layers matter a lot with respect to the algorithm, see (https://openreview.net/forum?id=LmUJqB1Cz8)\n\n- The authors also seem to have forgotten the effect of replay selection on the results. I conclude this by their choice reward function which they name as bowl function, which could also be a parabolic function too. Probably this is why they have doubts about their statements e.g. line 302-303, and section 5.3.\n\n- The implementation details mentioned in the appendix do not add much to the article. \n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "They need to be more elaborate about the limitations of the proposed technique/framework, secondly they should clearly mention the limitations of their solution within the proposed framework. \n", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "1KKNbxqrGXb", "reviewer": "Reviewer_DvqK", "summary": "This paper introduces a new open-source environment for training RL agents on controlling the voltage of a power distribution network. The problem is formulated as a multi-agent Dec-POMDP. Agents control PV inverters using only local measurements coming from sensors situated in the region they operate. Their task is to maintain the voltage within safety range. Several MARL algorithms are evaluated in three different power networks. The paper also investigates different reward functions.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The main contribution of this work is the introduction of a new benchmark domain for testing MARL algorithms. This consists of a simulator of a power network. The environment is built using existent network topologies and real data. I definitely agree with the authors' view on the importance of having realistic scenarios where RL methods can be tested appropriately before using them in real-world applications. It is very unfortunate that the code is not provided in the supplementary material. It is hard to assess how useful this work will be for the community without being able to test the environment.\n\nI miss some intuitive explanation of the dynamics of the problem. Some technical details are given in the Appendix but these are hard to follow for someone lacking in knowledge of the field. In particular, I am looking for details like, how much past information about the network state is useful for making decisions. How do the actions taken in the past affect the network in the long run. I.e. what's the horizon of the problem? All this could be illustrated using some sample trajectories coming from the standard operation of the network. To what extent do the changes in the policies of the agents in one region affect other regions? I.e how strongly coupled are the different regions in the network? This information could shed some light on why some MARL algorithms seem to perform better than others.\n\nIt is not clear whether an agent controls one or multiple PV inverters. Why are there multiple agents for each region? As far as I can see, all agents in a region receive the same information. If that’s the case, wouldn’t it be more effective to have all PV inverters within a region be controlled by a single policy? \n\nThe environment is partially observable, yet according to Section 5: MARL Algorithm Settings (line 283-287) the policies are modeled with MLPs receiving only the last two observations as input. Using only the last two observations doesn’t seem sufficient to satisfy the Markov assumption, especially considering the intrinsic non-stationarity of the problem (power generation and consumption fluctuate during the day).\n\nMinor details:\n\nh is first defined as a function that maps nodes to the corresponding subset of measures for that node (line 195) and then also used as the history length (line 228).\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "N/A", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents an multi-agent RL environment for active voltage control on power distribution networks. This could be a new interesting and practicality-oriented benchmark environment for multi-agent RL research. Most of the reviewers' concerns were (1) lack of access to the implementation of the simulator, which authors explicitly promised to make the code public if the paper gets accepted, and (2) lack of reference to another power network environment L2RPN / GridOp, which is not exactly active voltage control but should have been referenced. AC is recommending borderline accept, and if the paper gets accepted, please address (1) and provide detailed comparison to L2RPN / GridOp in the related work section, so that readers understand what exact power network problem the paper is tackling.
Decision: Accept (Poster)