Multi-Agent Reinforcement Learning for Active Voltage Control on Power Distribution Networks

1 Introduction . Multi-agent reinforcement learning ( MARL ) has demonstrated impressive performances on games ( e.g. , Chess [ 1 ] , StarCraft [ 2 , 3 ] , and etc . ) and robotics [ 4 ] . There are now increasing interests and thrusts to apply MARL in real-world problems . Power network is a natural test field for MARL algorithms . There are many problems in power networks involving the collaborative or competitive actions of a vast number of agents , such as market bidding [ 5 ] , voltage control , frequency control , and emergency handling [ 6 ] . One of the obstacles for applying MARL in power networks is the lack of transparency and guarantee , which is unacceptable considering the importance of reliable power supply for the whole society . Therefore , it is sensible to start with problems that are less sensitive to reliability but hard to be solved by conventional methods . Active voltage control in power distribution networks is such a candidate problem . The voltage control problem has been studied for years but only comes under spot light recently due to the increasing penetration of distributed resources , e.g . roof-top photovoltaics ( PVs ) [ 7 ] . The excessive active power injection may cause voltage fluctuations beyond the threshold of grid standards [ 8 ] . The voltage fluctuations can be relieved by exploiting the control flexibility of PV inverters themselves ∗Equal contributions . † Correspondence to Yunjie Gu who is also an honorary lecturer at Imperial College London . ‡ Tim C. Green is also the co-director of the Energy Futures Laboratory ( EFL ) . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . along with other controllable apparatuses , such as static var compensators ( SVCs ) and on load tap changers ( OLTCs ) . An elaborate scheme is needed to coordinate these multiple apparatuses at scale to regulate voltage throughout the network with limited local information , which is called active voltage control [ 9–12 ] . The active voltage control problem has many interesting properties . ( 1 ) It is a combination of local and global problem , i.e. , the voltage of each node is influenced by the powers ( real and reactive ) of all other nodes but the impact recedes with increasing distance between nodes . ( 2 ) It is a constrained optimisation problem where the constraint is the voltage threshold and the objective is the total power loss , but there is no explicit relationship between the constraint and the control action . ( 3 ) A distribution network has a radial topology involving a rich structure that can be taken as prior knowledge for control , but the node-branch parameters of the topology may be significantly uncertain . ( 4 ) Voltage control has a relatively large tolerance and less severe consequences if the control fails to meet standard requirements . There have been several attempts to apply MARL in active voltage control [ 6 , 13–16 ] . Each work on a particular case showed promising performances of state-of-the-art MARL approaches with modifications adapting to the active voltage control problem . However , it is not clear if these methods scale well to a larger network , and the robustness against different scenarios , such as penetration levels and load profiles , are yet to be investigated . There is no commonly accepted benchmark to provide the basis for fair comparison of different solutions . To facilitate further research on this topic , and to bridge the gap between the power community and MARL community , we present a comprehensive test-bench and open-source environment for MARL based active voltage control . We formally define the active voltage control problem as a Dec-POMDP [ 17 ] that is widely acknowledged in the MARL community . We present an environment in Python and construct 3 scenarios with real public data that span from small scale ( i.e . 6 agents ) to large scale ( i.e . 38 agents ) . The contributions of this paper are summarised as follows : ( 1 ) We formally define the active voltage control problem as a Dec-POMDP and develop an open-source environment.2 ( 2 ) We conduct large scale experimentation with 7 state-of-the-art MARL algorithms on different scenarios of the active voltage control problem . ( 3 ) We convert voltage constraints to barrier functions and observe the importance of designing an appropriate voltage barrier function from the experimental results . ( 4 ) By analysing the experimental results , we imply the possible challenges ( e.g . interpretability ) for MARL to solve the active voltage control problem and suggest potential directions for future works . 2 Related Work . Traditional Methods for Active Voltage Control . Voltage rising and fluctuation problem in distribution networks has been studied for 20 years [ 18 ] . The traditional voltage regulation devices such as OLTC and capacitor banks [ 10 ] are often installed at substations and therefore may not be effective in regulating voltages at the far end of the line [ 19 ] . The emergence of distributed generation , such as root-top PVs , introduces new approaches for voltage regulation by the active reactive power control of grid-connected inverters [ 11 ] . The state-of-the-art active voltage control strategies can be roughly classified into two categories : ( 1 ) reactive power dispatch based on optimal power flow ( OPF ) [ 20 , 21 ] ; and ( 2 ) droop control based on local voltage and power measurements [ 22 , 23 ] . Specifically , centralised OPF [ 24–26 ] minimises the power loss while fulfilling voltage constraints ( e.g . power flow equation defined in Eq.1 ) ; distributed OPF [ 27 , 28 ] used distributed optimization techniques , such as alternating direction method of multipliers ( ADMM ) , to replace the centralised solver . The primary limitation of OPF is the need of exact system model [ 29 ] . Besides , solving constrained optimisation problem is time-consuming , so it is difficult to react to the rapid change of load profile [ 30 ] . On the other hand , droop control only depends on its local measurements , but its performance relies on the manually-designed parameters and is often sub-optimal due to the lack of global information [ 30 ] . It is possible to enhance droop control by distributed algorithms , but extra communications are needed [ 31 , 19 ] . In this paper we investigate the possibility of applying MARL techniques on the active voltage control problem . Compared with the previous works on traditional methods , ( 1 ) MARL is model-free , so no exact system model is needed ; and ( 2 ) the response of MARL is fast so as to handle rapid changes of environments ( e.g . the intermittency of renewable energy ) . 2https : //github.com/Future-Power-Networks/MAPDN . Multi-Agent Reinforcement Learning for Active Voltage Control . In this section , we discuss the works that applied MARL to active voltage control problem in the power system community . [ 14 , 16 ] applied MADDPG with the reactive power of inverters or static var compensators ( SVCs ) as control actions . [ 32 ] applied MADDPG with a manually designed voltage inner loop , so that agents set reference voltage instead of reactive power as their control actions . [ 13 ] applied MATD3 also with reactive power as control actions . [ 15 ] applied MASAC , where both reactive power and the curtailment of active power are used as control actions . In the above works , distribution networks are divided into regions , with each region controlled by a single agent [ 33 ] . It is not clear if these MARL approaches scales well for increasing number of agents . In particular , it is not clear if each single inverter in a distribution network can behave as an independent agent . In this work , we model the active voltage control problem as a Dec-POMDP [ 17 ] , where each inverter is controlled by an agent . We propose Bowl-shape as a barrier function to represent voltage constraint as part of the reward . We build up an open-source environment for this specific problem that can be easily deployed with the existing MARL algorithms . Concurrent Works . L2RPN [ 34 ] is an environment mainly for the centralised control over power transmission networks with node-splitting and lines switch-off for topology management . Gym-ANM [ 35 ] is an environment for centralised power distribution network management . Compared with these 2 works , our environment mainly focuses on solving the active voltage control problem in power distribution networks , with the decentralised/distributed manner . Moreover , we provide more complicated scenarios with the real-world data . Finally , our environment can be easily and flexibly extended with more network topologies and data , thanks to the supports of PandaPower [ 36 ] and SimBench [ 37 ] . 3 Background . 3.1 Power Distribution Network . An electric power distribution network is illustrated in Figure 1 stage a to c. The electricity is generated from power plant and transmitted through transmission lines . Muti-stage transformers are applied to reduce the voltage levels while the electricity is being delivered to the distribution network . The electricity is then consumed by residential and industrial clients . A typical PV unit consists of PV panels and voltage-source inverters which can be installed either on roof-top or in the solar farm . Conventionally , there exist management entities such as distributed system operator ( DSO ) monitoring and operating the PV resources through the local communication channels . With emergent PV penetration , distribution network gradually grows to be an active participant in power networks that can deliver power and service to its users and the main grid ( see the bidirectional power flows in Figure 1 stage-d ) . System Model and Voltage Deviation . In this paper , we consider medium ( 10-24kV ) and low ( 0.23-1kV ) voltage distribution networks where PVs are highly penetrated . We model the distribution network in Figure 1 as a tree graph G = ( V , E ) , where V = { 0 , 1 , . . . , N } and E = { 1 , 2 , . . . , N } represent the set of nodes ( buses ) and edges ( branches ) respectively [ 24 ] . Bus 0 is considered as the connection to the main grid , balancing the active and reactive power in the distribution network . For each bus i ∈ V , let vi and θi be the magnitude and phase angle of the complex voltage and sj = pi + jqi be the complex power injection . Then the active and reactive power injection can be defined as follows : pPVi − pLi = v2i ∑ j∈Vi gij − vi ∑ j∈Vi vj ( gij cos θij + bij sin θij ) , ∀i ∈ V \ { 0 } qPVi − qLi = −v2i ∑ j∈Vi bij + vi ∑ j∈Vi vj ( gij sin θij + bij cos θij ) , ∀i ∈ V \ { 0 } ( 1 ) where Vi : = { j | ( i , j ) ∈ E } is the index set of buses connected to bus i . gij and bij are the conductance and susceptance on branch ( i , j ) . θij = θi − θj is the phase difference between bus i and j . pPVi and q PV i are the active power and reactive power of the PV on the bus i ( that are zeros if there is no PV on the bus i ) . pLi and q L i are the active power and reactive power of the loads on the bus i ( that are zeros if there is no loads on the bus i ) . Eq.1 can represent the power system dynamics which is essential for solving the power flow problem and active voltage control problem [ 38 ] ( details in Appendix A.1-A.2 ) . For safe and optimal operation , 5 % voltage deviation is usually allowed , i.e. , v0 = 1.0 per unit ( p.u . ) and 0.95 p.u . ≤ vi ≤ 1.05 p.u. , ∀i ∈ V \ { 0 } . When the load is heavy during the nighttime , the end-user voltage could be smaller than 0.95 p.u . [ 39 ] . In contrast , to export its power , large penetration of pPVi leads to reverse current flow that would increase vi out of the nominal range ( Figure 1-d ) [ 18 , 40 ] . Optimal Power Flow . In this paper , OPF is regarded as an optimization problem minimizing the total power loss subject to the power balance constraints defined in Eq.1 , PV reactive power limits , and bus voltage limits [ 41 ] . As the centralized OPF has full access to the system topology , measurements , and PV resources , it provides the optimal active voltage control performance and can be used as a benchmark method ( details in Appendix A.3 ) . However , the performance of the OPF depends on the accuracy of the grid model and the optimisation is time-consuming which makes it difficult to be deployed online . Droop Control . To regulate local voltage deviation , the standard droop control defines a piece-wise linear relationship between PV reactive power generation and voltage deviation at a bus equipped with inverter-based PVs [ 30 , 42 ] ( details in Appendix A.4 ) . It is a fully decentralised control and ignore both the total voltage divisions and the power loss .

Summary: In this work, the authors studied applying multi-agent reinforcement learning (MARL) to active voltage control in the power distribution networks, and targeting for bridging the gap between the power community and MARL community. Main contributions include:  1) develop an open-source environment of active voltage control for MARL and formally define this problem as a 65 Dec-POMDP; 2) conduct large scale experimentation with 8 state-of-the-art MARL algorithms on different scenarios of voltage control problem; 3) found valuable observations, e.g.,  notice the importance of converting voltage constraints to auxiliary reward functions and verify the sensitivity of MARL performances to the auxiliary reward functions, with a new bowl-shape auxiliary reward proposed which stands out in the test results; 4) by analyzing the experimental results, the authors imply the possible difficulties of applying MARL and the further research directions. The authors also conducted the comparison with traditional control methods, and empirically show the rationality and hope to use MARL as a solution for active voltage control.