Multi-Agent Reinforcement Learning for Active Voltage Control on Power Distribution Networks

1 Introduction . Multi-agent reinforcement learning ( MARL ) has demonstrated impressive performances on games ( e.g. , Chess [ 1 ] , StarCraft [ 2 , 3 ] , and etc . ) and robotics [ 4 ] . There are now increasing interests and thrusts to apply MARL in real-world problems . Power network is a natural test field for MARL algorithms . There are many problems in power networks involving the collaborative or competitive actions of a vast number of agents , such as market bidding [ 5 ] , voltage control , frequency control , and emergency handling [ 6 ] . One of the obstacles for applying MARL in power networks is the lack of transparency and guarantee , which is unacceptable considering the importance of reliable power supply for the whole society . Therefore , it is sensible to start with problems that are less sensitive to reliability but hard to be solved by conventional methods . Active voltage control in power distribution networks is such a candidate problem . The voltage control problem has been studied for years but only comes under spot light recently due to the increasing penetration of distributed resources , e.g . roof-top photovoltaics ( PVs ) [ 7 ] . The excessive active power injection may cause voltage fluctuations beyond the threshold of grid standards [ 8 ] . The voltage fluctuations can be relieved by exploiting the control flexibility of PV inverters themselves ∗Equal contributions . † Correspondence to Yunjie Gu who is also an honorary lecturer at Imperial College London . ‡ Tim C. Green is also the co-director of the Energy Futures Laboratory ( EFL ) . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . along with other controllable apparatuses , such as static var compensators ( SVCs ) and on load tap changers ( OLTCs ) . An elaborate scheme is needed to coordinate these multiple apparatuses at scale to regulate voltage throughout the network with limited local information , which is called active voltage control [ 9–12 ] . The active voltage control problem has many interesting properties . ( 1 ) It is a combination of local and global problem , i.e. , the voltage of each node is influenced by the powers ( real and reactive ) of all other nodes but the impact recedes with increasing distance between nodes . ( 2 ) It is a constrained optimisation problem where the constraint is the voltage threshold and the objective is the total power loss , but there is no explicit relationship between the constraint and the control action . ( 3 ) A distribution network has a radial topology involving a rich structure that can be taken as prior knowledge for control , but the node-branch parameters of the topology may be significantly uncertain . ( 4 ) Voltage control has a relatively large tolerance and less severe consequences if the control fails to meet standard requirements . There have been several attempts to apply MARL in active voltage control [ 6 , 13–16 ] . Each work on a particular case showed promising performances of state-of-the-art MARL approaches with modifications adapting to the active voltage control problem . However , it is not clear if these methods scale well to a larger network , and the robustness against different scenarios , such as penetration levels and load profiles , are yet to be investigated . There is no commonly accepted benchmark to provide the basis for fair comparison of different solutions . To facilitate further research on this topic , and to bridge the gap between the power community and MARL community , we present a comprehensive test-bench and open-source environment for MARL based active voltage control . We formally define the active voltage control problem as a Dec-POMDP [ 17 ] that is widely acknowledged in the MARL community . We present an environment in Python and construct 3 scenarios with real public data that span from small scale ( i.e . 6 agents ) to large scale ( i.e . 38 agents ) . The contributions of this paper are summarised as follows : ( 1 ) We formally define the active voltage control problem as a Dec-POMDP and develop an open-source environment.2 ( 2 ) We conduct large scale experimentation with 7 state-of-the-art MARL algorithms on different scenarios of the active voltage control problem . ( 3 ) We convert voltage constraints to barrier functions and observe the importance of designing an appropriate voltage barrier function from the experimental results . ( 4 ) By analysing the experimental results , we imply the possible challenges ( e.g . interpretability ) for MARL to solve the active voltage control problem and suggest potential directions for future works . 2 Related Work . Traditional Methods for Active Voltage Control . Voltage rising and fluctuation problem in distribution networks has been studied for 20 years [ 18 ] . The traditional voltage regulation devices such as OLTC and capacitor banks [ 10 ] are often installed at substations and therefore may not be effective in regulating voltages at the far end of the line [ 19 ] . The emergence of distributed generation , such as root-top PVs , introduces new approaches for voltage regulation by the active reactive power control of grid-connected inverters [ 11 ] . The state-of-the-art active voltage control strategies can be roughly classified into two categories : ( 1 ) reactive power dispatch based on optimal power flow ( OPF ) [ 20 , 21 ] ; and ( 2 ) droop control based on local voltage and power measurements [ 22 , 23 ] . Specifically , centralised OPF [ 24–26 ] minimises the power loss while fulfilling voltage constraints ( e.g . power flow equation defined in Eq.1 ) ; distributed OPF [ 27 , 28 ] used distributed optimization techniques , such as alternating direction method of multipliers ( ADMM ) , to replace the centralised solver . The primary limitation of OPF is the need of exact system model [ 29 ] . Besides , solving constrained optimisation problem is time-consuming , so it is difficult to react to the rapid change of load profile [ 30 ] . On the other hand , droop control only depends on its local measurements , but its performance relies on the manually-designed parameters and is often sub-optimal due to the lack of global information [ 30 ] . It is possible to enhance droop control by distributed algorithms , but extra communications are needed [ 31 , 19 ] . In this paper we investigate the possibility of applying MARL techniques on the active voltage control problem . Compared with the previous works on traditional methods , ( 1 ) MARL is model-free , so no exact system model is needed ; and ( 2 ) the response of MARL is fast so as to handle rapid changes of environments ( e.g . the intermittency of renewable energy ) . 2https : //github.com/Future-Power-Networks/MAPDN . Multi-Agent Reinforcement Learning for Active Voltage Control . In this section , we discuss the works that applied MARL to active voltage control problem in the power system community . [ 14 , 16 ] applied MADDPG with the reactive power of inverters or static var compensators ( SVCs ) as control actions . [ 32 ] applied MADDPG with a manually designed voltage inner loop , so that agents set reference voltage instead of reactive power as their control actions . [ 13 ] applied MATD3 also with reactive power as control actions . [ 15 ] applied MASAC , where both reactive power and the curtailment of active power are used as control actions . In the above works , distribution networks are divided into regions , with each region controlled by a single agent [ 33 ] . It is not clear if these MARL approaches scales well for increasing number of agents . In particular , it is not clear if each single inverter in a distribution network can behave as an independent agent . In this work , we model the active voltage control problem as a Dec-POMDP [ 17 ] , where each inverter is controlled by an agent . We propose Bowl-shape as a barrier function to represent voltage constraint as part of the reward . We build up an open-source environment for this specific problem that can be easily deployed with the existing MARL algorithms . Concurrent Works . L2RPN [ 34 ] is an environment mainly for the centralised control over power transmission networks with node-splitting and lines switch-off for topology management . Gym-ANM [ 35 ] is an environment for centralised power distribution network management . Compared with these 2 works , our environment mainly focuses on solving the active voltage control problem in power distribution networks , with the decentralised/distributed manner . Moreover , we provide more complicated scenarios with the real-world data . Finally , our environment can be easily and flexibly extended with more network topologies and data , thanks to the supports of PandaPower [ 36 ] and SimBench [ 37 ] . 3 Background . 3.1 Power Distribution Network . An electric power distribution network is illustrated in Figure 1 stage a to c. The electricity is generated from power plant and transmitted through transmission lines . Muti-stage transformers are applied to reduce the voltage levels while the electricity is being delivered to the distribution network . The electricity is then consumed by residential and industrial clients . A typical PV unit consists of PV panels and voltage-source inverters which can be installed either on roof-top or in the solar farm . Conventionally , there exist management entities such as distributed system operator ( DSO ) monitoring and operating the PV resources through the local communication channels . With emergent PV penetration , distribution network gradually grows to be an active participant in power networks that can deliver power and service to its users and the main grid ( see the bidirectional power flows in Figure 1 stage-d ) . System Model and Voltage Deviation . In this paper , we consider medium ( 10-24kV ) and low ( 0.23-1kV ) voltage distribution networks where PVs are highly penetrated . We model the distribution network in Figure 1 as a tree graph G = ( V , E ) , where V = { 0 , 1 , . . . , N } and E = { 1 , 2 , . . . , N } represent the set of nodes ( buses ) and edges ( branches ) respectively [ 24 ] . Bus 0 is considered as the connection to the main grid , balancing the active and reactive power in the distribution network . For each bus i ∈ V , let vi and θi be the magnitude and phase angle of the complex voltage and sj = pi + jqi be the complex power injection . Then the active and reactive power injection can be defined as follows : pPVi − pLi = v2i ∑ j∈Vi gij − vi ∑ j∈Vi vj ( gij cos θij + bij sin θij ) , ∀i ∈ V \ { 0 } qPVi − qLi = −v2i ∑ j∈Vi bij + vi ∑ j∈Vi vj ( gij sin θij + bij cos θij ) , ∀i ∈ V \ { 0 } ( 1 ) where Vi : = { j | ( i , j ) ∈ E } is the index set of buses connected to bus i . gij and bij are the conductance and susceptance on branch ( i , j ) . θij = θi − θj is the phase difference between bus i and j . pPVi and q PV i are the active power and reactive power of the PV on the bus i ( that are zeros if there is no PV on the bus i ) . pLi and q L i are the active power and reactive power of the loads on the bus i ( that are zeros if there is no loads on the bus i ) . Eq.1 can represent the power system dynamics which is essential for solving the power flow problem and active voltage control problem [ 38 ] ( details in Appendix A.1-A.2 ) . For safe and optimal operation , 5 % voltage deviation is usually allowed , i.e. , v0 = 1.0 per unit ( p.u . ) and 0.95 p.u . ≤ vi ≤ 1.05 p.u. , ∀i ∈ V \ { 0 } . When the load is heavy during the nighttime , the end-user voltage could be smaller than 0.95 p.u . [ 39 ] . In contrast , to export its power , large penetration of pPVi leads to reverse current flow that would increase vi out of the nominal range ( Figure 1-d ) [ 18 , 40 ] . Optimal Power Flow . In this paper , OPF is regarded as an optimization problem minimizing the total power loss subject to the power balance constraints defined in Eq.1 , PV reactive power limits , and bus voltage limits [ 41 ] . As the centralized OPF has full access to the system topology , measurements , and PV resources , it provides the optimal active voltage control performance and can be used as a benchmark method ( details in Appendix A.3 ) . However , the performance of the OPF depends on the accuracy of the grid model and the optimisation is time-consuming which makes it difficult to be deployed online . Droop Control . To regulate local voltage deviation , the standard droop control defines a piece-wise linear relationship between PV reactive power generation and voltage deviation at a bus equipped with inverter-based PVs [ 30 , 42 ] ( details in Appendix A.4 ) . It is a fully decentralised control and ignore both the total voltage divisions and the power loss .

Summary: This paper proposes a new environment for multi-agent reinforcement learning that addresses the problem of active voltage control on electric power distribution networks. The authors formulate the voltage control problem as a Dec-POMDP, and implement a number of different MARL baselines to assess their performance on the environment. They also assess the effect of different voltage loss functions on this environment.