Winning the L2RPN Challenge: Power Grid Management via Semi-Markov Afterstate Actor-Critic

1 INTRODUCTION . The power grid , an interconnected network for delivering electricity from producers to consumers , has become an essential component of modern society . For a safe and reliable transmission of electricity , it is constantly monitored and managed by human experts in the control room . Therefore , there has been growing interest in automatically controlling and managing the power grid . As we make the transition to sustainable power sources such as solar , wind , and hydro ( Rolnick et al. , 2019 ) , power grid management is becoming a very complex task beyond human expertise , calling for data-driven optimization . Yet , automatic control of a large-scale power grid is a challenging task since it requires complex yet reliable decision-making . While most approaches have focused on controlling the generation or the load of electricity ( Venkat et al. , 2008 ; Zhao et al. , 2014 ; Huang et al. , 2020 ) , managing the power grid through the topology control ( changing the connection of power lines and bus assignments in substations ) would be the ultimate goal . By reconfiguring the topology of the power grid , it can reroute the flow of electricity , which enables the transmission of electricity from the producers to consumers efficiently and thus prevent surplus production . There are preliminary studies of the grid topology control in the power systems literature ( Fisher et al. , 2008 ; Khodaei & Shahidehpour , 2010 ) , but due to its large , combinatorial , and non-linear nature , these methods do not provide a practical solution to be deployed to the real-world . On the other hand , deep Reinforcement Learning ( RL ) has shown significant progress in complex sequential decision-making tasks , such as Go ( Silver et al. , 2016 ) and arcade video games ( Mnih et al. , 2015 ) , purely from data . RL is also perceived as a promising candidate to address the challenges of power grid management ( Ernst et al. , 2004 ; Dimeas & Hatziargyriou , 2010 ; Duan et al. , 2020 ; Zhang et al. , 2020 ; Hua et al. , 2019 ) . In this regard , we present Semi-Markov ∗ : Equal contribution Afterstate Actor-Critic ( SMAAC ) , an RL algorithm that effectively tackles the challenges in power grid management . One of the main challenges in RL for the real-world scale power grid management lies in its massive state and action space . We address the problem by adopting a goal-conditioned hierarchical policy with the afterstate representation . First , we represent state-action pairs as afterstates ( Sutton & Barto , 2018 ) , the state after the agent has made its decision but before the environment has responded , to efficiently cover the large state-action space . The afterstate representation can be much more succinct than the state-action pair representation when multiple state-action pairs are leading to an identical afterstate . For example , in the case of controlling the topology of the power grid , a pair of a current topology and an action of topology modification can be represented as a reconfigured topology , since the topology is deterministically reconfigured by the action . Then the next state is determined by random external factors , such as the change of power demands in load . Second , we extend this idea to a hierarchical framework , where the high-level policy produces a desirable topology under the current situation , and the low-level policy takes care of figuring out an appropriate sequence of primitive topology changes . Combined together , our hierarchical policy architecture with afterstates facilitates effective exploration for good topology during training . Our algorithm ranked first in the latest international competition on training RL agents to manage power grids , Learning To Run a Power Network ( L2RPN ) WCCI 2020 . In this paper , we further evaluate our approach using Grid2Op , the open-source power grid simulation platform used in the competition , by training and testing the agent in 3 different sizes of power grids . We show that the agent significantly outperforms all of the baselines in all grids except for the small grid where the task was easy for all algorithms . 2 BACKGROUND . 2.1 GRID2OP ENVIRONMENT . We briefly overview Grid2Op , the open-source simulation platform for power grid operation used in the L2RPN WCCI 2020 challenge . Grid2Op models realistic concepts found in realworld operations used to test advanced control algorithms , which follow real-world power system operational constraints and distributions ( Kelly et al. , 2020 ) . The power grid is essentially a graph composed of nodes corresponding to substations that are connected to loads , generators , and power lines . The generator produces electricity , the load consumes electricity , and the power line transmits electricity between substations . The substation can be regarded as a router in the network , which determines where to transmit electricity . Grid2Op considers 2 conductors per substation , known as the double busbar system . This means that the elements connected to a substation , i.e . loads , generators , and power lines , can be assigned to one of the two busbars , and the power travels only over the elements on the same busbar . Thus , each substation can be regarded as being split into two nodes . The state of the power grid consists of various features such as a topology configuration ( the connectivity of each power line and the bus assignment in each substation ) , as well as the amount of power provided by each generator , required by each load , transmitted in each line , and so on . The power supplied by generators and demanded by loads changes over time , and the power transmitted in lines also changes according to the current topology configuration together with supply and demand . In addition , each line has its own capacity to transmit electricity and can be automatically disconnected when there is an overflow of electricity . The agent can apply actions on substations and lines to managing the power grid . The action on a substation , called bus assignment , assigns the elements in the substation to a busbar . The action on a line , called line switch , disconnects ( both ends of the line is assigned to neither bus ) a line or reconnects a disconnected line . The agent is allowed to perform one line switch or one bus assignment action per step , and can not successively perform actions on the same line or substation . The power grid is simulated for a given period , typically for several days at a 5-minute interval . The simulation can terminate prematurely when the agent fails to manage the grid , i.e . ( 1 ) the amount of power required by loads are not delivered , which can happen if there are too many disconnected lines , or ( 2 ) a disconnected subgraph is formed as a result of applying an action . This is reflected in the failure penalty when measuring the performance of the agent , given by the number of remaining simulation time steps upon termination . Another important performance metric is the power loss penalty , given by the amount of power that disappeared during transmitting due to resistive loss . Thus , the goal of the agent is to operate the power grid both safely and efficiently by minimizing the failure penalty and the power loss penalty . Figure 1 illustrates how the actions affect the state of the power grid using the bus assignment action as an example . The simulator provides 3 different sizes of power grids , ( 1 ) IEEE-5 is the power grid with 5 substations , ( 2 ) IEEE-14 is the power grid with 14 substations , and ( 3 ) L2RPN WCCI 2020 is the power grid with 36 substations . See Appendix A.1 for more details on the environment . 2.2 AFTERSTATES IN RL . Grid2Op provides a natural framework to use RL for operating power grids : we assume a Markov decision process ( MDP ) defined by ( S , A , p , r , γ ) to represent the RL task , where S is the state space , A is the action space , and p ( st+1|st , at ) is the ( unknown ) state transition probability , rt = r ( st , at ) ∈ R is the immediate reward , and γ ∈ ( 0 , 1 ) is the discount factor . We assume learning a stochastic policy π ( at|st ) , which is a probability distribution over actions conditioned on states . The state and action value functions under π are V π ( s ) = Eπ [ ∑ l≥0 γ lrt+l|st = s ] and Qπ ( s , a ) = Eπ [ ∑ l≥0 γ lrt+l|st = s , at = a ] respectively . As shown in Figure 1 in the previous section , the transition in Grid2Op comprises two steps : the topological change that results directly from the action , and then the rest of the state changes that arise from exogenous events . This motivates the use of the afterstate ( Sutton & Barto , 2018 ) , also known as the post-decision state in Approximate Dynamic Programming ( ADP ) ( Powell , 2007 ) , which refers to the state after the agent has made its decision but before the arrival of new information . Let us define the state S as ( T , X ) where T is the part of the state that is deterministically changed by an action , and X as independent or affected indirectly from an action . Following the modeling in ( Powell , 2007 ) , the transition is decomposed into two parts using fA and fE : st+1 = [ τt+1 , xt+1 ] = f E ( [ τt+1 , xt ] , et+1 ) , s at t = [ τt+1 , xt ] = f A ( [ τt , xt ] , at ) , ( 1 ) where τt+1 , the deterministic part of st+1 , is given by the the function fA ( st , at ) , and xt+1 , the stochastic part , is given by the function fE ( sat , et+1 ) where et+1 is the source of the randomness in the transition sampled from some unknown distribution pE . Note that et+1 itself can be included as a part in xt+1 . Using the afterstate has a number of advantages . For example , if the state and the action spaces are very large but the set of unique afterstates is relatively small , learning the value function of afterstates would be much more efficient . The value of an afterstate sa under policy π is defined as V π ( sa ) = Eπ [ ∑ l≥0 γ lrt+l|sa = fA ( st , at ) ] and its recursive form can be written as : V π ( satt ) = Eet+1∼pE , at+1∼π [ r ( st , at ) + γV π ( fA ( st+1 , at+1 ) ) |st+1 = fE ( satt , et+1 ) ] ( 2 ) The optimal afterstate value function and the optimal policy can be obtained by iteratively alternating between the policy evaluation by Eq . ( 2 ) and policy improvement : πnew ( st ) = argmax at [ V πold ( fA ( st , at ) ) ] ( 3 ) Note that we can not gain much from the afterstate representation when using the individual power grid operations as actions since they result in unique changes in the grid topology . However , we shall see that the afterstate becomes very powerful when we consider the sequences of grid operations as the action space , where their permutations result in identical changes in the final topology .

Summary: This paper proposes an effective method for managing power grid topology to increase efficiency. They use Transformer attention over a Graph Neural Network as the basic architecture, then propose a hierarchical technique in which the upper level learns to output goal network topologies, which are then implemented by a lower-level policy or a rule-based algorithm. An ablation study reveals that one of the most important components of the algorithm is using an "afterstate" representation, which learns a value function for the state after the agent changes the topology, but before the network is affected by random external factors, including supply and demand.