Winning the L2RPN Challenge: Power Grid Management via Semi-Markov Afterstate Actor-Critic
Authors: Deunsol Yoon
Sunghoon Hong
Byung-Jun Lee
Kee-Eung Kim
Venue: ICLR 2021 Spotlight (2021)
Keywords: power grid management
deep reinforcement learning
graph neural network

Abstract: Safe and reliable electricity transmission in power grids is crucial for modern society. It is thus quite natural that there has been a growing interest in the automatic management of power grids, exempliﬁed by the Learning to Run a Power Network Challenge (L2RPN), modeling the problem as a reinforcement learning (RL) task. However, it is highly challenging to manage a real-world scale power grid, mostly due to the massive scale of its state and action space. In this paper, we present an off-policy actor-critic approach that effectively tackles the unique challenges in power grid management by RL, adopting the hierarchical policy together with the afterstate representation. Our agent ranked ﬁrst in the latest challenge (L2RPN WCCI 2020), being able to avoid disastrous situations while maintaining the highest level of operational efﬁciency in every test scenarios. This paper provides a formal description of the algorithmic aspect of our approach, as well as further experimental studies on diverse power grids.

Published as a conference paper at ICLR 2021
WINNING THE L2RPN CHALLENGE: POWER GRID MANAGEMENT VIA SEMI-MARKOV AFTERSTATE ACTOR-CRITIC
Deunsol Yoon"1, Sunghoon Hong" Byung-Jun Lee Kee- Kee-Eung Kim
GGaauuate School of AI, KAIST, Daejeon, Republic of Korea
School of Computing, KAIST, Daejeon, Republic of Korea
Gaauss Labs Inc., Scoul, Republic of Korea dsyooo, shhong, bilee) @ai kaist. ac. kr, kekimokaist. ac.kr
ABSTRACT
Safe and reliable electricity transmission in power grids crucial for modern society. is thus quite natural that there has been a growing interest in the automatic management of power grids, exemplified by the Learning to Run a Power Network Challenge (L2RPN), modeling the problem learning (RL) task. However, it is highly challenging to manage real-world scale power grid, mostly due the massive scale of its state and action space. In this paper, we present an off-policy actor-critic approach that effectively tackles the unique challenges in power grid management by RI., adopting the hierarchical policy together with the afterstate representation. Our agent ranked first in the latest challenge (L2RPN WCCI 2020), being able to avoid disastrous situations while maintaining the highest level of operational efficiency in every test scenario. This paper provides formal description the algorithmic aspect our approach, well as further experimental studies diverse power grids.
reinforcement
INTRODUCTION
The power grid, an interconnected network for delivering electricity from producers to consumers, has become essential component of modern society. For safe and reliable transmission of electricity, is constantly monitored and managed by human experts in the control room. Therefore, there has been growing interest automatically controlling and managing the power grid. As make the transition to sustainable power sources such as solar, wind, and hydro (Rolnick er al.) 2019), power grid management is becoming very complex task beyond human expertise, calling for data-driven optimization.
Yet, automatic control of large-scale power grid is challenging task since it requires complex yet reliable decision-making While most approaches have focused on controlling the generation or the load of electricity (Venkat al.) 2008 Zhao et al., 2014 Huang et al., 2020). managing the power grid through the topology control (changing the connection of power lines and bus assignments in substations) would the ultimate goal. By reconfiguring the topology
the power grid, it can reroute the flow of electricity, which enables the transmission of electricity from the producers to consumers efficiently and thus prevent surplus production. There are preliminary studies of the grid topology control in the power systems literature (Fisher et al. 2008 Khodaei & Shahidehpour 2010), but due to its large, combinatorial, and non-linear nature, these methods do not provide a practical solution to be deployed the real-world.
On the other hand, deep Reinforcement Learning (RL) has shown significant progress in complex sequential decision-making tasks, such Go (Silver et al. 2016) and arcade video games (Mnih) ett al.] 2015). purely from data. RI. is also perceived as a promising candidate to address the challenges of power grid management (Ernst al. 2004 Dimeas & Hatziargyriou 2010; Duan 2020 Zhang et al. 2020; HHaa et al. 2019) In this regard, we present Semi-Markov et al.
Equal contribution

---

Published as a conference paper at ICLR 2021
10
geng
30
gen, =11
gen;
1
St
S St+1
17
at
e t++
load, 5
Loadz -00
load, -6
load2
188
Figure 1: An example of power grid with 4 substations, 2 generators, 2 loads, and 5 lines. Starting from the left, bus assignment action a reconfigures the grid and then the next state st+1 is determined by exogenous event 6t+1, such as the change of power demands in loads. The diagonal line was experiencing overflow, but the action ag shown to revert the overflow. The power loss also reduced from 15 to 13.
Afterstate Actor-Critic (SMAAC), an RL algorithm that effectively tackles the challenges in power grid management.
One of the main challenges in RL for the real-world scale power grid management lies in its massive state and action space. We address the problem by adopting goal-conditioned hierarchical policy with the afterstate representation. First, we represent state-action pairs as afterstates (Sutton & Barto 2018), the state after the agent has made its decision but before the environment has responded, to efficiently cover the large state-action space. The afterstate representation can be much more suceinct than the state-action pair representation when multiple state-action pairs leading to an identical afterstate. For example, in the case of controlling the topology of the power grid, pair of current topology and action of topology modification can be represented reconfigured topology, since the topology is deterministically reconfigured by the action. Then the next state is determined by random external factors, such the change of power demands in load. Second, we extend this idea to a hierarchical framework, where the high-level policy produces desirable topology under the current situation, and the low-level policy takes care of figuring out an appropriate sequence of primitive topology changes. Combined together, our hierarchical policy architecture with afterstates facilitates effective exploration for good topology during training.
Our algorithm ranked first in the latest international competition on training RL agents to manage power grids, Learning To Run a Power Network (L22RNN) WCCI 2020. In this paper, we further evaluate our approach using Grid2Op, the open-source power grid simulation platform used in the competition, by training and testing the agent different sizes power grids. We show that the agent significantly outperforms all of the baselines in all grids except for the small grid where the task was easy for all algorithms.
BACKGROUND
2.1 GRID2OP ENVIRONMENT
We briefly overview Grid2Op, the open-source simulation platform for power grid operation used in the L2RPN WCCI 2020 challenge. Grid2Op models realistic concepts found in real- world operations used test advanced control algorithms. which follow real-world power system operational constraints and distributions (Kelly et al. 2020).
The power grid essentially graph composed of nodes corresponding to substations that are connected to loads, generators, and power lines. 'he generator produces electricity, the load consumes electricity, and the power line transmits electricity between substations. The substation can be regarded as router in the network, which determines where to transmit electricity. Grid2Op considers conductors per substation, known as the double busbar system. This means that the elements connected to a substation, i.c. loads, generators, and power lines, can be assigned one of the two busbars, and the power travels only over the elements on the same busbar. Thus, each substation be regarded being split into nodes.
2

---

Published as a conference paper at ICLR 2021
The state of the power grid consists of various features such topology configuration (the connectivity of each power line and the bus assignment in each substation), as well as the amount of power provided by each generator, required by each load, transmitted in each line, and on. The power supplied by generators and demanded by loads changes over time, and the power transmitted in lines also changes according to the current topology configuration together with supply and demand. In addition, each line has its own capacity to transmit electricity and can be automatically disconnected when there is an overflow of electricity.
The agent can apply actions on substations and lines to managing the power grid. The action on substation, called bus assignment, assigns the elements in the substation to a busbar. The action on line, called line switch, disconnects (both ends of the line is assigned to neither bus) a line or reconnects a disconnected line. The agent allowed to perform one line switch or one bus assignment action per step, and cannot successively perform actions on the same line or substation.
The power grid is simulated for a given period, typically for several days a 5-minute interval. The simulation can terminate prematurely when the agent fails to manage the grid, i.e. (1) the amount of power required by loads are not delivered, which can happen if there are too many disconnected lines, or (2) a disconnected subgraph is formed as a result of applying the failure penalty when measuring the performance of the agent, given by the number of remaining simulation time steps upon termination. Another important performance metric is the power loss penalty, given by the amount of power that disappeared during transmitting due to resistive loss. Thus, the goal of the agent is operate the power grid both safely and efficiently by minimizing the failure penalty and the power loss penalty.
action. This is reflected in
Figure I fillustrates how the actions affect the state of the power grid using the bus assignment action as an example. The simulator provides 3 different sizes of power grids, (1) IEEE-5 is the power grid with 5 substations, (2) IEEE-14 14 the power grid with 14 substations, and (3) L2RPN WCCI 2020 is the power grid with 36 substations. See Appendix A. for more details on the environment.
2.2 AFTERSTATES IN RL
Grid2Op provides natural framework to use RL for operating power grids; we assume a Markov
decision process (MDP) defined by (S, r, 7) to represent the RL task, where S is the state space, A is the action space, and p(*++1|*;; ax) is the (unknown) state transition probability, r. = r(st, 0e) E IR is the immediate reward, and 7 E 10, is the discount factor. We assume learning a stochastic policy (a, (s) which is a probability distribution over actions conditioned on states. The state and action value functions under are (*) Ex Exx T' st s] and QT a)
E,, 1Ei0 7rr = 8, a = a) respectively.
As shown in Figure Tin in the previous section, the transition in Grid2Op comprises two steps: the topological change that results directly from the action, and then the rest of the state changes that arise from exogenous events. This motivates the use of the afterstate (Sutton & Barto 2018), also known as the post-decision state in Approximate Dynamic Programming (ADP) (Powell) 2007), which refers to the state after the agent has made its decision but before the arrival of new information.
Let us define the state as (T,X) where T is the part of the state that is deterministically changed by an action, and X as independent affected indirectly from an action. Following the modeling in (Powell 2007,, the transition is decomposed into two parts using f and fE:
S141 Ic+1] = f 1[T+1, 1x+1),
= [T
f ([Te, 2e), (1)
where T++1, the deterministic part of $t41, is given by the the function (s,, at), and r4+1, the stochastic part, is given by the function (st e1+1) where c++1 is the source of the randomness in the transition sampled from some unknown distribution p Note that e itself can be included as a part in 12+1-
Using the afterstate has a number of advantages. For example, if the state and the action spaces are very large but the set of unique afterstates is relatively small, learning the value function of afterstates would be much more efficient. The value of an afterstate s" under policy is defined as V= E, Eiz0 Tre4lss" = fA(s,, at) and its recursive form can be written as:
V (+ E aotio [r(st, at) + +VV*( ( (se+1, Gt+ ()=1)))8441 = 8 5 (681, ee41)]
(2)
3

---

Published as a conference paper at ICLR 2021
The optimal afterstate value function and the optimal policy can be obtained by iteratively alternating between the policy evaluation by Eq. (2) and policy improvement
πT Nnww (st) = arg max
(Voli (st,
(3)
Note that we cannot gain much from the afterstate representation when using the individual power grid operations as actions since they result in unique changes the grid topology. However, we shall see that the afterstate becomes very powerful when we consider the sequences of grid operations as the action space, where their permutations result in identical changes in the final topology.
APPROACH
We first present the state space, the action space, and the reward function modeled our approach. Then we briefly explain the unique challenge in Grid2Op and describe our approach tackle the challenge. Finally, we will describe the overall architecture of the RL agent.
3.1 MODELING STATES, ACTIONS AND REWARDS
State We also define the state in the Grid2Op environment as (T,X) where T is set of topology configuration (deterministically changed by action) and X as various features power demands and supplies (independent of the action), power being transmitted in each line (affected indirectly from the action) and on. The detail about the features of states used in this work is provided in Appendix A.I
Action We only consider bus assignment actions in our agent: we assume that it is desirable to have many lines connected possible since the overflow is less likely to occur when there many routes for the power delivery. Thus, for line switch actions, we simply follow the rule of always reconnecting the power lines whenever they get disconnected due the overflow.
Let us define the number of the substation as Noub and elements ith substation as Sub(i). Each end of lines, generators, and loads in the substation can be assigned one of two bushars, so the total number of actions is |A| INauut 2Sub(i) (i.e. each action chooses one of the substations and perform a bus assignment therein). Following the approach taken by the winner of the previous challenge L2RPN 2019 (Lan et al., 2019), we made our agent act (i.e. intervene) only in hazardous situations. The condition for being hazardous is determined by the existence of a line in which the power flow is larger than the threshold hyperparameter. This naturally yields a semi-MDP setting for RL (Sutton et al. 1999).
Reward We define the reward in intermediate time steps to be the efficiency of the power grid, defined by the ratio of the total load to the total production, i.e. prod, Note that if the ratio becomes greater than 1, the episode terminates with a large penalty for the failure since the production does meet the demand.
3.2 ACTOR-CRITIC ALGORITIM WITH AFTERSTATES
The main challenge of the Grid2Op environment is the large state and action spaces. For the power grid with 36 substations used in the L2RPN WCCI 2020 competition, there are about 70,000 actions that yield unique changes to the topology. We address this problem by adopting the actor-critic architecture, where the policy and the value function are represented by function approximators. In addition, we use the afterstate representation to capture many state-action pairs being led to an identical afterstate by leveraging the transition structure, shown in Figure I For notational simplicity, all the derivations assume MDP in this section, which shall be extended to the semi- MDP setting in the next section.
We use function approximators for the afterstate value function (s( and policy To (s)) parameterized by v and 0 respectively. The actor is trained to maximize J, and the critic to minimize Ly :
Jz (0) -EEr (.r (Vo as)]
4

---

Published as a conference paper at ICLR 2021
(v( = (o we)))D (Vff r(st, a1) - (o [V(lf (8t+1, (st+1;0-1)))) a+1))))
(5)
where the replay buffer D stores the transition tuple [st, r(8t, ar), 8++1] for off-policy learning. The actor and the critic are trained using Soft Actor-Critte Critic (SAC) (Haamojaa et al. 2018). Note that it learns a value function over an afterstate with a reconfigured topology, rather than a state-action pair, which is more succinet. Although the above equation defines state-value critic, we can still train off-policy since it is essentially an action-value critic (i.e. an afterstate is defined by state and an action).
Furthermore, we aim to apply the gradient estimator through a reparameterization trick similar to Haarnoja et al. (2018), since it is known to have lower variance than the likelihood ratio gradient estimator, resulting in stable learning. In order to update the actor via reparameterization trick, the transition f must be differentiable, but is not straightforward to define fA, which maps from the bus assignment actions to the topology configurations, as differentiable formula. In the next section, we will mitigate the problem by re-defining the action space.
3.3 EXTENSION TO GOAL-CONDITIONED HIERARCHICAL FRAMEWORK
is very challenging to take exploratory actions in the Grid2Op environment: if the agent takes random actions, the power grid would fail in few time steps. For example, the agent with the random policy would mostly fail in less than 10 time steps, whereas the agent with the no- op policy (naively maintaining the initial grid topology throughout time steps) would survive approximately 500 time steps on average. Thus, it is very difficult for the agent to explore diverse grid topology configurations that are significantly different from the initial ones, and thereby the random exploration policy (e.g. e-greddy) would be often stuck at bad local optima that executes only one or two actions. Therefore, more structured exploration a key to successful training.
To this end, we extend the afterstate actor-critic algorithm to a two-level hierarchical decision model by defining the goal topology configuration as the high-level action. Specifically, we define the high- level actions as the goal topology configuration g E {0, 1}" where n = Σi".. Sub(i), which learned by the high-level policy Th, This leads to the temporally extended afterstate representation, given by sft [Tedd gt, #r] = f ([T,,,)),, ge) where denotes the time a hazard occurs and denotes the time interval next hazard occurs. Note that we can now take full advantage of the afterstate representation since the equivalence of many different sequences of primitive actions (i.e. individual bus assignment actions) that lead to the identical topology are now captured by the goal topology configuration.
In addition, exploration with goal topology is more effective than with primitive actions since the policy only needs to focus where to go, i.e. the desirable topology under the current situation, without needing to care about how to get there, i.e. figuring out a suitable primitive action sequence that would yield the goal topology, with the help from an appropriate low-level policy. Finally, we can now use the reparameterization trick for the actor update in a straightforward manner since the result of f is merely a copy of the action ge-
The replay buffer D stores the transition tuple, [st, 9t, Tu:4dd; St+d] where Tur+d = Etr 7" "rr The high-level policy can be trained through the objective function of the actor and the critic written as:
J (0) Eg (V&s (1gt, re])]
(6)
Jv (v( = ED [(Vo(6')) Teatd - 7 "B gearan (e ]]
As for the low-level policy, it is relatively simple to find the action sequence that changes the current topology into the goal topology: we just need to identify the set of substations that requires changes in the bus assignment and make appropriate reassignments therein. Thus, we take a rule- based approach for the low-level policy, d, (se, 9t) where the rule determines the order of substations to execute bus assignment actions. For example, we could impose a priority on substations such that the substations with the least room in the capacity make their bus reassignment first because they are the ones requiring the most urgent interventions. In the experiments section, compare the results using various rules including learning- baased approach.
5

---

Published as a conference paper at ICLR 2021
Actor layer
It
Transformen
9t
Inpul grid
Graph modeling
Shared layer
Critic layer
m
V (s
T Sannssoon Be
9t
Figure 2: Overview of our model. The shared layer encodes 24, the actor layer outputs the desirable topology 9, given the current state s = [Tr,2,], Øz}. and the critic layer outputs the afterstate value given the afterstate o vT).
3.4 IMPLEMENTATION
In order to leverage the interconnection structure of the power grid, apply graph neural networks (GNN) (Scarselli et al. 2008). As illustrated in Figure 2] given the power grid with n substations, reshape 24 in the state 8t [Te, given
flat vector in Grid2Op, into (M, where M {0. 1) nx* is the adjacency matrix, and e, E R k is the node matrix with features. We adopted the transformer (Vaswani et al. (017)) as the GNN block, where the adjacency matrix M is used for masking out the attention weights of nodes, following the architecture proposed by byParisootto et al. (2020). The actor and the critic share the lower layers, consisting of GNN blocks and linear layers. Additionally, we add an entropy of policy to the objective function of the actor and the critic, following the SAC formulation. Details of the architecture provided in Appendix A.2
4 RELATED WORKS
The topology control of the power grid through line switch has been previously studied in the power systems literature. Previous works, (Fsher al. (2008) and Khodaei Shahidehpour (2010), solve the optimal transmission switching problem by mixed-integer programming. Since then, several heuristics have been introduced to tackle the computational cost (Fuller et al., 20122; Dehghanian et al.) 2015: Alhazmi et al. 2019). Recently, MMroot et al. (2018) explores bus assignment, more complex than the line switch, and presents an algorithm based on expert knowledge, which shows the utility of bus assignment. Their algorithm revert overflow with high probability of success and acceptable computational time. Han (2020) explores bus and line separation solve the problem of short circuit current reduction in power systems through RL. Marot et al. (2020) models the power grid management through line switch and bus assignment as RL task and releases open-source simulation called Grid2Op for power grid management in multi-step time horizons. Additionally. they held the international power grid management competition, L2RPN 2019 challenge, where JEEE- 14 is chosen for the competition environment, and Subramanian et al. (020) present simple deep RI. approach for IEBE-14.
find remedial bus assignment action that can
The winner of the 1,2PNN 2019 challenge (Lan al. 2019) tackles the problem through pre- training and guided exploration. They collect massive data sets from the simulator which can restore particular states, and pre-train an agent to generate a good initial policy. For exploration in the large action space, they use guided exploration instead of random exploration, where the agent simulates the top few actions with high action values before performs its action to the environment at every time step. They also design the agent to act only in hazardous situations, and they train it using dueling Deep Q-Networks (DQN) (Wang et al. 2016) and prioritized replay buffer (Schaul et al. 2016).

---

Published as a conference paper at ICLR 2021
Grid
Nuub
Nine
N gen
Moadd
[S|
[A]
71
IEEE-5
5
8
2
3
74
58
21
1EEE-14
14
20
6
11
194
160
57
L2RPN WCCI 2020
36
59
22
37
590
66810
177
Table 1: Characteristics of the grids. Neub. Ninner Ngens Noood is the total number of substations, lines, generators, and loads. [S] is the dimension of state, |.A] is the number of unitary bus assignment actions, n is the dimension of topology configuration.
The afterstate representation has been applied to address the resource allocation problems and dynamic routing problems. (Singh & Bertsekas (1996) formulate the dynamic channel allocation problem in the cellular network as a dynamic programming problem using the afterstate value function. More recently, there has been research utilizing the afterstate representation combined with ADP in a dynamic vehicle routing problem (Agussurja et al., 2019 Ulmer et al. 2019). [Shah] al.] (2020) also apply an afterstate-based deep RL method in ride-pool matching problem. The hierarchical framework has long held the promise to tackle complex RL tasks (Dayan & Hinton
1992 Parr & Russelll 1998 Barto & Mahadevan, 2003), and especially one of the prevailing approaches, the condiiindt hierarchical framework has recently achieved significant success various tasks, such as simulated and real- weal-world quadrupedal manipulation (Nachum et al., 2018a] 2020) and complex navigation (Levy et al., 2019, Zhang et al.) (2020). However, to the best of our knowledge, none of the works combines the afterstate representation with a hierarchical framework.
GNN has been widely used in deep RL to directly tackle graph-structured problems or to represent the interaction between entities in a state. You et al. (2018) formulates goal-directed graph generation as MDP and solves designing molecular structure with specific desired properties problem through an RL algorithm. WWang et al. (2018) apply GNN for continuous control by modeling controllable joints as nodes for graph, and the physical dependencies between joints as edges to capture underlying graph structure. Zambaldi et al. (2019) adopt GNN for a navigation and planning task where complex relational reasoning is required to represent pairwise interactions between objects in state, and Jiang et al. (2020) adopt it for learning cooperation in multi-agent environments by modeling agents as nodes graph where they communicate through GNN.
5 EXPERIMENTS
5.1 EXPERIMENTAL SETUP
Our experiments are conducted on the power grids, IEEE-5 (smallest), IEEE- and L2RPN WCCI 2020 (largest, used in the challenge), provided by Grid2Op. Details of each grid are provided in Table I Each grid has set of scenarios, and each scenario specifies the variations in the simulation such as the power supplies and demands at each time step. The length of each scenario is 864 time steps, which corresponds to 3 days at 5 minute time-resolution.
Since Grid2Op is relatively new to the research community, there are few RL methods applied to the grid topology control. Therefore we implement 3 baselines for performance comparison to verify the effectiveness of our method: (1) DDQN (Dueling DQN) has similar architecture as the last winner of the challenge, which learns the action-value function with the primitive action space (2) SAC is similar DDQN but utilizes maximum entropy exploration following SAC algorithm. (3) SMAACIAS is SMAAC without the afterstate representation, where we use action-value critic Q* g)- Thus, DDQN and SAC assume the MDP setting with primitive actions, SMAACIAS assumes the goal-conditioned semi-MDP setting but without the afterstate representation. We additionally compare our approach with the 3rd placed participant in the L2RPN WCCI 2020 grid (4) YZM, the only agent with publicly available code. This agent heuristically selects 596 actions among the primitive actions in prior and trains the agent with the reduced action space using Asynchronous Advantage Actor-Critic (Mnih et al., 2016). YZM additionally trains a backup agent
TThii algorithm is designed specifically for 1,2PNN WCCI 2020 grid and cannot he applied other grids.
7

---

Published as a conference paper at ICLR 2021
IEEE-5
IEEE-14
L2RPN WCC1 2020
100
100
20
50
0
0
-20
40
-40
-50
-60
SMAAT
20
-80
SMAACIAS
-100
SMAAGAA
-100
4
10
2
environment interactions (1o4]
environment Interactions (4)
environment Interactions (1e4)
Figure 3:
Training curves on 3 grids. Evaluation rollouts are performed every 1000 steps, and the
shaded area represents the standard error.
IEEE-5
1EEE-14
L2RPN WCCI 2020
SMAAC
98,18 0.31
69.66 = 10.62
55.26 5.82
DDQN
97.66 + 1.04
29.11 + 16.00
26.22 + 7.39
SAC
98.26 0.06
43.93 + 0.02
39.1 2.94
SMAAC(AS
93.79 + 0.58
14.91 + 14.08
11.58 + 2.74
YZM
28.34 + 4.31
Table 2: Performance measured by the scaled score on the 10 test scenarios averaged over 3 instances with standard error. Each the best policy obtained from the one with the highest performance in the validation scenarios during training.
with another set actions and invokes the backup agent when the base agent can lead to overflow or termination by using the simulation function
For fair comparison, all baselines except for YZM encode the input state through the same GNN architecture, and the agents get activated only in hazardous situations. The detail of implementation is provided in Appendix(A.3 AA. and the code is provided in
5.2 RESULTS
Figure 3 shows the total average scaled score of evaluation rollouts on the 10 validation scenario set during training: the scores are scaled in the range [-100,100], with the return of the no-op agent scaled and translated to 0. indicating how better the agent manages the power grid than the no-op agent terms of safety and power efficiency. Each algorithm was trained and evaluated for 3 runs for averaging the scores.
As shown in Figure all algorithms easily solve the smallest grid (IEEE-5). In the medium (IEEE- 14) and the large (L2RPN WCCI 2020) grids, both DDQN and SAC perform poorly. DDQN performs slightly better than the no-op agent the medium grid and worse than the no-op agent in the largest grid. Exploring with primitive actions is extremely difficult since most actions can lead to disastrous termination, and thereby it cannot find grids other than the initial one. This yields the DDQN to be stuck at bad local optima, not much better than the no-op agent. SAC performs slightly better than DDQN in the larger grids. This is due to the sophisticated optimization scheme in SAC that is shown affect number of other RL benchmark tasks. However, in Grid2Op. the performance was barely better than the no-op agent due to the same challenge faced by DDQN.
Perhaps surprisingly. the performance of SMAAC is no better than using primitive actions, although the hierarchical decision encourages deviating from the initial topology. Without the afterstate representation, the critic was not able to learn good action-value function due to the massive state and action spaces. YZM uniquely leverages the simulation function and can show good performance from the beginning. However, exploring primitive actions still hard even with
2The simulation function is predefined method in Grid2Op, which returns next state given action through approximate simulation.
8

---

Published as a conference paper at ICLR 2021
1 (ours)
2
3
5
6
Scaled score
75.72
66.21
48.62
26.60
17.98
4.31
0,07
CPU time (sec)
812.49
1406.45
1233.08
1322.02
116.43
96.56
118.58
Table 3: The top
leaderboard of the L2RPN WCCI 2020 Challenge among 50 participants.
the reduced set of actions, and it can be observed that YZM struggles to improve its performance. The performance on the test scenarios is provided Table 2 We provide qualitative analysis of how each agent behaves differently and how SMAAC remedies the hazardous power grid with detailed example in Appendix
On the contrary, our method learns significantly fast and outperforms all the baselines, effectively combining the benefits of the hierarchical decision model and the afterstate representation. Finally, Table 3 shows the leaderboard in the L2RPN WCCI 2020 challenge.
5.3 LOW-LEVEL RULE DESIGN
In this section, examine how the low-level policy affects the overall performance. (1) FIXED gives priority to substations randomly that predefined and fixed during training. We implement this low-level agent to find out whether our high- level agent manage the power network
Rules in IEEE-14
100
50
the poor low- level agent. (2) CAPA gives high priority to substations with
scaled
lines under high utilization of their capacity, which applies an action to substations that require urgent care. (3) DESC imposes a priority on large substations, i.e. many connected elements. A change a large substation can be seen making large change the overall topology with single action. (4) OPTI optimizes execution order by training, making the actor additionally output Neub values that represents the priority of substations. All rules achieve similar performance with overlapped confidence intervals except for FIXED.
-50
HXED
CAPA
-100
10
environment interections (1e4)
Figure 4: Comparison of 4 rules.
especially, CAPA converges fast compared to OPTI and DESC. hence we We assume that most of the rules could achieve similar final performances since SMAAC resilient to suboptimal low-level rules. By generating subgoals that include subset of intended topology reconfiguration, the high-level policy can adapt to suboptimal low-level rules to form an optimal policy overall. However, as the result of FIXED suggests, a very poorly designed lovel policy can lead to instability and degrade the performance.
As shown in Figure 4
use this low-level agent in Section 5.2
CONCLUSION
In this paper, we presented SMAAC, a deep RL approach demonstrated to be very effective for power grid management. SMAAC is actor-critic algorithm that combines the afterstate representation with a hierarchical decision model. This is very important for power grid management modeled by Grid2Op, where actions are too primitive for effective exploration and many permutations of action sequences lead to identical changes in the power grid topology. Besides, naive explorations with primitive actions are subject to immediate failure due the unique nature of power grid management. We empirically demonstrated that the presented method significantly outperforms several baselines in the real-world scale power grids, and ranked first in the latest international competition, 1. 1,2PN WCCI 2020 challenge. Our work shows the possibility of an intelligent agent that automatically operates the power grid for several days without expert help.
ACKNOWLEDGMENTS
This work was supported by the National Research Foundation (NRF) of Korea (NRF- 2019MBF2A1072238 and 22F=2001911A2C1087666 and the Ministry of Science and Information communication Technology (MSIT) of Korea (IITP No. 2019-0-00075, IITP No. 2020-0-00940 and UTTP No. 2017-0-01779 XAI).

---

Published as a conference paper at ICLR 2021
REFERENCES
Lucas Agussurja, Shih-Fen Cheng, and Hoong Chuin Lau. A state aggregation approach for stochastic multiperiod last-mile ride-sharing problems. Transp. Sci, 53(1):148-166, 2019. doi: 10.1287/trs..20 201c 8.0840. URL https::/// //doi. oww.10...1.... 1287 www.:12/8.....2000..... 2018 0840
M. Alhazmi, P. Dehghanian, S. Wang. and B. Shinde. Power grid optimal topology control considering correlations of system uncertainties. In 2019 IEEE/AA 55th Industrial and Commercial Power Systems Technical Conference (I CPS), pp. 1-7, 2019.
Andrew G. Barto and Sridhar Mahadevan. Recent advances in hierarchical reinforcement learning. Discrete Event Dynamic Systems. 13(1-2)441-7, Jamuary 2003, ISSN 0924-6703. doi: 10.1023/ A: A:1022140919877 URL LRL 44
Peter Dayan and Geoffrey E. Hinton. Feudal reinforcement learning In Advances in Neural Information Processing Systems 5, [NIPS Conference]. pp. 271-278, San Francisco, CA, USA, 1992. Morgan Kaufmann Publishers Inc. ISBN 1558602747.
Payman Dehghanian, Yaping Wang, Gurunath Gurrala, Erick Moreno-Centeno, and Mladen Kezunovic. Flexible implementation of power system corrective topology control. Electric Power Systems Research, 128:79-89 89, 11 2015.
A. L.. Dimeas and D. Hatziargyriou. Multi-agem reinforcement learning for microgrids. In IEEE PES General Meeting, pp. 1-8, 2010. doi: 10.1109/PES.2000555886.6.
J. Duan, D. Shi, R. Diao, H. Li, Z. Wang, B. Zhang, D. Bian, and Z. Yi. Deep-reinforcement- learning-base based autonomous voltage control for power grid operations. IEEE Transactions on Power Systems, 35(1):814-817 2020. doi: 10. N0.110//PWRS.2019.9000
D. Ernst, M. Glavic, and L. Wehenkel. Power systems stability control: reinforcement learning framework. IEEE Transactions on Power Systems, 19(1):427-435 2004. doi: 10.1109TTPWRS 2003.821457.
E. B. Fisher, R. P. O'Neill, and M. C. Ferris. Optimal transmission switching. IEEE Transactions on Power Systems, 2363):144-1335, 2008.
J. D. Fuller, R. Ramasra, and A. Cha. Fast heuristics for transmission-line switching. IEEE Transactions on Power Systems, 2773):1377 2012.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In Jennifer Dy and Andreas Krause (eds.), Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp. 1861-1870, Stockholmsmässan. Stockholm Sweden, 10-15 Jul 2018. PMLR.
Sangwook Han. Control Method of Buses and Lines Using Reinforcement Learning for Short Circuit Current Reduction. Sustainabiliry, 12(22) 188 November 2020. URL https://ideas.
a/ ottpeee......aaaaaaaaoooaaaaaaaa
Haochen Hua, Yuchao Qin, Chuantong Hao, and Junwei Cao. Optimal energy management
strategies for energy Internet via deep reinforcement learning approach. Applied Energy, 239 (C):598-609, 2019. doi: 10. ao..01//...penergy2001... URL https://idees repec. org/
html
Q. Huang, Huang, W. Hao, Tan, Fan, and Z. Huang. Adaptive power system emergency control using deep reinforcement learning. IEEE Transactions on Smart Grid, 1711)1171-1112,, 2020. doi: 10.1109/TSG.201
Jiechuan Jiang, Chen Dun, Tiejun Huang, and Zongqing Lu. Graph convolutional reinforcement learning. In International Conference on Learning Representations, 2020. URL https:// openreview wwtnnneview.net/eee wereet/oooummie Hww.okksSSSd
Adrian Kelly, Aidan O' Sullivan, Patrick de Mars, and Antoine Marot. Reinforcement learning for electricity network operation. ArXiv, abs/2003.07339, 2020.
10

---

Published as a conference paper at ICLR 2021
A. Khodaei and M. Shahidehpour. Transmission swithiiiiiiiissiitiiiiinnni in security unit commitment. IEEE Transactions on Power Systems, 25(4)1193-11945, 2010.
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Yoshua Bengio and Yann LeCun (eds.), International Conference on Learning Representations, 2015.
Tu Lan, Jiajun Duan, Bei Zhang, Di Shi, Zhiwei Wang, Ruisheng Diao, and Xiaohu Zhang, Ai- based autonomous line flow control via topology adjustment for maximizing time-series atcs. arXiv preprini arXiv: 111911..0466,, 2019.
Andrew Levy, Robert Platt, and Kate Saenko. Hierarchical reinforcement learning with hindsight. In International Conference on Learning Representations, 2019. URL net/ htte/oorum?idd ryzECoAcY?
A. Marot, B. Donnot, S. Tazi, and P. Panciatici. Expert system for topological remedial action discovery in smart grids. IET Conference Proceedings, pp. 43 (6 pp.)-33 (6 pp.)(l), January 2018. URL https: /tattps://diiiie //tgg//ggaaaaaaa wwiiiiaalllbbbrrr...laaaaaaaaaaa 10. 10.1009/cp.222 2018. 1875
org/ wttp////oooee/////eeeeeeeee
Antoine Marot, Benjamin Donnot, Camilo Romero, Luca Veyrin-Forrer, Marvin Lerousseau, Balthazar Donon, and Isabelle Guyon. Learning to run a power network challenge for training topology controllers. The Power Systems Computation Conference, 2020.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A. Rusu, Joel Veness, Mare G. Bellemare, Alex Graves, Martin Riedmiller, Andreas K. Fidjeland, Georg Ostrovski. Stig Petersen, Charles Beattie, Amir Sadik, Ioannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wierstra, Shane Legg, and Demis Hassabis. Human-level control through deep reinforcement learning. Nature, 518(7540):529-535 2015.
Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforcement learning. In Maria Florina Balean and Kilian Q. Weinberger (eds.), Proceedings of The 33th International Conference Machine Learning, pp. 1928-1937, 2016.
Ofir Nachum, Shixiang Gu, Honglak Lee, and Sergey Levine. Data-efficient hierarchical reinforcement learning. In Proceedings of the 32nd International Conference Neural Information Processing Systems, NIPS' 18, pp. 3307-3311311 Red Hook, NY, USA, 2018a. Curran Associates Inc.
Ofir Nachum, Shixiang (Shane) Gu, Honglak Lee, and Sergey Levine. Data-efficient hierarchical reinforcement learning. S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa- eesa-Bianchi, and R. Garnett (eds.), Advances in Neural Information Processing Systems 31, pp. 3303-3313. 2018b.
Ofrr Nachum, Michael Ahn, Hugo Ponte, Shixiang (Shane) Gu, and Vikash Kumar. Multi-agent manipulation via locomotion using hierarchical sim2real. In Leslie Pack Kaelbling, Danica Kragic, and Komei Sugiura (eds.), Proceedings of the Conference Robot Learning, volume 100 of Proceedings of Machine Learning Research, pp. 110-121, PMLR, 30 Oct-01 Nov 2020. URL http:::/procceeiiii////
ml.. wrwpspssss/..... http:////wwowo/aa/occseem www..prres...........
html
Emilio Parisotto, H Francis Song, Jack W Rae, Razvan Pascanu, Caglar Gulcehre, Siddhant M Jayakumar, Max Jaderberg, Raphael Lopez Kaufman, Aidan Clark, Seb Noury, et al. Stabilizing transformers for reinforcement learning. In Proceedings of The 37th International Conference on Machine Learning. 2020.
Ronald Parr and Stuart Russell. Reinforcement learning with hierarchies of machines. In Proceedings of the 1997 Conference on Advances in Neural Information Processing Systems 10, NIPS '97, pp. 1043-1049, Cambridge, MA, USA. 1998. MIT Press. ISBN 0262100762.
Warren B. Powell. Approximate Dynamic Programming: Solving the Curses of Dimensionality (Wiley Series in Probability and Statistics). Wilee--nterscience, USA, 2007.

---

Published as a conference paper at ICLR 2021
David Rolnick, Priya L. Donti, Lynn H. Kaack. Kelly Kochanski, Alexandre Lacoste, Kris Sankaran, Andrew Slavin Ross, Nikola Milojevie-Dupont, Natasha Jaques, Anna Waldman- Brown, Alexandra Luccioni, Tegan Maharaj, Evan D. Sherwin, S. Karthik Mukkavilli, Konrad P. Körding, Carla Gomes, Andrew Y. Ng. Demis Hassabis, John C. Platt, Felix Creutzig, Jennifer Chayes, and Yoshua Bengio. Tackling climate change with machine learning. CoRR, abs/1906.05433, 2019. URL http: pttp://arxiin org/ http/bbs/111000 00bb//190000433
Franco Scarselli, Marco Gori, Ah Chung Tsoi, Markus Hagenbuchner, and Gabriele Monfardini. The graph neural network model. IEEE Transactions on Neural Networks, 20(1))61 2008.
Tom Schaul, John Quan, loannis Antonoglou, and David Silver. Prioritized experience replay. In International Conference on Learning Representations, 2016.
Sanket Shah, Meghna Lowalekar, and Pradeep Varakantham. Neural approximate dynamic programming for on-demand ide-pooling. In The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2020, New York, NY, USA, February 7-12, 2020, pp. 507-515, AAAI Press, 2020. URL URL https::pp///.mmm //aaai/aaaaaa/aa otg/ojs////oooooooooooooooosro index. phpp /AAII wtt.a//aaatiiieeeie/e/e///
David Silver, Aja Huang, Chris J. Maddison, Arthur Guez, Laurent Sifre, George van den Driessche, Julian Schrittweeser, loannis Antonoglou, Veda Panneershelvam, Mare Lanctot, Sander Dieleman, Dominik Grewe, John Nham, Nal Kalchbrenner, Ilya Sutskever, Timothy Lillicrap, Madeleine Leach, Koray Kavukeuoglu, Thore Graepel, and Demis Hassabis. Mastering the game of Go with deep neural networks and tree search. Nature, 529(7587):48444555 489, 2016.
Satinder Singh and Dimitri Bertsekas. Reinforcement learning for dynamic channel allocation in cellular telephone systems. In Proceedings of the 9th International Conference on Neural Information Processing Systems, NIPS 96, pp. 974-980, Cambridge. MA, USA, 1996. MIT Press.
Medha Subramanian, Jan Viebahn, Simon Tindemans, Benjamin Donnot, and Antoine Marot. Exploring grid topology reconfiguration using a simple deep reinforcement learning approach. CoRR, abs/201 1. 13465, 2020. URL httph:t//aa/x:tt http:/awwiiv.....aaaaaaa abgg/bb////ecc 20110011110111
Richard S. Sutton and Andrew G. Barto. Reinforcement Learning: An Introduction. A Bradford Book, Cambridge, MA, USA, 2018.
Richard S. Sutton, Doina Precup, and Satinder Singh. Between mdps and semi-mdps: A framework for temporal abstraction in reinforcement learning. Artificial Intelligence, 112(1-2):181-2) 1999.
Marlin W. Ulmer, Barrett W. Thomas, and Dirk Mattfeld. Preemptive depot returns for dynamic same-day delivery. EURO Journal on Transportation and Logistics, 8(4):327 361, 2019. ISSN 2192-4376. doi: https://doi.org/111 10700000777711100000---000-00000 URL http:: / /www. sciencedi rect. com/ sci wnweeeceeeeeeee http://eeeeeeaaaaae/////////////
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems 30, pp. 5998-6008, 2017.
A. N. Venkat, I. A. Hiskens, J. B. Rawlings, and S. J. Wright. Distributed mpc strategies with application to power system automatic generation control. IEEE Transactions on Control Systems Technology, 16(6): 1206, 2008
Tingwu Wang, Renjie Liao, Jimmy Ba, and Sanja Fidler. Nervenet: Learning structured policy with graph neural networks. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 May 3, 2018, Conference Track Proceedingy. OpenReview.net, 2018. URL
net Sexrum?1d-SSSqMMECC
Ziyu Wang, Tom Schaul, Matteo Hessel, Hado Van Hasselt, Mare Lanctot, and Nando De Freitas. Dueling network architectures for deep reinforcement learning. In Proceedings of the 33th International Conference Machine Learning, pp. 1995-2003, 2016.
12

---

Published as a conference paper at ICLR 2021
Jiaxuan You, Bowen Liu, Zhitao Ying. Vijay Pande, and Jure Leskovec. Graph convolutional policy network for goal-directed molecular graph generation. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 31, pp. 6410-6421. Curran Associates, Inc., 2018. URL https::/proceeeding neuripuspp. rpss. cttpppppp/p.p////////pppp www.177778---pp...-- pdf
Vinícius Flores Zambaldi, David Raposo. Adam Santoro, Victor Bapst, Yujia Li, Igor Babuschkin, Karl Tuyls, David Reichert, Timothy Lillicrap, Edward Lockhart, Murray Shanahan, Victoria Langston, Razvan Pascanu, Matthew Botvinick, Oriol Vinyals, and Peter W. Battaglia. Deep reinforcement learning with relational inductive biases. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019.
Tianren Zhang, Shangqi Guo, Tian Tan, Xiaolin Hu, and Feng Chen. Generating adjacency- constrained subgoals hierarchical reinforcement learning. NeurIPS 2020, 2020.
Z. Zhang. D. Zhang, and R. C. Qiu. Deep reinforcement learning for power system applications: An overview. CSEE Journal of Power and Energy Systems, 6(1 (11::213-225 2020.
C. Zhao, U. Topcu, N. Li, and S. Low. Design and stability of load-side primary frequency control in power systems. IEEE Transactions on Automatic Control, 59(5):1177-1188, 2014.
13

---

Published as a conference paper at ICLR 2021
A APPENDIX
A.1 ENVIRONMENT DETAIL
Grid2Op provides a simulation for power grid operation in real-time over several days at 5-minute time-resolution. There are power grids, IEEE-S, IEEE-14, and L2RPN WCCI 2020 (a subgraph of IEEE-118), where each has different size.
We use 5 features provided by the environment and 1 feature defined by us, which we consider enough to represent the current state of the grid: Active power, rho, topology configuration, time step overflow, maintenance, hazard. The maintenance is boolean vector representing whether a line maintenance, and the hazard represent is also a boolean vector representing whether electricity flows of line is larger than a predefined threshold 8n, We use 0.9 for the threshold, and this threshold is same as the one we used for the hazardous state. Further details are provided in htm. html
State Space The state of the power grid consists of 12 features presented in Table 4
Action Space The agent can apply actions substations and lines to manage the power grid. The action on a substation, called bus assignment, assigns the elements the substation to a busbar. The action on a line, called line switch, disconnects (both ends of line is assigned to neither bus) a line or reconnects disconnected line. Let us define the number of lines in the power grid as Nime the number of substations Neub. and the number of elements in ith substation Sub(i). Then at each time step the agent selects an action a from the action space A where |A| = NNine + 2 x Naub 25h((1) 7
Nisne
Σ i=0
There are some rules that make the task more realistic and challenging in Grid2Op. Lines be automatically disconnected due to overflow of current, i.e. if more current flows than line can hold for 3 time steps, a line is automatically disconnected. There is cooldown time for each component, i.e. the agent cannot apply its action to the same component successively and it is reactivated after 3 time steps later. There is a stochastic event called maintenance that happens intermittently. During maintenance, a line is disconnected by force and cannot be reconnected.
Rule
Score The performance of agent be evaluated by score which consists of power loss penalty, failure penalty, and redispatching penalty, It is defined as
Score (prod: - loads) +
penalty +
redispatch,
(8)
t=taver
where tover is the time game occurs, prod, is total amount of power supply by all generators, load, is total amount of power demand by loads. prod, -load, stands for total amount of power loss. The failure penalty is given by the sum the large constant penalty for the remaining simulation time steps upon termination. The redispatching penalty is incurred when the agent do redispatching action, which controls generators produce more or less electricity. Since redispatching action always incurs the additional penalty, we do not consider this action, so the redispatching penalty is always 0 in our case. Therefore, the goal of the agent to operate the power grid both safely and efficiently by minimizing the failure penalty and the power loss penalty.
A.2 MODEL ARCHITECTURE
Given the state s [T, æ). we reshape x into (M, (). where M E [0,1) nx is the adjacency matrix, and à E R is the node matrix with & features. The shared layers consisted of L, GNN blocks that computes the node embedding of an input graph through transformer layers at their beginning. Given input nodes matrix R. a linear layer with ReLU activation increases input dimension & to embedding dimension hig. which maps & E Raxd to H E Tgn After the linear layer, I.,
2 Botth ends the line can be assigned one two buses in the substation, so 2 x Nise the number of reconnection. Each end of the line, generator, and load in the substation also can be assigned to one of two buses in the substation, so bus switching actions Σ Noub 2 2ub(()
14

---

Published as a conference paper at ICLR 2021
Name
Type
Size
Description
Date
int
6
The current year, month, day, hour of day, minute of hour, day of week. Active power magnitude.
Active power
float
Ngen + Niodd
+2 x Nisne
Reactive power
float
Ngen + Nioad
Reactive power magnitude.
+2 x Nisne
Voltage
float
Ngen + Nioad
Voltage magnitude.
+2 Nisne
Rho
float
Nisne
The capacity of each power line, which is defined as ratio between current flow and thermal limit.
Topology Configuration
int
N Ngen + Noad
For each element (load, generator, ends of line), it gives on which bus these elements is connected in its substation.
+2 x Nine
Line status
bool Nine
The status of each line, whether connected or disconnected.
Time step overflow
int Nisne
The number of time steps each line is overflowed.
Time before cooldown line
int Nisne
How much cooldown time for each line is left. An agent cannot perform its action on lines with cooldown.
Time before cooldown sub
int Nsub
How much cooldown time for each substation is left. An agent cannot perform its action on substations with cooldown.
Time next maintenance
int Nisne
The time of the planned maintenance. An agent cannot act on the lines with maintenance.
Duration
int
Nise
The number of time steps that the maintenance will last.
next maintenance
Table Details of features the state
transformer layers follow. The input of the transformer block at the 1th block is an embedding from the previous layer H'-1 and the adjaceney matrix A, H' Transformer (11'1 (I)
The actor's head consists of La transformer blocks and 2 linear layers. Given the final node embedding 4 from the shared layers, the transformer layers in actor's head takes it the input and outputs node embedding ' E Raxka, The first linear layer transforms 2D node embedding H to vector node embedding R" by reducing the embedding dimension ko 1, which is then concatenated with the current topology т to form the state and the next linear layers outputs mean and standard deviation of the normal distribution. We sample continuous values g' R from the normal distribution followed by tanh non-linearity and the desirable topology g E {0, 1}" constructed by assigning
values g' larger than predefined topology threshold ö, and otherwise. We empirically find out that an agent without the threshold has difficulty learning in the large grid. However, an appropriate threshold helps stable learning and fast convergence.
The critic's head has similar structure except for linear layers. There are L GNN blocks in the critic that also takes H and outputs H' E Raxk. After transforming it into a vector node embedding Th" by a linear layer, g' is concatenated to H and the following two linear layers take as the input and outputs a scalar value. The overall architecture is shown the Figure
Although the agent observes all substations, we reduce the goal dimension n to ñ. by restricting controllable substation. As we mentioned in inssubsection 3.1 we do not consider disconnection or reconnection. Therefore, the agent only controls substations that have more than 2 elements since there are only two possible cases, elements on a same bus (connection) or elements on a different bus (disconnection). For L2RPN WCCI 2020 grid, the agent acts substations that have more than 5 elements in order for fast convergence. As a result, the goal dimension is reduced from 21 to 16 in IEEE-5, from 57 42 in IEEE-14, 14, and from 177 to 79 in I. 1,2PNN WCCI 2020.
15

---

Published as a conference paper at ICLR 2021
Scenario1 of IEEE-14
Scenario2 OEEE
Scenariol of 128PN WCCI 2020
Scenario2 LRRPN WCCI 2020
25
20
15
3-
10
SNAAC
SMARCAAS
SAC
DDON
200
400
000
800
0
200
405
800
800
200
400
0.5
800
0
200
400
600
300
Mapps
stegs
ktaps
staps
Figure 5: Deviation from the initial grid. We measure distance between the initial topology and the current topology during rollout in 2 test scenarios for each grids.
A.3 IMPLEMENTATION DETAILS
For all models, we use state features, active power, tho, topology configuration, time step overflow, maintenance, and hazard, where dn 0.9. All the agent acts only when there rho of a line, ratio between current flow and thermal limit that is capacity of line, is larger than the 5, 0.9. Additionally the last states of a history are stacked to represent the input state, since the difference between the state at time step t and + 1 is not significantly different in Grid2Op. Since the first decimal place of reward does not change significantly (Loode varies from 0.85 to 0.99 most of time),
we transform it as (ood, x 10-9) 9) x 0.1 to use the second decimal place. Adam optimizer (Kingma & Ba 2015) is used for training with 5e - 5 learning rate and 128 batch size. We perform grid search find the best hyperparameters for each model.
SMAAC SMAAC is our proposed model, which learns the afterstate value function the goal space, namely topology configuration space. We use L, = 6 GNN blocks with embedding dimension k, 64/128 for shared layers. For actor's head, embedding dimension ki 64/128. For critic's head, we use L. = 1 GNN block with embedding dimension kn = 64/ /22 followed by linear layers with &.t- hidden units. We use 6, = 0/0.1 /0.15 In practice, the reward for the high-level policy in Equation 6 not discounted, Tr:++d Ei+d re and y used instead of ad following Nachum et al. (2018b). For the competition, we use k; 128, k. = 128, hi 128, 8- 0.35, and 7 is used as an extra input feature for the shared layers.
Lg 3 GNN blocks with
AS SMAAC AS is a baseline, which learns the action-value function of desired relative change in hierarchical framework. The overall architecture is similar SMAAC, but the critic takes both desired relative change and topology configuration to learn on state-action pairs. We use 8, = 0/0.1/0.15.
SAC SAC is a baseline, which learns the action-value function on the primitive bus assignment action space. We use Lg = 6 GNN blocks with embedding dimension k; = 64/128 for shared layers. For actor's head, use La 3 GNN blocks with embedding dimension k. 64/128. And, we use softmax to output categorical distribution while utilize relaxed categorical distribution in training. For critic's head, followed by concatenation with one-hot encoded action and one linear layer.
L., GNN block with embedding dimension k. 32/64/12
DDQN DDQN is a baseline, which learns the action-value function on the primitive bus assignment action space. It does not have separate actor but critic outputs |A| action-values. We use L, = 6 GNN blocks with embedding dimension kj = 64/128 for embedding layers. Following it, the critic has L. 1 GNN blocks with embedding dimension k2 64/128. Then, it utilizes the technique used by dueling DQN, namely we compute action-vall values through both value network and advantage network.
16

---

Published as a conference paper at ICLR 2021
Time step
Time step t+1
Time step L+2
0.96
11
0.89
11
0.86
G
5
0.16
0.79
0.20
0.26
12
9
9
0.00
0.42
13
8 0.05
13
8
0.83
13
Figure 6: Illustration of a part of IEEE-14 14 managed by our agent. Large circle refers to a substation, small circle with G or L refers to a generator or load, and dots in green yellow on substation refers 2 busbars. The numbers beside lines are line usage ratio of current to line capacity and the darker color of the line means large ratio.
Threshold 6n
Average scaled score
1.1
46,67 + 5.92
1.0
52.11 + 6.91
0.9
55.26 + 5.82 36.08 16.53
0.8
Table 5: Performance measured by the scaled score on the 10 test scenarios of L2RPN WCCI 2020 averaged over 3 instances with standard error. Each the best policy is obtained from the one with the highest performance in the validation scenarios during training.
A.4 QUALITATIVE ANALYSIS
In this section, we present qualitative analysis based on example behaviors of agents. Figure 5 indirectly shows how each agent behaves in 2 grids. In the medium grid, after our agent reaches to the certain grid, it keeps staying in that grid. We speculate that the agent finds the optimal grid during training where electricity could distribute evenly all the time so no further actions required. is reasonable in Grid2Op where single action can potentially destroy the grid. On the other hand, it shows different behavior in the large grid. It changes the topology configuration diversely to revert the hazardous situations since the agent cannot find the grid such as the one in the medium grid.
SAC, which shows the best performance among the baselines, only changes a few of the topology configuration by executing only one or two actions in both grids, since the initial topology is strong local optima Grid2Op. DDQN also shows similar behavior, but it shows worse performance than the SAC, since it changes the initial grid more. Likewise, SMAAC AS that changes the topology the most shows the worst performance. It is extremely difficult to find the hetter topology than the initial one in Grid2Op without the effective exploration. The efficient learning together with the effective exploration is the key to successful management.
We further examine how SMAAC learned to revert hazardous state back to the safe state. As shown in Figure 6) the line 4-5 (between substation 4 and 5) in / is in the hazardous situation. At time step + 1, our agent makes bus assignment change in the substation 12 by assigning line 12-13 to yellow busbar. As result, some amount of electricity that flows in the line 4-5 moves to line 4-3
meet the load demand in the substation 13 where electricity is supplied only from the line 8-13 due to the action in /++ 1. Then the last action at f + 2 that changes the substation 3 further disperses electricity from the substation 4. the end, more balanced distribution is achieved.
A.5 ACTIVATION OF THE AGENT
As mentioned in subsection 3.1 the agent acts only in hazardous situations, ie. there is line of which usage rate (ratio between current flow and thermal limit) larger than the threshold hyperparameter 5R. Note that the usage rate larger than 1.0 implies that line is overflowed.
17

---

Published as a conference paper at ICLR 2021
Table 5 shows how the final performance changes according to ò, in the test scenarios. If on is too high, c.g. 6n = 1.1, the agent may not be able to recover from the hazardous situation, and show relatively worse performance. On the other hand, the agent with 6G = 0.8 faces more diverse situations, requiring far more samples to reach the performance of the other agents with higher dn.
18


Reviews:
[{"review_id": "ph7fn3sknL", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 9, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Pros:\n\n- The paper presents an RL approach to train a model which may control a power grid. In that effort another actor-critic method coined as SMAAC is suggested which may become useful in other applications as well.\n- The authors used the concept of afterstates to reduce the huge state-space offered by L2RPN problem. They used Graph Neural networks along with the transformer based attention mechanism to achieve state of the art performance. \n- Formulating the hierarchical  decision model is very beneficial as it allows the exploration without violating the grid boundaries.\n- It is shown using the case study that the concept of afterstate has played crucial rule in the success of the model proposed. The same technique does not show good results with SMAAC\\AS, which is  SMAAC without afterstate. \n\nCons:\n- In section 3.1 it is said \"Thus, for line switch actions, we simply follow the rule always reconnecting the power lines whenever they get disconnected due to the overflow\" but this can also cause severe damage to the appliance. There is not enough detail provided on this, but I believe that re-connection must be done AFTER the grid state is improved so the appliances are not damaged.\n- According to Kundur 1994, a system goes through \"Alert state\" and \"Emergency state\" from normal state to get into \"In extremis state\".  The desired behavior would be to do restorative action in the \"Alert state\" to avoid any damage, if not possible then in \"Emergency state\". But authors have chosen to act only in hazardous situations (section 3.1). A little more discussion would be beneficial in this regard. Apparently, it does not seem difficult to act sooner, as one only has to reduce the width of allowed boundaries, but does this have any effect on exploration? that is not clearly discussed in the article.\n- There is a small spelling mistake in section 5.1 \"largest\" is written as \"largeest\"\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "Z6n7uTe8LkC", "reviewer": "AnonReviewer5", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary of the paper:\nThe paper introduced a new architecture and a new hierarchical reinforcement learning approach for the power grid control task (in form of the Power Network Challenge L2RPN). The paper starts with outlining the challenges of the problem, statespace size, action-set size and exploration issues. It then introduces the \"afterstate\" abstraction, which the authors argue is useful for the task because it should be more compact than the \"true\" MDP state representation. They then define how they model the resulting (semi) MDP, and how they adapt a Soft Actor Critic approach to their proposed hierarchical solution. In particular, their solution introduces a hierarchy in the policy space, where the higher level action is a representation of the desired power grid configuration, and the lower actions are the actual sequence of actions to achieve the desired outcome. The authors also argue that such a hierarchy helps with exploration, because exploration can mostly happen on the higher level only, reducing the explorative complexity. The paper then continues by introducing the function approximator used - a graph neural network with a transformer as the GNN block. The paper finishes by showing that the proposed approach outperforms several baselines by a big margin.\n\nCommentary on the goal of the paper:\n\nThe paper is highly application focused on very domain specific. However, given the overall relevance of the application - more efficient and sustainable power grids, I think that this limit in scope is perfectly justifiable.\n\nStrengths:\n- The paper clearly lays out the different challenges of the domain\n- the paper offers a well-justified, principled solution to each of the different challenges\n- the division of the action space in goal state and realization is highly appealing and could find application in other RL domains\n- the results are very strong\n- the authors show that their approach outperforms several others in an open benchmark\n\nWeaknesses:\n- the paper is a little difficult to follow if the reader is not intimately familiar with the domain; several key concepts of the domain are only explained late in the paper, but referenced quite early (in parcicular, it would be helpful to have a more thorough explanation of \"topology control\" earlier in the paper)\n- the paper has some minor editorial issues (typos; grammar; some citations are missing, in particular for soft actor Critic)\n- the paper convincingly argues that the problem should be modeled as an RL problem; since there are no other existing RL solutions to this domain, they create RL baselines that have not been explored in the literature. I find this misleading, as it suggests a \"strawman\" baseline. Given that the authors claim that their approach outperforms a supervised baseline in the competition, it would have been nice to have an SL baseline that is used to illustrate the differences between the two approaches.", "extra_scores": {}, "extra_text": {}}, {"review_id": "TjvFXQpgZrv", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes an effective method for managing power grid topology to increase efficiency. They use Transformer attention over a Graph Neural Network as the basic architecture, then propose a hierarchical technique in which the upper level learns to output goal network topologies, which are then implemented by a lower-level policy or a rule-based algorithm. An ablation study reveals that one of the most important components of the algorithm is using an \"afterstate\" representation, which learns a value function for the state after the agent changes the topology, but before the network is affected by random external factors, including supply and demand. \n\nStrengths:\n- The paper tackles a difficult, high-dimensional, real-world problem.\n- The architecture chosen is sophisticated and appropriate to the problem at hand. \n- The approach won the L2RPN WCCI 2020 competition, showing its effectiveness above other approaches in terms of both significant performance gains, and a significant reduction in computational complexity.\n- The experiments benchmark against a thorough set of baselines that use the same architecture, showing additional benefits for the proposed afterstate/topological approach. \n- The idea of the afterstate representation is interesting, and the authors provide an ablation study demonstrating its importance. \n- The discussion in Section 5.2 about why DDQN and SAC perform worse than a no-op policy was interesting. \n\nWeaknesses:\n- I was disappointed that this paper does not mention that an important benefit of increasing the efficiency of power grids through automation is to reduce energy consumption, and thereby reduce reliance on carbon intensive sources of energy, improve sustainability, and help fight climate change. These goals are mentioned in the description of the L2RPN WCCI challenge: https://competitions.codalab.org/competitions/24902, and are further discussed in https://arxiv.org/pdf/1906.05433.pdf. Instead of mentioning sustainability, the intro opens with \"The power grid [...] has become an essential component of modern society\". Because this paper describes the winning algorithm for the competition, it has the potential to be impactful. The authors should mention this potential benefit of their work, as it could help motivate other AI researchers to work on this important problem. If the authors can include a reference to the potential for increasing sustainability through better grid management, I would increase my score. \n- The clarity of the paper can be improved. For example, the intro talks about the difference between a state-action pair and an afterstate, but it is not apparent by that point in the paper what the difference is. Referencing topology vs. random external factors would be helpful here. Similarly, the terminology in Section 3.2 could be modified to improve clarity, perhaps by putting more emphasis on the fact that the paper essentially transforms the action space and learns a value function over topologies, rather than low-level actions. \n- More justification for the need for the reparameterization trick could be provided. \n", "extra_scores": {}, "extra_text": {}}, {"review_id": "17ElmqYr8Zf", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper reports an application of RL to a power station management problem. Notably this solution won a recent competition organized around this theme. More interestingly, it involves what seems to be an elegant representation that combines 2 ideas (afterstates and smdps) in a compelling way that I expect to have impact for other hierarchical RL problems.\n\nPros\n---\n-> Introduces a representation for a power grid management problem that I found elegant and appropriate for the problem. It has 2 parts: Use afterstates as many actions on the power network lead to the same afterstate by graph isomorphism. The second is the use of a semi-MDP. This combination seems to me to potentially be greater than the sum of its parts as the authors only partially discuss: The combinatorial explosion of the action space that sMDP's introduce are controlled by the afterstates. \n\n-> It won the L2RPN challenge, comparisons against standard algorithms like DDQN shows large gain.\n-> Exposition is mostly pretty clear.  I would have emphasized and developed more on the point I made in Pro#1 more.  \n\nCons\n---\n-> Curiously performance of other competitors were not presented. \n\n-> It is not entirely clear to me that the representational innovation (combining sMDPs+ above is in fact original. The authors seem somewhat timid in claiming it. Clearly delineating what the closest related work on this approach is would have helped. Perhaps authors can clarify in rebuttal.\n\nSection-wise comments\n---\n2.1: Given that this is an external competition, there is a presumption that this is a realistic/important problem setting. A full discussion of the connection to \"real\" power grid management is out of scope, but maybe a few cites on this topic would be helpful.\n\n   Also reading this paper, one wonders whether it could simply be generalized to a problem in dynamic graph routing i.e. nothing specific in the model about *power*\n\n3.1:\n   A little weirdness here where you say that the agent is designed to act in only hazardous situations which means its goal is roughly to keep the load ratio below 1. But the definition of reward would seem to encourage driving efficiency as high as possible, not just preventing hazard.\n\n\nsec 4:\n   as mentioned above, it would be useful (especially for ICLR audience) tot broaden related work beyond power management. RL on graph problems, other ideas to combine afterstates with heirarchical ?\n\nTable 2: Advisable to mention the metric used in the caption of the table, to make skimming the paper easy.\n\n5.2: The comparison to regular sMDP is very important, and drives home the value of afterstates as being crucial to making the heirarchical representation work.\n\n5.3: RAND seems like a bad name, since the order is randomized a priori than kept fixed, maybe call it FIXED?\n  Any analysis of why OPTI could not beat DESC?\n \n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: With reviewer scores of (7, 7, 9, 7), and with only one low-confidence score (R5's score of 7 with confidence of 2) it is obvious that the paper should be accepted.  
Decision: Accept (Spotlight)