Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: Summary of the paper:
The paper introduced a new architecture and a new hierarchical reinforcement learning approach for the power grid control task (in form of the Power Network Challenge L2RPN). The paper starts with outlining the challenges of the problem, statespace size, action-set size and exploration issues. It then introduces the "afterstate" abstraction, which the authors argue is useful for the task because it should be more compact than the "true" MDP state representation. They then define how they model the resulting (semi) MDP, and how they adapt a Soft Actor Critic approach to their proposed hierarchical solution. In particular, their solution introduces a hierarchy in the policy space, where the higher level action is a representation of the desired power grid configuration, and the lower actions are the actual sequence of actions to achieve the desired outcome. The authors also argue that such a hierarchy helps with exploration, because exploration can mostly happen on the higher level only, reducing the explorative complexity. The paper then continues by introducing the function approximator used - a graph neural network with a transformer as the GNN block. The paper finishes by showing that the proposed approach outperforms several baselines by a big margin.

Commentary on the goal of the paper:

The paper is highly application focused on very domain specific. However, given the overall relevance of the application - more efficient and sustainable power grids, I think that this limit in scope is perfectly justifiable.

Strengths:
- The paper clearly lays out the different challenges of the domain
- the paper offers a well-justified, principled solution to each of the different challenges
- the division of the action space in goal state and realization is highly appealing and could find application in other RL domains
- the results are very strong
- the authors show that their approach outperforms several others in an open benchmark

Weaknesses:
- the paper is a little difficult to follow if the reader is not intimately familiar with the domain; several key concepts of the domain are only explained late in the paper, but referenced quite early (in parcicular, it would be helpful to have a more thorough explanation of "topology control" earlier in the paper)
- the paper has some minor editorial issues (typos; grammar; some citations are missing, in particular for soft actor Critic)
- the paper convincingly argues that the problem should be modeled as an RL problem; since there are no other existing RL solutions to this domain, they create RL baselines that have not been explored in the literature. I find this misleading, as it suggests a "strawman" baseline. Given that the authors claim that their approach outperforms a supervised baseline in the competition, it would have been nice to have an SL baseline that is used to illustrate the differences between the two approaches.
review_title: A good paper with a very narrow scope

final_score: rating: 7: Good paper, accept
review_content: This paper reports an application of RL to a power station management problem. Notably this solution won a recent competition organized around this theme. More interestingly, it involves what seems to be an elegant representation that combines 2 ideas (afterstates and smdps) in a compelling way that I expect to have impact for other hierarchical RL problems.

Pros
---
-> Introduces a representation for a power grid management problem that I found elegant and appropriate for the problem. It has 2 parts: Use afterstates as many actions on the power network lead to the same afterstate by graph isomorphism. The second is the use of a semi-MDP. This combination seems to me to potentially be greater than the sum of its parts as the authors only partially discuss: The combinatorial explosion of the action space that sMDP's introduce are controlled by the afterstates. 

-> It won the L2RPN challenge, comparisons against standard algorithms like DDQN shows large gain.
-> Exposition is mostly pretty clear.  I would have emphasized and developed more on the point I made in Pro#1 more.  

Cons
---
-> Curiously performance of other competitors were not presented. 

-> It is not entirely clear to me that the representational innovation (combining sMDPs+ above is in fact original. The authors seem somewhat timid in claiming it. Clearly delineating what the closest related work on this approach is would have helped. Perhaps authors can clarify in rebuttal.

Section-wise comments
---
2.1: Given that this is an external competition, there is a presumption that this is a realistic/important problem setting. A full discussion of the connection to "real" power grid management is out of scope, but maybe a few cites on this topic would be helpful.

   Also reading this paper, one wonders whether it could simply be generalized to a problem in dynamic graph routing i.e. nothing specific in the model about *power*

3.1:
   A little weirdness here where you say that the agent is designed to act in only hazardous situations which means its goal is roughly to keep the load ratio below 1. But the definition of reward would seem to encourage driving efficiency as high as possible, not just preventing hazard.


sec 4:
   as mentioned above, it would be useful (especially for ICLR audience) tot broaden related work beyond power management. RL on graph problems, other ideas to combine afterstates with heirarchical ?

Table 2: Advisable to mention the metric used in the caption of the table, to make skimming the paper easy.

5.2: The comparison to regular sMDP is very important, and drives home the value of afterstates as being crucial to making the heirarchical representation work.

5.3: RAND seems like a bad name, since the order is randomized a priori than kept fixed, maybe call it FIXED?
  Any analysis of why OPTI could not beat DESC?
 

review_title: A good application of RL showing value of a novel representation

final_score: rating: 7: Good paper, accept
review_content: This paper proposes an effective method for managing power grid topology to increase efficiency. They use Transformer attention over a Graph Neural Network as the basic architecture, then propose a hierarchical technique in which the upper level learns to output goal network topologies, which are then implemented by a lower-level policy or a rule-based algorithm. An ablation study reveals that one of the most important components of the algorithm is using an "afterstate" representation, which learns a value function for the state after the agent changes the topology, but before the network is affected by random external factors, including supply and demand. 

Strengths:
- The paper tackles a difficult, high-dimensional, real-world problem.
- The architecture chosen is sophisticated and appropriate to the problem at hand. 
- The approach won the L2RPN WCCI 2020 competition, showing its effectiveness above other approaches in terms of both significant performance gains, and a significant reduction in computational complexity.
- The experiments benchmark against a thorough set of baselines that use the same architecture, showing additional benefits for the proposed afterstate/topological approach. 
- The idea of the afterstate representation is interesting, and the authors provide an ablation study demonstrating its importance. 
- The discussion in Section 5.2 about why DDQN and SAC perform worse than a no-op policy was interesting. 

Weaknesses:
- I was disappointed that this paper does not mention that an important benefit of increasing the efficiency of power grids through automation is to reduce energy consumption, and thereby reduce reliance on carbon intensive sources of energy, improve sustainability, and help fight climate change. These goals are mentioned in the description of the L2RPN WCCI challenge: https://competitions.codalab.org/competitions/24902, and are further discussed in https://arxiv.org/pdf/1906.05433.pdf. Instead of mentioning sustainability, the intro opens with "The power grid [...] has become an essential component of modern society". Because this paper describes the winning algorithm for the competition, it has the potential to be impactful. The authors should mention this potential benefit of their work, as it could help motivate other AI researchers to work on this important problem. If the authors can include a reference to the potential for increasing sustainability through better grid management, I would increase my score. 
- The clarity of the paper can be improved. For example, the intro talks about the difference between a state-action pair and an afterstate, but it is not apparent by that point in the paper what the difference is. Referencing topology vs. random external factors would be helpful here. Similarly, the terminology in Section 3.2 could be modified to improve clarity, perhaps by putting more emphasis on the fact that the paper essentially transforms the action space and learns a value function over topologies, rather than low-level actions. 
- More justification for the need for the reparameterization trick could be provided. 

review_title: Interesting applied work; please mention sustainability in the paper

final_score: rating: 9: Top 15% of accepted papers, strong accept
review_content: Pros:

- The paper presents an RL approach to train a model which may control a power grid. In that effort another actor-critic method coined as SMAAC is suggested which may become useful in other applications as well.
- The authors used the concept of afterstates to reduce the huge state-space offered by L2RPN problem. They used Graph Neural networks along with the transformer based attention mechanism to achieve state of the art performance. 
- Formulating the hierarchical  decision model is very beneficial as it allows the exploration without violating the grid boundaries.
- It is shown using the case study that the concept of afterstate has played crucial rule in the success of the model proposed. The same technique does not show good results with SMAAC\AS, which is  SMAAC without afterstate. 

Cons:
- In section 3.1 it is said "Thus, for line switch actions, we simply follow the rule always reconnecting the power lines whenever they get disconnected due to the overflow" but this can also cause severe damage to the appliance. There is not enough detail provided on this, but I believe that re-connection must be done AFTER the grid state is improved so the appliances are not damaged.
- According to Kundur 1994, a system goes through "Alert state" and "Emergency state" from normal state to get into "In extremis state".  The desired behavior would be to do restorative action in the "Alert state" to avoid any damage, if not possible then in "Emergency state". But authors have chosen to act only in hazardous situations (section 3.1). A little more discussion would be beneficial in this regard. Apparently, it does not seem difficult to act sooner, as one only has to reduce the width of allowed boundaries, but does this have any effect on exploration? that is not clearly discussed in the article.
- There is a small spelling mistake in section 5.1 "largest" is written as "largeest"

review_title: Great work

Meta-review: With reviewer scores of (7, 7, 9, 7), and with only one low-confidence score (R5's score of 7 with confidence of 2) it is obvious that the paper should be accepted.  
Decision: Accept (Spotlight)