MARL2Grid-TR: A Multi-Agent RL Benchmark in Power Grid Operations
Authors: Enrico Marchesini
Eva Boguslawski
Alessandro Leite
Christopher Amato
Matthieu DUSSARTRE
Marc Schoenauer
Benjamin Donnot
Priya L. Donti
Venue: ICLR 2026 Poster (2026)
Keywords: Multi-agent reinforcement learning
benchmark
power grids
Primary area: datasets and benchmarks
TL;DR: We introduce MARL2Grid, the first benchmark for multi-agent reinforcement learning in realistic power grid operations.

Abstract: Improving power grid operations is essential for enhancing flexibility and accelerating grid decarbonization. Reinforcement learning (RL) has shown promise in this domain, most notably through the Learning to Run a Power Network (L2RPN) competition series, but prior work has primarily focused on single-agent settings, neglecting the often decentralized, multi-agent nature of grid control.
We fill this gap with MARL2Grid-TR, the first multi-agent RL (MARL) benchmark for grid topology and redispatching, developed in collaboration with transmission system operators. Built on RTE France’s high-fidelity simulation platform, our benchmark supports decentralized control across substations and generators, with configurable agent scopes, observability settings, expert-informed heuristics, and safety-critical constraints.
The benchmark includes a suite of realistic scenarios that expose key challenges, such as coordination under partial information, long-horizon objectives, and adherence to hard physical constraints. Empirical results show that current MARL methods struggle under these real-world conditions. By providing a standardized, extensible platform, we aim to advance the development of scalable, cooperative, and safe learning algorithms for power grids.

Published as a conference paper at ICLR 2026
MARL2GRID-TR: A MULTI-AGENT RL BENCHMARK IN POWER GRID OPERATIONS
Enrico Marchesini' Eva Boguslawski
Alessandro Leite*
Christopher Amato Priya L. Donti
Matthieu Dussartre
Mare Schoenauer
Benjamin Donnot
Massachusetts Institute of Technology, Cambridge (MA), USA
TAU, INRIA and LISN (CNRS & Univ. Paris-Saclay), Orsay, France
RTE Réseau de Transport dl d'électricité, Puteaux, France
INSS Rouen Normandy, Univ. of Rouen Normandy, LITIS UR 4108, Rouen, France 5 Northeaster University, Boston (MA), USA
ABSTRACT
Improving power grid operations is essential for enhancing flexibility and accel- erating grid decarbonization. Reinforcement learning (RL) has shown promise in this domain, most notably through the Learning to Run a Power Network (L2RPN) competition series, but prior work has primarily focused on single-agent settings, neglecting the often decentralized, multi-agent nature of grid control. We fill this gap with MARL2GRID-TE TR, the first multi-agent RL (MARL) benchmark for grid topology and redispatching, developed in collaboration with transmission system operators. Built on RTE France's s high-fidelity simulation platform, our benchmark supports decentralized control across substations and generators, with configurable agent scopes, observability settings, expert-informed heuristics, and safety-critical constraints. The benchmark includes a suite of realistic scenarios that expose key challenges, such as coordination under partial information, long- horizon objectives. and adherence to hard physical constraints. Empirical results show that current MARL methods struggle under these real-world conditions. By providing standardized, extensible platform, aim advance the development of scalable, cooperative, and safe learning algorithms for power grids.
INTRODUCTION
Power grid operations are undergoing profound transformation to meet the global demands of decarbonization. The rapid rise of variable renewable energy (VRE) sources such wind and solar requires unprecedented levels of operational flexibility and reliability. To keep the lights on while integrating VRE at scale, system operators must increasingly rely on two families of control mechanisms: (i) topology optimization, which reconfigures grid connectivity to mitigate equipment failures; and (ii) redispatching and curtailment, which adjust generators and storage units' outputs to balance supply and demand in real time. These actions play an essential role in modern grid control. However, these actions difficult (or functionally unfeasible in the topological case) for human operators and traditional optimization-hn based solvers to properly handle, especially under VRE's uncertainty, and given flexible load profiles and long operating horizons (Marot al., 2022b).
Safe and efficient grid control thus requires solving a complex, high-dimensional problem in real time. Figure I clarifies the setup with simplified four-substation grid operated by two agents and interconnected by transmission lines (edges). Generators and loads are connected to buses within substations, and the power generated at each bus, which be redispatched or cur- tailed, flows through the network to meet demand-thh total amount of power required by the loads. Substations typically contain multiple buses that can be reconfigured via topological modifications
-maiing
" "Equal contribution; emarche@mit. edu, eva wwa.boggsswwstiiireerrrrere I Equal advising; dont i @mi edu,, benjammin.domott@rtee... www.oodrte--ee-ooooooo www.rrrttefffcceoomm "oode is available https: http///ppeeveeeeeeeeeeeeeeeeeeee net wwwp:/eto/oooommmmmmmmaam

---

Published as a conference paper at ICLR 2026
to modify the power flow. Both actions subject to many physical and operational constraints: generators have ramping constraints, transmission lines have thermal capacities, and substations have switching restrictions. Violating these constraints risks blackouts or costly economic losses.
Through the "Learning to Run a Power Network competition series" (L2RPN) (Marot et al., 2022b) and the recent RL2GRID benchmark (Marchesini et al., 2025), reinforcement learning (RL) has emerged as a promising paradigm for tackling grid control. However, these works model the problem as a single-agent task. In contrast, real-world grids are divided across multiple operators, and even within a single operator's area, the system can be decentralized. This motivates a multi-agent RL (MARL) perspective, where multiple RL agents act on different parts of the grid (U.S. DoE, 2024). This decentralization is key for dealing with the speed and scale required to man- age large amounts of VRE and flexible loads.
23GAEW
Pay-
TRMFY
My
-15AMW
-1IZWW
-200WWWW
1322OW
We present the first MARL benchmark for power grid topology and redis- patching control, namely MARL.2GRID-TR. De- signed in collaboration with transmission system operators (TSOs) and built on the French TSO's power simulation framework (Donnot, 2020), our benchmark captures the cooperative nature of power grids. Each agent controls a subset of substations and cooperates with others to satisfy demand while maintaining grid stability. To reflect modern grid challenges, we provide multiple scenarios with different action spaces (i.e., discrete topological or continuous redispatching and curtailment actions) that scale in size and number of agents, and support various observability regimes, from fully centralized to strictly local, where agents observe only what they control. We also incorporate multi-agent heuristic "idle" transition scheme to simplify the problem horizon under normal grid operations, and include safety-critical constraints such as load shedding, islanding, and line overloads. MARL2GRID-TR thus contributes: (i) A standardized suite of MARL tasks for discrete or continuous grid control; (ii) a PETTINGZoO interface (Terry al., 2021), with (optional) heuristic-based transitions and constrained formalizations; and (iii) reference implementations of popular baselines for reproducible evaluation and comparison.
188AWW
140
Appic
Figure I: Toy example of a grid controlled by two agents, where all transmission lines have maximum capacities of 210 M...W A "bus split" topological (discrete) action addresses an overloaded line (red).
Overall, MARL2GRID-TT introduces high-fidelity MARL benchmark for real power grids, pro- viding a foundation for developing the next generation of scalable, cooperative, and safe algorithms.
2 PRELIMINARIES AND RELATED WORK
Table I shows existing environments for studying RL in energy contexts. Most prior efforts target simplified settings, such as small-scale grids, low-voltage microgrids, or building districts (Chen et al.. 2022). For instance, 2020), GYM-ANM focuses on network management in distribution systems (Henry Ernst, 2021), and the ARPA-E Grid Optimization competition focuses offline optimization rather than online decision-making with RL (ARPA-E, 2023). Recently, RL2GRID (Marchesini et al., 2025) has es- tablished a standardized RL benchmark for grid control based on the French TSO's Grid2Op, high-fidelity power simulation framework (Donnot, 2020). Grid2Op captures crucial complexities
models microgrid-leve dynamics (Henri et al.,
Table 1: Comparison of RL benchmarks for power grid operations. MARL2Grid the only frame- work supporting large-scale realistic multi-agent settings for grid control with safety constraints.
Benchmark Environment
Scale Multi-agent Topology
Redisp. Curt. Constraints
PYTHON-MICROGRID (Henri al., 2020)
Small
X
GYM-ANM (Henry Emstt 2021)
Small
X
L2RPN (Marot al., 2020b)
Large
RL.2GRID (Marchesini et al., 2025)
Large
/
- TR (ours)
Large
/
2

---

Published as a conference paper at ICLR 2026
of real grids (e.g., non-linear power flows, uncertainty from VRE, and operational constraints), and has also served as the backbone for the L2RPN competitions, which establish RL as a promising solution for grid control. However, both RL2Grid and L2RPN adopt single-agent formulation that abstracts away the decentralized control structure of real transmission systems. Hence, they do not support varying observability regimes coordination among multiple agents, which are essential features for scalable and practical deployment in realistic power grids. MARL2GRID- TRR builds directly on the popular and realistic Grid2Op power simulation framework address these gaps.
2.1 MULTI-AGENT REINFORCEMENT LEARNING
We model MARL2Grid tasks as malti-agent Markov decision processes (MMDPs) (Boutilier, 1996) defined by the tuple (N, S, JeN, P, R, 7), where A is a finite set of agents, S is a finite set of states, A' is set of actions for agent i, P defines the transition dynamics over joint actions (A' + RR is the joint reward function, and y E [0. 1) is the
a = (a'...., ), R S x discount factor. At each time step, each agent selects an action based on its available information, and all agents cooperate to maximize the expected discounted return Ex Ex|E== R'sss az]], where denotes the joint policy. When agents can only observe information related to the substations they control, we extend the previous definition to a decentralized partially observable MDP (Dec-POMDP) (Oliehoek & Amato, 2016) with (fO') hee,, (ENN), where O' is the local observation space of agent and O S x {A') - A(O') defines its observation distribution. Each agent conditions its policy n' on the local accioonobseevatton history h' (or, depending on the degree of partial observability, on its own observation of E O³), maintaining the same objective.
=
Algorithms. A central paradigm in cooperative learning systems is centralized training with decen- tralized execution (CTDE), where agents leverage privileged information and centralized estimators during training while maintaining decentralized policies for deployment (Lowe et al., 2017). In value-based MARL, CTDE is often implemented through value factorization, where a centralized value function is decomposed into agent-wise utilities to guide coordination. Prominent examples include QMIX (Rashid et al., 2020) and QPLEX (Wang et al., 2021), the latter being widely adopted as strong baseline (Papoudakis et al., 2021). CTDE has also been applied in policy-gradient meth- ods. Algorithms such as MASAC and MAPPO (Bettini et al., 2024) employ centralized critics to stabilize learning and (potentially) improve performance, with MAPPO typically outperforming more complex approaches (Yu et al., 2022). Motivated by their widespread adoption and empirical success, MARL2GRID-THR TR includes QPLEX and MAPPO as representative baselines.
Power grid operations also come with safety constraints. Constrained MARL equips each agent i W with a set of auxiliary cost functions that capture constraint violations (Gu et al., 2021). Agent i maintains set of m' cost functions C == {cj JeN
Je(.....m where each cj S x {AA hee) + [0, 1] measures the occurrence of safeta-critical events such line overloads load shedding. Af- ter executing a, at time f, agents receive both task rewards and cost signals (st, The ob- jective is to maximize the expected return while ensuring that the cumulative discounted cost (j(T)) := Ir, [_>.o' ? (sc) a:)] S 1 Vj E {I,...,,m? m²) } remains below threshold 1 for every agent and cost index j. In practice, solving a constrained problem directly is difficult, causing most approaches to rely on Lagrangian relaxation. Dual variables are introduced to balance constraint sat- isfaction against reward maximization. Among these methods, Lagrangian MAPPO (LagrMAPPO) (Ling et al., 2022) has emerged as a strong baseline due to its simplicity, stability, and effectiveness across cooperative benchmarks (Ling et al., 2022; Aydeniz et al., 2024). For this reason, we adopt LagrMAPPO as our primary constrained baseline in 2ARD2GRIDDTR.
MARL2GRID
In multi-agent power grid operations, each agent generally acts on a subset of substations and must coordinate with others to ensure stable long-term operation. The episodes span from one simulated week to one month, and the agents make decisions at 5 5-minute intervals. At each step, an agent acts on its substations and observes global local information (based on the selected level of observabil- ity within the environment), contributing to the joint objective of maintaining safe and uninterrupted power delivery despite fluctuating demand, equipment failures, and physical constraints.
3

---

Published as a conference paper at ICLR 2026
Table 2: List of base grid environments and contingencies currently supported by MARL2GRID.
ID
Maintenance
Opponent
Subs.
Lines
Gens.
Loads
Ep. Length (steps)
Staat
bus14
14
20
11
8064
473
bus36
36
59
37
8064
1266
bus118
118
186
62
99
2017
4460
Environments. MARL2GRID-TH builds three Grid2Op power grids (referred to base grids). Table 2 summarizes their structure, and the number of substations, lines, and generators. Each base grid follows double bus architecture, meaning that every electrical component (generators, loads, and transmission lines) can connect to one of two buses within a substation. Some environments include Batteries (B), which can function both as generators (discharging) and loads (charging) in the continuous tasks. Environments also present operational contingencies designed to capture the disruptions faced by TSOs: Maintenance (M): Scheduled outages that agents can observe. During maintenance, transmission line is disconnected and remains unavailable until the maintenance window ends. (ii) Opponent (O): Unpredictable disturbances (e-g.. weather events) causing sudden line disconnections. These events are unobserved in advance, requiring agents to react in real time. disconnected line enters a cooldown period during which reconnection is not allowed.
Each grid in MARL2GRID-TR is partitioned among agents using the segmentation methods of Henka et al. (2022). Agents are assigned control over regions of the grid with strong internal con- nectivity and limited external interactions. This choice mirrors how TSOs structure control zones in practice, making our benchmark more realistic. The resulting 18 grid is in Table while we refer to Appendix C for the remaining grid configurations. At the same time, MARL2GRIDDTTR is designed to be flexible. Users can modify configuration files redefine zone assignments and explore alternative setups. Hence, the framework also supports a fully decentralized regime where every substation is controlled by its own agent. This config- uration allows researchers study the limits of coordination and scalability under higher agent counts. By supporting different configurations, MARL2GRID-TE facilitates the study of trade-offs between control and communication granularity, coordination complexity, and learning performance in multi-agent grid operations.
to-get assignment for the
Transition dynamics. Each environment transition driven by realistic yet synthetic time series of demand and generation, generated using ChroniX2Grid (Marot et al., 2020a). At the beginning of an episode, a random timestamp is sampled to initialize the grid, ensuring exposure to varied sea- sonal and temporal conditions. The environment then evolves step by step in process that mirrors real grid operations: (i) Exogenous stochastic events (e.g.. weather-induce
faults) are triggered ac- cording Grid2Op's predefined probabilistic models. (ii) Agents jointly execute their topological or redispatching and curtailment actions. (iii) The system updates cooldown counters and applies any scheduled maintenance events. (iv) Grid2Op's AC power flow solver computes the new system
Table 3: Agent--o-substatioon assignments and dimensionality of the busl 18 grid. (T stands for the topological case, R for the redispatching and curtailment one.)
Grid
Agent
Controlled Substations (IDs)
Lines
Gens.
Loads
|Obs (T/R)
|Actions (T/R)
[0-13, 116]
23
12
281 187
414/5
[14, 16-18, 29, 32, 37]
18
5
140 / 121
377 /3
2
[33-36]
10
1
3
61 51
73/
[38 41, 48]
18
5
155 127
65706 /
[42-47]
10
6
84 146
52/2
bus 1
[49-63, 65, 66]
32
14
249 119
1375 13
6
[23, 64, 68-72]
18
4
1
225
7
[67, 73-80, 115, 117]
24
6
8
218 163
2121 /
8
[81-101]
33
10
17
431 269
2640 / 10
[102-111]
15
186 243
145
[19-22, 24-28, 30-31, 112-114)
20
11
166 126
19515
11
[0- 1171 1 (redispatching agent for R)
186
62
99
-/ 1/33
-/20
We Grid2Op's grids data, spanning up to several years and covering various conditions.
4

---

Published as a conference paper at ICLR 2026
state. If the configuration is infeasible due to islanding unmet demand- the episode terminates. Otherwise, overloaded lines are monitored, and those exceeding limits for more than three consecu- tive steps automatically disconnected. (v) Finally, all grid variables (i.e., the state) updated, capturing the nonconvex, nonlinear, and stochastic dynamics of power systems. Depending on the observability regime, agents then receive either the full state or local observations.
Action space. Each base grid has two classes of tasks based on the selected action space.
For topology optimization (discrete action space), each agent modify the topology of the sub- stations it controls. Table 3 shows agent-substation assignments and dimensionality for the bus/ 18 grid. Agents can perform types of decisions: (i) switching the status of transmission lines (i.e., connecting or disconnecting them), and (ii) reassigning electrical components to one of the two buses within substation. While these operations correspond to simple remote switch commands in real power grids, they result in a high-dimensional space. Line switching introduces discrete action per line, whereas bus reassignments (or "uus--plitting") yield an exponentially large number of valid actions. The total number of discrete actions at double-bus substation with Ninns lines, N, generators, and N₁ loads is given by (Chauhan et al., 2023): N 2 2 +N N₂+ N -1 For example, substation #5 Figure 2, which contains 2 generators, load, and lines (7 elements total), has 63 distinct topological configurations- each representing unique combination of bus assignments. In larger grids such as bus36, a single substation can exceed 65,000 valid actions for a single agent. This combinatorial explosion makes traditional optimization approaches intractable and underscores the need for advanced MARL methods.
For redispatching and curtailment (continuous action space), the objective is to balance total genera- tion and demand at every time step. To reflect real-world operations, MARL2GRID- -RR introduces mixed agent structure, where: (i) decentralized agents manage the curtailment of renewable gener- ators and the charging/discharging of storage units within their areas, and (ii) a global redispatching agent adjusts the outputs of the other generators across the grid. The action space dimensionality thus scales linearly in the number of generators and storage units. For example, the action space size for the bus 18 grid is N Needssp + -Neun + N 69, where Neeiss the number of redispatchable generators, Neun the number of renewable generators, and Ns the number of storage units.
State space. The features of the state vector that are shared between the discrete and continuous tasks are listed in Table 4, including generator outputs, load demands. transmission line status and capacities. 3 In a centralized setting, each agent has access to the state (whose dimensionality reported in Table 2). In decentralized setting, agents observe only data corresponding to the sub- stations they directly control. Neighboring agents share partial information for lines that connect their substations. This decentralized structure better mirrors the realities of transmission system op- erations, where control centers operate with limited observability and coordination. Crucially, our codebase enables users to flexibly configure observability regimes for any base grid, allowing them to extend MARL2GRID- -RR and study coordination and learning under different paradigms.
The objective in grid operations is to ensure long-temn safety and efficiency. For topology optimization, MARI R.RGGRD---- adopts the reward design of Marchesini et al. (2025), developed in consultation with TSOs. It balances three components: R = a R Roveroadd + 1 Reas,, where a, 3, and 1 are weights specified in Appendix E. The three terms respectively encour- age survival, penalize overloads, and account for economic costs (formal definitions are provided
Reward function.
Table 4: List of features composing the state of power grid that are shared between the discrete and continuous cases. For brevity, n_ indicates "number of". gen stands for "generators."
Name(s)
Type Dim.
Description
p
float n_line
Transmission capacity of each line
gen-p
float n_gen
Gens real power
load.p
float n-load
Loads active load
line-status
bool n_line
Boolean flag for line connectivity
timestep overflow
int n line
Timesteps since line exceeded capacity
AAppendix B contains detailed overview of the task-specific features. See RTE France (2025) for more information about these features and their ranges.
5

---

Published as a conference paper at ICLR 2026
Grid aovironment
ldlle Beuritric
Sale reward
Alpl 2 Press
agent operations
30 recouneet!
observations
You Recornect lines
No No operations
Agents
action 3
Figure 2: Overview of the multi-agent idle heuristic.
in Appendix D). For redispatching and curtailment, we adopt the reward of Donnot (2025), which directly reflects line loading margins: R = 1 E where L. is the set of connected lines and is the loading of line I. Specifically, grid safety decreases as line flows approach thermal limits and this formulation yields better learning performance the continuous setting.
3.1 MUL.I--GENTT IDLE TRANSITIONS
Given the complexity and dimensionality of the tasks, MARL2GRRIDTH integrates expert- informed idle heuristic (I), illustrated in Figure 2, to reduce the effective decision horizon and sim- plify learning. This emulation of operational behavior modifies the transition dynamics, focusing learning on safety-critical situations. Our design builds on prior L2RPN solutions and Marchesini al. (2025), formalizing the heuristic transitions for the multi-agent case.
For topology optimization, the heuristic issues an idle action if all line loadings remain below a safety threshold Pmax- During idle phases, agent controls are suspended and the environment pro- gresses without intervention. When any line exceeds the threshold, control returns to the agents, who try to restore normal operation. the redispatching and curtailment case, the heuristic first attempts to reconnect any available transmission lines. If no reconnections are possible, the heuristic performs the same idle check as in the discrete case. Importantly, the heuristic does not replace agent learning transitions, during
but complements it: each agent action may trigger a sequence of heuristic-guiie which rewards continue to accrue. This design combines expert-in-ther in-the-loopp guidance with MARL flexibility, reducing redundant exploration, improving sample efficiency, and stabilizing training.
3.2 FOSTERING SAFE OPERATIONS VIA MULTI-AGENT CONSTRAINTS
MARL2GRID also includes constrained problem formalizations, in which agents have to jointly minimize safety violations under a shared set of constraints. In detail, local decisions made by one agent could affect the entire grid due to the highly coupled, nonlinear, and non-convex dynamics. This phenomenon, emphasized in our discussions with TSOs at the time development, motivated decision to adopt joint constraint formulation. Hence, constraint costs not assigned to the
individual agents but are instead accumulated globally and shared among all aggnts-mmrrond joint reward structure. This encourages agents reason beyond their local context and collectively maintain system-level safety, reflecting real-world operational practices. We focus on two primary classes of operational constraints, derived from major failure modes in real transmission grids, that lead to two types of constrained tasks for each base grid.
Load shedding and islanding (L). This constraint captures two critical failure modes: (i) insufficient generation to meet demand, and (ii) the formation electrical islands (discon- nected parts of the grid). Let (s a) and (s(s, a) denote the total demand and genera- tion, respectively, given the state s and the joint action a at given step. We define the load shedding indicator function: L(s, a) ((Po(s, a) < Pp (s. a)), and the islanding indica- tor based on the number of disconnected areas Nr a) as [ss, a) = (s, a) >> 0). The L(s, a) (s,, a), and episodes considered safe if the cumulative cost satisfies Eir Ci a) = O.
per-step cost is thus defined as (1,
Transmission line overload (0). This constraint captures two key failure modes in trans- mission networks: (i) thermal overloads, where flows exceed line capacity, and (ii) line disconnections caused by prolonged violations. Let (r. a) denote the power flow on line & at a given step, and Pinax (8, a) its thermal capacity limit. We define an over- load indicator function Oc(s.a) a) (Pr.e(s, > (s triggered when the line exceeds its thermal capacity, and disconnection indicator function De(s,a)

---

Published as a conference paper at ICLR 2026
1 (e disconnected due to overload), triggered when a line is disconnected by the envi- ronment due to sustained overload. The per-step cost across all transmission lines C is then Co a) Eiec (Or(s, a) + a)),, and the cumulative constraint is enforced as Ei= Co(s,a) a) S т, where 7 is a fixed threshold.
By formalizing multi-agent safety constraints, we aim to provide principled testbed for developing constrained MARL, algorithms capable of balancing grid performance with operational risk.
EXPERIMENTS
We evaluate popular MARL methods that often serve as building blocks for more advanced algo- rithms. Consistent with prior single-agent works (Marot et al., 2022a; Marchesini et al., 2025), topol- ogy optimization is substantially more challenging than the redispatching and curtailment setup. Due to the complexity of the task, our experiments focus primarily on the smaller bus/4 task for the topological setup, where we evaluate most algorithmic variations (c.g., the constrained algorithm) to then show how our best-perforing baseline fails on the more complex bus! 18 grid. Specifically, we evaluate: (i) QPLEX (Wang et al., 2021), (ii) MAPPO (Yu et al., 2022) with and without the idle heuristic, and LagrMAPPO (Gu et al., 2021) (on the constrained I. and O versions) in the bus /4 task; and (iii) MAPPO on the high-dimensional busl18 18 task. Despite decentralization being essential to reflect how TSOs operate real grids, we also evaluate a fully observable single-agent PPO controller and its lagrangian versions LagrPPO (on the constrained L and O versions) to verify whether cen- tralization would offer any advantage and to validate whether the challenges observed stem from the MARL decomposition or the intrinsically complex nature of the tasks. The redispatching and curtailment case comparatively easier, and the MAPPO baseline already achieves strong perfor- mance. For this reason, we report our evaluation for this case only for the bus/ 18 scenario, testing MAPPO, MASAC (Bettini et al., 2024) and PPO, augmented with the idle heuristic. Crucially, these differences in the evaluation are consistent with what has been done in previous single-agent works (Marot et al., 2022a; Marchesini et al., 2025).
Overall, this selection highlights the pressing challenges of topology optimization that motivate our benchmark, while showing that continuous redispatching, though important in practice, poses a comparatively simpler learning problem under our novel task formalization.
Experimental setup. Experiments were run on Xeon E5-2650 and Silver 4214R CPU nodes with 256-376GB of RAM. Baselines were implemented using custom code inspired by CleanRL's design and BenchMARL (Bettini et al., 2024), with hyperparameters selected via grid search (see Ap- pendix E). Unless otherwise noted, results correspond the average survival or reward of the grid over 100-episode windows, aggregated across 5 independent runs per method. Shaded regions indi- cate 95% bootstrapped confidence intervals. Survival is defined as the normalized fraction of time steps during which the grid remains functional, with a value of I indicating uninterrupted operation for a full episode. The experiments in this work required ~ 120,000 CPU hours to execute.
4.1 RESULTS
Topology Optimization (discrete). Overall, the baselines struggle to cope with the complexities of multi-agent topology optimization. Figure shows the training performance of the unconstrained baseline the 4us grid. MAPPO learns the most effective policy, maintaining good operations for roughly 84% of an episode. Moreover, PPO with full observability achieves lower survival than MAPPO, showing the benefits of decentralization, and QPLEX fails to sustain stable operation be- yond few dozen steps. Augmenting these baselines with the idle heuristic converges to a ~0% average survival. Hence, despite the effectiveness of the idle heuristic in multi-agent redispatching and curtailment tasks (see next section), this heuristic interacts poorly within decentralized control under combinatorial discrete action structure. Because control is decentralized, each agent sees only a subset of the grid and must coordinate with others through the environment's nonlinear AC coupling. The idle heuristic reduces the already limited windows during which agents can exper- iment with (and learn) multi-step coordinated reconfigurations across zones. In exponentially large discrete action space, where successful topological interventions are rare and require tempo- ral coordination, this loss of actuation opportunities severely hinders exploration and joint policy
Appendix provides an high-level description of all the haselines.

---

Published as a conference paper at ICLR 2026
Table 5: busl4 (diser.): Avg. survival for the trained base- lines on 2 years of test data.
LagiPRO (i)
LagrPPo (0)
MAPPO QPLEX-
LagMAAPO (1)
LagiMAPPO (0)
0.8
Aeg: Survival
0.20
0.6
0.15
0.4
Agent type
Avg. Surv.
0.2
DoNothing
0.18
0.0
0.05
40
2.5k
QPLEX
0.04
Global
167
Aug. Cost
MAPPO
0.79
Figure 4: busl4 (diser.): Avg. survival vs. cost at convergence for the constrained baselines.
PPO
0.38
Figure 3: bus14 (diser.): Avg.
survival over training.
LagrMAPPO (L]O) 0.19
LagrPPO (L|O) 0.04
improvement. Thus, while idle transitions accelerate learning in centralized single-agent settings (Marchesini et al., 2025), they can become detrimental in MARL topology control due to reduced exploration capacity and the need for tightly coupled multi-agent coordination.
Figure 4 shows the Pareto frontier of average survival versus cost for LagrMAPPO and LagrPPO with both types of constraint at convergence, with dashed lines indicating the thresholds. Despite having promising constraint satisfaction results, LagrMAPPO and LagrPPO fail to achieve good per- formance. The best performing LagrMAPPO (L) converges to roughly 21 average survival, while the single-agent baseline consistently achieves lower performance than the multi-agent counterpart. Finally, Table 5 shows the average survival at convergence for two years of data, for all baselines and for "DoNothing" agent that only executes idle actions. These long-horizon evaluations cor- roborate the training curves, confirming that MAPPO achieves good control while other methods fail to maintain reliable performance.
Figure analyzes how the unconstrained policies learn to control the grid in the complex discrete task (referring to Appendix F for similar analysis for the constrained case). We report two opera- tional metrics, margin and topology, each shown with 95% confidence intervals as average scores. The margin score (defined in Section 3) measures the cumulative available capacity across all con- nected transmission lines. Higher values indicate that agents maintain larger safety margins and greater flexibility to handle contingencies. Successful MAPPO policies consistently maximize mar- gins, and higher survival performance appears closely related to higher line capacity. The topology score quantifies deviations from the initial grid configuration (d Go), where G, is the topol- ogy at time t and d(G,, Go) is the Hamming distance from the initial configuration Gu. Values near O correspond to minimal changes, whereas increasingly negative values indicate substantial reconfigu- rations. Effective MAPPO agents exploit topological interventions stabilize operation. This result confirmed by the lower margins and topological changes of the single-agent PPO that also leads to lower grid survival. The analysis demonstrates how these agents strike a balance between maintain- ing transmission margins and performing topology reconfigurations to achieve good performance. Notably, even in the relatively small bus/4 system, the diffi- culty of learning safe and coor- dinated topological actions un- derscores the need for MARL
MAPPO OPLEX PPO
topology
margin (higher better)
400
Aug. Score
-20
advancements. This is con- firmed by our additional results in Appendix F, showing how our best performing solution, MAPPO, fails at controlling the topology in the more com- plex busl18 system. Notably, Marchesini et al. (2025) shows how the single-agent PPO base- line (with full observability) fails on bus 18, confirming that grid control challenges stem from the intrinsic structure of the topological task.
-40
200
-60
100
Global step
1e7
Global step
1e7
Figure 5: Avg. score for line margins (higher values mean better contingency management) and topological changes for the base- lines of Figure 3.
8

---

Published as a conference paper at ICLR 2026
Discrete Results Analysis. MAPPO achieves good performance in the busl4 setup, and our anal- ysis of operational metrics (Figure 5) shows that good policies reliably maximize line-loading mar- gins while performing topology reconfigurations that successfully relieve local congestion. These behaviors break down in larger grids, for which we identified four main reasons: (i) Exploration struggles in large combinatorial action spaces, where single substation may contain tens of thou- sands of valid configurations, and good multi-step reconfigurations become exceedingly rare. (ii) Agents have difficulty coordinating across electrically coupled zones: actions that increase mar- gins locally often overload distant lines (a challenge that does not appear in bus/4). (iii) Partial observability combined with delayed, global overload penalties creates severe redii--ssignmmnt problems as agents struggle to link distant or delayed outcomes to their own actions. (iv) Topology switches involve long-horizon irreversible consequences (cooldown timers, islanding, overload-to- disconnection logic), so early random actions often lead to unrecoverable states. As a result, we notice the learned policies do not succeed in increasing margins nor in discovering meaningful topo- logical changes in larger grids, directly explaining their poor performance. We extensively discuss avenues for future research directions related to these challenges in Section 5.
Redispatching and curtailment (continuous). In contrast to the topological task, the continuous setting does not involve exponential action spaces and requires optimally balancing generation and demand, making it inherently less complex and leading to higher performance. Figure 6 shows the learning curves of the baselines, each augmented with the heuristic from Figure 2, in the complex busl18 grid. For this scenario, we train on February data to expose agents to more challenging operating conditions. Because the continuous task reward is defined directly in terms of margin, we report average reward rather than survival avoid misinterpretation. Similar to the discrete case, MAPPO converges to strong performance, achieving ~ 58% average survival in our evaluation. The fully observable, single-agen agent PPO also achieves strong performance, but it is still inferior to MAPPO when both are trained for 3 million steps. However, in contrast to the topological setting, PPO surpasses MAPPO by 9% once trained to convergence, as shown Table 6 (although requiring roughly 10 million steps to reach this level, underscoring its lower sample efficiency). Table 6 reports average survival over a two-year test set, comparing the baselines to the same "DoNothing" agent used in the topological case, and a "RecoPowerline" agent that directly applies the heuristic of Figure 2. Notably, MASAC is unable to achieve the performance of its heuristic, whereas MAPPO and PPO confirm their superior performance, surviving twice long the "DoNothing" agent.
MАPPO w.w.n MASAC PPO
800
Table 6: busl18 (cont.) Avg. survival of the baselines on 2 years of test data.
Avg. Episade neward
Rtal consergenco P70
400
Agent type
Avg. Survival
DoNothing
0.29
200
RecoPowerline
0.34
0.5
1.0
1.5
2.0
2.5
3.0
MASAC
0.25
Global step
1e6
MAPPO
0.58
Figure 6: bus/18 18 (cont.) Avg. reward per episode during training.
PPO
0.67
5 CHALLENGES AND OPPORTUNITIES FOR MARL IN GRID OPERATIONS
Our work highlights the promise and limitations of MARL for power grid control. While MARL naturally reflects the decentralized structure of real-world operations and performs reasonably well in redispatching and curtailment tasks, our results show that popular MARI. algorithms are not suitable to address high-dimensional topology optimization. This gap underscores the need for new methods and evaluation paradigms that explicitly address the combinatorial action spaces, partial observability, and safety-critical constraints of realistic and long- horg-oorizon grid operations. Closing this gap is essential if MARL is to evolve from research prototype into a tool that supports TSOs in managing future decarbonized grids. Below, we outline key directions for such future research and how MARL can be deployed in grid operations.
Beyond imitation. Unlike many domains where imitation learning provides a strong starting point, grid topology optimization lacks reliable expert demonstrations operators themselves cannot op-

---

Published as a conference paper at ICLR 2026
timally solve the problem scale. This makes direct imitation infeasible. Instead, argue for the development of advanced heuristic-guicn guided MARL, where richer domain-inspired rules and ap- proximate dynamics models serve scaffolds to reduce exploration complexity while still allowing agents to learn effective policies.
Coordination under partial observability. In practice, each agent has only local view of the grid yet must coordinate implicitly with others to prevent cascading failures. Current MARL baselines struggle to balance local autonomy with system-wide safety. Advances in communication learning, coordination graphs, and multi-agent credit assignment are needed to ensure agents act collectively rather than at cross- pursosss
Scalability. The exponential growth of topology actions poses combinatorial barrier that is am- plified in the multi- aulti--agett setting, where action spaces interact across agents. Effective abstractions (e.g,, through hierarchical control, action pruning. or structured representations of topology) are thus crucial to scaling MARL to realistically sized grids. In MARL2Grid the 118-bus system already reaches meaningful scale for research: it is large enough to expose the core coordination, safety, and combinatorial challenges of realistic operator-level control, yet still tractable for large-scale ex- perimentation. Scaling to grids with thousands of buses remains an important long-term goal, but our findings indicate that substantial algorithmic advances are required before reaching that scale.
Realism, evaluation, simulation. Progress will also depend more realistic evaluations. While our benchmark includes long horizons, stochastic renewable fluctuations, and safety-critical con- straints, further realism is required (e.g., explicit N security). Evaluation should also go beyond average survival to assess economic impact, robustness under rare but critical contingencies, and cooperation in large, heterogeneous networks. Regarding simulation, the benchmark captures key operational constraints via Grid2Op's AC solver but omits fast transients, detailed inverter and pro- tection dynamics, and some action constraints. While larger grids can be configured, MARL train- ing on very large systems also remains computationally heavy. MARL approaches can move toward practical deployment only by coupling algorithmic advances with increasingly realistic benchmarks.
Deployment. While the power sector is rightly conservative, the joint development of MARL2GRID-TE TR with TSOs shows a clear interest in RL because traditional optimization tools struggle with the growing combinatorial and real-time complexity introduced by high VRE, frequent contingencies, and large reconfiguration spaces. Crucially, RL can address these challenges and be integrated within existing operator workflows and validated through offline simulation, shadow- mode deployment, and safety filters before a broader adoption in the industry.
In summary, MARL magnifies the core challenges of grid control (e.g.. combinatorial action spaces, strict safety constraints, and long horizons) while introducing new ones such as coordination under partial observability and the lack of expert demonstrations. Addressing these challenges will require going beyond standard MARL methods to design algorithms, heuristics, and evaluation protocols tailored to the unique demands of power system operations and decarbonization.
CONCLUSION
MARL2GRID-TE TR introduces the first multi-agent RL benchmark for realistic power grid opera- tions, covering both discrete topology optimization and continuous redispatching, curtailment, and storage control. By distributing control across agents responsible for subsets of substations, the benchmark reflects the cooperative structure of real-world grids while exposing key challenges: partial observability, high-dimensional action spaces, and safety-critical constraints such as load shedding, islanding. and line overloads.
The benchmark provides standardized tasks of increasing complexity, PErrINGZoo-compattie in- terfaces, heuristic-basse idle transitions, and constrained multi-agen aggnt training settings. Experiments show that while MARI. achieves promising performance in a subset of the proposed tasks and is a natural paradigm for distributed grid control, current methods struggle with scalability, coordination, and safety in most of these long-horizon scenarios.
We expect MARL2GRID-TR to serve foundation for developing, evaluating, and comparing cooperative MARL algorithms that can enable safe and efficient grid control under modern large amounts of (distributed) VRE and flexible loads.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGEMENTS
This work was supported in part by the AI2050 program at Schmidt Sciences (Grant G-24-66236), the MIT-IBM Watson AI Lab, the 'Fondo Italiano per la Scienza" project (Grant FIS-2024-05614) and the French National Research Agency (ANR) under Grant No. ANR-23-CPJ1-0099-0 The authors thank the reviewers for their insightful and constructive feedback, which has substantially improved the quality of this work.
ETHICS STATEMENT
This work introduces benchmark for MARL in realistic power grid operations. The benchmark is developed entirely on top of publicly available, synthetic data generated with the Grid2Op frame- work, ensuring that sensitive, private, or personally identifiable information is used. The envi- ronments model stylized versions of real-world power systems in collaboration with TSOs, but do not replicate proprietary security-critical grid infrastructure.
The primary goal this research is to advance the development safe, cooperative MARL methods in the context of power grid operations. While RL agents trained on our benchmark are not directly deployable in operational power grids, we acknowledge that methods for controlling critical infras- tructure must be carefully validated and subject to rigorous safety and regulatory oversight before practical use. By explicitly modeling safety-critiaa constraints (e.g.. load shedding, islanding, and line overloads), MARL.2RRID-TT aims encourage research directions that emphasize safety and reliability.
We believe that this work aligns with the ICLR Code of Ethics by supporting transparent, repro- ducible research and by fostering methods that can contribute positively to the reliable and decar- bonized operation of power systems.
REPRODUCIBILITY STATEMENT
We have taken several steps to ensure the reproducibility of our work. The full benchmark codebase will be released as anonymous supplementary code during the review process.
Detailed descriptions of the state and action spaces, reward functions, transition dynamics, and safety constraints are provided in Section 3 and Appendices B to D, while hyperparameter choices and grid search ranges are reported in Appendix E. All experiments were run on standard CPU clus- ters, with hardware details and data collection protocols documented in Section 4. For each baseline, we provide references to the original algorithm and describe how it was adapted the multi-agent power grid setting (Appendix A). Together, these materials ensure that all results presented in the paper can be independently verified and extended.

---

Published as a conference paper at ICLR 2026
REFERENCES
ARPA-E. Grid Optimization (GO) http:://ccompeeiiioo..eees 2023.
Ayhan Alp Aydeniz, Enrico Marchesini, Christopher Amato, and Kagan Tumer. Entropy seeking constrained mulliagent reinforcement learning. In 23rd International Conference on Autonomous Agents and Multiagent Systems, pp. 2141-2143, 2024.
Matteo Bettini, Amanda Prorok, and Vincent Moens. Benchmarl: Benchmarking multi-agent rein-
forcement learning. Journal of Machine Learning Research, 25(217):1-10, 2024.
Craig Boutilier. Planning, learning and coordination multiagent decision processes. 6th Con- Serencce Theoretical Aspects of Rationality and Knowledge, pp. 195-210, 1996.
Anandsingh Chauhan, Mayank Baranwal, and Ansuma Basumatary. Powrl: A reinforcement learn- ing framework for robust management of power networks. In AAAI, 2023.
Xin Chen, Guannan Qu, Yujie Tang, Steven Low, and Na Li, Reinforcement learning for selective key applications in power systems: Recent advances and future challenges. IEEE Transactions on Smart Grid, (3(4):233529955 2022.
B. Donnot. Grid2op- A testbed platform to model sequential decision making in power systems.. www/:tooooe/e.eee wwanceteeffreeeeeeeeeeeerr ptt.rrannngggdddd/// griddop, 2020.
B. Donnot. Grid2op linescapacttrrwward. https:://or.ddopprraattttt
ioreooccciii/...e... latest/user/ hwwbserrreewad... htw.aaddd..mlllmmmmmmmmmmmmlmmm. fgriddoprr Rew.reeeewwrdddee.eeeeeeeeeeee hww.eeeaaaaaeaaaccaaaaaaaaa 2025.
Shangding Gu, Jakub Grudzien Kuba, Munning Wen, Ruiqing Chen, Ziyan Wang, Zheng Tian, Jun Wang, Alois Knoll, and Yaodong Yang. Multi-agent constrained policy optimisation. arX.v:2110.02793. 2021.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In International Confer- ence on Machine Learning, 2018.
Noureddine Henka, Quentin Francois, Sami Tazi, Manuel Ruiz, and Patrick Panciatici. Power grid segmentation for local topological controllers. Power System Computation Conference (PSCC), 2022.
Gonzague Henri, Avishai Halev Tanguy Levent, Reda Alami, and Philippe Cordier. pym- grid: An open-source python microgrid simulator for applied artificial intelligence research. arXiv:201 1.00004,,,, 2020.
Robin Henry and Damien Ernst. Gym-anm: Open-source software to leverage reinforcement learn- ing for power system management in research and education. Software Impacts, 9, 2021.
Jiajing Ling, Arambam James Singh, Due Thien Nguyen, and Akshat Kumar. Constrained multia- gent reinforcement learning for large agent population. In ECML PKDD, 2022.
Ryan Lowe, Yi Wu, Aviv Tamar, Jean Harb, Pieter Abbeel, and Igor Mordatch. Multi-agent actor- critic for mixed cooperativ--competiive environments. In Conference on Neural Information Processing Systems, 2017.
Enrico Marchesini, Benjamin Donnot, Constance Crozier, Ian Dytham, Christian Merz, Lars Schewe, Nico Westerbeck, Cathy Wu, Antoine Marot, and Priya L. Donti. RI.2Grid: Bench- marking reinforcement learning power grid operations. arX1v:2503.3101,, 2025.
A. Marot, N. Megel, V. Renault, and M. Jothy. ChroniX2Grid The Extensive PowerGrid Time-serie Generator. https: / /tg/ttttttt//tt thub. com /t/////Doottt/oooooooo
d. 2020a.
Antoine Marot, Benjamin Donnot, Camilo Romero, Balthazar Donon, Marvin Lerousseau, Luca Seyrin-Forrer, and Isabelle Guyon. Learning to run a power network challenge for training topol- ogy controllers. Electric Power Systems Research, 189:106633 2020b.
12

---

Published as a conference paper at ICLR 2026
Antoine Marot, Benjamin Donnot, Karim Chaouache, Adrian Kelly, Qiuhua Huang, Ramij-Raja Hossain, and Jochen L Cremer. Learning to run power network with trust. Electric Power Systems Research, 212:108487, 2022a.
Antoine Marot, Adrian Kelly, Matija Naglic, Vincent Barbesant, Jochen Cremer, Alexandru Ste- fanov, and Jan Viebahn. Perspectives on future power system control centers for energy transition. Journal of Modern Power Systems and Clean Energy. (0(2):328-344) 2022b.
V. Mnih, K. Kavukeuoglu, D. Silver, A. Graves, Antonoglou, D. Wierstra, and M. Riedmiller. Playing atari with deep reinforcement learning. In Conference on Neural Information Processing Systems, 2013.
Frans A Oliehoek and Christopher Amato. A concise introduction to decentralized POMDPs. Springer, 2016.
Georgios Papoudakis, Filippos Christianos, Lukas Schäfer, and Stefano V Albrecht. Benchmarking multi-agent deep reinforcement learning algorithms in cooperative tasks. In Thirty-fifth Con- ference on Neural Information Processing Systems Datasets and Benchmarks Track (Round I), 2021.
Tabish Rashid, Gregory Farquhar, Bei Peng, and Shimon Whiteson. Weighted QMIX: expanding monotonic value function factorisation. In Conference on Neural Information Processing Systems, 2020.
RTE France. Dive into grid2op sequential decision process, 2025. URL https:h/tt://tt readthedocs wo/ee////ee wwwee//naaeetteeeeeeeeeeeeeeeeee /mdp. .tmll #s ome constraints Accessed: 2025-05- 15.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. In arXiv:1707006347, 2017.
J Terry, Benjamin Black, Nathaniel Grammel, Mario Jayakumar, Ananth Hari, Ryan Sullivan, Luis S Santos, Clemens Dieffendahl, Caroline Horsch, Rodrigo Perez- Vicente, et al. Pettingzoo: Gym for multi-agent reinforcement learning. Advances in Neural Information Processing Systems, 34: 15032-15043, 2021.
U.S. DoE. Distribution grid transformation. distribution-gri 2024.
https://wwy www.gyywwwweee.....e.....e....e. /ov/
Jianhao Wang, Zhizhou Ren, Terry Liu, Yang Yu, and Chongjie Zhang. QPLEX: duplex dueling multi-agent q-learning. In International Conference on Learning Representations, 2021.
Chao Yu, Akash Velu, Eugene Vinitsky, Yu Wang, Alexandre M. Bayen, and Yi Wu. The surprising effectiveness of MAPPO in cooperative, multi-agent games. In Conference on Neural Information Processing Systems, 2022.
13

---

Published as a conference paper at ICLR 2026
A MARL BASELINES
In this section, we briefly introduce the baseline MARL algorithms employed our evaluation, referring to the original papers for exhaustive details (Wang al., 2021; Yu et al., 2022; Gu al., 2021; Bettini et al., 2024).
QPLEX (Wang et al., 2021). QPLEX is value-based method designed for cooperative MARL. builds on the QMIX framework by introducing dueling network architecture. Each agent maintains its own local Quutility, while mixing network combines these into joint action-value function. This decomposition allows decentralized execution while maintaining centralized training. Similar DQN Mnih et (2013) in the single-agent case, QPLEX is restricted discrete action spaces, making it applicable to topology optimization tasks.
MАPPO (Yu et al., 2022) and LagrMAPPO (Gu et al., 2021). MAPPO extends PPO (Schulman et al., 2017) to the multi-agent setting using centralized critic and decentralized actors. Each agent learns own policy, while the centralized critic leverages global information to centralize training. The clipped surrogate objective from PPO ensures stable updates, balancing policy improvement and regularization. MAPPO can handle both discrete (topology optimization) and continuous (redis- patching and curtailment) actions, depending on the distribution chosen for the actor. LagrMAPPO augments MAPPO with a constraint-Ne handliinn mechanism: in addition to training the policy, it learns Lagrangian multipliers associated with each constraint (as Section 2). Policy updates then take gradient ascent steps in π and descent steps in X, trading off constraint satisfaction and task perfor- mance. This ensures penalties grow when constraints are violated and decay when constraints respected.
MASAC (Bettini et al., 2024). MASAC adapts SAC (Haarnoja al., 2018) to the multi-agent set- ting. combining centralized critics with decentralized actors. As in the single-agent SAC, MASAC jointly optimizes for expected return and policy entropy, encouraging exploration and robustness. Each agent learns stochastic policy, while the centralized critic leverages information across agents to reduce variance and improve stability. MASAC supports well continuous action spaces and therefore particularly suitable for our redispatching and curtailment tasks.
B STATE-SPACE
Tables and describe the remaining task-specific features composing agents' observations in the discrete and continuous case, respectively.
Table 7: List of additional features composing the state power grid for the discrete case. For brevity, indicates "number of", gen, sub stands for "generators" and "substations", respec- tively, and dim.topo is the size of the vector containing the current topology of the grid.
Name(s)
Type Dim.
Description
int float n_gen
Current simulation step
gen.theta
Gens real power and voltage angle Loads active load and voltage angle Topological vector of the grid; the bus to which each object is connected
load_0
float
load
topo-vect
int dim_topo
time cetoore.coodown lineinin int cime_befoore.cooldoow down__su int (time, duration] www.://mneettmmmteteaeaeeeae
nine n_sub
Line cooldown timer
Cooldown timer for substations Remaining time and duration of the next maintenance
int
n_line
14

---

Published as a conference paper at ICLR 2026
Table 8: List of additional features composing the state of power grid for the continuous case. For brevity, n. indicates "number of", gen, stor stands for "generators" and "storage units", respectively.
Name(s)
Type Dim.
Description
month
int
Month of the year Day of the week The time is Active power of each line Storage units charge
day-of.week
int 1
hour.of.day, minute.of.hour int
p-or storage storage_charge
float n-line
float n_stor
storage.pon power
float n_stor
Storage units power
curtailment
float n_gen
Curtailed power for each generator Limit imposed on each renewable generator
curtallment.limi
float n-gen
gen.p.before.curtar
float n-gen
Production there would have been without curtailment
target.dispatch
float n_gen
Targeted redispatching
actual.dispatch
float
n.gen
Implemented redispatching
C AGENT CONFIGURATIONS
Table reports the agent grid partitions for the bus14 and bus36 topology optimization (discrete) tasks. For these smaller grids, we focus exclusively on the discrete setting, which is substantially more challenging and already causes common MARL algorithms to struggle, even the simplest busl4 setup (see Section 4). By contrast, redispatching and curtailment (continuous) setups already achieve promising performance in the larger and more complex bus scenario, making the smaller cases not challenging enough to investigate in the continuous setting.
Table 9: Agent-to-substation assignments, number of controlled components, observation and action dimensions for the local observation setup of bus/4, bus36 (T stands for the topology case)
Grid
Agent Controlled Substations
Lines Gens. Loads
|Obs (T)
AAtions (T)
0
[0, 1, 2, 4]
8
3
71
61
bus14
1
[3, 6, 7, 8]
9
1
2
49
55
2
[5,9, 10, 11, 12, 13]
9
6
83
89
0
[0, 1, 2, 3, 3]
9
6
77
77
1
[6, 7, 8, 9, 16]
18
5
150
65642
bus36
[5. 10, 11, 12, 13, 15,
13
12
3
[17-31, 33, 34]
32
11
14
377
1119
D REWARD
In this section, we formally define the reward components for the discrete topological tasks. We recall the joint reward the agents get at each step R R + RR + R) While Rsavviee is cumulative positive constant, the overload and cost rewards are defined as:
(i) Overload: Penalizes line overloads and disconnections, and rewards available line capacity based on the difference between line flows and capacity limits. In unconstrained settings, disconnected lines incur fixed penalty. This is more formally defined as:
Pr. F.E
Roverlond
Σ max
F.E.
1( is disc.)
(1)
+
EEC
where РF. is the power flow on line E, PEIxx is its capacity limit, is a small constant to avoid divisions by O, and the indicator function returns 1 if the line is disconnected.. This term then normalized to lie within [11
15

---

Published as a conference paper at ICLR 2026
(ii) Cost: This component accounts for redispatching, curtailment, and storage usage, all of which induce operational costs. is defined as:
Resst = - [(P - PD) + Gredispl +
Stoorgee
Cmarginal
where Pg and PD denote the total power generated and total demand consumed at each step, respec- tively, with their difference representing transmission losses, Gediss corresponds to the redispatched power (i.e., the absolute deviation from scheduled generator setpoints), and Pstorage represents the power exchanged with storage units. All cost components scaled by the marginal generation cost Cmarginals defined as the cost per MWh of the most expensive generator currently producing power. This value is also normalized to lie in the range [-1,0].
E HYPERPARAMETERS
Table 10 lists the hyperparameters considered during initial grid search and the best- pest-performing parameters used to run the experiments in Section 4.
Table 10: Details of the grid search used to find the best-performing hyperparameters for each algorithm in the topology optimization (discrete) and redispatching and curtailment (continuous) cases.
Algorithm
Parameter
Grid search
Chosen value
Shared
No parallel
10. 20, 50 10, 20,
10
Max gradient norm
10
Discount y
0.9, 0.95, 0.99
0.99
Pmax
0.9, 0.95
0.9
Top. opt. reward
0.1, 0.5, 1.0 0.1, 0.5, 1.0
1.0
1.0
0.1, 0.5, 1.0
1.0
QPLEX
Train frequency
50, 100
100
Target network update
250, 500, 2500
2500
Buffer size
500000, 1000000
1000000
Batch size
128, 256
128
Learning
0,003, 0.0003, 0.00003
0.00003
e-decay fraction
0.1, 0.25 0.5
0.5
MAPPO (discrete case)
No steps (total) Nº minibatches
10000, 20000, 40000
20000
1, 8
N update epochs
20, 40, 80
80
Actor learning rate
3e-3, 3e-4, 3e-5
3e-5
Critic learning rate
3e-3, 3e-4, 3c-5
3e-5
c-clip
0.1, 0.2, 0.3
0.2
MAPPO (continuous case)
Batch size
3000, 6000, 9000
9000
N update epochs
5, 15, 30
30
Actor learning rate
3e-4, 3e-5, 3e-6
3e-5
Critic learning rate
3e-4, 3e-5, 3e-6
3e-5
LagrMAPPO
50
0 (L). 50 (O)
init
0.0, 1.0
0.0
learring rate
0.01, 0.025, 0.05
0.05
MASAC
Batch size
3000, 6000, 9000
9000
Minibatch size No optimizer steps
128, 256
256
1000, 2000
1000
Learning rate
3e-4, 3e-5, 3e-6
3e-4
16

---

Published as a conference paper at ICLR 2026
F ADDITIONAL PLOTS FOR SECTION 4
To complement the main results in the topology optimization (discrete) case, we evaluate the best- performing baseline, MAPPO, on the more complex busl18 system. Unlike in the smaller busl4 grid, where MAPPO manages to sustain operation for substantial fraction of the episode horizon, performance on bus/ 18 unsatisfactory. Figure summarizes the outcomes in terms of average survival at training time and analyzes the margin and topology scores for the trained policies. Sur- vival rates are close zero, indicating that MAPPO fails maintain stable operation for more than few steps. This is reflected in the margin metric, which remains consistently low and shows that agents are unable to preserve sufficient transmission capacity to handle contingencies. Similarly, the topology score indicates that agents rarely exploit meaningful structural reconfigurations: deviations from the initial configuration are minimal and do not translate into improved stability.
Overall, these results highlight the dramatic increase in difficulty when scaling from busl4 4 to bus118. Even our strongest baseline fails to discover effective strategies for coordinated topology optimization at this scale, reinforcing the conclusion that MARL-based grid control requires new algorithmic advances beyond current MARI. literature.
MAPPO
margin (higher = batter)
topology
Surveol
Avg. Score
-14
8.00
-16
0.002
-1.8
Global
Glotal
Glbbel
Figure 7: Results of the best performing baseline, MAPPO, in the topology optimization (discrete) busl 18 task. (Left) Average survival during training for the discrete case on the bus task. (Center)
Avg. margin score for the trained policy. (Right) Avg. topology score for the trained policy.
Moreover, Figure presents the same operational metries analysis as Fig- ure 5, but for the constrained base- line. LagrMAPPO with load shed- ding and islanding constraints (L) achieves higher performance than the transmission line overload strained version (O), despite operat- ing under stricter threshold. No- tably, these policies tend to converge on single topological modification that increases available margins, al- lowing the grid to remain operational for roughly 20% of the episode horizon.
LagrMAPPO (L) LagrMMPPMMPPP (O)
(igherr
Avr. Score
Global
Global
Figure 8: Average score for line margins and topological changes for the constrained algorithm Figure 4.
17


Reviews:
[{"review_id": "faJGRJuMeQ", "reviewer": "Reviewer_q5yx", "summary": "Modern power grids face increasing complexity due to renewable integration, requiring fast, decentralized control beyond traditional optimization. Prior RL benchmarks (like L2RPN, RL2GRID) model the grid as a single-agent problem, ignoring real-world decentralization. The paper introduces MARL2GRID, the first benchmark for multi-agent RL (MARL) in power grid operations, developed with transmission system operators (TSOs).", "questions": "I'd like to see a thorough discussion on how RL can be deployed in grid operations. Power grid is a very conservative industry, and it is more or less reluctant to emerging technologies, especially if existing technologies do just fine. I'd like to see some more discussions on what RL can achieve that traditional methods cannot.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "-\tPower grid control is challenging and important especially with more renewable integration. RL holds a big promiss, and this paper addresses a key gap for benchmarking decentralized MARL for realistic grid operations.\n-\tThis paper is built on industrial-grade Grid2Op simulations with realistic dynamics, long horizons, and stochastic disturbances.\n-\tThis paper considers safety constraint, which is important for grid operations.", "weaknesses": "-\tMore specifics about the intended power system use case. From my understanding, power system typically have a 3-layered control architecture, with primary control (droop) occurring at the fastest time scale, and then secondary control (AGC), followed by tertiary control. What time-scale is the benchmark and how it would fit into the existing control architecture of power systems?\n-\tScale of the benchmark. For applications like power systems, scale matters a lot and I would like to see a much larger scale (in the thousands/10k’s sized) power system benchmarks. \n-\tThe proposed benchmark also lacks diverse tasks to be named (MARL2GRID). As I mentioned, there are many control problems happening at the grid at different levels, including lower/fast level frequency/voltage control control at generators/inverters, mid level AGC as well as higher level day-ahead planning. For the benchmark to be named MARL2GRID, ideally it should consider more than 1 tasks that are representative of the complex grid operations.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "9lNDTbmj37", "reviewer": "Reviewer_eRUd", "summary": "This paper identifies the limitation of prior work in power grid operations, which has largely focused on single-agent settings, and proposes MARL2GRID, the first benchmark for multi-agent reinforcement learning (MARL) designed to reflect the decentralized nature of real-world grid control. Built upon the high-fidelity Grid2Op simulator, the benchmark introduces realistic tasks, including discrete topology optimization and continuous redispatching, along with safety-critical constraints. Through experiments, the authors demonstrate that existing MARL algorithms (e.g., MAPPO, QPLEX) struggle significantly with the scalability, coordination, and safety requirements of these tasks, particularly in the combinatorially complex topology optimization setting, thereby highlighting key challenges for future research.", "questions": "1, The role and efficacy of the \"idle heuristic\" are unclear. Was this heuristic only applied in the bus118 continuous task, or was it also used for the bus14 topology task where MAPPO performed well? To isolate and validate its contribution, could you provide an ablation study (e.g., MAPPO with vs. without the heuristic on the bus14 task) to quantify its actual impact on performance?\n\n2, To experimentally validate the core motivation for a MARL approach, could you please provide a performance comparison against a centralized single-agent RL baseline (e.g., single-agent PPO) on the bus14 and bus118 topology optimization tasks? This result is crucial for justifying the choice of the MARL paradigm over a simpler, centralized one.\n\n3, Performance drops dramatically from bus14 to bus118. To better understand the scalability limits of current MARL methods, could you provide results for the intermediate-sized bus36 grid in the topology optimization task? This would help clarify whether the performance degradation is gradual or a sharp cliff-edge effect at a certain complexity threshold.\n\n4, The results show that algorithms fail on large-scale tasks (Figure 7). Can you provide a more in-depth diagnostic analysis of these failures? For instance, is the failure primarily due to a lack of coordination, inefficient exploration, or credit assignment challenges under partial observability? A statistical breakdown of which safety constraints are most frequently violated would provide valuable guidance for future research.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1, The paper correctly identifies a critical gap between existing single-agent RL research (e.g., L2RPN) and the decentralized reality of power grid control, making the proposal of a MARL-specific benchmark both timely and significant. The design, developed in collaboration with Transmission System Operators (TSOs), adds a layer of realism and credibility to the proposed problem setting.\n\n2, The benchmark is well-structured, offering a suite of tasks across various grid scales (bus14, bus36, bus118), two distinct and crucial control paradigms (discrete topology and continuous redispatch), configurable observability regimes, and explicit safety constraints such as load shedding and line overloads (Sections 3, 3.2). This provides a rich and challenging testbed for future MARL research.\n\n3, The empirical results effectively highlight the brittleness of current MARL algorithms when faced with a complex, realistic task. For instance, while MAPPO achieves a 79% survival rate on the bus14 topology task (Table 5), its performance collapses to near-zero on the more complex bus118 grid (Figure 7), starkly illustrating the benchmark's difficulty and motivating the need for novel algorithmic solutions.", "weaknesses": "1, The paper's central premise is that the decentralized nature of power grids necessitates a MARL approach (Lines 014-016). However, this claim is not substantiated with experimental evidence. A critical baseline is fatally absent: a strong, centralized single-agent RL controller (e.g., single-agent PPO) that has access to the full state and action space. Without this comparison, it is impossible to determine whether the proposed multi-agent decomposition (1) offers any performance benefit, (2) is truly necessary for scalability, or (3) merely introduces unnecessary complexity that hinders performance. This is a major flaw in validating the paper's core motivation.\n\n2, The paper successfully demonstrates that current algorithms fail, particularly on large-scale topology optimization (Figure 7). However, it stops at reporting the phenomenon (e.g., \"MAPPO failed\") without providing a deep diagnostic analysis of why they fail. The root cause—be it challenges in credit assignment under partial observability, inefficient exploration in a combinatorial action space, or poor coordination—remains unexplored. This lack of insight limits the benchmark's utility in guiding future", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "fi0JQan11l", "reviewer": "Reviewer_2m4V", "summary": "The paper proposes a MARL simulation framework for power grid operations. By using realistic scenarios, it proposes a tool for evaluating \ndecentralized control of substations and generators. It supports various configurations, including partial information, delayed rewards, and physical constraints.", "questions": "- What are the limitations of the simulation in terms of the algorithms and the settings that it can support? There is an extensive discussion on why the current algorithms fail to address the complexities, but no account of the limitations of the simulation itself. \n\n- In the context of constraint handling, did you analyze the patterns of violations in different settings, and did you experiment with agent-local constraints?  \n\n- While I accept the CTDE paradigm here, can you speculate on the result of applying a fully decentralized approach?", "limitations": "", "rating": 8, "confidence": 2, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "- The first standardized MARL formulation for topology optimization and redispatching control, including safety constraints \n- Integration with PETTINGZOO for MARL compatibility and reproducibility.\n- Baseline evaluation of state-of-the-art MARL algorithms \n- Simulation environments based on real TSO data and topologies.\n- Fills an important gap - since previous power grid RL environments were single-agent. The combination of high-fidelity grid simulation and multi-agent formalism is novel and impactful.", "weaknesses": "- The title is misleading - the paper offers a simulation environment, not a rigorous account of how benchmarking should be performed (i.e., KPIs etc).\n- Evaluation focuses mainly on classic CTDE baselines with no ablation studies on the effect of observability regimes (centralized vs. local).\n- No explicit evaluation metrics on economic cost, safety violation frequency, or stability margins over time.\n- Minor writing issues (redundancy, figure clarity, inconsistent notation in equations).\n- Evaluation focuses on different SOTA algorithms, but since the focus is on the simulation, I would expect an account of its limitations.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)