SustainGym: Reinforcement Learning Environments for Sustainable Energy Systems
Authors: Christopher Yeh
Victor Li
Rajeev Datta
Julio Arroyo
Nicolas Christianson
Chi Zhang
Yize Chen
Mohammad Mehdi Hosseini
Azarang Golmohammadi
Yuanyuan Shi
Yisong Yue
Adam Wierman
Venue: NeurIPS 2023 Datasets and Benchmarks Poster (2023)
Keywords: reinforcement learning
sustainability
energy systems
multi-agent
distribution shift

Abstract: The lack of standardized benchmarks for reinforcement learning (RL) in sustainability applications has made it difficult to both track progress on specific domains and identify bottlenecks for researchers to focus their efforts. In this paper, we present SustainGym, a suite of five environments designed to test the performance of RL algorithms on realistic sustainable energy system tasks, ranging from electric vehicle charging to carbon-aware data center job scheduling. The environments test RL algorithms under realistic distribution shifts as well as in multi-agent settings. We show that standard off-the-shelf RL algorithms leave significant room for improving performance and highlight the challenges ahead for introducing RL to real-world sustainability tasks.

SustainGym: Reinforcement Learning Environments for Sustainable Energy Systems
Christopher Yeh', Victor Li", Rajeev Datta', .Juiio Arroyo', Nicolas Christianson', Chi Zhang", Yize Chen Mehdi Hosseini', Azarang Golmohammadi, Yuanyuan Shi?, Yisong Yue', Adam Wierman CCalifornia Institute of Technology
University of California, San Diego
Hong Kong University of Science and Technology
'Beyond Limits
[cyeh, vhli rraaaadaaaaaaarraraaaa yyue, adamw) calltcch.edu
cch20560csdd edu, yyshi Beng. ucsd. edu
Syizechendust. hk
[mhosseini, agolmohammadi @beyond. ai
Abstract
The lack of standardized benchmarks for reinforcement learning (RL) in sustain- ability applications has made difficult to both track progress on specific domains and identify bottlenecks for researchers to focus their efforts. In this paper, we present SustainGym, a suite of five environments designed to test the performance of RL algorithms on realistic sustainable energy system tasks, ranging from electric vehicle charging to carbon-awee aware data center job scheduling. The environments test RL algorithms under realistic distribution shifts well in multi-agent set- tings. We show that standard off-the-shelf RL algorithms leave significant room for improving performance and highlight the challenges ahead for introducing RL to real-world sustainability tasks.
Introduction
While reinforcement learning (RL) algorithms have demonstrated tremendous success in applications ranging from game-playing, e.g., Atari and Go, robotic control, e.g., (13,, most RL algorithms continue to only be benchmarked using toy environments-e. e.g. OpenAl Gym [4. These toy environments generally do not have realistic physical constraints, nor realistic environmental shifts over time. Furthermore, these environments are generally limited to single-agent systems, whereas real-world systems tend to involve coordination and/or competition between actors. The realism gap limits the reliable deployment of off-the-shelf RL algorithms in real-world systems.
Developing better RL algorithms address these challenges requires marking and comparing the performance different algorithms in real-world settings. Our inspiration comes from progress supervised machine learning (ML). where widespread adoption of break- through techniques was fueled by large datasets with standardized benchmarks, such as ImageNet for computer vision [5] and the GLUE benchmark for natural language processing 6 More recently, many supervised learning datasets have been created to address specific real-world sustainability challenges, such as monitoring global progress towards sustainable development goals 7
of empirically bench-
In this work, we introduce SustainGym, a suite of 5 RL environments that realistically model sustainability settings, summarized in Table
37th Conference on Neural Information Processing Systems (NeurIPS 2023) Track on Datasets and Benchmarks.

---

Table 1: Summary of environments included SustainGym and their features. The "Single agent" and Multi-agent rows indicate what an individual RL agent controls in that environment.
Env
EVChargingEny
ElectricitMurkeffn
DatacenterEan
CogenEnv
BuildingEnv
Control
charging rates for EV charging
market bids grid-connected battery storage system
virtual capacity curve for a carbon-aware data center
dispatch set points for turbines
heating supply for buildings
task
stations
Modeled after
charging
generic test case (IEE
(loosely) a Google data center
specific combined cycle gas generation plant in the U.S.
generic DoE commercial reference building models
networks at Caltech & JPL.
RTS-GMLC)
Single
all EV charging stations
single battery system
single data center
all 4 turbine units
all buildings
agent
Multi- agent
one charging station.
N/A
one turbine unit, cooperative
one room, cooperative
cooperative
Actions
discrete or continuous
discrete or continuous cost CO2
continuous
mixed discrete & continuous cost CO2
discrete or continuous
Rewards
cost CO2
penalty CO2
temperature difference energy use outdoor temperature
Distribution
MOER, EV arrivals
MOER, load
MOER
renewable wind penetration
shift
EVChargingEnv models the problem scheduling electric vehicle (EV) charging to meet
user needs while minimizing CO2 emissions.
models grid-scale battery storage system bidding into the elec- tricity market generate profit (through price arbitrage) and reduce CO2 emissions.
ElectricityMarreee
DatacenterEnv models datacenter deciding on a "virtual capacity curve" to shift flexible jobs towards times of day with lower CO2 emissions.
CogenEnv models a combined cycle cogeneration plant producing steam and electricity to meet local demand while minimizing fuel usage and ramp costs.
BuildingEnv models the thermal control of building energy systems to reduce the total electricity consumption while satisfying the user-specified temperature requirement.
key feature of SustainGym environments is their support for testing RL algorithms under realistic and natural exogenous distribution shifts, which generally fall under categories:
Shifts in demand. each environment, RI. agents choose actions to satisfy some "demand" that is often affected by the behavior of unmodeled agents. For example, in EVChargingEnv, the demand the amount of energy that needs to delivered to EVs that have arrived at the charging network. This demand changed significantly at the start of the COVID-19 pandemic when EV drivers changed their driving behaviors (Figure 2
2. Shifts in environmental parameters. Real- world environments are rarely static, and Sus- tainGym environments reflect changing environment parameters due to temporal and/or climate changes. For example, battery storage controller for Electrici settricittyie citerriettMarr makes decisions to minimize marginal CO2 emissions, but the distribution of CO2 emissions varies time power plants are added
removed from the electric grid (Figure
Notably, the distribution shifts reflected in SustainGym are unlike the "sim-to-real" or offline-vs- online RL distribution shifts that have been more commonly studied in the literature. The sim-to-real distribution shift comes from imperfect modeling of the environment, whereas the offline-to-onllee RL distribution shift is caused by a change in the policy used to generate trajectories. In contrast, the exogenous distribution shifts in SustainGym are not due to imperfect environments nor policy mismatches. but rather more fundamental changes in the transition dynamics of the Markov decision processes. Note that only the transition dynamics experience distribution shift; the state space, action space, and reward functions do not change.
Two other similar lines of work to the distribution shifts in SustainGym are nonstationary environments 8 and distributionally robust RL 19]. However, whereas nonstationary RL typically

---

PGE
SCE
0.6
2019
0.14
May 2023
May 2021
MOER RMSE
MOER
0.3
0.10
M
0.1
0.06
May
PGE. May 2021
0.04
SCE, May 2019
00:00
63:00
08.00
69:00
12.00
1580
18:00
21.00
ebren
030n
06:00
09en
12,00
1580
18:00
21:00
30
day
time
timwstaps ahwad
Figure 1: (left) MOER values from two different regions (a.k.. "balancing authorities")) in California, Pacific Gas & Electric (PGE) and Southern California Edison (SCH). Solid line is the mean MOER over all days month at given time of day. Shaded region is +1 std. dev. (right) MOER forecast error increases with the forecast horizon.
evaluates an RL agent's performance over the course of changing environment, SustainGym benchmarks RL agents' ability to generalize to new (unseen) distribution shifts. Distributionally robust RL generally assumes that the set of environmental distributions are known at training time, which is not necessarily the case for the settings considered in SustainGym.
In addition to modeling realistic distribution shifts, SustainGym is distinctive for its inclusion multi-agent interactions, physical constraints, and mixtures of discrete and continuous actions, as summarized in T
To demonstrate the use of the SustainGymm we perform experiments with off-the-shelf RI. algorithms. We find that these algorithms have mixed performance SustainGym. Furthermore, show that distribution shifts may reduce the performance of these algorithms significantly, demonstrating a need for more robust algorithms. Finally, comparisons against non-RL baselines and oracles show that RL has significant room for improvement.
Due to page constraints, the main text of this paper summarizes key design choices and experimental observations for SustainGym. Details can be found in Appendix Code, licenses, and instructions for using SustainGym can be found on GitHub !
Related Work. Prior work related SustainGym includes ConservationGymm, which focuses on ecological applications [10,, PowerGridWWorld for power system modeling and simulation [11, and CityLearn for simulation of demand response and urban energy management [2], among others. RL environments and algorithms for both EV charging (3355) and electricity markets [16118] have also been released. Compared to these works, the unique aspects of SustainGym are its focus on tracking estimated CO2 emissions and its ability to test RL algorithms in settings with challenging distribution shifts, physical constraints, and interactions between multiple agents. We expect SustainGym to serve as benchmark for the progress of RL algorithm development for sustainable energy systems.
2 Environments
This section introduces the environments in SustainGym and summarizes their design choices.
Marginal CO2 emissions. Three environments (EVChargingEnv, ElectricityMarrkeEMni DatacenterEnv) impose cost Poo, (in S/kgCO2) on the simulated CO2 emissions induced by the actions of an agent as result of changes in electricity consumption. To do so, our environments use data California's 's historical marginal operating emissions rate (MOER, in kgCO2/kWh), which the increase in CO2 emissions per increase in energy demand. The MOER at time is denoted m R.,, and the forecasts generated at time t for the next l time steps are denoted f R By default, we use k = 36. Figure shows how MOER values and their forecasts vary across time and between different regions in California.
3

---

Summer 2019
Fall 2019
2020
Summer
24:00
depetun ume
-30
-75
0.00
-90
9:00
8.00
16.00
24:00 0.00
8.00
16:00
24.00
9:00
8.00
16:00
24:00
0.00
8.00
16:00
24:00
ATTIVE
MET VM time
me
Figure 2: EV arrival vs. departure times for the Caltech EV charging network. Historical data is in blue, and logelikeldhood contours from 30-component GMM are in orange. The distribution of EV arrival and departure times changed noticeably when the COVID-19 pandemic began in early 2020.
2.1 EVChargingEnv
EVChargingEnv uses ACNSim [3] to simulate the charging of EVs based on actual data gathered from EV charging networks between fall 2018 and summer 2021 [19 20 ACNSim is a "digital twin" of actual EV charging networks Caltech and JPI., which have = 54 and 52 charging stations (abbrv. EVSEs, Electric Vehicle Supply Equipment), respectively. ACNSim accounts for nonlinear EV battery charging dynamics and unbalanced 3-phase AC power flows, and is thus very realistic. ACNSim (and therefore EVChargingEnv) can be extended to model other charging networks as well. When drivers charge their EVs, they provide an estimated time of departure and amount of energy requested. Because of network and power constraints, not all EVSES can simultaneously provide their maximum charging rates (a.k.a. "pilot signals").
Each episode starts at midnight and runs at 5-minute time steps for 24 hours. At each time step, the agent simultaneously decides all n EVSE pilot signals be executed for the duration of that time step. Its objective is to maximize charge delivery while minimizing carbon costs and obeying the network and power constraints.
Observation Space. An observation at time t is s(t) = (t, d, e, m,-1, m t E [0, 1] is the fraction of day. d E Z" is estimated remaining duration of each EV (in # of time steps). E BT is remaining energy demand of each EV (in kWh). If no EV is charging at EVSE i, then d; = 0 and
e, 0. If an EV charging at EVSE has exceeded the user-specified estimated departure time, then d, becomes negative, while e; may still be nonzero.
Action Space. EVChargingEnv exposes choice of discrete actions a(I) E 1,, repre- senting pilot signals scaled down by a factor of 8, or continuous actions a(I) E [0, 1)mh representing the pilot signal normalized by the maximum signal allowed M (in amps) for each EVSE. Physical infrastructure in charging network constrains the set Ar of feasible actions at each time step t 20 Furthermore, the EVSES only support discrete pilot signals, these physical constraints, EVChargingEnt can project an agent's is action a(t) into the convex hull of A and round it to the nearest allowed pilot signal, resulting in final normalized pilot signals ä(t). ACNSim processes a(t) and returns the actual charging rate Mā E R (in amps) delivered at each EVSE, well as the remaining demand (t + 1).
A, is nonconvex. To satisfy
Reward Function. The reward function is a sum of three components: r(t) = p(() -1v(l)-----lll)))
The profit term p(() aims to maximize energy delivered to the EVs. The constraint violation cost cv (t) penalizes network and power constraint violations. Finally, the CO2 emissions cost co (t). which is function of the MOER m. and charging action, aims to reduce emissions by encouraging the agent to charge EVs when the MOER is low.
Distribution Shift. EVChargingEnv supports real historical data as well as data sampled from a
30-component Gaussian Mixture Model (GMM) fit historical data. We fitted GMMs 4 disjoint historical periods, as defined in [21] [egures[land [ show the distribution of arrival and departure times in each of these 4 periods, for both the historical data as well as the GMM log-likelshoods. From these figures, it is evident that the pattern of user arrival and departure times changes over time, with the most drastic shift happening between Fall 2019 and Spring 2020, which when the COVID-19 19 pandemic began.

---

Multiagent Setting, The multiagent setting features n agents, each deciding the pilot signal for single EVSE. The reward split evenly among the agents. Each agent obtains the global observation, except that the estimated remaining durations and energy demands for other EVSES are delayed by la time steps.
2.2 ElectricittMarrketaia
ElectricityMarketänn simulates a realtime electricity market for 33 generators and I 80MWh battery storage system connected on a 24-bus congested transmission network based on the widely- used IEEE RTS-20 24 test case (22,, with 5-minute settlements and load data from IEEE CTL- is not modeled after any particular real-world transmission
23.. While Aeectricityarrettie network, the ETS-GGLC electricity load profile was designed to be representative of modern transmission network located in the southwestern U.S.
All participants submit bids to the market operator (MO) at every time step. Based on the bids, the MO solves the multi-timestep security-constrainee economic dispatch (SCED) problem which determines the price and amount of electricity purchased from (or sold by) each generator and battery to meet realtime electricity demand. Each episode runs for day, with 5-minute time intervals. The agent controls the battery system and is rewarded for submitting bids that result in charging (buy) when prices are low, and discharging (sell) when prices and CO2 emissions are high, thus performing price arbitrage.
(1, e, a(t Pt-,1----,,4-11 14- ( t [0, 1] represents the time of day. e E R4 is the agent's 's battery level (in MWh). alt 1) R is the previous action taken. It-1 R is the previous dispatch (in MWh) asked of the agent. Pt-1 E R4 is the previous price experienced by the agent (in S/MWh). 4-1 E 14 is the previous demand experienced by the agent (in MWh), while 1444--1 E R is the forecasted demand for the next As steps.
Observation Space. An observation
s()
Action Space. An agent action is a bid a(t) = (a",a") E R² x Rİ, representing prices (S/MWh) that the agent is willing pay (or receive) for charging (or discharging) per MWh of energy, for the next k + time steps starting from time t. The generators are assumed to always bid their fixed true cost of generation. The environment solves the optimal dispatch problem determine the electricity price pt and the agent's dispatch 2₁ E R. which is the amount of energy that the agent is obligated to sell into or buy from the grid within the next time step. The dispatch in turn determines the storage system's next energy level. We also provide a wrapper that discretizes the action space into 3 actions only: charge, do nothing, or discharge.
The reward function encourages the agent to maximize profit from charging decisions while minimizing associated carbon emissions. It is a sum of three components: r(t) (t(t) + (c( - (t). The revenue term rt(t) = pez, is the immediate revenue from the dispatch. The CO2 emissions reward term rt()) or incurred by the battery dispatch. The terminal cost er (t), which is nonzero only when = T. encourages the battery to have the same energy level at the end of the day as when it started. We also provide an option to delay all reward signals until the terminal time (intermediate rewards are 0).
Reward Function.
My represents the price of CO2 emissions displaced
Distribution Shift. Distribution shift for Electrico ci wityytaaeteeaaaee Env comes from changes in both electricity demand and MOER profiles between summer and winter months.
2.3 DatacenterEnv
DatacenterEny is simulator for carbon- job scheduling in datacenters, which aims to reduce the carbon emissions associated with electrcity usage in a datacenter. Carbon-aware job scheduling is premised upon two facts: (i) significant fraction of datacenter's workload (e.g., up to 50% in some
of Google datacenters [2 25 is comprised of low priority jobs whose execution can be delayed, and (ii) the carbon intensity of the electric grid fluctuates predictably over time. Therefore, if the execution of low priority workload is delayed to a time of day with "greener" energy, the datacenter's carbon emissions can be minimized.
DatacenterEny is loosely modeled after a Google datacenter. We assume that jobs are scheduled according to priority queue, with jobs spread evenly across the available machines. Following
5

---

Energy demand (MNV)
600
Energy demand (MWV)
600
Steam demand (klbarr)
1250
400
400
750
250
0h
4h
8h
12h
16h
20h
0h
0h
4h
8h
12h
16h
20h
0h
0h
12h
16h
20h
Oh
Time of day
Time
day
Time of day
Figure 3: Electricity demand without (left) and with wind (middle), and steam demand (right)
15 minute basis for 253 days in CogenEnv dataset, with random traces highlighted for each.
Radovanovic al. 26 we implement workload execution delay by artificially limiting the total datacenter capacity with a virtual capacity curve (VCC) at each time step. If more jobs are enqueued than the VCC permits, then the jobs must wait until the VCC is raised high enough allow the jobs to run. Simulation is carried out by replaying a sample of real job traces from a Google cluster from May 2019 (25). One timestep in the environment corresponds to one hour, and each episode lasts the whole month.
Observation Space. An observation s(t) (a(t - 1), de, n, m=3++23))) R227 contains the active VCC a(t - 1) set from the previous time step, currently rumning compute load d, number of jobs waiting to scheduled n, as well as the forecasted MOER for the next 24h f
Action Space. At time f, the agent sets the VCC, a(t) [0, 1]. for the next time step. This action denotes the fraction of the datacenter's maximum capacity allowed to be allocated by the scheduler.
Reward Function. The reward consists of two components that encourage the agents to trade-off between scheduling more jobs and reducing associated carbon emissions. The first component penalizes the agent when jobs are scheduled more than 24h after they were originally submitted. The second component is a carbon emissions cost. Formally, the reward is specified as
23
r(t) de + (1%24400 max 0. 0,0.97 - C
Σ a(t h)
h=
where d, my is the carbon emissions, C' is the datacenter's maximum capacity, and w; is the total job-hours of enqueued jobs on that day.
Distribution Shift. The distribution shift in DatacenterEny comes from changes in the MOER between 2019 and 2021.
2.4 CogenEnv
CogenEnv simulates the operation of a combined cycle gas power plant tasked with meeting local steam and energy demand. Conventional dispatchable generators suffer decreased efficiency as a result of frequent ramping, posing particular challenge as increasing penetrations of variable renewables necessitate larger and more frequent ramps to ensure of cogeneration resources requires balancing the competing objectives of minimizing fuel use, anticipating future ramp needs, and ensuring delivery of sufficient energy and steam to the grid.
balance. Thus, optimal operation
While CogenEnv models specific combined cycle gas generation plant in the U.S. (anonymized and location withheld for security reasons), the basic environment setup representative prototype of more general dispatchable resource generation control tasks, due to its complexity (the number of variables, mixed continuous/binary decisions, complementary trains of the plant all needing to be controlled together). In addition, the environment is readily modifiable to accommodate other cost structures (e.g., changing the relative magnitude of the constraint penalties vs. the ramping cost).
Observation Space. An observation takes the form
st)) (r, (t 1), Teatk, PEL+ky HEatk d destkr
d8 dettky TEttk [((+));
where r = 1/96 is the time (normalized by number of 15 minute intervals in a day), att 1) the agent's 's previous action, Thttk Pac-k and Hut+ are current and & forecast steps of temperature,

---

pressure, and relative humidity, respectively, d.t+ and d.t4 are current and k forecast steps of electricity and steam demand, respectively, and Tei+: and x TE are current and k forecast steps of electricity and fuel price, respectively.
Action Space. The action space is a vector a(t) IR'S specifying dispatch setpoints and other auxiliary variables for all turbines in the plant. Specifically, for each of three gas turbines, the agent specifies (a) a scalar turbine electricity output, (b) a scalar heat recovery steam flow, (c) binary evaporative cooler switch setting, and (d) a binary power augmentation switch setting. In addition, for the steam turbine, the agent specifies (a) scalar turbine electricity output, (b) a scalar steam flow through the plant condenser, and (c) an integer number of cooling tower bays employed.
Reward Function. The reward function is comprised of three components:
r(t) (ry(a(t); T,, P,, H.) + ry (a(((t);
1)) re(a(t);
Ty(a(t); Tr, Pt, Ha) is the generator fuel consumption in response to dispatch a(t). rr(a(t); 1)) the ramp cost, captured via an l norm penalty for any change in generator electricity dispatch between consecutive actions. re(a(t); dff, d²) is constraint violation penalty, penalizing any unmet electricity and steam demand, as well as any violation of the plant's dynamic operating constraints. The sum of these three components is negated to convert costs to rewards.
Distribution Shift. CogenEnv considers distribution shifts in the renewable generation profiles, and specifically, increasing penetration of wind energy. This increased variable renewable energy on the grid necessitates more frequent ramping in order to meet electricity demand, and may pose challenge for RL algorithms trained on electricity demand traces without such variability.
Multiagent Setting. The multiagent setting treats each turbine unit (each of the three gas turbines and the steam turbine) as an individual agent whose action is the turbine's ' electricity dispatch decision and auxiliary variable settings. The negative reward of each agent is the sum of the corresponding turbine unit's fuel consumption, ramp cost, and dynamic operating constraint penalty, as well as a shared penalty for unmet electricity and steam demand that is split evenly across agents. All agents observe the global observation.
2.5 BuildingEnv
BuildingEnv considers the control of the heat flow in multi-zone building so to maintain desired temperature setpoint. Building temperature simulation uses first-principled physics models. Users can either choose from pre-defined list of buildings (Office small, School primary, Apartment midrise, and Office large) and three climate types and cities (San Diego Tueson, New York) provided by the Department of Energy (DoE) Building Energy Codes Program (27] or define customized BuildingEnv environment by importing any self-defined EnergyPlus building models. Each episode runs for 1 day, with 5-minute time intervals (H 288, 5/60 hours).
Observation Space. For a building with M indoor zones, the state s(t) E RR. M+H contains observ- able properties of the building environment at timestep t:
s(t)
(t), (t()). (t),), QGH (t),
where Tt(t) is zone i's temperature at time step t, (t( is the heat acquisition from occupant's activities, QHI (t) the heat gain from the solar irradiance, and (t and Tz(() denote the ground and outdoor environment temperature. In practice, the agent may have access to all or part of the state variables for decision-making depending on the sensor setup. Note that the outdoor/ground temperature, room occupancy, and heat gain from solar radiance are time-varying uncontrolled variables from the environment.
Action Space. The action a(t) E 1]M sets the controlled heating supplied to each of the M zones, scaled [1,
The objective is to reduce energy consumption while keeping the temperature within a given comfort range. The default reward function is weighted &g reward, defined
Reward Function.
r(t) = (1 3) |a(()]l2 Balgra (t) T(t)ll2
where Trtargt (1) [7] target (4), (l),..., T Tataeet (1)) are the target temperatures and T(t) = (Tr (I)...., Tar [T) are the actual zonal temperatures. BuildingEnv also allows users to cus- tomize reward functions by changing the weight term 3 or the parameter p defining the lp norm.
7

---

Table Distribution shift experiments
Environment
What shifts
Original setting
Shifted setting
EVChargingEnv
EV sessions, MOER
Summer 2019
Summer 2021
DatacenterEnv
MOER
May 2019
May 2021
CogenEnv
Wind penetration
MW wind
300 MW wind
BuildingEnv
Ambient temperature
Summer 2004
Winter 2003
Users can customize the reward function to consider CO2 emissions and temperature constraints such as upper and lower temperature bounds.
Distribution Shift. BuildingEnv features distribution shifts in the ambient outdoor temperature profile Tg which varies with different seasons. BuildingEnv supports the distribution shifts due to the variation of seasons, located cities of the buildings, and can examine the challenges brought by such shifts in the RL environment.
Multiagent Setting. In the multiagent setting for Bui Buil BuildingEnv, each agent controls the heating action for single zone in the building. It must coordinate with other agents to maximize overall reward. Each agent obtains the same global observation and reward.
3
Experiments
For each of the environments in SustainGym, we implemented baseline non-RL algorithms as well as off-the-shelfe RL algorithms trained using either RLLib (28] Sasble-Bsseline33 (SB3) [29] For most environments, we tested off-policy soft actor-critic (SAC) 30 and on-policy proximal policy optimization (PPO) [31]. Note that neither RLLib nor SB3 has an implementation of SAC that supports mixed discrete and continuous actions, as found in CogenEnv. For EVCharg EicchrrgingEv, we also tested multi-agent implementations of PPO and SAC, where the same policy is shared across agents. Non-RL algorithms tested include random policies and model predictive control (MPC), which is a model-based controller. Detailed descriptions of the implementations for each algorithm, including syper-parameter tuning, given in Appendix B Finally, to test distribution shift, trained RL. agents both "original" and "shifted" environments, then compared their performance on the shifted environment, described in Table
Discussion and Conclusion
Our experiments, shown in Figure 4 demonstrate a wide range of outcomes for off-the-shelf RL algorithms, with single algorithm outperforming all the rest. In EVChargingEn' for example, most of the RL algorithms perform no better than random actions, with the exception multi-agent PPO with discrete actions. On DatacenterEny and Bui BuildingEnv, returns across the different RL algorithms. In contrast, model-based MPC algorithms, where available, tend to perform
notice a wider spread of
consistently than most RL algorithms.
In terms of distribution shift, we see a wide range of outcomes between agents trained on the original environments versus the shifted environments. Surprisingly. in CogenEnv, both single- agent and multi-agent policies trained on the shifted environment perform worse on the shifted environment than agents trained on the original environment. We believe this result may be due to the increased variability of shifted environment, making the shifted environment harder to learn in. In DatacenterEnv, the shift in MOER values shows essentially effect agent performance. In EVChargingEnv, agents trained on the shifted environment generally perform slightly better than agents trained the original environment. In BuildingEnv, agents trained on the shifted environment perform much better.
These results highlight that the distribution shifts present in SustainGym environments provide substantial opportunities for future research, including robust RL algorithms (32 as well as online learning under distribution shift. Developing RI. algorithms that are robust these natural distribution shifts will be critical for deploying RI. the real-world high-impact sustainability settings such as those modeled by SustainGym environments.

---

trained on
Summer 2019
Summer 2021
Daily Raturn
Offline optimal
Greedy
MPC-3
Rendom
PPO
PPO-d
SAC
MA-PPO
MA-PRO-d
MA-SAC
Algerithm
(a) EVChargingEny
150000
Daily Return ($)
-20
100000
Daily Return
40
50000
25000
trained
-80
2013
2021
-25000
Random
Algonitim
Algorithm
(b) ElectricityMarketeBnv
(c) DatacenterEny
1e7
Daily Return
Daily Return
-20
-30
trained on
0MW wind
-40
trained
Summer
-30
Winter
Random
MPO Algorithm
MA-PO
MPC
Random
SAC
PPO PPO discrete
A2C
A2C discrete
Algorithm
(d) CogenEnv
(c) BuildingEnv
Figure 4: Experimental results for all 5 environments comparing performance the shifted environ- ment between RI. algorithms trained on the original environment (blue) and RI. algorithms trained on the shifted environment (orange). Policies using discretized actions indicated with "-d", and multi-agent policies are prefixed with "MA-", For a complete description of the experiments, please Appendix B
Multi-agent RT. SustainGym is eurrently designed to support multi-agent RL in 3 environments, with the goal of upgrading all environments with multi-agent support in the future. In the two environments for which tested multi-agent RI. policies (EVChargingEnv and CogenEnv), the multi-agent PPO (MA-PPO) policies out-performed all other RL policies. We suspect that this may be because the action spaces these environments factorize well across agents, the multi-agent policies can learn more efficiently. Furthermore, we suspect that multi-agent policies may have the potential of performing better under distribution shift, since their environments are naturally non-stationary during training. We notice this to be true for both EVCharg "'ChargingEEv and CogenEnv: of the RL policies trained on the original environments, the multi-agent policies performed best when tested on the shifted environment.
Future work. SustainGym is under active development, with several key directions of future work:
Comprehensive support for multi-agent RL. Currently, only 3 out of the environ- ments support multi-agent RL. We are working to extend the two other environments ElectricityMaarketBi and DatacenterEnv to the multi-agent RI. setting. For ElectriicttyMarkeeEEnn we plan on introducing multiple batteries into the same trans- mission network, each controlled by separate competing agents. This will be the first

---

competitive multi-agent RL environment in SustainGym. (All other environments feature cooperative multi-agent RL.) For DatacenterEnv, we plan on introducing multiple datacen- ters spread across geographic regions to enable both temporal and geographic carbon-aware own agent.
load shifting. Each datacenter would
Different degrees of distribution shift. Currently, SustainGym environments feature a binary choice distribution shift: an original environment, and shifted environment. We plan on introducing more settings with varying degrees of distribution shift.
More environments. We welcome new environment ideas and contributions to SustainGym and working with potential collaborators to extend the scope of environments.
Limitations We conclude by acknowledging general limitations of SustainGym. First, SustainGym only captures very limited dimensions of sustainability (f.e., energy and CO2 emissions) and does not account for other aspects such as water usage and other pollutants associated with energy production. We welcome collaboration with experts in these other sustainability domains to help us improve the sustainability mission of SustainGym. Second, SustainGym is limited in the types of distribution shifts that are considered. Finally, while SustainGym environments have been designed to be reasonably representative of various sustainable energy settings, there is inevitably a gap between SustainGyym simulations and actual hardware systems. A detailed discussion of the representativeness, generalizability, and limitations of each environment can be found in Appendix
Acknowledgments and Disclosure of Funding
We would like to acknowledge Steven Low, Tongxin Li, Zachary Lee, Lucien Werner, Zaiwei Chen, Ivan Jimenez, Pieter Van Santvliet, and Ameera Abdelaziz for their feedback and input during the preparation of SustainGym. The model underlying CogenEnv was developed in partnership with Beyond Limits and Enexsa. Funding for this project comes from a variety of sources, including NSF awards CNS-2146814, CPS-2136197, CNS-21064033, EPCN-2200692, and NGSDI-2105648; an NSF Graduate Research Fellowship; Amazon A14Science Fellowship; the Resnick Sustainability Institute; Hellman Fellowship; Amazon Web Services; and Beyond Limits. This work partially supported by the Department of the Air Force under Air Force Contract No. MA47702-55--0000 Any opinions, findings, conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the Department of the Air Force. Notwithstanding any copyright notice, U.S. Government rights in this work are defined by DFARS 252.227-7013 or DEARS 252.227-7014 as detailed above. Use of this work other than specifically authorized by the U.S. Government may violate any copyrights that exist in this work.
10

---

References
[1] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A. Rusu, Joel Veness, Marc G. Bellemare, Alex Graves, Martin Riedmiller, Andreas K. Fidjeland, Georg Ostrovski, Stig Pe- tersen, Charles Beattie, Amir Sadik, Ioannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wierstra, Shane Legg, and Demis Hassabis. Human-lever control through deep inforcement learning. Nature, 518(7540):529- commaatticeess//ntuureeee htt.leessnauuuee422200
February 2015. ISSN 1476-4687. doi:
10. 10.1038/nature1445 14236. URL //www naturantrr.... com wwm/articlees.
[2] David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, Yutian Chen, Timothy Lillicrap, Fan Hui, Laurent Sifre. George van den Driessche, Thore Graepel, and Demis Hassabis. Mastering the game of Go without human knowledge. Nature, 550(7676):354-355 doi: (0.1038/naturc22270. URL hRttps/:://ww www.:/r nature
October 2017. ISSN 1476-4687.
com/ commaaticles///// naaur
[3] Jens Kober, J. Andrew Bagnell, and Jan Peters. Reinforcement learning in robotics: A sur-
vey. The International Journal of Robotics Research, 32(11):1238 11112228--744, September 2013. ISSN 0278-3649. doi: 10. 10.117700288644113495557 0278364913495721
URL https://dot wwweddi...ggg//////eee 1177/
[4] Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. OpenAl Gym. arXiv:1606.01500 [cs], June 2016. URL
http://arxiv.orrt http://axii..oogggb///b/i0 01540
[5] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg. and Li Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision, 115(3)211-2255 252, December 2015. ISSN 1573-1405. doi: 10.1007/s1 126007051112 URL https://doi /ttp::////oooor///////// 11t.://...100011111---------1110
[6] Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding. In International Conference on Learning Representations, September 2018. URL https wttp://wwweettoooonnnaiddaaa J4mm2225577
[7] Christopher Yeh, Chenlin Meng. Sherrie Wang, Anne Driscoll, Erik Rozi, Patrick Liu, Jihyeon Lee, Marshall Burke, David B. Lobell, and Stefano Ermon. SustainBench: Benchmarks for Monitoring the Sustainable Development Goals with Machine Learning. In Thirty- Conference Neural Information Processing Systems Datasets and Benchmarks Track (Round URL httphtttp/////eeeer//// net orummid-
2). 12 2021. doi: 10.48550/arXiv22111..8 55H3vCylqD
[8] Sindhu Padakandla, Prabuchandran K. J, and Shalabh Bhatnagar. Reinforcement Learning in Non-Stationary Environments. Applied Intelligence, 50(11):35903306, November 2020. ISSN doi: (0.10077/410489-000--01333 0924-669X, 1573-7497.
URL http://ra/// .rg. /abs/
19055009700
[9] Elena Smimova, Elvis Dohmatob, and Jérémie Mary. Distributtonally Robust Reinforcement Learning, June 2019. URL httpp::/aaxdw.org/ab////////////
[10] Marcus Lapeyrolerie, Melissa S. Chapman, Kari F. A. Norman, and Carl Boettiger. Deep Reinforcement Learning for Conservation Decisions, June 2021. URL http:://axiii.o////// org/ 0ba/2106.08277
[11] David Biagioni, Xiangyu Zhang, Dylan Wald, Deepthi Vaidhynathan, Rohit Chintala, Jennifer King, and Ahmed S. Zamzam. PowerGridworld: A Framework for Multi-Agent Reinforcement Learning in Power Systems. November 2021. doi: 10.48550/arXiv.2111 11.0969.. URLp
7/arxiv. / 01969
[12] Jose R. Vazquez-Canteli, Sourav Dey, Gregor Henze, and Zoltan Nagy. Cityl CityLearn: Standardiz- ing Research in Multi-Agent Reinforcement Learning for Demand Response and Urban Energy Management, December 2020. URL //tpxx/xxvvv/////e www.or/////////////o/ooooeeeeee/ abs abs/2012.10504

---

[13] Zachary J. Lee, Sunash Sharma, Daniel Johansson, and Steven H. Low. ACN-Sim: An Open- Source Simulator for Data- Driven Electric Vehicle Charging Research. IEEE Transactions on Smart Grid, 121)):513,, 11 2021. ISSN 1949-3061. doi: 10.1109/TSG.2221331133
[14] Georgios Karatzinis, Christos Korkas. Michalis Terzopoulos, Christos Tsaknakis, Aliki Ste- fanopoulou, lakovos Michailidis, and Elias Kosmatopoulos, Chargym: An EV Charging Station Model for Controller Benchmarking. IFIP Advances in Information and Communi- cation Technology. pages 241-252, Cham, 2022. Springer International Publishing. ISBN 978-3-031-083411 doi:
[15] Heba M. Abdullah, Adel Gastli, and Lazhar Ben-Brahim. Reinforcement Learning Based EV Charging Management Systems-A A Review. IEEE Access, 0411006-1553) 2021. ISSN 2 2169-3536. doi: 10. 21t////CCSSS222........
[16] Hao Wang and Baosen Zhang. Energy Storage Arbitrage in Real-Time Markets via Reinforce- ment Learning. In 2018 IEEE Power Energy Society General Meeting (PESGM), pages 1-5,
August 2018. doi: 10. 100/PESM..001...56666.6
[17] Hanchen Xu, Xiao Li, Xiangyu Zhang, and Junbo Zhang. Arbitrage of Energy Storage in Electricity Markets with Deep Reinforcement Learning, May 2019. URL http:httt:////iii abs/1904. 12232
[18] Jun Cao, Dan Harrold, Zhong Fan, Thomas Morstyn, David Healey, and Kang Li. Deep Reinforcement Learning- -ased Energy Storage Arbitrage With Accurate Lithium- Battery Degradation Model. IEEE Transactions on Smart Grid, 11(5):4513 4521, September 2020. ISSN 1949-3061. doi:
[19] Zachary J. Lee, Tongxin Li, and Steven H. Low. ACN-Data: Analysis and Applications of an Open EV Charging Dataset. In Proceedings of the Tenth ACM International Conference on Future Energy Systems, e-Energy 19, pages 139-149, New York, NY, USA, June 2019, Association for Computing Machinery. ISBN 978- 778-1-4503-6671-7 3328313. URL https: orgg 10. 1145 /2445//337772.. 3328333
doi: 10.1145/330777..
[20] Zachary J. Lee, George Lee, Ted Lee, Cheng Jin, Rand Lee, Zhi Low, Daniel Chang, Christine Ortega, and Steven H. Low. Adaptive Charging Networks: A Framework for Smart Electric Vehicle Charging. IEEE Transactions on Smart Grid, 12(5):4339 43500, September 2021.
ISSN 1949-3061. doi: 10. (0.1109/TSG.2221.307744 document/9409112
URL [Rtp:ttts://eeeexppooo .ieleorreeeeee....... org/
[21] Tongxin Li, Ruixiao Yang, Guannan Qu, Yiheng Lin, Steven Low, and Adam Wierman. Equip- ping Black-Box Policies with Model-Based Advice for Stable Nonlinear Control, June 2022.
URL http:://arxii/ wttp://wwwwwrpa/////ea/////// 01341
[22] Probability Methods Subcommittee, Ieee reliability test system. IEEE Trunsuctions on Power Apparatus and Systems, PAS-9966))2044220544 1979. doi: 10. 10.110900 10.100/TPPA.1199 1979.3199999
[23] Clayton Barrows, Aaron Bloom, Ali Ehlen, Jussi Ikäheimo, Jennie Jorgenson, Dheepak Krishna- murthy, Jessica Lau, Brendan McBennett, Matthew O'Connell, Eugene Preston, Andrea Staid, Gord Stephen, and Jean- Paul Watson. The IEEE Reliability Test System: A Proposed 2019 Update. IEEE Transactions on Power Systems, 35(1):119-12)) 127, January 2020. ISSN 1558-0679. doi: wttp://PRSS.209.9999999
[24] Abhishek Verma, Luis Pedrosa, Madhukar R. Korupolu, David Oppenheimer, Eric Tune, and John Wilkes. Large-scale cluster management Google with Borg. In Proceedings of the European Conference on Computer Systems (EuroSys), Bordeaux, France, 2015. URL https://researchh wwc.tl..oogggle.ee.. wooggle/pubs/ppb43330
[25] Muhammad Tirmazi, Adam Barker, Nan Deng, Md Ehtesam Haque, Zhijing Gene Qin, Steven Hand, Mor Harchol-Baltar Balter, and John Wilkes. Borg: the Next Generation. In EuroSys' 220 Heraklion, Crete, 2020. URL httpp:///reeaarr..oooleee///eeee
[26] Ana Radovanovic, Ross Koningstein, Ian Schneider, Bokan Chen, Alexandre Duarte, Binz Roy, Diyue Xiao, Maya Haridasan, Patrick Hung, Nick Care, et al. Barbon-aware computing for datacenters. IEEE Transactions Power Systems, 38(2): 2022.
12

---

[27] The Building Energy Codes Program. Prototype building models. energycodes, wtw.://dw...ooooooooooeeeeeeeeee building bodd----ee-ildddngdmddddddeeee models
URL https://wwy
[28] Eric Liang, Richard Liaw, Robert Nishihara, Philipp Moritz, Roy Fox, Ken Goldberg, Joseph Gonzalez, Michael Jordan, and Ion Stoica. Rllib: Abstractions for distributed reinforcement learning. In International Conference on Machine Learning, pages 5053-3062 PMLR, 2018.
[29] Antonin Rafin, Ashley Hill, Adam Gleave, Anssi Kanervisto, Maximilian Ernestus, and Noah Reliable Reinforcement Learning Implementations. Journal of Machine Learning Research, 22(268):1 2021. ISSN 1533-7928. http:///// org/
Dormann.
papers/v22/20- 1364. html
[30] Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft Actor-Critic: Off- Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor, August 2018. URL httttp://arxiv.org/aeea httgg/bb/18811.... 01290
[31] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal Policy Optimization Algorithms, August 2017. /RLLhhtpp//axiiii org/ httg/bb//777... 1tt./1070000880
[32] Kishan Panaganti, Zaiyan Xu, Dileep Kalathil, and Mohammad Ghavamzadeh. Ro- bust reinforcement learning using offline data. In S. Koyejo, Mohamed, A. Agar- wal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Informa- tion Processing Systems, volume 35, pages 32211-32224. Curran Associates, Inc., 2022. URL https: https://proceedinggs
mewproceeddinggs..uu.pppp....pp wwwrips.ripp.. ccp http:/apprrfeeleeee pdf
[33] Timothy P. Lillicrap. Jonathan J. Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning, July 2019. URL httt::///rriiii org/ 0abs//5099.22
[34] Aled James and Daniel Schien. A low carbon kubernetes scheduler. In 6th International Con- ference on ICT for Sustainability, ICT4S 2019, volume 2382 of CEUR Workshop Proceedings. CEUR-WS, June 2019.
[35] Nicolas Christianson, Christopher Yeh, Tongxin Li, Mahdi Torabi Rad, Azarang Golmoham- madi, and Adam Wierman. Robustifying machine-learned algorithms for efficient grid operation. In NeurlPS 2022 Workshop Tackling Climate Change with Machine Learning, 2022. URL
https://www. httpp::/www.cc.mtrecaaangg aat.aa///aaa at/papprrs/neurips2222/
[36] J. King, A. Clifton, and B. Hodge. Validation of Power Output for the WIND Toolkit. Technical Report NREL/TP-SD00-1177 1159354, September 2014. URL http://www osti gov/ serviets/purl/11553
[37] CAISO. What the duck curve tells us about managing green grid. Technical report, California Independent System Operator, November 2016.
[38] Chi Zhang, Yuanyuan Shi, and Yize Chen. BEAR:
Building Environment for Control and Reinforcement Learning. In Proceedings of the Fourteenth ACM International Conference on Future Energy Systems, e-Energy '23. arXiv, November 2022. doi: 10.48550/
arXiv.2211.1474. /ttpttpp//arxxi
abs 221122111121 14744
[39] US DOE. Engineering reference. httpss//ennrrgpplus wwt. http///sset//rreecssttoomm p pdff//ddf_sr wdff_v222.2. 1. wttp__222.......nnnnnee wtt.://nineeiinneeeeeeeee.eeepee 2022.
[40] Volodymyr Mnih, Adrià Puigdomènech Badia, Mehdi Mirza, Alex Graves, Timothy P. Lilli- crap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous Methods for Deep Reinforcement Learning, June 2016. Utt.http: http://axivv... org/ htrg/abs/1602.00 01783
13


Reviews:
[{"review_id": "luh666V5gf", "reviewer": "Reviewer_4BfE", "summary": "", "questions": "", "limitations": "I disagree that the authors have discussed the limitations of their work, the checklist states this is in Section 4. There are a number of limitations worth discussing, including but not limited to:\n- Some distribution shifts are modelled, others are not. There are many sources of distribution shift in the real world, which ones do you not capture? How do you estimate the effect of ignoring those?\n- The environments are based on specific implementations. E.g. the EV network uses data from Caltech and JPL charging stations. How does this generalize to other EV station? A discussion on how representative this is would be very welcomed.\n- Likewise, the generalizability of the other environments. Different electricity markets have very different rules for managing loads and offer spreads. How general is this implementation? Is it modelled on the California market? Are there any major world markets that are incompatible with this implementation? ", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "- The application of sustainable energy improvement is a signifiant and valuable one that requires further optimization. This helps in that endeavor.\n- The Gym is released on Github\n- The five environments cover a range of settings within sustainable infrastructure", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "WvKA5pg4w6", "reviewer": "Reviewer_P6Jo", "summary": "", "questions": "", "limitations": "Some limitations are given in Section 4. However, it would be interesting to split the limitations in two parts: those related to the environments, and those due to the algorithms.", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "The paper clearly describes each of these five environments.The definition of each enviroment is clearly stated and justified, in particular the fact of taking distribution shifts into account.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "JVLypqGEJ7", "reviewer": "Reviewer_pDRB", "summary": "", "questions": "", "limitations": "The limitations are discussed in Section 4.", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "* Sustainability is a paramount topic, and exploring how to harness RL to tackle sustainability tasks is a truly inspiring direction.\n* The introduced environments are built upon realistic sustainability tasks, offering significant practical value.\n* The descriptions of the environments are clear and detailed.\n* Overall, the paper is clearly written and easy to follow.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "ECIcq6EbsX", "reviewer": "Reviewer_E4iv", "summary": "", "questions": "", "limitations": "The discussion is very brief and I would expect to see some acknowledgment of the limitations of SustainGym. An obvious limitation is the lack of consideration of other sustainability dimensions. Sustainability goes beyond considerations of just energy and CO2 emissions and should consider factors such as water usage, eutrophication potential, etc., where possible. Another limitation is the representativeness of these sustainable energy tasks.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "- The development of SustainGym addresses the need for more realistic and sustainable RL environments that model real-world scenarios.\n- The environments test RL algorithms under distribution shifts and multi-agent settings, which can provide researchers with a better understanding of how algorithms would perform in real-world situations.\n- The paper offers an extensive evaluation of current off-the-shelf RL algorithms using SustainGym, which is beneficial for future research and benchmarking.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}]

Author rebuttal:

Meta-review: ### Summary And Contributions:
The authors present a set of environments to train and test reinforcement learning agents in the field of sustainable energy systems. The set, SustainGym, includes tasks of: EV charging, electricity market, datacenter carbon optimization, cogeneration plants and thermal building control. The sim also incorporates changes (distribution shifts) in demand and changes in environmental factors.

### Strengths:
- The application of sustainable energy improvement is a significant and valuable one that requires further optimization. This helps in that endeavor.
- The definition of each environment is clearly stated and justified, in particular the fact of taking distribution shifts into account.
- The Gym is released on Github.

### Opportunities For Improvement:
- The experiments conducted only use a small subset of existing RL algorithms, limiting the paper's ability to draw broader conclusions about RL's applicability to sustainable energy tasks.
- A more detailed discussion of multi-agent settings and physical constraints.
- Future work: how to expand the current 5 environments to incorporate and model other factors.

### Clarity:
- The paper is well written and clear with good sections and flow.

Overall, the reviewers agreed that SustainGym is a good idea, well formulated, clearly defining and new set of environments focused on sustainable energy systems.

Since the authors have incorporated the feedback provided by the reviewers and improved the paper sustainably as reviewers asked for, I'm recommending acceptance.
Decision: Accept (Poster)