SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$r_f(a(t);T_t,P_t,H_t)$  is the generator fuel consumption in response to dispatch  $a(t)$ .  $r_r(a(t);a(t - 1))$  is the ramp cost, captured via an  $\ell_1$  norm penalty for any change in generator electricity dispatch between consecutive actions.  $r_c(a(t);d_t^p,d_t^q)$  is a constraint violation penalty, penalizing any unmet electricity and steam demand, as well as any violation of the plant's dynamic operating constraints. The sum of these three components is negated to convert costs to rewards.
Distribution Shift. CogenEnv considers distribution shifts in the renewable generation profiles, and specifically, increasing penetration of wind energy. This increased variable renewable energy on the grid necessitates more frequent ramping in order to meet electricity demand, and may pose a challenge for RL algorithms trained on electricity demand traces without such variability.
Multiagent Setting. The multiagent setting treats each turbine unit (each of the three gas turbines and the steam turbine) as an individual agent whose action is the turbine's electricity dispatch decision and auxiliary variable settings. The negative reward of each agent is the sum of the corresponding turbine unit's fuel consumption, ramp cost, and dynamic operating constraint penalty, as well as a shared penalty for unmet electricity and steam demand that is split evenly across agents. All agents observe the global observation.
2.5 BuildingEnv
BuildingEnv considers the control of the heat flow in a multi-zone building so as to maintain a desired temperature setpoint. Building temperature simulation uses first-principled physics models. Users can either choose from a pre-defined list of buildings (Office small, School primary, Apartment midrise, and Office large) and three climate types and cities (San Diego, Tucson, New York) provided by the Department of Energy (DoE) Building Energy Codes Program [27] or define a customized BuildingEnv environment by importing any self-defined EnergyPlus building models. Each episode runs for 1 day, with 5-minute time intervals ( $H = 288$ ,  $\tau = 5/60$  hours).
Observation Space. For a building with  $M$  indoor zones, the state  $s(t) \in \mathbb{R}^{M + 4}$  contains observable properties of the building environment at timestep  $t$ :
$$
s (t) = \left(T _ {1} (t), \dots , T _ {M} (t), T _ {\mathrm {E}} (t), T _ {\mathrm {G}} (t), Q ^ {\mathrm {G H I}} (t), \bar {Q} ^ {\mathrm {p}} (t)\right),
$$
where  $T_{i}(t)$  is zone  $i$ 's temperature at time step  $t$ ,  $\bar{Q}^{\mathrm{p}}(t)$  is the heat acquisition from occupant's activities,  $Q^{\mathrm{GHI}}(t)$  is the heat gain from the solar irradiance, and  $T_{\mathrm{G}}(t)$  and  $T_{\mathrm{E}}(t)$  denote the ground and outdoor environment temperature. In practice, the agent may have access to all or part of the state variables for decision-making depending on the sensor setup. Note that the outdoor/ground temperature, room occupancy, and heat gain from solar radiance are time-varying uncontrolled variables from the environment.
Action Space. The action  $a(t) \in [-1, 1]^M$  sets the controlled heating supplied to each of the  $M$  zones, scaled to  $[-1, 1]$ .
Reward Function. The objective is to reduce energy consumption while keeping the temperature within a given comfort range. The default reward function is a weighted  $\ell_2$  reward, defined as
$$
r (t) = - (1 - \beta) \| a (t) \| _ {2} - \beta \| T ^ {\text {t a r g e t}} (t) - T (t) \| _ {2}
$$
where  $T^{\mathrm{target}}(t) = [T_1^{\mathrm{target}}(t), \dots, T_M^{\mathrm{target}}(t)]^\top$  are the target temperatures and  $T(t) = [T_1(t), \dots, T_M(t)]^\top$  are the actual zonal temperatures. BuildingEnv also allows users to customize reward functions by changing the weight term  $\beta$  or the parameter  $p$  defining the  $\ell_p$  norm.
Table 2: Distribution shift experiments
Environment	What shifts	Original setting	Shifted setting
EVChargingEnv	EV sessions, MOER	Summer 2019	Summer 2021
DatacenterEnv	MOER	May 2019	May 2021
CogenEnv	Wind penetration	0 MW wind	300 MW wind
BuildingEnv	Ambient temperature	Summer 2004	Winter 2003
Users can customize the reward function to consider  $\mathrm{CO}_{2}$  emissions and temperature constraints such as upper and lower temperature bounds.
Distribution Shift. BuildingEnv features distribution shifts in the ambient outdoor temperature profile  $T_{\mathrm{E}}$  which varies with different seasons. BuildingEnv supports the distribution shifts due to the variation of seasons, located cities of the buildings, and can examine the challenges brought by such shifts in the RL environment.
Multiagent Setting. In the multiagent setting for BuildingEnv, each agent controls the heating action for a single zone in the building. It must coordinate with other agents to maximize overall reward. Each agent obtains the same global observation and reward.
For each of the 5 environments in SustainGym, we implemented baseline non-RL algorithms as well as off-the-shelf RL algorithms trained using either RLLib [28] or Stable-Baselines3 (SB3) [29]. For most environments, we tested off-policy soft actor-critic (SAC) [30] and on-policy proximal policy optimization (PPO) [31]. Note that neither RLLib nor SB3 has an implementation of SAC that supports mixed discrete and continuous actions, as found in CogenEnv. For EVChargingEnv, we also tested multi-agent implementations of PPO and SAC, where the same policy is shared across agents. Non-RL algorithms tested include random policies and model predictive control (MPC), which is a model-based controller. Detailed descriptions of the implementations for each algorithm, including hyper-parameter tuning, are given in Appendix B. Finally, to test distribution shift, we trained RL agents in both "original" and "shifted" environments, then compared their performance on the shifted environment, as described in Table 2.
Our experiments, shown in Figure 4, demonstrate a wide range of outcomes for off-the-shelf RL algorithms, with no single algorithm outperforming all the rest. In EVChargingEnv, for example, most of the RL algorithms perform no better than random actions, with the exception of multi-agent PPO with discrete actions. On DatacenterEnv and BuildingEnv, we notice a wider spread of returns across the different RL algorithms. In contrast, model-based MPC algorithms, where available, tend to perform more consistently than most RL algorithms.
In terms of distribution shift, we see a wide range of outcomes between agents trained on the original environments versus the shifted environments. Surprisingly, in CogenEnv, both single-agent and multi-agent policies trained on the shifted environment perform worse on the shifted environment than agents trained on the original environment. We believe this result may be due to the increased variability of shifted environment, making the shifted environment harder to learn in. In DatacenterEnv, the shift in MOER values shows essentially no effect on agent performance. In EVChargingEnv, agents trained on the shifted environment generally perform slightly better than agents trained on the original environment. In BuildingEnv, agents trained on the shifted environment perform much better.
These results highlight that the distribution shifts present in SustainGym environments provide substantial opportunities for future research, including robust RL algorithms [32] as well as online learning under distribution shift. Developing RL algorithms that are robust to these natural distribution shifts will be critical for deploying RL in the real-world high-impact sustainability settings such as those modeled by SustainGym environments.
(a) EVChargingEnv
(b) ElectricityMarketEnv
(d) CogenEnv
(e) BuildingEnv
Figure 4: Experimental results for all 5 environments comparing performance on the shifted environment between RL algorithms trained on the original environment (blue) and RL algorithms trained on the shifted environment (orange). Policies using discretized actions are indicated with “-d”, and multi-agent policies are prefixed with “MA-”. For a complete description of the experiments, please see Appendix B.
Multi-agent RL. SustainGym is currently designed to support multi-agent RL in 3 environments, with the goal of upgrading all environments with multi-agent support in the future. In the two environments for which we tested multi-agent RL policies (EVChargingEnv and CogenEnv), the multi-agent PPO (MA-PPO) policies out-performed all other RL policies. We suspect that this may be because the action spaces in these environments factorize well across agents, so the multi-agent policies can learn more efficiently. Furthermore, we suspect that multi-agent policies may have the potential of performing better under distribution shift, since their environments are naturally non-stationary during training. We notice this to be true for both EVChargingEnv and CogenEnv: of the RL policies trained on the original environments, the multi-agent policies performed best when tested on the shifted environment.
Future work. SustainGym is under active development, with several key directions of future work:
- Comprehensive support for multi-agent RL. Currently, only 3 out of the 5 environments support multi-agent RL. We are working to extend the two other environments ElectricityMarketEnv and DatacenterEnv to the multi-agent RL setting. For ElectricityMarketEnv, we plan on introducing multiple batteries into the same transmission network, each controlled by separate competing agents. This will be the first
competitive multi-agent RL environment in SustainGym. (All other environments feature cooperative multi-agent RL.) For DatacenterEnv, we plan on introducing multiple datacenters spread across geographic regions to enable both temporal and geographic carbon-aware load shifting. Each datacenter would be its own agent.
- Different degrees of distribution shift. Currently, SustainGym environments feature a binary choice of distribution shift: an original environment, and a shifted environment. We plan on introducing more settings with varying degrees of distribution shift.
- More environments. We welcome new environment ideas and contributions to SustainGym and are working with potential collaborators to extend the scope of environments.
**Limitations** We conclude by acknowledging general limitations of SustainGym. First, SustainGym only captures very limited dimensions of sustainability (i.e., energy and  $\mathrm{CO}_{2}$  emissions) and does not account for other aspects such as water usage and other pollutants associated with energy production. We welcome collaboration with experts in these other sustainability domains to help us improve the sustainability mission of SustainGym. Second, SustainGym is limited in the types of distribution shifts that are considered. Finally, while SustainGym environments have been designed to be reasonably representative of various sustainable energy settings, there is inevitably a gap between SustainGym simulations and actual hardware systems. A detailed discussion of the representativeness, generalizability, and limitations of each environment can be found in Appendix B.
We would like to acknowledge Steven Low, Tongxin Li, Zachary Lee, Lucien Werner, Zaiwei Chen, Ivan Jimenez, Pieter Van Santvliet, and Ameera Abdelaziz for their feedback and input during the preparation of SustainGym. The model underlying CogenEnv was developed in partnership with Beyond Limits and Enexsa. Funding for this project comes from a variety of sources, including NSF awards CNS-2146814, CPS-2136197, CNS-2106403, EPCN-2200692, and NGSDI-2105648; an NSF Graduate Research Fellowship; an Amazon AI4Science Fellowship; the Resnick Sustainability Institute; Hellman Fellowship; Amazon Web Services; and Beyond Limits. This work is partially supported by the Department of the Air Force under Air Force Contract No. FA8702-15-D-0001. Any opinions, findings, conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the Department of the Air Force. Notwithstanding any copyright notice, U.S. Government rights in this work are defined by DFARS 252.227-7013 or DFARS 252.227-7014 as detailed above. Use of this work other than as specifically authorized by the U.S. Government may violate any copyrights that exist in this work.