Real-World Data and Calibrated Simulation Suite for Offline Training of Reinforcement Learning Agents to Optimize Energy and Emission in Buildings for Environmental Sustainability

Authors: Judah Goldfeder, John Sipple
Primary area: datasets and benchmarks
Keywords: Reinforcement Learning, HVAC Control, Simulator, RL Environment, Environmental Sustainability, Climate, Time-series prediction

Abstract:
Commercial office buildings contribute 17 percent of Carbon Emissions in the US, according to the US Energy Information Administration (EIA), and improving their efficiency will reduce their environmental burden and operating cost. A major contributor of energy consumption in these buildings are the Heating, Ventilation, and Air Conditioning (HVAC) devices. HVAC devices form a complex and interconnected thermodynamic system with the building and outside weather conditions, and current setpoint control policies are not fully optimized for minimizing energy use and carbon emission. Given a suitable training environment, a Reinforcement Learning (RL) agent is able to improve upon these policies, but training such a model, especially in a way that scales to thousands of buildings, presents many practical challenges. Most existing work on applying RL to this important task either makes use of proprietary data, or focuses on expensive and proprietary simulations that may not be grounded in the real world. We present the Smart Buildings Control Suite, the first open source interactive HVAC control dataset extracted from live sensor measurements of devices in real office buildings. The dataset consists of two components: six years of real-world historical data from three buildings, for offline RL, and a lightweight interactive simulator for each of these buildings, calibrated using the historical data, for online and model-based RL. For ease of use, our RL environments are all compatible with the OpenAI gym environment standard. We also demonstrate a novel method of calibrating the simulator, as well as baseline results on training an RL agent on the simulator, predicting real-world data, and training an RL agent directly from data. We believe this benchmark will accelerate progress and collaboration on building optimization and environmental sustainability research.

Decision: reject
Reviewer scores: 5, 3, 3, 5

Reviews and discussion:
<thread>
creation date:
1730421841603

thread_type:
official_review

<thread_object>
creator: reviewer_npuo

<content>
summary:
The paper introduces the Smart Buildings Control Suite, an open-source suite for offline training of RL agents for energy and emission optimization in buildings. It provides real-world HVAC data from three buildings and a calibrated simulation environment compatible with OpenAI's Gym. The suite aims to bridge the gap between simulation and real-world application by providing scalable tools that emulate building dynamics and facilitate RL training. The authors demonstrate a novel calibration method and showcase baseline results using RL and time-series models.

strengths:
- The availability of an open-source dataset and simulator addresses a critical gap in building energy optimization research by providing accessible benchmarks.
- The novel calibration method to align simulation with real-world data is a practical and relevant approach.
- The evaluation of predictive accuracy and its applicability for RL training adds to the suite's potential for broader adoption.
- The focus on optimizing building energy and emissions is timely and has significant implications for sustainable development.

weaknesses:
- While the findings are validated through simulations, it would be helpful to understand if the authors have plans for real-world validation in future work. Could the authors discuss how the insights from their simulations might translate to real-world performance, possibly supported by prior studies or existing literature?
- The dataset, limited to three buildings in California, constrains the applicability of the results. It would be beneficial for the authors to include or plan for the inclusion of diverse building types or climates. Could they consider expanding their dataset in future work to encompass various climate zones or building structures, such as commercial or mixed-use buildings?
-  The focus on temperature calibration is clear, but it would be valuable for the authors to explain the rationale behind this choice. To enhance the study, could they incorporate or discuss additional energy consumption or emissions metrics, such as total energy use, CO2 emissions, or HVAC load efficiency?
- The lack of discussion on recalibration or retraining frequency is noted. Could the authors elaborate on potential strategies for adaptive recalibration, or provide an estimation of how often recalibration or retraining would be needed, based on their current experience with the data or anticipated building condition variations?
- A comparison with established methods like OCTOPUS [1] could provide valuable context for the proposed calibration method. Could the authors elaborate on specific aspects that differentiate or improve upon their method compared to the OCTOPUS approach or other similar frameworks, highlighting innovations or strengths?
- The benchmarking results provided are limited, with minimal comparative data between the baseline and the SAC-trained agent. Could the authors expand this section by including additional metrics such as total energy consumption, carbon emissions, or setpoint deviations? Visualizations like performance graphs or distribution plots would also enhance the understanding of the agent’s impact on overall building performance.
1. Ding, Xianzhong, Wan Du, and Alberto Cerpa. "OCTOPUS: Deep reinforcement learning for holistic smart building control." In Proceedings of the 6th ACM international conference on systems for energy-efficient buildings, cities, and transportation, pp. 326-335. 2019.

questions:
- Could the authors provide more insights on how their simulator and RL agent would perform with different climates or building types?
- How robust is the simulator to noisy sensor data or partial information, which are common in real-world scenarios?
- Could the authors elaborate on the computational efficiency and practical deployment requirements of their suite?
- How adaptable is the calibration process to significant changes in building usage or occupancy patterns? Would recalibration require the same data volume and manual effort?
- Given that energy consumption and emissions are key objectives, why weren’t these included as primary metrics in the calibration?
- In the OCTOPUS paper[1], the calibration involved collecting detailed real-world data such as weather, occupancy schedules, and HVAC system parameters. Could the authors discuss how their calibration method compares to this approach?
- The Demonstration Benchmarking Results section reports limited metrics. Could the authors provide more comprehensive results including detailed metrics like energy consumption, carbon emissions, and setpoint deviations to better evaluate the policy's robustness and performance?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730609323643

thread_type:
official_review

<thread_object>
creator: reviewer_vsqv

<content>
summary:
The paper proposes a dataset and simulation approach for building HVAC system. Reinforcement learning algorithms are implemented to test the efficacy of proposed approaches. While the proposed approach is technically sound and shows its capability on particular settings, the paper organization is relatively hard to follow, and the contributions in terms of constructing a new dataset or apply RL algorithms are not significant enough for the machine learning community.

strengths:
* The paper collected six years of real-world historical data from three buildings, which is a valuable building data source.
* Proposed method considers the fine-grained ODE models.
* The proposed approach is claimed to strike a better balance between fidelity and execution speed.

weaknesses:
* The paper does not properly address the challenge mentioned by the author, especially on the claim that "but training such a (RL) model, especially in a way that scales to thousands of buildings, presents many practical challenges." As the paper only shows limited device modeling and results on few buildings, it is very hard to justify this claim on scalability.
* The claim on "first open source interactive HVAC control dataset" is not rigorous. There are already several datasets and approaches working in this field.For instance, CityLearn, BEAR: Physics-Principled Building Environment for Control and Reinforcement Learning, and EnergyPlus all provided datasets.
* The main texts only show temperature comparison and a brief spatial errors on one building. These results cannot justify why the proposed method is more appropriate for HVAC RL tasks.
* A more thorough evaluation of modeling parameters, computation time, control performance, generalization to other buildings, comparison with state-of-the-art approaches is highly needed. For instance, compare with other building simulation software in terms of computation time and solution accuracy.
* The paper does not provide further insights nor algorithm development for HVAC control tasks. It is not clear why the proposed simulator is a proper fit for such tasks.
* The optimization of emission is considered in a very simple setting. A further analysis of carbon emission rate, different configurations of HVAC devices such as electric heat pump, water heaters or gas-fired heaters are needed.
* There is a need to demonstrate more detailed results on the efficiency of RL algorithms on these tasks.

questions:
* How does the proposed framework compare to standard building simulation software such as EnergyPlus?
* How is the zone-to-zone heat interaction modeled in the paper?
* In Fig. 26, some predictions are really far off from historical data, can the authors comment on that?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730759626770

thread_type:
official_review

<thread_object>
creator: reviewer_lcet

<content>
summary:
The authors provide a quick thermal diffusion simulation of a building and show that it can be calibrated with real data to provide a simulation that is somewhat close to real world data. The authors provide this as a jumping off point for people interested in training RL agents on building HVAC. The authors show that calibration reduces some error.

strengths:
The framing of the problem is fair, and its an important problem to study. The notion of creating a fast and easily deployable RL training gym in building energy management is the correct path to study if RL is your goal.

weaknesses:
- Quick note – “One of the most significant factors limiting progress is the lack of a reliable public benchmark to test solutions against.” In my experience, a far greater obstacle is just how far behind the hardware of building operating systems, live sensors, etc., are. Teams I know that have studied this wind up spending months just trying to get building sensors to talk correctly to RL agents, or have building systems accept actions from agents. I do not think you would believe just how many bugs there are in controls to hardware and how difficult it is to reliably detect anomalous building readings, be robust to flatlines, etc. So, acknowledging that even the perfectly trained RL agent would not be able to be dropped into a building, regardless of the quality / speed / versatility of the RL simulator, is more true to the practice of this field -- this detail is swept under the rug in this paper, which makes me believe that you are unaware of it. This should be included in the intro but also limitations and future works. And, you should eventually give it a shot if you'd like to see how hard it is!  

- This RL gym is not referenced and seems to include more physics than your simulation. https://github.com/ugr-sail/sinergym. 
Indeed, barely any of the works in Table 1: https://www.sciencedirect.com/science/article/pii/S2666546820300203 are mentioned in your background. I think you reference this paper when referring to EnergyPlus; they seem to do just fine in training an RL agent using EnergyPlus, so I’m not sure what scalability or configuration challenges you’re referring to. Have you tried these other approaches first before developing your own sim? 

- You say that your simulation is physics inspired, but all I see is a thermal diffusion included via finite differences, and some building envelope information included. Fluid phase changes, solar radiance, and the effect of people / computers inside your building are all important physics to consider. These physics matter a lot! I don't see any discussion around how to include them (i.e. at the minimum, provide some thoughts on how to Monte Carlo the variation out could easily account for occupancy, for instance, and likely solar irradiance too. In practice, accounting for these additional physics scenarios doesn't even have to be a huge coding deal -- just some different scenarios that an MC setup can shuffle through.)  In a table, I would outline other major physics inputs and either say how they may be accounted for in future work, why they are unimportant, or what the trade-offs are in omitting them. 

- When you visualize the spatial error, you don’t provide any statistical tests to back up significance. There are no error bars on your error either. Etc. Etc. Etc. There is no attempt to demonstrate anything beyond qualitative measures of success. Thus, the results are unscientific and hand-wavy. This is the biggest reason I chose a negative score. There isn't even much in the ways of a qualitative comparison to your uncalibrated simulation, or a baseline existing simulation, etc. How do we know your results are better?

questions:
- You celebrate the fact that your n-step evaluation loss is novel, but to me, it reads like a very standard absolute value rollout error. Is the novelty the fact that you’ve summed over zones, and does that really deserve its own loss name? Furthermore, is summing over zones perhaps biasing your evaluation to zones that are smaller (seems like each zone is given equal weight in the sum)? Would normalizing by the size of the zone help? 

- There’s no evidence (or discussion) from this work that agent transfer can happen with this dataset. Is this dataset is beneficial to anyone training an agent outside of these specific buildings; i.e. what is the value of this work to the community? Sorry, but I don't see this paper as relevant to the ICLR community. Buildings related conferences like ACM BuildSys (specifically, the Reinforcement Learning for Energy Management workshop) would be good fits for this work. 

- how hard was it to instantiate a simulator with this specific building? I.e. how many engineer hours? 

- Where is your code?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730805097346

thread_type:
official_review

<thread_object>
creator: reviewer_3j5q

<content>
summary:
The work presents The Smart Buildings Control Suite, a benchmarking tool for the optimal control problem of HVAC devices for reducing energy consumption and environmental impact. The contribution is composed of two parts: a real-world historical HVAC dataset relative to three buildings and a highly customizable and scalable low-to-medium fidelity HVAC simulator. Authors employ the aforementioned tool, calibrated by the usage of real-world historical data, in three different ways: 
- training an RL agent and evaluating it against a baseline
- learning a dynamic model through an LSTM encoder-decoder network
- training an RL agent on the previously learned dynamic model.

strengths:
- Concept: HVAC consumption is a hot research topic and authors have clearly highlighted the problem and described the contributions of their work.
- Literature: the authors have presented an exhaustive review of works related to the topic.
- Presentation: the paper is clear and well-written, and the authors provide a very detailed description of every aspect of the framework.
- Collected data: the authors dispose of many real-world data (six years) from three different buildings which contribute to calibrating the simulator.
- Simulator: low-to-medium fidelity physics-based simulator, thought to be lightweight to foster RL experiments.

weaknesses:
- The RL agent is compared to a **control baseline which is never specified in the paper**. Understanding the policy of the baseline is essential to evaluate the RL method and authors may provide a detailed description of the baseline control strategy and its decision-making logic.
- Apart from the unspecified baseline, **the RL agent is not compared with other RL and non-RL strategies**. For example, even simple rule-based policies, such as a fixed action policy, can be considered as additional baselines and can prove the need for an RL solution for the tackled problem. Regarding RL methods, I could suggest trying PPO, A2C, and DDPG, for example.
- The simulator has been calibrated over two days of historical data and validated on a third one for building SB1. While this may be enough to capture building behavior across a day-night cycle, it surely **cannot be sufficient to deal with the building's yearly behavior**, which is affected by seasonal weather and the alternation of work days and weekends. Authors may discuss more in detail how they plan to address seasonal variations and long-term patterns.
- **The training of the RL agent is not completely clear.** From what the authors write, it seems that the agent is trained with the trajectories generated by the simulator calibrated on two days of data (thus almost 580 samples) and trained over 4000 samples, which seems too few for a deep RL agent to learn an optimal policy. I would suggest providing further details about the training process, including how the agent has been fed with simulated time series, how many samples have been used, and how train and test series have been split.
- Since each building behaves differently due to its structure, the **simulator has to be calibrated for each specific building and cannot be generalized**.
- Some **plots are not very clear**. In particular, figure 26 does not have an explicit legend, figure 27 is pixelated and labels on the axes are not readable, and figure 28 is pretty difficult to understand. 
- Supplementary materials containing the **codebase or scripts have not been provided**.

questions:
- I did not understand why the discount factor $\gamma$ is considered in the hyperparameter tuning phase of SAC, since $\gamma$ is not a hyperparameter of the problem, but rather a part of the MDP that defines the problem, thus definitely not something to tune.
- I would ask you to explain figure 18, since I don't understand if the lines represent the cumulative reward of the test episode or the total return across the training phase. Moreover, I am not convinced by the fact that the baseline line is perfectly horizontal. Could you provide an insight about that?
- I noticed that several simulator's hyperparameters are located on the edges of their interval range. Is it something you find reasonable? Are you fine with this outcome or could it lead to undesirable behaviors?
- Based on Figure 26, you stated in Appendix I, lines 2432-2433, that the LSTM model "provides good correspondence with the actual real data signals". However, some signals in these plots are significantly mismatched. Could you provide some numerical metrics and confidence intervals to corroborate your statement?
- I'm not convinced by Figure 20: the $y$ axis has all ticks set to $0.0$ and I cannot understand how to properly read the plot. Could you maybe add a line plot with confidence intervals to compare the baseline and SAC agent, highlighting the horizontal range considered "comfort temperature"? This would be nice to have also for the last experiment of SAC trained with the LSTM model, to confirm that the SAC policy maintains the comfort temperature by reducing carbon emissions, electrical energy, and natural gas energy consumption.
- I did not completely understand the sentence in lines 519-520: "we also trained an RL agent on the learned dynamics model, demonstrating the ability to learn a policy directly from data without involving the simulator". The LSTM model is a simulator, a data-driven one. Moreover, RL is based on training from data via a trial and error approach, thus there is no need for you to demonstrate this.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper presents the Smart Buildings Control Suite, a benchmarking tool that combines a real-world HVAC dataset and a configurable simulator for optimizing building energy consumption through reinforcement learning. While the work addresses an important practical challenge, it has significant limitations including inadequate experimental validation, insufficient comparison with existing solutions, and unclear differentiation from prior simulation approaches.