SlowGuess/NeurIPS_2023_OCR huggingface dataset:
(c)
Figure 2: Forecast uncertainty. Ground truth time series are truncated to previous 24 hours for visibility. Light blue lines are 10 samples from the predicted distribution. a-b) Successful commercial building forecasts. c-d) Failed residential building forecasts.
uncertainty. The best pretrained model Transformer (Gaussian) outperforms the best persistence method Persistence Ensemble on real commercial buildings. However, Persistence Ensemble has better accuracy than the pretrained models on real residential buildings. These results suggest synthetic pretraining is viable for commercial zero-shot STLF—see Sec. 6.2 for more discussion on residential buildings.
5.2 Transfer Learning
This task evaluates fine-tuning on a single target building for which 6 months of data has been collected. Our fine-tuning protocol is to train for a max of 25 epochs using the first 5 months and to use the last month for early stopping with a patience of 2 epochs. To ease the computational burden of fine-tuning a model on each building in the benchmark, we randomly sample 100 residential and 100 commercial buildings and fine-tune separately on each. We fine-tune the pretrained transformers across all sizes and train randomly initialized transformers. Table 4 displays results aggregated by the median over all buildings; see Fig. 4 for the Transformer-S and Transformer-M results. We also report performance for the pretrained models without fine-tuning.
Our results indicate that fine-tuning the pretrained transformers on limited data is likely to improve the performance on the target building. For statistical robustness, we compute average probability of improving NRMSE due to fine-tuning:  $P(X < Y) \coloneqq \frac{100}{N} \sum_{i=1}^{N} \mathbf{1}_{[X_i < Y_i]}$  where  $X_i$  is the pretrained + fine-tuned NRMSE for building  $i$  and  $Y_i$  is the pretrained NRMSE. The Transformer-M models have the highest  $P(X < Y)$ , with the Gaussian model achieving 98% and 73% respectively for commercial and residential buildings and the Tokens model scoring 70% and 62.5% (Fig. 4). For Transformer-L (Gaussian), this drops to 71% and 68% and to 61.5% and 39% for Transformer-L (Tokens). We found that fine-tuning all layers of both pretrained models was necessary—only fine-tuning the model's last layer led to slightly decreased performance (App. K). Encouragingly, the fine-tuned model (Transformer-L (Gaussian)) beats both LightGBM, a strong baseline in the low-data regime, and the Persistence Ensemble on both commercial and residential buildings.
5.3 Empirical Scaling Laws
Pretraining buildings: To characterize how size and diversity of the dataset impacts generalization, we trained the Transformer-M (17M) models on 1K, 10K, and 100K simulated buildings. We use early stopping to prevent the models trained on smaller datasets from overfitting. The Transformer-M model accuracy (Fig. 3a) and RPS (Fig. 3b) on commercial buildings roughly follows a power-law scaling with diminishing returns. Performance remains roughly constant across dataset scales for residential buildings (see Fig. 7 in App. J), which we attribute to the stronger sim-to-real distribution shift compared to commercial buildings. This suggests naively increasing the dataset size may not result in large improvements. Nevertheless, the best performance is achieved when using the full pretraining dataset of 900K buildings.
Model size: Here we compare the transformers of sizes S, M, and L. The NRMSE and RPS for real commercial buildings improves from S to M, but performance plateaus or decreases from the M to L models (Fig. 3c and Fig. 3d). We suspect this is due to distribution shifts and auto-correlation in the
Figure 3: Empirical scaling laws for zero-shot generalization on commercial buildings. Intervals are  $95\%$  stratified bootstrap CIs for the median across all buildings. a-b) Dataset scale vs. zero-shot performance. The trends appear to be power-laws with diminishing returns. c-d) Model size vs. zero-shot performance. Residential results are in App. J.
Gaussian (pretrained)
Gaussian (pretrained+FT)
Tokens (pretrained)
Tokens (pretrained+FT)
Figure 4: Model size vs. transfer learning. Pretrained vs. pretrained + fine-tuned (FT) performance for S, M, and L transformers. Intervals are  $95\%$  stratified bootstrap CIs of the median. The Transformer-M models show the most improvement after fine-tuning. The fine-tuned Transformer-M performance is comparable to the largest models. Improvement due to fine-tuning is less pronounced for the Transformer-L models, suggesting their zero-shot performance is saturated on this task.
time series that causes the largest models to overfit, despite using sliding windows to extract training samples and aggressive early stopping. The good RPS performance of the smallest Transformer (Tokens) model is likely due to training on quantized load values. Quantization helps generative models efficiently learn important structure in the data and ignore negligible information [37] (our tokenizer achieves a  $\sim 63\%$  dataset compression rate). Although, this does come at a cost of lower accuracy (Fig. 3c). While we show power-law scaling for residential building forecast accuracy (Fig. 7), the model performance may be saturated; see App. J for an in-depth discussion. We also show the relationship between model size and transfer learning performance in Fig. 4. The Transformer-M models demonstrate the largest performance gains due to fine-tuning, which confirms the high average probability of improvement scores (Sec. 5.2). The smaller impact of fine-tuning on Transformer-L performance suggests its zero-shot performance may be nearly saturated.
6.1 Findings
Pretraining on Buildings-900K leads to good zero-shot STLF performance on real commercial buildings. More investigation is needed to achieve similar results for residential buildings (Sec. 6.2).
Buildings-900K pretraining + fine-tuning on real buildings improves STLF for both commercial and residential buildings. Our best pretrained + fine-tuned baseline outperforms LightGBM and persistence in the BuildingsBench transfer learning task. Fine-tuning all model layers appears necessary, possibly to mitigate negative transfer caused by the distribution shift between simulated pretraining and real test data.
We observe an approximate power-law relationship with diminishing returns between dataset scale and zero-shot STLF for commercial buildings. Performance plateaus as model size increases, possibly due to overfitting and distribution shifts.
Pretraining on tokenized building loads instead of continuous loads achieves worse performance overall. However, the tokenized model is more stable to train, as it naturally handles a wide range of load values.
Pretraining with geospatial coordinates slightly improves generalization: Due to space constraints, the results of an ablation where the model ignores the building's latitude and longitude are provided in App. K. Briefly, improvements in accuracy were modest (0.1 - 1% NRMSE).
6.2 Residential STLF Challenges
Residential loads are inherently more uncertain and variable than commercial loads because they are more sensitive to occupant behaviour and changes in weather (e.g., temperature and humidity) [39]. This is evidenced by the relatively large persistence NRMSEs  $(77.88\% - 99.77\%)$  on BuildingsBench residential loads compared to commercial loads  $(16.68\% - 19.39\%)$ . However, rather than decreasing the value of BuildingsBench, our work enables exploring directions such as the inclusion of weather covariates, multi-variate formulations of STLF, and benchmarking of advanced approaches [20].
6.3 Limitations
While we expect BuildingsBench to stimulate research on generalizable STLF, our framework has limitations. First, pretraining on simulated data is fundamentally limited if deploying on real buildings is the goal. Mixing synthetic and real data during pretraining is an interesting direction to explore. Second, the pretraining and evaluation data is mainly representative of building energy consumption in the northwestern hemisphere. Expanding our framework to include data from other global regions is needed to comprehensively evaluate generalization. Third, the stochastic occupancy model used to simulate residential consumption behavior for the pretraining data is more predictable and less chaotic than real behavior, which increases the sim-to-real gap for residential buildings. Moreover, the occupancy model does not capture behavior patterns that may appear prominently in a specific geographic region (particularly outside of the U.S.), which may hurt the generalization capabilities when deploying the model in these locations. Finally, due to limited time, we only pretrained vanilla transformers on Buildings-900K. We encourage future work that compares these results with state-of-the-art transformers [31, 46]. We will maintain a leaderboard for BuildingsBench in our code repository, which will be updated with new model results.
In this work, we introduce BuildingsBench, which consists of a large-scale simulated dataset and a collection of real building datasets for benchmarking zero-shot STLF and transfer learning. Upon comparing the performance of pretrained transformers against persistence and traditional machine learning-based forecasting baselines, we observe promising results for commercial buildings and identify areas of improvement for residential buildings.
We plan to maintain and extend BuildingsBench in the following ways. As relevant EULP data becomes newly available, we will release updated versions of Buildings-900K. To ground improvements to benchmark tasks in a concrete downstream application, we will look into adding a reinforcement learning task to the benchmark for building control using STLF. Other promising directions for future work include exploring joint forecasting of weather and load and the impact of building metadata on performance. To facilitate this work, we plan to update the datasets with this auxiliary information. Overall, we hope that BuildingsBench will facilitate research for communities applying machine learning to the built environment, as well as those conducting foundational studies on large-scale pretraining and fine-tuning for time series.
This work was authored by the National Renewable Energy Laboratory (NREL), operated by Alliance for Sustainable Energy, LLC, for the U.S. Department of Energy (DOE) under Contract No. DE-AC36-08GO28308. This work was supported by the Laboratory Directed Research and Development (LDRD) Program at NREL. The views expressed in the article do not necessarily represent the views of the DOE or the U.S. Government. The U.S. Government retains and the publisher, by accepting the article for publication, acknowledges that the U.S. Government retains a nonexclusive, paid-up, irrevocable, worldwide license to publish or reproduce the published form of this work, or allow others to do so, for U.S. Government purposes. The research was performed using computational resources sponsored by the Department of Energy's Office of Energy Efficiency and Renewable Energy and located at the National Renewable Energy Laboratory.
The authors would like to thank Cong Feng and Alex Rybchuk for helping revise a draft version of this manuscript, as well as Dave Biagioni for providing valuable insights.
[1] Afram, A. and Janabi-Sharifi, F. Theory and applications of hvac control systems-a review of model predictive control (mpc). Building and Environment, 72:343-355, 2014.
[2] Amasyali, K. and El-Gohary, N. M. A review of data-driven building energy consumption prediction studies. Renewable and Sustainable Energy Reviews, 81:1192-1205, 2018. ISSN 13640321. doi: 10.1016/j.rser.2017.04.095. URL https://linkinghub.elsevier.com/ retrieve/pii/S1364032117306093.
[3] Amat Rodrigo, J. and Escobar Ortiz, J. skforecast, 5 2023. URL https://github.com/JoaquinAmatRodrigo/skforecast.
[4] Ardakanian, O., Huo, S., and Keshav, S. 6-second load measurement dataset, 2020. URL https://doi.org/10.5683/SP2/R4SVBF.
[5] Barker, S., Mishra, A., Irwin, D., Cecchet, E., Shenoy, P., Albrecht, J., et al. Smart*: An open data set and tools for enabling research in sustainable homes. *SustKDD*, August, 111(112):108, 2012.
[6] Box, G. E. and Cox, D. R. An analysis of transformations. Journal of the Royal Statistical Society: Series B (Methodological), 26(2):211-243, 1964.
[7] Chadoulos, S., Koutsopoulos, I., and Polyzos, G. C. One model fits all: Individualized household energy demand forecasting with a single deep learning model. In Proceedings of the Twelfth ACM International Conference on Future Energy Systems, pp. 466-474. ACM, 2021. ISBN 978-1-4503-8333-2. doi: 10.1145/3447555.3466587. URL https://dl.acm.org/doi/10.1145/3447555.3466587.
[8] Crawley, D. B., Lawrie, L. K., Winkelmann, F. C., Buhl, W. F., Huang, Y. J., Pedersen, C. O., Strand, R. K., Liesen, R. J., Fisher, D. E., Witte, M. J., et al. Energyplus: creating a new-generation building energy simulation program. Energy and buildings, 33(4):319-331, 2001.
[9] Drgona, J., Arroyo, J., Figueroa, I. C., Blum, D., Arendt, K., Kim, D., Ollé, E. P., Oravec, J., Wetter, M., Vrabie, D. L., et al. All you need to know about model predictive control for buildings. Annual Reviews in Control, 50:190-232, 2020.
[10] EIA. Energy information administration may 2023 monthly energy review. 2023. URL https://www.eia.gov/totalenergy/data/monthly/archive/00352305.pdf.
[11] Gage, P. A new algorithm for data compression. C Users Journal, 12(2):23-38, 1994.
[12] Gebru, T., Morgenstern, J., Vecchione, B., Vaughan, J. W., Wallach, H., Iii, H. D., and Crawford, K. Datasheets for datasets. Communications of the ACM, 64(12):86-92, 2021.
[13] Gneiting, T. and Katzfuss, M. Probabilistic forecasting. Annual Review of Statistics and Its Application, 1:125-151, 2014.
[47] Wu, N., Green, B., Ben, X., and O'Banion, S. Deep transformer models for time series forecasting: The influenza prevalence case. arXiv:2001.08317, 2020. URL http://arxiv.org/abs/2001.08317.
[48] Xu, X. and Meng, Z. A hybrid transfer learning model for short-term electric load forecasting. Electrical Engineering, 102(3):1371-1381, 2020. doi: 10.1007/s00202-020-00930-x. URL http://link.springer.com/10.1007/s00202-020-00930-x.
[49] Zeng, A., Chen, M., Zhang, L., and Xu, Q. Are transformers effective for time series forecasting? arXiv preprint arXiv:2205.13504, 2022.
[50] Zhang, G., Wei, C., Jing, C., and Wang, Y. Short-term electrical load forecasting based on time augmented transformer. International Journal of Computational Intelligence Systems, 15(1):67, 2022.
[51] Zhang, L., Plathottam, S., Reyna, J., Merket, N., Sayers, K., Yang, X., Reynolds, M., Parker, A., Wilson, E., Fontanini, A., et al. High-resolution hourly surrogate modeling framework for physics-based large-scale building stock modeling. *Sustainable Cities and Society*, 75:103292, 2021.
[52] Zhang, L., Wen, J., Li, Y., Chen, J., Ye, Y., Fu, Y., and Livingood, W. A review of machine learning in building load prediction. Applied Energy, 285:116452, 2021. ISSN 03062619. doi: 10.1016/j.apenergy.2021.116452. URL https://linkinghub.elsevier.com/retrieve/pii/S0306261921000209.
- Appendix A: Details of hosting, licensing, and maintenance
- Appendix B: Dataset datasheet
- Appendix C: BuildingsBenchmark benchmark datasets details
- Appendix D: Details of evaluation metrics
- Appendix E: Time series transformer details
- Appendix F: Training details
- Appendix G: Compute details
- Appendix H: Benchmark performance profiles
- Appendix I: Per-dataset results
- Appendix J: Additional empirical scaling law results
- Appendix K: Ablation studies
- Appendix L: Author responsibility statement
- Appendix M: Additional qualitative results
A Hosting, Licensing, and Maintenance
Our datasets and code are available via the following links:
- Github: https://github.com/NREL/BuildingsBench
- Documentation: https://nrel.github.io/BuildingsBench
- Datasets and Website: https://data.openei.org/submissions/5859
- Tutorials: https://nrel.github.io/BuildingsBench/tutorials/
- Dataset DOI: https://doi.org/10.25984/1986147
As described in Sec. 3 and Sec. 4, Buildings-900K and the BuildingsBench benchmark datasets are available for download under a CC-4.0 license and our code is available under a BSD 3-Clause license. We ensure the long-term availability and maintenance of the data by hosting it on the Open Energy Data Initiative (OEDI) platform (https://data.openei.org/about). The OEDI is a centralized repository for storing energy-related datasets derived from U.S. Department of Energy projects.
B Datasheet for Buildings-900K
Questions from Datasheet for Datasets (v8) [12].
B.1 Motivation
Q: For what purpose was the dataset created?
This dataset was created to research large-scale pretraining of models for short-term load forecasting (STLF). It specifically addresses a lack of appropriately sized and diverse datasets for pretraining STLF models. Buildings-900K, which consists of simulated building energy consumption time series, is derived from the National Renewable Energy Lab (NREL) End-Use Load Profiles (EULP) database. We emphasize that the EULP was not originally developed for studying STLF. Rather, it was developed as a general resource to "...help electric utilities, grid operators, manufacturers, government entities, and research organizations make critical decisions about prioritizing research and development, utility resource and distribution system planning, and state and local energy planning and regulation." [45].
Q: Who created the dataset (e.g., which team, research group) and on behalf of which entity (e.g., company, institution, organization)?
Researchers at NREL created Buildings-900K. Consent from the NREL developers of the EULP project was obtained in writing to extract, process, and redistribute a subset of the EULP. The EULP is a multiyear multi-institutional collaboration lead by NREL and is publicly available for download with a permissive CC-4.0 license.
Q: Who funded the creation of the dataset?
This dataset was developed with funding provided by the Laboratory Directed Research and Development program at NREL.
B.2 Composition
Q: What do the instances that comprise the dataset represent (e.g., documents, photos, people, countries)?
Annual hourly energy consumption from EnergyPlus [8] simulations. EnergyPlus is a high fidelity building energy consumption simulator which has been continuously maintained and updated for over 20 years by the U.S. Department of Energy.
Q: How many instances are there in total (of each type, if appropriate)?
There are  $\sim 1.8$  million annual time series instances (900K buildings  $\times 2$  years).
Q: Does the dataset contain all possible instances or is it a sample (not necessarily random) of instances from a larger set?
Buildings-900K consists of a subset of the EULP database. Specifically, we extract only the out.site_energy.total.energy Consumption time series from each upgrade  $= 0$  building in the 2021 version for weather-years AMY2018 and TMY3. The EULP contains additional end-use time series for each building, the EnergyPlus XML model files, the EnergyPlus weather files, and a variety of additional simulations under various electrification scenarios. We did not include these additional data in the initial version of Buildings-900K, although we may consider adding them in the future.
Q: What data does each instance consist of?
Each instance is stored in a Parquet file and has timestamp information and load values (in kWh). We also have spatial metadata that maps each building to a U.S. county.
Q: Is any information missing from individual instances?
Q: Are relationships between individual instances made explicit (e.g., users' movie ratings, social network links)?
Each building has a unique identifier and is assigned to a county. That county (or Public Use Microdata Area—PUMA) has its own unique identifier. Each PUMA contains many buildings that are geographically close to each other.
Q: Are there recommended data splits (e.g., training, development/validation, testing)?
- For test: We withhold all buildings in 4 counties from both AMY2018 and TMY3 weather-years.
- For validation: We withhold dates 2018-12-17 to 2018-12-31 from all buildings in the training set to use as held-out validation data (e.g., for early stopping).
- For pretraining: All remaining data is used for pretraining.
Q: Are there any errors, sources of noise, or redundancies in the dataset?
Q: Is the dataset self-contained, or does it link to or otherwise rely on external resources (e.g., websites, tweets, other datasets)?
It is self-contained.
Q: Does the dataset contain data that might be considered confidential (e.g., data that is protected by legal privilege or by doctor-patient confidentiality, data that includes the content of individuals' non-public communications)?
No. All simulated building models are created by sampling from conditional distributions over attributes calibrated to real data.
Q: Does the dataset contain data that, if viewed directly, might be offensive, insulting, threatening, or might otherwise cause anxiety?
Q: Does the dataset relate to people?
B.3 Collection Process
Q: How was the data associated with each instance acquired?
Buildings-900K time series instances are the outputs of extensively calibrated and validated EnergyPlus simulations. These simulations were created and run by the NREL EULP team on high performance computing resources. For completeness, we summarize here the steps taken by the EULP team to create the synthetic building models for these simulations, which are also described in the EULP documentation in more detail [45]. The synthetic buildings used to form large-scale U.S. residential and commercial building stock models were obtained from the ResStock<sup>1</sup> and ComStock<sup>2</sup> analysis tools. An extensive calibration and validation effort based on utility meter data from millions of customers, end-use submetering data, and additional private data sources helped to ensure the accuracy of the synthetic ResStock and ComStock models used in the EULP. A new stochastic occupant behavior model for the residential building stock was developed to help calibrate the simulations. Uncertainty quantification based on a learned surrogate [51] was also used for model calibration, in particular, to characterize sensitivity of simulation outputs (e.g., quantities of interest, or QOIs) with respect to different model parameters.
Q: What mechanisms or procedures were used to collect the data (e.g., hardware apparatus or sensor, manual human curation, software program, software API)?
Q: If the dataset is a sample from a larger set, what was the sampling strategy (e.g., deterministic, probabilistic with specific sampling probabilities)?
Buildings-900K consists of a hand-selected subset of the EULP database.
Q: Who was involved in the data collection process (e.g., students, crowdworkers, contractors) and how were they compensated (e.g., how much were crowdworkers paid)?
Q: Over what timeframe was the data collected?
A portion of Buildings-900K consists of building load time series simulated with weather from 2018.
Q: Were any ethical review processes conducted (e.g., by an institutional review board)?
B.4 Preprocessing/cleaning/labeling
Q: Was any preprocessing/cleaning/labeling of the data done (e.g., discretization or bucketing, tokenization, part-of-speech tagging, SIFT feature extraction, removal of instances, processing of missing values)?
The steps we followed to create Buildings-900K are as follows:
<https://resstock.nrel.gov
$^{2}$ https://comstock.nrel.gov
1. Select the out.site_energy.total.energy Consumption time series from each AMY2018 and TMY3 building.
2. Aggregate the 15-minute resolution energy consumption to hourly by summing the values at intervals of four consecutive timestamps XX:15, XX:30, XX:45, and XX+1:00 (e.g., 12:15, 12:30, 12:45, 13:00). Note that we use a sum aggregation here. When aggregating meter readings (in kWh) for real buildings (e.g., the BuildingsBench evaluation data), we take the average of the sub-hourly values.
3. Join the time series for all buildings in the same PUMA into a single Parquet table.
4. Save each PUMA-level Parquet table (compressed with Snappy).