SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Patrick Emami, Abhijeet Sahu, Peter Graf
National Renewable Energy Lab
{Patrick.Emami, Abhijeet.Sahu, Peter.Graf}@nrel.gov
Short-term forecasting of residential and commercial building energy consumption is widely used in power systems and continues to grow in importance. Data-driven short-term load forecasting (STLF), although promising, has suffered from a lack of open, large-scale datasets with high building diversity. This has hindered exploring the pretrain-then-fine-tune paradigm for STLF. To help address this, we present BuildingsBench, which consists of: 1) Buildings-900K, a large-scale dataset of 900K simulated buildings representing the U.S. building stock; and 2) an evaluation platform with over 1,900 real residential and commercial buildings from 7 open datasets. BuildingsBench benchmarks two under-explored tasks: zero-shot STLF, where a pretrained model is evaluated on unseen buildings without fine-tuning, and transfer learning, where a pretrained model is fine-tuned on a target building. The main finding of our benchmark analysis is that synthetically pretrained models generalize surprisingly well to real commercial buildings. An exploration of the effect of increasing dataset size and diversity on zero-shot commercial building performance reveals a power-law with diminishing returns. We also show that fine-tuning pretrained models on real commercial and residential buildings improves performance for a majority of target buildings. We hope that BuildingsBench encourages and facilitates future research on generalizable STLF. All datasets and code can be accessed from https://github.com/NREL/BuildingsBench.
Residential and commercial buildings in the United States are responsible for about  $40\%$  of energy consumption and  $35\%$  of greenhouse gas emissions [10]. Globally, these estimates are respectively  $30\%$  and  $27\%$  [22]. Building energy demand forecasting plays a part in reducing global emissions by helping to decarbonize the building sector.
Short-term load forecasting (STLF), which typically ranges from hour-ahead to a few days ahead, plays a multitude of critical roles. STLF can help match shifting energy supplies with customer demand, as well as aid energy markets with accurately setting prices based on forecasted supply/demand [19]. Accurate forecasts can be directly used by reinforcement learning [44] and model predictive control [1, 9] for optimal building energy management.
However, STLF remains a challenging problem, as energy demand can fluctuate heavily due to a variety of unobserved and exogenous factors. As such, data-driven methods have risen in popularity to address STLF [2, 52]. Interest in these techniques has also been spurred by a rise in deployments of advanced sensors (i.e., smart meters) that record building energy consumption. However, the number of instrumented buildings with publicly released data remains low. Moreover, it is typical for multiple years of historical data to be reserved for training and validation, with trained models tested
Figure 1: BuildingsBench gallery. Top row: commercial buildings (farthest left is simulated commercial Buildings-900K data). Second and third rows are residential buildings (farthest left in the second row is simulated residential Buildings-900K data).
on a single held-out year for the same building [28]. This incurs a lengthy data collection period per building that is not scalable. There is thus a lack of sufficiently large publicly available datasets (Table 1), which has hindered the investigation of large-scale pretraining (i.e., one model trained on every building [7, 17]).
In this work, we introduce BuildingsBench, an open-source platform for large-scale pretraining and for benchmarking zero-shot STLF [7, 17] and transfer learning for STLF [48, 14]. In zero-shot STLF, models forecast loads for unseen target buildings without any fine-tuning. This drastically reduces time-to-deployment on newly instrumented buildings. In transfer learning, a pretrained model is fine-tuned on a target building, assuming a limited yet realistic amount of data (e.g., 6 months).
As part of BuildingsBench, we introduce Buildings-900K, a dataset of nearly one million simulated time series, which approaches the scale of datasets in natural language processing and computer vision. This data is sourced from the NREL End-Use Load Profiles (EULP) database [45]. The EULP is a multiyear, multi-institution effort to create a statistically representative database of the entire U.S. building stock's energy consumption via careful calibration and validation of physics-based building simulations. We also provide an evaluation suite that combines 7 open datasets totalling over 1,900 real buildings (Table 2). Example time series from the simulated and real datasets are shown in Figure 1. Compared to other existing datasets (Table 1, Table 2), our large-scale pretraining and evaluation data contains both simulated and real building energy consumption time series spanning a wider range of geographic locations, years, and types of both residential and commercial buildings.
Our platform automates the evaluation of a variety of simple and advanced baselines on the two tasks. Our results on zero-shot STLF reveal that synthetically pretrained models achieve strong performance on real commercial buildings. We observe a smaller distribution shift between simulated
and real commercial buildings than residential buildings. We also show that pretrained models can further improve performance by fine-tuning on real commercial and residential building data. Buildings-900K also enables studying large-scale pretraining of transformers on geographical time series. The utility of transformers for forecasting has recently been questioned [49], but a lack of sufficiently large public time series datasets has made investigating this challenging. Our main finding is a power-law scaling with diminishing returns between dataset scale (size and diversity) and generalization (for commercial buildings). We expect that BuildingsBench will facilitate research on new modeling techniques and large-scale datasets for generalizable STLF.
To summarize, we contribute: 1) Buildings-900K, a simulated dataset for large-scale pretraining, 2) a platform for benchmarking zero-shot STLF and transfer learning, and 3) valuable insights on model pretraining for STLF.
2 Short-Term Load Forecasting
The BuildingsBenchmark considers the following univariate forecasting problem. Given  $H$  past load values  $x_{t-H:t}$  and  $H + T$  covariates  $z_{t-H:t+T}$  (defined in Section 3.1), we aim to predict the conditional distribution for  $T$  unobserved future load values  $y_{t+1:t+T}$ :
$$
p \left(y _ {t + 1}, \dots , y _ {t + T} \mid x _ {t - H}, \dots , x _ {t}, z _ {t - H}, \dots , z _ {t + T}\right). \tag {1}
$$
We consider a day-ahead STLF scenario with  $H = 168$  hours (one week) and  $T = 24$  hours. A primary goal of BuildingsBench is to study and evaluate STLF models, which learn a single set of parameters  $\theta$  shared by all buildings for the distribution in Eq. 1. We use a probabilistic formulation for our benchmark, as applications of STLF increasingly require uncertainty estimates, such as planning and scheduling of renewable energy sources for buildings [19].
3 The Buildings-900K Dataset
In this section, we introduce our dataset for pretraining STLF models.
Simulated data source: Our dataset is sourced from the NREL EULP database [45]. The EULP provides 15-minute resolution appliance-level consumption simulated with EnergyPlus [8, 29] for a base set of 900K building models (550K residential and 350K commercial) spread across all climate regions in the U.S. It aims to provide a statistically representative picture of the entire U.S. buildings stock at various aggregation levels (county, state, etc.) and under various electrification scenarios. The building simulations were extensively calibrated over a three-year
period with advanced metering data ( $\sim$ 2.3 million meters), data from 11 utilities, as well as other public/private datasets related to energy usage. Socio-economic building characteristics, including location based on Public Use Microdata Area (PUMA,  $\sim$ 2400 PUMAs in the U.S.—see inset), are sampled from distributions generated from U.S. Census survey responses. Please see Wilson et al. [45] for the complete description or App. B.3 for more details.
Processing and storage: To create Buildings-900K, we extract 900K total energy consumption time series (in kilowatt-hours (kWh)) from each of the non-upgraded buildings in the 2021 version of the EULP. To promote accessibility of our dataset, we also aggregate the 15-minute resolution to hourly to reduce the size. This data requires about 110 GB to store, significantly less than the entire EULP  $(70 + \mathrm{TB})$ . We store all buildings within each PUMA in a single Parquet file by year (there are two years, 2018 and an aggregated "typical meteorological year" (TMY) [45]) and by building type (residential/commercial), which amounts to 9,600 Parquet files. Each file has a column for the timestamp and a column for each building's energy consumption (8,760 rows per file). Processing the EULP to extract this data took  $\sim 3$  days with Apache PySpark on a 96-core AWS cloud instance.
Pretraining splits and loading: A validation set is created by withholding the final two weeks of 2018. The test set consists of buildings in four PUMAs that are withheld from both training and validation. All splits use a 24-hour sliding window to extract 192-hour load sub-sequences. Because shuffling large datasets of sub-sequences is computationally demanding, our platform provides a
Table 1: Comparing popular building energy consumption datasets to our dataset Buildings-900K. Our dataset has significantly more buildings (residential and commercial) located across a wide geographic area and spans multiple years, which enables studying large-scale pretraining for STLF.
# buildings	Open access	Residential	Commercial	Total hours	# Sites
Pecan Street [40]	1,000	×	✓	×	?	?
Electricity [41]	370	✓	×	✓	~12.9M	1
Building D.G.P. 2 [27]	1,636	✓	×	✓	~53.6M	19
Low Carbon London [30]	5,567	✓	✓	×	~97.5M	1
Buildings-900K (simulated)	900,000	✓	✓	✓	~15B	2400
Table 2: BuildingsBench STLF evaluation data.
Residential	Commercial	Years spanned	# Sites	Total days
Buildings-900K-Test (simulated)	915	565	TMY, 2018	4	1.1M
BuildingsBench (real)	953	970	2007–2018	10	1.2M
custom PyTorch [33] Dataset that creates an index file to map a shuffled list of integers to a subsequence. Each line of the index file is accessible in  $O(1)$  time with Python's `seek` function. Apache PyArrow is used to efficiently load the indexed building's time series into memory.
Hosting and licensing: Buildings-900K is hosted by the Open Energy Data Initiative and is available for download with a permissive CC-4.0 license (link available in App. A).
3.1 Feature Extraction
Beyond the load time series, we also extract these covariates on-the-fly to condition forecasts on:
- Calendar features—day of the week, day of the year, and hour of the day—are generated from the timestamp, which we then cyclically encode with sine and cosine.
- The latitude and longitude coordinate of the centroid of the building's PUMA (using metadata provided by the EULP) for encoding geospatial information.
- A binary feature for building type (residential (0) or commercial (1)).
4 BuildingsBench Evaluation Platform
BuildingsBench provides an open-source software platform for evaluating models on zero-shot and transfer learning for STLF using a collection of real building datasets. To avoid confusion, in our analysis (Sec. 5), we will use the name BuildingsBench to refer only to the real building evaluation data and explicitly state whether the results are for Buildings-900K-Test (simulated) or for BuildingsBench (real) (Table 2). We now describe each task in more detail.
Zero-shot STLF: For this task, pretrained models are asked to provide day-ahead forecasts for unseen simulated and real buildings. Our use of "zero-shot" here refers to a setting where only one week of historical data is presumed available, and is thus insufficient for training or fine-tuning a deep neural network. All available years are used for each test building.
Transfer learning: Our transfer learning task assumes 6 months of metered data has been collected for each target building, about  $\sim 180$  training samples. Fine-tuned models are then tasked with day-ahead forecasting for the next 6 months using a 24-hour sliding window.
4.1 Real Building Datasets
Here, we briefly describe the real building data used for evaluating the two tasks and defer additional details to App. C. Altogether, our real building benchmark has over 1,900 buildings and 1.2M days of energy usage.
Electricity [41]: 370 commercial buildings in Portugal with data spanning 2011-2014.
Building Data Genome (BDG) Project 2 [27]: 1,636 commercial buildings across 19 sites in 2016 and 2017. We include buildings from 4 U.S. sites (Panther, Fox, Bear, Rat).
Low Carbon London [30]: Energy consumption meter readings from 5,567 London, UK households between 2011-2014. To keep the overall number of residential and commercial buildings roughly the same, we keep a random sample of 713 buildings.
SMART [5]: Meter readings for 7 homes in western Massachusetts, U.S., between 2014-2016.
IDEAL [34]: Electricity meter data from 255 homes in Edinburgh, UK, between 2016-2018.
Individual household electric power consumption [15]: Energy consumption from a single household in Sceaux, Paris, between 2007-2010.
Borealis [4]: 6-second load measurements recorded for 30 homes in Waterloo, ON, in 2011-2012.
Processing and storage: We resample each time series to hourly if the data is sub-hourly. Smart meter time series typically have missing values, sometimes extending over weeks or months. Buildings with more than  $10\%$  of the data missing were not included. For included buildings, we linearly interpolate missing values, and if values are missing for a span greater than 1 week, we fill this with zeros. We also provide an option to exclude buildings with a max hourly consumption  $>5.1$  MW (only 15 from electricity) to keep the range of consumption values similar between Buildings-900K and BuildingsBench test data, as extrapolation is not our focus. A small fraction of spikes in the time series due to noisy meter readings are classified as outliers and filtered with a non-parametric distance-based sliding window algorithm [25] (see App. C.8). Each annual consumption time series per building is stored as a CSV file.
Hosting and licensing: We host the processed versions of each dataset, along with the original permissive licenses, alongside Buildings-900K under the same CC-4.0 license. Our code is open-sourced under a BSD-3 license.
We primarily evaluate task performance with two metrics, the normalized root mean square error (NRMSE) and the ranked probability score (RPS). The NRMSE (also known as the coefficient of variation of the RMSE) is widely used, as it captures the ability to predict the correct load shape. For a target building with  $M$  days of load time series,
$$
N R M S E := 1 0 0 \times \frac {1}{\bar {y}} \sqrt {\frac {1}{2 4 M} \sum_ {j = 1 , i = 1} ^ {M , 2 4} \left(y _ {i , j} - \hat {y} _ {i , j}\right) ^ {2}}, \tag {2}
$$
where  $\hat{y}$  is the predicted load,  $y$  is the actual load, and  $\bar{y}$  is the average actual load over all  $M$  days. In the appendix, we also report the normalized mean absolute error and normalized mean bias error (see descriptions in App. D).
The RPS is a well-known metric for uncertainty quantification in probabilistic forecasting [13]. It compares the squared error between two cumulative distribution functions (CDFs), the predicted CDF and the observation represented as a CDF. Define the indicator function  $\mathbf{1}_{x\leq y}$  as 1 if the actual load  $x$  is  $\leq y$  and 0 otherwise. The continuous RPS for a predicted CDF  $\hat{F}$  for the load at hour  $i$  is:
$$
R P S := \int_ {0} ^ {\infty} \left(\hat {F} _ {i} (y) - \mathbf {1} _ {y _ {i} \leq y}\right) ^ {2} d y. \tag {3}
$$
Our platform implements the closed-form RPS for Gaussian CDFs, as well as a discrete RPS for categorical distributions (used by baselines that discretize the positive real line for token-based load forecasting, see Sec. 4.3). These are formally defined in App. D.
4.3Baselines
Various baselines are implemented and benchmarked in the BuildingsBench platform. For zero-shot STLF, we pretrain a representative time series transformer [47] on Buildings-900K. Transformers have recently gained significant interest for STLF [17, 21, 36, 50]. The model is described in brief here and with more detail in App. E.
Transformer (Gaussian): This model is the original encoder-decoder transformer [43] repurposed for autoregressive time series forecasting, as proposed in Wu et al. [47]. It predicts a Gaussian distribution at each time step  $t + i$  conditioned on the past week and the previous  $i - 1$  predictions. We train it to minimize the Gaussian negative log-likelihood loss. Because our demand time series are highly non-Gaussian (periods of low consumption interspersed with bursts), the loss diverges during training when we use global standard scaling to normalize the data. The Box-Cox power transformation [6] removes this instability. To compute the RPS, we approximate a Gaussian in the unscaled space by backprojecting the scaled standard deviation (see App. D for details).
Transformer (Tokens): We also implement a transformer variant that uses tokenization to predict discrete load tokens. This baseline allows us to test whether quantizing loads into tokens is beneficial for large-scale pretraining. We also found that a comparison between tokenization and Gaussian time series transformers was missing from the literature. The model is trained to predict a categorical distribution over a vocabulary of load tokens by minimizing a multi-class cross-entropy loss. To quantize loads, we use a simple strategy—faisss-gpu [23] KMeans clustering with  $K = 8,192$  fit to the Buildings-900K training set. We merge clusters  $< 10$  Watts apart to obtain a compact vocabulary size of 3,747 tokens. See App. E for analysis on our tokenizer design.
Each transformer is pretrained jointly on residential and commercial buildings at three different sizes: Transformer-S (3M params), Transformer-M (17M params), and Transformer-L (160M params). Models are trained on 1 billion load hours with early stopping. See App. E for more architecture details, App. F for hyperparameter tuning details (for learning rate and batch size), and App. G for training compute requirements.
We also benchmark persistence forecasting on both the zero-shot STLF and transfer learning tasks:
Previous Day: For scenarios where the load conditions change relatively slowly, the previous day's load is a strong baseline for day-ahead STLF [42].
Previous Week: The 24-hour load profile on the same day from the previous week is used [17].
**Ensemble:** This persistence baseline computes a Gaussian distribution for each predicted hour  $t + i$  whose mean is the average load at hour  $t + i$  over the past 7 days:
$$
\hat {\mu} = \frac {1}{7} \sum_ {j = 1} ^ {7} x _ {(t + i - 2 4 j)}; \quad p \left(y _ {t + i} \mid x _ {t - H: t}\right) := \mathcal {N} \left(\hat {\mu}, \sqrt {\frac {1}{7} \sum_ {j = 1} ^ {7} \left(x _ {(t + i - 2 4 j)} - \hat {\mu}\right) ^ {2}}\right). \tag {4}
$$
In STLF, outperforming persistence is an indicator that a model is producing meaningful forecasts. On the transfer learning task, we also benchmark the following forecasting baselines by training them directly on the target building's 6 months of meter data with supervised learning:
LightGBM: The light gradient-boosting machine (LightGBM) [24] is a popular decision-tree-based algorithm suitable for STLF [32]. We use the multi-step forecasting implementation from skforecast [3] with 100 estimators and no max depth.
Linear regression, DLinear: Inspired by Zeng et al. [49], we benchmark a linear direct multi-step forecaster that regresses all 24 future values as a weighted sum of the past 168 values. We also implement DLinear, which decomposes the time series with a moving average kernel, applies a linear layer to each component, and then sums the two to get the final prediction.
RNN (Gaussian): This is an autoregressive encoder-decoder recurrent neural network inspired by Salinas et al. [38]. A multi-layer LSTM [18] first encodes the 168 past load values. The last encoder hidden state initializes the state of a multi-layer LSTM decoder. A linear layer maps each output of the decoder to Gaussian parameters.
5 Benchmark Results
Here, we analyze our baselines on the two benchmark tasks guided by the following questions:
1. Can models pretrained on Buildings-900K generalize to real buildings?
2. Does fine-tuning a pretrained model on limited data from a target building lead to improved performance?
Table 3: Zero-shot STLF results. Median accuracy (NRMSE) and ranked probability score (RPS) with best and second best highlighted. Lower is better. No fine-tuning is performed on any test building. Residential NRMSEs are naturally larger than commercial buildings, because the normalization factor—the building's average consumption per hour—is small. For example, Transformer-L (Gaussian) has an RMSE of  $0.72\mathrm{kWh}$  and an NRMSE of  $66.57\%$  on the residential Sceaux dataset.
Buildings-900K-Test (simulated)	BuildingsBench (real)
Commercial	Residential	Commercial	Residential
NRMSE (%)	RPS	NRMSE (%)	RPS	NRMSE (%)	RPS	NRMSE (%)	RPS
Persistence Ensemble	33.10	4.58	54.77	0.72	16.68	5.88	77.88	0.063
Previous Day Persistence	34.91	-	59.38	-	16.96	-	98.41	-
Previous Week Persistence	32.05	-	74.08	-	19.39	-	99.77	-
Transformer-L (Tokens)	12.91	2.45	44.29	0.875	14.46	5.62	95.34	0.152
Transformer-L (Gaussian)	15.10	2.07	43.52	0.578	13.31	5.23	79.34	0.072
Table 4: Transfer learning results. We show median accuracy (NRMSE) and ranked probability score (RPS) with best and second best highlighted. Transformer-S and Transformer-M results are in Fig. 4. Improvement due to fine-tuning.
Commercial buildings	Residential buildings
NRMSE (%)	RPS	NRMSE (%)	RPS
Not pretrained + Not fine-tuned
Persistence Ensemble	16.80	5.97	78.54	0.057
Previous Day Persistence	16.54	-	98.35	-
Previous Week Persistence	18.93	-	100.20	-
Not pretrained + Fine-tuned
Linear regression	25.18	-	89.98	-
DLinear	23.41	-	87.89	-
RNN (Gaussian)	41.79	15.28	96.75	0.078
LightGBM	16.02	-	80.07	-
Transformer-L (Tokens)	50.12	26.89	105.65	16.36
Transformer-L (Gaussian)	37.21	15.94	92.99	0.081
Pretrained + Not fine-tuned
Transformer-L (Tokens)	14.20	5.11	94.11	0.140
Transformer-L (Gaussian)	13.03	4.47	79.43	0.062
Pretrained + Fine-tuned
Transformer-L (Tokens)	14.07 (-0.13)	4.99 (-0.12)	94.53 (+0.42)	0.137 (-0.003)
Transformer-L (Gaussian)	12.96 (-0.07)	4.37 (-0.10)	77.20 (-2.23)	0.057 (-0.015)
3. How does the number of pretraining buildings affect zero-shot generalization?
4. How does the size of the pretrained model affect generalization?
5.1 Zero-Shot STLF
For zero-shot STLF, models are evaluated on day-ahead forecasting on all unseen simulated and real buildings without fine-tuning. Results for Transformer-L and other baselines are shown in Table 3 aggregated by the median over all buildings. Due to space constraints, Transformer-S and M results are shown in Figs. 3c-3d and App. Figs. 7-8. Performance profiles over all buildings (for comparing models by examining the tails of the forecast error distribution) and per-dataset metrics with  $95\%$  stratified bootstrap CIs are reported in App. H and I.
On unseen simulated buildings, the pretrained transformers outperform the persistence baselines in accuracy and uncertainty quantification. The average zero-shot STLF accuracy slightly improves from simulated to real commercial buildings  $(-0.1\%)$  NRMSE) and drops for residential buildings  $(+43\%)$  NRMSE). RPS is sensitive to the load magnitude, which makes it difficult to compute a sim-to-real gap for uncertainty quantification. See Fig. 2 for qualitative visualizations of the forecast