BuildingsBench: A Large-Scale Dataset of 900K Buildings and Benchmark for Short-Term Load Forecasting
Authors: Patrick Emami
Abhijeet Sahu
Peter Graf
Venue: NeurIPS 2023 Datasets and Benchmarks Poster (2023)
Keywords: Short-term load forecasting
climate change
time series forecasting
transformers
pretraining
TL;DR: 1) Buildings-900K, a synthetic large-scale training dataset for short-term load forecasting (STLF), 2) a benchmark for zero-shot and transfer learning on real buildings, and 3) insights on model pretraining for STLF.

Abstract: Short-term forecasting of residential and commercial building energy consumption is widely used in power systems and continues to grow in importance. Data-driven short-term load forecasting (STLF), although promising, has suffered from a lack of open, large-scale datasets with high building diversity. This has hindered exploring the pretrain-then-fine-tune paradigm for STLF. To help address this, we present BuildingsBench, which consists of: 1) Buildings-900K, a large-scale dataset of 900K simulated buildings representing the U.S. building stock; and 2) an evaluation platform with over 1,900 real residential and commercial buildings from 7 open datasets. BuildingsBench benchmarks two under-explored tasks: zero-shot STLF, where a pretrained model is evaluated on unseen buildings without fine-tuning, and transfer learning, where a pretrained model is fine-tuned on a target building. The main finding of our benchmark analysis is that synthetically pretrained models generalize surprisingly well to real commercial buildings. An exploration of the effect of increasing dataset size and diversity on zero-shot commercial building performance reveals a power-law with diminishing returns. We also show that fine-tuning pretrained models on real commercial and residential buildings improves performance for a majority of target buildings. We hope that BuildingsBench encourages and facilitates future research on generalizable STLF. All datasets and code can be accessed from https://github.com/NREL/BuildingsBench.

BuildingsBench: A Large-Scale Dataset of 900K Buildings and Benchmark for Short-Term Load Forecasting
Patrick Emami, Abhijeet Sahu, Peter Graf' National Renewable Energy Lab (Patrick. Emami,, Abhijeet.Saie Peter Graf)
Abstract
Short-term forecasting of residential and commercial building energy consumption is widely used in power systems and continues to grow importance. Data-driven short-term load forecasting (STLF), although promising, has suffered from a lack of open, large-scale datasets with high building diversity. This has hindered exploring the pretrain-then-fire fineefiittueeee paradigm for STLF. To help address this, we present BuildingsBench, which consists of: Buildings- -oatiingg--KKK,, large- scage-scalesoo dataset of 900K simulated buildings representing the U.S. building stock; and 2) an evaluation platform with over 1,900 real residential and commercial buildings from 7 open datasets. BuildingsBench benchmarks two under-explored tasks: zero-shot STLF, where pretrained model is evaluated on unseen buildings without fine-tuning, and transfer learning, where a pretrained model is fine-tuned on target building. The main finding of our benchmark analysis is that synthetically pretrained models generalize surprisingly well to real commercial buildings. An exploration of the effect of increasing dataset size and diversity on zero-shot commercial building performance reveals power-law with diminishing returns. We also show that fine- tuning pretrained models on real commercial and residential buildings improves performance for majority of target buildings. We hope that BuildingsBench encourages and facilitates future research on generalizable STLF, All datasets and code can be accessed from https:://// /ttgttttttuuu wow/NREL//Biul http:/BuuldinggBnrre
1 Introduction
Residential and commercial buildings the United States are responsible for about 40% of energy consumption and 35% of greenhouse gas emissions [10]. Globally, these estimates are respectively 30% and 27% [22]. Building energy demand forecasting plays a part reducing global emissions by helping to decarbonize the building sector.
load forecasting (STLF), which typically ranges from hour-ahead few days ahead, plays a multitude of critical roles. STLF can help match shifting energy supplies with customer demand, as well as aid energy markets with accurately setting prices based on forecasted sup- ply/demand [19]. Accurate forecasts can be directly used by reinforcement learning 44 and model predictive control
Short-term
9 for optimal building energy management.
However, STLF remains a challenging problem, as energy demand can fluctuate heavily due to a variety of unobserved and exogenous factors. As such, data-driven methods have risen in popularity to address STLF [2] (5). Interest in these techniques has also been spurred by a rise in deployments of advanced sensors (i.e., smart meters) that record building energy consumption. However, the number of instrumented buildings with publicly released data remains low. Moreover, it is typical for multiple years of historical data be reserved for training and validation, with trained models tested
37th Conference on Neural Information Processing Systems (NeurIPS 2023) Track on Datasets and Benchmarks.

---

Commercial Buildings 900K
Commercial Electrictt
Commercial BDG-2
80
900
240
800
230
load (XN
load (ANN)
700
220
600
210
500
200
300
190
200
100
timestamp
timestamp Residential SMART
timestamp
Residential Buildings-
1oad (k0h))
load (WWh)
Iced lced (KWh)
1.0
0.8
0.2
0.0
2917
Eimestamp
timestamp
Residential Sceaux
2.00
2.2
1.75
103d (00h)
1.0
lad (WWh))
1.50
load (KWP))
2.5
0.8
1.25
2.0
0.6
1.00
0.75
0.50
0.5
0.0
2003
2088
timestamp
6imestamp
Figure 1: BuildingsBench gallery. Top row: commercial buildings (farthest left is simulated commercial maildings-900K data). Second and third rows are residential buildings (farthest left in the second row is simulated residential Buildings-900 data).
on a single held-out year for the same building 28 This incurs a lengthy data collection period per building that not scalable. There is thus lack of sufficiently large publicly available datasets (Table T. which has hindered the investigation of large-scale pretraining (i.e., one model trained on every building [7
In this work, we introduce BuildingsBench, an open-source platform for large-scale pretraining and for benchmarking zero-shot STLF [7 and transfer learning for STLF 48 [4]. In zero-shot STLF, models forecast loads for unseen target buildings without any fine-tuning. This drastically reduces time-to-deploymen
newly instrumented buildings. In transfer learning, pretrained model is
fine-tuned on a target building, assuming a limited yet realistic amount data (e.g., 6 months).
As part of BuildingsBench, we introduce Buildings-900K, dataset of nearly one million simulated time series, which approaches the scale datasets in natural language processing and computer vision. This data is sourced from the NREL End-Use Load Profiles (EULP) database [45]. The EULP a multiycar, multi-instinn instiiuiinn effort to create statistically representative database of the entire U.S. building stock's 's energy consumption via careful calibration and validation of physics-based building simulations. We also provide an evaluation suite that combines open datasets totalling over 1,900 real buildings (Table Example time series from the simulated and real datasets are shown in Figure I Compared to other existing datasets (Table I Table 2 our large-scale pretraining and evaluation data contains both simulated and real building energy consumption time series spanning wider range of geographic locations, years, and types of both residential and commercial buildings.
Our platform automates the evaluation of a variety of simple and advanced baselines on the two tasks. Our results on zero-shot STLF reveal that synthetically pretrained models achieve strong performance real commercial buildings. We observe smaller distribution shift between simulated

---

and real commercial buildings than residential buildings. We also show that pretrained models can further improve performance by fine-tuning on real commercial and residential building data. Buildings-900k also enables studying large-scale pretraining of transformers on geographical time series. The utility of transformers for forecasting has recently been questioned [49], but a lack of sufficiently large public time series datasets has made investigating this challenging. Our main finding is a power-law scaling with diminishing returns between dataset scale (size and diversity) and generalization (for commercial buildings). We expect that BuildingsBench will facilitate research on new modeling techniques and large-scale datasets for generalizable STLF.
To summarize, we contribute: 1) Buildings-900K, a simulated dataset for large-scale pretraining, 2) platform for benchmarking zero-shot STLF and transfer learning. and 3) valuable insights on model pretraining for STLF.
2 Short-Term Load Forecasting
The BuildingsBench benchmark considers the following univariate forecasting problem. Given H past load values - and H + T covariates Zt- --+++ (defined in Section 3.1 we aim to predict the conditional distribution for T unobserved future load values 1++1:t+
p(u++1.... U++T Ft- HI
2t- H Z++T).
(1)
We consider day-ahead STLF scenario with H 168 hours (one week) and T 24 hours. primary goal of BuildingsBench is to study and evaluate STLF models, which learn a single set of parameters 0 shared by all buildings for the distribution Eq. [. We use a probabilistic formulation for our benchmark, as applications of STLF increasingly require uncertainty estimates, such as planning and scheduling of renewable energy sources for buildings 19
The Buildings-9009 Dataset
In this section, we introduce our dataset for pretraining STLF models.
Simulated data source: Our dataset is sourced from the NREL EULP database [5]. The EULP provides 15- minute resolution appliance-level consumption simulated with EnergyPlus [8 [29 for a base set of 900K building models (550K residential and 350K commercial) spread across all climate regions in the U.S. It aims to provide a statistically representative picture of the entire U.S. build- ings stock at various aggregation levels (county, state, etc.) and under various electrification scenarios. The building simulations were extensively calibrated over three- year
Buildings per PUMA
G-285
285 2344
334 350
288-40
period with advanced metering data (~ 2.3 million meters), data from 11 utilities, as well as other public/private datasets related to energy usage. Socio-cconomic building characteristics. including location based on Public Use Microdata Area (PUMA, 2400 PUMAs in the U.S.-see see inset), are sampled from distributions generated from U.S. Census survey responses. Please Wilson et al. [45 for the complete description or App. B.3 for more details.
To create Buildings-900K, we extract 900K total energy consumption time series (in kilowatt-hours (kWh)) from each of the non-upgraded buildings the 2021 version of the EULP. To promote accessibility of our dataset, we also aggregate the 15-minute resolution to hourly to reduce the size. This data requires about 110 GB to store, significantly less than the entire EULP (70+ TB). We store all buildings within each PUMA in single Parquet file by year (there are two years, 2018 and an aggregated "typical meteorological year" (TMY) 45) and by building type fresidential/commerria timestamp and a column for each building's s energy consumption (8,760 rows per file). Processing the EULP to extract this data took ~03 days with Apache PySpark a co-e AWS cloud instance.
Processing and storage:
which amounts to 9,600 Parquet files. Each file has a column for the
Pretraining splits and loading: A validation set created by withholding the final two weeks of 2018. The test set consists of buildings in four PUMAs that are withheld from both training and validation. All splits use a 24-hour sliding window to extract 192-hour load sub-sequences. Because shuffling large datascts of sub- uub-sequences is computationally demanding, our platform provides
3

---

Table 1: Comparing popular building energy consumption datasets to our dataset Buildings- Our dataset has significantly more buildings (residential and commercial) located across a wide geographic area and spans multiple years, which enables studying large-scale pretraining for STLE
# buildings
Open access
Residential
Commercial
Total hours # Sites
Pecan Street 40
1,000
Electricity 41
370
x
12.9M
Building D.G.P. 2 27
1,636
X
/
53.6M
19
Low Carbon London 30
5,567
X
97 5M
Buildings-900K (simulated)
900,000
~ 15B
2400
Table 2: BuildingsBench STLF evaluation data.
Residential
Commercial
Years spanned
# Sites
Total days
Buildings-900K-TT -est (simulated)
915
565
TMY, 2018 2007-2018
4
1.1M
BuildingsBench (real)
953
970
10
1.2M
custom PyTorch [33 Dataset that creates an index file to map a shuffled list of integers to sub- sequence. Each line of the index file is accessible in ((1) time with Python's seek function. Apache PyArrow is used to efficiently load the indexed building's 's time series into memory.
Hosting and licensing: Buildings-900 for download with permissive CC-4. license (link available in App. A).
hosted by the Open Energy Data Initiative and is available
3.1 Feature Extraction
Beyond the load time series, also extract these covariates on-the-fly to condition forecasts on:
Calendar daytures-day
week, day of the year, and hour of the day-are are generated from the timestamp, which then eyclically encode with sine and cosine.
The latitude and longitude coordinate of the centroid of the building's PUMA (using metadata provided by the EULP) for encoding geospatial information.
A binary feature for building type (residential (0) commercial (1)).
4 BuildingsBench Evaluation Platform
BuildingsBende provides an open-source software platform for evaluating models on zero-shot and transfer learning for STLF using collection of real building datasets. To avoid confusion, in our analysis (Sec. 5). we will use the name BuildingsBench to refer only to the real building evaluation data and explicitly state whether the results for Buildings-900K-Tes (simulated) or for BuildingsBench (real) (Table 2 We now describe each task in more detail.
Zero-shot STLF:
For this task, pretrained models are asked to provide day-ahead forecasts for unseen simulated and real buildings. Our use of "zero-shot" here refers to setting where only one week of historical data is presumed available, and thus insufficient for training fine-tuning deep neural network. All available years used for each test building.
Transfer learning: Our transfer learning task assumes 6 months of metered data has been collected for each target building. about ~ 180 training samples. Fine-tuned models are then tasked with day-ahead forecasting for the next 6 months using 24-hour sliding window.
4.1 Real Building Datasets
Here, we briefly describe the real building data used for evaluating the tasks and defer additional details to App.(c Altogether, our real building benchmark has over 1,900 buildings and 1.2M days of energy usage.
Electricity 41: 370 commercial buildings in Portugal with data spanning 2011-2014.

---

Building Data Genome (BDG) Project 2 [27]] 1,636 commercial buildings across 19 sites in 2016 and 2017. We include buildings from U.S. sites (Panther. Fox, Bear, Rat).
Low Carbon London [30]: Energy consumption meter readings from 5,567 London, UK house- holds between 201 2011-2014. To keep the overall number of residential and commercial buildings roughly the same, keep a random sample of 713 buildings.
SMART
5
Meter readings for homes in western Massachusetts, U.S., between 2014-2016.
IDEAL
34
Electricity meter data from 255 homes in Edinburgh, UK, between 2016-2018.
Individual household electric power consumption [15]: Energy consumption from single
household in Sceaux, Paris, between 2007-2010.
Borealis 6-second load measurements recorded for 30 homes in Waterloo, ON, 2011-2012.
Processing and storage: We resample each time series hourly if the data is sub-hourly. Smart meter time series typically have missing values, sometimes extending over weeks or months. Buildings with more than 10% of the data missing were not included. For included buildings, we linearly interpolate missing values, and if values missing for span greater than week, we fill this with zeros. We also provide an option to exclude buildings with a max hourly consumption 5.1 MW (only 15 from electricity) to keep the range of consumption values similar between Buildings-900K and BuildingsBench test data, as extrapolation is not our focus. A small fraction of spikes in the time series due to noisy meter readings are classified outliers and filtered with a non-pornmae non-parametric distance- based sliding window algorithm [25 (see App. C.8). Each annual consumption time series per building is stored as CSV file.
Hosting and licensing: We host the processed versions of each dataset, along with the original permissive licenses, alongside Buildings-900K under the same CC-4.0 license. Our code is open- sourced under a BSD-3 license.
4.2 Evaluation Metrics
We primarily evaluate task performance with two metrics, the normalized root mean square error (NRMSE) and the ranked probability score (RPS). The NRMSE (also known as the coefficient of variation of the RMSE) is widely used, as it captures the ability to predict the correct load shape. For a target building with days of load time series,
M,24
NRMSE = 100 x
Σ (us j=1,i=1
0i,)?
y
24M
where ý is the predicted load, y is the actual load, and ÿ is the average actual load over all M days. In the appendix, we also report the normalized mean absolute error and normalized mean bias error (see descriptions in App. D).
The RPS is well-known metric for uncertainty quantification in probabilistic forecasting 13 It compares the squared error between two cumulative distribution functions (CDFs), the predicted CDF and the observation represented as a CDF. Define the indicator function 1.50 as if the actual load S y and otherwise. The continuous RPS for a predicted CDF f for the load hour is:
RPS 1 (F:(v) 1y. Pdy
Our platform implements the closed-form RPS for Gaussian CDFs, as well as a discrete RPS for categorical distributions (used by baselines that discretize the positive real line for token-based load forecasting, see Sec. 4.3) These are formally defined in App. D
4.3 Baselines
Various baselines are implemented and benchmarked in the BuildingsBench platform. For zero-shot STLF, we pretrain a representative time series transformer 47 on Buildings-900K Transformers The model described in brief
have recently gained significant interest for STLF here and with more detail in App.
5

---

Transformer (Gaussian): This model is the original encoder- transformer 43 repurposed for autoregressive time series forecasting, as proposed in Wu 47 It predicts Gaussian distribution at each time step ( + i conditioned on the past week and the previous i predictions. We train it to minimize the Gaussian negative log-likelihoon loss. Because our demand time series are highly non-Gaussian (periods of low consumption interspersed with bursts), the loss diverges during training when we use global standard scaling to normalize the data. The Box-Co: power transformation [6] removes this instability. To compute the RPS, we approximate a Gaussian in the unscaled space by backprojecting the scaled standard deviation (see App. D for details).
We also implement transformer variant that uses tokenization to predict discrete load tokens. This baseline allows us test whether quantizing loads into tokens is beneficial for large-scale pretraining. We also found that comparison between tokenization and Gaussian time series transformers was missing from the literature. The model is trained to predict categorical distribution over a vocabulary of load tokens by minimizing a multi-class cross-entropy loss. To quantize loads, we use simple faites
Transformer (Tokens):
2 KMeans clustering with K = 8,192 fit to the Buildings-900K training set. We merge clusters < 10 Watts apart to obtain compact vocabulary size of 3,747 tokens. See App. for analysis on our tokenizer design.
Each transformer is pretrained jointly on residential and commercial buildings at three different sizes: Transformer-S (3M params), Transformer-M (17M params), and Transformer-L (160M params). Models are trained on 1 billion load hours with early stopping. See App. E for more architecture details. App. for hyperparameter tuning details (for learning rate and batch size), and App. G for training compute requirements.
We also benchmark persistence forecasting on both the zero-shot STLF and transfer learning tasks:
Previous Day: For scenarios where the load conditions change relatively slowly, the previous day's load a strong baseline for day-ahead STLF [42].
Previous Week: The 24-hour load profile on the same day from the previous week is used [7
Ensemble: This persistence baseline computes a Gaussian distribution for each predicted hour t + whose mean is the average load at hour t + i over the past 7 days:
7
T(t+ 245); p(yets/0 H:) == N((i j
Σ (5(44-21)) - 1)) 1)8).
(4)
1=1
In STLF, outperforming persistence is indicator that model is producing meaningful forecasts. On the transfer learning task, we also benchmark the following forecasting baselines by training them directly the target building's 6 months of meter data with supervised learning:
LightGBM: The light
machine (LightGBM) 24 is a popular
algorithm suitable for STLF (32. We use the multi-step forecasting implementation from skforecast
3 with 100 estimators and no max depth.
Linear regression, DLincar: Inspired by Zeng et al. 449 we benchmark a linear direct multi-step forecaster that regresses all 24 future values as a weighted sum of the past 168 values. We also implement DLinear, which decomposes the time series with a moving average kernel, applies linear layer to each component, and then sums the two get the final prediction.
RNN (Gaussian): This is an autoregressive encoder-decoder recurrent neural network inspired by Salinas et al. (38) A multi-layer LSTM [18] first encodes the 168 past load values. The last encoder hidden state initializes the state of multi-layer LSTM decoder. A linear layer maps each output of the decoder to Gaussian parameters.
5 Benchmark Results
Here, we analyze baselines the two benchmark tasks guided by the following questions:
1. Can models pretrained on Buildings-900k generalize to real buildings?
2. Does fine-tuning a pretrained model on limited data from a target building lead to improved performance?

---

Table Zero-shot STLF results. Median accuracy (NRMSE) and ranked probability score (RPS) with best and second best highlighted. Lower is better. No fine-tuning is performed on any test building. Residential NRMSEs are naturally larger than commercial buildings, because the normal- ization factor-the building's & average consumption per hour-is small). For example, Transformer-L. (Gaussian) has an RMSE of 0.72 kWh and an NRMSE of 66.57% on the residential Sceaux dataset.
9o0kdings-00K---- (simulated)
Buildings Bench (real)
Commercial
Residential
Commercial
Residential
NRMSE (%)
RPS
NRMSE (%)
RPS
NRMSE (%)
RPS
NRMSE (%)
RPS
Persistence Ensemble
33.10
4.58
54.77
0.72
16.68
5.88
77.88
0.063
Previous Day Persistence
34.91
59.38
16.96
98.41
Previous Week Persistence
32.05
74,08
19.39
99.77
Transformer-L. (Tokens)
12.91
2.45
44.29
0.875
14.46
5.62
95.34
0.152
Transformer-L (Gaussian)
10.10
2.07
43.52
0.578
13.31
5.23
79.34
0.072
Table 4: Transfer learning results. We show median accuracy (NRMSE) and ranked probability score (RPS) with best and second best highlighted. Transformer-S and Transformer- M results are in Fig. Improvement due to fine-tuning.
Commercial buildings
Residential buildings
NRMSE (%)
RPS
NRMSE (%)
RPS
Not pretrained Not fine-tuned
Persistence Ensemble
16.80
5.97
78.54
0.057
Previous Day Persistence
16.54
98.35
Previous Week Persistence
18.93
100.20
Not pretrained + Fine-tuned
Linear regression
25.18
89.98
DLinear
23.41
87.89
RNN (Gaussian)
41.79
15.28
96.75
0.078
LightGBM
16.02
80.07
Transformer-L. (Tokens)
50.12
26.89
105.65
16.36
Transformer-L. (Gaussian)
37.21
15.94
92.99
0.081
Pretrained Not fine-tuned
Transformer-L. (Tokens)
14.20 13.03
5.11
94.11 11
0.140
Transformer-L (Gaussian)
4.47
79.43
0.062
Pretrained Fine-tuned
Transformer-L (Tokens)
14.07 (-0.13)
4.99
(-0.1)
94.53 (4042)
0.137 (-0.003)
Transformer-L (Gaussian)
12.96 (-0.07)
4.37
(-0.10)
77.20 (2.25)
0.057
(-0.015)
3. How does the number of pretraining buildings affect zero-shot generalization?
4. How does the size of the pretrained model affect generalization?
5.1 Zero-Shot STLF
For zero-shot STLF, models are evaluated on day-ahead forecasting on all unseen simulated and real buildings without fine-tuning. Results for Transformer-L and other basclines are shown in Table 3 aggregated by the median over all buildings. Due to space constraints, Transformer-S and M results are shown in Figs. 3e 3d and App. Figs. 7 Performance profiles over all buildings (For comparing models by examining the tails of the forecast error distribution) and per-datasct metrics with 95% stratified bootstrap CIs are reported in App. and
On unseen simulated buildings, the pretrained transformers outperform the persistence baselines in accuracy and uncertainty quantification. The average zero-shot STLF accuracy slightly improves from simulated to real commercial buildings (-0.1% NRMSE) and drops for residential buildings (443% NRMSE). RPS sensitive to the load magnitude, which makes it difficult to compute sim-to-real gap for uncertainty quantification. See Fig. for qualitative visualizations of the forecast
7

---

Electricity
Bectriitty
2.5
Smart HomeD
Smart HomeD
350
GreundEuth
350
Ground tout
Grouns
Ground
300
TTrniforme (Gauasan)
Tharffomer (Tooenal
2.0
Warsfumer
2.0
load 000h
250
load (6001)
lood (lWnS)
2.5
lcad (lWin)
2.5
200
200
1.0
1.0
15.0
150
100
Hour
day
Hour of day (b)
Hour of
Hlour day
(a)
(c)
(d)
Figure 2: Forecast uncertainty. Ground truth time series truncated to previous 24 hours for visibility. Light blue lines are 10 samples from the predicted distribution. a-b) Successful commercial building forecasts. c-d) Failed residential building forecasts.
uncertainty. The best pretrained model Transformer (Gaussian) outperforms the best persistence method Persistence Ensemble on real commercial buildings. However, Persistence Ensemble has better accuracy than the pretrained models on real residential buildings. These results suggest synthetic pretraining is viable for commercial zero-shot STLF- see Sec. 6.2 for more discussion on residential buildings.
5.2 Transfer Learning
This task evaluates fine-tuning on single target building for which months of data has been collected. Our fine-tuning protocol is to train for max of 25 epochs using the first months and to use the last month for early stopping with patience of 2 epochs. To ease the computational burden of fine-tuning model each building in the benchmark, 100 commercial buildings and fine-tune separately on each. We tune the pretrained transformers across all sizes and train randomly initialized transformers. Table 4 displays results aggregated by the median over all buildings; see Fig. [ for the Transformer-S and Transformer-M results. We also report performance for the pretrained models without fine-tuning.
randomly sample 100 residential and
Our results indicate that fine-tuning the pretrained transformers on limited data is likely to improve the performance the target building. For statistical robustness, we compute average probability of improving NRMSE due fine-tuning: P(X < <)) 100
NN Ei= [xx,Y,) where X, is the pretrained + fine-tuned NRMSE for building i and Y; is the pretrained NRMSE. The Transformer-N M models have the highest P(X Y). with the Gaussian model achieving 98% and 73% respectively for commercial and residential buildings and the Tokens model scoring 70% and 62.5% (Fig. 4. For Transformer-L (Gaussian), this drops 71% and 68% and 61.5% and 39% for Transformer-L (Tokens). We found that fine-tuming all layers of both pretrained models was necessary only fine-tuning the model's last layer led to slightly decreased performance (App. Encouragingly. the fine-tuned model (Transformer-L (Gaussian)) beats both LightGBM, strong baseline in the low-data regime, and the Persistence Ensemble both commercial and residential buildings.
5.3 Empirical Scaling Laws
To characterize how size and diversity of the dataset impacts generalization, we trained the Transformer-M (17M) models on 1K, 10K, and 100K simulated buildings. We use carly stopping to prevent the models trained on smaller datasets from overfitting. The Transformer-M model accuracy (Fig. Bab and RPS (Fig. 3b) commercial buildings roughly follows plwer-law scaling with diminishing returns. Performance remains roughly constant across dataset scales for residential buildings (see Fig. 7 in App.D). which we attribute to the stronger sim-to-real distribution shift compared to commercial buildings. This suggests naively increasing the dataset size may not result in large improvements. Nevertheless, the best performance is achieved when using the full pretraining dataset 900K buildings.
Pretraining buildings:
Model size: Here we compare the transformers of sizes S, M, and L. The NRMSE and RPS for real commercial buildings improves from to M, but performance plateaus or decreases from the M models (Fig. 3c and Fig. 3d). We suspect this is due to distribution shifts and auto-comeeltion in the

---

Commercial
Commercal
Commercal
Commertial
0.24
Tranfermrr (Bianoo
Taasfurn
8.50
Tharatomer TBlann
Percetence
Panziamrr (ixeno)
Fanahemar (a
5.70
Teraboee (Guarart (Dtar:
NR0135
0.20
7.50
0.355
0.345
5.50
0.16
0.440
5.30
0.130
0.325
10'
protraining buildings
pnetraining buildings
parameter
parameter
(a)
(b)
(c)
(d)
Figure 3: Empirical scaling laws for zero-shot generalization on commercial buildings. Intervals 95% stratified bootstrap Cls for the median across all buildings. a-b) Dataset scale zero-shot performance. The trends appear to be power-laws with diminishing returns. c-d) Model size vs. zero-shot performance. Residential results are in App.
Residential
Commercial
Residential
Commertial
g 0.12
4.50
0.13
0.08
parameter count
parameter count
parameter count
parameter count
Goussion (pretrained)
Gaussian (pretrained FT)
Tokens (pretrained)
Takens (pnetrained FT)
Figure 4: Model size vs. transfer learning. Pretrained vs. pretrained fine-tuned (FT) performance for S, M, and transformers. Intervals are 95% stratified bootstrap CIs of the median. The Transformer-M models show the most improvement after fine-tuning. The fine-tuned Transformer-M performance is comparable to the largest models. Improvement due to fine-tuning is less pronounced for the Transformer-L models, suggesting their zero-shot performance is saturated this task.
time series that causes the largest models to overfit, despite using sliding windows to extract training samples and aggressive early stopping. The good RPS performance of the smallest Transformer (Tokens) model is likely due to training on quantized load values. Quantization helps generative models efficiently learn important structure in the data and ignore negligible information (37 (our tokenizer achieves a <~63% dataset compression rate). Although, this does come at a cost of lower accuracy (Fig. 3c). While we show power-law scaling for residential building forecast accuracy (Fig. 7 the model performance may be saturated; see Aprr an in-depth discussion. We also show the relationship between model size and transfer learning performance in Fig. 4 The Transformer-M models demonstrate the largest performance gains due to fine-tuning, which confirms the high average probability of improvement scores (Sec. 5.2). The smaller impact of fine-tuning on Transformer-L performance suggests its zero-shot performance may be nearly saturated.
6 Discussion
6.1 Findings
Pretraining Buildings-900k leads to good zero-shot STLF performance real commercial buildings. More investigation needed to achieve similar results for residential buildings (Sec. 6.2)
Buildings-900K pretraining + fine-tuning on real buildings improves STLF for both commercial and residential buildings. Our best pretrained + fine-tuned baseline outperforms LightGBM and persistence in the BuildingsBench transfer learning task. Fine-tuning all model layers appears necessary, possibly to mitigate negative transfer caused by the distribution shift between simulated pretraining and real test data.

---

We observe an approximate power-law relationship with diminishing returns between dataset scale and zero-shot STLF for commercial buildings. Performance plateaus as model size increases, possibly due to overfitting and distribution shifts.
Pretraining tokenized building loads instead of continuous loads achieves worse performance overall. However, the tokenized model is more stable to train, as it naturally handles wide range of load values.
Pretraining with geospatial coordinates slightly improves generalization: Due to space con- straints, the results of ablation where the model ignores the building's 's latitude and longitude are provided in App. K Briefly, improvements in accuracy were modest (0.1 1% NRMSE).
6.2 Residential STLF Challenges
Residential loads are inherently more uncertain and variable than commercial loads because they are more sensitive occupant behaviour and changes in weather (e.g., temperature and humidity) (39). This is evidenced by the relatively large persistence NRMSEs (77.8) 77..8%%-99.77 99.7%% on BuildingsBench residential loads compared commercial loads 116.68%-193399%1
However, rather than decreasing the value of BuildingsBench, our work enables exploring directions such as the inclusion of weather covariates, multi-variate formulations of STLF, and benchmarking of advanced approaches [20]
6.3 Limitations
While we expect BuildingsBench to stimulate research on generalizable STLF, our framework has limitations. First, pretraining on simulated data is fundamentally limited if deploying on real buildings the goal. Mixing synthetic and real data during pretraining is an interesting direction to explore. Second, the pretraining and evaluation data is mainly representative of building energy consumption the northwestern hemisphere. Expanding our framework to include data from other global regions is needed to comprehensively evaluate generalization. Third, the stochastic occupancy model used to simulate residential consumption behavior for the pretraining data is more predictable and less chaotic than real behavior, which increases the sim-to-real gap for residential buildings. Moreover, the occupancy model does not capture behavior patterns that may appear prominently in a specific geographic region (particularly outside of the U.S.), which may hurt the generalization capabilities when deploying the model in these locations. Finally, due to limited time, we only pretrained vanilla transformers Buildings-900k We encourage future work that compares these results with state- of-the-art transformers [31 (6). We will maintain a leaderboard for BuildingsBench in our code repository, which will be updated with new model results.
7 Conclusions and Future Work
In this work, we introduce BuildingsBench, which consists of a large-scale simulated dataset and collection of real building datasets for benchmarking zero-shot STLF and transfer learning. Upon comparing the performance of pretrained transformers against persistence and traditional machine learning-based forecasting baselines, we observe promising results for commercial buildings and identify areas of improvement for residential buildings.
We plan to maintain and extend BuildingsBench in the following ways. As relevant EULP data be- comes newly available, we will release updated versions of Buildings-900k. To ground improvements to benchmark tasks in a concrete downstream application, we will look into adding reinforcement learning task to the benchmark for building control using STLF. Other promising directions for future work include exploring joint forecasting of weather and load and the impact of building metadata on performance. To facilitate this work, we plan to update the datasets with this auxiliary information. Overall, we hope that BuildingsBench will facilitate research for communities applying machine learning to the built environment, as well as those conducting foundational studies on large-scale pretraining and fine-tuning for time series.
10

---

Acknowledgments and Disclosure of Funding
This work was authored by the National Renewable Energy Laboratory (NREL), operated by Alliance for Sustainable Energy. LLC, for the U.S. Department of Energy (DOE) under Contract No. DE- (136-08G028308. This work was supported by the Laboratory Directed Research and Development (LDRD) Program at NREL. The views expressed in the article do not necessarily represent the views of the DOE the U.S. Government. The U.S. Government retains and the publisher, by accepting the article for publication, acknowledges that the U.S. Government retains a nonexclusive, paid-up, irrevocable, worldwide license to publish or reproduce the published form of this work, or allow others to do so, for U.S. Government purposes. The research was performed using computational resources sponsored by the Department Energy's Office of Energy Efficiency and Renewable Energy and located at the National Renewable Energy Laboratory.
The authors would like to thank Cong Feng and Alex Rybchuk for helping revise a draft version of this manuscript, as well as Dave Biagioni for providing valuable insights.
References
[1] Afram, A. and Janabi-Sharif,, Theory and applications of hvac control systems-a review of model predictive control (mpc). Building and Environment, 72:343-355, 2014.
[2] Amasyali, K. and El-Gohary. N. M. A review of data-driven building energy consumption prediction studies. Renewable and Sustainable Energy Reviews, 81: 2018. ISSN 13640321. doi: 10.10010)) 005. URL httpss///hhhiiigguuubbbee elsevier com/
retrieve/ (Bcr1eeee/p//118860221770000
[3] Amat Rodrigo, J. and Escobar Ortiz, J. skforecast, 5 2023. URL https:// com/
JoaquinAAratRoddiggg/SSe orecast
[4] Ardakanian, O., Huo, S., and Keshav, 6-second load measurement dataset, 2020. URL
https://doi. org/ 10. 1tt.:/0000..33////////555555 5683/SP2/R4SVBB
[5] Barker, S., Mishra, A., Irwin, D., Cecchet, E., Shenoy, P., Albrecht, J., et al. Smart*: An open data set and tools for enabling research in sustainable homes. SustKDD, August, 111(112):108, 2012.
[6] Box, G. E. and Cox, D. R. An analysis of transformations. Journal of the Royal Statistical Society: Series B (Methodological), 26(2):21 1-2243,,11233 1964.
[7] Chadoulos, S., Koutsopoulos, I., and Polyzos, G. C. One model fits all: Individualized household energy demand forecasting with a single deep learning model. In Proceedings of the Twelfth ACM International Conference on Future Energy Systems, pp. 466-474. ACM, 2021. ISBN 978-1-4503-833344 doi: 10. 00.11453447555.3465555 URL htLLhttps:/// :/dpl:://l.... acm org/doi/10 1145/3447555. 3466587
[8] Crawley, D. B., Lawrie, L. K., Winkelmann, F. C., Buhl, W. F., Huang, Y. J., Pedersen, C. O., Strand, R. K., Liesen, R. J., Fisher, D. E., Witte, M. J., et al. Energyplus: creating a new- generation building energy simulation program. Energy and buildings, 33(4):319-331 2001.
[9] Drgoña, J., Arroyo, J., Figueroa, I. C., Blum, D... Arendt, K., Kim, D., Ollé, E. P., Oravec, J., Wetter, M., Vrabie, D. L.,, et al. All you need to know about model predictive control for buildings. Annual Reviews in Control, 50:190-232. 2020.
[10] EIA. Energy information administration may 2023 monthly energy review. 2023. URL .
https:// htops://www... wtwp//wwaaaa.../////
[11] Gage, P. A new algorithm for data compression. C Users Journal, 12(2):23-38, 1994.
[12] Gebru, T., Morgenstern, J., Vecchione, B., Vaughan, J. W., Wallach, H., Iii, H. D., and Crawford, K. Datasheets for datasets. Communications of the ACM, 64(12):86 92, 2021.
[13] Gneiting, T. and Katzfuss, M. Probabilistic forecasting. Annual Review of Statistics and Its Application, 1:125-151, 2014.

---

[14] He, Y., Luo, F., and Ranzi, G. Transferrable model-agnostic meta-learning for short-term houschold load forecasting with limited training data. IEEE Transactions on Power Systems, 37 (4):3177-3180, 2022. doi: 10.1109/TPWRS http///wWS..222.1166660 22..2.1192..99
[15] Hebrail, G. and Berard, A. Individual household electric power consumption. UCI Machine Learning Repository, 2012. DOI: 10. 24432/C58K54
[16] Hendrycks, D. and Gimpel, Bridging nonlinearities and stochastic regularizers with gaussian error linear units. arXiv preprint arXiv:1606.00m
2016.
[17] Hertel, M., Out S., Neumann, O., Schafer, B., Mikut, R., and Hagenmeyer, V. Transformer neural networks for building load forecasting. Tackling Climate Change with Machine Learning: workshop at NeurlPS 2022, 2022.
[18] Hochreiter, S. and Schmidhuber, Long short-term memory. Neural computation, 9(8): 1735- 1780, 1997.
[19] Hong, T. and Fan, S. Probabilistic electric load forecasting: A tutorial review. International Journal of Forecasting, 32(3):11--338, 2016. ISSN 01692070. doi: 10. 10 011. htt.:://ssiccciniioooooeeeeereer
[20] Hong, T., Pinson, P.. and Fan, S. Global energy forccasting competition 2012. International
Journal of Forecasting, 30(2):357-363, 2014. ISSN 0169-2070. doi: httihppps//doii.o///0000006..
jfforecaxt.011..7.000.. URL https::tp://wwt http://wwww.eeee........ooooaa/a $0169207013000745
[21] Huy, P. C., Minh, N. Q., Tien, D., and Anh, T. Q. Short-term electricity load forecast- ing based on temporal fusion transformer model. IEEE Access, 10:106296-106344, 2022. doi: 10. (0.110//ACEESSS2222.31110
URL Utt. ieee. wwweeorgg//ooooeee doccuuntttt/
9910162/
[22] IEA. Buildings. 2022. URL https: //ww. iea.ie.... ies.orggreportt//ulldiaa
[23] Johnson, J., Douze, M., and Jégou, H. Billion-scale similarity search with GPUs. IEEE Transactions on Big Data, 7(3):535-544 2019.
[24] Ke, G., Meng. Q., Finley, T., Wang, T., Chen, W., Ma, W., Ye, Q., and Liu, T.-Y. Lightgbm: A highly efficient gradient boosting decision tree. Advances in neural information processing systems, 30, 2017.
[25] Knox, M. and Ng, R. T. Algorithms for mining distancebased outliers in large datasets. In Proceedings of the international conference on very large data bases, pp. 392-403, Citeseer, 1998.
[26] Loshchilov, and Hutter, F. Decoupled weight decay regularization. In International Conference on Learning Representations, 2019,
[27] Miller, C., Kathirgamanathan, A., Picchetti, B., Arjunan, P., Park, J. Y., Nagy, Z., Raftery, P., Hobson, B. W., Shi, Z.,, and Meggers, F. The building data genome project 2, energy meter data from the ASHRAE great energy predictor III competition. Scientific Data, 7(1 (338:36, 2020.
ISSN 2052-4463. doi:
URL htttp://wwwt nature. com/
articlle//s41557-0000
[28] Mocanu, F., Nguyen, P. H., Gibescu, M., and Kling, W. L. Deep learning for estimating building energy consumption. Sustainable Energy, Grids and Networks, 6:91-99, 2016.
[29] National Renewable Energy Lab. Energyplus. 2023. URL /RLLasss///eeergplluu hetplllsssseeee/eeomomm
[30] Networks, U. P. houscholds, 2018
Smartmeter energy consumption data in london URL https :
gov. uk/dataset/
smart energy sesergy-use-daae
use- data- in- london households
[31] Nie, Y., H. Nguyen, N., Sinthong, P., and Kalagnanam, J. A time series is worth 64 words: Long- term forecasting with transformers. In International Conference on Learning Representations, 2023.
12

---

[32] Park, S., Jung, S., Jung, S., Rho, S., and Hwang, E. Sliding Sindow-basel lightgbm model for electric load forecasting using anomaly repair. The Journal of Supercomputing, 77(11): 12857-12878, Nov 2021. ISSN 1573-0484. doi: 10. 10.10077111227--01-0077777 URI.https //doi. 1007 0t10///112277-211 -3787777
https
[33] Paszke, A., Gross. S., Massa, F., Lerer, A., Bradbury. J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., Desmaison, A., Kopf, A., Yang. E., De- Vito, Z., Raison, M., Tejani, A., Chilamkurthy, S., Steiner, B., Fang, L.., Bai, J., and Chintala, S. PyTorch: An Imperative Style, High-Performance Deep Learn- ing Library. In Advances in Neural Information Processing Systems 32, pp. 8024- 8035. Curran Associates, Inc., 2019. URL http::/paperr neuriss. cctppppprr.cpcp/ library
9015 1:15--ytorch--ao aanichanniimmeren pdf
[34] Pullinger, M., Kilgour, J., Goddard, N., Berliner, N., Webb, L., Dzikovska, M., Lovell, H., Mann, J., Sutton, C., Webb, J., et al. The ideal household energy dataset, electricity, gas, contextual sensor data and survey data for 255 uk homes. Scientifie Data, 8(1):146, 2021.
[35] Radford, A., Narasimhan, K., Salimans, T., Sutskever, I., al. Improving language understand- ing by generative pre-training. 2018.
[36] Rathnayaka, P., Moraliyage. H., Mills, N., De Silva, D., and Jennings, A. Specialist vs generalist:
A transformer architecture for global forecasting energy time series. In 2022 15th International Conference on Human System Interaction (HSI), pp. 1-5. IEEE, 2022.
[37] Razavi, A., Van den Oord, A., and Vinyals, O. Generating diverse high-fidelity images with vq-vae-2. 2. Advances in neural information processing systems, 32, 2019.
[38] Salinas, D., Flunkert, V., and Gasthaus, J. DeepAR: Probabilistic forecasting with autoregres- sive recurrent networks. aeee::000041110 [cs, stat]. URL http:://rxa/ org/ abs// 1704 04110
[39] Stephen, B., Tang, X., Harvey, P. R., Galloway, S., and Jennett, K. I. Incorporating practice theory in sub-profile models for short term aggregated residential load forecasting. IEEE Transactions Smart Grid, 1991:199,-, 2015.
[40] Street, P. Pecan street dataport, 2021. URL //ww//ww// www.pecanstreet.org// org/datapott..tt..oaaa.ommo
[41] Trindade, A. 0lcerricty_oaaDDigram5010111111 12014. UCI Machine Learning Repository, 2015. DOI: 10. 24432/C58C86
[42] Tsakoumis, A., Vladov, S., and Mladenov, V. Daily load forecasting based on previous day load. In 6th Seminar Neural Network Applications in Electrical Engineering, pp. 83-86, 2002. doi: 10.100//EURELL200000007777
[43] Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L.. and Polosukhin, I. Attention is all you need. //arxiv. org/ abs 03762
[cs], 2017. URL http!
[44] Wei, T., Wang, Y., and Zhu, Q. Deep reinforcement learning for building hvac control. In Proceedings of the 54th annual design automation conference 2017, pp. 1-6, 2017.
[45] Wilson, E. J., Parker, A., Fontanini, A., Present, E., Reyna, J. L., Adhikari, R., Bianchi, C., CaraDonna, C., Dahlhausen, M., Kim, J., et al. End-use load profiles for the us building stock: Methodology and results of model calibration, validation, and uncertainty quantification. Technical report, National Renewable Energy Lab (NREL), 2021. URL wtl.https:/// hthpps:/www..r nrel gov/docs/Ey22osti 80889, pdf
[46] Wu, H., Hu, T., Liu, Y. Zhou, H., Wang, J., and Long, M. Timesnet: Temporal 2d-variation mod- eling for general time series analysis. In International Conference on Learning Representations, 2023.
13

---

[47] Wu, N., Green, B., Ben, X., and O O'Banion, S. Deep transformer models for time series forecasting: The influenza prevalence case. arXhv:2001.08317, 2020. URL Uttttp////arxi erg/abs/ 2ttas/2011.0000 08317
[48] Xu, X. and Meng, Z. A hybrid transfer learning model for short-term electric load forecasting. Electrical Engineering, 1023))1371-1381, 2020. doi: 10.1007/s00202-22 00000000000---00 x. URL
http://link:
springer www.e//gennnis.iingrrr...e com 100 /ttp://0000000000000000000000000
[49] Zeng, A., Chen, M., Zhang, L., and Xu, Q. Are transformers effective for time series forecasting? arXiv preprini 220x::2055.35044 2022.
[50] Zhang, G., Wei, C., Jing, C., and Wang. Y. Short-term electrical load forecasting based on time augmented transformer. International Journal of Computational Intelligence Systems, 15(1):67, 2022.
[51] Zhang, I.,, Plathottam, S., Reyna, J., Merket, N., Sayers, K., Yang. X., Reynolds, M., Parker, A., Wilson, E.,, Fontanini, A., et High-resolution hourly surrogate modeling framework for physics-based large-scale building stock modeling. Sustainable Cities and Society, 75:103292, 2021.
[52] Zhang, L., Wen, J., Li, Y., Chen, J., Ye, Y., Fu, Y., and Livingood. W. A review of machine learning in building load prediction. Applied Energy, 285:1 16452, 2021. ISSN 03062619. doi: 10. 10.1016// 1a1006/..eenergy222211.... 16452. (tLtsttts:///iin p11/S03362619210002999
elsevier www.:etrieveeeeooeeeoeeee..oooae
14

---

Appendix
Table of Contents
Appendix
Details of hosting, licensing, and maintenance
Appendix
Dataset datasheet
Appendix
BuildingsBench benchmark datasets details
Appendix
Details of evaluation metrics
Appendix
Time series transformer details
Appendix
Training details
Appendix
Compute details
Appendix
Benchmark performance profiles
Appendix
Per-dataset results
Appendix
Additional empirical scaling law results
Appendix Ablation studies
Appendix
Author responsibility statement
Appendix Additional qualitative results
A
Hosting, Licensing, and Maintenance
Our datasets and code are available via the following links:
Github: http://://gggattaoo///////////aa
D httpa///wwg.......o.......ooooaa
Datasets and Website: Vetiiig.hhgggg.//b.a....ooo/o//n
Tutorials: http:///weeeegggaa../////////aaa
Dataset DOI: /ttp:tgg///bb..g////222////////
As described in Sec. 3 and Sec. Buildings-900K and the BuildingsBench benchmark datasets are available for download under CC-4.0 license and our code is available under BSD 3-Clause license. We ensure the long-term availability and maintenance of the data by hosting it on the Open Energy Data Initiative (OEDI) platform (https: //data openei. org/ /agg/tt........... The OEDI centralized repository for storing energy-related datasets derived from U.S. Department of Energy projects.
B Datasheet for Buildings-900K
Questions from Datasheet for Datasets (v8)
B.1
Motivation
Q: For what purpose was the dataset created?
This dataset was created to research large-scale pretraining of models for short-term load forecasting (STLF). It specifically addresses a lack of appropriately sized and diverse datasets for pretraining STLF models. Buildings-900K, which consists of simulated building energy consumption time series, derived from the National Renewable Energy Lab (NREL) End-Use Load Profiles (EULP) database. We emphasize that the EULP not originally developed for studying STLF. Rather, was developed as a general resource to "...help electric utilities, grid operators, manufacturers, government entities, and research organizations make critical decisions about prioritizing research and development, utility resource and distribution system planning, and state and local energy planning and regulation." 455).
Q: Who created the dataset (e.g., which team, research group) and on behalf of which entity (e.g., company, institution, organization)?
15

---

Researchers at NREL created Buildings-900k Consent from the NREL developers of the EULP project was obtained in writing to extract, process, and redistribute a subset of the EULP. The EULP is a multiyear multi- multi-institutionall collaboration lead by NREL and is publicly available for download with permissive CC-4.0 license.
Q: Who funded the creation of the dataset?
This dataset was developed with funding provided by the Laboratory Directed Research and Develop- ment program at NREL.
B.2
Composition
Q: What do the instances that comprise the dataset represent (e.g., documents, photos, people, countries)?
Annual hourly energy consumption from EnergyPlus (8 simulations. EnergyPlus is a high fidelity building energy consumption simulator which has been continuously maintained and updated for over 20 years by the U.S. Department of Energy.
Q: How many instances are there in total (of each type, if appropriate)?
There are 1.8 million annual time series instances (900K buildings x 2 years).
Q: Does the dataset contain all possible instances or is it sample (not necessarily random) of instances from a larger set?
Buildings-900K consists of subset of the EULP database. Specifically, we extract only the out. site_ site__nergy_... total. potal.energe energy_consumplee consumption time series from each upgrade=0 building in the 2021 version for weather-years AMY2018 and TMY3. The EULP contains additional end-use time series for each building, the EnergyPlus XML model files, the EnergyPlus weather files, and variety of additional simulations under various electrification scenarios. We did not include these additional data in the initial version of Buildings-900K, although we may consider adding them in the future.
Q: What data does each instance consist of?
Each instance is stored in a Parquet file and has timestamp information and load values (in kWh). We also have spatial metadata that maps each building U.S. county.
Q: Is any information missing from individual instances?
No.
Q: Are relationships between individual instances made explicit (e.g., users' movie ratings, social network links)?
Each building has unique identifier and is assigned to a county. That county (or Public Use Microdata Area-PUMA) has its own unique identifier. Each PUMA contains many buildings that are geographically close to each other.
Q: Are there recommended data splits (e.g., training, development/valiiattoon testing)?
For test: We withhold all buildings in 4 counties from both AMY2018 and TMY3 weather- years.
We withhold dates 2018- 12-17 to 2018-12-3 from all buildings in the training set use as held-out validation data (e.g.. for early stopping).
For validation:
For pretraining: All remaining data is used for pretraining.
Q: Are there any errors, sources of noise, or redundancies in the dataset?
No.
Q: Is the dataset self-contained, does it link to otherwise rely on external resources (c.g., websites, tweets, other datasets)?
It is self-contained.
16

---

Q: Does the dataset contain data that might be considered confidential (e.g., data that is protected by legal privilege or by doctor-patient confidentiality, data that includes the content of individuals' non-public communications)?
No. All simulated building models created by sampling from conditional distributions attributes calibrated to real data.
Q: Does the dataset contain data that, if viewed directly, might be offensive, insulting, threaten- ing, or might otherwise cause anxiety?
No.
Q: Does the dataset relate to people?
No.
B.3
Collection Process
Q: How was the data associated with each instance acquired?
Buildings-900K time series instances are the outputs of extensively calibrated and validated En- ergyPlus simulations. These simulations were created and run by the NREL EULP team on high performance computing resources. For completeness, we summarize here the steps taken by the EULP team to create the synthetic building models for these simulations, which are also described in the EULP documentation more detail [45]. The synthetic buildings used form large-scale U.S. residential and commercial building stock models were obtained from the ResStock and ComStock analysis tools. extensive calibration and validation effort based on utility meter data from millions of customers, end-use submetering data, and additional private data sources helped to ensure the accu- racy of the synthetic ResStock and ComStock models used in the EULP. new stochastic occupant behavior model for the residential building stock was developed to help calibrate the simulations. Uncertainty quantification based on a learned surrogate [5[] was also used for model calibration, in particular, to characterize sensitivity of simulation outputs (e.g., quantities of interest, or QOIs) with respect to different model parameters.
Q: What mechanisms or procedures were used to collect the data (e.g., hardware apparatus or sensor, manual human curation, software program, software API)?
See above.
Q: If the dataset is a sample from larger set, what was the sampling strategy (e.g., deterministic, probabilistic with specific sampling probabilities)?
Buildings-900K consists of a hand-selected subset of the EULP database.
Q: Who was involved in the data collection process (e.g., students, crowdworkers, contractors) and how were they compensated (e.g., how much were crowdworkers paid)?
N/A
Q: Over what timeframe was the data collected?
portion of Buildings-900K consists of building load time series simulated with weather from 2018.
Q: Were any ethical review processes conducted (e-g., by an institutional review board)?
No.
B.4 Prepooesing/ddennnggaaelnngg
Q: Was any prepoocesing/ddeaingglaeele of the data done (e.g., discretization or bucketing, tokenization, part-of-speech tagging, SIFT feature extraction, removal of instances, processing of missing values)?
The steps we followed to create Buildings-900K are as follows:
httpe://resstock.ntt nrel. gov
https://comstock.nreg gov
17

---

1. Select the out site_energy... total. energy_consuumtion time series from each AMY2018 and TMY3 building.
2. Aggregate the 15-minute resolution energy consumption to hourly by summing the values at intervals of four consecutive timestamps XX:15, XX:30, XX:45, and XX+1:00 (e.g., 12:15, 12:30, 12:45, 13:00). Note that we use a sum aggregation here. When aggregating meter readings (in kWh) for real buildings (e.g., the BuildingsBench evaluation data), take the average of the sub-hourly values.
3. Join the time series for all buildings in the same PUMA into single Parquet table.
Save each PUMA-level Parquet table (compressed with Snappy).
Q: Was the "raw" data saved addition to the preprocesed/cllamc///hbeeee data (e-g,, to support unanticipated future uses)?
The raw data (the EULP) is publicly accessible for download o wwwp:ei.aaapeeeii.o.....oo
Q: Is the software used to preprrocss//deen//lbbl the instances available?
Yes, at http:////gggttbo..oco///////aaa/
B.5
Uses
Q: Has the dataset been used for any tasks already?
In this work, evaluate pretrained models zero-shot STLF and transfer learning for STLF (fine-tuning pretrained models on limited data from a target building).
Q: Is there repository that links to any or all papers or systems that use the dataset?
We will add links papers or systems that use the dataset https::/data. openei org/ submissions/ 5859 and
Q: What (other) tasks could the dataset be used for?
Pretrained models can sometimes serve as used for wide range of tasks, including:
feature extractors. These features can be
Predicting energy consumption a building based on its characteristics.
Classifying user behavior (e.g., whether building is currently occupied) based on their energy consumption.
Classifying building type based on energy consumption.
Load disaggregation, that is, predicting the energy consumption of individual appliances based on the total energy consumption of building.
Among others, potentially. We did not explore these tasks in this paper.
Q: Is there anything about the composition of the dataset or the way it was collected and preprocessed/clenneddbbbee that might impact future uses?
Buildings-900K is (in its current state) representative of buildings in the United States. While we find evidence that models pretrained Buildings-900K generalize buildings in other countries in the northwestern hemispheres, such as the UK and Canada, it may not be representative of building energy consumption patterns other regions of the world.
Q: Are there tasks for which the dataset should not be used?
None that the authors are currently aware of.
B.6 Distribution
Q: Will the dataset be distributed to third parties outside of the entity (c.g., company, institution, organization) on behalf of which the dataset was created?
Yes.
18

---

Q: How will the dataset will be distributed (e.g,, tarball on website, API, GitHub)?
It will be publicly available for download the Open Energy Data Initiative (OEDI) website https://dan http/a...a.aaaa....aaaamaa openei http:/uumissiionSSeaeee
Q: ill the dataset be distributed under copyright other intellectual property (IP) license, and/or under applicable terms of use (ToU)?
CC-4.0.
Q: Have any third parties imposed IP-based or other restrictions the data associated with the instances?
No.
Q: Do any export controls other regulatory restrictions apply the dataset individual instances?
No.
B.7
Maintenance
Q:
Who is supportinghosttngnnaattiiinng the dataset?
Code is hosted and publicly available on Github. NREL will maintain the code and datasets. The data is hosted on the Open Energy Data Initiative site, which is centralized repository for energy-related datasets derived from U.S. Department of Energy projects.
Q: How can the owner/curaoor/maaager of the dataset be contacted (e.g., email address)?
The lead maintainer at NREL is Patrick Emami (Patrick. Emami @nrel. gov).
Q: Will the dataset be updated (e-g., to correct labeling errors, add new instances, delete instances)?
Yes. A version naming system will be used to indicate updated versions. We are also maintaining GitHub repository for the associated benchmark (BuildingsBench) where notifications of updates will be posted.
Q: Will older versions of the dataset continue to be supportc//hostc/mainnaaieee?
Yes, older versions will remain available on the OEDI website.
Q: If others want extend/augment//uild on/contribute the dataset, there a mechanism for them to do so?
Not officially, but our benchmark code is open source and pull requests welcome.
C BuildingsBench Datasets
C.1
This dataset is available from the UCI Machine Learning Repository under a CC-4.0 license. contains 15-minute consumption time series (in kW) for 370 clients in Portugal from 2011 to 2014. The magnitudes of the loads are much larger than residential homes, thus, we consider these time series as commercial buildings. After filtering for missing data, we kept 359 buildings. We included this dataset in BuildingsBench because of its permissive license, popular use in machine-learning- based time series forecasting studies 49 and size.
C.2 The Building Data Genome Project 2
This dataset is available" under an MIT License. It consists of measurements from 3,053 meters from 1,636 commercial buildings over two years (2016 and 2017). One or more meters per building measured the total electrical, heating and cooling water, steam, solar energy, water, and irrigation
www.:daaaaagggeeeoggeeeeeee genome project-2/
19

---

usage. This dataset was curated by the Building Data Genome Project, consortium of academics and practitioners. We use the whole building electricity meter measurements from Bear, Fox, Panther and Rat sites, totalling 611 buildings (from the CSV file electricity selerriittty_eeeeeededeeee cav). This dataset was included in BuildingsBench because of its permissive license, size, and diversity.
C.3 Low Carbon London
This dataset is available under CC-4.0 license. It consists of consumption measurements from 5,567 London households that participated in a project led by UK Power Networks between November 2011 and February 2014. The dataset consists of half-hourly consumption (in kWh), a unique houschold identifier, and timestamps. Due to the large number of houscholds included in the dataset, we randomly subsampled 713 of them (so to keep the overall number of residential and commercial buildings in the benchmark approximately similar). We included this dataset because of its permissive license and size.
C.4 SMART
This dataset is available online unaccompanied by a specific license o of aggregate electricity data for 7 residential homes (which called Home ..B.C.D,,,G.HH) collected by researchers at UMass Amherst. After filtering out 2 homes due to missing data, we were left with Home B (years 2014-2016), Home C (2014-2016), Home D (2016), Home F (2014-2016) and Home G (2016). We included this dataset in BuildingsBench due to its accessibility and because the homes located in the U.S. (Western Massachusetts),
It contains meter measurements
the simulated homes in Buildings-900K.
C.5 IDEAL
This dataset is available [nder CC-4.0 license. It has meter readings for electric and gas usage, as well as room temperature, humidity, and boiler readings from 255 UK homes. We use the total home electricity measurements. The measurements were collected by researchers at the University of Edin- burgh. After our filtering process, we kept 219 homes. This dataset was included in BuildingsBench due to its permissive license and size.
C.6 Individual household electric power consumption (Sceaux)
This dataset is available via Kaggle under DbCL-1.0 license. It contains consumption measurements from a home in Sceaux, France between December 2006 and November 2010. It provides 7 types of meter readings at one minute resolution with timestamps. We use the global active power readings. This dataset was included in BuildingsBench due to its permissive license and due to its popularity both in the literature 28 and on Kaggle.
C.7 Borcalis
This dataset is available Junaer Jnder CCO-1.0 license. is 6-second resolution measurements of total real power consumption for households Waterloo, ON for at least three months between 2011 and 2012. After filtering out homes with excessive missing data, kept 15 of them. The data was collected by researchers at the University of Waterloo. We included this dataset in BuildingsBench due to its permissive license and size.
C.8 Outlier Removal
Here we provide additional details on outlier removal for noisy meter data. The algorithm we use compares each load value all others in sliding window of 24 hours and computes the absolute difference with respect to the 1-nearest neighbor (1-NN). To select an outlier threshold, we compute
htcps://data.London gov wow.:a//ataaee//mmmttttettotttte wnw.::eeteetteeeee-e--eee--eeeee data- haata--h--onddn--househhlllllll-
cs. wmass http:/idexx.,phi httpc//aaseeeee//eeeeeeee///iaaa
ac. wttp.aaa...../a/a//////////////
20

---

the average daily peak load and average daily base load, and compute the difference. If the 1-NN distance is larger than this difference, we classify the spike as an outlier and replace the spike with the median of the sliding window.
D Evaluation Metries
D.1 Additional Accuracy Metrics: NMAE, NMBE
NMAE: Similar to the NRMSE, the normalized mean absolute error (NMAE) measures the ability of a model to predict the correct load shape. However, it is less sensitive large errors than the NRMSE. For a building with M days of load time series,
M,21
NMAE = 100
Σ j=1,i=1
2hi Dial
(5)
24M
NMBE:
The normalized mean bias error (NMBE) is informative about model's tendency to over or under-estimate building loads, which is practically useful to measure. However, model that tends to over-estimate under-estimate the load by an equal amount can achieve an NMBE close to zero (i.e., positive and negative errors cancel). Therefore, it not useful indicator of accuracy. For building with M days of load time series,
M,24
NMBE 100
Σ
(yi
(is)
(6)
24M
j=1,1=1
We visualize performance profiles and dataset-specific results for all metrics in App. H and App. ! respectively.
D.2 Categorical Ranked Probability Score
Here, we define the ranked probability score (RPS) for categorical distributions. This is used to
compute the RPS for time series transformer variant that predicts categorical distribution
vocabulary of [V] discretized load tokens. Each load token is the centroid of a cluster, where clusters estimated via KMeans. See App. E for more details about the tokenizer.
In what follows, we assume that the vocabulary of load tokens is sorted increasing order by value, that is, 0 S vo < <... < ftvi-1. E V.
Let p(Y.,) be the predicted categorical distribution over V for hour i on day j. We assume this is normalized distribution. which is accomplished in practice by computing the softmax of the model's predicted logits. The corresponding CDF for p(Y..;) is F(Y.s), which is given by the cumulative sum of the \V sorted entries of p(Y..)) Then, let F(Y..) be the CDF of the ground truth discretized load. If the ground truth category is k E (O...., | }. the CDF F(Y.;) is 0 up to index k - 1 and 1 thereafter (resembling step function that goes from 0 to at ground truth load index k). Finally, define ukk) as the difference between the maximum load value and the minimum load value assigned load token u[k] by KMeans (recall, load tokens are centroids of ID KMeans clusters).
Then, the categorical RPS at hour for a building with M days of load time series is
M M---1
CatRPS,
1
Σ
-
M
D.3 Gaussian Ranked Probability Score
The continuous ranked probability score for Gaussian distributions can be evaluated in closed form. In particular, let fi. and ăi, be predicted mean and standard deviation, and Y/., be the ground truth value. Define 2,, 16 - Then, the Gaussian (C)RPS at hour for a building with M days
21

---

of load time series is
M
+ erf (2,5))
exp
1
GaussCRPS: = M
Ōij
Zijj
2
2v22
-
+
2
2
Vx
(8)
==1
Gaussian
Gaussian
Gaussian approximation of the inverse Box- Cox transform: Our time series transformer variant that predicts Gaussian distribution for each hour in the day-ahead forecast uses Box-Cox power transformation to normalize the load values. In App. K show that standard scaling was insufficient to remove training instabilities caused by the non-Gaussian load time series. Since the Gaussian distribution is learned in the Box-Cox transformed space, we cannot directly use the GaussCRPS metric to evaluate uncertainty quantification the original un-scaled space (i.e., with units of kWh). Our current solution, introduced here, is compute Gaussian distribution in the original un-scaled space that closely approximates the distribution given by the inverse Box-Cox transformation (for reasonable range of standard deviations). We leave exploring advanced non-approximate solutions for future work.
First, we define the Box-Cox transformation of random variable X
x (x(
if + 0 if X = 0,
fx()
(9)
and its inverse
(YA+ 1)
if X if i = O.
X= --
(10)
(ex((Y)
We estimate a Gaussian distribution in the un-scaled space that approximates the (power-normal) distribution of the samples inverted with /T1 Recall that Pi.j and o;j are the predicted mean and standard deviation in the Box-Cox transformed space. We take the inverse of the predicted to be fx (Pr.)) (Eq. 10 Then, the standard deviation is estimated with:
tt (pr.s + Ō (Prs) 0, f (Ar.)) fx (As.s Ōz.))
(11)
(12)
tt + oū.j.
oi 2
(13)
The Gaussian distribution the un-scaled space is thus ( off). We find that this Gaussian is a good approximation for the distribution of the samples obtained by applying Eq, TO
to samples drawn from N ((aaj, o²,5) ) for reasonably small values of øi.j- We compare these two distributions in Fig. 5 When the model is uncertain about the forecast and the standard deviation in
the un-scaled space oij is large, the true distribution in the un-scaled space is heavily skewed
right, making Gaussian poor approximation (Fig. 5c
E
Time Series Transformer Details
Table 5: Transformer architecture hyperparameters.
Predicted Distribution
# Layers
Heads
Embedding Dim
MLP Dim
Params
Transformer-S
Categorical
2
256
512
3.3M
Transformer-M
Categorical
3
512
1024
17.2M
Transformer-L
Categorical
768
2048
160.7M
Transformer-S
Gaussian
2
256
512
2.6M
Transformer-M
Gaussian
3
512
1024
15.8M
Transformer-L
Gaussian
12
768
2048
160.7M
In this section, we provide additional details about the time series transformer architecture used in this work. We use the original encoder-decoder model from Vaswani et al. (43) as implemented in the
22

---

35
5-2.71
0.01
samples (N 500)
6- 13.59
6 0.05
samples (N 5500
40
5=.7.32
d-02
30
25
samples
20
10
10
95
100
105
110
60
80
100
120
140
-100
100
load (kWh)
load (kWh)
load (KWW)
(a)
(b)
(c)
Figure 5: Gaussian approximation of the inverse Box-Cox. Visualization of the Gaussian dis- tribution (fff)((aa))..)))))) Ōi.j) (orange) in the un-scaled space of loads (in kWh) for computing the RPS when using Box-Cox scaling. For reasonably small standard deviations ā the un-scaled space, this Gaussian is a reasonable approximation of the power-normal distribution given by the inverse Box-Cox (blue). c) When the model is highly uncertain so that large, the power-normal is extremely right-skewed. In this case, the Gaussian approximation is inaccurate.
PyTorch nn. Transrrrrrraarn fomer module. This autoregressive model is trained to predict the load at each time step with teacher forcing. Following Wu et al. [77, the attention from peeking at the target value at each time step. The first value in the input sequence passed the decoder is the last value of the input sequence passed to the encoder. The Gaussian time series transformer uses a linear layer (with parameters shared across time steps) to transform the high-dimensional output of the decoder into a mean and standard deviation at each time step. The time series transformer trained on tokenized load values uses a linear layer to transform the decoder outputs into [V] logits for the cross-entropy loss, also shared across time steps. Our tokenizer based on KMeans (a stochastic algorithm) followed by token merging step. Depending on the random seed, multiple runs of the tokenizer will produce token vocabularies of different sizes. The tokenizer is described in detail in App. E.
masking in the decoder to prevent
The architecture hyperparameters for the L, M, and models are shown in Table The largest model has 12 layers, 12 heads, and an internal dimension of 768 selected to match Radford et al. [35]. but uses smaller feedforward dimension of 2048 to keep the number of parameters at approximately 10 We downsize these architecture hyperparameters to reduce the number of parameters by roughly an order of magnitude for the S and M models. Similar to Radford al. [35). all models use GELU [16 activations and N 0.02) weight initialization.
Model input encoding: Let E be the embedding dimension and be scaling factor given by E/25 /56 (e.g.. s = =3 for Transformer-L). The model inputs are encoded, concatenated, and passed to the transformer encoder follows:
day of year, day of week, hour of day: These are encoded into 2-dim space with (in( and cos(TTr) then projected with linear layer 32s-dim.
latitude, longitude: These scalars are each projected 32s-dim space with linear layers.
building type: A linear embedding layer is used to learn projection for each of residential and commercial building types to 32s-dim space.
continuous loads: When the load is continuous scalar, it is projected to 64s-dim space with a linear layer.
discrete load tokens: When the load is discrete token, a linear embedding layer is used to learn a projection for each of the [ tokens to 64s-dim space.
These various embeddings are concatenated into a feature vector of size E at each time step.
E.1 Load Tokenizer
We explore quantizing load values into a vocabulary of discrete tokens to closely mimic the application of transformers to natural language. The design of the tokenizer is detailed here. We use simple KMeans clustering with basic merging strategy inspired by Byte Pair Encoding [[] which merges
23

---

Algorithm Load tokenizer
// Inputs: loads ((n,1) 1) array), initial centroids, T 0.01 threshold (kWh)
kmeans faiss. Kmeans(())
kmeans. tomnsttrainnni (1oads)
// Kerge step. Initialize state.
sorted_centroidd sort (kmeans centroids)
currenti_centridd sorted_
merged_centroids=
lemp_centroids
cntereen_centroiin
// Iterate over sorted centroids in increasing order for 1 K do
// The next centroid is less than away, merge it
if coorted_centroidss/1 current_centroid then
temp-centroidd..ppe appendd ottp://oooo...pooooooooooooooooo [i]) else
temp_centroids [
current_centrow
[i]
// the last centroid to the new 1ist merged centroids
cerged_centroids (mennnn return kmeans, centroddde
Lood loicn Froouse
Load Tiwen Fresuencc
207
5000
Load Talzn Frequctcy
302
5000
--c00
Loud lakan
3000
2000
2000
Lo0
30
LOC0
1000
1000
Lood Tokens (small Iarge)
Load Tokers Gmall
large)
Load Tokens (omall large)
[V] 3,747 (K 8 8,192) (b) | 8,19 (No merging)
| 474 (K 512)
Figure 6: Load token frequency. We visualize histograms of token frequencies from a subsample of Buildings-900k, After merging K 8.19 into 3,747, there are some tokens that appear often near load values of zero while most other tokens have nearly equal usage. The mean absolute quantization error for commercial buildings is approximately 0.2 kWh whereas for residential buildings 0.003 kWh. b) Without merging, most tokens correspond to small load values and all appear to have roughly equal usage. When [V] is small (474), there few tokens for large load values.
clusters that are overly close to each other. We leave exploring other simple and sophisticated tokenization strategies for future work.
The tokenizer uses KMeans clustering on random subsample of load values from the Buildings- fit K clusters (the size of the subsample is chosen internally by faiss-gpul The merge step is as follows. First, we sort the K cluster centroids in increasing order and then iterate over them. For any cluster centroid i, if there are successive centroids that less than 0.01 kWh greater than i, we replace all of these centroids with a new centroid equal to their average. We provide pseudocode in Alg. M This algorithm is stochastic, such that running the tokenizer multiple times will produce vocabularies with different (but similar) sizes [ To tokenize continuous load value, we first look up the index of the original centroid it belongs to, then use this index look up the corresponding index in the list of merged centroids. See Fig. 6 a visual comparison of three different vocabularies produced by the tokenizer, and App. K. for ablation study results on the choice of K and the merging step.
900K training
F Training Details
In this section, we describe how training hyperparameters were selected for the transformer models. We use the loss on the Buildings-900K validation dataset for model selection. However, observed

---

that a better Buildings 900K validation loss does not necessarily indicate better downstream perfor- mance on the real cutt-o--disributioon evaluation data. We encourage exploring sophisticated model selection strategies that account for distribution shifts in future work.
Pretraining hyperparameters: The learning rate for all models uses linear warmup of 10K steps followed by cosine decay to zero. Models are trained on I billion load hours, as we observed in early runs that the validation loss would no longer improve after this point. We perform grid search over the max learning rate (6e-4, 6e-5, 6e-6] and the batch size [64, 128, 256] for the two Transformer-S models. For the Transformer-S (Gaussian) model, the highest learning rate and smallest batch size had the best validation loss. However, when applying these hyperparameters to the larger models, the high learning rate of 6e-4 caused training instability. Thus, we used the lower 6e-5 learning rate, which performed best with the smallest batch size of 64 (650K total gradient updates). Similarly for the Transformer-S (Tokens) model, the highest learning rate achieved the best validation loss, but for the larger models we used the lower 6e-5 learning rate with batch size of 64. We use the Adam W [26) optimizer with 31 = 0.9, 32 0.98, le-9, and weight decay of 0.01 following Radford et al. [35
Transfer learning hyperparameters: We fine-tune all layers of the transformers (as opposed to only the last logits layer-see App. K.4 for an ablation study on this) with a lower learning rate of 1e-6. We allow a maximum of 25 fine-tuning epochs with an early stopping patience of 2. We tune the learning rate for the Linear, DLinear, and RNN baselines by sweeping over 5 values: fle-2, 1e-3, le-4, le-5, le-6). All prettaetd models are trained for maximum of 100 epochs.
G Compute Details
Buildings-900K processing: Creating the Buildings-900K dataset by processing a subset of the EULP was accomplished with a 96-core AWS EC2 instance in 2-3 days.
The largest model variant, the Transformer-L, was trained on billion hours from Buildings-900k with one NVIDIA 40GB A100 GPU in 24 hours. Smaller model variants train faster, in about 18 hours. We use PyTorch automatic mixed precision training with gradient scaling, which is crucial for achieving fast training times.
Model pretraining:
The amount of time to run each benchmark task varies depending on model inference speed. This task uses all available years for every building the benchmark. For the largest model variant, it takes about 2-3 hours on one NVIDIA 40GB A100. By contrast, the persistence baselines run well under one hour.
Zero-shot STLF:
Transfer learning: Repeating the fine-tuming process for every building in the benchmark can be computationally demanding. For example, we sub-sample 200 total buildings (100 residential and 100 commercial) use for this task, and this takes the Transformer-L models 1,5-3.5 hours to complete on one NVIDIA 40GB A100. We estimate it would take over 15 hours to include all buildings in this task. Models that are fine-tuned from frozen pretrained weights finish this task faster, as we stop the fine-tuning process with early stopping using a patience of 2 epochs.
H Performance Profiles
Performance profiles plot the fraction of all buildings with performance greater than a threshold r for range of threshold values. These can be used to understand the tails of the error distributions across all buildings in the benchmark. For example, these plots show the fraction of buildings with extremely high forecasting errors.
Here we show performance profiles for commercial (Fig. Fig. 13 and residential (Fig. 14, Fig. 15 BuildingsBench buildings for the NRMSE, NMAE, NMBE, and RPS metrics. For reference, we also plot the performance profile of the Persistence Ensemble bascline. The methods that estimate the forecast with Gaussian distribution (the Persistence Ensemble and Transformer (Gaussian)) achieve noticeably better NMBE performance than Transformer (Tokens).
I Benchmark Dataset Results
Per-dataset zero-shot accuracy (NRMSE, NMAE, NMBE) visualized in Fig. 16
25

---

Residential
Residential
Residential
Residenkia
1:15
0.40
1.05
0.20
Tranfermrr (Thlen
1.05
NRASEE
NIDASE
0.95
0.95
0.85
0.08
10a
10' parameter count
protraining buildings
pnetraining buildings
parameter count
(a)
(b)
(c)
(d)
Figure 7: Empirical scaling laws for zero-shot generalization on residential buildings. Intervals are 95% stratified bootstrap Cls of the median. a-b) Dataset size zero-shot performance. c-d) Model size vs. zero-shot performance.
Commercial
Residential
Commertial
Residential
Pemibiess
0.30
Thanaforer (Gunnder))
NRNSE
NANOSE
3.50
0.48
0.20
0.44
10?
paramster
paramater
parameter
(a)
(b)
(c)
(d)
Figure 8: Model size vs. Buildings-900K -est zero-shot STLF performance. Intervals are 95% stratified bootstrap CIs of the median. The trends in (a)-(c) suggest superior generalization to the test data is achievable by larger models, which might be unsurprising due to the small distribution shift between the simulated pretraining and simulated test data (relative the large distribution shift between the simulated pretraining and real test data).
J Additional Empirical Scaling Law Results
The residential building results for the experiments where the Transformer-M models are trained under increasing dataset scale (size and number of unique pretraining buildings) and where model scales are compared are shown in Fig. 7 In most cases, the accuracy (NRMSE) and uncertainty quantification (RPS) independent of the dataset and model size. Exceptions the Transformer-M (Gaussian) performance on the smallest pretraining dataset size and the effect of model size on the Transformer (Gaussian), which appears to show a power-law scaling. By contrast, Fig. shows that the smallest Transformer-S (Gaussian) model achieves comparable performance to the larger variants on commercial buildings. A possible simple explanation is that the small model struggles to learn to jointly forecast for both residential and commercial buildings due to limited capacity. key benefit of the larger model, then, is unlocking the ability jointly forecast for both types of buildings. As discussed in Sec. 6.2 alternative modeling approaches that consider auxiliary inputs might be more successful for large-scale pretraining of residential building STLF. We believe this is an interesting future research direction. We also examine empirically how performance on the simulated Buildings-90n Mooeengs--O--TTeeee -Tet varies with model scale (Fig. R}. and observe trends that clearly indicate superior generalization is achievable with larger models in this setting.
K Ablation Studies
We explore the following aspects of the transformer models in this section:
App.
K.1
The size of the vocabulary and the impact of the merging step in the load tokenizer.
App.
K.2
Using standard scaling vs. Box-Cox scaling for training the Transformer (Gaus-
sian) model.
App.
K3
Impact of the lat, lon covariate generalization.
26

---

Table 6: Ablation study results on the zero-shot STLF task. Median accuracy (NRMSE) and ranked probability score (RPS). Results are for Transformer-L. (Tokens). We ablate the use of the lat, covariate and the size of the token vocabulary [V. The vocabulary with 8,192 tokens does not use merging (note the high RPS for residential buildings).
Buildings-9000 900K- Test (simulated)
BuildingsBench (real)
Commercial
Residential
Commercial
Residential
M
Lon
NRMSE (%)
NRMSE (%)
NRMSE (%)
RPS
NRMSE (%)
RPS
3,747
x
15.22
2.79
45.85
0.913
14.40
5.73
97.84
0.199
474
12.70
1.62
42.94
0.544
14.71
5.16
94.75
0.113
8,192
15.06
2.85
45.50
7.05
14.62
5.90
99.88
14.37
3,747
/
12.91
2,45
44.29
0.875
14,46
5.62
95.34
0.152
2
Box Cox scaler
Standard scaler
1
0
loss
-1
-2
Mont Manty
-3
-4
-5
50000 100000 150000 200000 step
Figure 9: Box-Cox scaling vs. standard scaling training stability. Standard normalizing the load values did not prevent the Transformer (Gaussian) model from suffering from training instability. Box-Cox scaling was successful at achieving stable training.
App.
K.4 Fine-tuning all layers vs. only the last layer for transfer learning.
K.1 Load Tokenizer
Our experiments conducted with merged vocabulary of size | 3,747 (K 8,192). To investigate the impact of using a smaller vocabulary, we trained the Transformer-L. model with a merged vocabulary of size V 474 (K 512) (Fig. [6cp. We also trained model without merging, thus, |V| 8,192 (K = 8, 192) (Fig. 6b). The results in TTale/6Jindicee small (V) 474), the accuracy on real commercial buildings somewhat deteriorates. Without merging ([V) 8, 192), the RPS on real residential buildings is significantly worse. Nearly all tokens correspond small load values, which causes the model to allocate probability large number of incorrect tokens whose load values are very close to the target value.
that when the vocabulary is
K.2 Standard Scaling Data Normalization
As can be seen in Fig. global standard normalization was not sufficient prevent training instability (large spikes in the loss) due to the highly non-Gaussian load values. By contrast, Box-Cox scaling successful stabilizing training.
K.3 Latitude, Longitude Covariate
In our experiments, the transformer models have latitude, longitude covariate that roughly localizes where building is in the world. The result of training Transformer-L (Tokens) without this covariate
27

---

Table Ablation study results on transfer learning task. Median accuracy (NRMSE) and ranked probability score (RPS). For statistical robustness, we compute average probability of improving NRMSE due to fine-tuning: P(X < Y)= == 100 E i= [xx where X; is the pretrained + fine- N
tuned NRMSE for building and Y; is the pretrained NRMSE. We compare fine-tuning only the last layer and fine-tuning all layers.
Commercial buildings
Residential buildings
NRMSE (%)
RPS
P(X <
NRMSE (%)
RPS
P(X<M
Fine-tune last layer Transformer-L. (Tokens)
14.18
5.07
53.5
93.51
0.136
47
Transformer-L (Gaussian)
12.93
4.50
51
79.57
0.063
36
Fine-tune all layers Transformer-L. (Tokens)
14.07
4.99
61.5
94.53
0.137
39
Transformer-L. (Gaussian)
12.96
4.37
71
77.20
0.057
68
are in T The buildings the Buildings-900K-1 set are from held-out PUMAS (i.e., counties) with never-before-seen latitude, longitude coordinates. Without latitude and longitude covariates, observe a negligible drop in zero-shot accuracy on real buildings, but a significant loss in accuracy on the simulated muildinng-900K-Tess split. As the majority of real buildings are located in non- U.S. out-of -uttr--ddistibutiinn locations, the latitude and longitude covariate appears to not be particularly helpful for generalizing to these buildings. This is despite mapping the non-U.S, latitude longitude coordinates to U.S. locations with similar climates, which we do to help avoid introducing errors due extrapolation. We believe further study on the geospatial aspects of generalizable building STLF promising use-case of our benchmark.
K.4
What Layers Get Fine-tuned?
For main transfer learning benchmark results, fine-tune all layers of the transformers. Here, compare the performance of these models with fine-tuning only the last layer. Table that when fine-tuning only the last linear layer (that output logits Gaussian parameters), the performance suffers, which can be best seen by comparing the respective probabilities of improvement. Our intuition is that since the pretraining data is synthetic, fine-tuning on real data may require updating all layers. We believe deeper investigation into transfer learning paradigms for time series is an important topic of study.
L Author Responsibility Statement
The authors confirm that they bear all responsibility in case of any violation of rights during the collection of the data or other work, and will take appropriate action when needed, e.g. to remove data with such issues. The authors also confirm the licenses provided with the data and code associated with this work.
28

---

M Additional Plots
We provide additional qualitative results here (Fig. T0 and Fig.
Thanftrmet (Quuodar)
Gecund
Electricity MT. 012
Geound Inen
Thanstormer (Gauedter)
Bectricity MT. 013
55
Ground truts
Electricity MT 014
Thanaformer (Gaunlann
Grannd truth
Electriiit 0T.015
Trrnffomtt (Gaussiant
300
120
60
1000 (1000)
25.
Icad (4M0)
300
load (Win)
to
(WWh)
80
200
40
60
150
40
25
20
20
19
18
Hour day
Hour day
Hour day
Hour day
HomeC
Smart HorneF
Smart HomeB
Smart HomeeG
1.8
Ground buth
(Guussian)
Transformr (Gaussan)
4.0
Sruundruum
1.6
Fanaformr 1saussian1
3.5
Tansformer (ooussiar)
1.4
losa (MNN))
3.0
2.5
lost
2.0
1.5
0.6
1.0
0.4
0.5
Hour
Hour
Hour day
350
400
Bdg -2:0x Fax, education Lilly
-Grundd
"hondfomer (scan))))
17.5
2:fox education, Virgin
Greand brtb
Tandfmmar Countian)
3000
2 Fox education, Willss
Ground tuth
Parsforer (aasoton
160
140
2:0g22fox
Ground Erssh
The Tranffomer (Gasadaa)
Frances
1080 (30h)
300
lood (NAP)
load IWWBs
120
25.0
25.00
100
200
10.0
2000
80
150
7.5
1500
100
2.5
Hour day
HouF day
Hbur of day
Hour day
1.75
1.50
Grunn buth
Vonstmme (ooussard
MAC000071
1.2
1.0
Groundbruth
Temioomer (Gaurcbnn
MAC001650
4.5
3.5
4.0
Tarstomerr (Gruecan
MAC001739
1.2
1.4
-Gound
Thanttomer (Cauoda))
MAC002141
1.25
Boad (2W0))
0.8
3.0
1.00
0.6
2.5
0.8
32.0
0.75
0.50
1.5
0.4
1.0
0:25
0.5
12
Hour day
Hour of day
day
Hour day
Figure 10: Qualitative results for Transformer-L (Gaussian). Ground truth time series are truncated to previous 24 hours for visibility. Light blue lines are 10 samples from the predicted distribution.
29

---

Electricity MT. 012
Bectricity MT 013
Electicctt MT 014
Electricitt NT. 015
100
Turrsoomet
100
80
120
1000 (dWn)
200
lead (KW0))
(OWN))
100
load (cWh)
150
40
40
40
50
20
T2
18
Smart HomeC
Smart Horne
Smart HomeB
Smart HomeG
.Ground truth
Ground tuth
Ground buis
*aansfomer Thieess
"Tansfomer Thkenss
-Goundd
Tarsfamo (bkers)
Tanslormr
106d (00h)
MWW DOO
lced BeWh:
2.5
1.0
losn
0.8
0.6
Hour day
Hour day
Hour of day
HrOuT day
35.0
8dg-2.fox Fox education Lily
22.5
20.0
2:fox Fox education Virgin
Griand buth
3500
3000
Bdg 2:fox Fox education Wills
160
140
0dg.2:fox Fox, looging Frances
Ground
Ieed 00000)
300
1000 E3Um!
load (KWN
120
250
12.5
2500
100
200
1.0.0
150
60
100
2.5
5.0
40
12
18 52
58
06
12 15
day
day
day
MAC000071
MAC001650
MACD01739
Ld MAC002141
1.75
Ground trth
Vonstmer Vivooe
1.2
Gound tuth
Pansitmme Thbken)
4.0
Groundtruth
Trandlomer (hoond
1.2
Taneeoomer Chooret
mrom
1.50
3.5
35 (1000)
1.25
Soad (WWW))
load (AWW))
2.5
0.8
1.00
0.775
0.6
1.5
1.0
0.25
day
day
Figure 11: Qualitative results for Transformer-L (Tokens). Ground truth time series are truncated to previous 24 hours for visibility. Light blue lines are 10 samples from the predicted distribution.
30

---

Commercial buildings NRMSE (%)
1.00
Persistence
% buildings with NRMSE
T-Tokens-L
T-Tokens-M
0.75
T-Tokens-S
T-Gaussian-L
T-Gaussian-M
0.50
T-Gaussian-S
0.25
0.00
0
20
40
60
80
100
(a)
Commercial buildings NMAE (%)
1.00
Persistence
T-Tokens-L
% buildings with NMAE
T-Tokens-M
0.75
T-Tokens-S
T-Gaussian-L
T-Gaussian-M
0.50
T-Gaussian-S
0.25
0.00
0
20
40
60
80
100
Figure 12: Real commercial building zero-shot NRMSE and NMAE performance profiles with 95%
bootstrap Cls. Curves closer to the bottom left are better.
31

---

Commercial buildings NMBE (%)
J 1.00
A
Persistence
T-Tokens-L
% buildings with NMBE
T-Tokens-M
0.75
T-Tokens-S
T-Gaussian-L
T-Gaussian-M
0.50
T-Gaussian-S
0.25
0.00
0
20
40
60
80
100
T
(a)
Commercial buildings RPS
1.00
Persistence
% buildings with RPS 7
T-Tokens-L
T-Tokens-M
0.75
T-Tokens-S
T-Gaussian-L
T-Gaussian-M
0.50
T-Gaussian-S
0.25
0.00
0
5
10
15
20
25
(b)
Figure 13: Real commercial building zero-shot NMBE and RPS performance profiles with 95%
bootstrap Cls. Curves closer to the bottom left are better.
32

---

Residential buildings NRMSE (%)
1.00
% buildings with NRMSE
0.75
Persistence
0.50
T-Tokens-L
T-Tokens-M
T-Tokens-S
0.25
T-Gaussian-L
T-Gaussian-M
T-Gaussian-S
0.00
O
20
40
60
80
100
T
(a)
Residential buildings NMAE (%)
A 1.00
% buildings with NMAE
0.75
0.50
Persistence
T-Tokens-L
T-Tokens-M
0.25
T-Tokens-S
T-Gaussian-L
T-Gaussian-M
T-Gaussian-S
0.00
0
20
40
60
80
100
Figure 14: Real residential building zero-shot NRMSE and NMAE performance profiles with 95%
bootstrap Cls. Curves closer to the bottom left are better.
33

---

Residential buildings NMBE (%)
A 1.00
Persistence
T-Tokens-L
% buildings with NMBE
T-Tokens-M
0.75
T-Tokens-S
T-Gaussian-L
T-Gaussian-M
0.50
T-Gaussian-S
0.25
0.00
0
20
40
60
80
100
T
(a)
Residential buildings RPS
1.00
Persistence
% buildings with RPS 7
T-Tokens-L
T-Tokens-M
0.75
T-Tokens-S
T-Gaussian-L
T-Gaussian-M
0.50
T-Gaussian-S
0.25
0.00
0.0
0.2
0.4
0.6
0.8
1.0
(b)
Figure 15: Real residential building zero-shot NMBE and RPS performance profiles with 95%
bootstrap Cls. Curves closer to the bottom left are better.
34

---

Sceaux
Borealis
Ideal
NRMGE
1.2
NRMBE
0.8
0.8
1.0
0.6
0.8
0.6
0.4
0.4
0.4
0.2
0.2
0.2
0.0
0.0
Previous. Day
Previous WNewk
Average Maoue
Thomer (Tokarm)
Thimer
Prevous Day
Prefious
Average
Thomee
Thmmer
Previous Day
Previous
Awerage Hour
Themer (Tokern))
Thrmer
(Tohama))
(Gasreant)
(Gasses
(a)
(b)
(c)
Lcl
Smart
1.0
NAMSE NMAT NMBE
1.2
NRMBE NVAE NUBSE
Electricity
NMAZ NM12
0.8
0.125
0.8
0.6
0.075
0.4
0.4
0.050
0.2
0.025
0.0
0.000
0.0
-0.025
Aucrage Weet
(Tobens)
(Con.asben)
Prexiux
Prectsux
Acerage
Thomae
Thrrerr
Prevkw
Prevkan
Averogo
Tyrre
Thome
(d)
Bdg-2
0.25
NRASS
MMAE
0.20
0.15
0.10
0.05
0.00
Previoua Day
Proy ou2
Average
Thrmar
Tlamar (Gounaber)
Figure 16: Per-dataset median zero-shot accuracy (NRMSE, NMAE, NMBE) with 95% bootstrap Cls. Lower is better.
35


Reviews:
[{"review_id": "fVLzGKQXN3", "reviewer": "Reviewer_aV5A", "summary": "", "questions": "", "limitations": "Yes, the authors have proposed solutions for the mentioned limitations.", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "1. The paper is well-organized and easy to follow.\n2. The proposed dataset comprises nearly 1M time series, which can greatly help the research in building large/foundation models for time series data, or for STLF tasks.\n3. The code is released and formulated well. People interested in STLF can utilize this tool to have a good start.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "jkOEvlb6BR", "reviewer": "Reviewer_YiaS", "summary": "", "questions": "", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "BuildingsBench provides a large-scale dataset of 900K buildings and an evaluation platform with over 1,900 real residential and commercial buildings from 7 open datasets, which can help researchers explore the pretrain-then-finetune paradigm for short-term load forecasting (STLF).\n\nThe authors show that synthetically pretrained models generalize surprisingly well to real commercial buildings, and that fine-tuning pretrained models on real commercial and residential buildings improves performance for a majority of target buildings.\n\nThe authors explore two under-explored tasks in STLF: zero-shot STLF, where a pretrained model is evaluated on unseen buildings without fine-tuning, and transfer learning, where a pretrained model is fine-tuned on a target building. This can help researchers better understand the generalizability of STLF models.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "2K8vDj8tZh", "reviewer": "Reviewer_ZMcD", "summary": "", "questions": "", "limitations": "It would be interesting to have some discussion on how transferable this dataset (and the benchmark platform) is to other countries. The authors discuss to some extent about the possible differences when fine tuning the model (pretrained with US data) with data from other countries, but not the other way around. For example, BuildingsBench comprises data from different geographic and cultural regions (which can drive residential consumption behavior), which perhaps do not translate directly to consumption patterns in the US (which buildings-900k only has). These cultural differences might be accentuated in the Low Carbon London dataset, where individuals in select London households were motivated to reduce their consumption (e.g., use less their appliances or prioritize high energy efficiency appliances; see the project summary report: https://innovation.ukpowernetworks.co.uk/projects/low-carbon-london/). Perhaps differences as these could explain the lower accuracy obtained in the residential buildings benchmarks.\n\nI see no potential negative societal impact.", "rating": 8, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "This paper addresses the need for a large dataset to develop STLF models (pretraining), with an emphasis on forecasting energy consumption on buildings unseen by the model, for which no fine tuning has been implemented (zero-shot). In addition to providing such dataset, this submission includes a detailed framework to benchmark the aforementioned zero-shot models. The authors present a solid and sound benchmark design, which utilizes a secondary dataset they have put together. This secondary dataset (BuildingsBench)—another novel contribution in this submission—provides real buildings data collected from 7 separate sources, which is also used to fine tune the models trained on simulated data.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "TyQAK6txKj", "reviewer": "Reviewer_cRNE", "summary": "", "questions": "", "limitations": "The authors have included a paragraph on the limitations of this benchmark. In addition, please also see my concerns and questions below.\n\n1. Page 2, Figure 1, the data from commercial buildings present clear periodical patterns, but the residential data look more \"unpredictable\". For the second and third rows, I saw many spikes in the time series data. How do the authors tell if these spikes are normal conditions or \"outliers\" in real-world residential data?\n\n2. Also, on Page 5, in the paragraph \"processing and storage\", the authors mention that they fill with zeros for the missing scenarios over one week. Is it possible to use some common matrix or tensor completion methods to approximate the missing values [1,2,3]? \n\nReferences:\n\n[1] Liu, J., Musialski, P., Wonka, P., & Ye, J. (2012). Tensor completion for estimating missing values in visual data. IEEE transactions on pattern analysis and machine intelligence, 35(1), 208-220.\n\n[2] Yu, H. F., Rao, N., & Dhillon, I. S. (2016). Temporal regularized matrix factorization for high-dimensional time series prediction. Advances in neural information processing systems, 29.\n\n[3] Chen, X., He, Z., & Sun, L. (2019). A Bayesian tensor decomposition approach for spatiotemporal traffic data imputation. Transportation research part C: emerging technologies, 98, 73-84.\n\n4. On page 5, for the real data, are there any noisy or outlier scenarios that need to be considered?\n\n5. As shown in Tables 3 and 4 on Page 7, the results show the challenges of the data from real residential buildings. The authors also provide several potential directions in Section 6.2 Residential STLF Challenges. Can the authors add more descriptions of the acceptable error range for this particular scenario? Based on my understanding, the normalized errors exceeding 100% are too large. ", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "1. This paper introduces a large-scale time series dataset, including both simulation data and real residential and commercial data. The major contribution comes from the 900K simulation data for both commercial and residential buildings and a benchmark that includes zero-shot/transfer learning tasks and several state-of-the-art baselines. \n\n2. The authors have provided the data and code links, as well as good documentation. ", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "psIUHO1uSo", "reviewer": "Reviewer_zkdi", "summary": "", "questions": "", "limitations": "1. There are typos in the paper. For example, \"STLF remains a challenging problem as energy demand can fluctuate heavily due to a\n31 variety of unobserved and exogenous factors\" ends with no period.\n2. The significance of this dataset against existing ones are not well explained.\n", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "1. It is quite useful to have a dataset for load forecasting. The dataset is interesting. \n2. The dataset is relatively large.\n3. The paper is well written.\n", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}]

Author rebuttal:

Meta-review: Based on the feedback from the reviewers and the responses from the authors, the paper titled "BuildingsBench" has received overall positive reviews. The paper introduces a substantial dataset for short-term load forecasting and provides valuable contributions to the field. The strengths of the paper include its organization, the size and quality of the dataset, and the clarity of the writing.

However, there were some common areas for improvement highlighted by the reviewers. These include addressing minor typos in the paper, providing a more comprehensive explanation of the dataset's significance compared to existing ones, and improving the clarity of the paper's relation to prior work.

The authors have responded to the reviewers' comments diligently and made appropriate revisions to the paper. They have corrected typos, clarified the dataset's significance, and improved the paper's relation to prior work. Additionally, they have made substantial efforts to address more specific concerns raised by reviewers, such as data preprocessing, evaluation metrics, baselines, data spikes, and missing values. The authors' responses have generally been thorough and effective.

Overall, the paper has been significantly improved based on the reviewers' feedback, and the authors have adequately addressed the concerns raised.
Decision: Accept (Poster)