SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Q: Was the "raw" data saved in addition to the preprocessed/cleaned/labeled data (e.g., to support unanticipated future uses)?
The raw data (the EULP) is publicly accessible for download https://data.openei.org/s3 viewer?bucket=oedi-data-lake&prefix=nrel-pds-building-stock.
Q: Is the software used to preprocess/clean/label the instances available?
Yes, at https://github.com/NREL/BuildingsBench.
B.5 Uses
Q: Has the dataset been used for any tasks already?
In this work, we evaluate pretrained models on zero-shot STLF and transfer learning for STLF (fine-tuning pretrained models on limited data from a target building).
Q: Is there a repository that links to any or all papers or systems that use the dataset?
We will add links to papers or systems that use the dataset to https://data.openei.org/submissions/5859 and https://nrel.github.io/BuildingsBench.
Q: What (other) tasks could the dataset be used for?
Pretrained models can sometimes serve as general-purpose feature extractors. These features can be used for a wide range of tasks, including:
- Predicting energy consumption of a building based on its characteristics.
- Classifying user behavior (e.g., whether a building is currently occupied) based on their energy consumption.
- Classifying building type based on energy consumption.
- Load disaggregation, that is, predicting the energy consumption of individual appliances based on the total energy consumption of a building.
Among others, potentially. We did not explore these tasks in this paper.
Q: Is there anything about the composition of the dataset or the way it was collected and preprocessed/cleaned/labeled that might impact future uses?
Buildings-900K is (in its current state) representative of buildings in the United States. While we find evidence that models pretrained on Buildings-900K generalize to buildings in other countries in the northwestern hemispheres, such as the UK and Canada, it may not be representative of building energy consumption patterns in other regions of the world.
Q: Are there tasks for which the dataset should not be used?
None that the authors are currently aware of.
B.6 Distribution
Q: Will the dataset be distributed to third parties outside of the entity (e.g., company, institution, organization) on behalf of which the dataset was created?
Q: How will the dataset will be distributed (e.g., tarball on website, API, GitHub)?
It will be publicly available for download on the Open Energy Data Initiative (OEDI) website at https://data.openei.org/submissions/5859.
Q: ill the dataset be distributed under a copyright or other intellectual property (IP) license, and/or under applicable terms of use (ToU)?
CC-4.0.
Q: Have any third parties imposed IP-based or other restrictions on the data associated with the instances?
Q: Do any export controls or other regulatory restrictions apply to the dataset or to individual instances?
B.7 Maintenance
Q: Who is supporting/hosting/maintaining the dataset?
Code is hosted and publicly available on Github. NREL will maintain the code and datasets. The data is hosted on the Open Energy Data Initiative site, which is a centralized repository for energy-related datasets derived from U.S. Department of Energy projects.
Q: How can the owner/curator/manager of the dataset be contacted (e.g., email address)?
The lead maintainer at NREL is Patrick Emami (Patrick.Emami@nrel.gov).
Q: Will the dataset be updated (e.g., to correct labeling errors, add new instances, delete instances)?
Yes. A version naming system will be used to indicate updated versions. We are also maintaining a GitHub repository for the associated benchmark (BuildingsBench) where notifications of updates will be posted.
Q: Will older versions of the dataset continue to be supported/hosted/maintained?
Yes, older versions will remain available on the OEDI website.
Q: If others want to extend/augment/build on/contribute to the dataset, is there a mechanism for them to do so?
Not officially, but our benchmark code is open source and pull requests are welcome.
C BuildingsBench Datasets
C.1 ElectricityLoadDiagrams20112014
This dataset is available from the UCI Machine Learning Repository<sup>3</sup> under a CC-4.0 license. It contains 15-minute consumption time series (in kW) for 370 clients in Portugal from 2011 to 2014. The magnitudes of the loads are much larger than residential homes, thus, we consider these time series as commercial buildings. After filtering for missing data, we kept 359 buildings. We included this dataset in BuildingsBench because of its permissive license, popular use in machine-learning-based time series forecasting studies [49], and size.
C.2 The Building Data Genome Project 2
This dataset is available<sup>4</sup> under an MIT License. It consists of measurements from 3,053 meters from 1,636 commercial buildings over two years (2016 and 2017). One or more meters per building measured the total electrical, heating and cooling water, steam, solar energy, water, and irrigation
<sup>3</sup>https://archive.ics.uci.edu/dataset/321/electricityloaddiagrams20112014
4https://github.com/buds-lab/building-data-genome-project-2/
usage. This dataset was curated by the Building Data Genome Project, a consortium of academics and practitioners. We use the whole building electricity meter measurements from Bear, Fox, Panther and Rat sites, totalling 611 buildings (from the CSV file electricity cleansed.csv). This dataset was included in BuildingsBench because of its permissive license, size, and diversity.
C.3 Low Carbon London
This dataset is available<sup>5</sup> under a CC-4.0 license. It consists of consumption measurements from 5,567 London households that participated in a project led by UK Power Networks between November 2011 and February 2014. The dataset consists of half-hourly consumption (in kWh), a unique household identifier, and timestamps. Due to the large number of households included in the dataset, we randomly subsampled 713 of them (so as to keep the overall number of residential and commercial buildings in the benchmark approximately similar). We included this dataset because of its permissive license and size.
C.4 SMART
This dataset is available online unaccompanied by a specific license<sup>6</sup>. It contains meter measurements of aggregate electricity data for 7 residential homes (which are called Home A,B,C,D,F,G,H) collected by researchers at UMass Amherst. After filtering out 2 homes due to missing data, we were left with Home B (years 2014–2016), Home C (2014–2016), Home D (2016), Home F (2014–2016), and Home G (2016). We included this dataset in BuildingsBench due to its accessibility and because the homes are located in the U.S. (Western Massachusetts), are the simulated homes in Buildings-900K.
C.5 IDEAL
This dataset is available<sup>7</sup> under a CC-4.0 license. It has meter readings for electric and gas usage, as well as room temperature, humidity, and boiler readings from 255 UK homes. We use the total home electricity measurements. The measurements were collected by researchers at the University of Edinburgh. After our filtering process, we kept 219 homes. This dataset was included in BuildingsBench due to its permissive license and size.
C.6 Individual household electric power consumption (Sceaux)
This dataset is available via Kaggle $^{8}$  under a DbCL-1.0 license. It contains consumption measurements from a home in Sceaux, France between December 2006 and November 2010. It provides 7 types of meter readings at one minute resolution with timestamps. We use the global active power readings. This dataset was included in BuildingsBench due to its permissive license and due to its popularity both in the literature [28] and on Kaggle.
C.7 Borealis
This dataset is available<sup>9</sup> under a CC0-1.0 license. It is 6-second resolution measurements of total real power consumption for 25 households in Waterloo, ON for at least three months between 2011 and 2012. After filtering out homes with excessive missing data, we kept 15 of them. The data was collected by researchers at the University of Waterloo. We included this dataset in BuildingsBench due to its permissive license and size.
C.8 Outlier Removal
Here we provide additional details on outlier removal for noisy meter data. The algorithm we use compares each load value to all others in a sliding window of 24 hours and computes the absolute difference with respect to the 1-nearest neighbor (1-NN). To select an outlier threshold, we compute
<sup>5</sup>https://data.london.gov.uk/dataset/smartmeter-energy-use-data-in-london-households
$^{6}$ https://traces.cs.umass.edu/index.php/smart/smart
<sup>7</sup>https://datashare.ed.ac.uk/handle/10283/3647
<sup>8</sup>https://www.kaggle.com/datasets/uciml/electric-power-consumption-data-set
<https://borealisdata.ca/dataset.xhtml?persistentId=doi:10.5683/SP2/R4SVBF>
the average daily peak load and average daily base load, and compute the difference. If the 1-NN distance is larger than this difference, we classify the spike as an outlier and replace the spike with the median of the sliding window.
D Evaluation Metrics
D.1 Additional Accuracy Metrics: NMAE, NMBE
NMAE: Similar to the NRMSE, the normalized mean absolute error (NMAE) measures the ability of a model to predict the correct load shape. However, it is less sensitive to large errors than the NRMSE. For a building with  $M$  days of load time series,
$$
N M A E := 1 0 0 \times \frac {1}{\hat {y}} \left[ \frac {1}{2 4 M} \sum_ {j = 1, i = 1} ^ {M, 2 4} \left| y _ {i, j} - \hat {y} _ {i, j} \right| \right]. \tag {5}
$$
NMBE: The normalized mean bias error (NMBE) is informative about a model's tendency to over or under-estimate building loads, which is practically useful to measure. However, a model that tends to over-estimate or under-estimate the load by an equal amount can achieve an NMBE close to zero (i.e., positive and negative errors cancel). Therefore, it is not a useful indicator of accuracy. For a building with  $M$  days of load time series,
$$
N M B E := 1 0 0 \times \frac {1}{\hat {y}} \left[ \frac {1}{2 4 M} \sum_ {j = 1, i = 1} ^ {M, 2 4} \left(y _ {i, j} - \hat {y} _ {i, j}\right) \right]. \tag {6}
$$
We visualize performance profiles and dataset-specific results for all metrics in App. H and App. I, respectively.
D.2 Categorical Ranked Probability Score
Here, we define the ranked probability score (RPS) for categorical distributions. This is used to compute the RPS for our time series transformer variant that predicts a categorical distribution over a vocabulary of  $|\mathcal{V}|$  discretized load tokens. Each load token is the centroid of a cluster, where clusters are estimated via KMeans. See App. E for more details about the tokenizer.
In what follows, we assume that the vocabulary of load tokens is sorted in increasing order by value, that is,  $0 \leq v_{0} < v_{1} < \dots < v_{|\mathcal{V}| - 1}, v_{i} \in \mathcal{V}$ .
Let  $p(\hat{Y}_{i,j})$  be the predicted categorical distribution over  $\mathcal{V}$  for hour  $i$  on day  $j$ . We assume this is a normalized distribution, which is accomplished in practice by computing the softmax of the model's predicted logits. The corresponding CDF for  $p(\hat{Y}_{i,j})$  is  $F(\hat{Y}_{i,j})$ , which is given by the cumulative sum of the  $|\mathcal{V}|$  sorted entries of  $p(\hat{Y}_{i,j})$ . Then, let  $F(Y_{i,j})$  be the CDF of the ground truth discretized load. If the ground truth category is  $k \in \{0, \dots, |\mathcal{V}| - 1\}$ , the CDF  $F(Y_{i,j})$  is 0 up to index  $k - 1$  and 1 thereafter (resembling a step function that goes from 0 to 1 at ground truth load index  $k$ ). Finally, define  $\bar{v}[k]$  as the difference between the maximum load value and the minimum load value assigned to load token  $v[k]$  by KMeans (recall, load tokens are centroids of 1D KMeans clusters).
Then, the categorical RPS at hour  $i$  for a building with  $M$  days of load time series is
$$
C a t R P S _ {i} := \frac {1}{M} \sum_ {j = 1} ^ {M} \sum_ {k = 0} ^ {| \mathcal {V} | - 1} \left(F \left(\hat {Y} _ {i, j}\right) [ k ] - F \left(Y _ {i, j}\right) [ k ]\right) ^ {2} \bar {v} [ k ]. \tag {7}
$$
D.3 Gaussian Ranked Probability Score
The continuous ranked probability score for Gaussian distributions can be evaluated in closed form. In particular, let  $\hat{\mu}_{i,j}$  and  $\hat{\sigma}_{i,j}$  be a predicted mean and standard deviation, and  $y_{i,j}$  be the ground truth value. Define  $z_{i,j} = \frac{y_{i,j} - \hat{\mu}_{i,j}}{\hat{\sigma}_{i,j}}$ . Then, the Gaussian (C)RPS at hour  $i$  for a building with  $M$  days
of load time series is
$$
G a u s s C R P S _ {i} := \frac {1}{M} \sum_ {j = 1} ^ {M} \hat {\sigma} _ {i, j} \left[ z _ {i, j} \left(2 \underbrace {\left(\frac {1 + \operatorname {e r f} (z _ {i , j})}{2 \sqrt {2}}\right)} _ {\text {G a u s s i a n C D F}} - 1\right) + 2 \underbrace {\left(\frac {\exp \frac {- z _ {i , j} ^ {2}}{2}}{\sqrt {2 \pi}}\right)} _ {\text {G a u s s i a n P D F}} - \frac {1}{\sqrt {\pi}} \right]. \tag {8}
$$
Gaussian approximation of the inverse Box-Cox transform: Our time series transformer variant that predicts a Gaussian distribution for each hour  $i$  in the day-ahead forecast uses a Box-Cox power transformation to normalize the load values. In App. K, we show that standard scaling was insufficient to remove training instabilities caused by the non-Gaussian load time series. Since the Gaussian distribution is learned in the Box-Cox transformed space, we cannot directly use the GaussCRPS metric to evaluate uncertainty quantification in the original un-scaled space (i.e., with units of kWh). Our current solution, introduced here, is to compute a Gaussian distribution in the original un-scaled space that closely approximates the distribution given by the inverse Box-Cox transformation (for a reasonable range of standard deviations). We leave exploring advanced non-approximate solutions for future work.
First, we define the Box-Cox transformation of a random variable  $X$
$$
Y = f _ {\lambda} (X) := \left\{ \begin{array}{l l} \frac {X ^ {\lambda} - 1}{\lambda} & \text {i f} \lambda \neq 0 \\ \ln (X) & \text {i f} \lambda = 0, \end{array} \right. \tag {9}
$$
and its inverse
$$
X = f _ {\lambda} ^ {- 1} (Y) := \left\{ \begin{array}{l l} (Y \lambda + 1) ^ {\frac {1}{\lambda}} & \text {i f} \lambda \neq 0 \\ \exp (Y) & \text {i f} \lambda = 0. \end{array} \right. \tag {10}
$$
We estimate a Gaussian distribution in the un-scaled space that approximates the (power-normal) distribution of the samples inverted with  $f_{\lambda}^{-1}$ . Recall that  $\hat{\mu}_{i,j}$  and  $\hat{\sigma}_{i,j}$  are the predicted mean and standard deviation in the Box-Cox transformed space. We take the inverse of the predicted mean to be  $f_{\lambda}^{-1}(\hat{\mu}_{i,j})$  (Eq. 10). Then, the standard deviation is estimated with:
$$
\hat {\sigma} _ {i, j} ^ {+} = f _ {\lambda} ^ {- 1} \left(\hat {\mu} _ {i, j} + \hat {\sigma} _ {i, j}\right) - f _ {\lambda} ^ {- 1} \left(\hat {\mu} _ {i, j}\right) \tag {11}
$$
$$
\hat {\sigma} _ {i, j} ^ {-} = f _ {\lambda} ^ {- 1} \left(\hat {\mu} _ {i, j}\right) - f _ {\lambda} ^ {- 1} \left(\hat {\mu} _ {i, j} - \hat {\sigma} _ {i, j}\right) \tag {12}
$$
$$
\tilde {\sigma} _ {i, j} \approx \frac {\hat {\sigma} _ {i , j} ^ {+} + \hat {\sigma} _ {i , j} ^ {-}}{2}. \tag {13}
$$
The Gaussian distribution in the un-scaled space is thus  $\mathcal{N}(f_{\lambda}^{-1}(\hat{\mu}_{i,j}),\tilde{\sigma}_{i,j}^{2})$ . We find that this Gaussian is a good approximation for the distribution of the samples obtained by applying Eq. 10 to samples drawn from  $\mathcal{N}(\hat{\mu}_{i,j},\hat{\sigma}_{i,j}^{2})$  for reasonably small values of  $\tilde{\sigma}_{i,j}$ . We compare these two distributions in Fig. 5. When the model is uncertain about the forecast and the standard deviation in the un-scaled space  $\tilde{\sigma}_{i,j}$  is large, the true distribution in the un-scaled space is heavily skewed to the right, making a Gaussian a poor approximation (Fig. 5c).
E Time Series Transformer Details
Table 5: Transformer architecture hyperparameters.
Predicted Distribution	# Layers	# Heads	Embedding Dim	MLP Dim	# Params
Transformer-S	Categorical	2	4	256	512	3.3M
Transformer-M	Categorical	3	8	512	1024	17.2M
Transformer-L	Categorical	12	12	768	2048	160.7M
Transformer-S	Gaussian	2	4	256	512	2.6M
Transformer-M	Gaussian	3	8	512	1024	15.8M
Transformer-L	Gaussian	12	12	768	2048	160.7M
In this section, we provide additional details about the time series transformer architecture used in this work. We use the original encoder-decoder model from Vaswani et al. [43] as implemented in the
(b)
Figure 5: Gaussian approximation of the inverse Box-Cox. Visualization of the Gaussian distribution  $\mathcal{N}(f^{-1}(\hat{\mu}_{i,j}),\tilde{\sigma}_{i,j})$  (orange) in the un-scaled space of loads (in kWh) for computing the RPS when using Box-Cox scaling. For reasonably small standard deviations  $\tilde{\sigma}$  in the un-scaled space, this Gaussian is a reasonable approximation of the power-normal distribution given by the inverse Box-Cox (blue). c) When the model is highly uncertain so that  $\tilde{\sigma}$  is large, the power-normal is extremely right-skewed. In this case, the Gaussian approximation is inaccurate.
PyTorch nn. Transformer module. This autoregressive model is trained to predict the load at each time step with teacher forcing. Following Wu et al. [47], we use masking in the decoder to prevent the attention from peeking at the target value at each time step. The first value in the input sequence passed to the decoder is the last value of the input sequence passed to the encoder. The Gaussian time series transformer uses a linear layer (with parameters shared across time steps) to transform the high-dimensional output of the decoder into a mean and standard deviation at each time step. The time series transformer trained on tokenized load values uses a linear layer to transform the decoder outputs into  $|\mathcal{V}|$  logits for the cross-entropy loss, also shared across time steps. Our tokenizer is based on KMeans (a stochastic algorithm) followed by a token merging step. Depending on the random seed, multiple runs of the tokenizer will produce token vocabularies of different sizes. The tokenizer is described in detail in App. E.1.
The architecture hyperparameters for the L, M, and S models are shown in Table 5. The largest model has 12 layers, 12 heads, and an internal dimension of 768 selected to match Radford et al. [35], but uses a smaller feedforward dimension of 2048 to keep the number of parameters at approximately  $10^{8}$ . We downsize these architecture hyperparameters to reduce the number of parameters by roughly an order of magnitude for the S and M models. Similar to Radford et al. [35], all models use GELU [16] activations and  $\mathcal{N}(0,0.02)$  weight initialization.
Model input encoding: Let  $E$  be the embedding dimension and  $s$  be a scaling factor given by  $\mathrm{E} / 256$  (e.g.,  $s = 3$  for Transformer-L). The model inputs are encoded, concatenated, and passed to the transformer encoder as follows:
- day of year, day of week, hour of day: These are encoded into a 2-dim space with  $\sin (\pi x)$  and  $\cos (\pi x)$  then projected with a linear layer to  $32s$ -dim.
- latitude, longitude: These scalars are each projected to  $32s$ -dim space with linear layers.
- building type: A linear embedding layer is used to learn a projection for each of residential and commercial building types to  $32s$ -dim space.
- continuous loads: When the load is a continuous scalar, it is projected to  $64s$ -dim space with a linear layer.
- discrete load tokens: When the load is a discrete token, a linear embedding layer is used to learn a projection for each of the  $|\mathcal{V}|$  tokens to 64s-dim space.
These various embeddings are concatenated into a feature vector of size  $E$  at each time step.
E.1 Load Tokenizer
We explore quantizing load values into a vocabulary of discrete tokens to closely mimic the application of transformers to natural language. The design of the tokenizer is detailed here. We use simple KMeans clustering with a basic merging strategy inspired by Byte Pair Encoding [11] which merges
Algorithm 1 Load tokenizer
// Inputs: loads ((n,1) array),  $K$  initial centroids,  $\tau = 0.01$  threshold (kWh)  
kmeans = faiss.Kmeans(K)  
kmeans.train(loads)  
// Merge step. Initialize state.  
sorted_centroids = sort(kmeans.centroids)  
current_centroid = sorted_centroids[0]  
merged_centroids = []  
temp_centroids = [current_centroid]  
// Iterate over sorted centroids in increasing order  
for  $i = 1$  to  $K$  do  
// The next centroid is less than  $\tau$  away, merge it  
if sorted_centroids[i] - current_centroid <  $\tau$  then  
temp_centroids.append(sorted_centroids[i])  
else  
merged_centroids.append(mean(temp_centroids))  
temp_centroids = [sorted_centroids[i]]  
current_centroid = sorted_centroids[i]  
// Add the last centroid to the new list of merged centroids  
merged_centroids.append(mean(temp_centroids))  
return kmeans, merged_centroids
(a)  $|\mathcal{V}| = 3,747$ $(K = 8,192)$
Figure 6: Load token frequency. We visualize histograms of token frequencies from a subsample of Buildings-900K. a) After merging  $K = 8,192$  into 3,747, there are some tokens that appear often near load values of zero while most other tokens have nearly equal usage. The mean absolute quantization error for commercial buildings is approximately  $0.2\mathrm{kWh}$  whereas for residential buildings it is  $0.003\mathrm{kWh}$ . b) Without merging, most tokens correspond to small load values and all appear to have roughly equal usage. c) When  $|\mathcal{V}|$  is small (474), there are few tokens for large load values.
(b)  $|\mathcal{V}| = 8,192$  (No merging)
(c)  $|\mathcal{V}| = 474$  ( $K = 512$ )
clusters that are overly close to each other. We leave exploring other simple and sophisticated tokenization strategies for future work.
The tokenizer uses KMeans clustering on a random subsample of load values from the Buildings-900K training set to fit  $K$  clusters (the size of the subsample is chosen internally by faiss-gpu). The merge step is as follows. First, we sort the  $K$  cluster centroids in increasing order and then iterate over them. For any cluster centroid  $i$ , if there are successive centroids that are less than  $\tau = 0.01$  kWh greater than  $i$ , we replace all of these centroids with a new centroid equal to their average. We provide pseudocode in Alg. 1. This algorithm is stochastic, such that running the tokenizer multiple times will produce vocabularies with different (but similar) sizes  $|\mathcal{V}|$ . To tokenize a continuous load value, we first look up the index of the original centroid it belongs to, then use this index to look up the corresponding index in the list of merged centroids. See Fig. 6 for a visual comparison of three different vocabularies produced by the tokenizer, and App. K.1 for ablation study results on the choice of  $K$  and the merging step.
F Training Details
In this section, we describe how training hyperparameters were selected for the transformer models. We use the loss on the Buildings-900K validation dataset for model selection. However, we observed
that a better Buildings-900K validation loss does not necessarily indicate better downstream performance on the real (out-of-distribution) evaluation data. We encourage exploring sophisticated model selection strategies that account for distribution shifts in future work.