Package: buildings-bench
Welcome to BuildingsBench!OverviewBuildingsBench is a platform forLarge-scale pretraining with the synthetic Buildings-900K dataset for short-term load forecasting (STLF). Buildings-900K is statistically representative of the entire U.S. building stock and is extracted from the NRELEnd-Use Load Profiles database.Benchmarking on two tasks evaluating generalization: zero-shot STLF and transfer learning for STLF.We provide an index-based PyTorch Dataset for large-scale pretraining, easy data loading for multiple real building energy consumption datasets as PyTorch Tensors or Pandas DataFrames, simple (persistence) to advanced (transformer) baselines, metrics management, and more.Read more about BuildingsBench in ourpaperanddocumentation.InstallationIf you aren't going to pretrain or evaluate models and just want access to the provided dataloaders, model code, metrics computation, etc., install the package with:pipinstallbuildings_benchFull installationOtherwise, clone this repository and install it in editable mode in a virtual environment or a conda environment.Create an environment withpython>=3.8, for example:conda create -n buildings_bench python=3.8.Install the package in editable mode withgitclonehttps://github.com/NREL/BuildingsBench.gitcdBuildingsBench
pipinstall-e".[benchmark]"Installing faiss-gpuDue to a PyPI limitation, we have to installfaiss-gpu(for KMeans) by directly downloading the wheel fromhttps://github.com/kyamagu/faiss-wheels/releases/.
Download the wheel for the python version you are using, then install it in your environment.For example:wgethttps://github.com/kyamagu/faiss-wheels/releases/download/v1.7.3/faiss_gpu-1.7.3-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl

pipinstallfaiss_gpu-1.7.3-cp38-cp38-manylinux2014_x86_64.whl[Optional] Installing LightGBMIf running the LightGBM baseline, you will need to install LightGBM.Follow instructionshereto installlightgbmfor your OS.Then installskforecastwithpip install skforecast==0.8.1.Download the datasets and metadataThe pretraining dataset and evaluation data is available for downloadhereas tar files, or can be accessed via AWS S3here. The benchmark datasets are < 1GB in size in total, but the pretraining data is ~110GB in size.The pretraining data is divided into 4 compressed filescomstock_amy2018.tar.gz: ~21GBcomstock_tmy3.tar.gz: ~21GBresstock_amy2018.tar.gz: ~33GBresstock_tmy3.tar.gz: ~33GBand one compressed file for the metadatametadata.tar.gzThe evaluation datasets are compressed into a single fileBuildingsBench.tar.gzDownload all files to a folder on a storage device with at least 250GB of free space. Then, decompress all of the downloaded files. There will be a new subdirectory calledBuildingsBench.This is the data directory, which is different than the Github code repository, although both folders are named "BuildingsBench".See the README fileBuildingsBench/metadata/README.md(inmetadata.tar.gz) for more information about how the BuildingsBench dataset directory is organized.Setting environment variablesSet the environment variableBUILDINGS_BENCHto the path where the data directoryBuildingsBenchis located (created when untarring the data files).This is not the path to this code repository.exportBUILDINGS_BENCH=/path/to/BuildingsBenchWandbIf usingwandb, set the following:WANDB_ENTITY: your wandb usernameWANDB_PROJECT: the name of your wandb project for this benchmarkRun testsVerify your local installation by running unit tests from the base directory:python-munittestUsageLoading a benchmark datasetThebuildings_benchpackage provides PyTorch Dataloaders for the pretraining and evaluation time series data as well as pandas integration to enable working with Dataframes.importtorchfrombuildings_benchimportload_torch_dataset,load_pandas_dataset,load_pretraining# Load a dataset generator to evaluate a PyTorch modelbuildings_dataset_generator=load_torch_dataset('bdg-2:panther')# Each building is a torch.utils.data.Datasetforbuilding_name,buildinginbuildings_dataset_generator:building_dataloader=torch.utils.data.DataLoader(building,batch_size=358,num_workers=4,shuffle=False)forsampleinbuilding_dataloader:x=sample['load']# context = x[:, :168], 1 week hourly of context# target = x[:, -24:], 24 hour target prediction# ...Running the benchmarkWe provide scripts in the./scriptsdirectory for pretraining and to run the benchmark tasks (zero-shot STLF and transfer learning), either withour provided baselinesor your own model. To use these scripts with your model you'll need to register your model with our platform.See thisstep-by-step tutorialfor more details.Make sure to have installed the benchmark in editable mode:pip install -e .[benchmark]Our benchmark assumes each model takes as input a dictionary of torch tensors with the following keys:{'load':torch.Tensor,# (batch_size, seq_len, 1)'building_type':torch.LongTensor,# (batch_size, seq_len, 1)'day_of_year':torch.FloatTensor,# (batch_size, seq_len, 1)'hour_of_day':torch.FloatTensor,# (batch_size, seq_len, 1)'day_of_week':torch.FloatTensor,# (batch_size, seq_len, 1)'latitude':torch.FloatTensor,# (batch_size, seq_len, 1)'longitude':torch.FloatTensor,# (batch_size, seq_len, 1)}Create a file calledyour_model.pywith your model's implementation, and make your model a subclass of the base model in./buildings_bench/models/base_model.py. Make sure to implement the abstract methods:forward,loss,load_from_checkpoint,predict,unfreeze_and_get_parameters_for_finetuning.Place this file under./buildings_bench/models/your_model.py.Import your model class and add your model's name to themodel_registrydictionary in./buildings_bench/models/__init__.py.Create a TOML config file under./buildings_bench/configs/your_model.tomlwith each keyword argument your model expects in its constructor (i.e., the hyperparameters for your model) and any additional args for the script you want to run.The TOML config file should look something like this:[model]# your model's keyword arguments[pretrain]# override any of the default pretraining argparse args here[zero_shot]# override any of the default zero_shot argparse args here[transfer_learning]# override any of the default transfer_learning argparse args hereSee./buildings_bench/configs/TransformerWithTokenizer-S.tomlfor an example.PretrainingWithout SLURMThe scriptpretrain.pyis implemented with PyTorchDistributedDataParallelso it must be launched withtorchrunfrom the command line and the argument--disable_slurmmust be passed.
See./scripts/pretrain.shfor an example.#!/bin/bashexportWORLD_SIZE=1NUM_GPUS=1torchrun\--nnodes=1\--nproc_per_node=$NUM_GPUS\--rdzv-backend=c10d\--rdzv-endpoint=localhost:0\scripts/pretrain.py--modelTransformerWithGaussian-S--disable_slurmThe argument--disable_slurmis not needed if you are running this script on a Slurm cluster as a batch job.This script will automatically log outputs towandbif the environment variablesWANDB_ENTITYandWANDB_PROJECTare set. Otherwise, pass the argument--disable_wandbto disable logging towandb.With SLURMTo launch pretraining as a SLURM batch job:exportWORLD_SIZE=$(($SLURM_NNODES*$SLURM_NTASKS_PER_NODE))echo"WORLD_SIZE="$WORLD_SIZEexportMASTER_PORT=$(expr10000+$(echo-n$SLURM_JOBID|tail-c4))echo"NODELIST="${SLURM_NODELIST}master_addr=$(scontrolshowhostnames"$SLURM_JOB_NODELIST"|head-n1)exportMASTER_ADDR=$master_addrecho"MASTER_ADDR="$MASTER_ADDRsrunpython3scripts/pretrain.py\--modelTransformerWithGaussian-SZero-shot STLFThis scriptscripts/zero_shot.pyand the script for transfer learningscripts/transfer_learning_torch.pydo not useDistributedDataParallelso they can be run withouttorchrun.python3 scripts/zero_shot.py --model TransformerWithGaussian-S --checkpoint /path/to/checkpoint.ptTransfer Learning for STLFWith pretrained models:python3 scripts/transfer_learning_torch.py --model TransformerWithGaussian-S --checkpoint /path/to/checkpoint.ptThe LightGBM baseline:python3 scripts/transfer_learning_lightgbm.pyBuildingsBench LeaderboardMetrics:NRMSE: Normalized Root Mean Squared Error (%)RPS: Ranked Probability ScoreGaussian Continuous Ranked Probability Score for Gaussian modelsCategorical Discrete Ranked Probability Score for token-based modelsPyTorch checkpoint files for all trained models can be downloaded in a single tar filehere.
We provide a tutorial on how to use these pretrained modelshere.Zero-shot STLFEval over all real buildings for all available years. Lower is better. Individual model checkpoints are available via AWS S3 for download by clicking on the model name.ModelCommercial NRMSE (%)Commercial RPSResidential NRMSE (%)Residential RPSTransformer-L (Gaussian)13.315.2379.340.072Transformer-M (Gaussian)13.285.2192.600.098Transformer-S (Gaussian)13.975.66102.300.118Transformer-L (Tokens)14.465.6295.340.152Transformer-M (Tokens)14.055.46100.490.203Transformer-S (Tokens)14.565.49101.180.085Persistence Ensemble16.685.8877.880.063Previous Day Persistence16.96-98.41-Previous Week Persistence19.39-99.77-Transfer Learning for STLFResults are over a sub-sample of 100 residential and 100 commercial buildings--see the list of buildings in the datasets metadata directory:BuildingsBench/metadata/transfer_learning_residential_buildings.csvandBuildingsBench/metadata/transfer_learning_commercial_buildings.csv.
Models are provided with the first 6 months of consumption data for fine-tuning and tested with a 24-hour sliding window on the next 6 months.
These results are for the Transformer-L models.ModelCommercial NRMSE (%)Commercial RPSResidential NRMSE (%)Residential RPSPretrained + Fine-tunedTransformer (Gaussian)12.964.3777.200.057Transformer (Tokens)14.074.9994.530.137Fine-tuned from random weightsTransformer (Gaussian)37.2115.9492.990.081Transformer (Tokens)50.1226.89105.6516.36LightGBM16.02-80.07-DLinear23.41-87.89-Linear Regression25.18-89.98-RNN41.7915.2896.750.078PersistencePersistence Ensemble16.805.9778.540.057Previous Day Persistence16.54-98.35-Previous Week Persistence18.93-100.20-CitationIf you use BuildingsBench in your research, please cite our preprint:@article{emami2023buildingsbench,
  title={BuildingsBench: A Large-Scale Dataset of 900K Buildings and Benchmark for Short-Term Load Forecasting},
  author={Emami, Patrick and Sahu, Abhijeet and Graf, Peter},
  journal={arXiv preprint arXiv:2307.00142},
  year={2023}
}