SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Pretraining hyperparameters: The learning rate for all models uses a linear warmup of 10K steps followed by cosine decay to zero. Models are trained on 1 billion load hours, as we observed in early runs that the validation loss would no longer improve after this point. We perform a grid search over the max learning rate  $\{6\mathrm{e - }4,6\mathrm{e - }5,6\mathrm{e - }6\}$  and the batch size  $\{64,128,256\}$  for the two Transformer-S models. For the Transformer-S (Gaussian) model, the highest learning rate and smallest batch size had the best validation loss. However, when applying these hyperparameters to the larger models, the high learning rate of 6e-4 caused training instability. Thus, we used the lower 6e-5 learning rate, which performed best with the smallest batch size of 64 (650K total gradient updates). Similarly for the Transformer-S (Tokens) model, the highest learning rate achieved the best validation loss, but for the larger models we used the lower 6e-5 learning rate with batch size of 64. We use the AdamW [26] optimizer with  $\beta_{1} = 0.9$ $\beta_{2} = 0.98$ $\epsilon = 1\mathrm{e - }9$  , and weight decay of 0.01 following Radford et al. [35].
Transfer learning hyperparameters: We fine-tune all layers of the transformers (as opposed to only the last logits layer—see App. K.4 for an ablation study on this) with a lower learning rate of 1e-6. We allow a maximum of 25 fine-tuning epochs with an early stopping patience of 2. We tune the learning rate for the Linear, DLinear, and RNN baselines by sweeping over 5 values: {1e-2, 1e-3, 1e-4, 1e-5, 1e-6}. All non-pretrained models are trained for a maximum of 100 epochs.
G Compute Details
- Buildings-900K processing: Creating the Buildings-900K dataset by processing a subset of the EULP was accomplished with a 96-core AWS EC2 instance in 2-3 days.
- Model pretraining: The largest model variant, the Transformer-L, was trained on 1 billion hours from Buildings-900K with one NVIDIA 40GB A100 GPU in 24 hours. Smaller model variants train faster, in about 18 hours. We use PyTorch automatic mixed precision training with gradient scaling, which is crucial for achieving fast training times.
- Zero-shot STLF: The amount of time to run each benchmark task varies depending on model inference speed. This task uses all available years for every building in the benchmark. For the largest model variant, it takes about 2-3 hours on one NVIDIA 40GB A100. By contrast, the persistence baselines run in well under one hour.
- Transfer learning: Repeating the fine-tuning process for every building in the benchmark can be computationally demanding. For example, we sub-sample 200 total buildings (100 residential and 100 commercial) to use for this task, and this takes the Transformer-L models 1.5-3.5 hours to complete on one NVIDIA 40GB A100. We estimate it would take over 15 hours to include all buildings in this task. Models that are fine-tuned from frozen pretrained weights finish this task faster, as we stop the fine-tuning process with early stopping using a patience of 2 epochs.
H Performance Profiles
Performance profiles plot the fraction of all buildings with performance greater than a threshold  $\tau$  for a range of threshold values. These can be used to understand the tails of the error distributions across all buildings in the benchmark. For example, these plots show the fraction of buildings with extremely high forecasting errors.
Here we show performance profiles for commercial (Fig. 12, Fig. 13) and residential (Fig. 14, Fig. 15) BuildingsBench buildings for the NRMSE, NMAE, NMBE, and RPS metrics. For reference, we also plot the performance profile of the Persistence Ensemble baseline. The methods that estimate the forecast with a Gaussian distribution (the Persistence Ensemble and Transformer (Gaussian)) achieve noticeably better NMBE performance than Transformer (Tokens).
I Benchmark Dataset Results
Per-dataset zero-shot accuracy (NRMSE, NMAE, NMBE) are visualized in Fig. 16.
Figure 7: Empirical scaling laws for zero-shot generalization on residential buildings. Intervals are  $95\%$  stratified bootstrap CIs of the median. a-b) Dataset size vs. zero-shot performance. c-d) Model size vs. zero-shot performance.
(a)
Figure 8: Model size vs. Buildings-900K-Test zero-shot STLF performance. Intervals are  $95\%$  stratified bootstrap CIs of the median. The trends in (a)-(c) suggest superior generalization to the test data is achievable by larger models, which might be unsurprising due to the small distribution shift between the simulated pretraining and simulated test data (relative to the large distribution shift between the simulated pretraining and real test data).
J Additional Empirical Scaling Law Results
The residential building results for the experiments where the Transformer-M models are trained under increasing dataset scale (size and number of unique pretraining buildings) and where model scales are compared are shown in Fig. 7. In most cases, the accuracy (NRMSE) and uncertainty quantification (RPS) are independent of the dataset and model size. Exceptions are the Transformer-M (Gaussian) performance on the smallest pretraining dataset size and the effect of model size on the Transformer (Gaussian), which appears to show a power-law scaling. By contrast, Fig. 3 shows that the smallest Transformer-S (Gaussian) model achieves comparable performance to the larger variants on commercial buildings. A possible simple explanation is that the small model struggles to learn to jointly forecast for both residential and commercial buildings due to limited capacity. A key benefit of the larger model, then, is unlocking the ability to jointly forecast for both types of buildings. As discussed in Sec. 6.2, alternative modeling approaches that consider auxiliary inputs might be more successful for large-scale pretraining of residential building STLF. We believe this is an interesting future research direction. We also examine empirically how performance on the simulated Buildings-900K-Test varies with model scale (Fig. 8), and observe trends that clearly indicate superior generalization is achievable with larger models in this setting.
K Ablation Studies
We explore the following aspects of the transformer models in this section:
- App. K.1) The size of the vocabulary and the impact of the merging step in the load tokenizer.
- App. K.2) Using standard scaling vs. Box-Cox scaling for training the Transformer (Gaussian) model.
App. K.3) Impact of the lat, Ion covariate on generalization.
Table 6: Ablation study results on the zero-shot STLF task. Median accuracy (NRMSE) and ranked probability score (RPS). Results are for Transformer-L (Tokens). We ablate the use of the lat,lon covariate and the size of the token vocabulary  $|\mathcal{V}|$ . The vocabulary with 8,192 tokens does not use merging (note the high RPS for residential buildings).
|V|	Lat,Lon	Buildings-900K-Test (simulated)	BuildingsBench (real)
Commercial	Residential	Commercial	Residential
NRMSE (%)	RPS	NRMSE (%)	RPS	NRMSE (%)	RPS	NRMSE (%)	RPS
3,747	✘	15.22	2.79	45.85	0.913	14.40	5.73	97.84	0.199
474	✓	12.70	1.62	42.94	0.544	14.71	5.16	94.75	0.113
8,192	✓	15.06	2.85	45.50	7.05	14.62	5.90	99.88	14.37
3,747	✓	12.91	2.45	44.29	0.875	14.46	5.62	95.34	0.152
Figure 9: Box-Cox scaling vs. standard scaling training stability. Standard normalizing the load values did not prevent the Transformer (Gaussian) model from suffering from training instability. Box-Cox scaling was successful at achieving stable training.
- App. K.4) Fine-tuning all layers vs. only the last layer for transfer learning.
K.1 Load Tokenizer
Our experiments are conducted with a merged vocabulary of size  $|\mathcal{V}| = 3,747$  ( $K = 8,192$ ). To investigate the impact of using a smaller vocabulary, we trained the Transformer-L model with a merged vocabulary of size  $|\mathcal{V}| = 474$  ( $K = 512$ ) (Fig. 6c). We also trained a model without merging, thus,  $|\mathcal{V}| = 8,192$  ( $K = 8,192$ ) (Fig. 6b). The results in Table 6 indicate that when the vocabulary is too small ( $|\mathcal{V}| = 474$ ), the accuracy on real commercial buildings somewhat deteriorates. Without merging ( $|\mathcal{V}| = 8,192$ ), the RPS on real residential buildings is significantly worse. Nearly all tokens correspond to small load values, which causes the model to allocate probability to a large number of incorrect tokens whose load values are very close to the target value.
K.2 Standard Scaling Data Normalization
As can be seen in Fig. 9, global standard normalization was not sufficient to prevent training instability (large spikes in the loss) due to the highly non-Gaussian load values. By contrast, Box-Cox scaling was successful at stabilizing training.
K.3 Latitude, Longitude Covariate
In our experiments, the transformer models have a latitude, longitude covariate that roughly localizes where a building is in the world. The result of training Transformer-L (Tokens) without this covariate
Table 7: Ablation study results on transfer learning task. Median accuracy (NRMSE) and ranked probability score (RPS). For statistical robustness, we compute average probability of improving NRMSE due to fine-tuning:  $P(X < Y) \coloneqq \frac{100}{N} \sum_{i=1}^{N} \mathbf{1}_{[X_i < Y_i]}$  where  $X_i$  is the pretrained + fine-tuned NRMSE for building  $i$  and  $Y_i$  is the pretrained NRMSE. We compare fine-tuning only the last layer and fine-tuning all layers.
Commercial buildings	Residential buildings
NRMSE (%)	RPS	P(X < Y)	NRMSE (%)	RPS	P(X < Y)
Fine-tune last layer
Transformer-L (Tokens)	14.18	5.07	53.5	93.51	0.136	47
Transformer-L (Gaussian)	12.93	4.50	51	79.57	0.063	36
Fine-tune all layers
Transformer-L (Tokens)	14.07	4.99	61.5	94.53	0.137	39
Transformer-L (Gaussian)	12.96	4.37	71	77.20	0.057	68
are in Table 6. The buildings in the Buildings-900K-Test set are from held-out PUMAs (i.e., counties) with never-before-seen latitude, longitude coordinates. Without latitude and longitude covariates, we observe a negligible drop in zero-shot accuracy on real buildings, but a significant loss in accuracy on the simulated Buildings-900K-Test split. As the majority of real buildings are located in non-U.S. out-of-distribution locations, the latitude and longitude covariate appears to not be particularly helpful for generalizing to these buildings. This is despite mapping the non-U.S. latitude longitude coordinates to U.S. locations with similar climates, which we do to help avoid introducing errors due to extrapolation. We believe further study on the geospatial aspects of generalizable building STLF is a promising use-case of our benchmark.
K.4 What Layers Get Fine-tuned?
For our main transfer learning benchmark results, we fine-tune all layers of the transformers. Here, we compare the performance of these models with fine-tuning only the last layer. Table 7 shows that when fine-tuning only the last linear layer (that output logits or Gaussian parameters), the performance suffers, which can be best seen by comparing the respective probabilities of improvement. Our intuition is that since the pretraining data is synthetic, fine-tuning on real data may require updating all layers. We believe a deeper investigation into transfer learning paradigms for time series is an important topic of study.
L Author Responsibility Statement
The authors confirm that they bear all responsibility in case of any violation of rights during the collection of the data or other work, and will take appropriate action when needed, e.g. to remove data with such issues. The authors also confirm the licenses provided with the data and code associated with this work.
M Additional Plots
We provide additional qualitative results here (Fig. 10 and Fig. 11).
Figure 10: Qualitative results for Transformer-L (Gaussian). Ground truth time series are truncated to previous 24 hours for visibility. Light blue lines are 10 samples from the predicted distribution.
Figure 11: Qualitative results for Transformer-L (Tokens). Ground truth time series are truncated to previous 24 hours for visibility. Light blue lines are 10 samples from the predicted distribution.
(a)
(b)
Figure 12: Real commercial building zero-shot NRMSE and NMAE performance profiles with  $95\%$  bootstrap CIs. Curves closer to the bottom left are better.
(a)
(b)
Figure 13: Real commercial building zero-shot NMBE and RPS performance profiles with  $95\%$  bootstrap CIs. Curves closer to the bottom left are better.
(a)
(b)
Figure 14: Real residential building zero-shot NRMSE and NMAE performance profiles with  $95\%$  bootstrap CIs. Curves closer to the bottom left are better.
(a)
(b)
Figure 15: Real residential building zero-shot NMBE and RPS performance profiles with  $95\%$  bootstrap CIs. Curves closer to the bottom left are better.
(g)
Figure 16: Per-dataset median zero-shot accuracy (NRMSE, NMAE, NMBE) with  $95\%$  bootstrap CIs. Lower is better.