SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Benchmark Datasets. To ensure a fair comparison, we evaluate on seven popular, real-world datasets across energy systems, economics, traffic, and weather: ECL (Electricity Consuming Load) [46], ETT 1 and 2 (Electricity Transformer Temperature) [52], Exchange [28], Illness [17], Traffic [39], and Weather [1]. These datasets are standard for benchmarking the performance of forecasting methods and are the same ones used by many of our baselines [36, 50, 52, 54]. The datasets are described in the Appendix E.
Figure 3: Analysis of performance vs. data budget and time budget, aggregated across datasets and prediction lengths. We plot the number of total MSE wins (left) where a higher value is better. Error bars show one standard deviation across training runs. Recall, ForecastPFN and Meta-N-BEATS see no training data for these series, only the length 36 input.
Experimental Setup. The two zero-shot methods, ForecastPFN and Meta-N-BEATS, were both trained offline before the evaluations on benchmark datasets. We train ForecastPFN once on our synthetic data, as described in Section 3, and then that model is used for all evaluations in this paper. For Meta-N-Beats, we use the best version as described in their paper [36], which is the one trained on M4 [31]. However, since the prediction length needs to be specified upfront, we train several Meta-N-Beats models, one for each prediction length considered below. For each time series, the zero-shot models see only the input of length 36 and then predict future timesteps. These training regimes are different from the other, non-zero shot models which are trained for each configuration.
We compare the zero-shot methods to the nine other non-zero-shot methods described above, even though this is not a fair comparison: the non-zero-shot methods are allowed to train on earlier portions of the time series used at test time, as is common in standard forecasting algorithms. We explore the performance of these algorithms across various resource constraints in two distinct directions: by restricting the training time budget and by restricting the amount of training data budget given to the non-zero-shot methods. We restrict the data budget of the non-zero-shot methods to lengths from 50 to 500. We restrict the time budget of the non-zero-shot methods to wall clock budgets from 1 to 120 seconds, where timing is calculated after data loading and only during the training loop.
These two restrictions follow naturally from the ways in which forecasting datasets and applications can be constrained in the real world. Many real-world forecasting applications have very few initial observations [6, 8, 16, 23]. For example, many businesses need demand forecasts or customer churn forecasts for a large number of products or customers, respectively, which are only size 100 daily series or size 36 monthly series. Furthermore, there is also a wide range of applications which require a low time (or computational) budget in resource constrained environments, such as on a CPU or on edge devices. For example, forecasting may be needed to predict energy demand in developing countries. Another application is as a forecasting data-exploration tool, allowing users to see instant forecasting visualizations as they navigate their dataset.
We also test all algorithms with different prediction lengths, or the number of steps into the future an algorithm must predict. Most deep-learning-based algorithms require this hyperparameter to be set before training, and then that prediction length is encoded into the algorithm's architecture. This is true for FEDFormer, Informer, Autoformer, the vanilla transformer, and Meta-N-BEATS, so we retrain these models for each experiment with a new prediction length. We choose prediction lengths from 6 to 48 as in prior work [54]. At test time, all algorithms are given series with input length of 36 and asked to predict into the future at these variable prediction lengths. For example, say that we have a time series  $\{(t,y_t)\}_{t=1}^{600}$ , a data budget of  $x$ , and a prediction length of  $\ell$ . We allow algorithms to use  $10\%$  of their data budget on validation. All non-zero-shot methods are allowed to train on  $\{(t,y_t)\}_{t=500-x}$ . Then, at test time, all algorithms see the 36 input data points and make a prediction length of  $\ell$ , e.g., input of  $\{(t,y_t)\}_{t=501}^{536}$  and make predictions for timesteps  $t = 537$  to  $537 + \ell$ .
Figure 4: Analysis of performance across prediction lengths for increasing data budgets (left subplot, data budget is 50 to the right subplot, data budget is 500), aggregated across datasets. Total MSE wins for each scenario is plotted with error bars as one standard deviation across training runs. Recall, ForecastPFN and Meta-N-BEATS see no training data for these series, only the length 36 input.
ECL	ETTh1	ETTh2	Exchange	Illness	Traffic	Weather
Data Budget = 50	Arima	1.840	0.341	1.610	1.175	5.045	2.738	0.042
Autoformer	1.289	0.570	1.107	0.331	1.353	2.119	0.520
FEDformer	0.683	0.403	1.020	0.233	1.240	1.318	0.330
ForecastPFN	1.075	0.127	0.330	0.058	1.091	1.971	0.009
Informer	1.252	0.982	0.890	2.834	8.584	3.627	0.436
Last	0.910	0.176	0.466	0.022	1.077	3.081	0.014
Mean	0.673	0.158	0.600	0.040	1.091	1.585	0.011
Meta-N-BEATS	0.909	0.177	0.480	0.023	1.301	2.913	0.014
Prophet	2.174	3.298	14.135	421.635	11.824	2.352	0.101
SeasonalNaive	0.453	0.203	0.554	0.028	1.410	0.653	0.017
Transformer	0.945	0.516	1.043	0.941	4.320	2.244	0.244
Data Budget = 500	Arima	1.969	0.200	1.099	1.175	4.848	1.661	0.042
Autoformer	0.513	0.144	0.338	0.072	0.737	0.526	0.210
FEDformer	0.480	0.133	0.352	0.068	0.707	0.523	0.188
ForecastPFN	1.075	0.127	0.330	0.058	1.091	1.971	0.009
Informer	0.453	0.144	0.253	0.529	4.394	1.084	0.224
Last	0.910	0.176	0.466	0.022	1.077	3.081	0.014
Mean	0.673	0.158	0.600	0.040	1.091	1.585	0.011
Meta N-BEATS	0.909	0.177	0.480	0.023	1.301	2.913	0.014
Prophet	15.668	3.228	4.960	13.346	3.462	1.556	0.057
SeasonalNaive	0.453	0.203	0.554	0.028	1.410	0.653	0.017
Transformer	0.541	0.139	0.274	0.280	3.482	0.935	0.016
Table 1: Average MSE values for each algorithm and each dataset on the shortest (50) and longest (500) data budgets. We observe that ForecastPFN is the majority winner on the smaller data budget and tied for best on the longer data budget. Thus, we see the robustness of the ForecastPFN method across a range of datasets and data budgets.
We conduct training runs under the Cartesian product of each prediction length and each data budget, as well as the Cartesian product of each prediction length and each time budget. We repeat these runs on every dataset and every algorithm. We conduct each run five times with different random seeds (in the randomness of the algorithm). However, ForecastPFN, Meta-N-BEATS, ARIMA, Prophet, and the baselines are deterministic algorithms, so they are only evaluated with one seed per run.
We plot the MSE in this section, and we give additional metrics in Appendix E. In order to aggregate results among all datasets and different random seeds, for each experiment, we compute the number of 'wins' (rank of 1) for each algorithm, as is commonly done in prior work [22, 33]. We plot the performance across data and time budgets in Figure 3, aggregated over datasets and prediction lengths, and tabulate the MSE of each model on each dataset in Table 1. Additionally, we plot the change in
performance for the prediction lengths on each data budget in Figure 4 and in Appendix E). We plot error bars as one standard deviation from the mean by averaging over the training runs.
Results and Discussion. We find that ForecastPFN significantly outperforms all ten other methods in the low data budget and low time budget settings (Figure 3). Specifically, we see that ForecastPFN is the best architecture by the number of MSE wins for time budgets 1 to 30. Similarly, ForecastPFN is the best architecture by the number of MSE wins for data budgets from 50 to 250. In the settings with higher data or time budget, ForecastPFN is still competitive with state-of-the-art methods. This story continues to hold across prediction lengths as well in Figure 4. ForecastPFN is the outright winner in nearly every prediction length for data budgets 50, 100, and 150, and becomes either outright best or very competitive in nearly every prediction length for budgets 200, 250, and 300. Additionally, we see from Table 1 that ForecastPFN achieves the lowest average MSE value on more datasets than any other algorithm, at a data budget of 50, and remains competitive at a data budget of 500. We also find that the simple baselines, SeasonalNaive and Last, often achieve second place behind ForecastPFN in the lowest training and runtime settings. This is not unexpected and is a testament to the challenge of predicting a series when given very little training data or training time.
The remarkable performance of ForecastPFN is despite the fact that, as explained above, the zero shot methods are at a disadvantage because the other methods were allowed to train on data from earlier in the time series. When ForecastPFN is compared to Meta-N-Beats, the only fair comparison, we see that ForecastPFN strictly outperforms Meta-N-Beats across all settings. We believe these strong results are due to our synthetic prior: by encoding multi-scale seasonal trends, global trends, and noise, across a variety of parameters, ForecastPFN learns very strong prior knowledge about the common patterns in time series. While  $\approx 250$  datapoints of a series contain enough complexity and noise for the training-based algorithms to struggle to make accurate predictions when learning the series 'from scratch', ForecastPFN's prior knowledge from synthetic series is significant enough to make better predictions when only given access to the 36 input points.
Furthermore, ForecastPFN is significantly faster than the other transformer-based methods such as FEDformer, Informer, and Autoformer. ForecastPFN makes predictions in a single forward pass, just 0.2 seconds; in order to reach the same accuracy as ForecastPFN, the other transformer-based methods need take over 100 times more runtime.
Figure 5: Robust scaling vs. min-max scaling.
Epoch	0	100	200	300
Train	ForecastPFN, lowest noise	23.71	1.484	0.878	0.591
ForecastPFN, low noise	0.359	0.050	0.039	0.384
ForecastPFN	0.207	0.049	0.037	0.032
Table 2: Loss values reported for three training runs of ForecastPFN where the amount of noise used to synthetically generate the ForecastPFN training data is moderated. In this table, we scale  $m_{\mathrm{noise}}$  to  $1/6$  for the lowest noise model, to  $2/3$  for the low noise model, and 1 for the ForecastPFN model.
Ablation Studies. Since ForecastPFN takes 30 GPU hours to train, we do not conduct substantial ablation studies. However, we are able to investigate two important aspects of the ForecastPFN algorithm: our customScaler and our noise model.
First, we investigate the impact of robust scaling (described in Section 3.3) on the performance of ForecastPFN. We compare robust scaling to min-max scaling. We conduct a training run of ForecastPFN with each scalar and plot the validation MSE loss on the synthetic dataset in Figure 5. We can clearly see that the robust scalar achieves a lower validation loss throughout training. We conclude the use of our robust scalar mitigates the effect of extreme values. While the presence of extreme values make it harder for the model to differentiate small-scale trends in a series, our robust scalar allows for better, faster convergence.
Finally, we investigate the impact of noise in our synthetic training data generation process (described in Section 3.3). We trained ForecastPFN with three scales of our noise parameter,  $m_{\mathrm{noise}}$ : 1 (original
ForecastPFN), 2/3 (low noise), and 1/6 (lowest noise). We do not change the scale of the noise in the validation data. In Table 2, we report the train and validation MSE on synthetic data for each model. Recall that we remove noise in the ground-truth predictions of the training data as a design decision that improves performance, which is why the scale looks different for the train and validation losses. We also look at removing the noise entirely in Appendix E.5 and come to similar conclusions about the level of noise. We can conclude from this ablation that the noise parameters we chose in our experiments provide for robust minimal train and validation losses, validating our approach.
5 Conclusions, Limitations, and Future Work
In this work, we introduced ForecastPFN, the first zero-shot forecasting model that is trained purely on synthetic data. Our novel synthetic data distribution incorporates multi-scale seasonal trends, linear and exponential global trends, and a noise distribution, capturing an extremely diverse range of time series. ForecastPFN is a prior-data fitted network, trained once, offline to approximate Bayesian inference, which can make predictions on a new time series dataset in a single forward pass. Through extensive experiments, we show that zero-shot predictions made by ForecastPFN are more accurate and faster compared to state-of-the-art forecasting methods, even when the other methods are allowed to train on hundreds of additional in-distribution data points.
Limitations and Future Work. While ForecastPFN has shown remarkable success in the zero-shot setting, there are still limitations as well as exciting directions left for future work. Due to the vanilla transformer architecture, ForecastPFN is limited by its training data to time series of fewer than 1000 points. However, given the recent advances in scaling transformers [3, 13, 26, 44], this is not inherently a problem for ForecastPFN. Currently, ForecastPFN only works for univariate time series. Once again, this is not an inherent limitation, and designing ForecastPFN for multivariate time series is a promising direction for future work. Furthermore, using exogenous features are interesting ideas for follow-up work.
Our synthetic prior was created to focus on 'human-like' or 'earth-like' time series (daily, weekly, yearly periods), so ForecastPFN may not perform well on uncommon periods such as 41, 89, or 97. ForecastPFN also only approximates the posterior predictive distribution and makes pointwise predictions. Outputting probabilistic predictions are an exciting area for future work. Finally, ForecastPFN is particularly well-suited to handling missing data (including large gaps), or varying time frequencies within the same series, in contrast to existing forecasting methods. An exciting future direction is to explore the potential to forecast time series in which the input contains large gaps or varying frequencies.
[1] Max-planck-institut für biogeochemie, jena wetterstation beutenberg / weather station beutenberg. https://www.bgc-jena.mpg.de/wetter/.
[2] Steven Adriaensen, Herilalaina Rakotoarison, Samuel Müller, and Frank Hutter. Efficient bayesian learning curve extrapolation using prior-data fitted networks. In Proceedings of the Annual Conference on Neural Information Processing Systems (NeurIPS), 2023.
[3] Iz Beltagy, Matthew E. Peters, and Arman Cohan. Longformer: The long-document transformer. arXiv:2004.05150, 2020.
[4] George EP Box and David A Pierce. Distribution of residual autocorrelations in autoregressive-integrated moving average time series models. Journal of the American statistical Association, 65(332):1509-1526, 1970.
[5] Salva Ruhling Cachay, Venkatesh Ramesh, Jason NS Cole, Howard Barker, and David Rolnick. Climart: A benchmark dataset for emulating atmospheric radiative transfer in weather and climate models. In Proceedings of the Annual Conference on Neural Information Processing Systems Datasets and Benchmarks Track (NeurIPS), 2021.
[6] Rosa María Cantón Croda, Damián Emilio Gibaja Romero, and Santiago Omar Caballero Morales. Sales prediction through neural networks for a small dataset. International Journal of Interactive Multimedia and Artificial Intelligence ..., 2019.
[44] Yi Tay, Mostafa Dehghani, Dara Bahri, and Donald Metzler. Efficient transformers: A survey. ACM Computing Surveys, 2022.
[45] Sean J Taylor and Benjamin Letham. Forecasting at scale. The American Statistician, 2018.
[46] Artur Trindade. ElectricityLoadDiagrams20112014. UCI Machine Learning Repository, 2015. DOI: https://doi.org/10.24432/C58C86.
[47] Ashish Vavwani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pages 5998-6008, 2017.
[48] Manuel Weber, Maximilian Auch, Christoph Doblander, Peter Mandl, and Hans-Arno Jacobsen. Transfer learning with time series data: a systematic mapping study. IEEE Access, 2021.
[49] Ruofeng Wen, Kari Torkkola, Balakrishnan Narayanaswamy, and Dhruv Madeka. A multihorizon quantile recurrent forecaster. arXiv preprint arXiv:1711.11053, 2017.
[50] Haixu Wu, Jiehui Xu, Jianmin Wang, and Mingsheng Long. Autoformer: Decomposition transformers with auto-correlation for long-term series forecasting. Proceedings of the Annual Conference on Neural Information Processing Systems (NeurIPS), 2021.
[51] Rui Ye and Qun Dai. Implementing transfer learning across different datasets for time series forecasting. Pattern Recognition, 2021.
[52] Haoyi Zhou, Shanghang Zhang, Jieqi Peng, Shuai Zhang, Jianxin Li, Hui Xiong, and Wancai Zhang. Informer: Beyond efficient transformer for long sequence time-series forecasting. In Proceedings of the AAAI Conference on Artificial Intelligence (AAAI), 2021.
[53] Jingbo Zhou, Xinjiang Lu, Yixiong Xiao, Jiantao Su, Junfu Lyu, Yanjun Ma, and Dejing Dou. Sdwpf: A dataset for spatial dynamic wind power forecasting challenge at kdd cup 2022. arXiv preprint arXiv:2208.04360, 2022.
[54] Tian Zhou, Ziqing Ma, Qingsong Wen, Xue Wang, Liang Sun, and Rong Jin. Fedformer: Frequency enhanced decomposed transformer for long-term series forecasting. In Proceedings of the International Conference on Machine Learning (ICML), 2022.
Time-series forecasting [9] has a multitude of applications such as healthcare [10, 20], climate science [5, 38], business [7, 34], finance [27, 42], and renewable energy [19, 53]. A variety of approaches have been used for time-series forecasting [23, 29], including traditional statistical methods [4, 43] and deep learning methods [50, 52, 54].
Forecasting. ARIMA [4] is a statistical method from the 1970s that is still popular today, which builds an autoregressive model based on Markov processes. Another popular statistical method in practice is Prophet [45], which incorporates non-linear trends and multi-scale seasonality. Recently, deep learning methods for time series forecasting have become popular [29]. While early deep learning methods made use of RNNs [11, 41], transformer models have become popular more recently [26, 50, 52], following the success of transformers on NLP [47]. FEDformer [54] incorporates Fourier transforms and the seasonal trend decomposition method into a transformer architecture. For a survey on deep learning methods for time-series forecasting, see [29, 30].
Many real-world forecasting applications have very few initial observations, sometimes just forty or fewer [6, 8, 16, 23]. However, there is much less work in this setting (referred to as 'zero-shot' forecasting [36]). Oreshkin et al. [36] use a model based off of N-BEATS [35], a model that was state of the art at the time, training on a (single) real-world time series dataset and testing on a different time series. However, it achieves substantially different performance based on which training dataset is used. Another zero-shot forecasting method [37] uses an RNN base, yet it is only empirically compared to traditional methods and on a fixed prediction length.
There is some work on transfer learning for time-series forecasting, but it is still limited [24]. Earlier works study transfer learning by pretraining a CNN architecture for forecasting on one time series, and fine-tuning on another time series [15, 51]. Fawaz et al. showed that some datasets can improve performance, while others can degrade performance, and they introduce a new method to predict the best source dataset. More recent work uses LSTMs and  $\mathrm{CNN + RNN}$  hybrid models, to achieve better results [48]. Global Forecasting Models (GFM) are a new type of forecasting model in which a single model is trained on all series from a single dataset [21, 40, 41, 49]. However, a key assumption is that these time series must be related in some way. For example, a retail company may train a single model for the time series of sales of thousands of different products.
Prior-Data Fitted Networks. Prior-data fitted networks (PFNs) are a recently-proposed paradigm for machine learning, in which a model is trained offline on small, synthetic datasets. Once trained, a PFN can infer the posterior predictive distribution (PPD) for test datapoints, given a new dataset, in a single forward pass.
Müller et al. [33] first introduced PFNs, showing that they provably approximate Bayesian inference, and demonstrating performance on very small datasets. Soon after, TabPFN [22] showed state-of-the-art empirical performance on datasets of up to size 1000 [32]. This new empirical finding was accomplished in part due to the synthetic prior, consisting of a large set of structural causal models. Recently, Adriaensen et al. [2] designed a zero-shot approach using aPFN for learning curve extrapolation; however, the setting for learning curve extrapolation is significantly different and arguably a special case of time-series forecasting. To the best of our knowledge, we are the first to design a zero-shot time-series forecasting model trained purely on synthetic data.
B Broader Societal Impact
We do not see any strongly negative broader societal impacts of this work. In fact, our hope is that this work will reduce the computational cost and CO2 emissions for forecasting. Although ForecastPFN takes 30 GPU hours to train, this is a one-time cost, which we have already done. Practitioners can now download and use our model, rather than training methods such as FedFormer and Informer from scratch. In general, we hope that our work encourages practitioners and researchers to conduct zero-shot (or few-shot) forecasting, rather than training models from scratch for each new dataset.
Our codebase is available at https://github.com/abacusai/forecastpfn. The README to the codebase contains instructions for installation and immediately using ForecastPFN to make predictions on a new dataset. We also include all code from this project, including the code for synthetic data generation, ForecastPFN training, and evaluation.
D Synthetic Data Details
We give the full details of the synthetic data series, including the parameters for daily, weekly, and monthly series.
Recall that the time series has a trend component, seasonal component, and noise component. The trend component is made up of a linear and exponential component with coefficients  $m_{\mathrm{lin}}, c_{\mathrm{lin}}, m_{\mathrm{exp}}, c_{\mathrm{exp}}$ . The seasonal component has a week, month, and year component where each comprises of coefficients  $p_{\mathrm{week}}, p_{\mathrm{month}}, p_{\mathrm{year}}$ . Finally, the noise in our model is derived from a Weibull distribution and is modeled such that the expected value of the noise model is 1, meaning in expectation the noise does not contribute to the seasonality or trend of the time series. Formally,
$$
\operatorname {s e a s o n a l} _ {\nu} (t) = 1 + m _ {\nu} \sum_ {f = 1} ^ {\lfloor p _ {\nu} / 2 \rfloor} \left[ c _ {f, \nu} \sin \left(2 \pi f \frac {t}{p _ {\nu}}\right) + d _ {f, \nu} \cos \left(2 \pi f \frac {t}{p _ {\nu}}\right) \right].
$$
Now we explain how to set all unspecified parameters above, for daily, weekly, and monthly series. For convenience, when creating a daily, weekly, or monthly series, a unit of time  $t$  is equal to 1, 7, or 30.5 days (which would be equivalent to scaling all the parameters below by 1, 7, or 30.5).
We set each  $m_{\nu}$  for  $\nu \in \{\text{week, month, year}\}$  separately for daily, weekly, and monthly, as follows.  $m_{\nu} = 0$  unless specified: