SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Samuel Dooley\*, Gurnoor Singh Khurana, Chirag Mohapatra, Siddartha Naidu, Colin White
1 Abacus.AI, 2 Caltech
The vast majority of time-series forecasting approaches require a substantial training dataset. However, many real-life forecasting applications have very little initial observations, sometimes just 40 or fewer. Thus, the applicability of most forecasting methods is restricted in data-sparse commercial applications. While there is recent work in the setting of very limited initial data (so-called 'zero-shot' forecasting), its performance is inconsistent depending on the data used for pretraining. In this work, we take a different approach and devise ForecastPFN, the first zero-shot forecasting model trained purely on a novel synthetic data distribution. ForecastPFN is a prior-data fitted network, trained to approximate Bayesian inference, which can make predictions on a new time series dataset in a single forward pass. Through extensive experiments, we show that zero-shot predictions made by ForecastPFN are more accurate and faster compared to state-of-the-art forecasting methods, even when the other methods are allowed to train on hundreds of additional in-distribution data points.
Time-series forecasting is an important and long-studied problem that has attracted significant attention for many years [9, 14, 18, 29]. Time-series forecasting has a wide variety of applications such as healthcare [10, 20], economics [28, 42], climate science [5, 38], and renewable energy [19, 53]. Many forecasting algorithms have been proposed, including traditional statistical methods [4, 12], and more recently, deep learning methods [29, 30, 52, 54].
Nearly all time-series forecasting approaches (including both traditional methods and deep learning methods) require a substantial number of initial time-series datapoints in order to learn patterns and make future predictions. However, many real-world forecasting applications have very few initial observations, sometimes just 40 or fewer [6, 8, 16, 23]. This setting is deemed 'zero-shot forecasting' in prior work [36], in contrast to the typical setting in which hundreds or thousands of observations from the target series are used for training. While there is recent work on zero-shot forecasting [36], its performance is inconsistent depending on the dataset used for pretraining.
In this work, we take a different approach and devise ForecastPFN, the first zero-shot method trained purely on a novel synthetic data distribution. ForecastPFN is a prior-data fitted network (PFN) [33]; PFNs are a recently-proposed paradigm, in which a model is pretrained offline on synthetic data generated from a prior to approximate Bayesian inference. PFNs have recently been used in a breakthrough result for tabular datasets [22]. However, there are significant challenges when designing a general and flexible PFN in the forecasting setting, namely, constructing a general time-series distribution, and tuning an architecture and training scheme that can learn it.
*Correspondence to: samuel@abacus.ai, crwhite@caltech.edu.
Figure 1: Left: standard forecasting setting, where a model trains on hundreds of datapoints for a time series, and tests on a future part of the same series. Right: ForecastPFN setting: a one-time, offline training routine on synthetic data, and then the model can make zero-shot predictions on new time series, without retraining or modifying the architecture weights.
We overcome these challenges first by designing a novel synthetic time-series distribution. We carefully design the synthetic distribution to be as general and diverse as possible, with a bias for common patterns seen in real-world series, while still being possible for a machine learning model to learn the distribution. Specifically, we design a modular synthetic model which incorporates multiscale seasonal trends, linear and exponential global trends, and a Weibull-based noise distribution, each of which are parameterized with enough variance to capture a diverse range of time series, while still being possible to learn. Furthermore, using synthetic training data consisting of a ground-truth trend and a separate noise component means that we can remove the noise in the labels when calculating the train loss, which significantly improves the training speed of our model. Next, we design a flexible transformer architecture that can scale across a wide range of time-series values. While many existing transformers for forecasting train with a fixed prediction length and only predict the next  $N$  timesteps into the future, ForecastPFN is trained to perform arbitrary queries – queries to any future timestep – allowing it to achieve zero-shot prediction for arbitrary prediction lengths.
We show that after a one-time, offline training phase on the synthetic dataset, ForecastPFN achieves strong zero-shot performance across a wide variety of real-world datasets, at a fraction of the runtime of other methods. ForecastPFN's zero-shot predictions outperforms state-of-the-art forecasting methods, even when these other methods are allowed to train on hundreds of additional in-distribution datapoints. For example, given a new time series, ForecastPFN makes future predictions after only seeing the input (typically the most-recent  $\approx 30$  points of the series) and outperforms state-of-the-art methods which were allowed to train on an additional 300 historical points in the series (see Figure 1 and Figure 3). This remarkable result is due to the structure of our synthetic priors: by encoding multi-scale seasonal trends, global trends, and noise, across a variety of parameters, our model is able to learn how to forecast general time series. Finally, ForecastPFN is fast, producing predictions on a brand new dataset in just a single forward pass, taking 0.2 seconds. This is over 100 times faster than existing state-of-the-art transformer methods, which require expensive training procedures. Our codebase and our model are available at https://github.com/abacusai/forecastpfn.
Our contributions. We summarize our main contributions below.
- We introduce ForecastPFN, a prior-data fitted network for forecasting, trained purely on a novel synthetic data distribution. ForecastPFN is zero-shot: after its initial pretraining, it can make predictions on a brand new dataset with no training data from that dataset.
- Through extensive experiments across a variety of datasets, we demonstrate that ForecastPFN, without any retraining, outperforms state-of-the-art methods in low-resource settings, even when the other methods are allowed to train on additional in-distribution datapoints. Furthermore, ForecastPFN is fast, making predictions on a new dataset in a single forward pass.
Time-series forecasting [9] has a multitude of applications throughout climate science [5, 38], healthcare [10, 20], business [7, 34], finance [27, 42], and renewable energy [19, 53]. A variety of approaches have been used for time-series forecasting [23, 29], including traditional statistical methods [4, 43] and deep learning methods [50, 52, 54].
ARIMA [4] is a statistical method from the 1970s that is still popular today, which builds an autoregressive model based on Markov processes. Another popular statistical method in practice is Prophet [45], which incorporates non-linear trends and multi-scale seasonality. Recently, deep learning methods for time series forecasting have become popular [29]. While early deep learning methods made use of RNNs [11, 41], transformer models have become popular more recently [26, 50, 52], following the success of transformers on NLP [47]. FEDformer [54] incorporates Fourier transforms and the seasonal trend decomposition method into a transformer architecture. For a survey on deep learning methods for time-series forecasting, see [29, 30], and for transfer learning for time-series forecasting, see [48].
Many real-world forecasting applications have very few initial observations, sometimes just forty or fewer [6, 8, 16, 23]. However, there is much less work in this setting (referred to as 'zero-shot' forecasting [36]). Oreshkin et al. [36] use a model based off of N-BEATS [35], a model that was state of the art at the time, training on a (single) real-world time series dataset and testing on a different time series. However, it achieves substantially different performance based on which training dataset is used. Another zero-shot forecasting method [37] uses an RNN base, yet it is only empirically compared to traditional methods and on a fixed prediction length. Recently, Adriaensen et al. [2] designed a zero-shot approach using a PFN for learning curve extrapolation; however, the setting for learning curve extrapolation is significantly different from time-series forecasting. To the best of our knowledge, we are the first to design a zero-shot time-series forecasting model trained purely on synthetic data. For additional related work, see Appendix A.
3 Prior-Data Fitted Networks for Forecasting
In this section, we give a background on prior-data fitted networks (PFNs), we introduce our synthetic data distribution, and we introduce ForecastPFN.
3.1 Background on Prior-Data Fitted Networks
A univariate time series is a sequence of observations collected over time,  $\{(t,y_t)\}_{t\in t}$ , in which  $t$  is a point in time, and  $y_{t}$  is the value of the series at time  $t$ . In this section, for simplicity, we assume  $t$  is a set of integers  $\{1,2,\dots ,T\}$ , so a time series can be written as  $D = \{(t,y_t)\}_{t = 1}^T$ . Now let  $\Phi$  denote a class of hypotheses, where each hypothesis  $\varphi \in \Phi$  is a mechanism for generating a time series. For example,  $\varphi$  may be defined by a function  $\psi :A\to \mathbb{R}$ , where  $A\subseteq \mathbb{N}$ , which generates an underlying base series; we generate the time series  $D = \{(t,y_t)\}_{t = 1}^T$  by  $y_{t} = \psi (t)\cdot z_{t}$  for all  $t\in A$ , where  $z_{t}$  is sampled from a noise distribution,  $z_{t}\sim \mu$ . In forecasting, we start with a partial time series  $D = \{(t,y_t)\}_{t = 1}^{T'}$ , and the goal is to predict the value of the series,  $\ell$  timesteps into the future.
The posterior predictive distribution (PPD) for a point  $y$  is the distribution of its values given time  $T$  and dataset  $D = \{(t,y_t)\}_{t=1}^{T'}$  (where  $T' < T$ ), denoted as  $p(\cdot \mid T, D)$ . It is computed for a particular point  $y$  by integrating over the set of hypotheses  $\Phi$  as follows:
$$
p (y \mid T, D) \propto \int_ {\Phi} p (y \mid T, \varphi) p (D \mid \varphi) p (\varphi) d \varphi . \tag {1}
$$
A PFN  $q_{\theta}$  is trained to approximate the PPD via synthetic prior-fitting [2, 33] as follows: iteratively sample a hypothesis  $\varphi \sim p(\varphi)$  according to its prior probability, and then use  $\varphi$  to sample a synthetic dataset  $D \sim p(D \mid \varphi)$ . We optimize the PFN's parameters  $\theta$  by making predictions on a held-out output from  $D$ . We split the dataset into a training input set and a training output set. For the case of forecasting, that is  $D_{\mathrm{in}} = \{(t,y_t)\}_{t=1}^{\ell}$  and  $D_{\mathrm{out}} = \{(t,y_t)\}_{t=\ell}^T$ . We use the PFN  $q_{\theta}$  to predict  $D_{\mathrm{out}}$  given  $D_{\mathrm{in}}$ . In the classification setting, we compute the cross entropy loss in expectation over the draw of the dataset  $D \sim p(D)$ , which trains the model to approximate the PPD in the long run [33]. Since forecasting is a regression task, we cannot use cross entropy loss, so we instead minimize
Figure 2: Two time series drawn from our synthetic distribution (Section 3.2). Each row shows the components that make up a time series: seasonal trends (far left), global trends (center left), and noise (center right), as well as the entire time series (far right). We plot only half of the series in order to show the seasonal trends clearly. See Appendix D for more examples.
the mean squared error loss  $\mathcal{L}_{\theta} = \mathbb{E}_{D\sim p(D)}\left[\sum_{t = \ell}^{T}(y_t - q_\theta (t,D_{\mathrm{in}}))^2\right]$ , where  $q_{\theta}(t,D_{\mathrm{in}})$  denotes the model's point-prediction given  $t$  and  $D_{\mathrm{in}}$ . Over many samples draws from  $p(\varphi)$ , this procedure trains the model to predict the mean of the posterior distribution  $p(\cdot \mid t,D_{\mathrm{in}})$ .
Note that the synthetic prior-fitting step is only done once, offline. At inference time, the resulting model is able to make zero-shot predictions: given a previously unseen input  $D_{\mathrm{in}}$ , predict the corresponding  $D_{\mathrm{out}}$  (see Figure 1). The model's weights are not updated during inference: the model makes predictions for a new dataset in a single forward pass. We describe the details of our PFN architecture later in Section 3.3. In the next section, we describe our synthetic prior.
3.2 Defining a Synthetic Prior for Time Series
Unlike prior work in forecasting, our model is not trained on any real-world data; it is trained purely on synthetic data. As discussed above, we assume that there is a prior distribution of time series,  $p(\varphi)$ , from which real-world time series are derived. Intuitively, this distribution includes components such as periodic (weekly, monthly, and yearly) trends, global trends, and noise. We show that when we model data from the distribution described below, we can adequately capture aspects of time-series data in order to produce a high-performance, zero-shot forecasting model.
We model our synthetic data with the simple premise that time series data have two independent components: underlying  $(\psi)$  and noise  $(z_{t})$ . We include time series for three scales: daily, monthly, and yearly, i.e., series where an observation is taken once a day, once a month, or once a year. Data are generated from each periodicity independently according to the following procedure. Now we describe daily series, and we describe the monthly and yearly time series in Appendix D.
We model the underlying time series as being comprised of a seasonal and a trend component. We see these as three independent aspects of time series data and model them as below, where the time series is the product of a trend and seasonality with an additional noise factor. The trend component is made up of a linear and exponential component with coefficients  $m_{\mathrm{lin}}$ ,  $c_{\mathrm{lin}}$ ,  $m_{\mathrm{exp}}$ ,  $c_{\mathrm{exp}}$ . The seasonal component has a week, month, and year component where each comprises of coefficients  $p_{\mathrm{week}}$ ,  $p_{\mathrm{month}}$ ,  $p_{\mathrm{year}}$ . Finally, the noise in our model is derived from a Weibull distribution and is modeled such that the expected value of the noise model is 1, meaning in expectation the noise does not contribute to the seasonality or trend of the time series. Formally,
$$
y _ {t} = \psi (t) \cdot z _ {t} = \operatorname {t r e n d} (t) \cdot \operatorname {s e a s o n a l} (t) \cdot z _ {t}, \text {w h e r e}
$$
$$
z _ {t} = 1 + m _ {\text {n o i s e}} (z - \bar {z}), \text {w h e r e} z \sim \operatorname {W e i b u l l} (1, k), \bar {z} = (\ln 2) ^ {1 / k}
$$
$$
\operatorname {t r e n d} (t) = \left(1 + m _ {\mathrm {l i n}} \cdot t + c _ {\mathrm {l i n}}\right) \left(m _ {\exp} \cdot c _ {\exp} ^ {t}\right)
$$
$$
\operatorname {s e a s o n a l} (t) = \operatorname {s e a s o n a l} _ {\text {w e e k}} (t) \cdot \operatorname {s e a s o n a l} _ {\text {m o n t h}} (t) \cdot \operatorname {s e a s o n a l} _ {\text {y e a r}} (t),
$$
$$
\operatorname {s e a s o n a l} _ {\nu} (t) = 1 + m _ {\nu} \sum_ {f = 1} ^ {\lfloor p _ {\nu} / 2 \rfloor} \left[ c _ {f, \nu} \sin \left(2 \pi f \frac {t}{p _ {\nu}}\right) + d _ {f, \nu} \cos \left(2 \pi f \frac {t}{p _ {\nu}}\right) \right],
$$
$$
\text {w h e r e} \nu \in \{\text {w e e k}, \text {m o n t h}, \text {y e a r} \}, \quad p _ {\text {w e e k}} = 7, p _ {\text {m o n t h}} = 3 0. 5, p _ {\text {y e a r}} = 3 6 5. 2 5.
$$
We choose multiplicative noise to better balance the amount of signal to noise across all series. If we had used additive noise, then the impact of the noise would have depended on the ratio of the scale of the base series and the scale of the noise. Since our base series have linear and exponential terms, additive noise would cause the signal-to-noise ratio to vary substantially over series based on the trend component. Furthermore, the Weibull distribution is a simple and natural method to parameterize between Gaussian and exponential distributions, two types of distributions that frequently come up in real-world time series.
The above synthetic data generation model is extremely general and can capture a variety of time series we would encounter in real world forecasting applications. The model contains a number of parameters, which we denote as
$$
\boldsymbol {\xi} = \left\{m _ {\text {l i n}}, c _ {\text {l i n}}, m _ {\text {e x p}}, c _ {\text {e x p}}, m _ {\text {n o i s e}}, c _ {1, \text {w e a k}}, d _ {1, \text {w e a k}}, \dots , c _ {p _ {\text {y e a r}} 2 /, \text {y e a r}}, d _ {p _ {\text {y e a r}} 2 /, \text {y e a r}} \right\}.
$$
Our full prior  $p(\varphi)$  consists of a product distribution over all elements of  $\xi$ , where
$$
m _ {\mathrm {l i n}}, m _ {\mathrm {e x p}} \sim \mathcal {N} (\mu_ {m}, \sigma_ {m}); \quad c _ {\mathrm {l i n}} \sim \mathcal {N} (0, \sigma_ {\mathrm {l i n}}); \quad c _ {\mathrm {e x p}} \sim \mathcal {N} (1, \sigma_ {\mathrm {e x p}}).
$$
For all pairs of  $\nu$  and  $f$ ,  $c_{f,\nu}$  and  $d_{f,\nu}$  are both drawn from  $\mathcal{N}(0,1/f)$ , such that these coefficients are inversely proportional to the harmonics of the series, and then they are evenly rescaled so that the sum of their squares equals 1.
Therefore, our synthetic data generation model contains six hyperparameters:  $\mu_{m},\sigma_{m},\sigma_{\mathrm{lin}},\sigma_{\mathrm{exp}}$  define the trend component, and  $m_{\mathrm{noise}},k$  define the noise component. See Figure 2 for examples of daily time series drawn from  $p(\varphi)$ , broken down into their different components.
3.3 ForecastPFN: a PFN for Zero-Shot Forecasting
Now we give the details for designing ForecastPFN, the first zero-shot forecasting method trained purely on synthetic data. We focus on the novel components inspired by the extreme challenge of training across a wide diversity of time-series scales with a single model.
Architecture Details. As with the original PFNs [2, 22, 33], we use a transformer [47] as the base architecture. We use an encoder-based transformer, consisting of one multi-head attention layer and two feedforward layers. This is in contrast to prior work on zero-shot forecasting, which used residual networks [36] or recurrent neural networks [37].
The transformer accepts data in the form of tokens  $(t, y_{t})$ . The time stamps  $t$  as part of the input data tokens are represented in terms of temporal features such as the year, month, day, day of the week, and day of the year. The series values are scaled as described in the robust scalar below. A query for ForecastPFN is extremely general: the transformer takes in a set of tokens  $(t, y_{t})$ , along with a query consisting of a single, arbitrary future date, and then it predicts the output of this query. Thus, the input to the model is a set of tokens and the output is a single prediction at a future time, specified by the user. The input does not need to be a fixed size, and the input tokens do not need to be contiguous.
The generality of ForecastPFN queries allows it to perform well at test time, on a very diverse range of datasets and settings. This is also in stark contrast to existing transformer models for forecasting, which typically are only set up to predict the next  $N$  steps in the current sequence.
Robust Scaling. One of the most challenging technical problems when training ForecastPFN is handling the extreme range of the scales of the time series, both in terms of the absolute values and the trends. This is a challenge both at training time and when computing the prediction of a time series, and partially stems from the inclusion of the multiplicative, exponential, and noise terms in our synthetic generation model. Conventional scaling techniques such as standardization (Z-score normalization) and min-max scaling techniques cannot apply: for example, not only does the absolute range differ greatly across training samples, but there is also large diversity in the number of outliers, and the scale of the pattern (i.e., periodicity and global trends) relative to the absolute range.
To handle these challenges, we perform three steps of outlier removal: (1) mask out any missing values, (2) standardize the data based on all non-outlier datapoints (defined as 2-sigma outliers), and (3) clip all 3-sigma outliers. This procedure allows us to put all time series into a relatively consistent range, while removing outliers that cause exploding gradients. Furthermore, when computing the loss of a time series, we scale the mean squared error (MSE) based on the maximum of the (scaled) input. The effect is a loss function that is in between MSE, which gives too much weight for series with a high range, and mean squared percentage error (MSPE), which gives too much weight for series with a small range. We show in Section 4 that robust scaling significantly boosts performance.
Training Details. We train ForecastPFN using the synthetic prior  $p(\varphi)$  defined in Section 3.2. During early-stage model development, we chose hyperparameters  $\mu_{m}, \sigma_{m}, \sigma_{\mathrm{lin}}, \sigma_{\mathrm{exp}}, m_{\mathrm{noise}}, k$  to create the most diverse data distribution possible, with the model still being able to train without diverging or stalling. For example, when the parameters defining the noise distribution become too large, the training data is too noisy and the model cannot converge. We specifically note that (1) all model development was done before evaluating on the seven real-world datasets defined the next section, and (2) all experiments were done with a single ForecastPFN model (specifically, after developing the model, we trained it once, and then used it for all experiments in the next section.
Crucially, since we use synthetic training data with a ground-truth trend and a separate noise component, we can remove the noise when calculating the train loss. This significantly improves the training speed of the model, since the model cannot learn from the noise, as it is independent of time (Appendix E.5).
We set the input length  $\ell = 100$  and the maximum prediction of 10 steps into the future (note: in the next section, we see that ForecastPFN is able to make accurate predictions significantly beyond these settings). We generate 300000 synthetic series to use as training data, each of length 200, and we use a sliding window of size 100 to obtain 101 prediction tasks per generated series. Each epoch consists of 1024000 tasks, and we trained the transformer for 600 epochs with the Adam optimizer [25] on a single Tesla V100 16GB GPU, which took 30 hours. This training is done offline, and only once, and we open-source the trained model at https://github.com/abacusai/forecastpfn. For the full development, implementation, and training details, see Appendix E.
We demonstrate the power of ForecastPFN by comparing it to various high-performing forecasting models across several datasets. We compare all models across different training time budgets, and we compare the zero-shot performance of ForecastPFN to other models with different data budgets.
Algorithmic Baselines. We compare against three simple numerical baselines: Last, Mean, and SeasonalNaive [23]; two high-performing traditional methods: ARIMA [4] and Prophet [23]; three state-of-the-art transformer-based methods: FEDformer [54], Autoformer [50], and Informer [52]; a vanilla Transformer method [47]; and one state-of-the-art zero-shot method: Meta-N-BEATS [36]. All of the methods were implemented using their default hyperparameters and official codebase, with the exception of ARIMA (which has no 'official' codebase), for which we use pdarima [43].