Title: ForecastPFN: Synthetically-Trained Zero-Shot Forecasting
Venue: NeurIPS 2023
Accepted: yes
Keywords: Forecasting

Zero-shot

Synthetic Data
Abstract: The vast majority of time-series forecasting approaches require a substantial training dataset. However, many real-life forecasting applications have very little initial observations, sometimes just 40 or fewer. Thus, the applicability of most forecasting methods is restricted in data-sparse commercial applications. While there is recent work in the setting of very limited initial data (so-called `zero-shot' forecasting), its performance is inconsistent depending on the data used for pretraining. In this work, we take a different approach and devise ForecastPFN, the first zero-shot forecasting model trained purely on a novel synthetic data distribution. ForecastPFN is a prior-data fitted network, trained to approximate Bayesian inference, which can make predictions on a new time series dataset in a single forward pass. Through extensive experiments, we show that zero-shot predictions made by ForecastPFN are more accurate and faster compared to state-of-the-art forecasting methods, even when the other methods are allowed to train on hundreds of additional in-distribution data points.

ForecastPFN: Synthetically-Trained Zero-Shot Forecasting

Abstract

The vast majority of time-series forecasting approaches require a substantial training dataset. However, many real-life forecasting applications have very little initial observations, sometimes just 40 or fewer. Thus, the applicability of most forecasting methods is restricted in data-sparse commercial applications. While there is recent work in the setting of very limited initial data (so-called 'zero-shot' forecasting), its performance is inconsistent depending on the data used for pretraining. In this work, we take a different approach and devise ForecastPFN, the first zero-shot forecasting model trained purely on a novel synthetic data distribution. ForecastPFN is a prior-data fitted network, trained to approximate Bayesian inference, which can make predictions on a new time series dataset in a single forward pass. Through extensive experiments, we show that zero-shot predictions made by ForecastPFN are more accurate and faster compared to state-of-the-art forecasting methods, even when the other methods are allowed to train on hundreds of additional in-distribution data points.

1 Introduction

Time-series forecasting is an important and long-studied problem that has attracted significant attention for many years [9, 14, 18, 29]. Time-series forecasting has a wide variety of applications such as healthcare [10, 20], economics [28, 42], climate science [5, 38], and renewable energy [19, 53]. Many forecasting algorithms have been proposed, including traditional statistical methods [4, 12], and more recently, deep learning methods [29, 30, 52, 54].

Nearly all time-series forecasting approaches (including both traditional methods and deep learning methods) require a substantial number of initial time-series datapoints in order to learn patterns and make future predictions. However, many real-world forecasting applications have very few initial observations, sometimes just 40 or fewer [6, 8, 16, 23]. This setting is deemed 'zero-shot forecasting' in prior work [36], in contrast to the typical setting in which hundreds or thousands of observations from the target series are used for training. While there is recent work on zero-shot forecasting [36], its performance is inconsistent depending on the dataset used for pretraining.

In this work, we take a different approach and devise ForecastPFN, the first zero-shot method trained purely on a novel synthetic data distribution . ForecastPFN is a prior-data fitted network (PFN) [33]; PFNs are a recently-proposed paradigm, in which a model is pretrained offline on synthetic data generated from a prior to approximate Bayesian inference. PFNs have recently been used in a breakthrough result for tabular datasets [22]. However, there are significant challenges when designing a general and flexible PFN in the forecasting setting, namely, constructing a general time-series distribution, and tuning an architecture and training scheme that can learn it.

Model

…

Traditional setting

Train on the same (single)

real-world series

Test on a new part of

the same

series

?

Train

Test

ForecastPFN

ForecastPFN setting

Train once, offline

with

synthetic

, diverse series

Zero-shot test on new,

unseen

, real-world series

?

Train

Test

…

…

…

We overcome these challenges first by designing a novel synthetic time-series distribution. We carefully design the synthetic distribution to be as general and diverse as possible, with a bias for common patterns seen in real-world series, while still being possible for a machine learning model to learn the distribution. Specifically, we design a modular synthetic model which incorporates multiscale seasonal trends, linear and exponential global trends, and a Weibull-based noise distribution, each of which are parameterized with enough variance to capture a diverse range of time series, while still being possible to learn. Furthermore, using synthetic training data consisting of a groundtruth trend and a separate noise component means that we can remove the noise in the labels when calculating the train loss, which significantly improves the training speed of our model. Next, we design a flexible transformer architecture that can scale across a wide range of time-series values. While many existing transformers for forecasting train with a fixed prediction length and only predict the next N timesteps into the future, ForecastPFN is trained to perform arbitrary queries - queries to any future timestep - allowing it to achieve zero-shot prediction for arbitrary prediction lengths.

We show that after a one-time, offline training phase on the synthetic dataset, ForecastPFN achieves strong zero-shot performance across a wide variety of real-world datasets, at a fraction of the runtime of other methods. ForecastPFN's zero-shot predictions outperforms state-of-the-art forecasting methods, even when these other methods are allowed to train on hundreds of additional in-distribution datapoints . For example, given a new time series, ForecastPFN makes future predictions after only seeing the input (typically the most-recent ≈ 30 points of the series) and outperforms state-of-the-art methods which were allowed to train on an additional 300 historical points in the series (see Figure 1 and Figure 3). This remarkable result is due to the structure of our synthetic priors: by encoding multi-scale seasonal trends, global trends, and noise, across a variety of parameters, our model is able to learn how to forecast general time series. Finally, ForecastPFN is fast , producing predictions on a brand new dataset in just a single forward pass, taking 0.2 seconds. This is over 100 times faster than existing state-of-the-art transformer methods, which require expensive training procedures. Our codebase and our model are available at https://github.com/abacusai/forecastpfn .

Our contributions. We summarize our main contributions below.

We introduce ForecastPFN, a prior-data fitted network for forecasting, trained purely on a novel synthetic data distribution. ForecastPFN is zero-shot : after its initial pretraining, it can make predictions on a brand new dataset with no training data from that dataset.

Through extensive experiments across a variety of datasets, we demonstrate that ForecastPFN, without any retraining, outperforms state-of-the-art methods in low-resource settings, even when the other methods are allowed to train on additional in-distribution datapoints. Furthermore, ForecastPFN is fast, making predictions on a new dataset in a single forward pass.

2 Related Work

Time-series forecasting [9] has a multitude of applications throughout climate science [5, 38], healthcare [10, 20], business [7, 34], finance [27, 42], and renewable energy [19, 53]. A variety of approaches have been used for time-series forecasting [23, 29], including traditional statistical methods [4, 43] and deep learning methods [50, 52, 54].

ARIMA [4] is a statistical method from the 1970s that is still popular today, which builds an autoregressive model based on Markov processes. Another popular statistical method in practice is Prophet [45], which incorporates non-linear trends and multi-scale seasonality. Recently, deep learning methods for time series forecasting have become popular [29]. While early deep learning methods made use of RNNs [11, 41], transformer models have become popular more recently [26, 50, 52], following the success of transformers on NLP [47]. FEDformer [54] incorporates Fourier transforms and the seasonal trend decomposition method into a transformer architecture. For a survey on deep learning methods for time-series forecasting, see [29, 30], and for transfer learning for time-series forecasting, see [48].

Many real-world forecasting applications have very few initial observations, sometimes just forty or fewer [6, 8, 16, 23]. However, there is much less work in this setting (referred to as 'zero-shot' forecasting [36]). Oreshkin et al. [36] use a model based off of N-BEATS [35], a model that was state of the art at the time, training on a (single) real-world time series dataset and testing on a different time series. However, it achieves substantially different performance based on which training dataset is used. Another zero-shot forecasting method [37] uses an RNN base, yet it is only empirically compared to traditional methods and on a fixed prediction length. Recently, Adriaensen et al. [2] designed a zero-shot approach using a PFN for learning curve extrapolation; however, the setting for learning curve extrapolation is significantly different from time-series forecasting. To the best of our knowledge, we are the first to design a zero-shot time-series forecasting model trained purely on synthetic data. For additional related work, see Appendix A.

3 Prior-Data Fitted Networks for Forecasting

In this section, we give a background on prior-data fitted networks (PFNs), we introduce our synthetic data distribution, and we introduce ForecastPFN.

3.1 Background on Prior-Data Fitted Networks

A univariate time series is a sequence of observations collected over time, { ( t, y t ) } t ∈ t , in which t is a point in time, and y t is the value of the series at time t . In this section, for simplicity, we assume t is a set of integers { 1 , 2 , . . . , T } , so a time series can be written as D = { ( t, y t ) } T t =1 . Now let Φ denote a class of hypotheses, where each hypothesis φ ∈ Φ is a mechanism for generating a time series. For example, φ may be defined by a function ψ : A → R , where A ⊆ N , which generates an underlying base series; we generate the time series D = { ( t, y t ) } T t =1 by y t = ψ ( t ) · z t for all t ∈ A , where z t is sampled from a noise distribution, z t ∼ µ. In forecasting, we start with a partial time series D = { ( t, y t ) } T ′ t =1 , and the goal is to predict the value of the series, ℓ timesteps into the future.

The posterior predictive distribution (PPD) for a point y is the distribution of its values given time T and dataset D = { ( t, y t ) } T ′ t =1 (where T ′ < T ), denoted as p ( · | T, D ) . It is computed for a particular point y by integrating over the set of hypotheses Φ as follows:

A PFN q θ is trained to approximate the PPD via synthetic prior-fitting [2, 33] as follows: iteratively sample a hypothesis φ ∼ p ( φ ) according to its prior probability, and then use φ to sample a synthetic dataset D ∼ p ( D | φ ) . We optimize the PFN's parameters θ by making predictions on a held-out output from D . We split the dataset into a training input set and a training output set. For the case of forecasting, that is D in = { ( t, y t ) } ℓ t =1 and D out = { ( t, y t ) } T t = ℓ . We use the PFN q θ to predict D out given D in. In the classification setting, we compute the cross entropy loss in expectation over the draw of the dataset D ∼ p ( D ) , which trains the model to approximate the PPD in the long run [33]. Since forecasting is a regression task, we cannot use cross entropy loss, so we instead minimize

Seasonal Trend

Global Trend

Noise

110

Full Time Series

100

1.050

100

1.04

75

1.025

90

50

80

70

25

0.975

0.96

60

MonthlyTrend—WeeklyTrend

Exponential-Global Trend-Linear

0

80-01-2238388-03.223-01

88-01-2202-03-88-03.223-01

20202

2020

Dates

Dates

Dates

Dates

Seasonal Trend

Global Trend

1.02

Noise

Full Time Series

3

1.2

1.01

1.1

1.5

1

0.9

0.99

0.8

MonthlyTrend—WeeklyTrend

Exponential—Global Trend-Linear

0.5

0.98

88-01-223-00-88-02.223-01

801-22303-8802.223-01

202002

Dates

Dates

Dates

Dates

the mean squared error loss L θ = E D ∼ p ( D ) [ ∑ T t = ℓ ( y t -q θ ( t, D in )) 2 ] , where q θ ( t, D in ) denotes the model's point-prediction given t and D in. Over many samples draws from p ( φ ) , this procedure trains the model to predict the mean of the posterior distribution p ( · | t, D in ) .

Note that the synthetic prior-fitting step is only done once, offline. At inference time, the resulting model is able to make zero-shot predictions: given a previously unseen input D in , predict the corresponding D out (see Figure 1). The model's weights are not updated during inference: the model makes predictions for a new dataset in a single forward pass. We describe the details of our PFN architecture later in Section 3.3. In the next section, we describe our synthetic prior.

3.2 Defining a Synthetic Prior for Time Series

Unlike prior work in forecasting, our model is not trained on any real-world data; it is trained purely on synthetic data. As discussed above, we assume that there is a prior distribution of time series, p ( φ ) , from which real-world time series are derived. Intuitively, this distribution includes components such as periodic (weekly, monthly, and yearly) trends, global trends, and noise. We show that when we model data from the distribution described below, we can adequately capture aspects of time-series data in order to produce a high-performance, zero-shot forecasting model.

We model our synthetic data with the simple premise that time series data have two independent components: underlying ( ψ ) and noise ( z t ). We include time series for three scales: daily, monthly, and yearly, i.e., series where an observation is taken once a day, once a month, or once a year. Data are generated from each periodicity independently according to the following procedure. Now we describe daily series, and we describe the monthly and yearly time series in Appendix D.

We model the underlying time series as being comprised of a seasonal and a trend component. We see these as three independent aspects of time series data and model them as below, where the time series is the product of a trend and seasonality with an additional noise factor. The trend component is made up of a linear and exponential component with coefficients m lin , c lin , m exp , c exp . The seasonal component has a week, month, and year component where each comprises of coefficients p week , p month , p year Finally, the noise in our model is derived from a Weibull distribution and is modeled such that the expected value of the noise model is 1, meaning in expectation the noise does not contribute to the seasonality or trend of the time series. Formally,

We choose multiplicative noise to better balance the amount of signal to noise across all series. If we had used additive noise, then the impact of the noise would have depended on the ratio of the scale of the base series and the scale of the noise. Since our base series have linear and exponential terms, additive noise would cause the signal-to-noise ratio to vary substantially over series based on the trend component. Furthermore, the Weibull distribution is a simple and natural method to parameterize between Gaussian and exponential distributions, two types of distributions that frequently come up in real-world time series.

The above synthetic data generation model is extremely general and can capture a variety of time series we would encounter in real world forecasting applications. The model contains a number of parameters, which we denote as

Our full prior p ( φ ) consists of a product distribution over all elements of ξ , where

For all pairs of ν and f , c f,ν and d f,ν are both drawn from N (0 , 1 /f ) , such that these coefficients are inversely proportional to the harmonics of the series, and then they are evenly rescaled so that the sum of their squares equals 1.

Therefore, our synthetic data generation model contains six hyperparameters: µ m , σ m , σ lin , σ exp define the trend component, and m noise , k define the noise component. See Figure 2 for examples of daily time series drawn from p ( φ ) , broken down into their different components.

3.3 ForecastPFN: a PFN for Zero-Shot Forecasting

Now we give the details for designing ForecastPFN, the first zero-shot forecasting method trained purely on synthetic data. We focus on the novel components inspired by the extreme challenge of training across a wide diversity of time-series scales with a single model.

Architecture Details. As with the original PFNs [2, 22, 33], we use a transformer [47] as the base architecture. We use an encoder-based transformer, consisting of one multi-head attention layer and two feedforward layers. This is in contrast to prior work on zero-shot forecasting, which used residual networks [36] or recurrent neural networks [37].

The transformer accepts data in the form of tokens ( t, y t ) . The time stamps t as part of the input data tokens are represented in terms of temporal features such as the year, month, day, day of the week, and day of the year. The series values are scaled as described in the robust scaler below. A query for ForecastPFN is extremely general: the transformer takes in a set of tokens ( t, y t ) , along with a query consisting of a single, arbitrary future date, and then it predicts the output of this query. Thus, the input to the model is a set of tokens and the output is a single prediction at a future time, specified by the user. The input does not need to be a fixed size, and the input tokens do not need to be contiguous.

The generality of ForecastPFN queries allows it to perform well at test time, on a very diverse range of datasets and settings. This is also in stark contrast to existing transformer models for forecasting, which typically are only set up to predict the next N steps in the current sequence.

Robust Scaling. One of the most challenging technical problems when training ForecastPFN is handling the extreme range of the scales of the time series, both in terms of the absolute values and the trends. This is a challenge both at training time and when computing the prediction of a time series, and partially stems from the inclusion of the multiplicative, exponential, and noise terms in our synthetic generation model. Conventional scaling techniques such as standardization (Z-score normalization) and min-max scaling techniques cannot apply: for example, not only does the absolute range differ greatly across training samples, but there is also large diversity in the number of outliers, and the scale of the pattern (i.e., periodicity and global trends) relative to the absolute range.

To handle these challenges, we perform three steps of outlier removal: (1) mask out any missing values, (2) standardize the data based on all non-outlier datapoints (defined as 2-sigma outliers), and (3) clip all 3-sigma outliers. This procedure allows us to put all time series into a relatively consistent range, while removing outliers that cause exploding gradients. Furthermore, when computing the loss of a time series, we scale the mean squared error (MSE) based on the maximum of the (scaled) input. The effect is a loss function that is in between MSE, which gives too much weight for series with a high range, and mean squared percentage error (MSPE), which gives too much weight for series with a small range. We show in Section 4 that robust scaling significantly boosts performance.

Training Details. We train ForecastPFN using the synthetic prior p ( φ ) defined in Section 3.2. During early-stage model development, we chose hyperparameters µ m , σ m , σ lin , σ exp , m noise , k to create the most diverse data distribution possible, with the model still being able to train without diverging or stalling. For example, when the parameters defining the noise distribution become too large, the training data is too noisy and the model cannot converge. We specifically note that (1) all model development was done before evaluating on the seven real-world datasets defined the next section, and (2) all experiments were done with a single ForecastPFN model (specifically, after developing the model, we trained it once, and then used it for all experiments in the next section.

Crucially, since we use synthetic training data with a ground-truth trend and a separate noise component, we can remove the noise when calculating the train loss. This significantly improves the training speed of the model, since the model cannot learn from the noise, as it is independent of time (Appendix E.5).

We set the input length ℓ = 100 and the maximum prediction of 10 steps into the future (note: in the next section, we see that ForecastPFN is able to make accurate predictions significantly beyond these settings). We generate 300 000 synthetic series to use as training data, each of length 200, and we use a sliding window of size 100 to obtain 101 prediction tasks per generated series. Each epoch consists of 1 024 000 tasks, and we trained the transformer for 600 epochs with the Adam optimizer [25] on a single Tesla V100 16GB GPU, which took 30 hours. This training is done offline, and only once, and we open-source the trained model at https://github.com/abacusai/forecastpfn . For the full development, implementation, and training details, see Appendix E.

4 Experiments

We demonstrate the power of ForecastPFN by comparing it to various high-performing forecasting models across several datasets. We compare all models across different training time budgets, and we compare the zero-shot performance of ForecastPFN to other models with different data budgets.

Algorithmic Baselines. We compare against three simple numerical baselines: Last, Mean, and SeasonalNaive [23]; two high-performing traditional methods: ARIMA [4] and Prophet [23]; three state-of-the-art transformer-based methods: FEDformer [54], Autoformer [50], and Informer [52]; a vanilla Transformer method [47]; and one state-of-the-art zero-shot method: Meta-N-BEATS [36]. All of the methods were implemented using their default hyperparameters and official codebase, with the exception of ARIMA (which has no 'official' codebase), for which we use pmdarima [43].

Benchmark Datasets. To ensure a fair comparison, we evaluate on seven popular, real-world datasets across energy systems, economics, traffic, and weather: ECL (Electricity Consuming Load) [46], ETT 1 and 2 (Electricity Transformer Temperature) [52], Exchange [28], Illness [17], Traffic [39], and Weather [1]. These datasets are standard for benchmarking the performance of forecasting methods and are the same ones used by many of our baselines [36, 50, 52, 54]. The datasets are described in the Appendix E.

50

100

300

500

0

5

10

15

20

Number of total MSE Wins per Data Budget

Data Budget

MSE Wins

Model

Arima

Autoformer

FEDformer

ForecastPFN

Informer

Last

Mean

Meta-N-BEATS

Prophet

SeasonalNaive

Transformer

1

10

100

0

5

10

15

20

Number of total MSE Wins per Time Budget

Time Budget

MSE Wins

Experimental Setup. The two zero-shot methods, ForecastPFN and Meta-N-BEATS, were both trained offline before the evaluations on benchmark datasets. We train ForecastPFN once on our synthetic data, as described in Section 3, and then that model is used for all evaluations in this paper. For Meta-N-Beats, we use the best version as described in their paper [36], which is the one trained on M4 [31]. However, since the prediction length needs to be specified upfront, we train several Meta-N-Beats models, one for each prediction length considered below. For each time series, the zero-shot models see only the input of length 36 and then predict future timesteps. These training regimes are different from the other, non-zero shot models which are trained for each configuration.

We compare the zero-shot methods to the nine other non-zero-shot methods described above, even though this is not a fair comparison: the non-zero-shot methods are allowed to train on earlier portions of the time series used at test time, as is common in standard forecasting algorithms. We explore the performance of these algorithms across various resource constraints in two distinct directions: by restricting the training time budget and by restricting the amount of training data budget given to the non-zero-shot methods. We restrict the data budget of the non-zero-shot methods to lengths from 50 to 500. We restrict the time budget of the non-zero-shot methods to wall clock budgets from 1 to 120 seconds, where timing is calculated after data loading and only during the training loop.

These two restrictions follow naturally from the ways in which forecasting datasets and applications can be constrained in the real world. Many real-world forecasting applications have very few initial observations [6, 8, 16, 23]. For example, many businesses need demand forecasts or customer churn forecasts for a large number of products or customers, respectively, which are only size 100 daily series or size 36 monthly series. Furthermore, there is also a wide range of applications which require a low time (or computational) budget in resource constrained environments, such as on a CPU or on edge devices. For example, forecasting may be needed to predict energy demand in developing countries. Another application is as a forecasting data-exploration tool, allowing users to see instant forecasting visualizations as they navigate their dataset.

We also test all algorithms with different prediction lengths, or the number of steps into the future an algorithm must predict. Most deep-learning-based algorithms require this hyperparameter to be set before training, and then that prediction length is encoded into the algorithm's architecture. This is true for FEDFormer, Informer, Autoformer, the vanilla transformer, and Meta-N-BEATS, so we retrain these models for each experiment with a new prediction length. We choose prediction lengths from 6 to 48 as in prior work [54]. At test time, all algorithms are given series with input length of 36 and asked to predict into the future at these variable prediction lengths. For example, say that we have a time series { ( t, y t ) } 600 t =1 , a data budget of x , and a prediction length of ℓ . We allow algorithms to use 10% of their data budget on validation. All non-zero-shot methods are allowed to train on { ( t, y t ) } 500 t =500 -x . Then, at test time, all algorithms see the 36 input data points and make a prediction length of ℓ , e.g., input of { ( t, y t ) } 536 t =501 and make predictions for timesteps t = 537 to 537 + ℓ .

10

20

30

40

50

0

1

2

3

4

50

10

20

30

40

50

100

10

20

30

40

50

150

10

20

30

40

50

200

10

20

30

40

50

250

10

20

30

40

50

300

10

20

30

40

50

500

MSE Wins per Prediction Length By Data Budget (50 to 500)

Prediction Length

MSE Wins

Arima

Autoformer

FEDformer

ForecastPFN

Informer

Last

Mean

Meta-N-BEATS

Prophet

SeasonalNaive

Transformer

We conduct training runs under the Cartesian product of each prediction length and each data budget, as well as the Cartesian product of each prediction length and each time budget. We repeat these runs on every dataset and every algorithm. We conduct each run five times with different random seeds (in the randomness of the algorithm). However, ForecastPFN, Meta-N-BEATS, ARIMA, Prophet, and the baselines are deterministic algorithms, so they are only evaluated with one seed per run.

We plot the MSE in this section, and we give additional metrics in Appendix E. In order to aggregate results among all datasets and different random seeds, for each experiment, we compute the number of 'wins' (rank of 1) for each algorithm, as is commonly done in prior work [22, 33]. We plot the performance across data and time budgets in Figure 3, aggregated over datasets and prediction lengths, and tabulate the MSE of each model on each dataset in Table 1. Additionally, we plot the change in

performance for the prediction lengths on each data budget in Figure 4 and in Appendix E). We plot error bars as one standard deviation from the mean by averaging over the training runs.

Results and Discussion. We find that ForecastPFN significantly outperforms all ten other methods in the low data budget and low time budget settings (Figure 3). Specifically, we see that ForecastPFN is the best architecture by the number of MSE wins for time budgets 1 to 30. Similarly, ForecastPFN is the best architecture by the number of MSE wins for data budgets from 50 to 250. In the settings with higher data or time budget, ForecastPFN is still competitive with state-of-the-art methods. This story continues to hold across prediction lengths as well in Figure 4. ForecastPFN is the outright winner in nearly every prediction length for data budgets 50, 100, and 150, and becomes either outright best or very competitive in nearly every prediction length for budgets 200, 250, and 300. Additionally, we see from Table 1 that ForecastPFN achieves the lowest average MSE value on more datasets than any other algorithm, at a data budget of 50, and remains competitive at a data budget of 500. We also find that the simple baselines, SeasonalNaive and Last, often achieve second place behind ForecastPFN in the lowest training and runtime settings. This is not unexpected and is a testament to the challenge of predicting a series when given very little training data or training time.

The remarkable performance of ForecastPFN is despite the fact that, as explained above, the zero shot methods are at a disadvantage because the other methods were allowed to train on data from earlier in the time series. When ForecastPFN is compared to Meta-N-Beats, the only fair comparison, we see that ForecastPFN strictly outperforms Meta-N-Beats across all settings. We believe these strong results are due to our synthetic prior: by encoding multi-scale seasonal trends, global trends, and noise, across a variety of parameters, ForecastPFN learns very strong prior knowledge about the common patterns in time series. While ≈ 250 datapoints of a series contain enough complexity and noise for the training-based algorithms to struggle to make accurate predictions when learning the series 'from scratch', ForecastPFN's prior knowledge from synthetic series is significant enough to make better predictions when only given access to the 36 input points.

Furthermore, ForecastPFN is significantly faster than the other transformer-based methods such as FEDformer, Informer, and Autoformer. ForecastPFN makes predictions in a single forward pass, just 0.2 seconds; in order to reach the same accuracy as ForecastPFN, the other transformer-based methods need take over 100 times more runtime.

Ablation Studies. Since ForecastPFN takes 30 GPU hours to train, we do not conduct substantial ablation studies. However, we are able to investigate two important aspects of the ForecastPFN algorithm: our custom scaler and our noise model.

First, we investigate the impact of robust scaling (described in Section 3.3) on the performance of ForecastPFN. We compare robust scaling to min-max scaling. We conduct a training run of ForecastPFN with each scaler and plot the validation MSE loss on the synthetic dataset in Figure 5. We can clearly see that the robust scaler achieves a lower validation loss throughout training. We conclude the use of our robust scaler mitigates the effect of extreme values. While the presence of extreme values make it harder for the model to differentiate small-scale trends in a series, our robust scaler allows for better, faster convergence.

Finally, we investigate the impact of noise in our synthetic training data generation process (described in Section 3.3). We trained ForecastPFN with three scales of our noise parameter, m noise: 1 (original

ForecastPFN), 2 / 3 (low noise), and 1 / 6 (lowest noise). We do not change the scale of the noise in the validation data. In Table 2, we report the train and validation MSE on synthetic data for each model. Recall that we remove noise in the ground-truth predictions of the training data as a design decision that improves performance, which is why the scale looks different for the train and validation losses. We also look at removing the noise entirely in Appendix E.5 and come to similar conclusions about the level of noise. We can conclude from this ablation that the noise parameters we chose in our experiments provide for robust minimal train and validation losses, validating our approach.

5 Conclusions, Limitations, and Future Work

In this work, we introduced ForecastPFN, the first zero-shot forecasting model that is trained purely on synthetic data. Our novel synthetic data distribution incorporates multi-scale seasonal trends, linear and exponential global trends, and a noise distribution, capturing an extremely diverse range of time series. ForecastPFN is a prior-data fitted network, trained once, offline to approximate Bayesian inference, which can make predictions on a new time series dataset in a single forward pass. Through extensive experiments, we show that zero-shot predictions made by ForecastPFN are more accurate and faster compared to state-of-the-art forecasting methods, even when the other methods are allowed to train on hundreds of additional in-distribution data points.

Limitations and Future Work. While ForecastPFN has shown remarkable success in the zero-shot setting, there are still limitations as well as exciting directions left for future work. Due to the vanilla transformer architecture, ForecastPFN is limited by its training data to time series of fewer than 1000 points. However, given the recent advances in scaling transformers [3, 13, 26, 44], this is not inherently a problem for ForecastPFN. Currently, ForecastPFN only works for univariate time series. Once again, this is not an inherent limitation, and designing ForecastPFN for multivariate time series is a promising direction for future work. Furthermore, using exogenous features are interesting ideas for follow-up work.

Our synthetic prior was created to focus on 'human-like' or 'earth-like' time series (daily, weekly, yearly periods), so ForecastPFN may not not perform well on uncommon periods such as 41, 89, or 97. ForecastPFN also only approximates the posterior predictive distribution and makes pointwise predictions. Outputting probabilistic predictions are an exciting area for future work. Finally, ForecastPFN is particularly well-suited to handling missing data (including large gaps), or varying time frequencies within the same series, in contrast to existing forecasting methods. An exciting future direction is to explore the potential to forecast time series in which the input contains large gaps or varying frequencies.

References

Max-planck-institut für biogeochemie, jena wetterstation beutenberg / weather station beutenberg. https://www.bgc-jena.mpg.de/wetter/ .

Steven Adriaensen, Herilalaina Rakotoarison, Samuel Müller, and Frank Hutter. Efficient bayesian learning curve extrapolation using prior-data fitted networks. In Proceedings of the Annual Conference on Neural Information Processing Systems (NeurIPS) , 2023.

Iz Beltagy, Matthew E. Peters, and Arman Cohan. Longformer: The long-document transformer. arXiv:2004.05150 , 2020.

George EP Box and David A Pierce. Distribution of residual autocorrelations in autoregressiveintegrated moving average time series models. Journal of the American statistical Association , 65(332):1509-1526, 1970.

Salva Rühling Cachay, Venkatesh Ramesh, Jason NS Cole, Howard Barker, and David Rolnick. Climart: A benchmark dataset for emulating atmospheric radiative transfer in weather and climate models. In Proceedings of the Annual Conference on Neural Information Processing Systems Datasets and Benchmarks Track (NeurIPS) , 2021.

Rosa María Cantón Croda, Damián Emilio Gibaja Romero, and Santiago Omar Caballero Morales. Sales prediction through neural networks for a small dataset. International Journal of Interactive Multimedia and Artificial Intelligence . . . , 2019.

Real Carbonneau, Kevin Laframboise, and Rustam Vahidov. Application of machine learning techniques for supply chain demand forecasting. European Journal of Operational Research , 184(3):1140-1154, 2008.

Vitor Cerqueira, Luis Torgo, and Carlos Soares. Machine learning vs statistical methods for time series forecasting: Size matters. arXiv preprint arXiv:1909.13316 , 2019.

Chris Chatfield. Time-series forecasting . CRC press, 2000.

Vinay Kumar Reddy Chimmula and Lei Zhang. Time series forecasting of covid-19 transmission in canada using lstm networks. Chaos, Solitons & Fractals , 135:109864, 2020.

Junyoung Chung, Caglar Gulcehre, KyungHyun Cho, and Yoshua Bengio. Empirical evaluation of gated recurrent neural networks on sequence modeling. arXiv preprint arXiv:1412.3555 , 2014.

Javier Contreras, Rosario Espinola, Francisco J Nogales, and Antonio J Conejo. Arima models to predict next-day electricity prices. IEEE transactions on power systems , 18(3):1014-1020, 2003.

Tri Dao, Dan Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. Flashattention: Fast and memory-efficient exact attention with io-awareness. In Proceedings of the Annual Conference on Neural Information Processing Systems (NeurIPS) , volume 35, pages 16344-16359, 2022.

Jan G De Gooijer and Rob J Hyndman. 25 years of time series forecasting. International journal of forecasting , 22(3):443-473, 2006.

Hassan Ismail Fawaz, Germain Forestier, Jonathan Weber, Lhassane Idoumghar, and PierreAlain Muller. Transfer learning for time series classification. In 2018 IEEE international conference on big data (Big Data) , 2018.

Simon James Fong, Gloria Li, Nilanjan Dey, Rubén González Crespo, and Enrique HerreraViedma. Finding an accurate early forecasting model from small dataset: A case of 2019-ncov novel coronavirus outbreak. arXiv preprint arXiv:2003.10776 , 2020.

Centers for Disease Control and Prevention. National, regional, and state level outpatient illness and viral surveillance. https://gis.cdc.gov/grasp/fluview/fluportaldashboard. html , 2020.

Carl Friedrich Gauss. Theoria motus corporum coelestium in sectionibus conicis solem ambientium , volume 7. FA Perthes, 1877.

Shahram Hanifi, Xiaolei Liu, Zi Lin, and Saeid Lotfian. A critical review of wind power forecasting methods-past, present and future. Energies , 13(15):3764, 2020.

Hrayr Harutyunyan, Hrant Khachatrian, David C Kale, Greg Ver Steeg, and Aram Galstyan. Multitask learning and benchmarking with clinical time series data. Scientific data , 6(1):96, 2019.

Hansika Hewamalage, Christoph Bergmeir, and Kasun Bandara. Global models for time series forecasting: A simulation study. Pattern Recognition , 124:108441, 2022.

Noah Hollmann, Samuel Müller, Katharina Eggensperger, and Frank Hutter. Tabpfn: A transformer that solves small tabular classification problems in a second. In Proceedings of the International Conference on Learning Representations (ICLR) , 2023.

Rob J Hyndman and George Athanasopoulos. Forecasting: principles and practice . OTexts, 2018.

Hassan Ismail Fawaz, Germain Forestier, Jonathan Weber, Lhassane Idoumghar, and PierreAlain Muller. Deep learning for time series classification: a review. Data mining and knowledge discovery , 33, 2019.

Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 , 2014.

Nikita Kitaev, Lukasz Kaiser, and Anselm Levskaya. Reformer: The efficient transformer. In Proceedings of the International Conference on Learning Representations (ICLR) , 2020.

Bjoern Krollner, Bruce J Vanstone, Gavin R Finnie, et al. Financial time series forecasting with machine learning techniques: a survey. In ESANN , 2010.

Guokun Lai, Wei-Cheng Chang, Yiming Yang, and Hanxiao Liu. Modeling long-and short-term temporal patterns with deep neural networks. In The 41st international ACM SIGIR conference on research & development in information retrieval , pages 95-104, 2018.

Bryan Lim and Stefan Zohren. Time-series forecasting with deep learning: a survey. Philosophical Transactions of the Royal Society A , 379(2194):20200209, 2021.

Amal Mahmoud and Ammar Mohammed. A survey on deep learning for time-series forecasting. Machine Learning and Big Data Analytics Paradigms: Analysis, Applications and Challenges , pages 365-392, 2021.

Spyros Makridakis, Evangelos Spiliotis, and Vassilios Assimakopoulos. The m4 competition: 100,000 time series and 61 forecasting methods. International Journal of Forecasting , 36(1):5474, 2020.

Duncan McElfresh, Sujay Khandagale, Jonathan Valverde, Ganesh Ramakrishnan, Vishak Prasad, Micah Goldblum, and Colin White. When do neural nets outperform boosted trees on tabular data? In Proceedings of the Annual Conference on Neural Information Processing Systems (NeurIPS) , 2023.

Samuel Müller, Noah Hollmann, Sebastian Pineda Arango, Josif Grabocka, and Frank Hutter. Transformers can do bayesian inference. In Proceedings of the International Conference on Learning Representations (ICLR) , 2022.

Maria Elena Nenni, Luca Giustiniano, and Luca Pirolo. Demand forecasting in the fashion industry: a review. International Journal of Engineering Business Management , 5(Godište 2013):5-36, 2013.

Boris N Oreshkin, Dmitri Carpov, Nicolas Chapados, and Yoshua Bengio. N-beats: Neural basis expansion analysis for interpretable time series forecasting. arXiv preprint arXiv:1905.10437 , 2019.

Boris N Oreshkin, Dmitri Carpov, Nicolas Chapados, and Yoshua Bengio. Meta-learning framework with applications to zero-shot time-series forecasting. In Proceedings of the AAAI Conference on Artificial Intelligence (AAAI) , 2021.

Bernardo Pérez Orozco and Stephen J Roberts. Zero-shot and few-shot time series forecasting with ordinal regression recurrent neural networks. arXiv preprint arXiv:2003.12162 , 2020.

Jaideep Pathak, Shashank Subramanian, Peter Harrington, Sanjeev Raja, Ashesh Chattopadhyay, Morteza Mardani, Thorsten Kurth, David Hall, Zongyi Li, Kamyar Azizzadenesheli, et al. Fourcastnet: A global data-driven high-resolution weather model using adaptive fourier neural operators. arXiv preprint arXiv:2202.11214 , 2022.

Caltrans PeMS. Caltrans performance measurement system (pems). US DOT , 2021.

Syama Sundar Rangapuram, Matthias W Seeger, Jan Gasthaus, Lorenzo Stella, Yuyang Wang, and Tim Januschowski. Deep state space models for time series forecasting. Advances in neural information processing systems , 31, 2018.

David Salinas, Valentin Flunkert, Jan Gasthaus, and Tim Januschowski. Deepar: Probabilistic forecasting with autoregressive recurrent networks. International Journal of Forecasting , 2020.

Omer Berat Sezer, Mehmet Ugur Gudelek, and Ahmet Murat Ozbayoglu. Financial time series forecasting with deep learning: A systematic literature review: 2005-2019. Applied soft computing , 90:106181, 2020.

Taylor G. Smith et al. pmdarima: Arima estimators for Python, 2017-. [Online; accessed <today>].

Yi Tay, Mostafa Dehghani, Dara Bahri, and Donald Metzler. Efficient transformers: A survey. ACM Computing Surveys , 2022.

Sean J Taylor and Benjamin Letham. Forecasting at scale. The American Statistician , 2018.

Artur Trindade. ElectricityLoadDiagrams20112014. UCI Machine Learning Repository, 2015. DOI: https://doi.org/10.24432/C58C86.

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems , pages 5998-6008, 2017.

Manuel Weber, Maximilian Auch, Christoph Doblander, Peter Mandl, and Hans-Arno Jacobsen. Transfer learning with time series data: a systematic mapping study. Ieee Access , 2021.

Ruofeng Wen, Kari Torkkola, Balakrishnan Narayanaswamy, and Dhruv Madeka. A multihorizon quantile recurrent forecaster. arXiv preprint arXiv:1711.11053 , 2017.

Haixu Wu, Jiehui Xu, Jianmin Wang, and Mingsheng Long. Autoformer: Decomposition transformers with auto-correlation for long-term series forecasting. Proceedings of the Annual Conference on Neural Information Processing Systems (NeurIPS) , 2021.

Rui Ye and Qun Dai. Implementing transfer learning across different datasets for time series forecasting. Pattern Recognition , 2021.

Haoyi Zhou, Shanghang Zhang, Jieqi Peng, Shuai Zhang, Jianxin Li, Hui Xiong, and Wancai Zhang. Informer: Beyond efficient transformer for long sequence time-series forecasting. In Proceedings of the AAAI Conference on Artificial Intelligence (AAAI) , 2021.

Jingbo Zhou, Xinjiang Lu, Yixiong Xiao, Jiantao Su, Junfu Lyu, Yanjun Ma, and Dejing Dou. Sdwpf: A dataset for spatial dynamic wind power forecasting challenge at kdd cup 2022. arXiv preprint arXiv:2208.04360 , 2022.

Tian Zhou, Ziqing Ma, Qingsong Wen, Xue Wang, Liang Sun, and Rong Jin. Fedformer: Frequency enhanced decomposed transformer for long-term series forecasting. In Proceedings of the International Conference on Machine Learning (ICML) , 2022.

A Additional Related Work

Time-series forecasting [9] has a multitude of applications such as healthcare [10, 20], climate science [5, 38], business [7, 34], finance [27, 42], and renewable energy [19, 53]. A variety of approaches have been used for time-series forecasting [23, 29], including traditional statistical methods [4, 43] and deep learning methods [50, 52, 54].

Forecasting. ARIMA [4] is a statistical method from the 1970s that is still popular today, which builds an autoregressive model based on Markov processes. Another popular statistical method in practice is Prophet [45], which incorporates non-linear trends and multi-scale seasonality. Recently, deep learning methods for time series forecasting have become popular [29]. While early deep learning methods made use of RNNs [11, 41], transformer models have become popular more recently [26, 50, 52], following the success of transformers on NLP [47]. FEDformer [54] incorporates Fourier transforms and the seasonal trend decomposition method into a transformer architecture. For a survey on deep learning methods for time-series forecasting, see [29, 30].

Many real-world forecasting applications have very few initial observations, sometimes just forty or fewer [6, 8, 16, 23]. However, there is much less work in this setting (referred to as 'zero-shot' forecasting [36]). Oreshkin et al. [36] use a model based off of N-BEATS [35], a model that was state of the art at the time, training on a (single) real-world time series dataset and testing on a different time series. However, it achieves substantially different performance based on which training dataset is used. Another zero-shot forecasting method [37] uses an RNN base, yet it is only empirically compared to traditional methods and on a fixed prediction length.

There is some work on transfer learning for time-series forecasting, but it is still limited [24]. Earlier works study transfer learning by pretraining a CNN architecture for forecasting on one time series, and fine-tuning on another time series [15, 51]. Fawaz et al. showed that some datasets can improve performance, while others can degrade performance, and they introduce a new method to predict the best source dataset. More recent work uses LSTMs and CNN+RNN hybrid models, to achieve better results [48]. Global Forecasting Models (GFM) are a new type of forecasting model in which a single model is trained on all series from a single dataset [21, 40, 41, 49]. However, a key assumption is that these time series must be related in some way. For example, a retail company may train a single model for the time series of sales of thousands of different products.

Prior-Data Fitted Networks. Prior-data fitted networks (PFNs) are a recently-proposed paradigm for machine learning, in which a model is trained offline on small, synthetic datasets. Once trained, a PFN can infer the posterior predictive distribution (PPD) for test datapoints, given a new dataset, in a single forward pass.

Müller et al. [33] first introduced PFNs, showing that they provably approximate Bayesian inference, and demonstrating performance on very small datasets. Soon after, TabPFN [22] showed stateof-the-art empirical performance on datasets of up to size 1000 [32]. This new empirical finding was accomplished in part due to the synthetic prior, consisting of a large set of structural causal models. Recently, Adriaensen et al. [2] designed a zero-shot approach using a PFN for learning curve extrapolation; however, the setting for learning curve extrapolation is significantly different and arguably a special case of time-series forecasting. To the best of our knowledge, we are the first to design a zero-shot time-series forecasting model trained purely on synthetic data.

B Broader Societal Impact

We do not see any strongly negative broader societal impacts of this work. In fact, our hope is that this work will reduce the computational cost and CO2 emissions for forecasting. Although ForecastPFN takes 30 GPU hours to train, this is a one-time cost, which we have already done. Practitioners can now download and use our model, rather than training methods such as FedFormer and Informer from scratch. In general, we hope that our work encourages practitioners and researchers to conduct zero-shot (or few-shot) forecasting, rather than training models from scratch for each new dataset.

C Reproducibility

Our codebase is available at https://github.com/abacusai/forecastpfn . The README to the codebase contains instructions for installation and immediately using ForecastPFN to make predictions on a new dataset. We also include all code from this project, including the code for synthetic data generation, ForecastPFN training, and evaluation.

D Synthetic Data Details

We give the full details of the synthetic data series, including the parameters for daily, weekly, and monthly series.

Recall that the time series has a trend component, seasonal component, and noise component. The trend component is made up of a linear and exponential component with coefficients m lin , c lin , m exp , c exp. The seasonal component has a week, month, and year component where each comprises of of coefficients p week , p month , p year Finally, the noise in our model is derived from a Weibull distribution and is modeled such that the expected value of the noise model is 1, meaning in expectation the noise does not contribute to the seasonality or trend of the time series. Formally,

Now we explain how to set all unspecified parameters above, for daily, weekly, and monthly series. For convenience, when creating a daily, weekly, or monthly series, a unit of time t is equal to 1, 7, or 30.5 days (which would be equivalent to scaling all the parameters below by 1, 7, or 30.5).

For all pairs of ν and f , c f,ν and d f,ν are both drawn from N (0 , 1 /f ) , such that these coefficients are inversely proportional to the harmonics of the series, and then they are evenly rescaled so that the sum of their squares equals 1.

We set each m ν for ν ∈ { week , month , year } separately for daily, weekly, and monthly, as follows. m ν = 0 unless specified:

Monthly: m year ∼ U ([0 , . 5]) , p year = 12 .

Finally,

As described in Section 3.3 and the next section (Appendix E.1), we set these parameters to create the most diverse data distribution possible, with the model still being able to train without diverging or stalling. For example, when the parameter for the noise distribution becomes too large, the model cannot converge. We also released a README for our synthetic data generation at https: //github.com/abacusai/forecastpfn .

For visualizations of daily time series drawn from this distribution, see Figure 2 and Figure 7.

E Additional Experimental Details

E.1 ForecastPFN Development

We estimate that the total computational budget used in this project, from initial research to running all the final evaluations, is 500 GPU hours. Our computation consisted mainly of designing the architecture and designing the synthetic data distribution. In this design phase, we used the train loss of the synthetic data as a signal, because many synthetic data distributions, and architectures, cause the model not to train, for example, setting the noise parameter too high. We also used the synthetic held-out validation set to make decisions, such as in Figure 5 and Appendix E.5.

The training of our ForecastPFN model is 30 GPU hours on a single Tesla V100 16GB GPU. We emphasize that this model training is only done once, and we released the trained ForecastPFN model.

For an ablation study, we tried to achieve non-trivial zero-shot performance with the FedFormer architecture. However, after significant effort, we were unable to achieve non-degenerate performance with the FedFormer architecture.

E.2 Implementation Details

ForecastPFN Implementation Details. We use an encoder-based transformer architecture, with two encoders. Each encoder consists of a multi-head attention layer with four heads, and two feedforward layers. The first feedforward layer is size 32 times the embedding dimension, and the second feedforward layer is size 8 times the embedding dimension.

Baseline Implementation Details. We compare ForecastPFN against ARIMA [4], Autoformer [50], FEDformer [54], Informer [52], Meta-N-BEATS [36], Prophet [45], and a vanilla Transformer method [47]. All of the methods were implemented using their default hyperparameters and official codebase. For ARIMA (which was designed in the 1970s and has no 'official' codebase), we used pmdarima [43]. The full implementation details of all methods can be found in our open source codebase. Furthermore, we compared to three baselines: Last (prediction is the last seen value in the series), Mean (prediction is the mean value in the series), and SeasonalNaive (prediction is the mean value from all prior instances of the same day of the week).

In Section 4, in the experiments with a restricted time budget, we restrict the time budget of the non-zero-shot methods to wall clock budgets from 1 to 120 seconds, where timing is calculated after data loading and only during the training loop. After each training step, we stop the training if the total budget was exceeded. In case an algorithm is unable to output any predictions within the data budget (such as ARIMA with a 1 second budget), we set the output to 0's.

E.3 Training Details

We generate 100 000 each of synthetic daily, weekly, and monthly series, each of length 200, and we use a sliding window of size 100 to obtain 101 prediction tasks per series. We sample 1024 tasks in a training step, and there are 1000 training steps in an epoch. We trained the transformer for 300 epochs on a V100 16GB GPU, which took 30 hours.

We use the Adam optimizer with a learning rate of 0 . 0001 , and MSE loss. We set the maximum prediction into the future to be 10 times the frequency of the training series.

E.4 Dataset Details

We use seven datasets for our experiments in Section 4. These datasets are standard for benchmarking the performance of forecasting methods and are the same ones used by the majority of our baselines [36, 50, 52, 54].

Illness is a dataset [17] containing influenza-like illness patients in the United States. It contains 966 datapoints, spaced weekly. Exchange is a currency exchange dataset [28], consisting of 7588 datapoints, spaced daily. Traffic is a dataset consisting of freeway congestion across California [39]. ETT (Electricity Transformer Temperature) is a electricity power deployment dataset [52] across 2 years from a power plant in China. The target value is oil temperature. Two separate series

are provided: ETT1 and ETT2. Each series represents data collected from two different electicity transformer stations. ECL (Electricity Consuming Load) [46] is an electricity consumption (Kwh) dataset across two years with a resolution of every hour. This dataset has a total of 26 304 hourly observations. Weather is a dataset [1] that contains local climate data from Germany, across three years. The datasets ETT, ECL, and Weather all natively have a sub-day resolution which we aggregate into daily observations by taking the sum of values in day.

E.5 Noise Removal Ablation

Recall that crucially, since we use synthetic training data with a ground-truth trend and a separate noise component, we remove the noise when calculating the train loss. Now, we give an ablation study to confirm that this design decision significantly improves the performance of our model.

We investigate the impact of this design decision on the performance of ForecastPFN by comparing the model with the noise removed (that which is included in the main body of the paper) with a training run conducted without the removal of the noise from the training loss calculation. In doing this, we calculate the validation MSE on the synthetic data for both models and observe 33.089 for the model without noise removed and 22.692 for the model with the noise removed. We conclude the use of the noise removal during training loss calculation leads to improvements on the synthetic validation set, and thus we deploy the use of noise removal in the final model.

This design decision significantly improves performance, and it can only be done when using synthetic data , rather than real data.

E.6 Training Time Analysis

Recall that during our benchmark, there were six models which required training data on a portion of each of the datasets before they were able to be tested. Further recall that we established a time budget for this training process, and varied that timing from 1 to 120 seconds. Some of these models were unable to fit to the trainig data during some of those training budgets. In Table 3, we calculate the percentage of configurations (dataset, prediction lengths, seeds) which failed for each model and training time budget. We see that Arima is the model which struggles the most at low time budgets.

E.7 Additional Experimental results

In Figure 16 and Figure 6, we plot the mean MSE rank per prediction length (similar to Figure 3 and 4 respectively, but for mean MSE rank instead of MSE wins).

In Figures 8 to 20, we give similar plots to Figures 3, 4 and 6 but instead of MSE, we plot MAE, MAPE, and MSPE. We also include plots for the evaluation of mean rank.

10

20

30

40

50

2.5

5

7.5

50

10

20

30

40

50

100

10

20

30

40

50

150

10

20

30

40

50

200

10

20

30

40

50

250

10

20

30

40

50

300

10

20

30

40

50

500

Mean MSE Rank per Prediction Length By Data Budget (50 to 500)

Prediction Length

Mean MSE Rank

Arima

Autoformer

FEDformer

ForecastPFN

Informer

Last

Mean

Meta-N-BEATS

Prophet

SeasonalNaive

Transformer

Figure 6: Analysis of performance across prediction lengths for increasing data budgets (left subplot, data budget is 50 to the right subplot, data budget is 500), aggregated across datasets. Mean MSE rank for each scenario is plotted with error bars as one standard deviation across training runs. Recall, ForecastPFN and Meta-N-BEATS see no training data for these series, only the length 36 input.

Seasonal Trend

Global Trend

2.5

Noise

0.09

Full Time Series

1.2

0.04

0.07

2

1.1

0.03

0.02

1.5

0.05

0.01

0.03

0.9

0

MonthlyTrendWeeklyTrend

-Exponential—Global Trend-Linear

008801-220888-02.22301

088801-22338323.01

Dates

Dates

Dates

Dates

Seasonal Trend

Global Trend

Noise

Full Time Series

1.4

1.2

2

1.2

1

1.1

1.5

1

0.5

0.8

0

0.Monthly Trend Weekly Trend

0.5

Exponential—Global Trend—Linear

8801-22383-88-2.223-01

3888-01-22303-88-02.223-01

80-01-22-03-03-83

202029

2022022022020202

Dates

Dates

Dates

Dates

Seasonal Trend

Global Trend

Noise

Full Time Series

1.050

1.2

0.7

1.5

1.025

1.1

0.6

0.5

0.9

0.975

0.4

0.5

0.8

0.950

0.3

MonthlyTrendWeeklyTrend

Exponential—Global Trend—Linear

88801-2238813.223-01

-223-01

202902902022020-

2029

20

2029

Dates

Dates

Dates

Dates

Seasonal Trend

Global Trend

Noise

Full Time Series

0.2

1.1

1.05

0.25

0.1

T

0.9

0

0.20

0.8

0.1

0.15

0.7

MonthlyTrend—WeeklyTrend

Exponential—Global TrendLinear

0.95

088801-2230888-3.23-01

18-801-2203-83-88-02.223-01

0-81-88-01-2202-03-88-03-223-01

801-2202-03-88-03.223-01

2022028

20220220202020

Dates

Dates

Dates

Dates

Seasonal Trend

Global Trend

Noise

Full Time Series

1.25

1.02

1.4

1

1.01

1.2

0.75

1.5

0.50

0.99

0.8

0.25

0.98

Exponential-Global Trend-Linear

881-22388132301

2020-

Dates

Dates

Dates

Dates

50

100

300

500

0

5

10

15

Number of total MAE Wins per Data Budget

Data Budget

MAE Wins

Model

Arima

Autoformer

FEDformer

ForecastPFN

Informer

Last

Mean

Meta-N-BEATS

Prophet

SeasonalNaive

Transformer

50

100

300

500

4

6

8

Mean MAE Rank per Data Budget

Data Budget

Mean MAE Rank

1

10

100

0

5

10

15

Number of total MAE Wins per Time Budget

Time Budget

MAE Wins

Model

Arima

Autoformer

FEDformer

ForecastPFN

Informer

Last

Mean

Meta-N-BEATS

Prophet

SeasonalNaive

Transformer

1

10

100

2

4

6

8

Mean MAE Rank per Time Budget

Mean MAE Rank

Time Budget

MAE Wins per Prediction Length By Data Budget (50 to 500)

10

20

30

40

50

0

1

2

3

4

50

10

20

30

40

50

100

10

20

30

40

50

150

10

20

30

40

50

200

10

20

30

40

50

250

10

20

30

40

50

300

10

20

30

40

500

Prediction Length

MAE Wins

Arima

Autoformer

FEDformer

ForecastPFN

Informer

Last

Mean

Meta-N-BEATS

Prophet

SeasonalNaive

Transformer

50

10

20

30

40

50

2.5

5

7.5

50

10

20

30

40

50

100

10

20

30

40

50

150

10

20

30

40

50

200

10

20

30

40

50

250

10

20

30

40

50

300

10

20

30

40

50

500

Mean MAE Rank per Prediction Length By Data Budget (50 to 500)

Prediction Length

Mean MAE Rank

Arima

Autoformer

FEDformer

ForecastPFN

Informer

Last

Mean

Meta-N-BEATS

Prophet

SeasonalNaive

Transformer

50

100

300

500

0

5

10

Number of total MAPE Wins per Data Budget

Data Budget

MAPE Wins

Model

Arima

Autoformer

FEDformer

ForecastPFN

Informer

Last

Mean

Meta-N-BEATS

Prophet

SeasonalNaive

Transformer

50

100

300

500

4

6

8

Mean MAPE Rank per Data Budget

Mean MAPE Rank

Data Budget

1

10

100

0

5

10

Number of total MAPE Wins per Time Budget

Time Budget

MAPE Wins

Model

Arima

Autoformer

FEDformer

ForecastPFN

Informer

Last

Mean

Meta-N-BEATS

Prophet

SeasonalNaive

Transformer

1

10

100

4

6

8

Mean MAPE Rank per Time Budget

Time Budget

Mean MAPE Rank

10

20

30

40

50

0

1

2

3

50

10

20

30

40

50

100

10

20

30

40

50

150

10

20

30

40

50

200

10

20

30

40

50

250

10

20

30

40

50

300

10

20

30

40

50

500

MAPE Wins per Prediction Length By Data Budget (50 to 500)

Prediction Length

MAPE Wins

Arima

Autoformer

FEDformer

ForecastPFN

Informer

Last

Mean

Meta-N-BEATS

Prophet

SeasonalNaive

Transformer

10

20

30

40

50

2.5

5

7.5

10

50

10

20

30

40

50

100

10

20

30

40

50

150

10

20

30

40

50

200

10

20

30

40

50

250

10

20

30

40

50

300

10

20

30

40

50

500

Mean MAPE Rank per Prediction Length By Data Budget (50 to 500)

Prediction Length

Mean MAPE Rank

Arima

Autoformer

FEDformer

ForecastPFN

Informer

Last

Mean

Meta-N-BEATS

Prophet

SeasonalNaive

Transformer

50

100

300

500

2

4

6

8

Mean MSE Rank per Data Budget

Data Budget

Mean MSE Rank

1

10

100

2

4

6

8

Mean MSE Rank per Time Budget

Time Budget

Mean MSE Rank

50

100

300

500

0

5

10

15

Number of total MSPE Wins per Data Budget

Data Budget

MSPE Wins

Model

Arima

Autoformer

FEDformer

ForecastPFN

Informer

Last

Mean

Meta-N-BEATS

Prophet

SeasonalNaive

Transformer

50

100

300

500

4

6

8

Mean MSPE Rank per Data Budget

Data Budget

Mean MSPE Rank

1

10

100

0

5

10

Number of total MSPE Wins per Time Budget

Time Budget

MSPE Wins

Model

Arima

Autoformer

FEDformer

ForecastPFN

Informer

Last

Mean

Meta-N-BEATS

Prophet

SeasonalNaive

Transformer

1

10

100

4

6

8

Mean MSPE Rank per Time Budget

Time Budget

Mean MSPE Rank

10

20

30

40

50

0

1

2

3

4

50

10

20

30

40

50

100

10

20

30

40

50

150

10

20

30

40

50

200

10

20

30

40

50

250

10

20

30

40

50

300

10

20

30

40

50

500

MSPE Wins per Prediction Length By Data Budget (50 to 500)

Prediction Length

MSPE Wins

Arima

Autoformer

FEDformer

ForecastPFN

Informer

Last

Mean

Meta-N-BEATS

Prophet

SeasonalNaive

Transformer

10

20

30

40

50

2.5

5

7.5

50

10

20

30

40

50

100

10

20

30

40

50

150

10

20

30

40

50

200

10

20

30

40

50

250

10

20

30

40

50

300

10

20

30

40

50

500

Mean MSPE Rank per Prediction Length By Data Budget (50 to 500)

Prediction Length

Mean MSPE Rank

Arima

Autoformer

FEDformer

ForecastPFN

Informer

Last

Mean

Meta-N-BEATS

Prophet

SeasonalNaive

Transformer

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe work present the ForecastPFG a zero-shot forecasting method trained using only synthetic dataset and it is evaluated on several real world dataset.\n\nSTRENGTHS:\nThe paper is well written and the steps well described.\nMoreover, I think that it could be an interesting approach when you have very few data.\n\nWEAKNESSES:\nWeakness are inserted in the Question section\n\nQUESTIONS:\nRows 71-74: This a repetition of what authors wrote in rows 16-20. This part should be improved with some details. The same in rows 83-85 and rows 23-24.\n\nThe relation between Zero shot forecasting, Transfer Learning in the forecasting context and the concept of Global Models is not discussed properly. I suggest authors to discuss more about this relation. \nRegarding global models authors can refer to recent works as: Januschowski et al., \"Criteria for classifying forecasting methods\", 2020; Buonanno et al., \"Global vs. Local Models for Short-Term Electricity Demand Prediction in a Residential/Lodging Scenario\", 2022; Montero-Manso, P.; Hyndman, R.J. \"Principles and algorithms for forecasting groups of time series: Locality and globality\", 2021, etc.\n\nRows 102-104\nAuthors use a multiplicative decomposition ($y_t = \\gamma(t) \\cdot z_t$). Why this choice and not the additive decomposition ($y_t = \\gamma(t) + z_t$)? How this choice impact on the results of the work?\n\nRow 109-110\nCould you clarify this tense? The $y_t = \\gamma(t) \\cdot z_t$ is always valid not only when $\\gamma(t)$ is deterministic. Moreover, if it is present $z_t$ sampled by a Gaussian, there are different possible output values and not only $y_T$. Do I miss something?\n\nRows 113-114\nThe athors wrote:\n\"We optimize the PFN’s parameters $\\theta$ by making predictions on a held-out test set from D\"--> Usually with test set is named the set used exclusively for the evaluation and that is not involved in learning of the model. In this case authors define a test set used to evaluate the loss used for the parameter tuning. I think that the name is misleading. Authors can evaluate to use different names (e.g. Training Input and Training Output) to identify the two parts of the timeseries used in the training phase.\n\nRows 129.\nIs not crucial to know the prior distribution of timeseries in order to have a good result on real dataset? In this work authors use a large dataset of timeseries that try to cover different situations but how the choice of the prior distribution impact on the results? Maybe this can be a idea for future works.\n\nRows 144\nIs there a motivation on the usage of Weibull distribution for the noise?\nHow a choice of a different noise distribution impacts on the results?\n\nRows 146-147\nNow $\\gamma(t)$ contains two contributions: trend and seasonal. In the row 102, instead, is written only trend. I got that in 102 authors want to introduce the general frame of the problem but I would not use the name trend in row 102.\n\nRows 169\n\"Predict the MSE of this query\" --> Does not the architecture predict the output? The next rows in fact the authors wrote \"the output is a single prediction at a future time\".\nAlso at row 178 it is written \"the loss\" and not the output.\n\nRows 180-183 \nIt is not clear to me why the classic scaling techniques are not appliable in this case. As discussed after, the problem is the outlier that has to be excluded since they can be very different across several timeseries. Please express better this issue.\n\nRows 188-191\nIs MSPE the mean squared prediction error? \nCould you describe mathematically the issue raised in this section because I didn't get it.\n\nRow 196\nHow authors evaluate the convergence?\n\nRow 235-236\nNot always the forecasting models are evaluated in retraining mode. For having a fairer comparison authors could force the forecasting model to not retrain during the evaluation test. This condition is, in fact, more similar to what happen in the real context where you don't retrain continuously your model once it is deployed. Moreover, this modality (no retrain) is followed in several works about forecasting. Using this modality (no retrain) we should have a fairer comparison between ForecastPFN and other models.\n\nRow 255\nCross product --> do you mean cartesian product?\n\nFigure 4\nIs the Time Budget the same of wall clock budgets defined in row 243\n\nRow 301\nIs not more important to compare classic standardization (without outlier removal) with robust scaling? The authors, instead, compared robust scaling with min-max scaling.\n\nLIMITATIONS:\nA limitation I see is the maximum number of training data used for the other models, limited to 500. It looks clear that with more other models outperform the ForecastPFN (Fig.5) while with few data ForecastPFN looks a good solution.\nMoreover, the training of ForecastPFN requires 30hr so it has a cost that should be taken in account for having a fair comparison with other approaches.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-24"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes to pre-train a deep learning model on synthetic data which follows characteristics of real world time series data. This model can then be used for any downstream forecasting dataset. The authors performed experiments to show that the proposed method performs well compared to existing classical and deep forecasting models in constrained scenarios (restricted number of training data, restricted amount of time for training).\n\nSTRENGTHS:\nThe paper proposes an exciting direction of pre-training deep forecasting models with synthetic data, which can be quickly adapted in a Bayesian manner on a new unseen real time series, and be able to perform accurate forecasts. The synthetic data is proposed using a prior distribution which has properties similar to real world data. Such a direction is exciting and of huge significance, unlocking the power of deep learning models to be pre-trained on large datasets. The paper is well written, neatly organised in a logical flow, and contains comprehensive experiments.\n\nWEAKNESSES:\n* While the premise of the paper is indeed exciting, I believe the claims made in the abstract and introduction are greatly exaggerated. Claims are made that the proposed method, ForecastPFN is able to beat SOTA forecasting methods. However, the experiments are only performed on severely handicapped scenarios (SOTA models are only given access to 100s of data points or maximum of 2 minutes of training). More effort should be made to highlight this handicapped scenario.\n* Claims that \"the zero shot methods are at a disadvantage because the other six methods were allowed to train on data from the  evaluation time series\" (lines 280 - 282) seem a little disingenuous since the zero shot methods were pre-trained extensively (30 hours of pre-training for ForecastPFN), whereas in certain cases the six methods were only allowed 1 second of training?\n* The claim in line 173 - 175, that existing transformer models have prediction length to be fixed is inaccurate. This is a feature of Transformers in the long sequence time series forecasting setting, but not true of time series Transformers in general. See [1] for one such example.\n* Only number of wins/ranking is given for the results, but not the actual MSE scores. The actual MSE values should be given for readers to understand whether the comparisons are meaningful at all.\n* Following up on the previous point, 2 more very simple baselines should be added - the naive forecast (current value is taken as the forecast) and the historical mean. The number of wins/rankings doesn't really matter if any of these models can't beat such naive baselines (which is a fair comparison regardless of the train/time budget).\n\n[1] Rasul, Kashif, et al. \"VQ-TR: Vector Quantized Attention for Time Series Forecasting.\" (2022).\n\nQUESTIONS:\n1. Are the existing methods (FEDformer, etc.) also pre-trained on the synthetic data? If not, it seems quite artificial to claim wins when these methods are only given 50 datapoints (1 batch?) for training, or 1 second to train. In fact, it seems unfair to compare to ForecastPFN which has 30 hours of pre-training on a huge synthetic dataset.\n2. How was Meta-N-BEATS trained on the M4 dataset? In the paper, 1 model was trained for each frequency. Which subset of M4 was the baseline trained on?\n3. What does number of MSE wins mean?\n4. What does training on 1 second mean? How is this implemented? Was a single gradient step even performed in this setting?\n5. What are the experiment details of the baseline models? What machine was used to run these baseline experiments? It seems a little iffy to compare the time budget in terms of seconds. How to compare 120 seconds on a H100 GPU vs a K80 GPU or even a CPU?\n6. How was the time restricted training for ARIMA/Prophet implemented? I understand for deep learning models you can easily stop the training, but what about ARIMA/Prophet if the model has not been fitted?\n\nPlease consider using number of batches/epochs trained instead of wall clock seconds.\n\nLIMITATIONS:\nThe paper doesn't adequately address limitations. The limitations given in the limitations section aren't really true limitations, but rather things that are out of scope of this paper. Some suggestions of limitations to consider:\n1. What if the proposed prior for synthetic data is completely different from the downstream real world data? Perhaps some sort of adversarial forecasting task?\n2. ForecastPFN doesn't compute the posterior predictive distribution, it is just an approximation.\n3. Does not consider probabilistic forecasting, which has more implications on the synthetic prior (not just the shape of time series, but also the distribution of data, need to consider some autocorrelation in the errors)", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 1, "DATE": "2023-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a zero-shot prior-data fitted network (PFN) for time-series forecasting. Existing works have challenges in designing a general and flexible PFN for general time-series distributions, and tuning an architecture and training scheme. This work overcomes these by designing a novel synthetic time-series distribution in training and using a transformer architecture for queries to arbitrary future timestep. Experiments have been presented to demonstrate the improved performance of the proposed model over comparison models.\n\nSTRENGTHS:\n1. The proposed work tackles an interesting setting where few initial observations are available in forecasting.\n2. The transformer-based network aims to predict arbitrary future time steps and the robust scaling deals with the extreme scale of time series, which increase the novelty of the methodology. \n3. The results seem compelling as well, with a clear outperformance of the proposed model over comparison baselines.\n\nWEAKNESSES:\n1. In Section 3.2, it is not clear what motivates the author to choose a multiplicative data generation model with seasonal and trend components as the synthetic prior. It would be appreciated if the author could elaborate more on that. Also, the ablation study could be more sufficient: the diversity of the synthetic data generated for training should be explored since the model relies on a general synthetic data distribution. \n2. The author could provide more details about the architecture in Section 3.3: 1) How does the network learn diverse temporal features of the input data? 2) How does the network deal with different input sizes? 3) What is the benefit of using transformer-based layers vs residual blocks or recurrent neural networks?\n3. In Section 4, the author should explain the definition of train budget and time budget. Are highlighted texts “training time” and “training data” related to them? It would be better to make them consistent.\n\nQUESTIONS:\nPlease see the weaknesses mentioned above.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nForecastPFN is a zero-shot forecasting model designed to overcome the limitations of traditional forecasting methods when dealing with data-sparse applications. Unlike most approaches, ForecastPFN is trained solely on synthetic data, which captures diverse time series patterns and incorporates multi-scale seasonal trends, global trends, and noise. This prior-data fitted network approximates Bayesian inference and enables accurate and fast predictions in a single pass. Extensive experiments presented in the paper demonstrate that ForecastPFN outperforms state-of-the-art methods across multiple datasets and forecast horizons, even when the latter are trained with significantly more data points.\n\nSTRENGTHS:\n1. The manuscript is well-written, and the presented ideas are easy to follow.\n2. The concept of training a foundational time series model solely on synthetic data is innovative and exciting. This approach not only addresses the issue of lower forecasting accuracy in data-sparse scenarios but also has positive environmental implications.\n3. The execution of this work is commendable. The paper provides detailed information on the generation of the synthetic dataset, the model architecture, and the training process. Additionally, the authors conducted an extensive set of experiments, exploring various model configurations, prediction horizons, and datasets from multiple domains\n\nWEAKNESSES:\nThe paper predominantly relies on plots to present the results, without including tables. While it is understandable that summarizing all the experiments in tabular form would be challenging due to the extensive nature of the research, it can be difficult for readers to extract meaningful information from a large number of plots. Additionally, certain plots, such as Fig 5, are scaled down to a point where they become challenging to interpret. The same issue extends to the plots presented in the appendix. To enhance clarity, it is recommended that the authors include some key results in tabular form and adjust the scale of the plots to improve readability for readers.\n\nQUESTIONS:\n1. Considering the focus on commercial forecasting applications in this work, it would be valuable to understand the reasons behind modeling ForecastPFN as a point forecaster instead of a probabilistic forecaster.\n2. In real-world forecasting applications, the presence of exogenous features can provide additional information that directly affects the accuracy of forecasts. How does ForecastPFN handle the incorporation of exogenous features?\n3. It would be interesting to know if the authors explored the option of fine-tuning ForecastPFN on the target data. This analysis could provide insights into the impact of fine-tuning on the performance of ForecastPFN in downstream forecasting tasks.\n\nLIMITATIONS:\nSee above", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}