ForecastPFN: Synthetically-Trained Zero-Shot Forecasting
Authors: Samuel Dooley
Gurnoor Singh Khurana
Chirag Mohapatra
Siddartha Venkat Naidu
Colin White
Venue: NeurIPS 2023 poster (2023)
Keywords: Forecasting
Zero-shot
Synthetic Data
TL;DR: We design the first synthetically-trained zero-shot forecasting model, which can outperform state-of-the-art approaches even without seeing training data.

Abstract: The vast majority of time-series forecasting approaches require a substantial training dataset. However, many real-life forecasting applications have very little initial observations, sometimes just 40 or fewer. Thus, the applicability of most forecasting methods is restricted in data-sparse commercial applications. While there is recent work in the setting of very limited initial data (so-called `zero-shot' forecasting), its performance is inconsistent depending on the data used for pretraining. In this work, we take a different approach and devise ForecastPFN, the first zero-shot forecasting model trained purely on a novel synthetic data distribution. ForecastPFN is a prior-data fitted network, trained to approximate Bayesian inference, which can make predictions on a new time series dataset in a single forward pass. Through extensive experiments, we show that zero-shot predictions made by ForecastPFN are more accurate and faster compared to state-of-the-art forecasting methods, even when the other methods are allowed to train on hundreds of additional in-distribution data points.

ForecastPFN: Synthetically-Trained Zero-Shot Forecasting
Samuel Dooley* 1, Gurnoor Singh Khurana', Chirag Mohapatra', Siddartha Naidu', Colin White
Abacus.AI, 2 Caltech
Abstract
The vast majority of time-series forecasting approaches require substantial train- ing dataset. However, many real-life forecasting applications have very little initial observations, sometimes just 40 or fewer. Thus, the applicability of most forecasting methods restricted in data-sparse commercial applications. While there is recent work in the setting of very limited initial data (so-called 'zero-shot' forecasting), its performance is inconsistent depending on the data used for pre- training. In this work, we take a different approach and devise ForecastPFN, the first zero-shot forecasting model trained purely on a novel synthetic data distribu- tion. ForecastPFN is prior-data fitted network, trained to approximate Bayesian inference, which can make predictions on a new time series dataset in a single forward pass. Through extensive experiments, made by ForecastPFN are more accurate and faster compared to state-of-the forecasting methods, even when the other methods are allowed to train on hundreds of additional in-distribution data points.
show that zero-shot predictions
1 Introduction
Time-series forecasting is important and long-studied problem that has attracted significant attention for many years [9, 14, 18, 29]. Time-scries forecasting has a wide variety of applications such as healthcare [10, 20], economies [28, 42], climate science [5, 38), and renewable energy [19, 53). Many forecasting algorithms have been proposed, including traditional statistical methods [4, 12], and more recently, deep learning methods [29, 30, 52, 54].
Nearly time-series forecasting approaches (including both traditional methods and deep learning methods) require substantial number of initial time-series datapoints in order to learn patterns and make future predictions. However, many real-world forecasting applications have very few initial observations, sometimes just 40 or fewer [6. 8, 16. 23]. This setting is deemed 'zero-shot forecasting' in prior work [36], in contrast to the typical setting in which hundreds or thousands of observations from the target series used for training. While there is recent work zero-shot forecasting [36], its performance is inconsistent depending on the dataset used for pretraining.
In this work, we take different approach and devise ForecastPFN, the first zero-shot method trained purely on novel synthetic data distribution. ForecastPFN is a prior-data fitted network (PFN) [33]; PFNs are recently-propost proposed paradigm, in which a model is pretrained offline on synthetic data generated from a prior to approximate Bayesian inference. PFNs have recently been used in breakthrough result for tabular datasets [22]. However, there are significant challenges when designing a general and flexible PFN in the forecasting setting, namely, constructing a general time-series distribution, and tuning architecture and training scheme that can learn it.
"Conespondence to:
ai, crwhite@caltech.c edu.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Traditional setting
ForecastPFN setting Train once, offlie with synthetic, diverse series
Train the same (single)
real-world scries
mmmm
MM
mmmm
mmmm
Trstte
Traato
www
?
Model
ForecastPFN
Test new of the same scries
Zero-shot test on new, unseen, real-world scries
Figure 1: Left: standard forecasting setting, where model trains hundreds of datapoints for time series, and tests on future part of the same series, Right: ForecastPFN setting: a one-time, offline training routine on synthetic data, and then the model can make zero-shot predictions on new time series, without retraining or modifying the architecture weights.
We overcome these challenges first by designing novel synthetic time-series distribution. We carefully design the synthetic distribution to be as general and diverse as possible, with a bias for common patterns seen in real-world series, while still being possible for a machine learning model to learn the distribution. Specifically, we design a modular synthetic model which incorporates multi- scale seasonal trends, linear and exponential global trends, and Weibull-based noise distribution, each of which are parameterized with enough variance to capture a diverse range of time series, while still being possible learn. Furthermore, using synthetic training data consisting of ground- truth trend and a separate noise component means that we can remove the noise in the labels when calculating the train loss, which significantly improves the training speed of our model. Next, design flexible transformer architecture that can scale across a wide range of time-series values. While many existing transformers for forecasting train with a fixed prediction length and only predict the next N timesteps into the future, ForecastPFN is trained to perform arbitrary queries queries to any future timestep allowing achieve zero-shot prediction for arbitrary prediction lengths.
We show that after a one-time, offline training phase on the synthetic dataset, ForecastPFN achieves strong zero-shot performance across wide variety of real-world datasets, fraction of the runtime of other methods. ForecastPFN's zero-shot predictions outperforms state-of-the-ar forecasting methods, when these other methods are allowed to train on hundreds of additional in-distribution datapoints. For example, given a new time series, ForecastPFN makes future predictions after only seeing the input (typically the most-recent & 30 points of the series) and outperforms state-of-the-art methods which were allowed to train on an additional 300 historical points in the series (see Figure I and Figure 3). This remarkable result is due to the structure of our synthetic priors: by encoding multi-scale seasonal trends, global trends, and noise, across a variety of parameters, our model is able learn how forecast general time series. Finally, ForecastPFN is fast. producing predictions on brand new dataset in just single forward pass, taking 0.2 seconds. This is over 100 times faster than existing state-of-the-ar transformer methods, which require expensive training procedures. Our codebase and our model are available at https://gtt thub. com/ wwwcccussffforccsttsoee
Our contributions. We summarize our main contributions below.
We introduce ForecastPFN, a prior-data fitted network for forecasting, trained purely on a novel synthetic data distribution. ForecastPFN is zero-shot: after its initial pretraining, it can make predictions on a brand new dataset with no training data from that dataset.
Through extensive experiments across a variety of datasets, we demonstrate that ForecastPFN, without any retraining, outperforms state-of-the-at methods in low-resource settings, even when the other methods are allowed to train on additional in-distribution datapoints. Furthermore, ForecastPFN fast, making predictions on a new dataset single forward pass.

---

2 Related Work
Time-series forecasting [9] has multitude of applications throughout climate science [5. 38], healthcare [10, 20]. business [7, 34], finance [27, 42], and renewable energy [19, 53]. A variety of approaches have been used for time-series forecasting [23, 291, including traditional statistical methods [4, 43] and deep learning methods [50, 52, 54].
ARIMA [4] is a statistical method from the 1970s that is still popular today, which builds an autoregressive model based on Markov processes. Another popular statistical method in practice is Prophet [45], which incorporates non-linear trends and multi-scale seasonality. Recently, deep learning methods for time series forecasting have become popular [29]. While early deep learning methods made use of RNNs [11, 41], transformer models have become popular more recently [26, 50, 52], following the success of transformers on NLP [47]. FEDformer [54] incorporates Fourier transforms and the seasonal trend decomposition method into a transformer architecture. For a survey on deep learning methods for time-series forecasting, see [29, 30J, and for transfer learning for time-series forecasting. see [48].
Many real-world forecasting applications have very few initial observations, sometimes just forty or fewer [6, 8, 16, 23]. However, there is much less work in this setting (referred to as 'zero-shot' forecasting [36]). Oreshkin et al. [36] use a model based off of N-BEATS [35], model that was state of the art at the time, training on (single) real-world time series dataset and testing on different time series. However, it achieves substantially different performance based on which training dataset used. Another zero-shot forecasting method [37] uses an RNN base. yet is only empirically compared to traditional methods and on fixed prediction length. Recently, Adriaensen et al. [2] designed a zero-shot approach using a PFN for learning curve extrapolation; however, the setting for learning curve extrapolation is significantly different from time-series forecasting. To the best of our knowledge, we are the first to design a zero-shot time-series forecasting model trained purely on synthetic data. For additional related work, see Appendix A.
3 Prior-Data Fitted Networks for Forecasting
In this section, we give a background on prior-data fitted networks (PFNs), we introduce our synthetic data distribution, and we introduce ForecastPFN.
3.1 Background on Prior-Data Fitted Networks
univariate time series is a sequence of observations collected over time, {tt, ye Feee. in which t is a point in time, and yı is the value of the series at time t. In this section, for simplicity, we assume t is set of integers {1. 2..... T). denote a class of hypotheses, where each hypothesis & E & is mechanism for generating a time series. For example, şo may be defined by a function sb : + IR, where C N, which generates an underlying base series; we generate the time series D = {(1,, (().))) by yt (t()) 2 for all i. E A, where 2t is sampled from a noise distribution, 2, ~ μ. In forecasting, we start with a partial time series D = {tt, >)) and the goal is to predict the value of the series, timesteps into the future.
so a time series can be written as D {t, yt) yt)) Now let &
The posterior predictive distribution (PPD) for a point y is the distribution of its values given time T and dataset D = {(t,, D²) (where T" < T), denoted as p (T D). It computed for a particular point y by integrating over the set of hypotheses T as follows:
[ ply TTg)x(D w))(p)dg.
plu T.
A PFN qo is trained to approximate the PPD via synthetic prior-fitting [2. 33] as follows: iteratively sample a hypothesis y ~ p(y)) according to its prior probability, and then use yp to sample synthetic dataset D p(D) (p). We optimize the PFN's parameters 6 by making predictions on a held-out output from D. We split the dataset into training input set and training output set. For the case of forecasting, that is Din = {(1, v:)) and D = {L1 w)) We use the PFN qo to predict Dew given Din- In the classification setting, we compute the cross entropy loss in expectation over the draw of the dataset D ~ p( D), which trains the model to approximate the PPD the long run [33]. Since forecasting is a regression task, cannot cross entropy loss, so we instead minimize
3

---

Seasonal Trend
Irend
210
Full Time Series
1,050
200
75
1.025
50
80
0.975
Monthly Weeldy
Expomential Giobal Trend Linear
Global Trend
1.00
1.2
M May
2.5
1.5
2
0.99
1.5
Monthly Irend
Weokly Trend 18
Exporential Clobel Trend Linear
0.98
Dates
Dates
Dates
Figure 2: Two time series drawn from our synthetic distribution (Section 3.2). Each row shows the components that make up a time series: seasonal trends (far left), global trends (center left), and noise (center right), well as the entire time series (far right). We plot only half of the series in order to show the seasonal trends clearly. See Appendix for more examples.
the mean squared error loss Co D (y 2 where qo (t, Dm)) denotes the model's point- ceint-prediction given i and Din Over many samples draws from p(y)), this procedure trains the model to predict the mean of the posterior distribution p(- Din).
Note that the synthetic prior-fitting step is only done once, offline. At inference time, the resulting model is able make zero-shot predictions: given a previously unseen input Dm. predict the corresponding Dout (see Figure 1). The model's weights are not updated during inference: the model makes predictions for new dataset in single forward pass. We describe the details of our PFN architecture later in Section 3.3. In the next section, we describe our synthetic prior.
3.2 Defining a Synthetic Prior for Time Series
Unlike prior work in forecasting, model is not trained any real-world data; it is trained purely on synthetic data. As discussed above, we assume that there is a prior distribution of time series, p(p)), from which real-world time series are derived. Intuitively, this distribution includes components such as periodic (weekly, monthly, and yearly) trends, global trends, and noise. We show that when we model data from the distribution described below, data in order to produce a bigh-perffrmanne
adequately capture aspects of series zero-shot forecasting model.
We model our synthetic data with the simple premise that time series data have two independent components: underlying (xb) and noise (zz). We include time series for three scales: daily, monthly, and yearly, i.e., series where an observation is taken once a day, once a month, or once year. Data are generated from each periodicity independently according to the following procedure. Now we describe daily series, and we describe the monthly and yearly time series in Appendix D.
We model the underlying time series as being comprised of seasonal and a trend component. We see these as three independent aspects of time series data and model them as below, where the time series is the product of trend and seasonality with an additional noise factor. The trend component is made up of a linear and exponential component with coefficients Min- Cin- Maxp Cexp- The seasonal component has week, month, and year component where each comprises of coefficients
Pweek, Pmonth, Finally, the noise in our model is derived from Weibull distribution and is modeled such that the expected value of the noise model 1, meaning expectation the noise does not contribute to the seasonality or trend of the time series. Formally,

---

y1 (t) 2 trend(t) seasonal(t)- 24, where
2t = + Mnoise (= 2) where Weibull(1, k), = (In 2)2 (t) = (1 + miin + Cin)) (Mexp Capp)
seasonal(t) (t) = seasonal (t)
seasonal (t) seasonallear where
[K0/2]
cgu 2tT ( + dj cos (2TT)² ( Pr
seasonal, my
Σ
[
Pv
f=1
where V E (week, month, year),
Pweek = 7, Pmonth = 30.5, Pyear 365.25.
We choose multiplicative noise to better balance the amount of signal to noise across all series. If we had used additive noise, then the impact of the noise would have depended on the ratio of the scale of the base series and the scale the noise. Since our base series have linear and exponential terms, additive noise would cause the signal-to-noise ratio vary substantially over series based on the trend component. Furthermore, the Weibull distribution is a simple and natural method to parameterize between Gaussian and exponential distributions, two types of distributions that frequently come up in real-world time series.
The above synthetic data generation model is extremely general and can capture variety of time series we would encounter in real world forecasting applications. The model contains a number of parameters, which we denote as
E Mia: Gin, Mexp: Gesps Mhoiss Ceewee, deeweeks
Cpyzz years dpyss/a.yonl
Our full prior p(p) consists of a product distribution over all elements of &, where
Min; Mexp (1mm om); Clin N'( omn) Cesp (N1,
For all pairs of and J, Cf and dz., are both drawn from (0 1/f), such that these coefficients are inversely proportional to the harmonics of the series, and then they are evenly rescaled so that the sum of their squares equals 1.
Therefore, our synthetic data generation model contains six hyperparameters: Pm, mm, Olin Texp define the trend component, and Mnoise, k define the noise component. See Figure 2 for examples of daily time series drawn from p(p), broken down into their different components.
3.3 ForecastPFN: PFN for Zero-Shot Forecasting
Now we give the details for designing ForecastPFN, the first zero-shot forecasting method trained purely synthetic data. We focus the novel components inspired by the extreme challenge of training across a wide diversity of time-series scales with single model.
Architecture Details. As with the original PFNs [2, 22, 33], we use a transformer [47] as the base architecture. We an encoder- based transformer, consisting of one multi-head attention layer and two feedforward layers. This is in contrast to prior work on zero-shot forecasting, which used residual networks [36] recurrent neural networks [37].
The transformer accepts data in the form of tokens (t, 31). The time stamps t as part of the input data tokens are represented in terms of temporal features such as the year, month, day, day of the week, and day of the year. The series values are scaled as described the robust scaler below. A query for ForecastPFN is extremely general: the transformer takes in a set of tokens (t, v1), along with query consisting of a single, arbitrary future date, and then it predicts the output of this query. Thus, the input to the model is set of tokens and the output is single prediction the user. The input does not need to be a fixed size, and the input tokens do not need to be contiguous.
future time, specified by
The generality of ForecastPFN queries allows it to perform well at test time, on very diverse range of datasets and settings. This also in stark contrast to existing transformer models for forecasting, which typically are only set up to predict the next N steps in the current sequence.
5

---

Robust Scaling, One of the most challenging technical problems when training ForecastPFN is handling the extreme range of the scales of the time series, both in terms of the absolute values and the trends. This is a challenge both at training time and when computing the prediction of a time series, and partially stems from the inclusion of the multiplicative, exponential, and noise terms in our synthetic generation model. Conventional scaling techniques such as standardization (Z-score normalization) and min-max scaling techniques cannot apply: for example, not only does the absolute range differ greatly across training samples, but there is also large diversity in the number of outliers, and the scale of the pattern (i.e., periodicity and global trends) relative to the absolute range.
To handle these challenges, we perform three steps of outlier removal: (!) mask out any missing values. (2) standardize the data based on all non-outlier datapoints (defined as 2-sigma outliers), and (3) clip all 3-sigma outliers. This procedure allows us to put all time series into a relatively consistent range, while removing outliers that cause exploding gradients. Furthermore, when computing the loss of time series, we scale the mean squared error (MSE) based on the maximum of the (scaled) input. The effect a loss function that is in between MSE, which gives too much weight for series with a high range, and mean squared percentage error (MSPE), which gives too much weight for series with a small range. We show in Section that robust scaling significantly boosts performance.
Training Details. We train ForecastPFN using the synthetic prior p(y) defined in Section 3.2. During early-stage model development, we chose hyperparameters Mmm m, Flin: eepp Mmoise: & to create the most diverse data distribution possible, with the model still being able to train without diverging or stalling. For example, when the parameters defining the noise distribution become too large, the training data is too noisy and the model cannot converge. We specifically note that (1) all model development was done before evaluating on the seven real-world datasets defined the next section, and (2) all experiments were done with a single ForecastPFN model (specifically, after developing the model, we trained it once, and then used it for all experiments in the next section.
Crucially, since we use synthetic training data with a ground-truie trend and separate noise component, we can remove the noise when calculating the train loss. This significantly improves the training speed of the model, since the model cannot learn from the noise, as it is independent of time (Appendix E.5).
We set the input length 8 = 100 and the maximum prediction of 10 steps into the future (note: in the next section, we see that ForecastPFN is able make accurate predictions significantly beyond these settings). We generate 300 000 synthetic series to use as training data, each of length 200, and we use a sliding window of size 100 to obtain 101 prediction tasks per generated series. Each epoch consists of 024 000 tasks, and we trained the transformer for 600 epochs with the Adam optimizer [25] on single Tesla V100 16GB GPU, which took 30 hours. This training is done offline, and only once, and we open- spen-source the trained model at https: hopps://githnn //github.....///////////// com/ For the full development, implementation, and training details, see Appendix E.
4 Experiments
We demonstrate the power of ForecastPFN by comparing it to various high-performing forecasting models across several datasets. We compare all models across different training time budgets, and we compare the zero-shot performance of ForecastPFN to other models with different data budgets.
Algorithmic Baselines. We compare against three simple numerical baselines: Last, Mean, and SeasonalNaive [23]; two high- -eeeforrrfr traditional methods: ARIMA [4] and Prophet [23]; three state-of-the-art art transformer-basee -asee methods: FEDformer [54], Autoformer [50]. and Informer [52]; a vanilla Transformer method [47]; and one ttte-of-the-ane zero-shot method: Meta-N-BEATS [36]. All of the methods were implemented using their default hyperparameters and official codebase, with the exception of ARIMA (which has no 'official" codebase). for which we use pmdarima [43].
Benchmark Datasets. To ensure a fair comparison, we evaluate on seven popular, real-world datasets across energy systems, economics, traffic, and weather: ECL (Electricity Consuming Load) [46), ETT I and 2 (Electricity Transformer Temperature) [52], Exchange [28], Illness [17], Traffic [39], and Weather [1]. These datasets are standard for benchmarking the performance of forecasting methods and are the same ones used by many of our baselines [36, 50, 52, 54]. The datasets are described in the Appendix E.

---

Number of total MSE Wins per Data Budget
Number of total MSE Wins per Time Budget
Model
Arima
Autatomer
15
ForecactPPM
MSE Wirs
Informer
MSE
BERTS
Proshet
Sessonaltaiie
50
100
300
500
Data Budget
Budget
Figure 3: Analysis of performance vs. data budget and time budget, aggregated across datasets and prediction lengths. We plot the number total MSE wins (left) where a higher value is better. Error bars show standard deviation across training runs. Recall, ForecastPFN and Meta-N-BEATS no training data for these series, only the length 36 input.
Experimental Setup. The two zero-shot methods, ForecastPFN and Meta-N-BEATS, were both trained offline before the evaluations on benchmark datasets. We train ForecastPFN once on our synthetic data, as described in Section 3, and then that model is used for all evaluations in this paper. For Meta-N-Beats, we use the best version as described in their paper [36], which the one trained on M4 [31]. However, since the prediction length needs to be specified upfront, we train several Meta-N- Beats models, one for each prediction length considered below. For each time series, the zero-shot models see only the input of length 36 and then predict future timesteps. These training regimes are different from the other, non-zero shot models which are trained for each configuration.
We compare the zero-shot methods to the nine other non zero-shot methods described above, even though this is not fair comparison: the non-zero-shot methods are allowed to train on earlier portions of the time series used at test time, as is common in standard forecasting algorithms. We explore the performance of these algorithms across various resource constraints in two distinct directions: by restricting the training time budget and by restricting the amount of training data budget given to the non-zero-shor methods. We restrict the data budget of the non-zero-shor methods to lengths from 50 to 500. We restrict the time budget of the non-zero-shot methods to wall clock budgets from I to 120 seconds, where timing calculated after data loading and only during the training loop.
These two restrictions follow naturally from the ways in which forecasting datasets and applications can be constrained in the real world. Many real-world forecasting applications have very few initial observations [6. 8, 16, 23]. For example, many businesses need demand forecasts or customer churn forecasts for large number of products or customers, respectively, which are only size 100 daily series or size 36 monthly series. Furthermore, there is also wide range of applications which require a low time (or computational) budget in resource constrained environments, such as on a CPU or on edge devices. For example, forecasting may be needed to predict energy demand in developing countries. Another application forecasting visualizations they navigate their dataset.
forecasting data-exploration tool, allowing users to see instant
We also test all algorithms with different prediction lengths, or the number of steps into the future an algorithm must predict. Most deep-learning- based algorithms require this hyperparameter to be set before training, and then that prediction length encoded into the algorithm's 's architecture. This is true for FEDFormer, Informer, Autoformer, the vanilla transformer, and N-BAATS,, so we retrain these models for each experiment with a new prediction length. We choose prediction lengths from 6 to 48 as in prior work [54]. At test time, all algorithms are given series with input length of 36 and asked to predict into the future these variable prediction lengths. For example, say that we have a time series {t. yt.) 1,00 data budget of J, and prediction length of f. We allow algorithms to use 10% their data budget on validation. All non-zero-shot methods are allowed to train on [(l, yt) ] 1=500-2- Then, at test time, all algorithms see the 36 input data points and make a prediction
536 length of é, e.g., input of {t. ye) 100 and make predictions for timesteps t = 537 to 537 + l.
7

---

MSE Wins per Prediction Length By Data Budget (50 to 500)
100
150
200
250
300
500
MSE
20
40
10
30
50
20
40
10
50 10
20
40
10
30
50
Prediction Length
ForecastPFN Informer
Prophet
Figure 4: Analysis of performance across prediction lengths for increasing data budgets (left subplot, data budget 50 to the right subplot, data budget is 500), aggregated across datasets. Total MSE wins for each scenario is plotted with error bars one standard deviation across training runs. Recall, ForecastPFN and Meta-N- N--EATS see no training data for these series, only the length 36 input.
ECL
ETTh
ETTh2
Exchange
Illness
Traffic
Weather
Arima
1,840
0.341
1.610
1.175
5.045
2.738
0.042
Autoformer
1.289
0.570
1.107
0.331
1.353
2.119
0.520
Data Budget 50
FEDformer
0.683
0.403
1.020
0.233
1.240
1.318
0.330
ForecastPFN
1.075
0.127
0.330
0,058
1.091
1.971
0.009
Informer
1.252
0.982
0.890
2.834
8.584
3.627
0,436
Last
0.910
0.176
0.466
0.022
1.077
3.081
0.014
Mean
0.673
0.158
0.600
0.040
1.091
1.585
0.011
Meta-N-BEATS
0.909
0.177
0.480
0.023
1.301
2.913
0.014
Prophet
2.174
3.298
14.135
421.635
11.824
2.352
0.101
SeavonalNaive
0.453
0.203
0.554
0.028
1.410
0.653
0.017
Transformer
0.945
0.516
1.043
0.941
4.320
2.244
0.244
Arima
1.969
0.200
1.099
1.175
4.848
1.661
0.042
Autoformer
0.513
0.144
0.338
0.072
0.737
0.526
0.210
Data Budget = 500
FEDformer
0.480
0.133
0,352
0.068
0.707
0.523
0.188
ForecastPFN
1.075
0.127
0.330
0,058
1.091
1.971
0.009
Informer
0.453
0.144
0.253
0.529
4.394
1.084
0.224
Last
0.910
0.176
0.466
0.022
1.077
3.081
0.014
Mean
0.673
0.158
0.600
0.040
1.091
1.585
0.011
Meta-N-BEATS
0.909
0.177
0.480
0,023
1.301
2.913
0.014
Prophet
15.668
3.228
4.960
13.346
3.462
1.556
0.057
SeasonalNaive 0.453
0.203
0.554
0.028
1.410
0.653
0.017
Transformer
0.541
0.139
0.274
0.280
3.482
0.935
0.016
Table 1: Average MSE values for each algorithm and each dataset on the shortest (50) and longest
(500) data budgets. We observe that ForecastPFN is the majority winner on the smaller data budget and tied for best on the longer data budget. Thus, we see the robustness of the ForecastPFN method across a range of datasets and data budgets.
We conduct training runs under the Cartesian product of each prediction length and each data budget, well the Cartesian product of each prediction length and each time budget. We repeat these runs on every dataset and every algorithm. We conduct each run five times with different random seeds (in the randomness of the algorithm). However, ForecastPFN, Meta-N-BEATS, ARIMA, Prophet, and the baselines are deterministic algorithms, so they are only evaluated with one seed per run.
We plot the MSE in this section, and we give additional metrics in Appendix E. In order to aggregate results among datasets and different random seeds, for each experiment, we compute the number is commonly done in prior work [22, 33]. We plot the
of "wins" (rank of 1) for each algorithm, performance across data and time budgets in Figure 3, aggregated over datasets and prediction lengths, and tabulate the MSE of each model each dataset in Table 1. Additionally, plot the change in

---

performance for the prediction lengths on each data budget in Figure 4 and in Appendix E). We plot error bars as one standard deviation from the mean by averaging over the training runs.
Results and Discussion. We find that ForecastPFN significantly outperforms all ten other methods in the low data budget and low time budget settings (Figure 3). Specifically, we that ForecastPFN is the best architecture by the number of MSE wins for time budgets to 30. Similarly, ForecastPFN the best architecture by the number of MSE wins for data budgets from 50 250. In the settings with higher data or time budget, ForecastPFN is still competitive with state-of-the-ar methods. This story continues hold across prediction lengths as well in Figure ForecastPFN is the outright winner in nearly every prediction length for data budgets 50, 100, and 150, and becomes either outright best or very competitive in nearly every prediction length for budgets 200, 250, and 300. Additionally, we see from Table 1 that ForecastPFN achieves the lowest average MSE value on more datasets than any other algorithm, at a data budget of 50, and remains competitive at a data budget of 500. We also find that the simple baselines, SeasonalNaive and Last, often achieve second place behind ForecastPFN in the lowest training and runtime settings. This is not unexpected and is a testament to the challenge of predicting a series when given very little training data or training time.
The remarkable performance of ForecastPFN is despite the fact that, as explained above, the zero shot methods are at disadvantage because the other methods were allowed to train on data from earlier the time series. When ForecastPFN is compared Meta-N-Boo NeBttts, the only fair comparison. we see that ForecastPFN strictly outperforms Meta-N-Bec N-Bects across all settings. We believe these strong results are due to our synthetic prior: by encoding multi-scale seasonal trends, global trends, and noise, across variety of parameters, ForecastPFN learns very strong prior knowledge about the common patterns in time series. While = 250 datapoints of series contain enough complexity and noise for the training-basse based algorithms to struggle to make accurate predictions when learning the series 'from scratch', ForecastPIN's s prior knowledge from synthetic series is significant enough to make better predictions when only given access to the 36 input points.
Furthermore, ForecastPFN is significantly faster than the other transformer-based methods such as FEDformer. Informer, and Autoformer. ForecastPFN makes predictions in single forward pass, just 0.2 seconds; in order to reach the same accuracy as ForecastPFN, the other transformer-bie methods need take over 100 times more runtime.
Scaler Ablation
0.1
Epoch
0
100
200
300
ForecastPFN, lowest noise
23.71
1.484
0.878
0.591
Val Loss
0.05
ForecastPFN, low noise
0.359
0.050
0.039
0.384
ForecastPFN
0.207
0.049
0.037
0.032
0.03
Max Scaler Robust Scaler @P
Table 2: Loss values reported for three training runs of ForecastPFN where the amount of noise used to synthetically generate the Fore- castPFN training data moderated. In this table, we scale mnoise to 1/6 for the lowest noise model, to 2/3 for the low noise model, and I for the ForecastPFN model.
Step
Figure 5: Robust scaling vs. min-max scaling.
Ablation Studies. Since ForecastPFN takes 30 GPU hours to train, we do not conduct substantial ablation studies. However, we are able to investigate two important aspects of the ForecastPFN algorithm: our custom scaler and our noise model.
First, investigate the impact of robust scaling (described in Section 3.3) on the performance of ForecastPFN. We compare robust scaling to min-max scaling. We conduct a training run of ForecastPFN with each scaler and plot the validation MSE loss the synthetic dataset in Figure 5. We can clearly see that the robust scaler achieves a lower validation loss throughout training. We conclude the use of our robust scaler mitigates the effect of extreme values. While the presence of extreme values make it harder for the model to differentiate small-scale trends in a series, our robust scaler allows for better, faster convergence.
Finally, we investigate the impact of noise in our synthetic training data generation process (described in Section 3.3). We trained ForecastPFN with three scales of our noise parameter, Mpoise: I (original

---

ForecastPFN), 2/3 (low noise), and 1/6 (lowest noise). We do not change the scale of the noise in the validation data. In Table we report the train and validation MSE on synthetic data for each model. Recall that we remove noise in the ground-truth predictions of the training data as a design decision that improves performance, which is why the scale looks different for the train and validation losses. We also look at removing the noise entirely in Appendix E.5 and come to similar conclusions about the level of noise. We can conclude from this ablation that the noise parameters we chose in our experiments provide for robust minimal train and validation losses, validating our approach.
5 Conclusions, Limitations, and Future Work
In this work, we introduced ForecastPFN, the first zero-shot forecasting model that is trained purely on synthetic data. Our novel synthetic data distribution incorporates multi-scale seasonal trends, linear and exponential global trends, and noise distribution, capturing an extremely diverse range of time series. ForecastPFN is prior-data fitted network, trained once, offline to approximate Bayesian inference, which can make predictions on new time series dataset in a single forward pass. Through extensive experiments, we show that zero-shot predictions made by ForecastPFN are more accurate and faster compared to state-of-the-art forecasting methods, even when the other methods are allowed to train on hundreds of additional in-distribution data points.
Limitations and Future Work. While ForecastPFN has shown remarkable success in the zero-shot setting, there are still limitations as well as exciting directions left for future work. Due to the vanilla transformer architecture, ForecastPFN is limited by its training data to time series of fewer than 1000 points. However, given the recent advances in scaling transformers [3, 13, 26, 44], this is not inherently a problem for ForecastPFN. Currently, ForecastPFN only works for univariate time series. Once again, this is not an inherent limitation, and designing ForecastPFN for multivariate time series a promising direction for future work. Furthermore, using exogenous features are interesting ideas for follow-up work.
Our synthetic prior was created focus on 'human-like" or 'carth-like' time series (daily, weekly, yearly periods), so ForecastPFN may not not perform well on uncommon periods such as 41, 89, or 97. ForecastPFN also only approximates the posterior predictive distribution and makes pointwise predictions. Outputting probabilistic predictions are an exciting area for future work. Finally, ForecastPFN is particularly well-suited handling missing data (including large gaps), or varying time frequencies within the same series, in contrast to existing forecasting methods. An exciting future direction is to explore the potential to forecast time series in which the input contains large gaps varying frequencies.
References
[1] Max-planck-instiuut für biogeochemie, jena wetterstation beutenberg weather station beuten- berg. https:tt////ttt //www. jogc-jena... jena.mp.........e.ee..e......... de/vetter/.
[2] Steven Adriaensen, Herilalaina Rakotoarison, Samuel Müller, and Frank Hutter. Efficient bayesian learning curve extrapolation using prior-data fitted networks. In Proceedings of the Annual Conference on Neural Information Processing Systems (NeurIPS), 2023.
[3] Iz Beltagy, Matthew E. Peters, and Arman Cohan. Longformer: The long documenttnttt transformer. arXiv:200..05150, 2020.
[4] George EP Box and David A Pierce. Distribution of residual autocorrelations in autoregressive- integrated moving average time series models. Journal of the American statistical Association, 65(332):1509-155
1526, 1970.
[5] Salva Rühling Cachay, Venkatesh Ramesh, Jason NS Cole, Howard Barker, and David Rolnick. Climart: A benchmark dataset for emulating atmospheric radiative transfer in weather and climate models. In Proceedings of the Annual Conference on Neural Information Processing Systems Datasets and Benchmarks Track (NeurIPS), 2021.
[6] Rosa María Cantón Croda, Damián Emilío Gibaja Romero, and Santiago Omar Ca- ballero Morales. Sales prediction through neural networks for small dataset. International Journal of Interactive Multimedia and Artificial Intelligence... .... 2019.
10

---

[7] Real Carbonneau, Kevin Laframboise, and Rustam Vahidov. Application of machine learning techniques for supply chain demand forecasting. European Journal of Operational Research, 184(3):110--115, 2008.
[8] Vitor Cerqueira, Luis Torgo, and Carlos Soares. Machine learning vs statistical methods for time series forecasting: Size matters. arXiv preprint arXiv:190..13116, 2019.
[9] Chris Chatfield. Time-series forecasting. CRC press, 2000.
[10] Vinay Kumar Reddy Chimmula and Lei Zhang. Time series forecasting of covid-19 transmission in canada using lstm networks. Chaos, Solitons & Fractals, 135:109864 2020.
[11] Junyoung Chung, Caglar Gulcehre, KyungHyun Cho, and Yoshua Bengio. Empirical evaluation of gated recurrent neural networks on sequence modeling. arXiv preprint arXiv:/412.3553
2014.
[12] Javier Contreras, Rosario Espinola, Francisco J Nogales, and Antonio J Conejo. Arima models to predict next-day electricity prices. IEEE transactions on power systems, 180)),011--102, 2003.
[13] Tri Dao, Dan Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. Flashattention: Fast and memory-efficient exact attention with io-awareness. Proceedings of the Annual Conference on Neural Information Processing Systems (NeurIPS), volume 35, pages 16344-16359, 2022.
[14] Jan G De Gooijer and Rob J Hyndman. 25 years of time series forecasting. International journal of forecasting, 22(3):443-477 473, 2006.
[15] Hassan Ismail Fawaz, Germain Forestier, Jonathan Weber, Lhassane Idoumghar, and Pierre- Alain Muller. Transfer learning for time series classification. In 2018 IEEE international conference on big data (Big Data), 2018.
[16] Simon James Fong, Gloria Li, Nilanjan Dey, Rubén González Crespo, and Enrique Herrera- Viedma. Finding an accurate early forecasting model from small dataset: A case of 2019-ncov novel coronavirus outbreak. arXiv preprint arXiv:2003. 10776, 2020.
[17] Centers for Disease Control and Prevention. National, regional, and state level outpatient illness and viral surveillance. https://ght cdc. gov/ grwsp/ wwwvilllliw//iie wtt.we////wwoooaaaasshiiiiiaaaaa html, 2020.
[18] Carl Friedrich Gauss. Theoria motus corporum coelestium sectionibus conicis solem ambien- fium, volume 7. FA Perthes, 1877.
[19] Shahram Hanifi, Xiaolei Liu, Zi Lin, and Saeid Lotfian. A critical review of wind power forecasting methods-past present and future. Energies, 13(15):3764, 2020.
[20] Hrayr Harutyunyan, Hrant Khachatrian, David C Kale, Greg Ver Steeg, and Aram Galstyan. Multitask learning and benchmarking with clinical time series data. Scientific data, 6(1):96, 2019.
[21] Hansika Hewamalage, Christoph Bergmeir, and Kasun Bandara. Global models for time series forecasting: A simulation study. Pattern Recognition, 124: 108441, 2022.
[22] Noah Hollmann, Samuel Müller, Katharina Eggensperger, and Frank Hutter. Tabpfn: A transformer that solves small tabular classification problems in a second. Proceedings of the International Conference on Learning Representations (ICLR), 2023.
[23] Rob J Hyndman and George Athanasopoulos. Forecasting: principles and practice, OTexts, 2018.
[24] Hassan Ismail Fawaz, Germain Forestier, Jonathan Weber, Lhassane Idoumghar, and Pierre- Alain Muller. Deep learning for time series classification: review. Data mining and knowledge discovery, 33, 2019.
[25] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint 1rXiv.141..6980, 2014.

---

[26] Nikita Kitaev, Lukasz Kaiser, and Anselm Levskaya. Reformer: The efficient transformer. In Proceedings of the International Conference on Learning Representations (ICLR), 2020.
[27] Bjoemn Krollner, Bruce J Vanstone, Gavin R Finnie, et al. Financial time series forecasting with machine learning techniques: a survey. In ESANN, 2010.
[28] Guokun I Lai, Wei-Cheng Chang, Yiming Yang, and Hanxiao Liu. Modeling long-and short-term temporal patterns with deep neural networks. In The 41st international ACM SIGIR conference on research & development in information retrieval, pages 95-104, 2018.
[29] Bryan Lim and Stefan Zohren. Time-series forecasting with deep learning: a survey. Philosoph- ical Transactions of the Royal Society A, 379(2194):2020020)) 2021.
[30] Amal Mahmoud and Ammar Mohammed. A survey on deep learning for series forecasting. Machine Learning and Big Data Analytics Paradigms: Analysis, Applications and Challenges, pages 365-392, 2021.
[31] Spyros Makridakis, Evangelos Spiliotis, and Vassilios Assimakopoulos. The m4 competition: 100,000 time series and 61 forecasting methods. International Journal of Forecasting, 36(1): 36(1):54- 74, 2020.
[32] Duncan McElfresh, Sujay Khandagale, Jonathan Valverde, Ganesh Ramakrishnan, Vishak Prasad, Micah Goldblum, and Colin White. When do neural nets outperform boosted trees on tabular data? In Proceedings of the Annual Conference Neural Information Processing Systems (NeurIPS), 2023.
[33] Samuel Müller, Noah Hollmann, Sebastian Pineda Arango, Josif Grabocka, and Frank Hutter. Transformers can do bayesian inference. In Proceedings of the International Conference Learning Representations (ICLR). 2022.
[34] Maria Elena Nenni, Luca Giustiniano, and Luca Pirolo. Demand forecasting the fashion industry: a review. International Journal of Engineering Business Management, 5(Godište 2013):5-36, 2013.
[35] Boris N Oreshkin, Dmitri Carpov, Nicolas Chapados, and Yoshua Bengio. N-beats: Neural basis expansion analysis for interpretable time series forecasting. arXiv preprint 1905. 2019.
[36] Boris N Oreshkin, Dmitri Carpov, Nicolas Chapados, and Yoshua Bengio. Meta-learning framework with applications to zero-shot time-series forecasting. In Proceedings of the AAAI Conference on Artificial Intelligence (AAAI), 2021.
[37] Bernardo Pérez Orozco and Stephen Roberts. Zero- shot and few-shot time series forecasting with ordinal regression recurrent neural networks. arXiv preprint arXiv:2003. 2020.
[38] Jaideep Pathak, Shashank Subramanian, Peter Harrington, Sanjeev Raja, Ashesh Chattopadhyay, Morteza Mardani, Thorsten Kurth, David Hall, Zongyi Li, Kamyar Azizzadenesheli, al. Fourcastnet: A global data-driven high-resolution weather model using adaptive fourier neural operators. arXiv preprint arXiv:2202. 12214 2022.
[39] Caltrans PeMS. Caltrans performance measurement system (pems). US DOT, 2021.
[40] Syama Sundar Rangapuram, Matthias W Seeger, Jan Gasthaus, Lorenzo Stella, Yuyang Wang, and Tim Januschowski. Deep state space models for time series forecasting. Advances in neural information processing systems, 31, 2018.
[41] David Salinas, Valentin Flunkert, Jan Gasthaus, and Tim Januschowski. Deepar: Probabilistic forecasting with autoregressive recurrent networks. International Journal of Forecasting, 2020.
[42] Omer Berat Sezer, Mehmet Ugur Gudelek, and Ahmet Murat Ozbayoglu. Financial time series forecasting with deep learning: A systematic literature review: 2005-2019. Applied soft computing, 90: 106181, 2020.
[43] Taylor G. Smith et al. pmdarima: Arima estimators for Python, 2017-. [Online; accessed <today>].
12

---

[44] Yi Tay, Mostafa Dehghani, Dara Bahri, and Donald Metzler. Efficient transformers: A survey, ACM Computing Surveys, 2022.
[45] Sean J Taylor and Benjamin Letham. Forecasting at scale. The American Statistician, 2018.
[46]
Artur Trindade. Elccrricty_ooaDDiarrrs201111111 12014. UCI Machine Learning Repository, 2015. DOI: https://bb..rgg/1.224//////3/
[47] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Fukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pages 5998-6008, 2017.
[48] Manuel Weber, Maximilian Auch, Christoph Doblander, Peter Mandl, and Hans-Arno Jacobsen. Transfer learning with time series data: systematic mapping study, leee Access, 2021.
[49] Ruofeng Wen, Kari Torkkola, Balakrishnan Narayanaswamy, and Dhruv Madeka. A multi- horizon quantile recurrent forecaster. arXiv preprint 1r1.: 2017.
[50] Haixu WU, Jiehui Xu, Jianmin Wang, and Mingsheng Long. Autoformer: Decomposition transformers with auto-correlation for long-term series forecasting. Proceedings of the Anmual Conference on Neural Information Processing Systems (NeurIPS), 2021.
[51] Rui Ye and Qun Dai. Implementing transfer learning across different datasets for time series forecasting, Pattern Recognition, 2021.
[52] Haoyi Zhou, Shanghang Zhang, Jieqi Peng, Shuai Zhang, Jianxin Li, Hui Xiong, and Wancai Zhang. Informer: Beyond efficient transformer for long sequence time-series forecasting. In Proceedings of the AAAI Conference on Artificial Intelligence (AAAI), 2021.
[53] Jingho Zhou, Xinjiang Lu, Yixiong Xiao, Jiantao Su, Junfu Lyu, Yanjun Ma, and Dejing Dou. Sdwpf: A dataset for spatial dynamic wind power forecasting challenge at kdd cup 2022. arXiv preprint arXiv:2208. 04360, 2022.
[54] Tian Zhou, Ziqing Ma Qingsong Wen, Xue Wang, Liang Sun, and Rong Jin. Fedformer: Frequency enhanced decomposed transformer for long-term series forecasting. In Proceedings of the International Conference on Machine Learning (ICML), 2022.
13

---

A Additional Related Work
Time-series forecasting [9] has multitude of applications such as healthcare [10, 20], climate science [5, 38], business [7, 34], finance [27, 42], and renewable energy [19, 53]. A variety of approaches have been used for time-series forecasting [23, 29], including traditional statistical methods [4. 43] and deep learning methods [50, 52, 54].
Forecasting. ARIMA [4] is a statistical method from the 1970s that is still popular today, which builds autoregressive model based Markov processes. Another popular statistical method in practice is Prophet [45], which incorporates non-linear trends and multi-scale seasonality. Recently, deep learning methods for time series forecasting have become popular [29]. While early deep learning methods made use of RNNs [11, 41J, transformer models have become popular more recently [26, 50, 52], following the success of transformers on NLP [47]. FEDformer [54] incorporates Fourier transforms and the seasonal trend decomposition method into a transformer architecture. For a survey on deep learning methods for time- series forecasting, see [29, 30].
Many real-world forecasting applications have very few initial observations, sometimes just forty or fewer [6. 8, 16, 23]. However, there is much less work in this setting (referred to 'zero-shot' forecasting [36]). Oreshkin et al. [36] use a model based off of N-BEATS [35], a model that was state of the art at the time, training on (single) real-world time series dataset and testing
different
time series. However, it achieves substantially different performance based on which training dataset used. Another zero-shot forecasting method [37] uses RNN base, yet it is only empirically compared to traditional methods and on a fixed prediction length.
There is some work on transfer learning for time-series forecasting, but it is still limited [24]. Earlier works study transfer learning by pretraining a CNN architecture for forecasting on one time series, and tunettunig another time series [15, 51]. Fawaz et al. showed that some datasets can improve performance, while others can degrade performance, and they introduce a new method to predict the best source dataset. More recent work uses LSTMs and CNN+RNN hybrid models, to achieve better results [48]. Global Forecasting Models (GFM) are a new type of forecasting model in which a single model is trained on all series from a single dataset [21, 40, 41, 49]. However, key assumption is that these time series must be related in some way. For example, a retail company may train a single model for the time series of sales of thousands of different products.
Prior-Data Fitted Networks. Prior-data fitted networks (PFNs) are a recently-proposed paradigm for machine learning, in which a model is trained offline on small, synthetic datasets. Once trained, a PFN can infer the posterior predictive distribution (PPD) for test datapoints, given new dataset, in single forward pass.
Müller et al. [33] first introduced PFNs, showing that they provably approximate Bayesian inference, and demonstrating performance on very small datasets. Soon after. TabPFN [22] showed state- of-the-art empirical performance on datasets of up to size 1000 [32]. This new empirical finding was accomplished in part due to the synthetic prior, consisting of a large set of structural causal models. Recently, Adriaensen et al. [2] designed zero-shot approach using a PFN for learning curve extrapolation; however, the setting for learning curve extrapolation is significantly different and arguably special case of time-series forecasting. To the best of our knowledge, we are the first to design a zero-shot time-series forecasting model trained purely on synthetic data.
B Broader Societal Impact
We do not see any strongly negative broader societal impacts of this work. In fact, our hope that this work will reduce the computational cost and CO2 emissions for forecasting. Although ForecastPFN takes GPU hours to train, this is a one-time cost, which we have already done. Practitioners can now download and use our model, rather than training methods such as FedFormer and Informer from scratch. In general, we hope that our work encourages practitioners and researchers to conduct zero-shot (or sho-shot) forecasting, rather than training models from scratch for each new dataset.
14

---

C Reproducibility
Our codebase is available at https://gttthiii hta//aaacssai/oreassppiiii The README to the codebase contains instructions for installation and immediately using ForecasIPFN to make predictions on a new dataset. We also include all code from this project, including the code for synthetic data generation, ForecastPEN training, and evaluation.
D Synthetic Data Details
We give the full details of the synthetic data series, including the parameters for daily, weekly, and monthly series.
Recall that the time series has a trend component, seasonal component, and noise compo- nent. The trend component is made up of linear and exponential component with coefficients Wiin Gin, Mexp. Cexp- The seasonal component has a week, month, and year component where each comprises of of coefficients Pweek, Pmonth Pyear Finally, the noise in our model is derived from Weibull distribution and is modeled such that the expected value of the noise model is 1, meaning in expectation the noise does not contribute to the seasonality or trend of the time series. Formally,
Y1 (t) 2 trend(t) seasonal()) 2e, where
2t Mmoise (= 2), where Weibull(1, k). z = (In 2)
(t) (1 min + Cim) (Mexp" (xxp)
seasonal(t) (t) = seasonal (t) seasonalloont (t) yeasonalyear(( where
[Pw/2] Σ
era 2 2 Po
2TT - p )
seasonal, my
dj cos
Now explain how to set all unspecified parameters above, for daily, weekly, and monthly series. For convenience, when creating a daily, weekly, or monthly series, unit of time is equal to 1, 7, or 30.5 days (which would be equivalent scaling all the parameters below by 1, 7, 30.5).
For pairs and f. Cf.v and dz are both drawn from (0 1/f), such that these coefficients are inversely proportional to the harmonics of the series, and then they are evenly rescaled so that the sum of their squares equals
We set each my for E {week, month, year) separately for daily, weekly, and monthly, as follows. m, 0 unless specified:
Daily: Mwwck ~ U(O0, 1]), mooohh ~ ([0,,,)),, Pwock = 7, Pmouth 30.5.
Weekly: mmonth [0, Myyar ((0,, .1],, Pmonth 2, Pyear 52.
Monthly: Myear U((0, Pyear = 12.
Finally,
Miin; Mexp
(0-00100.5 Oin ~ ~O(,0001); Cesp ~ (1100.005).
As described in Section 3.3 and the next section (Appendix E. we set these parameters to create the most diverse data distribution possible, with the model still being able to train without diverging or stalling. For example, when the parameter for the noise distribution becomes too large, the model cannot converge. We also released README for our synthetic data generation at https: //gi thub. wtt.:coooaaaaaaa
For visualizations of daily time series drawn from this distribution, Figure 2 and Figure 7.
15

---

E Additional Experimental Details
E.1 ForecastPEN Development
We estimate that the total computational budget used in this project, from initial research running all the final evaluations, is 500 GPU hours. Our computation consisted mainly of designing the architecture and designing the synthetic data distribution. In this design phase, we used the train loss of the synthetic data as signal, because many synthetic data distributions, and architectures, cause the model not to train, for example, setting the noise parameter too high. We also used the synthetic held-out validation set to make decisions, such as in Figure 5 and Appendix E.5.
The training of our ForecastPFN model is 30 GPU hours single Tesla V100 16GB GPU. We emphasize that this model training is only done once, and we released the trained ForecastPFN model.
For an ablation study, we tried to achieve non-trivial zero-shot performance with the FedFormer architecture. However, after significant effort, we were unable to achieve non-degenerate performance with the FedFormer architecture.
E.2 Implementation Details
ForecastPFN Implementation Details. We use an encoder-basee based transformer architecture, with two encoders. Each encoder consists of multi-head attention layer with four heads, and two feedforward layers. The first feedforward layer is size 32 times the embedding dimension, and the second feedforward layer is size times the embedding dimension.
Baseline Implementation Details. We compare ForecastPFN against ARIMA [4], Autoformer [50], FEDformer [54], Informer [52], Meta-N-BEATT [36], Prophet [45], and vanilla Transformer method [47]. All of the methods were implemented using their default hyperparameters and official codebase. For ARIMA (which was designed in the 1970s and has no 'official' codebase), we used pmdarima [43]. The full implementation details of all methods can found in our open source codebase. Furthermore, we compared to three baselines: Last (prediction is the last seen value in the series), Mean (prediction the mean value in the series), and SeasonalNaive (prediction is the mean value from all prior instances of the same day of the week).
In Section 4, in the experiments with restricted time budget, we restrict the time budget of the non-zero-shot methods to wall clock budgets from I to 120 seconds, where timing is calculated after data loading and only during the training loop. After each training step, we stop the training the total budget was exceeded. In case an algorithm is unable to output any predictions within the data budget (such as ARIMA with a second budget), we set the output to 0's.
E.3 Training Details
We generate 100.000 each of synthetic daily, weekly, and monthly series, each of length 200, and we use sliding window of size 100 to obtain 101 prediction tasks per series. We sample 1024 tasks training step, and there are 1000 training steps in an epoch. We trained the transformer for 300 epochs on a V100 16GB GPU, which took 30 hours.
We the Adam optimizer with learning rate of 0.0001, and MSE loss. We set the maximum prediction into the future to be 10 times the frequency of the training series.
E.4 Dataset Details
We use seven datasets for our experiments in Section 4. These datasets are standard for benchmarking the performance of forecasting methods and are the same ones used by the majority of our baselines [36, 50, 52, 54].
Illness is a dataset [17] containing influenza- like illness patients in the United States. It contains 966 datapoints, spaced weekly. Exchange a currency exchange dataset [28], consisting of 7588 datapoints, spaced daily. Traffic is a dataset consisting of freeway congestion across California [39]. ETT (Flectricity Transformer Temperature) a electricity power deployment dataset [52] across 2 years from power plant in China. The target value is oil temperature. Two separate series
16

---

Time Budget
1.0
5.0
10.0
15.0
30.0
45.0
60.0
120.0
Arima
9.68
77.42
88.71
92.9
100
100
100
100
Autoformer
70.97
97.74
100
100
100
100
100
100
FEDformer
71.94
99.68
100
100
100
99.35
100
100
Informer
72.58
100
100
100
100
100
97.42
100
Prophet 88.71
100
100
100
100
100
100
100
Transformer 66.77
100
100
100
98.39
100
100
100
Table 3: The percentage of configurations (dataset, prediction lengths, seeds) that failed, for each
time budget and each model which requires training or being fit.
are provided: ETT1 and ETT2. Each series represents data collected from two different electicity transformer stations. ECL (Electricity Consuming Load) [46] is electricity consumption (Kwh) dataset across two years with resolution of every hour. This dataset has a total of 26 304 hourly observations. Weather is dataset [1] that contains local climate data from Germany, across three years. The datasets ETT, ECL, and Weather all natively have a sub-day resolution which we aggregate into daily observations by taking the sum of values in day.
E.5
Noise Removal Ablation
Recall that crucially, since we use synthetic training data with a ground-truth trend and a separate noise component, remove the noise when calculating the train loss. Now, give ablation study to confirm that this design decision significantly improves the performance of our model.
We investigate the impact of this design decision the performance of ForecastPFN by comparing model with the noise removed (that which is included the main body the paper) with training conducted without the removal of the noise from the training loss calculation. In doing this, we calculate the validation MSE on the synthetic data for both models and observe 33.089 for the model without noise removed and 22.692 for the model with the noise removed. We conclude the use of the noise removal during training loss calculation leads to improvements on the synthetic validation set, and thus deploy the of noise removal in the final model.
This design decision significantly improves performance, and it can only be done when using synthetic data, rather than real data.
E.6 Training Time Analysis
Recall that during our benchmark, there were six models which required training data on portion of each of the datasets before they were able to be tested. Further recall that we established a time budget for this training process, and varied that timing from 1 to 120 seconds. Some of these models were unable to fit to the trainig data during some of those training budgets. In Table 3, we calculate the percentage of configurations (dataset, prediction lengths, seeds) which failed for each model and training time budget. We see that Arima is the model which struggles the most at low time budgets.
E.7 Additional Experimental results
In Figure 16 and Figure we plot the mean MSE rank per prediction length (similar Figure 3 and 4 respectively, but for mean MSE rank instead of MSE wins).
In Figures 8 to 20, give similar plots Figures 3, 4 and 6 but instead of MSE, we plot MAE, MAPE, and MSPE. We also include plots for the evaluation mean rank.
17

---

Mean MSE Rank per Prediction Length Data Budget (50 to 500)
50
100
150
250
300
500
7.5
2.5
10
20
30
40
50
10
20
30
40
50
10
20
30
40
50
10
20
30
40
50
10
20
30
40
50
10
20
30
40
50
10
20
30
40
50
Prediction Length Last
Arima Autaformer FEDfomer ForecastPFN
Informer
Mean
Meta-N-9EAIT Prophet PeessonslNaivee Transformer
Figure 6: Analysis of performance across prediction lengths for increasing data budgets (left subplot, data budget 50 to the right subplot, data budget is 500), aggregated across datasets. Mean MSE rank for each scenario is plotted with error bars as standard deviation across training runs. Recall, ForecastPFN and Meta-N-BEATS see no training data for these series, only the length 36 input.
18

---

Scasonal Trand
Global Irend
2.5
Noise
0.09
Tirre Seer les
0.07
w m Mole
0.03
0.02
1.5
WMM Ma Wiy
0.01
ND nt Thend Weekly Thend
Exporienti Riel - Cllobal Irend - Uinear
28
Dates
Dates
Dates
Dettes
Seasonal Trend
Irand
Notse
Full Time Serias
1.4
1.2
1.2
1.1
1.5
0.5
3.8
Manthy Weekly Trend
Exporientall Glabal Trend Linear
Seasonal Trand
Global Trend
Noise
Series
1,090
1.5
1.025
1.1
0.5
3.9
0.975
0.4
9.5
0.950
Monthly Tren
Weekly Teerd
Exporential tial Glabel Trend Uieer
Dates
Dates
Dates Nolis
Detes
Seasonal Trend
Time Ser
3.1
0.25
0.2
0.8
-0.3
0.35
0.7
Monthly Trend
Weekly Trend
Expomeriial Global Tren Linear
0.95
20
Dates
Dates
Seasonal Trend
Global Teerd
Notse
Series
1.25
1.4
2.01
MM
1.2
0.75
1.5
3.50
0.99
0.9
0.25
Monthy Wend Weekly Trend
Exborential Trend
Unear
Dales
Dates
Dutes
Detes
Figure 7: Five examples of time series drawn from the synthetic distribution we define in Section 3.2 (continuation of Figure 2). Each row shows the components that make up time series: seasonal trends (far left), global trends (center left). and noise (center right), as well as the entire time series (far right). We plot only half of the series order show the seasonal trends clearly.
19

---

Mean MAE Rank Data Budget
Number total MAE Wins per Data Budget
Modal
Auseformmr
Mean Rank
MAE 10
ForocastPin
Informer
Propint
Seacoraliatre
90
100
300
900
Budget
Data Budget
Figure 8: Analysis of performance vs. data budget, aggregated across datasets and prediction lengths. We plot the number of total MAE wins (left) where higher value is better and mean MAE rank (right) where a lower values is better. Error bars show one standard deviation across training runs. Recall, ForecastPFN and Meta-N-BEATS no training data for these series, only the length 36 input.
Mean MAE Rank Time Budget
Number of total MAE per Time Budget
Model
Auroformer
Informer
Mean
Mata N-BATS
Prophik
SeasoralNalee
Transfomer
15 Time Budget
100
Budget
Figure 9: Analysis of performance across time budgets, aggregated across datasets and prediction lengths. We plot the number of total MAE wins (left) where higher value is better and mean MAE rank (right) where a lower values is better. Error bars show one standard deviation across training Recall, ForecastPFN and Meta-N-BEATS 36 input.
training data for these series, only the length
20

---

MAE Wins per Prediction Length By Data Budget (50 to 500)
50
100
150
200
250
300
500
MAE
20
40
10
30
50
Prediction Length
Autoformer FEDfomer ForecastPFN Informer
Prophet
Transformer
Figure 10: Analysis of performance across prediction lengths for increasing data budgets (left subplot, data budget 50 to the right subplot, data budget 500), aggregated across datasets. Total MAE wins for each scenario is plotted with error bars as one standard deviation across training runs. Recall, ForecastPFN and Meta-N-BEATS see no training data for these series, only the length 36 input.
Mean MAE Rank per Prediction Length By Data Budget (50 to 500)
300
500
7.5
Mean
2.5
10
20
30
40
50
10
20
30
40
50
10
20
30
40
50
10
20
30
40
50
10
20
30
40
50
10
20
30
40
50
10
20
30
40
50
Prediction Length Last
Arima Autoformer FEDfomer ForecastPFN Informer
Mean
Meta-N-BEATS Prophet Peeesonallaivee Tansformer
Figure 11: Analysis performance across prediction lengths for increasing data budgets (left subplot, data budget is 50 to the right subplot, data budget is 500), aggregated across datasets. Mean MAE rank for each scenario is plotted with error bars as standard deviation training runs. Recall, ForecastPFN and N-BATS see no training data for these series, only the length 36 input.
Mean MAPE Rank per Data Budget
Number of total MAPE Wins per Data Budget
Model
10
Auroformer
FEDior mar
Mean MAPE
Moan
Prophat
Sexcorallatve Thansommr
50
150
302
500
50
100
500
Data Budget
Data Budget
Figure 12: Analysis of performance data budget, aggregated across datasets and prediction lengths. We plot the number of total MAPE wins (left) where a higher value is better and mean MAPE rank (right) where a lower values is better. Error bars show one standard deviation across training runs. Recall, ForecastPEN and Meta-N-BEATS see no training data for these series, only the length 36 input.
21

---

Mean MAPE Rank per Time Budget
Number total MAPE Wins Time Budget
10
TEDiermer
Mean Rank
Wins
TorecasPPTN
informer
Moan
Meta BEATS
Prophct
Seasorallestee
Transfomer
102
Time
Budget
Figure 13: Analysis of performance lengths. We plot the number of total MAPE wins (left) where a higher value is better and mean MAPE rank (right) where a lower values is better. Error bars show one standard deviation across training runs. Recall, ForecastPFN and Nota-N-BEATS see no training data for these series, only the length 36 input.
time budgets, aggregated across datasets and prediction
MAPE Wins Prediction Length By Data Budget (50 to 500)
100
150
200
250
300
500
MAPE
20
40
10
20
30
40
50
10
20
30
40
50
10
20
30
40
50
10
20
30
40
50
10
20
30
40
10
30
50
Prediction Length Mean
Autoformer FEDfomer ForecastPFN
Meta-N-BEATS
Prophat SeasonsNaaive Taastormer
Figure 14: Analysis of performance across prediction lengths for increasing data budgets (left subplot, data budget is 50 to the right subplot, data budget is 500), aggregated across datasets. Total MAPE wins for each scenario is plotted with enror bars standard deviation across training runs. Recall, ForecastPFN and Meta-N-BEATSS see no training data for these series, only the length 36 input.
Mean MAPE Rank per Prediction Length By Data Budget (50 500)
7.5
MAPE
2.5
10
30
50
20
50
20
20
40
10
10
30
50
30
50
Prediction Length
Arima Autoformer
FEDfomer
ForecastPPN
Informer
Lest
Mean
Meta N N-BEATS
Prophet
Seasona Naive
Transformer
Figure 15: Analysis performance across prediction lengths for increasing data budgets (left subplot, data budget is 50 to the right subplot, data budget is 500), aggregated across datasets. Mcan MAPE rank for each scenario is plotted with error bars as one standard deviation across training runs. Recall, ForecastPEN and Meta-N-BEL N-BEATS see training data for these series, only the length 36 input.
22

---

Mean MSE Rank Data Budget
Mean MSE Rank per Time Budget
Moan MSE Rank
Mean MSE Bark
300 300
Data Budget
Time Budget
Figure 16: Analysis performance vs. data budget and time budget, aggregated across datasets and prediction lengths. We plot the number of total MSE wins (left) where a higher value is better. Error bars show one standard deviation across training runs. Recall, ForecastPFN and Meta-N-BEATS see no training data for these series, only the length 36 input.
MSPE per Budget
Number total MSPE Wins per Data Budget
15
Model
Aunotemer
Mean Rank
vins
ForocastPFN
Informe
Meta N-BAAT
Prophet
Seacsraliatre
50
100
300
500
Budget
Data Budget
Figure 17: Analysis of performance vs. data budget, aggregated across datasets and prediction lengths. We plot the number of total MSPE wins (left) where higher value better and mean MSPE rank (right) where a lower values is better. Error bars show one standard deviation across training runs. Recall, ForecastPFN and Meta-N-BEATS no training data for these series, only the length 36 input.
23

---

Mean MSPE Rank per Time Budget
Number total MSPE Wins Time Budget
Mean MSPE Rank
TEDrermer
TorecastPTN
Informer
Mean
Meta BEATS
Proprit
Seasorallestee
Transfomer
102
Time
Budget
Figure 18: Analysis of performance lengths. We plot the number of total MSPE wins (left) where higher value is better and mean MSPE rank (right) where lower values is better. Error bars show one standard deviation across training runs. Recall, ForecastPFN and Beta-N-BEATS see no training data for these series, only the length 36 input.
time budgets, aggregated across datasets and prediction
MSPE Wins Prediction Length By Data Budget (50 to 500)
100
150
200
250
300
500
MSPE
20
40
10
20
30
40
50
10
20
30
40
50
10
20
30
40
50
10
20
30
40
50
10
20
30
40
10
30
50
Prediction Length Mean
FEDfomer Autoformer
ForecastPFN
Prophat SeasonsiNaive Taastormer
Figure 19: Analysis of performance across prediction lengths for increasing data budgets (left subplot, data budget is 50 to the right subplot, data budget is 500), aggregated across datasets. Total MSPE wins for each scenario is plotted with enror bars standard deviation across training runs. Recall, ForecastPFN and Meta-N-BEAT'S see no training data for these series, only the length 36 input.
Mean MSPE Rank per Prediction Length By Budget (50 500)
7.5
MSPE
2.5
10
30
50
50
20
40
10
10
30
50
30
50
Prediction Length
Arima Autoformer
FEDfomer
ForecastPPN Informer
Lest Mean Meta N N-BEATS
Prophet
Seasona Naive
Transformer
Figure Analysis performance across prediction lengths for increasing data budgets (left subplot, data budget is 50 the right subplot, data budget is 500), aggregated across datasets. Mean MSPE rank for each scenario is plotted with error bars as one standard deviation across training runs. Recall, ForecastPEN and Deta-N-BEATH see training data for these series, only the length 36 input.


Reviews:
[{"review_id": "4fzQYnBS2o", "reviewer": "Reviewer_MwwS", "summary": "ForecastPFN is a zero-shot forecasting model designed to overcome the limitations of traditional forecasting methods when dealing with data-sparse applications. Unlike most approaches, ForecastPFN is trained solely on synthetic data, which captures diverse time series patterns and incorporates multi-scale seasonal trends, global trends, and noise. This prior-data fitted network approximates Bayesian inference and enables accurate and fast predictions in a single pass. Extensive experiments presented in the paper demonstrate that ForecastPFN outperforms state-of-the-art methods across multiple datasets and forecast horizons, even when the latter are trained with significantly more data points.", "questions": "1. Considering the focus on commercial forecasting applications in this work, it would be valuable to understand the reasons behind modeling ForecastPFN as a point forecaster instead of a probabilistic forecaster.\n2. In real-world forecasting applications, the presence of exogenous features can provide additional information that directly affects the accuracy of forecasts. How does ForecastPFN handle the incorporation of exogenous features?\n3. It would be interesting to know if the authors explored the option of fine-tuning ForecastPFN on the target data. This analysis could provide insights into the impact of fine-tuning on the performance of ForecastPFN in downstream forecasting tasks.", "limitations": "See above", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The manuscript is well-written, and the presented ideas are easy to follow.\n2. The concept of training a foundational time series model solely on synthetic data is innovative and exciting. This approach not only addresses the issue of lower forecasting accuracy in data-sparse scenarios but also has positive environmental implications.\n3. The execution of this work is commendable. The paper provides detailed information on the generation of the synthetic dataset, the model architecture, and the training process. Additionally, the authors conducted an extensive set of experiments, exploring various model configurations, prediction horizons, and datasets from multiple domains", "weaknesses": "The paper predominantly relies on plots to present the results, without including tables. While it is understandable that summarizing all the experiments in tabular form would be challenging due to the extensive nature of the research, it can be difficult for readers to extract meaningful information from a large number of plots. Additionally, certain plots, such as Fig 5, are scaled down to a point where they become challenging to interpret. The same issue extends to the plots presented in the appendix. To enhance clarity, it is recommended that the authors include some key results in tabular form and adjust the scale of the plots to improve readability for readers.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the excellent feedback. We are glad to see that you find our idea innovative and exciting, and that you find the execution commendable. We also find that your comments will substantially improve the final version of our paper. We reply to each question below.\n\n**W1: No tables.** Thank you, we can easily add tables for all of our main experiments, and we are happy to do so. We present tables below which show the superiority of ForecastPFN on a majority of datasets at different train budgets with a prediction length of 24. \nWe have printed tables with MSE values for Train Budget = 50 below.\n\n|     |         ECL |        ETT1 |        ETT2 |    exchange |         ili |     traffic |     weather |\n|-------------|------------:|------------:|------------:|------------:|------------:|------------:|------------:|\n| Arima       |       1.844 |       0.344 |       1.590 |       1.175 |       5.039 |       2.696 |       0.041 |\n| Autoformer  |       2.368 |       0.850 |       1.299 |       0.381 |       1.858 |       3.803 |       0.371 |\n| FEDformer   |       0.899 |       0.762 |       1.175 |       0.786 |       1.549 |       2.331 |       0.817 |\n| ForecastPFN |       1.104 | **0.121** | **0.340** |       0.061 | **1.102** |       2.029 | **0.010** |\n| Informer    |       1.078 |       1.967 |       1.105 |       4.694 |      11.045 |       5.848 |      0.325 |\n| Last        |       0.889 |       0.191 |       0.492 | **0.024** |       1.487 |       3.030 |       0.017 |\n| Mean        | **0.658** |       0.174 |       0.649 |       0.042 | **1.102** | **1.558** |       0.012 |\n| Metalearn   |       0.871 |       0.192 |       0.500 |       0.025 |       1.572 |       2.795 |       0.014 |\n| Prophet     |       2.181 |       3.298 |      14.060 |     421.577 |      11.813 |       2.328 |       0.101 |\n| Transformer |       0.827 |       0.630 |       0.538 |       1.833 |       5.578 |       3.093 |       0.182 |\n\n\nAnd with Train Budget = 500 below. \n|     |         ECL |        ETT1 |        ETT2 |    exchange |         ili |     traffic |     weather |\n|-------------|------------:|------------:|------------:|------------:|------------:|------------:|------------:|\n| Arima       |       1.974 |       0.201 |       1.075 |       1.175 |       4.852 |       1.628 |       0.041 |\n| Autoformer  | **0.482** |       0.141 |       0.360 |       0.073 | **0.705** | **0.473** |       0.189 |\n| FEDformer   |       0.499 |       0.135 |       0.374 |       0.072 |       0.727 |       0.455 |       0.208 |\n| ForecastPFN |       1.104 | **0.121** | **0.340** |       0.061 |       1.102 |       2.029 | **0.010** |\n| Informer    |       0.442 |       0.144 |       0.263 |       0.518 |       4.978 |       1.028 |       0.186 |\n| Last        |       0.889 |       0.191 |       0.492 | **0.024** |       1.487 |       3.030 |       0.017 |\n| Mean        |       0.658 |       0.174 |       0.649 |       0.042 |       1.102 |       1.558 |       0.012 |\n| Metalearn   |       0.871 |       0.192 |       0.500 |       0.025 |       1.572 |       2.795 |       0.014 |\n| Prophet     |      15.698 |       3.226 |       4.907 |      13.347 |       3.478 |       1.530 |       0.056 |\n| Transformer |       0.621 |       0.157 |       0.277 |       0.351 |       3.576 |       0.804 |       0.012 |\n\n\nAll other tables have been put in the appendix of the paper. \n\n**Q1. Probabilistic forecaster.**\nThis is a great suggestion. Since this work already introduces a new paradigm, we wanted to start with the simplest regression setting, but we agree that adding probabilistic predictions is an exciting and natural area for future work. Towards that direction, we are training three ForecastPFN models to show that we can achieve probabilistic forecasts by ensembling (which is a strong technique for uncertainty calibration). For future work, it will be interesting to have the model itself output probabilities.\n\n**Q2. Exogenic features.** This is another great suggestion. Once again, we started out with a simpler setting so that we could clearly demonstrate our approach, and so that we could compare to the main experiments in prominent existing work such as FEDFormer and Informer. We think this is a great idea for future work. \n\n**Q3. Fine-tuning.** While this is once again a nice suggestion, we make three points. First, one of our main contributions is to create a universal forecasting method that can perform well on any new series, without any fine-tuning. Second, we believe that the settings where ForecastPFN stands out the most compared to prior work, is when there is very little inference time available and/or when there is very little training data available -- settings in which it would be very challenging to fine-tune. Third, we note that prior-data fitted networks (PFNs) are still a new research area just one year old, and fine-tuning has not yet been tried even for classification-based PFNs. From a theoretical standpoint, it is not clear that fine-tuning would maintain the desirable Bayesian properties of PFNs. Although, intuitively we agree that fine-tuning could help in practice.\n\nThank you very much, once again, for your positive view of our work and your excellent suggestions. If you find our responses satisfying, since we addressed your weaknesses, we respectfully ask that you consider increasing your score. On the other hand, we would be very happy to answer any follow-up or additional questions you have."}, {"review_id": "Lmr7gG8Xoc", "reviewer": "Reviewer_5AR9", "summary": "This paper proposes a zero-shot prior-data fitted network (PFN) for time-series forecasting. Existing works have challenges in designing a general and flexible PFN for general time-series distributions, and tuning an architecture and training scheme. This work overcomes these by designing a novel synthetic time-series distribution in training and using a transformer architecture for queries to arbitrary future timestep. Experiments have been presented to demonstrate the improved performance of the proposed model over comparison models.", "questions": "Please see the weaknesses mentioned above.", "limitations": "Yes.", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The proposed work tackles an interesting setting where few initial observations are available in forecasting.\n2. The transformer-based network aims to predict arbitrary future time steps and the robust scaling deals with the extreme scale of time series, which increase the novelty of the methodology. \n3. The results seem compelling as well, with a clear outperformance of the proposed model over comparison baselines.", "weaknesses": "1. In Section 3.2, it is not clear what motivates the author to choose a multiplicative data generation model with seasonal and trend components as the synthetic prior. It would be appreciated if the author could elaborate more on that. Also, the ablation study could be more sufficient: the diversity of the synthetic data generated for training should be explored since the model relies on a general synthetic data distribution. \n2. The author could provide more details about the architecture in Section 3.3: 1) How does the network learn diverse temporal features of the input data? 2) How does the network deal with different input sizes? 3) What is the benefit of using transformer-based layers vs residual blocks or recurrent neural networks?\n3. In Section 4, the author should explain the definition of train budget and time budget. Are highlighted texts “training time” and “training data” related to them? It would be better to make them consistent.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the excellent feedback. We are glad to see that you are positive towards our work, including our novel methodology and our results. We also find that your comments will substantially improve the final version of our paper. We reply to each question below.\n\n**W1.1. Multiplicative data generation model.**\nYes, additive vs. multiplicative noise is an interesting question. We chose multiplicative noise to better balance the amount of signal to noise across all series. If we used additive noise, then the impact of the noise depends on the ratio of the scale of the base series and the scale of the noise. Since our base series have linear and exponential terms, additive noise would cause the signal-to-noise ratio to vary substantially over series with a trend component. Therefore, we used multiplicative noise, which is a simpler parametrization to achieve a signal to noise ratio for a series. We now updated our paper to include this discussion, and we agree that including an ablation study is a good idea for future work.\n\n**W1.2. Ablation on the diversity of synthetic data.**\nThis is a great idea. Since each training procedure takes 30 GPU hours, it is challenging to do a thorough ablation study. As mentioned in Section 3.3, we performed light HPO on the synthetic data parameterization in the early stages of the project, using the training loss as a signal. We are currently working on a simple ablation study on the noise of the synthetic data, which we will finish during the NeurIPS discussion period.\n\n**W2.1. How does the network learn diverse temporal features.**\nThis is a good question. We chose to use a transformer architecture because there is a large body of empirical evidence across different areas of research that transformers learn spatio-temporal features (e.g., vision, speech and language). We used a fairly lightweight transformer (one multi-head attention layer and two feedforward layers; details in Section 3.3), and our results show that it learns diverse temporal trends. In fact, a concrete demonstration that we can do in the future would be to train our transformer to approximate a partial Fourier transform.\n\n**W2.2 How does the network deal with different input sizes.**\nAgain, since we use a transformer, it can easily deal with variable input lengths. In our experiments, we set a maximum input length to 200 just because of computation constraints that we imposed. A good experiment in future work would be to increase the maximum input length to, e.g., 5000 inputs, although this would be substantially slower.\n\n**W2.3 What is the benefit of using transformer-based layers vs residual blocks or recurrent neural networks.**\nNote that ForecastPFN is a fully pretrained model – similar to an LLM, but for forecasting. In other research areas that create “foundation models” for language modeling and even computer vision, transformers are by far the most common choice. However, as you say, we could also have tried out other types of architectures. One other reason is that transformers are currently very fast on hardware due to receiving a lot of attention from the research community. Furthermore, other models such as RNNs are inherently sequential and harder to saturate GPU compute; therefore, transformers are the most efficient and best-performing choice for ForecastPFN.\n\n**W3. Make definitions consistent.**\nThank you. We will clarify this point in Section 4 better. We define training time as the amount of wall clock time that is permitted for each model to train (1 to 120 seconds) and training budget as the amount of data points given to each model to train (50 to 500). This is given on lines 242-244 in the present manuscript, but we have rewritten the description to be more clear. \n\nThank you very much, once again, for your positive view of our work and your excellent suggestions. If you find our responses satisfying, since we addressed your weaknesses, we respectfully ask that you consider increasing your score. On the other hand, we would be very happy to answer any follow-up or additional questions you have."}, {"review_id": "k6PPpeoyGA", "reviewer": "Reviewer_3SYZ", "summary": "The paper proposes to pre-train a deep learning model on synthetic data which follows characteristics of real world time series data. This model can then be used for any downstream forecasting dataset. The authors performed experiments to show that the proposed method performs well compared to existing classical and deep forecasting models in constrained scenarios (restricted number of training data, restricted amount of time for training).", "questions": "1. Are the existing methods (FEDformer, etc.) also pre-trained on the synthetic data? If not, it seems quite artificial to claim wins when these methods are only given 50 datapoints (1 batch?) for training, or 1 second to train. In fact, it seems unfair to compare to ForecastPFN which has 30 hours of pre-training on a huge synthetic dataset.\n2. How was Meta-N-BEATS trained on the M4 dataset? In the paper, 1 model was trained for each frequency. Which subset of M4 was the baseline trained on?\n3. What does number of MSE wins mean?\n4. What does training on 1 second mean? How is this implemented? Was a single gradient step even performed in this setting?\n5. What are the experiment details of the baseline models? What machine was used to run these baseline experiments? It seems a little iffy to compare the time budget in terms of seconds. How to compare 120 seconds on a H100 GPU vs a K80 GPU or even a CPU?\n6. How was the time restricted training for ARIMA/Prophet implemented? I understand for deep learning models you can easily stop the training, but what about ARIMA/Prophet if the model has not been fitted?\n\nPlease consider using number of batches/epochs trained instead of wall clock seconds.", "limitations": "The paper doesn't adequately address limitations. The limitations given in the limitations section aren't really true limitations, but rather things that are out of scope of this paper. Some suggestions of limitations to consider:\n1. What if the proposed prior for synthetic data is completely different from the downstream real world data? Perhaps some sort of adversarial forecasting task?\n2. ForecastPFN doesn't compute the posterior predictive distribution, it is just an approximation.\n3. Does not consider probabilistic forecasting, which has more implications on the synthetic prior (not just the shape of time series, but also the distribution of data, need to consider some autocorrelation in the errors)", "rating": 5, "confidence": 3, "soundness": 1, "presentation": 3, "contribution": 3, "strengths": "The paper proposes an exciting direction of pre-training deep forecasting models with synthetic data, which can be quickly adapted in a Bayesian manner on a new unseen real time series, and be able to perform accurate forecasts. The synthetic data is proposed using a prior distribution which has properties similar to real world data. Such a direction is exciting and of huge significance, unlocking the power of deep learning models to be pre-trained on large datasets. The paper is well written, neatly organised in a logical flow, and contains comprehensive experiments.", "weaknesses": "* While the premise of the paper is indeed exciting, I believe the claims made in the abstract and introduction are greatly exaggerated. Claims are made that the proposed method, ForecastPFN is able to beat SOTA forecasting methods. However, the experiments are only performed on severely handicapped scenarios (SOTA models are only given access to 100s of data points or maximum of 2 minutes of training). More effort should be made to highlight this handicapped scenario.\n* Claims that \"the zero shot methods are at a disadvantage because the other six methods were allowed to train on data from the  evaluation time series\" (lines 280 - 282) seem a little disingenuous since the zero shot methods were pre-trained extensively (30 hours of pre-training for ForecastPFN), whereas in certain cases the six methods were only allowed 1 second of training?\n* The claim in line 173 - 175, that existing transformer models have prediction length to be fixed is inaccurate. This is a feature of Transformers in the long sequence time series forecasting setting, but not true of time series Transformers in general. See [1] for one such example.\n* Only number of wins/ranking is given for the results, but not the actual MSE scores. The actual MSE values should be given for readers to understand whether the comparisons are meaningful at all.\n* Following up on the previous point, 2 more very simple baselines should be added - the naive forecast (current value is taken as the forecast) and the historical mean. The number of wins/rankings doesn't really matter if any of these models can't beat such naive baselines (which is a fair comparison regardless of the train/time budget).\n\n[1] Rasul, Kashif, et al. \"VQ-TR: Vector Quantized Attention for Time Series Forecasting.\" (2022).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the excellent feedback. We are very glad to see that you view this direction as exciting and of huge significance, and that you found the paper well-written, neatly organized, and with comprehensive experiments. We very much appreciate your comments, which we believe will substantially improve the final version of our paper. We reply to each question below.\n\n**W1-2: “handicapped scenario” / “disingenuous claims.”**\nThank you for bringing up these points. We understand your points and have now made the experimental setup and settings very clear in our paper. We would like to bring up a few points for consideration. **The crux of this issue is that we focused on a novel paradigm that diverges significantly from prior work:** zero-shot forecasting; pretraining; and a setting that is important in practice: very limited in-distribution data and/or inference time. This makes it challenging to compare to prior work in general. To be clear, ForecastPFN is trained only once ever, and the weights are never changed when evaluating on new, unseen tasks. On the other hand, FEDFormer and the other transformers were not designed to be pretrained or to run zero-shot predictions. However, based on your comments, we *do* try out pretraining for FEDFormer. We find that after a week of attempting to train FEDFormer on the synthetic data, we were unable to achieve any non-degenerate performance on the 7 real-world datasets. Note that even training the ForecastPFN architecture (designed to be flexible and universal) was non-trivial: as described in the paper, we faced technical challenges such as handling scaling robustly, and a delicate parameterization of the synthetic data parameters.\n\nOverall, our work introduces a new paradigm in which prior works cannot easily fit, and our model also performs particularly well in an important setting -- the low resource setting, for which most prior work was not designed. We are now much more explicit with our claims by mentioning the exact training settings, and noting how these differ from the settings for which prior work was developed. Thank you very much for giving us the opportunity to further clarify these points in our paper.\n\n**W3. Transformer prediction length.**\nThank you for catching this! We have now corrected it.\n\n**W4. Raw MSE values.**\nThanks for this suggestion; we can easily add the raw MSE values. See [our reply to Reviewer MwwS](https://openreview.net/forum?id=tScBQRNgjk&noteId=Z6mb79DbuS) and also the one-page pdf.\n\n**W5. Add two additional baselines.**\nWe really like this suggestion. We added these two baselines; see the one-page pdf. We find that the historical mean achieves second place behind ForecastPFN in the lowest training and runtime settings. This is not unexpected and is a testament to the challenge of predicting a series when given very little training data or training time.\n\n**Q1. FEDFormer and other models were not pretrained.**\nThank you for this suggestion. We give our full answer in the first bullet point above: in general it is non-trivial to achieve non-degenerate zero-shot performance (even for specially designed architectures such as ForecastPFN), and we have not been able to do so for FEDFormer after a week of effort.\n\n**Q2. Meta-N-Beats setup**\nMeta-N-Beats was trained separately for each configuration of prediction length and dataset frequency, just like the other Transformer baselines. We use the subset of M4 according to the corresponding frequency. In contrast, the more flexible ForecastPFN can accommodate different frequencies and prediction lengths as a single model.\n\n**Q3 What is “number of MSE wins.”\nThis means that we compute the MSE for a given configuration and count the number of times each model has the best MSE. For example, in Figure 3, we count the number of wins for each model across 7 datasets and 7 prediction lengths, for each train budget.\n\n**Q4&6 Training on 1 second / Constrained budget for ARIMA/Prophet.**\nThank you, we will explain this better in the paper. Following the experimental setup of [TabPFN](https://arxiv.org/abs/2207.01848), we give each training run a time budget. After each training step, we stop the training if the total budget was exceeded. In case an algorithm is unable to output any predictions within the train budget (such as ARIMA with a 1 second budget), we set the output to 0’s.\n\n**Q5 Hardware used.**\nWe ran all the experiments ourselves on a V100; therefore, all of our results are fair to compare. We respectfully point out that since we include a diverse set of methods, the epoch/step times are not comparable, and so the best practice is to report the wallclock time using the exact same hardware, as we did. For example, [footnote 8 here](https://arxiv.org/abs/1909.02453) has a good discussion. We would be happy to include other compute resource metrics in the appendix while being explicit about the aforementioned caveats.\n\n**Limitations.**\nThank you for these great suggestions. We agree; our synthetic prior was created to focus on “human-like” or “earth-like” time series, but it would not do well on a purely “mathematical” time series with uncommon periods such as 41, 89, or 97. We also will clarify that our method only approximates the posterior predictive distribution and makes pointwise predictions. Since this is a new type of work, we focused on the pointwise predictions, but probabilistic predictions are an exciting area for future work.\n\nThank you very much, once again, for your excellent feedback. Your perspective and comments are very important for us to properly convey our work. We hope that you start to see our perspective on the main issues. Please let us know if you have any follow-up comments. If this starts to address your questions, especially since you agree that this direction is exciting and significant, we hope that you might reconsider giving our work a “strong reject” rating; we would very much appreciate it if you consider increasing your score."}, {"review_id": "pSnGYQhpcz", "reviewer": "Reviewer_458V", "summary": "The work present the ForecastPFG a zero-shot forecasting method trained using only synthetic dataset and it is evaluated on several real world dataset.", "questions": "Rows 71-74: This a repetition of what authors wrote in rows 16-20. This part should be improved with some details. The same in rows 83-85 and rows 23-24.\n\nThe relation between Zero shot forecasting, Transfer Learning in the forecasting context and the concept of Global Models is not discussed properly. I suggest authors to discuss more about this relation. \nRegarding global models authors can refer to recent works as: Januschowski et al., \"Criteria for classifying forecasting methods\", 2020; Buonanno et al., \"Global vs. Local Models for Short-Term Electricity Demand Prediction in a Residential/Lodging Scenario\", 2022; Montero-Manso, P.; Hyndman, R.J. \"Principles and algorithms for forecasting groups of time series: Locality and globality\", 2021, etc.\n\nRows 102-104\nAuthors use a multiplicative decomposition ($y_t = \\gamma(t) \\cdot z_t$). Why this choice and not the additive decomposition ($y_t = \\gamma(t) + z_t$)? How this choice impact on the results of the work?\n\nRow 109-110\nCould you clarify this tense? The $y_t = \\gamma(t) \\cdot z_t$ is always valid not only when $\\gamma(t)$ is deterministic. Moreover, if it is present $z_t$ sampled by a Gaussian, there are different possible output values and not only $y_T$. Do I miss something?\n\nRows 113-114\nThe athors wrote:\n\"We optimize the PFN’s parameters $\\theta$ by making predictions on a held-out test set from D\"--> Usually with test set is named the set used exclusively for the evaluation and that is not involved in learning of the model. In this case authors define a test set used to evaluate the loss used for the parameter tuning. I think that the name is misleading. Authors can evaluate to use different names (e.g. Training Input and Training Output) to identify the two parts of the timeseries used in the training phase.\n\nRows 129.\nIs not crucial to know the prior distribution of timeseries in order to have a good result on real dataset? In this work authors use a large dataset of timeseries that try to cover different situations but how the choice of the prior distribution impact on the results? Maybe this can be a idea for future works.\n\nRows 144\nIs there a motivation on the usage of Weibull distribution for the noise?\nHow a choice of a different noise distribution impacts on the results?\n\nRows 146-147\nNow $\\gamma(t)$ contains two contributions: trend and seasonal. In the row 102, instead, is written only trend. I got that in 102 authors want to introduce the general frame of the problem but I would not use the name trend in row 102.\n\nRows 169\n\"Predict the MSE of this query\" --> Does not the architecture predict the output? The next rows in fact the authors wrote \"the output is a single prediction at a future time\".\nAlso at row 178 it is written \"the loss\" and not the output.\n\nRows 180-183 \nIt is not clear to me why the classic scaling techniques are not appliable in this case. As discussed after, the problem is the outlier that has to be excluded since they can be very different across several timeseries. Please express better this issue.\n\nRows 188-191\nIs MSPE the mean squared prediction error? \nCould you describe mathematically the issue raised in this section because I didn't get it.\n\nRow 196\nHow authors evaluate the convergence?\n\nRow 235-236\nNot always the forecasting models are evaluated in retraining mode. For having a fairer comparison authors could force the forecasting model to not retrain during the evaluation test. This condition is, in fact, more similar to what happen in the real context where you don't retrain continuously your model once it is deployed. Moreover, this modality (no retrain) is followed in several works about forecasting. Using this modality (no retrain) we should have a fairer comparison between ForecastPFN and other models.\n\nRow 255\nCross product --> do you mean cartesian product?\n\nFigure 4\nIs the Time Budget the same of wall clock budgets defined in row 243\n\nRow 301\nIs not more important to compare classic standardization (without outlier removal) with robust scaling? The authors, instead, compared robust scaling with min-max scaling.", "limitations": "A limitation I see is the maximum number of training data used for the other models, limited to 500. It looks clear that with more other models outperform the ForecastPFN (Fig.5) while with few data ForecastPFN looks a good solution.\nMoreover, the training of ForecastPFN requires 30hr so it has a cost that should be taken in account for having a fair comparison with other approaches.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The paper is well written and the steps well described.\nMoreover, I think that it could be an interesting approach when you have very few data.", "weaknesses": "Weakness are inserted in the Question section", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the excellent and detailed feedback. We are glad to see your positive view of our work. Your comments and suggestions will greatly improve the final version of our paper. We reply to each question below.\n\n**Q1 Repetition of text.**\nThank you for catching this. We have now added more details to lines 71-74 and 83-85.\n\n**Q2: Relation to transfer learning and global models.**\nThank you for pointing this out. We will add a section on transfer learning, and another section on global models to our related work section. \n\n**Q3: Multiplicative decomposition.**\nYes, additive vs. multiplicative noise is an interesting question. We chose multiplicative noise to better balance the amount of signal to noise across all series. If we used additive noise, then the impact of the noise depends on the ratio of the scale of the base series and the scale of the noise. Since our base series have linear and exponential terms, additive noise would cause the signal-to-noise ratio to vary substantially over series with a trend component. Therefore, we used multiplicative noise, which is a simpler parametrization to achieve a signal to noise ratio for a series. We now updated our paper to include this discussion, and we agree that including an ablation study is a good idea for future work.\n\n**Q4 Clarify equations.**\nThank you for pointing this out. This is indeed a typo, because $p(y_T|T,\\varphi)=1$ is only true if there is a deterministic function $y_t=\\varphi(t)$ with no noise, and this is not the case we consider in Section 3.2.\n\n**Q5 Clarification on test set.**\nThank you for catching this! As you say, this is simply a naming mistake. We will now refer to the two parts of each training series as “training input” and “training output.”\n\n**Q6 Ablation on prior distribution.**\nWe agree that the choice of prior distribution is important and that it would be good to have an ablation study. It is challenging to do a thorough ablation study, since it takes 30 GPU hours to train a model. As mentioned in the paper, we did light tuning in the early stages of the project, using the training loss. We are currently working on a simple ablation study on the noise of the synthetic data, which we will finish during the NeurIPS discussion period.\n\n**Q7 Weibull distribution.**\nThis is a great question. Real world datasets seem to frequently have two types of noise: Gaussian and exponential; the latter is in the form of non-negative time series with high value outliers. The Weibull distribution is a simple and natural way to parametrize between these two distributions. While we do not claim the Weibull distribution is representative of the noise of all real world series, getting convergence on Weibull is already particularly non-trivial. Further study of the noise distribution is an interesting idea for future work.\n\n**Q8 Naming of phi.**\nWe agree that it is clearer to replace “trend” in line 102 with a different phrase, such as “base series.”\n\n**Q9 Prediction.**\nThanks for pointing this typo on rows 169 and 178. Yes, the architecture predicts the output, and then we compute the MSE of the output with the ground truth.\n\n**Q10 Classic scaling techniques.**\nYes, we will make this part clearer in the paper. Since ForecastPFN is designed to be a universal forecasting model that can handle a huge variety of time series, appropriately scaling the data becomes tricky. Our robust scaling procedure as described on line 184 gives the best level of standardization and outlier removal, which simpler methods such as min-max scaling or z-score normalization do not achieve.\n\n**Q11 MSPE.**\nBy MSPE, we are referring to the [mean squared percentage error](https://www.sktime.net/en/stable/api_reference/auto_generated/sktime.performance_metrics.forecasting.mean_squared_percentage_error.html). \nIt is a great idea to give the equations of MSE, MSPE, and our scaled MSE. We will make this change to the final version of our paper.\n\n**Q12 Convergence.**\nWe used the word convergence informally to mean that the training loss is much higher than the final trained ForecastPFN. So, if we set the noise of the synthetic data too high, the training loss is very high. We will fix this in the text.\n\n**Q13 Retraining.**\nThank you for pointing out this confusion. By “not a fair comparison”, we meant that the baseline methods are allowed to train on earlier time series from the same dataset. At test time, we do not permit the baseline methods to train on the test input. Indeed, we do the common technique of training the algorithms once per dataset and prediction length, and then, at test time, providing input to the model for $n$ timestamps and then asking the model to predict $l$ points in the future. We have updated the manuscript to clear up this confusion.\n\n**Q14 Cross product.**\nYes, in row 255 we meant to say the cartesian product. Thank you for spotting this.\n\n**Q15 Figure 4.**\nYes, “training time” in row 243 is the same as “time budget” in Fig. 4. We will make this clear in the paper.\n\n**Q16 Standardization.**\nThat is a good point. In future work, we can do a much larger ablation study for scaling methods and the data distribution. It is challenging to do a large ablation study because each ForecastPFN model takes 30 GPU hours of pretraining.\n\nThank you very much, once again, for your positive view of our work and your excellent suggestions. If you find our responses satisfying, we respectfully ask that you consider increasing your score. On the other hand, we would be very happy to answer any follow-up or additional questions you have."}]

Author rebuttal:
We thank all reviewers for their valuable feedback and suggestions. Our work introduces the first synthetically-trained, zero-shot, universal forecasting model, which performs particularly well in low-resource settings (small amount of in-distribution data, and/or low inference time budget), which are very important settings in practice. We appreciate that the reviewers find our method novel and exciting (​​**MwwS, 5AR9, 3SYZ**), our paper well-written and easy to follow (**MwwS, 3SYZ, 458V**), our experiments extensive and compelling (**5AR9, 3SYZ**), and the low-resource setting interesting and impactful (**MwwS, 5AR9, 458V**).

We would like to go over a few small points of confusion, just so that we are all on the same page. At a high level, we found that since our work is a substantially different paradigm from most prior work (which most reviewers mentioned as an exciting direction), there was also some confusion about our method and experimental setup. Please see our points below.
- **ForecastPFN was trained only once, ever.** Similar to foundation models such as GPT4, we only needed to train ForecastPFN once, and the weights are never updated even when evaluating on new, unseen time series.
- **ForecastPFN is a _universal_ forecasting model**. ForecastPFN, a single model, can be used for different input lengths, prediction lengths, and frequencies. It empirically performs well in different settings such as currency exchange, weather, consumer electricity usage, and illness trends. The time to output predictions for a brand new series is under one second -- just a model evaluation. To the best of our knowledge, ForecastPFN is the first model to have this level of universality, and the first to be pretrained on synthetic data.
- **We overcame significant technical challenges to create a universal, pretrained model**, such as handling dynamic scaling, designing a general and flexible architecture, and carefully designing and parameterizing the synthetic data, especially the noise distribution. **Other architectures such as FEDFormer are not able to pretrain with this level of generality** (see below).

We would also like to highlight the additional experiments we finished during the rebuttal period.
- **The FEDFormer architecture (the second-best algorithm in our experiments) cannot be pretrained with our synthetic data.** Based on the suggestions of reviewer 3SYZ, we (attempted to) pretrain FedFormer using our synthetic data. Although we spent significant time during the week-long rebuttal period, we so far are not able to get the FEDFormer architecture to achieve non-degenerate performance on the real-world datasets. Note that as we mentioned in a point above, it is non-trivial to design and train a model to achieve zero-shot, universal forecasting performance, and the ForecastPFN architecture was specifically designed for this goal.
- **We included several new tables of raw MSE values, as well as table versions of our main figures**. Based on the suggestions of reviewers MwwS and 3SYZ, we included raw MSE values as well as more tables, to better view our results. See these tables in [our reply to Reviewer MwwS](https://openreview.net/forum?id=tScBQRNgjk&noteId=Z6mb79DbuS) and in our one-page pdf.
- **We added two very simple baselines: historical mean, and previous value.** See the figures in our one-page pdf. We see that the historical mean baseline does very well, achieving second-best performance behind ForecastPFN when there is very low train budget or train time. This is a testament to how challenging it is to predict with very little training data or very little training time.

We would be very happy to keep the discussion going for any points that might be still unclear or any new comments. Thank you very much, once again, for giving suggestions for our paper.

Meta-review: This paper proposes a new approach for zero-shot forecasting in data-sparse applications by training a model purely on a novel synthetic data distribution with prior distribution similar in properties to real-world data.

It is clear that, if successful, this approach will be very useful for data-sparse applications. The idea is also interesting and has not received much attention in the past. Extensive experiments are reported showing significant improvement over the baselines. Moreover, the paper is generally well written. However, it does have much room for improvement as reflected by the comments and questions raised by the reviewers. We thank the authors for trying to answer all the questions raised and conducting further experiments when answering some of the questions.

While we generally agree that this work has merits to be accepted, we hope the authors will take the comments and suggestions seriously to revise the paper to make it appeal better to the research community.
Decision: Accept (poster)