SlowGuess/NeurIPS_2023_OCR huggingface dataset:
- Daily:  $m_{\text{week}} \sim U([0,1])$ ,  $m_{\text{month}} \sim U([0,.2])$ ,  $p_{\text{week}} = 7$ ,  $p_{\text{month}} = 30.5$ .
- Weekly:  $m_{\text{month}} \sim [0, .3]$ ,  $m_{\text{year}} \sim U([0, .1])$ ,  $p_{\text{month}} = 2$ ,  $p_{\text{year}} = 52$ .
Monthly:  $m_{\mathrm{year}} \sim U([0, .5])$ ,  $p_{\mathrm{year}} = 12$
$$
m _ {\text {l i n}}, m _ {\text {e x p}} \sim \mathcal {N} (-. 0 1, 0. 5); \quad c _ {\text {l i n}} \sim \mathcal {N} (0,. 0 1); \quad c _ {\text {e x p}} \sim \mathcal {N} (1, 0. 0 0 5).
$$
As described in Section 3.3 and the next section (Appendix E.1), we set these parameters to create the most diverse data distribution possible, with the model still being able to train without diverging or stalling. For example, when the parameter for the noise distribution becomes too large, the model cannot converge. We also released a README for our synthetic data generation at https://github.com/abacusai/forecastpfn.
For visualizations of daily time series drawn from this distribution, see Figure 2 and Figure 7.
E Additional Experimental Details
E.1 ForecastPFN Development
We estimate that the total computational budget used in this project, from initial research to running all the final evaluations, is 500 GPU hours. Our computation consisted mainly of designing the architecture and designing the synthetic data distribution. In this design phase, we used the train loss of the synthetic data as a signal, because many synthetic data distributions, and architectures, cause the model not to train, for example, setting the noise parameter too high. We also used the synthetic held-out validation set to make decisions, such as in Figure 5 and Appendix E.5.
The training of our ForecastPFN model is 30 GPU hours on a single Tesla V100 16GB GPU. We emphasize that this model training is only done once, and we released the trained ForecastPFN model.
For an ablation study, we tried to achieve non-trivial zero-shot performance with the FedFormer architecture. However, after significant effort, we were unable to achieve non-degenerate performance with the FedFormer architecture.
E.2 Implementation Details
ForecastPFN Implementation Details. We use an encoder-based transformer architecture, with two encoders. Each encoder consists of a multi-head attention layer with four heads, and two feedforward layers. The first feedforward layer is size 32 times the embedding dimension, and the second feedforward layer is size 8 times the embedding dimension.
Baseline Implementation Details. We compare ForecastPFN against ARIMA [4], Autoformer [50], FEDformer [54], Informer [52], Meta-N-BEATS [36], Prophet [45], and a vanilla Transformer method [47]. All of the methods were implemented using their default hyperparameters and official codebase. For ARIMA (which was designed in the 1970s and has no 'official' codebase), we used pdarima [43]. The full implementation details of all methods can be found in our open source codebase. Furthermore, we compared to three baselines: Last (prediction is the last seen value in the series), Mean (prediction is the mean value in the series), and SeasonalNaive (prediction is the mean value from all prior instances of the same day of the week).
In Section 4, in the experiments with a restricted time budget, we restrict the time budget of the non-zero-shot methods to wall clock budgets from 1 to 120 seconds, where timing is calculated after data loading and only during the training loop. After each training step, we stop the training if the total budget was exceeded. In case an algorithm is unable to output any predictions within the data budget (such as ARIMA with a 1 second budget), we set the output to 0's.
E.3 Training Details
We generate 100 000 each of synthetic daily, weekly, and monthly series, each of length 200, and we use a sliding window of size 100 to obtain 101 prediction tasks per series. We sample 1024 tasks in a training step, and there are 1000 training steps in an epoch. We trained the transformer for 300 epochs on a V100 16GB GPU, which took 30 hours.
We use the Adam optimizer with a learning rate of 0.0001, and MSE loss. We set the maximum prediction into the future to be 10 times the frequency of the training series.
E.4 Dataset Details
We use seven datasets for our experiments in Section 4. These datasets are standard for benchmarking the performance of forecasting methods and are the same ones used by the majority of our baselines [36, 50, 52, 54].
Illness is a dataset [17] containing influenza-like illness patients in the United States. It contains 966 datapoints, spaced weekly. Exchange is a currency exchange dataset [28], consisting of 7588 datapoints, spaced daily. Traffic is a dataset consisting of freeway congestion across California [39]. ETT (Electricity Transformer Temperature) is a electricity power deployment dataset [52] across 2 years from a power plant in China. The target value is oil temperature. Two separate series
Time Budget	1.0	5.0	10.0	15.0	30.0	45.0	60.0	120.0
Arima	9.68	77.42	88.71	92.9	100	100	100	100
Autoformer	70.97	97.74	100	100	100	100	100	100
FEDformer	71.94	99.68	100	100	100	99.35	100	100
Informer	72.58	100	100	100	100	100	97.42	100
Prophet	88.71	100	100	100	100	100	100	100
Transformer	66.77	100	100	100	98.39	100	100	100
Table 3: The percentage of configurations (dataset, prediction lengths, seeds) that failed, for each time budget and each model which requires training or being fit.
are provided: ETT1 and ETT2. Each series represents data collected from two different electricity transformer stations. ECL (Electricity Consuming Load) [46] is an electricity consumption (Kwh) dataset across two years with a resolution of every hour. This dataset has a total of 26304 hourly observations. Weather is a dataset [1] that contains local climate data from Germany, across three years. The datasets ETT, ECL, and Weather all natively have a sub-day resolution which we aggregate into daily observations by taking the sum of values in day.
E.5 Noise Removal Ablation
Recall that crucially, since we use synthetic training data with a ground-truth trend and a separate noise component, we remove the noise when calculating the train loss. Now, we give an ablation study to confirm that this design decision significantly improves the performance of our model.
We investigate the impact of this design decision on the performance of ForecastPFN by comparing the model with the noise removed (that which is included in the main body of the paper) with a training run conducted without the removal of the noise from the training loss calculation. In doing this, we calculate the validation MSE on the synthetic data for both models and observe 33.089 for the model without noise removed and 22.692 for the model with the noise removed. We conclude the use of the noise removal during training loss calculation leads to improvements on the synthetic validation set, and thus we deploy the use of noise removal in the final model.
This design decision significantly improves performance, and it can only be done when using synthetic data, rather than real data.
E.6 Training Time Analysis
Recall that during our benchmark, there were six models which required training data on a portion of each of the datasets before they were able to be tested. Further recall that we established a time budget for this training process, and varied that timing from 1 to 120 seconds. Some of these models were unable to fit to the training data during some of those training budgets. In Table 3, we calculate the percentage of configurations (dataset, prediction lengths, seeds) which failed for each model and training time budget. We see that Arima is the model which struggles the most at low time budgets.
E.7 Additional Experimental results
In Figure 16 and Figure 6, we plot the mean MSE rank per prediction length (similar to Figure 3 and 4 respectively, but for mean MSE rank instead of MSE wins).
In Figures 8 to 20, we give similar plots to Figures 3, 4 and 6 but instead of MSE, we plot MAE, MAPE, and MSPE. We also include plots for the evaluation of mean rank.
Mean MSE Rank per Prediction Length By Data Budget (50 to 500)
Figure 6: Analysis of performance across prediction lengths for increasing data budgets (left subplot, data budget is 50 to the right subplot, data budget is 500), aggregated across datasets. Mean MSE rank for each scenario is plotted with error bars as one standard deviation across training runs. Recall, ForecastPFN and Meta-N-BEATS see no training data for these series, only the length 36 input.
Figure 7: Five examples of time series drawn from the synthetic distribution we define in Section 3.2 (continuation of Figure 2). Each row shows the components that make up a time series: seasonal trends (far left), global trends (center left), and noise (center right), as well as the entire time series (far right). We plot only half of the series in order to show the seasonal trends clearly.
Figure 8: Analysis of performance vs. data budget, aggregated across datasets and prediction lengths. We plot the number of total MAE wins (left) where a higher value is better and mean MAE rank (right) where a lower values is better. Error bars show one standard deviation across training runs. Recall, ForecastPFN and Meta-N-BEATS see no training data for these series, only the length 36 input.
Figure 9: Analysis of performance across time budgets, aggregated across datasets and prediction lengths. We plot the number of total MAE wins (left) where a higher value is better and mean MAE rank (right) where a lower values is better. Error bars show one standard deviation across training runs. Recall, ForecastPFN and Meta-N-BEATS see no training data for these series, only the length 36 input.
MAE Wins per Prediction Length By Data Budget (50 to 500)
Figure 10: Analysis of performance across prediction lengths for increasing data budgets (left subplot, data budget is 50 to the right subplot, data budget is 500), aggregated across datasets. Total MAE wins for each scenario is plotted with error bars as one standard deviation across training runs. Recall, ForecastPFN and Meta-N-BEATS see no training data for these series, only the length 36 input.
Mean MAE Rank per Prediction Length By Data Budget (50 to 500)
Figure 11: Analysis of performance across prediction lengths for increasing data budgets (left subplot, data budget is 50 to the right subplot, data budget is 500), aggregated across datasets. Mean MAE rank for each scenario is plotted with error bars as one standard deviation across training runs. Recall, ForecastPFN and Meta-N-BEATS see no training data for these series, only the length 36 input.
Figure 12: Analysis of performance vs. data budget, aggregated across datasets and prediction lengths. We plot the number of total MAPE wins (left) where a higher value is better and mean MAPE rank (right) where a lower values is better. Error bars show one standard deviation across training runs. Recall, ForecastPFN and Meta-N-BEATS see no training data for these series, only the length 36 input.
Figure 13: Analysis of performance across time budgets, aggregated across datasets and prediction lengths. We plot the number of total MAPE wins (left) where a higher value is better and mean MAPE rank (right) where a lower values is better. Error bars show one standard deviation across training runs. Recall, ForecastPFN and Meta-N-BEATS see no training data for these series, only the length 36 input.
Figure 14: Analysis of performance across prediction lengths for increasing data budgets (left subplot, data budget is 50 to the right subplot, data budget is 500), aggregated across datasets. Total MAPE wins for each scenario is plotted with error bars as one standard deviation across training runs. Recall, ForecastPFN and Meta-N-BEATS see no training data for these series, only the length 36 input.
Figure 15: Analysis of performance across prediction lengths for increasing data budgets (left subplot, data budget is 50 to the right subplot, data budget is 500), aggregated across datasets. Mean MAPE rank for each scenario is plotted with error bars as one standard deviation across training runs. Recall, ForecastPFN and Meta-N-BEATS see no training data for these series, only the length 36 input.
Figure 16: Analysis of performance vs. data budget and time budget, aggregated across datasets and prediction lengths. We plot the number of total MSE wins (left) where a higher value is better. Error bars show one standard deviation across training runs. Recall, ForecastPFN and Meta-N-BEATS see no training data for these series, only the length 36 input.
Figure 17: Analysis of performance vs. data budget, aggregated across datasets and prediction lengths. We plot the number of total MSPE wins (left) where a higher value is better and mean MSPE rank (right) where a lower values is better. Error bars show one standard deviation across training runs. Recall, ForecastPFN and Meta-N-BEATS see no training data for these series, only the length 36 input.
Figure 18: Analysis of performance across time budgets, aggregated across datasets and prediction lengths. We plot the number of total MSPE wins (left) where a higher value is better and mean MSPE rank (right) where a lower values is better. Error bars show one standard deviation across training runs. Recall, ForecastPFN and Meta-N-BEATS see no training data for these series, only the length 36 input.
Figure 19: Analysis of performance across prediction lengths for increasing data budgets (left subplot, data budget is 50 to the right subplot, data budget is 500), aggregated across datasets. Total MSPE wins for each scenario is plotted with error bars as one standard deviation across training runs. Recall, ForecastPFN and Meta-N-BEATS see no training data for these series, only the length 36 input.
Figure 20: Analysis of performance across prediction lengths for increasing data budgets (left subplot, data budget is 50 to the right subplot, data budget is 500), aggregated across datasets. Mean MSPE rank for each scenario is plotted with error bars as one standard deviation across training runs. Recall, ForecastPFN and Meta-N-BEATS see no training data for these series, only the length 36 input.