GIFT-Eval: A Benchmark for General Time Series Forecasting Model Evaluation

Authors: Taha Aksu, Gerald Woo, Juncheng Liu, Xu Liu, Chenghao Liu, Silvio Savarese, Caiming Xiong, Doyen Sahoo
Primary area: datasets and benchmarks
Keywords: benchmark, time series forecasting, foundation models, forecasting, univariate forecasting, multivariate forecasting, pretraining data, deep learning, statistical models, foundation models, dataset

Abstract:
Time series foundation models excel in zero-shot forecasting, handling diverse tasks without explicit training. However, the advancement of these models has been hindered by the lack of comprehensive benchmarks. To address this gap, we introduce the **G**eneral T**I**me Series **F**orecas**T**ing Model **Eval**uation, **GIFT-EVAL**, a pioneering benchmark aimed at promoting evaluation across diverse datasets. GIFT-EVAL encompasses 28 datasets over 144,000 time series and 177 million data points, spanning seven domains, 10 frequencies, multivariate inputs, and prediction lengths ranging from short to long-term forecasts. To facilitate the effective pretraining and evaluation of foundation models, we also provide a non-leaking pretraining dataset containing approximately 230 billion data points. Additionally, we provide a comprehensive analysis of 20 baselines, which includes statistical models, deep learning models, and foundation models. We discuss each model in the context of various benchmark characteristics and offer a qualitative analysis that spans both deep learning and foundation models. We believe the insights from this analysis, along with access to this new standard zero-shot time series forecasting benchmark, will guide future developments in time series foundation models.

Decision: reject
Reviewer scores: 5, 5, 6, 5

Reviews and discussion:
<thread>
creation date:
1730623741786

thread_type:
official_review

<thread_object>
creator: reviewer_pz22

<content>
summary:
The existing benchmark evaluations are incomplete (lacking evaluation of foundation models), so the paper introduces the General Time Series Forecasting Model Evaluation (GIFT-Eval), including statistical models, deep learning models, and foundation models. It tests on 28 different test datasets and provides a large-scale pretraining dataset to better evaluate foundation models. Finally, the paper offers a qualitative analysis that spans both deep learning models and foundation models.

strengths:
S1: Model Level: GIFT-Eval provides a comprehensive evaluation for time series forecasting models, including statistical models, deep learning models, and foundation models.

S2: Experiment and Dataset Level: GIFT-Eval conducts extensive experiments, testing on 28 different test datasets, and also provides a non-leaking pretraining dataset to better evaluate foundation models

weaknesses:
W1: I believe that analyzing foundation models based on four time series characteristics—domain, frequency, prediction length, and the number of variates—combined with six time series features—trend, seasonality, entropy, Hurst, stability, and lumpiness—is not very meaningful, especially regarding the number of variates. I don't think it has any relation to these six features. Why not directly analyze the time series features for each test dataset, and then evaluate the performance of foundation models on various datasets to assess their strengths and weaknesses concerning these six features?

W2: The paper only analyzes the features of the test data. It is necessary to include an analysis of the pretraining data as well. This would help better assess whether the foundation models perform well on these features due to the presence of these characteristics in the pretraining data itself, the generalization ability of the foundation models, or other reasons.

W3: Please elaborate on the details of the data splitting for train/val/test datasets, specifically how the validation data is constructed. What does the statement "The final window of the training data serves as validation" mean, and why is a common splitting ratio like 7:1:2 or 6:2:2 not used?

W4: The experimental settings are unclear, for example: what are the input and output lengths for short-term, medium-term, and long-term forecasting? Are the foundation models performing zero-shot or full-shot forecasting in Tables 6-10? Why are common metrics like MSE and MAE not chosen for point forecasting?

W5: About experimental results: 1) PatchTST performs best on multivariate datasets, so why is Moirai, a model specifically designed for variable correlation, not used? Also, why do foundation models perform well on univariate data but not on multivariate data? 2) The paper states that Moirai has good prediction performance in short-term forecasting; does this conclusion contradict the original Moirai paper, which uses very long input sequences?

W6: The qualitative analysis is relatively limited; more in-depth analysis (not just visualization) may be needed to highlight the characteristics of the foundation models and draw more meaningful conclusions.

questions:
See W1-W6.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730659891780

thread_type:
official_review

<thread_object>
creator: reviewer_ah9k

<content>
summary:
In this work the authors introduce a large collective time series benchmark named GIFT-EVAL. They demonstrate the diversity and legitimacy of this benchmark with the analyses on both the characteristics of involved datasets and the benchmark results of state-of-the-art supervised and foundation time series models on it.

strengths:
This work address an important and pressing problem of the time series forecasting community that there lacks a large and comprehensive common benchmark. The collection and complication of the datasets itself is a significant result.

weaknesses:
Despite the comprehensive data analyses, the paper currently lacks the reasoning behind the selection of GIFT-EVAL components. And the empirical study follows suboptimal standards which makes the results less convincing. See questions.

questions:
Regarding the creation of GIFT-EVAL:
1. As mentioned in the weakness, GIFT-EVAL seems to be a straightforward ensemble of some available datasets. What are the reasons behind the selection of this particular ensemble vs the datasets, especially given the data analytics done on them?

Regarding benchmarking:
1. My major concern of the work: MAPE is well recognized as a bad point forecasting metric on its own, due to e.g. its favoring underprediction, sensitivity near ground truth 0 (this is recognized by the authors too), (see e.g., [1] and many community posts online). It would be helpful to switch to or also report other more robust metrics, e.g. MASE. 
2. In case normalized metrics are used, consider reporting geometric means [2].
3. Given the 6 properties of component datasets, it would be helpful to see the benchmark results sliced on these properties as well.

[1] Goodwin, Paul, and Richard Lawton. "On the asymmetry of the symmetric MAPE." International journal of forecasting 15.4 (1999): 405-408. 
[2] Fleming, Philip J., and John J. Wallace. "How not to lie with statistics: the correct way to summarize benchmark results." Communications of the ACM 29.3 (1986): 218-221.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730717243668

thread_type:
official_review

<thread_object>
creator: reviewer_sxef

<content>
summary:
This paper introduces GIFT-Eval, a benchmark aimed at evaluation for time series foundation models across diverse datasets. GIFT-Eval encompasses 28 datasets and divides them according to 4 characteristics and 6 features. It also includes a non-leaking pretraining dataset. It then evaluates 17 different baseline models, including 4 types of foundation models and some statistical and deep learning models on this benchmark. Based on the evaluation results, it discusses different models in the context of various benchmark characteristics and offers some qualitative analysis.

strengths:
* Foundation models are a new and promising research topic in time series forecasting. It is of value to establish a benchmark for evaluating these models.

* This paper does a lot of work in collecting large-scale data for evaluation and running various baseline models on these various datasets.

weaknesses:
* This paper emphasizes the inclusion of a non-leaking pretraining dataset, but its value and usage are not clear. Is this dataset used to re-train all foundation models instead of using their public checkpoints? Is it necessarily better than the original pretraining datasets of each foundation model? Since the application on downstream data is the main goal of foundation models, do we really need to keep consistency in pretraining to evaluate these models as discussed in the Introduction?

* Section 3.1.1 mentions covariates in time series forecasting, but it is unclear how are the covariates considered in this benchmark. Can all baselines in the benchmark perform forecasting with covariates?

* The divisions of some characteristics such as variates and frequencies are not reasonable enough, and it is unclear why we should evaluate methods according to these characteristics. For example, Multivariate v.s. Univariate may not be the intrinsic differences that influence forecasting, as even some multivariate datasets do not have strong correlations between variates. So simply claiming that some models perform best on multivariate or univariate datasets may be misleading.

* The experiments mainly point out that some specific models perform best in some specific datasets. There are not many consistent conclusions from the evaluations that help us understand the characteristics of different models. Some results or claims are also confusing. In Page 7, how can we get the conclusion that ‘foundation models consistently outperform both statistical and deep learning models’ from Table 6, since PatchTST also performs very well? In Page 8, what is the meaning of ‘This trend indicates that the fine-tuning of foundation models effectively captures longer-term dependencies’? In Page 9, why does Moirai, which is a foundation model considering multivariate correlation, perform best on univariate datasets instead?

* The qualitative results only show results in some special cases. It is doubtful whether these phenomena and analyses are general to all different data. It is also confusing that different data are used to evaluate foundation models and other models, which makes these models incomparable.

questions:
Please see Weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730717659395

thread_type:
official_review

<thread_object>
creator: reviewer_fpfr

<content>
summary:
This paper targets at a critical and longstanding challenge in time-series forecasting research, lacking a unified, comprehensive, and diverse benchmark for evaluation. To address this challenge, GIFT-Eval is developed, encompassing 28 datasets across seven domains and sampled in ten different frequencies.

strengths:
- Covering a diverse range of datasets and sampling frequencies, which can greatly facilitate our comprehensive understanding of existing time-series models
- Distinguishing pre-training and evaluation datasets, which can assist the comparison between supervised trained models (in-domain generalization) and pre-trained models (zero-shot forecasting)
- Conducting substantial experiments on these datasets (# experiments = # datasets x # sampling frequencies x  # models, such enormous computation!)

weaknesses:
Overall, I appreciate and respect this paper, given its important research focus and huge amounts of experiments performed.

However, I find the current version of this paper is still on an early stage. Below I elaborate on my significant concerns, which need to be properly addressed before it reaches an acceptance bar.

### 1. The Necessity of Additional Evaluation Datasets

MOIRAI [1] has already done an excellent job in collecting pre-training datasets and conducting comprehensive evaluations on widely recognized datasets typically used to evaluate time-series forecasting models in the literature. The datasets introduced in this paper heavily overlap with them while creating a new division for pre-training and downstream evaluation. Currently, this paper argues improvements in quantity, namely more evaluation datasets and sampling frequencies, are included in the evaluation. However, the necessity of introducing more evaluation datasets and how this approach reveals unique insights not mentioned in previous research is relatively weak.

Covering an excess of comparison datasets can create a significant burden for resource-limited research groups to perform further research and for reviewers to check and compare results. This may lead to redundant model training and evaluation, which is energy-inefficient. I strongly recommend the authors analyze the uniqueness and necessity of any dataset introduced as a new testbed.

For example, when incorporating a new dataset, does it cover distinctive patterns that rarely exist in existing benchmarks, thereby delivering unique insights on different pros and cons of model designs?

The same question extends to the different sampling frequencies applied to each dataset. Is it necessary to include all sampling frequencies for every dataset? Does including the most prominent sampling frequency, which may be defined by different business scenarios, already provide sufficient and fair comparisons of different models?

According to Tables 20, 21, and 22, the total comparison covers 28 datasets, each with multiple sampling frequencies. Are these experimental comparisons overly redundant? For instance, can you identify a compact set of evaluation datasets, covering unique datasets accompanied by selected sampling frequencies, that still reveal comprehensive information while being much more energy-efficient, user-friendly, and economical? Perhaps you could calculate the real rank for the result comparison matrix presented in these tables (# datasets x # models).

Moreover, analyzing the data patterns in Figure 1 is beneficial, but how does the pattern coverage of these new benchmark datasets differ from that of existing benchmarks? Do newly introduced datasets include more pronounced trends and seasonality, or large entropy, etc.?

### 2. Lack of Some Related Work and Baselines

The comparison with previous benchmarks omits some classic and recent studies specializing in probabilistic forecasting. For instance, gluon-ts [2] is a Python package for probabilistic time-series forecasting that also provides a robust interface for accessing multiple time-series datasets. Built on gluon-ts, pytorch-ts [3] includes more advanced probabilistic forecasting models based on deep generative models. ProbTS [4] is another benchmark study offering a unique perspective by comparing capabilities in delivering point versus probabilistic forecasts, short versus long forecasts, and associated preferences in methodological designs. Specifically, ProbTS should be compared in Table 1, as it is highly relevant to your work in comparing both classical and foundation models. It unifies comparison conditions, covers diverse forecasting horizons and data patterns, calculates dominant data characteristics (such as trend, seasonality, and non-Gaussianity), and associates them with the strengths and weaknesses of different model designs.

Moreover, other time-series foundation models have been developed beyond MOIRAI, chronos, and TimesFM. Notably, Timer [5] and UniTS [6] have been accepted at conference proceedings and have publicly released their implementations. These models should at least be discussed in the related work section and, ideally, be included in your experimental comparisons.

Additionally, the paper could benefit from including more advanced probabilistic forecasting baselines, such as TimeGrad [7], CSDI [8], and their predecessor GRU NVP [9]. ProbTS has highlighted the unique advantages of these methods in delivering short-term distributional forecasting. Moreover, a simple combination of GRU NVP with RevIN [10] has demonstrated very competitive performance for both short-term and long-term forecasting. Including these more powerful probabilistic models is crucial, as merely adding probabilistic heads over forecasting models like MOIRAI and DeepAR does not sufficiently capture complex data distributions that extend beyond closed-form probabilistic distribution functions.


### 3 Some Missing Details and Analyses in Experiments

The use of MAPE as an only metric for evaluating point forecasts is somewhat "biased." I recommend referring to N-BEATS [11] and including metrics such as sMAPE and ND (normalized deviation, equivalent to normalized MAE) for a more comprehensive evaluation.

Regarding hyperparameter search for deep learning baselines, there is a notable omission in tuning their lookback lengths. This can be an extremely critical factor to adjust, given the diversity of datasets and sampling frequencies. Appendix A indicates that this tuning was performed only for MOIRAI. The same process should be applied to other baselines, including supervised learning models and pre-trained foundation models.

Moreover, regarding three foundation models used in the experiments, only MOIRAI is re-trained on this new data split while TimesFM and Chronos are compared with their released model checkpoints. There is a risk of data leakage in these experiment comparisons. For example, TimesFM has leverage the Electricity dataset for pre-training while this dataset is also used for evaluation in this paper. This may explain why TimesFM performs extremely on "Electricity, short, H" (line 1285, Table 20), but this good performance may come from test data leakage. Therefore, to provide a fair and comprehensive comparison across TimeFM, Chronos, MOIRAI, and other representative time-series foundation models, re-training them following the new data split could be a necessary step.

When comparing supervised time-series models with zero-shot foundation models, it is crucial to investigate the effect of allowed lookback length on forecasting performance. As revealed in MOIRAI, the lookback length significantly influences model performance, as MOIRAI employs an additional hyper-parameter adaptation process on the lookback data, unlike others.

The current analysis lacks depth across multiple experimental configurations. Presently, only aggregated comparisons are provided, without detailed analyses to derive new insights. For instance, why do some foundation models yield better zero-shot forecasting results on certain datasets than models specifically tuned for those datasets? Is it because pre-training datasets encompass more related patterns, pre-trained models have a larger model capacity, or classical models are not well-tuned, particularly regarding lookback length, modeling layers, or model designs? Merely presenting results without in-depth analysis can be detrimental to the research community, as others may expend significant efforts to re-analyze numerous results, which should be addressed within this benchmark.

Additionally, there is a lack of explicit connection with existing benchmarks to validate the reliability of the experiments conducted. For example, evaluation datasets in Tables 20, 21, and 22 cover some classical datasets like ETTh, Electricity, Solar, and M4. Comparing your results on shared datasets with existing studies could demonstrate the reliability of your experimental protocols. Moreover, some widely used datasets in the literature, such as Traffic, Wikipedia, and Exchange, appear to be excluded. Please clarify the reasons for their exclusion.

I suggest maintaining existing classical benchmarks as they are while introducing new datasets and sampling frequencies that cover unique patterns. This approach allows for consistency with existing benchmarks, showcases the correct reproduction of existing models, demonstrates the necessity of new datasets and sampling frequencies, and reveals what can be discovered given this new benchmark.

[1] Unified Training of Universal Time Series Forecasting Transformers, https://arxiv.org/abs/2402.02592

[2] gluon-ts, https://www.jmlr.org/papers/v21/19-820.html

[3] pytorch-ts, https://github.com/zalandoresearch/pytorch-ts

[4] ProbTS, https://arxiv.org/abs/2310.07446

[5] Timer, https://arxiv.org/abs/2402.02368

[6] UniTS, https://arxiv.org/pdf/2403.00131

[7] Autoregressive Denoising Diffusion Models for Multivariate Probabilistic Time Series Forecasting, https://arxiv.org/abs/2101.12072

[8] CSDI: Conditional Score-based Diffusion Models for Probabilistic Time Series Imputation, https://arxiv.org/abs/2107.03502

[9] Multivariate Probabilistic Time Series Forecasting via Conditioned Normalizing Flows, https://arxiv.org/abs/2002.06103

[10] RevIN, https://openreview.net/forum?id=cGDAkQo1C0p

[11] N-BEATS, https://arxiv.org/abs/1905.10437

questions:
See weaknesses.

My critical concerns centered around the rationale of selecting/adding/filtering evaluation datasets and sampling frequencies, the missing discussion of some highly related studies, and incomplete experimental results and analyses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732273121195

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Paper Update Summary

comment:
We sincerely thank reviewers for their comments, which have helped us improve the paper. We have revised the manuscript and uploaded the updated PDF to the OpenReview system. Note that all changes in the paper are highlighted with red color and prepended with the capital letter R.

We also provide the anonymized code implementation alongside a subset of our dataset and sample notebooks, to allow interested reviewers to gain hands-on experience with our data: https://anonymous.4open.science/r/GIFT-Eval-1BFD/README.md.

Below, we provide a summary of the changes made:

1. Table 1: Added ProbTS as a comparison.  (Reviewer FpFr)
2. Section 2, RW: Added reference to more probabilistic + foundation models, and discussion for forecasting tools.  (Reviewer FpFr)
3. Tables 2-7 and 22-24 updated to report MASE instead of MAPE. (All Reviewers)
4. Tables 2-7, 16-24 are updated to use the public Moirai model instead of the retrained version on our pretraining data to ensure fair comparison to other foundation models.(Reviewer FpFr)
5. Section 4, Models add 3 more foundation models to the list of baselines.(Reviewer FpFr)
6. Appendix A: added clarification for context length (input length) setup.(Reviewer FpFr and Pz22)
7. Appendix B: Added Table 9 to show specific time series features of each dataset.(All Reviewers)
8. Table 14 updated to indicate past dynamic information of each test dataset in our benchmark.(Reviewer SxEF)
9. Appendix F.1, Added Table 16, further analysis of model families through the lens of time series features.(All Reviewers)
10. Tables 17 through 21 updated to report sMAPE, MASE, ND, MSE, MAE, CRPS and RANK metrics and three additional foundation models.(All Reviewers)
11. Added new appendix section F.5 and Figure 4 analyzing inter-variate correlation for multivariate datasets.(Reviewer SxEF)
12. Tables (previously 2-5 now 10-13) presenting statistics over each characteristic for test data is moved to Appendix D. (All Reviewers)
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper introduces GIFT-Eval, a benchmark designed to evaluate time series forecasting models, including statistical, deep learning, and foundation models. It features 28 datasets spanning diverse domains, frequencies, and prediction lengths, alongside a non-leaking pretraining dataset to ensure data integrity. While the benchmark’s scope and extensive experiments highlight significant effort, key issues persist. The connection between dataset characteristics and forecasting performance remains unclear, and inconsistent evaluation configurations, such as varying input lengths across models, raise fairness concerns. Additionally, the analysis lacks depth, providing limited actionable insights about model performance. The role and impact of the pretraining dataset on results also remain ambiguous. Despite its potential as a valuable tool for the research community, these unresolved issues lead to a borderline rejection recommendation, with encouragement for improvement in future iterations.