GIFT-Eval: A Benchmark for General Time Series Forecasting Model Evaluation

## Abstract

Abstract Time series foundation models excel in zero-shot forecasting, handling diverse tasks without explicit training. However, the advancement of these models has been hindered by the lack of comprehensive benchmarks. To address this gap, we introduce the G eneral T I me Series F orecas T ing Model Eval uation, GIFT-Eval, a pioneering benchmark aimed at promoting evaluation across diverse datasets.  GIFT-Eval encompasses 23 datasets over 144,000 time series and 177 million data points, spanning seven domains, 10 frequencies, multivariate inputs, and prediction lengths ranging from short to long-term forecasts. To facilitate the effective pretraining and evaluation of foundation models, we also provide a non-leaking pretraining dataset containing approximately 230 billion data points. Additionally, we provide a comprehensive analysis of 17 baselines, which includes statistical models, deep learning models, and foundation models. We discuss each model in the context of various benchmark characteristics and offer a qualitative analysis that spans both deep learning and foundation models. We believe the insights from this analysis, along with access to this new standard zero-shot time series forecasting benchmark, will guide future developments in time series foundation models. Code, data, and the leaderboard can be found at https://github.com/SalesforceAIResearch/gift-eval .

## 1 Introduction

The success of foundation model pretraining in language and vision modalities has catalyzed similar progress in time series forecasting. By pretraining on extensive time series datasets, a universal forecasting model can be developed, equipped to address varied downstream forecasting tasks across multiple domains, frequencies, prediction lengths, and number of variates in a zero-shot manner (Woo et al., ; Rasul et al., ; Ansari et al., ).

A critical aspect of foundation model research is creating a high-quality benchmark that includes large, diverse evaluation data, and preferably non-leaking pretraining data to fairly evaluate models and identify their weaknesses.
Research in Natural Language Processing (NLP) has produced key benchmarks such as GLUE, MMLU, etc. (Wang et al., ; Hendrycks et al., ; Srivastava et al., ; Chen et al., ), which are crucial for developing high-quality models.

Unlike NLP, time series foundation models lack a unified, diverse benchmark for fair comparison. For instance, Woo et al. () introduces LOTSA, which remains the largest collection of time series forecasting pre-training data to date. However, the proposed architecture, Moirai, is evaluated on existing benchmarks that are tailored to specific forecasting tasks, such as the LSF (Zhou et al., ) dataset for long-term forecast, and the Monash (Godahewa et al., ) dataset for univariate forecasts. Both datasets lack sufficient diversity in time series characteristics and forecasting tasks, making it challenging to evaluate the zero-shot capabilities of foundation models in handling broad and generalized forecasting scenarios. This limitation remains in the recent empirical evaluations of other foundation models, including those featured in benchmarks such as TimesFM, Chronos, and Lag-Llama (Das et al., ; Ansari et al., ; Rasul et al., ). Furthermore, the inconsistency in pretraining, training, and test splits across various foundation models complicates comparisons and poses a risk of data leakage during in-domain and out-of-domain evaluations. To accelerate the advancement for research on time series model, it is essential to establish a high-quality and diverse benchmark that supports universal forecasting evaluation.

To fill identified gaps, we introduce the General TIme Series ForecasTing Model Evaluation (GIFT-Eval), consisting of distinct pretraining and train/test components. The pretraining component features 88 datasets including 240 billion data points (Appendix E lists more details on pretraining data). The train/test component features 23 datasets encompassing 144,000 time series and 177 million data points across seven domains and 10 frequencies, with prediction lengths ranging from short to long-term, as well as univariate and multivariate forecasting settings.
Prior to our work, Qiu et al. () introduced TFB, a comprehensive dataset for time series forecasting. While it offered diversity in the number of variates and domains, it lacks
the evaluation of foundation models and accompanying pretraining data without leakage.
Our benchmark fills these gaps and it also includes a broader range of frequencies, a more diverse taxonomy, and a wider span of prediction lengths. We compare GIFT-Eval with other similar benchmarks in Table 1. Our contributions are three-fold:

- •
GIFT-Eval: We introduce a general time series forecasting benchmark that evaluates the zero-shot and universal forecasting capabilities of foundation models. We provide pretraining and train-test components that ensure diversity across multiple characteristics and time series features.
- •
Comprehensive Benchmarking: We design diverse forecasting tasks and evaluate 17 baselines that encompass statistical, deep learning, and foundational models on GIFT-Eval.
- •
Detailed Analysis: We provide insights into the strengths of different models on all aspects of GIFT-Eval including domains, frequencies, prediction lengths, and the number of variates. We further provide a qualitative analysis showing failure cases of both deep learning and foundation models. We believe these insights will contribute to the future development of foundation models.

## 2 Related work

**Table 1: Property comparisons of various forecasting benchmarks.**
| Benchmark<br>Property | Data | Forecasting Task | Evaluation |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Freq. Range | Num. of Domain | Pretraining data | Num. of var. | Pred. Len. | Benchmark Methods | Prob. Forecasting |  |
| Monash (Godahewa et al., ) | Secondly $\sim$ Yearly | 7 | No | Uni | Short | Stat./DL | No |
| TFB (Qiu et al., ) | Minutely $\sim$ Yearly | 6 | No | Uni/Multi | Short | Stat./DL | No |
| LTSF (Zeng et al., ) | Minutely $\sim$ Weekly | 5 | No | Multi | Long | Stat./DL | No |
| BasicTS+ (Shao et al., ) | Minutely $\sim$ Daily | 3 | No | Multi | Short/Long | Stat./DL | No |
| GIFT-Eval (our work) | Secondly $\sim$ Yearly | 7 | Yes | Uni/Multi | Short/Long | Stat./DL/FM | Yes |

##### Forecasting Methods

Time series forecasters can be broadly categorized into statistical methods, deep learning methods, and, more recently, foundation models. Statistical models rely solely on historical data statistics to predict future values. Among these, ARIMA (Box & Pierce, ), ETS (Hyndman et al., ), Theta (Garza et al., ), and VAR (Godahewa et al., ) are some of the most widely used ones. With the advent of deep learning technologies, models that apply these techniques to time series forecasting have emerged. Examples include DeepAR (Flunkert et al., ), N-BEATS (Oreshkin et al., ), and DLinear (Zeng et al., ), which utilize pre-transformer architectures. Additionally, transformer-based models such as PatchTST (Nie et al., ), Autoformer (Wu et al., ), and Crossformer (Zhang & Yan, ) have been developed. In the last few years, foundation models have been proposed, inspired by their success in other modalities like language and vision. The multivariate Moirai (Woo et al., ) forecaster, for instance, is based on an encoder-decoder architecture pretrained on a large dataset. Conversely, Chronos (Ansari et al., ) and TimesFM (Das et al., ) are univariate forecasters trained using a decoder-only model. VisionTS (Chen et al., ) is another univariate forecaster, unique in that it reformulates the forecasting task as an image reconstruction problem, leveraging a pretrained model from the image modality. However, the main bottleneck in building and evaluating these foundation models is the lack of a diverse and large benchmark dataset.

##### Forecasting Benchmarks

To address this challenge, several efforts have been made to develop extensive time series benchmarks.  Woo et al. () introduced LOTSA, which holds the title for the largest collection of open time series datasets, encompassing 231 billion data points across nine domains. Despite its vast size, the evaluation datasets reuse existing benchmarks from the time series forecasting community and still lack sufficient variety in terms of time series data characteristics and forecasting tasks, which our benchmark aims to augment. Ansari et al. () developed a dataset specifically structured for pretraining, in-domain evaluation, and zero-shot evaluation splits. However, their work is constrained by a limited range of prediction lengths (from 6 to 56), which excludes long-term forecasts, and it restricts the data to univariate forecasting. In contrast, our benchmark encompasses extensive multivariate scenarios and evaluates diverse data across various domains and frequencies. The corpus by Rasul et al. () presents a diverse array of domains, yet it comprises only univariate datasets totaling 8,000 time series. In contrast, GIFT-Eval dramatically expands this scope with 144,000 time series, enhancing the breadth and depth of the dataset. The benchmark by Qiu et al. () is closely aligned with our work in its aim to curate a diverse and comprehensive set of data. However, it lacks pretraining data, does not evaluate foundation models, and limits the taxonomy to time series features only. Our benchmark not only includes pretraining data (with zero-shot evaluation support) but also provides evaluations for foundation models and offers a taxonomy over both characteristics and time series properties. In summary, our benchmark, GIFT-Eval, builds upon and seeks to address the gaps identified in existing time series forecasting benchmarks. We provide a wider comparison with more benchmarks in Table 1. By providing a more diverse and extensive dataset, we aim to facilitate the development and evaluation of foundation models in time series forecasting.

## 3 GIFT-Eval

In this section, we first provide a background on time series forecasting tasks and define key characteristics and features of time series data. We then outline the design decisions behind the development of GIFT-Eval, concluding with an analysis that highlights the key features of its final distribution.

### 3.1 Background

We start by defining univariate and multivariate forecasting tasks. After that, we outline the fundamental characteristics of time series datasets which also influenced our data collection process, including domain, frequency, number of variates, and prediction length. We also introduce time series features as part of our data analysis.

#### 3.1.1 Time Series Forecasting

Time series forecasting is a task of predicting future values over one (univariate) or more (multivariate) variates given historical (most commonly real-valued) data which is sampled at regular time intervals. Suppose $D=(Y^{i},Z^{i})_{i=1}^{N}$ is a dataset of N time series where $Y^{i}=(y^{i}_{1},y^{i}_{2},\dots,y^{i}_{T_{i}})\in\mathbb{R}^{d_{y_{i}}\times T
_{i}}$ is the target time series with $d_{y_{i}}$ variates and $T_{i}$ time steps and $Z^{i}=(z^{i}_{1},z^{i}_{2},\dots,z^{i}_{T_{i}})\in\mathbb{R}^{d_{z_{i}}\times T
_{i}}$ are the set of covaraites with $d_{z_{i}}$ variates. Then the forecasting task can be modeled as the predictive distribution: $p(Y_{t:t+h}|Y_{t-l:t},Z_{t-l:t+h})$ where $l$ is the context length, and $h$ is the forecast horizon. Univariate forecasting is a special case where the target series is univariate (i.e., $d_{y_{i}}=1$), no covariates are used (i.e., $Z=\emptyset$), and only the historical values of the target time series are utilized for prediction.

#### 3.1.2 Time Series Characteristics and Features

##### Characteristics

Time series datasets possess inherent characteristics that define their structure, and common patterns observed in the data and even choices of modelling techniques. We believe a universal forecasting model should be able to perform irrespective of the domain from which the data is sourced, the granularity at which it was sampled, the length of the forecast horizon and whether it is univariate or multivariate. Thus in our study, we focus on these four characteristics: $(i)$ Domain, i.e., the field or industry from which the time series data originates, such as finance, healthcare or meteorology. The domain often has a direct effect on the nature of patterns.
Another crucial aspect is $(ii)$ the  frequency of observations, indicating the time intervals at which the data points are recorded – such as hourly, daily, monthly or annually. $(iii)$ Prediction length, or forecast horizon, is the number of future time steps for which predictions are expected.
Lastly, $(iv)$ the  number of variates pertains to the dimensionality of the time series data. A univariate time series consists of observations of a single variable over time, whereas a multivariate time series involves multiple interrelated variables. The number of variates adds complexity to the modelling process, as models need to account for dependencies among multiple time series. By ensuring diversity across these specific characteristics in our benchmark, we aim to encompass a wide array of real-life scenarios.

##### Features

Time series features (^1^11We use the python implementation of tsfeatures library (Garza et al., ) to calculate each feature.) are statistical properties that capture essential characteristics of the data. We have selected six such properties to analyze our benchmark, grouped into three categories based on the aspects they assess, c.f. Appendix B for a detailed explanation and formula of each feature. First, we chose two metrics for assessing the temporal attributes of each time series:$(i)$ Trend refers to the progression of the time series, indicating whether the data shows an overall increase, decrease or stability over time, where higher values indicate stronger trends.$(ii)$ Seasonal strength measures the extent to which regular, repeating patterns occur at specific intervals, such as daily cycles in energy consumption, or annual peaks in finance. The higher the value the more repeating patterns the data exhibits. Second, to assess the forecastability of the time series, we included two metrics:$(iii)$ Entropy measures the “forecastability” of a time series, where low values indicate a high signal-to-noise ratio and high values occur when a series is difficult to forecast. $(iv)$ Hurst exponent quantifies the long-term memory or persistence in a time series. It indicates whether future values are likely to be influenced by past trends, revert to the mean, or behave randomly, where higher values indicate more persistence. Lastly, to understand the regularity and variability within the time series, we selected two metrics: $(v)$ Stability assesses the inconsistency of the mean of the time series. In simpler terms, it can be defined as the variance of the means. Note that, unlike what the name suggests, lower values indicate more stable data. Finally, $(vi)$ Lumpiness quantifies the variability of the variance across different segments of the time series. A high value of lumpiness indicates significant fluctuations in variability, which can be challenging to model due to the inconsistent behavior of the data.

### 3.2 Datasets

To evaluate and advance universal time series forecasting methods, we have curated a comprehensive collection of datasets. Our compilation spans a wide array of domains with varying frequencies, numbers of variates, and prediction lengths. This diversity is crucial for assessing the generalization capabilities of forecasting models across different types of time series data. In the following sections, we provide detailed descriptions of GIFT-Eval and its unique splits, outlining their sources, and key properties. We also conduct a detailed analysis on the test data to gain a better understanding of the datasets’ characteristics and the distribution of time series features.

##### Train/Test Data

We curated the train/test portion of GIFT-Eval with 15 univariate and eight multivariate datasets, spanning sevem domains and 10 frequencies, totaling 144,000 time series and 177 million data points. We adhere to established prediction lengths for well-known datasets like M4 (Makridakis et al., ). For other datasets, we establish three prediction settings—short, medium, and long—based on frequency and domain, with medium and long settings extending the short-term length by factors of 10 and 15, respectively. To support models without multivariate forecasting, our framework flattens multivariate datasets for broader compatibility. Data is stored in the Arrow format (Richardson et al., ), ensuring efficient integration into deep learning pipelines. Our benchmark features 97 unique triplets of dataset, frequency, and length, with aggregated results for each model reported across these configurations. The sources of each dataset used in train/test split can be found in Appendix D

We structure the evaluation component of our benchmark by dedicating the final 10% of each dataset in train/test portion to testing, with the rest allocated for training. A non-overlapping rolling evaluation method is employed, setting a predetermined number of windows in the test split, each equal to the dataset’s prediction length. The final window of the training data serves as validation for tuning deep learning model hyperparameters.

##### Analysis and statistics over test data

Tables 3, 4, 2 and 5 present detailed statistics on the number of time series and total observations within each characteristic category of the test benchmark. Specifically, these tables break down the data by domain (Table 3), frequency (Table 4), prediction length (Table 2), and variate count (Table 5), offering a quantitative overview of the dataset’s composition. For the reader seeking more granular information, detailed dataset statistics are provided in Table 13.

**Table 2: GIFT-Eval Test data statistics aggregated by prediction length.**
| Pred. Length | 6 | 8 | 12 | 13 | 14 | 18 | 30 | 48 | 60 | 480 | 600 | 720 | 900 |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| # Series | 22,974 | 24,629 | 3,443 | 359 | 4,227 | 48,000 | 34,398 | 6,194 | 22 | 3,874 | 22 | 3,874 | 22 |
| # Obs | 845,109 | 2,525,512 | 201,042 | 371,579 | 10,023,836 | 11,246,411 | 1,447,848 | 131,125,706 | 194,369 | 129,375,020 | 194,369 | 129,375,020 | 194,369 |

**Table 3: GIFT-Eval Test data statistics aggregated by domain.**
| Domain | Econ/Fin | Energy | Healthcare | Nature | Sales | Transport | Web/CloudOps | Grand Total |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| # Series | 99,974 | 2,036 | 1,036 | 32,618 | 3,717 | 1,341 | 3,524 | 144,246 |
| # Obs | 25,266,415 | 74,119,755 | 129,408 | 3,154,921 | 671,707 | 38,028,955 | 16,610,251 | 157,981,412 |

**Table 4: GIFT-Eval Test data statistics aggregated by frequency.**
| Frequency | 10S | 10T | 15T | 5T | A | D | H | M | Q | W | Grand Total |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| # Series | 22 | 138 | 528 | 2,074 | 22,974 | 38,625 | 3,454 | 51,443 | 24,000 | 988 | 144,246 |
| # Obs | 194,369 | 7,253,424 | 52,498,336 | 49,105,728 | 845,109 | 11,471,684 | 22,268,218 | 11,447,453 | 2,406,108 | 490,983 | 157,981,412 |

**Table 5: GIFT-Eval Test data statistics aggregated by number of variates.**
| # Variates | 1 | 2 | 7 | 21 | Grand Total |
| --- | --- | --- | --- | --- | --- |
| # Series | 140,711 | 3,522 | 10 | 3 | 144,246 |
| # Obs | 141,133,451 | 16,575,619 | 210,488 | 61,854 | 157,981,412 |

We further analyze GIFT-Eval to understand the distribution and characteristics of the time series features across various datasets. Figure 1 illustrates the mean values of each time series features across different dataset characteristics. These heatmaps provide valuable insights into how metrics such as trend, seasonal strength, Hurst exponent, stability, and lumpiness vary across datasets with different domains, frequencies, prediction lengths, and numbers of variates. This visualization aids in identifying patterns and potential biases within the data, ensuring that the benchmark captures a diverse range of time series behaviors. It also facilitates fine-grained analysis of model performance across varying dataset characteristics, offering a comprehensive comparison.

Number of variates:  Figure 1(a) depicts that multivariate data exhibit higher stability and lumpiness values, suggesting more fluctuation in variance across different segments, indicating multivariate time series are more complex and potentially more challenging to model. Conversely, univariate series show stronger seasonal strength, reflecting more pronounced and regular repeating patterns, making them more predictable over certain periods. Note that the metrics on multivariate time series are calculated individually for each variate and aggregated for each dataset.

Figure: (a) Mean values of TS features across univariate and multivariate datasets.
Refer to caption: x1.png

Prediction Length:  Figure 1(b) shows that shorter prediction lengths have higher values for both trend and Hurst metrics. This suggests that time series with shorter forecast horizons exhibit stronger directional movements and greater persistence in their trends, making them potentially easier to predict. As the prediction length increases, the trend and hurst values tend to decrease significantly which makes forcasting harder. Notably, the stability values decrease from short to long indicating higher steadiness in long term while lumpiness increases suggesting higher fluctuations in different sections of the data.

Domain: Figure 1(c) reveals distinct patterns in the metrics. The Web/CloudOps, Sales, and Nature domains exhibit notably high lumpiness, indicating significant fluctuations in variance. This may reflect the volatile nature of online operations, sales dynamics and weather predictions. On the other hand, Transport shows the highest entropy and lowest trend values, indicating less predictability, likely due to the variable nature of transportation data influenced by numerous external factors. The Econ/Fin domain shows the highest trend values, indicating strong directional movements that may imply clear market trends or economic cycles. Finally, healthcare exhibits the highest Hurst and lowest lumpiness values, suggesting persistence in the data, possibly due to consistent patient trends or medical outcomes over time.

Frequency:  Figure 1(d) lists frequencties from highest to lowest. Data with very short intervals, such as secondly (S) and minutely (T) exhibit the lowest seasonal strengths and poor steadiness, indicative of the erratic and volatile nature typical at these granular levels. There is a noticeable increase in seasonal strength progressing from secondly and minutely data to hourly (H) and daily (D). Finally, yearly (A) and quarterly (Q) data demonstrate the strongest trends and Hurst values, with notably low lumpiness, suggesting increased persistence and high predictability. Notably, the yearly data lack seasonal strength measurements due to the tsfeatures library not providing seasonal strength for excessively long time series, a limitation commonly observed in low-frequency datasets.

Summary: These observations confirm that our benchmark is rich and diverse, representing a broad range of real-life time series scenarios. Our dataset encompasses various characteristics—such as differing levels of trend, seasonality, persistence, volatility, and complexity—across multiple domains and frequencies. For instance, we include data from domains with high volatility and significant fluctuations, as well as data exhibiting strong persistence and stability. We also cover a wide spectrum of frequencies, from high-frequency data with erratic patterns to low-frequency data with strong trends and greater predictability. In a similar manner, our metrics show diversity across variate types and prediction lengths. This diversity ensures that models are tested across various temporal behaviors, making our benchmark a robust platform for evaluating the general capabilities of unified models, particularly foundation models in time series forecasting.

##### Pretraining Dataset

We have also curated a pretraining dataset aligned with GIFT-Eval that has 71 univariate and 17 multivariate datasets, spanning seven domains and 13 frequencies, totaling 4.5 million time series and 230 billion data points. Notably this collection of data has no leakage issue with the train/test split and can be used to pretrain foundation models that can be fairly evaluated on GIFT-Eval. Further details on pretraining dataset can be found in Appendix E.

## 4 Experiments

In this section, we present the experimental evaluation of GIFT-Eval across various models.

##### Models

Time series forecasting training and inference may take different forms for different families of models. Statistical models make predictions by directly analyzing patterns in the historical data without a separate training phase. We incorporate five statistical models in our benchmark: Naive, Seasonal Naive (Hyndman & Athanasopoulos, ), Auto_Arima, Auto_ETS, and Auto_Theta (Garza et al., ) methods. Deep learning models require training a specific model instance for each dataset. Representing deep learning, we select 8 models: DeepAR (Flunkert et al., ), TFT (Lim et al., ), TiDE (Das et al., ), N-BEATS (Oreshkin et al., ), PatchTST (Nie et al., ), DLinear (Zeng et al., ), Crossformer (Zhang & Yan, ) and iTransformer (Liu et al., ).
To obtain both point and probabilistic forecasts, we either adapt models using gluonts (Alexandrov et al., ) with a small probabilistic head or implement our own modifications. We conduct an extensive hyperparameter search for each deep learning model, see Appendix A for details. We evaluate four foundation models on our benchmark: TimesFM (Das et al., ), Chronos (Ansari et al., ) available in tiny, small, and base sizes,  Moirai (Woo et al., ) available in small, base, and large sizes and VisionTS (Chen et al., ). These models all provide publicly accessible model parameters for direct use. However, it is important to note that pre-training datasets of  TimesFM, Chronos, and  Moirai exhibit partial data leakage issues for GIFT-Eval. Since  Moirai provides pretraining code, here we pretrain a series of  Moirai models using GIFT-Eval’s pretraining split to demonstrate its utility. We empirically investigate the impact of data leakage in Appendix F.2.
Further details on model-specific hyperparameters and tuning can be found in Appendix A.

For readability concerns, we omit results from Auto_ETS, DLinear and Crossformer models in the main tables, however, the reader may refer to Appendix F for results with all models available. For the same space concerns, we use abbreviations to replace each model in the tables. Here is a list of model$\rightarrow$abbreviation pairs for reference: Naive: Nv., Seasonal Naive: S.Nv., Auto_Arima: A.Ar., Auto_Theta: A.Th.,Auto_ETS: A.ETS, DeepAR: DeepAR, TFT: TFT, TiDE: TiDE, N-BEATS: N-BEATS, PatchTST: P.TST, iTransformer: iTrans.,DLinear: DLin., Crossformer: C.former, TimesFM: T.FM, VisionTS: Vis.TS, Chronos: Chr., ChronosSmall: Chr.S, ChronosBase: Chr.B, ChronosLarge: Chr.L, Moirai: Moi., MoiraiSmall: Moi.S, MoiraiBase: Moi.B, MoiraiLarge: Moi.L.

##### Evaluation setting

Performance is assessed using two metrics: the median Mean Absolute Percentage Error (MAPE) for point forecasts and the Continuous Ranked Probability Score (CRPS) (Gneiting & Raftery, ) for probabilistic forecasts (definition of both metrics are in Appendix C). To standardize comparison across benchmarks, both metrics are normalized against the Seasonal Naive baseline. To avoid skew from any single dataset, we employ a ‘Rank’ metric that assigns a numerical ranking to each model across all 97 configurations judging by their CRPS score. The average of these ranks is then reported as the final Rank for each model.

### 4.1 Results

We present results across five distinct parts. The first four parts aggregate the results by the key characteristics that guided the development of our benchmark: domain, prediction length, frequency, and number of variates, then conclude the section with aggregation of results across all configurations. For results on all datasets, frequency and prediction length combinations see Tables 20, 21 and 22.

##### Domain | Table 6

**Table 6: Results on GIFT-Eval aggregated by domain. The best results across each row are bolded, while the second best results are underlined.**
| Domain | Metric | Nv. | S.Nv. | A.Ar. | A.Th. | DeepAR | TFT | TiDE | N-BEATS | P.TST | iTrans. | T.FM | Vis.TS | Chr.S | Chr.B | Chr.L | Moi.S | Moi.B | Moi.L | Best |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Econ/Fin | MAPE | 1.22 | 1.00 | $9.37e^{-1}$ | 1.01 | 1.23 | 1.14 | 1.15 | $8.63e^{-1}$ | $9.46e^{-1}$ | 1.02 | $8.41e^{-1}$ | $9.93e^{-1}$ | $8.02e^{-1}$ | $8.02e^{-1}$ | $7.99e^{-1}$ | 1.04 | 1.27 | $8.80e^{-1}$ | Chr.L |
|  | CRPS | 1.36 | 1.00 | $8.25e^{-1}$ | $8.46e^{-1}$ | 1.41 | $8.48e^{-1}$ | 1.09 | $9.86e^{-1}$ | $8.15e^{-1}$ | $8.55e^{-1}$ | $7.29e^{-1}$ | 1.06 | $7.78e^{-1}$ | $7.68e^{-1}$ | $7.74e^{-1}$ | $8.53e^{-1}$ | 1.04 | $7.36e^{-1}$ | T.FM |
|  | Rank | $15.17$ | $14.83$ | $7.17$ | $7.83$ | $14.33$ | $7.83$ | $17.00$ | $12.67$ | $6.67$ | $8.33$ | $4.83$ | $16.50$ | $7.50$ | $6.33$ | $7.00$ | $10.83$ | $15.50$ | $3.17$ | Moi.L |
| Energy | MAPE | 1.23 | 1.00 | 1.02 | 1.35 | 1.68 | 1.09 | 1.30 | 1.17 | 1.00 | 1.24 | $9.90e^{-1}$ | 1.15 | $9.11e^{-1}$ | $9.09e^{-1}$ | $9.12e^{-1}$ | $8.99e^{-1}$ | $9.87e^{-1}$ | $8.91e^{-1}$ | Moi.L |
|  | CRPS | 1.72 | 1.00 | $8.63e^{-1}$ | 2.29 | 1.37 | $6.70e^{-1}$ | $8.82e^{-1}$ | $9.80e^{-1}$ | $6.52e^{-1}$ | $8.26e^{-1}$ | $7.04e^{-1}$ | $8.25e^{-1}$ | $6.85e^{-1}$ | $6.61e^{-1}$ | $6.59e^{-1}$ | $6.97e^{-1}$ | $6.35e^{-1}$ | $6.26e^{-1}$ | Moi.L |
|  | Rank | $18.22$ | $15.84$ | $12.28$ | $16.91$ | $14.69$ | $7.28$ | $10.47$ | $15.09$ | $5.75$ | $7.06$ | $8.25$ | $13.09$ | $8.59$ | $7.38$ | $7.25$ | $7.00$ | $5.69$ | $5.38$ | Moi.L |
| Healthcare | MAPE | 1.21 | 1.00 | $7.83e^{-1}$ | $9.67e^{-1}$ | $8.57e^{-1}$ | $7.90e^{-1}$ | $8.43e^{-1}$ | $7.70e^{-1}$ | $7.95e^{-1}$ | $9.02e^{-1}$ | $7.91e^{-1}$ | $8.03e^{-1}$ | $6.68e^{-1}$ | $7.25e^{-1}$ | $6.62e^{-1}$ | 1.06 | 1.15 | $8.17e^{-1}$ | Chr.L |
|  | CRPS | 1.29 | 1.00 | $6.31e^{-1}$ | $8.28e^{-1}$ | $8.64e^{-1}$ | $7.01e^{-1}$ | 1.16 | $8.69e^{-1}$ | $6.79e^{-1}$ | $7.65e^{-1}$ | $7.99e^{-1}$ | $7.96e^{-1}$ | $5.75e^{-1}$ | $5.94e^{-1}$ | $5.31e^{-1}$ | $9.26e^{-1}$ | 1.07 | $6.86e^{-1}$ | Chr.L |
|  | Rank | $17.80$ | $15.00$ | $7.80$ | $12.00$ | $10.40$ | $8.20$ | $14.40$ | $14.00$ | $8.60$ | $9.40$ | $7.80$ | $12.80$ | $5.80$ | $5.00$ | $4.20$ | $14.80$ | $15.80$ | $8.00$ | Chr.L |
| Nature | MAPE | 1.02 | 1.00 | $9.36e^{-1}$ | 5.15 | 1.34 | 1.35 | 1.65 | 2.05 | $9.76e^{-1}$ | 1.02 | 1.08 | 1.04 | $9.49e^{-1}$ | $8.09e^{-1}$ | $7.22e^{-1}$ | $8.64e^{-1}$ | 1.08 | $9.01e^{-1}$ | Chr.L |
|  | CRPS | 1.52 | 1.00 | $7.10e^{-1}$ | 1.03 | $7.84e^{-1}$ | $4.02e^{-1}$ | $6.49e^{-1}$ | $5.87e^{-1}$ | $4.04e^{-1}$ | $3.92e^{-1}$ | $3.82e^{-1}$ | $4.87e^{-1}$ | $4.57e^{-1}$ | $4.35e^{-1}$ | $4.33e^{-1}$ | $4.00e^{-1}$ | $4.22e^{-1}$ | $3.78e^{-1}$ | Moi.L |
|  | Rank | $19.27$ | $18.60$ | $14.40$ | $16.40$ | $11.87$ | $7.47$ | $13.47$ | $14.53$ | $7.27$ | $6.53$ | $5.73$ | $11.67$ | $9.67$ | $8.73$ | $8.20$ | $4.40$ | $4.47$ | $4.13$ | Moi.L |
| Sales | MAPE | $9.99e^{-1}$ | 1.00 | $7.72e^{-1}$ | $8.26e^{-1}$ | $7.39e^{-1}$ | $7.57e^{-1}$ | 1.00 | $7.26e^{-1}$ | $7.51e^{-1}$ | $7.59e^{-1}$ | $6.83e^{-1}$ | $8.11e^{-1}$ | $7.19e^{-1}$ | $7.01e^{-1}$ | $7.03e^{-1}$ | $6.72e^{-1}$ | $6.68e^{-1}$ | $6.71e^{-1}$ | Moi.B |
|  | CRPS | $9.35e^{-1}$ | 1.00 | $4.83e^{-1}$ | $5.03e^{-1}$ | $3.68e^{-1}$ | $3.64e^{-1}$ | $5.03e^{-1}$ | $4.26e^{-1}$ | $3.67e^{-1}$ | $3.71e^{-1}$ | $3.65e^{-1}$ | $5.23e^{-1}$ | $3.85e^{-1}$ | $3.85e^{-1}$ | $3.84e^{-1}$ | $4.62e^{-1}$ | $5.16e^{-1}$ | $4.06e^{-1}$ | TFT |
|  | Rank | $19.00$ | $19.25$ | $14.25$ | $14.50$ | $6.25$ | $8.00$ | $14.00$ | $11.00$ | $3.25$ | $4.75$ | $2.75$ | $15.50$ | $9.00$ | $7.25$ | $7.25$ | $8.50$ | $9.00$ | $5.25$ | T.FM |
| Transport | MAPE | 1.48 | 1.00 | 1.07 | 1.08 | $8.11e^{-1}$ | $8.22e^{-1}$ | $8.78e^{-1}$ | $7.60e^{-1}$ | $8.02e^{-1}$ | $8.27e^{-1}$ | $9.28e^{-1}$ | $8.70e^{-1}$ | $8.46e^{-1}$ | $8.53e^{-1}$ | $8.47e^{-1}$ | $9.10e^{-1}$ | $8.56e^{-1}$ | $9.28e^{-1}$ | N-BEATS |
|  | CRPS | 2.29 | 1.00 | $7.90e^{-1}$ | 1.48 | $5.54e^{-1}$ | $4.85e^{-1}$ | $5.70e^{-1}$ | $6.41e^{-1}$ | $5.06e^{-1}$ | $5.00e^{-1}$ | $5.78e^{-1}$ | $6.66e^{-1}$ | $6.02e^{-1}$ | $5.86e^{-1}$ | $5.85e^{-1}$ | $5.00e^{-1}$ | $4.78e^{-1}$ | $5.02e^{-1}$ | Moi.B |
|  | Rank | $20.13$ | $17.33$ | $15.87$ | $18.73$ | $6.47$ | $4.80$ | $10.40$ | $12.60$ | $5.80$ | $5.87$ | $8.13$ | $13.67$ | $10.07$ | $8.07$ | $8.40$ | $8.33$ | $6.13$ | $6.67$ | TFT |
| Web/CloudOps | MAPE | 1.11 | 1.00 | $8.94e^{-1}$ | $8.33e^{-1}$ | $8.57e^{-1}$ | 1.35 | $9.58e^{-1}$ | $6.42e^{-1}$ | $6.02e^{-1}$ | $7.19e^{-1}$ | 2.37 | $8.38e^{-1}$ | 1.15 | 1.30 | 1.33 | $7.97e^{-1}$ | 1.06 | $7.91e^{-1}$ | P.TST |
|  | CRPS | 1.19 | 1.00 | $9.24e^{-1}$ | $7.38e^{-1}$ | $7.81e^{-1}$ | $6.49e^{-1}$ | $6.73e^{-1}$ | $6.62e^{-1}$ | $5.18e^{-1}$ | $5.22e^{-1}$ | $9.76e^{-1}$ | $7.24e^{-1}$ | $7.52e^{-1}$ | $8.11e^{-1}$ | $7.91e^{-1}$ | $7.44e^{-1}$ | $7.68e^{-1}$ | $7.42e^{-1}$ | P.TST |
|  | Rank | $16.70$ | $16.35$ | $14.50$ | $12.10$ | $11.70$ | $5.95$ | $9.95$ | $10.30$ | $3.95$ | $4.50$ | $13.90$ | $10.65$ | $10.40$ | $11.35$ | $11.65$ | $8.50$ | $9.25$ | $8.35$ | P.TST |

The results across various domains demonstrate that foundation models consistently outperform both statistical and deep learning models. Notably, the Moirai and Chronos variants achieve top performance in most areas, except in the Web/CloudOps and Transport domains. As discussed in Section 3.2 the Transport domain features datasets with the highest entropy and lowest trend components, which are less predictable and Web/CloudOps is one of the domains to exhibit the highest lumpiness. This pattern suggests that foundation models may struggle with time series possessing such characteristics. In contrast, deep learning models like PatchTST and TFT excel in these challenging domains, possibly indicating a shortfall of the training data used for foundation models in these areas. The comparison of different foundation models yields inconsistent conclusions across various domains. We believe this inconsistency is related to the pre-training datasets utilized by the different foundation models and the varying contribution ratios associated with them.

##### Prediction length | Table 7

**Table 7: Results on GIFT-Eval aggregated by Prediction Length. The best results across each row are bolded, while the second best results are underlined.**
| Pred. Len. | Metric | Nv. | S.Nv. | A.Ar. | A.Th. | DeepAR | TFT | TiDE | N-BEATS | P.TST | iTrans. | T.FM | Vis.TS | Chr.S | Chr.B | Chr.L | Moi.S | Moi.B | Moi.L | Best |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Long | MAPE | 1.35 | 1.00 | 1.04 | 3.15 | 1.70 | 1.16 | 1.28 | 1.11 | $9.62e^{-1}$ | $9.96e^{-1}$ | 1.80 | 1.03 | 1.01 | $9.98e^{-1}$ | 1.00 | $9.19e^{-1}$ | 1.09 | 1.01 | Moi.S |
|  | CRPS | 2.11 | 1.00 | $8.43e^{-1}$ | 2.06 | $9.48e^{-1}$ | $4.53e^{-1}$ | $5.25e^{-1}$ | $6.64e^{-1}$ | $4.37e^{-1}$ | $4.55e^{-1}$ | $6.54e^{-1}$ | $5.46e^{-1}$ | $6.46e^{-1}$ | $6.20e^{-1}$ | $6.21e^{-1}$ | $5.61e^{-1}$ | $5.01e^{-1}$ | $5.39e^{-1}$ | P.TST |
|  | Rank | $19.62$ | $16.86$ | $15.10$ | $17.38$ | $12.71$ | $4.90$ | $8.76$ | $11.95$ | $4.43$ | $5.19$ | $11.29$ | $9.57$ | $11.86$ | $10.71$ | $11.05$ | $7.24$ | $5.43$ | $6.76$ | P.TST |
| Medium | MAPE | 1.30 | 1.00 | $9.93e^{-1}$ | 1.87 | 1.07 | 1.26 | 1.19 | 1.15 | $9.03e^{-1}$ | 1.02 | 1.52 | 1.02 | 1.23 | 1.38 | 1.33 | $9.31e^{-1}$ | 1.11 | $9.60e^{-1}$ | P.TST |
|  | CRPS | 2.08 | 1.00 | $8.63e^{-1}$ | 1.96 | $8.27e^{-1}$ | $5.29e^{-1}$ | $6.11e^{-1}$ | $7.60e^{-1}$ | $5.19e^{-1}$ | $5.29e^{-1}$ | $8.14e^{-1}$ | $6.69e^{-1}$ | $7.38e^{-1}$ | $7.77e^{-1}$ | $7.51e^{-1}$ | $6.61e^{-1}$ | $6.37e^{-1}$ | $6.25e^{-1}$ | P.TST |
|  | Rank | $19.29$ | $16.14$ | $14.57$ | $17.43$ | $10.52$ | $4.81$ | $9.52$ | $12.95$ | $4.00$ | $4.62$ | $10.81$ | $10.95$ | $11.48$ | $11.57$ | $10.90$ | $7.52$ | $6.38$ | $6.76$ | P.TST |
| Short | MAPE | 1.10 | 1.00 | $9.18e^{-1}$ | 1.13 | 1.08 | 1.05 | 1.12 | 1.03 | $8.04e^{-1}$ | $9.68e^{-1}$ | $9.29e^{-1}$ | $9.47e^{-1}$ | $7.81e^{-1}$ | $7.50e^{-1}$ | $7.52e^{-1}$ | $8.49e^{-1}$ | $9.41e^{-1}$ | $7.73e^{-1}$ | Chr.B |
|  | CRPS | 1.20 | 1.00 | $7.78e^{-1}$ | $9.56e^{-1}$ | 1.03 | $6.76e^{-1}$ | $9.17e^{-1}$ | $8.20e^{-1}$ | $6.28e^{-1}$ | $7.28e^{-1}$ | $6.45e^{-1}$ | $8.20e^{-1}$ | $5.97e^{-1}$ | $5.90e^{-1}$ | $5.86e^{-1}$ | $6.66e^{-1}$ | $7.08e^{-1}$ | $6.02e^{-1}$ | Chr.L |
|  | Rank | $17.22$ | $16.75$ | $11.87$ | $13.53$ | $11.89$ | $8.22$ | $13.42$ | $13.87$ | $6.87$ | $7.49$ | $6.64$ | $14.65$ | $7.25$ | $6.16$ | $6.27$ | $8.15$ | $8.84$ | $5.40$ | Moi.L |

**Table 8: Results on GIFT-Eval aggregated by frequency. The best results across each row are bolded, while second best results are underlined.**
| Freq. | Metric | Nv. | S.Nv. | A.Ar. | A.Th. | DeepAR | TFT | TiDE | N-BEATS | P.TST | iTrans. | T.FM | Vis.TS | Chr.S | Chr.B | Chr.L | Moi.S | Moi.B | Moi.L | Best |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| 10S | CRPS | 1.54 | 1.00 | 1.00 | $4.50e^{-1}$ | 1.03 | 1.05 | $9.85e^{-1}$ | $8.21e^{-1}$ | $7.32e^{-1}$ | $6.91e^{-1}$ | 1.82 | $9.34e^{-1}$ | 1.10 | 1.27 | 1.19 | 1.31 | 1.40 | 1.24 | A.Th. |
|  | MAPE | 1.97 | 1.00 | 1.00 | $6.49e^{-1}$ | 1.83 | 3.24 | 1.67 | $9.97e^{-1}$ | 1.06 | 1.31 | 6.23 | 1.10 | 2.42 | 2.95 | 3.02 | 1.65 | 2.34 | 1.55 | A.Th. |
|  | Rank | $16.33$ | $9.50$ | $8.50$ | $1.00$ | $11.17$ | $7.67$ | $10.17$ | $6.67$ | $5.00$ | $2.50$ | $19.33$ | $10.17$ | $10.83$ | $12.33$ | $11.67$ | $14.83$ | $16.50$ | $14.17$ | A.Th. |
| 5T | CRPS | 1.40 | 1.00 | 1.00 | 1.05 | $7.97e^{-1}$ | $5.57e^{-1}$ | $6.48e^{-1}$ | $7.26e^{-1}$ | $5.43e^{-1}$ | $5.43e^{-1}$ | $7.30e^{-1}$ | $7.34e^{-1}$ | $7.52e^{-1}$ | $7.49e^{-1}$ | $7.55e^{-1}$ | $5.36e^{-1}$ | $5.41e^{-1}$ | $5.32e^{-1}$ | Moi.L |
|  | MAPE | $8.44e^{-1}$ | 1.00 | 1.00 | $9.64e^{-1}$ | $6.72e^{-1}$ | $6.43e^{-1}$ | $7.95e^{-1}$ | $6.34e^{-1}$ | $5.87e^{-1}$ | $6.60e^{-1}$ | $8.20e^{-1}$ | $8.76e^{-1}$ | $7.80e^{-1}$ | $7.63e^{-1}$ | $7.72e^{-1}$ | $5.30e^{-1}$ | $6.00e^{-1}$ | $5.52e^{-1}$ | Moi.S |
|  | Rank | $17.17$ | $17.75$ | $16.25$ | $16.67$ | $13.25$ | $4.92$ | $10.08$ | $12.42$ | $4.75$ | $5.50$ | $11.92$ | $12.17$ | $11.17$ | $11.25$ | $11.92$ | $4.42$ | $4.75$ | $3.25$ | Moi.L |
| 10T | CRPS | 2.23 | 1.00 | 1.00 | 3.67 | $5.92e^{-1}$ | $4.12e^{-1}$ | $6.55e^{-1}$ | $7.81e^{-1}$ | $5.58e^{-1}$ | $5.91e^{-1}$ | $5.70e^{-1}$ | $5.01e^{-1}$ | $6.43e^{-1}$ | $5.55e^{-1}$ | $5.51e^{-1}$ | $5.92e^{-1}$ | $4.33e^{-1}$ | $4.95e^{-1}$ | TFT |
|  | MAPE | $7.59e^{-1}$ | 1.00 | 1.00 | 2.63 | $6.45e^{-1}$ | 1.20 | 1.16 | 1.26 | $8.81e^{-1}$ | $7.80e^{-1}$ | 1.02 | 1.08 | 1.51 | 1.27 | 1.03 | $5.83e^{-1}$ | $9.12e^{-1}$ | 1.05 | Moi.S |
|  | Rank | $19.33$ | $16.50$ | $15.50$ | $19.67$ | $10.67$ | $4.33$ | $11.83$ | $13.17$ | $7.33$ | $6.83$ | $8.33$ | $7.50$ | $12.17$ | $8.83$ | $8.00$ | $8.50$ | $5.00$ | $7.50$ | TFT |
| 15T | CRPS | 2.42 | 1.00 | $9.58e^{-1}$ | 1.76 | 1.68 | $7.32e^{-1}$ | $8.07e^{-1}$ | $9.89e^{-1}$ | $6.63e^{-1}$ | $6.58e^{-1}$ | $7.93e^{-1}$ | $8.78e^{-1}$ | $7.93e^{-1}$ | $7.67e^{-1}$ | $7.64e^{-1}$ | $7.93e^{-1}$ | $7.07e^{-1}$ | $6.76e^{-1}$ | iTrans. |
|  | MAPE | 1.44 | 1.00 | 1.10 | $9.89e^{-1}$ | 1.96 | 1.06 | $9.79e^{-1}$ | $9.64e^{-1}$ | $8.74e^{-1}$ | $9.00e^{-1}$ | 1.02 | $9.82e^{-1}$ | $9.61e^{-1}$ | $9.52e^{-1}$ | $9.44e^{-1}$ | 1.10 | 1.01 | $9.40e^{-1}$ | P.TST |
|  | Rank | $19.75$ | $15.17$ | $13.92$ | $17.00$ | $14.50$ | $6.58$ | $10.08$ | $15.00$ | $3.83$ | $3.50$ | $8.25$ | $12.83$ | $9.75$ | $8.50$ | $8.17$ | $8.67$ | $5.58$ | $4.42$ | iTrans. |
| H | CRPS | 1.84 | 1.00 | $7.77e^{-1}$ | 1.85 | $8.93e^{-1}$ | $4.87e^{-1}$ | $5.70e^{-1}$ | $6.65e^{-1}$ | $4.62e^{-1}$ | $4.76e^{-1}$ | $5.10e^{-1}$ | $6.11e^{-1}$ | $5.14e^{-1}$ | $5.06e^{-1}$ | $5.09e^{-1}$ | $4.78e^{-1}$ | $4.63e^{-1}$ | $4.99e^{-1}$ | P.TST |
|  | MAPE | 1.33 | 1.00 | $9.54e^{-1}$ | 3.08 | 1.18 | 1.11 | 1.25 | 1.34 | $9.28e^{-1}$ | $9.60e^{-1}$ | $9.82e^{-1}$ | $9.89e^{-1}$ | $7.33e^{-1}$ | $7.39e^{-1}$ | $7.48e^{-1}$ | $8.13e^{-1}$ | $9.07e^{-1}$ | $8.51e^{-1}$ | Chr.S |
|  | Rank | $19.65$ | $17.65$ | $15.48$ | $18.87$ | $10.87$ | $6.65$ | $10.58$ | $13.42$ | $4.97$ | $6.13$ | $8.35$ | $11.94$ | $8.58$ | $7.81$ | $8.00$ | $6.32$ | $5.29$ | $6.55$ | P.TST |
| D | CRPS | $7.98e^{-1}$ | 1.00 | $4.99e^{-1}$ | $5.75e^{-1}$ | $5.94e^{-1}$ | $4.14e^{-1}$ | $7.48e^{-1}$ | $5.71e^{-1}$ | $4.37e^{-1}$ | $4.95e^{-1}$ | $4.92e^{-1}$ | $5.56e^{-1}$ | $4.37e^{-1}$ | $4.21e^{-1}$ | $4.21e^{-1}$ | $4.13e^{-1}$ | $4.63e^{-1}$ | $3.92e^{-1}$ | Moi.L |
|  | MAPE | 1.00 | 1.00 | $8.53e^{-1}$ | $9.29e^{-1}$ | $8.01e^{-1}$ | $7.62e^{-1}$ | $9.97e^{-1}$ | $7.96e^{-1}$ | $7.52e^{-1}$ | $8.18e^{-1}$ | $7.80e^{-1}$ | $8.95e^{-1}$ | $7.37e^{-1}$ | $6.86e^{-1}$ | $6.96e^{-1}$ | $7.09e^{-1}$ | $7.45e^{-1}$ | $6.63e^{-1}$ | Moi.L |
|  | Rank | $17.60$ | $19.00$ | $11.00$ | $14.33$ | $11.07$ | $6.67$ | $13.53$ | $14.40$ | $7.20$ | $8.47$ | $5.60$ | $14.80$ | $8.07$ | $6.73$ | $6.67$ | $6.20$ | $7.47$ | $4.33$ | Moi.L |
| W | CRPS | $8.76e^{-1}$ | 1.00 | $7.49e^{-1}$ | $8.09e^{-1}$ | 1.35 | $7.79e^{-1}$ | 1.22 | 1.08 | $6.98e^{-1}$ | 1.33 | $6.31e^{-1}$ | $9.99e^{-1}$ | $5.55e^{-1}$ | $5.62e^{-1}$ | $5.52e^{-1}$ | $7.58e^{-1}$ | $7.48e^{-1}$ | $6.41e^{-1}$ | Chr.L |
|  | MAPE | 1.00 | 1.00 | $9.76e^{-1}$ | 1.08 | 1.86 | 1.04 | 1.77 | 1.45 | $9.07e^{-1}$ | 1.90 | $8.62e^{-1}$ | 1.14 | $7.08e^{-1}$ | $7.29e^{-1}$ | $7.07e^{-1}$ | $9.81e^{-1}$ | $9.50e^{-1}$ | $8.96e^{-1}$ | Chr.L |
|  | Rank | $13.62$ | $16.50$ | $9.88$ | $12.12$ | $12.75$ | $11.25$ | $13.12$ | $15.25$ | $8.00$ | $12.38$ | $4.88$ | $16.00$ | $5.25$ | $4.50$ | $4.38$ | $8.62$ | $9.75$ | $5.12$ | Chr.L |
| M | CRPS | 1.58 | 1.00 | $7.66e^{-1}$ | $8.82e^{-1}$ | 1.10 | $8.75e^{-1}$ | 1.24 | 1.01 | $8.48e^{-1}$ | $8.21e^{-1}$ | $7.38e^{-1}$ | 1.04 | $8.31e^{-1}$ | $8.61e^{-1}$ | $8.15e^{-1}$ | 1.22 | 1.50 | $9.17e^{-1}$ | T.FM |
|  | MAPE | 1.30 | 1.00 | $7.90e^{-1}$ | $9.20e^{-1}$ | 1.18 | 1.05 | 1.04 | $9.41e^{-1}$ | $9.90e^{-1}$ | 1.01 | $8.63e^{-1}$ | $9.38e^{-1}$ | $8.62e^{-1}$ | $8.95e^{-1}$ | $8.47e^{-1}$ | 1.38 | 1.89 | $9.97e^{-1}$ | A.Ar. |
|  | Rank | $19.20$ | $13.40$ | $6.20$ | $8.40$ | $11.80$ | $8.20$ | $15.40$ | $11.40$ | $7.60$ | $5.60$ | $3.60$ | $14.80$ | $8.60$ | $8.60$ | $8.20$ | $17.00$ | $19.20$ | $9.60$ | T.FM |
| Q | CRPS | $9.51e^{-1}$ | 1.00 | $8.23e^{-1}$ | $7.97e^{-1}$ | $8.41e^{-1}$ | $8.37e^{-1}$ | 1.02 | $9.72e^{-1}$ | $8.35e^{-1}$ | $7.97e^{-1}$ | $8.53e^{-1}$ | 1.05 | $8.46e^{-1}$ | $8.40e^{-1}$ | $8.40e^{-1}$ | $9.32e^{-1}$ | 1.13 | $7.88e^{-1}$ | Moi.L |
|  | MAPE | $9.30e^{-1}$ | 1.00 | $8.59e^{-1}$ | $8.38e^{-1}$ | $9.44e^{-1}$ | $9.37e^{-1}$ | 1.13 | $8.38e^{-1}$ | $9.37e^{-1}$ | $9.08e^{-1}$ | $8.80e^{-1}$ | $9.37e^{-1}$ | $8.24e^{-1}$ | $8.10e^{-1}$ | $8.10e^{-1}$ | 1.04 | 1.43 | $8.87e^{-1}$ | Chr.B |
|  | Rank | $14.00$ | $16.00$ | $5.00$ | $2.00$ | $10.00$ | $7.00$ | $17.00$ | $15.00$ | $6.00$ | $3.00$ | $12.00$ | $18.00$ | $11.00$ | $8.00$ | $9.00$ | $13.00$ | $20.00$ | $1.00$ | Moi.L |
| A | CRPS | $9.93e^{-1}$ | 1.00 | $9.42e^{-1}$ | $8.33e^{-1}$ | $8.19e^{-1}$ | $7.97e^{-1}$ | 1.12 | $9.71e^{-1}$ | $8.48e^{-1}$ | $8.48e^{-1}$ | $8.48e^{-1}$ | 1.15 | 1.01 | $9.78e^{-1}$ | $9.78e^{-1}$ | $8.26e^{-1}$ | $9.42e^{-1}$ | $7.75e^{-1}$ | Moi.L |
|  | MAPE | 1.00 | 1.00 | 1.02 | $9.37e^{-1}$ | 1.02 | $9.26e^{-1}$ | 1.21 | $9.03e^{-1}$ | 1.01 | 1.03 | $9.71e^{-1}$ | 1.09 | 1.00 | $9.77e^{-1}$ | $9.77e^{-1}$ | $9.77e^{-1}$ | 1.21 | $9.54e^{-1}$ | N-BEATS |
|  | Rank | $15.00$ | $16.00$ | $10.00$ | $6.00$ | $4.00$ | $2.00$ | $18.00$ | $12.00$ | $8.00$ | $7.00$ | $9.00$ | $19.00$ | $17.00$ | $13.00$ | $14.00$ | $5.00$ | $11.00$ | $1.00$ | Moi.L |

The analysis by prediction length reveals marked differences in performance across various forecast horizons. For short-term forecasts, foundation models, particularly the Moirai variants, consistently outperform other models, underscoring their robustness in handling immediate trends and fluctuations. As the prediction length extends to medium and long terms, the performance of TimesFM and Chronos have significantly declined. This is because the decoder-only architecture adopts recursive multi-step forecast strategy, leading to severe accumulation error issue. Meanwhile, deep learning models such as PatchTST demonstrate superior performance on medium and long terms forecast. This trend indicates that the fine-tuning of foundation models effectively captures longer-term dependencies, which are crucial for accurate predictions over extended periods. Thus despite the progress in foundational time series research, there remains a notable performance gap between deep learning and foundation models for medium to long-term predictions, suggesting an area ripe for further research.

##### Frequency | Table 8

In the analysis of model performance by data frequency, the highest frequency (secondly) is predominantly led by a statistical baseline, Auto_Theta model, indicating its strength in rapid, granular trend capture. As the frequency decreases, deep learning models like iTransformer, PatchTST, and TFT begin to assert dominance, particularly for minutely and hourly data granularities. However, from daily to yearly frequencies, foundation models consistently outperform other approaches, securing the best results across these settings. This pattern suggests that foundation models, with their extensive pretraining, are better equipped to leverage the broader patterns and slower dynamics typical of lower frequency data, enhancing their forecasting accuracy. Conversely, higher frequency data, which often contains more noise and less discernible patterns, poses challenges that foundation models are less suited to address, as evidenced by their performance relative to more specialized deep learning and statistical models.

##### Number of variates | Table 9

**Table 9: Results on GIFT-Eval aggregated by number of variates. The best results across each row are bolded, while the second best results are underlined.**
| Num. Var. | Metric | Nv. | S.Nv. | A.Ar. | A.Th. | DeepAR | TFT | TiDE | N-BEATS | P.TST | iTrans. | T.FM | Vis.TS | Chr.S | Chr.B | Chr.L | Moi.S | Moi.B | Moi.L | Best |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Multi.v | MAPE | 1.21 | 1.00 | $9.80e^{-1}$ | 2.41 | 1.54 | 1.40 | 1.44 | 1.43 | $8.75e^{-1}$ | 1.15 | 1.71 | 1.04 | 1.08 | 1.11 | 1.10 | $8.84e^{-1}$ | 1.09 | $9.32e^{-1}$ | P.TST |
|  | CRPS | 1.40 | 1.00 | $8.71e^{-1}$ | 1.19 | 1.18 | $6.09e^{-1}$ | $8.20e^{-1}$ | $7.35e^{-1}$ | $5.27e^{-1}$ | $6.22e^{-1}$ | $7.50e^{-1}$ | $7.01e^{-1}$ | $6.65e^{-1}$ | $6.83e^{-1}$ | $6.73e^{-1}$ | $6.26e^{-1}$ | $6.28e^{-1}$ | $6.27e^{-1}$ | P.TST |
|  | Rank | $17.70$ | $16.51$ | $13.91$ | $14.98$ | $14.00$ | $6.91$ | $11.40$ | $12.58$ | $5.02$ | $5.19$ | $10.07$ | $11.53$ | $9.51$ | $9.44$ | $9.56$ | $6.33$ | $6.28$ | $6.67$ | P.TST |
| Uni.v | MAPE | 1.19 | 1.00 | $9.45e^{-1}$ | 1.18 | $9.49e^{-1}$ | $8.92e^{-1}$ | $9.60e^{-1}$ | $7.96e^{-1}$ | $8.47e^{-1}$ | $8.53e^{-1}$ | $8.72e^{-1}$ | $9.34e^{-1}$ | $8.11e^{-1}$ | $8.06e^{-1}$ | $7.99e^{-1}$ | $8.80e^{-1}$ | $9.47e^{-1}$ | $8.10e^{-1}$ | N-BEATS |
|  | CRPS | 1.74 | 1.00 | $7.62e^{-1}$ | 1.59 | $8.02e^{-1}$ | $5.85e^{-1}$ | $7.23e^{-1}$ | $8.03e^{-1}$ | $5.92e^{-1}$ | $6.28e^{-1}$ | $6.30e^{-1}$ | $7.49e^{-1}$ | $6.18e^{-1}$ | $6.00e^{-1}$ | $5.95e^{-1}$ | $6.55e^{-1}$ | $6.63e^{-1}$ | $5.66e^{-1}$ | Moi.L |
|  | Rank | $18.57$ | $16.74$ | $12.56$ | $15.39$ | $10.00$ | $6.65$ | $11.70$ | $13.80$ | $6.28$ | $7.31$ | $7.33$ | $13.72$ | $8.89$ | $7.43$ | $7.31$ | $9.00$ | $8.59$ | $5.44$ | Moi.L |

In multivariate settings, deep learning models, particularly PatchTST (best) and iTransformer (second best), stand out by delivering the best scores across all evaluated metrics. Moirai outperforms other foundation models, as it is the only model that supports multivariate forecasting.
On the other hand, in univariate scenarios, foundation models, especially the large variant of Moirai, demonstrate superior performance over their deep learning counterparts. This suggests that foundation models, with their broader pretraining on diverse data sets, are particularly adept at extracting and leveraging predictive signals from single streams of data.

##### General | Table 10

**Table 10: Results on GIFT-Eval aggregated by all results. The best results across each row are bolded, while the second best results are underlined.**
| Metric | Nv. | S.Nv. | A.Ar. | A.Th. | DeepAR | TFT | TiDE | N-BEATS | P.TST | iTrans. | T.FM | Vis.TS | Chr.S | Chr.B | Chr.L | Moi.S | Moi.B | Moi.L | Best |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| MAPE | 1.20 | 1.00 | $9.61e^{-1}$ | 1.73 | 1.21 | 1.12 | 1.17 | 1.08 | $8.60e^{-1}$ | $9.85e^{-1}$ | 1.25 | $9.82e^{-1}$ | $9.28e^{-1}$ | $9.40e^{-1}$ | $9.30e^{-1}$ | $8.82e^{-1}$ | 1.01 | $8.64e^{-1}$ | P.TST |
| CRPS | 1.59 | 1.00 | $8.11e^{-1}$ | 1.41 | $9.69e^{-1}$ | $5.96e^{-1}$ | $7.66e^{-1}$ | $7.73e^{-1}$ | $5.63e^{-1}$ | $6.26e^{-1}$ | $6.83e^{-1}$ | $7.28e^{-1}$ | $6.38e^{-1}$ | $6.37e^{-1}$ | $6.29e^{-1}$ | $6.42e^{-1}$ | $6.48e^{-1}$ | $5.93e^{-1}$ | P.TST |
| Rank | $18.19$ | $16.64$ | $13.15$ | $15.21$ | $11.77$ | $6.76$ | $11.57$ | $13.26$ | $5.72$ | $6.37$ | $8.55$ | $12.75$ | $9.16$ | $8.32$ | $8.31$ | $7.81$ | $7.57$ | $5.99$ | P.TST |

**Table 11: Best and second best counts for each model across GIFT-Eval dataset configurations (97) according to the Rank metric. The best results across each row are bolded.**
|  | Moi.L | P.TST | iTrans. | Moi.B | C.former | T.FM | TFT | Moi.S | Chr.B | Chr.L | A.Th. | DeepAR | A.Ar. | Chr.S | N-BEATS | A.ETS | Nv. | S.Nv. | TiDE | DLin. | Vis.TS |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Best | 12 | 10 | 7 | 8 | 16 | 10 | 11 | 5 | 2 | 6 | 6 | 2 | 2 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
| Second Best | 12 | 10 | 13 | 11 | 2 | 6 | 5 | 10 | 9 | 3 | 1 | 5 | 3 | 3 | 2 | 2 | 0 | 0 | 0 | 0 | 0 |
| Total | 24 | 20 | 20 | 19 | 18 | 16 | 16 | 15 | 11 | 9 | 7 | 7 | 5 | 3 | 2 | 2 | 0 | 0 | 0 | 0 | 0 |

The comprehensive aggregation of results across the entire benchmark offers insightful performance distinctions. PatchTST emerges as the most dominant model, securing the top average scores in all metrics, with MoiraiLarge consistently following in second place. We also present the number of times each model achieves the best or second best results in Table 11. Crossformer appears most frequently as the best performer, and MoiraiLarge as the model that appears in top 2 most frequently. The discrepancy between ranking and count based evaluation suggests that certain datasets may disproportionately influence the metric-based results, which is not captured by the ranking-based outcomes.
Thus PatchTST offers reliable results across diverse datasets, making it a strong generalist. In contrast, MoiraiLarge delivers better performance on particular cases.

Some recent works  (Shi et al., ; ; Ansari et al., ) have verified
the scaling law in time series foundation models (i.e., larger model performs better), however, GIFT-Eval does not consistently support this conclusion. It only holds in the energy domain or univariate forecasting when only considering ranking metrics.

### 4.2 Qualitative Results / Failure Cases

In addition to the quantitative results discussed earlier, we present qualitative analyses by sharing forecasting samples across various datasets using both deep learning and foundation models. For the deep learning models, we selected four representatives: PatchTST and iTransformer, from recent transformer-based architectures, and DeepAR and N-BEATS, which are more traditional deep learning approaches. Regarding foundational models, we included Moirai to represent encoder-decoder architectures, Chronos as a decoder-only model, and VisionTS due to its unique method of representing the time series through image modality. By examining how these models perform on different datasets, we aim to provide deeper insights into their forecasting behaviors, strengths, and limitations.

Figure: (a) Deep learning forecasts sampled on Bizitobs_l2c, hourly dataset with medium prediction length.
Refer to caption: x5.png

The plots in Figures 2(a) and 2(b) show forecasts by deep learning models on the multivariate Bizitobs_l2c dataset (hourly, medium-term) and the univariate Solar dataset (ten-minutely, medium-term). In Figure 2(a) the irregular patterns challenge the models, with only PatchTST getting close to capturing some of the regular spikes accurately. DeepAR and N-BEATS perform reasonably but miss key periodic spikes, while iTransformer, despite its multivariate capability, oversimplifies the data into a sinusoidal pattern. In Figure 2(b), traditional models handle seasonal data better but still tend to underpredict, with N-BEATS producing a flat forecast in the second plot. PatchTST consistently outperforms others in both instances, showing robustness with both regular and irregular series, while iTransformer continues to underperform.

The plots in Figures 2(c) and 2(d) show forecasts by foundation models on two univariate datasets: M_DENSE (daily, long-term) and Solar (ten-minutely, medium-term). Figure 2(c) displays varying performance among the foundation models. Chronos shows a clear degradation in performance as the prediction horizon extends, struggling to maintain accuracy over time, while VisionTS captures spikes but misaligns them. Moirai offers smoother, more conservative forecasts, which may result in less sensitivity to extreme events but provide more consistent alignment with the general trend. In Figure 2(d) VisionTS predicts seasonal peaks but with timing shifts. On the other hand, both Moirai and Chronos struggle to capture the well-spaced regularity of the data, missing key trends altogether. These poor results across all foundation models (see Figure 2(b) vs Figure 2(d)) mirror the quantitative findings in Section 4.1, i.e. deep learning models outperform foundation models at higher frequencies. For more qualitative examples see Section F.3

## 5 Conclusion

We introduce GIFT-Eval, a benchmark designed to evaluate time series forecasting models with diversity across four key characteristics: domain, frequency, number of variates, and prediction length. We ensure additional diversity by verifying six statistical features across temporal attributes, forecastability, and regularity. In addition to the train/test dataset, we also provide a pretraining dataset with no leakage into our evaluation set. With this, we aim to provide the necessary ground for fairly comparing different families of models, including foundation models, across a diverse benchmark. We conduct comprehensive experiments with 17 baselines encompassing statistical, deep learning, and foundation models. Leveraging our detailed taxonomy, we provide insights into each model’s strengths relative to different characteristics. We also conduct a qualitative analysis highlighting failure cases in both deep learning and foundation models. GIFT-Eval is a comprehensive benchmark with fine-grained taxonomy that we hope will accelerate the development of new foundation time series forecasting models.

## Appendix A Experimental setup details

##### Statistical models

We utilize the statsforecast (Garza et al., ) library to implement all five statistical baselines: Naive, Seasonal Naive, Auto_Theta, Auto_ETS and Auto_Arima. Inference is performed on a CPU server equipped with 96 cores. For each dataset, a time limit of one day is set for the statistical model to complete its run. For any model that times out we halt it and replace its results with those from the Seasonal Naive model as a fallback. Given that some datasets in our benchmark are particularly long, we impose a maximum size constraint on each statistical baseline (set to 1000 with our time constraints), truncating the time series to this maximum size.

##### Deep learning models

**Table 12: Hyperparameter search range for deep learning baselines.**
|  | TiDE |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Parameters | num_layers_encoder | num_layers_decoder | hidden_dim | temporal_hidden_dim | decoder_output_dim | dropout_rate | lr |  |
| Search Range | [1,2] | [1,2] | [256,512,1024] | [64,128] | [8,16,32] | [0.0, 0.5] | [1e-5:1e-1] |  |
|  | N-BEATS |  |  |  |  | PatchTST |  |  |
| Parameters | loss_function | hidden_layer_units | share_weights_in_stack | nb_blocks_per_stack | lr | d_model | num_encoder_layers | lr |
| Search Range | ["mase", "mape", "smape"] | [256, 512, 1024, 2048] | [True, False] | [3, 4] | [1e-5:1e-1] | [128, 256, 512] | [2, 3, 4] | [1e-5:1e-1] |
|  | iTransformer |  |  | DeepAR |  |  | DLinear |  |
| Parameters | d_model | num_encoder_layers | lr | hidden_size | num_layers | lr | lr |  |
| Search Range | [128, 256, 512] | [2, 3, 4] | [1e-5:1e-1] | [20,25,…,80] | [1,2,3,4] | [1e-5:1e-1] | [1e-5:1e-1] |  |
|  | Crossformer |  |  | TFT |  |  |  |  |
| Parameters | d_model | n_heads | lr | num_heads | hidden_dim | lr |  |  |
| Search Range | [64,128,256] | [2,4,8] | [1e-5:5e-3] | [2,4,8] | [16,32,64] | [1e-5:1e-1] |  |  |

For all deeplearning models we either used models readily available in gluonts library (Alexandrov et al., ) or we write our own wrappers. Where feasible we also add a probabilistic forecasting head to the models. Where direct probabilistic outputs are not feasible, we generate probabilistic evaluations by converting point forecasts into sample forecasts using a single sample. To identify the optimal hyperparameters, we conducted a comprehensive search across all 97 runs included in GIFT-Eval. We employed the ray library (Moritz et al., ) to parallelize the search on a single GPU and used the optuna (Akiba et al., ) library to extend this parallelization across multiple GPU servers. We search for 15 trials for each deep learning model per each of the 97 runs. Table 12 lists the range of parameters we search for each model. On top of the listed parameters for each model, we also search for weight decay on all runs in the range: $[1e-8:1e-2]$. For the Crossformer model on the long term setting of Jena Weather dataset with both ten–minutely and hourly frequencies, we had to limit the search for d_model and n_heads, fixing them at 32 and 1, respectively. This adjustment was necessary because the model’s attention mechanism operates across multiple variates, leading to an OOM (Out of Memory) error due to the high number of variates present in this dataset.

##### Foundation models

For all foundation models we use their public versions available online and conduct zero-shot evaluation on our benchmark’s test-split. Since Moirai (Woo et al., ) provides multi-patch size projections and varying context lengths. We adopt the similar approach by defining a frequency-to-patch size mapping as follows:

- •
Yearly, Quarterly: 8
- •
Monthly: 8
- •
Weekly, Daily: 16
- •
Hourly: 32
- •
Minute-level: 32
- •
Second-level: 64

We search an optimal context length in the range $[1000,2000,4000,8000]$. We used the public available Moirai models from the corresponding HuggingFace repos, i.e., MoiraiSmall - [https://huggingface.co/Salesforce/moirai-1.1-R-small](https://huggingface.co/Salesforce/moirai-1.1-R-small), MoiraiBase - [https://huggingface.co/Salesforce/moirai-1.1-R-base](https://huggingface.co/Salesforce/moirai-1.1-R-base), MoiraiLarge - [https://huggingface.co/Salesforce/moirai-1.1-R-large](https://huggingface.co/Salesforce/moirai-1.1-R-large).

For Chronos, we mainly follow their official implementation(^2^22[https://github.com/amazon-science/chronos-forecasting/blob/main/scripts/evaluation/evaluate.py](https://github.com/amazon-science/chronos-forecasting/blob/main/scripts/evaluation/evaluate.py)) for evaluation: with the number of samples as 20. The models are loaded from the corresponding HuggingFace repos, e.g., ChronosTiny - [https://huggingface.co/amazon/chronos-t5-tiny](https://huggingface.co/amazon/chronos-t5-tiny), ChronosSmall - [https://huggingface.co/amazon/chronos-t5-small](https://huggingface.co/amazon/chronos-t5-small), ChronosBase- [https://huggingface.co/amazon/chronos-t5-base](https://huggingface.co/amazon/chronos-t5-base).

For TimesFM, we follow their official implementation(^3^33[https://github.com/google-research/timesfm/blob/master/experiments/long_horizon_benchmarks/run_eval.py](https://github.com/google-research/timesfm/blob/master/experiments/long_horizon_benchmarks/run_eval.py)) for evaluation. We set the context length for evaluation as 512 as mentioned in their paper since the maximum context length in training is 512.
Following their default setting in their example, we keep the input patch length as 32, the output patch length as 128, the number of layers as 20, and the model dimension as 1280.
TimesFM comes with only one model size, i.e., timesfm-1.0-200m, and we load the model from [https://huggingface.co/google/timesfm-1.0-200m](https://huggingface.co/google/timesfm-1.0-200m).

For VisionTS, we follow their official implementation(^4^44[https://github.com/Keytoyze/VisionTS/blob/main/eval_gluonts/run.py](https://github.com/Keytoyze/VisionTS/blob/main/eval_gluonts/run.py)) for evaluation. We set the context length as 2000, the norm constant as 0.4, the alignment constant as 0.4 according to their default settings. We use their implementation for seasonality detection to generate a candidate list and search an optimal seasonality parameter with the validation data.

##### Additional parameters and computational resources.

All experiments are conducted on eight NVIDIA A100 GPUs. For models that has gone through training the loss function and optimizer are set following their original implementation. Additionally we set the batch size to 128 and, number of batches per epoch to 100, and finally number of epochs to 50.

## Appendix B Details of Time Series Features

This section gives a detailed view of the time series features we used to analyze the test portion of our data in Section 3.2. We use tsfeatures library (Garza et al., ) to calculate each metric. Given the scale of our dataset, we limit each time series history to the most recent 500 data points before computing the respective features. The prediction length remains faithful to the original values specified for the dataset and is not clipped.

We also acknowledge that for some overly short time series, tsfeatures may output NaN (Not a Number) values for certain features—for example, the seasonal strength of some yearly time series data. In such cases, we exclude these NaN values during aggregation. Below we provide specific details for each feature used:

##### Trend

Using the STL (Seasonal and Trend decomposition using Loess) method, a time series $x_{t}$ is decomposed into a trend component $f_{t}$, multiple seasonal components $s_{i,t}$ for $i=1,\dots,M$, and a remainder component $e_{t}$:

$$ $x_{t}=f_{t}+s_{1,t}+\dots+s_{M,t}+e_{t},$ $$

where $M$ is the number of seasonal periods. The strength of the trend is quantified by comparing the variance of the remainder component $e_{t}$ to the combined variance of the trend and remainder components. Specifically, the strength of the trend is defined as:

$$ $\text{trend}=1-\frac{\operatorname{Var}(e_{t})}{\operatorname{Var}(f_{t}+e_{t} )}.$ $$

If the calculated value of trend is less than 0, it is set to 0; if it is greater than 1, it is set to 1. This measure indicates the proportion of the variability in the time series that is explained by the trend component, with values closer to 1 signifying a stronger trend.

##### Seasonal Strength

Following the same decomposition above the strength of each seasonal component is quantified by comparing the variance of the remainder $e_{t}$ to the combined variance of the seasonal component $s_{i,t}$ and the remainder.

For each seasonal component $s_{i,t}$, the strength of seasonality is defined as:

$$ $\text{seasonal_strength}_{i}=1-\frac{\operatorname{Var}(e_{t})}{\operatorname {Var}(s_{i,t}+e_{t})}.$ $$

If the calculated value of $\text{seasonal_strength}_{i}$ is less than 0, it is set to 0; if it is greater than 1, it is set to 1. For non-seasonal time series, $\text{seasonal_strength}=0$. This measure indicates the proportion of the variability in the time series that is explained by the $i$-th seasonal component, with values closer to 1 signifying stronger seasonality for that component.

##### Entropy

Entropy is defined as the Shannon entropy of the normalized spectral density estimate $\hat{f}(\lambda)$:

$$ $\text{Entropy}=-\int_{-\pi}^{\pi}\hat{f}(\lambda)\log\hat{f}(\lambda)\,d\lambda,$ $$

where $\hat{f}(\lambda)$ is an estimate of the spectral density of the data. A lower spectral entropy indicates a higher signal-to-noise ratio, meaning the time series has more predictable patterns and is easier to forecast. Conversely, a higher spectral entropy suggests that the series is more complex and difficult to predict.

##### Hurst Exponent

The Hurst exponent (hurst) is computed as $0.5$ plus the maximum likelihood estimate of the fractional differencing order $d$ by Haslett & Raftery (). The addition of $0.5$ ensures consistency with the traditional Hurst coefficient. The values of the Hurst exponent vary between 0 and 1, with higher values indicating a smoother trend, less volatility, and less roughness.

##### Stability

Stability measures the variability of the mean values across all tiles. It is defined as the variance of the means of the tiled windows. If the time series is divided into $N$ tiles and $\bar{x}_{i}$ represents the mean of the $i$-th tile, then the stability is calculated as:

$$ $\text{Stability}=\operatorname{Var}\left(\bar{x}_{1},\bar{x}_{2},\dots,\bar{x} _{N}\right).$ $$

A lower stability indicates that the means are consistent across tiles, suggesting a stable time series. A higher stability implies significant differences in means, indicating potential shifts or trends in the data.

##### Lumpiness

Lumpiness assesses the variability of the variances across all tiles. It is defined as the variance of the variances of the tiled windows. Let $s_{i}^{2}$ denote the variance of the $i$-th tile. Lumpiness is then computed as:

$$ $\text{Lumpiness}=\operatorname{Var}\left(s_{1}^{2},s_{2}^{2},\dots,s_{N}^{2} \right).$ $$

A higher lumpiness suggests that the variability within the tiles differs significantly, indicating that the time series may have periods of high and low volatility. A lower lumpiness means the variances are similar across tiles, pointing to a more homogeneous time series in terms of variability.

## Appendix C Evaluation Metrics

We use two metrics to evaluate performance of forecasters: Mean Absolute Percentage Error (MAPE) for point forecasting ability and Continous Ranked Probability Score (CRPS) for probabilistic forecasting. For both metrics we use gluonts library implementation to calculate final values (Alexandrov et al., ).

##### MAPE

MAPE is an evaluation metric used to measure the accuracy of forecasts in time series analysis. It is defined as the mean of the absolute percentage differences between the actual values $Y_{t}$ and the predicted values $\hat{Y}_{t}$. The formula for MAPE is:

$$ $\text{MAPE}=\frac{1}{n}\sum_{t=1}^{n}\left|\frac{Y_{t}-\hat{Y}_{t}}{Y_{t}} \right|,$ $$

where:

- •
$Y_{t}$ is the actual value at time $t$,
- •
$\hat{Y}_{t}$ is the forecasted value at time $t$,
- •
$n$ is the number of observations.

This metric expresses the forecast error as a percentage of the actual values, making it scale-independent and easy to interpret. However, it is sensitive to values of $Y_{t}$ that are zero or close to zero, as this can lead to division by zero or inflated error percentages.

##### CRPS

The Continuous Ranked Probability Score (CRPS) is a metric used in probabilistic forecasting to evaluate the accuracy of predicted cumulative distribution functions (CDFs) against observed values. Given a predicted distribution with CDF $F$ and a ground truth value $y$, the CRPS is defined as:

$$ $\text{CRPS}(F,y)=\int_{0}^{1}2\Lambda_{\alpha}(F^{-1}(\alpha),y)\,d\alpha,$ $$

where the quantile loss $\Lambda_{\alpha}(q,y)$ is defined as:

$$ $\Lambda_{\alpha}(q,y)=(\alpha-\mathbf{1}\{y<q\})(y-q).$ $$

In practice, computing the CRPS integral can be computationally intensive. To address this, we approximate the CRPS using a discrete sum over a finite set of quantile levels. This approximation, often referred to as the mean weighted quantile loss (Park et al., ), is given by:

$$ $\text{CRPS}\approx\frac{1}{K}\sum_{k=1}^{K}\text{wQL}[\alpha_{k}],$ $$

where $K$ is the number of quantile levels, and $\{\alpha_{1},\alpha_{2},\dots,\alpha_{K}\}$ are the selected quantile levels (e.g., $\alpha_{k}=0.1k$ for $k=1,2,\dots,9$ when $K=9$).

The weighted quantile loss $\text{wQL}[\alpha]$ for each quantile level $\alpha$ is calculated as:

$$ $\text{wQL}[\alpha]=2\frac{\sum_{t}\Lambda_{\alpha}(\hat{q}_{t}(\alpha),y_{t})} {\sum_{t}|y_{t}|},$ $$

where:

- •
$\hat{q}_{t}(\alpha)$ is the predicted $\alpha$-quantile at time step $t$,
- •
$y_{t}$ is the actual observed value at time $t$,
- •
$\Lambda_{\alpha}(\hat{q}_{t}(\alpha),y_{t})$ is the quantile loss at time $t$ for quantile level $\alpha$.

## Appendix D GIFT-Eval test datasets

In this section we provide comprehensive list of datasets used in test portion of GIFT-Eval along with original sources, for details regarding the pretraining portion see Appendix E. We utilize 10 open domain sources to curate the benchmark, here we list each one along with its properties in detail. We incorporate Jena Weather(^5^55[https://www.bgc-jena.mpg.de/wetter/](https://www.bgc-jena.mpg.de/wetter/)) dataset following Autoformer (Wu et al., ). We process BizITObs Application, Service, and L2C(^6^66[https://github.com/BizITObs/BizITObservabilityData/tree/main](https://github.com/BizITObs/BizITObservabilityData/tree/main)) following the pipeline in AutoMixer (Palaskar et al., ).These datasets consist of business and IT observability data, fusing both business KPIs and IT event channels into multivariate time series data. Within the same domain we also process Bitbrains datasets from Grid Workloads Archive (Shen et al., ). The Restaurant data is borrowed from Recruit Restaurant Forecasting Competition (Howard et al., ), The task associated with this dataset is to use reservation and visitation data to predict the total number of visitors to a restaurant for future dates. From Informer (Zhou et al., ) we utilize ETT1 and ETT2 datasets, which denote electricity transformer temperature and serve as an indicator used in the electricity power long-term deployment. Datasets for Transport domain are extracted from LibCity (Wang et al., ), which provides a collection of urban time series datasets. We utilize the solar dataset from LSTNet (Lai et al., ) where the task is to predict solar plant energy outputs. The second and last dataset for Sales data is by Mancuso et al. (). Monash (Godahewa et al., ) is a large collection of diverse time series datasets across many domains, we choose a subset of these datasets making sure there is no leak from pretrain to test split. Finally, from UCI ML Archive (Trindade, ) we use the electricity dataset which contains electricity consumption of 370 individual clients.
Table 13 lists all datasets, along with their source, frequency, prediction length and number of variates setup and presents various statistics from number of series, to series length, and also number of observations. We use last 10% of each timeseries in the test portion of our data for testing and keep the rest for training.

**Table 13: Individual statistics of GIFT-Eval benchmark across all datasets.**
|  |  |  |  |  | Series Length |  |  | Short-term | Med-term | Long-term |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Dataset | Source | Domain | Frequency | # Series | Avg | Min | Max | # Obs | Target Variates | Pred Length(S) | Windows | Pred Length(M) | Windows | Pred Length(L) | Windows |
| Jena Weather | Autoformer (Wu et al., ) | Nature | 10T | 1 | 52,704 | 52,704 | 52,704 | 52,704 | 21 | 48 | 20 | 480 | 11 | 720 | 8 |
| Jena Weather | Autoformer (Wu et al., ) | Nature | H | 1 | 8,784 | 8,784 | 8,784 | 8,784 | 21 | 48 | 19 | 480 | 2 | 720 | 2 |
| Jena Weather | Autoformer (Wu et al., ) | Nature | D | 1 | 366 | 366 | 366 | 366 | 21 | 30 | 2 |  |  |  |  |
| BizITObs - Application | AutoMixer (Palaskar et al., ) | Web/CloudOps | 10S | 1 | 8,834 | 8,834 | 8,834 | 8,834 | 2 | 60 | 15 | 600 | 2 | 900 | 1 |
| BizITObs - Service | AutoMixer (Palaskar et al., ) | Web/CloudOps | 10S | 21 | 8,835 | 8,835 | 8,835 | 185,535 | 2 | 60 | 15 | 600 | 2 | 900 | 1 |
| BizITObs - L2C | AutoMixer (Palaskar et al., ) | Web/CloudOps | 5T | 1 | 31,968 | 31,968 | 31,968 | 31,968 | 7 | 48 | 20 | 480 | 7 | 720 | 5 |
| BizITObs - L2C | AutoMixer (Palaskar et al., ) | Web/CloudOps | H | 1 | 2,664 | 2,664 | 2,664 | 2,664 | 7 | 48 | 6 | 480 | 1 | 720 | 1 |
| Bitbrains - Fast Storage | Grid Workloads Archive (Shen et al., ) | Web/CloudOps | 5T | 1,250 | 8,640 | 8,640 | 8,640 | 10,800,000 | 2 | 48 | 18 | 480 | 2 | 720 | 2 |
| Bitbrains - Fast Storage | Grid Workloads Archive (Shen et al., ) | Web/CloudOps | H | 1,250 | 721 | 721 | 721 | 901,250 | 2 | 48 | 2 |  |  |  |  |
| Bitbrains - rnd | Grid Workloads Archive (Shen et al., ) | Web/CloudOps | 5T | 500 | 8,640 | 8,640 | 8,640 | 4,320,000 | 2 | 48 | 18 | 480 | 2 | 720 | 2 |
| Bitbrains - rnd | Grid Workloads Archive (Shen et al., ) | Web/CloudOps | H | 500 | 720 | 720 | 720 | 360,000 | 2 | 48 | 2 |  |  |  |  |
| Restaurant | Recruit Rest. Comp. (Howard et al., ) | Sales | D | 807 | 358 | 67 | 478 | 289,303 | 1 | 30 | 1 |  |  |  |  |
| ETT1 | Informer (Zhou et al., ) | Energy | 15T | 1 | 69,680 | 69,680 | 69,680 | 69,680 | 7 | 48 | 20 | 480 | 15 | 720 | 10 |
| ETT1 | Informer (Zhou et al., ) | Energy | H | 1 | 17,420 | 17,420 | 17,420 | 17,420 | 7 | 48 | 20 | 480 | 4 | 720 | 3 |
| ETT1 | Informer (Zhou et al., ) | Energy | D | 1 | 725 | 725 | 725 | 725 | 7 | 30 | 3 |  |  |  |  |
| ETT1 | Informer (Zhou et al., ) | Energy | W-THU | 1 | 103 | 103 | 103 | 103 | 7 | 8 | 2 |  |  |  |  |
| ETT2 | Informer (Zhou et al., ) | Energy | 15T | 1 | 69,680 | 69,680 | 69,680 | 69,680 | 7 | 48 | 20 | 480 | 15 | 720 | 10 |
| ETT2 | Informer (Zhou et al., ) | Energy | H | 1 | 17,420 | 17,420 | 17,420 | 17,420 | 7 | 48 | 20 | 480 | 4 | 720 | 3 |
| ETT2 | Informer (Zhou et al., ) | Energy | D | 1 | 725 | 725 | 725 | 725 | 7 | 30 | 3 |  |  |  |  |
| ETT2 | Informer (Zhou et al., ) | Energy | W-THU | 1 | 103 | 103 | 103 | 103 | 7 | 8 | 2 |  |  |  |  |
| Loop Seattle | LibCity (Wang et al., ) | Transport | 5T | 323 | 105,120 | 105,120 | 105,120 | 33,953,760 | 1 | 48 | 20 | 480 | 20 | 720 | 15 |
| Loop Seattle | LibCity (Wang et al., ) | Transport | H | 323 | 8,760 | 8,760 | 8,760 | 2,829,480 | 1 | 48 | 19 | 480 | 2 | 720 | 2 |
| Loop Seattle | LibCity (Wang et al., ) | Transport | D | 323 | 365 | 365 | 365 | 117,895 | 1 | 30 | 2 |  |  |  |  |
| SZ-Taxi | LibCity (Wang et al., ) | Transport | 15T | 156 | 2,976 | 2,976 | 2,976 | 464,256 | 1 | 48 | 7 | 480 | 1 | 720 | 1 |
| SZ-Taxi | LibCity (Wang et al., ) | Transport | H | 156 | 744 | 744 | 744 | 116,064 | 1 | 48 | 2 |  |  |  |  |
| M_DENSE | LibCity (Wang et al., ) | Transport | H | 30 | 17,520 | 17,520 | 17,520 | 525,600 | 1 | 48 | 20 | 480 | 4 | 720 | 3 |
| M_DENSE | LibCity (Wang et al., ) | Transport | D | 30 | 730 | 730 | 730 | 21,900 | 1 | 30 | 3 |  |  |  |  |
| Solar | LSTNet (Lai et al., ) | Energy | 10T | 137 | 52,560 | 52,560 | 52,560 | 7,200,720 | 1 | 48 | 20 | 480 | 11 | 720 | 8 |
| Solar | LSTNet (Lai et al., ) | Energy | H | 137 | 8,760 | 8,760 | 8,760 | 1,200,120 | 1 | 48 | 19 | 480 | 2 | 720 | 2 |
| Solar | LSTNet (Lai et al., ) | Energy | D | 137 | 365 | 365 | 365 | 50,005 | 1 | 30 | 2 |  |  |  |  |
| Solar | LSTNet (Lai et al., ) | Energy | W-FRI | 137 | 52 | 52 | 52 | 7,124 | 1 | 8 | 1 |  |  |  |  |
| Hierarchical Sales | Mancuso et al. () | Sales | D | 118 | 1,825 | 1,825 | 1,825 | 215,350 | 1 | 30 | 7 |  |  |  |  |
| Hierarchical Sales | Mancuso et al. () | Sales | W-WED | 118 | 260 | 260 | 260 | 30,680 | 1 | 8 | 4 |  |  |  |  |
| M4 Yearly | Monash (Godahewa et al., ) | Econ/Fin | A-DEC | 22,974 | 37 | 19 | 284 | 845,109 | 1 | 6 | 1 |  |  |  |  |
| M4 Quarterly | Monash (Godahewa et al., ) | Econ/Fin | Q-DEC | 24,000 | 100 | 24 | 874 | 2,406,108 | 1 | 8 | 1 |  |  |  |  |
| M4 Monthly | Monash (Godahewa et al., ) | Econ/Fin | M | 48,000 | 234 | 60 | 2,812 | 11,246,411 | 1 | 18 | 1 |  |  |  |  |
| M4 Weekly | Monash (Godahewa et al., ) | Econ/Fin | W-SUN | 359 | 1,035 | 93 | 2,610 | 371,579 | 1 | 13 | 1 |  |  |  |  |
| M4 Daily | Monash (Godahewa et al., ) | Econ/Fin | D | 4,227 | 2,371 | 107 | 9,933 | 10,023,836 | 1 | 14 | 1 |  |  |  |  |
| M4 Hourly | Monash (Godahewa et al., ) | Econ/Fin | H | 414 | 902 | 748 | 1,008 | 373,372 | 1 | 48 | 2 |  |  |  |  |
| Hospital | Monash (Godahewa et al., ) | Healthcare | M | 767 | 84 | 84 | 84 | 64,428 | 1 | 12 | 1 |  |  |  |  |
| COVID Deaths | Monash (Godahewa et al., ) | Healthcare | D | 266 | 212 | 212 | 212 | 56,392 | 1 | 30 | 1 |  |  |  |  |
| US Births | Monash (Godahewa et al., ) | Healthcare | D | 1 | 7,305 | 7,305 | 7,305 | 7,305 | 1 | 30 | 20 |  |  |  |  |
| US Births | Monash (Godahewa et al., ) | Healthcare | W-TUE | 1 | 1,043 | 1,043 | 1,043 | 1,043 | 1 | 8 | 14 |  |  |  |  |
| US Births | Monash (Godahewa et al., ) | Healthcare | M | 1 | 240 | 240 | 240 | 240 | 1 | 12 | 2 |  |  |  |  |
| Saugeen | Monash (Godahewa et al., ) | Nature | D | 1 | 23,741 | 23,741 | 23,741 | 23,741 | 1 | 30 | 20 |  |  |  |  |
| Saugeen | Monash (Godahewa et al., ) | Nature | W-THU | 1 | 3,391 | 3,391 | 3,391 | 3,391 | 1 | 8 | 20 |  |  |  |  |
| Saugeen | Monash (Godahewa et al., ) | Nature | M | 1 | 780 | 780 | 780 |  | 1 | 12 | 7 |  |  |  |  |
| Temperature Rain | Monash (Godahewa et al., ) | Nature | D | 32,072 | 725 | 725 | 725 | 780 | 1 | 30 | 3 |  |  |  |  |
| KDD Cup 2018 | Monash (Godahewa et al., ) | Nature | H | 270 | 10,898 | 9,504 | 10,920 | 2,942,364 | 1 | 48 | 20 | 480 | 2 | 720 | 2 |
| KDD Cup 2018 | Monash (Godahewa et al., ) | Nature | D | 270 | 455 | 396 | 455 | 122,791 | 1 | 30 | 2 |  |  |  |  |
| Car Parts | Monash (Godahewa et al., ) | Sales | M | 2,674 | 51 | 51 | 51 | 136,374 | 1 | 12 | 1 |  |  |  |  |
| Electricity | UCI ML Archive (Trindade, ) | Energy | 15T | 370 | 140,256 | 140,256 | 140,256 | 51,894,720 | 1 | 48 | 20 | 480 | 20 | 720 | 20 |
| Electricity | UCI ML Archive (Trindade, ) | Energy | H | 370 | 35,064 | 35,064 | 35,064 | 12,973,680 | 1 | 48 | 20 | 480 | 8 | 720 | 5 |
| Electricity | UCI ML Archive (Trindade, ) | Energy | D | 370 | 1,461 | 1,461 | 1,461 | 540,570 | 1 | 30 | 5 |  |  |  |  |
| Electricity | UCI ML Archive (Trindade, ) | Energy | W-FRI | 370 | 208 | 208 | 208 | 76,960 | 1 | 8 | 3 |  |  |  |  |

## Appendix E GIFT-Eval pre-training datasets

The pre-training split of GIFT-Eval is constructed based on LOTSA (Woo et al., ), and we excluded certain datasets from it to form part of the evaluation set, making it more diverse and balanced.

Here is a brief discussion on each of the used sources: BuildingsBench (Emami et al., ) compiled datasets on residential and commercial building energy consumption. ClimateLearn (Nguyen et al., ) offered time series of various climate-related variables, including temperature, humidity, and multiple pressure levels. CloudOps TSF (Woo et al., ) introduced large-scale CloudOps time series datasets that capture key variables such as CPU and memory utilization. GluonTS (Alexandrov et al., ) provided a variety of datasets commonly used in time series forecasting. LargeST (Liu et al., ) sourced from the California Department of Transportation Performance Measurement System (PeMS) to date, which is widely used for traffic forecasting. LibCity (Wang et al., ) provided a collection urban spatio-temporal datasets. SubseasonalClimateUSA (Mouatadid et al., ) provided climate time series data at daily level. ProEnFo (Wang et al., ) introduced a range of datasets for load forecasting which include various covariates such as temperature, humidity, and wind speed. Monash (Godahewa et al., ) is a large collection of diverse time series datasets, the most popular source for building time series foundation models. LOTSA$_$Others (Woo et al., ) are complementary datasets collected by LOTSA to enhance the diversity.

The complete list of pre-training datasets and their respective sources, key properties are provided in  Table 14.

**Table 14: Pretraining datasets and their key properties.**
| Dataset | Source | Domain | Frequency | # Time Series | # Targets | # Covariates | # Obs. |
| --- | --- | --- | --- | --- | --- | --- | --- |
| BDG-2 Panther | BuildingsBench (Emami et al., ) | Energy | H | 105 | 1 | 0 | 919,800 |
| BDG-2 Fox | BuildingsBench (Emami et al., ) | Energy | H | 135 | 1 | 0 | 2,324,568 |
| BDG-2 Rat | BuildingsBench (Emami et al., ) | Energy | H | 280 | 1 | 0 | 4,728,288 |
| BDG-2 Bear | BuildingsBench (Emami et al., ) | Energy | H | 91 | 1 | 0 | 1,482,312 |
| Low Carbon London | BuildingsBench (Emami et al., ) | Energy | H | 713 | 1 | 0 | 9,543,348 |
| SMART | BuildingsBench (Emami et al., ) | Energy | H | 5 | 1 | 0 | 95,709 |
| IDEAL | BuildingsBench (Emami et al., ) | Energy | H | 219 | 1 | 0 | 1,265,672 |
| Sceaux | BuildingsBench (Emami et al., ) | Energy | H | 1 | 1 | 0 | 34,223 |
| Borealis | BuildingsBench (Emami et al., ) | Energy | H | 15 | 1 | 0 | 83,269 |
| Buildings900K | BuildingsBench (Emami et al., ) | Energy | H | 1,792,328 | 1 | 0 | 15,702,590,000 |
| CMIP6 | ClimateLearn (Nguyen et al., ) | Climate | 6H | 1,351,680 | 53 | 0 | 1,973,453,000 |
| ERA5 | ClimateLearn (Nguyen et al., ) | Climate | H | 245,760 | 45 | 0 | 2,146,959,000 |
| Azure VM Traces 2017 | CloudOpsTSF (Woo et al., ) | CloudOps | 5T | 159,472 | 1 | 2 | 885,522,908 |
| Borg Cluster Data 2011 | CloudOpsTSF (Woo et al., ) | CloudOps | 5T | 143,386 | 2 | 5 | 537,552,854 |
| Alibaba Cluster Trace 2018 | CloudOpsTSF (Woo et al., ) | CloudOps | 5T | 58,409 | 2 | 6 | 95,192,530 |
| Taxi | GluonTS (Alexandrov et al., ) | Transport | 30T | 67,984 | 1 | 0 | 54,999,060 |
| Uber TLC Daily | GluonTS (Alexandrov et al., ) | Transport | D | 262 | 1 | 0 | 47,087 |
| Uber TLC Hourly | GluonTS (Alexandrov et al., ) | Transport | H | 262 | 1 | 0 | 1,129,444 |
| Wiki-Rolling | GluonTS (Alexandrov et al., ) | Web | D | 47,675 | 1 | 0 | 40,619,100 |
| M5 | GluonTS (Alexandrov et al., ) | Sales | D | 30,490 | 1 | 0 | 58,327,370 |
| LargeST | LargeST (Liu et al., ) | Transport | 5T | 42,333 | 1 | 0 | 4,452,510,528 |
| PEMS03 | LibCity (Wang et al., ) | Transport | 5T | 358 | 1 | 0 | 9,382,464 |
| PEMS04 | LibCity (Wang et al., ) | Transport | 5T | 307 | 3 | 0 | 5,216,544 |
| PEMS07 | LibCity (Wang et al., ) | Transport | 5T | 883 | 1 | 0 | 24,921,792 |
| PEMS08 | LibCity (Wang et al., ) | Transport | 5T | 170 | 3 | 0 | 3,035,520 |
| PEMS Bay | LibCity (Wang et al., ) | Transport | 5T | 325 | 1 | 0 | 16,937,700 |
| Los-Loop | LibCity (Wang et al., ) | Transport | 5T | 207 | 1 | 0 | 7,094,304 |
| Beijing Subway | LibCity (Wang et al., ) | Transport | 30T | 276 | 2 | 11 | 248,400 |
| SHMetro | LibCity (Wang et al., ) | Transport | 15T | 288 | 2 | 0 | 1,934,208 |
| HZMetro | LibCity (Wang et al., ) | Transport | 15T | 80 | 2 | 0 | 146,000 |
| Q-Traffic | LibCity (Wang et al., ) | Transport | 15T | 45,148 | 1 | 0 | 264,386,688 |
| Subseasonal | SubseasonalClimateUSA (Mouatadid et al., ) | Climate | D | 862 | 4 | 0 | 14,097,148 |
| Subseasonal Precipitation | SubseasonalClimateUSA  (Mouatadid et al., ) | Climate | D | 862 | 1 | 0 | 9,760,426 |
| Covid19 Energy | ProEnFo (Wang et al., ) | Energy | H | 1 | 1 | 6 | 31,912 |
| GEF12 | ProEnFo  (Wang et al., ) | Energy | H | 20 | 1 | 1 | 788,280 |
| GEF14 | ProEnFo (Wang et al., ) | Energy | H | 1 | 1 | 1 | 17,520 |
| GEF17 | ProEnFo (Wang et al., ) | Energy | H | 8 | 1 | 1 | 140,352 |
| PDB | ProEnFo (Wang et al., ) | Energy | H | 1 | 1 | 1 | 17,520 |
| Spanish | ProEnFo (Wang et al., ) | Energy | H | 1 | 1 | 1 | 35,064 |
| BDG-2 Hog | ProEnFo (Wang et al., ) | Energy | H | 24 | 1 | 5 | 421,056 |
| BDG-2 Bull | ProEnFo (Wang et al., ) | Energy | H | 41 | 1 | 3 | 719,304 |
| BDG-2 Cockatoo | ProEnFo (Wang et al., ) | Energy | H | 1 | 1 | 5 | 17,544 |
| ELF | ProEnFo (Wang et al., ) | Energy | H | 1 | 1 | 0 | 21,792 |
| London Smart Meters | Monash (Godahewa et al., ) | Energy | 30T | 5,520 | 1 | 0 | 166,238,880 |
| Wind Farms | Monash (Godahewa et al., ) | Energy | T | 337 | 1 | 0 | 172,165,370 |
| Wind Power | Monash (Godahewa et al., ) | Energy | 4S | 1 | 1 | 0 | 7,397,147 |
| Solar Power | Monash (Godahewa et al., ) | Energy | 4S | 1 | 1 | 0 | 7,397,222 |
| Oikolab Weather | Monash (Godahewa et al., ) | Climate | H | 8 | 1 | 0 | 800,456 |
| Elecdemand | Monash (Godahewa et al., ) | Energy | 30T | 1 | 1 | 0 | 17,520 |
| Covid Mobility | Monash (Godahewa et al., ) | Transport | D | 362 | 1 | 0 | 148,602 |
| Kaggle Web Traffic Weekly | Monash (Godahewa et al., ) | Web | W | 145,063 | 1 | 0 | 16,537,182 |
| Extended Web Traffic | Monash (Godahewa et al., ) | Web | D | 145,063 | 1 | 0 | 370,926,091 |
| M1 Yearly | Monash (Godahewa et al., ) | Econ/Fin | Y | 106 | 1 | 0 | 3,136 |
| M1 Quarterly | Monash (Godahewa et al., ) | Econ/Fin | Q | 198 | 1 | 0 | 9,854 |
| M1 Monthly | Monash (Godahewa et al., ) | Econ/Fin | M | 617 | 1 | 0 | 44,892 |
| M3 Yearly | Monash (Godahewa et al., ) | Econ/Fin | Y | 645 | 1 | 0 | 18,319 |
| M3 Quarterly | Monash (Godahewa et al., ) | Econ/Fin | Q | 756 | 1 | 0 | 37,004 |
| M3 Monthly | Monash (Godahewa et al., ) | Econ/Fin | M | 1,428 | 1 | 0 | 141,858 |
| M3 Other | Monash (Godahewa et al., ) | Econ/Fin | Q | 174 | 1 | 0 | 11,933 |
| NN5 Daily | Monash (Godahewa et al., ) | Econ/Fin | D | 111 | 1 | 0 | 81,585 |
| NN5 Weekly | Monash (Godahewa et al., ) | Econ/Fin | W | 111 | 1 | 0 | 11,655 |
| Tourism Yearly | Monash (Godahewa et al., ) | Econ/Fin | Y | 419 | 1 | 0 | 11,198 |
| Tourism Quarterly | Monash (Godahewa et al., ) | Econ/Fin | Q | 427 | 1 | 0 | 39,128 |
| Tourism Monthly | Monash (Godahewa et al., ) | Econ/Fin | M | 366 | 1 | 0 | 100,496 |
| CIF 2016 | Monash (Godahewa et al., ) | Econ/Fin | M | 72 | 1 | 0 | 6,334 |
| Traffic Weekly | Monash (Godahewa et al., ) | Transport | W | 862 | 1 | 0 | 82,752 |
| Traffic Hourly | Monash (Godahewa et al., ) | Transport | H | 862 | 1 | 0 | 14,978,112 |
| Australian Electricity Demand | Monash (Godahewa et al., ) | Energy | 30T | 5 | 1 | 0 | 1,153,584 |
| Rideshare | Monash (Godahewa et al., ) | Transport | H | 2,304 | 1 | 0 | 859,392 |
| Sunspot | Monash (Godahewa et al., ) | Nature | D | 1 | 1 | 0 | 73,894 |
| Vehicle Trips | Monash (Godahewa et al., ) | Transport | D | 329 | 1 | 0 | 32,512 |
| Weather | Monash (Godahewa et al., ) | Climate | D | 3,010 | 1 | 0 | 42,941,700 |
| FRED MD | Monash (Godahewa et al., ) | Econ/Fin | M | 107 | 1 | 0 | 76,612 |
| Pedestrian Counts | Monash (Godahewa et al., ) | Transport | H | 66 | 1 | 0 | 3,130,762 |
| Bitcoin | Monash (Godahewa et al., ) | Econ/Fin | D | 18 | 1 | 0 | 74,824 |
| KDD Cup 2022 | LOTSA$_$Others (Woo et al., ) | Energy | 10T | 134 | 1 | 9 | 4,727,519 |
| GoDaddy | LOTSA$_$Others (Woo et al., ) | Econ/Fin | M | 3,135 | 2 | 0 | 128,535 |
| Favorita Sales | LOTSA$_$Others (Woo et al., ) | Sales | D | 111,840 | 1 | 0 | 139,179,538 |
| Favorita Transactions | LOTSA$_$Others (Woo et al., ) | Sales | D | 54 | 1 | 0 | 84,408 |
| China Air Quality | LOTSA$_$Others (Woo et al., ) | Nature | H | 437 | 6 | 0 | 5,739,234 |
| Beijing Air Quality | LOTSA$_$Others (Woo et al., ) | Nature | H | 12 | 11 | 0 | 420,768 |
| Residential Load Power | LOTSA$_$Others (Woo et al., ) | Energy | T | 271 | 3 | 0 | 145,994,559 |
| Residential PV Power | LOTSA$_$Others (Woo et al., ) | Energy | T | 233 | 3 | 0 | 125,338,950 |
| CDC Fluview ILINet | LOTSA$_$Others (Woo et al., ) | Healthcare | W | 75 | 5 | 0 | 63,903 |
| CDC Fluview WHO NREVSS | LOTSA$_$Others (Woo et al., ) | Healthcare | W | 74 | 4 | 0 | 41,760 |
| Project Tycho | LOTSA$_$Others (Woo et al., ) | Healthcare | W | 1,258 | 1 | 0 | 1,377,707 |

## Appendix F Finegrained Results

### F.1 Results with all models

In this section, we present results for all models, including those omitted from the main paper in section Section 4.1 due to space constraints. The results are displayed in the same aggregated form through Tables 15, 16, 17, 18 and 19. Furthermore, we provide non-aggregated results across all dataset, term and frequency combinations in Tables 20, 21 and 22.

### F.2 Data Leakage Effect in Foundation Models

Due to the use of different pre-training datasets by various foundation models, there is a partial data leakage issue when these models are evaluated against our GIFT-Eval. To ensure a fair comparison, we pre-trained a new series of Moirai models on our GIFT-Eval pre-training data and report the results from these models in the main paper. To further examine the impact of data leakage on foundation models, we also include the performance of the original Moirai models on our benchmark, both to facilitate replicability and to demonstrate the misleading effects that leakage can introduce. We call the original model Moirai-Leakage and use the abbreviation Moi Leak.. The results for all datasets affected by leakage are provided in  Table 23. In most cases, data leakage from training sets can boost performance on the corresponding test sets, with this effect becoming more pronounced as prediction length increases. This finding highlights the capacity of foundation models to memorize training data and underscores the critical importance of preventing data leakage when comparing time series foundation models on public benchmarks.

### F.3 Additional Qualitative Examples

In addition to the four examples shared in the main paper, we present three additional qualitative examples in Figure 3.  Figure 3(a) illustrates the forecasts of foundation models on the Bizitobs_l2c dataset (hourly, medium-term). Similar to previous observations, Chronos forecasts tend to degrade over longer time horizons. Unlike in earlier scenarios, Moirai also shows poor performance on this dataset, missing all the regular peaks and troughs. In contrast, the VisionTS model provides the forecast closest to the ground truth.  Figure 3(b) presents forecasts for a higher-frequency dataset: Electricity (15-minute intervals, long-term). Notably, Chronos excels on this dataset, showing better consistency than both Moirai and VisionTS. While Moirai performs reasonably well, it tends to predict some stationary changes (see the rightmost side of the upper plot) that are not aligned with the ground truth data. In contrast, VisionTS repeats a mistake observed in earlier datasets by predicting shifted peaks. The final plots in Figure 3(c) display forecasts from deep learning models on the same Electricity dataset (15-minute intervals, long-term). Compared to the foundation models, these deep learning models demonstrate poorer performance. Notably, the model that differs most by its forecast is DeepAR, which quickly flattens at the beginning of the prediction—a phenomenon also observed with another deep learning model, N-BEATS, in the Solar dataset example in Figure 2(b).

**Table 15: Results on GIFT-Eval with all models aggregated by domain. The best results across each row are bolded, while second best results are underlined.**
| Domain | Metric | Nv. | S.Nv. | A.Ar. | A.Th. | A.ETS | DeepAR | TFT | TiDE | N-BEATS | P.TST | iTrans. | DLin. | C.former | T.FM | Vis.TS | Chr.S | Chr.B | Chr.L | Moi.S | Moi.B | Moi.L | Best |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Econ/Fin | MAPE | 1.22 | 1.00 | $9.37e^{-1}$ | 1.01 | 1.07 | 1.23 | 1.14 | 1.15 | $8.63e^{-1}$ | $9.46e^{-1}$ | 1.02 | 1.11 | $3.43e^{1}$ | $8.41e^{-1}$ | $9.93e^{-1}$ | $8.02e^{-1}$ | $8.02e^{-1}$ | $7.99e^{-1}$ | 1.04 | 1.27 | $8.80e^{-1}$ | Chr.L |
|  | CRPS | 1.36 | 1.00 | $8.25e^{-1}$ | $8.46e^{-1}$ | $9.91e^{-1}$ | 1.41 | $8.48e^{-1}$ | 1.09 | $9.86e^{-1}$ | $8.15e^{-1}$ | $8.55e^{-1}$ | 1.13 | $1.27e^{2}$ | $7.29e^{-1}$ | 1.06 | $7.78e^{-1}$ | $7.68e^{-1}$ | $7.74e^{-1}$ | $8.53e^{-1}$ | 1.04 | $7.36e^{-1}$ | T.FM |
|  | Rank | $15.17$ | $14.83$ | $7.17$ | $7.83$ | $9.00$ | $14.33$ | $7.83$ | $17.00$ | $12.67$ | $6.67$ | $8.33$ | $17.50$ | $21.00$ | $4.83$ | $16.50$ | $7.50$ | $6.33$ | $7.00$ | $10.83$ | $15.50$ | $3.17$ | Moi.L |
| Energy | MAPE | 1.23 | 1.00 | 1.02 | 1.35 | 1.34 | 1.68 | 1.09 | 1.30 | 1.17 | 1.00 | 1.24 | 1.19 | $9.24e^{9}$ | $9.90e^{-1}$ | 1.15 | $9.11e^{-1}$ | $9.09e^{-1}$ | $9.12e^{-1}$ | $8.99e^{-1}$ | $9.87e^{-1}$ | $8.91e^{-1}$ | Moi.L |
|  | CRPS | 1.72 | 1.00 | $8.63e^{-1}$ | 2.29 | $4.31e^{23}$ | 1.37 | $6.70e^{-1}$ | $8.82e^{-1}$ | $9.80e^{-1}$ | $6.52e^{-1}$ | $8.26e^{-1}$ | $9.16e^{-1}$ | $1.16e^{1}$ | $7.04e^{-1}$ | $8.25e^{-1}$ | $6.85e^{-1}$ | $6.61e^{-1}$ | $6.59e^{-1}$ | $6.97e^{-1}$ | $6.35e^{-1}$ | $6.26e^{-1}$ | Moi.L |
|  | Rank | $18.22$ | $15.84$ | $12.28$ | $16.91$ | $15.81$ | $14.69$ | $7.28$ | $10.47$ | $15.09$ | $5.75$ | $7.06$ | $14.72$ | $14.25$ | $8.25$ | $13.09$ | $8.59$ | $7.38$ | $7.25$ | $7.00$ | $5.69$ | $5.38$ | Moi.L |
| Healthcare | MAPE | 1.21 | 1.00 | $7.83e^{-1}$ | $9.67e^{-1}$ | $8.06e^{-1}$ | $8.57e^{-1}$ | $7.90e^{-1}$ | $8.43e^{-1}$ | $7.70e^{-1}$ | $7.95e^{-1}$ | $9.02e^{-1}$ | $8.85e^{-1}$ | $4.55e^{2}$ | $7.91e^{-1}$ | $8.03e^{-1}$ | $6.68e^{-1}$ | $7.25e^{-1}$ | $6.62e^{-1}$ | 1.06 | 1.15 | $8.17e^{-1}$ | Chr.L |
|  | CRPS | 1.29 | 1.00 | $6.31e^{-1}$ | $8.28e^{-1}$ | $6.38e^{-1}$ | $8.64e^{-1}$ | $7.01e^{-1}$ | 1.16 | $8.69e^{-1}$ | $6.79e^{-1}$ | $7.65e^{-1}$ | $9.88e^{-1}$ | $7.01e^{1}$ | $7.99e^{-1}$ | $7.96e^{-1}$ | $5.75e^{-1}$ | $5.94e^{-1}$ | $5.31e^{-1}$ | $9.26e^{-1}$ | 1.07 | $6.86e^{-1}$ | Chr.L |
|  | Rank | $17.80$ | $15.00$ | $7.80$ | $12.00$ | $7.20$ | $10.40$ | $8.20$ | $14.40$ | $14.00$ | $8.60$ | $9.40$ | $16.00$ | $16.00$ | $7.80$ | $12.80$ | $5.80$ | $5.00$ | $4.20$ | $14.80$ | $15.80$ | $8.00$ | Chr.L |
| Nature | MAPE | 1.02 | 1.00 | $9.36e^{-1}$ | 5.15 | 1.21 | 1.34 | 1.35 | 1.65 | 2.05 | $9.76e^{-1}$ | 1.02 | 1.66 | 3.69 | 1.08 | 1.04 | $9.49e^{-1}$ | $8.09e^{-1}$ | $7.22e^{-1}$ | $8.64e^{-1}$ | 1.08 | $9.01e^{-1}$ | Chr.L |
|  | CRPS | 1.52 | 1.00 | $7.10e^{-1}$ | 1.03 | $7.63e^{7}$ | $7.84e^{-1}$ | $4.02e^{-1}$ | $6.49e^{-1}$ | $5.87e^{-1}$ | $4.04e^{-1}$ | $3.92e^{-1}$ | $5.66e^{-1}$ | 1.82 | $3.82e^{-1}$ | $4.87e^{-1}$ | $4.57e^{-1}$ | $4.35e^{-1}$ | $4.33e^{-1}$ | $4.00e^{-1}$ | $4.22e^{-1}$ | $3.78e^{-1}$ | Moi.L |
|  | Rank | $19.27$ | $18.60$ | $14.40$ | $16.40$ | $16.93$ | $11.87$ | $7.47$ | $13.47$ | $14.53$ | $7.27$ | $6.53$ | $14.47$ | $12.80$ | $5.73$ | $11.67$ | $9.67$ | $8.73$ | $8.20$ | $4.40$ | $4.47$ | $4.13$ | Moi.L |
| Sales | MAPE | $9.99e^{-1}$ | 1.00 | $7.72e^{-1}$ | $8.26e^{-1}$ | $9.02e^{-1}$ | $7.39e^{-1}$ | $7.57e^{-1}$ | 1.00 | $7.26e^{-1}$ | $7.51e^{-1}$ | $7.59e^{-1}$ | $8.00e^{-1}$ | 1.49 | $6.83e^{-1}$ | $8.11e^{-1}$ | $7.19e^{-1}$ | $7.01e^{-1}$ | $7.03e^{-1}$ | $6.72e^{-1}$ | $6.68e^{-1}$ | $6.71e^{-1}$ | Moi.B |
|  | CRPS | $9.35e^{-1}$ | 1.00 | $4.83e^{-1}$ | $5.03e^{-1}$ | $3.08e^{1}$ | $3.68e^{-1}$ | $3.64e^{-1}$ | $5.03e^{-1}$ | $4.26e^{-1}$ | $3.67e^{-1}$ | $3.71e^{-1}$ | $5.05e^{-1}$ | 7.66 | $3.65e^{-1}$ | $5.23e^{-1}$ | $3.85e^{-1}$ | $3.85e^{-1}$ | $3.84e^{-1}$ | $4.62e^{-1}$ | $5.16e^{-1}$ | $4.06e^{-1}$ | TFT |
|  | Rank | $19.00$ | $19.25$ | $14.25$ | $14.50$ | $17.25$ | $6.25$ | $8.00$ | $14.00$ | $11.00$ | $3.25$ | $4.75$ | $14.25$ | $20.75$ | $2.75$ | $15.50$ | $9.00$ | $7.25$ | $7.25$ | $8.50$ | $9.00$ | $5.25$ | T.FM |
| Transport | MAPE | 1.48 | 1.00 | 1.07 | 1.08 | 1.25 | $8.11e^{-1}$ | $8.22e^{-1}$ | $8.78e^{-1}$ | $7.60e^{-1}$ | $8.02e^{-1}$ | $8.27e^{-1}$ | $9.09e^{-1}$ | 2.13 | $9.28e^{-1}$ | $8.70e^{-1}$ | $8.46e^{-1}$ | $8.53e^{-1}$ | $8.47e^{-1}$ | $9.10e^{-1}$ | $8.56e^{-1}$ | $9.28e^{-1}$ | N-BEATS |
|  | CRPS | 2.29 | 1.00 | $7.90e^{-1}$ | 1.48 | $6.22e^{13}$ | $5.54e^{-1}$ | $4.85e^{-1}$ | $5.70e^{-1}$ | $6.41e^{-1}$ | $5.06e^{-1}$ | $5.00e^{-1}$ | $7.04e^{-1}$ | 2.82 | $5.78e^{-1}$ | $6.66e^{-1}$ | $6.02e^{-1}$ | $5.86e^{-1}$ | $5.85e^{-1}$ | $5.00e^{-1}$ | $4.78e^{-1}$ | $5.02e^{-1}$ | Moi.B |
|  | Rank | $20.13$ | $17.33$ | $15.87$ | $18.73$ | $18.33$ | $6.47$ | $4.80$ | $10.40$ | $12.60$ | $5.80$ | $5.87$ | $14.73$ | $10.47$ | $8.13$ | $13.67$ | $10.07$ | $8.07$ | $8.40$ | $8.33$ | $6.13$ | $6.67$ | TFT |
| Web/CloudOps | MAPE | 1.11 | 1.00 | $8.94e^{-1}$ | $8.33e^{-1}$ | 1.15 | $8.57e^{-1}$ | 1.35 | $9.58e^{-1}$ | $6.42e^{-1}$ | $6.02e^{-1}$ | $7.19e^{-1}$ | 1.68 | 3.91 | 2.37 | $8.38e^{-1}$ | 1.15 | 1.30 | 1.33 | $7.97e^{-1}$ | 1.06 | $7.91e^{-1}$ | P.TST |
|  | CRPS | 1.19 | 1.00 | $9.24e^{-1}$ | $7.38e^{-1}$ | $3.54e^{6}$ | $7.81e^{-1}$ | $6.49e^{-1}$ | $6.73e^{-1}$ | $6.62e^{-1}$ | $5.18e^{-1}$ | $5.22e^{-1}$ | $8.14e^{-1}$ | $7.28e^{-1}$ | $9.76e^{-1}$ | $7.24e^{-1}$ | $7.52e^{-1}$ | $8.11e^{-1}$ | $7.91e^{-1}$ | $7.44e^{-1}$ | $7.68e^{-1}$ | $7.42e^{-1}$ | P.TST |
|  | Rank | $16.70$ | $16.35$ | $14.50$ | $12.10$ | $17.05$ | $11.70$ | $5.95$ | $9.95$ | $10.30$ | $3.95$ | $4.50$ | $12.55$ | $11.35$ | $13.90$ | $10.65$ | $10.40$ | $11.35$ | $11.65$ | $8.50$ | $9.25$ | $8.35$ | P.TST |

**Table 16: Results on GIFT-Eval with all models aggregated by term length. The best results across each row are bolded, while second best results are underlined.**
| Pred. Len. | Metric | Nv. | S.Nv. | A.Ar. | A.Th. | A.ETS | DeepAR | TFT | TiDE | N-BEATS | P.TST | iTrans. | DLin. | C.former | T.FM | Vis.TS | Chr.S | Chr.B | Chr.L | Moi.S | Moi.B | Moi.L | Best |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Long | MAPE | 1.35 | 1.00 | 1.04 | 3.15 | 1.28 | 1.70 | 1.16 | 1.28 | 1.11 | $9.62e^{-1}$ | $9.96e^{-1}$ | 1.56 | $8.16e^{9}$ | 1.80 | 1.03 | 1.01 | $9.98e^{-1}$ | 1.00 | $9.19e^{-1}$ | 1.09 | 1.01 | Moi.S |
|  | CRPS | 2.11 | 1.00 | $8.43e^{-1}$ | 2.06 | $6.57e^{23}$ | $9.48e^{-1}$ | $4.53e^{-1}$ | $5.25e^{-1}$ | $6.64e^{-1}$ | $4.37e^{-1}$ | $4.55e^{-1}$ | $6.73e^{-1}$ | $5.07e^{-1}$ | $6.54e^{-1}$ | $5.46e^{-1}$ | $6.46e^{-1}$ | $6.20e^{-1}$ | $6.21e^{-1}$ | $5.61e^{-1}$ | $5.01e^{-1}$ | $5.39e^{-1}$ | P.TST |
|  | Rank | $19.62$ | $16.86$ | $15.10$ | $17.38$ | $18.67$ | $12.71$ | $4.90$ | $8.76$ | $11.95$ | $4.43$ | $5.19$ | $13.24$ | $8.29$ | $11.29$ | $9.57$ | $11.86$ | $10.71$ | $11.05$ | $7.24$ | $5.43$ | $6.76$ | P.TST |
| Medium | MAPE | 1.30 | 1.00 | $9.93e^{-1}$ | 1.87 | 1.59 | 1.07 | 1.26 | 1.19 | 1.15 | $9.03e^{-1}$ | 1.02 | 1.39 | $2.51e^{9}$ | 1.52 | 1.02 | 1.23 | 1.38 | 1.33 | $9.31e^{-1}$ | 1.11 | $9.60e^{-1}$ | P.TST |
|  | CRPS | 2.08 | 1.00 | $8.63e^{-1}$ | 1.96 | $8.31e^{6}$ | $8.27e^{-1}$ | $5.29e^{-1}$ | $6.11e^{-1}$ | $7.60e^{-1}$ | $5.19e^{-1}$ | $5.29e^{-1}$ | $7.66e^{-1}$ | $6.61e^{-1}$ | $8.14e^{-1}$ | $6.69e^{-1}$ | $7.38e^{-1}$ | $7.77e^{-1}$ | $7.51e^{-1}$ | $6.61e^{-1}$ | $6.37e^{-1}$ | $6.25e^{-1}$ | P.TST |
|  | Rank | $19.29$ | $16.14$ | $14.57$ | $17.43$ | $18.33$ | $10.52$ | $4.81$ | $9.52$ | $12.95$ | $4.00$ | $4.62$ | $13.86$ | $8.57$ | $10.81$ | $10.95$ | $11.48$ | $11.57$ | $10.90$ | $7.52$ | $6.38$ | $6.76$ | P.TST |
| Short | MAPE | 1.10 | 1.00 | $9.18e^{-1}$ | 1.13 | 1.03 | 1.08 | 1.05 | 1.12 | 1.03 | $8.04e^{-1}$ | $9.68e^{-1}$ | 1.14 | $1.30e^{9}$ | $9.29e^{-1}$ | $9.47e^{-1}$ | $7.81e^{-1}$ | $7.50e^{-1}$ | $7.52e^{-1}$ | $8.49e^{-1}$ | $9.41e^{-1}$ | $7.73e^{-1}$ | Chr.B |
|  | CRPS | 1.20 | 1.00 | $7.78e^{-1}$ | $9.56e^{-1}$ | $1.29e^{6}$ | 1.03 | $6.76e^{-1}$ | $9.17e^{-1}$ | $8.20e^{-1}$ | $6.28e^{-1}$ | $7.28e^{-1}$ | $8.76e^{-1}$ | $2.86e^{1}$ | $6.45e^{-1}$ | $8.20e^{-1}$ | $5.97e^{-1}$ | $5.90e^{-1}$ | $5.86e^{-1}$ | $6.66e^{-1}$ | $7.08e^{-1}$ | $6.02e^{-1}$ | Chr.L |
|  | Rank | $17.22$ | $16.75$ | $11.87$ | $13.53$ | $13.78$ | $11.89$ | $8.22$ | $13.42$ | $13.87$ | $6.87$ | $7.49$ | $15.15$ | $17.58$ | $6.64$ | $14.65$ | $7.25$ | $6.16$ | $6.27$ | $8.15$ | $8.84$ | $5.40$ | Moi.L |

**Table 17: Results on GIFT-Eval with all models aggregated by frequency. The best results across each row are bolded, while second best results are underlined.**
| Freq. | Metric | Nv. | S.Nv. | A.Ar. | A.Th. | A.ETS | DeepAR | TFT | TiDE | N-BEATS | P.TST | iTrans. | DLin. | C.former | T.FM | Vis.TS | Chr.S | Chr.B | Chr.L | Moi.S | Moi.B | Moi.L | Best |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| 10S | CRPS | 1.54 | 1.00 | 1.00 | $4.50e^{-1}$ | 2.99 | 1.03 | 1.05 | $9.85e^{-1}$ | $8.21e^{-1}$ | $7.32e^{-1}$ | $6.91e^{-1}$ | 1.13 | $9.63e^{-1}$ | 1.82 | $9.34e^{-1}$ | 1.10 | 1.27 | 1.19 | 1.31 | 1.40 | 1.24 | A.Th. |
|  | MAPE | 1.97 | 1.00 | 1.00 | $6.49e^{-1}$ | 1.77 | 1.83 | 3.24 | 1.67 | $9.97e^{-1}$ | 1.06 | 1.31 | 3.46 | $1.08e^{1}$ | 6.23 | 1.10 | 2.42 | 2.95 | 3.02 | 1.65 | 2.34 | 1.55 | A.Th. |
|  | Rank | $16.33$ | $9.50$ | $8.50$ | $1.00$ | $19.50$ | $11.17$ | $7.67$ | $10.17$ | $6.67$ | $5.00$ | $2.50$ | $11.67$ | $11.50$ | $19.33$ | $10.17$ | $10.83$ | $12.33$ | $11.67$ | $14.83$ | $16.50$ | $14.17$ | A.Th. |
| 5T | CRPS | 1.40 | 1.00 | 1.00 | 1.05 | 1.05 | $7.97e^{-1}$ | $5.57e^{-1}$ | $6.48e^{-1}$ | $7.26e^{-1}$ | $5.43e^{-1}$ | $5.43e^{-1}$ | $8.33e^{-1}$ | 1.04 | $7.30e^{-1}$ | $7.34e^{-1}$ | $7.52e^{-1}$ | $7.49e^{-1}$ | $7.55e^{-1}$ | $5.36e^{-1}$ | $5.41e^{-1}$ | $5.32e^{-1}$ | Moi.L |
|  | MAPE | $8.44e^{-1}$ | 1.00 | 1.00 | $9.64e^{-1}$ | 1.01 | $6.72e^{-1}$ | $6.43e^{-1}$ | $7.95e^{-1}$ | $6.34e^{-1}$ | $5.87e^{-1}$ | $6.60e^{-1}$ | $9.70e^{-1}$ | 1.27 | $8.20e^{-1}$ | $8.76e^{-1}$ | $7.80e^{-1}$ | $7.63e^{-1}$ | $7.72e^{-1}$ | $5.30e^{-1}$ | $6.00e^{-1}$ | $5.52e^{-1}$ | Moi.S |
|  | Rank | $17.17$ | $17.75$ | $16.25$ | $16.67$ | $15.50$ | $13.25$ | $4.92$ | $10.08$ | $12.42$ | $4.75$ | $5.50$ | $14.50$ | $11.42$ | $11.92$ | $12.17$ | $11.17$ | $11.25$ | $11.92$ | $4.42$ | $4.75$ | $3.25$ | Moi.L |
| 10T | CRPS | 2.23 | 1.00 | 1.00 | 3.67 | 2.15 | $5.92e^{-1}$ | $4.12e^{-1}$ | $6.55e^{-1}$ | $7.81e^{-1}$ | $5.58e^{-1}$ | $5.91e^{-1}$ | $6.52e^{-1}$ | 2.10 | $5.70e^{-1}$ | $5.01e^{-1}$ | $6.43e^{-1}$ | $5.55e^{-1}$ | $5.51e^{-1}$ | $5.92e^{-1}$ | $4.33e^{-1}$ | $4.95e^{-1}$ | TFT |
|  | MAPE | $7.59e^{-1}$ | 1.00 | 1.00 | 2.63 | 1.66 | $6.45e^{-1}$ | 1.20 | 1.16 | 1.26 | $8.81e^{-1}$ | $7.80e^{-1}$ | 1.30 | $1.99e^{1}$ | 1.02 | 1.08 | 1.51 | 1.27 | 1.03 | $5.83e^{-1}$ | $9.12e^{-1}$ | 1.05 | Moi.S |
|  | Rank | $19.33$ | $16.50$ | $15.50$ | $19.67$ | $16.67$ | $10.67$ | $4.33$ | $11.83$ | $13.17$ | $7.33$ | $6.83$ | $11.83$ | $11.50$ | $8.33$ | $7.50$ | $12.17$ | $8.83$ | $8.00$ | $8.50$ | $5.00$ | $7.50$ | TFT |
| 15T | CRPS | 2.42 | 1.00 | $9.58e^{-1}$ | 1.76 | $7.78e^{13}$ | 1.68 | $7.32e^{-1}$ | $8.07e^{-1}$ | $9.89e^{-1}$ | $6.63e^{-1}$ | $6.58e^{-1}$ | $9.37e^{-1}$ | 3.11 | $7.93e^{-1}$ | $8.78e^{-1}$ | $7.93e^{-1}$ | $7.67e^{-1}$ | $7.64e^{-1}$ | $7.93e^{-1}$ | $7.07e^{-1}$ | $6.76e^{-1}$ | iTrans. |
|  | MAPE | 1.44 | 1.00 | 1.10 | $9.89e^{-1}$ | 1.27 | 1.96 | 1.06 | $9.79e^{-1}$ | $9.64e^{-1}$ | $8.74e^{-1}$ | $9.00e^{-1}$ | 1.03 | 6.32 | 1.02 | $9.82e^{-1}$ | $9.61e^{-1}$ | $9.52e^{-1}$ | $9.44e^{-1}$ | 1.10 | 1.01 | $9.40e^{-1}$ | P.TST |
|  | Rank | $19.75$ | $15.17$ | $13.92$ | $17.00$ | $18.17$ | $14.50$ | $6.58$ | $10.08$ | $15.00$ | $3.83$ | $3.50$ | $14.08$ | $13.25$ | $8.25$ | $12.83$ | $9.75$ | $8.50$ | $8.17$ | $8.67$ | $5.58$ | $4.42$ | iTrans. |
| H | CRPS | 1.84 | 1.00 | $7.77e^{-1}$ | 1.85 | $4.45e^{23}$ | $8.93e^{-1}$ | $4.87e^{-1}$ | $5.70e^{-1}$ | $6.65e^{-1}$ | $4.62e^{-1}$ | $4.76e^{-1}$ | $6.81e^{-1}$ | 5.28 | $5.10e^{-1}$ | $6.11e^{-1}$ | $5.14e^{-1}$ | $5.06e^{-1}$ | $5.09e^{-1}$ | $4.78e^{-1}$ | $4.63e^{-1}$ | $4.99e^{-1}$ | P.TST |
|  | MAPE | 1.33 | 1.00 | $9.54e^{-1}$ | 3.08 | 1.33 | 1.18 | 1.11 | 1.25 | 1.34 | $9.28e^{-1}$ | $9.60e^{-1}$ | 1.35 | $5.68e^{9}$ | $9.82e^{-1}$ | $9.89e^{-1}$ | $7.33e^{-1}$ | $7.39e^{-1}$ | $7.48e^{-1}$ | $8.13e^{-1}$ | $9.07e^{-1}$ | $8.51e^{-1}$ | Chr.S |
|  | Rank | $19.65$ | $17.65$ | $15.48$ | $18.87$ | $18.77$ | $10.87$ | $6.65$ | $10.58$ | $13.42$ | $4.97$ | $6.13$ | $14.13$ | $11.00$ | $8.35$ | $11.94$ | $8.58$ | $7.81$ | $8.00$ | $6.32$ | $5.29$ | $6.55$ | P.TST |
| D | CRPS | $7.98e^{-1}$ | 1.00 | $4.99e^{-1}$ | $5.75e^{-1}$ | 9.35 | $5.94e^{-1}$ | $4.14e^{-1}$ | $7.48e^{-1}$ | $5.71e^{-1}$ | $4.37e^{-1}$ | $4.95e^{-1}$ | $6.15e^{-1}$ | $2.69e^{1}$ | $4.92e^{-1}$ | $5.56e^{-1}$ | $4.37e^{-1}$ | $4.21e^{-1}$ | $4.21e^{-1}$ | $4.13e^{-1}$ | $4.63e^{-1}$ | $3.92e^{-1}$ | Moi.L |
|  | MAPE | 1.00 | 1.00 | $8.53e^{-1}$ | $9.29e^{-1}$ | $9.33e^{-1}$ | $8.01e^{-1}$ | $7.62e^{-1}$ | $9.97e^{-1}$ | $7.96e^{-1}$ | $7.52e^{-1}$ | $8.18e^{-1}$ | $8.98e^{-1}$ | $1.55e^{2}$ | $7.80e^{-1}$ | $8.95e^{-1}$ | $7.37e^{-1}$ | $6.86e^{-1}$ | $6.96e^{-1}$ | $7.09e^{-1}$ | $7.45e^{-1}$ | $6.63e^{-1}$ | Moi.L |
|  | Rank | $17.60$ | $19.00$ | $11.00$ | $14.33$ | $14.33$ | $11.07$ | $6.67$ | $13.53$ | $14.40$ | $7.20$ | $8.47$ | $15.13$ | $18.40$ | $5.60$ | $14.80$ | $8.07$ | $6.73$ | $6.67$ | $6.20$ | $7.47$ | $4.33$ | Moi.L |
| W | CRPS | $8.76e^{-1}$ | 1.00 | $7.49e^{-1}$ | $8.09e^{-1}$ | $7.89e^{-1}$ | 1.35 | $7.79e^{-1}$ | 1.22 | 1.08 | $6.98e^{-1}$ | 1.33 | $9.75e^{-1}$ | $4.98e^{1}$ | $6.31e^{-1}$ | $9.99e^{-1}$ | $5.55e^{-1}$ | $5.62e^{-1}$ | $5.52e^{-1}$ | $7.58e^{-1}$ | $7.48e^{-1}$ | $6.41e^{-1}$ | Chr.L |
|  | MAPE | 1.00 | 1.00 | $9.76e^{-1}$ | 1.08 | $9.72e^{-1}$ | 1.86 | 1.04 | 1.77 | 1.45 | $9.07e^{-1}$ | 1.90 | 1.13 | 7.41 | $8.62e^{-1}$ | 1.14 | $7.08e^{-1}$ | $7.29e^{-1}$ | $7.07e^{-1}$ | $9.81e^{-1}$ | $9.50e^{-1}$ | $8.96e^{-1}$ | Chr.L |
|  | Rank | $13.62$ | $16.50$ | $9.88$ | $12.12$ | $10.38$ | $12.75$ | $11.25$ | $13.12$ | $15.25$ | $8.00$ | $12.38$ | $16.88$ | $20.38$ | $4.88$ | $16.00$ | $5.25$ | $4.50$ | $4.38$ | $8.62$ | $9.75$ | $5.12$ | Chr.L |
| M | CRPS | 1.58 | 1.00 | $7.66e^{-1}$ | $8.82e^{-1}$ | $7.72e^{-1}$ | 1.10 | $8.75e^{-1}$ | 1.24 | 1.01 | $8.48e^{-1}$ | $8.21e^{-1}$ | 1.20 | $6.80e^{1}$ | $7.38e^{-1}$ | 1.04 | $8.31e^{-1}$ | $8.61e^{-1}$ | $8.15e^{-1}$ | 1.22 | 1.50 | $9.17e^{-1}$ | T.FM |
|  | MAPE | 1.30 | 1.00 | $7.90e^{-1}$ | $9.20e^{-1}$ | $8.25e^{-1}$ | 1.18 | 1.05 | 1.04 | $9.41e^{-1}$ | $9.90e^{-1}$ | 1.01 | 1.08 | $1.05e^{1}$ | $8.63e^{-1}$ | $9.38e^{-1}$ | $8.62e^{-1}$ | $8.95e^{-1}$ | $8.47e^{-1}$ | 1.38 | 1.89 | $9.97e^{-1}$ | A.Ar. |
|  | Rank | $19.20$ | $13.40$ | $6.20$ | $8.40$ | $5.40$ | $11.80$ | $8.20$ | $15.40$ | $11.40$ | $7.60$ | $5.60$ | $15.80$ | $13.00$ | $3.60$ | $14.80$ | $8.60$ | $8.60$ | $8.20$ | $17.00$ | $19.20$ | $9.60$ | T.FM |
| Q | CRPS | $9.51e^{-1}$ | 1.00 | $8.23e^{-1}$ | $7.97e^{-1}$ | $7.98e^{-1}$ | $8.41e^{-1}$ | $8.37e^{-1}$ | 1.02 | $9.72e^{-1}$ | $8.35e^{-1}$ | $7.97e^{-1}$ | 1.11 | $1.20e^{2}$ | $8.53e^{-1}$ | 1.05 | $8.46e^{-1}$ | $8.40e^{-1}$ | $8.40e^{-1}$ | $9.32e^{-1}$ | 1.13 | $7.88e^{-1}$ | Moi.L |
|  | MAPE | $9.30e^{-1}$ | 1.00 | $8.59e^{-1}$ | $8.38e^{-1}$ | $8.59e^{-1}$ | $9.44e^{-1}$ | $9.37e^{-1}$ | 1.13 | $8.38e^{-1}$ | $9.37e^{-1}$ | $9.08e^{-1}$ | $9.86e^{-1}$ | 9.93 | $8.80e^{-1}$ | $9.37e^{-1}$ | $8.24e^{-1}$ | $8.10e^{-1}$ | $8.10e^{-1}$ | 1.04 | 1.43 | $8.87e^{-1}$ | Chr.B |
|  | Rank | $14.00$ | $16.00$ | $5.00$ | $2.00$ | $4.00$ | $10.00$ | $7.00$ | $17.00$ | $15.00$ | $6.00$ | $3.00$ | $19.00$ | $21.00$ | $12.00$ | $18.00$ | $11.00$ | $8.00$ | $9.00$ | $13.00$ | $20.00$ | $1.00$ | Moi.L |
| A | CRPS | $9.93e^{-1}$ | 1.00 | $9.42e^{-1}$ | $8.33e^{-1}$ | $8.04e^{-1}$ | $8.19e^{-1}$ | $7.97e^{-1}$ | 1.12 | $9.71e^{-1}$ | $8.48e^{-1}$ | $8.48e^{-1}$ | 1.22 | $1.03e^{2}$ | $8.48e^{-1}$ | 1.15 | 1.01 | $9.78e^{-1}$ | $9.78e^{-1}$ | $8.26e^{-1}$ | $9.42e^{-1}$ | $7.75e^{-1}$ | Moi.L |
|  | MAPE | 1.00 | 1.00 | 1.02 | $9.37e^{-1}$ | $9.37e^{-1}$ | 1.02 | $9.26e^{-1}$ | 1.21 | $9.03e^{-1}$ | 1.01 | 1.03 | 1.06 | 6.86 | $9.71e^{-1}$ | 1.09 | 1.00 | $9.77e^{-1}$ | $9.77e^{-1}$ | $9.77e^{-1}$ | 1.21 | $9.54e^{-1}$ | N-BEATS |
|  | Rank | $15.00$ | $16.00$ | $10.00$ | $6.00$ | $3.00$ | $4.00$ | $2.00$ | $18.00$ | $12.00$ | $8.00$ | $7.00$ | $20.00$ | $21.00$ | $9.00$ | $19.00$ | $17.00$ | $13.00$ | $14.00$ | $5.00$ | $11.00$ | $1.00$ | Moi.L |

**Table 18: Results on GIFT-Eval aggregated by number of variates. The best results across each row are bolded, while second best results are underlined.**
| Num. Var. | Metric | Nv. | S.Nv. | A.Ar. | A.Th. | A.ETS | DeepAR | TFT | TiDE | N-BEATS | P.TST | iTrans. | DLin. | C.former | T.FM | Vis.TS | Chr.S | Chr.B | Chr.L | Moi.S | Moi.B | Moi.L | Best |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Multi_v | MAPE | 1.21 | 1.00 | $9.80e^{-1}$ | 2.41 | 1.32 | 1.54 | 1.40 | 1.44 | 1.43 | $8.75e^{-1}$ | 1.15 | 1.62 | 2.65 | 1.71 | 1.04 | 1.08 | 1.11 | 1.10 | $8.84e^{-1}$ | 1.09 | $9.32e^{-1}$ | P.TST |
|  | CRPS | 1.40 | 1.00 | $8.71e^{-1}$ | 1.19 | $2.82e^{7}$ | 1.18 | $6.09e^{-1}$ | $8.20e^{-1}$ | $7.35e^{-1}$ | $5.27e^{-1}$ | $6.22e^{-1}$ | $7.82e^{-1}$ | $9.19e^{-1}$ | $7.50e^{-1}$ | $7.01e^{-1}$ | $6.65e^{-1}$ | $6.83e^{-1}$ | $6.73e^{-1}$ | $6.26e^{-1}$ | $6.28e^{-1}$ | $6.27e^{-1}$ | P.TST |
|  | Rank | $17.70$ | $16.51$ | $13.91$ | $14.98$ | $17.02$ | $14.00$ | $6.91$ | $11.40$ | $12.58$ | $5.02$ | $5.19$ | $13.49$ | $12.91$ | $10.07$ | $11.53$ | $9.51$ | $9.44$ | $9.56$ | $6.33$ | $6.28$ | $6.67$ | P.TST |
| Uni_v | MAPE | 1.19 | 1.00 | $9.45e^{-1}$ | 1.18 | 1.12 | $9.49e^{-1}$ | $8.92e^{-1}$ | $9.60e^{-1}$ | $7.96e^{-1}$ | $8.47e^{-1}$ | $8.53e^{-1}$ | 1.02 | $5.47e^{9}$ | $8.72e^{-1}$ | $9.34e^{-1}$ | $8.11e^{-1}$ | $8.06e^{-1}$ | $7.99e^{-1}$ | $8.80e^{-1}$ | $9.47e^{-1}$ | $8.10e^{-1}$ | N-BEATS |
|  | CRPS | 1.74 | 1.00 | $7.62e^{-1}$ | 1.59 | $2.55e^{23}$ | $8.02e^{-1}$ | $5.85e^{-1}$ | $7.23e^{-1}$ | $8.03e^{-1}$ | $5.92e^{-1}$ | $6.28e^{-1}$ | $8.29e^{-1}$ | $2.88e^{1}$ | $6.30e^{-1}$ | $7.49e^{-1}$ | $6.18e^{-1}$ | $6.00e^{-1}$ | $5.95e^{-1}$ | $6.55e^{-1}$ | $6.63e^{-1}$ | $5.66e^{-1}$ | Moi.L |
|  | Rank | $18.57$ | $16.74$ | $12.56$ | $15.39$ | $14.87$ | $10.00$ | $6.65$ | $11.70$ | $13.80$ | $6.28$ | $7.31$ | $15.22$ | $14.19$ | $7.33$ | $13.72$ | $8.89$ | $7.43$ | $7.31$ | $9.00$ | $8.59$ | $5.44$ | Moi.L |

**Table 19: Results on GIFT-Eval with all models aggregated by all datasets. The best results across each row are bolded, while second best results are underlined.**
| Metric | Nv. | S.Nv. | A.Ar. | A.Th. | A.ETS | DeepAR | TFT | TiDE | N-BEATS | P.TST | iTrans. | DLin. | C.former | T.FM | Vis.TS | Chr.S | Chr.B | Chr.L | Moi.S | Moi.B | Moi.L | Best |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| MAPE | 1.20 | 1.00 | $9.61e^{-1}$ | 1.73 | 1.20 | 1.21 | 1.12 | 1.17 | 1.08 | $8.60e^{-1}$ | $9.85e^{-1}$ | 1.28 | $3.05e^{9}$ | 1.25 | $9.82e^{-1}$ | $9.28e^{-1}$ | $9.40e^{-1}$ | $9.30e^{-1}$ | $8.82e^{-1}$ | 1.01 | $8.64e^{-1}$ | P.TST |
| CRPS | 1.59 | 1.00 | $8.11e^{-1}$ | 1.41 | $1.42e^{23}$ | $9.69e^{-1}$ | $5.96e^{-1}$ | $7.66e^{-1}$ | $7.73e^{-1}$ | $5.63e^{-1}$ | $6.26e^{-1}$ | $8.08e^{-1}$ | $1.65e^{1}$ | $6.83e^{-1}$ | $7.28e^{-1}$ | $6.38e^{-1}$ | $6.37e^{-1}$ | $6.29e^{-1}$ | $6.42e^{-1}$ | $6.48e^{-1}$ | $5.93e^{-1}$ | P.TST |
| Rank | $18.19$ | $16.64$ | $13.15$ | $15.21$ | $15.82$ | $11.77$ | $6.76$ | $11.57$ | $13.26$ | $5.72$ | $6.37$ | $14.45$ | $13.62$ | $8.55$ | $12.75$ | $9.16$ | $8.32$ | $8.31$ | $7.81$ | $7.57$ | $5.99$ | P.TST |

**Table 20: Results on all dataset configs for GIFT-Eval | Table 1/3. The best results across each row are bolded, while second best results are underlined.**
| Dataset,term,frequency | Metric | Nv. | S.Nv. | A.Ar. | A.Th. | A.ETS | DeepAR | TFT | TiDE | N-BEATS | P.TST | iTrans. | DLin. | C.former | T.FM | Vis.TS | Chr.S | Chr.B | Chr.L | Moi.S | Moi.B | Moi.L | Best |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| bitbrains_fast_storage, long, 5T | MAPE | $2.79e^{-1}$ | 1.00 | 1.00 | $5.79e^{-1}$ | 1.00 | $1.90e^{-1}$ | 1.04 | $5.12e^{-1}$ | $2.45e^{-1}$ | $4.10e^{-1}$ | $5.79e^{-1}$ | 1.34 | 1.14 | $6.36e^{-1}$ | $9.23e^{-1}$ | $2.83e^{-1}$ | $2.29e^{-1}$ | $2.49e^{-1}$ | $4.15e^{-1}$ | $4.97e^{-1}$ | $5.37e^{-1}$ | DeepAR |
| bitbrains_fast_storage, long, 5T | CRPS | 1.64 | 1.00 | 1.00 | 1.05 | 1.00 | $7.83e^{-1}$ | $5.69e^{-1}$ | $5.47e^{-1}$ | $6.13e^{-1}$ | $5.19e^{-1}$ | $5.35e^{-1}$ | 1.03 | $8.22e^{-1}$ | $6.25e^{-1}$ | $7.50e^{-1}$ | $5.50e^{-1}$ | $5.51e^{-1}$ | $5.60e^{-1}$ | $5.11e^{-1}$ | $4.92e^{-1}$ | $5.17e^{-1}$ | Moi.B |
| bitbrains_fast_storage, long, 5T | Rank | $21.00$ | $18.00$ | $16.00$ | $20.00$ | $17.00$ | $14.00$ | $10.00$ | $6.00$ | $11.00$ | $4.00$ | $5.00$ | $19.00$ | $15.00$ | $12.00$ | $13.00$ | $7.00$ | $8.00$ | $9.00$ | $2.00$ | $1.00$ | $3.00$ | Moi.B |
| bitbrains_fast_storage, medium, 5T | MAPE | $7.14e^{-1}$ | 1.00 | 1.00 | $8.47e^{-1}$ | 1.00 | $1.92e^{-1}$ | $3.85e^{-1}$ | $7.07e^{-1}$ | $2.53e^{-1}$ | $5.48e^{-1}$ | $6.93e^{-1}$ | $8.27e^{-1}$ | $9.74e^{-1}$ | $6.52e^{-1}$ | $9.43e^{-1}$ | $6.50e^{-1}$ | $5.74e^{-1}$ | $5.75e^{-1}$ | $5.77e^{-1}$ | $6.53e^{-1}$ | $5.79e^{-1}$ | DeepAR |
| bitbrains_fast_storage, medium, 5T | CRPS | 1.61 | 1.00 | 1.00 | 1.14 | 1.00 | $7.80e^{-1}$ | $4.80e^{-1}$ | $6.02e^{-1}$ | $6.62e^{-1}$ | $5.06e^{-1}$ | $5.57e^{-1}$ | $7.61e^{-1}$ | $8.03e^{-1}$ | $5.87e^{-1}$ | $7.06e^{-1}$ | $6.32e^{-1}$ | $6.33e^{-1}$ | $6.28e^{-1}$ | $4.95e^{-1}$ | $5.07e^{-1}$ | $4.87e^{-1}$ | TFT |
| bitbrains_fast_storage, medium, 5T | Rank | $21.00$ | $19.00$ | $17.00$ | $20.00$ | $18.00$ | $15.00$ | $1.00$ | $8.00$ | $12.00$ | $4.00$ | $6.00$ | $14.00$ | $16.00$ | $7.00$ | $13.00$ | $10.00$ | $11.00$ | $9.00$ | $3.00$ | $5.00$ | $2.00$ | TFT |
| bitbrains_fast_storage, short, 5T | MAPE | $4.10e^{-1}$ | 1.00 | 1.00 | $5.18e^{-1}$ | 1.00 | $2.07e^{-1}$ | $4.01e^{-1}$ | $3.36e^{-1}$ | $4.32e^{-1}$ | $3.64e^{-1}$ | $3.04e^{-1}$ | $4.57e^{-1}$ | $3.66e^{-1}$ | $3.27e^{-1}$ | $5.18e^{-1}$ | $3.63e^{-1}$ | $3.56e^{-1}$ | $3.56e^{-1}$ | $3.46e^{-1}$ | $3.86e^{-1}$ | $2.78e^{-1}$ | DeepAR |
| bitbrains_fast_storage, short, 5T | CRPS | $7.38e^{-1}$ | 1.00 | 1.00 | $6.04e^{-1}$ | 1.00 | $4.07e^{-1}$ | $3.73e^{-1}$ | $4.40e^{-1}$ | $5.14e^{-1}$ | $3.89e^{-1}$ | $3.79e^{-1}$ | $4.77e^{-1}$ | $5.49e^{-1}$ | $3.93e^{-1}$ | $5.12e^{-1}$ | $3.69e^{-1}$ | $3.83e^{-1}$ | $3.84e^{-1}$ | $3.61e^{-1}$ | $3.69e^{-1}$ | $3.45e^{-1}$ | Moi.L |
| bitbrains_fast_storage, short, 5T | Rank | $18.00$ | $21.00$ | $19.00$ | $17.00$ | $20.00$ | $11.00$ | $5.00$ | $12.00$ | $15.00$ | $9.00$ | $6.00$ | $13.00$ | $16.00$ | $10.00$ | $14.00$ | $3.00$ | $7.00$ | $8.00$ | $2.00$ | $4.00$ | $1.00$ | Moi.L |
| bitbrains_fast_storage, short, H | MAPE | 1.20 | 1.00 | $8.59e^{-1}$ | 1.32 | 1.25 | $2.00e^{-1}$ | 1.05 | $8.86e^{-1}$ | $8.28e^{-1}$ | $6.82e^{-1}$ | $6.14e^{-1}$ | 1.17 | 1.88 | 1.15 | 1.16 | $7.69e^{-1}$ | $6.97e^{-1}$ | $7.69e^{-1}$ | $4.81e^{-1}$ | $7.67e^{-1}$ | $7.82e^{-1}$ | DeepAR |
| bitbrains_fast_storage, short, H | CRPS | 1.12 | 1.00 | $7.81e^{-1}$ | 1.06 | $7.08e^{7}$ | $7.20e^{-1}$ | $5.51e^{-1}$ | $6.02e^{-1}$ | $7.52e^{-1}$ | $5.08e^{-1}$ | $5.13e^{-1}$ | $7.44e^{-1}$ | $8.50e^{-1}$ | $6.47e^{-1}$ | 1.44 | $5.69e^{-1}$ | $5.76e^{-1}$ | $5.86e^{-1}$ | $5.36e^{-1}$ | $5.98e^{-1}$ | $6.63e^{-1}$ | P.TST |
| bitbrains_fast_storage, short, H | Rank | $19.00$ | $17.00$ | $15.00$ | $18.00$ | $21.00$ | $12.00$ | $4.00$ | $9.00$ | $14.00$ | $1.00$ | $2.00$ | $13.00$ | $16.00$ | $10.00$ | $20.00$ | $5.00$ | $6.00$ | $7.00$ | $3.00$ | $8.00$ | $11.00$ | P.TST |
| bitbrains_rnd, long, 5T | MAPE | 1.79 | 1.00 | 1.00 | 2.47 | 1.00 | $4.65e^{-1}$ | $3.07e^{-1}$ | 1.46 | 1.63 | $4.03e^{-1}$ | $7.67e^{-1}$ | 2.21 | 1.96 | 1.00 | 1.25 | 1.61 | 1.58 | 1.62 | $1.47e^{-1}$ | $8.71e^{-1}$ | $4.37e^{-1}$ | Moi.S |
| bitbrains_rnd, long, 5T | CRPS | 1.83 | 1.00 | 1.00 | 1.24 | 1.00 | $5.21e^{-1}$ | $4.84e^{-1}$ | $6.20e^{-1}$ | $8.22e^{-1}$ | $5.15e^{-1}$ | $5.41e^{-1}$ | 1.01 | $8.84e^{-1}$ | $6.25e^{-1}$ | $7.53e^{-1}$ | $8.37e^{-1}$ | $8.14e^{-1}$ | $8.29e^{-1}$ | $4.36e^{-1}$ | $5.36e^{-1}$ | $4.57e^{-1}$ | Moi.S |
| bitbrains_rnd, long, 5T | Rank | $21.00$ | $18.00$ | $16.00$ | $20.00$ | $17.00$ | $5.00$ | $3.00$ | $8.00$ | $12.00$ | $4.00$ | $7.00$ | $19.00$ | $15.00$ | $9.00$ | $10.00$ | $14.00$ | $11.00$ | $13.00$ | $1.00$ | $6.00$ | $2.00$ | Moi.S |
| bitbrains_rnd, medium, 5T | MAPE | $2.09e^{-1}$ | 1.00 | 1.00 | $9.16e^{-1}$ | 1.00 | $1.43e^{-1}$ | $3.89e^{-1}$ | $6.84e^{-1}$ | $2.93e^{-1}$ | $1.83e^{-1}$ | $4.65e^{-1}$ | 1.31 | 2.65 | $7.38e^{-1}$ | 1.21 | $1.08e^{-1}$ | $1.60e^{-1}$ | $1.51e^{-1}$ | $9.05e^{-2}$ | $1.17e^{-1}$ | $1.35e^{-1}$ | Moi.S |
| bitbrains_rnd, medium, 5T | CRPS | 1.60 | 1.00 | 1.00 | 1.17 | 1.00 | $5.13e^{-1}$ | $4.98e^{-1}$ | $5.76e^{-1}$ | $5.55e^{-1}$ | $4.92e^{-1}$ | $5.25e^{-1}$ | $8.02e^{-1}$ | $9.29e^{-1}$ | $5.83e^{-1}$ | $7.21e^{-1}$ | $4.88e^{-1}$ | $5.11e^{-1}$ | $5.05e^{-1}$ | $4.71e^{-1}$ | $4.70e^{-1}$ | $4.37e^{-1}$ | Moi.L |
| bitbrains_rnd, medium, 5T | Rank | $21.00$ | $19.00$ | $17.00$ | $20.00$ | $18.00$ | $9.00$ | $6.00$ | $12.00$ | $11.00$ | $5.00$ | $10.00$ | $15.00$ | $16.00$ | $13.00$ | $14.00$ | $4.00$ | $8.00$ | $7.00$ | $3.00$ | $2.00$ | $1.00$ | Moi.L |
| bitbrains_rnd, short, 5T | MAPE | $5.38e^{-1}$ | 1.00 | 1.00 | $7.74e^{-1}$ | 1.00 | $2.95e^{-1}$ | $5.74e^{-1}$ | $7.11e^{-1}$ | $2.17e^{-1}$ | $4.39e^{-1}$ | $4.23e^{-1}$ | $5.67e^{-1}$ | 1.25 | $4.43e^{-1}$ | $7.93e^{-1}$ | $4.46e^{-1}$ | $4.56e^{-1}$ | $4.52e^{-1}$ | $3.61e^{-1}$ | $3.54e^{-1}$ | $3.26e^{-1}$ | N-BEATS |
| bitbrains_rnd, short, 5T | CRPS | $7.89e^{-1}$ | 1.00 | 1.00 | $6.74e^{-1}$ | 1.00 | $5.06e^{-1}$ | $4.42e^{-1}$ | $5.27e^{-1}$ | $5.96e^{-1}$ | $4.31e^{-1}$ | $4.57e^{-1}$ | $5.19e^{-1}$ | $5.85e^{-1}$ | $4.72e^{-1}$ | $5.91e^{-1}$ | $4.48e^{-1}$ | $4.61e^{-1}$ | $4.60e^{-1}$ | $4.11e^{-1}$ | $4.10e^{-1}$ | $4.01e^{-1}$ | Moi.L |
| bitbrains_rnd, short, 5T | Rank | $18.00$ | $21.00$ | $19.00$ | $17.00$ | $20.00$ | $11.00$ | $5.00$ | $13.00$ | $16.00$ | $4.00$ | $7.00$ | $12.00$ | $14.00$ | $10.00$ | $15.00$ | $6.00$ | $9.00$ | $8.00$ | $3.00$ | $2.00$ | $1.00$ | Moi.L |
| bitbrains_rnd, short, H | MAPE | $6.76e^{-1}$ | 1.00 | $4.59e^{-1}$ | $9.50e^{-1}$ | 1.00 | $2.59e^{-1}$ | $3.03e^{-1}$ | $7.18e^{-1}$ | $3.44e^{-1}$ | $2.90e^{-1}$ | $3.07e^{-1}$ | 2.05 | $9.92e^{-1}$ | $5.70e^{-1}$ | $9.81e^{-1}$ | $2.30e^{-1}$ | $3.61e^{-1}$ | $2.96e^{-1}$ | $1.42e^{-1}$ | $1.95e^{-1}$ | $1.50e^{-1}$ | Moi.S |
| bitbrains_rnd, short, H | CRPS | 1.12 | 1.00 | $6.72e^{-1}$ | 1.06 | 1.00 | $4.50e^{-1}$ | $5.00e^{-1}$ | $5.46e^{-1}$ | $5.50e^{-1}$ | $4.64e^{-1}$ | $4.58e^{-1}$ | $8.23e^{-1}$ | $8.00e^{-1}$ | $5.03e^{-1}$ | $7.92e^{-1}$ | $4.72e^{-1}$ | $5.12e^{-1}$ | $5.08e^{-1}$ | $4.18e^{-1}$ | $4.48e^{-1}$ | $4.05e^{-1}$ | Moi.L |
| bitbrains_rnd, short, H | Rank | $21.00$ | $19.00$ | $14.00$ | $20.00$ | $18.00$ | $4.00$ | $8.00$ | $12.00$ | $13.00$ | $6.00$ | $5.00$ | $17.00$ | $16.00$ | $9.00$ | $15.00$ | $7.00$ | $11.00$ | $10.00$ | $2.00$ | $3.00$ | $1.00$ | Moi.L |
| bizitobs_application, long, 10S | MAPE | 1.04 | 1.00 | 1.00 | $7.70e^{-5}$ | $1.79e^{-4}$ | $1.20e^{-4}$ | $3.61e^{-4}$ | $1.09e^{-4}$ | $1.02e^{-4}$ | $8.64e^{-5}$ | $9.35e^{-5}$ | $1.13e^{-4}$ | $2.01e^{-4}$ | $4.55e^{-4}$ | $9.07e^{-5}$ | $2.49e^{-4}$ | $2.36e^{-4}$ | $2.25e^{-4}$ | $2.16e^{-4}$ | $2.17e^{-4}$ | $2.08e^{-4}$ | A.Th. |
| bizitobs_application, long, 10S | CRPS | 1.01 | 1.00 | 1.00 | $3.65e^{-2}$ | $2.76e^{-1}$ | $8.52e^{-2}$ | $5.71e^{-2}$ | $6.00e^{-2}$ | $6.40e^{-2}$ | $5.58e^{-2}$ | $5.58e^{-2}$ | $7.20e^{-2}$ | $7.87e^{-2}$ | $1.32e^{-1}$ | $6.63e^{-2}$ | $9.59e^{-2}$ | $9.52e^{-2}$ | $8.62e^{-2}$ | $1.14e^{-1}$ | $1.11e^{-1}$ | $1.06e^{-1}$ | A.Th. |
| bizitobs_application, long, 10S | Rank | $21.00$ | $20.00$ | $19.00$ | $1.00$ | $18.00$ | $10.00$ | $4.00$ | $5.00$ | $6.00$ | $3.00$ | $2.00$ | $8.00$ | $9.00$ | $17.00$ | $7.00$ | $13.00$ | $12.00$ | $11.00$ | $16.00$ | $15.00$ | $14.00$ | A.Th. |
| bizitobs_application, medium, 10S | MAPE | 2.21 | 1.00 | 1.00 | $6.16e^{-1}$ | 2.61 | 1.16 | 4.55 | 1.11 | $9.51e^{-1}$ | 1.02 | 1.09 | 1.40 | 2.03 | 4.07 | 1.00 | 3.71 | 3.45 | 3.12 | 2.69 | 2.89 | 2.50 | A.Th. |
| bizitobs_application, medium, 10S | CRPS | 2.11 | 1.00 | 1.00 | $5.73e^{-1}$ | 3.84 | 1.28 | 1.13 | 1.08 | 1.12 | 1.13 | 1.07 | 1.34 | 1.16 | 3.25 | 1.24 | 1.86 | 2.79 | 2.28 | 2.00 | 2.37 | 1.83 | A.Th. |
| bizitobs_application, medium, 10S | Rank | $16.00$ | $3.00$ | $2.00$ | $1.00$ | $21.00$ | $11.00$ | $8.00$ | $5.00$ | $6.00$ | $7.00$ | $4.00$ | $12.00$ | $9.00$ | $20.00$ | $10.00$ | $14.00$ | $19.00$ | $17.00$ | $15.00$ | $18.00$ | $13.00$ | A.Th. |
| bizitobs_application, short, 10S | MAPE | 1.60 | 1.00 | 1.00 | $4.84e^{-1}$ | 2.16 | 1.74 | 3.87 | 1.71 | 1.16 | $9.47e^{-1}$ | $6.07e^{-1}$ | 3.26 | 3.11 | 1.82 | $9.97e^{-1}$ | 1.38 | 1.26 | 1.22 | 1.37 | 1.86 | 1.67 | A.Th. |
| bizitobs_application, short, 10S | CRPS | 1.10 | 1.00 | 1.00 | $2.84e^{-1}$ | 1.38 | 1.84 | 2.57 | 1.57 | 1.24 | $6.32e^{-1}$ | $5.92e^{-1}$ | 2.28 | 1.03 | 1.59 | 1.20 | $8.82e^{-1}$ | $8.91e^{-1}$ | $9.05e^{-1}$ | $9.45e^{-1}$ | 1.16 | 1.03 | A.Th. |
| bizitobs_application, short, 10S | Rank | $12.00$ | $9.00$ | $8.00$ | $1.00$ | $16.00$ | $19.00$ | $21.00$ | $17.00$ | $15.00$ | $3.00$ | $2.00$ | $20.00$ | $10.00$ | $18.00$ | $14.00$ | $4.00$ | $5.00$ | $6.00$ | $7.00$ | $13.00$ | $11.00$ | A.Th. |
| bizitobs_l2c, long, 5T | MAPE | $9.67e^{-1}$ | 1.00 | 1.00 | $9.64e^{-1}$ | $9.72e^{-1}$ | $9.80e^{-1}$ | $7.69e^{-1}$ | $8.23e^{-1}$ | $6.68e^{-1}$ | $5.94e^{-1}$ | $4.32e^{-1}$ | $7.83e^{-1}$ | $4.26e^{-1}$ | $9.90e^{-1}$ | $5.01e^{-1}$ | $9.72e^{-1}$ | $9.43e^{-1}$ | $9.66e^{-1}$ | $9.30e^{-1}$ | $8.65e^{-1}$ | $8.59e^{-1}$ | C.former |
| bizitobs_l2c, long, 5T | CRPS | $8.78e^{-1}$ | 1.00 | 1.00 | $9.38e^{-1}$ | $8.86e^{-1}$ | 1.07 | $7.00e^{-1}$ | $9.18e^{-1}$ | $8.10e^{-1}$ | $4.81e^{-1}$ | $3.68e^{-1}$ | $9.69e^{-1}$ | $4.35e^{-1}$ | 1.09 | $5.15e^{-1}$ | 1.10 | 1.07 | 1.10 | $8.74e^{-1}$ | $8.13e^{-1}$ | $8.00e^{-1}$ | iTrans. |
| bizitobs_l2c, long, 5T | Rank | $10.00$ | $16.00$ | $15.00$ | $13.00$ | $11.00$ | $17.00$ | $5.00$ | $12.00$ | $7.00$ | $3.00$ | $1.00$ | $14.00$ | $2.00$ | $19.00$ | $4.00$ | $21.00$ | $18.00$ | $20.00$ | $9.00$ | $8.00$ | $6.00$ | iTrans. |
| bizitobs_l2c, long, H | MAPE | 1.03 | 1.00 | $1.69e^{-1}$ | $4.20e^{-1}$ | $4.37e^{-1}$ | $1.26e^{-1}$ | $1.25e^{-1}$ | $1.81e^{-1}$ | $1.12e^{-1}$ | $1.28e^{-1}$ | $2.68e^{-1}$ | $1.57e^{-1}$ | $1.16e^{-1}$ | $2.18e^{-1}$ | $1.43e^{-1}$ | $2.83e^{-1}$ | $2.70e^{-1}$ | $2.53e^{-1}$ | $1.96e^{-1}$ | $1.76e^{-1}$ | $2.40e^{-1}$ | N-BEATS |
| bizitobs_l2c, long, H | CRPS | $6.98e^{-1}$ | 1.00 | $4.32e^{-1}$ | $4.50e^{-1}$ | $6.10e^{-1}$ | $1.86e^{-1}$ | $1.57e^{-1}$ | $2.61e^{-1}$ | $2.63e^{-1}$ | $1.60e^{-1}$ | $5.32e^{-1}$ | $2.32e^{-1}$ | $1.91e^{-1}$ | $3.98e^{-1}$ | $2.04e^{-1}$ | $4.29e^{-1}$ | $4.05e^{-1}$ | $4.16e^{-1}$ | $3.08e^{-1}$ | $2.46e^{-1}$ | $3.62e^{-1}$ | TFT |
| bizitobs_l2c, long, H | Rank | $20.00$ | $21.00$ | $16.00$ | $17.00$ | $19.00$ | $3.00$ | $1.00$ | $8.00$ | $9.00$ | $2.00$ | $18.00$ | $6.00$ | $4.00$ | $12.00$ | $5.00$ | $15.00$ | $13.00$ | $14.00$ | $10.00$ | $7.00$ | $11.00$ | TFT |
| bizitobs_l2c, medium, 5T | MAPE | $6.67e^{-1}$ | 1.00 | 1.00 | $7.29e^{-1}$ | $6.62e^{-1}$ | 1.03 | $5.74e^{-1}$ | $5.98e^{-1}$ | $6.27e^{-1}$ | $5.99e^{-1}$ | $5.52e^{-1}$ | $6.40e^{-1}$ | $4.42e^{-1}$ | $6.71e^{-1}$ | $5.38e^{-1}$ | $6.98e^{-1}$ | $7.12e^{-1}$ | $7.18e^{-1}$ | $6.79e^{-1}$ | $5.81e^{-1}$ | $6.37e^{-1}$ | C.former |
| bizitobs_l2c, medium, 5T | CRPS | $6.77e^{-1}$ | 1.00 | 1.00 | $7.83e^{-1}$ | $6.70e^{-1}$ | 1.11 | $6.53e^{-1}$ | $6.81e^{-1}$ | $9.53e^{-1}$ | $6.26e^{-1}$ | $6.00e^{-1}$ | $9.25e^{-1}$ | $6.21e^{-1}$ | $8.47e^{-1}$ | $6.49e^{-1}$ | $8.77e^{-1}$ | $9.13e^{-1}$ | $9.36e^{-1}$ | $7.11e^{-1}$ | $7.04e^{-1}$ | $7.66e^{-1}$ | iTrans. |
| bizitobs_l2c, medium, 5T | Rank | $7.00$ | $20.00$ | $19.00$ | $12.00$ | $6.00$ | $21.00$ | $5.00$ | $8.00$ | $18.00$ | $3.00$ | $1.00$ | $16.00$ | $2.00$ | $13.00$ | $4.00$ | $14.00$ | $15.00$ | $17.00$ | $10.00$ | $9.00$ | $11.00$ | iTrans. |
| bizitobs_l2c, medium, H | MAPE | $6.25e^{-1}$ | 1.00 | $6.44e^{-1}$ | $8.87e^{-1}$ | $7.31e^{-1}$ | $6.12e^{-1}$ | $6.09e^{-1}$ | $6.25e^{-1}$ | $3.53e^{-1}$ | $4.11e^{-1}$ | $4.09e^{-1}$ | $4.94e^{-1}$ | $3.56e^{-1}$ | 1.42 | $3.54e^{-1}$ | $9.38e^{-1}$ | $8.31e^{-1}$ | $8.56e^{-1}$ | $7.25e^{-1}$ | $6.37e^{-1}$ | $7.06e^{-1}$ | N-BEATS |
| bizitobs_l2c, medium, H | CRPS | $6.18e^{-1}$ | 1.00 | $5.73e^{-1}$ | $6.28e^{-1}$ | $7.39e^{-1}$ | $2.43e^{-1}$ | $2.43e^{-1}$ | $3.47e^{-1}$ | $2.96e^{-1}$ | $1.85e^{-1}$ | $1.87e^{-1}$ | $2.80e^{-1}$ | $2.18e^{-1}$ | $6.03e^{-1}$ | $2.13e^{-1}$ | $5.49e^{-1}$ | $5.58e^{-1}$ | $5.65e^{-1}$ | $3.99e^{-1}$ | $3.27e^{-1}$ | $4.86e^{-1}$ | P.TST |
| bizitobs_l2c, medium, H | Rank | $18.00$ | $21.00$ | $16.00$ | $19.00$ | $20.00$ | $5.00$ | $6.00$ | $10.00$ | $8.00$ | $1.00$ | $2.00$ | $7.00$ | $4.00$ | $17.00$ | $3.00$ | $13.00$ | $14.00$ | $15.00$ | $11.00$ | $9.00$ | $12.00$ | P.TST |
| bizitobs_l2c, short, 5T | MAPE | $3.94e^{-1}$ | 1.00 | 1.00 | $3.97e^{-1}$ | $3.69e^{-1}$ | $7.61e^{-1}$ | $3.73e^{-1}$ | $5.47e^{-1}$ | $3.90e^{-1}$ | $3.24e^{-1}$ | $3.62e^{-1}$ | $3.03e^{-1}$ | $4.95e^{-1}$ | $4.18e^{-1}$ | $4.53e^{-1}$ | $4.27e^{-1}$ | $4.01e^{-1}$ | $4.18e^{-1}$ | $4.15e^{-1}$ | $4.39e^{-1}$ | $3.40e^{-1}$ | DLin. |
| bizitobs_l2c, short, 5T | CRPS | $3.06e^{-1}$ | 1.00 | 1.00 | $3.05e^{-1}$ | $2.88e^{-1}$ | $6.83e^{-1}$ | $2.94e^{-1}$ | $4.73e^{-1}$ | $3.82e^{-1}$ | $2.83e^{-1}$ | $2.94e^{-1}$ | $3.07e^{-1}$ | $5.69e^{-1}$ | $3.06e^{-1}$ | $4.27e^{-1}$ | $3.32e^{-1}$ | $3.20e^{-1}$ | $3.22e^{-1}$ | $3.24e^{-1}$ | $3.50e^{-1}$ | $2.86e^{-1}$ | P.TST |
| bizitobs_l2c, short, 5T | Rank | $7.00$ | $21.00$ | $20.00$ | $6.00$ | $3.00$ | $19.00$ | $5.00$ | $17.00$ | $15.00$ | $1.00$ | $4.00$ | $9.00$ | $18.00$ | $8.00$ | $16.00$ | $13.00$ | $10.00$ | $11.00$ | $12.00$ | $14.00$ | $2.00$ | P.TST |
| bizitobs_l2c, short, H | MAPE | $9.49e^{-1}$ | 1.00 | $7.50e^{-1}$ | $9.78e^{-1}$ | $9.12e^{-1}$ | $7.17e^{-1}$ | $5.59e^{-1}$ | $3.43e^{-1}$ | $4.75e^{-1}$ | $3.18e^{-1}$ | $3.31e^{-1}$ | $4.94e^{-1}$ | $3.74e^{-1}$ | $7.09e^{-1}$ | $4.06e^{-1}$ | $6.51e^{-1}$ | $7.01e^{-1}$ | $7.87e^{-1}$ | $5.51e^{-1}$ | $5.43e^{-1}$ | $5.10e^{-1}$ | P.TST |
| bizitobs_l2c, short, H | CRPS | $9.07e^{-1}$ | 1.00 | 1.02 | $9.46e^{-1}$ | $9.22e^{-1}$ | 1.47 | $7.48e^{-1}$ | $4.09e^{-1}$ | $5.37e^{-1}$ | $4.05e^{-1}$ | $3.58e^{-1}$ | $6.27e^{-1}$ | $5.32e^{-1}$ | $9.05e^{-1}$ | $6.16e^{-1}$ | $7.99e^{-1}$ | $8.75e^{-1}$ | $9.16e^{-1}$ | $7.61e^{-1}$ | $6.83e^{-1}$ | $9.85e^{-1}$ | iTrans. |
| bizitobs_l2c, short, H | Rank | $14.00$ | $19.00$ | $20.00$ | $17.00$ | $16.00$ | $21.00$ | $9.00$ | $3.00$ | $5.00$ | $2.00$ | $1.00$ | $7.00$ | $4.00$ | $13.00$ | $6.00$ | $11.00$ | $12.00$ | $15.00$ | $10.00$ | $8.00$ | $18.00$ | iTrans. |
| bizitobs_service, long, 10S | MAPE | 1.78 | 1.00 | 1.00 | 1.20 | 2.06 | 4.17 | 2.56 | 3.03 | 1.39 | 1.60 | 2.77 | 5.52 | $1.50e^{1}$ | $1.60e^{1}$ | 1.73 | 2.51 | 2.69 | 2.47 | 2.16 | 3.59 | 1.90 | S.Nv. |
| bizitobs_service, long, 10S | CRPS | 2.13 | 1.00 | 1.00 | $9.25e^{-1}$ | 6.67 | 1.25 | 1.00 | 1.16 | 1.09 | 1.02 | $9.71e^{-1}$ | 1.21 | 1.22 | 2.45 | 1.08 | 1.66 | 1.67 | 1.68 | 2.06 | 1.90 | 1.99 | A.Th. |
| bizitobs_service, long, 10S | Rank | $19.00$ | $4.00$ | $3.00$ | $1.00$ | $21.00$ | $12.00$ | $5.00$ | $9.00$ | $8.00$ | $6.00$ | $2.00$ | $10.00$ | $11.00$ | $20.00$ | $7.00$ | $13.00$ | $14.00$ | $15.00$ | $18.00$ | $16.00$ | $17.00$ | A.Th. |
| bizitobs_service, medium, 10S | MAPE | 1.68 | 1.00 | 1.00 | $8.62e^{-1}$ | 2.29 | 1.91 | 1.65 | 2.25 | 1.56 | 1.53 | 2.00 | 5.06 | $1.97e^{1}$ | 9.96 | 1.37 | 5.00 | 8.71 | 9.38 | 2.44 | 4.16 | 1.77 | A.Th. |
| bizitobs_service, medium, 10S | CRPS | 1.96 | 1.00 | 1.00 | $5.64e^{-1}$ | 4.61 | $9.05e^{-1}$ | $9.01e^{-1}$ | 1.02 | $8.93e^{-1}$ | $9.28e^{-1}$ | $8.64e^{-1}$ | 1.09 | 1.21 | 2.24 | 1.00 | 1.41 | 1.49 | 1.49 | 1.71 | 1.79 | 1.40 | A.Th. |
| bizitobs_service, medium, 10S | Rank | $19.00$ | $8.00$ | $7.00$ | $1.00$ | $21.00$ | $5.00$ | $4.00$ | $10.00$ | $3.00$ | $6.00$ | $2.00$ | $11.00$ | $12.00$ | $20.00$ | $9.00$ | $14.00$ | $16.00$ | $15.00$ | $17.00$ | $18.00$ | $13.00$ | A.Th. |
| bizitobs_service, short, 10S | MAPE | 3.52 | 1.00 | 1.00 | $7.31e^{-1}$ | 1.52 | 1.98 | 6.83 | 1.93 | $9.20e^{-1}$ | 1.25 | 1.42 | 5.55 | $2.49e^{1}$ | 5.53 | 1.48 | 1.93 | 1.59 | 1.94 | 1.24 | 1.55 | 1.47 | A.Th. |
| bizitobs_service, short, 10S | CRPS | $9.15e^{-1}$ | 1.00 | 1.00 | $3.18e^{-1}$ | 1.16 | $8.10e^{-1}$ | $6.13e^{-1}$ | 1.02 | $5.32e^{-1}$ | $6.30e^{-1}$ | $5.90e^{-1}$ | $7.90e^{-1}$ | 1.08 | 1.27 | 1.02 | $6.77e^{-1}$ | $6.80e^{-1}$ | $6.75e^{-1}$ | 1.03 | 1.08 | 1.08 | A.Th. |
| bizitobs_service, short, 10S | Rank | $11.00$ | $13.00$ | $12.00$ | $1.00$ | $20.00$ | $10.00$ | $4.00$ | $15.00$ | $2.00$ | $5.00$ | $3.00$ | $9.00$ | $18.00$ | $21.00$ | $14.00$ | $7.00$ | $8.00$ | $6.00$ | $16.00$ | $19.00$ | $17.00$ | A.Th. |
| car_parts, short, M | MAPE | $9.95e^{-1}$ | 1.00 | $8.59e^{-1}$ | $8.02e^{-1}$ | $7.61e^{-1}$ | $9.70e^{-1}$ | 1.14 | 1.17 | 1.10 | 1.17 | 1.16 | $8.49e^{-1}$ | $8.89e^{-1}$ | $9.04e^{-1}$ | $8.74e^{-1}$ | $9.00e^{-1}$ | $8.97e^{-1}$ | $9.13e^{-1}$ | $8.57e^{-1}$ | $8.70e^{-1}$ | $8.04e^{-1}$ | A.ETS |
| car_parts, short, M | CRPS | 1.45 | 1.00 | $7.50e^{-1}$ | $7.79e^{-1}$ | $7.79e^{-1}$ | $5.54e^{-1}$ | $5.17e^{-1}$ | $7.15e^{-1}$ | $5.93e^{-1}$ | $5.81e^{-1}$ | $5.81e^{-1}$ | $7.33e^{-1}$ | 8.72 | $5.93e^{-1}$ | $8.55e^{-1}$ | $5.99e^{-1}$ | $6.22e^{-1}$ | $6.22e^{-1}$ | $9.42e^{-1}$ | 1.15 | $7.27e^{-1}$ | TFT |
| car_parts, short, M | Rank | $20.00$ | $18.00$ | $13.00$ | $15.00$ | $14.00$ | $2.00$ | $1.00$ | $10.00$ | $5.00$ | $4.00$ | $3.00$ | $12.00$ | $21.00$ | $6.00$ | $16.00$ | $7.00$ | $8.00$ | $9.00$ | $17.00$ | $19.00$ | $11.00$ | TFT |
| covid_deaths, short, D | MAPE | 1.00 | 1.00 | $6.38e^{-1}$ | 1.02 | $5.87e^{-1}$ | 1.13 | $6.74e^{-1}$ | 1.17 | $6.77e^{-1}$ | $9.92e^{-1}$ | $7.35e^{-1}$ | $7.42e^{-1}$ | $2.23e^{3}$ | 1.32 | $9.40e^{-1}$ | $8.80e^{-1}$ | $8.87e^{-1}$ | $8.95e^{-1}$ | $8.50e^{-1}$ | 1.11 | $6.88e^{-1}$ | A.ETS |
| covid_deaths, short, D | CRPS | 1.02 | 1.00 | $2.38e^{-1}$ | $7.56e^{-1}$ | $2.60e^{-1}$ | 1.42 | $2.99e^{-1}$ | 1.37 | $5.68e^{-1}$ | $5.38e^{-1}$ | $4.70e^{-1}$ | $5.04e^{-1}$ | $5.62e^{1}$ | 1.63 | $4.64e^{-1}$ | $4.86e^{-1}$ | $3.62e^{-1}$ | $3.39e^{-1}$ | $3.98e^{-1}$ | $7.80e^{-1}$ | $3.20e^{-1}$ | A.Ar. |
| covid_deaths, short, D | Rank | $17.00$ | $16.00$ | $1.00$ | $14.00$ | $2.00$ | $19.00$ | $3.00$ | $18.00$ | $13.00$ | $12.00$ | $9.00$ | $11.00$ | $21.00$ | $20.00$ | $8.00$ | $10.00$ | $6.00$ | $5.00$ | $7.00$ | $15.00$ | $4.00$ | A.Ar. |
| electricity, long, 15T | MAPE | 2.58 | 1.00 | 1.00 | 1.35 | 1.00 | 1.51 | $7.70e^{-1}$ | 1.04 | $8.55e^{-1}$ | $7.45e^{-1}$ | $9.05e^{-1}$ | 1.18 | $3.13e^{1}$ | 1.25 | $9.15e^{-1}$ | $9.65e^{-1}$ | $8.65e^{-1}$ | $8.70e^{-1}$ | 1.16 | $9.45e^{-1}$ | $8.60e^{-1}$ | P.TST |
| electricity, long, 15T | CRPS | 3.71 | 1.00 | 1.00 | 3.11 | 1.00 | 1.20 | $6.50e^{-1}$ | $7.75e^{-1}$ | $9.53e^{-1}$ | $6.30e^{-1}$ | $6.69e^{-1}$ | 1.00 | 1.90 | 1.06 | $8.14e^{-1}$ | $7.56e^{-1}$ | $7.33e^{-1}$ | $7.30e^{-1}$ | 1.17 | $7.83e^{-1}$ | $6.85e^{-1}$ | P.TST |
| electricity, long, 15T | Rank | $21.00$ | $15.00$ | $12.00$ | $20.00$ | $13.00$ | $18.00$ | $2.00$ | $8.00$ | $11.00$ | $1.00$ | $3.00$ | $14.00$ | $19.00$ | $16.00$ | $10.00$ | $7.00$ | $6.00$ | $5.00$ | $17.00$ | $9.00$ | $4.00$ | P.TST |
| electricity, long, H | MAPE | 1.51 | 1.00 | 1.00 | 1.35 | 1.93 | $9.20e^{-1}$ | 1.22 | 1.20 | $5.98e^{-1}$ | $9.28e^{-1}$ | $9.56e^{-1}$ | 1.40 | $1.51e^{1}$ | $9.34e^{-1}$ | 1.13 | $8.42e^{-1}$ | $8.56e^{-1}$ | $8.64e^{-1}$ | 1.12 | $9.17e^{-1}$ | $8.64e^{-1}$ | N-BEATS |
| electricity, long, H | CRPS | 3.52 | 1.00 | 1.00 | 1.58 | $1.38e^{25}$ | $9.26e^{-1}$ | $4.94e^{-1}$ | $6.26e^{-1}$ | $6.63e^{-1}$ | $5.47e^{-1}$ | $5.14e^{-1}$ | 1.07 | $7.58e^{-1}$ | $5.32e^{-1}$ | $6.68e^{-1}$ | $5.63e^{-1}$ | $5.53e^{-1}$ | $5.63e^{-1}$ | $4.63e^{-1}$ | $5.19e^{-1}$ | $5.37e^{-1}$ | Moi.S |
| electricity, long, H | Rank | $20.00$ | $17.00$ | $16.00$ | $19.00$ | $21.00$ | $15.00$ | $2.00$ | $11.00$ | $12.00$ | $7.00$ | $3.00$ | $18.00$ | $14.00$ | $5.00$ | $13.00$ | $10.00$ | $8.00$ | $9.00$ | $1.00$ | $4.00$ | $6.00$ | Moi.S |
| electricity, medium, 15T | MAPE | 1.88 | 1.00 | 1.00 | 1.07 | 1.00 | 1.19 | $8.88e^{-1}$ | $8.70e^{-1}$ | 1.31 | $7.49e^{-1}$ | $8.37e^{-1}$ | 1.16 | $2.14e^{1}$ | 1.15 | $7.86e^{-1}$ | $8.56e^{-1}$ | $8.19e^{-1}$ | $8.23e^{-1}$ | 1.21 | $9.44e^{-1}$ | $7.91e^{-1}$ | P.TST |
| electricity, medium, 15T | CRPS | 3.47 | 1.00 | 1.00 | 2.65 | 1.00 | $9.60e^{-1}$ | $7.57e^{-1}$ | $7.71e^{-1}$ | 1.42 | $6.92e^{-1}$ | $7.02e^{-1}$ | 1.15 | 2.00 | 1.11 | $8.39e^{-1}$ | $7.72e^{-1}$ | $7.68e^{-1}$ | $7.61e^{-1}$ | 1.06 | $8.87e^{-1}$ | $7.25e^{-1}$ | P.TST |
| electricity, medium, 15T | Rank | $21.00$ | $14.00$ | $12.00$ | $20.00$ | $13.00$ | $11.00$ | $4.00$ | $7.00$ | $18.00$ | $1.00$ | $2.00$ | $17.00$ | $19.00$ | $16.00$ | $9.00$ | $8.00$ | $6.00$ | $5.00$ | $15.00$ | $10.00$ | $3.00$ | P.TST |
| electricity, medium, H | MAPE | 1.53 | 1.00 | 1.00 | 1.06 | 1.00 | 1.49 | $7.91e^{-1}$ | $7.96e^{-1}$ | $5.50e^{-1}$ | $6.33e^{-1}$ | $7.24e^{-1}$ | 1.24 | $1.17e^{1}$ | $5.66e^{-1}$ | $8.94e^{-1}$ | $5.56e^{-1}$ | $6.72e^{-1}$ | $6.69e^{-1}$ | $8.58e^{-1}$ | $6.07e^{-1}$ | $7.03e^{-1}$ | N-BEATS |
| electricity, medium, H | CRPS | 3.58 | 1.00 | 1.00 | 1.63 | 1.00 | 2.91 | $5.82e^{-1}$ | $6.08e^{-1}$ | $7.37e^{-1}$ | $5.19e^{-1}$ | $5.49e^{-1}$ | 1.32 | 1.38 | $5.25e^{-1}$ | $7.18e^{-1}$ | $5.62e^{-1}$ | $5.59e^{-1}$ | $5.50e^{-1}$ | $5.45e^{-1}$ | $5.53e^{-1}$ | $5.40e^{-1}$ | P.TST |
| electricity, medium, H | Rank | $21.00$ | $16.00$ | $14.00$ | $19.00$ | $15.00$ | $20.00$ | $10.00$ | $11.00$ | $13.00$ | $1.00$ | $5.00$ | $17.00$ | $18.00$ | $2.00$ | $12.00$ | $9.00$ | $8.00$ | $6.00$ | $4.00$ | $7.00$ | $3.00$ | P.TST |
| electricity, short, 15T | MAPE | 1.17 | 1.00 | 1.00 | $7.15e^{-1}$ | 1.20 | $7.35e^{-1}$ | 1.12 | $9.24e^{-1}$ | $8.74e^{-1}$ | $8.51e^{-1}$ | $7.72e^{-1}$ | $9.37e^{-1}$ | $1.74e^{1}$ | $7.72e^{-1}$ | $9.54e^{-1}$ | $6.19e^{-1}$ | $5.86e^{-1}$ | $5.86e^{-1}$ | $8.34e^{-1}$ | $8.54e^{-1}$ | $6.56e^{-1}$ | Chr.B |
| electricity, short, 15T | CRPS | 1.34 | 1.00 | 1.00 | $8.48e^{-1}$ | 1.06 | $9.21e^{-1}$ | 1.12 | $8.73e^{-1}$ | 1.09 | $8.12e^{-1}$ | $7.52e^{-1}$ | 1.07 | $2.04e^{1}$ | $7.88e^{-1}$ | 1.02 | $5.98e^{-1}$ | $5.61e^{-1}$ | $5.56e^{-1}$ | $8.36e^{-1}$ | $8.55e^{-1}$ | $6.30e^{-1}$ | Chr.L |
| electricity, short, 15T | Rank | $20.00$ | $14.00$ | $13.00$ | $9.00$ | $16.00$ | $12.00$ | $19.00$ | $11.00$ | $18.00$ | $7.00$ | $5.00$ | $17.00$ | $21.00$ | $6.00$ | $15.00$ | $3.00$ | $2.00$ | $1.00$ | $8.00$ | $10.00$ | $4.00$ | Chr.L |
| electricity, short, D | MAPE | 1.00 | 1.00 | 1.11 | 1.10 | 1.13 | $8.69e^{-1}$ | 1.17 | $9.70e^{-1}$ | $9.62e^{-1}$ | $9.16e^{-1}$ | $8.84e^{-1}$ | 1.16 | $1.70e^{1}$ | $8.08e^{-1}$ | $8.31e^{-1}$ | $8.35e^{-1}$ | $8.38e^{-1}$ | $8.38e^{-1}$ | 1.03 | 1.13 | $7.27e^{-1}$ | Moi.L |
| electricity, short, D | CRPS | $8.52e^{-1}$ | 1.00 | $6.82e^{-1}$ | $7.25e^{-1}$ | $9.10e^{-1}$ | $6.43e^{-1}$ | $6.85e^{-1}$ | $6.36e^{-1}$ | $9.02e^{-1}$ | $6.79e^{-1}$ | $6.29e^{-1}$ | 1.39 | $3.95e^{1}$ | $6.33e^{-1}$ | $7.48e^{-1}$ | $5.82e^{-1}$ | $5.01e^{-1}$ | $4.97e^{-1}$ | $5.60e^{-1}$ | $5.24e^{-1}$ | $4.82e^{-1}$ | Moi.L |
| electricity, short, D | Rank | $16.00$ | $19.00$ | $12.00$ | $14.00$ | $18.00$ | $10.00$ | $13.00$ | $9.00$ | $17.00$ | $11.00$ | $7.00$ | $20.00$ | $21.00$ | $8.00$ | $15.00$ | $6.00$ | $3.00$ | $2.00$ | $5.00$ | $4.00$ | $1.00$ | Moi.L |
| electricity, short, H | MAPE | 1.44 | 1.00 | 1.00 | $9.81e^{-1}$ | $8.71e^{-1}$ | $4.44e^{-1}$ | 1.01 | $8.51e^{-1}$ | $9.20e^{-1}$ | $5.87e^{-1}$ | $6.75e^{-1}$ | 1.02 | $1.52e^{1}$ | $5.18e^{-1}$ | $8.15e^{-1}$ | $5.67e^{-1}$ | $4.33e^{-1}$ | $4.46e^{-1}$ | $7.69e^{-1}$ | $6.25e^{-1}$ | $5.37e^{-1}$ | Chr.B |
| electricity, short, H | CRPS | 2.70 | 1.00 | 1.00 | 1.62 | $9.10e^{-1}$ | $8.61e^{-1}$ | $8.17e^{-1}$ | $7.92e^{-1}$ | 1.13 | $7.20e^{-1}$ | $6.71e^{-1}$ | 1.03 | $2.48e^{1}$ | $5.90e^{-1}$ | $8.57e^{-1}$ | $6.43e^{-1}$ | $5.90e^{-1}$ | $5.94e^{-1}$ | $7.07e^{-1}$ | $7.09e^{-1}$ | $6.76e^{-1}$ | T.FM |
| electricity, short, H | Rank | $20.00$ | $16.00$ | $15.00$ | $19.00$ | $14.00$ | $13.00$ | $11.00$ | $10.00$ | $18.00$ | $9.00$ | $5.00$ | $17.00$ | $21.00$ | $2.00$ | $12.00$ | $4.00$ | $1.00$ | $3.00$ | $7.00$ | $8.00$ | $6.00$ | Chr.B |
| electricity, short, W | MAPE | 1.00 | 1.00 | $9.58e^{-1}$ | 1.00 | $9.72e^{-1}$ | $9.06e^{-1}$ | 1.02 | $9.69e^{-1}$ | $9.27e^{-1}$ | $9.69e^{-1}$ | $9.97e^{-1}$ | $9.27e^{-1}$ | $2.47e^{1}$ | $9.13e^{-1}$ | $9.34e^{-1}$ | $8.60e^{-1}$ | $8.67e^{-1}$ | $8.71e^{-1}$ | $9.51e^{-1}$ | $9.55e^{-1}$ | $9.27e^{-1}$ | Chr.S |
| electricity, short, W | CRPS | 1.00 | 1.00 | 1.01 | 1.02 | $9.92e^{-1}$ | $9.29e^{-1}$ | 1.08 | 1.14 | 1.24 | $9.65e^{-1}$ | 1.10 | 1.12 | $2.22e^{2}$ | $8.88e^{-1}$ | 1.12 | $5.26e^{-1}$ | $4.99e^{-1}$ | $4.92e^{-1}$ | $8.68e^{-1}$ | $8.85e^{-1}$ | $6.14e^{-1}$ | Chr.L |
| electricity, short, W | Rank | $12.00$ | $11.00$ | $13.00$ | $14.00$ | $10.00$ | $8.00$ | $15.00$ | $19.00$ | $20.00$ | $9.00$ | $16.00$ | $17.00$ | $21.00$ | $7.00$ | $18.00$ | $3.00$ | $2.00$ | $1.00$ | $5.00$ | $6.00$ | $4.00$ | Chr.L |

**Table 21: Results on all dataset configs for GIFT-Eval | Table 2/3. The best results across each row are bolded, while second best results are underlined.**
| Dataset,term,frequency | Metric | Nv. | S.Nv. | A.Ar. | A.Th. | A.ETS | DeepAR | TFT | TiDE | N-BEATS | P.TST | iTrans. | DLin. | C.former | T.FM | Vis.TS | Chr.S | Chr.B | Chr.L | Moi.S | Moi.B | Moi.L | Best |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| ett1, long, 15T | MAPE | 1.19 | 1.00 | 1.00 | 1.20 | 1.33 | 4.72 | $8.37e^{-1}$ | $8.10e^{-1}$ | $7.82e^{-1}$ | $8.37e^{-1}$ | $8.47e^{-1}$ | $8.20e^{-1}$ | $8.04e^{-1}$ | $8.12e^{-1}$ | $8.29e^{-1}$ | $8.88e^{-1}$ | $8.60e^{-1}$ | $8.72e^{-1}$ | $8.78e^{-1}$ | $8.50e^{-1}$ | $7.72e^{-1}$ | Moi.L |
| ett1, long, 15T | CRPS | 1.81 | 1.00 | 1.00 | 3.51 | 2.65 | 5.61 | $7.07e^{-1}$ | $7.27e^{-1}$ | 1.09 | $6.24e^{-1}$ | $6.19e^{-1}$ | $8.66e^{-1}$ | $8.64e^{-1}$ | $9.04e^{-1}$ | $8.08e^{-1}$ | 1.14 | 1.01 | $9.27e^{-1}$ | $7.20e^{-1}$ | $6.59e^{-1}$ | $7.58e^{-1}$ | iTrans. |
| ett1, long, 15T | Rank | $18.00$ | $14.00$ | $13.00$ | $20.00$ | $19.00$ | $21.00$ | $4.00$ | $6.00$ | $16.00$ | $2.00$ | $1.00$ | $10.00$ | $9.00$ | $11.00$ | $8.00$ | $17.00$ | $15.00$ | $12.00$ | $5.00$ | $3.00$ | $7.00$ | iTrans. |
| ett1, long, H | MAPE | 2.13 | 1.00 | $8.26e^{-1}$ | 1.01 | 2.13 | 1.50 | 1.00 | 1.48 | $7.66e^{-1}$ | 1.69 | 1.17 | 1.22 | 1.02 | $7.25e^{-1}$ | 1.45 | $9.58e^{-1}$ | 1.11 | 1.17 | $9.74e^{-1}$ | 1.77 | 1.19 | T.FM |
| ett1, long, H | CRPS | 1.60 | 1.00 | $6.98e^{-1}$ | 3.15 | $7.27e^{2}$ | $7.61e^{-1}$ | $5.08e^{-1}$ | $5.08e^{-1}$ | $9.20e^{-1}$ | $4.82e^{-1}$ | $4.82e^{-1}$ | $5.89e^{-1}$ | $9.25e^{-1}$ | $5.15e^{-1}$ | $5.91e^{-1}$ | $5.84e^{-1}$ | $5.68e^{-1}$ | $5.68e^{-1}$ | $4.63e^{-1}$ | $4.98e^{-1}$ | $5.75e^{-1}$ | Moi.S |
| ett1, long, H | Rank | $19.00$ | $18.00$ | $14.00$ | $20.00$ | $21.00$ | $15.00$ | $5.00$ | $6.00$ | $16.00$ | $3.00$ | $2.00$ | $12.00$ | $17.00$ | $7.00$ | $13.00$ | $11.00$ | $8.00$ | $9.00$ | $1.00$ | $4.00$ | $10.00$ | Moi.S |
| ett1, medium, 15T | MAPE | 1.22 | 1.00 | 1.00 | $9.77e^{-1}$ | 1.35 | $9.17e^{-1}$ | $8.11e^{-1}$ | $8.80e^{-1}$ | $9.70e^{-1}$ | $7.84e^{-1}$ | $8.17e^{-1}$ | $8.31e^{-1}$ | $8.08e^{-1}$ | $8.89e^{-1}$ | 1.01 | $9.43e^{-1}$ | $9.71e^{-1}$ | $9.31e^{-1}$ | $8.92e^{-1}$ | $8.79e^{-1}$ | $8.22e^{-1}$ | P.TST |
| ett1, medium, 15T | CRPS | 1.75 | 1.00 | 1.00 | 3.21 | 2.46 | $8.95e^{-1}$ | $7.02e^{-1}$ | $8.10e^{-1}$ | 1.22 | $7.10e^{-1}$ | $7.13e^{-1}$ | $9.86e^{-1}$ | $9.40e^{-1}$ | $9.35e^{-1}$ | 1.05 | 1.11 | 1.08 | 1.07 | $8.86e^{-1}$ | $7.50e^{-1}$ | $8.72e^{-1}$ | TFT |
| ett1, medium, 15T | Rank | $19.00$ | $13.00$ | $12.00$ | $21.00$ | $20.00$ | $8.00$ | $1.00$ | $5.00$ | $18.00$ | $2.00$ | $3.00$ | $11.00$ | $10.00$ | $9.00$ | $14.00$ | $17.00$ | $16.00$ | $15.00$ | $7.00$ | $4.00$ | $6.00$ | TFT |
| ett1, medium, H | MAPE | 4.14 | 1.00 | 1.41 | 1.08 | 4.14 | 1.77 | 1.53 | 2.26 | 5.57 | 1.80 | 3.11 | 2.87 | 2.70 | 1.01 | 2.84 | $2.08e^{-1}$ | $9.70e^{-1}$ | $8.64e^{-1}$ | $4.68e^{-1}$ | 2.04 | 1.32 | Chr.S |
| ett1, medium, H | CRPS | 1.56 | 1.00 | $7.11e^{-1}$ | 3.06 | $2.85e^{1}$ | $9.91e^{-1}$ | $5.85e^{-1}$ | $5.41e^{-1}$ | $8.33e^{-1}$ | $5.06e^{-1}$ | $5.24e^{-1}$ | $8.43e^{-1}$ | $6.81e^{-1}$ | $5.63e^{-1}$ | $6.89e^{-1}$ | $6.06e^{-1}$ | $6.11e^{-1}$ | $6.17e^{-1}$ | $5.20e^{-1}$ | $5.17e^{-1}$ | $5.69e^{-1}$ | P.TST |
| ett1, medium, H | Rank | $19.00$ | $18.00$ | $14.00$ | $20.00$ | $21.00$ | $17.00$ | $8.00$ | $5.00$ | $15.00$ | $1.00$ | $4.00$ | $16.00$ | $12.00$ | $6.00$ | $13.00$ | $9.00$ | $10.00$ | $11.00$ | $3.00$ | $2.00$ | $7.00$ | P.TST |
| ett1, short, 15T | MAPE | 1.47 | 1.00 | 1.00 | $7.99e^{-1}$ | 1.48 | 1.10 | $9.42e^{-1}$ | $7.82e^{-1}$ | $7.44e^{-1}$ | $8.20e^{-1}$ | $9.05e^{-1}$ | $7.47e^{-1}$ | $9.95e^{-1}$ | $7.79e^{-1}$ | $8.20e^{-1}$ | $8.36e^{-1}$ | $7.93e^{-1}$ | $7.63e^{-1}$ | 1.19 | $8.18e^{-1}$ | $6.57e^{-1}$ | Moi.L |
| ett1, short, 15T | CRPS | 1.60 | 1.00 | 1.00 | 1.70 | 1.59 | 1.33 | 1.02 | $9.29e^{-1}$ | 1.05 | $7.93e^{-1}$ | $7.88e^{-1}$ | $9.67e^{-1}$ | 1.08 | $8.01e^{-1}$ | 1.04 | $9.00e^{-1}$ | $8.22e^{-1}$ | $8.13e^{-1}$ | 1.06 | $8.76e^{-1}$ | $7.43e^{-1}$ | Moi.L |
| ett1, short, 15T | Rank | $20.00$ | $12.00$ | $11.00$ | $21.00$ | $19.00$ | $18.00$ | $13.00$ | $9.00$ | $15.00$ | $3.00$ | $2.00$ | $10.00$ | $17.00$ | $4.00$ | $14.00$ | $8.00$ | $6.00$ | $5.00$ | $16.00$ | $7.00$ | $1.00$ | Moi.L |
| ett1, short, D | MAPE | 1.00 | 1.00 | $8.97e^{-1}$ | $8.15e^{-1}$ | $9.74e^{-1}$ | $7.62e^{-1}$ | 1.29 | 2.00 | 1.32 | $9.66e^{-1}$ | 1.77 | 1.08 | 1.12 | 1.75 | 1.80 | 1.27 | 1.06 | 1.17 | 1.02 | $9.66e^{-1}$ | 1.04 | DeepAR |
| ett1, short, D | CRPS | $7.92e^{-1}$ | 1.00 | $5.42e^{-1}$ | $6.62e^{-1}$ | $6.49e^{-1}$ | $5.69e^{-1}$ | $6.41e^{-1}$ | 2.06 | $7.51e^{-1}$ | $5.90e^{-1}$ | $6.68e^{-1}$ | $7.30e^{-1}$ | $7.42e^{-1}$ | $5.44e^{-1}$ | $7.79e^{-1}$ | $6.99e^{-1}$ | $7.51e^{-1}$ | $7.84e^{-1}$ | $5.63e^{-1}$ | $5.57e^{-1}$ | $5.53e^{-1}$ | A.Ar. |
| ett1, short, D | Rank | $19.00$ | $20.00$ | $1.00$ | $10.00$ | $9.00$ | $6.00$ | $8.00$ | $21.00$ | $16.00$ | $7.00$ | $11.00$ | $13.00$ | $14.00$ | $2.00$ | $17.00$ | $12.00$ | $15.00$ | $18.00$ | $5.00$ | $4.00$ | $3.00$ | A.Ar. |
| ett1, short, H | MAPE | 1.62 | 1.00 | $9.12e^{-1}$ | 1.15 | 1.69 | $9.84e^{-1}$ | $9.35e^{-1}$ | $8.58e^{-1}$ | $8.53e^{-1}$ | $7.82e^{-1}$ | $8.11e^{-1}$ | $8.16e^{-1}$ | 1.05 | $8.44e^{-1}$ | $9.28e^{-1}$ | $9.30e^{-1}$ | $7.56e^{-1}$ | $8.25e^{-1}$ | $8.70e^{-1}$ | $9.46e^{-1}$ | $8.86e^{-1}$ | Chr.B |
| ett1, short, H | CRPS | 1.63 | 1.00 | $8.92e^{-1}$ | 2.67 | 1.76 | $9.32e^{-1}$ | $7.96e^{-1}$ | $8.88e^{-1}$ | $9.96e^{-1}$ | $7.60e^{-1}$ | $7.76e^{-1}$ | 1.02 | 1.14 | $8.36e^{-1}$ | $9.64e^{-1}$ | $8.88e^{-1}$ | $7.76e^{-1}$ | $7.92e^{-1}$ | $7.96e^{-1}$ | $7.72e^{-1}$ | $8.48e^{-1}$ | P.TST |
| ett1, short, H | Rank | $19.00$ | $16.00$ | $12.00$ | $21.00$ | $20.00$ | $13.00$ | $7.00$ | $11.00$ | $15.00$ | $1.00$ | $4.00$ | $17.00$ | $18.00$ | $8.00$ | $14.00$ | $10.00$ | $3.00$ | $5.00$ | $6.00$ | $2.00$ | $9.00$ | P.TST |
| ett1, short, W | MAPE | 1.00 | 1.00 | 1.08 | 1.04 | $9.76e^{-1}$ | 1.49 | 1.80 | 1.03 | 1.42 | 1.01 | 1.12 | 1.17 | 1.71 | 1.27 | 1.30 | $9.17e^{-1}$ | $9.76e^{-1}$ | $9.66e^{-1}$ | 1.33 | 1.18 | 1.07 | Chr.S |
| ett1, short, W | CRPS | $9.23e^{-1}$ | 1.00 | $9.02e^{-1}$ | $9.44e^{-1}$ | $8.20e^{-1}$ | 2.03 | 1.20 | $8.40e^{-1}$ | 1.10 | $9.56e^{-1}$ | $8.67e^{-1}$ | 1.32 | 1.68 | $9.08e^{-1}$ | 1.17 | $9.38e^{-1}$ | $9.23e^{-1}$ | $9.35e^{-1}$ | $8.17e^{-1}$ | $8.37e^{-1}$ | $7.87e^{-1}$ | Moi.L |
| ett1, short, W | Rank | $10.00$ | $15.00$ | $7.00$ | $13.00$ | $3.00$ | $21.00$ | $18.00$ | $5.00$ | $16.00$ | $14.00$ | $6.00$ | $19.00$ | $20.00$ | $8.00$ | $17.00$ | $12.00$ | $9.00$ | $11.00$ | $2.00$ | $4.00$ | $1.00$ | Moi.L |
| ett2, long, 15T | MAPE | 1.24 | 1.00 | 1.00 | 1.12 | 1.48 | 3.31 | 1.13 | 1.20 | $9.25e^{-1}$ | $9.32e^{-1}$ | $9.69e^{-1}$ | 1.05 | 1.17 | 1.00 | 1.04 | 1.10 | 1.08 | 1.09 | 1.07 | $9.25e^{-1}$ | 1.09 | N-BEATS |
| ett2, long, 15T | CRPS | 1.72 | 1.00 | 1.00 | 1.02 | 1.92 | 1.84 | $6.61e^{-1}$ | $7.70e^{-1}$ | $7.64e^{-1}$ | $5.95e^{-1}$ | $6.12e^{-1}$ | $8.55e^{-1}$ | $8.67e^{-1}$ | $7.21e^{-1}$ | $8.06e^{-1}$ | $7.82e^{-1}$ | $8.12e^{-1}$ | $8.48e^{-1}$ | $6.55e^{-1}$ | $5.90e^{-1}$ | $6.73e^{-1}$ | Moi.B |
| ett2, long, 15T | Rank | $19.00$ | $17.00$ | $16.00$ | $18.00$ | $21.00$ | $20.00$ | $5.00$ | $9.00$ | $8.00$ | $2.00$ | $3.00$ | $14.00$ | $15.00$ | $7.00$ | $11.00$ | $10.00$ | $12.00$ | $13.00$ | $4.00$ | $1.00$ | $6.00$ | Moi.B |
| ett2, long, H | MAPE | 1.15 | 1.00 | 1.11 | 1.27 | 1.13 | 2.19 | 1.28 | 1.11 | 1.14 | 1.12 | 1.11 | 1.24 | 1.37 | $9.79e^{-1}$ | 1.18 | $9.47e^{-1}$ | $9.84e^{-1}$ | 1.01 | $9.74e^{-1}$ | $9.68e^{-1}$ | 1.16 | Chr.S |
| ett2, long, H | CRPS | 1.22 | 1.00 | $9.48e^{-1}$ | 1.17 | 1.15 | $6.83e^{-1}$ | $4.81e^{-1}$ | $4.46e^{-1}$ | $5.44e^{-1}$ | $4.53e^{-1}$ | $4.36e^{-1}$ | $5.75e^{-1}$ | $5.75e^{-1}$ | $4.36e^{-1}$ | $5.64e^{-1}$ | $4.25e^{-1}$ | $4.74e^{-1}$ | $4.84e^{-1}$ | $3.66e^{-1}$ | $3.52e^{-1}$ | $4.25e^{-1}$ | Moi.B |
| ett2, long, H | Rank | $21.00$ | $18.00$ | $17.00$ | $20.00$ | $19.00$ | $16.00$ | $10.00$ | $7.00$ | $12.00$ | $8.00$ | $5.00$ | $15.00$ | $14.00$ | $6.00$ | $13.00$ | $3.00$ | $9.00$ | $11.00$ | $2.00$ | $1.00$ | $4.00$ | Moi.B |
| ett2, medium, 15T | MAPE | 1.12 | 1.00 | 1.00 | $9.88e^{-1}$ | 1.80 | 3.17 | 1.06 | 1.15 | 1.02 | $8.67e^{-1}$ | $9.03e^{-1}$ | 1.10 | 1.05 | $9.45e^{-1}$ | 1.09 | $9.88e^{-1}$ | $9.70e^{-1}$ | $9.70e^{-1}$ | $9.88e^{-1}$ | $9.94e^{-1}$ | 1.01 | P.TST |
| ett2, medium, 15T | CRPS | 1.69 | 1.00 | 1.00 | 1.05 | 2.27 | 1.80 | $7.27e^{-1}$ | 1.16 | $9.65e^{-1}$ | $6.55e^{-1}$ | $6.55e^{-1}$ | 1.06 | $9.30e^{-1}$ | $7.83e^{-1}$ | 1.08 | $8.18e^{-1}$ | $8.53e^{-1}$ | $8.81e^{-1}$ | $7.13e^{-1}$ | $7.06e^{-1}$ | $7.20e^{-1}$ | P.TST |
| ett2, medium, 15T | Rank | $19.00$ | $14.00$ | $13.00$ | $15.00$ | $21.00$ | $20.00$ | $6.00$ | $18.00$ | $12.00$ | $1.00$ | $2.00$ | $16.00$ | $11.00$ | $7.00$ | $17.00$ | $8.00$ | $9.00$ | $10.00$ | $4.00$ | $3.00$ | $5.00$ | P.TST |
| ett2, medium, H | MAPE | 1.20 | 1.00 | 1.31 | 1.16 | 1.20 | 2.11 | 1.14 | 1.17 | $9.10e^{-1}$ | 1.00 | $9.80e^{-1}$ | 1.02 | 1.07 | $9.35e^{-1}$ | 1.07 | $9.40e^{-1}$ | $9.60e^{-1}$ | $9.45e^{-1}$ | $9.55e^{-1}$ | $8.86e^{-1}$ | 1.02 | Moi.B |
| ett2, medium, H | CRPS | 1.22 | 1.00 | 1.02 | 1.18 | 1.13 | 1.17 | $5.06e^{-1}$ | $6.39e^{-1}$ | $5.39e^{-1}$ | $5.19e^{-1}$ | $4.90e^{-1}$ | $6.89e^{-1}$ | $5.98e^{-1}$ | $5.23e^{-1}$ | $6.64e^{-1}$ | $5.64e^{-1}$ | $5.48e^{-1}$ | $5.19e^{-1}$ | $4.73e^{-1}$ | $4.44e^{-1}$ | $5.23e^{-1}$ | Moi.B |
| ett2, medium, H | Rank | $21.00$ | $16.00$ | $17.00$ | $20.00$ | $18.00$ | $19.00$ | $4.00$ | $13.00$ | $9.00$ | $6.00$ | $3.00$ | $15.00$ | $12.00$ | $8.00$ | $14.00$ | $11.00$ | $10.00$ | $5.00$ | $2.00$ | $1.00$ | $7.00$ | Moi.B |
| ett2, short, 15T | MAPE | 1.07 | 1.00 | 1.00 | $8.10e^{-1}$ | 1.05 | 3.78 | 1.04 | $8.82e^{-1}$ | $9.28e^{-1}$ | $7.97e^{-1}$ | $7.84e^{-1}$ | $8.50e^{-1}$ | $8.95e^{-1}$ | $8.56e^{-1}$ | $9.08e^{-1}$ | $8.63e^{-1}$ | $7.97e^{-1}$ | $7.91e^{-1}$ | $9.15e^{-1}$ | $9.41e^{-1}$ | $9.15e^{-1}$ | iTrans. |
| ett2, short, 15T | CRPS | 1.26 | 1.00 | 1.00 | $8.00e^{-1}$ | 1.07 | 3.92 | $8.39e^{-1}$ | $9.67e^{-1}$ | 1.13 | $7.87e^{-1}$ | $7.31e^{-1}$ | 1.06 | 1.02 | $8.00e^{-1}$ | 1.06 | $7.58e^{-1}$ | $7.42e^{-1}$ | $7.49e^{-1}$ | $8.63e^{-1}$ | $8.53e^{-1}$ | $7.76e^{-1}$ | iTrans. |
| ett2, short, 15T | Rank | $20.00$ | $14.00$ | $13.00$ | $7.00$ | $18.00$ | $21.00$ | $9.00$ | $12.00$ | $19.00$ | $6.00$ | $1.00$ | $16.00$ | $15.00$ | $8.00$ | $17.00$ | $4.00$ | $2.00$ | $3.00$ | $11.00$ | $10.00$ | $5.00$ | iTrans. |
| ett2, short, D | MAPE | 1.00 | 1.00 | $8.85e^{-1}$ | 1.04 | 1.01 | 1.49 | $9.28e^{-1}$ | $8.05e^{-1}$ | $8.32e^{-1}$ | 1.02 | $9.34e^{-1}$ | 1.37 | 1.08 | $9.83e^{-1}$ | 1.02 | 1.21 | $8.75e^{-1}$ | $9.47e^{-1}$ | $8.56e^{-1}$ | $9.81e^{-1}$ | $7.45e^{-1}$ | Moi.L |
| ett2, short, D | CRPS | $7.46e^{-1}$ | 1.00 | $6.10e^{-1}$ | $8.00e^{-1}$ | $7.71e^{-1}$ | 1.01 | $4.66e^{-1}$ | $6.34e^{-1}$ | $6.83e^{-1}$ | $6.39e^{-1}$ | $6.29e^{-1}$ | 1.06 | $7.22e^{-1}$ | $5.51e^{-1}$ | $6.68e^{-1}$ | $4.72e^{-1}$ | $4.51e^{-1}$ | $4.42e^{-1}$ | $5.12e^{-1}$ | $6.10e^{-1}$ | $4.80e^{-1}$ | Chr.L |
| ett2, short, D | Rank | $16.00$ | $19.00$ | $8.00$ | $18.00$ | $17.00$ | $20.00$ | $3.00$ | $11.00$ | $14.00$ | $12.00$ | $10.00$ | $21.00$ | $15.00$ | $7.00$ | $13.00$ | $4.00$ | $2.00$ | $1.00$ | $6.00$ | $9.00$ | $5.00$ | Chr.L |
| ett2, short, H | MAPE | 1.24 | 1.00 | 1.12 | 1.18 | 1.20 | 1.48 | 1.14 | 1.21 | $8.76e^{-1}$ | $9.59e^{-1}$ | $9.93e^{-1}$ | $8.97e^{-1}$ | 1.23 | $9.03e^{-1}$ | $8.83e^{-1}$ | $9.10e^{-1}$ | $8.76e^{-1}$ | $8.90e^{-1}$ | $8.97e^{-1}$ | $9.03e^{-1}$ | $9.31e^{-1}$ | N-BEATS |
| ett2, short, H | CRPS | 1.20 | 1.00 | $9.48e^{-1}$ | 1.08 | 1.09 | 1.30 | $8.31e^{-1}$ | 1.36 | $9.68e^{-1}$ | $7.86e^{-1}$ | $7.99e^{-1}$ | $9.32e^{-1}$ | 1.14 | $7.84e^{-1}$ | $9.32e^{-1}$ | $7.63e^{-1}$ | $7.59e^{-1}$ | $7.62e^{-1}$ | $7.36e^{-1}$ | $7.27e^{-1}$ | $8.00e^{-1}$ | Moi.B |
| ett2, short, H | Rank | $19.00$ | $15.00$ | $13.00$ | $16.00$ | $17.00$ | $20.00$ | $10.00$ | $21.00$ | $14.00$ | $7.00$ | $8.00$ | $11.00$ | $18.00$ | $6.00$ | $12.00$ | $5.00$ | $3.00$ | $4.00$ | $2.00$ | $1.00$ | $9.00$ | Moi.B |
| ett2, short, W | MAPE | 1.00 | 1.00 | 1.73 | 1.99 | 1.16 | 8.07 | 1.86 | 7.84 | 4.92 | 1.71 | 8.44 | 2.34 | 2.48 | 1.39 | 1.92 | 1.07 | $9.43e^{-1}$ | $8.52e^{-1}$ | $9.67e^{-1}$ | 1.16 | 1.39 | Chr.L |
| ett2, short, W | CRPS | $7.93e^{-1}$ | 1.00 | $8.05e^{-1}$ | $9.47e^{-1}$ | $6.80e^{-1}$ | 4.31 | $9.47e^{-1}$ | 4.21 | 1.74 | $8.40e^{-1}$ | 4.96 | 1.15 | 1.25 | $6.57e^{-1}$ | 1.08 | $4.54e^{-1}$ | $4.53e^{-1}$ | $4.18e^{-1}$ | $5.14e^{-1}$ | $5.44e^{-1}$ | $6.21e^{-1}$ | Chr.L |
| ett2, short, W | Rank | $9.00$ | $14.00$ | $10.00$ | $12.00$ | $8.00$ | $20.00$ | $13.00$ | $19.00$ | $18.00$ | $11.00$ | $21.00$ | $16.00$ | $17.00$ | $7.00$ | $15.00$ | $3.00$ | $2.00$ | $1.00$ | $4.00$ | $5.00$ | $6.00$ | Chr.L |
| hierarchical_sales, short, D | MAPE | 1.00 | 1.00 | $6.09e^{-1}$ | $7.64e^{-1}$ | $7.27e^{-1}$ | $5.52e^{-1}$ | $5.48e^{-1}$ | 1.16 | $5.16e^{-1}$ | $5.38e^{-1}$ | $5.36e^{-1}$ | $6.71e^{-1}$ | $9.75e^{-1}$ | $5.11e^{-1}$ | $6.85e^{-1}$ | $6.08e^{-1}$ | $5.53e^{-1}$ | $5.52e^{-1}$ | $5.08e^{-1}$ | $4.93e^{-1}$ | $5.23e^{-1}$ | Moi.B |
| hierarchical_sales, short, D | CRPS | $7.37e^{-1}$ | 1.00 | $3.11e^{-1}$ | $4.10e^{-1}$ | $3.94e^{-1}$ | $2.54e^{-1}$ | $2.54e^{-1}$ | $4.53e^{-1}$ | $3.08e^{-1}$ | $2.50e^{-1}$ | $2.53e^{-1}$ | $3.46e^{-1}$ | 2.01 | $2.43e^{-1}$ | $3.50e^{-1}$ | $2.62e^{-1}$ | $2.54e^{-1}$ | $2.54e^{-1}$ | $2.46e^{-1}$ | $2.45e^{-1}$ | $2.44e^{-1}$ | T.FM |
| hierarchical_sales, short, D | Rank | $19.00$ | $20.00$ | $13.00$ | $17.00$ | $16.00$ | $9.00$ | $10.00$ | $18.00$ | $12.00$ | $5.00$ | $6.00$ | $14.00$ | $21.00$ | $1.00$ | $15.00$ | $11.00$ | $8.00$ | $7.00$ | $4.00$ | $3.00$ | $2.00$ | T.FM |
| hierarchical_sales, short, W | MAPE | 1.00 | 1.00 | $7.44e^{-1}$ | $8.14e^{-1}$ | 1.13 | $6.78e^{-1}$ | $5.92e^{-1}$ | $9.68e^{-1}$ | $5.85e^{-1}$ | $6.03e^{-1}$ | $6.15e^{-1}$ | $9.27e^{-1}$ | 2.67 | $5.71e^{-1}$ | $8.56e^{-1}$ | $5.74e^{-1}$ | $6.01e^{-1}$ | $5.92e^{-1}$ | $5.81e^{-1}$ | $5.85e^{-1}$ | $5.97e^{-1}$ | T.FM |
| hierarchical_sales, short, W | CRPS | $8.08e^{-1}$ | 1.00 | $4.71e^{-1}$ | $4.60e^{-1}$ | $6.23e^{-1}$ | $3.68e^{-1}$ | $3.71e^{-1}$ | $5.17e^{-1}$ | $4.26e^{-1}$ | $3.48e^{-1}$ | $3.61e^{-1}$ | $5.65e^{-1}$ | $1.59e^{1}$ | $3.33e^{-1}$ | $4.94e^{-1}$ | $3.56e^{-1}$ | $3.56e^{-1}$ | $3.54e^{-1}$ | $3.51e^{-1}$ | $3.60e^{-1}$ | $3.50e^{-1}$ | T.FM |
| hierarchical_sales, short, W | Rank | $19.00$ | $20.00$ | $14.00$ | $13.00$ | $18.00$ | $10.00$ | $11.00$ | $16.00$ | $12.00$ | $2.00$ | $9.00$ | $17.00$ | $21.00$ | $1.00$ | $15.00$ | $7.00$ | $6.00$ | $5.00$ | $4.00$ | $8.00$ | $3.00$ | T.FM |
| hospital, short, M | MAPE | 1.04 | 1.00 | $8.63e^{-1}$ | $8.24e^{-1}$ | $8.58e^{-1}$ | $9.36e^{-1}$ | $9.48e^{-1}$ | $9.06e^{-1}$ | $8.33e^{-1}$ | $8.97e^{-1}$ | $8.67e^{-1}$ | $8.97e^{-1}$ | $4.38e^{1}$ | $8.50e^{-1}$ | $9.57e^{-1}$ | $8.88e^{-1}$ | $8.84e^{-1}$ | $8.84e^{-1}$ | 1.14 | 1.34 | $9.06e^{-1}$ | A.Th. |
| hospital, short, M | CRPS | 1.38 | 1.00 | $9.54e^{-1}$ | $8.85e^{-1}$ | $8.51e^{-1}$ | $9.92e^{-1}$ | $9.28e^{-1}$ | 2.06 | 1.09 | 1.02 | $8.32e^{-1}$ | 1.22 | $2.90e^{2}$ | $8.38e^{-1}$ | 1.12 | $8.99e^{-1}$ | $8.94e^{-1}$ | $9.04e^{-1}$ | 1.19 | 1.38 | $9.17e^{-1}$ | iTrans. |
| hospital, short, M | Rank | $19.00$ | $12.00$ | $10.00$ | $4.00$ | $3.00$ | $11.00$ | $9.00$ | $20.00$ | $14.00$ | $13.00$ | $1.00$ | $17.00$ | $21.00$ | $2.00$ | $15.00$ | $6.00$ | $5.00$ | $7.00$ | $16.00$ | $18.00$ | $8.00$ | iTrans. |
| jena_weather, long, 10T | MAPE | 1.09 | 1.00 | 1.00 | 5.57 | 1.40 | 1.01 | 1.41 | 1.66 | 3.73 | 1.14 | 1.08 | 1.71 | 1.68 | 1.50 | 1.04 | 1.41 | 1.13 | 1.12 | $9.53e^{-1}$ | 1.19 | 1.97 | Moi.S |
| jena_weather, long, 10T | CRPS | 3.52 | 1.00 | 1.00 | 1.39 | $9.28e^{-1}$ | $4.70e^{-1}$ | $1.70e^{-1}$ | $2.79e^{-1}$ | $4.41e^{-1}$ | $2.18e^{-1}$ | $2.10e^{-1}$ | $3.05e^{-1}$ | $2.75e^{-1}$ | $2.26e^{-1}$ | $2.18e^{-1}$ | $3.15e^{-1}$ | $2.62e^{-1}$ | $2.53e^{-1}$ | $2.08e^{-1}$ | $1.91e^{-1}$ | $2.40e^{-1}$ | TFT |
| jena_weather, long, 10T | Rank | $21.00$ | $19.00$ | $18.00$ | $20.00$ | $17.00$ | $16.00$ | $1.00$ | $12.00$ | $15.00$ | $5.00$ | $4.00$ | $13.00$ | $11.00$ | $7.00$ | $6.00$ | $14.00$ | $10.00$ | $9.00$ | $3.00$ | $2.00$ | $8.00$ | TFT |
| jena_weather, long, H | MAPE | 1.27 | 1.00 | 1.33 | $3.13e^{1}$ | 2.68 | 4.54 | 3.63 | 3.74 | 5.35 | 1.91 | 1.72 | 5.45 | 4.76 | 2.88 | 1.82 | $8.34e^{-1}$ | $8.50e^{-1}$ | $8.29e^{-1}$ | 1.44 | 2.11 | 1.48 | Chr.L |
| jena_weather, long, H | CRPS | 1.91 | 1.00 | $3.85e^{-1}$ | 2.16 | $1.14e^{9}$ | $3.29e^{-1}$ | $1.51e^{-1}$ | $2.04e^{-1}$ | $2.19e^{-1}$ | $1.27e^{-1}$ | $1.27e^{-1}$ | $2.32e^{-1}$ | $1.92e^{-1}$ | $1.49e^{-1}$ | $1.36e^{-1}$ | $1.21e^{-1}$ | $1.25e^{-1}$ | $1.21e^{-1}$ | $1.06e^{-1}$ | $9.97e^{-2}$ | $9.60e^{-2}$ | Moi.L |
| jena_weather, long, H | Rank | $19.00$ | $18.00$ | $17.00$ | $20.00$ | $21.00$ | $16.00$ | $11.00$ | $13.00$ | $14.00$ | $8.00$ | $7.00$ | $15.00$ | $12.00$ | $10.00$ | $9.00$ | $5.00$ | $6.00$ | $4.00$ | $3.00$ | $2.00$ | $1.00$ | Moi.L |
| jena_weather, medium, 10T | MAPE | 1.06 | 1.00 | 1.00 | 3.03 | 1.84 | $7.13e^{-1}$ | 1.16 | 1.64 | $9.45e^{-1}$ | $8.67e^{-1}$ | $8.85e^{-1}$ | 1.50 | 1.36 | 1.16 | $9.92e^{-1}$ | 3.53 | 2.40 | $9.62e^{-1}$ | $9.13e^{-1}$ | 1.09 | 1.56 | DeepAR |
| jena_weather, medium, 10T | CRPS | 3.18 | 1.00 | 1.00 | 1.26 | $7.58e^{-1}$ | $2.65e^{-1}$ | $1.88e^{-1}$ | $3.94e^{-1}$ | $3.22e^{-1}$ | $2.34e^{-1}$ | $2.27e^{-1}$ | $3.55e^{-1}$ | $3.33e^{-1}$ | $2.42e^{-1}$ | $2.44e^{-1}$ | $3.23e^{-1}$ | $2.74e^{-1}$ | $2.71e^{-1}$ | $2.23e^{-1}$ | $1.99e^{-1}$ | $2.46e^{-1}$ | TFT |
| jena_weather, medium, 10T | Rank | $21.00$ | $19.00$ | $18.00$ | $20.00$ | $17.00$ | $9.00$ | $1.00$ | $16.00$ | $12.00$ | $5.00$ | $4.00$ | $15.00$ | $14.00$ | $6.00$ | $7.00$ | $13.00$ | $11.00$ | $10.00$ | $3.00$ | $2.00$ | $8.00$ | TFT |
| jena_weather, medium, H | MAPE | 1.20 | 1.00 | $9.88e^{-1}$ | $1.72e^{1}$ | 1.30 | 1.08 | 5.46 | 4.56 | 5.12 | 2.24 | 2.52 | 3.45 | 2.27 | 1.91 | 1.66 | $7.88e^{-1}$ | $7.29e^{-1}$ | $8.41e^{-1}$ | 1.32 | 1.65 | 1.19 | Chr.B |
| jena_weather, medium, H | CRPS | 1.91 | 1.00 | $4.34e^{-1}$ | 1.71 | $7.65e^{3}$ | $1.61e^{-1}$ | $1.49e^{-1}$ | $2.49e^{-1}$ | $1.99e^{-1}$ | $1.41e^{-1}$ | $1.24e^{-1}$ | $1.92e^{-1}$ | $1.71e^{-1}$ | $1.34e^{-1}$ | $1.40e^{-1}$ | $1.41e^{-1}$ | $1.44e^{-1}$ | $1.46e^{-1}$ | $1.09e^{-1}$ | $1.13e^{-1}$ | $1.28e^{-1}$ | Moi.S |
| jena_weather, medium, H | Rank | $20.00$ | $18.00$ | $17.00$ | $19.00$ | $21.00$ | $12.00$ | $11.00$ | $16.00$ | $15.00$ | $7.00$ | $3.00$ | $14.00$ | $13.00$ | $5.00$ | $6.00$ | $8.00$ | $9.00$ | $10.00$ | $1.00$ | $2.00$ | $4.00$ | Moi.S |
| jena_weather, short, 10T | MAPE | $4.77e^{-1}$ | 1.00 | 1.00 | $7.77e^{-1}$ | $4.64e^{-1}$ | $4.57e^{-1}$ | $5.79e^{-1}$ | $8.29e^{-1}$ | 1.08 | $4.42e^{-1}$ | $5.51e^{-1}$ | 1.04 | 2.55 | $4.77e^{-1}$ | $6.48e^{-1}$ | $4.69e^{-1}$ | $4.14e^{-1}$ | $4.29e^{-1}$ | $4.46e^{-1}$ | $4.97e^{-1}$ | $4.55e^{-1}$ | Chr.B |
| jena_weather, short, 10T | CRPS | 1.86 | 1.00 | 1.00 | $8.39e^{-1}$ | $5.30e^{-1}$ | $4.05e^{-1}$ | $4.46e^{-1}$ | 1.19 | $6.71e^{-1}$ | $4.12e^{-1}$ | $2.55e^{-1}$ | $8.32e^{-1}$ | 8.19 | $2.34e^{-1}$ | $3.92e^{-1}$ | $3.03e^{-1}$ | $2.81e^{-1}$ | $2.82e^{-1}$ | $3.22e^{-1}$ | $3.25e^{-1}$ | $2.99e^{-1}$ | T.FM |
| jena_weather, short, 10T | Rank | $20.00$ | $18.00$ | $17.00$ | $16.00$ | $13.00$ | $10.00$ | $12.00$ | $19.00$ | $14.00$ | $11.00$ | $2.00$ | $15.00$ | $21.00$ | $1.00$ | $9.00$ | $6.00$ | $3.00$ | $4.00$ | $7.00$ | $8.00$ | $5.00$ | T.FM |
| jena_weather, short, D | MAPE | 1.00 | 1.00 | $5.99e^{-1}$ | $6.50e^{-1}$ | $7.34e^{-1}$ | $3.71e^{-1}$ | $8.98e^{-1}$ | $9.27e^{-1}$ | 1.03 | $5.48e^{-1}$ | $9.15e^{-1}$ | $7.57e^{-1}$ | $9.44e^{-1}$ | $4.20e^{-1}$ | $3.79e^{-1}$ | $3.64e^{-1}$ | $3.84e^{-1}$ | $3.84e^{-1}$ | $4.26e^{-1}$ | $4.36e^{-1}$ | $4.73e^{-1}$ | Chr.S |
| jena_weather, short, D | CRPS | $7.10e^{-1}$ | 1.00 | $2.70e^{-1}$ | $2.75e^{-1}$ | $2.49e^{-1}$ | $2.10e^{-1}$ | $2.33e^{-1}$ | 1.12 | $6.50e^{-1}$ | $1.77e^{-1}$ | $3.16e^{-1}$ | $2.46e^{-1}$ | $2.38e^{-1}$ | $1.98e^{-1}$ | $2.48e^{-1}$ | $1.70e^{-1}$ | $1.65e^{-1}$ | $1.64e^{-1}$ | $1.62e^{-1}$ | $1.83e^{-1}$ | $1.55e^{-1}$ | Moi.L |
| jena_weather, short, D | Rank | $19.00$ | $20.00$ | $15.00$ | $16.00$ | $14.00$ | $9.00$ | $10.00$ | $21.00$ | $18.00$ | $6.00$ | $17.00$ | $12.00$ | $11.00$ | $8.00$ | $13.00$ | $5.00$ | $4.00$ | $3.00$ | $2.00$ | $7.00$ | $1.00$ | Moi.L |
| jena_weather, short, H | MAPE | $9.73e^{-1}$ | 1.00 | 1.08 | $1.09e^{1}$ | 1.05 | 2.27 | 1.34 | 3.88 | 7.29 | 1.32 | 1.71 | 2.81 | 1.88 | 1.96 | 1.42 | $9.65e^{-1}$ | $8.01e^{-1}$ | $8.69e^{-1}$ | 1.32 | 1.40 | 1.14 | Chr.B |
| jena_weather, short, H | CRPS | 1.78 | 1.00 | $8.27e^{-1}$ | 1.71 | $9.13e^{-1}$ | 4.04 | $2.79e^{-1}$ | $8.79e^{-1}$ | $5.64e^{-1}$ | $2.90e^{-1}$ | $3.07e^{-1}$ | $4.97e^{-1}$ | $3.95e^{-1}$ | $2.77e^{-1}$ | $3.33e^{-1}$ | $2.69e^{-1}$ | $2.67e^{-1}$ | $2.64e^{-1}$ | $2.60e^{-1}$ | $2.49e^{-1}$ | $2.50e^{-1}$ | Moi.B |
| jena_weather, short, H | Rank | $20.00$ | $18.00$ | $15.00$ | $19.00$ | $17.00$ | $21.00$ | $8.00$ | $16.00$ | $14.00$ | $9.00$ | $10.00$ | $13.00$ | $12.00$ | $7.00$ | $11.00$ | $6.00$ | $5.00$ | $4.00$ | $3.00$ | $1.00$ | $2.00$ | Moi.B |
| kdd_cup_2018, long, H | MAPE | $8.15e^{-1}$ | 1.00 | $8.59e^{-1}$ | $9.63e^{-1}$ | 1.00 | 3.73 | $7.40e^{-1}$ | $8.52e^{-1}$ | $6.13e^{-1}$ | $7.06e^{-1}$ | $7.32e^{-1}$ | 1.01 | 8.15 | $8.74e^{-1}$ | $9.04e^{-1}$ | 1.06 | $8.81e^{-1}$ | $9.19e^{-1}$ | $7.36e^{-1}$ | $7.85e^{-1}$ | $8.44e^{-1}$ | N-BEATS |
| kdd_cup_2018, long, H | CRPS | 1.00 | 1.00 | $8.40e^{-1}$ | $7.76e^{-1}$ | 1.00 | $8.72e^{-1}$ | $4.02e^{-1}$ | $4.22e^{-1}$ | $5.28e^{-1}$ | $3.82e^{-1}$ | $3.83e^{-1}$ | $4.66e^{-1}$ | $9.44e^{-2}$ | $4.30e^{-1}$ | $4.62e^{-1}$ | $5.70e^{-1}$ | $4.99e^{-1}$ | $5.09e^{-1}$ | $3.74e^{-1}$ | $3.64e^{-1}$ | $3.78e^{-1}$ | C.former |
| kdd_cup_2018, long, H | Rank | $20.00$ | $21.00$ | $17.00$ | $16.00$ | $19.00$ | $18.00$ | $7.00$ | $8.00$ | $14.00$ | $5.00$ | $6.00$ | $11.00$ | $1.00$ | $9.00$ | $10.00$ | $15.00$ | $12.00$ | $13.00$ | $3.00$ | $2.00$ | $4.00$ | C.former |
| kdd_cup_2018, medium, H | MAPE | $9.27e^{-1}$ | 1.00 | $9.76e^{-1}$ | $7.38e^{-1}$ | 1.00 | $8.60e^{-1}$ | $6.83e^{-1}$ | $6.34e^{-1}$ | $5.18e^{-1}$ | $5.90e^{-1}$ | $6.28e^{-1}$ | $7.01e^{-1}$ | 4.43 | $7.32e^{-1}$ | $6.65e^{-1}$ | $8.96e^{-1}$ | $8.23e^{-1}$ | $8.23e^{-1}$ | $5.90e^{-1}$ | $5.82e^{-1}$ | $6.28e^{-1}$ | N-BEATS |
| kdd_cup_2018, medium, H | CRPS | 1.05 | 1.00 | $8.97e^{-1}$ | $8.34e^{-1}$ | 1.00 | $4.66e^{-1}$ | $4.97e^{-1}$ | $4.91e^{-1}$ | $6.95e^{-1}$ | $4.66e^{-1}$ | $4.60e^{-1}$ | $5.77e^{-1}$ | $2.50e^{-1}$ | $5.19e^{-1}$ | $5.53e^{-1}$ | $7.44e^{-1}$ | $7.00e^{-1}$ | $6.93e^{-1}$ | $4.69e^{-1}$ | $4.54e^{-1}$ | $4.59e^{-1}$ | C.former |
| kdd_cup_2018, medium, H | Rank | $21.00$ | $20.00$ | $18.00$ | $17.00$ | $19.00$ | $5.00$ | $9.00$ | $8.00$ | $14.00$ | $6.00$ | $4.00$ | $12.00$ | $1.00$ | $10.00$ | $11.00$ | $16.00$ | $15.00$ | $13.00$ | $7.00$ | $2.00$ | $3.00$ | C.former |
| kdd_cup_2018, short, D | MAPE | 1.00 | 1.00 | $9.42e^{-1}$ | $9.69e^{-1}$ | 1.14 | $9.90e^{-1}$ | $8.18e^{-1}$ | 2.33 | $8.61e^{-1}$ | $8.93e^{-1}$ | $8.98e^{-1}$ | $9.86e^{-1}$ | 8.40 | $9.20e^{-1}$ | 1.18 | $8.49e^{-1}$ | $8.32e^{-1}$ | $8.32e^{-1}$ | $8.90e^{-1}$ | $8.71e^{-1}$ | $9.07e^{-1}$ | TFT |
| kdd_cup_2018, short, D | CRPS | $7.60e^{-1}$ | 1.00 | $4.43e^{-1}$ | $5.17e^{-1}$ | $1.16e^{1}$ | $4.31e^{-1}$ | $4.28e^{-1}$ | 1.14 | $5.96e^{-1}$ | $4.52e^{-1}$ | $4.52e^{-1}$ | $5.43e^{-1}$ | 3.92 | $4.28e^{-1}$ | $6.20e^{-1}$ | $5.77e^{-1}$ | $5.66e^{-1}$ | $5.65e^{-1}$ | $4.36e^{-1}$ | $4.31e^{-1}$ | $4.34e^{-1}$ | TFT |
| kdd_cup_2018, short, D | Rank | $17.00$ | $18.00$ | $7.00$ | $10.00$ | $21.00$ | $3.00$ | $1.00$ | $19.00$ | $15.00$ | $9.00$ | $8.00$ | $11.00$ | $20.00$ | $2.00$ | $16.00$ | $14.00$ | $13.00$ | $12.00$ | $6.00$ | $4.00$ | $5.00$ | TFT |
| kdd_cup_2018, short, H | MAPE | $9.51e^{-1}$ | 1.00 | 1.00 | $8.38e^{-1}$ | 1.13 | $4.78e^{-1}$ | $7.39e^{-1}$ | $8.94e^{-1}$ | 1.14 | $8.38e^{-1}$ | $9.01e^{-1}$ | $8.38e^{-1}$ | $1.00e^{1}$ | $7.96e^{-1}$ | 1.06 | $8.10e^{-1}$ | $7.82e^{-1}$ | $7.75e^{-1}$ | $5.87e^{-1}$ | $6.17e^{-1}$ | $6.07e^{-1}$ | DeepAR |
| kdd_cup_2018, short, H | CRPS | 1.01 | 1.00 | 1.00 | $9.50e^{-1}$ | 2.08 | $9.25e^{-1}$ | $8.35e^{-1}$ | $8.53e^{-1}$ | 1.22 | $8.18e^{-1}$ | $8.26e^{-1}$ | 1.04 | 5.49 | $7.98e^{-1}$ | 1.09 | $8.21e^{-1}$ | $8.21e^{-1}$ | $8.19e^{-1}$ | $7.23e^{-1}$ | $6.96e^{-1}$ | $7.12e^{-1}$ | Moi.B |
| kdd_cup_2018, short, H | Rank | $16.00$ | $15.00$ | $14.00$ | $13.00$ | $20.00$ | $12.00$ | $10.00$ | $11.00$ | $19.00$ | $5.00$ | $9.00$ | $17.00$ | $21.00$ | $4.00$ | $18.00$ | $8.00$ | $7.00$ | $6.00$ | $3.00$ | $1.00$ | $2.00$ | Moi.B |
| loop_seattle, long, 5T | MAPE | 1.49 | 1.00 | 1.00 | 1.10 | 1.47 | 1.81 | $8.30e^{-1}$ | 1.08 | $9.00e^{-1}$ | 1.07 | 1.08 | 1.05 | $6.65e^{-1}$ | 1.37 | $8.30e^{-1}$ | 1.40 | 1.36 | 1.36 | $7.52e^{-1}$ | $7.78e^{-1}$ | $8.13e^{-1}$ | C.former |
| loop_seattle, long, 5T | CRPS | 2.64 | 1.00 | 1.00 | 1.69 | 1.75 | 1.34 | $6.41e^{-1}$ | $7.37e^{-1}$ | $8.54e^{-1}$ | $6.96e^{-1}$ | $6.77e^{-1}$ | $9.56e^{-1}$ | $1.56e^{-1}$ | 1.08 | $8.76e^{-1}$ | 1.09 | 1.04 | 1.04 | $5.72e^{-1}$ | $5.69e^{-1}$ | $5.89e^{-1}$ | C.former |
| loop_seattle, long, 5T | Rank | $21.00$ | $13.00$ | $12.00$ | $19.00$ | $20.00$ | $18.00$ | $5.00$ | $8.00$ | $9.00$ | $7.00$ | $6.00$ | $11.00$ | $1.00$ | $16.00$ | $10.00$ | $17.00$ | $14.00$ | $15.00$ | $3.00$ | $2.00$ | $4.00$ | C.former |
| loop_seattle, long, H | MAPE | 1.39 | 1.00 | 1.28 | 1.01 | 1.26 | $5.91e^{-1}$ | $5.74e^{-1}$ | $7.87e^{-1}$ | $5.45e^{-1}$ | $6.60e^{-1}$ | $7.06e^{-1}$ | $6.09e^{-1}$ | $2.28e^{-1}$ | $8.60e^{-1}$ | $6.85e^{-1}$ | $6.55e^{-1}$ | $6.21e^{-1}$ | $6.21e^{-1}$ | $9.70e^{-1}$ | $8.47e^{-1}$ | $6.89e^{-1}$ | C.former |
| loop_seattle, long, H | CRPS | 2.27 | 1.00 | $7.88e^{-1}$ | 1.91 | 1.74 | $2.79e^{-1}$ | $2.79e^{-1}$ | $3.55e^{-1}$ | $3.26e^{-1}$ | $2.80e^{-1}$ | $2.81e^{-1}$ | $3.58e^{-1}$ | $4.24e^{-2}$ | $3.95e^{-1}$ | $3.60e^{-1}$ | $3.64e^{-1}$ | $3.33e^{-1}$ | $3.32e^{-1}$ | $3.73e^{-1}$ | $3.29e^{-1}$ | $3.15e^{-1}$ | C.former |
| loop_seattle, long, H | Rank | $21.00$ | $18.00$ | $17.00$ | $20.00$ | $19.00$ | $2.00$ | $3.00$ | $11.00$ | $7.00$ | $4.00$ | $5.00$ | $12.00$ | $1.00$ | $16.00$ | $13.00$ | $14.00$ | $10.00$ | $9.00$ | $15.00$ | $8.00$ | $6.00$ | C.former |
| loop_seattle, medium, 5T | MAPE | 1.62 | 1.00 | 1.00 | 1.37 | 1.61 | 1.01 | 1.14 | $9.96e^{-1}$ | $9.96e^{-1}$ | 1.08 | 1.20 | 1.02 | 2.71 | 1.41 | 1.40 | 1.42 | 1.41 | 1.42 | $7.72e^{-1}$ | $8.26e^{-1}$ | $8.53e^{-1}$ | Moi.S |
| loop_seattle, medium, 5T | CRPS | 2.66 | 1.00 | 1.00 | 1.95 | 1.93 | $9.59e^{-1}$ | $7.45e^{-1}$ | $7.49e^{-1}$ | $9.59e^{-1}$ | $7.70e^{-1}$ | $7.75e^{-1}$ | 1.02 | $4.96e^{-1}$ | 1.23 | 1.23 | 1.42 | 1.43 | 1.43 | $6.15e^{-1}$ | $6.24e^{-1}$ | $6.50e^{-1}$ | C.former |
| loop_seattle, medium, 5T | Rank | $21.00$ | $12.00$ | $11.00$ | $20.00$ | $19.00$ | $9.00$ | $5.00$ | $6.00$ | $10.00$ | $7.00$ | $8.00$ | $13.00$ | $1.00$ | $14.00$ | $15.00$ | $16.00$ | $17.00$ | $18.00$ | $2.00$ | $3.00$ | $4.00$ | C.former |
| loop_seattle, medium, H | MAPE | 1.60 | 1.00 | 1.11 | $9.40e^{-1}$ | 1.39 | $7.06e^{-1}$ | $5.92e^{-1}$ | $7.96e^{-1}$ | $6.27e^{-1}$ | $7.16e^{-1}$ | $7.81e^{-1}$ | $7.21e^{-1}$ | $2.28e^{-1}$ | $8.76e^{-1}$ | $6.37e^{-1}$ | $6.97e^{-1}$ | $6.72e^{-1}$ | $6.72e^{-1}$ | $9.45e^{-1}$ | $8.61e^{-1}$ | $7.16e^{-1}$ | C.former |
| loop_seattle, medium, H | CRPS | 2.22 | 1.00 | $7.48e^{-1}$ | 1.89 | 1.65 | $3.49e^{-1}$ | $3.34e^{-1}$ | $4.25e^{-1}$ | $4.30e^{-1}$ | $3.44e^{-1}$ | $3.31e^{-1}$ | $5.05e^{-1}$ | $6.75e^{-2}$ | $4.85e^{-1}$ | $4.38e^{-1}$ | $4.30e^{-1}$ | $4.08e^{-1}$ | $4.09e^{-1}$ | $4.21e^{-1}$ | $3.76e^{-1}$ | $3.71e^{-1}$ | C.former |
| loop_seattle, medium, H | Rank | $21.00$ | $18.00$ | $17.00$ | $20.00$ | $19.00$ | $5.00$ | $3.00$ | $11.00$ | $13.00$ | $4.00$ | $2.00$ | $16.00$ | $1.00$ | $15.00$ | $14.00$ | $12.00$ | $8.00$ | $9.00$ | $10.00$ | $7.00$ | $6.00$ | C.former |

**Table 22: Results on all dataset configs for GIFT-Eval | Table 3/3. The best results across each row are bolded, while second best results are underlined.**
| Dataset,term,frequency | Metric | Nv. | S.Nv. | A.Ar. | A.Th. | A.ETS | DeepAR | TFT | TiDE | N-BEATS | P.TST | iTrans. | DLin. | C.former | T.FM | Vis.TS | Chr.S | Chr.B | Chr.L | Moi.S | Moi.B | Moi.L | Best |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| loop_seattle, short, 5T | MAPE | 1.06 | 1.00 | 1.00 | $9.00e^{-1}$ | 1.02 | $9.85e^{-1}$ | $9.38e^{-1}$ | 1.09 | $9.62e^{-1}$ | 1.04 | 1.07 | 1.12 | 2.21 | 1.19 | 1.15 | $9.85e^{-1}$ | $9.77e^{-1}$ | $9.77e^{-1}$ | $8.69e^{-1}$ | $8.38e^{-1}$ | $8.31e^{-1}$ | Moi.L |
| loop_seattle, short, 5T | CRPS | 1.44 | 1.00 | 1.00 | 1.01 | 1.03 | $8.91e^{-1}$ | $8.01e^{-1}$ | $9.01e^{-1}$ | $9.89e^{-1}$ | $8.12e^{-1}$ | $8.12e^{-1}$ | 1.22 | 5.64 | $9.29e^{-1}$ | 1.08 | $8.66e^{-1}$ | $8.61e^{-1}$ | $8.64e^{-1}$ | $6.52e^{-1}$ | $6.44e^{-1}$ | $6.51e^{-1}$ | Moi.B |
| loop_seattle, short, 5T | Rank | $20.00$ | $15.00$ | $14.00$ | $16.00$ | $17.00$ | $10.00$ | $4.00$ | $11.00$ | $13.00$ | $6.00$ | $5.00$ | $19.00$ | $21.00$ | $12.00$ | $18.00$ | $9.00$ | $7.00$ | $8.00$ | $3.00$ | $1.00$ | $2.00$ | Moi.B |
| loop_seattle, short, D | MAPE | 1.00 | 1.00 | $8.50e^{-1}$ | $8.01e^{-1}$ | $7.07e^{-1}$ | $5.58e^{-1}$ | $5.32e^{-1}$ | $5.08e^{-1}$ | $4.96e^{-1}$ | $5.16e^{-1}$ | $5.06e^{-1}$ | $4.99e^{-1}$ | $8.21e^{-1}$ | $4.88e^{-1}$ | $5.29e^{-1}$ | $5.56e^{-1}$ | $5.04e^{-1}$ | $5.04e^{-1}$ | $5.36e^{-1}$ | $5.17e^{-1}$ | $4.94e^{-1}$ | T.FM |
| loop_seattle, short, D | CRPS | $7.86e^{-1}$ | 1.00 | $5.93e^{-1}$ | $5.48e^{-1}$ | $4.46e^{-1}$ | $3.97e^{-1}$ | $3.68e^{-1}$ | $3.37e^{-1}$ | $4.08e^{-1}$ | $3.53e^{-1}$ | $3.48e^{-1}$ | $4.08e^{-1}$ | 2.77 | $3.24e^{-1}$ | $4.29e^{-1}$ | $3.68e^{-1}$ | $3.40e^{-1}$ | $3.40e^{-1}$ | $3.53e^{-1}$ | $3.53e^{-1}$ | $3.31e^{-1}$ | T.FM |
| loop_seattle, short, D | Rank | $19.00$ | $20.00$ | $18.00$ | $17.00$ | $16.00$ | $12.00$ | $11.00$ | $3.00$ | $13.00$ | $7.00$ | $6.00$ | $14.00$ | $21.00$ | $1.00$ | $15.00$ | $10.00$ | $4.00$ | $5.00$ | $9.00$ | $8.00$ | $2.00$ | T.FM |
| loop_seattle, short, H | MAPE | 1.76 | 1.00 | 1.00 | 1.02 | 1.57 | $7.49e^{-1}$ | $8.65e^{-1}$ | $9.53e^{-1}$ | $8.19e^{-1}$ | $9.06e^{-1}$ | 1.04 | $8.54e^{-1}$ | $3.06e^{-1}$ | 1.04 | $8.54e^{-1}$ | $7.84e^{-1}$ | $7.60e^{-1}$ | $7.60e^{-1}$ | 1.19 | 1.08 | $8.95e^{-1}$ | C.former |
| loop_seattle, short, H | CRPS | 1.69 | 1.00 | 1.00 | 1.53 | 1.45 | $6.07e^{-1}$ | $6.74e^{-1}$ | $7.35e^{-1}$ | $8.33e^{-1}$ | $7.07e^{-1}$ | $6.91e^{-1}$ | $9.14e^{-1}$ | 1.44 | $7.59e^{-1}$ | $8.96e^{-1}$ | $6.41e^{-1}$ | $6.11e^{-1}$ | $6.10e^{-1}$ | $7.83e^{-1}$ | $7.20e^{-1}$ | $6.49e^{-1}$ | DeepAR |
| loop_seattle, short, H | Rank | $21.00$ | $17.00$ | $16.00$ | $20.00$ | $19.00$ | $1.00$ | $6.00$ | $10.00$ | $13.00$ | $8.00$ | $7.00$ | $15.00$ | $18.00$ | $11.00$ | $14.00$ | $4.00$ | $3.00$ | $2.00$ | $12.00$ | $9.00$ | $5.00$ | DeepAR |
| m4_daily, short, D | MAPE | 1.00 | 1.00 | $9.41e^{-1}$ | 1.03 | $9.29e^{-1}$ | 1.38 | 1.03 | 1.16 | 1.03 | 1.01 | 1.05 | 1.10 | $4.55e^{1}$ | $9.00e^{-1}$ | 1.14 | $8.74e^{-1}$ | $9.12e^{-1}$ | $9.05e^{-1}$ | 1.10 | 1.36 | $9.45e^{-1}$ | Chr.S |
| m4_daily, short, D | CRPS | $9.21e^{-1}$ | 1.00 | $8.72e^{-1}$ | $8.87e^{-1}$ | 1.08 | 1.15 | $8.60e^{-1}$ | 1.21 | 1.08 | $8.60e^{-1}$ | $8.45e^{-1}$ | 1.09 | $2.76e^{2}$ | $8.04e^{-1}$ | 1.25 | $8.11e^{-1}$ | $8.34e^{-1}$ | $8.42e^{-1}$ | $9.06e^{-1}$ | 1.22 | $8.15e^{-1}$ | T.FM |
| m4_daily, short, D | Rank | $12.00$ | $13.00$ | $9.00$ | $10.00$ | $14.00$ | $17.00$ | $8.00$ | $18.00$ | $15.00$ | $7.00$ | $6.00$ | $16.00$ | $21.00$ | $1.00$ | $20.00$ | $2.00$ | $4.00$ | $5.00$ | $11.00$ | $19.00$ | $3.00$ | T.FM |
| m4_hourly, short, H | MAPE | 2.42 | 1.00 | 1.04 | 1.43 | 1.88 | 1.40 | 1.87 | 1.25 | $8.33e^{-1}$ | 1.07 | 1.13 | 1.66 | $1.15e^{2}$ | $6.54e^{-1}$ | $8.08e^{-1}$ | $5.50e^{-1}$ | $5.44e^{-1}$ | $5.44e^{-1}$ | $8.14e^{-1}$ | $8.21e^{-1}$ | $6.79e^{-1}$ | Chr.L |
| m4_hourly, short, H | CRPS | 3.44 | 1.00 | $8.53e^{-1}$ | 1.05 | 1.76 | 3.37 | 1.02 | 1.21 | 1.27 | $9.87e^{-1}$ | 1.04 | 1.40 | $1.07e^{2}$ | $5.32e^{-1}$ | $9.97e^{-1}$ | $6.41e^{-1}$ | $6.18e^{-1}$ | $6.53e^{-1}$ | $6.81e^{-1}$ | $6.78e^{-1}$ | $5.62e^{-1}$ | T.FM |
| m4_hourly, short, H | Rank | $20.00$ | $11.00$ | $8.00$ | $14.00$ | $18.00$ | $19.00$ | $12.00$ | $15.00$ | $16.00$ | $9.00$ | $13.00$ | $17.00$ | $21.00$ | $1.00$ | $10.00$ | $4.00$ | $3.00$ | $5.00$ | $7.00$ | $6.00$ | $2.00$ | T.FM |
| m4_monthly, short, M | MAPE | $9.90e^{-1}$ | 1.00 | $8.39e^{-1}$ | $8.02e^{-1}$ | $8.80e^{-1}$ | 1.50 | 1.10 | 1.02 | $8.59e^{-1}$ | $8.75e^{-1}$ | 1.00 | $9.38e^{-1}$ | 6.61 | $8.70e^{-1}$ | 1.06 | $8.65e^{-1}$ | $8.85e^{-1}$ | $8.80e^{-1}$ | 1.28 | 1.68 | $9.38e^{-1}$ | A.Th. |
| m4_monthly, short, M | CRPS | 1.02 | 1.00 | $7.77e^{-1}$ | $7.78e^{-1}$ | $7.92e^{-1}$ | 1.46 | $8.97e^{-1}$ | 1.07 | $9.68e^{-1}$ | $8.10e^{-1}$ | $9.05e^{-1}$ | 1.02 | $4.09e^{1}$ | $7.69e^{-1}$ | 1.10 | $8.17e^{-1}$ | $8.25e^{-1}$ | $8.17e^{-1}$ | 1.03 | 1.29 | $8.10e^{-1}$ | T.FM |
| m4_monthly, short, M | Rank | $15.00$ | $13.00$ | $2.00$ | $3.00$ | $4.00$ | $20.00$ | $10.00$ | $17.00$ | $12.00$ | $6.00$ | $11.00$ | $14.00$ | $21.00$ | $1.00$ | $18.00$ | $8.00$ | $9.00$ | $7.00$ | $16.00$ | $19.00$ | $5.00$ | T.FM |
| m4_quarterly, short, Q | MAPE | $9.30e^{-1}$ | 1.00 | $8.59e^{-1}$ | $8.38e^{-1}$ | $8.59e^{-1}$ | $9.44e^{-1}$ | $9.37e^{-1}$ | 1.13 | $8.38e^{-1}$ | $9.37e^{-1}$ | $9.08e^{-1}$ | $9.86e^{-1}$ | 9.93 | $8.80e^{-1}$ | $9.37e^{-1}$ | $8.24e^{-1}$ | $8.10e^{-1}$ | $8.10e^{-1}$ | 1.04 | 1.43 | $8.87e^{-1}$ | Chr.B |
| m4_quarterly, short, Q | CRPS | $9.51e^{-1}$ | 1.00 | $8.23e^{-1}$ | $7.97e^{-1}$ | $7.98e^{-1}$ | $8.41e^{-1}$ | $8.37e^{-1}$ | 1.02 | $9.72e^{-1}$ | $8.35e^{-1}$ | $7.97e^{-1}$ | 1.11 | $1.20e^{2}$ | $8.53e^{-1}$ | 1.05 | $8.46e^{-1}$ | $8.40e^{-1}$ | $8.40e^{-1}$ | $9.32e^{-1}$ | 1.13 | $7.88e^{-1}$ | Moi.L |
| m4_quarterly, short, Q | Rank | $14.00$ | $16.00$ | $5.00$ | $2.00$ | $4.00$ | $10.00$ | $7.00$ | $17.00$ | $15.00$ | $6.00$ | $3.00$ | $19.00$ | $21.00$ | $12.00$ | $18.00$ | $11.00$ | $8.00$ | $9.00$ | $13.00$ | $20.00$ | $1.00$ | Moi.L |
| m4_weekly, short, W | MAPE | 1.00 | 1.00 | $9.25e^{-1}$ | $9.98e^{-1}$ | $9.35e^{-1}$ | 1.13 | $9.47e^{-1}$ | 1.12 | $7.18e^{-1}$ | $7.76e^{-1}$ | 1.01 | $9.43e^{-1}$ | $2.20e^{1}$ | $7.70e^{-1}$ | $9.18e^{-1}$ | $6.97e^{-1}$ | $6.83e^{-1}$ | $6.79e^{-1}$ | 1.01 | 1.12 | $8.76e^{-1}$ | Chr.L |
| m4_weekly, short, W | CRPS | $8.38e^{-1}$ | 1.00 | $6.86e^{-1}$ | $7.35e^{-1}$ | $7.13e^{-1}$ | $8.51e^{-1}$ | $6.80e^{-1}$ | $9.05e^{-1}$ | $6.51e^{-1}$ | $5.53e^{-1}$ | $6.93e^{-1}$ | $9.57e^{-1}$ | $1.14e^{2}$ | $5.68e^{-1}$ | $7.87e^{-1}$ | $5.43e^{-1}$ | $5.14e^{-1}$ | $5.16e^{-1}$ | $7.41e^{-1}$ | $9.52e^{-1}$ | $6.63e^{-1}$ | Chr.B |
| m4_weekly, short, W | Rank | $15.00$ | $20.00$ | $9.00$ | $12.00$ | $11.00$ | $16.00$ | $8.00$ | $17.00$ | $6.00$ | $4.00$ | $10.00$ | $19.00$ | $21.00$ | $5.00$ | $14.00$ | $3.00$ | $1.00$ | $2.00$ | $13.00$ | $18.00$ | $7.00$ | Chr.B |
| m4_yearly, short, A | MAPE | 1.00 | 1.00 | 1.02 | $9.37e^{-1}$ | $9.37e^{-1}$ | 1.02 | $9.26e^{-1}$ | 1.21 | $9.03e^{-1}$ | 1.01 | 1.03 | 1.06 | 6.86 | $9.71e^{-1}$ | 1.09 | 1.00 | $9.77e^{-1}$ | $9.77e^{-1}$ | $9.77e^{-1}$ | 1.21 | $9.54e^{-1}$ | N-BEATS |
| m4_yearly, short, A | CRPS | $9.93e^{-1}$ | 1.00 | $9.42e^{-1}$ | $8.33e^{-1}$ | $8.04e^{-1}$ | $8.19e^{-1}$ | $7.97e^{-1}$ | 1.12 | $9.71e^{-1}$ | $8.48e^{-1}$ | $8.48e^{-1}$ | 1.22 | $1.03e^{2}$ | $8.48e^{-1}$ | 1.15 | 1.01 | $9.78e^{-1}$ | $9.78e^{-1}$ | $8.26e^{-1}$ | $9.42e^{-1}$ | $7.75e^{-1}$ | Moi.L |
| m4_yearly, short, A | Rank | $15.00$ | $16.00$ | $10.00$ | $6.00$ | $3.00$ | $4.00$ | $2.00$ | $18.00$ | $12.00$ | $8.00$ | $7.00$ | $20.00$ | $21.00$ | $9.00$ | $19.00$ | $17.00$ | $13.00$ | $14.00$ | $5.00$ | $11.00$ | $1.00$ | Moi.L |
| m_dense, long, H | MAPE | 2.19 | 1.00 | $9.16e^{-1}$ | 2.29 | 1.45 | $5.38e^{-1}$ | $5.00e^{-1}$ | $8.90e^{-1}$ | $7.15e^{-1}$ | $5.82e^{-1}$ | $6.10e^{-1}$ | 1.14 | 5.26 | $6.27e^{-1}$ | $6.75e^{-1}$ | $5.02e^{-1}$ | $5.18e^{-1}$ | $5.22e^{-1}$ | $9.10e^{-1}$ | $6.14e^{-1}$ | 1.69 | TFT |
| m_dense, long, H | CRPS | 2.61 | 1.00 | $4.89e^{-1}$ | 2.59 | 1.07 | $2.36e^{-1}$ | $2.08e^{-1}$ | $2.93e^{-1}$ | $4.40e^{-1}$ | $2.17e^{-1}$ | $2.45e^{-1}$ | $4.69e^{-1}$ | $1.15e^{-1}$ | $2.45e^{-1}$ | $3.01e^{-1}$ | $2.41e^{-1}$ | $2.45e^{-1}$ | $2.52e^{-1}$ | $3.26e^{-1}$ | $2.59e^{-1}$ | $4.53e^{-1}$ | C.former |
| m_dense, long, H | Rank | $21.00$ | $18.00$ | $17.00$ | $20.00$ | $19.00$ | $4.00$ | $2.00$ | $11.00$ | $14.00$ | $3.00$ | $7.00$ | $16.00$ | $1.00$ | $8.00$ | $12.00$ | $5.00$ | $6.00$ | $9.00$ | $13.00$ | $10.00$ | $15.00$ | C.former |
| m_dense, medium, H | MAPE | 2.17 | 1.00 | $9.01e^{-1}$ | 1.38 | 1.43 | $4.70e^{-1}$ | $4.55e^{-1}$ | $7.92e^{-1}$ | $5.10e^{-1}$ | $5.32e^{-1}$ | $5.64e^{-1}$ | $7.81e^{-1}$ | 3.78 | $5.52e^{-1}$ | $6.19e^{-1}$ | $4.64e^{-1}$ | $4.81e^{-1}$ | $4.83e^{-1}$ | $7.05e^{-1}$ | $5.93e^{-1}$ | 1.18 | TFT |
| m_dense, medium, H | CRPS | 2.51 | 1.00 | $5.32e^{-1}$ | 2.53 | $9.56e^{-1}$ | $2.46e^{-1}$ | $2.38e^{-1}$ | $3.51e^{-1}$ | $3.84e^{-1}$ | $2.65e^{-1}$ | $2.86e^{-1}$ | $3.99e^{-1}$ | $2.78e^{-1}$ | $2.76e^{-1}$ | $3.78e^{-1}$ | $2.67e^{-1}$ | $2.84e^{-1}$ | $2.80e^{-1}$ | $3.30e^{-1}$ | $3.03e^{-1}$ | $4.53e^{-1}$ | TFT |
| m_dense, medium, H | Rank | $20.00$ | $19.00$ | $17.00$ | $21.00$ | $18.00$ | $2.00$ | $1.00$ | $12.00$ | $14.00$ | $3.00$ | $9.00$ | $15.00$ | $6.00$ | $5.00$ | $13.00$ | $4.00$ | $8.00$ | $7.00$ | $11.00$ | $10.00$ | $16.00$ | TFT |
| m_dense, short, D | MAPE | 1.00 | 1.00 | $8.85e^{-1}$ | $8.46e^{-1}$ | $8.42e^{-1}$ | $5.89e^{-1}$ | $5.22e^{-1}$ | $5.85e^{-1}$ | $4.82e^{-1}$ | $5.18e^{-1}$ | $5.10e^{-1}$ | $6.80e^{-1}$ | 8.46 | $4.94e^{-1}$ | $5.57e^{-1}$ | $5.93e^{-1}$ | $5.10e^{-1}$ | $5.18e^{-1}$ | $5.26e^{-1}$ | $5.45e^{-1}$ | $6.32e^{-1}$ | N-BEATS |
| m_dense, short, D | CRPS | $7.72e^{-1}$ | 1.00 | $4.59e^{-1}$ | $4.29e^{-1}$ | $4.18e^{-1}$ | $2.60e^{-1}$ | $2.61e^{-1}$ | $3.09e^{-1}$ | $2.98e^{-1}$ | $2.38e^{-1}$ | $2.45e^{-1}$ | $4.18e^{-1}$ | $1.03e^{1}$ | $2.39e^{-1}$ | $3.20e^{-1}$ | $2.95e^{-1}$ | $2.55e^{-1}$ | $2.54e^{-1}$ | $2.64e^{-1}$ | $2.65e^{-1}$ | $3.10e^{-1}$ | P.TST |
| m_dense, short, D | Rank | $19.00$ | $20.00$ | $18.00$ | $17.00$ | $15.00$ | $6.00$ | $7.00$ | $12.00$ | $11.00$ | $1.00$ | $3.00$ | $16.00$ | $21.00$ | $2.00$ | $14.00$ | $10.00$ | $5.00$ | $4.00$ | $8.00$ | $9.00$ | $13.00$ | P.TST |
| m_dense, short, H | MAPE | 1.93 | 1.00 | 1.00 | 1.20 | 1.14 | $4.94e^{-1}$ | $5.43e^{-1}$ | $7.09e^{-1}$ | $5.94e^{-1}$ | $6.91e^{-1}$ | $5.96e^{-1}$ | $7.80e^{-1}$ | 5.17 | $6.04e^{-1}$ | $8.41e^{-1}$ | $5.30e^{-1}$ | $5.24e^{-1}$ | $5.22e^{-1}$ | $6.98e^{-1}$ | $6.46e^{-1}$ | $7.07e^{-1}$ | DeepAR |
| m_dense, short, H | CRPS | 1.92 | 1.00 | 1.00 | 1.95 | $8.61e^{-1}$ | $4.56e^{-1}$ | $4.95e^{-1}$ | $5.98e^{-1}$ | $6.76e^{-1}$ | $6.16e^{-1}$ | $5.34e^{-1}$ | $7.62e^{-1}$ | 4.48 | $4.98e^{-1}$ | $8.58e^{-1}$ | $4.98e^{-1}$ | $4.88e^{-1}$ | $4.88e^{-1}$ | $6.05e^{-1}$ | $5.55e^{-1}$ | $5.94e^{-1}$ | DeepAR |
| m_dense, short, H | Rank | $19.00$ | $18.00$ | $17.00$ | $20.00$ | $16.00$ | $1.00$ | $4.00$ | $10.00$ | $13.00$ | $12.00$ | $7.00$ | $14.00$ | $21.00$ | $6.00$ | $15.00$ | $5.00$ | $2.00$ | $3.00$ | $11.00$ | $8.00$ | $9.00$ | DeepAR |
| restaurant, short, D | MAPE | 1.00 | 1.00 | $8.74e^{-1}$ | $9.24e^{-1}$ | $9.93e^{-1}$ | $7.55e^{-1}$ | $7.46e^{-1}$ | $7.18e^{-1}$ | $7.08e^{-1}$ | $6.91e^{-1}$ | $7.24e^{-1}$ | $7.52e^{-1}$ | 1.44 | $7.48e^{-1}$ | $8.29e^{-1}$ | $7.92e^{-1}$ | $7.52e^{-1}$ | $7.57e^{-1}$ | $7.40e^{-1}$ | $7.22e^{-1}$ | $7.58e^{-1}$ | P.TST |
| restaurant, short, D | CRPS | $7.43e^{-1}$ | 1.00 | $3.99e^{-1}$ | $3.63e^{-1}$ | $1.21e^{2}$ | $2.98e^{-1}$ | $3.13e^{-1}$ | $3.27e^{-1}$ | $3.77e^{-1}$ | $2.89e^{-1}$ | $2.87e^{-1}$ | $3.75e^{-1}$ | 3.97 | $2.92e^{-1}$ | $3.95e^{-1}$ | $3.22e^{-1}$ | $3.08e^{-1}$ | $3.08e^{-1}$ | $3.11e^{-1}$ | $3.07e^{-1}$ | $3.04e^{-1}$ | iTrans. |
| restaurant, short, D | Rank | $18.00$ | $19.00$ | $17.00$ | $13.00$ | $21.00$ | $4.00$ | $10.00$ | $12.00$ | $15.00$ | $2.00$ | $1.00$ | $14.00$ | $20.00$ | $3.00$ | $16.00$ | $11.00$ | $7.00$ | $8.00$ | $9.00$ | $6.00$ | $5.00$ | iTrans. |
| saugeen, short, D | MAPE | 1.00 | 1.00 | $9.47e^{-1}$ | 1.12 | 1.32 | 1.03 | $7.03e^{-1}$ | $7.22e^{-1}$ | $6.94e^{-1}$ | $9.13e^{-1}$ | $8.77e^{-1}$ | 1.33 | 2.25 | $8.15e^{-1}$ | 1.24 | $6.71e^{-1}$ | $6.90e^{-1}$ | $6.41e^{-1}$ | $7.35e^{-1}$ | $7.09e^{-1}$ | $6.20e^{-1}$ | Moi.L |
| saugeen, short, D | CRPS | $7.76e^{-1}$ | 1.00 | $7.48e^{-1}$ | $8.87e^{-1}$ | $7.90e^{-1}$ | $7.59e^{-1}$ | $5.56e^{-1}$ | $5.27e^{-1}$ | $6.34e^{-1}$ | $5.41e^{-1}$ | $5.54e^{-1}$ | $8.13e^{-1}$ | $6.13e^{-1}$ | $5.53e^{-1}$ | $7.72e^{-1}$ | $5.13e^{-1}$ | $5.73e^{-1}$ | $5.57e^{-1}$ | $5.03e^{-1}$ | $5.21e^{-1}$ | $5.29e^{-1}$ | Moi.S |
| saugeen, short, D | Rank | $17.00$ | $21.00$ | $14.00$ | $20.00$ | $18.00$ | $15.00$ | $9.00$ | $4.00$ | $13.00$ | $6.00$ | $8.00$ | $19.00$ | $12.00$ | $7.00$ | $16.00$ | $2.00$ | $11.00$ | $10.00$ | $1.00$ | $3.00$ | $5.00$ | Moi.S |
| saugeen, short, M | MAPE | 1.47 | 1.00 | $7.67e^{-1}$ | 1.00 | $8.47e^{-1}$ | 1.46 | $9.20e^{-1}$ | 1.06 | $7.10e^{-1}$ | $9.66e^{-1}$ | $9.03e^{-1}$ | 1.20 | $8.49e^{-1}$ | $8.65e^{-1}$ | $8.45e^{-1}$ | $9.05e^{-1}$ | $7.73e^{-1}$ | $7.87e^{-1}$ | 1.93 | 3.60 | $9.86e^{-1}$ | N-BEATS |
| saugeen, short, M | CRPS | 1.73 | 1.00 | $7.33e^{-1}$ | $8.38e^{-1}$ | $7.24e^{-1}$ | 1.55 | $7.64e^{-1}$ | $9.53e^{-1}$ | $8.72e^{-1}$ | $8.36e^{-1}$ | $7.10e^{-1}$ | 1.10 | $4.79e^{-1}$ | $7.37e^{-1}$ | $9.21e^{-1}$ | 1.04 | $9.17e^{-1}$ | $9.33e^{-1}$ | 1.23 | 1.59 | $8.83e^{-1}$ | C.former |
| saugeen, short, M | Rank | $21.00$ | $15.00$ | $4.00$ | $8.00$ | $3.00$ | $19.00$ | $6.00$ | $14.00$ | $9.00$ | $7.00$ | $2.00$ | $17.00$ | $1.00$ | $5.00$ | $12.00$ | $16.00$ | $11.00$ | $13.00$ | $18.00$ | $20.00$ | $10.00$ | C.former |
| saugeen, short, W | MAPE | 1.00 | 1.00 | $6.57e^{-1}$ | 1.07 | 1.00 | $5.55e^{-1}$ | $6.52e^{-1}$ | $5.95e^{-1}$ | $5.46e^{-1}$ | $6.58e^{-1}$ | $6.13e^{-1}$ | 1.04 | 1.64 | $5.32e^{-1}$ | $8.45e^{-1}$ | $4.07e^{-1}$ | $3.98e^{-1}$ | $3.99e^{-1}$ | $5.12e^{-1}$ | $5.32e^{-1}$ | $4.84e^{-1}$ | Chr.B |
| saugeen, short, W | CRPS | $8.58e^{-1}$ | 1.00 | $6.42e^{-1}$ | $8.58e^{-1}$ | 1.05 | $4.64e^{-1}$ | $5.74e^{-1}$ | $6.12e^{-1}$ | $6.71e^{-1}$ | $5.66e^{-1}$ | $5.45e^{-1}$ | $7.87e^{-1}$ | 2.97 | $4.47e^{-1}$ | $7.05e^{-1}$ | $5.64e^{-1}$ | $5.53e^{-1}$ | $5.51e^{-1}$ | $5.36e^{-1}$ | $5.60e^{-1}$ | $5.10e^{-1}$ | T.FM |
| saugeen, short, W | Rank | $18.00$ | $19.00$ | $13.00$ | $17.00$ | $20.00$ | $2.00$ | $11.00$ | $12.00$ | $14.00$ | $10.00$ | $5.00$ | $16.00$ | $21.00$ | $1.00$ | $15.00$ | $9.00$ | $7.00$ | $6.00$ | $4.00$ | $8.00$ | $3.00$ | T.FM |
| solar, long, 10T | MAPE | $4.33e^{-1}$ | 1.00 | 1.00 | 4.10 | $4.33e^{-1}$ | $8.70e^{-1}$ | 1.98 | 1.43 | $4.16e^{-1}$ | 1.35 | 1.25 | 1.47 | $1.20e^{11}$ | 1.06 | 1.80 | $9.18e^{-1}$ | 1.06 | 1.08 | $4.59e^{-1}$ | $8.74e^{-1}$ | $8.61e^{-1}$ | N-BEATS |
| solar, long, 10T | CRPS | 1.97 | 1.00 | 1.00 | 8.45 | 3.73 | $6.98e^{-1}$ | $4.82e^{-1}$ | $4.54e^{-1}$ | 1.28 | $4.31e^{-1}$ | $5.47e^{-1}$ | $7.44e^{-1}$ | $3.92e^{-2}$ | $8.94e^{-1}$ | $7.15e^{-1}$ | 1.15 | $9.52e^{-1}$ | $9.50e^{-1}$ | 1.06 | $5.80e^{-1}$ | $7.65e^{-1}$ | C.former |
| solar, long, 10T | Rank | $19.00$ | $15.00$ | $14.00$ | $21.00$ | $20.00$ | $7.00$ | $4.00$ | $3.00$ | $18.00$ | $2.00$ | $5.00$ | $9.00$ | $1.00$ | $11.00$ | $8.00$ | $17.00$ | $13.00$ | $12.00$ | $16.00$ | $6.00$ | $10.00$ | C.former |
| solar, long, H | MAPE | $1.88e^{-1}$ | 1.00 | $8.16e^{-1}$ | 5.29 | $9.49e^{-1}$ | 1.10 | 1.17 | 1.93 | $1.87e^{-1}$ | 1.68 | 1.15 | 1.50 | $5.18e^{1}$ | 1.37 | $9.40e^{-1}$ | $6.53e^{-1}$ | $6.12e^{-1}$ | $6.32e^{-1}$ | $7.50e^{-1}$ | $8.99e^{-1}$ | $6.68e^{-1}$ | N-BEATS |
| solar, long, H | CRPS | 2.35 | 1.00 | $4.13e^{-1}$ | 4.98 | 1.17 | $2.59e^{-1}$ | $2.73e^{-1}$ | $2.92e^{-1}$ | $6.80e^{-1}$ | $2.40e^{-1}$ | $2.50e^{-1}$ | $4.16e^{-1}$ | $3.30e^{-3}$ | $3.89e^{-1}$ | $2.96e^{-1}$ | $3.00e^{-1}$ | $3.16e^{-1}$ | $3.05e^{-1}$ | $2.50e^{-1}$ | $2.45e^{-1}$ | $2.22e^{-1}$ | C.former |
| solar, long, H | Rank | $20.00$ | $18.00$ | $15.00$ | $21.00$ | $19.00$ | $7.00$ | $8.00$ | $9.00$ | $17.00$ | $3.00$ | $5.00$ | $16.00$ | $1.00$ | $14.00$ | $10.00$ | $11.00$ | $13.00$ | $12.00$ | $6.00$ | $4.00$ | $2.00$ | C.former |
| solar, medium, 10T | MAPE | $8.88e^{-1}$ | 1.00 | 1.00 | 1.30 | 4.81 | $5.87e^{-1}$ | 1.30 | 1.15 | $2.99e^{-1}$ | 1.12 | $5.62e^{-1}$ | 1.19 | 1.21 | $8.45e^{-1}$ | 1.01 | $9.69e^{-1}$ | 1.17 | 1.16 | $3.54e^{-1}$ | $7.42e^{-1}$ | $7.14e^{-1}$ | N-BEATS |
| solar, medium, 10T | CRPS | 1.78 | 1.00 | 1.00 | 7.35 | 5.95 | $6.29e^{-1}$ | $4.70e^{-1}$ | $5.12e^{-1}$ | 1.30 | $4.62e^{-1}$ | $7.12e^{-1}$ | $7.16e^{-1}$ | $3.50e^{-1}$ | $8.08e^{-1}$ | $5.85e^{-1}$ | 1.03 | $8.90e^{-1}$ | $8.83e^{-1}$ | $9.70e^{-1}$ | $5.45e^{-1}$ | $7.41e^{-1}$ | C.former |
| solar, medium, 10T | Rank | $19.00$ | $16.00$ | $15.00$ | $21.00$ | $20.00$ | $7.00$ | $3.00$ | $4.00$ | $18.00$ | $2.00$ | $8.00$ | $9.00$ | $1.00$ | $11.00$ | $6.00$ | $17.00$ | $13.00$ | $12.00$ | $14.00$ | $5.00$ | $10.00$ | C.former |
| solar, medium, H | MAPE | $1.91e^{-1}$ | 1.00 | $8.47e^{-1}$ | 1.83 | $4.81e^{-1}$ | $9.12e^{-1}$ | $7.77e^{-1}$ | $8.09e^{-1}$ | $1.89e^{-1}$ | 1.19 | 1.00 | 1.33 | $5.27e^{1}$ | 1.04 | $7.96e^{-1}$ | $8.66e^{-1}$ | $8.17e^{-1}$ | $8.17e^{-1}$ | $7.67e^{-1}$ | $7.86e^{-1}$ | $6.83e^{-1}$ | N-BEATS |
| solar, medium, H | CRPS | 2.35 | 1.00 | $4.39e^{-1}$ | 4.83 | 1.67 | $2.77e^{-1}$ | $2.60e^{-1}$ | $2.71e^{-1}$ | $7.87e^{-1}$ | $2.71e^{-1}$ | $2.95e^{-1}$ | $4.35e^{-1}$ | $3.11e^{-2}$ | $3.35e^{-1}$ | $3.29e^{-1}$ | $2.84e^{-1}$ | $2.80e^{-1}$ | $2.80e^{-1}$ | $2.82e^{-1}$ | $2.84e^{-1}$ | $2.56e^{-1}$ | C.former |
| solar, medium, H | Rank | $20.00$ | $18.00$ | $16.00$ | $21.00$ | $19.00$ | $6.00$ | $3.00$ | $5.00$ | $17.00$ | $4.00$ | $12.00$ | $15.00$ | $1.00$ | $14.00$ | $13.00$ | $10.00$ | $8.00$ | $7.00$ | $9.00$ | $11.00$ | $2.00$ | C.former |
| solar, short, 10T | MAPE | $6.05e^{-1}$ | 1.00 | 1.00 | 1.01 | $9.86e^{-1}$ | $2.30e^{-1}$ | $7.75e^{-1}$ | $2.75e^{-1}$ | 1.10 | $3.66e^{-1}$ | $3.52e^{-1}$ | $8.93e^{-1}$ | $8.41e^{-1}$ | 1.09 | $9.98e^{-1}$ | 1.78 | 1.47 | 1.46 | $3.75e^{-1}$ | 1.07 | $7.18e^{-1}$ | DeepAR |
| solar, short, 10T | CRPS | 1.10 | 1.00 | 1.00 | 2.74 | 1.01 | 1.08 | $7.19e^{-1}$ | 1.10 | $6.70e^{-1}$ | 1.59 | 1.59 | $9.58e^{-1}$ | 3.42 | 1.01 | $8.56e^{-1}$ | $7.38e^{-1}$ | $6.73e^{-1}$ | $6.69e^{-1}$ | $7.72e^{-1}$ | $7.60e^{-1}$ | $6.78e^{-1}$ | Chr.L |
| solar, short, 10T | Rank | $16.00$ | $12.00$ | $11.00$ | $20.00$ | $13.00$ | $15.00$ | $5.00$ | $17.00$ | $2.00$ | $19.00$ | $18.00$ | $10.00$ | $21.00$ | $14.00$ | $9.00$ | $6.00$ | $3.00$ | $1.00$ | $8.00$ | $7.00$ | $4.00$ | Chr.L |
| solar, short, D | MAPE | 1.00 | 1.00 | $8.70e^{-1}$ | $8.61e^{-1}$ | $8.55e^{-1}$ | $7.69e^{-1}$ | $9.13e^{-1}$ | 1.11 | $9.57e^{-1}$ | $8.96e^{-1}$ | 1.22 | $8.78e^{-1}$ | $4.27e^{-1}$ | $9.57e^{-1}$ | 1.14 | 1.04 | 1.03 | 1.05 | $9.04e^{-1}$ | $8.42e^{-1}$ | $9.30e^{-1}$ | C.former |
| solar, short, D | CRPS | $7.38e^{-1}$ | 1.00 | $3.73e^{-1}$ | $3.78e^{-1}$ | $3.71e^{-1}$ | $9.01e^{-1}$ | $3.66e^{-1}$ | $4.65e^{-1}$ | $5.94e^{-1}$ | $3.79e^{-1}$ | 1.15 | $5.06e^{-1}$ | 2.81 | $3.80e^{-1}$ | $5.67e^{-1}$ | $4.45e^{-1}$ | $4.31e^{-1}$ | $4.33e^{-1}$ | $3.74e^{-1}$ | $3.78e^{-1}$ | $3.79e^{-1}$ | TFT |
| solar, short, D | Rank | $17.00$ | $19.00$ | $3.00$ | $5.00$ | $2.00$ | $18.00$ | $1.00$ | $13.00$ | $16.00$ | $8.00$ | $20.00$ | $14.00$ | $21.00$ | $9.00$ | $15.00$ | $12.00$ | $10.00$ | $11.00$ | $4.00$ | $6.00$ | $7.00$ | TFT |
| solar, short, H | MAPE | $2.58e^{-1}$ | 1.00 | 1.00 | 1.77 | $5.14e^{-1}$ | 1.18 | $9.61e^{-1}$ | $8.35e^{-1}$ | 1.72 | 1.18 | $8.53e^{-1}$ | 1.59 | $7.14e^{1}$ | 1.58 | 1.18 | 1.29 | 1.23 | 1.21 | $7.16e^{-1}$ | 1.14 | $8.97e^{-1}$ | Nv. |
| solar, short, H | CRPS | 2.12 | 1.00 | 1.00 | 3.71 | 1.72 | $6.19e^{-1}$ | $5.84e^{-1}$ | $6.05e^{-1}$ | $7.93e^{-1}$ | $5.41e^{-1}$ | $6.78e^{-1}$ | $8.07e^{-1}$ | $3.31e^{-1}$ | $6.42e^{-1}$ | $6.61e^{-1}$ | $5.49e^{-1}$ | $5.32e^{-1}$ | $5.37e^{-1}$ | $5.75e^{-1}$ | $5.91e^{-1}$ | $5.41e^{-1}$ | C.former |
| solar, short, H | Rank | $20.00$ | $18.00$ | $17.00$ | $21.00$ | $19.00$ | $11.00$ | $8.00$ | $10.00$ | $15.00$ | $5.00$ | $14.00$ | $16.00$ | $1.00$ | $12.00$ | $13.00$ | $6.00$ | $2.00$ | $3.00$ | $7.00$ | $9.00$ | $4.00$ | C.former |
| solar, short, W | MAPE | 1.00 | 1.00 | $7.57e^{-1}$ | $7.73e^{-1}$ | $6.57e^{-1}$ | 1.14 | $4.27e^{-1}$ | $8.64e^{-1}$ | 1.56 | $7.38e^{-1}$ | $8.74e^{-1}$ | $7.57e^{-1}$ | 1.87 | $7.67e^{-1}$ | 1.43 | $5.50e^{-1}$ | $7.64e^{-1}$ | $7.67e^{-1}$ | 1.26 | 1.07 | $9.64e^{-1}$ | TFT |
| solar, short, W | CRPS | $8.90e^{-1}$ | 1.00 | $6.44e^{-1}$ | $6.57e^{-1}$ | $5.89e^{-1}$ | 1.03 | $4.83e^{-1}$ | $7.92e^{-1}$ | 1.83 | $6.86e^{-1}$ | $8.60e^{-1}$ | $8.90e^{-1}$ | $3.58e^{1}$ | $6.65e^{-1}$ | 1.67 | $5.25e^{-1}$ | $6.82e^{-1}$ | $6.86e^{-1}$ | 1.17 | $9.36e^{-1}$ | $8.35e^{-1}$ | TFT |
| solar, short, W | Rank | $14.00$ | $16.00$ | $4.00$ | $5.00$ | $3.00$ | $17.00$ | $1.00$ | $10.00$ | $20.00$ | $9.00$ | $12.00$ | $13.00$ | $21.00$ | $6.00$ | $19.00$ | $2.00$ | $7.00$ | $8.00$ | $18.00$ | $15.00$ | $11.00$ | TFT |
| sz_taxi, long, 15T | MAPE | 2.83 | 1.00 | 2.55 | 1.55 | 1.66 | 1.46 | 2.41 | 1.66 | 1.82 | 1.70 | 1.83 | 1.94 | $2.39e^{-12}$ | 2.63 | 1.89 | 2.43 | 2.48 | 2.54 | 2.30 | 2.42 | 2.24 | C.former |
| sz_taxi, long, 15T | CRPS | 4.33 | 1.00 | $7.18e^{-1}$ | 1.14 | $9.33e^{14}$ | $5.16e^{-1}$ | $4.35e^{-1}$ | $5.69e^{-1}$ | $5.83e^{-1}$ | $5.07e^{-1}$ | $5.09e^{-1}$ | $7.15e^{-1}$ | $2.20e^{-1}$ | $4.30e^{-1}$ | $4.93e^{-1}$ | $4.96e^{-1}$ | $4.78e^{-1}$ | $4.78e^{-1}$ | $3.86e^{-1}$ | $3.77e^{-1}$ | $3.75e^{-1}$ | C.former |
| sz_taxi, long, 15T | Rank | $20.00$ | $18.00$ | $17.00$ | $19.00$ | $21.00$ | $13.00$ | $6.00$ | $14.00$ | $15.00$ | $11.00$ | $12.00$ | $16.00$ | $1.00$ | $5.00$ | $9.00$ | $10.00$ | $7.00$ | $8.00$ | $4.00$ | $3.00$ | $2.00$ | C.former |
| sz_taxi, medium, 15T | MAPE | $5.29e^{-1}$ | 1.00 | $6.56e^{-1}$ | $3.55e^{-1}$ | $7.72e^{-1}$ | $5.22e^{-1}$ | $5.45e^{-1}$ | $5.98e^{-1}$ | $6.08e^{-1}$ | $5.10e^{-1}$ | $6.03e^{-1}$ | $5.34e^{-1}$ | $2.09e^{-13}$ | $5.04e^{-1}$ | $5.98e^{-1}$ | $6.56e^{-1}$ | $6.61e^{-1}$ | $6.67e^{-1}$ | $6.03e^{-1}$ | $7.14e^{-1}$ | $6.51e^{-1}$ | C.former |
| sz_taxi, medium, 15T | CRPS | 4.27 | 1.00 | $7.73e^{-1}$ | 1.17 | $1.74e^{8}$ | $4.63e^{-1}$ | $4.54e^{-1}$ | $5.57e^{-1}$ | $6.92e^{-1}$ | $4.85e^{-1}$ | $4.58e^{-1}$ | $6.52e^{-1}$ | $4.27e^{-1}$ | $5.13e^{-1}$ | $5.90e^{-1}$ | $6.15e^{-1}$ | $5.90e^{-1}$ | $5.88e^{-1}$ | $4.69e^{-1}$ | $4.60e^{-1}$ | $4.67e^{-1}$ | C.former |
| sz_taxi, medium, 15T | Rank | $20.00$ | $18.00$ | $17.00$ | $19.00$ | $21.00$ | $5.00$ | $2.00$ | $10.00$ | $16.00$ | $8.00$ | $3.00$ | $15.00$ | $1.00$ | $9.00$ | $13.00$ | $14.00$ | $12.00$ | $11.00$ | $7.00$ | $4.00$ | $6.00$ | C.former |
| sz_taxi, short, 15T | MAPE | 1.00 | 1.00 | 1.00 | $9.31e^{-1}$ | 1.11 | 1.09 | 1.15 | $9.39e^{-1}$ | $7.27e^{-1}$ | $9.01e^{-1}$ | $6.24e^{-1}$ | 1.22 | $6.83e^{-12}$ | $6.79e^{-1}$ | $9.39e^{-1}$ | $3.86e^{-1}$ | $5.40e^{-1}$ | $4.37e^{-1}$ | 1.11 | $8.40e^{-1}$ | $8.24e^{-1}$ | C.former |
| sz_taxi, short, 15T | CRPS | 2.14 | 1.00 | 1.00 | $9.32e^{-1}$ | $9.42e^{-1}$ | $7.09e^{-1}$ | $7.18e^{-1}$ | $7.80e^{-1}$ | $9.09e^{-1}$ | $6.70e^{-1}$ | $6.93e^{-1}$ | $8.77e^{-1}$ | 6.67 | $6.67e^{-1}$ | $9.45e^{-1}$ | $7.80e^{-1}$ | $7.64e^{-1}$ | $7.64e^{-1}$ | $6.99e^{-1}$ | $6.86e^{-1}$ | $6.86e^{-1}$ | T.FM |
| sz_taxi, short, 15T | Rank | $20.00$ | $19.00$ | $18.00$ | $15.00$ | $16.00$ | $7.00$ | $8.00$ | $12.00$ | $14.00$ | $2.00$ | $5.00$ | $13.00$ | $21.00$ | $1.00$ | $17.00$ | $11.00$ | $9.00$ | $10.00$ | $6.00$ | $3.00$ | $4.00$ | T.FM |
| sz_taxi, short, H | MAPE | $6.25e^{-1}$ | 1.00 | $8.58e^{-1}$ | $5.14e^{-1}$ | 1.35 | $5.96e^{-1}$ | $7.42e^{-1}$ | $7.87e^{-1}$ | $5.90e^{-1}$ | $6.15e^{-1}$ | $6.84e^{-1}$ | $6.84e^{-1}$ | 2.18 | $6.06e^{-1}$ | $8.58e^{-1}$ | $6.39e^{-1}$ | $7.74e^{-1}$ | $7.10e^{-1}$ | $7.68e^{-1}$ | $7.16e^{-1}$ | $6.97e^{-1}$ | A.Th. |
| sz_taxi, short, H | CRPS | 2.12 | 1.00 | $7.42e^{-1}$ | 1.01 | 7.07 | $6.07e^{-1}$ | $6.29e^{-1}$ | $8.52e^{-1}$ | $8.30e^{-1}$ | $6.29e^{-1}$ | $6.16e^{-1}$ | $8.78e^{-1}$ | 9.21 | $5.98e^{-1}$ | $7.95e^{-1}$ | $6.51e^{-1}$ | $6.51e^{-1}$ | $6.46e^{-1}$ | $6.55e^{-1}$ | $6.42e^{-1}$ | $6.38e^{-1}$ | T.FM |
| sz_taxi, short, H | Rank | $19.00$ | $17.00$ | $12.00$ | $18.00$ | $20.00$ | $2.00$ | $5.00$ | $15.00$ | $14.00$ | $4.00$ | $3.00$ | $16.00$ | $21.00$ | $1.00$ | $13.00$ | $10.00$ | $9.00$ | $8.00$ | $11.00$ | $7.00$ | $6.00$ | T.FM |
| temperature_rain, short, D | MAPE | 1.00 | 1.00 | $9.02e^{-1}$ | 1.13 | 1.19 | $5.07e^{-1}$ | $4.98e^{-1}$ | $4.55e^{-1}$ | 1.14 | $6.06e^{-1}$ | $4.41e^{-1}$ | 1.14 | 4.11 | $3.02e^{-1}$ | $8.56e^{-1}$ | $2.82e^{-1}$ | $2.42e^{-1}$ | $2.15e^{-1}$ | $1.55e^{-1}$ | $1.97e^{-1}$ | $1.81e^{-1}$ | Moi.S |
| temperature_rain, short, D | CRPS | $7.79e^{-1}$ | 1.00 | $4.26e^{-1}$ | $4.67e^{-1}$ | $4.63e^{-1}$ | $4.18e^{-1}$ | $3.63e^{-1}$ | $4.28e^{-1}$ | $5.15e^{-1}$ | $3.95e^{-1}$ | $3.88e^{-1}$ | $5.09e^{-1}$ | 3.72 | $3.56e^{-1}$ | $4.79e^{-1}$ | $3.85e^{-1}$ | $3.74e^{-1}$ | $3.74e^{-1}$ | $3.46e^{-1}$ | $3.50e^{-1}$ | $3.45e^{-1}$ | Moi.L |
| temperature_rain, short, D | Rank | $19.00$ | $20.00$ | $12.00$ | $15.00$ | $14.00$ | $11.00$ | $5.00$ | $13.00$ | $18.00$ | $10.00$ | $9.00$ | $17.00$ | $21.00$ | $4.00$ | $16.00$ | $8.00$ | $7.00$ | $6.00$ | $2.00$ | $3.00$ | $1.00$ | Moi.L |
| us_births, short, D | MAPE | 1.00 | 1.00 | $8.40e^{-1}$ | $8.72e^{-1}$ | $8.56e^{-1}$ | $2.70e^{-1}$ | $1.62e^{-1}$ | $3.32e^{-1}$ | $2.39e^{-1}$ | $2.56e^{-1}$ | $2.70e^{-1}$ | $3.34e^{-1}$ | $8.72e^{-1}$ | $2.85e^{-1}$ | $2.90e^{-1}$ | $2.27e^{-1}$ | $2.20e^{-1}$ | $2.27e^{-1}$ | $3.64e^{-1}$ | $3.04e^{-1}$ | $2.74e^{-1}$ | TFT |
| us_births, short, D | CRPS | $8.33e^{-1}$ | 1.00 | $5.13e^{-1}$ | $5.19e^{-1}$ | $5.17e^{-1}$ | $1.91e^{-1}$ | $1.12e^{-1}$ | $2.15e^{-1}$ | $2.02e^{-1}$ | $1.73e^{-1}$ | $1.85e^{-1}$ | $2.85e^{-1}$ | $4.85e^{-1}$ | $1.98e^{-1}$ | $2.47e^{-1}$ | $1.61e^{-1}$ | $1.53e^{-1}$ | $1.57e^{-1}$ | $2.61e^{-1}$ | $2.28e^{-1}$ | $1.94e^{-1}$ | TFT |
| us_births, short, D | Rank | $20.00$ | $21.00$ | $17.00$ | $19.00$ | $18.00$ | $7.00$ | $1.00$ | $11.00$ | $10.00$ | $5.00$ | $6.00$ | $15.00$ | $16.00$ | $9.00$ | $13.00$ | $4.00$ | $2.00$ | $3.00$ | $14.00$ | $12.00$ | $8.00$ | TFT |
| us_births, short, M | MAPE | 1.99 | 1.00 | $6.21e^{-1}$ | 1.17 | $7.77e^{-1}$ | 1.02 | 1.16 | 1.05 | 1.21 | 1.04 | 1.09 | 1.53 | $5.92e^{-1}$ | $8.25e^{-1}$ | $9.51e^{-1}$ | $7.52e^{-1}$ | 1.03 | $7.72e^{-1}$ | 1.72 | 1.98 | 1.35 | C.former |
| us_births, short, M | CRPS | 2.32 | 1.00 | $6.19e^{-1}$ | 1.13 | $7.14e^{-1}$ | $9.35e^{-1}$ | 1.27 | 1.38 | 1.51 | $9.94e^{-1}$ | 1.08 | 1.92 | $1.15e^{-1}$ | $7.50e^{-1}$ | 1.18 | $7.98e^{-1}$ | 1.05 | $7.98e^{-1}$ | 1.71 | 2.07 | 1.25 | C.former |
| us_births, short, M | Rank | $21.00$ | $9.00$ | $2.00$ | $12.00$ | $3.00$ | $7.00$ | $15.00$ | $16.00$ | $17.00$ | $8.00$ | $11.00$ | $19.00$ | $1.00$ | $4.00$ | $13.00$ | $6.00$ | $10.00$ | $5.00$ | $18.00$ | $20.00$ | $14.00$ | C.former |
| us_births, short, W | MAPE | 1.00 | 1.00 | $9.53e^{-1}$ | $9.53e^{-1}$ | $9.53e^{-1}$ | $9.31e^{-1}$ | 1.01 | $7.54e^{-1}$ | $8.92e^{-1}$ | $7.89e^{-1}$ | 1.55 | $9.22e^{-1}$ | 2.18 | $6.81e^{-1}$ | $8.79e^{-1}$ | $5.91e^{-1}$ | $5.99e^{-1}$ | $5.30e^{-1}$ | 1.24 | 1.00 | $8.66e^{-1}$ | Chr.L |
| us_births, short, W | CRPS | $8.94e^{-1}$ | 1.00 | $8.33e^{-1}$ | $8.47e^{-1}$ | $8.47e^{-1}$ | $7.87e^{-1}$ | $8.98e^{-1}$ | $7.59e^{-1}$ | $9.72e^{-1}$ | $6.71e^{-1}$ | 1.26 | 1.01 | 4.08 | $5.79e^{-1}$ | $9.58e^{-1}$ | $5.32e^{-1}$ | $5.14e^{-1}$ | $4.59e^{-1}$ | 1.06 | $9.12e^{-1}$ | $7.50e^{-1}$ | Chr.L |
| us_births, short, W | Rank | $12.00$ | $17.00$ | $9.00$ | $11.00$ | $10.00$ | $8.00$ | $13.00$ | $7.00$ | $16.00$ | $5.00$ | $20.00$ | $18.00$ | $21.00$ | $4.00$ | $15.00$ | $3.00$ | $2.00$ | $1.00$ | $19.00$ | $14.00$ | $6.00$ | Chr.L |

**Table 23: Moirai vs Moirai-Leakage results on datasets that LOTSA collection and our GIFT-Eval has in common.**
| Dataset | Model | Short | Medium | Long |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- |
| MAPE | CRPS | MAPE | CRPS | MAPE | CRPS |  |  |
| hierarchical_sales, D | Moi Leak.B | 0.51 | 0.24 | NA | NA | NA | NA |
| hierarchical_sales, D | Moi.B | 0.49 | 0.25 | NA | NA | NA | NA |
| hierarchical_sales, D | Moi Leak.L | 0.53 | 0.25 | NA | NA | NA | NA |
| hierarchical_sales, D | Moi.L | 0.52 | 0.24 | NA | NA | NA | NA |
| hierarchical_sale, D | Moi Leak.S | 0.50 | 0.25 | NA | NA | NA | NA |
| hierarchical_sales, D | Moi.S | 0.51 | 0.25 | NA | NA | NA | NA |
| loop_seattle, 5T | Moi Leak.B | 0.67 | 0.57 | 0.42 | 0.37 | 0.50 | 0.38 |
| loop_seattle, 5T | Moi.B | 0.84 | 0.64 | 0.83 | 0.62 | 0.78 | 0.57 |
| loop_seattle, 5T | Moi Leak.L | 0.66 | 0.51 | 0.33 | 0.31 | 0.46 | 0.36 |
| loop_seattle, 5T | Moi.L | 0.83 | 0.65 | 0.85 | 0.65 | 0.81 | 0.59 |
| loop_seattle, 5T | Moi Leak.S | 0.84 | 0.69 | 0.75 | 0.57 | 0.70 | 0.53 |
| loop_seattle, 5T | Moi.S | 0.87 | 0.65 | 0.77 | 0.61 | 0.75 | 0.57 |
| loop_seattle, D | Moi Leak.B | 0.50 | 0.34 | NA | NA | NA | NA |
| loop_seattle, D | Moi.B | 0.52 | 0.35 | NA | NA | NA | NA |
| loop_seattle, D | Moi Leak.L | 0.51 | 0.35 | NA | NA | NA | NA |
| loop_seattle, D | Moi.L | 0.49 | 0.33 | NA | NA | NA | NA |
| loop_seattle, D | Moi Leak.S | 0.53 | 0.35 | NA | NA | NA | NA |
| loop_seattle, D | Moi.S | 0.54 | 0.35 | NA | NA | NA | NA |
| loop_seattle, H | Moi Leak.B | 0.96 | 0.68 | 0.54 | 0.50 | 0.49 | 0.26 |
| loop_seattle, H | Moi.B | 1.08 | 0.72 | 0.65 | 0.55 | 0.59 | 0.30 |
| loop_seattle, H | Moi Leak.L | 0.84 | 0.61 | 0.53 | 0.45 | 0.47 | 0.23 |
| loop_seattle, H | Moi.L | 0.89 | 0.65 | 0.71 | 0.59 | 1.18 | 0.45 |
| loop_seattle, H | Moi Leak.S | 1.22 | 0.80 | 0.73 | 0.64 | 0.70 | 0.35 |
| loop_seattle, H | Moi.S | 1.19 | 0.78 | 0.70 | 0.60 | 0.71 | 0.33 |
| m_dense, D | Moi Leak.B | 0.78 | 0.35 | NA | NA | NA | NA |
| m_dense, D | Moi.B | 0.55 | 0.27 | NA | NA | NA | NA |
| m_dense, D | Moi Leak.L | 0.67 | 0.32 | NA | NA | NA | NA |
| m_dense, D | Moi.L | 0.63 | 0.31 | NA | NA | NA | NA |
| m_dense, D | Moi Leak.S | 0.58 | 0.28 | NA | NA | NA | NA |
| m_dense, D | Moi.S | 0.53 | 0.26 | NA | NA | NA | NA |
| m_dense, H | Moi Leak.B | 0.54 | 0.50 | 0.49 | 0.26 | 0.53 | 0.22 |
| m_dense, H | Moi.B | 0.65 | 0.55 | 0.59 | 0.30 | 0.61 | 0.26 |
| m_dense, H | Moi Leak.L | 0.53 | 0.45 | 0.47 | 0.23 | 0.49 | 0.21 |
| m_dense, H | Moi.L | 0.71 | 0.59 | 1.18 | 0.45 | 1.69 | 0.45 |
| m_dense, H | Moi Leak.S | 0.73 | 0.64 | 0.70 | 0.35 | 0.71 | 0.31 |
| m_dense, H | Moi.S | 0.70 | 0.60 | 0.71 | 0.33 | 0.91 | 0.33 |
| restaurant | Moi Leak.B | 0.70 | 0.29 | NA | NA | NA | NA |
| restaurant | Moi.B | 0.72 | 0.31 | NA | NA | NA | NA |
| restaurant | Moi Leak.L | 0.75 | 0.30 | NA | NA | NA | NA |
| restaurant | Moi.L | 0.76 | 0.30 | NA | NA | NA | NA |
| restaurant | Moi Leak.S | 0.74 | 0.31 | NA | NA | NA | NA |
| restaurant | Moi.S | 0.74 | 0.31 | NA | NA | NA | NA |
| sz_taxi, 15T | Moi Leak.B | 0.90 | 0.69 | 0.71 | 0.33 | 2.16 | 0.38 |
| sz_taxi, 15T | Moi.B | 0.84 | 0.69 | 0.64 | 0.46 | 2.42 | 0.38 |
| sz_taxi, 15T | Moi Leak.L | 0.78 | 0.69 | 0.71 | 0.46 | 2.14 | 0.38 |
| sz_taxi, 15T | Moi.L | 0.82 | 0.69 | 0.60 | 0.47 | 2.24 | 0.38 |
| sz_taxi, 15T | Moi Leak.S | 0.95 | 0.69 | 0.65 | 0.47 | 2.12 | 0.38 |
| sz_taxi, 15T | Moi.S | 1.11 | 0.70 | 0.60 | 0.47 | 2.30 | 0.39 |
| sz_taxi, H | Moi Leak.B | 0.64 | 0.62 | NA | NA | NA | NA |
| sz_taxi, H | Moi.B | 0.72 | 0.64 | NA | NA | NA | NA |
| sz_taxi, H | Moi Leak.L | 0.63 | 0.64 | NA | NA | NA | NA |
| sz_taxi, H | Moi.L | 0.70 | 0.64 | NA | NA | NA | NA |
| sz_taxi, H | Moi Leak.S | 0.65 | 0.66 | NA | NA | NA | NA |
| sz_taxi, H | Moi.S | 0.77 | 0.65 | NA | NA | NA | NA |

Figure: (a) Foundation model forecasts sampled on Bizitobs_l2c hourly dataset with medium prediction length.
Refer to caption: x9.png

## Peer Reviews

confidence: 4
contribution: 3
initial_rating: 5
presentation: 3
questions: See weaknesses.

My critical concerns centered around the rationale of selecting/adding/filtering evaluation datasets and sampling frequencies, the missing discussion of some highly related studies, and incomplete experimental results and analyses.
soundness: 2
strengths: - Covering a diverse range of datasets and sampling frequencies, which can greatly facilitate our comprehensive understanding of existing time-series models
- Distinguishing pre-training and evaluation datasets, which can assist the comparison between supervised trained models (in-domain generalization) and pre-trained models (zero-shot forecasting)
- Conducting substantial experiments on these datasets (# experiments = # datasets x # sampling frequencies x  # models, such enormous computation!)
summary: This paper targets at a critical and longstanding challenge in time-series forecasting research, lacking a unified, comprehensive, and diverse benchmark for evaluation. To address this challenge, GIFT-Eval is developed, encompassing 28 datasets across seven domains and sampled in ten different frequencies.
weaknesses: Overall, I appreciate and respect this paper, given its important research focus and huge amounts of experiments performed.

However, I find the current version of this paper is still on an early stage. Below I elaborate on my significant concerns, which need to be properly addressed before it reaches an acceptance bar.

### 1. The Necessity of Additional Evaluation Datasets

MOIRAI [1] has already done an excellent job in collecting pre-training datasets and conducting comprehensive evaluations on widely recognized datasets typically used to evaluate time-series forecasting models in the literature. The datasets introduced in this paper heavily overlap with them while creating a new division for pre-training and downstream evaluation. Currently, this paper argues improvements in quantity, namely more evaluation datasets and sampling frequencies, are included in the evaluation. However, the necessity of introducing more evaluation datasets and how this approach reveals unique insights not mentioned in previous research is relatively weak.

Covering an excess of comparison datasets can create a significant burden for resource-limited research groups to perform further research and for reviewers to check and compare results. This may lead to redundant model training and evaluation, which is energy-inefficient. I strongly recommend the authors analyze the uniqueness and necessity of any dataset introduced as a new testbed.

For example, when incorporating a new dataset, does it cover distinctive patterns that rarely exist in existing benchmarks, thereby delivering unique insights on different pros and cons of model designs?

The same question extends to the different sampling frequencies applied to each dataset. Is it necessary to include all sampling frequencies for every dataset? Does including the most prominent sampling frequency, which may be defined by different business scenarios, already provide sufficient and fair comparisons of different models?

According to Tables 20, 21, and 22, the total comparison covers 28 datasets, each with multiple sampling frequencies. Are these experimental comparisons overly redundant? For instance, can you identify a compact set of evaluation datasets, covering unique datasets accompanied by selected sampling frequencies, that still reveal comprehensive information while being much more energy-efficient, user-friendly, and economical? Perhaps you could calculate the real rank for the result comparison matrix presented in these tables (# datasets x # models).

Moreover, analyzing the data patterns in Figure 1 is beneficial, but how does the pattern coverage of these new benchmark datasets differ from that of existing benchmarks? Do newly introduced datasets include more pronounced trends and seasonality, or large entropy, etc.?

### 2. Lack of Some Related Work and Baselines

The comparison with previous benchmarks omits some classic and recent studies specializing in probabilistic forecasting. For instance, gluon-ts [2] is a Python package for probabilistic time-series forecasting that also provides a robust interface for accessing multiple time-series datasets. Built on gluon-ts, pytorch-ts [3] includes more advanced probabilistic forecasting models based on deep generative models. ProbTS [4] is another benchmark study offering a unique perspective by comparing capabilities in delivering point versus probabilistic forecasts, short versus long forecasts, and associated preferences in methodological designs. Specifically, ProbTS should be compared in Table 1, as it is highly relevant to your work in comparing both classical and foundation models. It unifies comparison conditions, covers diverse forecasting horizons and data patterns, calculates dominant data characteristics (such as trend, seasonality, and non-Gaussianity), and associates them with the strengths and weaknesses of different model designs.

Moreover, other time-series foundation models have been developed beyond MOIRAI, chronos, and TimesFM. Notably, Timer [5] and UniTS [6] have been accepted at conference proceedings and have publicly released their implementations. These models should at least be discussed in the related work section and, ideally, be included in your experimental comparisons.

Additionally, the paper could benefit from including more advanced probabilistic forecasting baselines, such as TimeGrad [7], CSDI [8], and their predecessor GRU NVP [9]. ProbTS has highlighted the unique advantages of these methods in delivering short-term distributional forecasting. Moreover, a simple combination of GRU NVP with RevIN [10] has demonstrated very competitive performance for both short-term and long-term forecasting. Including these more powerful probabilistic models is crucial, as merely adding probabilistic heads over forecasting models like MOIRAI and DeepAR does not sufficiently capture complex data distributions that extend beyond closed-form probabilistic distribution functions.


### 3 Some Missing Details and Analyses in Experiments

The use of MAPE as an only metric for evaluating point forecasts is somewhat "biased." I recommend referring to N-BEATS [11] and including metrics such as sMAPE and ND (normalized deviation, equivalent to normalized MAE) for a more comprehensive evaluation.

Regarding hyperparameter search for deep learning baselines, there is a notable omission in tuning their lookback lengths. This can be an extremely critical factor to adjust, given the diversity of datasets and sampling frequencies. Appendix A indicates that this tuning was performed only for MOIRAI. The same process should be applied to other baselines, including supervised learning models and pre-trained foundation models.

Moreover, regarding three foundation models used in the experiments, only MOIRAI is re-trained on this new data split while TimesFM and Chronos are compared with their released model checkpoints. There is a risk of data leakage in these experiment comparisons. For example, TimesFM has leverage the Electricity dataset for pre-training while this dataset is also used for evaluation in this paper. This may explain why TimesFM performs extremely on "Electricity, short, H" (line 1285, Table 20), but this good performance may come from test data leakage. Therefore, to provide a fair and comprehensive comparison across TimeFM, Chronos, MOIRAI, and other representative time-series foundation models, re-training them following the new data split could be a necessary step.

When comparing supervised time-series models with zero-shot foundation models, it is crucial to investigate the effect of allowed lookback length on forecasting performance. As revealed in MOIRAI, the lookback length significantly influences model performance, as MOIRAI employs an additional hyper-parameter adaptation process on the lookback data, unlike others.

The current analysis lacks depth across multiple experimental configurations. Presently, only aggregated comparisons are provided, without detailed analyses to derive new insights. For instance, why do some foundation models yield better zero-shot forecasting results on certain datasets than models specifically tuned for those datasets? Is it because pre-training datasets encompass more related patterns, pre-trained models have a larger model capacity, or classical models are not well-tuned, particularly regarding lookback length, modeling layers, or model designs? Merely presenting results without in-depth analysis can be detrimental to the research community, as others may expend significant efforts to re-analyze numerous results, which should be addressed within this benchmark.

Additionally, there is a lack of explicit connection with existing benchmarks to validate the reliability of the experiments conducted. For example, evaluation datasets in Tables 20, 21, and 22 cover some classical datasets like ETTh, Electricity, Solar, and M4. Comparing your results on shared datasets with existing studies could demonstrate the reliability of your experimental protocols. Moreover, some widely used datasets in the literature, such as Traffic, Wikipedia, and Exchange, appear to be excluded. Please clarify the reasons for their exclusion.

I suggest maintaining existing classical benchmarks as they are while introducing new datasets and sampling frequencies that cover unique patterns. This approach allows for consistency with existing benchmarks, showcases the correct reproduction of existing models, demonstrates the necessity of new datasets and sampling frequencies, and reveals what can be discovered given this new benchmark.

[1] Unified Training of Universal Time Series Forecasting Transformers, https://arxiv.org/abs/2402.02592

[2] gluon-ts, https://www.jmlr.org/papers/v21/19-820.html

[3] pytorch-ts, https://github.com/zalandoresearch/pytorch-ts

[4] ProbTS, https://arxiv.org/abs/2310.07446

[5] Timer, https://arxiv.org/abs/2402.02368

[6] UniTS, https://arxiv.org/pdf/2403.00131

[7] Autoregressive Denoising Diffusion Models for Multivariate Probabilistic Time Series Forecasting, https://arxiv.org/abs/2101.12072

[8] CSDI: Conditional Score-based Diffusion Models for Probabilistic Time Series Imputation, https://arxiv.org/abs/2107.03502

[9] Multivariate Probabilistic Time Series Forecasting via Conditioned Normalizing Flows, https://arxiv.org/abs/2002.06103

[10] RevIN, https://openreview.net/forum?id=cGDAkQo1C0p

[11] N-BEATS, https://arxiv.org/abs/1905.10437

---

confidence: 5
contribution: 2
initial_rating: 5
presentation: 2
questions: Please see Weaknesses.
soundness: 2
strengths: * Foundation models are a new and promising research topic in time series forecasting. It is of value to establish a benchmark for evaluating these models.

* This paper does a lot of work in collecting large-scale data for evaluation and running various baseline models on these various datasets.
summary: This paper introduces GIFT-Eval, a benchmark aimed at evaluation for time series foundation models across diverse datasets. GIFT-Eval encompasses 28 datasets and divides them according to 4 characteristics and 6 features. It also includes a non-leaking pretraining dataset. It then evaluates 17 different baseline models, including 4 types of foundation models and some statistical and deep learning models on this benchmark. Based on the evaluation results, it discusses different models in the context of various benchmark characteristics and offers some qualitative analysis.
weaknesses: * This paper emphasizes the inclusion of a non-leaking pretraining dataset, but its value and usage are not clear. Is this dataset used to re-train all foundation models instead of using their public checkpoints? Is it necessarily better than the original pretraining datasets of each foundation model? Since the application on downstream data is the main goal of foundation models, do we really need to keep consistency in pretraining to evaluate these models as discussed in the Introduction?

* Section 3.1.1 mentions covariates in time series forecasting, but it is unclear how are the covariates considered in this benchmark. Can all baselines in the benchmark perform forecasting with covariates?

* The divisions of some characteristics such as variates and frequencies are not reasonable enough, and it is unclear why we should evaluate methods according to these characteristics. For example, Multivariate v.s. Univariate may not be the intrinsic differences that influence forecasting, as even some multivariate datasets do not have strong correlations between variates. So simply claiming that some models perform best on multivariate or univariate datasets may be misleading.

* The experiments mainly point out that some specific models perform best in some specific datasets. There are not many consistent conclusions from the evaluations that help us understand the characteristics of different models. Some results or claims are also confusing. In Page 7, how can we get the conclusion that ‘foundation models consistently outperform both statistical and deep learning models’ from Table 6, since PatchTST also performs very well? In Page 8, what is the meaning of ‘This trend indicates that the fine-tuning of foundation models effectively captures longer-term dependencies’? In Page 9, why does Moirai, which is a foundation model considering multivariate correlation, perform best on univariate datasets instead?

* The qualitative results only show results in some special cases. It is doubtful whether these phenomena and analyses are general to all different data. It is also confusing that different data are used to evaluate foundation models and other models, which makes these models incomparable.

---

confidence: 5
contribution: 3
initial_rating: 5
presentation: 3
questions: Regarding the creation of GIFT-EVAL:
1. As mentioned in the weakness, GIFT-EVAL seems to be a straightforward ensemble of some available datasets. What are the reasons behind the selection of this particular ensemble vs the datasets, especially given the data analytics done on them?

Regarding benchmarking:
1. My major concern of the work: MAPE is well recognized as a bad point forecasting metric on its own, due to e.g. its favoring underprediction, sensitivity near ground truth 0 (this is recognized by the authors too), (see e.g., [1] and many community posts online). It would be helpful to switch to or also report other more robust metrics, e.g. MASE. 
2. In case normalized metrics are used, consider reporting geometric means [2].
3. Given the 6 properties of component datasets, it would be helpful to see the benchmark results sliced on these properties as well.

[1] Goodwin, Paul, and Richard Lawton. "On the asymmetry of the symmetric MAPE." International journal of forecasting 15.4 (1999): 405-408. 
[2] Fleming, Philip J., and John J. Wallace. "How not to lie with statistics: the correct way to summarize benchmark results." Communications of the ACM 29.3 (1986): 218-221.
soundness: 2
strengths: This work address an important and pressing problem of the time series forecasting community that there lacks a large and comprehensive common benchmark. The collection and complication of the datasets itself is a significant result.
summary: In this work the authors introduce a large collective time series benchmark named GIFT-EVAL. They demonstrate the diversity and legitimacy of this benchmark with the analyses on both the characteristics of involved datasets and the benchmark results of state-of-the-art supervised and foundation time series models on it.
weaknesses: Despite the comprehensive data analyses, the paper currently lacks the reasoning behind the selection of GIFT-EVAL components. And the empirical study follows suboptimal standards which makes the results less convincing. See questions.

---

confidence: 4
contribution: 3
initial_rating: 5
presentation: 2
questions: See W1-W6.
soundness: 3
strengths: S1: Model Level: GIFT-Eval provides a comprehensive evaluation for time series forecasting models, including statistical models, deep learning models, and foundation models.

S2: Experiment and Dataset Level: GIFT-Eval conducts extensive experiments, testing on 28 different test datasets, and also provides a non-leaking pretraining dataset to better evaluate foundation models
summary: The existing benchmark evaluations are incomplete (lacking evaluation of foundation models), so the paper introduces the General Time Series Forecasting Model Evaluation (GIFT-Eval), including statistical models, deep learning models, and foundation models. It tests on 28 different test datasets and provides a large-scale pretraining dataset to better evaluate foundation models. Finally, the paper offers a qualitative analysis that spans both deep learning models and foundation models.
weaknesses: W1: I believe that analyzing foundation models based on four time series characteristics—domain, frequency, prediction length, and the number of variates—combined with six time series features—trend, seasonality, entropy, Hurst, stability, and lumpiness—is not very meaningful, especially regarding the number of variates. I don't think it has any relation to these six features. Why not directly analyze the time series features for each test dataset, and then evaluate the performance of foundation models on various datasets to assess their strengths and weaknesses concerning these six features?

W2: The paper only analyzes the features of the test data. It is necessary to include an analysis of the pretraining data as well. This would help better assess whether the foundation models perform well on these features due to the presence of these characteristics in the pretraining data itself, the generalization ability of the foundation models, or other reasons.

W3: Please elaborate on the details of the data splitting for train/val/test datasets, specifically how the validation data is constructed. What does the statement "The final window of the training data serves as validation" mean, and why is a common splitting ratio like 7:1:2 or 6:2:2 not used?

W4: The experimental settings are unclear, for example: what are the input and output lengths for short-term, medium-term, and long-term forecasting? Are the foundation models performing zero-shot or full-shot forecasting in Tables 6-10? Why are common metrics like MSE and MAE not chosen for point forecasting?

W5: About experimental results: 1) PatchTST performs best on multivariate datasets, so why is Moirai, a model specifically designed for variable correlation, not used? Also, why do foundation models perform well on univariate data but not on multivariate data? 2) The paper states that Moirai has good prediction performance in short-term forecasting; does this conclusion contradict the original Moirai paper, which uses very long input sequences?

W6: The qualitative analysis is relatively limited; more in-depth analysis (not just visualization) may be needed to highlight the characteristics of the foundation models and draw more meaningful conclusions.

Meta-review: The paper introduces GIFT-Eval, a benchmark designed to evaluate time series forecasting models, including statistical, deep learning, and foundation models. It features 28 datasets spanning diverse domains, frequencies, and prediction lengths, alongside a non-leaking pretraining dataset to ensure data integrity. While the benchmark’s scope and extensive experiments highlight significant effort, key issues persist. The connection between dataset characteristics and forecasting performance remains unclear, and inconsistent evaluation configurations, such as varying input lengths across models, raise fairness concerns. Additionally, the analysis lacks depth, providing limited actionable insights about model performance. The role and impact of the pretraining dataset on results also remain ambiguous. Despite its potential as a valuable tool for the research community, these unresolved issues lead to a borderline rejection recommendation, with encouragement for improvement in future iterations.

Decision: Reject