Venue: NeurIPS 2021 Track Datasets_and_Benchmarks Round2

Reviews:

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: summary and contributions: - a set of 25 multivariate time series datasets
- a new format to store time series
- Benchmark tests using a good amount of different methods
strengths: - A new repository dedicated to time series, especially multivariate time series
- A new format designed to store time series
- A benchmark that presents results for a good amount of time series methods
weaknesses: - Only seven of the datasets were not yet public
correctness: No incorrectnesses were found.
clarity: Very well written
relation to prior work: Only 23 of the 30 datasets were not previously publicly available. Anyhow, these 23 datasets were in different formats. 
additional feedback:
review_title: A new repository with 25 datasets on multivariate time series plus 5 very long univariate time series

final_score: rating: 6: Marginally above acceptance threshold
review_content: summary and contributions: This paper presents the Monash Time Series Forecasting Archive, which contains 58 datasets (derived from 30 primary datasets, including 5 single very long time-series) across diverse domains and time-series attributes. Among them, 23 datasets are re-processed from publicly available datasets and 7 datasets are published by authors. In the experiments, the paper evaluates 13 different baseline forecasting models using a fixed origin evaluation scheme with 10 error metrics. Moreover, it gives a comprehensive analysis of the dataset characteristics through feature extraction and PCA. The effort of building this archive has a certain contribution to the research community of time series. 

strengths: 1.	It collects and releases a collection of datasets from different domains. The attributes of time-series are very diverse, including various length, frequency, containing or not containing missing values, and univariate/multivariate. It will facilitate new design of time-series forecasting models in a comprehensive and complete evaluating protocol.
2.	It gives a comprehensive analysis of the dataset characteristics through feature extraction and PCA. 
3.	It notifies the metrics problem of previous evaluation of time-series forecasting. To address these issues, the authors analyze the performance of baseline model with 10 error metrics and suggest that the selection of error metrics should be decided by specific applications. 
weaknesses: 1.	The paper only compares 13 baselines, while some recently published works are missed. It would be more informative to compare with more state-of-the-art models of time-series forecasting, such as Informer [1], StemGNN [2], and ST-GCN [3]. Moreover, as new time-series forecasting methods continuously emerge, it’s better to describe the unified evaluation protocol in more detail and provide an interface to implement new forecasting algorithms in a well-controlled setting.

2.	Time-series lack a unified metrics for evaluation. This paper raises this problem but leaves the decision to the owners of specific applications. To enable a fair comparison, at least the author should discuss the limitations of certain metrics and give suggestions about which metrics could be used as primary metrics for a certain kind of time-series. 

[1] Zhou, Haoyi, et al. "Informer: Beyond efficient transformer for long sequence time-series forecasting." Proceedings of AAAI. 2021.

[2] Cao, Defu, et al. "Spectral Temporal Graph Neural Network for Multivariate Time-series Forecasting." Proceedings of the NeurIPS 2020.

[3] Yu, Bing, Haoteng Yin, and Zhanxing Zhu. "Spatio-Temporal Graph Convolutional Networks: A Deep Learning Framework for Traffic Forecasting." IJCAI. 2018.

correctness: The dataset is constructed in a sound way. The experiments are performed reasonably. 
clarity: This paper is well written. 

relation to prior work: It has discussed the relation to prior works. 
additional feedback: Refer to Weakness. 

Figure 1 visualizes distinct density populations for various datasets. Therefore, it would be interesting to discuss the performance of baselines in terms of different density populations.
review_title: Monash Time Series Forecasting Archive

final_score: rating: 6: Marginally above acceptance threshold
review_content: summary and contributions: The paper introduces an archive consisting of a total of 30 time series datasets. 7 of the dataset are newly introduced in this paper remaining datasets have already been publically available. The paper also introduces TSF formate which is a new formate to save time series data. The paper benchmarked different machine learning methods both traditional methods and neural methods.
strengths: - The paper is well written and clear.
- This benchmark has huge potential in advancing state of the art forecasting methods (if processed correctly).
- The code is clear and can be easily reproduced.
- Looking at the previous reviews in the supplementary material provided by the authors; it seems like most issues have been addressed in the new draft.
weaknesses: - The paper shows that statical methods tend to outperform neural networks in most datasets which is quite surprising since given enough capacity and data a neural network should be able to perform as well as simple statical methods. Since this was quite surprising for me I looked into the code to reproduce the results and one thing I noticed is that data is downloaded from the archive and passed to the networks. I believe the data is in its raw form (please correct me if I am wrong) so no data preprocessing is done whatsoever before feeding it into the networks which might be the reason behind such results.    Preprocessing timeseries is tricky especially that different datasets might need different processing steps.

A complete forecasting benchmark would involve the following:
Download data -> Process data -> Feed to models -> evaluate models.

Currently, step two is missing which might lead to undesirable behavior, especially for neural networks.
correctness: Currently, the benchmark is missing preprocessing data phase which might lead to incorrect results when comparing different machine learning methods
clarity: Yes
relation to prior work: Yes
additional feedback: - Please clarify, is the data on the archive saved as the raw data or preprocessed?
- Are all the results for univariate forecasting? using only a single variable or all available features are used as inputs? Since it seems that neural network only get univariate features (take covid death as an example)

########Post rebuttal

Thank you for the clarification.
Given that preprocessing is done within the GluonTS, I raise my score to 6.
review_title: Strong datasets but unfair benchmarking due to lack of preprocessing for neural models

Meta-review: All the reviewers agree on acceptance. The authors are encouraged to use the reviewer feedback to improve the paper for its camera-ready version.
Decision: Accept