Venue: NeurIPS 2021 Track Datasets_and_Benchmarks Round1

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: This paper provides a repository of multi-univariate time series, as well as an evaluation on these datasets to showcase their utility. This contrasts most available time series datasets, which are typically univariate or classification problems for multivariate time series. This repository brings together many datasets, to get a more comprehensive set of multi-univariate time series to facilitate research on time series prediction and forecasting. 

*Note: I gave these datasets the name “multi-univariate time series”, because the authors did not give them a name. To easily reference this in this review, I decided to make up a name, though it is clearly not an ideal name. 
strengths: This repository is highly needed. Though an anecdote, my group has often been in search of multivariate time series, often finding few suitable options. This work focuses on datasets, with multiple related univariate time series (i.e., multi-univariate), with forecasting still occurring on each univariate series. Though a different problem setting, this dataset nonetheless expands the set of available time series datasets. 

Further, steps towards repositories for multi-univariate and multivariate time series is critical for machine learning. Arguably, most real data includes time, and making more time series data available may help the whole community refocus on this common and important setting.

There are a variety of datasets, some of which are reasonably long (often time series can be quite short) and some of which have many series (again, providing more information to make forecasting feasible). The more data, the more it is feasible to use learning approaches.

The benchmarking results, with basic models, is important and useful. This statement in the work is true: “many works in the machine learning space are notoriously weak when it comes to proper benchmarking for time series forecasting”. Improving evaluation would be a useful outcome of this paper. 

weaknesses: There could be a more clear contrast between multivariate time series and this multi-univariate setting. Can these datasets also be used for multivariate time series forecasting? Are they simply different time series, that are related but not perfectly temporally related? (e.g., I might not be able to stack them to multivariate inputs and outputs?) 

The following is a minor weakness, more so about typical time series datasets; it is a call for this work to be even more ambitious. Time series datasets, in the past, have seemed limited by the mindset of time series forecasting. Filtering approaches, or expert-driven approaches, have been used for which having less data point and/or less frequent sampling was reasonable. This repository could be further improved by discussing any limitations in how time series data is gathered, and considering atypical sources of time series data with high frequency sampling (e.g., robotics). People have a predisposition to presume that sampling every 30 minutes is sufficient (for example, in one of the energy datasets in this repo), whereas learning algorithms might be able to benefit from more frequent sampling. 

Similarly, to attract more to researchers to time series analysis, it would be useful to include a more familiar strategy to those outside traditional time series analysis. For example, it could use useful to include a recurrent neural network approach, with additional inputs that reflect the seasonality. And it could be useful to explain why this setting is of interest, and how it compares to other time series problems (again, say the different between multivariate and this setting). 

correctness: The work is correct.

clarity: The paper is written clearly. 

Section 4.2 about the different metrics could be improved. There is a big block of text describing the metrics. I did not understand snaive, and the following sentence:  “As therewith the MASE compares the forecasts of longer horizons  (up to one week) with the in-sample snaïve forecasts obtained with shorter horizons (one day), the MASE values of multi-seasonal datasets are considerably greater than one across all baselines. ”

Further, the section is set up to say: this is hard and an open question. Instead it could be set up to say: here is how we address this issue. 

Organizationally, it would be clearer to have one section about the Feature Analysis and one about the Baseline Results. The separation into methods and results is reasonably common, but nonetheless these are such different experiments its worth compartmentalizing into sections. 
relation to prior work: Previous repositories are discussed. This does indeed appear to be the first repository focused on multi-univariate time series, with forecasting in mind. Many previous multivariate time series are classification problems.  
additional feedback: The paper says: “Furthermore, the error measures are not directly comparable across datasets as we consider different forecasting horizons with different datasets.” —> I wonder if there is a sensible way to provide normalized metrics, that also normalize for horizon.
review_title: Official Review

final_score: rating: 5: Marginally below acceptance threshold
review_content: summary and contributions: The paper creates an archive containing a total of 26 time series datasets, the datasets are stored in tsf formate. The paper then benchmarks multiple statical methods for time series forecasting reporting 10 error metrics for a particular dataset.

TSF formate is a new formate to save time series data introduced in the paper.

strengths: There is no standard datasets for time series forecasting comparison so such a benchmark is highly needed by the  research community.
The paper included 26 datasets which is a very decent number.
For fair comparison the paper used 10 different metrics to compare methods.

weaknesses: The contribution of the paper is minimal. The paper does not introduce any new datasets, it only collects existing datasets in one location in a "tsf" formate.

Right now (Sunday, July 4th, 11:20 AM EST) there is no code in python to reproduce the results or evaluate methods on the datasets there is only a single file data_loader.py that shows how to load the convert "tsf" into a python dataframe.

The paper compared forecasting methods including, Exponential Smoothing, ARIMA, Simple Exponential Smoothing, Theta, TBATS, DHR-ARIMA and PR; all of which are statistical methods no machine learning or deep learning methods were benchmarked. Please see [1] as an example.

The paper proposed the use of 10 error metrics for evaluation; however, there is no way to reproduce the results (no code and not enough details in paper).

[1] Bauer, André, et al. "Libra: A Benchmark for Time Series Forecasting Methods." Proceedings of the ACM/SPEC International Conference on Performance Engineering. 2021.
correctness: From the paper description, it seems that the evaluation methods and experiment are designed appropriately. However, there is no way to reproduce the results.
clarity: Yes
relation to prior work: Work very similar to [1]

[1] Bauer, André, et al. "Libra: A Benchmark for Time Series Forecasting Methods." Proceedings of the ACM/SPEC International Conference on Performance Engineering. 2021.

additional feedback: This paper has the potential to be very useful for the research community. However, it still needs some work.
It would be very beneficial to show a complete pipeline that includes:
A) Loading data
B) Processing data
C) Applying method
D) Evaluate using different metrics


Since you are submitting the paper to Neural Information Processing Systems, consider benchmarking state of the art neural networks.
review_title: Collecting many existing datasets into one location

final_score: rating: 5: Marginally below acceptance threshold
review_content: summary and contributions: As opposed to single time series forecasting, global or multivariate time series forecasting trains a single model across all series that need to be forecast to exploit cross-series information in the forecasting tasks. This work aims to fill the gap in global or multivariate time series forecasting by creating a suitable archive that contains 20 publicly available datasets, each of which has time series from similar sources. A benchmarking study of some baseline forecasting methods using the archive is reported based on 10 different error metrics.

strengths: * Compared with some related previous attempts, this archive contains more time series datasets and covers a wider range of application sectors.

* This work also introduces a new format, based on Weka’s ARFF file format, for storing time series data to overcome some of the limitations of the .ts format used in the sktime time series repository.

weaknesses: * Modern deep learning methods for time series forecasting, such as encoder-decoder recurrent neural networks and graph neural networks, have not been included in the benchmarking study.

* The work does not offer much insight for the performance comparison of different forecasting methods. It presents 10 error metrics, but some guidance on how to make use of them under what situations is missing.

correctness: The evaluation is done correctly, but what is also needed is how to choose from the 10 error metrics when given a specific forecasting task to do.

clarity: The paper is clearly written.

relation to prior work: There are some related previous attempts as reviewed in Section 1. This work aims to include more datasets in the archive and have higher diversity in terms of the application sector.

additional feedback: Post-rebuttal comments:
While the authors are moving in the right direction to make their benchmarking study more complete, I don't feel comfortable with recommending for acceptance based solely on promises made by the authors to include the other experiment results in the final version. I would rather wait until a complete version becomes ready before making a decision. The authors are recommended to try it again in the second round.
review_title: Benchmark for supporting global or multivariate time series forecasting

Meta-review: This paper posed a difficult decision since the reviewers were generally in favor of the submission due to the need of a benchmark in this area. However, one concern raised by two reviewers is the lack of baselines implemented in Python, especially baselines based on neural network models. NeurIPS is certainly not limited to a specific language or class of methods, and I find it a valuable aspect of the submission that they evaluated a wide range of methods implemented in R. Nevertheless, it is crucial for a benchmark to include relevant baselines.

The authors have already started the process of adding more baselines, which will significantly strengthen the paper. This leads to the complicated situation where the submission has to be partially evaluated on promises made during the response phase. The fact that we are in the first submission round of the datasets & benchmark track with a second submission date at the end of August opens a unique possibility here to give the authors more time to incorporate the reviewers' feedback but still submit to this year's NeurIPS. This will then allow reviewers to judge the comprehensively updated version of the paper.

So while I recommend rejecting the paper in the first round of the datasets & benchmarks track, I *strongly* encourage the authors to submit their updated paper to the second round (the current submission deadline is August 27th). This should be sufficient time to implement the changes suggested by the reviewers and will make the paper a strong submission that adds a valuable benchmark. Moreover, if the authors make it easy to load the data in both Python and R with the same preprocessing etc. and provide detailed instructions for running the baselines irrespective of the language used, the submission can play an important role in bridging the Python and R worlds of time series forecasting.

Decision: Reject