Venue: NeurIPS 2021 Track Datasets_and_Benchmarks Round2

Reviews:

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: summary and contributions: This work contributes 131 different chaotic dynamic system simulations and 16 precomputed simulations. It also includes code for resimulation and computation of features of the dynamic system such as lyapunov exponent. 
strengths: This dataset seems great for benchmarking Ml models and deep learning models which can model dynamics continuously or discretely. As their tests show this can include recurrent neural networks, transformers and other statistical model.  Because of known ground truth one can test interpretability and ability to reverse engineer or identify systems as well. 
weaknesses: The suite uses chaotic dynamic systems which may be hard for ML methods in general, non-chaotic dynamics could be included as well in future versions. 

I would have liked to see a test on neural ODE systems, these seem ideally suited for this benchmark dataset. 

The github has a bunch of todos particularly with regards to lyapunov exponent calculation, hopefully these can be addressed soon.
correctness: The dataset is soundly constructed. 
clarity: The paper is well written. One could clarify how these test cases could aid the development of machine learning models. Presumably chaos prediction does not always have to be the end goal, but rather this could be a test of dynamic neural network capabilities. 
relation to prior work: Previous work is adequately discussed. 
additional feedback: I would include some non-chaotic systems as well.
review_title: Dynamics systems benchmarks

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: summary and contributions: The paper introduces a dataset consisting  of 131 known chaotic dynamical systems. The paper benchmarks the performance of 16 forecasting models including deep learning, statical, classical and machine learning methods on forecasting chaotic time series task. The paper shows the usefulness of dataset in transfer learning of to improve the classification accuracy of UCR datasets. The paper shows that training can be accelerated by encouraging the network to oversample sparser regions of the underlying attractor.  Finally the paper use the proposed dataset for data-driven modeling in the form of a symbolic regression task.

strengths: - The paper creates a huge dataset repository that can easily be extended.
- The paper compares multiple methods for time series forecasting of chaotic systems and show that Neural Networks tend to outperform statical and traditional methods on this particular task.
- The paper shows that the datasets can be used for transfer learning on different classification tasks for UCR dataset (this is my favorite contribution) since the usage of chaotic time series to improve the accuracy of non chaotic is novel and has many useful applications.
- The paper shows how training can be accelerated by upsampling sparser regions which reduced the time and  error compared to training full epoch.
- The paper showed that this dataset can be utilized for data-driven modeling in the form of a symbolic regression task.
weaknesses: - The usage of term interpretable might not be misleading, the data itself is interpretable due to its mathematical properties however it us unclear how this data can improve or benchmark model interpretability in time series [1]. 
-  As mentioned in the paper, all included chaotic systems that have previously appeared in published works.



[1]Ismail, Aya Abdelsalam, et al. "Benchmarking deep learning interpretability in time series predictions." arXiv preprint arXiv:2010.13924 (2020).
correctness: Yes
clarity: Yes
relation to prior work: Yes
additional feedback: Minnor comments:
Line 292: I think you meant table 1 instead of table 3
review_title: Cha

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: &nbsp;

The paper introduces a set of 131 chaotic dynamical systems spanning application domains including climatology, astrophysics and biochemistry. The author makes a strong case for the niche which the dataset fills within the canon of literature on general time series analysis; through illustrative experiments the author showcases some unique properties of chaotic dynamical systems whilst also highlighting use-cases for the dataset as a general tool for transfer learning and data augmentation.

I believe the dataset is an interesting and useful contribution to the machine learning community and as such veer on the side of acceptance. I have a few minor concerns which, if addressed in the rebuttal, may warrant an increased score.

&nbsp;
strengths: &nbsp;

1. The dataset has demonstrable utility for transfer learning on general time series classification tasks, a subject of broad interest to the wider ML community.

2. The author does a good job in highlighting some of the unique properties of chaotic dynamical systems that may encourage the development of bespoke methods for forecasting in this domain, an example of which being the application of importance sampling.

3. The dataset is clearly documented with a full list of all systems comprising the dataset described in the supplementary material. The codebase is in general well documented (see minor comments below). 

4. Ideas for future applications are dispersed throughout the paper.

5. The paper is well written and is, in general, understandable by a lay machine learning audience (see minor comments below).

&nbsp;
weaknesses: &nbsp;

## __MAJOR POINTS__

&nbsp;

1. It may be helpful to provide a **glossary of terms** (perhaps in the SI) including basic definitions of terms such as attractor (e.g. set of states to which a system evolves), strange attractor (attractor exhibiting fractal structure) etc. This should help make the subject matter more accessible to a lay machine learning audience.

2. Some explanation of the links between strange attractors and chaotic systems may be beneficial in the introduction e.g. the existence of **strange attractors which are not chaotic** [1].

3. In relation to Task 1: Forecasting, while the findings are interesting, it may be helpful to users to provide some ideas about how to use this task as a benchmark for developing models that are appropriate for forecasting in chaotic dynamical systems?

4. For Task 4: Data-driven model inference and prediction, is the implication that the benchmark should be extended to be more challenging through the inclusion of aspects such as measurement noise? The implication seems to be that the chaotic systems benchmark in its current form is too simple to differentiate between state-of-the-art models?

&nbsp;

## __MINOR POINTS__

&nbsp;

1) Renaming the sections may make the paper easier to read e.g. having section 3.1 as the conclusion in a separate section 4.

2) Increasing the fontsize of figure 4 for example, should be possible given the extra page for the camera-ready version.

3) How are the errobands in Figure 3 computed?

4) Table 1, how are the standard errors computed?

5) Task 3 appears to refer to a Table 3 that doesn't exist?

6) The readability of Table 1 would be improved if the task was defined in the caption alongside a description of how the standard errors were computed.

7) In the conclusion, it is stated that importance sampling is an application of the dataset but is this limited to applications involving chaotic attractors? The use of transfer learning for general time series classification appears to be more general.

8) In the section on related work, is it necessary to cite general time series benchmarks? It seems that this literature may be too large to provide a detailed account of all benchmarks e.g. MIMIC [2] in healthcare for example.

&nbsp;

## __REFERENCES__

&nbsp;

1) Grebogi et al., Strange attractors that are not chaotic. Physica D: Nonlinear Phenomena. 1984.

2) Johnson et al., MIMIC-III, a freely accessible critical care database. Scientific Data. 2016. 

&nbsp;
correctness: &nbsp;

To the best of my knowledge the results of the author appear to be correct. The statistical hypothesis tests (under the assumption of appropriate hyper-parameter tuning) lend credence to the claims made by the author.

&nbsp;
clarity: &nbsp;

Overall very good, minor points noted above.

&nbsp;
relation to prior work: &nbsp;

Adequately addressed.

&nbsp;
additional feedback: &nbsp;

All points covered in the main response.

&nbsp;
review_title: Interesting Dataset with Dual Applications to Chaotic Dynamical Systems and General Time Series Analysis

final_score: rating: 6: Marginally above acceptance threshold
review_content: summary and contributions: This work aims to benchmark temporal models of varying complexities for a multitude of chaotic systems across different domains. The paper provides mathematical details regarding each of the chaotic system they use in the benchmark and allows for a number of variations per chaotic system, namely, (a) sampling granularity, (b) initial conditions, (c) multivariate vs univariate state space, and (d) presence or absence of brownian noise in the dynamics. Using this data, the paper constructs a number of tasks like future predictability, model acceleration with importance sampling, transfer learning and symbolic regression.
strengths: The proposed benchmark is a nice idea of consolidating advances in temporal models on chaotic systems. Typically, most works consider Lorenz or Chua systems and the presence of this benchmark would systematically scale testing of models to a wide variety of systems. Not only is it a consolidated system of many chaotic systems but it also allows for different variations regarding each, eg. presence of brownian noise and granularity of sampling. This could lead to a systematic study of generalization and transfer learning across various variations and prove to be a pretty good test-bed for testing upcoming ML models.
weaknesses: While the work introduces an interesting dataset and a good set of tasks, I believe that in its current form it is not thorough enough for me to give it a high score. The benchmark provides 131 different dynamical systems along with variations but there is no analysis done on the performance of models across some fundamental different properties. For example, it would be nice to see how performance of different benchmarked models vary on increasing granularity of sampling, or on the presence of brownian noise, or across different fractal dimensions. Such an analysis would definitely make the work stronger and also provide more detailed avenues of where which type of models outperform the others.
correctness: The dataset is constructed in a sound way and the experimental design is appropriate. The authors consider hyperparameter tuning for each model and dataset and provide ample information regarding the different chaotic systems they use in their work. 
clarity: I found the paper to be on the harder side to read through and understand well. I think some sections outlining the basics of chaotic systems as well as a dedicated section for interpreting all the results of multiple tasks would make it easier for people to read and understand. It would also be better if the authors use more subsections within the sections to better outline different details of the works (example - Section 2 can be broken into submodules defining variations, mathematical properties, etc. I think an improved structure and more refined writing would make it a much stronger candidate.
relation to prior work: It is clearly discussed that the proposed work systematically consolidates different chaotic systems across various fields, thus allowing for better testing of models than considering just one or two chaotic systems like Lorenz.
additional feedback: The authors propose a great benchmark that would help the ML community as a whole. With more experimentation and clear writing, I think it will be a good benchmark for people to start using. I'd be happy to revise my score if some of my concerns are addressed.

**Update**: Increased the score based on the rebuttal provided.
review_title: Review of the Chaotic Systems Benchmark

Meta-review: All reviewers agreed that this dataset makes good contribution. It includes many known chaotic dynamical systems, benchmarking 16 forecasting models with many ML methods (not just DL). It show cases usefulness in transfer learning, symbolic regression and others. Authors has responded to reviewers’ comments appropriately (eg adding glossary terms).
Decision: Accept