Venue: NeurIPS 2021 Track Datasets_and_Benchmarks Round1

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: summary and contributions: The paper provides a  end-to-end pipeline for time-series analysis fo the HiRID dataset an ICU dataset.
This pipeline involves the preprocessing of data, proposing possible tasks where this dataset can be used, finally evaluating models.
Tasks proposed were a variety of classification and regression tasks.
The pipeline allows hand selection of feature. 
The paper used the proposed pipeline  to compare LightGBM, Logistic Regression, LSTM, GRU,TCN and Transformer models.
strengths: One of the major problems with time series datasets is that unlike image datasets they are not standardized.  The results reported in the paper highly depend on the preprocessing step and generally machine learning researchers do not want to spend time processing data;  which leads to focus on images and language where there are many standard datasets for specific tasks eventually  ​leading to less research in this area.  


This  paper  puts  the data in  machine  learning-friendly  format allowing them to focus on the research  part  making it extremely beneficial and will eventually lead to progress in research in this area.

The time series tasks proposed reflect real world challenges; also there is a variety of regression and classification tasks. 
weaknesses: There is nothing particularly novel proposed by this paper. The dataset already exists, metrics are commonly for the mentioned tasks. Model comparison is pretty straight forward.


correctness: Yes
clarity: Yes paper is will written.
relation to prior work: The difference between HiRID-ICU-Benchmark and MIMIC-III is compared.
additional feedback: Additional experiments comparing forecasting models might be interesting for example:
Forecasting accuracy at  different horizons.
Forecasting when training data length << test data length.
review_title: Timeseries ICU Dataset

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: summary and contributions: Here the authors present a number of contributions related to the HiRID-ICU database, including:

- a standardized pipeline for pre-processing, with two stages: one with just simple pre-processing (e.g. artifact removal) and one with pre-selected features for machine learning
- a series of tasks that can be used for comparison of different approaches to analyzing the ICU data (such as organ failure, mortality, etc.)
- benchmark results on the listed tasks above   
strengths: **Significance:** this contribution is significant as it addresses an important need in the ICU space, namely a way to readily compare performance of different approaches with a standardized set of tasks and pre-processing. The temporal resolution may also enable improved performance over other datasets.

**Relevance:** the contribution is highly relevant to the broader community as managing ICU patients will only continue to be an important clinical need 

**Accessibility and Accountability:** the documentation is mostly clear. 

**Ethical/Social Implications:** managing ICU is an important social problem.
weaknesses: There are important details about the demographics of the participants and some of the methods that may limit the contribution. A major weakness of the paper is lack of clarity and details in the methods. In particular:

- it would be very helpful if the information about the different signals used could be included in the manuscript. I had to search for it in the documentation.  At least linking to here would be helpful: https://hirid.intensivecare.ai/Data-details-1ff9c433b9894904b1dbd7652be4b11c
- the authors should include more information about the pre-processing stage, and how they were able to verify that the pre-processing results were appropriate. This is especially important considering the varied nature of the signals. For example, I was wondering whether the heart rate data were extracted from e.g. continuous ECG measurements by the researchers in generation of the HiRID dataset. If this is beyond the scope of present work please include a line referring to the original HiRID paper.
- the authors should explain what they mean by "density of the measurement" (the rest of the features seem self-explanatory)
- the authors should discuss the scaling factors used for the continuous variables. How was the scaling conducted to insure comparison across participants without losing important information such as whether the patient is in tachycardia?
- there does not appear to be any summary of the demographics of the patients included in the dataset. Given that this dataset is intended to be used for machine learning, understanding how is included in the dataset and who is not is essential.

correctness: Claims made in the submission appear mostly correct. I focused on evaluation of the details of the ICU datasets and not as much on the model testing as that is not my background. The tasks seem diverse and appropriate.
clarity: What is written in the paper is mostly clear, other than what was missing as described in weaknesses.
relation to prior work: Yes, the authors discuss previous datasets and attempts for tasks and benchmarks for ICU data in detail.
additional feedback: The footnotes 4 and 5 appear to be on the incorrect page.
review_title: Solid contribution with need for more information before acceptance

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: This work provides a modular and standard pipeline to pre-process the public ICU time-series dataset (HiRID), creates six clinically relevant prediction benchmark tasks, and evaluates regular machine learning methods (LightGBM, Logistic Regression) and deep learning methods (LSTM, GRU, TCN, and Transformer).
strengths: The benchmark tasks in this work have several desired features:
- the tasks are highly clinical oriented and diverse (two regression tasks; three binary classification tasks and one multi-class task)

- the standard pre-processing pipeline is provided, which is critical for reproducibility

- open-sourced baselines
weaknesses: Major
1. There is a high requirement to access the benchmark dataset (e.g., need to join a course and take a few hours to train). Could you directly make the pre-processed data publicly available by Google Drive or Dropbox? If it is not allowed by the data holder, could you provide some simulated data to allow users to run and test your code?
2. It would be better to provide a more user-friendly tutorial on the GitHub repo. with a small subset of the (simulated) data, e.g., Jupyter-notebook.  
3. Please provide the download link to baseline results (e.g., trained models, predictions).
4. Please provide specific explanations or direct links to the background of the dataset, which can release the burden for the newcomers that want to enter this domain. For example, what is the meaning of each variable? What are the clinical backgrounds/values of the benchmark tasks? How to understand the evaluation metrics?... Although these answers can be obtained by google or the dataset paper, it will be more convenient for future users if you provide them on the GitHub readme.

Minor 
1. Please put Table 1 on Page 3 or on the top of Sec. 3.2. 
Furthermore, what are the relationships of the six tasks? Please provide more motivations.
2. line 122: We provide more details about it in the README section of the Software Repository...
Do you mean the GitHub repo? https://github.com/ratschlab/HIRID-ICU-Benchmark/blob/master/README.md?  I do not find the preprocessing "details" in the README except for the command. Please add a footnote to specify the link of details.
3. Figure 1 Title: The explanation should follow the pipeline (i.e., Red, Blue, Green, Grey)
4. Add statistical significance analysis in Table 3 and Table 4.
5. Provided some data samples in the manuscript will be better. 
correctness: Yes.
clarity: The paper is well written and easy to follow.
relation to prior work: Yes. It was presented in Sec. 2.
additional feedback: Please see the detailed comments in the weaknesses part.
review_title: This work sets up six benchmark tasks based on existing ICU time series data and it needs improvements on details to be more user-friendly.

Meta-review: The reviewers all liked the paper.  The dataset will provide more information for the medical and machine learning community. The authors' response clarified some important points.  The authors are strongly invited to integrate these points in the final version. 
Decision: Accept