Venue: NeurIPS 2022 Track Datasets and Benchmarks

Reviews:

final_score: rating: 9: Top 15% of accepted papers, strong accept
review_content: summary and contributions: This dataset paper aim to provide a post-processing prediction system data that can enhances the wether prediction. The dataset covers 20 years and has 10 ensemble members. 
strengths: - The data is composed of 20 years wether data extracted for 10 ensembles that is obtained for a weather simulator. 
- The authors provide new ways of measuring the prediction correction task.
- The dataset includes important wether prediction parameters like height above sea level, pressure values, etc.
-
weaknesses: - Is the numerical weather prediction approach the most famous for weather prediction?
- Line 91, why did you use these measurements? Are there other parameters that you could have used?
- Line 173, similarly, why did you use these parameters?
- Why did you stop the data range at 2017 although we are in 2022?
correctness: Yes, the data is constructed by extracting weather data from a global weather simulator. 
clarity: Yes.
relation to prior work: Yes, although it would be nice to expand on the advantage of ensemble processing methods over deterministic data-driven forecasting.
additional feedback: None.
review_title: Official review for ENS-10

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: - The authors present a post-processing ensemble prediction system to improve weather forecasting.
- The task is to correct the prediction by post-processing the ensemble members.
- The metrics used in the paper are: the Continuous Ranked Probability Score (CRPS) and Extreme Event Weighted Continuous Ranked Probability Score (EECRPS). The authors proposed the last one, and its work is to assess the correction improvement in skill for extreme event prediction.
- In case of baseline methods: Raw, Ensemble Model Output Statistics (EMOS), Multi-layer perception(MLP), and U-Net.
strengths: - Explains the challenge when predicting weather and the necessity to use ensemble processing methods instead of deterministic data-driven forecasting.
- The authors use reforecasts that run routinely at ECMWF to simulate weather over the past 20 years; this way, the resulting dataset can serve as a measure to skill a forecast system.
weaknesses: - Weather prediction is challenging because of its multiple variables, which makes it difficult to evaluate its usefulness or advantages. 
correctness: The paper is coherent when explaining data collection and metrics for its evaluation.
clarity: The paper is well written, very technical in terms of weather but easy to follow.
relation to prior work: Using ensemble processing methods instead of deterministic data-driven forecasting.
additional feedback: Do the authors consider other models for baseline models?
review_title: Official review

final_score: rating: 6: Marginally above acceptance threshold
review_content: summary and contributions: I think the dataset serves an important purpose. Part of the reason why "physics-based" models have been more popular in weather prediction is the lack of large and principled datasets to train data-driven models. This dataset alleviates this challenge to some extent and is an important contribution. I am broadly in support of accepting the paper.  That being said, I have some questions about the dataset. My rating is conditional on the questions being answered. 
strengths: Overall, a strong dataset paper. The lack of large scale datasets is a major gap for the community working on data-driven weather forecasting, so this is an important paper.
weaknesses: 1. From the perspective of releasing open-source datasets, wouldn't it be better to include all 10 prognostic variables per grid point? The reduction of features can be done by the users of the dataset.
2. I am a bit confused about using ERA5 as ground truth data for the exact weather at prediction time. What exactly are the labels in the data?
3. The authors write in the supplementary material that there are no sources of errors or noise in the dataset. Does the original data not suffer from noisy measurements?  Also, the spatial discretization must include some noise, right? The authors should clarify this. 
4. Other than the e Prediction Correction task, what other tasks can the dataset be used for?
5. What guided the choice of the baselines? 
correctness: The claims look correct.
clarity: The paper is well-written overall. 
relation to prior work: Yes.
additional feedback: NA
review_title: Important dataset for weather prediction

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: This article introduces and makes available an open-source weather prediction dataset spanning 20 years. The dataset can be readily utilized for ensemble prediction correction by other Machine Learning models for weather forecasting tasks. This work provides support for many weather forecasting tasks being carried out in other domains. The importance of ensemble weather correction has been demonstrated by a predictive task using the introduced dataset.
strengths: * The authors have provided a useful open source dataset (ENS-10) for ensemble weather prediction correction for 20 years along with a well-maintained website.
* A new error metric (EECRPS) has been introduced to evaluate predictions under extreme weather events.
* The dataset has been used in a predictive task of prediction correction using several Machine Learning models.
weaknesses: * As the authors discuss on page 7 of the manuscript, there exist many models in the literature for ensemble post-processing of weather forecasts. Nevertheless, the authors have attempted only a few of them (particularly only 2 Machine Learning methods in their predictive task), which seems a limited benchmarking attempt in my opinion.
* The benchmarking study focuses only on ensemble correction using different techniques. However, to make a fair comparison, other direct data-driven weather forecasting methods could also have been tried out (as the authors discuss on page 2), to establish the importance of performing ensemble correction and thus the usage of this dataset in the first place.
* The introduced dataset is 20 years long but only until 2017. How is it extended beyond 2017 for more recent weather forecasting work?
correctness: No incorrectness issues were found.
clarity: Well-written article.
relation to prior work: Prior work is sufficiently discussed.
additional feedback: The authors mention that 'ENS-10 consists of the atmospheric fields that appear most relevant in this context'. However, it is not clear or mentioned, on what basis the relevance of these atmospheric fields is evaluated.
review_title: A dataset of ensemble weather forecast correction by using downstream Machine Learning methods.

final_score: rating: 5: Marginally below acceptance threshold
review_content: summary and contributions: The paper discusses a dataset and metrics for mesoscale weather forecasting bias correction. The topic is undoubtedly very important and has broader societal implications in general.
strengths: -- The paper is overall well written
-- The topic is very important and is not widely explored by the ML community.
weaknesses: There is nothing wrong with the paper but nothing stands out either. The data set is quite standard and there are multiple other sources to get a similar dataset. The metrics and methods are also conventional. It is not clear/not discussed what the ML community can do with this data beyond forecasting and bias correction. 
correctness: The experiments are appropriate and the dataset is discussed properly.
clarity: The paper is clearly written.
relation to prior work: No proper discussion of competing data sets.
additional feedback: Do authors plan to regularly update these data? If yes, how will the maintenance be organized?
review_title: A dataset for validation of weather prediction 

Meta-review: metareview: This is a high quality, high impact dataset for weather forecasting, with a novel task and metrics to measure the quality of prediction by forecast ensembles. The reviewers are in agreement that this paper should be accepted. Congratulations.
confidence: 4: The area chair is confident but not absolutely certain
Decision: Accept