HR-Extreme: A High-Resolution Dataset for Extreme Weather Forecasting

Authors: Nian Ran, Peng Xiao, Yue Wang, Wesley Shi, Jianxin Lin, Qi Meng, Richard Allmendinger
Primary area: datasets and benchmarks
Keywords: Weather Forecast Dataset, Extreme Weather, Deep Learning, Numerical Weather Prediction

Abstract:
The application of large deep learning models in weather forecasting has led to
significant advancements in the field, including higher-resolution forecasting and
extended prediction periods exemplified by models such as Pangu and Fuxi. Despite
these successes, previous research has largely been characterized by the neglect
of extreme weather events, and the availability of datasets specifically curated for
such events remains limited. Given the critical importance of accurately forecasting
extreme weather, this study introduces a comprehensive dataset that incorporates
high-resolution extreme weather cases derived from the High-Resolution Rapid
Refresh (HRRR) data, a 3-km real-time dataset provided by NOAA. We also
evaluate the current state-of-the-art deep learning models and Numerical Weather
Prediction (NWP) systems on HR-Extreme, and provide a improved baseline
deep learning model called HR-Heim which has superior performance on both
general loss and HR-Extreme compared to others. Our results reveal that the
errors of extreme weather cases are significantly larger than overall forecast error,
highlighting them as an crucial source of loss in weather prediction. These findings
underscore the necessity for future research to focus on improving the accuracy of
extreme weather forecasts to enhance their practical utility

Decision: accept (poster)
Reviewer scores: 5, 8, 6, 8

Reviews and discussion:
<thread>
creation date:
1730000265437

thread_type:
official_review

<thread_object>
creator: reviewer_tggo

<content>
summary:
This paper introduces a high-resolution dataset, called HR-Extreme, for numerical weather forecasting under extreme weather conditions, an area often overlooked in weather forecasting literature. The authors also present a baseline deep learning model alongside the dataset, called HR-Heim, which outperforms state-of-the-art weather models in extreme weather conditions.

strengths:
* The authors provide a high-resolution dataset for numerical weather forecasting under extreme weather conditions. The need for such dataset has been evident for some time, as highlighted by the low performance analysis of SOTA weather forecasting models.
* The construction of the dataset is well-motivated, and the authors provide a clear and thorough explanation of both the data collection and construction processes.
* Additionally, the authors provide a baseline deep learning model called HR-Heim, which is inspired by a SOTA numerical weather forecasting model, to specifically excel and outperform SOTA models under extreme weather conditions.

weaknesses:
* In Section 2.2, the authors discuss some datasets that share similarities (to a certain extent) with the proposed one. It would have been beneficial to illustrate the performance of the SOTA models and the proposed HR-Heim on some of these datasets. Specifically, the last dataset introduced by Liu et al., which also includes certain extreme weather conditions, could have been included in the experiments to support the need for HR-Extreme and support the performance of HR-Heim. Given that HR-Heim outperforms SOTA methods in HR-Extreme, we would expect to see similar results on other datasets with extreme weather conditions.
* As the authors also state as a limitation, this kind of study should include more than just a single-step prediction analysis. Given the increased difficulty of predicting extreme weather events in the long term, an accuracy comparison between HR-Heim and SOTA methods across varying time horizons could be valuable.

questions:
* Could you extend the dataset analysis to include predictions for more than a single hour ahead?
* As an extension to the previous question, could you analyze and compare the performance of state-of-the-art models and HR-Heim for both short-term and long-term forecasting?
* Could you further evaluate the performance of HR-Heim on a similar extreme weather dataset?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730573288171

thread_type:
official_review

<thread_object>
creator: reviewer_klu6

<content>
summary:
The authors present a new dataset of labelled extreme weather events over the continental US based on a high resolution (3km) numerical forecast product. They compare the events from a numerical weather prediction (NWP) model as well as two baseline ML based weather models and a newly proposed variant. The authors claim improved skill in their new variant compared to the other baselines.

strengths:
The constructed dataset could provide a useful evaluation of extreme events in new ML based weather prediction models which tend to be evaluated on larger scale statistics, potentially hiding biases in such events. It is based on an operational high-resolution dataset with a mixture of automated and manual labelling.

weaknesses:
There are a number of technical and fundamental weaknesses that undermine the above strengths.

Major issues:
1) Extreme events are, by definition, rare. While a database of such events derived from a high-resolution reanalysis product could provide a useful starting point for evaluation, the specific events are much less useful without a probabilistic understanding or measure of their likelihood at different lead times. i.e. It is probably more likely that any of the given events *wouldn't* have happened given a similar atmospheric state if the conditions were encountered again. Without a probabilistic understanding of the probability of an event (based on lots of comparable events that weren't extreme), this dataset has very little value as presented.
 2) This leads to my second concern regarding the evaluation setup. Given the dataset extracts the atmospheric state at t=0, -1 and -2 hours, I presume the evaluations happen from an atmospheric state of t=-2 hours and run forward? This is no longer forecasting, but nowcasting and quite a different task. Since the atmospheric state already has the extreme event and the model just needs to advect it correctly. I have no idea how the NWP model is compared in this setting since presumably the authors don't run this explicitly with the extracted state? Or perhaps they do? Also, since the authors use the same NWP as was used to create the HRRR dataset, why doesn't it perform as well as the other models? At what resolution are the models run, are they run globally, over the US, or only over the event region? Are the global Fuxi and Pangu models retrained for these regions?
 3) Related to this, to what extent is this dataset used to train the different models? Does HR-Helm get to see some or any extreme event data during training? Given the issues described in (1), how do you avoid overfitting?

One more minor issue is that I would like to see Figure 4 presented for the same day for all 4 models, with a separate figure for the other day for all 4 models in the appendix. Currently it's impossible to fairly compare the skill of the models (although it seems like the NWP already does better than HR-Helm across the two examples).

questions:
Please provide specific responses to each of the concerns and questions raised above in the weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730620438284

thread_type:
official_review

<thread_object>
creator: reviewer_jbrg

<content>
summary:
This work utilizes high-resolution HRRR data to create the HR-Extreme dataset, which encompasses a comprehensive set of 17 extreme weather types. The aim is to provide a more specialized dataset for evaluating the performance of weather forecasting models. To achieve this goal, the authors employed unsupervised clustering and manual filtering methods to develop a complete feature map of extreme events in a machine learning-ready format for the continental United States. The dataset was then used to assess the 1-hour forecasting capabilities of existing medium-range prediction models in comparison to the HR-Heim model proposed in this study.

strengths:
1. **Significance**: It fills the gap in benchmarking for extreme weather assessment in deep learning-based medium-range weather forecasting tasks.

2. The dataset is clearly introduced, and it will be fully open-sourced.

3. The authors present comprehensive experiments and baseline results.

weaknesses:
1. The description of the dataset generation process lacks clarity in some areas: a more detailed introduction of the clustering method is needed, including how records from different sources are handled and the hyperparameters of the algorithm.

2. There are also unclear aspects in the experimental description:  
   a) The baselines are models trained on globally coarse-resolution grids; Was there any further fine-tuning on this dataset? What preprocessing steps were taken?  
   b) What is the training strategy for HR-Heim? Does it use the same hyperparameter settings on the original dataset and HR-Extreme?

questions:
1. Was the result of the clustering algorithm verified by domain experts? Is there corresponding uncertainty detection and assessment?

2. What is the basis for the "types of events without specific ranges or those not related to obvious variations in feature map predictions"?

3. Can data from different sources be used separately?
</content>
</thread_object>

</thread>
<thread>
creation date:
1731172421014

thread_type:
official_review

<thread_object>
creator: reviewer_wjpu

<content>
summary:
Extreme weather forecasting is a crucial problem for the whole world. With the rise of deep learning-based weather forecasting models, the effectiveness of them on extreme weathers are not well analyzed.  This paper targets on providing a new benchmark for extreme weather forecasting. Authors employ the HRRR data and utilize the extreme events record in three sources to crop the extreme feature from the original HRRR dataset. Experiments are conducted with four baselines to show the performance.

strengths:
Pros: 

1. Extreme weather forecasting evaluation is an important research problem. 
2. The provided dataset introduces 17 extreme events, which are comprehensive. 
3. Authors also release the code for generating the data.

weaknesses:
Cons:

1. It is not clear how the HR-heim model is trained. 
2. Considering the dataset is a processed version of HRRR, it would be helpful to provide the geo-location of the extreme data to facilitate more diverse use from users.  
3. While the dataset is valuable, there is almost no analysis are present, especially compared to the ERA5 dataset, which is not insightful.

questions:
please refer to the weakness
</content>
</thread_object>

</thread>
<thread>
creation date:
1732261922765

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Comments and Revision Summary

comment:
We thank all reviewers for your insightful reviews and helpful suggestions, and thank all reviewers for their recognition of the importance, comprehensiveness and contributions of our work.  The revised version of our paper is submitted and we give a summary of our revision of the paper below,:
- According to the reviewer's suggestion, we have added additional statistical analysis of physical variables in normal and extreme events showing the ability of our dataset. We have also added additional experiments of NWP and HR-Heim at more lead times, the results show HR-Heim consistently outperform NWP model at different lead times.
- Many reviewers have concerns about the model training, we have clarified that all deep learning models (Pangu, Fuxi and HR-Heim) are trained with same level of parameters, same hyperparameters. And none of them are fine-tuned on HR-Extreme, enabling fair comparison. This clarification is also added in the paper.
- Some reviewers have questions about more details of extreme events of our dataset, we have clarified that all kinds of details of extreme events can be found in the index file generated along with the dataset. This is clarification is also added to the paper.
- We have improved our writing to present more clearly our primary goals, contributions and significance.
- At last, I want to thank all reviewers again for your constructive feedback that makes this paper better.
</content>
</thread_object>

</thread>


Metareview:
metareview:
The authors introduce a high-resolution weather dataset for extreme weather based on the HRRR public data from NOAA (numerical weather prediction). They do this by processing HRRR through unsupervised clustering and filtering and creating a comprehensive benchmark of many extreme weather events over the US. The data is in ML training ready format and open-sourced. They further introduce a new DL baseline model that outperforms other SOTA DL models on their dataset.

Strengths: Timely dataset for extreme weather forecasting was acknowledged as a strong contribution by all reviewers, dataset is comprehensive, results show the shortcomings for current DL models

Weaknesses: The HR-Heim model contribution as a baseline model is not comprehensive, technical clarity can be improved

The paper could be improved by primarily more discussion/text based on the reviewer's comments and my own reading. For example:

* analysis on ERA5 - The author's rebuttal is satisfactory in that ERA5 is a coarse (but global) dataset. It could be beneficial to show this shortcoming of ERA5 to motivate the new dataset - missing extreme events, smooth/coarse prediction of events, etc through similar visual images.
* NWP comparisons - discussion on why the HR-Heim model is better than NWP given that NWP is used to create the dataset in the first place - this could be a discussion on details of the HRRR dataset (analysis vs reanalysis states), how the NWP model works in the forecasting mode, what additional information the HR-Heim model sees (as well as other DL models) and why you expect this setting is close to the actual forecasting setting for NWP prediction centers. 
* HR-Heim vs other models - discussion on why HR-Heim is a better architecture than the other models, given that they are all trained on the same dataset - what parts of HR-Heim contribute to high resolution prediction and what parts could be switched with other backbones. Typically, these are answered systematically with ablations but since the authors reduce this contribution in favor of the dataset primarily, a discussion will be beneficial.
* Model training discussion - while the author's rebuttal clarifies partially how the models were trained, it is beneficial to also state training times, training resources. This also related to the previous point where the compute/memory costs comparisons across models will inform the reader as to the actual cost of getting the increased accuracy on extreme events. 
* Minor: Fig resolution for Fig 1 can be improved and in general font sizes for figures can be bigger to allow for easy readability.

The above improvements (primarily text changes) will help cater to the broader ML for science ICLR audience.