HR-Extreme: A High-Resolution Dataset for Extreme Weather Forecasting
Authors: Nian Ran
Peng Xiao
Yue Wang
Wesley Shi
Jianxin Lin
Qi Meng
Richard Allmendinger
Venue: ICLR 2025 Poster (2025)
Keywords: Weather Forecast Dataset
Extreme Weather
Deep Learning
Numerical Weather Prediction
Primary area: datasets and benchmarks
TL;DR: We present HR-Extreme, a high-resolution dataset for evaluating extreme weather forecasting accuracy, enhancing the practical utility of SOTA models.

Abstract: The application of large deep learning models in weather forecasting has led to
significant advancements in the field, including higher-resolution forecasting and
extended prediction periods exemplified by models such as Pangu and Fuxi. Despite
these successes, previous research has largely been characterized by the neglect
of extreme weather events, and the availability of datasets specifically curated for
such events remains limited. Given the critical importance of accurately forecasting
extreme weather, this study introduces a comprehensive dataset that incorporates
high-resolution extreme weather cases derived from the High-Resolution Rapid
Refresh (HRRR) data, a 3-km real-time dataset provided by NOAA. We also
evaluate the current state-of-the-art deep learning models and Numerical Weather
Prediction (NWP) systems on HR-Extreme, and provide a improved baseline
deep learning model called HR-Heim which has superior performance on both
general loss and HR-Extreme compared to others. Our results reveal that the
errors of extreme weather cases are significantly larger than overall forecast error,
highlighting them as an crucial source of loss in weather prediction. These findings
underscore the necessity for future research to focus on improving the accuracy of
extreme weather forecasts to enhance their practical utility

Published as a conference paper at ICLR 2025
HR-EXTREME: A HIGH-RESOLUTION DATASET FOR EXTREME WEATHER FORECASTING
Nian Ran', Peng Xiao², Yue Wang'; Wenlei , Jiaxin Lin', Qi QiMeng', Richard Allmendinger
Zhongguancun Academy
2 Human University
3 Microsoft Research
*Chinese Academy of Sciences University of Manchester
ABSTRACT
The application of large deep learning models in weather forecasting has led sigmificant advancements the field, including higher-resolution forecasting and extended prediction periods exemplified by models such Pangu and Fuxi. De- spite these successes, previous research has largely been characterized by the neglect of extreme weather events, and the availability of datasets specifically curated for such events remains limited. Given the critical importance of accu- rately forecasting extreme weather, this study introduces comprehensive dataset that incorporates high-resolution extreme weather cases derived from the High- Resolution Rapid Refresh (HRRR) data, a 3-km real-time dataset provided by NOAA. We also evaluate the current state-of-the-art deep learning models and Numerical Weather Prediction (NWP) systems on HR-Exxreme, and provide a improved baseline deep learning model called HR-Heim which has superior per- formance on both general loss and HR-Extreme compared to others. Our results reveal that the eтгors of extreme weather cases are significantly larger than overall forecast error, highlighting them as an crucial source of loss in weather prediction. These findings underscore the necessity for future research to focus on improving the accuracy of extreme weather forecasts to enhance their practical utility.
INTRODUCTION
Weather forecasting is crucial scientific endeavor that influences various aspects of human life, from daily activities to disaster management and agricultural planning. The ability predict weather conditions accurately can mitigate the impact of natural disasters, optimize resource management, and improve public safety. Traditionally, Numerical Weather Prediction (NWP) models (European) Centre for Medium-Ramme cornerstone of weather forecasting by explicitly solving the large-scale Partial Differential Equations (PDE). These models rely on mathematical formulations of atmospheric dynamics and physical processes to predict future states of the atmosphere based on current observations. While NWP models have achieved significant success, they are extremely computationally intensive, requiring substantial supercomputer power due to vast amount of data propressing and complex PDEs to simulate atmospheric conditions. Additionally, they have limitations in capturing the intricacies of certain weather phenomena.
Weather Forecasts (ECMWF) 2024 Bauer et al. 2015) have been the
In recent years, the advent of deep learning has revolutionized weather forecasting by offering alter- native approaches that can potentially overcome some of the limitations of traditional NWP models by implicitly solving large-scale PDEs. Deep learning models such Pangu- Weather (Bi et al.) 2023), Fuxi (Chen et al. 2023b), and FourCastNet (Pathak et al.. 2022) have shown promising re- sults in generating high-resolution weather forecasts. FourCastNet is Fourier-based neural network model, which leverages Adaptive Fourier Neural Operators and vision transformer (ViT) to produce high-resolution, global weather forecasts. Pangu-Weate Weather leverages large-scale neural networks to
"Corresponding author.yuewnn@@@rrggeeed... ficenn.all manchester.cc.uk)

---

Published as a conference paper at ICLR 2025
provide accurate and timely predictions. Fuxi integrates multi-scale spatiotemporal features to hance forecast accuracy by effectively capturing the complex interactions within the atmosphere.
Despite these advances, one of the most critical and challenging aspects of weather forecasting is the accurate prediction of extreme weather events. Extreme weather, such as hurricanes, tornadoes, and severe storms, poses significant risks to life and property. Accurate forecasting of such events is essential for effective disaster preparedness and response. However, existing models and datasets often fall short in this regard. Current datasets lack sufficient types of extreme weather cases (Racah) et al. 2017), or irrelevant data forms (Liu et al., 2023) Yu et al. 2023b; Zhu et al. 2016), or not very specific (Liu al. [021a), leading to models that are less reliable about extreme weather. On the other hand, current state-of-the-an (SOTA) deep learning models in weather forecasting have either been tested on only a limited range of extreme weather events or even lack testing this area (Bi et al., 2023; Chen et al., 2023b; Patthak et al. 2022). This gap underscores the urgent need for specialized datasets and models that focus on extreme weather prediction.
In this work, we address this gap by proposing a comprehensive dataset that includes high-resolution extreme weather cases derived from the NiihhRResolutin Rapid Refresh (HRRR) (National Oceanic and Atmospheric Administration (NOAA) 2024) data, a 3-km real-time dataset provided by the Na- tional Oceanic and Atmospherie Administration (NOAA), Our dataset, called HR-Extreme, includes high-resolution feature maps with dimensions (69, 320, 320) where 69 represents the number of physical variables as channels and (320, 320) denotes the size of each feature map, with each pixel corresponding to 3km by 3km area aims to improve the accuracy of weather forecasting models in predicting extreme weather events. This dataset is built on U.S. area because of its complete and rich storm data. Our key contributions are:
We propose weather forecasting dataset called HR-Extreme specifically for evaluating extreme weather cases, based on HRRR data with a 3-km resolution, which provides a sig- nificantly higher resolution than previous most-used dataset ERA5, which is 31-km resolu- tion, and is suitable for evaluating and improving SOTA deep learning and physics- models.
Our dataset includes a comprchensive set of 17 extreme weather types, such as strong winds, heavy rains, hail, tornadoes, and extreme temperatures, while previous work typi- cally evaluates only one or a few types of extreme weather.
We provide an extensive evaluation of SOTA models on our dataset, including visualiza- tion and detailed analysis. Additionally, we provided an improved version of deep learning model called HR- based on HRRR data as a baseline, inspired by FuXi (Chen et al., 2023b) and MagViTv2 (Yu et al., 2023a), which outperforms SOTA methods on both nor- mal weather forecasting and extreme weather evaluation in terms of one-hour prediction.
RELATED WORK
Historically, NWP models have been the gold standard, utilizing explicit solutions of PDEs to sim- ulate atmospheric states. Although NWP models, such those from the ECMWF, have achieved considerable success, they are computationally intensive and often struggle with the intricacies of complex weather phenomena.
2.1 ADVANCEMENT IN MODELS
Recently, weather forecasting has seen transformative advances with the integration of machine learning. Keisler (2022) utilizes graph neural networks to achieve successful short-term and medium-range forecasting on a 1.0" latitude/longitude grid. GraphCast (Lam et al., 2022) demon- strates that ML-baaed methods can compete with traditional weather forecasting techniques on ERAS data, while ClimaX (Nguyen et al., 2023a) establishes a foundation model for weather pre- diction by pretraining and fine-tuning on several datasets using transformer-based architecture Vaswani et al. 2017). FengWu (Chen et al. 2023a) also employs a transformer architecture, ad- dressing global medium-range forecasting as multi-modal, multi-task problem and outperforming GraphCast. Subsequently, Feng WengWuGGHR (Han et al. 2024) became the first data-driven method successfully forecast 0.09* horizontal resolution. Stormer (Nguyen et al. 2023ch further
2

---

Published as a conference paper at ICLR 2025
analyzes the key components that contribute the success of transformers in this task. Addition- ally, GenCast and SEEDS (Price et al., 2023) emulate ensemble weather forecasts using diffusion models, enabling the generation of large ensembles that preserve the statistical properties and pre- dictive skills of physics-based ensembles. In contrast, the NeuralGCM (Kochkov et al.) 2024) model integrates differentiable dynamical core for atmospheric dynamics with machine learning compo- nents, achieving significant computational efficiency and improved accuracy for both short-term and long-term predictions compared traditional GCMs. Meanwhile, ClimODE (Verma et al. 2024) is a physics-informed neural ODE model that integrates quantification, delivering state-of-the of-the-art performance in both global and regional forecasting.
dynamics and uncertainty
Models like Pangu-Weather Weather (Bi et al. 2023) and FuXi (Chen et al. 2023b) represent significant strides in this domain. Pangu- -eather leverages three-dimensional neural networks with Earth- specific priors to effectively manage complex weather patterns, achieving higher accuracy and ex- tended prediction periods. FuXi employs a cascaded machine learning system to provide 15-day global forecasts, demonstrating superior performance compared to traditional ECMWF models, par- ticularly in extending the lead times for key weather variables.
Nevertheless, critical challenge persisting in accurately weather forecasting is extreme weather events, which include hurricanes, toradoess, severe storms and etc. These events pose sigmificant risks to life and property, which makes precise forecasting essential for effective disaster preparation and response. However, recent advanced models, such as GraphCast, GenCast, FourCastNet (Pathak jet al., 2022), Pangu, Fuxi, including physics-basee based models only involve few types of extreme weather or only state the abilities in extreme weather prediction. Although NowcastNet (Zhang et al.) 2023) is designed for extreme weather, it only predicts extreme precipitation. The reasons include insufficient attention to extreme weather events and the lack of comprehensive and systemactic dataset that covers various types of extreme weather over long period.
2.2 RELATED DATASET
Several datasets have been developed to address this challenge; however, they often fall short in com- prehensiveness and resolution, or relevance. The ExtremeWeather dataset (Racah et al. 2017) is notable effort, utilizing 16 variables, such as surface temperature and pressure, detect and localize extreme weather events through object detection techniques and 3D convolutional neural networks (CNNs). However, this dataset is limited three specific weather phenomena, such as tropical de- pression and tropical cyclones, and does not encompass a broad range of extreme weather types. Similarly, the EWELD dataset (Liu al.. 2023) focuses on electricity consumption and weather conditions, providing a temporal dataset that supports weather analysis over 15-minute intervals. However, EWELD provides time-series numerical data (signals) which is completely different from feature maps in HRRR data, and therefore it is not applicable in weather forecasting problem.
ClimSim YYu et al. 2023b) and ClimateNet (Prabhat et al., 202T) represent further advances in dataset development. ClimSim aims to bridge the gap between physics- based and machine learn- ing models by providing multi-scale climate simulations with over 5.7 billion pairs of multivariate input and output vectors. Despite its scale, ClimSim is more geared towards hybrid MI.-physics research rather than specific extreme weather forecasting and multi mhaln-channell feature map prediction. ClimateNet, on the other hand, provides an expert-labeled dataset designed for high-precision anal- yses of extreme weather events, such as tropical cyclones and atmospheric rivers. The most relevant work is severe convective weather dataset proposed by Liu et al. (2021)), which includes ground observation data, soundings, and multi-channe' satellite data for various types of extreme weather in China. Although extensive, their feature maps are of lower resolution, contain too much redundant data, involve fewer types of extreme weather, provide less accurate location information of extreme events, and different countries, compared to our dataset.
In conclusion, existing datasets often lack the necessary resolution and variety of extreme weather examples, which are essential for evaluating SOTA robust models. To address this gap, this study introduces novel dataset derived from the High-Resolutton Rapid Refresh (HRRR) data, a 3-km real-time dataset provided by NOAA. The HRRR dataset is significantly more detailed than previous datasets, offering high-resolution weather forecasting and a rich set of physical channels. This dataset is designed to enhance the predictive capability of weather forecasting with SOTA large deep learning models, particularly for accurately forecasting extreme weather events.
3

---

Published as a conference paper at ICLR 2025
3 DATASET CONSTRUCTION
3.1 DATASET OUTLINE
In the context of weather forecasting, model predicts multi-channel feature map, where each channel represents physical variable, such as temperature at two meters above ground, vertical wind speed, humidity specific atmospheric pressure level. The dimensions of these feature maps depend on the resolution of the original data and the detection range. For example, each pixel in the feature maps in Pangu and Fuxi (Bi al. 2023 Chen et al. 2023b) is 0.25 x 0.25%, constructing a grid of 721 X 1440 latitude-longitude points representing the global area. Although previous work has focused mainly on ECMWF ERAS reanalysis data (Hersbach al. 2020) (0.25% x 0.25 per pixel), our dataset is derived from HRRR (National Oceanic and Atmospheric Adminis tration (NOAA) 2024) data provided by NOAA. This dataset features a 3-km resolution, is updated hourly, and offers cloud-resolving. convection-allowing atmospheric data with approximately ten times higher resolution than ERAS.
Our dataset comprises 17 types of extreme weather events that occurred in 2020, including extreme temperatures, hail, tornadoes, heavy rain, marine winds and etc. For each event, we specifically crop the area and utilize series of feature maps with dimensions of (69, 320, 320) to cover both spatial and temporal axes. An event is thus expressed as (n, f, c, w, h). where n is the number of 320 x 320 feature maps covering the event area, f is the timestamp, is the number of physical variables, and w and h denote width and height, respectively. In our case, we set = 3 and that for particular hour when extreme event occurs, group the feature maps with this hour and two hours before. This approach accommodates models that take two previous timestamps of feature maps as input and predict the current timestamp. However, in our interface, users can adjust any number of hours before and after the timestamp for different usages.
69, which means
The file naming convention used "date "date-type www.tyyppeyyyppyt htype2t...tyeNN LyppeemmmX.........llipppa minX .iimmmnn Y mmaxX max Y",, for example "2020070100 Toil++orradoo762... 766.466666666666 821 821__511.pp2 where "date" is in the format "yyyymmddhh" and "ttpe" denotes the event types included in this area, connected by "+" "minX, minY, maxX, max are the indices of the area extracted from the original HRRR product, such that applying m:max, minX:maxX] on HRRR data will directly obtain this area. This format allows users easily evaluate models in terms of different spans and event types, well visual- izations. In the following sections, Section 3.2 will elaborate on how each type of extreme weather is collected and processed, and Section 3.3 will describe the creation of feature maps covering the area and the interface to generate additional years data beyond current storage.
Variable msl
Definition
Unit
Range
Mean Sea Level Pressure
Pa K
2t
Temperature 2 above ground
U U-component Wind Speed 10 m
10u
m/s
above ground
V-component Wind Speed 10 m
10v
m/s
above ground
hgtn
Geopotential Height
gpm
U-component Wind Speed
m/s
At 50, 100, 150, 200. 250, 300, 400, 500, 600, 700, 850, 925, 1000 millibars, 13 levels in total
v
V-component Wind Speed
m/s
Temperature Specific Humidity
K kg/kg
q
Table I: Summary of the 69 physical variables in the dataset
3.2
DATA COLLECTION
Our dataset focuses extreme weather events in the U.S. based on NOAA HRRR data. The original feature maps measure 1799 pixels in width and 1059 pixels in height, covering latitudes from 2 to 52.6 and longitudes from 225.9 to 299.1%, collected by the Herbie Python library (Blaylock)
4

---

Published as a conference paper at ICLR 2025
2024
The sources of extreme weather cases are divided into three categories based extreme
weather types.
NOAA Storm Events Database The first source is the NOAA Storm Events Database (National Centers for Environmental Information (NCHI) 2024). which includes a wide range of storms and significant weather phenomena from 1950 to 2024, recorded by NOAA's National Weather Service (NWS). While most events from this database include information on location and spatial range, some lack these critical details. Additionally, certain event types, such avalanches and high surf, or events requiring long-term predictions like droughts, do not significantly impact the physical variables predicted by the models. To ensure greater accuracy, we have filtered out these events. Common types of extreme events not covered here will be supplemented by the other two sources. Otherwise the extensive and detailed records this database facilitate the identification of areas and time spans through provided ranges and timestamps directly.
NOAA Storm Prediction Center The NOAA Storm Prediction Center (National Oceanic and At- mospheric Administration (NOAA), Storm Prediction Center (SPC) 2024) records daily reports of hail, tornadoes, and wind from various sources, including local NWS offices, public reports, agen- cies, and weather stations. However, the lack of event classification means that no specific ranges or time spans are provided, and multiple events are often mixed together. This complicates the creation of clear and accurate areas necessary for training and evaluating deep learning models accurately and effectively. To address this, we employ an unsupervised clustering algorithm to classify the events. After extensive case studies, we determined that DBSCAN (Ester al., (996) is the most suitable for this task, as illustrated in Figure | For each timestamp. user reports are treated as 2D points based on normalized latitude and longitude on the x and y axes. DBSCAN identifies clusters based on point density, forming cluster if there are enough points in close proximity, We carefully tune the hyperparameters of DBSCAN to create more intuitive clusters and to filter out noisy points more accurately as shown in Figure | minor events or errors that are not significant enough to warrant creating a separate cropped area for evaluation.
Noisy points are filtered out because they likely represent
Class
Class
Class
Class 4
Cass
Nose
Class
class
Kmeans
DBSCAN
Figure 1: The performance of clustering by KMeans (right) and DBSCAN (left). It can be seen that the results of DBSCAN are more accurate and reliable, and the approach also identifies noisy points effectively.
Manually Filtered Extreme Temperatures As extreme temperature events not covered in the Storm Events Database and are absent from the Storm Prediction Center, we manually created cases for unusual temperatures over large areas. Climateleeam (Nguyen al. 2023b) uses the 5th and
5

---

Published as a conference paper at ICLR 2025
95th percentiles of the 7-day localized mean surface temperature for each pixel threshold to filter extreme temperature cases, similar to our approach. We also calculate the mean temperature for each pixel and use the 5th and 95th percentiles and heat at C2 and 37 respectively, because exposure to such conditions can quickly cause discomfort injury, and these temperatures timestamp, we count the number of pixels in the feature map of the variable "temperature 2 meters above ground" that exceed these thresholds. If the count is below 100, we discard the timestamp since events in such timestamps are likely to be localized phenomena such as volcanic eruptions or fires rather than the widespread heat or cold events we aim to predict. Subsequently, we apply DBSCAN each timestamp to further classify the events and eliminate noisy points.
reference, and select the thresholds for extreme cold
uncommon over large areas in the U.S. For each
3.3 DATA CROPPING AND INTERFACE
69 physical thennei
Grop
1059
320
Masks
320
Covered by squares
1799
Event area cropped and covered by series of squares considering the fixed size fed to the neural network, masks are provided to ensure only event area are calculated.
Figure 2:
Alongside the dataset generated by code, provide index file that contains details loca- tion, range, type of extreme weather, and time span by integrating information from above three data source. Users easily convert this information from the index file longitude and latitude coordi- nates with our open-souree code. Based on the information from the index file, the original feature maps covering the US are cropped into series of sub-images that specifically focus on extreme events. This approach allows models to train and evaluate on extreme events more precisely and exclusively. The entire pipeline illustrated in Figure 2 To account for uncertainties records and user reports, we slightly increased the range of each event. Each target area is covered by series of squares with dimensions (320, 320) to ensure compatibility with neural networks, and masks are provided to ensure that only the target area is considered in loss calculations. When a cover image reaches the edge of the sampling image, is moved completely inside the edges instead adding constant value paddings, with masks adjusted accordingly. This ensures optimal performance of deep learning models, which are trained without constant padding for this specific task. We provide data for extreme events that occurred in 2020, totaling 5 TiB of memory. Data after July 2020 is designated as the evaluation set. Due to substantial memory requirements and varying user needs, also offer code interface for generating the dataset, allowing users create datasets for any selected year, with any number of hours before and after for each piece of data, ensuring scalability and compatibility for different prediction strategies.
3.4 DATA AVAILABILITY
The HRRR data is in U.S. Govermment Work license, which means that the data is in the pub- lie domain and can be freely used, distributed, and modified without any restrictions. Our dataset hottps:/huuggina
co http//aaassss..... /Ninnainnnnnnnnnn anRanl Maeaannn/RR-BBB (RReextreee and code (https:)
//github. com/ Hwtp//uskyNiinn/in /RR Exxtreme will be available upon the accpetance this pa- per.

---

Published as a conference paper at ICLR 2025
4 EVALUATION
4.1 MODELS
NWP Numerical Weather Prediction models make predictions on HRRR data by solving mathemat- ical equations that describe atmospheric dynamics and thermodynamics (Olson al. 2022: Bauer et al., 2015). In this work, if refers to the WRF-ARRwv.99 model (Skamarock et al. 2008) in HRRRv4. They assimilate observational data to create an initial atmospheric state, then use this data to initialize model variables. The model integrates these equations forward in time to produce forecasts of various atmospheric parameters at high resolution.
Pangu Pangu- Weather utilizes 3D Earth-specific transformer (3DEST) architecture based on Tran-Transformer (Liu al., (021b). The 3DEST model incorporates height information as an ad- ditional dimension, enabling it to effectively capture atmospheric state relationships across different pressure levels. Predictions are made by processing reanalysis weather data through hierarchical temporal aggregation strategy, which reduces cumulative forecast errors.
FuXi FuXi employs a cascaded deep learning architecture for weather forecasting. It features three components: cube embedding to reduce input dimensions, a U-Transformer using Swin Transformer V2 (Liu et al. 2022) blocks for data processing, and fully connected layer for predictions. The cascade involves three models optimized for different time windows (0-5, 5-10, and 10-15 days), using outputs from shorter lead time models as inputs for longer ones to reduce forecast errors.
HR-Heim The architecture of HR-Heim follows conventional structure with encoder, series of transformer layers, and a decoder, inspired by the FuXi architecture (Chen et al. 2023b). For the encoder, we utilize causal convolutions from Mag ViTv2 capture spatial-temporal input Yu al., 2023a). The transformer segment consists of multiple stacked SwinTransformer blocks (Liu et al. 2022). Unlike typical Vision Transformer decoders that use a simple MLP with 1 convolution, which can hinder resolution, our decoder progressively upscales the feature map from # W to the target size H W through a series of steps. Each step resolves details its specific resolution level, incorporating convolutional layers and upsampling operations to enhance prediction quality. More details about the architecture and its effectiveness are explained in the supplementary materials.
4.2 COMPUTE RESOURCES AND EXPERIMENT SETUP
The dataset creation process is resource-efficient, leveraging improved code efficiency and multi- threading capabilities. Generating the dataset for half-year period requires approximately hours on 42 CPU machines, utilizing around 2.5 TB of memory. HR-Ex MRR-Extreme is designed to be machine learning (ML) ready, allowing users simply load a file and use keys to retrieve inputs. targets, and masks to create tensors for model. All models were evaluated the evaluation of deep learning model for half a year's data taking approximately hours. For our experiments, we set the batch size to 8. Given HR-Extrenn IR--Extemeess ML-ready format, it is straightforward to use, with no additional parameter setup required.
an Nvidia A A100 80G GPU, with
4.3 EXPERIMENTS
RMSE on original set
RMSE on HR-Extreme
Mean increasement of RMSE of each variable
Model Type
HR-Heim
1.40
1.60
34.30 %
Pangu
2.77
10.42
394.23%
Fuxi
2.39
6.10
121.81 %
NWP 2.35
3.27
78.08%
Table 2: Evaluation NWP and deep learning models on extreme weather compared to on normal datasct.
All models (Pangu, Fuxi, and HR-Heim) were trained on HRRR data spanning the U.S. from January 2019 to June 2020, from scratch. They were trained under identical parameters and same level of model parameters, and hyperparameter tuning was applied to HR-Heim. Furthermore,
7

---

Published as a conference paper at ICLR 2025
HR-Heim
NWP
1090 1.N00
Lime
FuXi
Pangu
Oriinall
Figure 3: The comparison of each variahle original test set and HR-Fin ERR-xxtemme evaluated on four models, where the blue are normalized RMSE original test and green areas normalized RMSE on HR- Extreme.
none of these models were fine-tuned on our HR-Extreme dataset, ensuring a fair basis for compar- ison and evaluation. We first evaluated the NWP model and the deep learning model the original test set spanning from July 2020 to the end of 2020. Subsequently. these models were assessed and compared on HR-Extreme during the same period, illustrated in Table 2 The losses on HR- Extreme for all models increased significantly compared to general losses. The highest increase was observed in Pangu, which experienced nearly fourfold rise, while the smallest increase was noted in HR-Heim, at 34.3%. This underscores the substantial impact of extreme weather on model performance.
The complete loss for each variable, evaluated on both the original test set and HR- Extrexee and the comparisons are shown in Table and Table 5 in the appendix A.2] The best results in these tables on the original test set are highlighted in black bold font, while those on HR-Extreme are highlighted red bold font. In summary, HR- HiR-Heim exhibits superior performance on both the original dataset and -R-EExreme across nearly every variable. In terms of original test set, only 4 out of 69 variable losses are slightly higher than the results of NWP, and they are and V component of wind speed, temperature and specific humidity at 50 millibars. On HR-Extreme, only the humidity 50 and 100 millibars of HR-He Reii are slightly higher than that of NWP and FuXi respectively, demonstrating HR- strong baseline HRRR data in terms of both general and extremes predictions.
8

---

Published as a conference paper at ICLR 2025
From Figure where the blue areas represent normalized RMSE the original test set and the green areas represent normalized RMSE on HR-Extreme, it is evident that the losses for nearly all variables of all models losses on the original set. However, HR-Heim performs most stably in both the original test set and HR-Extreme, with both green and blue areas of HR-Heim being dramatically smaller than those of other models. Notably, NWP shows more stability in terms of extreme weather detection com- pared previous SOTA deep learning models, Pangu and FuXi, showing the demand of improving extreme weather predictions of deep learning models. We conducted additional experiments with
HR-Extteme increase significantly, often by many times compared to
RMSE of NWP on original set
RMSE of NWP HR-Extreme
RMSE of HR-Heim on HR-Extreme
Lead time
1
2.35
3.27
1.60
2
3.55
4.12
2.29
3
4.63
5.20
2.86
4
5.49
6.04
3.43
Table 3: RMSE of NWP and HR-HHeim on HR-EExtreee with different lead times
varying lead times for the NWP and HR-Heim models. The results show that losses gradually increase with longer lead times. Notably, the HR-Heim model consistently and significantly outper- forms the NWP model across all lead times, achieving even lower losses than the NWP model on the original dataset. This demonstrates the HR-Heim model's effectiveness as a robust baseline for high-resolution predictions (e.g., HRRR) and extreme event analysis HR-Ex TR--xtreme.
4.4
CASE STUDY
2020081520 NWP prediction error heatmap
2020081520 Pangu prediction heatmap
2020070222 HR-Heim prediction error heatmap
2020070222 FuXi prediction error heatmap
Figure The mean error heatmap of variables on entire U.S for four mdools.
Figure presents the mean enror heatmap visualization of all variables with normalized loss across the entire U.S. Normalizing losses ensures each variable equally contributes to the eгтоr heatmap despite differing scales. The upper left and right panels depict NWP and Pangu predictions, spectively, at 8 p.m. on August 15, 2020, while the lower left and right panels show HR- ERR-Extreme and Fuxi predictions 10 p.m. July 2, 2020. Different models used for single timestamp

---

Published as a conference paper at ICLR 2025
for comparison, and two timestamps are included for a more comprehensive demonstration. A red bounding box indicates the occurrence of a specific type of extreme event, and overlapping bounding boxes denote multiple extreme events in the same area.
In the upper left heatmap. bright areas covered with several bounding boxes indicate regions with high losses where multiple extreme weather events occurring. The upper right heatmap features a very large bounding box with significant losses, which means Pangu makes significant errors in predicting this extreme event. Comparing the upper images highlights the superior generalizability of HR-Extreme, demonstrating its capacity to reveal the varying abilities of different models to detect extreme weather. Specifically, it shows that NWP's predictions for extreme events outperform Pangu's s in this case, particularly regarding the largest bounding box, which corresponds to excessive heat. This comparison confirms that the most significant loss contributions are effectively identified by extreme weather.
The lower heatmaps further prove that the most prominent loss contributions are accurately situated within the bounding boxes. However, the most apparent loss areas are not always precisely centered within the bounding boxes, likely due to the movement of extreme weather events and HR- HRR-Extreme's basis reports. Despite this, HR-Extreme exhibits excellent generalizability for both physical- based and deep-learning based methods. We also provide more case studies of physical varibles in different extreme events in appendix A.I
4.5 LIMITATION
Aside from excessive heat and cold extremes, HR-Extreme identifies bounding boxes primarily based user reports, including inputs from individuals, weather stations, and agencies. This re- liance introduces uncertainty in identifying the range and span of extreme weather events, particu- larly in regions with sparse populations insufficient detection devices, such mountains, marine areas, and deserts. Consequently, many events may be omitted. Although HR-Extreme encompasses wide range of extreme weather events, it does not account for some large weather phenomena, such as tropical depressions and tropical eyclones et al. 2017). These phenomena are also crucial for predicting extreme weather events.
CONCLUSION AND FUTURE WORK
In this work, we present a comprehensive fine-grained dataset that encompasses 17 types of ex- treme weather events in the US, derived from HRRR data, which is updated hourly and features 3-km resolution. Due the incomplete records of extreme events in the NOAA database (National Centers for Environmental Information (NCEI) 2024), we utilized records from the NOAA Storm Prediction Center (National Oceanic and Atmospheric Administration (NOAA), Storm Prediction Center (SPC) 2024) and employed unsupervised clustering along with manual filtering to compile a complete set of extreme events that significantly impact HRRR feature maps. These events are anticipated to be accurately predicted by both deep learning and physical models, ensuring precise identification of extreme weather events. HR-Extreme serves as a novel dataset for evaluating the performance of weather forecasting models, particularly in practical applications. Our experiments indicate that the misprediction of extreme or "unusual" events significantly contributes to the overall prediction losses, revealing deficiencies in current model performance and underscoring the under- valuation of these events. Our proposed model outperforms SOTA transformers and NWP models in single-step step prediction accuracy on both original dataset and extreme weather dataset.
For future work, since the current dataset is primarily built on user reports, developing more and im- proved methods combining with user reports to identify the time span and range of extreme weather events could yield a more accurate dataset. In addition, incorporating precursors of extreme weather events as well as large weather phenomenons such as tropical cyclones into the dataset could provide an even more comprehensive training framework. Furthermore, it would be beneficial to fine-tune models specifically on the extreme weather dataset to enhance their practical utility. Regarding HR- Heim, although it has superior performance compared to both SOTA deep learning models and NWP model. it is only trained and tested on single hour prediction, but still serves as a strong baseline on HRRR data. Thus, improved version of HR-Heim predict longer period can be developed.
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Peter Bauer, Alan Thorpe, and Gilbert Brunet. The quiet revolution of numerical weather prediction. Nature, 525(7567):4-555, 2015. doi: 10.1038/nature14999 14956. URL httpp:://o...ccgg///
1038/ http//aturee99900
Kaifeng Bi, Lingxi Xie, Hengheng Zhang, Xin Chen, Xiaotao Gu, and Qi Tian. Accurate medium- range global weather forecasting with 3d neural networks. Nature, 619(7970):533-538, July doi: 00.100//41186-023-00000 1038/ 1038/s4111 0010//415866--6 023--66------0 06185--33663333
2023. ISSN 1476-4687.
URL https::/dd..ozg///////
Brian K. Blaylock. Herbie: Retrieve Numerical Weather Prediction Model Data, May 2024. URL
https: wttp://dooooooooo .wr./rr///o / 5281//00oooo. 11111 866
Kang Chen, Tao Han, Junchao Gong, Lei Bai, Fenghua Ling, Jing-Jia Luo, Xi Chen, Leiming Ma, Tianning Zhang, Rui Su, et al. Fengwu: Pushing the skillful global medium-range weather forecast beyond 10 days lead. arXiv preprint Stt::2304.029446. 2023a.
Lei Chen, Xiaohui Zhong, Feng Zhang, Yuan Cheng, Yinghui Xu, Yuan Qi, and Hao Li. Fuxi: cascade machine learning forecasting system for 15-day global weather forecast. upj Cli- mate and Atmospheric Science, 6(1): November 2023b. ISSN 2397-3722. doi: 10.1038/
$41612-002-00512-1 UtLhttps::/
org/10..///.o 1038/ 44612 023 00512-1
Martin Ester, Hans-Peter Kriegel, Jörg Sander, and Xiaowei Xu. A density-based algorithm for discovering clusters in large spatial databases with noise. In Proceedings of the Second Inter- national Conference on Knowledge Discovery and Data Mining, KDD'96, pp. 226-231. AAAI
Press, 1996.
European Centre for Medium-Range Weather Forecasts (ECMWF), Bemwf global weather forecast- ing system, 2024. URL hhttps://wwwt wwwwf. http://////////efooesssss/////// cessed: 2024-05-18.
Ac-
Tao Han, Song Guo, Fenghua Ling, Kang Chen, Junchao Gong, Jingjia Luo, Junxia Gu, Kan Dai, Wanli Ouyang, and Lei Bai. Fengwu-ghr: Learning the kilometer-scale medium-range global weather forecasting. arXiv preprint arXiv:2402.00055. 2024.
Hans Hersbach, Bill Bell, Paul Berrisford, Shoji Hirahara, András Horányi, Joaquín Muñoz Sabater, Julien Nicolas, Carole Peubey, Raluca Radu, Dinand Schepers, Adrian Simmons, Cornel Soci, Saleh Abdalla, Xavier Abellan, Gianpaolo Balsamo, Peter Bechtold. Gionata Biavati, Jean Bidlot, Massimo Bonavita, and J.-N Thépaut. The era5 global reanalysis. Quarterly Journal of the Royal Meteorological Society, 146, 06 2020. doi: 10.1002/qj.3805.
Ryan Keisler. Forecasting global weather with graph neural networks. arXiv:2202.07575. 2022.
arXiv preprint
Dmitrii Kochkov, Janni Yuval, Ian Langmore, Peter Norgaard, Jamie Smith, Griffin Mooers, Milan Klöwer, James Lottes, Stephan Rasp. Peter Düben, al. Neural general circulation models for weather and climate. Nature, pp. 1-7, 2024.
Remi Lam, Alvaro Sanchez-Gonzaler, Matthew Willson, Peter Wirnsberger, Meire Fortunato, Fer- ran Alet, Suman Ravuri, Timo Ewalds, Zach Eaton-Rosen, Weihua Hu, et al. Graphcast: Learning skillful medium-range global weather forecasting. arXiv preprint 221.
2022.
Guolong Liu, Jinjie Liu, Yan Bai, Chengwei Wang, Haosheng Wang, Huan Zhao, Gaoqi Liang, Jun- hua Zhao, and Jing Qiu. Eweld: A large-scale industrial and commercial load dataset in extreme weather events. Scientific Data, 10(1))615, September 2023. ISSN 2052-4463, doi: 1038/ htt.:///r.......000//1100000-000 /tt.://000.....././/////./..//..
-415977-02-005033-6
Na Liu, Anyuan Xiong, Qiang Zhang, et al. Development of basic dataset of severe convective weather for artificial intelligence training. Journal of Applied Meteorological Science, 32(5): 530-541, 2021a. doi: 10. 1
Z. Liu, H. Hu, Lin, al. Swin transformer v2: Scaling up capacity and resolution. Proceedings of the IEFE/CVF conference computer vision and pattern recognition, pp. 12009-12019, 2022.

---

Published as a conference paper at ICLR 2025
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows, 2021b.
National Centers for Environmental Information (NCE)). NOAA Storm Events Database, 2024.
URL httph:://ww/wtttt ncdc http://www.....a.aooo/o////oooaa Accessed: 2024-05-20.
National Oceanic and Atmospheric Administraationn (NOAA), High-resolution rapid refresh (hrrr),
2024. UKL/stppss//ppolleeeeeettttt.
/hrrr/ Accessed: 2024-05-18
National Oceanic and Atmospheric Administration (NOAA), Storm Prediction Center (SPC).
NOAA Storm Prediction Center Severe Weather Event Reports, 2024. URL https://wwwt
spc www.aaa...o...o http://lmo//// http:///lmo////orrttaaa
Accessed: 2024-05-20.
Tung Nguyen, Johannes Brandstetter, Ashish Kapoor, Jayesh K Gupta, and Aditya Grover. Climax:
A foundation model for weather and climate. arXiv preprini (rXiv:2301.10344 2023a.
Tung Nguyen, Jason Jewik, Hritik Bansal, Prakhar Sharma, and Aditya Grover. Climatelearn:
Benchmarking machine learning for weather and climate modeling, 2023b. URL https:
//axxiv.org/abs//20000
Tung Nguyen, Rohan Shah, Hritik Bansal, Troy Arcomano, Sandeep Madireddy, Romit Maulik, Veerabhadra Kotamarthi, Ian Foster, and Aditya Grover. Scaling transformer neural networks for skillful and reliable medium-range weather forecasting. arXiv preprint arXiv:23/2.0368 2023c.
Joseph B. Olson, Ming Hu, Tatiana G. Smirnova, Terra Ladwig. Jaymes S. Kenyon, Ravan Ah- madov, David D. Turner, Jeffrey Duda, and Trevor I. Alcott. The high-resolution rapid refresh (hrrr): An hourly updating convection convection-allowing forecast model. part i: Motivation and system description. Weather and Forecasting, 37(8) 1371-1399, 2022. doi: 10.1175/VAA----1-005...
URL https: /ttp:://doo..oom http://oooo.rg//////// 1175/ WAF -
Jaideep Pathak, Shashank Subramanian, Peter Harrington, Sanjeev Raja, Ashesh Chattopadhyay, Morteza Mardani, Thorsten Kurth, David Hall, Zongyi Li, Kamyar Azizzadeneshelii Pedram Hassanzadeh, Karthik Kashinath, and Animashree Anandkumar. Fourcastnet: A global data- driven high-resolution weather model using adaptive fourier neural operators, 2022.
Prabhat, K. Kashinath, M. Mudigonda, S. Kim, L. Kapp-Schwoerer, A. Graubner, E. Karais- mailoglu, L. von Kleist, T. Kurth, A. Greiner, A. Mahesh, K. Yang, C. Lewis, Chen, A. Lou, S. Chandran, B. Toms, W. Chapman, K. Dagon, C. A. Shields, T. O'Brien, M. Wehner, and W. Collins. Climatenet: an expert-labeled open dataset and deep learning architecture for en- abling preecirceii analyses of extreme weather. Geoscientific Model Development, 14(1): 107-124, 2021. doi: 10.5194/gmd- 14- 102-2021. htt/htttss://mmm.coope articles /ttpclees//////////3
org/
Ilan Price, Alvaro Sanchez-Gonzalez, Ferran Alet, Timo Ewalds, Andrew El-Kadi, Jacklynn Stott, Shakir Mohamed, Peter Battaglia, Remi Lam, and Matthew Willson. Gencast: Diffusion-basee ensemble forecasting for medium-range weather. arXiv preprint arXiv:2312.15796 2023.
Evan Racah, Christopher Beckham, Tegan Maharaj. Samira Ebrahimi Kahou, Mr. Prabhat, and Chris Pal. Extremeweather: A large-scale climate dataset for semi-supervised detec- tion, localization, and understanding of extreme weather events. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Ad- vances in Neural Information Processing Systems, volume 30. Curran Associates. Inc., 2017. URL https: (55
papper files//aperr2011/
William C Skamarock, Joseph Klemp, Jimy Dudhia, David O Gill, Dale Barker, Michael G Duda, Xiang-Yu Yu Huang, Wei Wang, Jordan G Powers, et al. A description of the advanced research wrf version 3. NCAR technical note, 475(125):10-5065, 2008.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, E ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In I. Guyon, U. Von Luxburg, Bengio. H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Ad- vances in Neural Information Processing Systems, volume 30. Curran Associates, Inc.,
12

---

Published as a conference paper at ICLR 2025
2017. URL httper//ppoeeela...sseress/scs/s (tta://pe_iieeeppppp//00101011 File/ FLL23322225777111135544aaa222223
Yogesh Verma, Markus Heinonen, and Vikas Garg. Climode: Climate and weather forecasting with physics-informed neural odes. arXiv preprint arXiv:2404.1002 2024.
L. Yu, Lezama, N. Gundavarapu, et al. Language model beats Liffusion-tokenizz is key to visual generation. arXiv preprint arXiv:2310.057.33 2023a.
Sungduk Yu, Walter Hannah, Liran Peng. Jerry Lin, Mohamed Aziz Bhouri, Ritwik Gupta, Bjöm Lütjens, Justus C. Will, Gunnar Behrens, Julius Busecke, Nora Loose, Charles Stern, Tom Beucler, Bryce Harrop, Benjamin Hillman, Andrea Jenney, Savannah L.. Ferretti, Nana Liu, Animashree Anandkumar, Noah Brenowitz, Veronika Eyring, Nicholas Geneva, Pierre Gentine, Stephan Mandt, Jaideep Pathak, Akshay Subramaniam, Carl Vondrick, Rose Yu, Laure Zanna, Tian Zheng, Ryan Abernathey, Fiaz Ahmed, David Bader, Pierre Baldi, Elizabeth Barnes, Christopher Bretherton, Peter Caldwell, Wayne Chuang, Yilun Han, YU HUANG. Fernando Iglesias-Suares Sanket Jantre, Karthik Kashinath, Marat Khairoutdinov. Thorsten Kurth, Nicholas Lutsko. Po-Lun Ma, Griffin Mooers, David Neelin, David Randall, Sara Shamekh, Mark Taylor, Nathan Urban, Janni Yuval, Guang Zhang, and Mike Pritchard. Climsim: A large multi-scale dataset for hybrid physics-ml climate emulation. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (eds.), Advances in Neural Information Processing Systems, volume 36, pp. 22070-22084. Curran Associates, Inc., 2023b. URL https:s//proceeddngs..eeee wtwp://weee.iaeeeeeee.......oooe httpppppeee..leeeepp//////////
45 4455c001822227563500002222223333 www.appppppaaaaatt..a.a and
Bac.aaattt__addBBeecaaaassss Benchmarks.
pdf
Yuchen Zhang, Mingsheng Long, Kaiyuan Chen, Lanxiang Xing, Ronghua Jin, Michael I. Jordan, and Jianmin Wang. Skilful nowcasting of extreme precipitation with nowcastnet. Nature, 619
(7970):526 532, July 2023. ISSN 1476-4687. doi: 10001////111800220666661111 URL https: 7/doi oww. 10. 1038/ 4 586 wewee44111------11
Ziqi Zhu, Li Zhuo, Panling Qu, Kailong Zhou, and Jing Zhang. Extreme weather recognition using convolutional neural networks. 2016 IEEE International Symposium on Multimedia (ISM), pp. 621-625, 2016. doi: 10. 0.1109//SM.2006.00133
A APPENDIX
A.1 ADDITIONAL. VARIABLE CASE STUDY
We conducted statistical analysis comparing values between normal and extreme cases as shown in figure To mitigate seasonal and regional effects, we randomly selected 50 timestamps within 15 days of the extreme event's start or end date, ensuring at least 24 hours separation from the event occurrence, within the same area. Extreme variables were averaged over the event duration, and details of the variables are provided in Table I For clarity, the analysis focused on variables closest the ground. Extreme means differ significantly from normal cases, demonstrating the dataset's ability to statistically capture extreme values and events. Different types of extreme events exhibit distinct characteristies. while similar types show consistent features. During heavy rainfall, humidity increases rapidly, often with fluctuations in pressure and wind speeds. Thunderstorm winds display more complex variability temperature, wind speed, and pressure. Cases of hail, lightning, and marine strong winds reveal different patterns: hail and lightning, which co-occur in this case, show significant increases in humidity, mean sea level pressure, and geopotential height, while marine strong winds show drops in mean sea level pressure and geopotential height, with notable increase in humidity.
A.2 DETAILED VARIABLE LOSS AND TYPE DESCRIPTION
13

---

Published as a conference paper at ICLR 2025
Otew
Marolaal
1200
1,000
1,000
3.000
1.000
1.3000
0,000
0.000
1100
hormal values ep us Extreme values
Hill Lightiing [Case
Homol valuess Extt Dotreme values Martne Strong Wind Kass
Figure 5: Case studies for physical variables in normal and extreme cases in the same area.
14

---

Published as a conference paper at ICLR 2025
HR-
HR-
Pangu HR- Extreme Loss
FuXi HR-
NWP
Heim
Pangu General
FuXi General Loss
NWP General Loss
Heim
HR-
Variable
HR-
General
Extreme Loss
Extreme
Extreme
Loss
Loss
Loss
Loss
msl
27.00
27.88
37.09
36.06
40.48
78.61
60.30
80.07
2t
0.40
0.47
0.81
0.74
0.86
1.63
0.52
1.19
10u
0.59
0.75
0.81
0.90
1.09
0.89
1.32
10v
0.59
0.77
0.82
0.93
0.78
1.12
0.89
1.36
bgtn 5000
6.48
6.80
8.35
8.53
10.25
25.09
8.05
10.58
hgtn.
5.93
7.39
9.54
28.76
7.04
8.56
hgtn_ 15000
5.14
5,80
6.55
7.15
10.70
35.26
6.83
8.22
hgtn_20000
20000
4.77
5.54
6.77
7.53
10.54
36.72
6.50
8.12
25000
4.32
5.10
272.94
9.84
34.94
6.12
7.80
hgtn_30000
3.95
4.69
5.19
14,26
9.01
30.41
5.82
7.38
40000
3.30
3.93
4.31
17.91
7.38
22.87
5.37
6.84
ngtt_500000
2.85
3.39
4.06
13.11
5.95
18.27
5.18
6.73
hgtn_ 60000
2.58
3.02
3.35
10.66
4.98
14.14
5.11
6.64
hggn_70000 70000
2.40
2.69
3.05
83.58
4.21
10.82
5.06
6.42
85000
2.27
2.44
2.88
28.27
3.45
7.05
5.06
6.51
hgtn_ .2500
2.31
2.44
3.24
39.93
3.40
6.46
5.15
6.86
hgtn_I 100000
2.26
2.37
46.46
44.29
3.41
6.46
5.10
6.90
u_5000
0.49
0.52
0.71
0.69
0.75
1.32
0.48
0.93
u_ 10000
0.47
0.55
0.69
0.74
0.76
1.69
0.49
1.03
u. 15000
0.67
0.83
0.90
1.06
0.97
2.34
0.73
1.45
u 20000
0.96
1.20
1.27
1.84
1.35
2.95
1.08
1.96
u. 2500000
0.92
1.14
1.22
13.53
1.35
3.02
1.03
1.89
1,30000
0.83
1.04
1.09
3.14
1.24
2.71
0.91
1.67
u. 4040000
0.69
0.93
0.91
3.55
1.03
2.38
0.75
1.42
500000
0.62
0.93
0.89
2.94
0.89
2.20
0.70
1.42
u. 1660000
0.58
0.89
2.75
0.82
1.87
1.39
u. 70000
0.59
0.88
0.79
6.60
0.80
1.70
0.69
1.36
1L85500
0.59
0.89
0.78
4.37
0.79
1.57
0.73
1.40
u. 1.92500
0.55
0.86
1.28
4.15
0.72
1.42
u. 100000
0.45
0.60
3.12
2.64
0.57
0.96
0.63
1.04
v_5000
0.52
0.53
0.71
0.68
0.65
0.86
0.47
0.95
v. 1,00000
0.47
0.54
0.70
0.70
1.21
1.00
v_15000
0.68
0.82
0.89
1,03
0.96
1.97
0.72
1.43
2020000
0.98
1.19
1.28
1.90
1.34
2.69
1.07
1.95
v. 250000
0.94
1.13
1.25
11.74
1.36
2.63
1.04
1.92
v_30000
0.84
1.03
1.11
3.10
1.25
2.38
0.92
1.72
7,40000
0.70
0.93
0.93
3.64
1.02
2.06
0.76
1.48
Table 4: Losses of each variable for each model
15

---

Published as a conference paper at ICLR 2025
HR-
HR-
Pangu HR- Extreme Loss
FuXi HR-
NWP HR-
Heim
Pangu General
FuXi General
NWP General Loss
Heim
Variable
HR-
General
Extreme Loss
Extreme
Extreme
Loss
Loss
Loss
Loss
Loss
v_50000
0.63
0.94
0.90
3.06
0.90
1.92
0.71
1.48
v_60000
0.59
0.91
0.81
2.82
0.82
1.77
1.45
v. v_70000
0.60
0.90
0.81
6,52
0.80
1.60
0.69
1.40
v 1.85000
0.60
0.91
0.80
4.78
0.80
1.60
0.74
1.43
0.92000
0.88
1.36
4.80
0.76
0.74
1.47
v. 10000000
0.46
0.62
4.04
3.34
0.60
1.01
0.65
1,08
L5000
0.32
0.35
0.43
0.43
0.51
0.63
0.29
0.63
L,0000
0.24
0.29
0.39
0.90
0.57
t_ (15000
0.23
0.29
0.33
0,36
0.36
0.82
0.26
0,49
L,20000
0.31
0.34
0.41
0.46
0.43
0.63
0.35
0.43
t 25000
0.25
0.28
0.34
5.42
0.38
0.83
0.29
0.38
L L30000
0.21
0.26
0.28
1.29
0.34
0.92
0.25
0.39
400000
0.21
0.28
0.28
I.57
0.35
0.96
0.25
0.43
5,50000
0.19
0.27
0.31
1.43
0.35
0.89
0.24
0.44
L60000
0.18
0.26
0.28
1.47
0.33
0.87
0.22
0.43
70000
0.21
0.30
0.30
7.29
0.38
0.98
0.26
0.51
LS5000
0.28
0.35
0.44
2.64
0.49
1.15
0.38
0.65
1.92000
0.31
0.37
1.0S
3.02
0.53
1.20
0.43
0.75
1_100000
0.29
0.36
9.79
3.11
0.52
1.19
0.48
0.95
q-5000
6.30e-08
5.61e-08
7.99e-08
6.99e-08
8.89e-08
1 1.49e-07
5.03e-08
5.51c-08
q_ 10000
1.87e-07
3.86c-07
2.42c-07
3.66e-07
4.20c-07
6.99c-07
4.45c-07
6.63c-07
q 15000
6.13e-07
1.17e-06
7.38c-07
1.18c-06
8.85e-07
1.66c-06
9.66e-07
2.13c-06
9 1,20000
3.02e-06
5.68c-06
3.86e-06
7.74e-06
3.94e-06
8. 8.34e-06
4.73e-06
1.05e-05
q 2,25000
9.16e-06
1.72e-05
1.17e-05
9.99c-05
1.21c-05
2.72c-05
1,47c-05
3.31c-05
q q-30000
2.03e-05
3.85c-05
2.58e-0S
1.02e-04
2.70e-05
6.43e-05
3.17e-05
7.52e-05
q 4.0000
5.82e-05
1.13e-04
7.41c-05
3.92c-04
7.7lc-05
1.93c-04
8.62c-05
2.15c-04
q_50000
1.13e-04
2.16e-04
1.53c-04
7.44c-04
1,46c-04
3.74c-04
1,56c-04
3.72c-04
q 9_60000
1.79c-04
3.34c-04
2.23e-04
1.05e-03
2.26e-04
5.60e-04
2.32e-04
5.10e-04
q 7,70000
2.56e-04
4.76e-04
3.14c-04
3.60c-03
3.20c-04
7.35c-04
3.22c-04
6.76c-04
q 1,85000
4.13e-04
6.60c-04
5.04e-04
3.27e-03
5.02e-04
9.85e-04
4.83e-04
9.04e-04
q. 1,92500
4.57c-04
5.77c-04
1.28e-03
3.37e-03
3.78e-04
1.03e-03
5.44e-04
8.06e-04
q. 100000
3.59e-04
4.32e-04
5.13c-03
2.44c-03
4.97c-04
9.81c-04
4,99c-04
8.01c-04
Table 5: Losses of each variable for each model, follow Table
16

---

Published as a conference paper at ICLR 2025
Variable
Definition
May be triggered by intense rain after wildfires. A slurry of loose soil, rock, organic malter, and water, similar to wet concrete, which is capable of holding particles larger than gravel in suspension.
Debris Flow
life-threa,,ning, rapid rise water into a normally dry area beginning within minutes to multiple hours of the causative event (e.g., intense rainfall, dam failure, ice
Flash Flood
jam).
Any high flow, overflow, or inundation by water which causes damage. In general, this would mean the inundation normally dry area caused by an increased water level in an established watercourse, or ponding of water, associated with heavy rainfall A rotating, visible extension of cloud pendant from convective cloud with circulation not reaching the ground. is precursor
Flood
severe weather events. The wind shear caused by (sudden changes in wind speed and direction) will
Funnel Cloud
endanger aviation.
Hail
Frozen precipitation in the form of halls or irregular lumps of ice. Unusually large amount of rain which does not cause a Flash Flood or Flood event, but causes damage
Heavy Rain
A sudden electrical discharge from a thunderstorm, resulting in fatality, injury, and/or damage
Lightning
Hail 3/4 of an inch in diameter or larger, occurring over the waters and bays of the occan, Great I Lakes, and other lakes with assigned specific Marine Forecast Zones Non-convective, sustained winds trequent gusts of 48 knots (55 mph) or more, resulting in a fatality, injury, or damage, over the waters and bays of the ocean, Great Lakes, and other lakes with assigned specific Marine Forecast Zones.
Marine Hail
Marine High Wind
Non-convective, sustained winds frequent gusts up to 47 knots (54 mph), resulting in fatality, injury, or damage, occurring over the waters and bays of the occan, Great Lakes, and other lakes with assigned specific Marine Forecast Zones.
Marine Strong
Wind
Marine
Winds, associated with thunderstorms, occurring over the waters and bays of the occan, Great Lakes, and other lakes with assigned specific Marine Forecast Zones with speeds of at least 34 knots (39 mph) for 2 hours or less.
Thunderstorm
Wind
Winds, arising from convection (occurring within 30 minutes of lightning being observed or detected), with speeds of at least 50 knots (58 mph), or winds of any speed (non- movere thunderstorm winds below 50 knots)
Thunderstorm Wind
A violently rotating column of air, extending or from cumuliform cloud or underneath cumuliform cloud, to the ground, and often (but not always) visible as a
Tornado
condensation funnel.
A rotating column of air, pendant from convective cloud, with its circulation extending from cloud hase the water surface of bays and waters of the Great Lakes, and other lakes with assigned Marine Forecast Zones.
Waterspout
Severe thunderstorm wind, strong wind that causes damage, which is reported and recorded by NOAA Storm Prediction Center (National Oceamic and Atmospheric Administration (NOAA), Storm Prediction Center (SPC) 2024).
Wind
Large area of excessive heat above 37 degrees celsius in US during daytime, manually filtered.
Heat
Large area of excessive cold below -29 degress celsius in US during nighttime, manually filtered.
Cold
Table 6: Explaination of each type of extreme weather included in HR-Ex HR-Extreme
17


Reviews:
[{"review_id": "r389JZ1OuR", "reviewer": "Reviewer_wJpu", "summary": "Extreme weather forecasting is a crucial problem for the whole world. With the rise of deep learning-based weather forecasting models, the effectiveness of them on extreme weathers are not well analyzed.  This paper targets on providing a new benchmark for extreme weather forecasting. Authors employ the HRRR data and utilize the extreme events record in three sources to crop the extreme feature from the original HRRR dataset. Experiments are conducted with four baselines to show the performance.", "questions": "please refer to the weakness", "limitations": "", "rating": 5, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "Pros: \n\n1. Extreme weather forecasting evaluation is an important research problem. \n2. The provided dataset introduces 17 extreme events, which are comprehensive. \n3. Authors also release the code for generating the data.", "weaknesses": "Cons:\n\n1. It is not clear how the HR-heim model is trained. \n2. Considering the dataset is a processed version of HRRR, it would be helpful to provide the geo-location of the extreme data to facilitate more diverse use from users.  \n3. While the dataset is valuable, there is almost no analysis are present, especially compared to the ERA5 dataset, which is not insightful.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "1C0CRZU292", "reviewer": "Reviewer_Jbrg", "summary": "This work utilizes high-resolution HRRR data to create the HR-Extreme dataset, which encompasses a comprehensive set of 17 extreme weather types. The aim is to provide a more specialized dataset for evaluating the performance of weather forecasting models. To achieve this goal, the authors employed unsupervised clustering and manual filtering methods to develop a complete feature map of extreme events in a machine learning-ready format for the continental United States. The dataset was then used to assess the 1-hour forecasting capabilities of existing medium-range prediction models in comparison to the HR-Heim model proposed in this study.", "questions": "1. Was the result of the clustering algorithm verified by domain experts? Is there corresponding uncertainty detection and assessment?\n\n2. What is the basis for the \"types of events without specific ranges or those not related to obvious variations in feature map predictions\"?\n\n3. Can data from different sources be used separately?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. **Significance**: It fills the gap in benchmarking for extreme weather assessment in deep learning-based medium-range weather forecasting tasks.\n\n2. The dataset is clearly introduced, and it will be fully open-sourced.\n\n3. The authors present comprehensive experiments and baseline results.", "weaknesses": "1. The description of the dataset generation process lacks clarity in some areas: a more detailed introduction of the clustering method is needed, including how records from different sources are handled and the hyperparameters of the algorithm.\n\n2. There are also unclear aspects in the experimental description:  \n   a) The baselines are models trained on globally coarse-resolution grids; Was there any further fine-tuning on this dataset? What preprocessing steps were taken?  \n   b) What is the training strategy for HR-Heim? Does it use the same hyperparameter settings on the original dataset and HR-Extreme?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "BtnGWgwOtH", "reviewer": "Reviewer_kLu6", "summary": "The authors present a new dataset of labelled extreme weather events over the continental US based on a high resolution (3km) numerical forecast product. They compare the events from a numerical weather prediction (NWP) model as well as two baseline ML based weather models and a newly proposed variant. The authors claim improved skill in their new variant compared to the other baselines.", "questions": "Please provide specific responses to each of the concerns and questions raised above in the weaknesses.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The constructed dataset could provide a useful evaluation of extreme events in new ML based weather prediction models which tend to be evaluated on larger scale statistics, potentially hiding biases in such events. It is based on an operational high-resolution dataset with a mixture of automated and manual labelling.", "weaknesses": "There are a number of technical and fundamental weaknesses that undermine the above strengths.\n\nMajor issues:\n1) Extreme events are, by definition, rare. While a database of such events derived from a high-resolution reanalysis product could provide a useful starting point for evaluation, the specific events are much less useful without a probabilistic understanding or measure of their likelihood at different lead times. i.e. It is probably more likely that any of the given events *wouldn't* have happened given a similar atmospheric state if the conditions were encountered again. Without a probabilistic understanding of the probability of an event (based on lots of comparable events that weren't extreme), this dataset has very little value as presented.\n 2) This leads to my second concern regarding the evaluation setup. Given the dataset extracts the atmospheric state at t=0, -1 and -2 hours, I presume the evaluations happen from an atmospheric state of t=-2 hours and run forward? This is no longer forecasting, but nowcasting and quite a different task. Since the atmospheric state already has the extreme event and the model just needs to advect it correctly. I have no idea how the NWP model is compared in this setting since presumably the authors don't run this explicitly with the extracted state? Or perhaps they do? Also, since the authors use the same NWP as was used to create the HRRR dataset, why doesn't it perform as well as the other models? At what resolution are the models run, are they run globally, over the US, or only over the event region? Are the global Fuxi and Pangu models retrained for these regions?\n 3) Related to this, to what extent is this dataset used to train the different models? Does HR-Helm get to see some or any extreme event data during training? Given the issues described in (1), how do you avoid overfitting?\n\nOne more minor issue is that I would like to see Figure 4 presented for the same day for all 4 models, with a separate figure for the other day for all 4 models in the appendix. Currently it's impossible to fairly compare the skill of the models (although it seems like the NWP already does better than HR-Helm across the two examples).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4ydmhU3ohp", "reviewer": "Reviewer_tGGo", "summary": "This paper introduces a high-resolution dataset, called HR-Extreme, for numerical weather forecasting under extreme weather conditions, an area often overlooked in weather forecasting literature. The authors also present a baseline deep learning model alongside the dataset, called HR-Heim, which outperforms state-of-the-art weather models in extreme weather conditions.", "questions": "* Could you extend the dataset analysis to include predictions for more than a single hour ahead?\n* As an extension to the previous question, could you analyze and compare the performance of state-of-the-art models and HR-Heim for both short-term and long-term forecasting?\n* Could you further evaluate the performance of HR-Heim on a similar extreme weather dataset?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "* The authors provide a high-resolution dataset for numerical weather forecasting under extreme weather conditions. The need for such dataset has been evident for some time, as highlighted by the low performance analysis of SOTA weather forecasting models.\n* The construction of the dataset is well-motivated, and the authors provide a clear and thorough explanation of both the data collection and construction processes.\n* Additionally, the authors provide a baseline deep learning model called HR-Heim, which is inspired by a SOTA numerical weather forecasting model, to specifically excel and outperform SOTA models under extreme weather conditions.", "weaknesses": "* In Section 2.2, the authors discuss some datasets that share similarities (to a certain extent) with the proposed one. It would have been beneficial to illustrate the performance of the SOTA models and the proposed HR-Heim on some of these datasets. Specifically, the last dataset introduced by Liu et al., which also includes certain extreme weather conditions, could have been included in the experiments to support the need for HR-Extreme and support the performance of HR-Heim. Given that HR-Heim outperforms SOTA methods in HR-Extreme, we would expect to see similar results on other datasets with extreme weather conditions.\n* As the authors also state as a limitation, this kind of study should include more than just a single-step prediction analysis. Given the increased difficulty of predicting extreme weather events in the long term, an accuracy comparison between HR-Heim and SOTA methods across varying time horizons could be valuable.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)