RiteeshGrandhi/sciclaimeval-shared-task huggingface dataset:
Post-NWP Optimization.
We refer to the optimization problem implicit in the post-processing of NWP model ouput for precipitation forecasting as post-NWP optimization, drawing a connection to skillful precipitation optimization. Post-NWP optimization has several key properties that differentiates it from a typical weather-forecast optimization problem ( Figure 1 ):

Influential Variable Selection. In NWP, each pixel on grid has a list of variables expressing the state regarding atmospheric feature ( Table 1 ).

Sensitivity to Hyperparameter Settings. NWP data can be regarded as tabular data. In this sense, as Kadra et al. [ 25 ] observe, finding the optimal cocktail with tuning the modern regularization techniques can boost the performance comparable to the state-of-the-art models in end-to-end DL models.

Robust Architectures. We expect a new dedicated architecture by considering the spatial and temporal characteristics of our benchmark.

Non-Learning Points of AWS. AWS is not installed at all locations due to space and cost limitations. Therefore, its value for some regions are sparse, which needs to be overcome for the supervision of NWP.

Class-Imbalance. The distribution of precipitation amounts are highly asymmetric. Despite rare occurrences of heavier rain, it can bring significant real-world damages.
Organization.
The remainder of this paper is organized as follows. In Section 2 , we discuss the related literature on traditional NWP approaches, data assimilation, and deep learning-based weather forecasting. In Section 3 , we describe our benchmark KoMet thoroughly. In Section 4 , we formulate the problem settings for the use of KoMet, and Section 5 exhibits the baseline experimental results. Finally, Section 6 concludes the paper.
Numerical Weather Prediction (NWP).
NWP models explain and predict the future changes of the atmospheric conditions through solving dynamics and physics partial differential equations. Since the advent of the NWP models, weather forecasting highly relies on them according to their steady progress over time [ 33 ; 4 ] . NWP applications have been involved into air quality [ 76 ] , solar [ 41 ; 75 ; 67 ] , wind power [ 11 ; 8 ] , wildfire [ 13 ] , anomalies (i.e. severe weather events) [ 14 ; 35 ; 73 ; 50 ] , and hydrological forecasting. The last of these, hydrological forecasting, includes precipitation, humidity [ 15 ] , drought [ 39 ; 62 ] , and evapotranspiration [ 34 ] forecasting.
Despite its raving utility in real world, NWP faces some endemic challenges [ 64 ; 4 ] . Firstly, the quality of NWP models can be determined with how accurate the given observations of present atmospheric states (initial condition) is [ 27 ; 58 ; 68 ; 38 ] . In addition, the computational and power demands of NWP grow in lockstep with the resolution of the forecast, creating a trade-off between forecast accuracy, which necessitates higher levels of resolution, and forecasting time. Lastly, some models utilize the machine learning approach for prediction while there remains a lack of considering two types of uncertainty: aleatoric uncertainty and epistemic uncertainty [ 12 ] .
Data Assimilation (DA).
DA is the process of combining every possible source of information for the accurate prediction of each status.
Generally, DA serves as an initial condition for computing the NWP in weather forecasting [ 27 ; 36 ] .
Various observation platforms have been evolving through weather station, radar, and satellite [ 73 ; 63 ; 15 ; 5 ; 6 ] whose observation equipment is also becoming more sophisticated. To reinforce model uncertainty, variational data assimilation methods have been emerged as a line of works [ 66 ; 40 ] .
Data assimilation methods [ 66 ; 40 ] have been done on such observations. Three-dimensional variational methods (3DVar) [ 10 ; 17 ; 31 ] , four dimentional approaches (4DVar) [ 43 ] , ensemble Kalman filter [ 21 ; 32 ] are general examples of DA techniques used in global NWP workflow.
Deep Learning-based Weather Forecasting.
Deep learning rises as the one way to mitigate the shortcomings of NWP.
DL enables its success by conjugating vast amount of available data as well as geometrically increased computational power of tensor processing units (e.g. GPUs or TPUs) [ 24 ] .
Because deluge of meteorological data has been accumulated since the establishment of weather stations, there are an increasing demand of efforts to adopt modern DL techniques for weather forecasting [ 56 ; 53 ; 48 ; 51 ] . DL-based weather forecast approach can be categorized into two folds: end-to-end DL approach and NWP-DL hybrid approach. End-to-end DL approaches [ 30 ; 46 ; 49 ; 71 ; 74 ; 47 ] learn the representation from meteorological observations using DNN without the access of physics knowledge.
On the other side, NWP-DL hybrid approaches [ 45 ; 71 ; 19 ; 65 ; 20 ; 22 ; 20 ] integrate theoretical models (NWP) and data-driven models (DL), and thus encompass the explainability as well as accurate expressivity.
KoMet Dataset
The KoMet dataset is comprised of predictions from GDAPS-KIM, a global numerical weather prediction model operated by the Korea Meterological Administration (KMA 1 1 1 https://www.kma.go.kr/eng/index.jsp ), as well as AWS observations which serve as ground-truth precipitation data.
Input: GDAPS-KIM
GDAPS-KIM is a global numerical weather prediction model designed to improve the prediction accuracy of weather phenomena with a particular focus on the Korean Peninsula [ 57 ] . GDAPS-KIM provides hourly predictions of various atmospheric variables. This is achieved by assimilating observations from all over the globe to compute the initial state, and subsequently solving dynamics and physics equations to predict future states.

GDAPS-KIM prediction data is provided in the following format: \mathbf{X}\in\mathbb{R}^{T\times L\times C\times W\times H} where each dimension corresponds to the origin time (T) at which the simulation took place, lead time (L) between the origin time and the target prediction time, the variable index (C), and the spatial dimensions of the prediction map (W, H). Each GDAPS-KIM instance in KoMet dataset is a daily simulation executed at 00 UTC. We provide predictions made between July 1st and August 31st (inclusive) of 2020 and 2021, i.e., T=124 days, when rainfall is most intensive due to the seasonal characteristics of the region. We provide predictions with lead times ranging from 0 to 89 hours, i.e., L=90 . We provide data on C=122 atmospheric variables for the Korean Peninsula and the surrounding region lying within [32.94°N, 39.06°N] and [124.00°E, 132.00°E]. GDAPS-KIM has a resolution of 12 km \times 12 km, and thus its spatial dimension is 65 \times 50.

GDAPS-KIM variables can be categorized into two types: Pres variables and Unis variables. While Pres variables include values for various altitudes corresponding to specific air pressure levels (i.e., different isobaric surfaces), Unis data does not.
Each GDAPS-KIM instance has 5 Pres variable types at 22 isobaric surfaces (22 \times 5=110 variables) and 12 Unis variables ( Table 1 ), thus it has 122 variables at each pixel.
Despite its rich variables, using all variables may not always be a silver bullet owing to the high correlation among themselves and noisiness.
Furthermore, as the assimilated initial condition used for numerical prediction has significant spatial- and temporal-variability, it is more prone to inaccuracies compared to other regions, leading to cascading errors.
In this paper, for the sake of benchmark experiments, we use 12 variables 2 2 2 Unis 6 variables: rain, q2m, rh2m, t2m, tsfc, ps; Pres 6 variables: T and rh_liq at 500, 700, 850 hpa. out of 122 through the advice of KMA experts.
Ground-Truth: AWS Observations
Surface observations from AWS provide ground-truth data on hourly precipitation. While AWS observation data includes various weather measurements such as temperature, wind speed, and solar radiation, the precipitation is only utilized for supervision. Unlike GDAPS-KIM predictions,
only several pixels have AWS observations as weather stations are sparsely installed.
Even for such stations, observations are occasionally omitted due to miscellaneous reasons, leading to NaN values.

Raw AWS observation data is provided in the following format: \mathbf{Y}_{raw}\in\mathbb{R}^{h\times S} where each dimension corresponds to the hour (h) at which the measurement of accumulated rainfall (from the preceding hour) is taken and the weather station index (S). AWS observations are recorded every hours at which GDAPS-KIM predictions are provided, with some additional leeway. More precisely, hourly observations are ranging from July 1st, 00:00 UTC to September 3rd, 23:00 UTC, totaling h=3120 hours, from 484 stations ( S=484 ). To facilitate use of AWS data to train GDAPS-KIM post-processing models, we use the location metadata of each weather station to convert AWS data into the spatial grid format used in GDAPS-KIM. The format is \mathbf{Y}\in\mathbb{R}^{h\times W\times H} where each dimension corresponds to the hour (h) at which the measurement of accumulated rainfall is taken, and the spatial dimensions of the prediction map (W, H; identical to GDAPS-KIM).

We investigate the characteristics of AWS considering the sparsity of AWS installation, temporal and spatial rainfall distributions. Figure 2 provides the distributions of hourly and daily average of rainfall amounts on each year. As Figure 2 shows, hourly rainfall events occur sporadically, and these characteristics vary greatly from year to year. For the spatial distribution, among the 65\times 50=3250 pixels comprising the GDAPS-KIM grid, there are only 484 pixels where ground-truth data is available, and even many of those are concentrated to certain locations such as metropolitan areas ( Figure 3 ). These challenges can pose a significant challenge for training calibration models, as the supervisory signal from ground-truth values is limited to specific locations and time.
Inputs.
In our dataset, GDAPS-KIM predictions are provided in NumPy format. These predictions are propagated to deep models, following normalization; each variable is linearly scaled based on min-max values derived from the entire dataset .
Outputs.
We formulate the precipitation forecast problem as a pointwise classification task pertaining to three categorical classes: non-rain, rain, and heavy rain ( Table 2 ).
Following this, the AWS observation data is pre-processed into 2D array format according to the grid used in GDAPS-KIM, respectively. The location of each station is determined within each grid based on the location metadata of AWS stations and grid specifications for GDAPS-KIM. Of course, our benchmark also supports predicting rainfall as a regression task or classification tasks having more categories. However, the evaluation should be based on the Table 2 , which is generally used in South Korea [ 59 ] .
Dataset Split
We split the data temporally into three non-overlapping datasets by repeatedly using approximately 4 days for training followed by 2 days for validation and 2 days for testing. Separation by month or year could cause a shift in distribution between datasets. With reference to Sønderby et al. [ 60 ] , this category of temporal split is utilized. Because the data scale is small, more proportions are devoted to validation and testing compared to Sønderby et al. [ 60 ] .

Due to the overlap between the predicted instances at the same time from different GDAPS-KIM simulations, it is not straightforward to perform this split. For example, there are multiple predictions for April 21st, 2021 2:00PM, including a prediction ran at midnight the same day with lead time of 14 hours, and one ran at midnight of April 20th with 38 hour lead time, to name a few. In this paper, we follow the strategy that divides among prediction simulations, based on the point in time in which the predictions were ran. Since predictions from GDAPS-KIM simulations are highly conditioned on the initial state, our benchmark uses the aforementioned scheme to prevent predictions from the same initial state to leak into different data splits.
Problem Formulation.
In this work, we consider the following optimization model:

\min_{{\bm{w}}}\Big{\{}\mathcal{L}({\bm{w}};\mathcal{D})\triangleq\mathbb{E}_{(\mathbb{X}_{t},\mathbb{Y}_{t})\sim\mathcal{D}}[\ell(\mathbb{X}_{t},\mathbb{Y}_{t};{\bm{w}})]\Big{\}}\vspace{-10pt} (1)

where \mathcal{L} is the objective function parameterized by {\bm{w}} on the dataset \mathcal{D} having the input as the NWP outputs \mathbb{X}_{t} , the corresponding rainy output \mathbb{Y}_{t} at time t , and \ell is the classification loss between the output of the neural network and the ground-truth. \mathbb{X}_{t} is a sequence of NWP predictions printed at time t , {\bm{x}}_{(t,0)},{\bm{x}}_{(t,1)},\cdots,{\bm{x}}_{(t,L-1)} where {\bm{x}}_{(t,i)} is the prediction at time t+i based on a simulated NWP output at time t , L is the length of lead time which means the prediction of additional hour per simulation, \mathbb{Y}_{t} is a sequence of AWS observations {\bm{y}}_{t},{\bm{y}}_{t+1},\cdots,{\bm{y}}_{t+L-1} . In this approach, the prediction of the neural network at time t is defined as the following probabilistic forecast:

 f(\tau;\mathbb{X}_{t_{0}},{\bm{w}},ws)=\mathbb{P}({\bm{y}}_{\tau}|{\bm{x}}_{(t,\Delta-ws+1)},\cdots,{\bm{x}}_{(t,\Delta-1)},{\bm{x}}_{(t,\Delta)})\vspace{-10pt} (2)

where f(\cdot;{\bm{w}}) is the neural network parameterized by {\bm{w}} , t is the time at which the forecast is made, \Delta=\tau-t_{0} , ws is the window size which is the number of consequent observations used for inference, and \mathbb{P}(\cdot) is the probability matrix whose each pixel in grid indicates the probability of class among {‘non-rain’, ‘rain’, ‘heavy rain’}.
Algorithm Description.
Here, we describe one inference step (say the \tau -th; \tau\geq t ) of the proposed post-NWP optimization task. Firstly, the subset \mathbb{X}_{t} is reorganized with channel-wise concatenation: [{\bm{x}}_{(t,\Delta-ws+1)};\cdots;{\bm{x}}_{(t,\Delta-1)};{\bm{x}}_{(t,\Delta)}]\in\mathbb{R}^{(C*ws)\times W\times H} where W,H is the width and height of the grid map and C is the number of variables in each pixel–and thus each instance after reorganization is 3D-shaped. We then create batches based on the modified \mathbb{X}_{t} and perform forward and backward propagation. The model is trained over the entire dataset until convergence, or pre-fixed total epochs.
Temporal Dependencies on Window Size & Lead Time.
The problem formulation of post-NWP optimization elicits two important temporal parameters that influence the learning problem. First, the window size of NWP predictions used for post-processing affects the richness of the input information. While larger window size allows the post-processing model to consider ample information, it may be more difficult to optimize the model. Another important temporal component is lead time. Predictions from different lead times exhibit distinct characteristics, leading to differences in evaluation performance. Taking this into consideration, it may be beneficial to train separate post-processing models for different lead times. For our baseline analysis, we focus on a single model trained to handle all possible lead times, as a representative baseline.
Evaluations.
For evaluation, the validity of forecasting algorithms are assessed using diverse statistical metrics, which are commonly used for precipitation forecasting.
Following common practice in the multi-classification settings [ 18 ] , such metrics can be calculated based on the number of true positives ( TP_{k} ), false positives ( FP_{k} ), true negatives ( TN_{k} ), and false negatives ( FN_{k} ) for some generic class k :

Accuracy (ACC) returns an overall measure of how much the model is correctly predicting on the entire set of data.

Probability of Detection (POD) is a recall calculated as \frac{TP_{k}}{TP_{k}+FP_{K}} .

Critical Success Index (CSI) [ 14 ] is a categorical score that considers more aspects of the confusion matrix similar with F1-score having the value as \frac{TP_{k}}{TP_{k}+FN_{k}+FP_{k}} .

False Alarm Ratio (FAR) [ 3 ] is the number of false alarms per the total number of warnings or alarms, also known as the probability of false detection. It is computed as \frac{FP_{k}}{TP_{k}+FP_{k}}

Bias is defined as the ratio of the observed frequency of occurrence of a phenomenon to the frequency of the occurrence predicted by the forecast model. \frac{TP_{k}+FP_{k}}{TP_{k}+FN_{k}} . If it has a value greater than 1, it means that the frequency of occurrence predicted by the forecasting model is greater than the frequency of occurrence of the actual phenomenon, and therefore more frequent prediction is made. The more accurate the forecast, the closer this index is to 1.
Benchmarking neural network architectures
Various models have been adopted or designed to effectively extrapolate future weather status from past and current state. Here, we provide three baseline architectures for precipitation forecasting: U-Net [ 52 ] , ConvLSTM [ 47 ] , and MetNet [ 60 ; 16 ] (detailed explanations are provided in Appendix).

Table 3 shows the results for various lead times ranging from 6 to 87 hours, according to the changes of the architectures. Compared to the statistics of GDAPS-KIM for predicting class ‘rain’, the qualities of post-processed predictions with the three networks are consistently improved while that of MetNet is the best. In contrast, deep learning-based post-processing rather hinders performance for ‘heavy rain’ prediction. In particular, MetNet faces severe performance degradation. We leave this phenomenon as a challenge to be addressed. Figure 4 illustrates the results of CSI scores for precipitation of \geq 0.1 (mm/h) over lead time. Here, MetNet also outperforms other baseline models.
Influential Variable Selection
As listed in Table 1 , GDAPS-KIM contains many variables. Naively using all available variables may impede generalization, therefore it is important to select the most prominent variables. Variable selection experimentation can be done in various ways, and we propose a baseline experiment in which the variables are divided into Pres type and Unis type groups and considered as manipulated variables and control variables, respectively. An example is illustrated in the Appendix, and we observe that selecting certain significant variables performs better than simply using more variables.
Sampling Strategies for Class-Imbalance Issue
As a solution to the class-imbalance issue, we provide two probabilistic data sampling techniques for target transformation: (1) Under-Sampling for ‘No Rain’ Points that uses only a fraction p of the ‘no rain’ points used for learning, to reduce the no precipitation points, and (2) Balancing for ‘Rain Points’ that maintains the ratio between ‘no rain’ points and ‘rain’ points to 1:p by under-sampling ‘no rain’ points, when there are excessive ‘no rain’ points.

Figure 5 depicts the performance of models trained with different sampling strategies. Although the performance change differs depending on the architecture, it can be confirmed that generalization can be further facilitated if the data distribution is adjusted fairly. It is expected that an approach that manipulates the loss function can also be effective [ 29 ; 7 ] .
Window Size.
The first row of Figure 6 shows the CSI scores as the window size changes. In ‘rain’, the CSI score is relatively unchanged, while the ‘heavy rain’ case is shown to be extremely sensitive.
Weight Decay.
The second row of Figure 6 shows the CSI scores according to the changes in magnitude of weight decay. When comparing the performance of rain and heavy rain, we see differing trends according to increase in weight decay.

The results above suggest the need for combinatorial optimization among a variety of hyperparameters such as batch size, optimizer, beta of batch normalization, and learning rate.
Conclusion & Future Work
In this paper, we present a new benchmark dataset, termed KoMet , for a hybrid NWP-DL workflow that post-processes the rain prediction from South Korea’s NWP models under the supervision of surface-level observations from AWS. Our work engages in the solution of precipitation forecasting. We believe that our work opens the door to develop robust and explainable forecasting services that benefit the broad society. Our benchmark dataset will lower the barriers to entry for those in the DL community to contribute to research in precipitation forecasting.
Acknowledgments and Disclosure of Funding
This work was supported by the Korea Meteorological Administration Research and Development Program "Development of AI techniques for Weather Forecasting" under Grant (KMA2021-00121) and Institute of Information & communications Technology Planning & Evaluation (IITP) grant funded by the Korea government(MSIT) (No.2019-0-00075, Artificial Intelligence Graduate School Program(KAIST)). We thank Jaehoon Oh and Sangmook Kim for discussing the pipelines of precipitation forecasting and Yun Am Seo (Jeju National University) and Min-Gee Hong (National Institute of Meteorological Sciences) for discussing the data pre-processing.
Security.
Weather information is an important factor in a country’s economic and military security. Nevertheless, general users do not have access to up-to-date NWP prediction data, so if access to live NWP data is well protected, the security problem will be sufficiently prevented.
Environment.
While our approach brings more benefits in generalizing precipitation forecasting, it still relies on NWP simulations which require vast amounts of computation, and thus energy consumption. The same is true for training and running deep learning models for post-processing. However, we believe that the positive impact of accurate weather forecasts on energy consumption and planning far outweighs these costs and will lead to more environmentally conscious energy policies.
Limitations.
Although we illustrate the advantages of our benchmark dataset, the bottleneck lies in the accuracy of GDAPS-KIM as well as AWS. We only consider the summer season of South Korea, specifically, July and August. We do not investigate other NWP datasets. We also do not consider the precipitation nowcasting task, with lead times of less than 3 hours, as this would limit the deep model’s window size to less than 3, possibly hindering model performance.
Future Directions.
In future work, we aim to explore more robust training strategies under the data scarcity scenario. Furthermore, thorough investigation is needed for the selection of important variables in Table 1 . In addition, we intend to develop our methods in other seasons such as Spring, Fall, and Winter. Lastly, we plan to explore methods to tackle precipitation nowcasting, using a limited window size.
Data Preprocessing.
We refer to the settings in [ 16 ] . For the transformation of GDAPS-KIM, we only apply Min-Max normalization to each variable. We employ various transformations on the ground-truth AWS targets to mitigate the sparsity of observations. In all of our experiments, we apply linear interpolation between available observations to fill in the non-learning points, using functions provided by the scipy package. This is illustrated in Figure 7 . To tackle the class imbalance issue, we study two sampling strategies in subsection 5.3 . Details of sampling strategies are explained in the following paragraphs.
Under-Sampling for ‘No Rain’ Points.
As the ratio of ‘no rain’ is significantly higher than that of ‘rain’ and ‘heavy rain’, we consider a simple sampling strategy of randomly sampling a subset of ‘no rain‘ points within each sample, at a rate of p . The remaining ‘no rain’ points are discarded during training, i.e., set to nan. We use a sampling ratio of p=0.20 .