RiteeshGrandhi/sciclaimeval-shared-task huggingface dataset:
Balancing for ‘Rain’ Points.
This technique considers the ratio of ‘rain’ and ‘heavy rain’ to ‘no rain’ within each feature map. Specifically, let R be the number of points with ‘rain’ or ‘heavy rain’, and N be the number of ‘no rain’ points. Given the target rate p , we randomly sample ‘no rain’ points to match p=N/R as closely as possible. We also use a sampling ratio of p=0.20 for this method. The specific strategy applied to each sample is as follows:

Case 1. R=0 : The sample is not used for training.

Case 2. (R×p)<N : (N-R×p) ‘no rain’ points are discarded from the sample.

Case 3. (R×p)>=N : The sample is used as is.
Training Settings.
We use the Adam optimizer with the default learning rate of 0.001, and beta parameters of 0.9 and 0.999. We train the models for 20 epochs and select the best epoch based on the highest CSI performance on the validation set. Reported performances are based on the test set, unless otherwise specified. We use a window size of 3 and lead times ranging from 6 to 87 hours. We use two Pres type variables ( T , rh_liq ) at three isobaric planes,{500hPa, 700hPa, 850hPa}; and six variables of Unis type ( rain, q2m, rh2m, t2m, tsfc, ps ). These baseline settings are used in all experiments, unless otherwise stated.
Resources.
We run our experiments on NVIDIA GeForce RTX 3090 Ti GPUs. Each training run takes approximately 30 minutes.
G.1 Influential Variables
We conduct an ablation study to investigate the influence of each individual variable. We start with a baseline set of variables, curated under the guidance of weather forecasters in South Korea. This includes all baseline variables used in our other experiments, as well as one additional Unis variable: pbltype . We then measure the performance of our baseline U-Net model when adding or subtracting individual variables used for training. We consider the u , v and u10m , v10m variables jointly, as they both pertain to wind. For Pres variables, we use values corresponding to three isobraic surfaces at {500hPa, 700hPa, 850hPa}. We report our results in Table 4 and find that while some metrics such as accuracy are relatively unaffected, others such as CSI and bias vary depending on the choice of variables.
G.2 Lead Time on Heavy Rain Prediction
Figure 8 illustrates the results of CSI scores for precipitation of \geq 10.0 (mm/h) over lead time. Unlike the trends in rain prediction, interestingly GDAPS-KIM outperforms other baseline DL models. It seems that heavy rain prediction is a more ardous challenge to be addressed.
G.3 Learning Rate
Figure 9 shows the CSI scores according to changes in learning rate. When comparing the performance of ‘rain’ and ‘heavy rain’, we see different trends based on increase in learning rate.
G.4 Learning Curve
We visualize the learning curves of the networks according to changes in window size ( Figure 10 , Figure 11 , Figure 12 ).
G.5 Generalization for Different Times of the Month
We evaluate the deep models trained on our benchmark for different times of the year, ranging from July 2020 to July 2021. Our benchmark dataset covers only summer for 2020 and 2021, and thus it harbors concerns about the generalization of different seasons. While the data of other months is not provided owing the limit of license, it is available for the analysis of CSI distribution according to the lead time as well as month, as a substitute of raw data.

Figure 13 and Figure 14 show the CSI distribution by lead time and month from ConvLSTM and U-Net, respectively. For the lead time, both models has larger CSI values of early lead time than those of latter and the overall curve has inflection points at similar time compared to the curve of rain ratio. On the other hand, the curve of heavy rain CSI is significantly similar with the graph of heavy rain ratio distribution on lead time. For the month, it can be seen that the annual average rain CSI is near 0.3, but the performance is extremely low in winter in both models. This seems to be the case, however, because the amount of rain in winter is close to zero. This phenomenon is more pronounced in heavy rain.
G.6 Training on Other Multiple Categorical Classes
Our framework allows users to specify categorical classes in training as they wish if 0.1 and 10 are included in the threshold list. By referring to the other regions [ 47 , 16 , 56 ] , we conduct an experiments with different categorical classes while the evaluation is verified with the standard threshold for rain and heavy rain used in our framework. Followings are the threshold lists with the number of classes:

V1: [0.1, 2.0, 10.0], 4 classes

V2: [0.1, 5.0, 10.0], 4 classes

V3: [0.1, 0.5, 5.0, 10.0], 5 classes

V4: [0.1, 0.5, 5.0, 10.0, 30.0], 6 classes

Table 5 shows the performance according to the changes of the threshold list. As the class of middle rain is newly created, it can be seen that the CSI can be changed. Furthermore, when the number of classes in the middle rain is further increased, it can be seen that the csi of the heavy rain increases significantly with some degradation in rain CSI.
G.7 Pointwise Architecture without the Use of Spatial Information
We conduct an experiment with methods that do not use spatial information for NWP correction. While our benchmark architectures include both spatial encoder and temporal encoder, it is important to check the needs of spatial neural processing. Table 6 shows the results for various lead times from 6 to 87 hours, according to the presence of spatial kernel operator in neural networks. Here, we design pointwise architecture having no spatial kernel. Pointwise architecture with LSTM indicates the modified version of ConvLSTM whose spatial encoder is replaced with pointwise architecture. For the NWP correction, as Table 6 shows, using only pointwise operator lead to worse optima than our benchmark models. In other words, it can be seen that the methodology considering the spatial information is more effective.
Figure 1: Overview of challenges for KoMet . The central part shows our main workflow; as GDAPS-KIM simulations are released timely, a deep neural network aims to post-process the model output for rain forecast. Challenges can be categorized into five folds: (1) influential variable selection that selects the key features for data pre-processing, (2) sensitivity to hyperparameter settings, (3) robust architectures, (4) non-learning points on the ground-truth labels, and (5) class-imbalance issues where the number of precipitation areas are fewer than the number of other cases.; Overview of challenges for KoMet . The central part shows our main workflow; as GDAPS-KIM simulations are released timely, a deep neural network aims to post-process the model output for rain forecast. Challenges can be categorized into five folds: (1) influential variable selection that selects the key features for data pre-processing, (2) sensitivity to hyperparameter settings, (3) robust architectures, (4) non-learning points on the ground-truth labels, and (5) class-imbalance issues where the number of precipitation areas are fewer than the number of other cases.
(a) Precipitation Ratio by Date (2020); (b) Precipitation Ratio by Date (2021); (c) Precipitation Ratio by Hour (2020); (d) Precipitation Ratio by Hour (2021); Temporal distribution of the rain ratio among all AWS observations . ‘Rain’ refers to hourly precipitation between 0.1mm and 10mm, while ‘heavy rain’ refers to that above 10mm. In (c), (d), the left axis refers to the ratio of ‘rain’ and the right axis is the ratio of ‘heavy rain’, in percentages.
(a) No Rain Ratio (2020); (b) Rain Ratio (2020); (c) Heavy Rain Ratio (2020); (d) No Rain Ratio (2021); (e) Rain Ratio (2021); (f) Heavy Rain Ratio (2021); Spatial distribution of precipitation from AWS observations in South Korea. ‘Rain’ refers to hourly precipitation between 0.1mm and 10mm, while ‘heavy rain’ refers to that above 10mm.
Figure 4: CSI scores of GDAPS-KIM and baseline models for a class ‘rain’. Scores are given for predictions corresponding to specific lead times, ranging from 6 to 20 hours.; CSI scores of GDAPS-KIM and baseline models for a class ‘rain’. Scores are given for predictions corresponding to specific lead times, ranging from 6 to 20 hours.
(a) Acc; (b) CSI; (c) Bias; Performance of the models according to the changes of sampling strategies. Vanilla indicates baseline training strategy without resampling of AWS targets. The metrics are averaged over 10 different random seeds and error bars indicate the 95% confidence intervals.
(a) U-Net, window size; (b) ConvLSTM, window size; (c) MetNet, window size; (d) U-Net, weight decay; (e) ConvLSTM, weight decay; (f) MetNet, weight decay; (a), (b), (c): CSI performance according to changes in window size in hours; (d), (e), (f): CSI performance according to changes in weight decay.
(a) Original; (b) Interpolated; Visualization of AWS observations of precipitation before and after interpolation, on April 23, 2020 06:00 UTC. The unit of precipitation is mm/h.
(a) Heavy Rain; CSI scores of GDAPS-KIM and baseline models for binary heavy rain classification. Scores are given for predictions corresponding to specific lead times, ranging from 6 to 20 hours.
(a) U-Net; (b) ConvLSTM; (c) MetNet; CSI performance according to changes in learning rate.
(a) U-Net, window size=1; (b) U-Net, window size=3; (c) U-Net, window size=6; U-Net: CSI learning curve according to window size.
(a) Conv-LSTM, window size=1; (b) Conv-LSTM, window size=3; (c) Conv-LSTM, window size=6; Conv-LSTM: CSI learning curve according to window size.
(a) MetNet, window size=1; (b) MetNet, window size=3; (c) MetNet, window size=6; MetNet: CSI learning curve according to window size.
(a) Rain CSI by lead time; (b) Heavy rain CSI by lead time; (c) Rain CSI by month; (d) Heavy rain CSI by month; CSI distribution of ConvLSTM outputs from July 2020 to July 2021. Orange line indicates the rain CSI of the model and the blue dot line is the distribution of observed rain ratio from AWS.
(a) Rain CSI by lead time; (b) Heavy rain CSI by lead time; (c) Rain CSI by month; (d) Heavy rain CSI by month; CSI distribution of U-Net outputs from July 2020 to July 2021. Orange line indicates the rain CSI of the model and the blue dot line is the distribution of observed rain ratio from AWS.
List of variables contained in the benchmark dataset. Pres data contains atmospheric features at different pressure levels, and Unis data contains some other features as well as those in Pres data but does not at certain pressure levels. † indicates the variable with integer type.
Statistics in the KoMet benchmark.
Evaluation metrics of KoMet and baseline models for precipitation while 12 variables are utilized for the training. Best performances are marked in bold.
Ablation study on the binary classification performance of our baseline U-Net model for ‘rain’ and ‘heavy rain’ depending on the selection of variables. Each row represents an addition (+) or a subtraction (-) of a single variable from our curated set.
Evaluation metrics of KoMet with the MetNet for precipitation while 12 variables are utilized for the training, according to the changes of the number of categorical classes during training.
Evaluation metrics of KoMet and pointwise architectures for precipitation while 12 variables are utilized for the training.