# Icing Detection over East Asia from Geostationary Satellite Data Using Machine Learning Approaches
## 4 Results and Discussion
### Model Performance  
The RF- and MLL-based icing detection models were developed using PIREP-based reference data for COMS and Himawari-8. Since the existing FIT and KMA models require level 2 products to generate icing masks, they were tested only with COMS data. Figure 4 shows the accuracy assessment results for the two theoretical (i.e., existing) models and the four machine learning models. When the COMS data were used, the RF model yielded the highest POD (\(\sim\)87.1%), OA (\(\sim\)79.5%), and TSS (\(\sim\)62.9%) among the four tested models, followed by the MLL model. However, the RF model resulted in a relatively high POFD (\(\sim\)24.3%) compared to the KMA model (7.6%). The standard errors of the accuracy metrics (i.e., POD, OA, and TSS) of the RF model were slightly higher than those of the MLL model, which implies that MLL showed more consistent performance than RF for icing detection. This might be due to the small number of samples for RF, resulting in slight overfitting for some folds. On the other hand, the rule-based models (i.e., FIT and KMA) consistently performed worse than the RF and MLL models, resulting in low standard errors (average \(\sim\)3.5% indicating stable performance by fold) with OAs of 67.1% and 67.5%, and TSSs of 27.1% and 10.0%, respectively. Overall, the RF icing detection model resulted in the best performance among the four models for COMS. When the Himawari-8 data were used, the results were similar to the COMS-based models; the RF model produced better performance than the MLL model. However, the accuracy difference between the RF and MLL models for Himawari-8 was smaller than that for COMS. The standard errors of the performance metrics for the Himawari-8 models were much higher than those for COMS-based models (Figure 4). For example, the standard errors of the mean TSS for the machine-learning-based COMS models were 6.5% and 5.1% (i.e., for the RF and MLL models, respectively), while those for the Himawari-8 models were 14.5% and 9.6%. Such differences in the standard errors are possibly due to the much smaller number of samples for Himawari-8 than COMS. This implies that the RF and MLL models for Himawari-8 might not be robust, often resulting in varied performance for the samples even though they produced high accuracy. It is not appropriate to directly compare these results to those from other studies as different data were used. However, the accuracy metrics from the proposed approaches are higher than those from the literature. For example, [PERSON] et al. [22] evaluated the KMA model using COMS-PIREP data, resulting in a POD of 57.9%, a POFD of 27.5%, an OA of 72.46%, and a TSS of 30.4%. Similarly, the FIT model was evaluated using GOES-PIREP data, yielding a POD of 62%, a POFD of 39%, an OA of 58%, and a TSS of 4% in the daytime [15]. PIREPs were used as reference data in both studies. Figures 5 and 6 depict the distribution of the potential icing areas produced from the six models as examples for selected icing and non-icing PIREP cases. Among the six models, the FIT model identified more low cloud areas as potential icing regions compared to the other models. This pattern often occurred when the phase of low clouds was SLW for other cases (not shown). However, such a pattern was not found where the cloud phase was neither water nor mixed. Among the empirical machine learning models, the Himawari-8-based models classified less areas as potential icing clouds than the COMS based models when the same algorithm was used. This is possibly because a smaller number of training samples were available for the Himawari-8 models (Table 3), which often results in overfitting [62, 63]. In particular, the ranges of the input values of the icing samples for the Himawari-8-based models were generally smaller than those for COMS-based models. The RF models produced a more scattered icing pattern in a number of small patches, while the MLL models showed more clustered icing regions. These patterns were consistently found in other cases (not shown).