## 3 Experiments and Analysis  
### Model Evaluation Criteria  
The efficacy of a model is validated by evaluation organizations using specific model evaluation criteria. When faced with the dual challenges of imbalanced sample categories and the complexity of building a capability evaluation model, it's crucial to choose appropriate metrics to gauge the model's classification efficacy. While loss functions measure how well a classification model anticipates the future, evaluation metrics serve a similar purpose. However, the key difference is that evaluation metrics are not integrated into the model's training phase. Accuracy stands as the most frequently employed metric in traditional classification algorithms. Yet, when dealing with imbalanced data, the limited sample size of certain classes might have minimal impact on accuracy. It means high accuracy might still be observed even if all low-capability samples are misclassified. This contradicts the pragmatic needs of capability evaluation, which emphasizes the importance of low-risk samples [31]. To effectively assess the classification prowess of the proposed model, this study leverages three popular metrics: Accuracy, F1 score, and AUC. The computation of these metrics is anchored on the confusion matrix presented in Table 1.  
Table 1 illustrates the following metrics:  
The terms True Positive (TP), False Positive (FP), False Negative (FN), and True Negative (TN) represent the number of correctly and incorrectly classified positive and negative samples, respectively. The formula for Accuracy, which depicts the ratio of correctly classified samples to the total number of samples, is given by  
\[Accuracy=\frac{TP+TN}{TP+FP+TN+FN} \tag{18}\]  
The F1 score combines both precision and recall. To achieve a high F1 score, the model must accurately and comprehensively classify positive samples. It's defined as  
\[F_{1}=2*\frac{Precision*Recall}{Precision+Recall} \tag{19}\]  
_Precision_ and _Recall_ are described by:  
\[Precision=\frac{TP}{TP+FP} \tag{20}\]  
\[Recall=\frac{TP}{TP+FN} \tag{21}\]  
The _AUC_ (_Area Under the Curve_) is defined as the area beneath the _ROC_ (_Receiver Operating Characteristic_) curve. The ROC curve's horizontal axis represents the _False Positive Rate_ (_FPRate_), and its vertical axis represents the _True Positive Rate_ (_TPRate_). These rates are defined as [28]:  
\[FPRate=\frac{FP}{FP+TN} \tag{22}\]  
\[TPRate=\frac{TP}{TP+FN} \tag{23}\]  
From Equations (22) and (23), we discern that the _TPRate_ signifies the ratio of positive samples accurately classified, whereas the _FPRate_ depicts the ratio of negative samples mistakenly identified as positive. Thus, the AUC considers the model's evaluation efficacy for different sample classes. It remains effective even when certain classes have very few samples, a situation where Accuracy and F1 score may falter. This makes the AUC particularly apt for evaluating imbalances in sample counts, such as when assessing the capabilities of weather disaster response agencies. \begin{table}
\begin{tabular}{c c c} \hline \hline
**Prediction/Real** & **1** & **0** \\ \hline
1 & True (_TP_) & False Negative (_FN_) \\
0 & False Positive (_FP_) & True Negative (_TN_) \\ \hline \hline \end{tabular}
\end{table} 
Table 1: Confusion matrix. Figure 1 presents a scenario where the _TPRate_ exceeds the _FPRate_. In this case, the AUC value surpasses 0.5. This implies that the model is more likely to correctly classify positive samples than to misclassify negative samples. It suggests that the model can reliably differentiate between positive and negative samples. Figure 2 illustrates the scenario where the _TPRate_ equals the _FPRate_, and the value of AUC stands at 0.5. In this situation, the proportion of positive samples accurately classified as positive by the ability evaluation model equals the proportion of negative samples falsely classified as positive.