## 4 Experimental Results  
In this section, we show our model performance results using known skill score metrics composed of different combinations between true positive (TP), false negative (FN), true negative (TN) and false positive (FP) ratios. The confusion matrix result is shown in Figure 8, where our model gains 80% and 85.7% prediction for TP and TN, respectively.  
The metrics we used were:  
\[\text{Precision}=\frac{\text{TP}}{\text{TP}+\text{FP}} \tag{3}\]  
\[\text{Recall}=\frac{\text{TP}}{\text{TP}+\text{FN}} \tag{4}\]  
\[\text{HSS}=\frac{2\cdot\left[\left(\text{TP}\times\text{TN}\right)-\left( \text{FN}\times\text{FP}\right)\right]}{\left(\text{TP}+\text{FN}\right)\cdot \left(\text{FN}+\text{TN}\right)+\left(\text{TP}+\text{FN}\right)\cdot\left( \text{FP}+\text{TN}\right)} \tag{5}\]  
Figure 8: The confusion matrix for the SVM model results extracted from the training set (**left**), and the test set (**right**).  
\[\text{Accuracy}=\frac{\text{TP}+\text{TN}}{\text{P}+\text{N}} \tag{6}\]  
\[\text{TSS}=\frac{\text{TP}}{\text{TP}+\text{FN}}-\frac{\text{FP}}{\text{FP}+ \text{TN}} \tag{7}\]  
where precision is the fraction of relevant instances among the retrieved instances, the recall is the fraction of relevant instances that were retrieved, and the accuracy is the proportion of correct predictions among the total number of cases examined. The Heidke skill score (HSS) evaluates the fractional improvement of the prediction accuracy relative to some set of controls or reference predictions. It is normalized by the total range of possible improvement over the standard (which basically means it can be compared with different datasets). The range of the HSS is defined as: HSS = 1 is a perfect prediction; HSS = 0 shows no skill. If HSS < 0, the prediction is worse than the reference prediction. The true skill statistics (TSS) compares both probabilities of the true prediction and the false prediction. The range of the TSS is between \(-1\) and \(+1\), where value 0 means that the algorithm has no ability to predict. High positive values indicate that the algorithm performs well, and negative values indicate contradictory behavior, suggesting that the labels should be reversed. The following (Table 1) are the results of the different skill score metrics we used for our best model:  
Another method to evaluate the results of the different models during the optimization process is the receiver operating characteristic (ROC) curve, which is a graphical plot that illustrates the diagnostic ability of a binary classifier system as its hyperparameters are varied. The ROC curve is created by plotting the true positive rate (TPR) against the false positive rate (FPR) at various hyperparameter settings. The ROC analysis provides tools to select possibly optimal models and to discard suboptimal ones independent of the cost context or the class distribution. Below is the ROC curve for the best model obtained during the hyperparameter optimization process (Figure 9).  
\begin{table}
\begin{tabular}{c c c c c} \hline \hline  & **Precision** & **Recall** & **HSS** & **Accuracy** & **TSS** \\ \hline Training Set & 0.76 & 0.609 & 0.419 & 0.7095 & 0.419 \\ (n = 84) & 0.85 & 0.8 & 0.657 & 0.8285 & 0.657 \\ \hline \hline \end{tabular}
\end{table}
Table 1: Skill score metrics.  
Figure 9: ROC curve of the models we obtained during the hyperparameter optimization process.