## 4 Discussion
Under such circumstances, advanced metrics such the ROC curve and AUC should be used to provide a wider understanding of the model's performance over the training and test sets, i.e., during modeling and validation. Figure 4 shows the ROC curves for the selected EXP2-LR model considering the NEG class label, separately for the training and test sets. The calculated AUC values for the training and test sets were 0.974213 and 0.979435, respectively. A high AUC value means that the trained model returns a great result during the validation phase. It also means that the selected model is applicable. The trends of the two NEG curves were similar. Both curves quickly reached the top, almost fully sensitive, when the 1-specificity value was close to 0.1; 1-specificity is also known as the FP rate and represents the probability of a Type I error or false alarm event. In addition, the NEG curves for the test set were steeper than those for the training set. Figure 5 shows the ROC curves and AUC values for the EXP2-LR model considering the POS class label, separately for the training and test sets. Compared to the curves for the NEG class label, the curves for the POS class label rose more quickly at the beginning until 1-specificity was reached at a value of 0.2. The POS curves go downward when the 1-specificity value was between 0.02 and 0.5. After that, the curves tended to flatten. \begin{table}
\begin{tabular}{l c c c c} \hline \hline
**Source** & \(\beta\) & \(\beta_{opt}\) & \(\beta_{opt,payoff}\) & \(X_{c}\) \\ \hline [PERSON] and [PERSON] (2016) & 0.0000 & 0.0357 & 0.0001 & \(-\)2.8854 \\ [PERSON], [PERSON], and [PERSON] (2016) & 0.0000 & 0.0324 & 0.0001 & \(-\)2.7828 \\ [PERSON], et al. (2016) & 0.0000 & 0.0265 & 0.0001 & \(-\)2.7267 \\ [PERSON] and [PERSON] (2017) & 0.0000 & 0.0359 & 0.0001 & \(-\)2.7597 \\ [PERSON], et al. (2019)—Under-sampled & 0.0000 & 0.0550 & 0.0002 & \(-\)2.8358 \\ [PERSON], [PERSON], et al. (2019)—Actual Data & 0.0013 & 0.0375 & 0.0013 & \(-\)2.1683 \\ [PERSON], and [PERSON] (2019) & 0.0000 & 0.0279 & 0.0001 & \(-\)2.7469 \\ EXP2-LR (in this paper) & 0.0013 & 0.0044 & 0.0007 & \(-\)1.8992 \\ \hline \hline \end{tabular}
\end{table} 
Table 8: Comparison of the proposed approach to methods reported in previous studies based on \(\beta\) values and \(X_{c}\) over the test set. Figure 4: Receiver operating characteristic (ROC) curves and area under the curve (AUC) values of EXP2-LR for the NEG class label: (**a**) training set and (**b**) test set. Based on the above results, it was proved that the selected EXP2-LR model was outperformed on the APS data than other machine learning models created from the previous articles. Further, under the data conditions proposed as EXP2, as a candidate model, k-NN also can generate a higher ACC (99.44%) and F-measure (63.64%), and a bit weaker MCC (65.91%) than previous models (Tables 5 and 7). When the existence of missing data is a kind of reasonable behavior, our proposed sorted missing percentage method can achieve better modeling results. According to the original structure of data that is not changed much, these outperformed results can be explained that our proposed method was with stability and reliability. Our proposed method also has limitations in practice. First, our proposed method cannot meet the needs of automation. Typically, either single or multiple imputation methods can quickly calculate and fill up the missing data through existed software or programming. Considering the variety of data sources and the respect to the original data structure, our proposed method must first sort the attributes by missing percentage and must also check whether the distributions of the training and the test sets are consistent. These two manipulations are required, the manual and subjective checks. The second limitation is the decision of the cut-off point, and it relies on the advice of experienced experts.