## 4 Experimental Results
### Feature Importance  
We conducted additional experiments to identify and compare important variables of the ML and DL models. The relative importance of 210 independent variables for the four ML models (RF, GBM, XGBoost, LightGBM) was calculated using the permutation importance module of the eil5 package. Variables with positive feature importance values were considered important variables because they significantly influence the predictive model. Figure 6 shows the top 10 important variables based on the average of the feature importance of the four ML models. The most important variable in the ML model was \"Days of application for loan execution in the last year\". Meanwhile, as mentioned in Section 3.2, the feature importance of the DL model was calculated by adding a CancelOut layer after the input layer of the XmCNN model. To calculate the final feature importance of the DL model, the XmCNN model to which the CancelOut layer was added was independently trained 10 times, and the weight values of the CancelOut layer were averaged. Figure 7 shows the top 10 important variables based on the feature importance of the DL model. In contrast ML models, the most important variable in the DL model was the \"Percentage of one-time loan execution in the last year\". The important variable sets of the ML model and DL model were different. As shown in Figure 8, the intersection of ML and DL important features increased with increasing top N percent of feature importance. In the top 20% of the feature importance criteria, 9 out of 42 variables (see Table 10) were considered important simultaneously in ML and DL, which corresponded to 4.28% of the total featu \begin{table}
\begin{tabular}{l c c c c c c} \hline \hline
**Ensemble Model** & **FPR** & **FNR** & **Recall** & **Precision** & **Accuracy** & **F1-Score** \\ \hline RF, SVM, GBM, XGBoost, LightGBM & 0.0831 & 0.3631 & 0.7769 & 0.7810 & 0.8624 & 0.7789 \\ SVM, GBM, XGBoost, LightGBM & 0.0739 & 0.3714 & 0.7773 & 0.7921 & 0.8682 & 0.7843 \\ SVM, GBM, LightGBM & 0.0584 & 0.3856 & 0.7780 & 0.8138 & 0.8779 & 0.7938 \\ SVM, GBM, XGBoost & 0.1073 & 0.2622 & 0.8152 & 0.7790 & 0.8625 & 0.7945 \\ RF, SVM, GBM & 0.0840 & 0.3168 & 0.7996 & 0.7928 & 0.8707 & 0.7961 \\ \hline \(\text{CNN}_{\text{S(3)}},\text{CNN}_{\text{S(4)}},\text{CNN}_{\text{S(5)}}\) & 0.0537 & 0.1992 & 0.8735 & 0.8671 & 0.9179 & 0.8703 \\ \(\text{CNN}_{\text{S(5)}},\text{XmCNN}\) & 0.0519 & 0.2031 & 0.8725 & 0.8693 & 0.9187 & 0.8709 \\ \(\text{CNN}_{\text{S(3)}},\text{CNN}_{\text{S(5)}},\text{XmCNN}\) & 0.0527 & 0.1992 & 0.8741 & 0.8689 & 0.9188 & 0.8714 \\ \(\text{DNN}_{\text{S(3)}},\text{CNN}_{\text{S(3)}},\text{CNN}_{\text{S(4)}}, \text{CNN}_{\text{S(5)}},\text{XmCNN}\) & 0.0525 & 0.1999 & 0.8738 & 0.8690 & 0.9188 & 0.8714 \\ \hline \(\text{CNN}_{\text{S(3)}},\text{CNN}_{\text{S(4)}},\text{CNN}_{\text{S(5)}},\text{XmCNN}\) & 0.0517 & 0.1992 & 0.8745 & 0.8704 & 0.9196 & 0.8724 \\ \hline \hline \end{tabular}
\end{table} 
Table 9: Performance evaluation of ensemble models. the important features of ML and DL increased rapidly, but the intersection of the upper intervals according to the important features was relatively small. In other words, when ML and DL models were trained, the features they focused on were different. Finally, as shown in Table 11, we investigated the performance of ML and DL models according to the feature selection used. The features corresponding to the top N percent based on feature importance of ML and DL were selected and used for modeling. We experimented with XGBoost, LightGBM, and the proposed XmCNN model. The XGBoost model performed the best when using only the top 60% of features based on feature importance, while the LightGBM model and XmCNN model performed the best when using all features. In particular, the proposed XmCNN model outperformed ML models trained with all features, even if only the top 20% based on feature importance were used. Figure 6: Top 10 important features in machine learning models. Figure 7: Top 10 important features in deep learning model.