# Robust Cross-Validation of Predictive Models Used in Credit Default Risk
## 7 Results and Discussion
### Best Model Selection  
Apart from analyzing the capability of the validators to rank and discriminate among a wide range of credit default models with different predictive performances in terms of AUR, it is also interesting to check if the same validators have the ability to select the best ones. It could happen that a validator that differentiates well among intermediate models is not able to choose the best one or the top ones. To check this, the number of times the model selected as the best one was in fact the best model in the unseen population has been counted for each validation technique, as shown in Table 6. In case of any draw in the number of successes, the number of second and third-best models correctly selected was used to break the tie and rank the validators. Again, it is confirmed that the top of the rank is led by robust cross-validation methods, although the representative splitting cross-validation performs surprisingly much better than expected according to the overall selection error shown in Table A2 of Appendix A. It is also surprising how much the stratified bootstrapping has improved in this rank compared with the results shown in Table A2. Then, to make a more robust assessment about which validators perform better when selecting good predictors, Table 7 shows the total number of times the model selected as the best one by each validator was in fact among the top five performances in the unseen population for all experiments in all datasets. To break any tie in the results, the validator with more second or third-best options in the true top five was preferred. As shown in Table 7, the robust cross-validation methods are still leading the ranking as the most reliable in general for model selection purposes when the validated models are good predictors. In addition, Monte Carlo cross-validation seems to be a trustworthy alternative when using 20% of the data for testing. Something important to highlight is that distribution optimally balanced stratified cross-validation, enhanced with the weight of evidence for categorical variables, is usually among the best validators, even for different values of the k-fold parameter. This is an \begin{table}
\begin{tabular}{c c c c c c} \hline \hline
**Validator** & **Num.Best** & **Validator** & **Num.Best** & **Validator** & **Num.Best** \\ \hline CV:RSCV/2 & 96 & CV:SCV/5/50 & 77 & CV:DB-SCV2/10/50 & 71 \\ CV:DOB-WOE/10/50 & 91 & CV:DOB-WOE/20/50 & 77 & CV:DB-SCV2/15/50 & 71 \\ CV:DOB-SCV/10/50 & 81 & CV:DB-SCV1/2/50 & 77 & CV:SCV/10/50 & 71 \\ CV:DOB-SCV/2/50 & 81 & CV:DB-SCV2/20/50 & 77 & CV:kfold/15/50 & 70 \\ CV:DOB-WOE/15/50 & 80 & HO:MCCV/0.5/1000 & 75 & CV:RSCV/10/50 & 70 \\ BTS:sboot/0/1000 & 80 & BTS:b632+/0/1000 & 75 & CV:SCV/2/50 & 69 \\ CV:DB-WOE2/10/50 & 80 & CV:DOB-SCV/15/50 & 75 & CV:kfold/20/50 & 69 \\ CV:DOB-WOE/5/50 & 80 & CV:SCV/15/50 & 75 & BTS:boot/0/1000 & 69 \\ CV:DB-WOE2/2/50 & 80 & CV:DB-SCV1/15/50 & 75 & CV:RSCV/20/50 & 68 \\ CV:RSCV/15/50 & 80 & CV:DB-WOE2/5/50 & 75 & BT5:b632/0/1000 & 68 \\ HO:MCCV/0.2/1000 & 79 & CV:DOB-SCV/5/50 & 74 & CV:SCV/20/50 & 68 \\ HO:SMCCV/0.1/1000 & 79 & CV:kfold/10/50 & 73 & HO:MCCV/0.1/1000 & 66 \\ CV:DB-WOE2/20/50 & 79 & CV:kfold/2/50 & 73 & CV:MS-SCV/15/50 & 64 \\ BTS:sb632/0/1000 & 79 & BTS:sb632+/0/1000 & 73 & CV:MS-SCV/10/50 & 62 \\ HO:SMCCV/0.5/1000 & 78 & CV:DB-SCV2/5/50 & 72 & CV:MS-SCV/5/50 & 61 \\ CV:RSCV/5/50 & 78 & HO:SMCCV/0.2/1000 & 72 & CV:LOOCV & 59 \\ CV:DOB-SCV/20/50 & 78 & CV:DB-SCV2/2/50 & 72 & CV:MS-SCV/20/50 & 56 \\ CV:DB-SCV1/5/50 & 78 & CV:kfold/5/50 & 72 & CV:MS-SCV/2/50 & 48 \\ CV:DB-WOE2/15/50 & 77 & CV:DB-SCV1/20/50 & 72 & InSample/AIC/0/0 & 4 \\ CV:DB-SCV1/10/50 & 77 & CV:DOB-WOE/2/50 & 72 & InSample/BIC/0/0 & 2 \\ \hline \hline \end{tabular}
\end{table} 
Table 6: Number of times the best model was correctly selected by each validator, summing up the results of all experiments in all credit datasets used. important advantage because the selection of the \(k\) parameter is usually subject to expert judgement, and it is not clear when to choose one or another value.