# Robust Cross-Validation of Predictive Models Used in Credit Default Risk
## 7 Results and Discussion
### Result Stability  
Robust cross-validation methods achieve better results than other techniques in exchange for more computational requirements. This is due to the need to calculate the nearest neighbors of the observations being partitioned, which extends the validation execution time. Then, more stable results do not necessarily mean any computational saving, but it is possible to measure the stability of the cross-validation estimations in terms of the number of repetitions needed before reaching a final estimate within a certain tolerance level. Just to confirm this expected behavior, Figure 3 represents the average number of repetitions, after which all changes in the estimated AUR are below 0.001 for each validation technique when validating the model with the best averaged true AUR in each dataset. The validators in Figure 3 on the horizontal axis are in the same order as in Table 2 to visualize that there is a clear trend and therefore a relationship between the validator performance and the stability of its estimations: more stable results correlate with lower selection errors. The main exceptions to this rule are the two fuzzy methods (FNkfold and FUkfold), which need more repetitions than expected by the trend in all the datasets due to their fuzzy nature and therefore more volatile results. Figure 3 shows that the distribution optimally balanced stratified cross-validation gives the more stable estimations of the AUR on average. This is coherent with the performance shown by this type of validation method, which is usu \begin{table}
\begin{tabular}{l c c c c c} \hline \hline
**Validator** & **Rank 1–5** & **Validator** & **Rank 1–5** & **Validator** & **Rank 1–5** \\ \hline CV:DOB-WOE/10/50 & 449 & CV:kfold/15/50 & 433 & BTS:sb632/0/1000 & 423 \\ CV:DOB-WOE/5/50 & 449 & BTS:b632/0/1000 & 433 & BTS:boot/0/1000 & 423 \\ HO:MCCV/0.2/1000 & 445 & CV:DOB-SCV/15/50 & 432 & CV:DB-SCV1/2/50 & 422 \\ CV:DB-WOE2/5/50 & 445 & HO:SMCCV/0.1/1000 & 431 & CV:DOB-SCV/2/50 & 421 \\ CV:DOB-SCV/10/50 & 444 & CV:SCV/15/50 & 431 & CV:DOB-WOE/2/50 & 420 \\ CV:DOB-WOE/15/50 & 443 & CV:RSCV/15/50 & 430 & CV:RSCV/20/50 & 420 \\ CV:DB-SCV1/5/50 & 442 & CV:DB-WOE2/20/50 & 430 & HO:MCCV/0.5/1000 & 416 \\ CV:RSCV/5/50 & 441 & CV:DB-SCV2/20/50 & 430 & CV:DB-SCV2/2/50 & 415 \\ CV:DOB-SCV/20/50 & 439 & CV:kfold/10/50 & 430 & CV:SCV/2/50 & 412 \\ CV:DB-WOE2/15/50 & 438 & CV:DB-SCV2/5/50 & 430 & CV:kfold/2/50 & 409 \\ CV:DB-WOE2/10/50 & 437 & HO:SMCCV/0.2/1000 & 430 & CV:RSCV/2 & 391 \\ CV:kfold/5/50 & 437 & CV:DB-WOE2/2/50 & 429 & CV:LOOCV & 391 \\ CV:DOB-WOE/20/50 & 436 & BTS:sb632+/0/1000 & 428 & HO:MCCV/0.1/1000 & 357 \\ BTS:sboot/0/1000 & 435 & CV:DB-SCV1/20/50 & 428 & CV:MS-SCV/15/50 & 351 \\ CV:DB-SCV1/10/50 & 435 & CV:kfold/20/50 & 428 & CV:MS-SCV/10/50 & 345 \\ CV:SCV/5/50 & 435 & HO:SMCCV/0.5/1000 & 427 & CV:MS-SCV/20/50 & 337 \\ CV:DB-SCV2/15/50 & 435 & BTS:b632+/0/1000 & 426 & CV:MS-SCV/5/5/50 & 327 \\ CV:DOB-SCV/5/50 & 434 & CV:SCV/20/50 & 426 & CV:MS-SCV/2/50 & 180 \\ CV:DB-SCV2/10/50 & 434 & CV:SCV/10/50 & 425 & InSample/BIC/0/0 & 65 \\ CV:DB-SCV1/15/50 & 433 & CV:RSCV/10/50 & 425 & InSample/AIC/0/0 & 7 \\ \hline \hline \end{tabular}
\end{table} 
Table 7: Number of times the best model selected by each validator was among the true top five models in terms of AUR calculated in the unseen population. The counting sums up the results of all experiments in all credit datasets used. model selection, particularly the enhanced algorithm that uses the weight of evidence for the distance between categorical covariates (see Table A2 in Appendix A). The maximally shifted cross-validation methods need more repetitions than any other cross-validation method tested because their more heterogeneous partitions give place to a larger volatility of the AUR estimations. The representative splitting cross-validation with \(k=2\) is not represented because the DUPLEX algorithm always divides the sample into two halves in the same way, and no repetition can be performed in this case.