# Robust Cross-Validation of Predictive Models Used in Credit Default Risk
## 7 Results and Discussion
### Limitations of the Analysis  
The overall rank of Table A2 in Appendix A could be considered the main contribution of this study because it is the comparison of all validation methods using all the paired tests in all the considered datasets. From the overall rank, it can be stated that there is statistical evidence of the better performance of robust validation methods because the 7 best-ranking positions correspond only to the more robust techniques, and any other robust validator with a rank greater than 8 has less convenient \(k\) parameter values different from 5 and 10. Apart from this qualitative conclusion based on extensive quantitative results, one of the limitations faced in the analysis is that making a pure quantitative assessment of the benefits of a robust validation is not easy or feasible because those benefits depend on many circumstances. For example, the selection error measures, via the normalized Kendall Tau, how many positions each validator has altered the correct order of model performances, but how much the selection of one model instead of another one impacts some final generalization error will depend on the particular models considered, the available data, and the severity of the impact of wrong predictions in each practical application. To provide a simple visualization of the clear division in model selection performance between robust and non-robust validators when there are data limitations, Figure 4 represents the selection error averaged per each rank for the validators in the first 12 ranks of Table A2. The last validators in the ranking that corresponds to the RSCV with \(k=2\) (which is deterministic), the maximally shifting cross-validations, and the in-sample methods are not represented in the graph to keep the visualization in a better scale, as the selection error is up to five times larger than the smallest one. From Figure 4 or Table A2, it can be seen that the variation in the model selection error among the existing validation techniques in ranks 8 to 12, such as stratified CV, bootstrapping, Monte-Carlo CV, leave-one-out CV and even the fuzzy methods tested, is similar in magnitude to the difference found between validators in rank 8 and validators in rank 1 or 2. This means that the improvement in model selection using the best robust validators could be as large as the improvements made when choosing between two different methods from the existing validation methodologies and variants. Then, the benefits of robust model selection are potentially relevant in practice, and at least as relevant as the best improvements found among the methods used as potential validation alternatives. \begin{table}
\begin{tabular}{c c c c c c c c c c c} \hline \hline \multirow{2}{*}{**Validator**} & \multicolumn{2}{c}{**All Datasets**} & \multicolumn{2}{c}{**Taiwan**} & \multicolumn{2}{c}{**German**} & \multicolumn{2}{c}{**Australian**} & \multicolumn{2}{c}{**Lending Club**} \\ \cline{2-11}  & **Error** & **Rank** & **Error** & **Rank** & **Error** & **Rank** & **Error** & **Rank** & **Error** & **Rank** \\ \hline CV:DOB-WOE/10/50 & 0.1676 & 1 & 0.0999 & 1 & 0.1144 & 1 & 0.1211 & 1 & 0.3348 & 1 \\ CV:DOB-WOE/10/10 & 0.1688 & 2 & 0.1006 & 2 & 0.1147 & 1 & 0.1218 & 1 & 0.3379 & 1 \\ CV:kfold/10/1000 & 0.1731 & 3 & 0.1076 & 3 & 0.1154 & 1 & 0.1221 & 2 & 0.3473 & 2 \\ CV:SCV/10/1000 & 0.1732 & 3 & 0.1076 & 3 & 0.1153 & 1 & 0.1220 & 1 & 0.3478 & 2 \\ \hline \hline \end{tabular}
\end{table} 
Table 8: Selection error and ranking of selected validation methods given by the normalized Kendall Tau, averaged over 120 random samples taken from each credit dataset. An overall ranking is calculated using all the experiments made in all credit datasets as the final ranking. In fact, previous studies making comparisons between bootstrapping, leave-one-out, cross-validation with different k-fold parameters, and other techniques generally reach qualitative conclusions [5,7,11,12,15,17,24,27] without being able to quantify the potential benefits.