# Robust Cross-Validation of Predictive Models Used in Credit Default Risk
## 7 Results and Discussion
### Model Selection  
To make a one-dimensional ranking for the performance of the validators in a given dataset, first, each validator has been ordered in a list according to its average selection error on the 150 experiments from lower to higher values. Then, a minimum rank value has been calculated for each validator, rejecting the null hypothesis that its selection error is equal or lower than the selection error of some previous validator using a one-sided Wilcoxon paired test: if the Wilcoxon test is positive compared to the previous validator in the list, the rank is increased by one. Otherwise, the one-sided Wilcoxon test is evaluated against the following previous validator in the list until a positive paired test is found, supporting the alternative hypothesis that the selection error of the validator is greater at a 95% confidence level. This is considered a minimum rank value for the validator. Then, the rank of the validator is determined as the maximum between the rank of the previous validator in the ordered list and the minimum rank found for it. The resulting ranking in each dataset is shown in Table 12 of Appendix A. To make a validator performance ranking based on all the credit datasets, an overall rank has been calculated in the same way, performing a single Wilcoxon test in all the experiments of all the datasets, as shown in Table 12 of Appendix A. The validators have been ordered according to this overall rank from best to worst performance for model selection. In this way, it can be seen in Table 12 that only 20 different ranks were found in a list with 65 different validation techniques. This means that many times the Kendall Tau distance is not different enough to consider one validator better than the next one. Something similar happens with the results of the individual credit datasets shown in the same table. Table 14 shows that there are more draws in the ranking of the validators in the individual datasets, different from Taiwan, because the samples and populations are smaller, probably with less complex default patterns, which prevents them from easily discriminating among the performances of the validators. This is also the reason why the selection error is larger in these datasets. Despite some differences found in the performance of the validation methodologies throughout the credit datasets, it was found that the robust cross-validation techniques are usually in the best positions in the rankings. In particular, the enhancement made to the DOB-SCV using the weight of evidence to calculate the distances for categorical variables presents better results than the original method for the same value of the k-fold parameter. In fact, looking at the overall ranking, almost all variants of the robust cross-validation algorithms are in the top positions for any value of the k-fold parameter, making this type of validator very convenient in situations where there is no clue about what \(k\) value to choose. It is remarkable that the robust methods present better performance than the rest of the validation techniques, even for high values of the \(k\) parameter, supporting the initial hypothesis that homogeneous partitioning makes more stable test sets, allowing the use of more data for training without harming the results too much. Of course, it can be seen from Table 14 that there is still an impact of the \(k\) parameter, and for this credit default use case, \(k=10\) and \(5\) were found to be the more suitable values, aligned with the findings of other research in different contexts. Roughly speaking, following the more robust cross-validation variants, there is a large group of validators in positions eight and nine of the overall rank, mainly with traditional and fuzzy cross-validation techniques. There are also some robust methods in this group with \(k=2\), as the negative impact of this parameter is too large to be mitigated by the optimal partitioning.