# Robust Cross-Validation of Predictive Models Used in Credit Default Risk
## 6 Experimental Framework
### Validator Performance Assessment and Comparison  
To statistically determine whether a particular validator can be considered better than others at discriminating between good and bad models, the validation algorithms will beconsidered as if they were classification algorithms whose task is to order a set of predictive models in a ranking according to their estimated performance.  
Then, it is necessary to assess how good the ranking provided by a particular validator is compared to the true model ranking that can be built based on the model performance in the population not used to train the models.  
To achieve this, the Kendall Tao ranking distance can be used, which counts the number of discrepancies between two ordered lists or rankings. This distance metric can be defined as the number of discordant pairs between two ordered lists \(L_{1}\) and \(L_{2}\), that is, the number of pairs in which the order given by \(L_{1}\) to the two elements of each pair is different from the order given by \(L_{2}\) to the same elements.  
Let \(R_{1}(i)\) be the rank of element \(i\) in list \(L_{1}\) and \(R_{2}(i)\) the rank of element \(i\) in list \(L_{2}\); then the Kendall Tau distance \(KT(L_{1},L_{2})\) between the two lists can be calculated using the following expression:  
\[KT(L_{1},L_{2})=\Bigl{|}\Bigl{\{}(i,j):i<j,\Bigl{[}R_{1}(i)<R_{1}(j)\ \bigwedge\ R_{2}(i)>R_{2}(j)\Bigr{]}\ \bigvee\Bigl{[}R_{1}(i)<R_{1}(j)\ \bigwedge\ R_{2}(i)>R_{2}(j)\Bigr{]}\Bigr{\}}\bigr{|}\]  
The distance is zero in cases where the two lists have the same order, while the maximum distance is achieved when one list has the reverse ordering of the other. This maximum value is \(m(m-1)/2\), where m is the number of elements of the lists. Then, a normalized Kendall Tau distance in the interval [0, 1] can be obtained by dividing \(KT\) by \(m(m-1)/2\).  
The normalized Kendall Tau distance has been used as a measure of the error that each validator makes when ranking the models by their estimated AUR in comparison with the ranking based on the true AUR of the models. From now on, this metric will be referred to as the \"selection error\" of the validation algorithm and will be used to assess the validator's performance for model selection purposes.  
This approach allows the assessment of each validator's performance using a set of models with a wide range of different performances, since in practical applications, there will be situations where the models to be selected will perform well, and others where the AUR of the models will be lower.  
In this study, a validator will be considered worse than others for a specific credit default dataset if its selection error is statistically larger when measured on a sufficient number of samples taken randomly from the dataset.  
One may think about comparing the selection error of each validator averaged over many samples, but there is no easy way to assess the variance of the selection error to make a proper comparison. Then, the selection error of all the validators needs to be compared in each of the experimental samples one by one, which allows to perform a paired test of the validator performances. Therefore, validation results will differ due to the particular validation method used, avoiding any variability coming from the sample randomness.  
To this end, the non-parametric paired Wilcoxon one-sided rank test was used to compare the selection errors with a confidence level of 95% using the R function wilcox.test() over 150 experiments performed in each of the default credit datasets considered. In addition, an overall paired comparison using all the 600 samples was performed, applying a single Wilcoxon test to all the samples of all datasets.