# Robust Cross-Validation of Predictive Models Used in Credit Default Risk
## 6 Experimental Framework
### Model Set  
In consumer credit approval, logistic regressions are the market standard for probability of default models. This makes it possible to reach practical, applicable conclusions without the need to test other types of models, such as neural networks, random forest, SVM, or others not used in this business context.  
A set of 63 models was created systematically from different subsets of explanatory variables to diversify the model complexity and the validation results. This allowed us to test the validators for model selection in a wide range of model performances, typically from AURs less than 0.6 to more than 0.8. In addition, as the number of models created was  
\begin{table}
\begin{tabular}{c c c c c} \hline
**Dataset Name** & **Records** & **Features** & **Event Rate** & **Reference** \\ \hline Taiwan & 30,000 & 23 & 22\% & [31] \\ German & 1000 & 20 & 29\% & [32] \\ Australian & 690 & 14 & 44\% & [33] \\ Lending Club 2009 & 5281 & 13 & 14\% & [34] \\ \hline \end{tabular}
\end{table}
Table 1: Public credit datasets used: name of the dataset, number of observations, number of variables, global default rate, and reference.  
quite large, most of the time there were models with similar performance results, which made the model selection more difficult, resulting in different model performance rankings for similar validation algorithms.  
To automate the model set generation, first, an ordered list of all variables was created for each credit dataset. The list order was determined by decreasing variable importance measured with the Akaike Information Criterion AIC. First, the list was divided into three equal parts (top, middle, bottom) or sub-lists; then, four subsets were created from each one of the three sub-lists. These subsets contained all variables, none of them, the variables in even positions of the sub-list, and the variables in odd positions. Taking all possible combinations formed by one subset from each sub-list, a total of 4\({}^{3}-1=63\) variable combinations were made, excluding the empty set.  
In this way, the models trained in each of the experiments covered a wide range of predictive performance and model complexity, going from a few of the worst explanatory variables to all the variables available in each credit datasets.  
For each experiment, each combination of covariates was used on a logistic regression trained with the whole available sample of each credit dataset using the \"glm\" function of the \"Stats\" package in R version 4.4.1. Details and syntax of this function can be found on the official R documentation website [36]. In fact, since the underlying population used for sampling was known, it was possible to find out the true performance of each of the models trained by applying them to the remaining observations not included in each sample.  
Table A1 in Appendix A shows the expected true AUR performance of each logistic regression considered, calculated by averaging over 150 experiments of each credit dataset, and their corresponding standard deviation. These details give an overview of the large variety of model performances tested in this study, where the averaged true AUR goes from 0.50 to 0.93, allowing us to challenge the model selection capabilities of the validation algorithms in many potential scenarios, thus providing strong support to the conclusions.  
The explanatory variables corresponding to each model are shown in Table A1 as a combination of the subsets that correspond to the top, middle, and bottom sub-lists of the variable list ordered by increasing AIC. The subsets of each third of the list have been labelled as follows: \"A\" = all variables of the corresponding sub-list; \"N\" = none of the variables; \"O\" = variables in odd positions of the sub-list; and \"E\" = variables in even positions.