# Robust Cross-Validation of Predictive Models Used in Credit Default Risk
## 6 Experimental Framework
### Credit Default Datasets  
The set of experiments was made with several consumer credit datasets from developed countries of distinct geographical areas to test the validation methodologies with potentially different credit default patterns. These datasets have been used in previous research and are publicly available to make the conclusions comparable with any other future or past studies.  
Table 1 shows the datasets used, their size, and bad credit rates, as well as the source repository where they can be found. The event rates of these credit datasets go from 14% to 44%. These high rates are not normally found in credit products; for example, typical rates for credit cards usually vary between 1% and 7% [30], while rates higher than 10% were observed only during the \"credit-crunch\" of 2007-2008 for some products. However, it is a common practice in predictive modeling of credit defaults to make an oversampling  
of delinquency events that allows reaching ratios of bad/good credits from 1:1 to 1:5, which facilitates the modeling procedure. So, event rates after oversampling generally vary between 17% and 50%. In this sense, the range of event rates of the credit data used in the experiments aligns very well with those over-sampling conditions given in practice.  
We have to make sure that the three issues faced in the validation of consumer credit default models are also present in the current experiments. The imbalanced data issue is already taken into consideration due to the low default rate of most of the datasets, usually making it more difficult to both build and validate predictive models [9; 17; 35].  
To include data scarcity in the experiments, samples of 500 records have been used for the German and Lending Club datasets. The Australian dataset has only 690 records in total, so a sample size of 230 observations was used, representing one-third of the available data. For the Taiwan dataset, 500 observations are probably not representative enough to make modeling exercises, so samples of 10% of the population were used to have sufficient data to build predictive models, but still small enough to be considered as a potential data shortage situation.  
The explanatory variables included in the datasets mainly concern socio-demographic, financial, and professional information of the applicant, as well as characteristics related to the credit application (purpose, amount, duration...).  
Due to the large number of categorical levels of some variables of the Lending Club 2009 file, the original categorical explanatory variables of this dataset were aggregated in 10 bins when there were more than 10 levels, merging the bins with more similar weight of evidence according to Section 2.1 and using the resulting woe of the merged bin as the new variable value.  
Finally, the covariate shift issue will be a result of the data scarcity because the test/train data splits will significantly decrease the representativeness of these datasets. This will lead to partitions with biased covariates if almost all the observations of the experimental sample are needed to fully capture the data patterns of the underlying population.  
This means that the hold-out test samples and the partitioning of cross-validation techniques could induce significant deviations between the distributions of the test and training sets, which can cause undesired, unstable results. This situation should be better managed by robust partitioning techniques, which is the hypothesis to be tested in this work.