# Robust Cross-Validation of Predictive Models Used in Credit Default Risk
###### Abstract  
Model validation is a challenging Machine Learning task, usually more difficult for consumer credit default models because of the availability of small datasets, the modeling of low-frequency events (imbalanced data), and the bias in the explanatory variables induced by the train/test sets split of the validation techniques (covariate shift). While many methodologies have been developed, cross-validation is perhaps the most widely accepted, often being part of the model development process by optimizing the hyperparameters of predictive algorithms. This experimental research focuses on evaluating existing robust cross-validation variants to address the issues of validating credit default models. In addition, some improvements to those methods are proposed and compared with a wide range of validation techniques, including fuzzy methods. To reach solid and practical conclusions, this work limits its scope to logistic regression, as it is the best-practice modeling technique in real-world applications of this context. It is shown that  
Many methods have been investigated and proposed to select the combination of explanatory variables of the final model that maximizes predictive performance or minimizes model error. More simplistic methods like hold-out test samples make a deterministic estimation of error bias, while more advanced methods like cross-validation (CV) and bootstrapping (BTS) are stochastic in nature, considering both bias and variance of the model error [1; 2; 3; 4; 5; 6; 7; 8]. The less computational requirements, better practicality, and good results of cross-validation techniques have made CV the most popular stochastic validation methodology for model error estimation, yielding a broad range of CV variants and benchmarking studies over the last decades [6; 9; 10; 11]. The main motivation of this research is that, in the particular case of credit default risk models, there is a simultaneous concurrence of several validation difficulties: the predicted events have low frequency and high severity, while the available data sample is usually small and always imbalanced because the number of defaulters is much smaller than non-defaulters due to the nature of the credit events, leading to a covariate-shift when cross-validation techniques are used. This results in less stable model performance estimations that increase the risk of not selecting the best model. There are studies that address these issues separately [12; 13; 14], and others propose solutions that are not feasible for all credit default explanatory variables [15], while there are very few robust methods that face all the issues simultaneously [9; 16; 17]. All the above facts support the motivation basis of this work, in a context where a failure in the prediction of an event could be critical, as happens in credit lending, where the economic impact of a bad loan is many times the profit of any performing loan for the financial institution. In fact, it is even worse for the credit applicant, and it could mean the bankruptcy of any person. In addition, the referenced studies in validation methodologies do not tackle this context specifically, and most of the work conducted specifically for credit lending usually proposes and assesses different modeling techniques without focusing on the validation aspect of the problem. Then, we can pose the questions: Are some existing methods more appropriate than others to validate credit default models? In that case, is it possible to improve those methods even further? This paper conducts experimental research to test the hypothesis that reducing the volatility of the results of the validation process using more robust cross-validation methods, the estimations should outperform any other validation methodology when tackling all the above issues simultaneously for model selection in the consumer credit default risk context. In addition, some enhancements are proposed and tested to better leverage categorical explanatory variables, leading to the best results in data-scarce situations.