# Robust Cross-Validation of Predictive Models Used in Credit Default Risk
## 5 Robust Cross-Validation Methods  
As stated in the introductory section, the objective of this work is to identify potentially better validation methodologies to be used for credit default data and models. As discussed previously, cross-validation is the most accepted validation method nowadays, so the  
Figure 2: Classification of validation methods based on fundamental characteristics. research methodology is to identify potentially robust cross-validation methods, those with more stable expected estimations, to compare them with the rest of the validation methodologies for model selection purposes. Going further in the research, another goal is to identify new variants of the robust methods by improving the strategy used to reduce the volatility of the estimations. Among the different methodological variants of cross-validation presented in the previous section, only a few can be considered robust methods because they were designed to make more homogeneous partitioning, potentially resulting in more stable and less volatile estimations. This could be beneficial when validating credit default models, where data scarcity, imbalanced data, and covariate shift are generally present, because these three issues increase the volatility of the results. Then, robust cross-validation methodologies of interest to be tested and analyzed in this research are DB-SCV and DOB-SCV. Nevertheless, they have some weaknesses that are going to be addressed here to potentially improve their performance for model selection. The main problem of these methods for validating credit default models is that they rely on Euclidean distances to distribute the sample observations more uniformly. This is a problem when dealing with categorical variables because a Euclidean distance cannot be calculated, and consumer credit models usually have several categorical explanatory variables. Euclidean distance in several dimensions can only be calculated properly for quantitative attributes that are comparable in their magnitudes. Both DB-SCV and DOB-SCV deal with this issue by using a \"binarized\" distance for the categorical variables between two observations. This distance is 0 when both observations have the same category and 1 otherwise, considering all possible categories as equidistant from each other. If there are Pcateg categorical covariates, the binarized distance between two observations \(X\) and \(Z\) is calculated similarly to a Euclidean distance as:  
\[D_{categ}(X,Z)=\sqrt{\sum_{i=1}^{Pcateg}\delta_{i}(X,Z)}\ :\ \delta_{i}(X,Z)=1\ if\ \ X_{i}\
eq Z_{i};\ \delta_{i}(X,Z)=0\ if\ \ X_{i}=Z_{i}\]  
To calculate the Euclidean distance between observations using continuous covariates, the values of the covariates must be normalized first to make them comparable. This normalization is made by considering the observed range of values in the sample. Then, the Euclidean distance between two observations \(X\) and \(Z\) when they have \(p_{cont}\) continuous covariates is calculated as:  
\[D_{cont}(X,\ Z)=\sqrt{\sum_{i=1}^{Pcat}\left(\frac{X_{i}-\text{min}\ (X_{i})}{ \text{max}(X_{i})-\text{min}\ (X_{i})}-\frac{Z_{i}-\text{min}\ (Z_{i})}{\text{max}(Z_{i})-\text{min}\ (Z_{i})} \right)^{2}}\]  
In case the two observations \(X\) and \(Z\) have both continuous and categorical variables, the distance between them is calculated following the expression:  
\[D(X,Z)=\sqrt{D_{cont}(X,Z)^{2}+D_{categ}(X,Z)^{2}}\]  
In order to enhance the performance of these validators, it is proposed here to use the weight of evidence of the categorical variables to calculate a Euclidean distance for \(D_{categ}\). This could lead to better assessing the similarity or the distance between observations when categorical variables are involved, making them more fairly comparable in terms of the same quantitative magnitude. This does not mean that the two variables must be used as the model covariates. The covariates in the model could be the two variables, the binarized dummy variables, or any other transformed variable.