# Robust Cross-Validation of Predictive Models Used in Credit Default Risk
## 3 Model Validation Methodologies
### Cross-Validation  
Usually, the hold-out test sample technique is used to validate a model when data in excess are available, taking out a large enough independent test set separated from the training data to avoid performance overestimation. For those situations in which there is data scarcity, a smaller validation set should be used to allow for enough training data. However, this can add significant bias to the validation results due to the lower representativeness of the test set. The cross-validation method (CV) was specifically designed to tackle this situation. The basis of cross-validation is to use all the \(N\) observations in the sample \(\Omega\) to validate the classification model, calculating its performance on all the available data. Then, the k-fold cross-validation method performs a random partition of the data into \(k\) subsets of equal size \(\{\Omega_{1},\Omega_{2},\dots\Omega_{k}\}\) as shown in Figure 1, and uses each subset \(\Omega_{j}\) of \(N/k\) elements (test set) to validate a model that has been previously trained with the remaining \(N(1-1/k)\) observations of \(\Omega\backslash\Omega_{j}\) (train set). Thus, all the data are used once each time, either to train the model or to validate it. Finally, the average of the performances obtained in the \(k\) test sets is used as an estimation of the model's performance:  
\[AUR_{CV}(k)=\frac{1}{k}\sum_{j=1}^{k}AUR\big{(}\text{PD}_{j};\Omega_{j}\big{)}\]  
where \(AUR(\text{PD}_{j};\Omega_{j})\) is the AUR calculated in the \(\Omega_{j}\) dataset using the PD\({}_{j}\) model trained in \(\Omega\backslash\Omega_{j}\). The k-fold cross-validation technique can be applied iteratively by repeating the process \(r\) times. That is, once an estimate of the performance has been obtained by cross-validation, a different partition of the available data can be made to estimate again the performance of the predictive model in the same way. Finally, all the results given by the \(r\) repetitions are averaged to obtain a more stable final estimate of the predictive model performance:  
\[AUR_{CV}(k,r)=\frac{1}{r}\sum_{i=1}^{r}AUR_{CV}^{r}(k)\]  
where \(AUR_{CV}^{r}(k)\) is \(AUR_{CV}(k)\) calculated for a particular k-fold partition corresponding to the repetition \(r\). Given the flexibility to choose the number of partition segments, the number of repetitions, and the method used for partitioning, a significant direct impact of this parameter setting is expected on the CV results [24]. According to several studies, some of the most interesting results obtained may be the following:  
* The number of partitions most commonly accepted as the best choice is \(k=10\) or even \(k=5\) when compared to other typical values such as \(k=2\) or \(k=N\)[2, 25], although this should not be taken as a rigorous rule of universal application [10]. * The iterative repetition of the CV estimates converges as the number of repetitions increases [9] and significantly improves the validation results by reducing their variance [5, 8, 10, 26, 27]. * The variance of the CV estimator decreases as the size of the data sample increases [8, 26]. * Usually, there is not enough information about the population and the bias of the data samples to calibrate the validation result to the true model performance. For this reason, cross-validation could overestimate or underestimate the actual performance of the predictive model, making it a more appropriate technique for model selection than for model assessment [7, 13, 14]. The versatility of the CV methodology allows for continuous research on how to leverage this technique to improve the predictions of increasingly sophisticated artificial intelligence algorithms that are applicable in many specialized fields, including credit default risk. As will be explained in Section 4, many CV variants have been developed from the original methodology described above. Possibly, stratified cross-validation (SCV) is the most popular and commonly used variant among all of them due to its simplicity and potential benefits.