# Robust Cross-Validation of Predictive Models Used in Credit Default Risk
## 7 Results and Discussion
### Sample Size Impact  
It is expected that the advantages of using robust validation methods will tend to disappear when increasing the size of the available dataset because the greater representativeness of the training and test sets will increase the homogeneity of the partitions for all kinds of partitioning methods. To test this hypothesis, the same experiments were carried out with double and triple-sized samples for the Lending Club 2009 dataset, which is the one with larger selection errors. As these samples are more representative of the underlying population patterns, the selection errors are reduced significantly for all validation methods. The full validator comparison for sample sizes of 500, 1000, and 1500 observations is given in Table 3 of Appendix A. To compare more clearly the effect of sample size for the cross-validation methods, Tables 4 and 5 show separately the results for two groups corresponding to \(k=5\) and \(k=10\), respectively. The methods without a specific k-fold parameter value have also been added to the comparisons. \begin{table}
\begin{tabular}{c c c c c c c} \hline \hline \multirow{2}{*}{**Validator**} & \multicolumn{3}{c}{**Lending Club—N = 500**} & **Lending Club—N = 1000** & **Lending Club—N = 1500** \\ \cline{2-7}  & **Error** & **Rank** & **Error** & **Rank** & **Error** & **Rank** \\ \hline CV:DOB-WOE/5/50 & 0.3362 & 1 & 0.2491 & 1 & 0.2223 & 2 \\ CV:DOB-SCV/5/50 & 0.3388 & 2 & 0.2462 & 1 & 0.2191 & 1 \\ CV:DB-SCV1/5/50 & 0.3430 & 3 & 0.2535 & 2 & 0.2285 & 3 \\ CV:DB-SCV2/5/50 & 0.3448 & 3 & 0.2557 & 3 & 0.2298 & 4 \\ CV:DB-WOE2/5/50 & 0.3468 & 3 & 0.2557 & 3 & 0.2309 & 5 \\ \hline \hline \end{tabular}
\end{table} 
Table 4: Impact of the sample size on the selection error and ranking of cross-validation methods with \(k=5\) or non-predefined k-fold parameter for the Lending Club 2009 credit dataset. Selection error is given by the normalized Kendall Tau averaged over 150 random samples of sizes \(N=500\), 1000, and 1500. Tables 4 and 5 show that there is no significant enhancement using the weight of evidence of the categorical covariates for the distribution optimally balanced stratified cross-validation methods when the sample size is large enough. Nevertheless, the robust validators DOB-SCV and DB-SCV are still at the top of the performance ranking, with some advantage for the DOB variant. In Table 4, where \(k=5\), other non-robust methods scale up in the performance list, diluting some of the benefits of the homogeneous partitioning against the random resampling or fuzzy methods. In the case of \(k=10\), this is not yet clearly seen because the test sets are smaller and therefore more unstable than in the \(k=5\) case, still giving room for the benefits of homogeneous partitioning. \begin{table}
\begin{tabular}{c c c c c c c} \hline \hline \multirow{2}{*}{**Validator**} & \multicolumn{2}{c}{**Lending Club—N = 500**} & \multicolumn{2}{c}{**Lending Club—N = 1000**} & \multicolumn{2}{c}{**Lending Club—N = 1500**} \\ \cline{2-7}  & **Error** & **Rank** & **Error** & **Rank** & **Error** & **Rank** \\ \hline HO:FMCCV/0/1000 & 0.3468 & 4 & 0.2557 & 3 & 0.2301 & 4 \\ HO:MCCV/0.2/1000 & 0.3474 & 4 & 0.2550 & 3 & 0.2311 & 5 \\ HO:SMCCV/0.2/1000 & 0.3478 & 4 & 0.2565 & 4 & 0.2314 & 5 \\ CV:RSCV/5/50 & 0.3481 & 4 & 0.2555 & 3 & 0.2309 & 5 \\ CO:SCV/5/50 & 0.3481 & 4 & 0.2558 & 3 & 0.2309 & 5 \\ HO:FSMCCV/0/1000 & 0.3483 & 4 & 0.2563 & 4 & 0.2309 & 5 \\ CV:kfold/5/50 & 0.3484 & 4 & 0.2550 & 3 & 0.2304 & 4 \\ CV:RndCV/0/50 & 0.3497 & 4 & 0.2574 & 4 & 0.2313 & 5 \\ CV:FNkfold/0/50 & 0.3503 & 4 & 0.2570 & 4 & 0.2301 & 4 \\ CV:FUkfold/0/50 & 0.3521 & 5 & 0.2582 & 4 & 0.2302 & 4 \\ CV:LOOCV & 0.3605 & 5 & 0.2713 & 5 & 0.2496 & 5 \\ CV:MS-SCV/5/50 & 0.4544 & 6 & 0.3822 & 6 & 0.3255 & 6 \\ \hline \hline \end{tabular}
\end{table} 
Table 4: _Cont._