Additional file 1 of Leave-one-out cross-validation, penalization, and differential bias of some prediction model performance measures—a simulation study
Creators: name: Geroldinger, Angelika
name: Lusa, Lara
name: Nold, Mariana
name: Heinze, Georg
Publisher: name: figshare
Year: 2023
Subjects: subject: Statistics
subject: FOS: Mathematics

description: Additional file 1: S1. Problems in resampling techniques associated with small samples. S2. Data generating mechanism. S3. A side remark on the simple bootstrap: resampling may increase the optimism. Figure S1. Independently validated (solid line) and leave-one-out crossvalidated (dashed line) c-statistics for different penalization strengths in ridge regression on six artificially constructed data sets. The data were created in the same way as for one of the scenarios in our simulation study (null scenario, sample size of 50, marginal event fraction of 0.25). The x-axis shows the tuning parameter in ridge regression (lambda in the R package glmnet) with higher values corresponding to stronger penalization. For each data set we fitted 96 ridge regression models corresponding to a series of log-equidistant tuning values. As in our simulation study, the independently validated c-statistics were obtained by validating the models on an independent data set consisting of 100,000 observations. As expected, the independently validated c-statistics are very close to the true value of 0.5. LOO, leave-one-out crossvalidation; IV, independently validated. Figure S2. Mean and root mean squared differences (RMSD) between Brier scores (BS) computed by data resampling techniques and independently validated (IV) BS for three different model estimators for the simulation settings with 50 observations, an event fraction of 0.25 and either no or strong effects. The Monte Carlo standard errors of both, the mean difference and of the root mean squared difference (x100), were smaller than 0.2 for all scenarios. ML, maximum likelihood; FL, Firth’s logistic regression; RR, ridge regression. LOO, leave-one-out crossvalidation; 5-fold, 5-fold crossvalidation; enhBT, enhanced bootstrap; .632+, .632+ bootstrap; app, apparent estimate. Table S1. Construction of explanatory variables in the simulation study, following Binder H, Sauerbrei W, Royston P. Multivariable Model-Building with Continuous Covariates: 1. Performance Measures and Simulation Design. Germany: University of Freiburg; 2011. Square brackets […] indicate that the argument is truncated to the next integer towards 0. The indicator function 1{…} is equal to 1 if the argument is true and 0 otherwise. Table S2. Mean and standard deviation (x100) of independently validated (IV) c-statistics for different model estimators and all simulation scenarios. The standard deviation strongly depends on the number of new observations (in our case 100 000) used to estimate the IV c-statistics. Table S3. Mean and standard deviation (x100) of independently validated (IV) discrimination slope for different model estimators and all simulation scenarios. The standard deviation strongly depends on the number of new observations (in our case 100 000) used to estimate the IV discrimination slope. Table S4. Mean and standard deviation (x100) of independently validated (IV) Brier score for different model estimators and all simulation scenarios. The standard deviation strongly depends on the number of new observations (in our case 100 000) used to estimate the IV Brier score. Table S5. Percentage of separated data sets for the twelve simulation scenarios in the full data sets, in the data sets used for model fitting in leave-one-outcrossvalidation, leave-pair-out crossvalidation and 5-fold crossvalidation, respectively, and in the bootstrap data sets. Table S6. Mean difference and root mean squared difference (x100) between winsorized c-statistics computed by different resampling techniques and the independently validated (IV) value (as presented in Table S2) for simulation scenarios with sample size of 50 and event fraction of 0.25. Resampled c-statistics were winsorized by replacing values smaller than 0.5 by 0.5. Figure 3 shows the analogous results for the untransformed c-statistics.