Additional file 1 of Large-scale evaluation of k-fold cross-validation ensembles for uncertainty estimation
Creators: name: Dutschmann, Thomas-Martin
name: Kinzel, Lennart
name: ter Laak, Antonius
name: Baumann, Knut
Publisher: name: figshare
Year: 2024
Subjects: subject: Medicine
subject: Biological Sciences not elsewhere classified
subject: Information Systems not elsewhere classified
subject: Mathematical Sciences not elsewhere classified
subject: Plant Biology
subject: FOS: Biological sciences
subject: Computational Biology

description: Additional file 1: Figure S1. Overview of the predictive performances (R2) of the 200-member ensembles, for all datasets and each combination of featurization and modeling technique. Brighter colors correspond to larger values. Figure S2.1. Differences in predictive performances between subsampling by 2-fold CV and by bootstrapping for the eight selected datasets. Each row corresponds to the comparisons between results of one dataset, each plot in a row shows the results for a specific modeling technique. Within each plot, a pair of bars represents the results for a specific featurization, also indicated by the color. The left bar of each pair shows R2 when generating 200 members by 2-fold, the right bar when using bootstrapping. Figure S2.2. Differences in UQ performances between subsampling by 2-fold CV and by bootstrapping for the eight selected datasets. Each row corresponds to the comparisons between results of one dataset, each plot in a row shows the results for a specific modeling technique. Within each plot, a pair of bars represents the results for a specific featurization, also indicated by the color. The left bar of each pair shows ρ when generating 200 members by 2-fold, the right bar when using bootstrapping. Figure S3.1. Raw predictions. Figure S3.2. Predictions after removing those that are outside the acceptable interval. Figure S3.3. Predictions when removing ’Kappa3’ from the dataset featurized as RDKit descriptors before machine learning. Figure S4. Overview of the UQ performances (ρ) of the 200-member ensembles, for all datasets and each combination of featurization and modeling technique. Brighter colors correspond to larger values. Figure S5.1. Raw cumulative member curves for predictive performance (a) and UQ performance (b), for all descriptors, modeling P03372, using SNN ensembles. Figure S5.2. Median cumulative member curves for predictive performance (a) and UQ performance (b), for all descriptors, obtained from 200 permutations of the 200 members per setting. Figure S5.3. Michaelis-Menten functions, fitted to the median cumulative member curves for predictive performance (a) and UQ performance (b), for all descriptors. The median cumulative member curves are shown as thin black lines. The estimated saturation point for each fitted curve (i.e., the first ensemble size where the corresponding gain in performance falls below 0.0001) is depicted as vertical bar. Figure S6. Median ensembles at saturation against full ensemble performance, for predictive performance (a) and UQ performance (b), for all datasets (32 points in each plot). Figure S7. Overview of the predictive performances (R2) and UQ performances (ρ) of the 10-fold CV single DNN models, for all datasets and each featurization. Brighter colors correspond to larger values. Figure S8. Overview of the predictive performances (R2) and UQ performances (ρ) of the 10-fold CV single RF models, for all datasets and each featurization. Brighter colors correspond to larger values. Table S1. All data sets used for evaluation. The original number of compounds refers to the number of measurement points in the raw files, the number of compounds after pipeline to the preprocessed files of which descriptor values were computed from. Table S2. Python packages with versions used for this study.