TITLE: Benchmarking missing-values approaches for predictive models on health databases
AUTHORS: Alexandre Perez-Lebel, Gaël Varoquaux, Marine Le Morvan, Julie Josse, Jean-Baptiste Poline

[DESCRIPTION]
<div class = "text-blocks"><div class = "text-block">BACKGROUND</div><div class = "text-block">As databases grow larger, it becomes harder to fully control their collection, and they frequently come with missing values: incomplete observations. These large databases are well suited to train machine-learning models, for instance for forecasting or to extract biomarkers in biomedical settings. Such predictive approaches can use discriminative --rather than generative-- modeling, and thus open the door to new missing-values strategies. Yet existing empirical evaluations of strategies to handle missing values have focused on inferential statistics.</div><div class = "text-block">RESULTS</div><div class = "text-block">Here we conduct a systematic benchmark of missing-values strategies in predictive models with a focus on health databases: two electronic health record datasets, a population brain imaging one, and a health survey. Using gradient-boosted trees, we compare native support for missing values with simple and state-of-the-art imputation prior to learning. We investigate prediction accuracy and computational time. For prediction after imputation, we find that adding an indicator to express which values have been imputed is important, suggesting that the data are missing not at random. Elaborate missing values imputation can improve prediction compared to simple strategies but requires longer computational time on large data. Learning trees that model missing values --with missing incorporated attribute-- leads to robust, fast, and well-performing predictive modeling.</div><div class = "text-block">CONCLUSIONS</div><div class = "text-block">Native support for missing values in supervised machine learning predicts better than state-of-the-art imputation with much less computational cost. When using imputation, it is important to add indicator columns expressing which values have been imputed.</div></div>

[STEPS]
?. [Introduction]
This protocol details the experiments run in the article Benchmarking missing-values approaches for predictive models on health databases, Perez-Lebel et al. 2021. Documented code is available on GitHub:And can be installed through the following steps:We benchmarked two sets of 9 supervised predictive methods on 13 prediction tasks taken from 4 health databases.
?. [Prediction]
ScaleTo study the influence of the scale on the results, we decided to work on 4 sizes of the training set: 2 500, 10 000, 25 000 and 100 000. For each one of these sizes are run the following operations.Outer cross-validationFirst, 5 train sets are randomly selected with the appropriate number of samples (2 500, 10 000, 25 000 or 100 000). For each train set, the test set is composed of all the remaining examples. Note that the size of the test set is considerably larger with a train set of 2 500 samples than with 100 000. For each of the 5 folds, the methods are fitted on the train set and tested on the test set. For imputation-based methods, missing values of both train and test sets are first imputed with the imputer fitted on the training set only. Then, a cross-validated hyper-parameter tuning of the predictive model's parameters is performed on the imputed train set. Once the best model is obtained, it is tested against the imputed test set and its score is reported (accuracy or R²). Methods that do not require imputation follow the same pipeline with the imputation step skipped.To draw the 5 folds, we used StratifiedShuffleSplit (resp. ShuffleSplit) from scikit-learn for classifications (resp. regressions).Evaluating a method on a prediction task is done by running:Be careful to replace placeholders {task_name}, {method_id} and {T} by the name of the task, the ID or name of the method and the trial ID (0 to 4) respectively.Example:Results are dumped in the results/ folder.
[CPU hours to run the full benchmark]
?. [Prediction]
Imputation4 imputation methods are available:Imputation with the mean.Imputation with the median.Iterative imputation.Imputation with the nearest neighbors.For each of them, new binary features can be added to the data. This binary mask encodes whether a value was originally missing or not.The imputer is fitted on the train set only and both the train and test sets are then imputed with the fitted imputer. Doing so avoids leaking information from the train set to the test set and then helps to avoid overfitting.We used SimpleImputer, IterativeImputer and KNNImputer from scikit-learn.
?. [Prediction]
Cross-validated hyper-parameters tuningHyper-parameters of the predictive models are tuned in a cross-validated manner on the train set (imputed or not).We used GridSearchCV from scikit-learn to perform the cross-validated hyper-parameters tuning. The cross-validation folds are drawn with StratifiedShuffleSplit (resp. ShuffleSplit) from scikit-learn for classifications (resp. regressions).
?. [Data]
Each one of the 4 databases needs to be downloaded separately from their respective source project. Access to Traumabase, UK BioBank and MIMIC-III, requires an application. NHIS is freely available. Once downloaded, data path of each database can be updated in the TB.py, UKBB.py,MIMIC.py and NHIS.py files which are in the database/ folder of the project.
?. [Prediction tasks]
From these databases, we defined 13 prediction tasks. That is, a set of input features and an outcome to predict. All features of each task belong to the same database.Available tasks can be obtained with:Names of the available tasks are:
TB/death_pvalsTB/platelet_pvalsTB/hemoTB/hemo_pvalsTB/septic_pvalsUKBB/breast_25UKBB/breast_pvalsUKBB/skin_pvalsUKBB/parkinson_pvalsUKBB/fluid_pvalsMIMIC/septic_pvalsMIMIC/hemo_pvalsNHIS/bmi_pvals
?. [Data]
The Traumabase Group (TB) is a collaboration studying major trauma. The database gathers information from 20 French trauma centers on more than 20 000 trauma cases from admission until discharge from critical care. Data collection started in 2010 and is still ongoing in 2020. We used records spanning from 2010 to 2019. We defined 5 prediction tasks on this database, 4 classifications and 1 regression. Data can be obtained by contacting the team on the Traumabase website.
?. [Data]
UK Biobank (UKBB) is a major prospective epidemiology cohort with biomedical measurements. It provides health information on more than 500 000 United-Kingdom participants aged between 40 to 69 years from 2006 to 2010. We defined 5 tasks on this database, 4 classifications and 1 regression.The data are available upon application as detailed on the UK BioBank website.
?. [Data]
The Medical Information Mart for Intensive Care (MIMIC) database is an Intensive Care Unit (ICU) dataset developed by the MIT Lab for Computational Physiology. It comprises deidentified health data associated with about 60 000 ICU admissions recorded at the Beth Israel Deaconess Medical Center of Boston, United States, between 2001 and 2012. It includes demographics, vital signs, laboratory tests, medications, and more. We defined 2 classification tasks on this database. The data can be accessed via an application described on the MIMIC website. Note that, as of the time of writing, the completion of an online MIT course is required for the application. We used the 1.4 version of the data in the project.
?. [Data]
The National Health Interview Survey (NHIS) is a major data collection program of the National Center for Health Statistics (NCHS), part of the Centers for Disease Control and Prevention (CDC) in the United States. It aims to monitor the health of the population. Since 1957, it collects data from United-States population. We used the 2017 edition, summing up to approximately 35 000 households containing about 87 500 persons. We defined 1 regression task on this database.It is freely-accessible on the NHIS website.
?. [Results]
Once the results of all the methods are obtained, they are gathered in a single CSV file using the following command:This creates a scores.csv file in the scores/ folder.The aggregated results obtained during our experiment are given in our repository. This allows to reproduce the figures and tables and to analyze further the results without needing the original data.
?. [Figures and tables]
Most of the figures and tables of our article can be easily reproduced without requiring the original data (based on saved results only).Some figures however need the data to be build. All figures and tables are saved in the graphics/ folder of the repository.The main figure can be reproduced with:
?. [Predictive methods]
36 predictive methods are available. The list of their IDs and names can be obtained running:IDs and names of the available methods are:
0: Classification1: Classification_Logit2: Regression3: Regression_Ridge4: Classification_imputed_Mean5: Classification_Logit_imputed_Mean6: Regression_imputed_Mean7: Regression_Ridge_imputed_Mean8: Classification_imputed_Mean+mask9: Classification_Logit_imputed_Mean+mask10: Regression_imputed_Mean+mask11: Regression_Ridge_imputed_Mean+mask12: Classification_imputed_Med13: Classification_Logit_imputed_Med14: Regression_imputed_Med15: Regression_Ridge_imputed_Med16: Classification_imputed_Med+mask17: Classification_Logit_imputed_Med+mask18: Regression_imputed_Med+mask19: Regression_Ridge_imputed_Med+mask20: Classification_imputed_Iterative21: Classification_Logit_imputed_Iterative22: Regression_imputed_Iterative23: Regression_Ridge_imputed_Iterative24: Classification_imputed_Iterative+mask25: Classification_Logit_imputed_Iterative+mask26: Regression_imputed_Iterative+mask27: Regression_Ridge_imputed_Iterative+mask28: Classification_imputed_KNN29: Classification_Logit_imputed_KNN30: Regression_imputed_KNN31: Regression_Ridge_imputed_KNN32: Classification_imputed_KNN+mask33: Classification_Logit_imputed_KNN+mask34: Regression_imputed_KNN+mask35: Regression_Ridge_imputed_KNN+mask
?. [Feature selection]
11 tasks have their features automatically selected with a simple ANOVA-based univariate test of the link of each feature to the outcome (task name ends with "_pvals" in the code and "_screening" in the article). The 2 remaining tasks have their feature manually defined following the choices of experts in prior studies.
?. [Feature selection]
ANOVA-based feature selectionCategorical features are first one-hot encoded. Then, the ANOVA-based univariate test is performed on one third of the samples which are then discarded. We kept the 100 encoded features having the smallest 100 p-values. Once the features are selected, the cross-validated prediction is performed on the remaining two thirds of the samples. For these tasks, there are 5 trials during which the samples on which the selection test is performed are redrawn, and the prediction each time fitted on the new remaining samples and the new selected features.We used f_classif and f_regression from the feature_selection module of scikit-learn.For each of these tasks, p-values of the test can be computed for each each trial by running:Be careful to replace placeholders {task_name} and {T} by the name of the task and the trial ID (0 to 4) respectively.Example:
?. [Feature selection]
Manual selection following expertsFeatures for the hemorrhagic shock prediction (task named TB/hemo) in the Traumabase database are defined following Jiang et al.:Features for the breast cancer prediction (task named UKBB/breast_25) are defined following Läll et al.:There is only 1 trial for these tasks.