Getting Started With Data Imputation Using Autoimpute | by Haider Waseem | Towards Data Science

A large majority of datasets in the real world contain missing data. This leads to an issue since most Python machine learning models only work with clean datasets. As a result, analysts need to figure out how to deal with the missing data before proceeding on to the modeling step. Unfortunately, most data professionals are mainly focused on the modeling aspect and they do not pay much attention to the missing values. They usually either just drop the rows with missing values or rely on simple data imputation (replacement) techniques such as mean/median imputation. Such techniques can negatively impact model performance. This is where the Autoimpute library comes in — it provides you a framework for the proper handling of missing data.
Univariate imputation: Impute values using only the target variable itself, for example, mean imputation.Multivariate imputation: Impute values based on other variables, such as, using linear regression to estimate the missing values based on other variables.Single imputation: Impute any missing values within the dataset only once to create a single imputed dataset.Multiple imputation: Impute the same missing values within the dataset multiple times. This basically involves running the single imputation multiple times to get multiple imputed datasets (explained with a detailed example in the next section).
Univariate imputation: Impute values using only the target variable itself, for example, mean imputation.
Multivariate imputation: Impute values based on other variables, such as, using linear regression to estimate the missing values based on other variables.
Single imputation: Impute any missing values within the dataset only once to create a single imputed dataset.
Multiple imputation: Impute the same missing values within the dataset multiple times. This basically involves running the single imputation multiple times to get multiple imputed datasets (explained with a detailed example in the next section).
Now let’s demonstrate how to tackle the issue of missingness using the Autoimpute library. This library provides a framework for handling missing data from the exploration phase up until the modeling phase. The image below shows a basic flowchart of how this process works on regression using multiple imputation.
In the above image, the raw dataset is imputed three times to create three new datasets, each one having its own new imputed values. Separate regressions are run on each of the new datasets and the parameters obtained from these regressions are pooled to form a single model. This process can be generalized to other values of ‘n’ (number of imputed datasets) and various other models.
In order to understand one major advantage of obtaining multiple datasets, we must keep in mind that the missing values are actually unknown and we are not looking to obtain the exact point estimates for them. Instead, we are trying to capture the fact that we do not know the true value and that the value could vary. This technique of having multiple imputed datasets containing different values helps in capturing this variability.
We’ll start off by importing the required libraries.
import numpy as npimport pandas as pdimport matplotlib.pyplot as pltfrom scipy.stats import norm, binomimport seaborn as snsfrom autoimpute.utils import md_pattern, proportionsfrom autoimpute.visuals import plot_md_locations, plot_md_percentfrom autoimpute.visuals import plot_imp_dists, plot_imp_boxplotsfrom autoimpute.visuals import plot_imp_swarmfrom autoimpute.imputations import MultipleImputer
The complete code for this article can be downloaded from this repository: https://github.com/haiderwaseem12/Autoimpute
For demonstration purposes, we create a dummy dataset with 1000 observations. The dataset contains two variables; predictor ‘x’ and response ‘y’. Forty percent of the observations in ‘y’ are randomly replaced by missing values while ‘x’ is fully observed. The correlation between ‘x’ and ‘y’ is approximately 0.8. A scatter plot of the data is shown below.
As a starting point, it is important to examine the missingness of the data in order to extract patterns from it. This helps us in understanding the missingness and choosing the appropriate imputation technique. The visualization techniques provided by the Autoimpute library for this task are demonstrated below.
The results of the function plot_md_percent(df) are shown below. This can be used to visualize the percentage of missingness for each feature in the dataset. In our case, we can see that ‘x’ is completely filled while y is 40% missing.
Another function plot_md_locations(df) helps us visualize the locations of missing data within the DataFrame by showing missing rows as white bars. This plot can come in extremely handy while trying to find patterns to the missingness.
Since we removed values randomly from the ‘y’ column, our data has no pattern to the missingness. We can see that the values of column ‘y’ are missing randomly over all values of ‘x’.
However, in some cases, this plot could help us make sense of the missingness. For example, if we saw more white bars near the bottom, we would know that the probability of missingness increases as the value of x increases.
Once we have explored the missingness, we can proceed onto the imputation stage.
The function MultipleImputerprovides us with multiple imputations for our dataset. This function can be used in an extremely simple way and performs reasonably well, even with its default arguments.
imputer = MultipleImputer()                 #initialize the imputerimputations = imputer.fit_transform(df)     #obtain imputations
However, the function is extremely flexible and can be customized in a number of ways. Some of the commonly used arguments are discussed below:
n: Number of imputations (number of new imputed datasets to be created).strategy: The imputation method can be specified using the strategy column. The function provides us with various imputation methods ranging from simple univariate techniques such as mean imputation to other more advanced multivariate ones such as Predictive Mean Matching. If no strategy is specified, the default method is applied. The default method depends on the column data type.predictors: This argument can be used to set which columns are to be used for the imputation of specific columns. If no predictor is specified, all columns are used which is the default option.imp_kwgs: This argument can be used to specify any further parameters that might be needed to customize a certain strategy.
n: Number of imputations (number of new imputed datasets to be created).
strategy: The imputation method can be specified using the strategy column. The function provides us with various imputation methods ranging from simple univariate techniques such as mean imputation to other more advanced multivariate ones such as Predictive Mean Matching. If no strategy is specified, the default method is applied. The default method depends on the column data type.
predictors: This argument can be used to set which columns are to be used for the imputation of specific columns. If no predictor is specified, all columns are used which is the default option.
imp_kwgs: This argument can be used to specify any further parameters that might be needed to customize a certain strategy.
An example of the usage of MultipleImputer and some of its arguments is given below. Let's assume we have a dataset with four columns: gender, salary, education, and age.
MultipleImputer(    n=10,    strategy={'salary':'pmm', 'gender':'binary logistic'},    predictors={'salary':'all', 'gender':['salary','age']},    imp_kwgs={'pmm':{'fill_value':'random', 'neighbors':5}}    )
This function will create ten imputations. Salary and gender are to be imputed using predictive mean matching and binary logistic, respectively. To impute salary, all columns will be used; whereas, for gender, only salary and age variables will be utilized. Lastly, the PMM strategy will use a random fill value and the number of neighbors will be set to five.
Note: Autoimpute follows the same API as Scikit-learn which makes the code familiar to a lot of Python users.
We start off with the most basic type of imputation: Mean Imputation. Since mean imputation is very commonly used, we implement this first and then compare it with another technique provided by the Autoimpute library.
mi_mean = MultipleImputer(n=5, strategy="mean", seed=101)        imp_mean = mi_mean.fit_transform(df)
Autoimpute also provides us with some visualization techniques to see how imputed values have affected our dataset. We will use these plots to compare the performance of different techniques.
plot_imp_swarm(d=imp_mean,             mi=mi_mean,             imp_col=”y”,             title=”Imputed vs Observed Dists after Mean Imputation”             )
From the swarm plot above, we can see that all the imputed values are exactly the same. Since mean imputation replaces each missing value by the column mean, and the mean remains the same each time a column is imputed, this technique gives us the exact same results no matter how many times we impute a column. As a result, imputing by mean multiple times does not introduce any variance to the imputations. This observation is further supported by the two plots below as we can clearly see that the distributions for all five imputations are also exactly the same.
Moreover, the plots below also show how mean imputation can alter the spread of the data. We still have the same mean as the original data but the data is now more tightly centered around the mean, leading to lower variance. This may lead to the model being unreasonably confident in making predictions around the mean value.
plot_imp_dists(d=imp_mean,                mi=mi_mean,                imp_col="y",               separate_observed=False,                 hist_observed=True,               title="Distributions after Mean Imputation"               )plot_imp_boxplots(d=imp_mean,                   mi=mi_mean,                  imp_col="y",                  title="Boxplots after Mean Imputation"                  )
Now we take a look at a more advanced technique: Predictive Mean Matching (PMM) imputation. This method uses a Bayesian Regression Model to predict values for the imputations. The observed values that are closest to the prediction are then found and one of those values is then randomly replaced as the imputation.
Note: Having a detailed understanding of how the PMM algorithm works is not required for the following demonstration. However, if you are unfamiliar with the algorithm and wish to understand it, please take a look at the following link: https://statisticalhorizons.com/predictive-mean-matching.
mi_pmm = MultipleImputer(n=5, strategy="pmm", seed=101) imp_pmm = mi_pmm.fit_transform(df)
We will use the same visualization techniques used above in order to demonstrate the performance of PMM imputation.
plot_imp_swarm(d=imp_pmm,              mi=mi_pmm,             imp_col=”y”,             title=”Imputed vs Observed Dists after Mean Imputation”             )
The first thing that we notice when we look at the swarm plot is that the imputed values are spread over the actual values of ‘y’. This helps preserve the distribution and we can clearly see from the plots below that the distributions of the imputed columns are similar to the actual column. Even though the distributions slightly vary across the different imputations, the mean and variance are both similar to the actual column.
The second observation that we make from the plot above is that the imputed values vary across the different columns. This observation is further supported by the two plots below which clearly show us that all five imputed columns have slightly different distributions. This helps us in capturing the variability of the missing values.
plot_imp_dists(d=imp_pmm,                mi=mi_pmm,                imp_col="y",               separate_observed=False,                 hist_observed=True,               title="Distributions after Mean Imputation"               )plot_imp_boxplots(d=imp_pmm,                   mi=mi_pmm,                  imp_col="y",                  title="Boxplots after Mean Imputation"                  )
In our demonstration, PMM imputation performed better than mean imputation. This is the case for most datasets. However, PMM imputation is not always the best technique since it has its own drawbacks, for instance, it can be computationally expensive. In scenarios where the number of missing values is small, even simple techniques like mean imputation could give optimal results.
As with everything in machine learning, there is no single optimal technique. The optimal technique varies from case to case depending on the characteristics of the given dataset. Finding the optimal technique depends on a combination of domain knowledge and experimentation. The Autoimpute library provides us with an easy way to experiment with various imputation strategies and find out the one that works best.
The complete code for this article can be downloaded from this repository: https://github.com/haiderwaseem12/Autoimpute