A brief guide to data imputation with Python and R | by Pavlo Horbonos | Towards Data Science

We all know, that data cleaning is one of the most time-consuming stages in the data analysis process. We need to acquire missing values, check their distribution, figure out the patterns, and make a decision on how to fill the spaces. At this point you should realize, that identification of missing data patterns and correct imputation process will influence further analysis. So, let me introduces a few technics for the common analysis languages: R and Python.
Before we start the imputation process, we should acquire the data first and find the patterns or schemes of missing data. In simple words, there are two general types of missing data: MCAR and MNAR.
MNAR (missing not at random) is the most serious issue with data. It means, that we need to find the dependencies between missing features, and start the data gathering process. It turns in some kind of analysis step, which involves the work with different data sources, analysis of connections, and search of alternative data.
MCAR (missing completely at random) means that there are no deep patterns in missing values, so we can work with that and decide if some rows/features may be removed or imputed. It is something we can deal with but only within empirical borders because there can be too much missing data (in the percentage of total records).
So, in illustration purposes we will use the next toy-example:
We can see the impact on multiple missing values, numeric, and categorical missing values.
R programming language has a great community, which adds a lot of packages and libraries to the R development warehouse. So, that’s not a surprise, that we have the MICE package. It includes a lot of functionality connected with multivariate imputation with chained equations (that is MICE algorithm). You can dive deep into the documentation for details, but I will give the basic example.
Firstly, let’s see the pattern of the missing data on our toy-example mentioned above:
library(mice)md.pattern(df_test)
Mice package has built-in tool md.pattern(), which shows the distribution of missing values and combinations of missing features. You can read more about this tool in my previous article about missing data acquainting with R. Also this function gives us a pretty illustration:
Work with a mice-imputer is provided within two stages. At the first stage, we prepare the imputer, and at the second stage, we apply it.
Imputation preparation includes prediction methods choice and including/excluding columns from the computation. We just need to rewrite the default imputation method for necessary columns through the $method property. In our case, we used mean (unconditional mean) for first and third columns, pmm (predictive mean matching) for the fifth column, norm (prediction by Bayesian linear regression based on other features) for the fourth column, and logreg (prediction by logistic regression for 2-value variable) for the conditional variable. See more in the documentation for the mice() method and by the command methods(your_mice_instance). We have also excluded the second column from the algorithm.
Now we are ready for the second stage: reuse current mice instance as the input value for the real imputer:
imputation <- mice(df_test, method=init$method,                             predictorMatrix=init$predictorMatrix,                             maxit=10,                            m = 5,                            seed=123)
One of the main features of the MICE package is generating several imputation sets, which we can use as testing examples in further ML models. So, we will be able to choose the best fitting set. In our example we have m=5, so the algorithm generates 5 imputed datasets. You can read more about the work with generated datasets and their usage in your ML pipeline in this article by the author of the package.
The last step is to run the algorithm with the concrete number of the imputed dataset:
imputed <- complete(imputation, 2)
You can see all generated sets within the $imp property of your mice instance. Though, I have chosen the second of the generated sets:
Python has one of the strongest support from the community among the other programming languages. There is the especially great codebase for data science packages. You may find several imputation algorithms in the famous scikit-learn package.
This package also supports multivariate imputation, but as the documentation states — it is still in experimental status. So, let’s see a less complicated algorithm: SimpleImputer. Nevertheless, the imputer component of the sklearn package has more cool features like imputation through K-nearest algorithm, so you are free to explore it in the documentation.
We will use the same toy-example. Of course, a simple imputation algorithm is not so flexible and gives us less predictive power, but it still handles the task. I will skip the part of missing data checking since it is the same as in the previous example. Nevertheless, you can check some good idioms in my article about missing data in Python.
from sklearn.impute import SimpleImputerimpNumeric = SimpleImputer(missing_values=np.nan, strategy='mean')impCategorical = SimpleImputer(missing_values=np.nan,                                strategy='most_frequent')
We have chosen the mean strategy for every numeric column and the most_frequent for the categorical one. You can read more about applied strategies on the documentation page for SingleImputer. You may also notice, that SingeImputer allows to set the value we treat as missing. The further process is much shorter than in R: imputer classes have the same fit-transform procedure as other sklearn components. So, again, we set imputation strategies for every column (except the second):
And here is our imputed dataset:
You are free to experiment, compare, and choose the best one among R and Python implementations.
Data clearing is just the beginning of the analysis process, but mistakes at this stage may become catastrophic for further steps. Make the data clean and see the working code from the article on my Github:
github.com
Also, make sure, you haven’t missed my other data cleaning articles: