A Better Way to Handle Missing Values in your Dataset: Using IterativeImputer (PART I) | by Gifari Hoque | Towards Data Science

IntroductionReasoning with MissingnessHow does it work?- Univariate Imputation- Multivariate ImputationIterative Imputer- How to be the “Chosen One”- The Process Behind the IterationsGetting Started- The Problem with Single Imputation MethodsHow Single Imputation vs. Iterative Imputation can Affect the Distribution of Your DatasetConclusion- References- Final Words
Introduction
Reasoning with Missingness
How does it work?- Univariate Imputation- Multivariate Imputation
Iterative Imputer- How to be the “Chosen One”- The Process Behind the Iterations
Getting Started- The Problem with Single Imputation Methods
How Single Imputation vs. Iterative Imputation can Affect the Distribution of Your Dataset
Conclusion- References- Final Words
Missing data is a major problem when it comes to making profitable business decisions, solving health problems, better understanding a population, etc. Not only is missing data a problem for humans, but it’s also a big problem for the machine we feed the dataset to.
Here’s a hypothetical situation: imagine your boss handing you a book with a number of pages missing every few pages. And now imagine that you HAVE to read the book and give your analysis for each chapter, and a final analysis for the whole book no matter what or you’ll get fired. Now, this may not be surprising for some of you, as some of you may have bosses that actually make you do similar things like this, but how exactly would you handle this situation? Without asking any questions, you accept the task and carry on. I know, now you’re thinking, “I’ll just search this book up online,” and you do. You search everywhere, only to find out that this book doesn’t exist anywhere else, except on your desk right beside you.
You’re only left with one choice, and so, you start to read. You come across the first missing page as you see a torn page in the middle of the book. The book jumps from page number 14 to page number 17. You shrug it off and continue on with the book while also making assumptions about what could’ve possibly happened on pages 15–16. You continue on reading the book and you encounter more missing pages, except now, the book jumps from page 38 to page 45. You can’t believe your eyes, as you start to read page 45. The story is no longer adding up...
You eventually reach the near end of the book and you find out that the ending is not actually the ending. That’s right, a few more pages missing in the end. Having “finished” the book, you look back at the analyses you’ve made for each chapter to try to make a conclusion of your own for the book.
Your dataset is like a story that you give your machine learning model to read, and each row is like a chapter. If your dataset has missing values, your model won’t be able to fully understand what’s going on and may make false predictions.
There are several ways of handling missing data including, but not limited to: ignoring the missing data, removing the row/column depending on the mass of missingness in the row/column, replacement with a single digit (0, 1, any other number, mean, median), grouped replacement, etc.
Having a better understanding of the reasoning for the missingness of your data can help you determine what type of imputation method you can use. In general, there are three types of missing data1:
Missing Completely At Random (MCAR): The values in your dataset are missing completely at random. This is when there is no clear reasoning as to why a certain value in your dataset is missing. An example of this is if a surveyee forgets to answer a question on a voluntary survey. Handling this situation is relatively simple, we can replace the missing data with a mean/median replacement — the data missing is recoverable.
Missing At Random (MAR): The values in your dataset are missing at random. This is when we can determine some correlation to why the data value may be missing. An example of this is if a certain question in a survey is blank for multiple surveyees of the same gender. A way in which we can handle this situation is by using other features to do a grouped mean/median replacement— the data missing is still recoverable.
Missing Not At Random (MNAR): The values in your dataset are not missing at random. This is when we can see a clear pattern to the missing values. An example of this is if a certain question category in a survey is left blank by surveyees because of the question itself, as it may be a sensitive question to the surveyee (the missingness depends on the missing data) — the data missing will be hard to recover unless further research is done. Unlike the other two types of missing data, MNAR is nonignorable.
Before I get into the fun stuff, I’ll have to do a bit more explaining so grab a cup of coffee or a scoop of pre-workout (or two 😏) or whatever you’re into because it’s time to focus.
Let’s start with the basics.
Missing data can cause research or experiments to be biased, can make data analysis more gruesome, and can reduce the quality of your hypothesis testing and results. It’s always better to do more research to reduce your data biases than just say “whatever” and continue on to your analysis. In my opinion, this defeats the purpose of why you first started your experimentation. In the end, not only will your results have bad quality, you may even end up misleading others. That’s why we see a lot of “myths” today. It’s one thing to create a “clickbait” title, but creating a clickbait title and spewing myths is on a completely different level.
Now let’s say you can’t do any more research because it’s out of your power to be able to do so. What can you do about the data that’s missing in your dataset? You can remove some parts of your data which may cause bias, but what if you don’t have much data to work with so you can’t afford to remove data? That’s when you find other means and look into imputing values to replace missing data.
What is imputation? As defined in Wikipedia,
In statistics, imputation is the process of replacing missing data with substituted values.
As stated earlier, we can replace (impute) missing values using several different approaches. The approaches boil down to two different categories of imputation algorithms: univariate imputation and multivariate imputation2.
Single imputation methods are known as univariate imputation. As stated in scikit-learn’s documentation:
One type of imputation algorithm is univariate, which imputes values in the i-th feature dimension using only non-missing values in that feature dimension.
This means that univariate imputation methods only use the non-missing values in a chosen feature (chosen by you), and impute the missing values in the same feature using either a constant that you provide or by using a statistical measurement (mean/median/mode) of that chosen feature.
For example, in Python, a simple one-liner would do the justice:
>>> data['chosenFeature'].fillna(data['chosenFeature'].mean(), inplace = True)"""Where `data` is your dataset and `chosenFeature` is the column that you choose to replace the missing NaN values for by using the current mean of the non-missing values in the column."""
You can also use scikit-learn’s SimpleImputer class to impute missing values. To learn more about this class, click here.
Multiple imputations can be used in cases where the data are MCAR, MAR, and even when the data are MNAR. Multiple imputation methods are known as multivariate imputation.
Multivariate imputation algorithms use the entire set of available feature dimensions to estimate the missing values.
Basically, the purpose of multivariate imputation is to use other features (columns) in the dataset to predict the missing value(s) in the current feature.
You may be thinking “Well Gifari, you’re telling me that multiple imputation algorithms use other columns, but how does the algorithm know which feature to choose first to impute values on if we have multiple features (columns) with missing values?” Hold your horses there, Billy (no offense to all the Billy’s out there), that’s a very good question, and I’ll get into that in the next chapter where I’ll talk about how you can be the chosen one.
Single imputation methods, as you may have guessed, don’t take uncertainty into account. Single imputation methods don’t care about “what ifs”; it has just one goal: to get rid of missing values. This is when we use multiple imputation methods, where not only are we getting rid of missing values, we’re also considering the “what ifs”.
Although the methodology of multiple imputations may sound difficult to implement, this is the 21st century, where our lives are the most technologically advanced it’s ever been. If you’re reading this from the 22nd, hopefully Medium started supporting superscripts of letters and subscripts by then, and thank you for taking the time to read this article, but I don’t think you should be here. I’m pretty sure there’re more advanced imputation methods in your century and thereon.
There are several statistical packages available which allow you to easily perform multiple imputations. For example, if you’re programming in R, then you have the MICE (Multivariate Imputation by Chained Equations) or the missForest packages readily available for your use. If you’re programming in Python, you have scikit-learn’s IterativeImputer package, which was originally inspired by the MICE package in R.
Iterative Imputer is a multivariate imputing strategy that models a column with the missing values (target variable) as a function of other features (predictor variables) in a round-robin fashion and uses that estimate for imputation. The source code can be found on GitHub by clicking here.
NOTE: This estimator is still experimental for now: the predictions and the API might change without any deprecation cycle. To use it, you need to explicitly import enable_iterative_imputer:
>>> # explicitly require this experimental feature>>> from sklearn.experimental import enable_iterative_imputer # noqa>>> # now you can import normally from sklearn.impute>>> from sklearn.impute import IterativeImputer
There are a number of parameters that you can see in IterativeImputer’s documentations. It requires no parameter inputs as the default parameters are already set for you, meaning that, after you finish importing necessary packages, you can simply introduce an Iterative Imputer by running:
imp = IterativeImputer()
The “current feature” is chosen based on the imputation_order given where the default order is “ascending.” This means that the feature chosen first will be the feature containing the least missing values. After imputing all missing values in the chosen feature, the feature no longer contains missing values, so it is no longer the feature with the least missing values, hence the imputer will move onto the next feature with the least missing values.
The other orders are:
“descending”: features with most missing values to fewest
“roman”: left features to right features
“arabic”: right features to left features
“random”: a random order for each round
Now that you know how to be the “Chosen One”, let’s understand the process behind the iterations.
Iterative Imputer initially initializes the missing values with the value passed for initial_strategy, where the initial strategy is the “mean” for each feature. The imputer then uses an estimator (where the default estimator used is Bayesian Ridge) at each step of the round-robin imputation. At each step, a feature column is chosen as the target variable y and the other feature columns are treated as predictor variables X. The missing values are then predicted for each feature. This process is repeated for at most “max_iter” number of times, where the default is 10 times (rounds). The reason why I say “at most” and not “exactly” is because early stopping is enabled due to the default parameterization of sample_posterior=False.
The stopping criterion is met once max(abs(X_t-X_{t-1}))/max(abs(X[known_vals])) < tol, where X_t is X at iteration t.
This essentially means that for a single missing value, there are at most max_iter number of predictions and the iterations stop once the difference between the previous iteration prediction and current iteration prediction for a value is smaller than the given tol value (default = 1e-3).
You can mess around with several other parameters, so be sure to try them out on your own dataset! Now enough of this nerdy stuff, let’s see some action.
Now that you have somewhat of an understanding of IterativeImputer, let me introduce you to two different datasets that I’ll be using as examples:
PIMA Indians Diabetes Database: For the first example, I’m going to be using the diabetes dataset which can be found on Kaggle by clicking here. You can also view my notebook on Kaggle which can be used hand in hand along with this example. You can view/edit the notebook by clicking here.
The notebook focuses more on different imputation methods as well as what I call “hybrid methods” for imputing missing values. In the end, it compares the accuracy scores of multiple models training on the different datasets derived from the different imputation methods. It’s a lengthy notebook, but I was very interested in this project. It was created as a machine learning and iterative imputer tutorial.
Stock Market Data: For the second example, I’ll be using Iterative Imputer on the stock market data. Unfortunately, this example will have to have its own article due to the amount of information. I don’t want to spoil this yet as I found this very interesting. I’ve also made a notebook for this example on Kaggle after writing this article which uses functions to simplify the process which I’ll be going over in the second article. You can play along with the notebook by clicking here.
Click here to read the second article. I would really like it if you check it out after finishing reading this one!
I lied when I said I was done with the nerdy stuff. I can’t help it sometimes. How about a little pre-game before proceeding into my first example?
When you impute a single digit to missing values within a feature, depending on how much of your data is missing in the certain feature, you increase the likelihood of causing your data to have more kurtosis. Kurtosis is the measurement of the “tailedness” of the distribution. Like skewness, kurtosis is another measurement used to describe the shape of the data distribution. For simplicity, when one thinks of how “symmetrical” the distribution is, they usually think about the skewness of the data. When one thinks about how “curvy” or “pointy” the distribution is, they usually think about the kurtosis of the data. Here is a visual example of the differences in skewness/kurtosis of a distribution that I’ve coded (weird flex but ok):
The kurtosis of any univariate normal distribution is three (when not using Fisher’s definition)3. To learn more about skewness and kurtosis, click here.
Now you’re ready to head to the party.
For my first example, I’ll be using the PIMA Indians Diabetes Database which can be found on Kaggle, for free. The dataset contains several missing values throughout the dataset (which are masked by zeros). My updated dataset replaces the zeros with NaNs. In particular, I’ll only be working with two features that had the most missing values: “SkinThickness” and “Insulin”. More specifically, out of the 768 rows in this dataset, “SkinThickness” had 227 missing values, and “Insulin” had 374 missing values. Isn’t that crazy.
Note: You don’t need to know the importance of these two features for this example.
I’ve created a notebook on Kaggle which can be used hand in hand with this article. The notebook dives more into different imputation methods including what I like to call, “hybrid methods”. The notebook was created as a machine learning and iterative imputer tutorial, and you can view/edit the notebook by clicking here.
Let’s now look at the distribution of these two features:
The skewness and kurtosis for these two distributions are as follows:
Skewness for “SkinThickness”: 0.690619
Skewness for “Insulin”: 2.166464
Kurtosis for “SkinThickness”: 2.935491
Kurtosis for “Insulin”: 6.370522
Let’s now see how our distribution plot, skewness, and kurtosis change after doing a mean-imputation to our missing values. Without even thinking, we can imagine our distribution plots for both features having more kurtosis because we’re imputing a single value 227 times for “SkinThickness” and a single different value 374 times for “Insulin” (when we only have 768 rows to work with).
Note: The mean for “SkinThickness” is 29.153420 and the mean for “Insulin” is 155.548223.This is for our dataset with NaNs.
The skewness and kurtosis for these two distributions are as follows:
Skewness for “SkinThickness”: 0.837608
Skewness for “Insulin”: 3.380019
Kurtosis for “SkinThickness”: 5.430987
Kurtosis for “Insulin”: 16.232455
Without a surprise, we see S̶O̶M̶E̶ A DRASTIC rise in kurtosis, especially for “Insulin”.
Let’s now use IterativeImputer to impute the missing values for these two features in our dataset containing NaNs. I’ll be using RandomForestRegressor() as the estimator for IterativeImputer.
# Introduce IterativeImputer with an estimator>>> imp = IterativeImputer(estimator=RandomForestRegressor())# Fit to the dataset containing missing values>>> imp.fit(ndf)# Transform the dataset containing missing values>>> df = pd.DataFrame(imp.transform(ndf), columns = ndf.columns)"""Where `ndf` is the dataset containing missing values"""
Let’s now look at the distribution for the two features after imputations.
The skewness and kurtosis for these two distributions are as follows:
Skewness for “SkinThickness”: 0.679155
Skewness for “Insulin”: 2.052863
Kurtosis for “SkinThickness”: 3.039844
Kurtosis for “Insulin”: 6.778861
We can see that the kurtosis for both features isn’t as crazy as they were when we imputed the mean of the two features to their corresponding missing values. Not only that, we can see that the skewness and kurtosis levels for both features stayed relatively similar to the levels in which the dataset containing NaNs had.
In general, the multivariate approach is preferred over the univariate approach, but I personally think this depends on your own needs and goals for your project. In this article, we saw the difference between a mean-imputation and iterative imputation using the PIMA Indians Diabetes Database. In the second article, you will be able to see how another way of using IterativeImputer by using the stock market data.
I do hope that reading this article was worth your time. This was my first article written on Medium, but as far as I can see, it won’t be my last. I highly encourage you to check out the second part of this article by clicking on the link below. I hope you can make use of IterativeImputer, and I hope that you find at least one of my two examples to be useful.
gifari.medium.com
Feel free to check out my notebook which I’ve written on Kaggle for where I go more in depth on IterativeImputer on the PIMA Indians Diabetes Database.
www.kaggle.com
[1]: Wikipedia. Missing Data (click to go back)
[2]: Scikit-learn. Imputation of Missing Values (click to go back)
[3]: Wikipedia. Kurtosis (click to go back)
If you notice any errors in my article, please do leave a comment and reach out to me so I can get it fixed ASAP! I don’t want to mislead anyone or teach anyone the wrong. To me, any criticism is positive criticism, that’s one of the best ways I learn. I’m also always open to tips and suggestions.
If you enjoyed reading this article, follow me on Medium and Kaggle where I’ll be posting more content from time to time. Also, feel free to connect with me on LinkedIn. If you think I deserve a coffee for my work, please do buy me one! Any support helps me keep going.
As always, thanks again for stopping by.