Dealing with missing data in data science projects | by Mahbubul Alam | Towards Data Science

It’s about connecting dots..... they are real in data science.
Missing data always existed and has always been a problem. People typically deal with them on an ad hoc, case-by-case basis depending on how large is the dataset and how big is the problem.
It is gaining increasing attention recently due to the proliferation of data and the inevitable missing values that come with them. People are being creative about how to deal with them, even academic papers are being published about missing data problems and methods to deal with them.
The purpose of this article is to summarise some of those discussions and techniques and demonstrate how to navigate that in a data science project. So bear with me.
Missing data are values that are not recorded in a dataset. They can be a single value missing in a single cell or missing of an entire observation (row). Missing data can occur both in a continuous variable (e.g. height of students) or a categorical variable (e.g. gender of a population).
Missing data are common in any field of natural or behavioral science, but it is particularly commonplace in social sciences research data.
In programming languages, missing values are represented as NA or NaN or simply an empty cell in an object.
So where do the missing values come from, and why do they even exist?
Let’s give an example. You are administering a questionnaire survey among a sample of respondents; and in the questionnaire, you are asking a question about household income. Now, what if a respondent refuses to answer that question? Would you make that up or rather leave the field empty? You’d probably leave that cell empty — creating an instance of missing value.
Or a missing value can originate simply due to human errors. Someone may have entered data that was erroneous (remember outliers?) that was later detected and removed during data cleaning or exploratory data analysis (EDA).
In another example, some time series data — particularly survey datasets — might contain missing observations. This can happen simply because no survey took place in one or more years in a series.
Missing values are definitely undesirable but it is difficult to quantify the magnitude of effects in statistical and machine learning projects. If it’s a large dataset and a very small percentage of data is missing the effect may not be detectable at all.
However, if the dataset is relatively small, every data point counts. In these situations, a missing data point means loss of valuable information.
In any case, generally missing data creates imbalanced observations, cause biased estimates, and in extreme cases, can even lead to invalid conclusions.
Prevention is better than cure!
So minimizing the occurrence of missing data at production is the most effective way to deal with them. If you are conducting a customer survey, make efforts not to leave any field empty by following up on unanswered questions. If the sample size is small, determine the accepted threshold of missing values. If the threshold crossed, expand the sample size. Design experiments or collect data explicitly aiming to minimize missing values.
If inevitable to avoid the following are some other common ways to deal with missing data:
Case deletion: if the dataset is relatively large delete the complete record with a missing value
Substitution: substitute missing cells with (a) column mean, (b) mean of nearest neighbors, (c) moving average, or (c) filling with the last observation
Statistical imputation: a regression can be an effective way to determine the value of missing cell given other information in the dataset
Sensitivity analysis: if the sample is small or missing values are relatively large then conduct a sensitivity analysis with multiple variations of outcomes.
So how it works in practice? Below are a few suggestive examples with codes in Python. For this demonstration purpose, I am using a dataset of Starbucks locations across the continental United States.
Import pandas and numpy libraries and the dataset as pandas dataframe.
# import librariesimport pandas as pdimport numpy as np# import datadf = pd.read_csv("starbucks_location.csv")
You can check the total number of missing values in each column with a single line of code.
df.isnull().sum()
You could also check the missing values as a percent of all observations.
df.isnull().sum()*100/len(df)
One treatment of missing values is just dropping the observations entirely if the dataset is relatively large and missing cases are few.
# drop all rows containing nulldf.dropna()# drop all columns containing nulldf.dropna(axis=1)# drop columns with less than 5 na valuesdf.dropna(axis=1, thresh=5)
You could replace the missing values with appropriate data as you wish. There is a range of ways to do that, see what works for your objectives best.
# replace all na values with -9999df.fillna(-9999)# fill na values with NaNdf.fillna(np.NaN)# fill na values with stringsdf.fillna("data missing")# fill na values with mean of columnsdf.fillna(df.mean())# replace na values with mean value of specific columnsdf["col1"] = df["col1"].fillna(df.mean)
If it’s a time series data you could interpolate or use a moving average.
# interpolation of missing values of all columnsdf.interpolate()# interpolation of missing values of specific columnsdf["col"] = df["col"].interpolate()# using moving average or rolling windowdf['col'] = df['col'].rolling(window = 5).mean
In this article I summarised the origin of missing data, the problems they cause and ways to eliminate/deal with them. Some statistical packages automatically eliminate missing data before performing statistical analysis, but that’s not always best to rely on. As a data scientist, you need to have a better handle on how to work with missing observations and I highlighted some industry best practices in this article.