How to Deal with Missing Data in Python | by Chaitanya Baweja | Towards Data Science

Last year I was working on a project that dealt with Twitter data. We had around a billion tweets in our database. I needed to clean our data so that we can run some machine learning models on it.
Here is a snapshot of the Dataset:
Weird Unicode characters (emojis), random missing values with changing identifiers, etc. I still have nightmares about that project.
But, that’s how real-world datasets are. You get missing values, random characters, wrong data types, etc. According to some statistics, data scientists end up spending 80% of their time organizing and cleaning data.
A common occurrence in a data-set is missing values. This can happen due to multiple reasons like unrecorded observations or data corruption.
In this tutorial, we will walk through many different ways of handling missing values in Python using the Pandas library.
Pandas library provides a variety of functions for marking these corrupt values. We will study how we can remove or impute these values.
All the examples in this tutorial were tested on a Jupyter notebook running Python 3.7. We will be using NumPy and Pandas in this tutorial.
There is an accompanying Jupyter notebook for this tutorial here.
I would highly recommend setting up a virtual environment with all the required libraries for testing. Here is how you can do it.
$ virtualenv missing_data$ source ./missing_data/bin/activate$ pip3 install pandas numpy
We will be working with a small Employee Dataset for this tutorial.
Download the dataset in CSV format from my Github repo and store it in your current working directory: employees.csv
Let us import this dataset into Python and take a look at it.
Output:
We read the CSV file into a Pandas DataFrame. The .head() method returns the first five rows of the DataFrame.
The dataset has 1000 observations with six variables as given below:
There are two types of missing values in every dataset:
Visible errors: blank cells, special symbols like NA (Not Available), NaN (Not a Number), etc.Obscure errors: non-corrupt but invalid values. For example, a negative salary or a number for a name.
Visible errors: blank cells, special symbols like NA (Not Available), NaN (Not a Number), etc.
Obscure errors: non-corrupt but invalid values. For example, a negative salary or a number for a name.
Note: The name rule might need some modification for Elon’s son X Æ A-12.
The employee dataset has multiple missing values. Let us take a closer look:
You would notice values like NA, NaN, ? and also blank cells. These represent missing values in our dataset.
Let us print the first 10 rows of the ‘Salary’ column.
print(df['Salary'].head(10))
We depict both a snapshot from our dataset and the output of the above statement in the images below.
Pandas automatically marks blank values or values with NA as NaN (missing values).
Pandas also assigns an index value to each row. Values containing NaN are ignored from operations like mean, sum, etc.
While this works for NA and blank lines, Pandas fails to identify other symbols like na, ?, n.a., n/a. This can be seen in the ‘Gender’ column:
As earlier, Pandas takes care of the blank value and converts it to NaN. But it is unable to do so for the n.a. in the 6th row.
With multiple users manually feeding data into a database, this is a common issue. We can pass all of these symbols in the .read_csv() method as a list to allow Pandas to recognize them as corrupt values. Take a look:
Till now, our missing values had unique identifiers which, made them pretty easy to catch. But what happens when we get an invalid data type.
For example, if we are expecting a numeric value but, the user inputs a string like ‘No’ for salary. Technically, this is also a missing value.
I have designed a function that allows me to check for invalid data types in a column. Suppose I wish to ensure that all values in the ‘Salary’ column are of type int. I will use the following:
After this, values in the salary column will be of type int with NaN wherever invalid entries occurred.
In Pandas, we have two functions for marking missing values:
isnull(): mark all NaN values in the dataset as True
notnull(): mark all NaN values in the dataset as False.
Look at the code below:
# NaN values are marked Trueprint(df[‘Gender’].isnull().head(10))# NaN values are marked False print(df[‘Gender’].notnull().head(10))
We can use the outputs of the isnull and notnull functions for filtering.
Let us print all rows for which Gender is not missing.
# notnull will return False for all NaN valuesnull_filter = df['Gender'].notnull()# prints only those rows where null_filter is Trueprint(df[null_filter])
isnull and notnull can also be used to summarize missing values.
print(df.isnull().values.any())# OutputTrue
print(df.isnull().sum())# OutputFirst Name            70Gender               149Salary                 5Bonus %                4Senior Management     71Team                  48
Now that we have marked all missing values in our dataset as NaN, we need to decide how do we wish to handle them.
The most elementary strategy is to remove all rows that contain missing values or, in extreme cases, entire columns that contain missing values.
Pandas library provides the dropna() function that can be used to drop either columns or rows with missing data.
In the example below, we use dropna() to remove all rows with missing data:
# drop all rows with NaN valuesdf.dropna(axis=0,inplace=True)
inplace=True causes all changes to happen in the same data frame rather than returning a new one.
To drop columns, we need to set axis = 1.
We can also use the how parameter.
how = 'any’: at least one value must be null.
how = 'all’: all values must be null.
Some code examples using the how parameter:
Removing rows is a good option when missing values are rare. But this is not always practical. We need to replace these NaNs with intelligent guesses.
There are many options to pick from when replacing a missing value:
A single pre-decided constant value, such as 0.
Taking value from another randomly selected sample.
Mean, median, or mode for the column.
Interpolate value using a predictive model.
In order to fill missing values in a datasets, Pandas library provides us with fillna(), replace() and interpolate() functions.
Let us look at these functions one by one using examples.
We will use fillna() to replace missing values in the ‘Salary’ column with 0.
df['Salary'].fillna(0, inplace=True)# To check changes call# print(df['Salary'].head(10))
We can also do the same for categorical variables like ‘Gender’.
df['Gender'].fillna('No Gender', inplace=True)
This is a common approach when filling missing values in image data. We use method = 'pad’ for taking values from the previous row.
df['Salary'].fillna(method='pad', inplace=True)
We use method = 'bfill’ for taking values from the next row.
df['Salary'].fillna(method='bfill', inplace=True)
A common sensible approach.
# using mediandf['Salary'].fillna(df['Salary'].median(), inplace=True)#using meandf['Salary'].fillna(int(df['Salary'].mean()), inplace=True)
Note: Information about other options for fillna is available here.
The replace method is a more generic form of the fillna method. Here, we specify both the value to be replaced and the replacement value.
# will replace NaN value in Salary with value 0  df['Salary'].replace(to_replace = np.nan, value = 0,inplace=True)
interpolate() function is used to fill NaN values using various interpolation techniques. Read more about the interpolation methods here.
Let us interpolate the missing values using the Linear Interpolation method.
df['Salary'].interpolate(method='linear', direction = 'forward', inplace=True) print(df['Salary'].head(10))
Whether we like it or not, real world data is messy. Data cleaning is a major part of every data science project.
In this tutorial, we went over techniques to detect, mark and replace missing values.
Often data is missing due to random reasons like data corruption, signal errors, etc. But some times, there is a deeper reason for this missing data.
While we discussed replacement using mean, median, interpolation, etc, missing values usually have a more complex statistical relationship with our data. I would recommend going through this guide for more details.
I hope that after reading this tutorial, you spend less time cleaning data and more time exploring and modeling. Happy to answer all your queries in the comments below.
Python Pandas — Missing Data
DataCamp Course: Dealing with Missing Data in Python
Machine Learning Mastery — How to Handle Missing Data with Python
Chaitanya Baweja aspires to solve real world problems with engineering solutions. Follow his journey on Twitter and Linkedin.