How to Spot Hidden Missing Values With Python
By: Giulio Laurenti
Tags: Python, Missing Data, Data Cleaning, Data Science, Pandas

How to Spot Hidden Missing Values With Python

Know the signs that indicate that your data set still contains missing values.

Image by lpittman from Pixabay

Missing values are a common issue in real-world data. They occur when no information is available for one or more variables in an observation.

Errors in data entry, information not available at the time of data collection and corrupted records are only a few sources of missing values.

Independently of their origin, missing values can introduce bias in the analysis, reduce the power of machine learning models and overall affect the validity of our conclusions. For these reasons, handling missing values is a critical step in data cleaning.

In this post for beginners, we will focus on how to detect missing values.

I created a small dataset called health, which contains four variables.

gender: male; female

male; female min_pressure: blood pressure between heartbeats

blood pressure between heartbeats max_pressure: blood pressure during heartbeats

blood pressure during heartbeats body mass index (BMI): underweight; normal weight; overweight; obese

We will apply techniques commonly used to detect missing values to the health dataset and then see how to spot the missing values that are not detected by these techniques.

Let’s import the necessary libraries.

import pandas as pd

import matplotlib.pyplot as plt

Load and explore the dataset.

health = pd.read_csv('health.csv') health.shape

The dataset has fifty rows and four columns. Let’s print the first and the last five rows, as well as five rows selected at random.

health.head()

health.tail()

health.sample(5)

The dataset has no visible missing values.

When working on a new dataset, we must gain familiarity with the variables and understand what values to expect in them (i.e. data type and range). Knowing the variables can help us identify anomalies that may indicate that the dataset is not yet ready for the analysis.

Once we understand the dataset, we can start cleaning the data.

Detecting missing values

Pandas isnull() and isna() are two functions commonly used to detect missing values. They return the boolean value True if the cell contains a missing value and False if the cell contains a normal value.

health.isnull().head()

notnull() and notna() work in the opposite way of isnull() and isna(). They return True when the value is not missing and False when the value is missing.

health.notnull().head()

Here I will use isnull() and notnull(), but isna() and notna() would produce the same results.

isnull().sum().any() returns True if the dataset contains at least one missing value and False if it contains no missing values.

health.isnull().sum().any()

isnull().any() returns True if a column contains at least one missing value.

Gender is the only variable with missing values.

isnull().sum().sum() returns the number of missing values in the dataset.

isnull().sum().sum()

isnull().sum() returns the number of missing values in each column.

health.isnull().sum()

The dataset has 3 missing values in the variable gender.

If we prefer to visualize the missing values, we can use the missingno library.

import missingno as mso

The matrix and the bar chart are the two plots in missingno used to display missing values.

plt.figure(figsize = (16, 5)) ax0 = plt.subplot(1,2,1)

ax1 = plt.subplot(1,2,2) mso.matrix(health, ax = ax0, sparkline = False, fontsize = 12)

mso.bar(health, ax = ax1, fontsize = 12) plt.subplots_adjust(wspace = 0.3)

In the matrix, the white lines represent the position of missing values. This plot makes it easy to see if an observation has multiple missing values or if the missing values in the dataset follow a pattern.

In the bar plot, the length of the bars is proportional to the number of non-null values.

Visualizing missing values is particularly useful when working with a high number of columns.

How do isnull() and notnull() identify missing values?

From the documentation of isnull(), we can see that the function identifies the following labels:

NaN in numeric arrays

NaN, None in object arrays

NaT in datetime

notnull() does the opposite: it recognizes values not labelled as NaN, None or NaT.

So for isnull() and notnull() to work, the missing values must have specific labels.

Does this mean that we can only use NaN or None to indicate missing values in our datasets? Not exactly.

From the documentation of read_csv(), we can see that the following labels are automatically interpreted as NaN when loading a dataset: ‘’, ‘#N/A’, ‘#N/A N/A’, ‘#NA’, ‘-1.#IND’, ‘-1.#QNAN’, ‘-NaN’, ‘-nan’, ‘1.#IND’, ‘1.#QNAN’, ‘<NA>’, ‘N/A’, ‘NA’, ‘NULL’, ‘NaN’, ‘n/a’, ‘nan’, ‘null’.

The labels above, however, are not the only ones used to indicate missing values. Other common labels include ?,- and 0, which are not automatically converted into NaN by read_csv(), and are not recognized by isnull() and notnull().

But if isnull() and notnull() cannot identify these labels, and we do not see them when exploring the dataset (hidden), how can we spot these missing values? The answer is that we can follow the clues they leave behind.

Detecting hidden missing values

Pay attention to the info() function

The info() function returns general information about the dataset including the number of observations, column names, count of non-null values, data type and memory usage.

health.info()

The variable max_pressure has an object data type even though it contains integers. Pandas assigns the object data type to columns with text or mixed data (numeric and non-numeric). So max_pressure must contain at least one cell with mixed or non-numeric values.

To spot non-numeric values in a Series, we can use the Series.str.isnumeric() function, which checks if all characters in each string of the Series are numeric.

Using the code below, we can filter out the observations with non-numeric values in max_pressure.

non_numeric = health[health["max_pressure"].str.isnumeric() == False]

We can now count the number of these observations using len(), and print the non-numeric values using unique().

print(f"Count non-numeric : {len(non_numeric)}")

print(f"Non numeric values: {non_numeric['max_pressure'].unique()}")

Four observations contain either ? or - in the max_pressure column.

We can replace ? and - with NaN using replace().

health['max_pressure'].replace(['?', '-'], np.NaN, inplace = True)

Checking again for missing values.

health.isnull().sum()

Pay attention to the summary statistics

Summary statistics, created using the describe function, can also help identify missing values.

health.describe()

min_pressure is the blood pressure between heartbeats. In this variable, a min value of 0 does not make any sense, and it likely indicates missing values.

Check how many entries have a value of 0 in min_pressure.

(health['min_pressure'] == 0).sum()

Two observations have a 0 value in min_pressure. We can replace the 0s with NaN.

health['min_pressure'].replace(0, np.NaN, inplace = True)

By default, describe() returns the summary statistics of numeric variables, but we can use the include argument to display other data types such as object or categorical.

Here I will set the include argument to object.

health.describe(include = ‘object’).T

The number of unique values in both gender and bmi does not seem correct. Let’s print the unique values in both variables.

for column in health[['gender', 'bmi']]:

print(f"{column}: {health[column].unique()}")

The variable gender contains missing values as - and 0, as well as the NaN previously identified using isnull(), while bmi contains missing values as ? and ‘ ‘.

Notice that ‘ ‘ was not converted into NaN by read_csv() because, unlike ‘’, it is not an empty string but a string containing a space.

Now we can replace the missing values with NaN.

health.replace({’gender’:[’-’, 0], 'bmi’: [’?’, ' ']}, np.NaN, inplace = True)

Applying isnull().sum() again, we will have the number of missing values in each column.

health.isnull().sum()

Only a few steps back, we were confident that our dataset had only 3 missing values, but by paying attention to the right parameters, we spotted 11 missing values that were not detected by isnull() or notnull().

If 11 missing values can go undetected in a small dataset like this, imagine what could happen in a dataset with hundreds of thousands of rows and hundreds of columns, and how this could affect our conclusions.