Visualizing the Nothing. How to visualize the incompleteness of... | by Thiago Carvalho | Towards Data Science

It’s hard to know what to do with null values in our data. Most times, it’s easier to drop them and follow with what’s left.
But they can eventually have meaning and can be investigated. Taking some time to have a good look at them can often bring a better understanding of how the data was collected and even reveal some patterns in it.
In this article, we’ll explore how to visualize all the NULLs in our datasets and get a look at what insights we can extract from doing so.
I’ll run the code in Jupyer Lab, and I’ll use Pandas, Numpy, Missingno, and Matplotlib for this example.
The dataset will be California Jail Profile Survey, which contains monthly county-level data from 1995 to 2018.
import pandas as pdf = 'data/california_jail_county_monthly_1995_2018.csv'df = pd.read_csv(f)
After loading the dataset to Pandas, we can look at one of its convenient methods for dealing with Nulls.
We can use .isnull followed by a .sum and get the number of missing values.
df.isnull().sum()
That’s already useful since it gives us an idea of which fields we can rely on, but there are better ways of visualizing this, let’s try using Missingno.
Missingno is a library for visualizing incompleteness in a dataset, it works on top of Matplotlib and Seaborn and is effortless to use.
import missingno as msno
We’ll start with a simple bar chart. Instead of comparing that extensive list of numbers, we’ll use rectangles and their sizes.
msno.bar(df)
That’s great! In a single method, just passing the data frame, we already have a visual representation of our missing values.
This bar chart represents the number of records in each column. Here we have two scales, one telling the exact amount of values in the field (right) and the other representing the proportion to the total number of rows.
Starting with a bar chart allows us to quickly tell if we’ll need to spend more time analyzing the null values or not.
It isn’t the case with our data, but if you have a blessed dataset without any null values or with an acceptable amount of them, you could stop by now and follow up with your analysis.
Let’s try a different visualization for missing values.
msno.matrix(df)
A matrix tells us exactly where the missing values are, in our example, the data is sorted with the newest records on top.
We can already have some valuable insights by looking at the matrix and knowing how our data is arranged. Here we can see that the data collected changed at some point, where fields such as ‘avg inmates get mental health bed’ on the right of the matrix were discontinued, and others were added.
Pretty cool, the matrix also has a small viz on the right that represents the completeness of the rows — They even mark the max and min amount of missing values.
Now let’s get a look at .heatmap, which will help us understand how the fields relate to each other regarding incompleteness.
msno.heatmap(df)
Okay, things just got complicated — Let’s simplify it.
This plot represents the correlation between the null values by column.
Column A has a value, B also has a value, which means a strong positive correlation or a blue 1 in the chart.
Column A has a value, but B has a null, which means a strong negative correlation, or a red -1 in the chart.
We noticed earlier that there was some change in the way data was collected, and here we can see it again. The new variables have a positive correlation with themselves but a negative correlation to the old fields.
Even though we can visualize that relationship by looking at the matrix, heatmaps make them more evident. It’s not always that we’ll have our rows sorted in a convenient way for us to realize the pattern.
In this dataset, we can see that the fields of the survey are almost grouped.
So, if one question is answered, we’ll very likely have the answers for all items of that group.
Okay, I want to go through one more way of visualizing how the fields relate in matters of nulls.
msno.dendrogram(df)
We saw some group behaviour with the heatmaps, but primarily because of the dataset arrangement.
If the rows are sorted, we can notice some correlation between the fields using the matrix chart, but this would be harder if they’re randomly positioned.
The same thing goes with heatmaps. If the columns are randomly positioned, we would have a more challenging time figuring out the pattern among groups of variables.
The dendrogram uses a hierarchical clustering algorithm to bin the fields by their correlation. *the relationship we saw with the heatmaps.
So it’s like figuring out which fields are highly related to each other in matters of nullity, then testing how those groups of variables relate to themselves and so on.
Again we can notice the discontinued fields, but we can get a more unobstructed view of which variables may be more reliable.
The chart illustrates how the groups connect, where connections farther from zero represent combinations of variables that are less similar in nullity.
Missingno is an excellent tool for quickly visualizing missing values; we only used one line of code for each chart and got a pretty decent result.
The tool has some filtering functions to select and arrange the variables we want to plot. And it allows us to customize some aspects of the chart, all on the same statement.
For whatever you can’t customize directly with the Missingno, you can always go to Matplotlib.
from IPython.core.display import display, HTMLimport matplotlib.pyplot as pltimport numpy as np# define matrix for the 13 columns with most nullsm = msno.matrix(df, figsize=(25, 14),                filter='bottom', n=13,                 color=(0.3, 0.3, 0.5))# get ticks names and positionscols = [i.get_text() for i in m.axes.get_xticklabels()]ticks = np.arange(0,len(cols))# plot new ticksplt.xticks(ticks, labels=ticks+1, rotation=0, ha='center')# title n plotplt.title('Null values matrix\n'.upper(), loc='left', fontsize=22)plt.show()# empty string for the HTML and template for divhtml_table = ''template = '<div style="border-style:solid; display: inline-block; width: 325px; padding-left: 5px; border-width: thin" >{} - {}</div>'table_order = [1, 6, 10,               2, 7, 11,               3, 8, 12,               4, 9, 13,               5]# draw tablefor i in table_order:    html_table += template.format(i, cols[i-1])HTML(html_table)
Great, we got a look at the basics of visualizing and analyzing the completeness of a dataset. It’s interesting to see how much we can learn about the data by visualizing their missing values.
Thanks for reading my article. I hope you enjoyed it.
Here you can find more tutorials like this about Python and data visualization.