Visualizing Intersections and Overlaps with Python | by Thiago Carvalho | Towards Data Science

A prevalent task in any data analysis is comparing multiple sets of something. You may have lists of IPs for each landing page of your website, clients who bought certain items from your store, multiple answers from a survey, and so many others.
This article will use Python to explore ways to visualize overlaps and intersections of sets, the possibilities, and their advantages and disadvantages.
For the next examples, I’ll use a dataset from the Data Visualization Society 2020 Census.
I’m using the survey because it has many different types of questions, where some are multiple-choice questions with multiple answers, like the bellow.
Let’s say we plot a count for each answer. We would have a total in our chart bigger than the total number of respondents, which can be hard for our audience to understand, raising questions or even making the viewers skeptical about the data.
For example, if we had 100 respondents and three possible answers — A, B, and C.
We could have something like this:50 answered — A and B;25 answered — A and C;25 answered — A;
It looks confusing; even if we explain to the audience that a respondent could select more than one response, it’s hard to grasp what this chart represents.
Besides that, with this visualization, we don’t have any information about how the answers intersect. For example, it’s impossible to tell that nobody selected all three options.
Let’s start with a simple and very familiar solution, Venn diagrams. I’ll use Matplotlib-Venn for this task.
import pandas as pdimport numpy as npimport matplotlib.pyplot as pltfrom matplotlib_venn import venn3, venn3_circlesfrom matplotlib_venn import venn2, venn2_circles
Now let’s load the dataset and prepare the data we want to analyze.
The question we’ll check is, “Which of these best describes your role as a data visualizer in the past year?”.
The answers to this question are distributed in 6 columns, one for each response.
If the respondent selected that answer, the field will have a text. If not, it’ll be empty.
We’ll convert that data to 6 lists containing the indexes of the users that selected each response.
df = pd.read_csv('data/2020/DataVizCensus2020-AnonymizedResponses.csv')nm = 'Which of these best describes your role as a data visualizer in the past year?'d1 = df[~df[nm].isnull()].index.tolist() # independentd2 = df[~df[nm+'_1'].isnull()].index.tolist() # organizationd3 = df[~df[nm+'_2'].isnull()].index.tolist() # hobbyd4 = df[~df[nm+'_3'].isnull()].index.tolist() # studentd5 = df[~df[nm+'_4'].isnull()].index.tolist() # teacherd6 = df[~df[nm+'_5'].isnull()].index.tolist() # passive income
Venn diagrams are straightforward to use and understand.
We need to pass the sets with the key/ids we’ll analyze. If it’s an intersection of two sets, we use Venn2; if it's three sets, we use Venn3.
venn2([set(d1), set(d2)])plt.show()
Great! With Venn Diagrams, we can clearly display that 201 respondents selected A and didn’t select B, 974 selected B and didn’t select A, and 157 selected both.
We can even customize some aspects of the chart.
venn2([set(d1), set(d2)],       set_colors=('#3E64AF', '#3EAF5D'),       set_labels = ('Freelance\nConsultant\nIndependent contractor',                     'Position in an organization\nwith some dataviz job responsibilities'),      alpha=0.75)venn2_circles([set(d1), set(d2)], lw=0.7)plt.show()
venn3([set(d1), set(d2), set(d5)],      set_colors=('#3E64AF', '#3EAF5D', '#D74E3B'),       set_labels = ('Freelance\nConsultant\nIndependent contractor',                     'Position in an organization\nwith some data viz job responsibilities',                    'Academic\nTeacher'),      alpha=0.75)venn3_circles([set(d1), set(d2), set(d5)], lw=0.7) plt.show()
That’s great, but what if we want to display the overlaps of more than 3 sets?
Well, there are a couple of possibilities. We could use multiple diagrams, for example.
labels = ['Freelance\nConsultant\nIndependent contractor',          'Position in an organization\nwith some data viz\njob responsibilities',           'Non-compensated\ndata visualization hobbyist',          'Student',          'Academic/Teacher',          'Passive income from\ndata visualization\nrelated products']c = ('#3E64AF', '#3EAF5D')# subplot indexestxt_indexes = [1, 7, 13, 19, 25]title_indexes = [2, 9, 16, 23, 30]plot_indexes = [8, 14, 20, 26, 15, 21, 27, 22, 28, 29]# combinations of setstitle_sets = [[set(d1), set(d2)], [set(d2), set(d3)],               [set(d3), set(d4)], [set(d4), set(d5)],               [set(d5), set(d6)]]plot_sets = [[set(d1), set(d3)], [set(d1), set(d4)],              [set(d1), set(d5)], [set(d1), set(d6)],             [set(d2), set(d4)], [set(d2), set(d5)],             [set(d2), set(d6)], [set(d3), set(d5)],             [set(d3), set(d6)], [set(d4), set(d6)]]fig, ax = plt.subplots(1, figsize=(16,16))# plot textsfor idx, txt_idx in enumerate(txt_indexes):    plt.subplot(6, 6, txt_idx)    plt.text(0.5,0.5,             labels[idx+1],              ha='center', va='center', color='#1F764B')    plt.axis('off')# plot top plots (the ones with a title)for idx, title_idx in enumerate(title_indexes):    plt.subplot(6, 6, title_idx)    venn2(title_sets[idx], set_colors=c, set_labels = (' ', ' '))    plt.title(labels[idx], fontsize=10, color='#1F4576')# plot the rest of the diagramsfor idx, plot_idx in enumerate(plot_indexes):    plt.subplot(6, 6, plot_idx)    venn2(plot_sets[idx], set_colors=c, set_labels = (' ', ' '))plt.savefig('venn_matrix.png')
That’s ok, but it didn’t really solve the problem. We can’t tell if there’s someone who selected all answers, nor can we tell the intersection of three sets.
What about a Venn with four circles?
Here is where things start to get complicated.
In the above image, there is no intersection for only blue and green. To solve that, we can use ellipses instead of circles.
I’ll use PyVenn for the next example.
from venn import vennsets = {    labels[0]: set(d1),    labels[1]: set(d2),    labels[2]: set(d3),    labels[3]: set(d4)}fig, ax = plt.subplots(1, figsize=(16,12))venn(sets, ax=ax)plt.legend(labels[:-2], ncol=6)
Alright, there it is!
But, we lost a critical encoding in our diagram — the size.
The blue (807) is smaller than the yellow (62), which doesn’t help much in visualizing the data. We can use the legends and the labels to figure what is what, but using a table would be clearer than this.
There are a few implementations of area proportional Venn diagrams that can handle more than three sets, but I couldn’t find any in Python.
Layout wise, it’s hard to make a Venn diagram that can display more than three sets properly.
But there’s another solution available. Upset plots are a great way of displaying the intersection of multiple sets. They’re not so intuitive to read as Venn diagrams but do get the job done.
I’ll use the library UpSetPlot for the next example, but first, let’s prepare the data.
upset_df = pd.DataFrame()col_names = ['Independent', 'Work for Org', 'Hobby', 'Student', 'Academic', 'Passive Income']nm = 'Which of these best describes your role as a data visualizer in the past year?'for idx, col in enumerate(df[[nm, nm+'_1', nm+'_2', nm+'_3', nm+'_4', nm+'_5']]):    temp = []    for i in df[col]:        if str(i) != 'nan':            temp.append(True)        else:            temp.append(False)    upset_df[col_names[idx]] = temp    upset_df['c'] = 1example = upset_df.groupby(col_names).count().sort_values('c')example
With the data properly arranged, we only need one method to draw our chart, and we’re done.
upsetplot.plot(example['c'], sort_by="cardinality")plt.title('Which of these best describes your role as a data visualizer in the past year?', loc='left')plt.show()
Awesome! On the top, we have bars showing how many times a combination appeared. On the bottom, there is a matrix showing which combination each bar represents, and on the bottom left, we have a horizontal bar chart representing the total size of each set.
That’s a lot of information, but the well-organized layout makes it simple to extract insights.
Even with my poorly written labels, we can easily tell that most people selected ‘working for an organization.’
The second most common answer wasn’t even displayed in the previous Venn diagrams: the number of people who didn’t select any answer.
Overall visualizing sets and their intersections can be a thought task, but we have some good options for tackling it.
I prefer Venn Diagrams when dealing with a small number of sets and Upset plots when it's more than three. It’s always a good idea to explain what the visualization is displaying and how to read the charts you’re presenting, especially in cases like this where the charts aren’t so friendly.
Thanks for reading my article. Here, you can find more tutorials about DataViz with Python.