Cohort Analysis: From SQL to Plotly visualization
By: Ismael Masharo
Tags: Plotly, Visualization, Cohorts, Sql, Data Analysis

Placing ourselves in context

Picture an e-commerce platform. It provides services. Users start to sign up. Some of them are frequently interacting, some of them leave after a month or two. How do we summarise this behavior? What kind of report could we perform to get value out of the collected data? Well, this is what Cohort Analysis is all about.

What’s a Cohort?

It’s a group of people who share a common characteristic over a time frame . Say again? Take for example this bluish group of students:

Photo by Mikael Kristenson on Unsplash

Imagine all of them graduated in 2018. This group is a cohort . They all share the year of graduation.

Okay, so what is Cohort Analysis?

It’s the analysis of these individuals clustered in a given period, how frequently they interact with a service and how these interactions vary over time.

In a more formal definition, it’s one of the existing mechanisms used in behavioral analysis to identify trends in customer retention and its interaction and evolution over time with a given service (let’s say our e-commerce site).

What questions does it answer?

Let’s consider the user registration date and a monthly time frame for clustering users. The questions will be the following:

What percentage of users registered in month Y is still active after X months?

Here users registered in month Y is a Cohort . The analysis is made on its behavior X months after its registration.

How do user interactions vary from one cohort to another?

While the first question analyzes one cohort and its evolution over time , here we are focusing on how do different cohorts are compared to one another .

What’s the output and how does it look like?

The outcome of the analysis is summarized as a Cohort Table .

Monthly cohort table from users activity in 2018. Built with Plotly.

The horizontal axis represents the evolution of independent cohorts over time while the vertical axis compares the difference between them.

The values in each cell are the percentage of active users in a given cohort month vs numbers of months ahead intersection. This is known as the user retention percentage.

Let’s dig a little bit deeper.

How to interpret this report?

We will take the values of the plot above for our example. Let’s also consider the number of users registered in January and February to be 450 and 1010 respectively.

Then, analyzing the cohort of January, the story will be:

From the users registered in January, we had a 24.89% user retention, where only 112 made an interaction. Five months after its registration, only 39 (8.67%) are still active.

What was the question about? We wanted to know how does a set of users interact with our platform and not only that but the evolution of these interactions.

Now let’s compare January and February cohorts:

Taking a look at the percentage of active user in each cohort we can see an 11% increase in retention, with 112 active users out of 450 in January against 363 out of 1010 the next month.

What are we seeing here? How two different cohorts vary from one another.

Getting a little bit technical

We’ve been discussing some of the main concepts about cohort analysis. Now lets briefly talk about how can we build such a report.

Where do I get the data? It’s probably stored in a database

The source of the analysis will be a database model with the following schema:

Here, user holds information on the registered users whereas interaction records the different user interactions and the date they were performed.

Almost a Cohort Table

It’ll be demanding to construct the cohort table directly by querying the database. So, we will first go from the schema to something like the following:

We could then use a data manipulation tool like Excel or Tableau to perform the finale tweaks and get the cohort table. I’ve used pandas for this task.

Let’s check the columns above:

cohort month is equivalent to the Y variable (users analyzed per month) and number of months ahead to the X one (evolution after a given number of months).

is equivalent to the variable (users analyzed per month) and to the one (evolution after a given number of months). total user has the same value for a given month since it is the number of users analyzed in that cohort.

has the same value for a given month since it is the number of users analyzed in that cohort. The value of retention percentage is what we’re aiming to get from the analysis.

SQL your closest ally

As we saw, to start building our cohort table we need to perform some aggregations and groupings over a database. It’ll deviate the aim of this post explaining the process for constructing such a query. For more detail check out this great post from Holistycs where they describe the steps in more detail.

Here’s the query from where we get our summarize table:

Making Cohort Tables Interactive

The SQL query returns something we could handle with a little effort for a couple of cohorts. But imagine having dozens of cohorts, then you’ll have dozens of dozens of values for each of them. Not so easy to handle now.

Luckily for us, there are some pretty neat tools we could use to manipulate the data. I’ve used pandas for merging, stacking, and reshaping the table obtained from the SQL. You can take a look at the code and experiment with your own data. Here’s how the final table looks like:

This is the actual Cohort Table . It looks nice but it can be nicer if we could interact with it. That’s were Plotly shines displaying the table as an interactive plot. You can then customize what to display on hover, the color gradient, the legend, embed it in your site, and so on. Again you can check out the code for the steps to build it.

So, here’s the final visualization:

Interactive cohort table served with Chart Studio.

Finale thoughts

Remember cohort analysis is a tool for start tackling other interesting questions like why does this happen? what do we do about it? how do we profit out of it? With this, you could continue with further richer analysis.

This has been my first post! Hope you found it useful.

Happy analysis!