Cohort analysis with Pandas and Seaborn
By: João Pedro S. Macalós
Tags: Data Science, Seaborn, Pandas, Cohort Analysis

Cohort analysis with Pandas and Seaborn

Introduction

The objective of this notebook is to show, step by step, how to implement a cohort analysis with Pandas and Seaborn.

Coming from an R background, I like working with pipes and to chain commands to modify a DataFrame. The idea of writing this article came when I was trying to implement this analysis with Python and struggled with the existing tutorials. Therefore, the main objective of this post is to present the code — if you are curious about cohort analysis, there are plenty of articles available online.

Cohort analysis

What is a cohort analysis ?

Cohort analysis is a type of analysis that follows a group of customers over time. For example, instead of analyzing our customers individually, we group them by date of sign-up in our app. In this way, we can analyze whether a marketing campaing implemented in a given month was effective or not.

Why does it matter ?

Cohort analyses matter because churn is one of the main problems faced by companies nowadays, and every information that can lead to less churning is a valuable business information.

Olist data

To show how to implement a cohort analysis with Pandas and Seaborn , I will work with the Olist dataset. It is available on Kaggle and provide a good example of an e-commerce dataset. The schema of the data is available in the link above.

The Olist plataform connects small business owners (the sellers) with customers from all over Brazil. Understanding whether sellers come back to the platform after their first sale is a crucial business question for Olist.

Therefore, I will make the cohorts based on the month of the first sale of each seller. With this cohorts, I will track the proportion of sellers from each cohort that came back to the platform in the subsequent months.

Hence, we must connect the olist_sellers_dataset where the id of each seller is located with the olist_orders dataset where we can find the date of each sale ( order_purchase_timestamp ) in the platform. These two tables are connected through the olist_order_items_dataset .

1. Import the required packages

2. Load the data

3. Join the data

Join the datasets, convert date to correct type, remove observations outside the analysis boundaries, and select relevant columns:

4. Create the cohorts

To find the cohorts, we must find the first appearance ( order_purchase_timestamp ) of each individual seller ( seller_id ). To do that, I'll order the dataset by date of sale and drop the duplicates in the seller_id column.

5. Merge and calculate delta

In the next step, we need to:

Merge the seller_cohort DataFrame with df on the seller_id column;

DataFrame with on the column; Assign a column with the months of each sale;

Assign a column that calculates the difference in months between the date of sale and the cohort ( delta ).

Before proceeding with the analysis, it’s important to visualize the size of each cohort we just created to see if there are no clear problems in the data:

As we can see, there were some variability in the number of new sellers joining Olist in the period under analysis, but nothing really outstanding that invalidate our exercise in this post.

6. Count sellers in each cohort per delta and calculate the proportion

The next step consist in:

Counting the unique sellers in each delta by each cohort;

Counting the total sellers in each cohort (the number of sellers in the first month, i.e., when delta is equal to zero);

Calculating the proportion of sellers present in each cohort afterwards.

7. Pivot the DataFrame

Now we have all the information we need to calculate make the cohort analysis, but it is in the long format. The following step consists in applying a pivot method to reshape it:

8. Plot with Seaborn

With the pivoted column, we can simply use the sns.heatmap function to visualize the cohorts:

Conclusion

It’s interesting to note that the first cohort was relatively more loyal to Olist than the cohorts that followed, with a participation rate 6 months after their first sale in the platform that was similar to the participation rate of the other cohorts. It’s also noteworthy that the cohort “2017–07” was also loyal to the brand. Now it is up to the marketing team of Olist to investigate further why some of the cohorts presented churn rates much lower than the others and act upon this information.

Full code

The notebook with the full code of this analysis can be found here: https://github.com/joaomacalos/Medium/blob/main/cohort-pandas-seaborn.ipynb