Cohort Analysis with Python. A data clustering skill that every... | by Joe Tran | Towards Data Science

If you are a data analyst working for an e-commerce company, there is a high chance that one of your tasks at work is to discover insights from customer data to improve customer retention.
However, the customer data is massive and each customer behaves differently. Customer A who was acquired in March 2020 displays a different behaviour from customer B who was acquired in May 2020. Therefore, it is essential to group customers into different clusters and then investigate the behaviour of each cluster over time. This is called cohort analysis.
Cohort Analysis is the data analytic technique in understanding the behaviour of a special group of customers over a period of time.
In this article, I will not go into details the theory of cohort analysis. If you do not know what cohort analysis is all about, I would strongly recommend you read this blog first.
This article is more to show you how to segment customers into different cohorts and observe the retention rate for each cohort over a period of time.
Let’s get into it!
You can download the data here.
import pandas as pdimport matplotlib.pyplot as pltimport seaborn as snsdf = pd.read_csv('sales_2018-01-01_2019-12-31.csv')df
first_time = df.loc[df['customer_type'] == 'First-time',]final = df.loc[df['customer_id'].isin(first_time['customer_id'].values)]
Here is it not wise to just simply select df.loc[df['customer_type']]. Let me explain why. In this data, under the customer_type column, First_time refers to new customer, whereas Returning refers to returning customer. So if I first bought on 31 Dec 2019, the data will show me as new customer on 31 Dec 2019 but as returning customer on my 2nd, 3rd, ... time. The cohort analysis looks at new customers and their subsequent purchases. Therefore, if we simply use df.loc[df['customer_type']=='First-time',] we would be ignoring the subsequent purchases of the new customers, which is not a correct way to analyse cohort behaviour.
Thus, what I did here was that, first create a list of all the first-time customers and store it as first_time. Then from the original customer dataframe df select only those customers whose IDs fall within the first_time customers group. By doing this, we can make sure that we obtain the data that has only first-time customers and their subsequent purchases.
Now, let’s drop the customer_type column because it is not necessary anymore. Also, convert the day column into correct date-time format
final = final.drop(columns = ['customer_type'])final['day']= pd.to_datetime(final['day'], dayfirst=True)
final = final.sort_values(['customer_id','day'])final.reset_index(inplace = True, drop = True)
def purchase_rate(customer_id):    purchase_rate = [1]    counter = 1    for i in range(1,len(customer_id)):          if customer_id[i] != customer_id[i-1]:                 purchase_rate.append(1)                 counter = 1          else:                 counter += 1                 purchase_rate.append(counter)    return purchase_ratedef join_date(date, purchase_rate):    join_date = list(range(len(date)))    for i in range(len(purchase_rate)):           if purchase_rate[i] == 1:                 join_date[i] = date[i]          else:                 join_date[i] = join_date[i-1]    return join_datedef age_by_month(purchase_rate, month, year, join_month, join_year):    age_by_month = list(range(len(year)))    for i in range(len(purchase_rate)):          if purchase_rate[i] == 1:              age_by_month[i] = 0          else:              if year[i] == join_year[i]:                 age_by_month[i] = month[i] - join_month[i]              else:                 age_by_month[i] = month[i] - join_month[i] + 12*(year[i]-join_year[i])     return age_by_month
purchase_rate function will determine whether that is a 2nd, 3rd, 4th purchase of each customer.
join_date function allows us to identify the date the customer joins.
age_by_month function gives us how many months from the current purchase of a customer to the first time purchase.
Now the input is ready. Let’s create cohorts.
final['month'] =pd.to_datetime(final['day']).dt.monthfinal['Purchase Rate'] = purchase_rate(final['customer_id'])final['Join Date'] = join_date(final['day'], final['Purchase Rate'])final['Join Date'] = pd.to_datetime(final['Join Date'], dayfirst=True)final['cohort'] = pd.to_datetime(final['Join Date']).dt.strftime('%Y-%m')final['year'] = pd.to_datetime(final['day']).dt.yearfinal['Join Date Month'] = pd.to_datetime(final['Join Date']).dt.monthfinal['Join Date Year'] = pd.to_datetime(final['Join Date']).dt.year
final['Age by month'] = age_by_month(final['Purchase Rate'], final['month'],final['year'],final['Join Date Month'],final['Join Date Year'])
cohorts = final.groupby(['cohort','Age by month']).nunique()cohorts = cohorts.customer_id.to_frame().reset_index()   # convert series to framecohorts = pd.pivot_table(cohorts, values = 'customer_id',index = 'cohort', columns= 'Age by month')cohorts.replace(np.nan, '',regex=True)
How to interpret this tableTake cohort 2018–01 as an example. In Jan 2018, there were 462 new customers. Out of these 462, 121 customers came back and purchased in Feb 2018, 125 in Mar 2018 and so on and so forth.
for i in range(len(cohorts)-1):    cohorts[i+1] = cohorts[i+1]/cohorts[0]cohorts[0] = cohorts[0]/cohorts[0]
cohorts_t = cohorts.transpose()cohorts_t[cohorts_t.columns].plot(figsize=(10,5))sns.set(style='whitegrid')plt.figure(figsize=(20, 15))plt.title('Cohorts: User Retention')sns.set(font_scale = 0.5) # font sizesns.heatmap(cohorts, mask=cohorts.isnull(),cmap="Blues",annot=True, fmt='.01%')plt.show()
That is it. Hope you guys enjoyed and picked up something from this article. If you have any questions, feel free to put them down in the comment section below. Thank you for your read. Have a great day and happy new year 🎉🎉🎉