Modeling customer churn for an e-commerce company with Python | by Collin Ching | Towards Data Science

It’s more cost effective to retain existing customers than to acquire new ones, which is why it’s important to track customers at high risk of turnover (churn) and target them with retention strategies.
In this project, I’ll build a customer churn model based off of data from Olist, a Brazilian e-commerce site. I’ll use that to identify high risk customers and inform retention strategies and marketing experiments.
There’s one complication with e-commerce. While it’s straightforward to measure churn for a contractual (subscription-based) business, churns aren’t explicitly observed in non-contractual businesses (e-commerce). In these scenarios, probabilistic models come in handy for estimating time of customer death. The probabilistic model that I’ll use is the BG/NBD model from the Lifetimes package.
import pandas as pdimport numpy as npimport datetime as dtimport seaborn as snsimport matplotlib.pyplot as pltfrom lifetimes.utils import *from lifetimes import BetaGeoFitter,GammaGammaFitterfrom lifetimes.plotting import plot_probability_alive_matrix, plot_frequency_recency_matrix, plot_period_transactions, plot_cumulative_transactions,plot_incremental_transactionsfrom lifetimes.generate_data import beta_geometric_nbd_modelfrom lifetimes.plotting import plot_calibration_purchases_vs_holdout_purchases, plot_period_transactions,plot_history_aliveorders = pd.read_csv(‘brazilian-ecommerce/olist_orders_dataset.csv’)items = pd.read_csv(‘brazilian-ecommerce/olist_order_items_dataset.csv’)cust = pd.read_csv(‘brazilian-ecommerce/olist_customers_dataset.csv’)
The lifetimes package relies on recency-frequency-monetary (RFM) analysis to model churn and customer lifetime value (CLV). To make our models, we’ll need a a dataframe that consists of recency, frequency, and monetary columns. The definitions of each are below.
Recency: time between initial purchase and most recent (last) purchase
Frequency: number of repeat purchases made by a customer (total purchases — 1)
Monetary: total spent on purchases
Customer ID information will come from cust. Order date will come from orders. Price will come from items.
print(cust.columns)
There are two columns used to identify customers. customer_id is a customer ID token that is generated for every order. If the same customer makes multiple orders, he has multiple customer_id identifiers. What we want to use for this analysis is customer_unique_id, which is unique to each purchaser and can be used to track their purchases over time.
Here is the distribution of purchases made by customers.
cust.groupby('customer_unique_id').size().value_counts()
The majority of customers made only a single purchase.
orders = pd.merge(orders,cust[['customer_id','customer_unique_id']],on='customer_id')orders.columns
In the items dataset, each item in an order gets its separate row. The price column refers cumulative order purchase rather than individual item price. Since I only need order price, I'll keep the first item from each order.
print(items.columns)
items.drop_duplicates('order_id',keep='first',inplace=True)
Next, I’ll join orders with items to append price information.
transaction_data = pd.merge(orders,items,'inner','order_id')transaction_data = transaction_data[['customer_unique_id','order_purchase_timestamp','price']]## convert timestamp to date; only need the daytransaction_data['date'] = pd.to_datetime(transaction_data['order_purchase_timestamp']).dt.datetransaction_data = transaction_data.drop('order_purchase_timestamp',axis=1)transaction_data.head()
Now that I have my transaction data, I want to convert this into dataframe with the RFM variables that I mentioned in the introduction. The Lifetimes package has a function for converting transaction data into an RFM DataFrame.
summary = summary_data_from_transaction_data(transaction_data,'customer_unique_id','date',monetary_value_col='price',)summary.describe()
summary.head()
The summary function converted customer transactions into an aggregated table. Many of the customers have frequency, recency, and monetary = 0, like customer 0000366f3b9a7992bf8c76cfdf3221e2. That's because Lifetimes only considers customers who have made repeat purchases into account.
Using days as time periods (can also be defined as weeks or months), variables are defined like so for the Lifetimes model:
frequency: # of days in which a customer made a repeat purchase
T: customer's age in days
recency: customer's age in days at time of most recent purchase
monetary_value: mean of a customer's purchases, excluding the 1st purchase
frequency excludes the customer's first purchase because that is considered the day the customer is born. Afterwards, you can begin to question whether or not that customer is alive.
summary[summary['frequency']>0].head()
transaction_data[transaction_data['customer_unique_id']=='004288347e5e88a27ded2bb23747066c']
Note how customer 004288347e5e88a27ded2bb23747066c made two purchases with Olist but his frequency is 1 and monetary_value is $87.90 based on how frequency and monetary_value are defined.
We’re going to use the Beta-Geometric/NBD (BG/NBD) model for customer churn. The BG/NBD model is an adaptation of the Pareto/NBD model. Both models describe repeat purchasing patterns in businesses where customer turnover is unobserved; however, the BG/NBD is much more computationally feasible.
Assumptions of the BG/NBD model:
A customer’s relationship has two phases: “alive” for an unobserved period of time, then “dead”
While alive, the number of transactions made by a customer follows a Poisson distribution with transaction rate lambda
Heterogeneity in lambda follows a gamma distribution
After any transaction, a customer dies with probability p; the probability that a customer dies after a number of transactions follows a geometric distribution
p follows a beta distribution
Lambda and p vary independently across customers
For more information on the BG/NBD model, check out this paper by Peter Fader and this post by Cam Davidson-Pilon.
bgf = BetaGeoFitter(penalizer_coef=0.0)bgf.fit(summary['frequency'], summary['recency'], summary['T']);
plot_frequency_recency_matrix(bgf);
plot_probability_alive_matrix(bgf);
Next we want to evaluate the model to see how well it performs in the future. I’ll split the data into a training (calibration) period and a holdout (observation) period, train the BG/NBD model and evaluate performance with four plots that Peter Fader outlines in this talk (@ 26:10). These plots are:
1) Calibration period histogram: does the model fit the training data?
2) Cumulative transaction plot: does the model predict cumulative sales well?
3) Incremental transaction plot: does the model capture the overall trend in transactions?
4) Conditional expectations plot: can the model predict the number of purchases a customer will make based on the training data?
plot_period_transactions(bgf).set_yscale('log');
The model is fairly representative of the real data up until four repeat transactions. There are few customers who make more purchases.
summary_cal_holdout = calibration_and_holdout_data(transaction_data, 'customer_unique_id', 'date',calibration_period_end='2017-09-03', observation_period_end='2018-09-03' )
We can evaluate how the dataset works by plotting both of them.
bgf.fit(summary_cal_holdout['frequency_cal'], summary_cal_holdout['recency_cal'], summary_cal_holdout['T_cal'])plot_cumulative_transactions(bgf, transaction_data, 'date', 'customer_unique_id', 730, 365);
The red line represents the boundary between the calibration period on the left and the holdout period on the right. As you can see, the BG/NBD model does a pretty swell job at predicting cumulative transactions.
plot_incremental_transactions(bgf, transaction_data, 'date', 'customer_unique_id', 730, 365);
This plot shows that the model does a decent job capturing general trends in the data.
plot_calibration_purchases_vs_holdout_purchases(bgf, summary_cal_holdout);
The model performs well up to three calibration period purchases, but diverges from the holdout data because of the distribution of the data.
cust.groupby('customer_unique_id').size().value_counts()
Less than 1% of customers have made four or more purchases, so there’s not much data for the BG/NBD model to learn about customers who have made many repeat transactions.
In practice, I would consider collecting more data if I were to proceed with modeling customer churn. But for learning purposes, it will still be a good exercise to predict churn.
The BG/NBD model assumes that death can only occur after a repeat purchase, since the customer leaving occurs during a purchase and the first purchase is reserved to signal a customer’s birth.
Because of this, customers with only one transactions will have a 100% probability of being alive, which is suspect. To account for this limitation, we’ll only predict churn risk on customers who have made at least one repeat transaction.
df = summary[summary['frequency']>0]df['prob_alive'] = bgf.conditional_probability_alive(df['frequency'],df['recency'],df['T'])sns.distplot(df['prob_alive']);
From here, we can visualize customers based on the probability that they’re “alive”. Using domain knowledge we might be able to set a threshold for customers who probably have already churned, and also identify customers who are at risk for churning, but haven’t yet disappeared.
Next, I would like to set a decision threshold for customer churn. This is an opportunity to inject personal expertise or talk with domain experts. Assume I speak with the sales and marketing managers, and we agree to consider a customer with <10% chance of being alive to have churned.
df['churn'] = ['churned' if p < .1 else 'not churned' for p in df['prob_alive']]sns.countplot(df['churn']);
A little over 92% of customers have churned, meaning that there’s a lot of opportunity for improvement regarding retention.
We can assume that the customers who have churned are already lost. But what is interesting in a business setting is the customers who are at high risk for churn, but haven’t churned yet. Later on, it might still be a good idea to apply different treatments to the churned group.
If I can identify them, maybe I can encourage the marketing team to target them with promotions.
sns.distplot(df[df['churn']=='not churned']['prob_alive']).set_title('Probability alive, not churned');
It seems reasonable to bucket customers with 80% or more churn risk to be considered high risk for churn.
df['churn'][(df['prob_alive']>=.1) & (df['prob_alive']<.2)] = "high risk"df['churn'].value_counts()
Now that I have these churn groupings, I can move forward and apply special treatments to these groups. Ideally there would be more data and a bigger population of high-risk customers.
We’ve modeled churn risk in a non-contractual setting, and now have three customer segments — not churned, high risk, and churned. This could feed into a dashboard to give stakeholders a glimpse of “at-risk” customers. It also provides three different groups that we can run specific actions. Some ideas:
1) Reach out to churned customers to figure out why they left.
2) Send different types of targeted emails and special offers to the high risk group. If the sample size of high risk customers is large enough, you could split off a few small treatment groups and compare how their retention and CLV change with different promotional or customer relationship strategies.
3) Determine the the highest value customers in the non-churn group, and serve them additional benefits to ensure that they remain loyal customers.
Thanks for reading! If you want to follow along with the code, here’s the GitHub repo. Always open to feedback and questions. Comment below!