What’s a Customer Worth? Modelling Customers Lifetime Value For Non-Contractual Business with Python | by Susan Li | Towards Data Science

In E-Commerce or retail business, the relationship between businesses and customers are non-contractual relationship. In the non-contractual world, customers do go away, but they do so silently; they have no need to tell us they are leaving. This makes for a much trickier CLV calculation. We have to look at the time since a customer’s last transaction and ask a question: is the customer alive but dormant, or is the customer “dead” (“alive” means customers interact with us, “die” means they become inactive as customers)?
So, here we are: modelling customer lifetime value for non-contractual business.
The analysis is drawn upon Dr. Peter Fader of Wharton. We will not dive deeply into the math of CLV in this analysis. However, if you are interest, here is the Paper.
The data we will use is the same as when we did Market Basket Analysis and Customer Segmentation — Online retail dataset that can be downloaded from UCI Machine Learning Repository.
import pandas as pdimport warningswarnings.filterwarnings('ignore')df = pd.read_excel("Online_Retail.xlsx")df.head()
As usual, we have some cleaning to do, then create a new dataframe that only contains CustomerID, InvoiceDate (remove the time) and add a new column — sales:
import datetime as dtdf['InvoiceDate'] = pd.to_datetime(df['InvoiceDate']).dt.datedf = df[pd.notnull(df['CustomerID'])]df = df[(df['Quantity']>0)]df['Sales'] = df['Quantity'] * df['UnitPrice']cols_of_interest = ['CustomerID', 'InvoiceDate', 'Sales']df = df[cols_of_interest]print(df.head())print(df.head())print(df['CustomerID'].nunique())
For the CLV models, the following nomenclature is used:
Frequency represents the number of repeat purchases the customer has made. This means that it’s one less than the total number of purchases.
T represents the age of the customer in whatever time units chosen (daily, in our dataset). This is equal to the duration between a customer’s first purchase and the end of the period under study.
Recency represents the age of the customer when they made their most recent purchases. This is equal to the duration between a customer’s first purchase and their latest purchase. (Thus if they have made only 1 purchase, the recency is 0.)
The following analysis is conducted in Python using Lifetimes packagedeveloped by Cameron Davidson-Pilon, data scientist at Shopify, and the code was largely borrowed from Lifetimes documentation.
from lifetimes.plotting import *from lifetimes.utils import *from lifetimes.estimation import *data = summary_data_from_transaction_data(df, 'CustomerID', 'InvoiceDate', monetary_value_col='Sales', observation_period_end='2011-12-9')data.head()
There are 4339 customers under our analysis.
CustomerID 12346 made 1 purchase only (no repeat), so his frequency and recency are 0, and his age is 325 days (e.g. the duration between his first purchase and the end of the period in the analysis).
data['frequency'].plot(kind='hist', bins=50)print(data['frequency'].describe())print(sum(data['frequency'] == 0)/float(len(data)))
Among all customers in our data, more than 35% of them only made purchase once (no repeat).
from lifetimes import BetaGeoFitterbgf = BetaGeoFitter(penalizer_coef=0.0)bgf.fit(data['frequency'], data['recency'], data['T'])print(bgf)
<lifetimes.BetaGeoFitter: fitted with 4339 subjects, a: 0.00, alpha: 69.01, b: 1.18, r: 0.83>
Visualizing our frequency/recency matrix
Consider: a customer has made purchase every day for four weeks straight, and then we haven’t heard from him in months. What are the chances he is still “alive”? Pretty small, right? On the other hand, a customer who historically made purchase once a quarter, and again last quarter, is likely still alive. We can visualize this relationship using the frequency/recency matrix, which computes the expected number of transactions an artificial customer is to make in the next time period, given his recency (age at last purchase) and frequency (the number of repeat transactions he has made).
from lifetimes.plotting import plot_frequency_recency_matriximport matplotlib.pyplot as pltfig = plt.figure(figsize=(12,8))plot_frequency_recency_matrix(bgf)
If a customer has made 120 purchases, and his latest purchase was when he was approximately 350 days old (i.e. Recency: the duration between her first transaction and her latest transaction is 350 days), then he is our best customer (bottom-right).
Customers who have purchased a lot and purchased recently will likely be the best customers in the future. We will never have enough of them.
Customers who have purchased a lot but not recently (top-right corner), have probably gone.
There are also another type of customers that around (40, 300) that represents the customer who buys infrequently, and we have not seen him recently, so he might buy again. However, we are not sure if he has gone or just between purchases.
We can predict which customers are surely alive:
from lifetimes.plotting import plot_probability_alive_matrixfig = plt.figure(figsize=(12,8))plot_probability_alive_matrix(bgf)
Customers who have purchased recently are almost surely “alive”.
Customers who have purchased a lot but not recently, are likely to have dropped out. And the more they bought in the past, the more likely they have dropped out. They are represented in the upper-right.
We are ranking customers from “highest expected purchases in the next period” to lowest. Models expose a method that will predict a customer’s expected purchases in the next period using their history.
t = 1data['predicted_purchases'] = bgf.conditional_expected_number_of_purchases_up_to_time(t, data['frequency'], data['recency'], data['T'])data.sort_values(by='predicted_purchases').tail(5)
Listed above are our top 5 customers that the model expects them to make purchases in the next day. The predicted_purchases column represents their expected number of purchases while the other three columns represent their current RF metrics. The BG/NBD model believes these individuals will be making more purchases within the near future as they are our current best customers.
Assessing model fit
from lifetimes.plotting import plot_period_transactionsplot_period_transactions(bgf)
Not bad, out model does not suck. So, we can continue on with our analysis.
We now partition the dataset into a calibration period dataset and a holdout dataset. This is important as we want to test how our model performs on data not yet seen (just like cross-validation in machine learning practice).
from lifetimes.utils import calibration_and_holdout_datasummary_cal_holdout = calibration_and_holdout_data(df, 'CustomerID', 'InvoiceDate',                                        calibration_period_end='2011-06-08',                                        observation_period_end='2011-12-9' )   print(summary_cal_holdout.head())
from lifetimes.plotting import plot_calibration_purchases_vs_holdout_purchasesbgf.fit(summary_cal_holdout['frequency_cal'], summary_cal_holdout['recency_cal'], summary_cal_holdout['T_cal'])plot_calibration_purchases_vs_holdout_purchases(bgf, summary_cal_holdout)
In this plot, we separate the data into both a in-sample (calibration) and validation (holdout) period. The sample period consists the beginning to 2011–06–08; the validation period spans from 2011–06–09 to 2011–12–09. The plot groups all customers in the calibration period by their number of repeat purchases (x-axis) and then averages over their repeat purchases in the holdout period (y-axis). The green and blue line presents the model prediction and actual result of the y-axis respectively. As we can see, our model is able to very accurately predict the customer base’s behaviour out of the sample, the model under-estimates at 4 purchases and after 5 purchases.
Based on customer history, we can now predict what an individual’s future purchases might look like:
t = 10individual = data.loc[12347]bgf.predict(t, individual['frequency'], individual['recency'], individual['T'])
0.1572775101289126
Our model predicts that customer 12347’s future transaction is 0.157 in 10 days.
Given a customer transaction history, we can calculate their historical probability of being alive, according to our trained model. For example, we want to look at our best customer’s transactions history and see the probability of being alive:
from lifetimes.plotting import plot_history_aliveimport matplotlib.pyplot as pltfig = plt.figure(figsize=(12,8))id = 14606days_since_birth = 365sp_trans = df.loc[df['CustomerID'] == id]plot_history_alive(bgf, days_since_birth, sp_trans, 'InvoiceDate')
Our best customer is definitely alive, however, he had several hiatus in the second half of the year 2011.
fig = plt.figure(figsize=(12,8))id = 14729days_since_birth = 365sp_trans = df.loc[df['CustomerID'] == id]plot_history_alive(bgf, days_since_birth, sp_trans, 'InvoiceDate')
On the other hand, this customer had only purchased once and has dropped off for a long time with a 100% chance that he is dead.
We are now taking into account the economic value of each transaction. To estimate this we use the Gamma-Gamma submodel presented by Dr. Peter Fader and Dr. Bruce Hardie from Wharton, to predict likely spend per transaction in the future at the customer level.
We are only estimating the customers who had at least one repeat purchase with us. Therefore, we are estimating for 2,790 customers.
returning_customers_summary = data[data['frequency']>0]print(returning_customers_summary.head())print(len(returning_customers_summary))
from lifetimes import GammaGammaFitterggf = GammaGammaFitter(penalizer_coef = 0)ggf.fit(returning_customers_summary['frequency'],        returning_customers_summary['monetary_value'])print(ggf)
<lifetimes.GammaGammaFitter: fitted with 2790 subjects, p: 2.10, q: 3.45, v: 485.57>
After applying Gamma-Gamma model, now we can estimate average transaction value for each customer. Phew!
print(ggf.conditional_expected_average_profit(        data['frequency'],        data['monetary_value']    ).head(10))
Jupyter notebook can be found at Github. Happy Weekend!