An RFM Analysis with Python | Wenling Yao | Towards Data Science

It’s time to brush up on my Python skills! 👩🏻‍💻
This week I want to go techie and share a recent Python exercise using the RFM framework. It is a method used to determine customer value by looking at three dimensions:
Recency: when is the last time the user takes an action (e.g., login, place an order)?
Frequency: how many times does the user take this action?
Monetary value: what is the sum of monetary value from this user throughout his/her lifetime?
My work was heavily inspired by many helpful posts on the same dataset like this one. With this post, I want to share my RFM segmentation analysis and focus on some tips that I found useful in quickly bringing out the business value from the model output.
Convert absolute recency, frequency, and monetary values into relative ones to reduce the bias associated with the actual customer lifetime.
Generate simple, heuristic business logic to deliver business value in a quick manner.
Join the segments with other datasets to gain more insights into your customer base.
Before we dive into details, I want to give a quick look into how our dataset looks like and what problems we aim to resolve.
Our raw data is a table of transaction records with the following fields:
<class 'pandas.core.frame.DataFrame'>Int64Index: 406829 entries, 0 to 541908Data columns (total 8 columns): #   Column       Non-Null Count   Dtype  ---  ------       --------------   -----   0   InvoiceNo    406829 non-null  object  1   StockCode    406829 non-null  object  2   Description  406829 non-null  object  3   Quantity     406829 non-null  int64   4   InvoiceDate  406829 non-null  object  5   UnitPrice    406829 non-null  float64 6   CustomerID   406829 non-null  float64 7   Country      406829 non-null  object dtypes: float64(2), int64(1), object(5)memory usage: 27.9+ MB
Usually, we would aggregate the data on the customer level.
We want to apply the RFM framework to this aggregate dataset and answer the following questions:
Is there a correlation between Recency, Frequency, and Monetary Value?How should we define our most valuable customers (MVC)? What percentage of our customers are the most valuable customers?Are we able to create distinguishing segments and design CRM campaigns accordingly to improve customer engagement and/or monetization?
Is there a correlation between Recency, Frequency, and Monetary Value?
How should we define our most valuable customers (MVC)? What percentage of our customers are the most valuable customers?
Are we able to create distinguishing segments and design CRM campaigns accordingly to improve customer engagement and/or monetization?
One thing I noticed in many tutorials is that absolute numbers are used for the calculation. In other words, we have the following:
Recency = the interval between the last action and today
Frequency = the number of actions between the user’s signup date and today
Monetary value = the sum of revenue within the user’s lifetime
After this step, one typical approach is to calculate the Recency/Frequency/Monetary Score by creating buckets based on the quartiles of each individual column. For example, on the column “Frequency” we do the following:
customer_data["FrequencyScore"] = pd.cut(customer_data["Frequency"],                                        bins=[-1,                                          np.percentile(customer_data["Frequency"], 25),                                              np.percentile(customer_data["Frequency"], 50),                                              np.percentile(customer_data["Frequency"], 75),                                              customer_data["Frequency"].max()],                                        labels=[1, 2, 3, 4]).astype("int")
Note that for Recency Score we need to give inverse labels as the more active the customer is the lower the value of Recency. :)
Once we obtain the scores of each individual dimension, we calculate the overall RFM score by summing up the three scores. The higher the overall RFM score (in a range from 3 to 12), the more valuable the customer.
customer_data["RFM"] = customer_data["RecencyScore"] + customer_data["FrequencyScore"] + customer_data["MonetaryScore"]
This method is easy to implement, yet it is prone to bias associated with the actual customer lifetime. For example, compared to the customers who sign up in the last 6 months, those who signed up 12 months ago would by nature have lower recency (assuming generally decreasing user engagement over lifetime), higher frequency, and higher monetary value (assuming cohorts are homogenous in terms of user activity and monetization). Hence, the RFM score of customers who signed up 12 months may be overestimated.
To account for this bias, I would first normalize Recency, Frequency, and Monetary Value before calculating individual scores. For example, I would do Relative Frequency = Recency / Lifetime.
customer_data["RelFrequency"] = customer_data["Frequency"] / customer_data["Lifetime"]
Then I would calculate the Frequency Score based on the Relative Frequency:
customer_data["FrequencyScore"] = pd.cut(customer_data["RelFrequency"],                                        bins=[-1,                                              np.percentile(customer_data["RelFrequency"], 25),                                              np.percentile(customer_data["RelFrequency"], 50),                                              np.percentile(customer_data["RelFrequency"], 75),                                              customer_data["RelFrequency"].max()],                                        labels=[1, 2, 3, 4]).astype("int")
In my notebook, you can find rfm_abs the RFM segmentation using absolute values and rfm_rel using relative values. Here I attach only the median customer lifetime of each RFM segment under these two approaches. As we can see, the segments from the RFM segmentation using absolute values have a larger variance in customer lifetime whereas those from the RFM segmentation using relative values have rather a constant customer lifetime. More specifically, with the 1st approach we see that those with an RFM score of 5 and 6 (thus at the low bound of activity) have a significantly shorter lifetime (and thus are more recent customers). This approach seems to “penalize” the more recent customers for their lower frequency and monetary amount.
On the other hand, our approach manages to reduce the bias associated with customer lifetime in RFM segmentation!
So far so good!
Many tutorials end up here with generating RFM segmentation labels (3–12) with the following summary statistics:
This table answers our first two questions:
There is indeed a positive correlation across Recency, Frequency, and Monetary Value.Our MVCs are those with a total RFM score of 11 and 12, i.e., they on average had more than 7 transactions, generated more than 2K in their lifetime, and had their last transaction within the last 2 weeks. Such customers account for ~16% of our total customer base.
There is indeed a positive correlation across Recency, Frequency, and Monetary Value.
Our MVCs are those with a total RFM score of 11 and 12, i.e., they on average had more than 7 transactions, generated more than 2K in their lifetime, and had their last transaction within the last 2 weeks. Such customers account for ~16% of our total customer base.
However, in real life, this output might not be sufficient for creating quick business value. On one hand, the buckets we get are not so different from each other and there is a chance we can merge some segments. On the other hand, the logic itself is rather complex — if we want to strictly follow the logic we need to deploy the notebook into production, which requires significant engineering effort with high uncertainty of generating substantial business value.
We want to move fast. Therefore, I would like to generate some simple and heuristic logic based on the RFM segmentation results above and apply them to production for analytical and operational purposes. Note that there is no perfect logic — here I only share one set of criteria that I consider as the most differentiating.
customer_data["last_order_within_l60d"] = customer_data["Recency"]<60 # Had transactions in the last 60 dayscustomer_data["more_than_two_orders"] = customer_data["Frequency"]>2 # Logged in more than twicecustomer_data["value_higher_than_2k"] = customer_data["MonetaryValue"]>2000 # Sum of value higher than 2Kcustomer_data.groupby(["last_order_within_l60d", "more_than_two_orders", "value_higher_than_2k"]).count()["Lifetime"]
Based on the branching above, I define five final segments (in the brackets you can find the segment sizes):
High Engagement & High Value (523) are those who had their last transactions in the last 2 months, made more than 2 transactions, and contributed more than 2K on average.High Engagement & Low Value (876) are those who are as active as Group1 but contributed less than 2K.Recent Activity & Low Frequency (811) are those who had their last transactions in the last 2 months but made no more than 2 transactions. These customers also have a lower monetary value. Apparently, most customers in this group are recently acquired customers (their average lifetime is only 45 days).“Old” Activity & High Frequency (523) are those who did not have any transactions in the last 2 months. However, on average they had more than 2 transactions. These customers were very active at the beginning of their journey but are not that active recently. We need to find ways to reengage them.Low Engagement & Low Value (1426) are those who did not have any transactions in the last 2 months, made no more than 2 transactions, and contributed the lowest revenue.
High Engagement & High Value (523) are those who had their last transactions in the last 2 months, made more than 2 transactions, and contributed more than 2K on average.
High Engagement & Low Value (876) are those who are as active as Group1 but contributed less than 2K.
Recent Activity & Low Frequency (811) are those who had their last transactions in the last 2 months but made no more than 2 transactions. These customers also have a lower monetary value. Apparently, most customers in this group are recently acquired customers (their average lifetime is only 45 days).
“Old” Activity & High Frequency (523) are those who did not have any transactions in the last 2 months. However, on average they had more than 2 transactions. These customers were very active at the beginning of their journey but are not that active recently. We need to find ways to reengage them.
Low Engagement & Low Value (1426) are those who did not have any transactions in the last 2 months, made no more than 2 transactions, and contributed the lowest revenue.
There are many use cases based on the learnings we generate above. Here I want to elaborate on two use cases I have used at work.
One way is to attach this customer value score as a new dimension to BI datasets (user profiles, tracking events, purchase records, etc). Combined with existing metrics and dashboards, this can help you answer questions like:
What are the socioeconomic (income, occupation, etc) and demographic traits (age, gender, marital status, etc) of our MVCs? This sheds light on how we can target and acquire more of such customers.
Do our MVCs have a significantly different user journey compared with other groups? How can we optimize the journey experience for our MVC?
Do our MVCs have a significant market basket compared with other groups? This may give us some inspiration for improving recommendations for them.
The other use case is to send this label to a CRM tool and use it for customized engagement communication. Take the final segments I generate above as one example: for Group 2 High Engagement & Low Value and Group 3 Recent Activity & Low Frequency we should focus on monetization (e.g., number of orders, average order value) by personalized recommendation while for Group 4 “Old” Activity & High Frequency we should try to re-engage them and bring them back to our website/application. Combined with features from other datasets we are able to design targeted content and medium to reach individual customers and maximize conversions.
That’s it. Let me know your thoughts & feedback!