Divide and Conquer: segment your customers using RFM analysis | by Gabriel Signoretti | Towards Data Science

Most people who run an online business must know the importance of gathering insights from the user’s data generated in their platform. Be it to understand some usability issues, customer preferences, general buying behavior and much more. Thus, if you are building an eCommerce, it is essential to store data on your orders, customers and their transactions. One of the main purposes of storing such data is to analyze the behavior of customers and design more profitable strategies accordingly.
In our company is no different! We have a very data-centric culture and are always looking for new ways to improve our products based on the information we can extract from it. One of our core products is a drink delivery app that started operating in the second trimester of 2019. The app in question is a marketplace for stores and local producers to sell beer, wine, drinks and similar products like snack foods and the like.
As we are a startup and this product is still fairly recent in the market, it presents some unique challenges and developing it is an ever-evolving process. We are currently implementing various new features and bugfixes. However, data generated since our launch still serves as a great source for learning more about our customers and maybe the platform itself.
In addition to that, the pursuit of integrating business intelligence into the system also motivates this work. In this series, we will be looking at 2 different methods for segmenting customers into separate groups based on their similarities: The classical RFM analysis (explored in this first article) and K-Means clustering (explored in the followup article).
Throughout this series we will try to answer some of these questions:
What are the existing user personas (groups)?
What characteristics do these user groups have (average ticket, frequency, recency, location, etc.)?
Of course, as the data is proprietary and includes some sensitive information, we will not be able to provide the original dataset for you to follow along. In addition to that, some of the data presented will be modified or normalized beforehand.
But fear not! You can find some similar datasets on Kaggle, such as the Retail Data Analytics, the Online Retail Data Set and the Retail Transaction Data datasets, for example. In addition to that, the purpose of these articles is more about teaching you some of the theory and coding than the data and results themselves.
So, let's dive right in!
To perform this analysis we used the Python programming language and some basic data science libraries. Although it is not strictly required for you to understand all the bits and pieces of the code to follow along, it is recommended that you are at least a bit familiar with programming in general.
With that in mind, here are some of the concepts that we will be using throughout the series:
Python and some basic understanding of the Numpy, Pandas and Scikit-Learn APIs;
Some statistics (nothing too fancy);
Basic understanding of some Machine Learning concepts and jargons such as clustering
With all that out of the way, let’s get started with our discussion
Customer segmentation is an important part of developing a successful business. Customers have different kinds of needs and, as the customer and transaction base grows, it becomes increasingly harder to understand the individual requirements of each one of them. At that point, you should be able to identify those differences and act on them. However, there are many ways to perform this segmentation. The one we will be talking about today is called RFM.
RFM is one of the most widely used techniques for selecting significant customers. It is a very popular customer segmentation technique that uses the customers’ past purchase behavior to divide them into different groups based on similarities.
The name RFM stands for Recency, Frequency, and Monetary value. Recency (R) represents the number of days since the customer's last purchase; Frequency (F) represents the number of times the customer has purchased in the timeframe being analyzed; and, finally, Monetary (M) represents the total amount spent by the customer in the same period.
After calculating the Recency, Frequency, and Monetary values for each customer, we need to bin them into 3 or more distinct categories. With this, we can rank the different types of customers based on these categories.
Let's take a look at that in a more practical scenario, shall we?
We will start with a dataset containing information about individual orders performed on our platform. The data is represented in a transaction level i.e. each row of the dataset contains features associated with a single transaction, such as date, time, payment method, client id of the user that made the purchase and etc.
Some important features of the dataset are highlighted below:
order_id: the order unique identifier ;
store_id: the id of the store in which the order was made;
customer_id: the id of the customer that performed the order;
payment_option_id: the identifier of the payment option used;
status: an integer that represents the current status of the order;
created: the date and time in which the order was created;
modified: the date and time in which the order status was last modified
shipping_price: self-explanatory
total_price: the order total price, including shipping
latitude & longitude: the location where the order was delivered
After loading the order's data set, we then need to group and summarize it to obtain customer-level data for each individual customer, such as the number of orders made, the total amount spent, frequency, recency and etc. With this information, we can proceed to perform the RFM analysis. The code used to load the dataset and perform the grouping is highlighted below:
# load the order datasetorders = pd.read_csv('data/orders-27-11-2019.csv')# convert 'created' and 'modified' columns to datetime orders['created'] = pd.to_datetime(orders['created'])orders['modified'] = pd.to_datetime(orders['modified'])# create a snapshot date with today's datesnapshot_date = max(orders.created) + datetime.timedelta(days=1)# create functions to get recency and tenuredef get_recency(x):    last_purchase = x.max()    return (snapshot_date - last_purchase).daysdef get_tenure(x):    first_purchase = x.min()    return (snapshot_date - first_purchase).days# aggregate data by the customerscustomers = orders.groupby('customer_id').agg(    recency=('created', get_recency),    tenure=('created', get_tenure),    frequency=('order_id', 'count'),    total_value=('total_price', 'sum'),    mean_value=('total_price', 'mean'),)# show 5 samples of the grouped dataframecustomers.sample(5)
Remember that we masked the real values, so that is why some of them may seem strange. Notice also that we added 2 new columns that we haven't discussed before: the tenure, which represents the time since the customer first purchase and mean value, which is pretty self-explanatory. These 2 columns will be used to improve the clustering presented in the next article.
The next thing we need to do is to segment the recency, frequency, and total_value into the categories that we discussed earlier. For our use case, we decided to split each feature into 4 quartiles that roughly divide the sample into 4 segments of equal proportion. We called these scores R, F, and M respectively. The code used to do so is shown below:
# use only the necessary columnsrfm = customers[['customer_id', 'recency', 'frequency', 'total_value']]# recency quartile segmentationr_labels = range(4, 0, -1)recency = rfm['recency']r_quartiles, bins = pd.qcut(recency, 4, labels=r_labels, retbins=True)rfm = rfm.assign(R=r_quartiles.values)# frequency quartile segmentationf_labels = range(1, 5)frequency = rfm['frequency'].rank(method='first') # rank to deal with duplicate valuesf_quartiles, bins = pd.qcut(frequency, 4, labels=f_labels, retbins=True)rfm = rfm.assign(F = f_quartiles.values)# monetary value quartile segmentationm_labels = range(1, 5)monetary = rfm['total_value']m_quartiles, bins = pd.qcut(monetary, 4, labels=m_labels, retbins=True)rfm = rfm.assign(M = m_quartiles.values)# show 5 samples of the newly created scoresrfm[['R', 'F', 'M']].sample(5)
To simplify the analysis, it is important that we combine the 3 different scores (R, F, and M) to create a single metric.
There are a few approaches available. The first one is to create an RFM Segment by concatenating the 3 digits from the individual scores to form a 3 character string that goes from 111 (lowest possible score in all three metrics) to 444 (highest possible score in all three metrics). The drawback of this method is the creation of many distinct segments (4x4x4 = 64 segments) which are not so easy to differentiate and prioritize (who is more valuable, a 432 or a 234 customer?).
Another possibility is to sum the 3 individual scores to create the RFM Score, a number ranging from 3 (lowest possible score in all metrics) to 12 (highest possible score in all metrics). Here the drawback is that customers with different buying habits (e.g. from different RFM Segments) can fall on the same score bins. For example, both customers in segments 431 and 134 would get a score of 8. On the other hand, we end up with less distinct scores to compare (4+4+4 = 12 scores), each of them with equal relevance.
# Build RFM Segment and RFM Scoredef join_rfm(x):     return str(x['R']) + str(x['F']) + str(x['M'])rfm['segment'] = rfm.apply(join_rfm, axis=1)rfm['score'] = rfm[['R','F','M']].sum(axis=1)# show 5 samplesrfm[['R', 'F', 'M', 'segment', 'score']].sample(5)
We can further divide the customers into RFM tiers by binning together ranges of scores. For example, we can say that customers with scores ranging from 3 to 5 are Bronze, from 5 to 9 are Silver and from 9 to 12 are Gold.
# group into different tiersdef get_tier(df):    if df['score'] >= 9:        return 'Gold'    elif (df['score'] >= 5) and (df['score'] < 9):        return 'Silver'    else:        return 'Bronze'rfm['tier'] = rfm.apply(get_tier, axis=1)rfm[['R', 'F', 'M', 'segment', 'score', 'tier']].sample(5)
Following these simple steps, you have successfully segmented your customer base! Where can you go from here?
You can start by building some summaries to see the means and standard deviations of the variables for each different score, segment, and tier:
# Summary metrics per RFM Scorescore_summary = rfm.groupby('score').agg(    mean_recency=('recency', 'mean'),    std_recency=('recency', 'std'),    mean_frequency=('frequency', 'mean'),    std_frequency=('frequency', 'std'),    mean_monetary=('total_value', 'mean'),    std_monetary=('total_value', 'std'),       samples=('customer_id', lambda x: len(x)*100/len(rfm.score))).round(2)# Get the 10 segments with most customerspopular_segments = rfm.segment.value_counts()[:10].index.tolist()# Summary metrics for the 10 most popular RFM Segmentssegment_summary = rfm[rfm.segment.isin(popular_segments)].groupby('segment').agg(    mean_recency=('recency', 'mean'),    std_recency=('recency', 'std'),    mean_frequency=('frequency', 'mean'),    std_frequency=('frequency', 'std'),    mean_monetary=('total_value', 'mean'),    std_monetary=('total_value', 'std'),       samples=('customer_id', lambda x: len(x)*100/len(rfm.score))).round(2)# Summary metrics per RFM Tiertier_summary = rfm.groupby('tier').agg(    mean_recency=('recency', 'mean'),    std_recency=('recency', 'std'),    mean_frequency=('frequency', 'mean'),    std_frequency=('frequency', 'std'),    mean_monetary=('total_value', 'mean'),    std_monetary=('total_value', 'std'),       samples_percentage=('customer_id', lambda x: len(x)*100/len(rfm.score))).round(2)
The lowest recency, highest frequency, and the highest monetary amounts can be classified as our best customers.
On the other hand, customers with high recency are the ones who stopped visiting the store for some reason. The business should focus on finding and understanding those reasons and also some methods to reactivate those sleeping customers.
Finally, customers with a low-frequency are the ones that do not buy very often. If they have low recency they may be new customers and the business should focus on retaining them for future purchases. If they have high recency, on the other hand, they also can be categorized as customers needing reactivation.
With this, we get to the end of our discussion on RFM segmentation. But you should not stop exploring!
Additionally to segmenting customers, the RFM criterion can be used by the business to asses the customers buying patterns and assess the effectiveness of different marketing strategies by analyzing the responses of those customers.
In the next part of the series, we will dive into another form of customer segmentation: clustering using the K-Means.
The steps involved in RFM analysis can be summarized as such:
Create an RFM table;
Calculate the Recency, Frequency and Monetary values for each customer;
Segment the different values into categories (in our case from 1 to 4) creating the R, F and M scores;
Create combined metrics such as RFM Segment, RFM Score, RFM tier from the 3 base scores calculated on the previous step;
Create summaries to visualize these groups;