Using K-means to segment customers based on RFM Variables
By: Jasneek Chugh
Tags: K Means, Rfm Analysis, Python, Customer Segmentation, Is688spr21a04

Dividing customers into different segments based on the RFM (Recency-Frequency-Monetary) score, in python

Customer Segmentation- RFM

Every person is different and so is their behavior as customers. Imagine you are the owner of a shop. It doesn’t matter if you own an e-commerce or a supermarket. It doesn’t matter if it is a small shop or a big company such as Google or Microsoft, it’s always better to know your customers.

Coming from a business family background, I have always seen my father facing problem in targeting the right customers and building a good relationship with the customer to optimize sales. During my summer vacations, I have helped my father with our business, and I have always observed that every customer has different needs, interest and spending habits. Some were the regular and frequent buyer, some weren’t. I have realised that it’s very difficult for businesses to personalize marketing campaigns for each customer

My father always believes in taking feedback, personalizing communication, providing responsive support, educating the customers with resources to know the customer better and provide them with their respective needs. This has been helping him to retain customers, increase their lifetime value and optimize sales.

Nowadays never techniques have come up in the market for segmenting customers and also Customer segmentation very important to make a decision, what action needed to increase revenue, build a good relationship with the customer and optimize sales, for a business. I’ll show you how to do RFM analysis for analyzing customer value.

The main focus of this blog would be to show how we can segment customers into different clusters using the K-means algorithm. But, first I would be doing RFM analysis to get the desired values and those features will be used as an input in K-means, to determine similarity. K-Means uses Euclidean distance as a distance metric to calculate the distance between the data points.

I will show you the main calculations of RFM analysis only so that this blog doesn’t go that big.

You can also check out the full code on my GitHub repository here.

Let’s Begin!

To create a good customer segmentation, I will be using the famous Online retail dataset by UCI Machine Learning Library. The data set contains all the transactions occurring between 01/12/2010 and 09/12/2011 for a UK-based and registered non-store online retail. It has 8 columns and 541909 rows.

The data set has attributes like- InvoiceNo, StockCode, Description, Quantity, InvoiceDate, UnitPrice, CustomerID and Country. But, we will be mainly making use of features like CustomerID, UnitPrice, Quantity, InvoiceDate etc and also we would need to make new features as well to calculate the RFM score and those features will be used further for finding the similarity to assign each customer to a cluster using k-means.

Importing Python Libraries and dependencies

import pandas as pd

import numpy as np

import matplotlib.pyplot as plt

import seaborn as sns from scipy import stats

from sklearn.preprocessing import StandardScaler

from sklearn.manifold import TSNE

from sklearn.cluster import KMeans import plotly.offline as pyoff

import plotly.graph_objs as go import datetime as dt

import feature_engine from feature_engine.outliers import Winsorizer import warnings

warnings.filterwarnings("ignore")

Data Preparation/Data Cleaning

First, we need to clean the data to make it into a more usable format.

Reading the data into a pandas DataFrame.

There are 8 columns and 541909 rows in our data.

Description of the DataSet

Created a custom function that returns the desc, shape and data types of our data.

We can see that there are many nulls values in CustomerID. CustomerID is one of the main features. We need to remove them as there is no way we can get the number of CustomerID

There are few records with UnitPrice<0 and Quantity<0. We need to remove them from the analysis.

UnitPrice<0

Quantity<0. InvoiceNo starting with ‘C’

The min and max value for Quantity is 80995, this could represent cancelled or returned orders.

The UnitPrice also have few negative values which are uncommon, these transactions could represent cancelled orders by customers or bad debt incurred by the business.

Bad debt adjustments will be dropped from the dataset as these do not represent actual sales.

There are about 25% of Null CustomerID in the data.

We will be using the K-means algorithm in this project later and k-means is not able to deal with missing values. Therefore, We will drop those records with missing CustomerID.

The proportion of Customers- country wise

As customer clusters may vary by geography, we will restrict the data to only United Kingdom customers, which contains most of our customer’s historical data (91% of customers)

df = df[df.Country == 'United Kingdom']

Removing the negative values from UnitPrice and Quantity

df = df[df.Quantity > 0]

df = df[df.UnitPrice > 0] #Removing the Null values from the data.

df = df[pd.notnull(df['CustomerID'])]

Cleaning the Date Column

df['InvoiceDate'] = pd.to_datetime(df['InvoiceDate'])

df['InvoiceYearMonth'] = df['InvoiceDate'].map(lambda date: 100*date.year + date.month)

df['Date'] = df['InvoiceDate'].dt.strftime('%Y-%m')

Now, all our data is clean.

We have all the crucial information we need:

- Customer ID

- Unit Price

- Quantity

- Invoice Date

Exploratory Data Analysis

#Function to plot

def plot_df(df, x, y, title="", xlabel='Date', ylabel='Orders', dpi=100):

plt.figure(figsize=(16,5), dpi=dpi)

plt.gca().set(title=title, xlabel=xlabel, ylabel=ylabel)

plt.plot(x, y, color='tab:Blue', marker='o')

plt.show() #plotting

plot_df(df_agg, x=df_agg.Date, y=df_agg.Quantity,title='Orders in 2011')

Number of Orders throughout 2011

The number of orders started increasing from the third quarter and the maximum number of orders are in November

Calculating Revenue

Revenue = Order Count * Average Revenue per Order

df['Revenue'] = df['Quantity']*df['UnitPrice']

Monthly Revenue

To plot the graph for Revenue monthly