Customer Segmentation Tutorial: Data Science in the Industry
By: Andre Ye
Tags: Data Analysis, Machine Learning, Statistics, AI, Data Science

Step 1: Loading and Cleaning the Data

The data can be loaded with the pandas library, using ISO-8859–1 encoding.

import pandas as pd

data = pd.read_csv('/kaggle/input/ecommerce-data/data.csv',encoding='ISO-8859-1')

data.head()

Calling len(data) yields 541909 , meaning that there are 541,909 rows in the data. Running len(data[‘CustomerID’].unique()) returns 4373 , meaning that there are 4,373 distinct customers. Ideally, our final data will have 4,373 rows.

We need to deal with missing values. By checking len(data) — len(data.dropna()) , which returns the number of rows that would be dropped if we deleted any row that had a na value. This outputs 135080 , meaning that between 1/5 and 1/4 of the data would be missing if we simply dropped any row with a na value.

So where are most of these missing values coming from? The answer: missing CustomerIDs . The code data[‘CustomerID’].isnull().sum() , which checks how many missing values there are in the column CustomerID, returns 135080 . All of the rows that would have been dropped using data.dropna() had a missing CustomerID .

Normally, we would want to impute the data when so much of it has missing values. However, due to the nature of IDs, which are put on a linear scale (as numbers) but are inherently classes, imputing methods like averaging or taking a median won’t work. We may want to try KNN, but there isn’t much data for it to work off of, and may be inaccurate.

For the purposes of this project, I’ll be dropping all rows that have na values. However, this may not be wise in a serious business project because of selection bias — say that all the customers whose IDs are na belong to some additional cluster or provide valuable information that is discarded because their IDs are unknown. In a more formal business setting, it may be wiser to try and impute the IDs through a method like Decision Tree, training the classifier to recognize a customer’s ID based on its purchases. For this scenario, a high-variance algorithm needs to be used due to the astronomical number of classes.

data = data.dropna()

data.head()

After dropping the missing values, our data is len(data) = 406829 rows long. That’s still a considerable amount of data.

The InvoiceNo and StockCode columns are irrelevant. We can remove them:

data.drop(['InvoiceNo','StockCode'],axis=1,inplace=True)

data.head()

Next, we’ll be vectorizing the Description column to cluster products into groups. To prepare the data for this, we’ll need to make the words lowercase and remove any punctuation, for example in row 4 with “RED WOOLY HOTTIE WHITE HEART.”

Let’s make a function that cleans the description. It uses regular expressions to remove punctuation and lowers capital letters.

import re

def clean(description):

return re.sub(r'[^\w\s]','',description).lower()

As a demonstration of the function:

It’s ready to go! Pandas’ .apply function lets us apply a function to a column.

data['Description'] = data['Description'].apply(clean)

data.head()

Great! The text is ready to vectorize. Two last things — we need to reset the index and quickly create a new column. Because we removed all na values, there are going to be some holes in the indexes. If let untamed, it will cause problems for us in the future.

data = data.reset_index().drop('index',axis=1)

In the future, we will need to know the total amount of money spent on a product — rephrased, the unit cost times the quantity of items.

data['Total'] = data['UnitPrice'] * data['Quantity']

We’re ready to move on!