Segmentation of Online Shop Customers
By: Andreas Stöckl
Tags: Visualization, Data Science, Online Marketing, Ebusiness, Machine Learning

Segmentation of Online Shop Customers

With Web Analytics Data and k-Means Clustering

Identifying clusters of similar Customers

In this article I will describe how we can segment customers based on web analytics data from an online shop. Based on the results, on-site personalization can be realized and targeted campaigns can be started for the users in the segments.

On the way there, we will first explore the data in more detail (“Explorative Data Analysis”), then do suitable preprocessing of the data, calculate the segmentation, and finally visualize the clusters. For the calculations we will use Google Colab.

Data

www.kaggle.com

The data is from the Kaggle data platform and contains web tracking data for one month (Oct. 2019) from a large multi-category online shop. Each line in the file represents an event. There are different types of events, such as page views, shopping cart actions, and purchases.

The record contains information about:

event_time / When was the event triggered? (UTC)

event_type / view, shopping cart, purchase

product_id / product ID

category_id / category ID

category_code / category name

brand / brand name

price / price

user_id / customer ID

user_session / session ID

The data is available as a CSV file as an export from a customer database platform for analysis.

Let’s import the data:

First look at the data

There are over 42 million records available for the month of October 2019.

First 10 records of the dataset

This data comes from over 3 million visitors. Over 166,000 different products were purchased.

Example Customer Journey

To show an example of a customer visit, we look at all entries that exist for a certain Session_id and try to interpret them:

Example Customer Journey

The user has viewed several iPhones

An iPhone purchased with 1 click (without shopping cart event)

Considered 2 unknown products of the brand arena

Viewing some Apple headphones and buying one

Afterward he visited a more expensive one, but decided not to buy it.

Example Customer History

To view all actions of a specific user in that month, we filter all records with his User ID.

Example Customer History

Explorative data analysis

How many events were recorded in the web analysis on each day?

Number of events recorded over time in Oct. 2019

Number of event types

Which events occur in the data, how often?

Type of events

The majority of the data consists of page views 96%, the rest of the data consists of shopping cart and purchase actions.

Features of the visitors

We calculate the most important features for each visitor and put them together in a table.

Pageviews

Visits

Number of purchased products

Number of products in the shopping cart

Total expenditure

Expenditure per visit

Pageviews per visit

Shopping cart actions per visit

We filter the purchases from the actions

In the next step, we filter the purchases from the data in order to be able to analyze them more precisely. We save the result in a separate table.

Key figures on purchases

How many products are purchased by one buyer?

What is the average purchase value per buyer?

On average, each buyer makes slightly more than 2 purchases.

The average buying value per buyer is 773.85.

Brand popularity

From which brands are products bought?

Let’s look at a bar chart of the top 10 brands.

Popular Brands

For further analysis, we group the purchases into groups of the most common brands (the top 5). And the rest into a group “Others”.

We calculate for each buyer the share of purchases in the 6 brand categories, and store them in the buyer table.

Product categories

Which product categories are available?

The product category exists in the form of a hierarchical code. We extract the first level and save it as a separate characteristic.

Purchases per top-level category

There are 13 main categories. We add the share of the purchase price in each of the main categories as additional features to the table of buyers.

Adding purchase characteristics to the characteristics of all visitors

We now add the purchase characteristics to the table of all visitors, and thus obtain a table with all visitors and characteristics.

So we have the data of 3,022,290 users, of which we have stored 27 characteristics each.

Limitation of the number of users

In order to keep the calculation of the clusters and the visualization within limits, we will limit ourselves to the first 50,000 users in the following.

Conversion to matrix format for cluster calculation

Before we can start the calculations for clustering, we have to convert the data into the appropriate format as a 2-dimensional array.

Scaling of the data

To ensure that all characteristics are present on a uniform size scale, the matrix is scaled by shifting by the mean value and dividing by the standard deviation.

Calculation of customer segments with different numbers of clusters

The “k-Means method” is used to calculate the segments. It is a method used for cluster analysis, where a set of objects has to form a number of k groups, which have to be given in advance.

Since we are dealing with a very large amount of data, we use the “mini-batch” variant of the procedure, which uses only part of the data in each iteration to calculate the new cluster centers.

How to set the optimal number of clusters (“k-value”)?

We calculate the clustering for different k-values, and then search for the best value. The calculated silhouette score is a measure for the quality of the clustering. The closer the value is to 1, the better the quality of the clustering. We use it to determine the number of clusters.

Silhouette score for different k values

Now we calculate the clusters with the determined optimal cluster number.

And consider the number of customers allocated to each segment.

Cluster sizes

Visualization of the clusters

To get an impression of the clustering, we create a visualization with the method “tSNE”. t-Distributed Stochastic Neighbor Embedding (tSNE) is a technique for dimensionality reduction and is particularly well suited for the visualization of high-dimensional data sets.

Cluster visualization with tSNE

Now let’s calculate the visualization with a much smaller number of clusters. It is much more difficult for the procedure to separate the individual areas into different segments.

tSNE with 5 clusters

Characterization of the segments

In order to enable an interpretation of the segments, we create graphical representations that show, for example, the characteristics of categories for each segment at once as a “radar chart”. This can help to interpret the meaning of the segments.

Categories in clusters

For example, a segment that has high purchase shares in the area of “Children” and “Sports”, and others with purchases in “Electronics”.