Using Machine Learning to Identify Customer Segments
Tags: Unsupervised Learning, Machine Learning, Data Science, Customer Segmentation

Project Overview and Problem Statement

Identifying customer segments is an important and popular application in business and data science, in which being able to identify possible consumers can help corporates use their resources more effectively when making marketing decisions.

The goal of this project is to identify customer segments that can be used as targeted marketing groups for a mail-order company based in Germany. The main hypothesis of this project is that overrepresented clusters in the customer dataset would indicate the core user base of the mail-order company, and the general population with the same characteristics of those within the overrepresented cluster can be targeted as possible consumers.

Data Understanding

There are four main datasets associated with this project:

Udacity_AZDIAS_Subset.csv: Demographics data for the general population of Germany; 891211 persons (rows) x 85 features (columns).

Udacity_CUSTOMERS_Subset.csv: Demographics data for customers of a mail-order company; 191652 persons (rows) x 85 features (columns).

Data_Dictionary.md: Detailed information file about the features in the provided datasets.

AZDIAS_Feature_Summary.csv: Summary of feature attributes for demographics data; 85 features (rows) x 4 columns

Each row of the demographics files represents a single person, but also includes information outside of individuals, including information about their household, building, and neighborhood. I used this information to cluster the general population into groups with similar demographic properties.

Data Preparation

Assessing Missing Data by Converting Missing Value Codes to NaNs

I started the process of data preparation by converting the missing value codes to NaNs in the actual dataset. The codes for missing or unknown values were obtained from the document of detailed information on the features. After converting all the missing or unknown values in the dataset to NaN, I wanted to work on identifying the pattern of missing values in the dataset. First, I wanted to find how much missing data is present in each column, so I worked on finding the outliers columns, which are columns that are in different proportions in terms of values that are missing.

Investigating patterns in the amount of missing data in each column

After visualizing the distribution of missing value, we can see that there are some features with missing data more than most of the attributes in the dataset. For this dataset, it seems that the outliers are the features that have above 200,000 missing data.

Based on the observation above, I made the decision of removing the following columns: ‘AGER_TYP’, ‘GEBURTSJAHR’, ‘TITEL_KZ’, ‘ALTER_HH’, ‘KK_KUNDENTYP’, and ‘KBA05_BAUMAX’, as they had above 200,000 missing data.

Second, I wanted to assess the missing values in each row:

After visualizing the distribution of missing values in each row, I found the number of rows with many missing values to be 93,154 or 10.45 % of all data, I sat the threshold to be 30.

After dividing the data into two subsets, one for data points that are above 30 missing values, and a second subset for points below the threshold, I decided to visualize the distributions of non-missing features between the data with many missing values and the data with few or no missing values. I selected 5 columns to plot the comparison.

From the analysis above, I came to the conclusion that the distribution of data with many missing values is very different from data with few or no missing values, meaning that the data could be special and should be kept for now.

Selecting and Re-Encoding Features

Since the unsupervised learning techniques to be used will only work on data that is encoded numerically, I had to make a few encoding changes or additional assumptions to be able to make progress.

Re-encode binary categoricals that have non-numerical values.

Drop Multi-level categoricals from the dataset.

Re-engineer mixed-type features by constructing new variables.

Feature Transformation

Applying Feature Scaling

Before I apply dimensionality reduction techniques to the data, I needed to perform feature scaling so that the principal component vectors are not influenced by the natural differences in scale for features. So I used imputer to deal with NaN values by replacing them with the median value and then applied scalar transformation to normalize all the values in the dataset.

Performing Dimensionality Reduction

On the scaled data, I applied principal component analysis, which is a dimensionality reduction technique, in order to find the vectors of maximal variance in the data.

From the analysis above, looking at the variance explained by each component, I found that 95% of the variability appeared in the first 50 components, thus, I decided to select only the top 50 components for PCA.

Interpreting Principal Components

I wanted to further investigate the components by checking out the weight of each variable on the first few components. So I mapped each weight to their corresponding feature name, then sorted the features according to the weight. The most interesting features for each principal component, then, were those at the beginning and end of the sorted list as they represent the positive and negative correlation with the component respectively.

Component 1

Positive

PLZ8_ANTG3 (Number of 6–10 family houses in the PLZ8 region). CAMEO_INTL_2015_Wealth (Wealth and financial status of households) EWDICHTE (Density of households per square kilometer) ORTSGR_KLS9 (Size of community) FINANZ_SPARER (Financial typology)

Negative

KBA05_GBZ (Number of buildings in the microcell) KBA05_ANTG1 (Number of 1–2 family houses in the microcell) PLZ8_ANTG1 (Number of 1–2 family houses in the PLZ8 region) MOBI_REGIO (Movement patterns) FINANZ_MINIMALIST (Financial typology)

From the graph, we can conclude that component 1 is correlated with the wealth and financial status of a household, in addition to the size of house and family number. We can also assume that component 1 has negative correlations with financial saving habits.

Component 2

Positive

ALTERSKATEGORIE_GROB (Estimated age based on given name analysis) FINANZ_VORSORGER (Financial typology) SEMIO_ERL (Personality typology) SEMIO_LUST (Personality typology) RETOURTYP_BK_S (Return type)

Negative

SEMIO_PFLICHT (Personality typology) FINANZ_UNAUFFAELLIGER (Financial typology) FINANZ_SPARER (Financial typology) SEMIO_REL (Personality typology) PRAEGENDE_JUGENDJAHRE_decade (Dominating decade of person’s youth)

For component 2, we can see that age, personality traits and financial behaviors are correlated with Component 2, as it seems that financial behaviors such as purchase return type and financial prevention behaviors have a positive relationship with our component.

Component 3

Positive

SEMIO_VERT (Personality typology) SEMIO_SOZ (Personality typology) SEMIO_KULT (Personality typology) SEMIO_FAM (Personality typology) PLZ8_ANTG4 (Number of 10+ family houses in the PLZ8 region)

Negative

SEMIO_ERL (Personality typology) SEMIO_KRIT (Personality typology) SEMIO_DOM (Personality typology) ANREDE_KZ (Gender) SEMIO_KAEM (Personality typology)

For component 3, we can see that the personality traits have a big influence on component 3, as it appears on both the positive and negative side of the graph.

Clustering Using K-Means

K-means algorithm is an unsupervised machine learning algorithm that identifies the number of centroids and then allocates every data point to the nearest cluster while keeping the centroids as small as possible. To determine the optimal number of clusters into which the data may be clustered, I used the elbow method. However, from the analysis above it, appeared that there is no clear elbow, so I decided to go with 20 clusters as that was where it seemed to decrease endlessly.

Comparing Customer Data to Demographics Data

Finally, I applied all the previous steps on the customer dataset to compare the results with the general population. I wanted to find which cluster or clusters are overrepresented and underrepresented in the customer dataset compared to the general population. I did that by selecting one overrepresented cluster and one underrepresented cluster and found the main characteristics of the people within each group.

From the analysis above, we can see that cluster 1 is overrepresented in the customer data, the main characteristics of the mail-order company customers are as follow:

Estimated age based on given name analysis is between 30–45 years old (ALTERSKATEGORIE_GROB = 2.394718).

Gender is male (ANREDE_KZ = 1.574314)

A person whose minimalist habits are high (FINANZ_MINIMALIST = 2.890335)

Also from the previous analysis, we can see that cluster 8 is underrepresented in the customer data, the main characteristics of the customers are as follow:

Estimated age based on given name analysis is younger than 30 years old (ALTERSKATEGORIE_GROB = 1.882568).

Gender is male (ANREDE_KZ = 1.928417)

A person whose minimalist habits are very high (FINANZ_MINIMALIST = 1.093893)

Conclusion

In this project, I applied unsupervised learning techniques to identify segments of the population that form the core customer base for a mail-order sales company in Germany. These segments can then be used by Arvato to direct marketing campaigns towards audiences that will have the highest expected rate of returns.

I found this project to be interesting as it contains real data in which it has many areas that can be further explored. Additionally, I found data exploration and wrangling to be quite challenging but reflective of real-life data science applications.