Unsupervised and supervised learning for customer segmentation — Part 1
By: Alejandro Galindo Medina
Tags: Unsupervised Learning, Machine Learning, Supervised Learning, Customer Segmentation

Part 1 Unsupervised Learning

As mentioned, regarding the unsupervised model part, I used two datasets for the task:

azdias: Contains demographic information about the general population with shape of 891,221 rows and 366 columns

Contains demographic information about the general population with shape of 891,221 rows and 366 columns customers: Contains demographic information about company-only individuals with shape of 191,652 rows and 369 columns

Like most tasks that involve machine learning models, the first step one must go through is data cleaning. For this project, all the data cleaning steps were put inside a function, which is rather long but if interested, the function is contained inside the GitHub repo where all the code used for the project is located.

After all the cleaning was done, I proceeded to impute and scale the data. For the former, I decided to make the imputation based on the most frequent value as pretty much all the missing data columns were of ordinal type, and for scale, I decided to go with sci-kit-learn standard scaler class, which standardize features by removing their mean and scale them to unit variance, this step is very important as ML models tend to learn better from the data this way. This resulted in two data frames: df representing azdias after preprocessing, and df2 representing customers after preprocessing.

Before applying the model, a principal component analysis was performed on the data to reduce its dimensionality. To perform this task, I used sci-kit learn’s own PCA class, fitted it on df and plotted the explained variance per principal component in order to choose the number of features to shrink the data frame into.

# Initialize PCA and fit it on df

pca_init_time = time.time()

pca = PCA()

pca = pca.fit(df)

# Predict on df and df2

pca = PCA(n_components = 230)

pca = pca.fit(df)

df_pca = pca.transform(df)

df2_pca = pca.transform(df2)

After fitting PCA on the general population and plotted the explained variance per component, I chose 230 to be the final features for the dataset to shrink into, as they explain roughly more than 90% of the variance in the dataset as seen in the graph. The two resulting data frames after doing all the preprocessing and dimensionality reduction, were df_pca and df2_pca, the former representing the general population, and the latter representing the company’s own customers dataset.

Now for interesting part, to extract which parts of the general population are potential new customers to the company, I fitted the K-Means algorithm on df_pca with 25 different clusters, and predict this same fitted model on both df_pca and df_pca2 to later check which groups from the general population are the ones that over-represent and under-represent the data in the customers dataset.

# K-Means algorithm with 25 clusters on df_pca

model = KMeans(n_clusters = 25)

model = model.fit(df_pca) #Make cluster prediction on both df_pca and df2_pca

pred = model.predict(df_pca)

pred2 = model.predict(df2_pca)

With these two predictions, I proceeded and created one final data frame, containing information on the proportion of each of cluster (1–25) on each of pred and pred2 in order to see which cluster or group is the one that is over-representing and under-representing the customers dataset, in other words, this over-represented cluster will contain the demographic information of the individuals that are best suited to become company’s new customers, and the under-represented one will show otherwise.

def cluster_prop(predictions):

s_pred = pd.Series(predictions)

pred_counts = s_pred.value_counts(normalize = True)

ind = pred_counts.index.values

pred_counts = pd.DataFrame(columns = [‘Counts’],

data =pred_counts.values, index = ind)

pred_counts = pred_counts.assign(prop = lambda x: x/len(pred))

return pred_counts #Apply cluster prop on pred and pred2 to get their proportions

pred_df = cluster_prop(pred)

pred2_df = cluster_prop(pred2) #Create new dataframe containing both pred and pred2

comparative_df = pd.DataFrame({'azdias_cluster_prop': pred_df.prop,

'customers_cluster_prop': pred2_df.prop,

'Cluster': range(0,25)}) Plot comparative_df:

By watching the graph, cluster 7 clearly shows to be the one over-representing the customer's data, and cluster 6 the one that under-represents it. With these cluster numbers now on hand, the next step is to get the information contained inside each of them in order to create a report on what the characteristics of the individuals that are inside both groups look like.

In order to achieve this, I had to reverse-engineer the data contained inside df_pca to get the demographic information of the individuals that are part of the chosen clusters.

# Reverse engineering for cluster 7

pca_cluster = df_pca[pred == 7]inverse_pca = pca.inverse_transform(pca_cluster) inverse_scale = scaler.inverse_transform(inverse_pca)data = np.round(inverse_scale) df3 = pd.DataFrame(data = data, columns = df.columns) # Reverse engineering for cluster 6

pca_cluster = df_pca[pred == 6]inverse_pca = pca.inverse_transform(pca_cluster) inverse_scale = scaler.inverse_transform(inverse_pca)

data = np.round(inverse_scale) df4 = pd.DataFrame(data = data, columns = df.columns)

From df3 I chose these six features below to explain what kind of demographic information cluster 7 and cluster 6 contain:

ANREDE_KZ: Gender

HH_EINKOMMEN_SCORE: Estimated household net income

OST_WEST_KZ: Flag indicating east or west

Wealth_Encode: Level of income in each household

Life_Stage_Encode: Type of family

MOBI_REGIO: Moving patterns

I divided the analysis into two parts: Popular, containing information on the individuals that are most receptive to the company’s products or campaigns (cluster 7) and Unpopular, containing information of the least receptive individuals (cluster 6).

Popular:

Anrede_kz: Both men and women. (1: men, 2: women)

Hh_einkommen_score: Persons with very high and high income. (2,3)

Ost_west_kz: People from the west. (1)

Wealth_encode: Wealthy and prosperous households. (4,5)

life_stage_encode : Mature couples.(4)

Mobi_regio : Very little to none movement. (4,5)

Unpopular:

Anrede_kz: Women. (2)

Hh_einkommen_score: People with average and low income. (5,6)

Ost_west_kz: West living people. (1)

wealth_encode: Poorer and less affluent households. (1,2)

life_stage_encode : Pre-Family Couples & Singles.(1)

Mobi_regio : High mobility and very high mobility. (1,2)

Conclusion:

Popular: The best individuals to target after the unsupervised learning model can be described as: mature to elder couples that have considerable income and wealth, and are well settled.

Unpopular: The worst individuals for the company to target are: pre-family couples or singles (women) that are young, don’t have much income and wealth and have high mobility.

A more in-depth analysis can be made by accessing information from the rest of the features, which can show more detailed information on each of the clusters, but for time's sake, only these five features were presented.

To check Part 2 you can click here.