How To Acquire New Customers Utilizing Machine Learning?
By: Laila Shahreen
Tags: Unsupervised Learning, Machine Learning, Data Science, Clustering

Customer Segmentation Part

This part contains data understanding, preprocessing of demographics of general population data(cleaning, feature scaling and building proper cleaning functions so that used for any similar dataset) and make them ready for fitting into model. Let’s look into the datasets and features information files for clear perception.

Data:

There are four data files associated with this project:

Udacity_AZDIAS_052018.csv: Demographics data for the general population of Germany; 891 211 persons (rows) x 366 features (columns).

Udacity_CUSTOMERS_052018.csv: Demographics data for customers of a mail-order company; 191 652 persons (rows) x 369 features (columns).

Udacity_MAILOUT_052018_TRAIN.csv: Demographics data for individuals who were targets of a marketing campaign; 42 982 persons (rows) x 367 (columns).

Udacity_MAILOUT_052018_TEST.csv: Demographics data for individuals who were targets of a marketing campaign; 42 833 persons (rows) x 366 (columns).

The CUSTOMERS file contains three extra features (CUSTOMER_GROUP, ONLINE_PURCHASE, and PRODUCT_GROUP), which provides broad information about the customers depicted in the file.Furthermore two descriptive files were provided to give further insight into the features of both datasets:

DIAS Information Levels — Attributes (2017): provides information level, description and additional notes for each feature

DIAS Attributes — Values (2017): provides description and a value/meaning map for each feature.

The attributes’ files were helpful to fathom the properties of the features. There was mismatch in description with demographics datasets’ features and features attributes’ files and many features were not available in the later file. That made it a laborious task to identify all the attributes and store them as a Features_info.csv file. Finally it was a manual task to create such file with the help of DIAS Information Levels — Attributes (2017) file and DIAS Attributes — Values (2017) file.It sorts the features alphabetically, and maps both a type (categorical, mixed, numeric, ordinal) and values corresponding to missing information to each feature.

Data Preprocessing

General population data was transformed into a clean one by the following steps. There are six steps and the first five steps were put into a function to utilize in customers datasets for further analysis.

Step 1: Drop not described features

After looking into the details of the Features_info.csv file, the missing attributes were dropped from azdias data frame. There were 334 feature columns available after cleaning.

Step 2: Convert missing values

Next step was to find out missing values in azdias data frame using the features_info.csv file that contains all values corresponding to missing information. These placeholders have to be converted to NaN values to the already existing NaN values in the datasets.

Step 3: Drop features with high percentage of unknown values

At this step, I dropped the columns those have more than 20% missing value. Since these columns are not many in numbers compared to other portions of the data frame, 20% was a good threshold for doing such elimination of features. The following figure explains better reason.

Figure1. Features with missing value percent

Step 4: Drop rows with high percentage of unknown values

The above process was followed in identifying missing value in each row as well. Then the dataset was split and saved into two datasets containing above and below the threshold of missing value (30% in this case of row) in the original dataframe. The below figure shows the details.

Figure2. Features with missing value in row

Step 5: Re-encode features

The remaining dataset contains four types of features: categorical, mixed, numeric and ordinal. The majority of the features are numeric or ordinal and could be left without re-encoding.

An assessment of the 26 categorical features showed that four of them were dropped during earlier preprocessing steps already. Three more (CAMEO_DEU_2015, LP_FAMILIE_FEIN, LP_STATUS_FEIN) could be dropped from the original dataset because of redundancy (fine and rough feature available). Only one feature (OST_WEST_KZ) had to be one hot encoded. All others had numeric values and weren’t changed for simplification reasons.

Furthermore the four mixed features were assessed. Feature PRAEGENDE_JUGENDJAHRE was split into the new features DECADE and MOVEMENT, as well as feature CAMEO_INTL_2015 was split into WEALTH and LIFE_STAGE. The information contained in the two remaining mixed features (LP_LEBENSPHASE_FEIN, LP_LEBENSPHASE_GROB) was redundant to other features and somehow not clearly structured, therefore it was decided to drop both features.

Step 6: Imputing and Scaling

Now it is time for imputation since it ensures that there will not be any missing information so that our machine learning model can work purposefully.Missing values were imputed with the median of the corresponding feature. Because of the majority of categorical and ordinal features it is superior to imputing mean values. Afterwards the features were standardized.

Complications

Imputing missing values with NaN was not an easy and fast task.The first approach was iterating manually over each column and row: its drawbacks were source code complexity and bad performance. Finally the mask function of pandas’ DataFrame solved both problems.