Arvato Solutions Customer Segmentation and Prediction
By: Julian Gomez
Tags: Udacity, Oversampling, Data Science, Supervised Learning, Unsupervised Learning

Part 3 — Supervised Learning

After performing cluster analysis on the German demographics and customer data sets to determine which customers are more likely to be customers of the mail-order company, the next stage is to apply supervised learning to a separate data file. This data set is like the previous two, but every individual was targeted for a mail-out campaign. Furthermore, it includes a response variable that represents whether an individual became a customer of the mail-order company after the campaign. That being said, the data serves as a training set that will be used to build an optimal predictive algorithm for a separate data set for testing, which will be used in a Kaggle competition.

Figure 10 — Customer response frequency in training data (image by author)

As the training data set has almost 43,000 samples, figure 10 shows that the response variable is significantly unbalanced. In other words, about 1.24% of the targeted individuals became customers of the company. Since this is the target variable that the prediction models will use, it would be a challenge to classify whether an individual will become a customer in the testing data. A way to remedy this is by implementing sampling methods in addition to data preprocessing and training/testing splits before training the predictive models.

Data Preprocessing

Once the 300 predictors and the response variable are separated from each other, the mail-out training data was split into a training, validation, and testing set based on a validation ratio of 0.1 and a test ratio of 0.2. These three splits serve to assess the candidate models before using the best one on the entire mail-out data.

Following the splits, the training, validation, and testing sets were each transformed with imputation, standardization, and PCA reduction to 154 components. For this analysis, these processed data sets will be used to assess three prediction algorithms: logistic regression, AdaBoost, and gradient boosting classifiers. The process consists of performing cross-validation on the models using a ROC score, rather than an f1-score, but not before implementing an additional modification to the training data.

Resampling — ADASYN

Before implementing models on significantly unbalanced data, a resampling method will be essential for an accurate model. The minority response — RESPONSE = 1 — is so rare that an arbitrary model would learn close to nothing about the outcome, resulting in a weak model. To make the positive response more important, the models need to get to know it better, that’s where oversampling comes in.

The two possible approaches to resampling are undersampling and oversampling. Undersampling consists of balancing the data such that observations of the majority class — RESPONSE = 0 — are randomly reduced to the same size as that of the minority class. However, for this data and in several applications, a major downside is that significant omissions of majority class observations result in the loss of important information, which could be crucial for the learning process of a model. Oversampling, on the other hand, generally duplicates minority class samples to the same size of the majority class instead of omitting samples from the majority class. However, this tactic could most likely overfit the data due to high repetitions of minority samples¹.

To pick between the two resampling tactics, oversampling is better since that way more information is kept rather than dropped². Even so, the imblearn package offers a few enhanced oversampling methods that reduce the downside of general oversampling.

Figure 11 — KNN generation of a minority sample for ADASYN (image by imblearn)

A good technique used is ADASYN — Adaptive Synthetic Sampling Approach. This resampling technique aims to oversample the minority class samples by generating synthetic versions of them. Consider a sample x_i, which becomes a new sample x_new. This new sample is generated through the original sample’s k-nearest neighbors. From all possible neighbors, x_zi, is selected and the new observation is calculated as follows:

Where lambda is a random number between 0 and 1. This new sample generally lies on the line between x_i and x_z ³. Figure 11 illustrates this process. Furthermore, ADASYN uses a density distribution as a criterion to decide the number of synthetic samples for the minority samples. It does so by adaptively changing the weights of the different minority samples while taking into account any skewed distributions⁴.

The ADASYN resampling was fit on the split training data, resulting in a total of over 61044 samples rather than 30,000.

Logistic Regression

Figure 12 — Logistic regression classification report for validation and testing sets (image by author)

From a default logistic regression model on the resampled training data, the validation and testing scores are similar to one another. The recall score is 0.43 and the precision score is 0.2 for the positive responses. This means that 43% of the individuals who are actually customers were classified correctly as 2% of individuals who were predicted as customers were actually customers. The validation ROC score is 0.53.

For the testing data, 55% of actual customers were predicted correctly and 2% of the predicted customers were correctly classified. The ROC score is 0.59, higher than that of the validation set.

AdaBoost Classifier

Figure 13— AdaBoost classification report for validation and testing sets (image by author

From a default AdaBoost model on the resampled training data, the validation and testing scores are similar to one another. The recall score is 0.26 and the precision score is 0.2 for the positive responses. This means that 26% of the individuals who are actually customers were predicted correctly and 2% of individuals who were predicted as customers were actually customers. The validation ROC score is 0.54.

For the testing data, 26% of actual customers were predicted correctly and 2% of predicted customers were correctly classified. The ROC score is 0.54, about the same but slightly higher than that of the validation set.

Extreme Gradient Boosting Classifier

Figure 14— Extreme Gradient Boosting classification report for validation and testing sets (image by author)

From a default extreme gradient boosting model on the resampled training data, the validation and testing scores are similar to one another. The recall score is 0.07 and the precision score is 0.1 for the positive responses. This means that 7% of the individuals who are actually customers were predicted correctly and 1% of individuals who were predicted as customers were actually customers. The validation ROC score is 0.47, which is less than the other two models.

In the testing data, 10% of actual customers were predicted correctly and 1% of predicted customers were correct. The ROC score is 0.49, which is higher than that of the validation set.

Model Selection

Based on the preliminary information on the general models above, if one model can be picked, the best one moving forward would be the logistic regression model. Since the precision scores are the same for all three models through validation and testing, another metric to look at would be the recall and ROC scores. Focusing on the positive response, even if it is the minority class, logistic regression has the highest recall and ROC scores in the testing and validation sets.

However, for this analysis, the goal of the supervised model is not to pick which model generally predicts best. Although extreme gradient boosting could give a ROC score higher than logistic regression, it turned out to be the lowest and significantly lower than that of the logistic regression. So, does this mean that logistic regression would give a higher ROC score than extreme gradient boosting on the real testing set? To find out, the three listed models would need to be cross-validated and optimized to see if the application to the mail-out testing data will give a higher score to logistic regression than the other two ensemble models.

Cross-Validation Model Tuning

Logistic Regression

Once a generic logistic regression model is initiated, it will be applied through a grid search cross-validation method to find the optimal parameters for the entire training data. The general model is defined in a pipeline containing the three data transformations, and the ADASYN oversampling in addition to the model. The set of parameters for this pipeline model consists of three separate instances.

lr_parameters = [

{'model__penalty':['l1'],

'model__C': np.arange(0.1, 1.1, 0.1),

'model__max_iter':[300, 500, 800, 1300],

'model__solver':['saga']

},



{'model__penalty':['l2'],

'model__C': np.arange(0.1, 1.1, 0.1),

'model__max_iter':[100, 200, 300, 500, 800],

'model__solver':['newton-cg','lbfgs']

},



{'model__penalty':['l2'],

'model__C': np.arange(0.1, 1.1, 0.1),

'model__max_iter':[300, 500, 800, 1300],

'model__solver':['sag']

}



]

Logistic regression accepts four different penalties, but for this case, two will be used — l2 (the default) and l1. The parameter dictionary structure consists of one l1 set and two slightly different l2 sets. Each one contains the C coefficient ranging from 0.1 to 1.1 by 0.1, max iterations ranging from 100 to 1300 depending on the penalties, and different solvers. The first instance corresponds to l1, which only accepts ‘saga’ or ‘liblinear’ solvers. Since this model is going to be applied to a large data set, only ‘saga’ will be used. The second instance is for l2 with ‘newton-cg’ and ‘lbfgs’. The third instance is l2 with solver ‘sag’, which handles large data sets with maximum iterations of at least 300. After the grid search was complete, logistic regression with a coefficient of 0.8, 1300 maximum iterations, and a ‘sag’ solver yielded a ROC score of 0.63.

LogisticRegression(C=0.8, max_iter=1300, solver='sag')

As a result, the prediction of this algorithm on the mail-out testing set resulted in 37.6% of individuals predicted as customers. The final submissions gave a score of 0.609.

AdaBoost Classifier

Contrary to logistic regression, the AdaBoost parameter search will be conducted through a randomized search. Although there are two parameters to search through, several fits can last a long time to fit. The learning rate values are organized like a Fibonacci sequence from 0.1 to 0.8, including the default value of 1. The number of estimators is organized the same way from 100 to 300, plus the value of 50.

ab_parameters = {'model__learning_rate': [0.1, 0.2, 0.3, 0.5, 0.8, 1],

'model__n_estimators': [50, 100, 200, 300]

}

The randomized search on the AdaBoost classifier consists of 12 iterations where the model with the highest score has a learning rate of 0.2 and 200 estimators.

AdaBoostClassifier(learning_rate=0.2, n_estimators=200,

random_state=0)

The prediction of this algorithm on the mail-out testing set resulted in 22% of individuals predicted as customers. The final submissions gave a score of 0.579.

Extreme Gradient Boosting Classifier

The parameter search for this model is different from the other two. Since extreme gradient smoothing is more complex than AdaBoost both in parameters and efficacy. The pipeline is defined the same way as in the other two models. Out of the 27 different parameters for this algorithm, I chose the five most important ones — maximum depth, alpha, subsample, number of trees, and learning rate.

xgb_parameters = {

'model__max_depth': np.arange(3, 13, 1),

'model__alpha': np.arange(0.1, 0.8, 0.05),

'model__subsample': [None, 0.1, 0.25, 0.5, 0.75, 1],

'model__n_estimators': np.arange(10, 200, 10),

'model__learning_rate': np.linspace(0.01, 0.8, 10)

}

After defining the parameters, the cross-validation will be performed through a randomized search instead of a grid search. The XGBoost parameters defined here would cost a lot of time and computational effort. Therefore, a limited but randomized search would improve both of those issues with a certain number of iterations. For this randomized search, 45 iterations were set to increase the chances of capturing an optimal model, since there would be hundreds of combinations here that could last for days with a grid search. In the end, the optimal XGBoost classifier has the following parameters, in addition to the default parameters that were not in the search.

XGBClassifier(alpha=0.7, base_score=0.5, booster=None, colsample_bylevel=1,

colsample_bynode=1, colsample_bytree=1,

gamma=0, gpu_id=-1, importance_type='gain',

interaction_constraints=None,

learning_rate=0.09777777777777778,

max_delta_step=0, max_depth=3,

min_child_weight=1, missing=nan,

monotone_constraints=None, n_estimators=90,

n_jobs=0, num_parallel_tree=1, random_state=0,

reg_alpha=0.699999988, reg_lambda=1,

scale_pos_weight=1, subsample=1,

tree_method=None, validate_parameters=False,

verbosity=None)

The prediction of this algorithm on the mail-out testing set resulted in 20% of individuals predicted as customers. The final submissions gave a score of 0.587.

Out of the three models, the model that attained the highest score on the Kaggle competition turned out to be the logistic regression model. Just like the basic classification reports.