Unsupervised and supervised learning for customer segmentation — Part 2
By: Alejandro Galindo Medina
Tags: Unsupervised Learning, Machine Learning, Supervised Learning, Customer Segmentation

The Task

The evaluation metric used for this competition was the AUC score, which tells us how well a model can distinguish between two diagnostic groups, in our case would be the individuals that are prospects of becoming new members to the ones that are not.

The dataset involved is composed of 42,962 observations (individuals) and 367 features (characteristics) from which, only 532 observations where people that became actual customers, which is extremely low, to put it in context, this would mean that the probability of picking one person at random, send him the campaign, and him then becoming a new customer is about 0.012 percent!

With this, we can say that the dataset is really imbalanced, and to illustrate this more, after preprocessing (which you can find the code for it here) I decided to build a Decision Tree Classifier and apply it into the data, which would usually perform well in this type of tasks, but the AUC score that I got from the model was only about~0.51, which is almost the lowest anyone can get from the metric, meaning that the model is doing awful at trying to distinguish one class from another.

#Untuned Decision Tree Classifier

DT = DecisionTreeClassifier()

DT.fit(X_train, y_train)

pred_dt = DT.predict_proba(X_test)[:,1]

roc_auc_score(y_test, pred_dt) Result:

Untuned Decision Tree score: 0.51404593009540434

The Model

To address this issue, another approach must be taken. The way I did it, was to train several algorithms, adjust their class_weight hyperparameter when possible, and check their AUC performance, in order to build a more robust model with the information gathered by them.

An extra note here: As the data is imbalanced, one can make use of re-sampling techniques, I personally tried a few approaches, but with no luck, so a decided and went with the presented model, which at the end, gave me the results i was looking for.

# Initialize the classifiers

logreg = LogisticRegression(random_state=123, penalty = 'l1',

C = 4.0, class_weight={0:1, 1:41})

Ada = AdaBoostClassifier(learning_rate = 0.1, random_state = 123)

NB = GaussianNB()

Tree = DecisionTreeClassifier(min_samples_split = 10,

min_samples_leaf = 15, random_state =42,

max_depth=5, class_weight='balanced')

RF = RandomForestClassifier(n_estimators=300, max_depth=5,

max_features = 'sqrt', min_samples_leaf=15,

min_samples_split=8,random_state = 123,

class_weight={0:1, 1:41})

GBoost = GradientBoostingClassifier(n_estimators=1000,

learning_rate=0.003, max_depth=5,

max_features = 'sqrt', min_samples_leaf=15,

min_samples_split=10, loss='deviance',

random_state = 42)

GBoostr = GradientBoostingRegressor(random_state = 42)

Lasso = make_pipeline(RobustScaler(), Lasso(alpha =0.0005,

random_state=1)) #function for getting classifiers auc

def fit_pred_auc(model):

model.fit(X_train, y_train)

pred = model.predict_proba(X_test)[:,1]

auc = roc_auc_score(y_test, pred)

return auc #function for getting linear model auc

n_folds = 5

def auc_cv(model):

kf = KFold(n_folds, shuffle=True,

random_state=42).get_n_splits(X_train.values)

auc= cross_val_score(model, X_train.values, y_train,

scoring="roc_auc", cv = kf)

return(auc) score = auc_cv(GBoostr)

score = fit_pred_auc(logreg)

score = auc_cv(Lasso)

score = fit_pred_auc(NB)

score = fit_pred_auc(Ada)

score = fit_pred_auc(Tree)

score = fit_pred_auc(RF)

score = fit_pred_auc(GBoost)

Results: GBoostr score: 0.7386



Logistic Regression score: 0.6104



Lasso score: 0.6546



Naive Bayes score: 0.6382



Ada Boost score: 0.7626



Decision Tree score: 0.7328



Random Forest score: 0.7431



Gradient Boosting score: 0.7640

As seen above, the best AUC scores I got were from the Gradient Boosting Trees Classifier, and the AdaBoost Classifier, this comes as no surprise, as ensembling algorithms tend to perform better than single algorithms (except for neural networks) which are clearly explained in this article.

From here, there are several approaches to go for (stacking techniques, ensembling, hyperparameter tuning, neural networks, etc), I decided to go for a simple average stacking model using the top two best performers indicated earlier, as I feel that combining the information from these two models could give my final score the extra boost needed for improving.

#stacking class

class StackingModels(BaseEstimator, ClassifierMixin,

TransformerMixin):

def __init__(self, models):

self.models = models



# define clones to fit the data

def fit(self, X, y):

self.models_ = [clone(x) for x in self.models]



# Training

for model in self.models_:

model.fit(X, y) return self



#predictions

def predict(self, X):

predictions = np.column_stack([

model.predict(X) for model in self.models_

])

return np.mean(predictions, axis=1)



def predict_proba(self, X):

predictions_proba = np.column_stack([

model.predict_proba(X)[:,1] for model in self.models_

])

return [np.mean(i) for i in predictions_proba] #stacking model

stacked_models = StackingModels(models = (Ada, GBoost))

stacked_models.fit(X_train, y_train)

preds_sm = stacked_models.predict_proba(X_test) Results:

preds_sm AUC score: 0.7737

The final AUC score I got from the stacking model was 0.7737 which is a huge improvement from the ~0.51 AUC score of the normal approached Decision Tree at the beginning, and also it is an improvement from the top previous model score, which was the Gradient Boosting Tree Classifier with an AUC score of 0.7640.

After choosing this as my final model, I proceeded to use it and predict on the given test data, which positioned me at the 15th place in the leaderboard with an AUC score of .79930, having the top score at .8081.

Further improvements

As mentioned earlier, the way build the final model is not the only option, as explained, there are another couple of approaches that one can use to improve the score, the first and most straightforward, would be to tune the hyperparameters of the models by using grid-search for example, and re-use my stacking class. Another approach can be to create a different architecture of the stacking class, by adding meta-learners as explained in this article by Marios Michailidis (KazAnova). Also, if you are familiar with deep learning libraries or even with sci-kit learn, you can create a neural network model, which are known to outperform many classic ML algorithms in these type of tasks.

Conclusion

This project was about finding the groups of individuals within the general population, and distinguish the ones that contain new potential customers, to the ones that not, using machine learning techniques.

There were two different approaches to each different task presented to solve the problem, the first was unsupervised learning and the second supervised learning, and in both the results were really satisfactory, as in the first one, the model could find the characteristics of the group that make up the core customer base of the company, and also the characteristics of the group that the company should avoid.

For the second task, the model build did a pretty decent job at learning to distinguish the individuals that are going to become new customers to the ones that not, and was able to throw predictions that ultimately positioned me in a good rank in the competition.

With this, I wrap up the two-part series story, I hope you liked it, and if you have any suggestions or thoughts, you can put them in the comments below, they will be very much appreciated.