XGBoost for Multi-class Classification | by Ernest Ng | Towards Data Science

Customer touch points are your brand’s points of customer contact, from start to finish. For example, customers may find your business through mail discount, SMS, email promotions etc. There are many types of touch points depending on your company’s marketing team!
Touch point prediction is an important tool similar to customer segmentation because it allows for a company’s marketing efforts to be better served if they target specific customer groups with touch points that have a higher chance of ending in a purchase. Through their customers’ profile, companies will have a deeper understanding of the customers’ preferences and execute accurately tailored marketing materials towards each of them.
The aim of this project is to predict the next best touch point action for each customer based on their profile and previous touch point. Let us see what we have to work with!
I like taking notes when I am exploring a dataset, this allows me to keep a checklist of things to clean/explore further later in the project. From the first look, we can see that there are missing values in the SocialMedia column and under the touch points column, we see a sequence of touch points that might have led to a purchase.
Check for missing values and imputation!
We see that 25% of entries do not have touch points and 2.9% of entries do not have credit rating. Missing touch points could mean that the customers purchased without having to go through any online promotion links etc. The lack of credit rating might point to new customers.
I also noted that missing values under SocialMedia column were denoted with empty spaces.
Treating missing values for each column:1. SocialMedia — change ‘ ‘ to ‘U’ (denotes Unknown social media status)2. creditRating — change NaN to ‘New’ (denotes new customers)3. touchpoints — I assume that the touch points are stated in order from left to right, so the last value is the most recent touchpoint for a customer before making a purchase. I will hence take only the last value as labels to predict which touchpoint should be assigned for a future customer
data = df[df['nTouchpoints']!=0].reset_index().drop('index', axis=1)data['recent_touchpoint'] = data['touchpoints'].apply(lambda x: x.split()[-1])data.recent_touchpoint.value_counts()
I would like to explore the relationship of the segment variable with other variables in my dataset. Some possibilities could be segmentation based on income, average spending, credit rating or a combination. We have to check for any multicollinearity between any of our variables. Multicollinearity reduces the precision of the estimate coefficients, which weakens the statistical power of our regression model. I will explore these relationships with graphs and heat maps.
There isn’t a general pattern we can observe with average spending over each credit ratings as seen from each line plot for P1 to P4. The purple line shows us mean average spending per credit rating group and its 95% confidence interval. Comparing both plots, it seems that the high earners with credit rating of 6 spends less than others, and low earners with credit rating of 7 spends more than others.
We see that number of touch points is highly correlated with average spending however the correlation is low-moderate below 0.5. Since the degree of multicollinearity is not severe, I will leave the variables untouched.
Skewed distributions can be a pain in the ass when you are building your model, especially with outliers in the way. The best way to check for any outliers is to plot the distribution! A general rule in Machine Learning is to ensure that all our numerical variables are approximately in the same range and normally distributed so we have to do normalisation/standardisation.
Some observations that I made:
1. Age distribution looks normally distributed with slight left skew2. Number of married > single > divorced > unknown customers.3. Approximately same number of customers in each category for segment and SocialMedia.4. Distribution of customers across credit ratings looks normal with slight right skew.5. Most customers go through only 1 touchpoint6. Distribution of income looks normal.7. Average spending distribution can be considered exponential but there are obviously outliers that we have to deal with.
I remove all customers earning $18156.7 because our median aveSpend is only at 91.52 which is far from the max value. Leaving them in the data will only skew our aveSpend distribution. When removing outliers, we must ensure that the mean/median will not largely affected and take note that we do not introduce any bias.
Note that the max-min range of all 3 variables are very different from one another. When we do further analysis, like multivariate linear regression, for example, the attributed income will intrinsically influence the result more due to its larger value. Hence, it is important to standardise and normalise the data to bring all variables to the same range.
I use a Robust Scaler (QuantileTransformer): similar to normalisation but it instead uses the interquartile range, so that it is robust to outliers.
We use one-hot encoding to convert categorical variables (marital, segment, SocialMedia, creditRating) into binary variables. We can use the in built OneHotEncoder from sklearn but I chose to write my own functions for the same purpose!
Now our numerical variables follow a normal distribution. We will start labelling our data using the most recent touchpoint. Let us see how many possible labels are there in our data.
Create a mapping from labels to a unique integer and vice versa for labelling and prediction later
For our model data, select columns we want to use in our model and I will use stratified sampling to retrieve them. For more details on stratified sampling, I explained the procedure in my previous post — Keras, Tell Me The Genre Of My Book.
As the baseline model, I used Random Forest. A baseline is a method that uses heuristics, simple summary statistics, randomness, or machine learning to create predictions for a dataset. You can use these predictions to measure the baseline’s performance (e.g., accuracy) — this metric will then become what you compare any other machine learning algorithm against. In this case, I used multi class logistic loss since we predicting the probabilities of the next touchpoint, I want to find the average difference between all probability distributions. In addition, I also used micro F1-score since we have imbalanced classes of labels.
I picked Random Forest Classifier simply because it runs fast and I am able to use GridSearchCV to iterate to the best model possible efficiently. After initialising and tuning my RandomForestClassifier model with GridSearchCV, I got a train accuracy of 1.0 and test accuracy of 0.77688 which shows overfitting.
Our Random Forest Classifier seems to pay more attention to average spending, income and age.
XGBoost is a decision-tree-based ensemble Machine Learning algorithm that uses a gradient boosting framework. In prediction problems involving unstructured data (images, text, etc.) artificial neural networks tend to outperform all other algorithms or frameworks. However, when it comes to small-to-medium structured/tabular data, decision tree based algorithms are considered best-in-class right now.
The implementation of XGBoost offers several advanced features for model tuning, computing environments and algorithm enhancement. It is capable of performing the three main forms of gradient boosting (Gradient Boosting (GB), Stochastic GB and Regularised GB) and it is robust enough to support fine tuning and addition of regularisation parameters.
This ensemble method seeks to create a strong classifier based on previous ‘weaker’ classifiers. By adding models on top of each other iteratively, the errors of the previous model are corrected by the next predictor, until the training data is accurately predicted or reproduced by the model. Long story short, we are updating the model using gradient descent! XGBoost has been the not-so-secret recipe to winning many Kaggle competitions so now you know why this method is so popular amongst Machine Learning enthusiast.
For our initial model, this were the results I got.
As we can see, XGBoost already outperforms Random Forest on the first model iteration. Let us start fine tuning our model, although I will not go into details on how I tune my model. More information on step-by-step tuning can be found here!
Parameters that I tuned:
min_child_weight: Minimum number of samples that a node can represent in order to be split further
max_depth: Tune this to avoid our tree from growing too deep and resulting in overfitting
reg_alpha: Degree of regularisation
Compared to our first iteration of the XGBoost model, we managed to improve slightly in terms of accuracy and micro F1-score. We achieved lower multi class logistic loss and classification error!
We see that a high feature importance score is assigned to ‘unknown’ marital status. This could be due to the fact that there are only 44 customers with ‘unknown’ marital status, hence to reduce bias, our XGBoost model assigns more weight to ‘unknown’ feature.
I will tune gamma, subsample and colsample_bytree and learning rate if I had enough computational power. Due to the limited time I have, I only focus on max_depth and reg_alpha (applying regularisation to reduce overfitting). Try it out and play around with the parameters!
Full code implementation can be found on my Github here!
Thank you for reading my article. I would appreciate it if I could get comments on how I can improve my Data Science projects and I am always looking to collaborate with anyone with an interest in Machine Learning too :)