Who is a Potential Customer?. Online purchasing intention prediction... | by Pei Guo | Towards Data Science

Every day online shopping sites receive a huge number of visits. However, only a small portion of the visits turned into purchases.
What if the shopping sites know which customer groups are more likely to make purchases?
They will be able to launch target promotions that benefit both the websites and the customers. Today we will explore this problem using the power of machine learning. The dataset is from the research of C. Okan Sakar. We will use different methodologies from the research in this project. If you would like to reproduce this project, you can find all the resources in my Github.
The dataset consists of 12,330 sessions(rows) of online shopping visits to a website. The data is within a 1 year period to avoid any tendency. 18 features are used to describe each session, divided into numeric and categorical features. Feature “Revenue” is the label of our binary classification problem. Our goal is to use all the other 17 features below to predict the label “Revenue”, that is to see whether a visit session will end up with a transaction or not.
Here are the descriptions of the features, referenced to the paper.
It’s worth note that Google Analytics metrics are used in numeric features: bounce rate measures whether a customer leaves a page without further visiting another one; exit rate tells you how often a page is the last page of a session; page value shows which page contributed more to the site’s revenue.
We will follow the workflow above to find the best model for customer purchase intension prediction.
We need to make a lot of decisions on model selections and feature selections in this project. The decisions are made based on these metrics: Accuracy, Recall, Precision, and F1. Let me explain them using our business case.
In our dataset, there is a customer behind each online visit session. If we predict “Revenue” is 1, it means this customer who initiated the session is more likely to make a purchase. However, our prediction is not always correct. The table below shows the different scenarios.
Assumptions: 1. $100 revenue for each order placed and $5 cost for each coupon sent. 2. A likely customer with a coupon will make a purchase and an unlikely customer with a coupon will NOT make a purchase.
As seen from the above table, we want more “true positive(TP)” to happen and to avoid “false negative(FN)” as much as possible. How do we apply this to a machine learning model? The classification metrics:
Recall score: TP/(TP+FN)
Of all the likely customers, how many of them the model can identify correctly. A high recall requires both high TP and low FN. It can identify a large portion of likely customers to reduce lost sales. Meanwhile more unlikely customers(FP) will also receive the coupons. This is fine when the lost sales cost is greater than the coupon cost.
Precision score: TP/(TP+ FP)
Of all the “likely” customers our model predicted, how many of them are actually correct, not false positive. A high precision score requires both high TP and low FP. It can make sure a larger portion of predicted likely customers are correct. More coupons are used, fewer wasted coupons. However more other likely customers will be ignored to cause lost sales.
F1 score: 2*recall*precisoin/(recall+precision)
A value that balances recall and precision scores with consideration of both.
Accuracy score: (TP+TN)/(TP +TN+FN+FP)
The percentage of the predicted results are correct. This metric only works when the classes are balanced. We don’t use it for unbalanced data.
As a result of the above, in this project, we will use the Recall score as our main metric when making decisions on model selections. When recall scores are similar, we will also consider F1 scores.
First, let’s clean the data:
Usually only a small portion of the visits ended up into transactions, we suspect the data is imbalanced. After testing it below, we found the transaction conversion rate is only 18.3%. This means later we need to make a balance adjustment on the data for modeling.
The categorical data need to be converted into numeric for further analysis.
The data set is highly imbalanced. All the models tend to fit towards the majority class, so in order to improve the model performance, we need to balance the data first. I used the SMOTE oversizing technique, the result is that both classes have the same number of samples.
Now we can use Accuracy as a metric for a balanced dataset. I chose 7 classifiers as model candidates. I split the dataset into 80% train and 20% test. Then the train set is fitted into each model without tuning. Here we get a baseline accuracy ranking. I will choose the top 4 models as my baseline models: Random forest, EXTRA trees, ADAboost and logistic regression. We now prioritize on accuracy to recall, because we need unbiased models. Then later select high recall models within the high accuracy models.
Feature engineering and model tuning will be performed later on the 4 models. For code details please refer to my Github page.
My goal is to reduce the number of features and increase the metric values at the same time. With fewer features, the cost of data collection is reduced, as well as the computing time and cost. Three feature selection techniques were used to give a comprehensive decision: information values, sklearn’s SelectFromModel(), and the correlation values.
Which features can give us more information in prediction than other features? Let’s try sklearn’s SelectFromModel() to filter the importance weights of the features. This gives us three features as important features. We will keep them and continue exploring the other two methods for feature selection.
'ProductRelated_Duration', 'ExitRates', 'PageValues'
The heatmap on the left shows the correlations among all the features. The darker the color, the higher the correlation. What we want to see is that the features are very correlated with the label “Revenue” but not correlated with one another. But some features are highly correlated with another feature. This means only one of the correlated features can be kept. How do we know which one to keep? The information values (IV) will help us.
Now, we will use information values to rank all the features and find the ones that have medium to strong prediction power.
The function below will calculate the information value of each feature.
With all the calculations above, now we have the results below:
The tables above show the 8 features kept for further modeling:
Administrative_Duration, Informational, ProductRelated_Duration, ExitRates, Special day, Month, VisitorType, PageValues
How did I make the decisions?
The flowchart on the left is my thought process.
A feature with IV below 0.1 is removed.For the rest, if a feature is selected by the sklearn filter, we keep it.Then the rest is removed if it is highly correlated with the kept features.Then last portion if a feature is not correlated with another, we keep it. If it is, the one with higher information value is kept.
A feature with IV below 0.1 is removed.
For the rest, if a feature is selected by the sklearn filter, we keep it.
Then the rest is removed if it is highly correlated with the kept features.
Then last portion if a feature is not correlated with another, we keep it. If it is, the one with higher information value is kept.
Now we have fewer features, let’s see if they can help improve our models:
We have great results! All the recall values are improved greatly after feature selection and the other metrics look good as well.
We have 7 categorical features. Even though they are converted to numeric values earlier, all the values of a categorical feature are in one column. What if we make each value its own feature? Let’s dummy the categorical feature variable! After processing the data, now we have 29 features and 1 label.
Why we don’t use the dummy variables in the 1st round feature selection? Because it is not effective to dummy the low information value features and make too many new features. That will lower the models’ performance.
Now we will use the same techniques in the 1st round feature selection process to select the new features. For details, please refer to my Github.
Administrative_Duration, Informational_bins_page_1_plus, ProductRelated_Duration, ExitRates, Month 11, VisitorType_bins_return, PageValues_bins_above_0
Now the heat map above shows the 7 features + 1 label after a new round of feature selection. Note that no two features are correlated (this is great), and the Pagevalues_bins_above_0 feature is very relevant to Revenue
Now let’s see the performance of these new features on the models.
We can see that EXTRA trees and AdaBoost have improved recall scores while the other two models don’t perform as well as before. EXTRA trees model has a lower recall than AdaBoost but a higher F1. So we will keep both models for now for the next step, model tuning.
Now we have narrowed down to two models: EXTRA trees and AdaBoost. By tuning their hyper-parameters, we could potentially increase the recall score even further. For model tuning techniques, I will use RandomizedSearchCV first. It will randomly run the model through a list of hyper-parameters we defined. Once we have the best performance through RandomizedSearch, we will use GridSearch to further improve the model by running through a narrower list of hyper-parameters. For details, please check my Github.
The picture above shows that both models were improved by using a randomized search, then grid search. The final recall scores only have 0.002 difference. We then look at F1 and accuracy scores. EXTRA trees model has much higher scores in both metrics. As a result, it is our final model.
In this project, we built a binary classification machine learning model to predict the purchase intension of customers on an online shopping site. After several rounds of feature selections and model selections, we determine the best model for our data is EXTRA trees and the features to fit in the model are as below.
EXTRA trees model we built in this project can capture 83% of the likely customers. This means if the marketing team send the coupons to all the likely customers predicted by this model, the coupons will cover 83% of all the actual likely customer in the market. This model can definitely help the marketing team make effective marketing strategies!
Thank you for your time in reading this article. I hope it gives you some inspirations in your own project. If you like my article, please give it a clap. Connect with me on Linkedin is always welcomed! Happy learning!