Starbucks Customer profiling using CatBoost and Bokeh | by Hardik B. | Towards Data Science

In this post, I explore the customer data provided by Starbucks in collaboration with Udacity to understand customer behaviors in light of potential rewards. The Starbucks mobile app sends out different kinds of incentive offers to its customers to keep them engaged with the brand.
The motivation of this post is to discuss a solution approach taken to analyze the data and machine learning solution that predicts customer responsiveness.
I have used a machine learning library CatBoost for a multiclass classification problem and Bokeh as a primary visualization tool.
The GitHub Repo of the project can be found here.
This data set contains simulated data that mimics customer behavior on the Starbucks rewards mobile app. Once every few days, Starbucks sends out an offer to users of the mobile app. An offer can be merely an advertisement for a drink or an actual offer, such as a discount or BOGO (buy one get one free). Some users might not receive any offers during certain weeks.
Every offer has a validity period before the offer expires. As an example, a BOGO offer might be valid for only five days. You’ll see in the data set that informational offers have a validity period even though these ads are merely providing information about a product; for example, if an informational offer has seven days of validity, you can assume the customer is feeling the influence of the offer for seven days after receiving the advertisement.
[Project Overview is taken from Udacity’s Capstone Project for DataScience NanoDegree Program]
Data Sets
The data is contained in three files:
portfolio.json — containing offer ids and metadata about each offer (duration, type)
profile.json — demographic data for each customer
transcript.json — records for transactions, offers received, offers viewed, and offers complete
The aim is to analyze individual customer behaviors to identify unusual patterns. The finding of such an analysis should help Starbucks business to re-evaluate its rewards program based on specific outcomes.
I aim to provide analysis targeting the following business questions:
Can we classify customers based on their responsiveness?
Can we identify high paying customers who don’t care about the rewards program?
Can we identify customers who haven’t made any transactions in the past?
Can we visualize customers’ interaction with offers and spending activities to gain interesting insights?
There are three different datasets, which I explore one by one and identify any issues like missing information, extra/redundant information, other inconsistencies. I accomplish this by making good use of visualization as necessary. At the end of this step, I have a clean dataset ready for further exploration.
I combine all the three datasets into one single set and perform further analysis through advanced visualization. Based on this, I frame a classification problem to tackle through some machine learning algorithms.
I have chosen a CatBoost algorithm, a multiclass classification algorithm. It identifies the relations between highly co-related features automatically and comes with built-in visualizations. It is straightforward to train and evaluate the model using it. The final results are usually better with default parameters because CatBoots uses horizontal decision trees internally. Since we are dealing with classification and the error of misclassification doesn’t result in any adverse effects, in this case, model accuracy should be our standard evaluation criteria, including the confusion matrix.
In the end, I evaluate the results of the overall analysis, including machine learning model evaluation.
By looking at the ‘channels’ columns, it makes sense to disaggregate each channel values into separate columns and encode it using 0|1 identifiers.
Let’s create dummies for the ‘offer_type’ column and encode it and also drop the original column as we no longer require it.
Here each offer is always sent via the ‘email’ channel. Therefore this feature is only informative and may not add any value to the predictive power of any machine learning algorithm.
It is obvious that ‘age’ value 118 is a placeholder for the missing value.Further analysis reveals 2175 missing entries for gender, age, and income columns together.
‘became_member_on’ column is a timestamped date, let’s transform it into a more readable format and also extract the number of days since the customer has signed up.
Let’s create some visualizations to analyze the demographic data
The left chart shows that there is a steady increase in the number of customers over the years. However, there is a steep fall at the end of 2017. This could also be due to sampling to prepare the dataset.
Surprisingly, there are many customers who fall into relatively lower-income groups and very few customers who fall into relatively high-income groups.
The pair chart shows that there are not many female customers, and the female group is relatively older and has a higher income.
Specific Observations
There exist no customers in the age group [20, 30] having an income higher than 80K.
There exists no customer in the age group [40, 50] having an income higher than 100K.
All the customer who became a member more than 2000 days ago has income less than 100K.
The quick look at the transcript dataset reveals that the ‘value’ column is encoded as a dictionary. If the event is related to offers, then the ‘value’ column encode offer id, and in case of transaction event, it encodes the amount of transaction.
Therefore, it makes sense to decode the ‘value’ column into separate columns to understand the relationship between offers and the amount spent by the customers.
First of all, we should calculate the total number of offers completed and the total number of offers viewed by each customer.The data includes ‘offers completed’ by customers, knowingly or unknowingly. The customers who have unknowingly completed offers should not be sent out further offers. This group represents profitable customers for the company.
We combine our transcript dataset with the customer demographics (profile) and portfolio data.
After several preprocessing steps applied to the joint dataset,
The above chart displays the number of offers viewed against timestamp in hours. There are six peaks where the number of viewed offers increases dramatically.
I have embedded the number of offers received (correct received offer count is four times the marked one) in the chart. When new offers get received, the count of viewed offer increases and then slowly decreases over time. Generally speaking, most offers are viewed within 24 hours of receipt.
The above chart shows the number of completed offers against a timestamp in hours. It follows a similar pattern as the graph before because new offers are sent six times in 24 day period, and hence the chart has six peaks.
Between each peak, the chart decreases gradually, just like viewed offers chart. However, the decrease is not as smooth as for the viewed offer chart. There are some local picks during the decline.
The above chart combines a number of transactions and offers completed together. There is a local peak between each ‘offer sent out interval’ for transactions. This implies that when offers are received, customers quickly perform some transactions resulting in offer completion.
At time stamp 0, there exist around 600 transactions. Around 200 of them contribute to ‘offer completion’ (number of offers completed at timestamp 0), which is roughly a 25% conversion rate. This figure is roughly in agreement with the ratio of the total number of completed offers to the total number of transactions => 138953/33579)*100 = 24.16%.
There exist some high-value transactions (above 100 USD). It could be a large order from individuals for special events or from corporate clients. Such a high-value transaction leads to all of the current offers associated with a customer to complete.
In reality, these transactions were not motivated to complete the offers, and therefore it should be considered side effects. Sending out offers to such high paying customers would not lead to an increase or decrease in their purchasing behavior. Therefore we should remove such high-value transactions from the dataset.
Next, we should also find out and remove non-responsive customers. I define non-responsive customer = no received offered viewed + no transaction made.
After some further preprocessing steps, I found 422 customers having made no transaction. Out of them, 412 has received and viewed offers while 10 of them have received and not viewed any offers.
Similarly, analyzing the transaction amount, the following observations were collected:
There are some transactions > 1000 USD.
There are many more transactions in the range [0.05,50] than the rest of the range.
Any transaction greater than 50 USD can be treated as a high-value transaction and not necessarily motivated to complete the offers.
In this section, we look at the customer’s interaction with offers through some advanced visualizations. Based on that, I define some calculated features.
Here in the above chart, we have a customer who has completed all the offers sent out to him/her. It usually takes more than one transaction to complete an offer if those transactions are low value.
Sometimes, the customer views the offer instantly while sometimes, he/she views it at a later time.
Out of 4 offers completed, only 1 (last completed) is not viewed by the customer.
Here we have two high paying customers side by side who has completed all the received offers.
Interestingly, for customer ‘9fa’ after completing the first offer, he/she has made five transactions amounting in total around 75 USD without any pending offer to complete. It demonstrates that those transactions were not motivated to complete the offer and show customers spending tendencies in the absence of any offers.
Based on the above analysis I derive the following custom features1. Percentage of Offers completed by a given customer2. Absolute count of Offers completed by a given customer3. Percentage of Offers viewed by a given customer4. Absolute count of Offers viewed by a given customer5. Total amount spent by a given customer
Considerations:
Our problem falls into the category of multi-class classification. The multi-class classification problem can be summarised as below:Given a dataset with instances xi together with N classes where every instance xi belongs precisely to one class yi is a problem targeted for a multiclass classifier.After the training and testing, we have a table with the correct class yi and the predicted class ai for every instance xi in the test set. So for every instance, we have either a match (yi=ai) or a miss (yi≠ai).Assuming we have balanced class distribution in our training set, evaluation using a confusion matrix together with the average accuracy score should be sufficient. However, F1-score can also be used for the evaluation of the multi-class problem.
Since the cost of misclassification is not high in our case (sending an offer to the non-responsive customer doesn’t cost the company extra money), F1-score is not necessary.
In this project, I prefer to use the confusion matrix and the average score as our evaluation measures.Confusion Matrix:A confusion matrix shows the combination of the actual and predicted classes. Each row of the matrix represents the instances in a predicted class, while each column represents the instances in an actual class. It is a good measure of whether models can account for the overlap in class properties and understand which classes are most easily confused.
Accuracy:Percentage of total items classified correctly- (TP+TN)/(N+P) TP: True PositiveTN: True Negative N: Negative P: Positive
For unbalanced class distribution, I have provided weights to each class label, and CatBoost automatically handles it.
I have created a custom target label indicating customer’s responsiveness.
For binary classification, it would simply classify the customer based on the percentage of offers completed. If the offer completed percentage is >50 %, a customer belongs to a ‘responsive’ class. Otherwise, it belongs to the ‘unresponsive’ class.
Later, I extend the problem to multiclass classification. For this, I have created 3-class, 4-class, and 5-class labels by assigning the percentage of offers completed to a specific class and assigning an appropriate label to each class.
For 5-class problem, the label encoding looks like this:
encoding = {
‘responsive’: 4, ‘very_responsive’: 0, ‘moderately_responsive’: 3, ‘very_moderately_responsive’:2, ‘unresponsive’:1 }
1. Model Evaluation with fixed values of hyperparameters (Base Model)
I evaluate CatBoost Classifier with following fixed hyperparameters on all classification problems (class_2, class_3, class_4, class_5)
number of iterations = 2000
loss_function = [‘MultiClass’]
early_stopping_rounds = 50
eval_metric = ‘Accuracy’
and the rest of the parameter values as default provided by CatBoost Classifier.
2. Model Evaluation with hyperparameters found using GridSearch (Tuned Model)
In this round of experiments, I wrote a custom GridSearch function, which finds the best values for each given hyperparameter ranges and returns the model hyperparameters with the best average accuracy on training data.
For CatBoost Multiclassifier, there are numerous hyperparameters to tune. An extensive list can be found here:
I selected only the following hyperparameters with a specified range(found via some research on CatBoost website and Kaggle) to find the model with the best performance.
- iterations = [1000,3000]- loss_function =  ['Logloss','MultiClass','MultiClassOneVsAll']- depth = [4,6,8]      - early_stopping_rounds = [10, 20, 50]
I evaluate four multiclass classification models and document the results. It seems that training accuracy is a reliable indicator of the model performance rather than test accuracy.
This shows that the base model actually performed better than the model with parameters found via GridSearch. It could be due to the fact that range selection was not optimized. In future work, evaluating a model using RadomizedGridSearch could be interesting.
Providing too many parameters to GridSearch lead to a very slow search (more than 7 hours for one model), therefore I have reduced the parameter range, and then it took on average 20 minutes for searching the best parameters.
Initially, I faced some challenges about cleaning up transactions and identifying how to use transaction data for data modeling. The critical insight was to identify that the same timeline relates to different events. That allowed me to plot some great customer journey visualizations and subsequent custom feature creations.
Overall, I find it exciting to work on this project. I learned lots of stuff regarding data analysis and particularly, data visualization using Bokeh.
It took some initial trial and error to make a model using CatBoost work. Even though I haven’t changed much of the default parameters, the results were excellent. Part of the reason is due to the fact that I have included features like ‘the number of offers viewed’ in the training set, which is an excellent indicator of the offer completion rate, and based on it, I have encoded target classes.
Future refinement
I suggest trying out custom categorical features to train a model.
Also, removing some custom features from the dataset and evaluating the model would be interesting.
Concerning feature engineering, there could be features developed regarding how much the customer has spent before viewing offers.
Many thanks for reading the article. If you are interested in more detail about the project, please have a look at the project repository here.