How to target promotions with conversion prediction model to maximize Net Incremental Revenue?
Tags: Data Analysis, Predictive Analytics, Marketing, Data Science, Machine Learning

Identifying the right set of customers to be targeted for the promotion is the key for a successful campaign Image Source

Every time a company chooses to promote some product through tactics like offering discounts or running a digital ad campaign, there is a certain cost as well as some potential revenue earning opportunity associated with it. If the company is not careful in choosing the right set of customers to be receiving the promotion, it can end up losing a lot of money without earning much in return.

How to quantify the net returns from promotion campaign?

Let’s assume that company has access to promote its product to a set of people. They can be seen as existing customers or just an audience accessible through social media, search engine ads, email or some other communication mechanism.

Assume that company runs an experiment where it takes a random small sample of these people to derive statistical conclusions from.

It further divides this sample into two groups of roughly equal size. Which person gets assigned to which group is also random.

To one of these groups, the company chooses to send promotion and for simplicity let’s assume that all of them receive it. We call this group the treatment group. For e.g. company chooses to send discount coupon to all of the members in this group and everyone receives and sees it.

Let’s consider the cost associated with one person receiving promotion. For e.g. Cost Per Click charged by social media platform.

Consider the number of people who chose to purchased the product after receiving promotion.

The company generates revenue from the purchase of the product by the person who received promotion.

The second group is the one to which the company does not send promotion. We call this group the control group.

In this control group, there will be some people who didn’t receive the promotion and still they purchased the product.

The company generated revenue from the purchase of the product by the person who did not receive promotion.

Assume that company runs this experiment over a certain period and at the end of that period it has these numbers. It can then formulate the returns from this promotion campaign like this.

Incremental Response Rate (IRR) quantifies the amount of change in the response or purchase rate of people after receiving promotion.

Net Incremental Revenue (NIR) quantifies the change in the net revenue earned by company after running the promotional campaign while also taking into consideration the cost associated with it.

I believe both the formulations above intuitively make sense that they capture the effectiveness of the promotional campaigns really well.

The goal of the company will be to maximize both of these parameters IIR and NIR.

For different types of companies or scenarios, IRR can capture any positive action from person as response such as signup or purchase, etc.

For a product selling company, NIR reflects the tangible outcome of the promotional campaign in terms of immediate monetary gains.

Imagine a scenario, where the number of people who choose to purchase the product after receiving the promotion is roughly same (or possibly less than) the number of people who didn’t receive promotion and still bought the product. In this case, if the revenue generated per person in both the treatment and control groups is the same, then the Net Incremental Revenue for this campaign is negative. This means the company made monetary losses by running this promotional campaign!

How can a company maximize the IIR and NIR?

If we assume that cost of sending promotion to a person and the revenue generated from the purchase of the person who received the promotion are both constants, then the only two parameters that we have left for tuning are the minimising number of people who receive the promotion and maximising the number of people who choose to respond positively (e.g. make a purchase) after receiving the promotion.

The company should send promotions to only those individuals who are highly likely to respond positively only after receiving promotion.

How can the company decide whom to send promotion to?

From the data collected through the experiment, the company can classify its accessible audience consisting of potential purchasers into two categories: those who made the purchase after they received promotion vs. the rest.

Note, that the rest will include those people to whom it’s not worth sending promotion. These include:

Those who purchased the product without receiving promotion. This is important because there is a cost of sending out promotion and the company does not want to spend it on those who might have purchased the product anyways. Those who received the promotion and yet did not purchase. These are people on whom company spent money for sending out promotion and yet they chose to not purchase the product. There might be a possibility that these people got pissed off for some reason (may be privacy concern) after seeing the company targeting them with promotion and consciously chose not purchase it just for that reason. Those who did not receive the promotion and did not purchase product. The company would have little data derived from the experiment to make judgement on whether to send them promotion or not. Conservative approach would be to not send promotion to such people.

The problem translates to deciding which person falls under which category. Applying machine learning based statistical model trained on data derived from experiment can help. If the company has some personal data related to the potential audience like demographic, age, gender, location, purchase history, financial status, marital status, education, employment background, interests, etc. then it can use it for identifying which subsets of these people are more likely to purchase the product after receiving promotion.

Let’s walk through an example

To further explain this technique, I would take example of a sample dataset provided by Starbucks.

You can find my code for this project in this notebook here and the project is available on GitHub.

Happy Hour, Starbucks’ invite only promotion Image Source

It used to provide this dataset to Data Scientist job candidates. It contains 120,000 rows with seven numerical variables named V1 to V7 about each person that they could target in their experiment. It has two additional variables indicating whether the person received promotion and whether she purchased the product.

In case of Starbucks, the revenue earned per purchase is 10$ from both types of customer who did and did not receive the promotion. The cost of sending promotion to one person was 0.15$. Therefore in this scenario, the formula for NIR translates to something like this:

To evaluate the performance of our prediction model, it’s essential to split available dataset from the experiment results into training and test dataset. The test dataset must not be utilised in any form for training the model. It must only be used at the end of training for evaluating the model against some pre-determined parameters, in our case IIR and NIR. In case, we want to evaluate our prediction model during the training phase, we can split out our training dataset into further two splits: training and validation datasets. The model can be trained with some hyperparameters on training dataset and evaluated on validation dataset to choose the optimal values for hyper parameters.

There are several ways how this dataset can be used for training machine learning based statistical model. I would discuss some of the approaches that gave great results with this dataset.

First Approach

As the company is interested in sending promotion to only those people who will purchase product only after receiving promotion, we can define a new binary variable called Response that will only be set to True if the person has both the variables namely, Promotion , which indicates whether she received promotion and Purchase , which indicates whether she purchased the product, are True . For, the people with rest of the combinations of values of Promotion and Purchase variables, the value assigned to Response will be False . After enriching the dataset with this new target variable, a machine learning model like Regularised Gradient Boosting (XGBoost) can be used to train the dataset by treating variables V1 to V7 as input variables and Response as output variable.

Checking and dealing with imbalance in dataset

After defining this new Response variable I decided to check for any imbalance in values assigned to it. As it is usually the case with marketing campaigns where only a small percentage of people out of those receiving promotion choose to act upon it. In this dataset, the ratio between number of people assigned True vs. False to values to target variable Response is around 1:116. This indicates a huge imbalance in the dataset.

If we choose to ignore this imbalance and proceed to train a machine learning model on this dataset, it can achieve accuracy of 0.99 out of 1, just by giving output of Response variable to False in all the cases it receives in the test. It can be misleading to conclude that the model is performing really well in terms of accuracy. Specifically such model wont’t be able to identify the people at all to whom the company should send promotion.

There several ways to deal with imbalanced dataset

Oversampling the data with output variable equal to the minority class by inclusion of new synthetically generated data points (for e.g. using SMOTE technique) Oversampling the data with output variable equal to the minority class by replacement during sampling. The decision boundaries near data points belonging to minority class may not generalise very well in this case. Downsampling the data with output variable equal to the majority class. In this case, we are dropping some data that the model could be trained on. Using class weighting mechanism available in machine learning model for assigning different weights to samples for training based on the value of target variable

For this dataset, I decided to use the Synthetic Minority Over-sampling Technique, where the balance in the dataset is achieved by introducing new data points between existing data points assigned to minority class with some variation in the input variables values.

For this dataset, I decided to train Regularised Gradient Boosting model (XGBoost) on this dataset to predict the Response variable for a person with attributes specified as V1 to V7 as input variables.

The XGBoost has large number of hyperparameters that needs to set based on cross validations. I decided to tune some of the important ones and chose to train the model with those values that performed the best in terms of F1 score on the cross validation tests.

The F1 score that the model received on validation set was around 0.028311, which is considered pretty low. However, we must remember that the true evaluation metric that matters to us in our problem are IIR and NIR.

I went ahead with XGBoost model trained on best parameters and used it for making prediction on test dataset for deciding whether the given person is someone who will make purchase after receiving promotion from the company. If yes, then it will recommend sending promotion to that person.

On the test dataset, this model achieved IIR = 0.0206 and NIR = 259.25.

Starbucks mentioned in the dataset description that their team achieved the IIR = 0.0188 and NIR = 189.45

The results achieved from our prediction model suggests that this approach can be a highly effective way of addressing problem of such type, where the company needs decision support system on deciding whom to send promotion.

Approach 2

Another approach is to evaluate for each person, what is incremental probability of person making purchase after receiving promotion as compared to without receiving promotion. If the incremental probability is sufficiently high for the given person, our algorithm will recommend sending promotion to that person.

Formula for calculating Incremental Probability (IP)

To implement this approach, we need to add Promotion , as an indicator variable to whether the person received promotion or not, as input variable in addition to personal data that we have about each person. The target output variable will be indicating whether the person took desired action or not.

In our example of Starbucks dataset, we will use variables V1 to V7 and Promotion as input variables and Purchase as target output variable.

Here is how the decision on whether to send promotion or not is to be made in this approach. We first calculate the probability of person making purchase after receiving promotion and without receiving promotion by setting Promotion input variable as True or False respectively and calculating the difference between the two probabilities. If the difference turns out to be greater than some threshold value, in that case we can send promotion to the person. The value of threshold can be decided using Hyper Parameter optimization technique while using NIR formula for calculating the return value of the objective function to be minimized i.e. the score can be -NIR.

The dataset has ratio for Purchase variable values between True and False values to be around 1:80. This indicates a high imbalance in dataset. I used SMOTE technique to oversample the minority class to bring both of the classes to equal ratio in training dataset.

Here I have used log loss as evaluation metric while tuning number of estimators as a hyper parameter of XGBoost as I am more interested in calculating accurate probabilities of person giving certain response than just predicting response class. From this probabilities, I can calculate incremental probability of person purchasing some product after receiving promotion.

The trained XGBoost model with optimal hyper parameters received log loss score of 0.07 on validation dataset. When it comes to log loss, the smaller it is the better the quality of prediction performance of the model.

I used this model to give recommendations on the test dataset and calculated, the to be IIR = 0.018826 and NIR = 98.30. Comparing it to what the Starbucks claimed its team having achieved of IIR = 0.0188 and NIR = 189.45, we can say that this model is showing decent performance, but not as great as Approach 1 or the one claimed to be achieved by Starbucks. However, in different problem scenario, it can be considered a worthy option for consideration.

Approach 3 that can be tried

There is a two models based approach suggested in literature related to uplift marketing. In this approach, we train two separate Purchase prediction models: one for the people receiving promotion and another for those who didn’t. Then we calculate the difference in probability for each person making purchase with vs. without receiving promotion. If the difference is sufficiently high, we can choose to send promotion to that person. However, a caveat pointed out by some for this approach is that it trains two separate models, each one can have its own scale of probabilities generated in output and the error in correctly predicting the incremental probability doubles as we are using two predictive models. I have left this approach to interested reader (or myself) for trying sometime in future on this dataset.

I hope you found this post insightful.

You can find my code for this project in this notebook here and the project with Starbucks dataset is available on GitHub.