Why Every Marketer Should Consider Uplift Modeling
By: Michael Firn
Tags: Uplift Modeling, Uplift Predictions, Editors Pick, Machine Learning, AI

Why Every Marketer Should Consider Uplift Modeling

By Michael Firn

Uplift modeling is an increasingly important tool in the modern marketer’s toolkit of machine learning technology. Uplift predictions, although applicable across a variety of business applications, are particularly powerful for marketers since they address one of the core goals of marketing: how will each customer respond to a particular intervention? In this article, we discuss Uplift modeling for marketing use cases, and compare the performance of various common Uplift algorithms on a public dataset.

Prescriptive Data: Moving from Information to Optimization

Modern marketing teams have a variety of data science and machine learning techniques at their disposal. These techniques can be generally grouped into three buckets: descriptive, predictive, or prescriptive. Uplift models fall into the prescriptive category, providing marketers with knowledge of which marketing treatment they should apply to each user in order to encourage some desired downstream behavior.

Image by author

What is Uplift Modeling?

Uplift modeling is a prescriptive technique which predicts how each customer is likely to respond to a marketing action. Based on these predictions, marketing teams can determine which customers should be targeted in order to maximize the ROI of their campaign.

Defining an Uplift prediction involves specifying two inputs:

Intervention: Which marketing action(s) would you like to measure the impact of?

Outcome: What is the conversion event that your intervention is meant to influence?

For example, if we are considering offering a discount (the intervention) with the goal of driving customer transactions (the outcome), our Uplift prediction for Customer A might tell us: “A discount will increase Customer A’s probability of purchasing by 10% compared to if we did not offer the discount.”

In order to create an Uplift model, we must have data from a randomized control trial in which a random set of customers were targeted with the intervention (the “treatment group”) and another random set of customers were not targeted (the “control group”). An Uplift model uses this information to differentiate between four types of users (see image below). If the model is able to accurately identify Persuadables and Sleeping Dogs, marketers can determine which customers to target in order to drive incremental conversions with a marketing campaign.

Image by author

Some Real World Examples

The advantages of the Uplift approach extend to many common marketing problems. A few examples are listed below -

Image by author

Comparing the Performance of Different Uplift Algorithms

Although Uplift Modeling is a relatively new technique, there are several algorithms which have been shown to perform well in practical settings. We compared these algorithms on the publicly available dataset from a competition hosted by X5 and Retail Hero in order to gauge their relative performance. The goal of the competition was to determine which customers should be targeted through an SMS campaign by predicting whose probability of purchase would be most positively impacted by the campaign.

How to Evaluate the Performance of Uplift Models

A key challenge in evaluating Uplift models comes from the lack of ground truth. That is, we observe the outcome for a given customer in the treatment group, but we don’t know what the outcome would have been had the customer been in the control group. To overcome this counterfactual limitation, we evaluate the performance of an Uplift model over a group of customers within a holdout test set.

One common evaluation technique is visualized through an “Uplift Bins” plot (see Figure 1 below). These plots are created through a three-step process:

Sort all test set customers in descending order of predicted Uplift. Divide users into k equal-width bins, where bin #1 represents users with the highest predicted Uplift and bin k represents users with the lowest predicted Uplift. Within each bin, compute two conversion rates: (1) users within the bin who received treatment (Treatment Conversion Rate), and (2) users within the bin who did not receive treatment (Control Conversion Rate). Calculate actual uplift within each bin by subtracting (Treatment Conversion Rate — Control Conversion Rate). Note that actual uplift is represented as a difference in percentage points.

A high-quality Uplift model will show actual uplift decreasing monotonically and sharply across bins. If we see this pattern, it indicates that we observed a large positive uplift for the set of users with high Uplift predictions, and a small uplift (even negative) for users with low uplift predictions.

This pattern is illustrated in the hypothetical plot with five bins in Figure 1. The first two bins show positive uplift, indicating that users within these bins (Persuadables) convert at a higher rate when treated. Bins 3 and 4 show negligible uplift, indicating that treatment has essentially no effect for these users (Sure Things and Lost Causes). Bin 5 shows negative uplift, indicating that treatment reduces the likelihood that these users convert (Sleeping Dogs).

Figure 1: A hypothetical Uplift Bins plot, which shows the difference in treatment vs. control group conversion rates within each of five predicted uplift bins. (Image by author)

Common Uplift Algorithms

In this section we describe various Uplift algorithms. This not a comprehensive list and new Uplift algorithms are actively being developed by researchers.

Solo Model — Train a single classifier to predict the outcome of interest, and include as a feature a dummy variable indicating whether each user received treatment. During inference, each user is scored twice: once with the value of the treatment flag set to 1, and once set to 0. The Uplift prediction for User A is defined as User A’s prediction with treatment = 1 minus User A’s prediction with treatment = 0. Read more here.

Two Models — Train two classifiers to predict the outcome of interest. Classifier #1 is trained using only the treatment group, while Classifier #2 is trained using only the control group. The Uplift prediction for User A is defined as User A’s prediction from Classifier #1 minus User A’s prediction from Classifier #2. Read more here.

Class Transformation — Train a single classifier based on a modified class label Z, where:

Z = 1 if Treatment = 1 & Conversion = 1

Z = 1 if Treatment = 0 & Conversion = 0

Z = 0 if Treatment = 1 & Conversion = 0

Z = 0 if Treatment = 0 & Conversion = 1

This modified label ensures that all Persuadables are in the positive label group, while all Sleeping Dogs are in the negative label group. The Uplift prediction for User A is defined as User A’s prediction from this classifier. Read more here.

Regression Transformation — Train a single regression model based on a modified class label Z, where:

Z = 2 if Treatment = 1 & Conversion = 1

Z = 0 if Treatment = 0 & Conversion = 0

Z = -2 if Treatment = 1 & Conversion = 0

Z = 0 if Treatment = 0 & Conversion = 1

This modified label ensures that Persuadables are differentiated from Sleeping Dogs. The Uplift prediction for User A is defined as User A’s prediction from this model. Read more here.

Propensity Deviation — Train a single regression model based on the following label for users in the treatment group:

Let Wi = user i’s a priori probability of conversion, before treatment is applied (i.e. the output of a probabilistic classifier).

Let Ci = a binary outcome indicating whether the user ended up converting post-treatment

The regression label for user i is defined as Wi — C.

To the extent that a user’s predicted conversion deviates from their actual conversion, this label attributes that deviation to the treatment. The Uplift prediction for User A is defined as User A’s prediction from this model.

Results

In order to smooth variation, performance is reported as the average of five runs, where each run involves training on a randomly selected 80% of the data (training set) and evaluating performance on the remaining 20% (test set).

(Image by author)

Figure 2: Uplift Bins. For this dataset, the Regression Transformation method offered the strongest overall performance (based on the sharpest decrease in actual Uplift from Bin #1 to Bin #5). The Propensity Deviation method was the least successful, though all five algorithms were able to extract signal from the data. Observed uplift was positive in almost every bin, indicating that the models found a relatively small number of “Sleeping Dogs”. Different algorithms may be best suited for different problems/datasets, but more research is required to understand when each approach works best.

When do Uplift Models work best?

Although Uplift models often perform well for a broad set of marketing use cases, they tend to work best under certain conditions. Specifically, Uplift models stand the best chance for strong performance when -

The marketing team’s goal is to impact short-term user behavior (e.g. a transaction). If instead, the goal is to impact longer-term metrics like CLV, Uplift may not be up to the task. The team has powerful marketing incentives at its disposal. If an intervention can’t meaningfully sway customer behavior, there may not be any Uplift for even the best model to find.

Conclusion

Uplift modeling is still an active area of research in the data science community, but its practical applications are quickly gaining steam in the marketing world. Uber, Wayfair, and Fidelity Investments count themselves among the growing list of companies that rely extensively on Uplift modeling. A number of Uplift techniques already show significant promise in driving ROI for marketers, and progress is likely to accelerate as more innovative businesses explore this approach.