Model Lift — the missing link. How to talk about Machine Learning... | by Andrzej Szymanski, PhD | Towards Data Science

Half of the success in data modelling is the perception of our model by the stakeholder/audience. The critical point is to understand their expectation and address them in our presentation, using simple language, understandable by a broad, non-technical audience. Usually, the audience are the members of marketing and/or finance team who want to use the model in some kind of cross-sell or retention campaign. They are usually more interested in how much the model can improve campaign performance rather than how good the model is itself. It is a subtle difference which I will try to explain in a moment.
As Data Scientists, we have a tendency to concentrate on the overall model performance and by using the ROC curve and confusion matrix tell the story of how good the model is in differentiating between positives and negatives. However, this is not always the best approach.
The understanding of how the model will be deployed in real life is the critical point, which determines how the model should be evaluated.
ROC is a very good evaluation method when it comes to identifying fraudulent behaviour, building a credit scorecard or identifying patients with cancer. These are the cases when we need to correctly identify each existing or a new record.
However, in a marketing or political environment, the majority of campaigns target only a narrow group of our base — those most open to the message (target group). Because of that, the priority is to understand how the model performs within that group. The performance outside the group is irrelevant. Therefore, using ROC, which evaluates the whole base, is not necessarily the best option.
Secondly, our audience may not understand a confusion matrix or ROC. For them, AUC equal to 0.6 or 0.75 is an abstract, irrelevant figure. Talking about specificity and sensitivity, won’t work. We have to start broadcasting on the same frequency as our audience, speak in their own language. It is us who have to adjust, not them! Remember, you are there not to impress them with your knowledge, but to pass a clear message about the findings of your model. The message should be understood within seconds. To achieve that, we need something easier to understand than ROC and something we can easily put a value on. This is where Model Lift shines.
The definition of genius is taking the complex and making it simpleAlbert Einstein
Model Lift tells us by how much we can improve the campaign by using our model. It tells us, what is the chance of finding positives (goods) in our selection and how does it compare to finding them on a random basis? The comparison of success ratio using our model and random selection is the Model Lift.
What is great about Model Lift, is that we can immediately assign financial value to our model. If the Model Lift in our top decile is equal, for example to 3, we can say that using the top decile of our model, the campaign can generate 3 times more revenue than a campaign based on a random selection. Model Lift is basically a Return on Investment (ROI), where investment is the model we have built.
To obtain Model Lift, we need to follow these steps:
Obtain a model probability score for each record in our base.
Sort the base out using probability score and split it into equal groups. Deciles seems to be most frequent selection, therefore, for the rest of the article I will refer to these groups as deciles.
Calculate the proportion of positives in each decile (decile Response Rate)
Calculate the average Response Rate
Calculate Model Lift
To illustrate it, let’s take an example of a direct marketing campaign of a Portuguese bank. The data came from UCI Machine Learning repository and are available here. After running data cleansing, features reduction, and resampling, I ended up with this dataset. The full code including data cleansing can be found here. Our objective is to identify customers likely to give a positive response to the offer. Our dependent variable is y, where 1= positive reaction(positives) and 0=no/negative reaction(negatives).
One of the features in the dataset (duration of a call) is causing a data leakage. Duration of a call can be the result of a purchase not a reason for it. Secondly, it is unknown until data for the campaign are selected and calls made. Therefore it should be excluded from the data selected for modelling. However, as I want to concentrate on Model Lift itself, I decided to leave all variables as they are in the dataset.
First, we need to split the data into train and test samples
df= pd.read_csv('banking_campaign.csv')X = df.iloc[:,1:]y = df.iloc[:,0]X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=0)
I have selected Gradient Boosting for modelling
from sklearn.ensemble import GradientBoostingClassifierfrom sklearn.metrics import classification_reportGBM = GradientBoostingClassifier(learning_rate=0.01,                                       n_estimators=1000,                                      max_depth=6,                                       min_samples_split=50,                                       min_samples_leaf=25,                                       subsample=0.8,                                      max_features=10,                                       random_state=10)GBM.fit(X_train,y_train)predictors=list(X_train)
Let’s have a brief look at the overall model performance.
print('Accuracy of the GBM on test set: {:.3f}'.format(GBM.score(X_test, y_test)))pred=GBM.predict(X_test)print(classification_report(y_test, pred))from sklearn.metrics import roc_auc_scoreprint (roc_auc_score(y_test, pred))
The ROC AUC score of the model is 0.75 but let’s ignore it.
The first step to obtain Model Lift is getting probabilities. To do so we use predict_proba.
y_pred2=GBM.predict_proba(X_test)
The first value in y_pred2 is probability of 0 (negatives), the second probability of 1 (positives).
To calculate the Model Lift, run the code below. The code produces a pandas data frame which will be used for the final evaluation of the model and for building a graphical presentation.
What you need to do is to call the function, specifying your test dependent variable (y_test), its predicted probabilities (y_pred2) and number of groups you want to split the base into (I decided to use deciles, so 10).
ModelLift=lift(y_test,y_pred2,10)
The output looks as follow:
Scr_grp represents our deciles, sorted and split using probability score. Now we can compare the distribution of positives and negatives across the deciles. You can see the diminishing number of positives as we are going down the deciles. We have 800 positives in decile 1 compared with 1 in decile 10! That means our model is doing the right job.
Now we can calculate the response rate (resp_rate). Response rate is the proportion of positives in a decile. For decile 1 it is 800/(800+436)=0.65. Again, you can see that the response rate is quickly diminishing down the decile groups.
Finally, we need to calculate lift. For that we need the overall proportion of positives and negatives in the test group, which is: 1376/12357 = 0.11. Now the lift for the decile 1 is 5.84 (0.65/0.11). This means that a campaign based on decile 1 selection can be 5.84 times more successful than a campaign based on random selection.
The other features in the table are:cmltv_p_perc: cumulative % of positivescmltv_n_perc: cumulative % of negativescmltv_rand_p_perc: cumulative % of random positivecmltv_resp_rate: cumulative response ratecmltv_lift: cumulative liftKS: Kolmogorov-Smirnov test
Now compare these two messages:
Now, you tell me which message our audience will be excited about!
Creating a chart is the best way to pass the message across, but the audience should be able to grasp the idea within seconds. Therefore I would avoid anything more complicated, like for example gain charts.
Three questions which we need to answer when presenting the results are:
Is the model working?What is the relationship between selection and model performance?What is the optimal selection?
Is the model working?
What is the relationship between selection and model performance?
What is the optimal selection?
Let’s adjust the output a little bit. I have changed the dataframe index into something more informative, like Decile1 to 10 and selected only variables in scope.
dec = ['Decile 1','Decile 2','Decile 3','Decile 4','Decile 5','Decile 6','Decile 7','Decile 8','Decile 9','Decile 10',]MLift=ModelLift[['Positives','Negatives','cmltv_lift','KS']].copy()MLift.index = (dec)
We can quickly visualise how the model is working using the code below.
import cufflinks as cfcf.go_offline()cf.set_config_file(offline=False, world_readable=True)MLift[['Positives','Negatives']].iplot(kind='bar',yTitle='Volume',xTitle='Model decile', title='Positives & Negatives by model decile')
The relationship between selection size and performance can be done using cumulative lift.
MLift[['cmltv_lift']].iplot(kind='bar',color='LightSkyBlue',yTitle='Lift',xTitle='Model decile', title='Cumulative Lift', yrange=[1.11, 6])
As we increase the targeted group, we can expect smaller ROI (Model Lift). For example, if we decide to use decile 1 and 2 together, ROI will drop from 5.84 (decile 1 only ) to 2.87. Still, the campaign will be almost 3 times more successful than a random selection!
The optimal selection is identified using Kolmogorov-Smirnov test, which in simple language evaluates the difference between cumulative positives and cumulative negatives. The test finds the point in our data where we can have maximum positives with minimum negatives. To illustrate this I used to build gain charts but I have changed my mind and I now think bar charts are so much easier to read!
MLift[['KS']].iplot(kind='bar',color='DarkSlateGrey', yTitle='Separation',xTitle='Model decile', title='Target separation')
In our example the best split is around decile 2 and 3. At this point, we have got the maximum of positives with the minimum of negatives. This is the point at which we can maximise the revenue using our model.
Summarising, the 3 main advantage of Model Lift over ROC are:
· Even an average model can still present some value and seriously contribute to marketing campaigns.
· Model Lift can be directly translated into revenue.
· Model Lift is easy to understand and less confusing than confusion matrix!