Meaningful Metrics: Cumulative Gains and Lyft Charts | by RAFFI SAHAKYAN | Towards Data Science

Nowadays, all major companies rely heavily on their data science capabilities. Business data units are becoming larger and more sophisticated in terms of the complexity and diversity of analysis. However, the success of delivering data science solutions into business reality largely depends on the interpretability of findings. Even if the developed models provide outstanding accuracy scores, they may be neglected if they do not match the demands of various business stakeholders.
All beloved ROC AUC score is not informative enough for the business unit since it is abstract for non-technical managers. For the latter reason, I will present two meaningful metrics that every analyst should take into consideration when illustrating the results of their models: Cumulative Gains and Lift charts. I will demonstrate the example of a customer churn case study in the telecommunications industry.
Imagine a business scenario, when a telecom company decides to minimize the customer attrition rate by giving out one month of unlimited data usage to 20% of its customers. One of the managers suggests, for the promotion to be unbiased, to send the gift to every 5th customer ordered by their last names. The decision is almost there: unless someone proposes a better strategy for this marketing promotion.
Having the sample data of customers (actually can be accessed from my GitHub), we would decide to build a RandomForestClassifier model to predict the customer churn before this promotion begins. Then we will preprocess data, remove unnecessary columns, eliminate multicollinearity, and visualize. Final steps: model training and optimization with GridSearchCV. Yet, this article is not about the project; it is about the interpretability of your findings. The case question remains unanswered.
For this, I will present the cumulative gains and lift curves. The cumulative gains curve is an evaluation curve that assesses the performance of the model and compares the results with the random pick. It shows the percentage of targets reached when considering a certain percentage of the population with the highest probability to be target according to the model. In python, we are provided with scikitplot library, that will make the plot for us.
import scikitplot as skplt# Deriving Class probabilitiespredicted_probabilities = model_rf_best_params.predict_proba(x_test)# Creating the plotskplt.metrics.plot_cumulative_gain(y_test, predicted_probabilities)
First, we order all the observations according to the output of the model. On the left-hand side of the horizontal axis, we place the observations with the highest probability to be target according to the model and vice versa for the right-hand side. In our case, at the 20% point of the horizontal axis, the 80% observations with the highest probability are located. On the vertical axis, the curve indicates which percentage of all targets is included in this curve. So, if we can target the 20% of the observations, the model will ensure that 80% of the churners in total are in this group, while the random pick would provide only the 20% of the targets.
In addition to the cumulative gains curve, the lift curve is a widely used visualization of model performance. In definitive terms, the lift is a measure of the performance of a targeting model at predicting or classifying cases as having an enhanced response (for the population as a whole), measured against a random choice targeting model. Constructing a lift curve follows a similar process as forming the cumulative gain curve. Indeed, it is derived from the gain chart. First, we order observations on the horizontal axis with the highest probability of being a target on the left and vice versa for the right-hand side. On the vertical axis, the lift curve indicates how many times more than average targets are included in this group. Lift is calculated as the ratio of Cumulative Gains from classification and random models. Consider the lift at 20%(the desired target of promotion); we can notice that the top 20% of observations contain 80% of targets. If the average incidence of targets is 20%, so the lift is 4. Thus, the model allows addressing four times more targets for this group, compared with addressing without the model, that is randomly.
#Scikitplot library is there to helpskplt.metrics.plot_lift_curve(y_test, predicted_probabilities)
Eventually, we are ready to present the findings for the marketing strategy. With the help of our model, we can target 80% of churners if the company delivers free promotion to 20% of its customers. While with a random pick, we would be able to focus only on 20% of churners.
So, we were able to observe that the Cumulative Gain and Lift charts are useful tools for decision making and model evaluation. They are relatively easy to interpret not only by professionals with a technical background but by others as well. Eventually, those charts can be applied in various fields, which are market segment targeting, financial budgeting, human resource evaluation, etc.
For full implementation, please refer to my GitHub repository and feel free to use this data for the same analysis yet with other classification models: GradientBoostingClassifier, LogisticRegression, etc.
Try, Fail, Learn, Succeed.