Churn Predictions for Credit Cardholders: Handling Imbalanced Data Set
By: Mari Galdina
Tags: Logistic Regression, Data Science, Towards Data Science, Random Forest, Data

Photo by Dan Meyers on Unsplash

Credit cards can offer convenience for spending, along with the potential to earn valuable rewards or to enjoy travel benefits. Many credit card issuers offer attractive sign-up bonuses on credit cards to entice consumers to apply for the card. It is their pros and cons at the same time. You can receive a good bonus, but later annual fees may eat into it. So, the credit card business possesses a high risk and high profit for both sides: bank and customer.

The customer churn of credit cards has already become the problem to solve in the urgent need. To support the bank to reduce the churn rate, we need to predict which customers are at high risk of churn. Because we know it costs more to acquire new customers than it does to retain existing ones.

My customer data set is from Kaggle, where we can review customer attributes such as gender, age, tenure, balance, education level, estimated salary, and if they stopped the subscription or not.

# checking table

customers = pd.read_csv('/kaggle/input/credit-card-customers/BankChurners.csv')

customers

Before analyzing the data and forecasting, I checked the data on missing values and data types. Data contains 10127 rows (customers) and 21 columns (features). Our data set is pretty clean and doesn’t have duplicates. But when I check for outliers, I found that columns Credit Limit on the Credit Card, Open to Buy Credit Line (Average of last 12 months), and Total Transaction Amount have outliers.

I fixed this problem using the quantile method.

Next step to check: is this data set balanced?

Checking balanced

How do we know if the model is imbalanced or not?

An imbalanced dataset is one where the number of observations belonging to one group or class is significantly higher than those belonging to the other classes. But in real data sets, there is always some degree of imbalance.

And how we can see on the plot my dataset looks imbalanced.

Numbers of Churn

We can confirm it by a total of customer churn from the dataset. 84% of customers stay with their credit cards, 16% —churn.

Why imbalanced dataset a big deal?

Unfortunately, machine learning (ML) algorithms are very likely to produce faulty classifiers when they are trained with imbalanced datasets. They are all created for balanced data. So, if we try to forecast churn for this data set without any preparation, we get a likelihood of the wrong classification of the minority class.

How to work with an imbalanced dataset?

There are a few techniques to solve the class imbalance problem.

use the right evaluation metrics (confusion matrix, precision, recall, F1-score);

use the right technique to modify the unequal data (undersampling, oversampling).

I decided to make the first step into forecasting from Logistic Regression. And I started from the model without applying any technique for data. Why? I think evaluation metrics analysis give me some idea of how to work with my dataset.

Logistic Regression #1

After k-fold cross-validation and some regulation, I found that better to use:

penalty = ‘l2’ (L2 regularization usually has better results than L1 regularization, because it prevent from overfitting);

solver=’lbfgs’ (solver is a good first choice for most cases);

C=100 (high er values of C correspond to less regularization)

And I have results:

the accuracy score is very good 91%;

the recall score is relatively low — 61%, as well as the precision score — 78%. This is a relatively bad recall score.

In the second step, I still used Logistic Regression with undersampling for my data set. I use the same parameters for the model.

Undersampling: this technique balances the imbalanced dataset by reducing the size of the class which is in abundance.

The result for logistic regression when I use undersampling has a different value of evaluation metrics because it is merged with random samples from 8500 existing customers. One of my result:

Accuracy is: 85.0%

Recall Score is: 86.0%

Precision Score is: 52.0%

F1 Score is: 65.0%

The precision score goes down compared with the previous model result (52% vs. 78%), but accuracy still good — 85%. What we really should improve is recall, because recall has a large extent for business.

SMOTE

In the third step, I analyzed the Logistic Regression model with oversampling. Oversampling helps me to upsample the number of churn samples to match them with the regular customer sample size. This data transformation gives the model a better chance of catching small details for the high score.

Oversampling: this method involves increasing the number of minority class to the size of the majority class.

Synthetic Minority Over-sampling Technique is an oversampling method that uses the nearest neighbors algorithm to generate new and synthetic data that we can use for training our model.

# import library for sampling

from imblearn.over_sampling import SMOTE oversampler = SMOTE()

Results:

Accuracy is: 86.0%

Recall Score is: 86.0%

Precision Score is: 54.0%

F1 Score is: 66.0%

Unfortunstly, oversamoling for logistic regression doesn’t shown good result on this data set. How we can see on evaluation matrix, accuracy still very good, but recall and precision still on the same level. Good thing for this mode is each time I get the same score. Why? SMOTE is a generally rull for dataset and here I can try to work with another parametrs more.

Random Forest Model

For this data set, I create a few more models: Random Forest, SVM, Naive Bayes. I like to work for datasets like this with random forest models because they have a good speed and clear results. So, I want to show you the evalustion matrix.

Results for Random Forest Model:

Accuracy : 93.05%

Recall : 95.61%

Precision : 96.09%

It’s look like good relativly results for our task.

Summary

In this blog, I try to show how data and techniques and their transformation techniques affect the models. In my opinion, oversampling is always better than undersampling, because we have stationary results.