Credit card fraud detection with Logistic Regression, Random Forest, XGBoost and LightGBM
By: Han Yi Jiang
Tags: Logistic Regression, Credit Card Fraud, Lightgbm, Random Forest, Xgboost

Imbalanced Data

Data Counts

As shown in the chart above, our data set is extremely imbalanced with 99% of the data points are normal legal transactions and less than 1% of data points are a fraud. If we make predictions using accuracy, the model will not be helpful as a very high accuracy can be achieved without catching any fraud. As a result, we decided to use metrics such as AUC and recall which serve as better measures for measuring the performance of our models.

Pre-processing

Since the data is extremely imbalanced, we need to re-sample the original data to prevent the model from biasing towards choosing 0s. We adopted two sampling techniques for handling imbalanced data, under-sampling and SMOTE over-sampling.

Under Sampling

With the code above, we were able to get balanced data with an under sampling approach which gives us 50% normal transaction and 50% fraud transaction. As a result, more than 99% of the data is not used due to the highly imbalance nature of our data.

After plotting the under sampled data points, we can see a clear clustering among the legal and fraud transactions.

Over Sampling:

By using a built-in library SMOTE, we were able to generate more fraud transactions to balance the distribution of transactions. In short, SMOTE stands for Synthetic Minority Oversampling Technique. This is a statistical technique for increasing the number of minority cases in a dataset through generating them from a distribution. Below is the plot of the data after SMOTE oversampling.

As shown in the plot, the legal transactions are clustered densely in the middle of all transactions.

Logistic Regression

We started off with the default logistic regression model. With the default parameters, Logistic Regression without any tuning returns a 99.903% of accuracy.

This looks impressive, however when we look at some of the other score is not this high.

The Recall Score is 0.520

F1_Score: 0.650

Precision Score: 0.864

ROC-AUC Score: 0.908

ROC Curve

Confusion Matrix for default Logistic Regression

We started off with the default logistic regression model. With the default parameters, Logistic Regression without any tuning returns a 99.903% of accuracy. This looks impressive, however when we look at some of the other score is not this high. With default logistic regression model: we predicted 56856 normal transactions correctly, and 51 fraud transactions correctly, however, we also predicted 47 transactions as normal, but they are actually fraud. So although we have a 99.9% accuracy, the algorithm wrongly classified about 1/2 of the fraudulent transactions. In-this case, we will focus on trying to decrease the number of False Negative, because not detecting fraud is way worse than false detecting.

Under Sample

With that in mind, we start to try out other things to improve the recall score. We first tried with under sample. By reducing the model to 50% normal transaction and 50% fraud transaction, we get a much smaller dataset. With this small dataset, we fit the model once again with logistic regression.

Under sample matrix 1

From this you can see, we have a 100% accuracy, which looks fantastic. But, now we have to run the model with under sampled train and test in the original dataset.

Under Sample Matrix 2

Recall: 0.337

Precision: 0.003

ROC-AUC: 0.559

From the matrix above, you can see the result from under sampling didn’t help, it not only reduced accuracy, even the recall score got much lower.

SMOTE For Balancing Data:

Since we know the data is imbalanced, we decide to try out SMOTE for balancing the data. With SMOTE, we have a reduced accuracy score with only 0.9858.

Now we look at some other metrics,

Recall Score:0.888

Precision Score: 0.097.

ROC-AUC Score: 0.965

ROC-AUC Plot for SMOTE

SMOTE Matrix

With SMOTE, we have way more false positive counts, we have a much lower on the False Negative counts. In this case, we only have 11 missed fraud transaction, which is really good. As a result, we now have 795 false detection, in our opinion, that is totally fine result. But obviously, if we can reduce the number of false detection, that would be even better.

HYPERPARAMETER TUNING

By using Grid Search, we were able to find the best parameter for Logistic Regression. With hyper tuned model, now we have a 99.11% in accuracy.

Now we look at other metrics:

Recall Score: 0.8878

Precision Score:0.1495

ROC-AUC Score: 0.9779

Tuned Model ROC Plot

Tuned Model Matrix

With Hyperparameter tuning, our tuned model performed much better. Although the number of missed fraud detection didn’t get any lower, the number of false detection went down by 300.

So, in the end, our Logistic Regression model was able to detect 56369 normal transactions correctly, 87 fraud transaction correctly, false accusing 495 normal transactions as fraud, and last but not least, only miss detecting 11 fraud transactions.