Credit Card Fraud Prediction using Machine Learning
By: Sharon Lim
Tags: Logistic Regression, Benfords Law, Fraud, Naive Bayes, Machine Learning

Removal of Extreme Outliers

In order to obtain a good model performance, there is a need to identify and remove extreme outliers from features that have a high correlation with our classes. This clean-up process helps to ensure that the observations best represent the problem.

One of the techniques is to use the Interquartile Range Method as shared in Dr Jason Brownlee’s Machine Learning Mastery blog post:

A good statistic for summarizing a non-Gaussian distribution sample of data is the Interquartile Range, or IQR for short. The IQR is calculated as the difference between the 75th and the 25th percentiles of the data and defines the box in a box and whisker plot. The IQR can be used to identify outliers by defining limits on the sample values that are a factor k of the IQR below the 25th percentile or above the 75th percentile. The common value for the factor, k, is the value 1.5.

The threshold for removing outliers has been determined at 1.5 to prevent the risk of information loss, which will cause the models to have a lower accuracy (ie. under-fitting).

Model 2: Optimized Logistic Regression Model

The logistic regression model is optimized through Grid search, an exhaustive search over specified parameter values for an estimator. It selects combinations of hyperparameters from a grid, evaluates them using cross-validation on the training data, and returns the values that perform the best.

We have selected the following model parameters for the grid search:

penalty: Used to specify the norm used in the penalization. C: Each of the values in Cs describes the inverse of regularization strength. If Cs is as an integer, then a grid of Cs values are chosen in a logarithmic scale between 1e-4 and 1e4. Smaller values specify stronger regularization.

Based on the grid search, the best hyperparameter values were not the defaults. This shows the importance of tuning a model for a specific dataset. Each dataset will have different characteristics, and the model that does best on one dataset will not necessarily do the best across all datasets.

Use the Best Model to Optimize Logistic Regression

Although the performance achieved by the optimized model was slightly below that of the baseline model, it represented a more realistic fit of learning curve instead of an almost perfect 100% accuracy hit for both the training and test data.

Model 3: Naïve Bayes Model

According to the Scikit-Learn documentation,

BernoulliNB implements the naïve bayes training and classification algorithms for data that is distributed according to multivariate Bernoulli distributions; i.e., there may be multiple features but each one is assumed to be a binary-valued (Bernoulli, boolean) variable. Therefore, this class requires samples to be represented as binary-valued feature vectors; if handed any other kind of data, a BernoulliNB instance may binarize its input (depending on the binarize parameter).

As compared to the optimized logistic regression model, it appears that the naïve bayes model has a less ideal fit of the learning curve with a lower accuracy rate achieved by using the trained model on the test data.

Performance Evaluation of Models

1. Classification Report

The classification report shows a representation of the main classification metrics on a per-class basis and gives a deeper intuition of the classifier behavior over global accuracy, which can mask functional weaknesses in one class of a multi-class problem. The metrics are defined in terms of true and false positives, and true and false negatives.

Precision is the ability of a classifier not to label an instance positive that is actually negative. For each class, it is defined as the ratio of true positives to the sum of true and false positives.

For all instances classified positive, what percent was correct?

Recall is the ability of a classifier to find all positive instances. For each class, it is defined as the ratio of true positives to the sum of true positives and false negatives.

For all instances that were actually positive, what percent was classified correctly?

The F1-score is a weighted harmonic mean of precision and recall such that the best score is 1.0 and the worst is 0.0. Generally, F1-scores are lower than accuracy measures as they embed precision and recall into their computation. As a rule of thumb, the weighted average of the F1-score should be used to compare classifier models, not global accuracy.

Support is the number of actual occurrences of the class in the specified dataset. Imbalanced support in the training data may indicate structural weaknesses in the reported scores of the classifier and could indicate the need for stratified sampling or re-balancing. Support does not change between models but instead diagnoses the evaluation process.

In general, the optimized logistic regression model has performed well in the above metrics. In particular, with SMOTE performed to maintain a balanced dataset, a significant observation was noted in the minority class (ie. label ‘1’ representing fraudulent transactions), where recall scores had improved 64%, from 0.55 to 0.90, by using the optimized logistic regression model as compared to its baseline model.

With a higher precision and recall scores, the optimized logistic regression model was able to correctly label instances that were indeed positive. Out of these instances which were actually positive, the optimized logistic regression model had classified them correctly to a large extent. This directly translates into a higher F1-score as a weighted harmonic mean of precision and recall.

2. Confusion Matrix

The confusion matrix takes a fitted scikit-learn classifier and a set of test x and y values and returns a report showing how each of the test values predicted classes compare to their actual classes. These provide similar information as what is available in a classification report, but rather than top-level scores, they provide deeper insight into the classification of individual data points.