Credit Card Fraud — A Machine Learning Approach
By: Mateus Nascimento
Tags: Smote, Supervised Learning, Machine Learning, Imbalanced Data, Isolation Forest

Credit Card Fraud — A Machine Learning Approach

Machine Learning can be really helpful in detecting credit card frauds

Introduction

This is an approach that I used to deal with the problem of predicting fraudulent transactions while also dealing with a highly imbalanced dataset. In this solution I used re-sampling algorithms as well as one unsupervised algorithm to tackle this problem. Everything will be thoroughly explained in the respective section regarding each section of the approach.

The Dataset

This is a very famous dataset. It’s basically Imbalanced Learning 101. I’ll give a brief rundown of its main characteristics but if you want to know some more, check out its Kaggle page.

Our dataset, first of all, has a tremendous imbalance regarding its two transaction classes, fraudulent and non-fraudulent. We have 284807 transactions in the whole dataset and only 492 of them are fraudulent. That makes up for less than 1%. Other than that, when looking at our columns, we can see that, for the sake of privacy, 28 out of 31 columns have been anonymized and normalized. The other 3 are the Class, the Time and the Amount of each transaction.

The Problem

So, our main problem is: how to make our model learn with data that doesn’t make that possible. It’s clear that the best choice for our model, in order to be accurate would be to label every single example as non-fraudulent, and it would still get more than 99% accuracy!

It’s clear that the problem relies not solely on accuracy, but also recall. I will not get into details of classification metrics in this article since there is a good amount of material online.

Therefore, how can we tackle this problem? We can either diminish the imbalance of use algorithms that are good to detect outliers. That’s exactly what we’re going to do.

Changing Our Data: Re-sampling methods

Re-sampling consists of creating more instances of a class or removing instances of another class in order to reduce imbalance. Respectively, those techniques are called oversampling and under-sampling. Let’s start with oversampling.

It’s important to notice that resampling must always be done after splitting our dataset. Why? When resampling a dataset that wasn’t split, you run the risk of allowing the same type of observations to be present in both the test and the train dataset, leading to over-fitting.

Oversampling

Oversampling consists of copying instances of the minority class in order to equal or increase its amount when compared to the majority class. There are some methods of using this, we will focus on three:

Random Oversampling

SMOTE

Adaptative Synthetic Sampling

Each one will be explained in detail when used.

Random Oversampling

This algorithm is really simple, so we’ll start with it. Random Oversampling randomly replicates the minority class instances. It has some drawbacks, which we’ll get into later.

I am using the imblearn library since it has a lot of models already implemented.

ros = RandomOverSampler(random_state=42) X_res, y_res = ros.fit_resample(X_train, y_train) X_res = pd.DataFrame(X_res, columns = X_train.columns y_res = pd.Series(y_res)

log_reg = LogisticRegression() log_reg.fit(X_res, y_res) predictions = log_reg.predict(X_test)

recall = recall_score(y_test, predictions) accuracy = accuracy_score(y_test, predictions) f1 = f1_score(y_test, predictions) print(recall, accuracy, f1)

# we got:

#recall - 0.925

# accuracy - 0.9667406932752824

# f1 - 0.08896369319547967

Not bad! Our recall went from 0 to 92% accurate. That’s pretty good!

But, as everything in Machine Learning, there is a drawback. Randomly adding data can lead to severe overfitting, since we can replicate the same example multiple times. Let’s see how can we address this with other models.

SMOTE

To avoid the over-fitting problem, Chawla et al. (2002) propose the Synthetic Minority Over-sampling Technique (SMOTE). This method is considered a state-of-art technique and works well in various applications. This method generates synthetic data based on the feature space similarities between existing minority instances. In order to create a synthetic instance, it finds the K-nearest neighbors of each minority instance, randomly selects one of them, and then calculate linear interpolations to produce a new minority instance in the neighborhood.

sm = SMOTE(sampling_strategy = 0.6, random_state = 41, k_neighbors = 5) X_smote, y_smote = sm.fit_resample(X_train, y_train) X_smote = pd.DataFrame(X_smote, columns = X_train.columns) y_smote = pd.Series(y_smote) recall = recall_score(y_test, predictions) accuracy = accuracy_score(y_test, predictions) f1 = f1_score(y_test, predictions) print(recall, accuracy, f1)

0.7272727272727273 0.9968652037617555 0.6153846153846153

ADASYN

ADASYN is, like SMOTE, a synthetic data generator. It also generates data through a K-Neighbors perspective, but it also takes into account the density of a certain neighbor of K-Neighbors. I’ll explain what this means. First let’s see, step by step, how this algorithm works.

Step 0

First, let’s set some variables.

Let ml and mg be, respectively, the amount of samples for the minority and majority classes. Also, let m=ml+mg. We will define dx as a certain predefined threshold ratio between the two, defined as dx=ml/mg

Step 1

Calculate the current ratio between the two classes using:

If d<dx, then run the algorithm.

Step 2

Calculate the total number of synthetic data to be generated. This number is represented by G and its formula is here:

Where β is the desired ratio between the two classes after ADASYN. β=1 means that after ADASYN we will have the same amount for both classes. Let’s see an example.

Imagine that we have 10 instances of the majority class and 5 of the minority one. That is, mg=10 and ml=5. We want to have an equal amount of the majority and minority classes at the end of the algorithm. Therefore, β=1. Notice that when we plug in our values in the formula, we get exactly 5. It’s that simple.

Step 3

Now we calculate the density of the majority class of each K-Neighborhood that is associated with each member of the minority class. Wait, what? Relax, it just sounds fancy. This is where ADASYN diverges from SMOTE. This density, or ri defines how predominant the majority class is in a specific neighborhood, and this determines how many synthetic minority instances will be generated in that neighborhood. Therefore, unlike SMOTE, ADASYN does not attribute the same weights to each minority member.

But, how do we calculate it? It’s really simple. When you define a number for K, that will define the K-Neighborhood for a determined instance, and for that instance i, we will calculate its ri like this:

Where mi is the amount of majority instances in that neighborhood and K is your K value.

Step 4

Normalize the ri values. This step is mainly a precursor to make step 5 easier.

Step 5

Now, we calculate the amount of synthetic examples to be generated per neighborhood.

Because ri is higher for neighborhoods dominated by majority class examples, more synthetic minority class examples will be generated for those neighborhoods. Hence, this gives the ADASYN algorithm its adaptive nature; more data is generated for “harder-to-learn” neighborhoods.

Step 6

Now we are actually going to generate Gi data for each neighborhood. To do this, we select the minority example from that neighborhood, xi. Then, randomly select another minority example within that neighborhood xz. The new synthetic sample example can be calculated using:

Where λ is a random number between 0 and 1, and si is the new synthetic example.

That’s it! That’s ADASYN!

Now, let’s use it.

ada = ADASYN(random_state = 41, sampling_strategy = 0.6, n_neighbors = 5) X_ada, y_ada = ada.fit_resample(X_train, y_train) X_ada = pd.DataFrame(X_ada, columns = X_train.columns) y_ada = pd.Series(y_ada) log_reg = LogisticRegression() log_reg.fit(X_ada, y_ada) predictions = log_reg.predict(X_test) recall = recall_score(y_test, predictions) accuracy = accuracy_score(y_test, predictions) f1 = f1_score(y_test, predictions) print(recall, accuracy, f1) 0.7272727272727273 0.9987460815047022 0.7999999999999999

These are the oversampling techniques used here. We can clearly see that we made progress. Jumping from 0 to 87% recall is pretty good with algorithms that didn’t get too complex. The drawback that follows every single one of these Oversampling techniques is that our happiness may be founded upon overfitting. Since this type of data itself is rare, we can, for now, only hope that this model actually did a good job.

Under-sampling

Opposed to oversampling, under-sampling changes the proportion between the classes by removing examples from the majority class until we have a more even distribution. By approaching the problem like this, we clearly have different algorithms and different difficulties, which we’ll get into.

These are the oversampling techniques used here. We can clearly see that we made progress. Jumping from 0 to 87% recall is pretty good with algorithms that didn’t get too complex. The drawback that follows every single one of these Oversampling techniques is that our happiness may be founded upon overfitting. Since this type of data itself is rare, we can, for now, only hope that this model actually did a good job.

Random Undersampling

As the name suggests, Random Undersampling removes example randomly from the dataset. It follows the same logic as Random Oversampling. Again, we will be using imblearn.

rus = RandomUnderSampler(sampling_strategy = 0.6, random_state = 41) X_rus, y_rus = rus.fit_resample(X_train, y_train) X_rus = pd.DataFrame(X_rus, columns = X_train.columns) y_rus = pd.Series(y_rus) log_reg = LogisticRegression() log_reg.fit(X_rus, y_rus) predictions = log_reg.predict(X_test) recall = recall_score(y_test, predictions) accuracy = accuracy_score(y_test, predictions) f1 = f1_score(y_test, predictions) print(recall, accuracy, f1) 0.8181818181818182 0.9952978056426333 0.5454545454545455

Because of its simple nature, Random Oversampling also easily shows us its problems. By randomly removing examples, we might be throwing away valuable data. Data that is representative of the real problem that we are trying to solve. We simply do not have any mechanisms that are built in the algorithm to deal with the possible loss of information.

CNN

First, let’s try to think what it means to reduce data efficiently. What type of data can be removed from a dataset without affecting its quality? A quick answer comes to the mind of anyone who’s slightly familiar with the world of Machine Learning: redundant.

This makes a lot of sense and it’s really simple. If we are removing examples that tell us something that we already know, then we are not losing anything. This is the logic behind the CNN algorithm. By the way, CNN means Condensed Nearest Neighbor.

So, imagine D as the original training set and C as a set with all of the minority examples and a few randomly chosen members from the majority class. CNN then classifies all instances in D by finding its nearest neighbor in C and adding it to C if it is miss-classified. Thus, we can see that redundant instances are effectively eliminated in the resulting dataset C as any instance correctly labeled by its nearest neighbors is not added to the dataset.

It’s important to realize that this approach does not allow any choice of ratio between the two classes.

Let’s use it.

cnn = CondensedNearestNeighbour(random_state = 41) X_cnn, y_cnn = cnn.fit_resample(X_train, y_train) X_cnn = pd.DataFrame(X_cnn, columns = X_train.columns) y_cnn = pd.Series(y_cnn) log_reg = LogisticRegression() log_reg.fit(X_cnn, y_cnn) predictions = log_reg.predict(X_test) recall = recall_score(y_test, predictions) accuracy = accuracy_score(y_test, predictions) f1 = f1_score(y_test, predictions) print(recall, accuracy, f1) 0.83 0.9956812935052624 0.4029126213592233

Outlier Detection

Up until now, we’ve been dealing with the imbalance of our dataset from a perspective of more data. If we get more data, then our lives will be easier. That’s a good starting point. But there are other strategies to be taken.

When we consider our dataset, the fraud transactions consist less than 1% of total entries. Therefore it is, by definition, an outlier. Therefore, we could apply algorithms that are really good at detecting an outlier and then label its predictions of outliers as fraudulent transactions.

Isolation Forest

As the name suggest, Isolation Forest is an ensemble method. It uses decision trees in order to detect the ouliers in a dataset. But how does it do that? Let’s get into the details.

Isolation Forest comes from the assumption that outliers will be easier to detect in a Decision Tree as they are easier to isolate. This can seems really intuitive first-hand for some people and not so much for others. I know that for me it wasn’t. Take a look at the following images, they explain it very well.

I got this image from https://towardsdatascience.com/@eryk.lewinson

The algorithm samples the training set and creates a forest for that sample, based on randomly selected attributes. It does this iteratively for many subsets of the data and then we calculate the anomaly score for each data point in all of the generated trees.

What is the anomaly score? It’s a mathematical formula that calculates, as it says, the degree of anomaly of a particular data example. How does it do that? It takes into account the strategy mentioned earlier that outliers will be easier to be separated. Here’s the formula:

Where h(x) is the path length of observation x, c(n) is the average path length of unsuccessful search in a Binary Search Tree and n is the number of external nodes.

We then calculate this for each tree and then calculate the mean anomaly score.

This is how it works. Now let’ test it. We will take the values that are outputted by the algorithm and assign it, through a threshold, to be either 0 (negative) or 1 (positive).

isf = IsolationForest(n_estimators = 100, random_state = 41) isf.fit(X_train) predictions = isf.predict(X_test) predictions = pd.Series(predictions) # These are predicted values of each one, now let's transalte these scores into our actual predictions predictions = predictions.apply(lambda x: 1 if x == -1 else 0) recall = recall_score(y_test, predictions) accuracy = accuracy_score(y_test, predictions) f1 = f1_score(y_test, predictions) print(recall, accuracy, f1) 0.9424083769633508 0.8998885211941399 0.03059923501912452

With this, we got our best recall!

Conclusion

In this article we have seen a few methods of dealing with imbalanced data, but there are so much more. I came up with a lot more algorithms when searching this topic so I suggest you too go look for it, if you’re interested. You can look for Local Outlier Factor Algorithm and also Extended Isolation Forest.

I also implemented a more detailed resolution of this problem in a notebook, which you can find right here. In it I picked a model and did some more correlation research and also some Grid Search. Feel free to read it too.

Thanks for reading!