Fraud and Anomaly Detection with Artificial Neural Networks using Python3 and Tensorflow. | by Saleh Alkhalifa | Towards Data Science

Over the last few years, there has been a increasing trend in demand for the application of anomaly detection models within the field of data science — especially when it comes to the detection of fraudulent vs non-fraudulent actions. Within the following dataset, we will explore the use of a number of different predictive models, each with varying complexity. As with every good data science project, we will first examine the dataset, preprocess our data, explore the contents, train a number of models, and finally review and evaluate the results.
For the purposes of this tutorial, the dataset we will be using consists of credit card transactions by European cardholders. You can access the dataset as well as the code from my GitHub. There are approximately 30 features within the dataset, as well as a column consisting of the binary classes. We will use the features and classes of these observations to develop a supervised predictive model allowing us to determine whether a new transaction is fraudulent or not.
As always, there are a number of first steps any data scientist should complete to understand the content and overall quality of the dataset at hand. We begin with a quick examination of the shape to have an idea of the number of rows and columns we are working with, as well as some general statistics concerning them.
import pandas as pdimport numpy as npdf = pd.read_csv("./data/creditcard.csv")df.head()print(df.shape)df.describe()
The dataset is fairly sized with 284,807 rows of data, and 31 columns. Let us take a closer look at the columns:
df.columns
We immediately notice that we are working with already preprocessed data. There are approximately 28 vectors worth of information which the provider of the dataset must have developed in order to mask the sensitive information behind the transactions. We also notice that there are two other features: Time and Amount. Let us now take a much closer look at the data. First, we will check the quality of the dataset for any null values.
df.isnull().sum()
Since the dataset as a while seems to be generally in tact, we can now explore some of the specifics. With a dataset of this magnitude, its difficult to believe that it would be well balanced with both fraudulent and non-fraudulent transactions. Lets take a closer look at the distribution of the two:
df.Class.value_counts()
As suspected, there are over 284,315 non-fraudulent transactions, and 492 fraudulent ones. This dataset is highly imbalanced, and that will need to be addressed later on in this analysis — perhaps within the architecture of the ANN. One of the best way to understand the data when it comes to content and correlation very quickly is by creating a pairplot showing all the columns plotted against one another. Luckily, completing this visualization is made easy using the seaborn library.
plt.figure(figsize=(20,10))sns.pairplot(df[[“Time”,”V1", “Amount”]])
In this diagram we examine the time, amount and first of the many vectors within the dataset. We notice little to no correlation with one another, however, we do see some interesting trending. Notice how some of the data points create an almost linear line going from right to left when graphing Time vs V1. Also notice how there are two general ‘clusters’ of data when graphing Amount vs Time. If you are following this tutorial in Python, go ahead and create a similar visualization with the other vector features. Are there any other observations worth noting?
The next step is to take a closer look at a scatter plot of Time vs V1, which can be accomplished using the same Seaborn library.
plt.figure(figsize=(20,5))sns.scatterplot(df[“Time”], df[“V1”], hue=df[“Class”])
Interesting! The supposed linearity we observed in an earlier diagram seems to be a consistent trend for many of the fraudulent cases (orange), relative to the non-fraudulent ones (blue). Notice that many of them chain-together with a hand of them scattered here and there. The consistency between that is indicative of V1 having captured information that can be used to separate class 0 from class 1. Perhaps the other vectors may offer similar value? If you are following this tutorial in Python, go ahead and graph Time vs V2, V3, ..., and so on. Do they show similar or more interesting results?
With the data explored, we can now move on to preprocessing. Given that the dataset was mostly clean and did not contain any nulls, we will not need to do any further cleaning on the dataset. Notice that the vector columns are of a much smaller magnitude relative to their Time and Amount counterparts. This drastic change in scale could affect the model’s performance, and must be addressed. For this, we can use the StandardScaler library from Scikit-Learn. Next, we will split the data into out X (input) and y (output).
import pandas as pdfrom sklearn.preprocessing import StandardScalerscaler = StandardScaler()X = df.iloc[:, :-1]y = df.Class.values.ravel()X_scaled = scaler.fit_transform(X)
With the data scaled and separated, we can now split the data into testing and training subsets. We split the data in order use the training portion to train the model, and the testing portion to ensure that the models performance is consistent on never-before-seen data. This ensures that the model is generalizing and not overfitting. Please see my previous article for more detailed information in regards to overfitting.
from sklearn.model_selection import train_test_splitX_train, X_test, y_train, y_test = train_test_split(X, y, test_size = 0.25, random_state=42)
Within this section, we will investigate the design and performance of three distinct types of models. We will examine a Random Forests Classifier (DTC), an XGBoost Classifier (XGB), as well as an Artificial Neural Network (ANN).
We will begin with XGBoost, a model that has dominated the machine learning space over the last few years when it comes to Kaggle competitions. The XGB model is quite well suited for our purposes, so we will begin by importing the model from sklearn, and fitting it to our training data. We then use the classification_report method from sklearn to check the models performance.
from xgboost import XGBClassifierxgb = XGBClassifier()xgb.fit(X_train, y_train)y_pred = xgb.predict(X_test)print(classification_report(y_test, y_pred))
We notice from the results that the model is performing quite well, especially for the case of the non-fraudulent (0) entries. However, we notice significantly different results for the fraudulent (1) cases with a precision of 0.91, a recall of 0.80, and an overall F1-score of 0.85. Perhaps we can do better with an ensemble model?
Next we will utilize the Random Forests Classifier, which an ensemble learning method consisting of the construction of a number of decision trees. Please see my previous article for more detailed information in regards to Random Forest Classifiers. We begin by importing the classifier from the sklearn library, and fitting the model with the training data. We then use the classification_report method from sklearn to check the models performance.
from sklearn.ensemble import RandomForestClassifierfrom sklearn.metrics import classification_reportrfc = RandomForestClassifier()rfc.fit(X_train, y_train)y_pred = rfc.predict(X_test)print(classification_report(y_test, y_pred))
Relative to the XGB model, we managed to improve our precision by 3% thus boosting the f1-score by 1%. However, we still see a stagnant recall of 80%. While the results are directly indicative of the content of the data, it is likely indicative of the great imbalance we saw between the two classes. Perhaps this can be remedied by an ANN?
We begin the process of preparing the ANN by importing the tensorflow model. Please note that for the purposes of this tutorial, I will be using tensorflow 2.2.4.
from tensorflow import keras
We then prepare our model using the Sequential class in Keras allowing us to group a linear stack of layers into the model based on the order in which they are received. After some experimentation, I found that four Dense layers sandwiched with two Dropout layers showed promising results. Let us go ahead and build this model.
model = keras.Sequential( [ keras.layers.Dense(32, activation=”relu”, input_shape=(X_train.shape[-1],)), keras.layers.Dense(128, activation=”relu”), keras.layers.Dropout(0.3), keras.layers.Dense(256, activation=”relu”), keras.layers.Dropout(0.3), keras.layers.Dense(1, activation=”sigmoid”), ])model.summary()
Notice that within the model, we create a number of Dense Layers such as:
keras.layers.Dense(128, activation=”relu”)
A Dense layer is a neural network layer which takes in a number of different arguments. Within the scope of our model, it takes in two arguments: ‘128’ which is the number of units, and ‘relu’ which is the activation function.
We can also call the summary() function to display a better representation of the model’s architecture. Notice that the Output shape (middle column) is representative of the layer’s units. We start with 32 (the number of columns in the dataset), and then proceed to increase this to 128, then 256, and then drastically shrink this down to 1, which represents the output of the model.
With the model prepared, we will now compile the model using the compile function within Keras. We will assign a generic Adam optimizer to control the learning rate, and assign the loss as binary crossentropy given that we are working with a binary dataset.
model.compile( optimizer=keras.optimizers.Adam(1e-2),  loss=”binary_crossentropy”,  metrics=[“accuracy”])
With the model compiled, we can now fit the model with the dataset. We will use the training set as input, and assign a batch_size of 2048 which will run with 30 epochs.
history = model.fit( X_train, y_train, batch_size=2048, epochs=30, verbose=2, validation_split = 0.2)
Fun Fact: Notice that the batch_size was assigned a value of 2048 which is a power of 2, such as the many other powers of 2:
You will also notice that the units within the model (32, 128, 256) are also all powers of 2. This was not coincidental, but by design. There exists an issue of alignment of virtual processes onto the physical processors of a GPU. The physical processes are often a power of 2, and using a number that is not a power of two often leads to poor performance. Therefore, it is considered best practice to use powers of two within the model to achieve an optimized performance.
With the model trained, we can either review the output of the model, call the history’s keys and review the metrics captured. Either way, we notice that we have achieved a relatively high accuracy, and a very minimal loss which means the model is working. However, upon reviewing the recall and precision, the predictive capabilities here are similar to those of the RFC and XGB models. Perhaps a more complex model is needed?
We will go ahead and prepare a second ANN model, however, we will add a number of convolutional layers within the model with increasing units. We will also introduce another class from Keras known as BatchNormalization which applies a transformation in order to maintain the mean output close to 0, whereas the output of the standard deviation is close to 1.
model_3.add(Conv1D(32, 2, activation=’relu’, input_shape=X_test_d[0].shape))model_3.add(BatchNormalization())model_3.add(Dropout(0.1))model_3.add(Conv1D(64, 2, activation=’relu’))model_3.add(BatchNormalization())model_3.add(Dropout(0.2))model_3.add(Conv1D(128, 2, activation=’relu’))model_3.add(BatchNormalization())model_3.add(Dropout(0.3))model_3.add(Flatten())model_3.add(Dropout(0.4))model_3.add(Dense(512, activation=’relu’))model_3.add(Dropout(0.5))model_3.add(Dense(1, activation=’sigmoid’))
Once again, we will compile the model and fit the data to our training set. We assign a validation split of 20%, and a batch_size of 16. We will let the model train for 20 epochs, and monitor the performance while training.
model_3.compile(optimizer=’adam’, loss=’binary_crossentropy’, metrics=METRICS)history = model_3.fit(X_train_d, y_train,  validation_split=0.2, batch_size=16,  epochs=20,  )
After the training period, we notice that the model did not achieve the highly impressive performance we were hoping for, however, it did manage to show some improvements relative to the RFC and XGB models. We notice that the loss was minimal indicating that the model was in fact ‘learning’. We also notice that the accuracy was maximized at the range of 99.94% towards the end of the training process. We also notice a precision of around 85%, and a recall of around 83% towards the end of the model’s training process. In the end, we were able to increase the model’s recall by 3%.
After reviewing the performance of the models, we notice a few things. We were able to maintain a high accuracy, and we were able to increase the recall by 3% at the expense of the precision metric. We notice that the recall metric shows some high fluctuations throughout the training cycle. Notice that at epoch 7.5 we reach a minimum of 55%, which slowly decreases over time as we reach the 20th epoch. It is likely with more training perhaps towards epochs 30 or 40, we will see the recall metric stabilizing and possibly increasing alongside precision. If you are following this tutorial in Python, I suggest you train the model beyond 20 epochs and see if the lines converge and show superior performance after a longer duration. In the end, we managed to successfully develop a number of models such as the RFC, XGB, and ANN which are able to detect anomalies with a great deal of success.