Anomaly Detection using Autoencoders | by Renu Khandelwal | Towards Data Science

Learn what are AutoEncoders, how they work, their usage, and finally implement Autoencoders for anomaly detection.
AutoEncoder is a generative unsupervised deep learning algorithm used for reconstructing high-dimensional input data using a neural network with a narrow bottleneck layer in the middle which contains the latent representation of the input data.
Autoencoder consists of an Encoder and a Decoder.
Encoder network: Accepts high-dimensional input data and translates it to latent low-dimensional data. The input size to an Encoder network is larger than its output size.
Decoder network: The Decoder network receives the input from the Encoder coder’s output. Decoder’s objective is to reconstruct the input data. The output size of a Decoder network is larger than its input size.
The Autoencoder accepts high-dimensional input data, compress it down to the latent-space representation in the bottleneck hidden layer; the Decoder takes the latent representation of the data as an input to reconstruct the original input data.
Dimensionality Reduction. The Encoder encodes the input into the hidden layer to reduce the dimensionality of linear and nonlinear data; hence it is more powerful than PCA.
Recommendation Engines
Anomaly Detection: Autoencoders tries to minimize the reconstruction error as part of its training. Anomalies are detected by checking the magnitude of the reconstruction loss.
Denoising Images: An image that is corrupted can be restored to its original version.
Image recognition: Stacked autoencoder are used for image recognition by learning the different features of an image.
Image generation: Variational Autoencoder(VAE), a type of autoencoders, is used to generate images.
Read about different types of Autoencoder here.
Follow the following steps to detect anomalies in a high-dimension dataset. You can apply this to unbalanced datasets too.
During the training, input only normal transactions to the Encoder. The bottleneck layer will learn the latent representation of the normal input data.
The Decoder will use the bottleneck layers output to reconstruct the normal transactions of the original input data.
A fraudulent transaction will be different from a normal transaction. The Autoencoder will have trouble reconstructing the fraudulent transaction, and hence the reconstruction error will be high.
You can flag a new transaction is fraudulent based on a specified threshold value for the reconstruction error.
Dataset used here is Credit Card Fraud Detection from Kaggle.
Import required libraries
import pandas as pdimport numpy as npimport tensorflow as tfimport matplotlib.pyplot as pltimport seaborn as snsfrom sklearn.model_selection import train_test_splitfrom sklearn.preprocessing import StandardScalerfrom sklearn.metrics import confusion_matrix, recall_score, accuracy_score, precision_scoreRANDOM_SEED = 2021 TEST_PCT = 0.3LABELS = ["Normal","Fraud"]
Read the dataset
I had downloaded the data from Kaggle and stored it in the local directory.
dataset = pd.read_csv("creditcard.csv")
Exploratory Data Analysis
#check for any  nullvalues print("Any nulls in the dataset ",dataset.isnull().values.any() )print('-------')print("No. of unique labels ", len(dataset['Class'].unique()))print("Label values ",dataset.Class.unique())#0 is for normal credit card transaction#1 is for fraudulent credit card transactionprint('-------')print("Break down of the Normal and Fraud Transactions")print(pd.value_counts(dataset['Class'], sort = True) )
Visualize the dataset
plotting the number of normal and fraud transactions in the dataset.
#Visualizing the imbalanced datasetcount_classes = pd.value_counts(dataset['Class'], sort = True)count_classes.plot(kind = 'bar', rot=0)plt.xticks(range(len(dataset['Class'].unique())), dataset.Class.unique())plt.title("Frequency by observation number")plt.xlabel("Class")plt.ylabel("Number of Observations");
Visualizing the amount for normal and fraud transactions.
# Save the normal and fradulent transactions in separate dataframenormal_dataset = dataset[dataset.Class == 0] fraud_dataset = dataset[dataset.Class == 1]#Visualize transactionamounts for normal and fraudulent transactionsbins = np.linspace(200, 2500, 100)plt.hist(normal_dataset.Amount, bins=bins, alpha=1, density=True, label='Normal')plt.hist(fraud_dataset.Amount, bins=bins, alpha=0.5, density=True, label='Fraud')plt.legend(loc='upper right')plt.title("Transaction amount vs Percentage of transactions")plt.xlabel("Transaction amount (USD)")plt.ylabel("Percentage of transactions");plt.show()
Create train and test dataset
Checking on the dataset
Time and Amount are the columns that are not scaled, so applying StandardScaler to only Amount and Time columns. Normalizing the values between 0 and 1 did not work great for the dataset.
sc=StandardScaler()dataset['Time'] = sc.fit_transform(dataset['Time'].values.reshape(-1, 1))dataset['Amount'] = sc.fit_transform(dataset['Amount'].values.reshape(-1, 1))
The last column in the dataset is our target variable.
raw_data = dataset.values# The last element contains if the transaction is normal which is represented by a 0 and if fraud then 1labels = raw_data[:, -1]# The other data points are the electrocadriogram datadata = raw_data[:, 0:-1]train_data, test_data, train_labels, test_labels = train_test_split(    data, labels, test_size=0.2, random_state=2021)
Normalize the data to have a value between 0 and 1
min_val = tf.reduce_min(train_data)max_val = tf.reduce_max(train_data)train_data = (train_data - min_val) / (max_val - min_val)test_data = (test_data - min_val) / (max_val - min_val)train_data = tf.cast(train_data, tf.float32)test_data = tf.cast(test_data, tf.float32)
Use only normal transactions to train the Autoencoder.
Normal data has a value of 0 in the target variable. Using the target variable to create a normal and fraud dataset.
train_labels = train_labels.astype(bool)test_labels = test_labels.astype(bool)#creating normal and fraud datasetsnormal_train_data = train_data[~train_labels]normal_test_data = test_data[~test_labels]fraud_train_data = train_data[train_labels]fraud_test_data = test_data[test_labels]print(" No. of records in Fraud Train Data=",len(fraud_train_data))print(" No. of records in Normal Train data=",len(normal_train_data))print(" No. of records in Fraud Test Data=",len(fraud_test_data))print(" No. of records in Normal Test data=",len(normal_test_data))
Set the training parameter values
nb_epoch = 50batch_size = 64input_dim = normal_train_data.shape[1] #num of columns, 30encoding_dim = 14hidden_dim_1 = int(encoding_dim / 2) #hidden_dim_2=4  learning_rate = 1e-7
Create the Autoencoder
The architecture of the autoencoder is shown below.
#input Layerinput_layer = tf.keras.layers.Input(shape=(input_dim, ))#Encoderencoder = tf.keras.layers.Dense(encoding_dim, activation="tanh",                                activity_regularizer=tf.keras.regularizers.l2(learning_rate))(input_layer)encoder=tf.keras.layers.Dropout(0.2)(encoder)encoder = tf.keras.layers.Dense(hidden_dim_1, activation='relu')(encoder)encoder = tf.keras.layers.Dense(hidden_dim_2, activation=tf.nn.leaky_relu)(encoder)# Decoderdecoder = tf.keras.layers.Dense(hidden_dim_1, activation='relu')(encoder)decoder=tf.keras.layers.Dropout(0.2)(decoder)decoder = tf.keras.layers.Dense(encoding_dim, activation='relu')(decoder)decoder = tf.keras.layers.Dense(input_dim, activation='tanh')(decoder)#Autoencoderautoencoder = tf.keras.Model(inputs=input_layer, outputs=decoder)autoencoder.summary()
Define the callbacks for checkpoints and early stopping
cp = tf.keras.callbacks.ModelCheckpoint(filepath="autoencoder_fraud.h5",                               mode='min', monitor='val_loss', verbose=2, save_best_only=True)# define our early stoppingearly_stop = tf.keras.callbacks.EarlyStopping(    monitor='val_loss',    min_delta=0.0001,    patience=10,    verbose=1,     mode='min',    restore_best_weights=True
Compile the Autoencoder
autoencoder.compile(metrics=['accuracy'],                    loss='mean_squared_error',                    optimizer='adam')
Train the Autoencoder
history = autoencoder.fit(normal_train_data, normal_train_data,                    epochs=nb_epoch,                    batch_size=batch_size,                    shuffle=True,                    validation_data=(test_data, test_data),                    verbose=1,                    callbacks=[cp, early_stop]                    ).history
Plot training and test loss
plt.plot(history['loss'], linewidth=2, label='Train')plt.plot(history['val_loss'], linewidth=2, label='Test')plt.legend(loc='upper right')plt.title('Model loss')plt.ylabel('Loss')plt.xlabel('Epoch')#plt.ylim(ymin=0.70,ymax=1)plt.show()
Detect Anomalies on test data
Anomalies are data points where the reconstruction loss is higher
To calculate the reconstruction loss on test data, predict the test data and calculate the mean square error between the test data and the reconstructed test data.
test_x_predictions = autoencoder.predict(test_data)mse = np.mean(np.power(test_data - test_x_predictions, 2), axis=1)error_df = pd.DataFrame({'Reconstruction_error': mse,                        'True_class': test_labels})
Plotting the test data points and their respective reconstruction error sets a threshold value to visualize if the threshold value needs to be adjusted.
threshold_fixed = 50groups = error_df.groupby('True_class')fig, ax = plt.subplots()for name, group in groups:    ax.plot(group.index, group.Reconstruction_error, marker='o', ms=3.5, linestyle='',            label= "Fraud" if name == 1 else "Normal")ax.hlines(threshold_fixed, ax.get_xlim()[0], ax.get_xlim()[1], colors="r", zorder=100, label='Threshold')ax.legend()plt.title("Reconstruction error for normal and fraud data")plt.ylabel("Reconstruction error")plt.xlabel("Data point index")plt.show();
Detect anomalies as points where the reconstruction loss is greater than a fixed threshold. Here we see that a value of 52 for the threshold will be good.
Evaluating the performance of the anomaly detection
threshold_fixed =52pred_y = [1 if e > threshold_fixed else 0 for e in error_df.Reconstruction_error.values]error_df['pred'] =pred_yconf_matrix = confusion_matrix(error_df.True_class, pred_y)plt.figure(figsize=(4, 4))sns.heatmap(conf_matrix, xticklabels=LABELS, yticklabels=LABELS, annot=True, fmt="d");plt.title("Confusion matrix")plt.ylabel('True class')plt.xlabel('Predicted class')plt.show()# print Accuracy, precision and recallprint(" Accuracy: ",accuracy_score(error_df['True_class'], error_df['pred']))print(" Recall: ",recall_score(error_df['True_class'], error_df['pred']))print(" Precision: ",precision_score(error_df['True_class'], error_df['pred']))
As our dataset is highly imbalanced, we see a high accuracy but a low recall and precision.
Things to further improve precision and recall would add more relevant features, different architecture for autoencoder, different hyperparameters, or a different algorithm.
Autoencoder can be used as an anomaly detection algorithm when we have an unbalanced dataset where we have a lot of good examples and only a few anomalies. Autoencoders are trained to minimize reconstruction error. When we train the autoencoders on normal data or good data, we can hypothesize that the anomalies will have higher reconstruction errors than the good or normal data.