Unsupervised Learning For Anomaly Detection | by Vardaan Bajaj | Towards Data Science

In this post, we’ll be going through:
The Need for Anomaly DetectionBaseline Algorithm for Anomaly Detection with underlying MathematicsEvaluating an Anomaly Detection AlgorithmExtending Baseline Algorithm for a Multivariate Gaussian Distribution and the use of Mahalanobis DistanceDetection of Fraudulent Transactions on a Credit Card Dataset available on Kaggle
The Need for Anomaly Detection
Baseline Algorithm for Anomaly Detection with underlying Mathematics
Evaluating an Anomaly Detection Algorithm
Extending Baseline Algorithm for a Multivariate Gaussian Distribution and the use of Mahalanobis Distance
Detection of Fraudulent Transactions on a Credit Card Dataset available on Kaggle
In the previous post, we had an in-depth look at Principal Component Analysis (PCA) and the problem it tries to solve. To consolidate our concepts, we also visualized the results of PCA on the MNIST digit dataset on Kaggle. In each post so far, we discussed either a supervised learning algorithm or an unsupervised learning algorithm but in this post, we’ll be discussing Anomaly Detection algorithms, which can be solved using both, supervised and unsupervised learning methods. Here though, we’ll discuss how unsupervised learning is used to solve this problem and also understand why anomaly detection using unsupervised learning is beneficial in most cases.
Anomaly is a synonym for the word ‘outlier’. Anomaly detection (or outlier detection) is the identification of rare items, events or observations which raise suspicions by differing significantly from the majority of the data. Anomalous activities can be linked to some kind of problems or rare events such as bank fraud, medical problems, structural defects, malfunctioning equipment etc.
According to a research by Domo published in June 2018, over 2.5 quintillion bytes of data were created every single day, and it was estimated that by 2020, close to 1.7MB of data would be created every second for every person on earth. And in times of CoViD-19, when the world economy has been stabilized by online businesses and online education systems, the number of users using the internet have increased with increased online activity and consequently, it’s safe to assume that data generated per person has increased manifold. In simple words, the digital footprint for a person as well as for an organization has sky-rocketed. The servers are flooded with user activity and this poses a huge challenge for all businesses. Fraudulent activities in banking systems, fake ids and spammers on social media and DDoS attacks on small businesses have the potential to collapse the respective organizations and this can only be prevented if there are ways to detect such malicious (anomalous) activity. Since there are tonnes of ways to induce a particular cyber-attack, it is very difficult to have information about all these attacks beforehand in a dataset. But, since the majority of the user activity online is normal, we can capture almost all the ways which indicate normal behaviour. And from the inclusion-exclusion principle, if an activity under scrutiny does not give indications of normal activity, we can predict with high confidence that the given activity is anomalous.
Let us understand the above with an analogy. In the world of human diseases, normal activity can be compared with diseases such as malaria, dengue, swine-flu, etc. for which we have a cure. SarS-CoV-2 (CoViD-19), on the other hand, is an anomaly that has crept into our world of diseases, which has characteristics of a normal disease with the exception of delayed symptoms. Had the SarS-CoV-2 anomaly been detected in its very early stage, its spread could have been contained significantly and we wouldn’t have been facing a pandemic today. Since SarS-CoV-2 is an entirely new anomaly that has never been seen before, even a supervised learning procedure to detect this as an anomaly would have failed since a supervised learning model just learns patterns from the features and labels in the given dataset whereas by providing normal data of pre-existing diseases to an unsupervised learning algorithm, we could have detected this virus as an anomaly with high probability since it would not have fallen into the category (cluster) of normal diseases. I hope this gives enough intuition to realize the importance of Anomaly Detection and why unsupervised learning methods are preferred over supervised learning methods in most cases for such tasks.
There are different types of anomaly detection algorithms but the one we’ll be discussing today will start from feature-by-feature probability distribution and how it leads us to using Mahalanobis Distance for the anomaly detection algorithm.
One of the most important assumptions for an unsupervised anomaly detection algorithm is that the dataset used for the learning purpose is assumed to have all non-anomalous training examples (or very very small fraction of anomalous examples). This might seem a very bold assumption but we just discussed in the previous section how less probable (but highly dangerous) an anomalous activity is. Also, the goal of the anomaly detection algorithm through the data fed to it is to learn the patterns of a normal activity so that when an anomalous activity occurs, we can flag it through the inclusion-exclusion principle. With this thing in mind, let’s discuss the anomaly detection algorithm in detail.
Data points in a dataset usually have a certain type of distribution like the Gaussian (Normal) Distribution. Not all datasets follow a normal distribution but we can always apply certain transformation to features (which we’ll discuss in a later section) that convert the data’s distribution into a Normal Distribution, without any kind of loss in feature variance. Set of data points with Gaussian Distribution look as follows:
From the histogram above, we see that data points follow a Gaussian Probability Distribution and most of the data points are spread around a central (mean) location. When the frequency values on y-axis are mentioned as probabilities, the area under the bell curve is always equal to 1. Before we continue our discussion, have a look at the following normal distributions.
What do we observe? All the line graphs above represent Normal Probability Distributions and still, they are different. This is because each distribution above has 2 parameters that make each plot unique: the mean (μ) and variance (σ2) of data. The red, blue and yellow distributions are all centered at 0 mean, but they are all different because they have different spreads about their mean values. On the other hand, the green distribution does not have 0 mean but still represents a Normal Distribution. From this, it’s clear that to describe a Normal Distribution, the 2 parameters, μ and σ2 control how the distribution will look like. As a matter of fact, 68% of data lies around the first standard deviation (σ) from the mean (34% on each side), 26.2 % data lies between the first and second standard deviation (σ) (13.1% on each side) and so on. This means that roughly 95% of the data in a Gaussian distribution lies within 2 standard deviations from the mean. We can use this to verify whether real world datasets have a (near perfect) Gaussian Distribution or not.
Real world data has a lot of features. If each feature has its data distributed in a Normal fashion, then we can proceed further, otherwise, it is recommended to convert the given distribution into a normal one. The following figure shows what transformations we can apply to a given probability distribution to convert it to a Normal Distribution. The resultant transformation may not result in a perfect probability distribution, but it results in a good enough approximation that makes the algorithm work well.
Remember the assumption we made that all the data used for training is assumed to be non-anomalous (or should have a very very small fraction of anomalies). We’ll put that to use here. We saw earlier that almost 95% of data in a normal distribution lies within two standard-deviations from the mean. Since the likelihood of anomalies in general is very low, we can say with high confidence that data points spread near the mean are non-anomalous. And since the probability distribution values between mean and two standard-deviations are large enough, we can set a value in this range as a threshold (a parameter that can be tuned), where feature values with probability larger than this threshold indicate that the given feature’s values are non-anomalous, otherwise it’s anomalous. Consider that there are a total of n features in the data. For a feature x(i) with a threshold value of ε(i), all data points’ probability that are above this threshold are non-anomalous data points i.e. non-anomalous data points w.r.t. a particular feature are represented as:
Where P(X(i): μ(i), σ(i)) represents the probability of a given training example for feature X(i) which is characterized by the mean of μ(i) and variance of σ(i).
The above case flags a data point as anomalous/non-anomalous on the basis of a particular feature. In reality, we cannot flag a data point as an anomaly based on a single feature. Only when a combination of all the probability values for all features for a given data point is calculated can we say with high confidence whether a data point is an anomaly or not. This scenario can be extended from the previous scenario and can be represented by the following equation.
A data point is deemed non-anomalous when
Now that we know how to flag an anomaly using all n-features of the data, let us quickly see how we can calculate P(X(i)) for a given normal probability distribution. For that, we also need to calculate μ(i) and σ2(i), which is done as follows.
where m is the number of training examples and n is the number of features.
We now have everything we need to know to calculate the probabilities of data points in a normal distribution. Let’s go through an example and see how this process works. Consider data consisting of 2 features x1 and x2 with Normal Probability Distribution as follows:
If we consider a data point in the training set, then we’ll have to calculate it’s probability values wrt x1 and x2 separately and then multiply them in order to get the final result, which then we’ll compare with the threshold value to decide whether it’s an anomaly or not. To better visualize things, let us plot x1 and x2 in a 2-D graph as follows:
The combined probability distribution for both the features will be represented in 3-D as follows:
The resultant probability distribution is a Gaussian Distribution. Now, if we consider a training example around the central value, we can see that it will have a higher probability value rather than data points far away since it lies pretty high on the probability distribution curve. We saw earlier that approximately 95% of the training data lies within 2 standard deviations from the mean which led us to choose the value of ε around the border probability value of second standard deviation, which however, can be tuned depending from task to task. This indicates that data points lying outside the 2nd standard deviation from mean have a higher probability of being anomalous, which is evident from the purple shaded part of the probability distribution in the above figure.
Any anomaly detection algorithm, whether supervised or unsupervised needs to be evaluated in order to see how effective the algorithm is. Since the number of occurrence of anomalies is relatively very small as compared to normal data points, we can’t use accuracy as an evaluation metric because for a model that predicts everything as non-anomalous, the accuracy will be greater than 99.9% and we wouldn’t have captured any anomaly. This is completely undesirable. Our requirement is to evaluate how many anomalies did we detect and how many did we miss. Predicting a non-anomalous example as anomalous will do almost no harm to any system but predicting an anomalous example as non-anomalous can cause significant damage. One metric that helps us in such an evaluation criteria is by computing the confusion matrix of the predicted values. A confusion matrix is a summary of prediction results on a classification problem. The number of correct and incorrect predictions are summarized with count values and broken down by each class. This is the key to the confusion matrix. The confusion matrix shows the ways in which your classification model is confused when it makes predictions. It gives us insight not only into the errors being made by a classifier but more importantly the types of errors that are being made.
A true positive is an outcome where the model correctly predicts the positive class (non-anomalous data as non-anomalous). Similarly, a true negative is an outcome where the model correctly predicts the negative class (anomalous data as anomalous). A false positive is an outcome where the model incorrectly predicts the positive class (non-anomalous data as anomalous) and a false negative is an outcome where the model incorrectly predicts the negative class (anomalous data as non-anomalous). In the case of our anomaly detection algorithm, our goal is to reduce as many false negatives as we can. Lower the number of false negatives, better is the performance of the anomaly detection algorithm.
We have missed a very important detail here. The anomaly detection algorithm we discussed above is an unsupervised learning algorithm, then how do we evaluate its performance? While collecting data, we definitely know which data is anomalous and which is not. The reason for not using supervised learning was that it cannot capture all the anomalies from such a limited number of anomalies. That is why we use unsupervised learning with inclusion-exclusion principle. Suppose we have 10,040 training examples, 10,000 of which are non-anomalous and 40 are anomalous. This data will be divided into training, cross-validation and test set as follows:
Training set: 8,000 non-anomalous examples
Cross-Validation set: 1,000 non-anomalous and 20 anomalous examples
Test set: 1,000 non-anomalous and 20 anomalous examples
This distribution will enable us to capture as many patterns that occur in non-anomalous data points and then we can compare and contrast them with 20 anomalies, each in cross-validation and test set. The point of creating a cross validation set here is to tune the value of the threshold point ε.
We need to know how the anomaly detection algorithm analyses the patterns for non-anomalous data points in order to know whether there is a further scope of improvement. Let’s consider a data distribution in which the plotted points do not assume a circular shape, like the following.
All the red points in the image above are non-anomalous examples. If we consider the point marked in green, using our intelligence we will flag this point as an anomaly. But, the way we the anomaly detection algorithm we discussed works, this point will lie in the region where it can be detected as a normal data point.
The anomaly detection algorithm discussed so far works in circles. The inner circle is representative of the probability values of the normal distribution close to the mean. The second circle, where the green point lies is representative of the probability values that are close the first standard deviation from the mean and so on. This is undesirable because every time we won’t have data whose scatter plot results in a circular distribution in 2-dimensions, spherical distribution in 3-dimensions and so on. We need an anomaly detection algorithm that adapts according to the distribution of the data points and gives good results. Turns out that for this problem, we can use the Mahalanobis Distance (MD) property of a Multi-variate Gaussian Distribution (we’ve been dealing with multivariate gaussian distributions so far).
The Mahalanobis distance (MD) is the distance between two points in multivariate space. In a regular Euclidean space, variables (e.g. x, y, z) are represented by axes drawn at right angles to each other. The distance between any two points can be measured with a ruler. For uncorrelated variables, the Euclidean distance equals the MD. However, if two or more variables are correlated, the axes are no longer at right angles, and the measurements become impossible with a ruler. In addition, if you have more than three variables, you can’t plot them in regular 3D space at all. The MD solves this measurement problem, as it measures distances between points, even correlated points for multiple variables.
The Mahalanobis distance measures distance relative to the centroid — a base or central point which can be thought of as an overall mean for multivariate data. The centroid is a point in multivariate space where all means from all variables intersect. The larger the MD, the further away from the centroid the data point is.
To use Mahalanobis Distance for anomaly detection, we don’t need to compute the individual probability values for each feature. Instead, we can directly calculate the final probability of each data point that considers all the features of the data and above all, due to the non-zero off-diagonal values of Covariance Matrix Σ while calculating Mahalanobis Distance, the resultant anomaly detection curve is no more circular, rather, it fits the shape of the data distribution.
Mahalanobis Distance is calculated using the formula given below.
Once the Mahalanobis Distance is calculated, we can calculate P(X), the probability of the occurrence of a training example, given all n features as follows:
Where |Σ| represents the determinant of the covariance matrix Σ.
The values μ and Σ are calculated as follows:
Finally, we can set a threshold value ε, where all values of P(X) < ε flag an anomaly in the data. However, this value is a parameter and can be tuned using the cross-validation set with the same data distribution we discussed for the previous anomaly detection algorithm.
Before concluding the theoretical section of this post, it must be noted that although using Mahalanobis Distance for anomaly detection is a more generalized approach for anomaly detection, this very reason makes it computationally more expensive than the baseline algorithm. Also, we must have the number training examples m greater than the number of features n (m > n), otherwise the covariance matrix Σ will be non-invertible (i.e. Σ^-1 would become undefined).
Finally we’ve reached the concluding part of the theoretical section of the post. We understood the need of anomaly detection algorithm before we dove deep into the mathematics involved behind the anomaly detection algorithm. I recommend reading the theoretical part more than once if things are a bit cluttered in your head at this point, which is completely normal though. Now, let’s take a look back at the fraudulent credit card transaction dataset from Kaggle, which we solved using Support Vector Machines in this post and solve it using the anomaly detection algorithm.
In this section, we’ll be using Anomaly Detection algorithm to determine fraudulent credit card transactions. Dataset for this problem can be found here. One thing to note here is that the features of this dataset are already computed as a result of PCA. This helps us in 2 ways:
(i) The confidentiality of the user data is maintained.
(ii) The features in the dataset are independent of each other due to PCA transformation.
Let’s start by loading the data in memory in a pandas data frame.
import numpy as npimport pandas as pdimport matplotlib.pyplot as pltfrom sklearn import preprocessingfrom sklearn.neighbors import LocalOutlierFactorfrom sklearn.metrics import confusion_matrix, classification_report,accuracy_scoreimport osfor dirname, _, filenames in os.walk(‘/kaggle/input’):  for filename in filenames    print(os.path.join(dirname, filename))
df = pd.read_csv("/kaggle/input/creditcardfraud/creditcard.csv")df.head()
We proceed with the data pre-processing step. Let’s have a look at how the values are distributed across various features of the dataset.
df.describe()
The original dataset has over 284k+ data points, out of which only 492 are anomalies. Training the model on the entire dataset led to timeout on Kaggle, so I used 20% of the data ( > 56k data points ).
print(df.shape)data= df.sample(frac = 0.2,random_state=1)print(data.shape)
The data has no null values, which can be checked by the following piece of code.
df.isnull().values.any()
Let us plot normal transaction v/s anomalous transactions on a bar graph in order to realize the fraction of fraudulent transactions in the dataset. Additionally, also let us separate normal and fraudulent transactions in datasets of their own.
num_classes = pd.value_counts(df['Class'], sort = True)num_classes.plot(kind = 'bar')plt.title("Transaction Class Distribution")plt.xticks(range(2), ["Normal", "Fraud"])plt.xlabel("Class")plt.ylabel("Frequency")fraud = df[df['Class'] == 1]normal = df[df['Class'] == 0]print(fraud.shape, normal.shape)
In the dataset, we can only interpret the ‘Time’ and ‘Amount’ values against the output ‘Class’. Let us see, if we can find something observations that enable us to visibly differentiate between normal and fraudulent transactions.
f, (ax1, ax2) = plt.subplots(2, 1, sharex=True)f.suptitle('Time of transaction v/s Amount by Class type')ax1.scatter(fraud.Time, fraud.Amount)ax1.set_title('Fraud')ax2.scatter(normal.Time, normal.Amount)ax2.set_title('Normal')plt.xlabel('Time (in secs)')plt.ylabel('Amount')plt.xlim((0, 20000))plt.show()
f, (ax1, ax2) = plt.subplots(2, 1, sharex=True)f.suptitle('Amount per transaction by class')bins = 10ax1.hist(fraud.Amount, bins = bins)ax1.set_title('Fraud')ax2.hist(normal.Amount, bins = bins)ax2.set_title('Normal')plt.xlabel('Amount ($)')plt.ylabel('Number of Transactions')plt.xlim((0, 20000))plt.yscale('log')
From the first plot, we can observe that fraudulent transactions occur at the same time as normal transaction, making time an irrelevant factor. From the second plot, we can see that most of the fraudulent transactions are small amount transactions. This is however not a huge differentiating feature since majority of normal transactions are also small amount transactions.
Before proceeding further, let us have a look at how many fraudulent and non-fraudulent transactions do we have in the reduced dataset (20% of the features) that we’ll use for training the machine learning model to identify anomalies.
fraud = data[data['Class']==1]normal = data[data['Class']==0]anomaly_fraction = len(fraud)/float(len(normal))print(anomaly_fraction)print("Fraud Cases: " + str(len(fraud)))print("Normal Cases: " + str(len(normal)))
We have just 0.1% fraudulent transactions in the dataset. This means that a random guess by the model should yield 0.1% accuracy for fraudulent transactions. We’ll, however, construct a model that will have much better accuracy than this one. You might be thinking why I’ve mentioned this here. I’ll refer these lines while evaluating the final model’s performance.
Recall that we learnt that each feature should be normally distributed in order to apply the unsupervised anomaly detection algorithm. Let us plot histograms for each feature and see which features don’t represent Gaussian distribution at all.
data.hist(figsize=(15,15), bins = 64)plt.show()
From the above histograms, we can see that ‘Time’, ‘V1’ and ‘V24’ are the ones that don’t even approximate a Gaussian distribution. Let’s drop these features from the model training process. We were going to omit the ‘Time’ feature anyways.
columns = data.columns.tolist()target=columns[-1]columns = columns[:-1]X_train = data.iloc[:45000, :-1]y_train = data.iloc[:45000, -1]X_test = data.iloc[45000:, :-1]y_test = data.iloc[45000:, -1]print(X_train.shape, X_test.shape)print(y_train.shape, y_test.shape)
Let us use the LocalOutlierFactor function from the scikit-learn library in order to use unsupervised learning method discussed above to train the model.
model = LocalOutlierFactor(contamination=anomaly_fraction)y_train_pred = model.fit_predict(X_train)y_train_pred[y_train_pred == 1] = 0y_train_pred[y_train_pred == -1] = 1y_test_pred = model.fit_predict(X_test)y_test_pred[y_test_pred == 1] = 0y_test_pred[y_test_pred == -1] = 1
Now that we have trained the model, let us evaluate the model’s performance by having a look at the confusion matrix for the same as we discussed earlier that accuracy is not a good metric to evaluate any anomaly detection algorithm, especially the one which has such a skewed input data as this one.
import itertoolsclasses = np.array(['0','1'])def plot_confusion_matrix(cm, classes,title='Confusion matrix', cmap=plt.cm.Blues):plt.imshow(cm, interpolation='nearest', cmap=cmap)plt.title(title)plt.colorbar()tick_marks = np.arange(len(classes))plt.xticks(tick_marks, classes, rotation=45)plt.yticks(tick_marks, classes)fmt = 'd'thresh = cm.max() / 2.for i, j in itertools.product(range(cm.shape[0]), range(cm.shape[1])):plt.text(j, i, format(cm[i, j], fmt),horizontalalignment="center",color="white" if cm[i, j] > thresh else "black")plt.tight_layout()plt.ylabel('True label')plt.xlabel('Predicted label')
The above function is a helper function that enables us to construct a confusion matrix. We’ll plot confusion matrices to evaluate both training and test set performances.
cm_train = confusion_matrix(y_train, y_train_pred)plot_confusion_matrix(cm_train,["Normal", "Fraud"])
cm_test = confusion_matrix(y_test_pred, y_test)plot_confusion_matrix(cm_test,["Normal", "Fraud"])
We see that on the training set, the model detects 44,870 normal transactions correctly and only 55 normal transactions are labelled as fraud. This is quite good, but this is not something we are concerned about. We can see that out of the 75 fraudulent transactions in the training set, only 14 have been captured correctly whereas 61 are misclassified, which is a problem. When we compare this performance to the random guess probability of 0.1%, it is a significant improvement form that but not convincing enough. Even in the test set, we see that 11,936/11,942 normal transactions are correctly predicted, but only 6/19 fraudulent transactions are correctly captured.
When I was solving this dataset, even I was surprised for a moment, but then I analysed the dataset critically and came to the conclusion that for this problem, this is the best unsupervised learning can do. One reason why unsupervised learning did not perform well enough is because most of the fraudulent transactions did not have much unusual characteristics regarding them which can be well separated from normal transactions. This is supported by the ‘Time’ and ‘Amount’ graphs that we plotted against the ‘Class’ feature. And I feel that this is the main reason that labels are provided with the dataset which flag transactions as fraudulent and non-fraudulent, since there aren’t any visibly distinguishing features for fraudulent transactions. The accuracy of detecting anomalies on the test set is 25%, which is way better than a random guess (the fraction of anomalies in the dataset is < 0.1%) despite having the accuracy of 99.84% accuracy on the test set.
print('Total fraudulent transactions detected in training set: ' + str(cm_train[1][1]) + ' / ' + str(cm_train[1][1]+cm_train[1][0]))print('Total non-fraudulent transactions detected in training set: ' + str(cm_train[0][0]) + ' / ' + str(cm_train[0][1]+cm_train[0][0]))print('Probability to detect a fraudulent transaction in the training set: ' + str(cm_train[1][1]/(cm_train[1][1]+cm_train[1][0])))print('Probability to detect a non-fraudulent transaction in the training set: ' + str(cm_train[0][0]/(cm_train[0][1]+cm_train[0][0])))print("Accuracy of unsupervised anomaly detection model on the training set: "+str(100*(cm_train[0][0]+cm_train[1][1]) / (sum(cm_train[0]) + sum(cm_train[1]))) + "%")
print('Total fraudulent transactions detected in test set: ' + str(cm_test[1][1]) + ' / ' + str(cm_test[1][1]+cm_test[1][0]))print('Total non-fraudulent transactions detected in test set: ' + str(cm_test[0][0]) + ' / ' + str(cm_test[0][1]+cm_test[0][0]))print('Probability to detect a fraudulent transaction in the test set: ' + str(cm_test[1][1]/(cm_test[1][1]+cm_test[1][0])))print('Probability to detect a non-fraudulent transaction in the test set: ' + str(cm_test[0][0]/(cm_test[0][1]+cm_test[0][0])))print("Accuracy of unsupervised anomaly detection model on the test set: "+str(100*(cm_test[0][0]+cm_test[1][1]) / (sum(cm_test[0]) + sum(cm_test[1]))) + "%")
The entire code for this post can be found here.
That’s it for this post. This post also marks the end of a series of posts on Machine Learning. It was a pleasure writing these posts and I learnt a lot too in this process. I believe that we understand things only as good as we teach them and in these posts, I tried my best to simplify things as much as I could. Mathematics got a bit complicated in the last few posts, but that’s how these topics were. Thanks for reading these posts.