What are techniques to detect anomalies | Towards Data Science

Anomaly detection is the identification of rare items, events, or observations that raise suspicions by differing significantly from the majority of the data. Typically the anomalous items will translate to some kind of problem such as credit card fraud, network intrusion, medical diagnostic, system health monitor.
Anomaly detection works on two basic premise
Anomalies only occur very rarely in the data.
Their features differ from normal instances significantly.
The simplest approach to identifying irregularities in data is to flag the data points that deviate from common statistical properties of distribution, including mean, median, mode, and quartiles.
One of the most popular ways is the Interquartile Range (IQR). IQR is a concept in statistics that is used to measure the statistical dispersion and data variability by dividing the dataset into quartiles.
In simple words, any dataset or any set of observations is divided into four defined intervals based upon the values of the data and how they compare to the entire dataset. A quartile is what divides the data into three points and four intervals.
Image Source: Wikipedia
Interquartile Range (IQR) is important because it is used to define the outliers. It is the difference between the third quartile and the first quartile (IQR = Q3 -Q1). Outliers, in this case, are defined as the observations that are below (Q1 − 1.5x IQR) or above (Q3 + 1.5x IQR)
Image Source: Wikipedia
np.percentile is baked functionality in Python
q75, q25 = np.percentile(x, [75 ,25])iqr = q75 - q25
IQR technique doesn’t work in the following scenarios
The pattern is based on seasonality. This involves more sophisticated methods, such as decomposing the data into multiple trends to identify the change in seasonality.
The pattern is based on seasonality. This involves more sophisticated methods, such as decomposing the data into multiple trends to identify the change in seasonality.
2. The definition of abnormal or normal may frequently change, as malicious adversaries constantly adapt themselves
Clustering is one of the most popular concepts in the domain of unsupervised learning. The underlying premise is similar data points tend to belong to similar groups or clusters, as determined by their distance from local centroids.
K-means is a widely used clustering algorithm. It creates ‘k’ similar clusters of data points. Data instances that fall outside of these groups could potentially be marked as anomalies. Other clustering algorithms such as hierarchal clustering and DB scan can also be used to detect outliers.
The way the K-means algorithm works is as follows:
Specify the number of clusters K.Initialize centroids by first shuffling the dataset and then randomly selecting K data points for the centroids without replacement.Calculate distances between the centroids and the data points.Keep iterating until there is no change to the centroids. i.e assignment of data points to clusters isn’t changing.
Specify the number of clusters K.
Initialize centroids by first shuffling the dataset and then randomly selecting K data points for the centroids without replacement.
Calculate distances between the centroids and the data points.
Keep iterating until there is no change to the centroids. i.e assignment of data points to clusters isn’t changing.
Initialize random centroids
Initialize random centroids
You start the process by taking three(as we decided K to be 3) random points (in the form of (x, y)). These points are called centroids which is just a fancy name for denoting centers. Let’s name these three points — C1, C2, and C3 so that you can refer them later.
2. Calculate distances between the centroids and the data points
Next, you measure the distances of the data points from these three randomly chosen points. A very popular choice of distance measurement function, in this case, is the Euclidean distance.
Briefly, if there are n points on a 2D space(just like the above figure) and their coordinates are denoted by (x_i, y_i), then the Euclidean distance between any two points ((x1, y1) and(x2, y2)) on this space is given by:
Suppose the coordinates of C1, C2 and C3 are — (-1, 4), (-0.2, 1.5), and (2, 2.5) respectively. Let’s now write a few lines of Python code which will calculate the Euclidean distances between the data-points and these randomly chosen centroids. We start by initializing the centroids.
# Initialize the centroidsc1 = (-1, 4)c2 = (-0.2, 1.5)c3 = (2, 2.5)
Next, we write a small helper function to calculate the Euclidean distances between the data points and centroids.
# A helper function to calculate the Euclidean distance between the data points and the centroidsdef calculate_distance(centroid, X, Y):    distances = []            # Unpack the x and y coordinates of the centroid    c_x, c_y = centroid            # Iterate over the data points and calculate the distance using the           # given formula    for x, y in list(zip(X, Y)):        root_diff_x = (x - c_x) ** 2        root_diff_y = (y - c_y) ** 2        distance = np.sqrt(root_diff_x + root_diff_y)        distances.append(distance)            return distances
We can now apply this function to the data points and assign the results accordingly.
# Calculate the distance and assign them to the DataFrame accordinglydata['C1_Distance'] = calculate_distance(c1, data.X_value, data.Y_value)data['C2_Distance'] = calculate_distance(c2, data.X_value, data.Y_value)data['C3_Distance'] = calculate_distance(c3, data.X_value, data.Y_value)# Preview the dataprint(data.head())
3. Compare, assign, mean and repeat
This is fundamentally the last step of the K-Means clustering algorithm. Once you have the distances between the data points and the centroids, you compare the distances and take the smallest ones. The centroid to which the distance for a particular data point is the smallest, that centroid gets assigned as the cluster for that particular data point.
Let’s do this programmatically.
# Get the minimum distance centroids    data['Cluster'] = data[['C1_Distance', 'C2_Distance', 'C3_Distance']].apply(np.argmin, axis =1)    # Map the centroids accordingly and rename them    data['Cluster'] = data['Cluster'].map({'C1_Distance': 'C1', 'C2_Distance': 'C2', 'C3_Distance': 'C3'})    # Get a preview of the data    print(data.head(10))
Now comes the most interesting part of updating the centroids by determining the mean values of the coordinates of the data points (which should be belonging to some centroid by now). Hence the name K-Means. This is how the mean calculation looks like:
The following lines of code do this for you:
# Calculate the coordinates of the new centroid from cluster 1x_new_centroid1 = data[data['Cluster']=='C1']['X_value'].mean()y_new_centroid1 = data[data['Cluster']=='C1']['Y_value'].mean()# Calculate the coordinates of the new centroid from cluster 2x_new_centroid2 = data[data['Cluster']=='C3']['X_value'].mean()y_new_centroid2 = data[data['Cluster']=='C3']['Y_value'].mean()# Print the coordinates of the new centroidsprint('Centroid 1 ({}, {})'.format(x_new_centroid1, y_new_centroid1))print('Centroid 2 ({}, {})'.format(x_new_centroid2, y_new_centroid2))
This process is repeated until the coordinates of the centroids do not get updated anymore.
K-means algorithm is popular and used in a variety of applications such as image compression, document classification. The goal of K-mean is to group data points into distinct non-overlapping subgroups. It does a very good job when the clusters have a kind of spherical shape. However, it suffers as the geometric shapes of clusters deviate from spherical shapes. Moreover, it also doesn’t learn the number of clusters from the data and requires it to be pre-defined.
Isolation Forest is an unsupervised learning algorithm that belongs to the ensemble decision trees family. This approach is different from all previous methods. All the previous ones were trying to find the normal region of the data then identifies anything outside of this defined region to be an outlier or anomalous. This method works differently. It explicitly isolates anomalies instead of profiling and constructing normal points and regions by assigning a score to each data point. It takes advantage of the fact that anomalies are the minority data points and that they have attribute-values that are very different from those of normal instances.
IsolationForest ‘isolates’ observations by randomly selecting a feature and then randomly selecting a split value between the maximum and minimum values of the selected feature.
Since recursive partitioning can be represented by a tree structure, the number of splittings required to isolate a sample is equivalent to the path length from the root node to the terminating node.
This path length averaged over a forest of such random trees, is a measure of normality and our decision function.
Random partitioning produces noticeable shorter paths for anomalies. Hence, when a forest of random trees collectively produces shorter path lengths for particular samples, they are highly likely to be anomalies.
This algorithm works great with very high dimensional datasets and it proved to be a very effective way of detecting anomalies.
This paper covers full details on how isolation forest works.
import numpy as npimport matplotlib.pyplot as pltfrom sklearn.ensemble import IsolationForestrng = np.random.RandomState(42)# Generate train dataX = 0.3 * rng.randn(100, 2)X_train = np.r_[X + 2, X - 2]# Generate some regular novel observationsX = 0.3 * rng.randn(20, 2)X_test = np.r_[X + 2, X - 2]# Generate some abnormal novel observationsX_outliers = rng.uniform(low=-4, high=4, size=(20, 2))# fit the modelclf = IsolationForest(max_samples=100, random_state=rng)clf.fit(X_train)y_pred_train = clf.predict(X_train)y_pred_test = clf.predict(X_test)y_pred_outliers = clf.predict(X_outliers)# plot the line, the samples, and the nearest vectors to the planexx, yy = np.meshgrid(np.linspace(-5, 5, 50), np.linspace(-5, 5, 50))Z = clf.decision_function(np.c_[xx.ravel(), yy.ravel()])Z = Z.reshape(xx.shape)plt.title("IsolationForest")plt.contourf(xx, yy, Z, cmap=plt.cm.Blues_r)b1 = plt.scatter(X_train[:, 0], X_train[:, 1], c='white',                 s=20, edgecolor='k')b2 = plt.scatter(X_test[:, 0], X_test[:, 1], c='green',                 s=20, edgecolor='k')c = plt.scatter(X_outliers[:, 0], X_outliers[:, 1], c='red',                s=20, edgecolor='k')plt.axis('tight')plt.xlim((-5, 5))plt.ylim((-5, 5))plt.legend([b1, b2, c],           ["training observations",            "new regular observations", "new abnormal observations"],           loc="upper left")plt.show()
Source: scikit-learn.org
This post provides an overview of different techniques to detect anomalies in data. It ranges from the usage of simple statistical methods like standard deviation to the unsupervised learning algorithms like Isolation Forest. Every method has its advantages and disadvantages. E.g. Interquartile (IQR) doesn’t work for seasonal patterns, K-Means clustering is good at grouping data into distinct non-overlapping subgroups. It does a very good job when the clusters have a kind of spherical shape. Isolation Forest offers a contrarian approach to detect anomalies. It takes advantage of the fact that anomalies are the minority data points and that they have attribute-values that are very different from those of normal instances. To be a good practitioner, it’s good to know the assumptions behind algorithms/methods so that you would have a pretty good idea about the strength and weaknesses of each method. This will help you decide when to use each method and under what circumstances.