Outlier or Inlier: Types of Algorithms to Detect Anomalous Behavior
By: Pınar Ersoy
Tags: AI, Machine Learning, Data Science, Anomaly Detection, Algorithms

Anomaly detection is a critical problem that has been researched within diverse research areas and application disciplines. This article aims to construct a structured and comprehensive overview of the selected algorithms for anomaly detection by targeting data scientists, data analysts, and machine learning specialists as an audience.

Concept of Anomaly Detection

An unexpected change that performs highly divergent attitudes from other observations in a time period can be represented as abnormal behavior. In other words, Anomaly Detection can be defined as the measure of specifying the outliers in the existing dataset which acts considerably different from the rest of the data points by profiling them as non-conforming normal points.

Anomalous points might be produced by errors in the data; however, it could point out to a historically or currently existing unidentified or hidden process or behavior by Hawkins.

As the publicly available data volume reaches in mass amounts, outlier detecting algorithms are modified to run on these data sets to be able to predict the unusual patterns. For instance, a “suspiciously high” count of login trials might outline a possible cyber intrusion or a considerable increase in incoming network traffic can be pointed to malicious activity in network systems. Considering these activities, they hold a shared aspect that they are “appealing” and “unusual” to the data scientists and data analysts. The “curiosity” or real-life applicability of anomalies is an essential element of anomaly detection.

Types of Anomalies

There exist three different kinds of anomalies in the literature.

Figure 1. Types & Examples of Anomaly Detection. (Image by the author)

Descriptions can be found below:

1. Point Anomaly: An anomaly when a distinct item in a dataset is largely dissimilar from others corresponding to its attributes.

Figure 2. The point anomaly is marked with red. (Image by the author)

2. Contextual Anomalies: An anomaly which has a divergence that points to a context-based knowledge. This kind of anomaly may not be recognized when the contextual information is absent.

Figure 3. The contextual anomalous point that can be explained in its context. (Image by Author)

3. Collective Anomalies: Anomalies that are composed of multiple related instances of elements that may not constitute an anomalous point individually. The collective summation of specific events is considered while analyzing outlier behaviors.

Figure 4. Collective Anomaly Detection highlighted in the red line. (Image by the author)

Table of Contents

1. Statistical Approach

1.1. Minimum Covariance Determinant (MCD)

1.2. Principle Component Analysis (PCA) 2. Distance-based Approach

2.1. Local Outlier Factor (LOF)

2.2. Novelty Detection Local Outlier Factor (ND LOF)

2.3. Mahalanobis Distance (MDist) 3. Density-based Approach

3.1. Density-Based Spatial Clustering of Applications with Noise (DBSCAN)

3.2. Ordering Points To Identify the Clustering Structure (OPTICS) 4. Isolation-based Approach

4.1. Isolation Forest (iForest) 5. Classification-based Approach

5.1. One-Class SVM

1. STATISTICAL APPROACH

1.1. Minimum Covariance Determinant (MCD)

Minimum Covariance Determinant (MCD) acts as the covariance estimator that is to be applied to Gaussian-distributed data. It basically searches for the subset of a specified number of data points whose covariance matrix contains the lowest determinant.

Because of the geometrical representation of the covariance matrix, the MCD algorithm tends to learn a rotationally symmetrical shape and works best with elliptically symmetric unimodal distributions. For this reason, it would be more performant to apply this algorithm while detecting outliers on the dataset which belongs to a unimodal distribution, so it is not advised to be used with multi-modal data. The more the size of the data and unimodality gets lower, the more the performance of the algorithm diminishes.

For the formulation and the detailed parameter explanations, please kindly visit this article.

Figure 5. Classical Covariance Matrix (red) vs MCD Matrix (blue). (Image by the author)

Python Library for Implementation:

Algorithm:

H: random subsample from X,

[(I + J + 1) / 2 <= H <= I]



By using XH, the column-wise row vector mean can be computed as µH

For the covariance:: SH = (XH.T * XH) * (H -1)^(-1) The determinant of SH can be computed.

The squared Mahalanobis Distance for each observation in X as

mi = (xi - µH)*SH^(-1) * (xi - µH).T After the computations, XH can be assigned as the subset of H observations with the smallest Mahalanobis Distance.

1.2. Principal Component Analysis (PCA)

This statistical methodology builds up the essentials of multivariate data analysis that uses an orthogonal transformation to apply on a set of observations of probably correlated variables into a set of values of uncorrelated variables in a linear way. It serves a multivariate table as a smaller set of variables to be able to inspect trends, bounces, and outliers. This type of high-level analysis may expose the interconnection between observations and variables.

As one of the statistical anomaly detection approaches, it can be applied to, for instance, mark fraudulent transactions by evaluating applicable features to define what establishes as normal observation and assigning distance metrics to detect possible cases that serve as outliers/anomalies. This kind of approach allows training a model using the present unbalanced dataset.

Figure 6. 2D PCA plot of clusters. (Image by the author)

Python Library for Implementation:

Algorithm:

X = N × m # Sample Covariance Matrix for each data point xi

X = subtract mean x from each column vector xi in X

Σ = (X*X).T

return { λi, ui }i=(1..k) * Top-k vectors are more crucial eigenvectors

2. DENSITY-BASED APPROACHES

2.1. Density-Based Spatial Clustering of Applications with Noise (DBSCAN)

The Density-Based Spatial Clustering of Applications with Noise (DBSCAN) aims to detect anomalous data points with the help of the density of its encircled space that is detached by regions of low-density observations.

Major concepts of this methodology can be listed as density and connectivity which are calculated in terms of local distribution of their nearest neighbors.

There exists two key parameters of the algorithm are:

distance threshold — epsilon (ɛ): The given radius of the neighborhoods around a data point

The given radius of the neighborhoods around a data point min_samples: The minimum number of data points in a neighborhood to create a cluster

With the help of these parameters, the algorithm can be broken down in the following steps:

Detecting the points in the ε (eps neighborhood of every point and identify the core points with more than minimum points neighbors.

(eps neighborhood of every point and identify the core points with more than minimum points neighbors. Find the connected components of inlier points on the neighbor graph, ignoring all non-core points.

Assign each non-inlier point to a nearby cluster if the cluster is an ε (eps neighbor, otherwise assign it to noise.

A naive implementation of this requires storing the neighborhoods in step 1, thus requiring substantial memory. The original DBSCAN algorithm does not require this by performing these steps for one point at a time.

There are two parameters in the DBSCAN algorithm:

ε (epsilon): A point p is a core point if it has neighbors within a given radius MinPts (minimum number of points): The least number of neighbors to form a cluster

Python Library for Implementation:

Algorithm:

DBSCAN(Data, Epsilon, MinPts)

Cluster_Label = 0

for each unvisited observation Point in dataset Data

assign Point as visited

Neighbors = getNeighbors (Point, Epsilon)

if sizeof(Neighbors) < MinPts

assign Point as Outlier

else

Cluster_Label = next cluster

extendClusterSize(Point, Neighbors, Cluster_Label, Epsilon, MinPts)



extendClusterSize(Point, Neighbors, Cluster_Label, Epsilon, MinPts)

add Point to cluster Cluster_Label

for each point Point' in Neighbors

if Point' is not visited

assign Point' as visited

Neighbors' = getNeighbors(Point', Epsilon)

if sizeof(Neighbors') >= MinPts

Neighbors = Neighbors joined with Neighbors'

if Point' is not yet member of any cluster

add Point' to cluster Cluster_Label

2.2. Ordering Points To Identify the Clustering Structure (OPTICS)

Ordering points to identify the clustering structure (OPTICS) can be defined as an algorithm for outlying density-based groups in dimensional data.

It is an approach for detecting density-based clusters in spatial data. Its algorithm methodology is not only related to DBSCAN but also it labels one of DBSCAN algorithm’s crucial gap in identifying differing density-based useful clusters.

There are some key points in the descriptions of the parameters of DBSCAN, and OPTICS as core point, core distance, and reachability distance:

Core point: A point that resides at the center of the cluster can be accepted as a core point by providing minimally MinPts data points are located within its neighbors .

A point that resides at the center of the cluster can be accepted as a core point by providing minimally MinPts data points are located within its neighbors Core Distance: The minimum length of radius required to group provided observations as a core point .

The minimum length of radius required to group provided observations as a . Reachability Distance: This distance metric represents the shortest length that is defined with respect to the cluster’s central point to another data point which cannot be smaller than the core distance. In the below depiction, the reachability distance between (c, a) is assigned as x, since their distance is smaller than the core distance which is not acceptable.

Figure 7. Metrics depictions. (Image by the author)

Python Library for Implementation:

Algorithm:

OPTICS(DB, Epsilon, MinPts)

for each observation Point in dataset DB

Point.reachabilityDistance = UNDEFINED

for each unprocessed point Point of DB

Neighbors = getNeighbors(Point, Epsilon)

assign Point as visited

output Point to the ordered list

if coreDistance(Point, Epsilon, MinPts) != UNDEFINED then

Queue = empty priority queue

updateCluster(Neighbors, Point, Queue, Epsilon, MinPts)

for each next Q in Queue

Neighbors' = getNeighbors(Q, Epsilon)

assign Q as visited

output Q to the ordered list

if coreDistance(Q, Epsilon, MinPts) != UNDEFINED do

updateCluster(Neighbors', Q, Queue, Epsilon, MinPts) updateCluster(N, Point, Queue, Epsilon, MinPts) is

coreDist = coreDistance(Point, Epsilon, MinPts)

for each Observation in N

if Observation is not visited then

newReachabilityDistance = max(coreDist, dist(Point, Observation))

if Observation.reachabilityDistance == UNDEFINED then

Observation.reachabilityDistance = newReachabilityDistance

Queue.insert(Observation, newReachabilityDistance)

if newReachabilityDistance < Observation.reachabilityDistance then

Observation.reachabilityDistance = newReachabilityDistance

Queue.moveObservation(Observation, newReachabilityDistance)

3. DISTANCE-BASED APPROACHES

3.1. Local Outlier Factor (LOF)

The Local Outlier Factor can be described as the calculation that scans through the neighbors of a certain point to discover its density and examine this to the density of other points to measure the distance between them.

It calculates a score matching the degree of irregularity of the observations and measures the local density differentiation of a specified data point comparing to its neighbors. The logic is to identify the samples that have a considerably minor density than their surrounded-points.

Figure 8. Point P has a higher LOF score since its density is lower than its surrounded densities. (Image by the author)

Inlier points are local in the local outlier factor score builds upon how detached a single data node is regarding the encircling region. More accurately, the condition of being local is assigned by k-nearest neighbors, whose gap between points is used to calculate the local density.

By contrasting the inner density of a point to the local densities of its neighbors, one can specify samples which allocates a distinctively lower density than their neighbors can be called as outliers.

Figure 9. LOF score visualization. (Image by the author)

Python Library for Implementation:

Algorithm:

LOF = null

for each observation Point in DB

KNNeighbors = kDistance(Distance, K)

LocalReacbilityDistance = reachDistanceK(KNNeighbors, K)

for each Point in KNNeighbors

tempLOF[i] = sum((LocalReacbilityDistance[Obs ∈ N (Point)]) / LocalReacbilityDistance[i]) / |N(Point)|

LOF = max(LOF, tempLOF)

return top(LOF)

3.2. Novelty Detection with Local Outlier Factor

Novelty detection can be accepted as the description of a novel (new) or undiscovered patterns in the data. The novelties revealed are not considered as anomalous data points; instead, they are been practiced to the data model.

The complicatedness of present highly integrated systems is similar to a restricted comprehension of the connections between the different system components that can be collected. An unavoidable effect of this situation is the presence of an enormous number of potential unusual forms, some of which may not be established as rational which causes regular multi-class classification schemes improper for these operations. A possible resolution to this issue is proposed by the algorithm of novelty detection, in which a definition of being an inlier is trained by creating a model with various examples depicting positive states. Formerly undiscovered patterns are then tested by contrasting them with the model of accepted-inliers, usually proceeded in some mode of novelty score. A novelty score may be generated for these formerly unseen data points with the help of a decision threshold score. The points which substantially diverge from this decision threshold may be treated as outliers.

In other words, novelty detection can be described as the process of acknowledging that test data deviates in some way from the training data. Its practical influence and challenging essence have guided many suggestions for being proposed. These approaches are mostly applied to datasets in which a huge number of cases of the inlier or positive state is available and where there are scarce data to define outlier or negative state.

Figure 10. One-class (left) and multi-class (right) novelty detection. (Image by the author)

Python Library for Implementation:

3.3. Mahalanobis Distance

For the given data source that holds standard normal distribution, the corresponding mean shall be zero while variance shall be observed as 1.

The standard normal distribution has differentiative specifications from the normal distribution that can be expressed by means of the normal distribution having two distinct parameters as μ=0 and σ=1 with a representation of N(μ,σ) → N(0,1).

Figure 11. Normal Distribution (left) vs Standard Normal Distribution (right). (Image by the author)

Mahalanobis Distance is one of the outlier detection approaches which provides a simple means of detecting outliers in multidimensional data where the distance is between a point and a distribution and not between two distinct points.

Figure 12. A point and a distribution. (Image by author)

Python Library for Implementation:

Algorithm:

First Version:



Mahalanobis = [(Y – X)T * SampCov -1 * (Y – X)] ^ 0.5 - X, Y : Pair of observations

- SampCov : Sample covariance matrix

Alternative Version:



Distance_n = [((xn – x̄)^t)*(C)^(-1) * (xn – x̄)] ^ 0.5



- xn = an observation vector

- x̄ = arithmetic mean vector

4. ISOLATION-BASED APPROACH

4.1. Isolation Forest

Isolation Forest also called iForest recognizes abnormal activities as opposed to profiling typical data points. Isolation Forest, similar to any tree ensemble methodology, is based on decision trees. Among these trees, partitions are generated by first arbitrarily choosing a component and afterward choosing a randomly selected value in the scope of min and max of in a specified feature set. The algorithm is based on the idea that outliers represent data points that are few and different.

There exist two variables in this algorithm. One is the number of trees to be built and the size of sub-sampling which enables the following subjects:

(i) to accomplish a low time-complexity with a reduced memory necessity

(ii) to carry out the impacts of swamping, labeling nonoutlier as an outlier, and masking, an unidentified outlier

With the help of these parameters, the isolation forest method does not take into consideration the distance and density metrics to catch anomalous behavior. This approach adds an advantage to iForest over distance-centered or density-centered methodologies since it excludes an extensive load of distance measurement.

Isolation forest works very efficiently in case of having a narrow-sized data set to profile outliers.

Below, the left-hand side graph shows the formed tree after the algorithm is executed. The blue line represents the normal data points while the red line can be accepted as the tree with anomalous data points. On the second graph, a schematic example of a tree and a plotted forest is shown with red-colored lines out abnormal data point existence while blue ones are normal ones.

Figure 13. Isolation Forest visualization with outlier and inlier points. (Image by the author)

4.2. Formulation:

Step 1: Sampling for training a model

Step 2: Random selection of a value that is between minimum and maximum of chosen data points

Step 3: Repeating of step 2 of binary splitting to continue create a forest

Step 4: Continuous feed of data resource to calculate anomaly score for every generated tree and their corresponding average path lengths

At the end of the average path sizes, an outlier is expected to have a shorter path length compared to an inlier.

Figure 14. Average steps to generate an isolation forest by detecting inliers and outliers. (Image by the author)

Python Library for Implementation:

Algorithm:

t = null (the empty tree) if numberOfRow(Leaf) == 1 then return t

Randomly select li a feature of Leaf

Randomly select a split observation O ∈ (min(li), max(li))

Add to t the node N li,Point

Assign Leaf_l and Leaf_R as the matrix

Repeat the looping with Leaf = Leaf_l

Add the obtained tree as the left child of main tree t

Repeat the looping with Leaf = Leaf_r

Add the obtained tree as the right child of main tree t

5. CLASSIFICATION-BASED APPROACHES

5.1. One-Class SVM

The One-Class SVM can be accepted as sensitive to anomalous points which leads to poor performance outlier detection by using the training data with outliers. This predictor is correctly matched for novelty detection when the training set is not surrounded by outlier data points which aim to detect new observation as an outlier instead of detecting the noisy data in the existing data.

A One-Class SVM might give beneficial results in multi-dimensional situations depending on the value of its hyperparameters. The aim of this algorithm to detach data points from the origin to maximize the distance between the subspace dimension to the origin.

This situation proceeds in a result set of binary outputs which grab areas in the input space in which the probability density of the data locates.

In this way, the function returns +1 for inliers in a limited region which contains the space by occupying training observations and −1 for outliers.

Figure 15. Outlier SVM Classifier with (1) for inliers, (-1) for outliers. (Image by the author)

Python Library for Implementation:

Algorithm: