Anomaly Detection Techniques
By: Shabari Girish
Tags: Anomaly Detection, Data Science, Supervised Learning, Machine Learning, Outliers

Our next segment involves Machine Learning with Anomaly Detection. There are 3 main approaches to detect anomalies.

Determination of outliers without previous data information (anomalies). This is analogous to unsupervised clustering. Supervised machine learning issue with usable labels for both regular and anomalous classes. As anomalies are expected to be very few, this could be considered as binary classification with imbalanced data. Learning only the labels of the normal class makes it a semi-supervised approach

Trying to balance the imbalanced data will not always serve the purpose of anomaly detection. So we need various other methods…..

I am choosing 5 algorithms from multiple categories (Linear, Proximity-based, Probabilistic, Outlier Ensembles) to explain how they perform anomaly detection.

The ABOD algorithm is based on the angle variance among the difference vectors of data objects in the dataset. Thus, in comparison to pure distance-based methods, are the consequences of the ‘curse of dimensionality.’

The angle between the PX and the PY from point P is somewhat smaller than the angles of other points Q and R in the diagram above for the outlined point P. The corner of the most distant data points is less than the angle of the near data points. If you think more, the difference between the farthest data points (of all possible angles to the other data points) and closer data points would be smaller. Thus, the data point is treated as an outlier with less angle variation. Angles in large dimensions are more stable than distances. The angle and the distance between the point are separated during actual use so that the distance is also taken.

KNN is a monitored ML algorithm sometimes used in data science for classification problems (also regression issues sometimes). It is one of the most basic and commonly used algorithms with strong use cases.

The underlying principle is that identical observations are close together and outliers generally represent solo observations, that they are more distant from the cluster with similar observations.

Although KNN is a supervised ML algorithm, it takes an unsupervised approach completely based on the threshold value of distance when it comes to anomaly detection.

They are like every other tree ensemble form, based on decision trees. Partitions are generated in these trees, by first choosing a feature randomly and then selecting a split value randomly between the lowest and highest value of the chosen function.

A random function is first chosen to construct a branch of the tree. Next, a random split value is chosen (between min and maximum value). If this function has a lower value, then the chosen one follows the left branch, otherwise the right branch. This process takes place until the maximum depth is isolated or defined.

The outliers are essentially less common and distinct in terms of values than normal observations (they lie further away from the regular observations in the feature space).

This is why they can be found closer to the tree root by using such a random partitioning (shorter average path).

Histogram Based Outlier Score takes on the freedom of the function and measures the degree of deviations by histogram construction. A histogram can be calculated for each function, measured individually, and averaged at the end of multivariate anomaly detection.

A One-Class Vector Support Machine is an unsupervised learning algorithm trained on ‘normal’ data, the negative examples in our case. It understands the borders of these points and is thus capable of classifying certain points beyond the boundary as outliers, you guessed.

It is difficult to train an unsupervised learning algorithm and the One-Class SVM is no exception. The nu parameter can be the proportion of outliers that you intend to see, the gamma parameter smooths up the contour lines.