5 Outlier Detection Techniques that every “Data Enthusiast” Must Know | by Prakhar Mishra | Towards Data Science

Outliers are those observations that differ strongly(different properties) from the other data points in the sample of a population. In this blog, we will go through 5 Outlier Detection techniques that every “Data Enthusiast” must know. But before that let’s take a look and understand the source of outliers.
There are multiple reasons why there can be outliers in the dataset, like Human errors(Wrong data entry), Measurement errors(System/Tool error), Data manipulation error(Faulty data preprocessing error), Sampling errors(creating samples from heterogeneous sources), etc. Importantly, detecting and treating these Outliers is important for learning a robust and generalizable machine learning system.
The Z-score(also called the standard score) is an important concept in statistics that indicates how far away a certain point is from the mean. By applying Z-transformation we shift the distribution and make it 0 mean with unit standard deviation. For example — A Z-score of 2 would mean the data point is 2 standard deviation away from the mean.
Also, z-score of any data point can be calculated as shown below —
Z-score(i) = (x(i) -mean) / standard deviation
It assumes that the data is normally distributed and hence the % of data points that lie between -/+1 stdev. is ~68%, -/+2 stdev. is ~95% and -/+3 stdev. is ~99.7%. Hence, if the Z-score is >3 we can safely mark that point to be an outlier. Refer to below fig.
You can implement it using python as shown below —
import numpy as npdata = [1, 2, 3, 2, 1, 100, 1, 2, 3, 2, 1]threshold = 3mean = np.mean(data)std = np.std(data)z_score_outlier = [i for i in data if (i-mean)/std > threshold]print (z_score_outlier)>> 100   (outlier)
You can also use in-built functions provided by Scikit learn and Scipy in Python. (Link in the Resources section)
In Local Outlier Factor (LOF), the idea revolves around the concept of local regions. Here, we calculate and compare the local density of the focus point with the local density of its neighbours. If we find that the local density of the focus point is very low compared to its neighbours, that would kind of hint that the focus point is isolated in that space and is a potential outlier. The algorithm depends on the hyperparameter K, which decides upon the number of neighbours to consider when calculating the local density. This value is bounded between 0 (no neighbour) and the total points (all points being neighbour) in the space.
The local density function is defined as the reciprocal of average reachability distance, where, average reachability distance is defined as the average distance from the focus point to all points in the neighbour.
LOF = average local density of neighbors / local density of focus point
If,
LOF ≈ 1 similar density as neighbors
LOF < 1 higher density than neighbors (normal point)
LOF > 1 lower density than neighbors (anomaly)
Also, the below diagram shows the calculation of LOF and Local Density for a sample focus point(dark blue) in the space. Here, K=3(neighbours), d(distance) can be calculated as euclidean, manhattan, etc.
You can implement it using python’s Scikit-learn library as shown below —
from sklearn.neighbors import LocalOutlierFactordata = [[1, 1], [2, 2.1], [1, 2], [2, 1], [50, 35], [2, 1.5]]lof = LocalOutlierFactor(n_neighbors=2, metric='manhattan')prediction = lof.fit_predict(data)>> [ 1,  1,  1,  1, -1,  1]
Please feel free to explore the library for hyper-parameter tuning, etc.(Link in the Resources section)
Also, if you want to keep enjoying reading awesome articles related to Data Science and Machine Learning, you can always purchase the Medium membership through my referral link :)
I already have a pretty detailed blog post on Geometric Models for Outlier Detection, where I primarily focus on Angle-Based Techniques(ABOD) and Depth-Based Techniques(Convex Hull). You can check it out here. The below figure shows a pictorial view for both the techniques detecting outliers —
Isolation Forest is a tree-based algorithm that tries to find out outliers based on the concept of decision boundaries(just like we have for decision trees). The idea over here is to keep splitting the data at random thresholds and feature till every point gets isolated(it’s like overfitting a decision tree on a dataset). Once the isolation is achieved we chunk out points that got isolated pretty early during this process. And we mark these points as potential outliers. If you see this intuitively, the farther a point is from the majority, the easier it gets to isolate, whereas, isolating the points that are part of a group would require more cuts to isolate every point.
If you see in the fig. below, we randomly select feature and value to where we cut. And just after 4 cuts, we were able to isolate the outlier points, which means, these nodes would be occurring very early during our tree construction phase.
You can implement it using python’s Scikit-learn library as shown below —
from sklearn.ensemble import IsolationForestdata = [[1, 1], [2, 2.1], [1, 2], [2, 1], [50, 35], [2, 1.5]]iforest = IsolationForest(n_estimators=5)iforest.fit(data)actual_data = [[1, 1.5]]iforest.predict(actual_data)>> 1   (Normal)outlier_data = [[45, 55]]iforest.predict(outlier_data)>> -1   (Outlier)
Please feel free to explore the library for hyper-parameter tuning, etc.(Link in the Resources section)
Autoencoders are Neural network architectures that are trained to reproduce the input itself. It consists of two trainable components namely — Encoder and Decoder. Where the goal of the encoder is to learn a latent representation of the input(original dimension to low dimension) and the goal of the decoder is to learn to reconstruct the input from this latent representation(low dimension to original dimension). So for the autoencoder to work well, both of these components should optimize on their respective tasks.
Autoencoders are widely used for detecting anomalies. A typical intuition behind how this works is that if a point in feature space lies far away from the majority of the points(meaning it holds different properties, for example — dog images clustered around a certain part of the feature space and cow image lies pretty far from that cluster), in such cases, the autoencoder learns the dog distribution (because of the count of dog images would be very high compared to cows — that’s why it’s an anomaly, hence model would majorly focus on learning the dog cluster). This means, the model would be able to more or less correctly re-generate the dog images leading to low loss values, whereas, for the cow image it would generate high loss(because that’s something odd it saw for the first time and the weights it has learnt is mostly to reconstruct dog images). We use these reconstruction loss values as the anomaly scores, so Higher the scores, the higher the chances of input being an anomaly.
In the figure below — Consider green dots as dogs and blue dot as cows. We train our system with this input distribution and expect it to output the same. But we can see that the model more or less perfectly re-generated the yellow dots but could not fit well for the blue one(as it was not in the same feature range as yellow’s)
In the figure below — During inference when we give a yellow dot to our model, it’s able to re-generate it will less error(signal for normal point), whereas for blue one it returns a high error(signal for anomaly/outlier) due to incompetence to re-generate the same.
You can implement it using python’s PyOD library. I recommend you to follow this blogpost for the same.(Link in the Resources section)
In Outlier Detection using In-degree Number (ODIN), we calculate the in-degree for each of the data points. Here, in-degree is defined as the number of nearest neighbour sets to this point belongs. Higher this value, the more the confidence of this point belonging to some dense region in the space. Whereas, on the other side, a lesser value of this would mean that it’s not part of many nearest neighbour sets and is kind of isolated in the space. You can think of this method to be the reverse of KNN.
Here in the figure, we set the value of k(nearest neighbours)=3, the redpoint belong to just one nearest neighbour set which is of bluepoint, whereas, all other points are part of more than 1 nearest neighbour sets. Hence, we conclude that redpoint is an outlier.
You can implement it using python’s package-outlier library as shown below —
import package_outlier as podata = [[1, 1], [2, 2.1], [1, 2], [2, 1], [50, 35], [2, 1.5]]result = po.LocalOutlierFactorOutlier(data)print (result)
Please feel free to explore the library for hyper-parameter tuning, etc. (Link in the Resources section)
Also if research papers interest you then you can checkout some research paper summaries that I have written.
PyOD: https://pypi.org/project/pyod/Scikit-Learn: https://scikit-learn.org/stable/index.htmlpackage-outlier: https://pypi.org/project/package-outlier/
PyOD: https://pypi.org/project/pyod/
Scikit-Learn: https://scikit-learn.org/stable/index.html
package-outlier: https://pypi.org/project/package-outlier/
I hope you enjoyed reading this. If you’d like to support me as a writer, consider signing up to become a Medium member. It’s just $5 a month and you get unlimited access to Medium.
That’s it for this blog. Thank you for your time!