3 Simple Outlier/Anomaly Detection Algorithms every Data Scientist needs | by Vagif Aliyev | Towards Data Science

I’m sure you have come across a few of the following scenarios:
Your model is not performing as you wanted it to.You can’t help but notice that some points seem to differ greatly from the rest.
Your model is not performing as you wanted it to.
You can’t help but notice that some points seem to differ greatly from the rest.
Well congratulations, because you might have outliers in your data!
In statistics, an outlier is a data point that differs significantly from other observations. From the figure above, we can clearly see that while most points lie in and around the linear hyperplane, a single point can be seen diverge from the rest. This point is an outlier.
For example, take a look at the list below:
[1,35,20,32,40,46,45,4500]
Here, it is clearly easy to see that 1 and 4500 are outliers in the dataset.
Usually, outliers can occur from one of the following scenarios:
Sometimes they can occur by chance, possibly because of a measurement error.Sometimes they can occur in the data as data will rarely be 100% clean without any outliers present.
Sometimes they can occur by chance, possibly because of a measurement error.
Sometimes they can occur in the data as data will rarely be 100% clean without any outliers present.
Here are several reasons:
Linear models
Linear models
Let’s suppose you have some data, and you want to predict house prices from it using Linear Regression. A possible hypothesis could look like this:
In this case, we are actually fitting the data too well(overfitting). However, note how all the points are situated are roughly the same range.
Now, let’s see what happens when we add an outlier.
Clearly, we see how our hypothesis has shifted, and therefore, inference will be much worse that it would be without the outlier. Linear models include:
Perceptron
Linear + Logistic Regression
Neural Networks
KNN
2. Data imputing
A common scenario is to have missing data, and one of two approaches can be taken:
Remove instances with missing rowsImpute data using a statistical method
Remove instances with missing rows
Impute data using a statistical method
If we were to go with the second option, we could have problematic imputations, as outliers can greatly change the values of statistical methods . For example, going back to our fictional data with no outliers:
# Data with no outliersnp.array([35,20,32,40,46,45]).mean() = 36.333333333333336# Data with 2 outliersnp.array([1,35,20,32,40,46,45,4500]).mean() = 589.875
Clearly this analogy is quite extreme, but the idea remains the same; outliers in our data is usually a problem, as outliers can cause serious problems in statistical analysis and modelling. However, in this article, we will be looking at a few ways into how we can detect and combat them.
Density-based spatial clustering of applications with noise(or, more simply, DBSCAN) is actually an unsupervised clustering algorithm, just like KMeans. However, one of its uses is also being able to detect outliers in data.
DBSCAN is popular because it can find non-linearly separable clusters, which can’t be done with KMeans and Gaussian Mixtures. It works well when clusters and dense enough, and are separated by low-density regions.
The algorithm defines clusters as continuous regions of high density. The algorithm is quite simple:
For each instance, it counts how many instances are located within a small distance ε (epsilon) from it. This region is called the instance’s ε- neighbourhood.If the instance has more than min_samples instances located in its ε-neighbourhood, then it is considered a core instance. This means that the instance is located in high density region(a region with many instances inside it.)All instances inside a core instance’s ε-neighbourhood are assigned to the same cluster. This may include other core instances, therefore a single long sequence of neighbouring core instances forms a single cluster.Any instances that are not a core instance, or are not located in any core instance’s ε-neighbourhood are outliers.
For each instance, it counts how many instances are located within a small distance ε (epsilon) from it. This region is called the instance’s ε- neighbourhood.
If the instance has more than min_samples instances located in its ε-neighbourhood, then it is considered a core instance. This means that the instance is located in high density region(a region with many instances inside it.)
All instances inside a core instance’s ε-neighbourhood are assigned to the same cluster. This may include other core instances, therefore a single long sequence of neighbouring core instances forms a single cluster.
Any instances that are not a core instance, or are not located in any core instance’s ε-neighbourhood are outliers.
The DBSCAN algorithm is very easy to use thanks to Scikit-Learn’s intuitive API. Let’s see an example of the algorithm in action:
from sklearn.cluster import DBSCAN from sklearn.datasets import make_moonsX, y = make_moons(n_samples=1000, noise=0.05)dbscan = DBSCAN(eps=0.2, min_samples=5)dbscan.fit(X)
Here we will instantiate DBSCAN with a ε-neighbourhood length of 0.05, and 5 to be the minimum number of samples required for an instance to be considered a core instance
Remember, we do not pass in our labels as it is an unsupervised algorithm. We can see the labels the labels that the algorithm produced using the following command:
dbscan.labels_OUT:array([ 0,  2, -1, -1,  1,  0,  0,  0, ...,  3,  2,  3,  3,  4,  2,  6,  3])
Note how some labels have values equal to -1: these are the outliers.
The DBSCAN does not have a predict method, only a fit_predict method, meaning that it can’t cluster new instances. Instead, we can use a different classifier to train and predict on. For this example, let’s use a KNN:
from sklearn.neighbors import KNeighborsClassifierknn = KNeighborsClassifier(n_neighbors=50)knn.fit(dbscan.components_, dbscan.labels_[dbscan.core_sample_indices_])X_new = np.array([[-0.5, 0], [0, 0.5], [1, -0.1], [2, 1]])knn.predict(X_new)OUT:array([1, 0, 1, 0])
Here, we fit the KNN classifier on the core samples and their respective neighbours.
However, we run into one problem; we have given the KNN data without any outliers. This is problematic, as it will force KNN to choose a cluster for new instances, even if the new instance is indeed an outlier.
To combat this, we leverage the leverage the kneighbors method of the KNN classifier, which, given a set of instances, returns the distances and indices of the k nearest neighbours of the training set. We can then set a maximum distance, and if an instance exceeds that distance, we qualify it as an outlier:
y_dist, y_pred_idx = knn.kneighbors(X_new, n_neighbors=1)y_pred = dbscan.labels_[dbscan.core_sample_indices_][y_pred_idx]y_pred[y_dist > 0.2] = -1y_pred.ravel()OUT:array([-1, 0, 1, -1])
Here we have discussed and implemented DBSCAN for anomaly detection. DBSCAN is great because it’s quick, has only two hyperparameters and is robust to outliers.
An IsolationForest is an ensemble learning anomaly detection algorithm, that is especially useful at detecting outliers in high dimensional datasets. The algorithm basically does the following:
It creates a Random Forest in which Decision Trees are grown randomly: at each node, features are picked randomly, and it picks a random threshold value to split the dataset into two.It continues to chop away the dataset until all instances end up being isolated from each other.An anomaly is usually far away from other instances, so, on average(across all Decision Trees), it becomes isolated in less steps that normal instances.
It creates a Random Forest in which Decision Trees are grown randomly: at each node, features are picked randomly, and it picks a random threshold value to split the dataset into two.
It continues to chop away the dataset until all instances end up being isolated from each other.
An anomaly is usually far away from other instances, so, on average(across all Decision Trees), it becomes isolated in less steps that normal instances.
Again, thanks to Scikit-Learn’s intuitive API, we can easily implement the IsolationForest class. Let’s see an example of the algorithm in action:
from sklearn.ensemble import IsolationForestfrom sklearn.metrics import mean_absolute_errorimport pandas as pd
We will also import mean_absolute_error to measure our error. For the data, we will use a dataset that can be obtained from Jason Brownlee’s GitHub:
url='https://raw.githubusercontent.com/jbrownlee/Datasets/master/housing.csv'df = pd.read_csv(url, header=None)data = df.values# split into input and output elementsX, y = data[:, :-1], data[:, -1]
Before we fit an Isolation Forest, let’s try fit a vanilla Linear Regression model on the data and get our MAE:
from sklearn.linear_model import LinearRegressionlr = LinearRegression()lr.fit(X,y)mean_absolute_error(lr.predict(X),y)OUT:3.2708628109003177
A relatively good score. Now, let’s see if the Isolation Forest can improve the score by removing anomalies!
First, we will instantiate our IsolationForest:
iso = IsolationForest(contamination='auto',random_state=42)
The most important hyperparameter in the algorithm is probably the contamination parameter, which is used to help estimate the number of outliers in the dataset. This is a value between 0.0 and 0.5 and by default is set to 0.1
However, it is essentially a randomised Random Forest, so all the hyperparameters of a random forest can also be used in the algorithm.
Next, we will fit the data to the algorithm:
y_pred = iso.fit_predict(X,y)mask = y_pred != -1
Note how we also filter out predictions values = -1, as just like in DBSCAN, these are considered as outlier.
Now, we will reassign the X and Y with the outlier-filtered data:
X,y = X[mask,:],y[mask]
And now let’s try fit our Linear Regression model to the data and measure the MAE:
lr.fit(X,y)mean_absolute_error(lr.predict(X),y)OUT:2.643367450077622
Wow, a good decrease in cost. This clearly demonstrates the power of the Isolation Forest.
While Boxplots are quite a common way to identify outliers, I really find that the latter is probably the most underrated method to identify outliers. But before we get into the Tuckey Method, let’s talk about Boxplots:
Boxplots essentially provide a graphical way to display numerical data through quantiles.It is a very simple yet effective way to visualise outliers.
The upper and lower whiskers show the boundaries of the distribution, and anything above or below is considered to be an outlier. In the figure above, anything above ~80 and below ~62 is considered to be an outlier.
Essentially, the box plot works by splitting the dataset into 5 parts:
Min: the lowest data point in the distribution excluding any outliers.
Max: the highest data point in the distribution excluding any outliers.
Median (Q2 / 50th percentile) : the middle value of the dataset.
First quartile (Q1 / 25th percentile) : is the median of the lower half of the dataset.
Thirds quartile (Q3 / 75th percentile) : is the median of the upper half of the dataset.
The Interquartile Range (IQR) is important as it is what defines outliers. Essentially, it is the following:
IQR = Q3 - Q1Q3: third quartileQ1: first quartile
In a boxplot, a distance of 1.5 * IQR is measured out and encompasses the higher observed points of the dataset. Similarly, a distance of 1.5 * IQR is measured out on the lower observed points of the dataset. Anything outside these distances is an outlier. More specifically:
If observed points are below (Q1 − 1.5 * IQR) or the boxplot lower whisker, then they are considered outliers.
Similarly, if observed points are above (Q3 + 1.5 * IQR) or the boxplot upper whisker, then they are also considered outliers.
Let’s see how we can detect outliers using Boxplots in Python!
import matplotlib.pyplot as pltimport seaborn as snsimport numpy as npX = np.array([45,56,78,34,1,2,67,68,87,203,-200,-150])y = np.array([1,1,0,0,1,0,1,1,0,0,1,1])
Let’s plot a boxplot of our data:
sns.boxplot(X)plt.show()
So we see that we have a median value of 50 and 3 outliers in our data, according to our boxplot. Let’s get rid of these points:
X = X[(X < 150) & (X > -50)]sns.boxplot(X)plt.show()
Here, I have basically set a threshold, so that all points less than -50 and greater than 150 will be excluded. And the result; an even distribution!
The tuckey method outlier detection is actually a non-visual method of the box plot; the method is the same, except that there is no visualisation.
The reason I sometimes prefer this method, as oppose to the boxplot, is because sometimes taking a look at the visualisation and making a rough estimate of what the threshold should be set as is not really effective.
Instead, we can code an algorithm that can actually return the instances that it defined as outliers.
The code for the implementation is the following:
import numpy as npfrom collections import Counterdef detect_outliers(df, n, features):    # list to store outlier indices    outlier_indices = []    # iterate over features(columns)    for col in features:        # Get the 1st quartile (25%)        Q1 = np.percentile(df[col], 25)        # Get the 3rd quartile (75%)        Q3 = np.percentile(df[col], 75)        # Get the Interquartile range (IQR)        IQR = Q3 - Q1        # Define our outlier step        outlier_step = 1.5 * IQR       # Determine a list of indices of outliers       outlier_list_col = df[(df[col] < Q1 - outlier_step) |     (df[col] > Q3 + outlier_step)].index   # append outlier indices for column to the list of outlier indices         outlier_indices.extend(outlier_list_col)   # select observations containing more than 2 outliers    outlier_indices = Counter(outlier_indices)            multiple_outliers = list(k for k, v in outlier_indices.items() if v > n)return multiple_outliers# detect outliers from list of featureslist_of_features = ['x1', 'x2']# params dataset, number of outliers for rejection, list of features Outliers_to_drop = detect_outliers(dataset, 2, list_of_features)
Basically, this code does the following:
For every feature, it obtains:
For every feature, it obtains:
The 1st Quartile
The 3rd Quartile
The IQR
2. Next, it defines the outlier step, which, just like in boxplots, is 1.5 * IQR
3. It detects outliers by:
Seeing if the observed point is < Q1 — outlier step
Seeing if the observed point is Q3 + outlier step
4. It then checks selects observations that have k outliers(in this case, k = 2)
To summarise, there exists many outlier detection algorithms, but we went through the 3 most common ones: DBSCAN, IsolationForest and Boxplots. I encourage you to :
Try these methods on the Titanic Dataset. Which one detects outliers best?Look for other outlier detections methods and see if they perform better or worse than the ones you initially tried.
Try these methods on the Titanic Dataset. Which one detects outliers best?
Look for other outlier detections methods and see if they perform better or worse than the ones you initially tried.
I really do appreciate my followers, and I do hope to constantly write and give everybody great food for thought. For now, however, I must say goodbye ;}