3 Simple Outlier/Anomaly Detection Algorithms every Data Scientist needs
By: Vagif Aliyev
Tags: Machine Learning, Artificial Intelligence, Data Science, Data, Data Visualization

3 Simple Outlier/Anomaly Detection Algorithms every Data Scientist needs

Get an in-depth understanding about outlier detection and how you can implement 3 simple, intuitive and powerful outlier detection algorithms in Python

Photo By Scott.T on Flickr

I’m sure you have come across a few of the following scenarios:

Your model is not performing as you wanted it to. You can’t help but notice that some points seem to differ greatly from the rest.

Well congratulations, because you might have outliers in your data!

What are Outliers?

Photo can be found in StackExchange

In statistics, an outlier is a data point that differs significantly from other observations. From the figure above, we can clearly see that while most points lie in and around the linear hyperplane, a single point can be seen diverge from the rest. This point is an outlier.

For example, take a look at the list below:

[1,35,20,32,40,46,45,4500]

Here, it is clearly easy to see that 1 and 4500 are outliers in the dataset.

Why is there outliers in my data?

Usually, outliers can occur from one of the following scenarios:

Sometimes they can occur by chance, possibly because of a measurement error. Sometimes they can occur in the data as data will rarely be 100% clean without any outliers present.

Why are outliers a problem?

Here are several reasons:

Linear models

Let’s suppose you have some data, and you want to predict house prices from it using Linear Regression. A possible hypothesis could look like this:

Photo By Author

In this case, we are actually fitting the data too well(overfitting). However, note how all the points are situated are roughly the same range.

Now, let’s see what happens when we add an outlier.

Photo By Author

Clearly, we see how our hypothesis has shifted, and therefore, inference will be much worse that it would be without the outlier. Linear models include:

Perceptron

Linear + Logistic Regression

Neural Networks

KNN

2. Data imputing

Photo by Ehimetalor Akhere Unuabona on Unsplash

A common scenario is to have missing data, and one of two approaches can be taken:

Remove instances with missing rows Impute data using a statistical method

If we were to go with the second option, we could have problematic imputations, as outliers can greatly change the values of statistical methods . For example, going back to our fictional data with no outliers:

# Data with no outliers

np.array([35,20,32,40,46,45]).mean() = 36.333333333333336 # Data with 2 outliers

np.array([1,35,20,32,40,46,45,4500]).mean() = 589.875

Clearly this analogy is quite extreme, but the idea remains the same; outliers in our data is usually a problem, as outliers can cause serious problems in statistical analysis and modelling. However, in this article, we will be looking at a few ways into how we can detect and combat them.

Solution 1: DBSCAN

Photo By Wikipedia

Density-based spatial clustering of applications with noise(or, more simply, DBSCAN) is actually an unsupervised clustering algorithm, just like KMeans. However, one of its uses is also being able to detect outliers in data.

DBSCAN is popular because it can find non-linearly separable clusters, which can’t be done with KMeans and Gaussian Mixtures. It works well when clusters and dense enough, and are separated by low-density regions.

A high-level overview of how DBSCAN works

The algorithm defines clusters as continuous regions of high density. The algorithm is quite simple:

For each instance, it counts how many instances are located within a small distance ε (epsilon) from it. This region is called the instance’s ε- neighbourhood. If the instance has more than min_samples instances located in its ε-neighbourhood, then it is considered a core instance. This means that the instance is located in high density region(a region with many instances inside it.) All instances inside a core instance’s ε-neighbourhood are assigned to the same cluster. This may include other core instances, therefore a single long sequence of neighbouring core instances forms a single cluster. Any instances that are not a core instance, or are not located in any core instance’s ε-neighbourhood are outliers.

DBSCAN in Action

The DBSCAN algorithm is very easy to use thanks to Scikit-Learn’s intuitive API. Let’s see an example of the algorithm in action:

from sklearn.cluster import DBSCAN

from sklearn.datasets import make_moons X, y = make_moons(n_samples=1000, noise=0.05)

dbscan = DBSCAN(eps=0.2, min_samples=5)

dbscan.fit(X)

Here we will instantiate DBSCAN with a ε-neighbourhood length of 0.05, and 5 to be the minimum number of samples required for an instance to be considered a core instance

Remember, we do not pass in our labels as it is an unsupervised algorithm. We can see the labels the labels that the algorithm produced using the following command:

dbscan.labels_ OUT:

array([ 0, 2, -1, -1, 1, 0, 0, 0, ..., 3, 2, 3, 3, 4, 2, 6, 3])

Note how some labels have values equal to -1: these are the outliers.

The DBSCAN does not have a predict method, only a fit_predict method, meaning that it can’t cluster new instances. Instead, we can use a different classifier to train and predict on. For this example, let’s use a KNN:

from sklearn.neighbors import KNeighborsClassifier knn = KNeighborsClassifier(n_neighbors=50)

knn.fit(dbscan.components_, dbscan.labels_[dbscan.core_sample_indices_]) X_new = np.array([[-0.5, 0], [0, 0.5], [1, -0.1], [2, 1]]) knn.predict(X_new) OUT:

array([1, 0, 1, 0])

Here, we fit the KNN classifier on the core samples and their respective neighbours.

However, we run into one problem; we have given the KNN data without any outliers. This is problematic, as it will force KNN to choose a cluster for new instances, even if the new instance is indeed an outlier.

To combat this, we leverage the leverage the kneighbors method of the KNN classifier, which, given a set of instances, returns the distances and indices of the k nearest neighbours of the training set. We can then set a maximum distance, and if an instance exceeds that distance, we qualify it as an outlier:

y_dist, y_pred_idx = knn.kneighbors(X_new, n_neighbors=1)

y_pred = dbscan.labels_[dbscan.core_sample_indices_][y_pred_idx]

y_pred[y_dist > 0.2] = -1

y_pred.ravel() OUT:

array([-1, 0, 1, -1])

Here we have discussed and implemented DBSCAN for anomaly detection. DBSCAN is great because it’s quick, has only two hyperparameters and is robust to outliers.

Solution 2: IsolationForest

Photo By Author

An IsolationForest is an ensemble learning anomaly detection algorithm, that is especially useful at detecting outliers in high dimensional datasets. The algorithm basically does the following:

It creates a Random Forest in which Decision Trees are grown randomly: at each node, features are picked randomly, and it picks a random threshold value to split the dataset into two. It continues to chop away the dataset until all instances end up being isolated from each other. An anomaly is usually far away from other instances, so, on average(across all Decision Trees), it becomes isolated in less steps that normal instances.

IsolationForest in Action

Again, thanks to Scikit-Learn’s intuitive API, we can easily implement the IsolationForest class. Let’s see an example of the algorithm in action:

from sklearn.ensemble import IsolationForest

from sklearn.metrics import mean_absolute_error

import pandas as pd

We will also import mean_absolute_error to measure our error. For the data, we will use a dataset that can be obtained from Jason Brownlee’s GitHub:

url=' https://raw.githubusercontent.com/jbrownlee/Datasets/master/housing.csv' df = pd.read_csv(url, header=None) data = df.values

# split into input and output elements

X, y = data[:, :-1], data[:, -1]

Before we fit an Isolation Forest, let’s try fit a vanilla Linear Regression model on the data and get our MAE:

from sklearn.linear_model import LinearRegression lr = LinearRegression()

lr.fit(X,y) mean_absolute_error(lr.predict(X),y) OUT:

3.2708628109003177

A relatively good score. Now, let’s see if the Isolation Forest can improve the score by removing anomalies!

First, we will instantiate our IsolationForest:

iso = IsolationForest(contamination='auto',random_state=42)

The most important hyperparameter in the algorithm is probably the contamination parameter, which is used to help estimate the number of outliers in the dataset. This is a value between 0.0 and 0.5 and by default is set to 0.1

However, it is essentially a randomised Random Forest, so all the hyperparameters of a random forest can also be used in the algorithm.

Next, we will fit the data to the algorithm:

y_pred = iso.fit_predict(X,y)

mask = y_pred != -1

Note how we also filter out predictions values = -1, as just like in DBSCAN, these are considered as outlier.

Now, we will reassign the X and Y with the outlier-filtered data:

X,y = X[mask,:],y[mask]

And now let’s try fit our Linear Regression model to the data and measure the MAE:

lr.fit(X,y)

mean_absolute_error(lr.predict(X),y) OUT:

2.643367450077622

Wow, a good decrease in cost. This clearly demonstrates the power of the Isolation Forest.

Solution 3: Boxplots + The Tuckey Method

While Boxplots are quite a common way to identify outliers, I really find that the latter is probably the most underrated method to identify outliers. But before we get into the Tuckey Method, let’s talk about Boxplots:

Boxplots

Photo By Wikipedia

Boxplots essentially provide a graphical way to display numerical data through quantiles.It is a very simple yet effective way to visualise outliers.

The upper and lower whiskers show the boundaries of the distribution, and anything above or below is considered to be an outlier. In the figure above, anything above ~80 and below ~62 is considered to be an outlier.

How Boxplots work

Essentially, the box plot works by splitting the dataset into 5 parts:

Photo from StackOverflow

Min : the lowest data point in the distribution excluding any outliers.

: the lowest data point in the distribution excluding any outliers. Max : the highest data point in the distribution excluding any outliers.

: the highest data point in the distribution excluding any outliers. Median (Q2 / 50th percentile) : the middle value of the dataset.

: the middle value of the dataset. First quartile (Q1 / 25th percentile) : is the median of the lower half of the dataset.

: is the median of the lower half of the dataset. Thirds quartile (Q3 / 75th percentile) : is the median of the upper half of the dataset.

The Interquartile Range (IQR) is important as it is what defines outliers. Essentially, it is the following:

IQR = Q3 - Q1 Q3: third quartile

Q1: first quartile

In a boxplot, a distance of 1.5 * IQR is measured out and encompasses the higher observed points of the dataset. Similarly, a distance of 1.5 * IQR is measured out on the lower observed points of the dataset. Anything outside these distances is an outlier. More specifically:

If observed points are below (Q1 − 1.5 * IQR) or the boxplot lower whisker, then they are considered outliers.

Similarly, if observed points are above (Q3 + 1.5 * IQR) or the boxplot upper whisker, then they are also considered outliers.

Photo By Wikipedia

Boxplots in Action

Let’s see how we can detect outliers using Boxplots in Python!

import matplotlib.pyplot as plt

import seaborn as sns

import numpy as np X = np.array([45,56,78,34,1,2,67,68,87,203,-200,-150])

y = np.array([1,1,0,0,1,0,1,1,0,0,1,1])

Let’s plot a boxplot of our data:

sns.boxplot(X)

plt.show()

Photo By Author

So we see that we have a median value of 50 and 3 outliers in our data, according to our boxplot. Let’s get rid of these points:

X = X[(X < 150) & (X > -50)] sns.boxplot(X)

plt.show()

Photo By Author

Here, I have basically set a threshold, so that all points less than -50 and greater than 150 will be excluded. And the result; an even distribution!

The Tuckey Method Outlier Detection

The tuckey method outlier detection is actually a non-visual method of the box plot; the method is the same, except that there is no visualisation.

The reason I sometimes prefer this method, as oppose to the boxplot, is because sometimes taking a look at the visualisation and making a rough estimate of what the threshold should be set as is not really effective.

Instead, we can code an algorithm that can actually return the instances that it defined as outliers.

The code for the implementation is the following:

import numpy as np

from collections import Counter def detect_outliers(df, n, features):

# list to store outlier indices

outlier_indices = [] # iterate over features(columns) for col in features:

# Get the 1st quartile (25%)

Q1 = np.percentile(df[col], 25)

# Get the 3rd quartile (75%)

Q3 = np.percentile(df[col], 75)

# Get the Interquartile range (IQR)

IQR = Q3 - Q1 # Define our outlier step

outlier_step = 1.5 * IQR # Determine a list of indices of outliers outlier_list_col = df[(df[col] < Q1 - outlier_step) | (df[col] > Q3 + outlier_step)].index # append outlier indices for column to the list of outlier indices

outlier_indices.extend(outlier_list_col) # select observations containing more than 2 outliers

outlier_indices = Counter(outlier_indices)

multiple_outliers = list(k for k, v in outlier_indices.items() if v > n) return multiple_outliers # detect outliers from list of features

list_of_features = ['x1', 'x2']

# params dataset, number of outliers for rejection, list of features Outliers_to_drop = detect_outliers(dataset, 2, list_of_features)

Basically, this code does the following:

For every feature, it obtains:

The 1st Quartile

The 3rd Quartile

The IQR

2. Next, it defines the outlier step, which, just like in boxplots, is 1.5 * IQR

3. It detects outliers by:

Seeing if the observed point is < Q1 — outlier step

Seeing if the observed point is Q3 + outlier step

4. It then checks selects observations that have k outliers(in this case, k = 2)