Detecting outliers with PyOD. As the name suggests, outliers are... | by Valentina Alto | Towards Data Science

As the name suggests, outliers are datapoint which differs significantly from the rest of your observations. In other words, they are far away from the average path of your data.
In statistics and Machine Learning, detecting outliers is a pivotal step, since they might affect the performance of your model. Namely, imagine you want to predict the return of your company based on the amount of sold units.
To do so, you want to fit your data with simple linear regression, hence you proceed with the OLS strategy to find your parameter alpha and beta (to read more about Linear Regression, you can read my former article here):
Nice, but what if, among your data, there was an outlier?
As you can see, the regression line is steeper since it wants to capture also the effect of that outlier. As a result though, it was less accurate in fitting al the other data points. One possible way of resolution might be, once detected all the outliers (we will see how to do so later on) simply removing them or, a bit more sophisticatedly, substituting them with the mean or median value of that feature.
However, there is a pitfall in this procedure. Indeed, outliers (at least, not all of them) are not necessarily a bad thing. Indeed, they can contain relevant information of deviations from the average pattern which might repeat in the future. Hence, a model which does not capture this information will not be able to prevent or at least model those deviations while making predictions.
Namely, imagine a factory working with many machines. These latter need to be monitored over time: let’s say that you are going to keep track only of their pressure. It will likely follow a given pattern, depending on the moment of the day and the amount of workflow. Then, you suddenly detect a peak of the value of the pressure, then everything comes back to normality. Intuitively, are you going to neglect what you’ve just seen, or are you going to dwell on this peak? Maybe this peak is anticipating a technical problem, hence you’d be better to incorporate this information and use it to prevent potential failures.
Hence, special attention needs to be used in outliers detection, especially in those fields where there are highly costly consequences in case of missing them (think about fraud detection, cybersecurity and so forth). That’s why it is necessary to use specific algorithms for anomaly detection, rather than simple visualization methods, like histogram and boxplot.
That’s where PyOD comes to aid.
PyOD is a Python package which you can easily install via pip install pyod on your console. It offers a variety of functions or algorithms to detect outliers in an efficient way, each of them returning a so-called outlying score: it will label each datapoint with a number that will be compared to an internal threshold to determine whether that datapoint is an outlier. Let’s examine some of them.
Angle-based Outlier Detection (ABOD): it considers the relationship between each point and its neighbor(s), without considering the relationships among these neighbors. The variance of its weighted cosine scores to all neighbors could be viewed as the outlying score. The ABOD method is especially useful for high-dimensional data, as the angle is a more robust measure than the distance in high-dimensional space.
K-Nearest Neighbor: it uses the math behind the classification algorithm KNN. Indeed, for any data point, the distance to its kth nearest neighbor could be viewed as the outlying score. PyOD supports three KNN detectors: largest, mean and median, which use as outlying score, respectively, the distance of the kth neighbor, the average of all the k neighbors and the median distance to k neighbors.
Isolation forest: it is built following the theory of decision tree and random forest. In this case though, the root is randomly selected, and the idea is that outliers should be lying close to the root. That’s because a normal point needs many partitions to be identified, while an outlier will be easily identified with fewer partitions.
Local Correlation Integral (LOCI): it is very effective for detecting outliers and groups of outliers. It provides a LOCI plot for each point which summarizes a lot of the information about the data in the area around the point.
Let’s see an implementation in Python for the KNN.
First of all, let’s have a very brief recap of how the KNN actually works.
The KNN is probably the easiest algorithm for classification tasks. It assumes as core idea that similar observations should be near to each other. What does it mean concretely? Imagine we have the following 2-classes observations:
Our classes are squares and triangles. Now, we face a new occurrence (the circled observation) and we want to determine whether it is a triangle or a square:
The KNN, as anticipated, reasons in terms of ‘similarity’ which, in mathematical terms, could be translated as ‘distance’. For example, here we can compute the Euclidean distance between our green observation and some nearest observations (the number of those latter is a value we have to set before training the model, often indicated with K). Then, we simply compute the number of nearest neighbors for each class: the highest number determines the membership of the new observation.
Namely, if we set K=3 we will face the following:
We can count two squares and one triangle, hence the new data point is supposed to be a square. Nice rule of thumb: try to set your K equal to odd numbers, so that you can avoid a half and half computation.
Now let’s implement it in Python. First, PyOD provides some embedded functions to create a random dataset with a fraction of outliers:
import numpy as npfrom scipy import statsimport matplotlib.pyplot as pltfrom pyod.models.knn import KNNfrom pyod.utils.data import generate_dataoutlier_fraction = 0.1  n_train = 200 n_test = 100X_train, y_train, X_test, y_test = generate_data(n_train=n_train, n_test=n_test, contamination=outlier_fraction)#let's plot our train and test setfeature_1_train = X_train[:,[0]].reshape(-1,1)feature_2_train = X_train[:,[1]].reshape(-1,1)feature_1_test = X_test[:,[0]].reshape(-1,1)feature_2_test = X_test[:,[1]].reshape(-1,1)# scatter plot plt.scatter(feature_1_train,feature_2_train)plt.scatter(feature_1_test,feature_2_test)plt.xlabel('feature_1')plt.ylabel('feature_2')
Now let’s train and make predictions with our KNN model.
knn=KNN(contamination=outlier_fraction)knn.fit(X_train)# get the prediction labels and outlier scores of the training datay_train_pred = knn.labels_  y_train_scores = knn.decision_scores_ # get the prediction on the test datay_test_pred = knn.predict(X_test)  y_test_scores = knn.decision_function(X_test)# no of errors in test setn_errors = (y_test_pred != y_test).sum()print('No of Errors in test set: {}'.format(n_errors))#accuracy in test setprint('Accuracy in test set: {}'.format((n_test-n_errors)/n_test))
Nice, our algorithm was able to correctly classify almost all the outliers as such. Let’s also visualize the fitting procedure:
from pyod.utils import exampleexample.visualize(knn, X_train, y_train, X_test, y_test, y_train_pred, y_test_pred, show_figure=True, save_figure=False)
As anticipated, once properly detected our outliers, we have to decide how to deal with them or, in other words, how to incorporate the information contained in them. Modeling outliers is far from being easy and it is an open topic in statistics and data science. To give an example, an interesting field of study is the Extreme value theory or extreme value analysis (EVA). It is a branch of statistics dealing with the extreme deviations from the median of probability distributions. It seeks to assess, from a given ordered sample of a given random variable, the probability of events that are more extreme than any previously observed.
This article was about the detection procedure and I’m not going to dwell on the ‘modeling’ step. However, if you are interested in this topic, I recommend these further readings: