Identifying Outliers with Local Outlier Probabilities | by Valentino Constantinou | Towards Data Science

Locating outliers is important in a variety of data science tasks, from performing predictive modeling to determining outliers in historical data. Many types of approaches exist to identifying outliers in your data. Local Outlier Probabilities, or LoOP, is just one approach. While not capable of detecting more complex anomalies that occur well within the distribution of the data, LoOP is an ideal approach for detecting anomalies that fall outside of the typical multivariate distribution.
The outlier score of each sample is called the Local Outlier Probability. It measures the local deviation of density of a given sample with respect to its neighbors as Local Outlier Factor (LOF) of which it is based, but provides normalized outlier scores in the range [0,1]. These outlier scores are directly interpretable as a probability of an object being an outlier. Since Local Outlier Probabilities provides scores in the range [0,1], it is straight forward to interpret the results according to the application.
Like LOF, it is local in that the anomaly score depends on how isolated the sample is with respect to the surrounding neighborhood. Locality is given by k-nearest neighbors, whose distance is used to estimate the local density. By comparing the local density of a sample to the local densities of its neighbors, one can identify samples that lie in regions of lower density compared to their neighbors and thus identify samples that may be outliers according to their Local Outlier Probability.
We can illustrate how LoOP works by using Iris dataset and the PyNomaly library, a Python implementation that also provides just-in-time (JIT) compilation via Numba for large datasets (it is Python only by default). The Iris dataset contains only 150 observations, 50 within each class of Iris flower. One of the classes, or Iris flower types, is linearly separable from the others — the other two are not linearly separable from one another. This can be seen by applying a clustering algorithm such as Density-Based Spatial Clustering of Applications with Noise (DBSCAN).
# library importsimport matplotlib.pyplot as pltfrom mpl_toolkits.mplot3d import Axes3Dimport pandas as pdfrom pydataset import datafrom sklearn.cluster import DBSCAN# import the data and remove any non-numeric columnsiris = pd.DataFrame(data('iris'))iris = pd.DataFrame(iris.drop('Species', 1))# apply DBSCAN clusteringdb = DBSCAN(eps=0.9, min_samples=10).fit(iris)iris['labels'] = db.labels_# and plot the clustersfig = plt.figure(figsize=(7, 7))ax = fig.add_subplot(111, projection='3d')ax.scatter(iris['Sepal.Width'], iris['Petal.Width'], iris['Sepal.Length'],c=iris['labels'], cmap='Set1', s=50)ax.set_xlabel('Sepal.Width')ax.set_ylabel('Petal.Width')ax.set_zlabel('Sepal.Length')plt.show()plt.clf()plt.cla()plt.close()
When applying LoOP to the data, it is intuitive to expect that values along the edges of the scatter field to be identified as more likely to be outliers since these fall outside the typical values in the data’s distribution. By default, PyNomaly applies LoOP across all observations (illustrated with varying distance metrics used), but can also be applied with respect to each cluster if desired. Two parameters — the extent and n_neighbors (number of neighbors) — directly influence the resulting LoOP scores, and can be directly specified:
from PyNomaly import loop# using the same iris dataset as above...m = loop.LocalOutlierProbability(iris, extent=3, n_neighbors=20).fit()scores = m.local_outlier_probabilitiesprint(scores)
The extent parameter controls the sensitivity of the scoring in practice. The parameter corresponds to the statistical notion of an outlier defined as an object deviating more than a given lambda (extent) times the standard deviation from the mean. A value of 2 implies outliers deviating more than 2 standard deviations from the mean, and corresponds to 95.0% in the empirical “three-sigma” rule. The question to ask is whether it is more reasonable to assume outliers in your data are 1, 2, or 3 standard deviations from the mean, and select the value likely most appropriate to your data and application.
The n_neighbors parameter, on the other hand, defines the number of neighbors to consider about each sample (neighborhood size) when determining its Local Outlier Probability with respect to the density of the sample’s defined neighborhood. The idea number of neighbors to consider is dependent on the input data. However, the notion of an outlier implies it would be considered as such regardless of the number of neighbors considered. One potential approach is to use a number of different neighborhood sizes and average the results for reach observation. Those observations which rank highly with varying neighborhood sizes are more than likely outliers. Another is to select a value proportional to the number of observations, such an odd-valued integer close to the square root of the number of observations in your data. The ideal choice, again, varies depending on the data and its characteristics.
Local Outlier Probabilities (LoOP) is a useful approach to be aware of and to apply in certain contexts, and has since been extended to work in specific use cases such as with streaming data (an extension available in PyNomaly, together with more examples of LoOP in practice). This approach can help identify anomalies defined according to their suitability within the multivariate distribution of a set of data, and is often a great starting point for identifying less-complex data outliers (anomalies) in a practical, programmable, and interpretable way.