Statistical techniques for anomaly detection
By: Mahbubul Alam
Tags: Machine Learning, Data Science, Anomaly Detection, Outlier Detection, Fraud Detection

1) Z-score

Z-score is probably the simplest one yet an useful statistical measure for anomaly detection. In a statistical distribution, Z-score tells you how far is a given data point from the rest of the crowd. Technically speaking, Z-score measures how many standard deviations away a given observation is from the mean. A Z-score of 2 means that the data point is 2 standard deviation far from the mean.

To calculate Z-score you only need two parameters: mean and standard deviation, and they are fairly easy to get in any programming language (for example in Python you just apply the function describe() to a data frame).

Once these two parameters are obtained, Z-score for any given data point is calculated using the following simple formula:

2) Modified Z-score

Z-score is easy to calculate using mean and standard deviation, but it has its own limitations. There are a few situations when it’s not an ideal technique, for example:

the data are not normally distributed

data/sample size is small

In addition to these, Z-score is sensitive to extreme values (because one of its parameters — mean — itself is sensitive to extreme values), so it may fail to screen outliers appropriately.

To overcome these shortcomings, several modifications are made to the standard Z-score:

median is used instead of the mean as a parameter, since the median is less sensitive to outliers

Median Absolute Deviation (MAD) is used in lieu of standard deviation

The values are multiplied by a constant so that it is approximately equal to SD (for normally distributed data, MAD is approximately equal to 0.6745*SD)

Putting them all together, the equation looks like the following:

3) Interquartile Range (IQR)

If you arrange data from small to large, the mid-point is called the median. The median splits data into two halves. The mid-points of each of these halves is called a quartile.

In other words, you can split data into 3 quartiles — 1st, 2nd and 3rd (the 2nd quartile has a name for it— the median). So the Interquartile Range is the distance between 1st and 3rd quartiles. The theory behind anomaly detection using IQR is that, if a data point is too far from the 1st and 3rd quartile, it probably is an outlier.

IQR can be used standalone for outlier detection, but boxplots below use the same algorithmic theory and are probably more intuitive than IQR.

4) Boxplot

Boxplot provides a better graphical representation of IQR, but also provides additional information.

In the boxplot below, the length of the box is IQR, and the minimum and maximum values are represented by the whiskers. The whiskers are generally extended into 1.5*IQR distance on either side of the box. Therefore, all data points outside these 1.5*IQR values are flagged as outliers.

I wrote a separate article on boxplot for outlier detection with an example and a small snippet of Python, you might want to check that out.

5) Histogram

The final statistical tool for outlier detection is the distribution plot or histogram. It’s another way of tracking outliers both visually and programmatically. Take a look at the following figure for example:

Distribution of data to detect potential outliers

From this distribution plot alone it is pretty clear that while most values in the dataset are grouped together, there are quite a few values that seem to be out of the ordinary. So you could filter those values and check whether these are anomalies or not.

However, if the distribution of the data is skewed, sometimes you may need to apply data transformation techniques (e.g. logarithmic or square roots). For example, if we apply a logarithm to the same data as above, the data becomes approximately normally distributed and tells a different story about outliers.

Log transformed data to filter potential outliers

Summary

The purpose of this article was to introduce five simple statistical techniques — z-score, modified z-score, IQR, boxplot and histogram — that are commonly used in data science as coarse filters for outlier/anomaly detection. Sometimes these simple techniques are good enough for anomaly detection, but sometimes we need to move on to more sophisticated machine learning algorithms — kNN, SVM, DBSCAN etc. — a subject of my future articles (stay tuned!).

If you like my articles feel free to follow me on Twitter.