Practical Guide to Outlier Detection Methods | by A. Tayyip Saka | Towards Data Science

I am going to talk about the details of four outlier detection methodologies implemented in R Studio. I will be mentioning what the outlier is, why it is important and why outliers occur. These methods are as follows:
Tukey’s MethodTwitter Anomaly DetectionZ-score MethodMean Absolute Deviation(MADe) Method
Tukey’s Method
Twitter Anomaly Detection
Z-score Method
Mean Absolute Deviation(MADe) Method
To put simply, an outlier is a data point that differs greatly (much smaller or larger than) from other values in a dataset. Outliers may be because of random variation or may demonstrate something scientifically interesting. In any event, we should not simply eliminate the outlying observation before a careful investigation. It is significant to understand them and their occurrence in the right context of the study to be able to cope with them.
Why do outliers occur?
There are several reasons for outliers: 1. Some observations in the sample are extreme; 2. The data are inappropriately scaled; 3. Errors were made on data entry.
Dataset and Pre-Processing
My public dataset that shows weekly sales of the company from 2014 to 2017 provides “year”, “week” and “sales” columns as shown below.
The dimension of a dataset is 201 x 3 and data includes NA values. I looked at the week of blank values then fulfilled blanks by taking an average of observations whose week is the same with blank values in the dataset.
dim(sales_data)[1] 201   3sum(is.na(sales_data$Sales))[1] 4
The line graph points out sales of beverage in Turkey starting from 2014 to 2017.
Now, it is time to dive deeper into outlier detection methods.
Tukey’s Method(Box Whisker)
Tukey’s Method(Box Whisker)
Minimum value: The smallest value in the data set(Q1–1.5*IQR)First quartile: The value below which the lower 25% of the data are contained (Q1)Median value: The middle number in a range of numbers (Q2)Third quartile: The value above which the upper 25% of the data are contained (Q3)Maximum value: The largest value in the data set (Q3+1.5*IQR)InterQuartile Range(IQR): Range of observations in a dataset (Q3-Q1)
Minimum value: The smallest value in the data set(Q1–1.5*IQR)
First quartile: The value below which the lower 25% of the data are contained (Q1)
Median value: The middle number in a range of numbers (Q2)
Third quartile: The value above which the upper 25% of the data are contained (Q3)
Maximum value: The largest value in the data set (Q3+1.5*IQR)
InterQuartile Range(IQR): Range of observations in a dataset (Q3-Q1)
This method is particularly useful for indicating whether a distribution is skewed and whether there are potential unusual observations in the data set.
“ggplot” is one of the powerful data visualization packages in R; hence I used “ggplot” package to draw box-plot of sales. As shown above, blue points are identified as outliers and red point shows the median of a dataset. The data is left-skewed.
The model detected 8 outliers in the dataset as shown on the left table. I can clearly say that these data points differ from the remaining of the dataset. In fact, the sales value of outlier points is above 115,000.
2. Twitter Anomaly Detection
AnomalyDetection is an open-source R package to detect anomalies which is robust, from a statistical standpoint, in the presence of seasonality and an underlying trend. The AnomalyDetection package can be used in a wide variety of contexts such as new software release, user engagement posts, and financial engineering problems. The underlying algorithm — known as Seasonal Hybrid ESD builds upon the Generalized ESD test for detecting anomalies. It can be used to find both global as well as local anomalies.
It can be seen below that I implemented Twitter Anomaly and then the model found 6 outliers in the dataset in comparison with Tukey’s method by taking the alpha value as 0.05. Outlier points are identified as a circle in the table.
The table below shows which data points labeled as outliers. The index cumulatively shows week numbers of sales. These outliers belong to generally years 2014 and 2015.
3. Z-score Method
Z-score finds the distribution of data where mean is 0 and the standard deviation is 1. The Z-score method relies on the mean and standard deviation of data to gauge the central tendency and dispersion. This is problematic on several occasions since the mean and standard deviation are highly affected by outliers.
From a statistical perspective, using a cut-off of 1.96 would give you an equivalent p-value of p = 0.05. As a result, this method detected 8 extreme points among 201 observations. The Z-score method accords with Box-Whisker on this dataset. Most of these points appeared in the year 2014.
4. MADe Method
In statistics, the median absolute deviation (MAD) is a robust measure of the variability of univariate data. Besides, MAD is similar to the standard deviation but it is less sensitive to extreme values in the data than the standard deviation.
I first computed the median, and then for each data point, I computed the distance between that value and the median. The MAD is defined as the median of these distances. Then, this quantity (MAD) needs to be multiplied by 1.4826 to assure it approximates the actual standard deviation.
Median and MADe are employed to detect outliers by applying formulas below.
As a result of the MADe method, 9 outliers are found method as shown above.
Final Notes
I performed four outlier detection methods and each method may produce different results on a dataset. Therefore, you must select one of them to observe outliers or can label the most common points among all methods as extreme points. In that case, there are 6 common extreme points among methods I mentioned above.
As a next step, outlier points will be transformed by considering different ways before forecasting modeling. This will result in a more accurate forecasting model.
You can see all codes here and next story will be coming soon ...