Unsupervised Machine Learning Approaches for Outlier Detection in Time Series, Using Python
By: Kirsten Perry
Tags: Anomaly Detection, Time Series Analysis, Python, Data Science, Machine Learning

In this post, I cover some of my favorite methods for detecting outliers in time series data. There are many different approaches for detecting anomalous data points; for the sake of brevity, I only focus on unsupervised machine learning methods in this post.

The anomaly/outlier detection algorithms covered in this article include:

Low-pass filters: taking the centered rolling average of a time series, and removing anomalies based on Z-score Isolation forests Seasonal-extreme studentized deviate (S-ESD) algorithm One class support vector machines (SVM’s)

So what is an ‘anomaly’ in a time series, and why do we care about about detecting anomalies in time series sequences?

I deal with time series data a lot, and it’s not uncommon for data sets to experience unexpected drops or spikes, flat lines, or phase shifts. Each of these situations qualifies as an ‘anomaly’ — something out of the ordinary when compared to the behavior of the sequence as a whole.

Detecting anomalies in a time series is important for a variety of reasons. Sometimes anomalies are indicative of certain patterns, such as spikes or drops in a trend during a specific day or time. Other times, anomalies are ‘false’ readings, which happen because of an instrumentation or meter error. Often, we want to isolate anomalies from time series data because they skew ‘average’ behaviors — average rate of change, average value, average distribution, etc.

All of the algorithms presented in this article have a binary output: a data point in a time series is either anomalous or it isn’t.

Time Series Example

In this article, we compare the results of several different anomaly detection methods on a single time series. The time series that we will be using is the daily time series for gasoline prices on the U.S. Gulf Coast, which is retrieved using the Energy Information Administration (EIA) API.

For more background on using the EIA’s free API to retrieve energy-related data in Python, check out this tutorial.

We pull and visualize the time series using the following Python code:

Just to spice things up, we pepper the time series with a few outliers. Hopefully, these points will be picked up by the anomaly detection algorithms.

Gasoline Price Time Series, with artificially generated anomalies in yellow

Now that we’ve selected our time series and added a few outliers, let’s test our anomaly detection algorithms on it.

Simple Low-Pass Filter: Taking a Rolling Average, and Removing Anomalies based on Z-Score

One of the simplest (but still highly effective) methods that I use for detecting anomalies goes as follows:

Implementing a centered rolling average of the time series data Calculating the individual Z-score of each data point in the time series, compared to the rolling average Removing data points that are more than a certain number of standard deviations away from the rolling average (normally 2 to 3 standard deviations away, but depends on the data behavior)

Let’s try this method with an example.

Code Example

As mentioned above, we use the time series for gasoline prices on the US Gulf Coast as our example time series set.

It is important to note that the rolling average that we implement for our low-pass filter is centered. This means that a rolling average with x-length is the mean of x/2 data points before, and x/2 data points after. For example, if we implement a 60-point rolling average at value t, then we find the mean of the data points ranging between (t-30) and (t+30). Using a centered rolling average helps to account for large shifts in the time series from both ends.

We generate the low-pass filter and visualize anomalies using the following code:

Gasoline time series, color-coded by detected low-pass filter anomalies. Anomalous data points are in yellow, and normal data points are in purple.

In examining the above results, the low pass filter performs fairly well. Of the nine outliers added to the time series, the low pass filter successfully detects six, plus a few other data points that look anomalous.

Isolation Forests

Isolation forests are an unsupervised extension of the popular random forest algorithm. The building blocks of isolation forests are isolation trees with a binary outcome (is/is not an outlier).

When an isolation forest is built, the algorithm splits each individual data point off from all other data points. The easier it is to isolate a single point in space from all other points, the more likely it is an outlier (because it’s far away from all other data points). If a data point is an in-lier, it will be closely surrounded by other data points, and will take more splits to isolate (1). See the graphic below as an illustration.

Each line represents a split performed by the algorithm to isolate individual data points. In the first figure, X0 is an outlier, and is isolated in ~3 splits. In the second figure, Xi is an in-lier, and requires many more splits to isolate from all other data points.

Image courtesy of https://www.semanticscholar.org/paper/Anomaly-Detection-through-on-line-Isolation-Forest%3A-Susto-Beghi/c321685dd03fa1a54e99b4c046950b96c3b0f040/figure/1.

For further information on isolation forests, check out this tutorial by T. Fuertes.

Code Example

Again we use the gasoline time series, but this time we apply the isolation forest algorithm to detect anomalies. We use the IsolationForest() model available in the scikit-learn package to build a model and test it on our data set:

Gasoline time series, color-coded by detected Isolation Forest anomalies. Anomalous data points are in yellow, and normal data points are in purple.

Like the low-pass filter, the isolation forest detects most of the artificially generated anomalies; however, it classifies more false positives (data points labelled as anomalies when they are actually not). The algorithm has a particularly difficult time recognizing large time series shifts (see late 2014 time period as an example).

Seasonal-Extreme Studentized Deviate (S-ESD) Algorithm

A group of data scientists as Twitter initially developed the S-ESD algorithm. One of the original (and most popular) implementations of the algorithm is in the R ‘AnomalyDetection’ package. Luckily, the R package has been adapted several times over for use in Python.

The algorithm is simple to understand, and, depending on the time series used, quite robust. It works as follows:

A time series is decomposed using STL (season-trend-loess) decomposition. The ESD algorithm is run on the resulting loess time series to detect anomalies. The loess time series represents noise in the system, after trend and seasonal behavior have been filtered out.

Let’s try an example to demonstrate.

Code Example

Before we implement the S-ESD algorithm, let’s decompose the gasoline price time series to get an idea of the time series’ trend, seasonality, and noise:

STL Decomposition of Gulf Coast Gasoline Price Time Series, 2014-Present

In the above code snippet, we decompose the time series on a yearly frequency. When detecting anomalies, the decomposed time series that we really care about is the Residual Time Series (or loess time series). This time series represents noise in the sequence, after seasonality and trend have been accounted for.

Following decomposition, we apply the Extreme Studentized Deviate (ESD) test on the residual time series to detect outliers. One of the main advantages of the ESD Test is that it only requires an upper bound for the total number of outliers to detect; this is in contrast to the Grubbs Test, where the number of outliers must be declared exactly (2).

For more information on the Extreme Studentized Deviate (ESD) algorithm, check out this link.

In the code snippet below, we perform S-ESD anomaly detection on the time series, which includes STL decomposition and outlier detection using ESD:

Gasoline Time Series, color coded on S-ESD anomalies, with outliers in yellow and all other data points in purple

The results for S-ESD algorithm are not too promising, based on the results above. The algorithm only successfully manages to identify one of the nine generated anomalies. I’ve previously had success with this algorithm using other time series sets, but it appears to struggle with this particular example.

One-Class Support Vector Machines (SVM’s)

One-class SVM’s are the unsupervised version of support vector machines, as they are trained on only one class (the “normal” class). Because the data is unlabeled, one-class support vector machines “infer the properties of the normal cases, and from these properties predict which examples are unlike the normal examples” (3). One-class SVM’s use a binary function that maps the probability density of data across a plane, with high probability regions set to +1 and low probability regions set to -1 (4).

Check out this article by Hsu et al for more in-depth information on one-class support vector machines.

Code Example

Once again we turn to scikit-learn to build our model; this time, for one-class SVM’s. We generate a couple variables to plug into the algorithm to improve performance--mainly, a centered 6-point rolling average of the gasoline price time series. We then run the original time series and its rolling average through the one_class_SVM_anomaly_detection() function. In the function, we scale each variable, and train the unsupervised model. Finally, we classify each data point, and visualize using a scatter plot:

Gasoline time series, color-coded by detected one-class SVM anomalies. Anomalous data points are in yellow, and normal data points are in purple.

The one-class SVM performs relatively well, detecting six out of the nine artificially generated outliers. However, like the isolation forest, it detects several false positives in the sequence.

Conclusions

In this article, we compared the results of a variety of anomaly detection techniques — isolation forests, low-pass filters, one class SVM’s, and the S-ESD algorithm. Surprisingly, when comparing the results side-by-side, the algorithm with the best performance (at least for this example) is a simple low-pass filter. The algorithm with the worst performance is the S-ESD algorithm.

It is important to note that these algorithms perform differently for each individual time series. Just because the low-pass filter works best with this example, doesn’t mean it works best for all time series. To achieve optimal results, try all the above options before making a final decision on which algorithm to use.

This concludes my tutorial on unsupervised machine learning methods for time series anomaly detection. Thanks for reading!

For the full Python code used in this tutorial, check out the following Github repo: https://github.com/kperry2215/unsupervised_anomaly_detection_time_series

Sources

T. Fuertes. April 4, 2018. Isolation forest: the art of cutting off from the world. https://quantdare.com/isolation-forest-algorithm/ Engineering Statistics Handbook. Generalized ESD Test for Outliers. https://www.itl.nist.gov/div898/handbook/eda/section3/eda35h3.htm Azure Machine Learning Studio. May 5, 2019. One-Class Support Vector Machine. https://docs.microsoft.com/en-us/azure/machine-learning/studio-module-reference/one-class-support-vector-machine Roemer Vlasveld. July 12, 2013. Introduction to One-class Support Vector Machines. http://rvlasveld.github.io/blog/2013/07/12/introduction-to-one-class-support-vector-machines/

Check out some of my other time series tutorials and articles: