How to use machine learning for anomaly detection and condition monitoring
By: Vegard Flovik
Tags: Machine Learning, Data Analysis, Predictive Analytics, Towards Data Science, Statistics

Technical section:

It is hard to cover the topics of machine learning and statistical analysis for anomaly detection without also going into some of the more technical aspects. I will still avoid going too deep into the theoretical background (but provide some links to more detailed descriptions). If you are more interested in the practical applications of machine learning and statistical analysis when it comes to e.g. condition monitoring, feel free to skip ahead to the “Condition monitoring use-case” section.

Approach 1: Multivariate statistical analysis

Dimensionality reduction using principal component analysis: PCA

As dealing with high dimensional data is often challenging, there are several techniques to reduce the number of variables (dimensionality reduction). One of the main techniques is principal component analysis (PCA), which performs a linear mapping of the data to a lower-dimensional space in such a way that the variance of the data in the low-dimensional representation is maximized. In practice, the covariance matrix of the data is constructed and the eigenvectors of this matrix are computed. The eigenvectors that correspond to the largest eigenvalues (the principal components) can now be used to reconstruct a large fraction of the variance of the original data. The original feature space has now been reduced (with some data loss, but hopefully retaining the most important variance) to the space spanned by a few eigenvectors.

Multivariate anomaly detection

As we have noted above, for identifying anomalies when dealing with one or two variables, data visualization can often be a good starting point. However, when scaling this up to high-dimensional data (which is often the case in practical applications), this approach becomes increasingly difficult. This is fortunately where multivariate statistics comes to help.

When dealing with a collection of data points, they will typically have a certain distribution (e.g. a Gaussian distribution). To detect anomalies in a more quantitative way, we first calculate the probability distribution p(x) from the data points. Then when a new example, x, comes in, we compare p(x) with a threshold r. If p(x)<r, it is considered as an anomaly. This is because normal examples tend to have a large p(x) while anomalous examples tend to have a small p(x)

In the context of condition monitoring, this is interesting because anomalies can tell us something about the “health state” of the monitored equipment: Data generated when the equipment approaches failure, or a sub-optimal operation, typically have a different distribution than data from “healthy” equipment.

The Mahalanobis distance

Consider the problem of estimating the probability that a data point belongs to a distribution, as described above. Our first step would be to find the centroid or center of mass of the sample points. Intuitively, the closer the point in question is to this center of mass, the more likely it is to belong to the set. However, we also need to know if the set is spread out over a large range or a small range, so that we can decide whether a given distance from the center is noteworthy or not. The simplistic approach is to estimate the standard deviation of the distances of the sample points from the center of mass. By plugging this into the normal distribution we can derive the probability of the data point belonging to the same distribution.

The drawback of the above approach was that we assumed that the sample points are distributed about the center of mass in a spherical manner. Were the distribution to be decidedly non-spherical, for instance ellipsoidal, then we would expect the probability of the test point belonging to the set to depend not only on the distance from the center of mass, but also on the direction. In those directions where the ellipsoid has a short axis the test point must be closer, while in those where the axis is long the test point can be further away from the center. Putting this on a mathematical basis, the ellipsoid that best represents the set’s probability distribution can be estimated by calculating the covariance matrix of the samples. The Mahalanobis distance (MD) is the distance of the test point from the center of mass divided by the width of the ellipsoid in the direction of the test point.

In order to use the MD to classify a test point as belonging to one of N classes, one first estimates the covariance matrix of each class, usually based on samples known to belong to each class. In our case, as we are only interested in classifying “normal” vs “anomaly”, we use training data that only contains normal operating conditions to calculate the covariance matrix. Then, given a test sample, we compute the MD to the “normal” class, and classify the test point as an “anomaly” if the distance is above a certain threshold.

Note of caution: Use of the MD implies that inference can be done through the mean and covariance matrix — and that is a property of the normal distribution alone. This criteria is not necessarily fulfilled in our case, as the input variables might not be normal distributed. However, we try anyway and see how well it works!

Approach 2: Artificial Neural Network

Autoencoder networks

The second approach is based on using autoencoder neural networks. It is based on similar principles as that of the above statistical analysis, but with some slight differences.

An autoencoder is a type of artificial neural network used to learn efficient data codings in an unsupervised manner. The aim of an autoencoder is to learn a representation (encoding) for a set of data, typically for dimensionality reduction. Along with the reduction side, a reconstructing side is learnt, where the autoencoder tries to generate from the reduced encoding a representation as close as possible to its original input.

Architecturally, the simplest form of an autoencoder is a feedforward, non-recurrent neural network very similar to the many single layer perceptrons which makes a multilayer perceptron (MLP) — having an input layer, an output layer and one or more hidden layers connecting them — but with the output layer having the same number of nodes as the input layer, and with the purpose of reconstructing its own inputs.

Figure 2: Autoencoder network

In the context of anomaly detection and condition monitoring, the basic idea is to use the autoencoder network to “compress” the sensor readings to a lower-dimensional representation, which captures the correlations and interactions between the various variables. (Essentially the same principle as the PCA model, but here we also allow for non-linear interactions between the variables).

The autoencoder network is then trained on data representing the “normal” operating state, with the goal of first compressing and then reconstructing the input variables. During the dimensionality reduction, the network learns the interactions between the various variables and should be able to re-construct them back to the original variables at the output. The main idea is that as the monitored equipment degrades, this should affect the interaction between the variables (e.g. changes in temperatures, pressures, vibrations, etc.). As this happens, one will start to see an increased error in the networks re-construction of the input variables. By monitoring the re-construction error, one can thus get an indication of the “health” of the monitored equipment, as this error will increase as the equipment degrades. Similar to the first approach of using the Mahalanobis distance, we here use the probability distribution of the reconstruction error to identify whether a data point is normal or anomalous.