“Isolation Forest”: The Anomaly Detection Algorithm Any Data Scientist Should Know
By: Samuele Mazzanti
Tags: Machine Learning, Isolation Forests, Anomaly Detection, Unsupervised Learning, Data Science

“Isolation Forest” is a brilliant algorithm for anomaly detection born in 2009 (here is the original paper). It has since become very popular: it is also implemented in Scikit-learn (see the documentation).

In this article, we will appreciate the beauty in the intuition behind this algorithm and understand how exactly it works under the hood, with the aid of some examples.

“Why is anomaly detection so hard?”

Anomaly (or outlier) detection is the task of identifying data points that are “very strange” compared to the majority of observations.

This is useful in a range of applications, from fault detection to discovery of financial frauds, from finding health issues to identifying unsatisfied customers. Moreover, it can also be beneficial for machine learning pipelines, since it has been proven that removing outliers leads to an increase in model accuracy.

What makes anomaly detection so hard is that it is an unsupervised problem. In other words, we usually don’t have labels telling us which instances are actually “anomalies”. Or rather, even if we had labels, it would be very hard to frame anomaly detection as a supervised problem. In fact:

anomalies are rare;

anomalies are novel;

anomalies are different from each other.

For all these reasons, supervised techniques typically make a bad fit with anomaly detection.

What is so special about Isolation Forest

The traditional approach to anomaly detection was roughly:

Describe how “normal instances” look like (this usually involves cluster analysis). Label all instances that don’t fit into those profiles as outliers.

The innovation introduced by Isolation Forest is that it starts directly from outliers rather than from normal observations.

The core idea is that it should be very easy to “isolate” anomalies based on the caracteristics that make them unique.

Technically, this translates into the fact that, if we fit a decision tree on all the observations, outliers should be found closer to the root of the tree than “normal” instances.

What does it mean? Let’s make this clear with an example.

Suppose that we have a dataset containing data about all the 7,932,843,214 humans alive right now. We have as many variables as we want: age, net worth, place of residence, job title…

What are the outliers in such a dataset? Keep in mind that outliers are not necessarily wrong data: they are just data points that are very different from the rest of the population. In this example, Jeff Bezos is for sure an outlier.

Now imagine that we could fit a decision tree such that each terminal leaf contains one and only one person. In other words, this tree is completely unpruned. If the assumption behind Isolation Forest is correct, then Jeff Bezos will be found closer to the tree root than, say, myself.

A completely unpruned decision tree, in which each terminal leaf identifies a single person in the world [Image by Author]

Being an outlier, Jeff Bezos is easier to isolate: it’s enough to ask “is he worth more than 170 billion $?” to retrieve him among almost 8 billion humans. On the other hand, since I am by far more ordinary than Jeff Bezos, you would probably need at least 10 True/False question to narrow down the search space until you find me.

Looking under the hood

Now that we have seen the main intuition behind Isolation Forest, let’s try to understand the exact mechanics of the algorithm, with the aid of some simple data points.

import pandas as pd df = pd.DataFrame({

'x': [1.0, 1.1, 1.2, 1.3, 1.4, 1.5, 2.0],

'y': [2.1, 2.4, 3.0, 2.6, 2.2, 2.8, 3.7]

}, index = ['A', 'B', 'C', 'D', 'E', 'F', 'G'])

Some data points [Image by Author]

Items from A to F represent a quite compact cloud of points: they are “normal” data points. Compared to these instances, G is probably an outlier: it has anomalous values both for x and y.

Isolation Forest is based on trees, so let’s fit a tree on these data:

A tree fit randomly on the data points. [Image by Author]

Note that this tree has been grown in a random fashion.

The most fundamental concept here is the depth of the leaf at which each element is found. For example, in this tree, the observation called G (our outlier) is at depth 1 (e.g. 1 level from the root node), whereas C is at depth 3.

The idea behind Isolation Forest is that, on average, outliers will be closer to the root node (i.e. at a lower depth) than normal instances.

As often in machine learning, the key is iteration. In fact, if we randomly fit many decision trees, and then take an average of the depth of each observation over the different trees, we find an “average depth” that represents an empirical measure of “outlierness”.

Isolation Forest in Scikit-learn

Let’s see an example of usage through the Scikit-learn’s implementation.

from sklearn.ensemble import IsolationForest iforest = IsolationForest(n_estimators = 100).fit(df)

If we take the first 9 trees from the forest ( iforest.estimators_[:9] ) and plot them, this is what we get:

First 9 (out of 100) decision trees of Isolation Forest. The number next to each data point is its depth in the tree. [Image by Author]

Taking a look at these first 9 trees, we can already see a pattern: G tends to be at a much lower depth (1.44 on average) than any other point. Indeed, the second point is A with an average depth of 2.78.

Conceptually, this is exactly how the algorithm works: a lower average depth means a higher likelihood of being an outlier.

However, in practice, we cannot use average depth, since the depth of a tree depends on the number of samples it has been fit on. For this reason, we need a formula that also take into account the total number of instances. This is the formula proposed in the paper:

Score of “outlierness” proposed by Liu. [Screenshot from here]

where n is the number of instances, h(x) is the depth at which the data point is found in a particular tree (E(h(x)) is its average over different trees), and H is the armonic number.

s(x, n) is a number between 0 and 1, where the higher the score the more likely it is an outlier.

Note: Scikit-learn’s implementation returns the opposite of the score defined above. So what said above is still valid, but with negative sign.

On our small dataset, the scores are given by:

scores = iforest.score_samples(df)

Let’s see the scores estimated for each of our points:

Scores estimated by Isolation Forest [Image by Author]

As we expected, G is more likely to be an outlier, since its score is lower than all the other scores.