Isolation Forest from Scratch. Implementation of Isolation forest from... | by Carlos Mougan | Towards Data Science

Lately, I have been reading about Isolation Forest and its performance in outlier/anomaly detection. After carefully reading about the algorithm (and not finding any vanilla tutorial of my taste). I decided to code it from scratch in the simplest possible way in order to grasp the algorithm better.
The goal of this post is that after reading it you can understand Isolation Forest in-depth, its strength, weakness, parameters and you are able to use it whenever you consider with knowledge of the algorithm.
For this blog/implementation, I have used this paper about Isolation forest for the pseudo-code, this Extended Isolation Forest paper for the visualizations (that corresponds with this other blog post) and using this youtube tutorial example of Random Forest implementation from Sebastian Mantey
Before getting to the code I believe that there is the need for a bit of theory.
Isolation Forest is used for outlier/anomaly detection
Isolation Forest is an Unsupervised Learning technique (does not need label)
Uses Binary Decision Trees bagging (resembles Random Forest, in supervised learning)
This method isolates anomalies from normal instances, for doing this the following assumptions for anomalies are made:
They are a minority consisting of fewer instances
They have attribute-values that are different from normal instances
In other words, anomalies are “few and different.”
Because of these first two assumptions, anomalies are susceptible to be isolated and this makes them fall closer to the root of the tree.
Isolation Forest builds an ensemble of Binary Trees for a given dataset. Anomalies, due to their nature, they have the shortest path in the trees than normal instances.
Isolation Forest converges quickly with a very small number of trees and subsampling enables us to achieve good results while being computationally efficient.
The overall code strategy will be the following. First coding a tree, then doing a forest of trees (ensembling) and finally measuring how far a certain instance goes in each tree and determining whether it is or not an outlier.
Let’s start with the tree
The input will be a sample of the data, the current tree height, and the maximum depth.
For the output, we will have a built tree.
To make it easier to follow I am working with pandas data frames, even if it is not optimal in terms of performance, makes it easier to follow for regular users.
Selecting a feature(column) of the data
def select_feature(data):     return random.choice(data.columns)
Select a random value within the range
def select_value(data,feat):    mini = data[feat].min()    maxi = data[feat].max()    return (maxi-mini)*np.random.random()+mini
Split data
def split_data(data, split_column, split_value):    data_below = data[data[split_column] <= split_value]    data_above = data[data[split_column] >  split_value]        return data_below, data_above
All together: The Isolation Tree.
The idea is the following: selecting a feature, a value of the feature, and splitting the data. If there is only one data point in the branch or the tree has reached the maximum depth: stop.
def isolation_tree(data,counter=0, max_depth=50,random_subspace=False):        # End Loop if max depth or isolated    if (counter == max_depth) or data.shape[0]<=1:        classification = classify_data(data)        return classification        else:        # Counter        counter +=1                # Select feature        split_column = select_feature(data)                # Select value        split_value = select_value(data,split_column)        # Split data        data_below, data_above = split_data(data,split_column,split_value)                # instantiate sub-tree              question = "{} <= {}".format(split_column, split_value)        sub_tree = {question: []}                # Recursive part        below_answer = isolation_tree(data_below, counter,max_depth=max_depth)        above_answer = isolation_tree(data_above, counter,max_depth=max_depth)                if below_answer == above_answer:            sub_tree = below_answer        else:            sub_tree[question].append(below_answer)            sub_tree[question].append(above_answer)                return sub_tree
Once we have our tree, we have to make a forest of them. The pseudo-code is the following:
Given an input data, a number of trees and a sampling size (how many data is fed to each tree), we fit as many trees as desired and return a forest.
The actual code:
def isolation_forest(df,n_trees=5, max_depth=5, subspace=256):    forest = []for i in range(n_trees):        # Sample the subspace        if subspace<=1:            df = df.sample(frac=subspace)        else:            df = df.sample(subspace)        # Fit tree        tree = isolation_tree(df,max_depth=max_depth)                # Save tree to forest        forest.append(tree)        return forest
After having built an Isolation Forest, we have to evaluate instances, for that we define Path Length
Given an instance and an Isolation Tree, we compute how many nodes does each instance goes through before being isolated or reaching maximum depth.
The pseudo code is extracted from Isolation forest paper.
def pathLength(example,iTree,path=0,trace=False):    # Initialize question and counter    path=path+1    question = list(iTree.keys())[0]    feature_name, comparison_operator, value = question.split()            # ask question    if example[feature_name].values <= float(value):        answer = iTree[question][0]    else:        answer = iTree[question][1]              # base case    if not isinstance(answer, dict):        return path        # recursive part    else:        residual_tree = answer        return pathLength(example, residual_tree,path=path)return path
This is just counting how many nodes an instance goes through given how data has been stored before.
Let’s play a little bit around to see
mean = [0, 0]cov = [[1, 0], [0, 1]]  # diagonal covarianceNobjs = 2000x, y = np.random.multivariate_normal(mean, cov, Nobjs).T#Add manual outlierx[0]=3.3y[0]=3.3X=np.array([x,y]).TX = pd.DataFrame(X,columns=['feat1','feat2'])plt.figure(figsize=(7,7))plt.plot(x,y,'bo');
We will train and evaluate with our data.
iForest = isolation_forest(X,n_trees=20, max_depth=100, subspace=256)# Evaluate one instancedef evaluate_instance(instance,forest):    paths = []    for tree in forest:        paths.append(pathLength(instance,tree))    return paths
We select one normal instance of the data (selected at random) and an outlier (remember that we hardcoded an outlier at the beginning as the first row). we see how far in the tree does it fall and plot the results.
outlier = evaluate_instance(X.head(1),iForest)normal  = evaluate_instance(X.sample(1),iForest)
In blue a random normal sample, in red an outlier.
np.mean(outlier)# 4.85np.mean(normal)# 12.55
We can see that there is a visual and statistical difference between the normal sample and the outlier in how deep instance falls in each tree.
From the paper of Extended Isolation Forest, there is another visualization in a radial form that provides a clear intuition of the outlier
A more formal way to evaluate the forest is using the anomaly score (s) of an instance (x):
Where H(i) is the harmonic number and it can be estimated by ln(i) + 0.5772156649 (Euler’s constant). As c(n) is the average of h(x) given n, we use it to normalize h(x).
def c_factor(n) :    """    Average path length of unsuccesful search in a binary search     tree given n points        Parameters    ----------    n : int        Number of data points for the BST.    Returns    -------    float        Average path length of unsuccesful search in a BST            """    return 2.0*(np.log(n-1)+0.5772156649) - (2.0*(n-1.)/(n*1.0))def anomaly_score(data_point,forest,n):    '''    Anomaly Score        Returns    -------    0.5 -- sample does not have any distinct anomaly    0 -- Normal Instance    1 -- An anomaly    '''    # Mean depth for an instance    E = np.mean(evaluate_instance(data_point,forest))        c = c_factor(n)        return 2**-(E/c)
Using this method we can get a mathematical definition of what is an outlier, but I believe that stopping here already allows you to have an intuition of how does the algorithm work, what are the parameters, and for which cases it might be successful.
Feel free to drop any comments or to DM message me with suggestions or anything else.
This blog has been done following
[1] Isolation forest paper for the pseudo-code from Fei Tony Liu, Kai Ming Ting and Zhi-Hua Zhou
[2] Extended isolation forest for the visualizations (that corresponds with this other blog post)
[3] Youtube Random Forest implementation from Sebastian Mantey
The full code, notebook, and images can be seen in this repository