Isolation Forest is the best Anomaly Detection Algorithm for Big Data Right Now | by Andrew Young | Towards Data Science

Isolation forest or “iForest” is an astoundingly beautiful and elegantly simple algorithm that identifies anomalies with few parameters. The original paper is accessible to a broad audience and contains minimal math. In this article, I will explain why iForest is the best anomaly detection algorithm for big data right now, provide a summary of the algorithm, history of the algorithm and share a code implementation.
Best-in-class performance that generalizes. iForest performs better than most other outlier detection (OD) algorithms across a variety of datasets, based on ROC performance and Precision. I took benchmark data from the authors of the Python Outlier Detection package (PyOD; original paper) and applied row-wise green-red gradient conditional formatting in Excel. Dark green indicates the best performing algorithm for a dataset and dark red indicates the worst performing:
We see that iForest leads across most datasets and in aggregate, as shown by colors for the mean, median and standard deviation rows that I calculated. The same pattern of superior results for iForest holds true for precision @ N performances:
Scalability. iForest is the fastest performing algorithm for its performance. PCA and Histogram-based Outlier Score (HBOS; paper, poster) are expectedly faster across all datasets. k-nearest-neighbors (KNN) is much slower and scales terribly with more observations, N.
I’ve successfully constructed isolation forests with datasets containing 100M observations with 36 columns on the order of minutes in a cluster environment. I cannot say the same for sk-learn’s KNN().
I will provide a grossly simplified summary of the 10 page paper with the following overview:
Most other outlier detection (OD) algorithms seek to build a profile of “normal” instances then flag instances that don’t fit that profile of normality. iForest explicitly isolates anomalous records by taking advantage of inherent properties of anomalies: they have unusual values for the set of covariates.Existing methods are constrained to low dimensional data and small data size due to computational expense. Case in point: ever try sklearn.neighbor.KNeighborsClassifier on big data? :)Additionally, iForest has “a low constant and low memory requirement” i.e. low overhead. Specifics: the number of external nodes is n since each observation, n, is isolated by itself. The total number of internal nodes is clearly n-1, and the total number nodes is 2n-1. Hence, we see why the memory requirement is bounded and grows linearly with n.Isolation tree node definition: T is either a childless external node or an internal node with one test and exactly two daughter nodes (Tl, Tr). To build an iTree, we recursively divide X by randomly selecting an attribute q and a split value p until either: (i) the tree reaches a height limit, (ii) all observations are isolated at their own external node, or (iii) all data have the same values for all attributes.Path length. The path length h(x) of an observation x is measured by the number of edges x traverses an iTree from the root node until transversal is terminated at an external node. E(h(x)) is the average of h(x) from a collection of isolation trees. An anomaly score, s(x, n), can be derived from the average path length, E(h(x)): s(x, n) = 2^[− E(h(x)) / c(n)]. Basically, there is a monotonic relationship between s and E(h(x)) (see appendix at end for details and a helpful figure illustrating their relationship). I won’t get into the term c(n) so I can keep this brief but it is a constant for any given, static data set.Only requires user to set two variables: the number of trees to build and the sub-sampling size. The authors present experiments with generated Gaussian distribution data that show how convergence for mean path length is achieved relatively quickly with few trees and small subsamples.Small subsampling (sample of a sample) size addresses swamping and masking issues. Swamping and masking are caused by input data that is too large for the purposes of anomaly detection. Swamping is when an “normal” observation is mistaken for an “anomalous” one because it is surrounded by anomalies and masking is the opposite. In other words, when a tree is fed a sample consisting of a majority of anomalies, a normal data point might look anomalous. The authors present examples of this phenomenon with mammography data.Small subsamples allows each isolation tree to be specialized,as each sub-sample includes a different set of anomalies oreven no anomalyiForest doesn’t depend on any distance or density-based measures to identify anomalies so it is fast and computationally inexpensive, which leads to the next pointLinear time complexity, O(n). Informally, this means that the running time increases at most linearly with the size of the input. This is pretty good:
Most other outlier detection (OD) algorithms seek to build a profile of “normal” instances then flag instances that don’t fit that profile of normality. iForest explicitly isolates anomalous records by taking advantage of inherent properties of anomalies: they have unusual values for the set of covariates.
Existing methods are constrained to low dimensional data and small data size due to computational expense. Case in point: ever try sklearn.neighbor.KNeighborsClassifier on big data? :)
Additionally, iForest has “a low constant and low memory requirement” i.e. low overhead. Specifics: the number of external nodes is n since each observation, n, is isolated by itself. The total number of internal nodes is clearly n-1, and the total number nodes is 2n-1. Hence, we see why the memory requirement is bounded and grows linearly with n.
Isolation tree node definition: T is either a childless external node or an internal node with one test and exactly two daughter nodes (Tl, Tr). To build an iTree, we recursively divide X by randomly selecting an attribute q and a split value p until either: (i) the tree reaches a height limit, (ii) all observations are isolated at their own external node, or (iii) all data have the same values for all attributes.
Path length. The path length h(x) of an observation x is measured by the number of edges x traverses an iTree from the root node until transversal is terminated at an external node. E(h(x)) is the average of h(x) from a collection of isolation trees. An anomaly score, s(x, n), can be derived from the average path length, E(h(x)): s(x, n) = 2^[− E(h(x)) / c(n)]. Basically, there is a monotonic relationship between s and E(h(x)) (see appendix at end for details and a helpful figure illustrating their relationship). I won’t get into the term c(n) so I can keep this brief but it is a constant for any given, static data set.
Only requires user to set two variables: the number of trees to build and the sub-sampling size. The authors present experiments with generated Gaussian distribution data that show how convergence for mean path length is achieved relatively quickly with few trees and small subsamples.
Small subsampling (sample of a sample) size addresses swamping and masking issues. Swamping and masking are caused by input data that is too large for the purposes of anomaly detection. Swamping is when an “normal” observation is mistaken for an “anomalous” one because it is surrounded by anomalies and masking is the opposite. In other words, when a tree is fed a sample consisting of a majority of anomalies, a normal data point might look anomalous. The authors present examples of this phenomenon with mammography data.
Small subsamples allows each isolation tree to be specialized,as each sub-sample includes a different set of anomalies oreven no anomaly
iForest doesn’t depend on any distance or density-based measures to identify anomalies so it is fast and computationally inexpensive, which leads to the next point
Linear time complexity, O(n). Informally, this means that the running time increases at most linearly with the size of the input. This is pretty good:
11. For the reader with relevant background, the following diagrams from the paper make it explicitly clear how the algorithm works:
The well-read reader knows that the lag time between a great new idea and wider adoption can be decades long. For example, the logistic function was discovered in 1845, re-discovered in 1922 (read more) and is now regularly used by modern-day data scientists for logistic regression. The lag time between a new idea and its broader adoption has decreased in recent decades, but it is still arguably a long period of time. iForest was first shared in 2008 and commercially viable applications weren’t released until late 2018! Timeline below:
12/2008 — Original paper for iForest released (paper)07/2009 — iForest authors last modify their code implementation (code)10/2018 — h2o team makes iForest available to R and Python users (code)01/2019 — PyOD releases outlier detection (OD) toolkit for Python users (code, paper)08/2019 — LinkedIn engineering team releases Spark/Scala implementation of iForest (code, press release)
Since this article is about big data, I assume a cluster environment. My implementation below works with data stored in AWS S3 but can be minimally modified to be used elsewhere. Code scaffolding (code to QA objects, debug, etc.) mostly omitted. Need help installing packages for an AWS cluster environment? See my Medium article: How to Set-up a cost-effective AWS EMR cluster and Jupyter Notebooks for SparkSQL
I have found iForest to easily and quickly work with 7.5M rows and 36 features, finishing calculations on the order of minutes.
Python (h2o):
The following histogram is produced by the code above. For the observations in my DataFrame predictions_fullX , most observations are not anomalous as evidenced by the frequency of larger mean path lengths.
If your data has labels you’d like to corroborate with iForest, you can compare the distributions of the sets of normal instances with anomalous instances and with the original dataset for further inference. For example, you can look at counts by different combinations of features in the original data set, like so:
N = df.count()df[['v1', 'v2', 'id']].groupby(['v1', 'v2']).count() / Ndf[['v1', 'v3', 'id']].groupby(['v1', 'v3']).count() / N...
and compare with normal/anomalous sets of instances as determined by iForest as shown below:
I did all of that and also exported my data to Excel to quickly produce some cumulative distribution functions like so:
Python (pyod) (I’ll add code if requested in the comments!)
Update (15 Dec 2020), PyOD code:
There are plenty of great examples on the PyOD github (link) so I don’t see much need for me to make one up. If you have a specific request for PyOD code that isn’t available on the GitHub repo, please let me know in the comments. Github doesn’t show you what the results should look like, so my value-add can be showing you what the code should output, in case you’re having trouble getting even that far.
The output for https://github.com/yzhao062/pyod/blob/master/examples/ocsvm_example.py should look like:
Scala (forthcoming)
I used the LinkedIn Team’s implementation of iForest (github) for Scala since PyOD and h2o are for Python. Presently not able to get it to work but will update when I can.
F. T. Liu, K. M. Ting, and Z.-H. Zhou. Isolation forest. In: Proceedings of the 8th IEEE International Conference on Data Mining (ICDM’08), Pisa, Italy, 2008, pp.413–422. [code] This paper won the Theoretical/Algorithms Runner-Up Best Paper Award at IEEE ICDM’08
Zhao, Y., Nasrullah, Z. and Li, Z., 2019. PyOD: A Python Toolbox for Scalable Outlier Detection. Journal of machine learning research (JMLR), 20(96), pp.1–7.
Andrew Young is an R&D Data Scientist Manager at Neustar. For context, Neustar is an information services company that ingests structured and unstructured text and picture data from hundreds of companies in the domains of aviation, banking, government, marketing, social media and telecommunications to name several. Neustar combines these data ingredients then sells a finished dish with added value to enterprise clients for purposes like consulting, cyber security, fraud detection and marketing. In this context, Mr. Young is a hands-on lead architect on a small R&D data science team responsible for building, optimizing and maintaining a system feeding all products and services responsible for $1+ billion in annual revenue for Neustar. Follow Andrew on LinkedIn to stay up-to-date on the latest trends in data science!
Every Data Scientist Should Know: The Bias-Variance Trade-off Generalization is Wrong
Must-Know Presentation Tools for the Effective Data Scientist
Algorithm main points/summary, point #5 details:
when E(h(x)) → c(n), s → 0.5 (the observation is not an obvious anomaly)
when E(h(x)) → 0, s → 1 (anomaly)
when E(h(x)) → n − 1, s → 0 (not an anomaly)
so to summarize the above in the words of the paper:
(a) if instances return s very close to 1, then they are definitely anomalies,
(b) if instances have s much smaller than 0.5, then they are quite safe to be regarded as normal instances, and
© if all the instances return s ≈ 0.5, then the entire sample does not really have any distinct anomaly.
Helpful figure that illustrates the relationship between anomaly score, s, and average path length E(h(x))