Anomaly detection with Isolation Forest, Spark and Scala
Tags: Zeppelin, Machine Learning, Spark, Big Data, Docker

Background

Isolation Forest(iForest) is unsupervised machine learning algorithm which optimized for anomaly/outlier detection. iForest uses tree structure for modeling data, iTree isolates anomalies closer to the root of the tree as compared to normal points. A anomaly score is calculated by iForest model to measure the abnormality of the data instances. The higher, the more abnormal. More details about iForest can be found in the paper Isolation Forest and the blog post here.

In this post I’m gonna use iForest algorithm to detect anomalies on network traffic data. In my previous post I have discussed about detecting anomalies on network traffic data by using K-Means clustering with Zeppelin notebook and docker . In this post also I will run iForest algorithm on Zeppelin notebook with docker. All the source codes and dataset which related to this post available in gitlab. Please clone the repo and continue the post.

Zeppelin docker

I have run Zeppelin in my local machine with docker. Following is the docker-compose.yml to run Zeppelin. It defines two docker volumes which keeps the datasets and notebooks.

Dataset

I’m gonna analyze network data anomalies and find the attack behaviors in the network by using iForest algorithm. To do that I had to generate an attack dataset. To generate the dataset I have run a simple Scala based HTTP REST service and do real attacks to it by using Metasploit . Then captured all the packets coming to the server via TShark . I wrote a blog post about capturing network data with TShark in here. The captured dataset published as a .CSV file in gitlab. Please download the dataset and put it in the Zeppelin docker container data directory volume. Then it will be available inside Zeppelin docker container. For an example I have put it on /private/var/services/spark/zeppelin/data directory in my local machine.

Run Zeppelin

I can start Zeppelin container via docker-compose up -d zeppelin command. Then I can browse the Zeppelin web in http:localhost:8080 url. Zeppelin web provides interface to create notebooks, import notebooks, data visualizations. I have created a notebook named iforest . The iForest spark program written in that notebook. I have used Scala language to write the spark job. More details about configuring Zeppelin notebook can be found in my previous post.

iForest library

I have used iForest library which implemented by LinkedIn. The library repository can be found in github. By default iForest library does not comes with spark, so I have downloaded the jar file of iForest from maven repository and loaded it in the notebook. I have included the jar file in gitlab as well. You can get it from there if want. Since I’m running Zeppelin with docker, I need to copy the jar file into Zeppelin docker container. I have done it by using docker volume. I have put the jar file on /private/var/services/spark/zeppelin/data directory in my local machine which is the data directory volume of Zeppelin. Then it will be available inside Zeppelin docker container.

We can load external dependencies or jars into Zeppelin notebook by using %dep or %spark.dep (if using Spark) commands. More details of loading dependencies can be found in Zeppelin documentation. Following is the way I have loaded the iForest jar file into the notebook.

Load dataset

To build iForest model I have loaded the dataset into spark DataFrame . The dataset locates in /usr/zeppelin/data/tshark.csv path inside the docker container. Following is the way to load the dataset into DataFrame.

Prepare dataset

Next I have done various data preparations and added feature columns into the dataset. I have used packet count , average packet length , average frame length , packet local anomaly , frame local anomaly as the features. To generate the features I have grouped the dataset with frame_time (in 1 minute time interval), ip_src and frame_protocols . Then calculated the packet count , average packet length , average frame length , packet local anomaly , frame local anomaly and generated a new DataFrame. This DataFrame joined with original DataFrame to obtain the dataset with full feature set. Following is the way to do that. I have added comments in each phase in the code to make thing more clearer.

iForest model

In order to train and test the iForest model, the dataset need to be split into a training dataset and a test dataset. 70% of the data is used to train the model, and 30% will be used for testing. The iForest model built with training dataset and evaluated with test dataset. When building the iForest model I have used various model parameters such as setNumEstimators , setMaxFeatures . You can find more details about these parameters from here. The trained DataFrame contains predicted_label , outlier_score columns. I have created new DataFrame with these fields with sequence id column and saved in a SQL view for visualization.

Visualization

Zeppelin support to do the data visualization via SQL queries. I can execute SQL queries with data on the SQL view and generate various graphs/charts from the query output. Following are some example charts that I have used to visualize prediction output of iForest model. select id, predicted_label from iforest is the SQL query that I have used to select the relevant data. iforest is the SQL view name that I have previously created.

Full spark job

Following is the full source code of the spark job. The spark jobs can exported as JSON documents from Zeppelin web. I have exported this job as a JSON document and published in gitlab. If want you can download it and directly import it into Zeppelin web.

Reference