Outlier Detection: An ETL Tutorial with Spark | by Edward Elson Kosasih | Towards Data Science

Part of the Industry 4.0 framework is to make sure that manufacturers have more visibility over what’s going on with their machines in the factory floors. This is why Industry 4.0 works tightly with Internet of Things. IoT allows large scale real-time data collection from sensors that are installed in production equipment possible. Nevertheless, having good Data Collection Agents alone isn’t sufficient. We need an automated way of extracting, analyzing and summarizing information from the large data stream, since it’s impossible for humans to do it manually. In big data terminology, this process is often referred to as ETL (Extract-Transform-Load).
Today, we’ll discuss one family of algorithm that I have personally seen to be useful in the industry: outlier detection. The idea is to find any abnormal measurements from the data stream and highlight them to the domain experts e.g. process engineers. I will share an implementation of a basic anomaly detection algorithm in Spark. I could have done the same tutorial with python’s pandas-dataframe, but unfortunately once we deal with big dataset (whose size is way larger than memory space), the latter is no longer suitable.
In this post, I’ll use scala as my choice of programming language. I personally like to use scala when I write codes for production. Python is faster to write, hence useful for prototyping. However, its’ lack of static typing could make things more difficult to debug. On the other hand, scala is more similar to Java, albeit less verbose (it runs on JVM too). But let’s leave this pro-and-cons debate for another session.
This is a simple dummy dataset that I use. Suppose we have data stream from 2 sensors. How can we automatically capture the two anomalous dots that are present below?
The model that we use finds region of values whose probability of occurrence are low under the distribution that has been fitted to the observed data. We assume that our sensors are unimodal gaussian in nature. With that, we can calculate the two thresholds that are six sigma away from the distribution’s mean.
Visually, the thresholds are fitted in this manner. Any measurements above the Upper Limit (around 25) or below the Lower Limit (around 15) are deemed as outliers.
Now to implement this in Spark, we first import all of the library dependencies.
import org.apache.spark.sql.{Column, SQLContext}import org.apache.spark.{SparkConf, SparkContext}import org.apache.spark.sql.types._import org.apache.spark.sql.functions._
We need to wrap all of our functions inside an object with a main function (This might remind you of Java). We can actually write this in a different style, one that looks more like a script. Scala has an interpreter called REPL that allows programmers to execute codes line-by-line. Nevertheless in production it’s good to keep things organized in this manner.
/**  * Sample Anomaly Detection with Spark  */object AnomalyDetection {    def main(args: Array[String]): Unit = {      // the rest of the functions go here    }}
Now if you’re running spark in Windows, you might stumble upon some exceptions saying that the program failed to locate the winutil binary. In that case, you need to download the winutils.exe (this is how to do it) and set the following system property inside your code.
System.setProperty("hadoop.home.dir", "C:/winutil/")
Next, we need to configure spark, sparkContext and sqlContext. If we use HBase (a NoSQL database) with ZooKeeper, we’ll need to configure the spark.hbase.host parameter to point the corresponding host address.
/** Configure Spark */val conf = new SparkConf()  .setAppName("anomalyDetection")  .setMaster("local[8]")conf.set("spark.hbase.host", "zookeeper.default.svc.cluster.local")val sc = new SparkContext(conf)sc.setLogLevel("WARN")val sqlContext = new SQLContext(sc)
We now assume that our data comes in a csv format. It has also been saved in a file called test.csv. We first specify the data schema explicitly. Note than in production, data could also be obtained from a database and message broker (e.g. MQTT, Kafka etc...).
/** read csv */val customSchema = StructType(Array(  StructField("sensorId", StringType, true),  StructField("values", DoubleType, true)))
Then, we read the csv file into a Spark DataFrame. Here we can see that there are only two columns: sensorId and values.
// load datasetval df = sqlContext.read  .format("com.databricks.spark.csv")  .option("header", "true")  .schema(customSchema)  .load("test.csv")df.printSchema()
root | — sensorId: string (nullable = true) | — values: double (nullable = true)
We would like to calculate the distribution profile for each sensorID, particularly the Upper and Lower Outlier Thresholds. To do that, we need to group the dataframe by sensorId, followed by aggregating each sensor data’s mean and standard deviation accordingly. We can then create 2 new columns, one for each outlier threshold.
// calculate statisticsval statsDF = df  .groupBy("sensorId")  .agg(mean("values").as("mean"), stddev("values").as("stddev"))  .withColumn("UpperLimit", col("mean") + col("stddev")*3)  .withColumn("LowerLimit", col("mean") - col("stddev")*3)statsDF.printSchema()
root | — sensorId: string (nullable = true) | — mean: double (nullable = true) | — stddev: double (nullable = false) | — UpperLimit: double (nullable = true) | — LowerLimit: double (nullable = true)
We would now like to find which sensor readings are anomalous in the original dataframe. Since the information live in two different dataframes, we need to join them using the sensorId column as a common index.
// join the two dataframeval joinDF = df.join(statsDF, usingColumns = Seq("sensorId"))
Lastly, we can filter rows whose values lie beyond the range enclosed by the outlier thresholds. Voila! We managed to capture the two anomalous points.
// outlierDetectionval detectOutlier = (values: Column, UpperLimit: Column, LowerLimit: Column) => {  // outliers are points lying below LowerLimit or above upperLimit  (values < LowerLimit) or (values > UpperLimit)}val outlierDF = joinDF  .withColumn("isOutlier", detectOutlier(col("values"), col("UpperLimit"), col("LowerLimit")))  .filter(col("isOutlier"))outlierDF.show()
(Optional) We could also write output dataframe into another table (or csv file) for further processing. This completes the whole ETL process.
outlierDF  .repartition(1)  .write.format("com.databricks.spark.csv")  .save("result")
The above method outputs a folder called result, with the following files inside it. Normally data will be split into multiple csvs (each with a different part name). But since we have repartitioned the dataframe into 1, all data is collected into one partition (or file).
The final step is to close the sparkContext.
// close sparkContextsc.stop()
Spark comes with a web interface (by default in port 4040) that could be used to see how the ETL process runs. I find this useful, especially in identifying bottleneck functions.
We have seen how a typical ETL pipeline with Spark works, using anomaly detection as the main transformation process. Note that some of the procedures used here is not suitable for production. For example, CSV input and output are not encouraged. Normally we would use Hadoop Distributed File System (HDFS) instead. The latter could be wrapped under a database too e.g. HBase. Nonetheless, the main programming paradigm stays the same.