Rapid integration testing for Spark ETL pipelines
By: Christian Johnston
Tags: Hadoop, Programming, Scala, Backend Development, Spark

This process, typically known as ETL (Extract Transform and Load), involves a number of different Hadoop technologies such as Hive, a data warehouse built on top of HDFS and used to hold the intermediary storage tables, as well as Spark, a distributed-computing framework which operates on immutable DataFrames. The power of ETL is that, due to the separation of the pipeline stages, as well as the natural properties of relational data, there is ample opportunity for parallelism:

Data Parallelism : Splitting a sequential file into smaller data files to provide parallel access, à la Hadoop.

: Splitting a sequential file into smaller data files to provide parallel access, à la Hadoop. Pipeline Parallelism : Simultaneous running of several components on the same data stream. e.g. The transformation stage on data block X can be run at the same time as the extract stage for data block X-1 .

: Simultaneous running of several components on the same data stream. e.g. The transformation stage on data block can be run at the same time as the extract stage for data block . Component Parallelism: Simultaneous running of multiple processes on different data streams in the same job.

In this blog I am going to talk about a specific problem that we faced with our ETL pipelines, specifically how a user tests any configuration changes that they make.

How our ETL is configured

At Panaseer, the data pipelines begin by ingesting data from a number of different data sources such as ServiceNow, Qualys and Microsoft AD. The flow for collecting and processing data from each of these data sources is unique, meaning we have to write and maintain a large amount of Spark code. As a solution to manage this vast complexity, we created a JSON-based language to manage the ETL flows as config. We have generic table readers (e.g. HiveReader) and loaders as well as configurable transformers which can be chained together to create pipelines of arbitrary length and complexity. Here is an example code snippet of this ETL config:

stages = [

{

stage = "read"

readerClass = HiveReader

props {

tableName = "my_table"

filterExpr = "column_1 is not null and column_1 != ''"

}

},

{

stage = "withDataSet"

dataSetIDs = ["my_table"]

transforms-loads = [

{

stage = "transform"

transformerClass = ingest.transform.ConfigurableTransformer

props {

Path = "transformations/custom_transformation.conf"

}

}

]

},

...

We find this useful for a number of reasons. Firstly, it allows us to reuse a lot of code easily. For example, we have a configurable transformer which we can pass specific transformations (more config) into and it will be converted into Spark code and be applied to the DataFrame. Secondly, the config code is much more high-level and declarative than lower-level Spark code- it is clear what this code is doing and most of the low-level technical implementation details are hidden away.

The problem

Previously, when any user wanted to test an ETL pipeline configuration change, they were forced to deploy their code to a dev or staging environment with our chosen Hadoop distribution installed.

The flow was something like this:

Make necessary configuration changes locally to ETL config files Sync (deploy) repository with the target environment​ Without any confidence the configuration is correct:​

Run the corresponding ETL job

Tail the logs while waiting for Spark Context to initialise​

Check the logs to see if there were any issues​

4. If there was an error, go back to step 1 until no more errors were found

The issues with this flow were obvious. Firstly, the time taken to configure, deploy and run these ETLs could take up to 15 minutes resulting in a very slow feedback loop, minimising developer productivity. A second issue is that, due to the dependence on Hadoop services (Hive, Zookeeper, Hbase, Yarn etc.), it was impossible to test the ETLs locally. This was an issue as, apart from the transformation unit tests, there was no easy way we could tests these pipelines end-to-end and plug it into our CI workflows. We needed a solution!

What were the requirements

ETL must be able to be run locally with no dependency on any external service. e.g. Zookeeper, Hive, HBase. The solution must provide a considerable performance speedup, improving the development experience and developer productivity. The solution must allow us to be able to test our ETL pipelines E2E and be runnable in CI.

Solution high-level view

The solution we came up with was to have a tool, similar to how the AWS Glue Crawler works, to scan Hive and HBase and extract the schemas for the underlying tables. These schemas are then stored in a central schema repository, which we call the Schema Registry. We decided to use raw files in source control, rather than a document database or some other central repository, so a user could easily access these files in the same place as the ETL code and they could copy the files to their local development environment for offline use.

Here is a diagram illustrating the flow described above: