SparkJob testing with Live Data
By: Yoav Nordmann
Tags: Data, Testing, Spark, Pipeline

There is this one spark job which gave us quite a headache. In the beginning, it seemed quite simple. Extract the data, aggregate it in some way or another, filter it, enrich the data, and in the end, save it to a database. The real “challenge” here was to loop over the source data a couple of times and create different aggregations to be saved in the resulting database, in the most efficient manner. Simple enough. The whole pipeline was written in Scala on Apache Spark with not too many lines of code.

There were even some tests. Unit Tests. Scala Test, of course, ensuring the different modules and functions performed as expected. Some of the tests were logical test and for others, we extracted some real data, saved in a file, and read into the tests.

And then we started deployment and all hell broke loose.

Testing data-driven applications/pipelines

For a service application, the input is much more defined, the environment is much more controlled. A request which does not conform to a specific standard is more likely to be discarded than anything else. There is mostly some form of interaction between the client and the server providing a form of feedback for problematic input.

None of the above is true for data-driven applications, specifically data pipelines. Of course, the internal logic can be tested with Unit tests, but events cannot be simply discarded when not conforming to a specific standard. The pipeline must not fail, the data must not be lost.

For this reason, it is not enough to just write some tests with specific input, a snapshot of the data. It is imperative to test the pipeline using real live data at any given time.

So there I was, trying to figure out how to tell my team leader how my pipeline is correct, it is the data that is wrong.

Data Pipeline Integration Tests

So my team leader Eran Shemesh had this great idea. Run a pipeline, which tests your pipeline. It is quite easy when you think of it, although getting all the smaller pieces to work together is quite tedious. So here it goes:

1. Break apart your pipeline into testable pieces such as ETL

2. Write another pipeline that uses the main pipeline as an API

3. Your test should test the outcome of the main pipeline functionality

Using this setup, I can run my testing pipeline in the same environment as the main pipeline, even extracting the same data and run my tests on real live data.

Depending on how I’ve written my code, I can control which data is being loaded and which data is being run through the transformation functions. I could even control the amount of data that is being run in this testing pipeline. But most importantly, I can run this against real live data and gain higher confidence in mission-critical pipelines.

And since this is a testing pipeline, I will want to run this automatically using my CI system

OK, Nice, But what can we test

Well, the interesting thing is, we can write tests that were previously unavailable to us in a local testing environment. Remember, our tests run in a spark environment. Following is a list of possible tests that can be run:

I can run many different pipelines just to see if the data matches.

I can do some statistical analysis of the previous data and check if the new data is within those bounds.

I can run my tests on previous data and compare the output to the current real output.

What we cannot test

On the other hand, it is important to note what we cannot test.

You have no control over the data which is being processed. There could be “missing” data, e.g. data points not present while running your testing pipeline. Or what about a batch job running for 2 hours. Are 2 hours of live data significant enough to for us to determine the tests passed?

As with all tests, they are complementary, not instead of. It will not replace your regular Unit Tests or any other tests running sample data through your pipeline.