Data Quality Libraries: The Right Fit
By: Jan Vanbuel
Tags: Data Validation, Data Quality, Data Pipeline

Greatest common divisor

From a bird’s-eye view, all three libraries can be used to implement the following workflow:

Data quality validation workflow

Step 1: ingest a batch of data to be validated

Step 2: fetch a set of rules that your data should comply with. In case you don’t have a set of rules available yet, each of the libraries offers the capability to propose a few simple rules by profiling your data.

Step 3: calculate statistics of your data and compare against the schema/set of rules to validate the data

Step 4: store the validation results and/or calculated metrics

Step 5: Take action based on the validation result (e.g. quarantine data, send notifications, …)

Besides data validation, all three libraries offer the possibility of profiling your data (to some extent) without validating it. Now that we know their similarities, let’s see what sets them apart.

1. TensorFlow Data Validation

TensorFlow Data Validation (TFDV) is part of the TensorFlow Extended (TFX) suite, developed and used by Google for their production ML pipelines. TFDV runs on top of Apache Beam; so in principle you have access to the multiple Beam runners (Spark, Flink, Dataflow, …) to execute your validation pipeline.

Being developed by Google, it doesn’t come as a surprise that TFDV works well out-of-the-box with Google’s own data formats (TFRecord), and with the Beam DataFlow runner. To use the Spark runner, however, you have to be willing to put in some extra work. Native support is also provided for pandas data frames and csv files, but if you want other data sources, you’ll have to write your own connector.

Compared to the other two libraries, TFDV attributes more weight to data profiling. In each validation run, TFDV calculates statistics of all columns, unless specified otherwise. If your dataset is large, this could be computationally quite expensive. Great Expectations and Deequ, on the other hand, only calculate statistics that are necessary for validation, and provide profiling on the side.

Output-wise, everything in TFDV is Protobuf, Google’s own wire format. The set of constraints your data should comply with? schema.proto . Validations? anomalies.proto . Statistics? statistics.proto . Additionally, TFDV also gives you the option to write these protobufs to plain text files.

A big plus for TFDV is its visualisation capabilities; the library makes use of Google’s Facets Overview, which allows you to either visualize a single batch of data, or two batches of data side by side. You can include these Facets in a jupyter notebook, or embed them in your website.

2. Great Expectations

If you’ve read the eponymous novel by Charles Dickens, it’s hard not to chuckle when pulling Great Expectations (GE) from the PyPi repository*:

pip install great-expectations

At its core, GE’s goal is to validate your data by letting you define expectations about your data. In contrast to TFDV, it tries to make this as easy as possible by providing multiple backends (pandas, Spark, SQL), and multiple datasources and stores(file system, S3, data warehouses such as Redshift, BigQuery, Snowflake, …). All of this information (i.e. the data sources, stores, expectation suites, …) is stored in a single configuration yaml file.

Although the library exposes a Python API, it’s more convenient to use Great Expectations’ CLI to get started. After configuring your data source(s), and optionally allowing GE to infer a few simple expectations, you’re launched into a jupyter notebook which allows you to quickly iterate and fine-tune your expectations. GE also gives you the option to define your own custom expectations, both at the column and individual record level.

GE integrates out of the box with Slack so you can get notified when an expectation fails. Just pass a Slack web hook in the configuration yaml file and you’re done. In addition, GE comes with a static website, the so called data docs, to visualize the results of a validation job. You can either self-host these data-docs, or tell GE to publish the static website to S3.

Compared to TFDV, GE focuses less on profiling, and more on validation. Still, GE does offer basic profiling, providing you with a summary of the statistics of your data. Similarly, GE can generate a draft expectation suite based on these statistics, but expects you to fine-tune the suite in a notebook.

Because of its many features, the Great Expectations docs are a bit unwieldy. Of the three libraries, we found GE’s documentation to lack in clarity and conciseness.

3. Deequ

Lastly, there is Amazon’s Deequ library. Deequ is written in Scala, and built on-top of Spark. Where GE goes at lengths to accommodate you with all kinds of sources and stores for your data, Deequ keeps it simple: you’re responsible for getting and storing your data, the only thing you have to do is make sure that a some point it should reside within a Spark DataFrame. There are no bells and whistles, no fancy visualisations and no integrations with notification tools and the like.

There are, however, two features which set Deequ apart, namely the differential or stateful calculation of metrics, and its anomaly detection capabilities.

The stateful calculation of metrics allows you to update metrics for datasets which are subject to change (think of an OLTP database that grows during the day). Instead of recalculating the metrics for the entire dataset, Deequ persists the already calculated metrics in a file system, either locally or on S3, and then aggregates these with the metrics of newer data.

The anomaly detection feature of Deequ stores metrics for different batches in a MetricsRepository (again, in a local file system or S3). In addition to detecting anomalies based on absolute or relative rates of change, or simple thresholds (which is what TFDV and GE do), Deequ also gives you the option to detect anomalies based on a running average and standard deviation of those metrics.

Scala’s steeper learning curve compared to Python makes it more of a niche language. Taking this into consideration, we suspect Deequ to be mostly prevalent in the data engineering community. It’s a library written by data engineers, for data engineers.

Conclusion

It goes without saying that each of the libraries has its own strengths and weaknesses; your choice of library will depend on the needs of your specific use case. Are you building ML pipelines, a GCP enthusiast, and do you speak fluently protobuf? TFDV FTW. Do you want to include a validation step in several pipelines, with different sources and stores, and a bunch of other integrations? Get a notification in case of failure and be able to visualise what failed and why? Great Expectations might be your thing. Do you want something more simple, yet powerful? Do you need more advanced anomaly detection or incremental computation of your metrics? Deequ is right up your alley.

P.S.: It should be said that none of the libraries actually runs your data pipelines —they’re pipeline libraries and not pipeline platforms. In the end, when you’re ready to go into production, you’ll need to submit the validation job to a workflow orchestrator such as Airflow, Luigi, Kubeflow, ...

* The main protagonist in Charles Dickens’ novel ‘Great Expectations’ is named Pip.