Great Expectations: The Data Testing Tool – Is This the Answer to Our Data Quality Needs? | by Karen Bajador Valencia | Towards Data Science

I’m a data engineer currently working in the banking industry. I play a big part in delivering an essential project that focuses on risk assessments to protect the institution and its customers from financial crimes. It is particularly crucial to produce periodical risk reports that are accurate, complete, consistent, time-relevant and standardised to perform correct analysis and mitigate inherent risks. Being unable to do so will result in serious consequences not only to the bank but also to its stakeholders and individual clients. Incorporating a robust data quality process in data pipelines is a must and choosing an efficient tool to utilise for this purpose is definitely going to be a game-changer.
Great Expectations is a tool that helps teams eliminate pipeline debt through data testing, documentation and profiling. I’ve been meaning to explore and evaluate this tool to determine if it is feasible to integrate with our existing environment and whether it could potentially ameliorate apparent constraints in our current process and prevent major refactoring efforts when we migrate data platforms in the future.
To deliver downstream metrics for analysis, we have to process hundreds of millions of data from a variety of source systems that goes through different layers of transformation. We leverage storage and compute suitable for Big data such as AWS S3 and AWS EMR, use Jupyter and Pyspark as ETL tools, and load the final metrics into AWS Redshift.
The pipeline does five main things: prepare the dependencies, determine the order of execution, execute the notebooks, validate test results, and lastly, load the final metrics into the Data warehouse.
Each notebook creates a single data entity. To run some quality checks, aggregated data is queried and loaded into a DataFrame and unit tests are executed against it. The results of the tests are logged into S3 and loaded during the test validation phase to assess coverage.
This process currently works but there are a few things that I think could be done better.
Unit-test code exists in each notebook. But most notebooks are pretty much running exactly the same unit tests. This results in duplication of code.
It would be nice to have a user-friendly static website shared across the team that shows what exactly is going on with our data in any environment. In our current setup, all we get is a long list of test results outputted in the console when the pipeline is executed.
We are anticipating a big move from Redshift and Spark environment to Snowflake soon which means we’ll have to ditch our notebooks that are mainly written in Pyspark and where unit tests currently reside, and possibly switch to DBT to do our data transformations. We’ll have to find a way to run the tests against a new data source.
Sure we can always refactor the code to generate a reusable test module that is decoupled and implements dependency injection so that it doesn’t matter if the source of data changes. Documentation, on the other hand, is not trivial to implement. It would be lovely if there is such a tool that will help overcome the current limitations and offers way more beneficial features.
Unit tests or expectations translated directly into documentationAdd extra information about the unit testDocumentation can be hosted in S3Ready-to-use common data quality test functionsSupport most of the data quality checks we already doSupport for multiple Pandas/Spark DataFrame and Snowflake Data sourceExtensibleIntegration with orchestration tools like Airflow or DBT
Unit tests or expectations translated directly into documentation
Add extra information about the unit test
Documentation can be hosted in S3
Ready-to-use common data quality test functions
Support most of the data quality checks we already do
Support for multiple Pandas/Spark DataFrame and Snowflake Data source
Extensible
Integration with orchestration tools like Airflow or DBT
We know how powerful and convenient documentation is but nobody wants to maintain separate documentation just to ensure it is synced with the latest code. I prefer an out-of-the-box feature that generates documentation out of unit tests defined.
Great expectations: Supported; Score: 1/8
I want to be able to elaborate on what is exactly being tested instead of generic descriptions to make it comprehensible enough for anyone who checks the data quality reports.
According to the style guide, it is recommended to use google style docstrings.
Great expectations: Supported; Score: 2/8
AWS S3 is a convenient location to host static websites and since we already use S3, might as well use the same to deploy the shared documentation.
By default, docs are stored locally but can also be deployed in a cloud blob storage including S3.
Great expectations: Supported; Score: 3/8
It would greatly expedite the refactoring process if there are already native unit test functions that we can leverage to avoid having to write dynamic tests from scratch.
Great expectations: Supported; Score: 4/8
I’ve searched for available expectations that potentially match our most common data quality checks at least semantically.
Great expectations: Mostly supported; Score: 5/8
It should work for both our current and future environment with minimal effort to refactor.
Data sources supported: Pandas DataFrame, Spark DataFrame, and Snowflake via SQLAlchemy
Great expectations: Supported; Score: 6/8
Even though it supports most of our data quality expectations out of the box, it should be flexible enough to be able to write custom tests for those that it doesn’t natively support.
Great expectations: Supported; Score: 7/8
It should support integration with the existing orchestration platforms in our data ecosystem.
Validations can be run via Airflow through BashOperator or PythonOperator. There is also an extension package for DBT as well.
Great expectations: Supported; Score: 8/8
Looks like Great Expectations looks good on paper but will it actually do the work? Let’s find out! Follow me in my journey as I try out the tool and explore its capabilities. Watch out for the next sequel in this blog series.
Data quality unit tests in Pyspark using Great Expectations