Monitoring Data Quality in a Data Lake Using Great Expectations and Allure-Built Serverless | by Bvolodarskiy | Towards Data Science

The artificial intelligence, machine learning, and big data industries are rapidly growing. They are no longer just about hype — all three industries have matured to the point where they are able to help various organizations drive growth and deliver tangible results. In this context, the quality of associated solutions, and especially, the quality of data that powers those solutions, becomes paramount.
At Provectus, we understand that the quality of data in AI makes all the difference. For that purpose, we use various services for data quality monitoring that can help us identify and keep track of data trends, detect data shifts, and more. All of these services are helpful in identifying risks and preventing expenses. The theory behind software testing implies that the sooner you find a bug and fix it, the less money will be spent. In the case of AI and ML, the research is clear that good data is much more useful than coefficients in models.
Today’s AI reality is that it is no longer enough to simply make data pipelines. Handling your ML models and data more closely resembles a standard IT routine, with all of its processes. So how can you assess the quality of data efficiently and at scale?
We already have some experience with two open-source frameworks for Data QA: Deequ and Great Expectations, but mostly on premise, not in cloud. This article provides some perspective on using Great Expectations to monitor data quality in the AWS Cloud.
Great Expectations (GE) is an open-source data quality framework based on Python. GE enables engineers to write tests, review reports, and assess the quality of data. It is a plugable tool, meaning you can easily add new expectations and customize final reports. GE can be easily integrated with ETLs such as Airflow, and it has AWS support. The latter is important to Provectus, an AWS Premier Consulting Partner, because we are now working to add Pandas profiling to our mix of data quality monitoring tools. Pandas is perfect for analyzing data and auto generating simple test suites for GE.
The Allure framework is a classic IT tool for testing and reporting. It is a good option for managers and non-technical professionals because it allows them to handle data in a more comfortable way, compared to a strictly technical GE. In this implementation, we use both Allure and GE reports by means of a self-written adapter from GE to Allure.
Solution Architecture
At the beginning, we built a POC based on the following reference-to-architecture.
In the implementation we used in one of our previous projects (Client Under NDA). We relied on an Airflow-based solution, and also on our preconceived strategy for Data QA with GE. In this case, however, we needed to collect large amounts of data to structure it, and to come up with a user-friendly dashboard, enabling C-level employees to handle the data. From that perspective, the quality of data once again presented itself as critical.
Our first step was to select the right dataset, and we reached out to the data owner to find out what exactly we needed to check in it. We then wrote several local test suites: a technical one to provide technical details about null vs empty values in columns, and another to look into business and operational issues. After that, the GE config was changed to an AWS (Amazon S3) integration, with the data uploaded to its storage. For operational excellence we used docker images and ECR for Lambda. We tested it and began to investigate the best way to integrate it to the existing data pipeline. We decided to use Step Functions with AWS Lambda as an ETL.
Here is the final solution.
Solution Overview
Retrieve data from data sources using AWS Lambda and AWS GlueIn parallel, run AWS Lambda with Pandas Profiling and GE Test Suite for each dataset, and store/serve each as a Static S3 WebsiteIn AWS Lambda, convert GE results to Allure results, and store them in Amazon S3In AWS Lambda, generate Allure reports from Allure results, stored and served in Amazon S3By using AWS, Lambda sends reports to the Slack channelResults are pushed to Amazon DynamoDB (you can also use Amazon S3 for this to reduce costs)Data is crawled from Amazon DynamoDB by using Amazon AthenaData is pulled with Amazon Quicksight
Retrieve data from data sources using AWS Lambda and AWS Glue
In parallel, run AWS Lambda with Pandas Profiling and GE Test Suite for each dataset, and store/serve each as a Static S3 Website
In AWS Lambda, convert GE results to Allure results, and store them in Amazon S3
In AWS Lambda, generate Allure reports from Allure results, stored and served in Amazon S3
By using AWS, Lambda sends reports to the Slack channel
Results are pushed to Amazon DynamoDB (you can also use Amazon S3 for this to reduce costs)
Data is crawled from Amazon DynamoDB by using Amazon Athena
Data is pulled with Amazon Quicksight
Step Functions:
Bear in mind that we don’t want (or need) to stop the implementation of the usual data engineering steps, but we can deploy all stacks to Cloudformation by using Serverless.
*** — sensitive information
To implement GE on AWS you need to:
Change the GE config(great_expectation.yml) datasource, sources, and data_docs
Change the GE config(great_expectation.yml) datasource, sources, and data_docs
datasources:  pandas_s3:    class_name: PandasDatasource    batch_kwargs_generators:      pandas_s3_generator:        class_name: S3GlobReaderBatchKwargsGenerator        bucket: ***        assets:          your_first_data_asset_name:            prefix: ***/             regex_filter: .*     module_name: great_expectations.datasource    data_asset_type:      class_name: PandasDataset      module_name: great_expectations.datasetexpectations_S3_store:    class_name: ExpectationsStore    store_backend:      class_name: TupleS3StoreBackend      bucket: '***'      prefix: '***/great_expectations/expectations/'  validations_S3_store:    class_name: ValidationsStore    store_backend:      class_name: TupleS3StoreBackend      bucket: '***'      prefix: '***/great_expectations/uncommitted/validations/'  evaluation_parameter_store:    class_name: EvaluationParameterStoreexpectations_store_name: expectations_S3_storevalidations_store_name: validations_S3_storeevaluation_parameter_store_name: evaluation_parameter_storedata_docs_sites:  s3_site:    class_name: SiteBuilder    show_how_to_buttons: false    store_backend:      class_name: TupleS3StoreBackend      bucket: ***    site_index_builder:      class_name: DefaultSiteIndexBuilder      renderer:        module_name: custom_data_docs.renderers.custom_table_renderer        class_name: CustomTableRenderer      show_cta_footer: true
2. Create test suites and write tests; remember to change the path to necessary files
# ### Column Expectation(s)# #### `id`# In[ ]:batch.expect_column_values_to_match_regex(column='id',                                          regex="^\d+$")# #### `lastNameFirstName`# In[ ]:batch.expect_column_values_to_match_regex(column='lastNameFirstName',                                          regex="([a-zA-Z .-]){2,30}[,]+[ ]+([a-zA-Z .-]){2,30}$")# #### `customTeam`# In[ ]:# In[ ]:batch.expect_column_values_to_match_regex(column='customTeam',                                          regex="^[a-zA-Z -]+$")
3. If you want to use AWS Lambda,wrap your code as lambda_handler or
def handler(event, link):    context = DataContext(os.path.join(BASE_DIR, 'great_expectations'))    expectation_suite_name = "***"    suite = context.get_expectation_suite(expectation_suite_name)    suite.expectations = []    batch_kwargs = {'data_asset_name': '***', 'datasource': 'pandas_s3', 'path': 's3a://***/'+event[0],'link':link}    batch = context.get_batch(batch_kwargs, suite)    batch.head()
4. Create requirements.txt and dockerfile
importlib-metadata==2.0great-expectations==0.13.19s3fs==2021.6.0python-dateutil==2.8.1botocore==1.20.49boto3==1.17.1aiobotocore==1.3.0requests==2.25.1decorator==4.4.2pyarrow==2fastparquetawslambdaricawswranglerpandas_profilingFROM python:3.6-stretchRUN apt-get update \ && apt-get upgrade -y \ && apt-get install -yRUN apt-get install -y python3-devRUN apt-get install -y python3-pipRUN pip3 install --upgrade cythonRUN pip3 install setuptools_scmADD ./requirements.txt ./COPY  edit_bamboo_hr_technic.py ./COPY edit_bamboo_hr.py ./COPY test_pandas_profiler.py ./COPY bamboohr_report_data_test.py ./COPY great_expectations /great_expectationsRUN pip3 install  -r requirements.txtENTRYPOINT ["/usr/local/bin/python", "-m", "awslambdaric"]CMD ["bamboohr_report_data_test.handler"]
In Amazon S3, use a static Website option. If you want to add more security, implement Amazon CloudFrontDeploy to Cloudformation by using a Serverless tool. First, you need to ‘docker-compose build’ and then to ‘sls deploy’. If you want to have a single Lambda with GE, you need to create an ECR repository and to go through the instructions to push your docker image. After that, create an AWS Lambda image and select it.
In Amazon S3, use a static Website option. If you want to add more security, implement Amazon CloudFront
Deploy to Cloudformation by using a Serverless tool. First, you need to ‘docker-compose build’ and then to ‘sls deploy’. If you want to have a single Lambda with GE, you need to create an ECR repository and to go through the instructions to push your docker image. After that, create an AWS Lambda image and select it.
GE Reports portal:
GE Report:
For Pandas Profiling, you can use this code:
def profile_data(event, runs):    file_path = event[0]    s3 = boto3.resource("s3")    bucket = s3.Bucket('***')    df = wr.s3.read_parquet(path='s3://***/')    profile = ProfileReport(df, title="Pandas Profiling Report",    minimal=True)    report = profile.to_html()    bucket.put_object(Key='***', ContentType='text/html')
We deploy GE and Pandas Profiling on the same Lambda, to cut costs and to simplify the pipeline.
Pandas Profiling Report:
Allure:
Pre: you need to map GE json results to Allure json resultsCreate a docker file and requirements txtCreate sh script and use to interact with Allure from Python script, because Allure needs Java under the hood
Pre: you need to map GE json results to Allure json results
Create a docker file and requirements txt
Create sh script and use to interact with Allure from Python script, because Allure needs Java under the hood
FOLDER=$1rm -r /tmp/resultrm -r /tmp/allure-reportaws s3 sync s3://***/allure/"$FOLDER"/result /tmp/resultallure-2.14.0/bin/allure generate /tmp/result --clean -o tmp/allure-reportaws s3 sync /tmp/allure-report s3://***/allure/"$FOLDER"/allure-report
4. Deploy it
Allure report:
We also have an integration between Gitlab issues, to make it easier to create issues after tests have been run.
Amazon Athena and Amazon DynamoDB:
To store data, use a pivot table, and to crawl it, use Amazon Athena.
To push data to storage, we can use Amazon DynamoDD (or Amazon SS).
Access to Amazon DynamoDBCreate a tableGo to Amazon Athena in the same region as Amazon DynamoDBConnect a new DataSource (i.e. select a created DynamoDB Table)Test the connection in Query Editor
Access to Amazon DynamoDB
Create a table
Go to Amazon Athena in the same region as Amazon DynamoDB
Connect a new DataSource (i.e. select a created DynamoDB Table)
Test the connection in Query Editor
QuickSight:
Go to Amazon QuicksightSelect Datasets, go to new datasetsSelect Amazon AthenaSelect your workgroup and a datasource name
Go to Amazon Quicksight
Select Datasets, go to new datasets
Select Amazon Athena
Select your workgroup and a datasource name
5. Create a data source
6. Run a new analysis
In the existing implementation, we use two types of charts
Line for bug trends — percentage of failed tests per selected source per date, which you can choose by filters
Line for bug trends — percentage of failed tests per selected source per date, which you can choose by filters
2. Bar for suite and sources status
3. Also a pivot table, where you can click on raw to open the reporting menu and review each data qa result
Slack bot sends a notification with links about every pipeline
This article demonstrates how to build and implement a data quality approach in your data pipelines on AWS using StepFunctions, Lambda, GE, Allure, and Pandas Profiling. It explains the what and the how of these tools and services, and provides a robust approach for using them to ensure data quality, simplify pipeline debugging, and reduce costs, without increasing the architectural complexity of the data pipelines.
Data Quality Assurance with Great Expectations and Kubeflow PipelinesOur pip package for map GE results to Allure results