Package: dq-module
dq-moduledq-module is a tool which can be used to perform validations and profiling on the datasets.This tool is compatible with two run_enginespysparkandpolars.Features1. Data ValidationThis library contains aSingleDatasetQualityCheck()class which can used to validate the dataset against a defined set of rules.
This class contains the following rules which can be used to validate the dataset:null_checkschema_checkrange_min_checkrange_max_checklength_checkunique_records_checkunique_keys_checkallowed_values_checkmin_length_checkcolumn_count_checknot_allowed_values_checkdate_range_checkregex_checkrow_count_checkHow To Useimportdataqualitycheckasdq
fromdatetimeimportdate
importtimeStep 1: Adding a datasourceWe have 4 classes which can be used to connect to a datasource:AzureBlobDF()- This class can be used to interact with the datasources on azure blob storage .DatabricksFileSystemDF()- This class can be used to interect with the datasources on  databricks filesystem.DatabricksSqlDF()- This class can be used to interact with the datasources on  databricks databases.LocalFileSystemDF()- This class can be used to interact with the datasources on your local filesystem.Each of the above class provides the functionalities to read and write from the respective datasources.Example:Pass the configuration of the  blob connector  inblob_connector_configand
add a datasource by defining adata_read_obanddata_write_ob.blob_connector_config={"storage_account_name":"rgmdemostorage","container_name":"cooler-images","sas_token":<vaild-sas-token>}data_read_ob=dq.AzureBlobDF(storage_name=blob_connector_config["storage_account_name"],sas_token=blob_connector_config["sas_token"])data_write_ob=dq.DatabricksFileSystemDF()tables_listis a dictionary that contains the list of sources along with the container_name , source_type , layer , source_name , filename , read_connector_method and latest_file_path  for the tables on which the validations has to be applied .#This is optional.It is required when you are calling individual rules.tables_list={}Step 2: Add a DataContextInstantiate a DataContext by passingtables_list,interaction_between_tables,data_read_ob,data_write_ob,data_right_structure,job_id,time_zone,no_of_partitionandoutput_db_name.
You can also pass therun_enginewith which you want to apply the quality checks. By default, the run_engine ispyspark.dq_ob=dq.SingleDatasetQualityCheck(tables_list={},interaction_between_tables=[],data_read_ob=data_read_ob,data_write_ob=data_write_ob,data_right_structure='file',job_id=<pipeline_run_id>,time_zone=None,output_db_name=<database_name_where_report_has_to_be_written>,no_of_partition=4)Step 3:Passing arules_diagnosys_summery_file_pathandconfig_dfas an input and apply validations on various columns of respective table defined in theconfig_df.Here is an sample of theconfig_df.container_namesource_typelayersource_namefilenamerule_namecolumn_to_be_checkedvaluedate_column_configdate_format_dictionaryruletypeactiveread_connector_methodlatest_file_pathoutput_folder_structurefailed_schema_source_listrgm-data-qualitysharepointprocessedneilsen_data_folderneilsen_datanull_checkprod_brandnullnullnullMandatory1blobpath-to-fileprocessed/rgm-data-qualitysharepointprocessedneilsen_data_folderneilsen_datarange_min_checkinventory_kg_avg10nullnullNot Mandatory1blobpath-to-fileprocessed/rgm-data-qualitysharepointprocessedneilsen_data_folderneilsen_datarange_max_checkinventory_kg_avg1000nullnullNot Mandatory1blobpath-to-fileprocessed/Example:rules_diagnosys_summery_folder_path=<folder-path-for-the-output-report>config_df=spark.read.option("header",True).csv(<path-to-the-config>)dq_ob.apply_validation(config_df,write_summary_on_database=True,failed_schema_source_list=[],output_summary_folder_path=rules_diagnosys_summery_folder_path)2. Data ProfilingWe can generate a detailed summary statistics such as mean, median, mode, list of uniques, missing count etc. of a dataset using theDataProfile()class.This class can also be used to recommend some  data quality rules  based on the profiling report generated on the dataset.How To UseStep 1: Add a Datasourcedata_write_ob=dq.DatabricksSqlDF()data_write_ob=dq.DatabricksSqlDF()Step 2: Add a DataContextimportpytztime_zone=pytz.timezone('US/Central')dq_ob=dq.DataProfile(tables_list=tables_list,interaction_between_tables=[],data_read_ob=data_read_ob,data_write_ob=data_write_ob,data_right_structure='table',job_id=<pipeline_run_id>,time_zone=time_zone,no_of_partition=4,output_db_name=<database_name_where_report_has_to_be_written>,run_engine='polars')Step 3:Passconfig_dfas an input and apply data profiling on various columns of respective table defined in theconfig_df.# You can create a config_df in pyspark/polars run_engine directly also rather than reading as a csv.config_df=spark.createDataFrame([{"container_name":"rgm-data-quality","source_type":"sharepoint","layer":"raw","source_name":"data_quality_db","filename":"neilsen_data_parquet","latest_file_path":"data_quality_db.neilsen_data_parquet","read_connector_method":"databricks sql","output_folder_structure":"processed/data_profiling_test/"}])# Generating a data profiling report.dq_ob.apply_data_profiling(source_config_df=config_df,write_consolidated_report=True)# Generating a data profiling report as well as recommending the quality rules based on the profiling report.rules_config=dq_ob.data_profiling_based_quality_rules(config_df,list_of_columns_to_be_ignored)3. Consistency CheckYou can check the consistency of common columns between two tables using theConsistencyCheck()class.How To UseStep 1: Add a datasource.data_read_ob=dq.DatabricksFileSystemDF()data_write_ob=dq.AzureBlobDF(storage_name=blob_connector_config["storage_account_name"],sas_token=blob_connector_config["sas_token"])Step 2: Add a DataContextdq_ob=dq.ConsistencyCheck(tables_list={},interaction_between_tables=[],data_read_ob=data_read_ob,data_write_ob=data_write_ob,data_right_structure='file',job_id=<pipeline_run_id>,time_zone=None,no_of_partition=4,output_db_name=<database_name_where_report_has_to_be_written>)Step 3:Passconfig_dfandoutput_report_folder_pathas an input and apply consistency check.
Here is a sample consistency check config.container_namebase_table_source_typebase_table_layerbase_table_source_namebase_table_filenamebase_table_col_namebase_table_file_pathmapped_table_source_typemapped_table_layermapped_table_source_namemapped_table_filenamemapped_table_col_namemapped_table_file_pathread_connector_methodoutput_folder_structurergm-data-qualitysharepointprocessedscantrackbutters_margarins_baseBARCODEdbfs:/FileStore/Tables/sharepoint/scantrack/butters_margarins_base/2023/01/12/butters_margarins_base_01_12_2023.csvsharepointprocessedscantrackbutters_margarins_masterBARCODEdbfs:/FileStore/Tables/sharepoint/scantrack/butters_margarins_master/2023/01/12/butters_margarins_master_01_12_2023.csvdbfsprocessed/data_consistenct_test/rgm-data-qualitysharepointprocessedscantrackcheese_baseBARCODEdbfs:/FileStore/Tables/sharepoint/scantrack/cheese_base/2023/01/12/cheese_base_01_12_2023.csvsharepointprocessedscantrackcheese_masterBARCODEdbfs:/FileStore/Tables/sharepoint/scantrack/cheese_master/2023/01/12/cheese_master_01_12_2023.csvdbfsprocessed/data_consistenct_test/config_df=spark.read.option("header",True).csv(<path-to-the-consistency-check-config>)output_report_folder_path=<folder-path-for-the-output-report>

dq_ob.apply_consistency_check(config_df,output_report_folder_path)