Package: data-kalite
Data-KaliteKalite means Quality in Turkish.This is a framework on top of open source Great Expectations PySpark Suite with additional features.About Great Expectations:It supports Spark 3.0It has built in Metrics and Validations Suite that provide information[Optional] Can be hooked up with the GE HTML static pagesOn Top of GE:Ability to return a standardized DataFrame for both Metrics and Validations that can be pushed to DWH for historical analysisRuns all validations first before failing the pipelineReturns a Validation generated standardized DataFrameFailing the pipeline when one of the validation has failedLogs validation through loggerSupport to add metadata column for tracking purposes (dag_id etc)Support to pass optional pipeline args (batch_date etc)Deriving validations through configurations (YAML configs or Dict)Easy to run default metricsAbility to use lower level api for metrics and validationsUsageRequires:3.7.0 < Python < 3.11.0Few ways:Download from Pypipip install data-kaliteClone and integrate in your codebaseClone and build your python package:pip install .for local installpython setup.py bdist_wheelfor distributionMetricsHow to run metrics:Read:LimitationsTo run default metrics;fromkalite.application.default_metricsimportDefaultMetricssource_data="<set_dataframe>"metadata={"dag_id":"<set>","dag_task_id":"<set>","dag_run_id":"<set>","pipeline_args":{"<set>":"<set>"},# OPTIONAL}# run() returns a DataFrame# set unique_count_threshold depending on data size: Details [here](https://github.com/MJFND/data-kalite/blob/main/kalite/functions/metrics.py#L27-L29)metrics=DefaultMetrics(source_data=source_data,metadata=metadata,unique_count_threshold=5).run()metrics.show(10,False)Implementing Metrics:fromtypingimportListfromkalite.data_classes.metrics_dataimportMetricsDatafromkalite.functions.metricsimportGeMetrics,MetricsclassTempM(Metrics):defmetrics(self)->List[MetricsData]:returnGeMetrics(self.source_data).\get_column_unique_count("column_A").\get_column_max("column_B").\result()source_data="<set_dataframe>"metadata={"dag_id":"<set>","dag_task_id":"<set>","dag_run_id":"<set>","pipeline_args":{"<set>":"<set>"},# OPTIONAL}# run() returns a DataFrame# set unique_count_threshold depending on data size: Details [here](https://github.com/MJFND/data-kalite/blob/main/kalite/functions/metrics.py#L27-L29)metrics=TempM(source_data=source_data,metadata=metadata,unique_count_threshold=2).run()metrics.show(10,False)Refer to unit test cases on usage.Metrics sample result:idcreated_atpipeline_argsdag_iddag_run_iddag_task_idmetricsresultscolumn_namecolumn_value53d924311faca5490a94d6f2d51ae7d62023-05-29 23:12:35.351{id -> 123, flag -> True}mydag123456mytaskget_column_unique_count0.0event_datenull9c209f282a8a91f4a2c07b59406e68cf2023-05-29 23:12:35.351{id -> 123, flag -> True}mydag123456mytaskget_column_value_count10.0sourceunknown5f57c0104e1fab302b134bac84b7c2d22023-05-29 23:12:35.351{id -> 123, flag -> True}mydag123456mytaskget_column_median10.0device_os_versionnullafc8be1f1a2fc296f2133ebc1cca45f72023-05-29 23:12:35.351{id -> 123, flag -> True}mydag123456mytaskget_rate0.9device_os_nameAndroid3d10cc1b98a617e54a8af6eed33677d42023-05-29 23:12:35.351{id -> 123, flag -> True}mydag123456mytaskget_column_unique_count0.0typenullValidationsHow to run validations:Read:LimitationsTo run validations,validationsfunction must be implemented.fromtypingimportListfromkalite.data_classes.validations_dataimportValidationsDatafromkalite.functions.validationsimportGeValidations,ValidationsclassTempV(Validations):defvalidations(self)->List[ValidationsData]:returnGeValidations(self.source_data).\expect_column_to_exist("column_A").\expect_column_values_to_be_of_type("column_A","StringType").\expect_column_value_to_exist("column_A","XYZ").\expect_column_values_to_not_have_one_unique_count("column_A").\result()# source_data = "<set_dataframe>"metadata={"dag_id":"<set>","dag_task_id":"<set>","dag_run_id":"<set>","pipeline_args":{"<set>":"<set>"},# OPTIONAL}# run() returns a DataFramevalidations=TempV(source_data=source_data,metadata=metadata).run()validations.show(10,False)Refer to unit test cases on usage.Config Driven ValidationsA simple validator service on top of validations that allows to perform validations via a config dynamically.
A config should be in a standardDictformat as shown underconfig_driven_validations.py.Calling the function is easy, just pass config param as Dict, could be derived from a YAML or JSON. Benefits of decoupling from code is you can keep these validations in a centralized Data Contract repository.YAML Config:validations:<column_name>:-validation_1-validation_2..-validation_nRefer to unit test resource for full example:linkFunction require config to be dictionary, it can be sourced from YAML or JSON as long as structure is followed.fromkalite.application.config_driven_validationsimportConfigDrivenValidationssource_data="<set_dataframe>"metadata={"dag_id":"<set>","dag_task_id":"<set>","dag_run_id":"<set>","pipeline_args":{"<set>":"<set>"},# OPTIONAL}config="<set_dict>"# Details: https://github.com/MJFND/data-kalite/blob/main/kalite/application/config_driven_validations.py#L20-L28# run() returns a DataFramevalidations=ConfigDrivenValidations(source_data=source_data,metadata=metadata,config=config).run()validations.show(10,False)Refer to unit test cases on usage.Validation sample result:idcreated_atpipeline_argsdag_iddag_run_iddag_task_idvalidationsresultscolumn_namege_metadata40a2037a2298c8a86aa9c5c92c3a9db92023-05-29 16:20:32.975{id -> 123, flag -> True}mydag123456mytaskexpect_column_to_existPASSdecision{"meta": {}, "result": {}, "success": true, "expectation_config": {"meta": {}, "kwargs": {"column": "decision", "result_format": "BASIC"}, "expectation_type": "expect_column_to_exist"}, "exception_info": {"raised_exception": false, "exception_traceback": null, "exception_message": null}}f5af2ed95a5ac827cf7d5020e44fee6e2023-05-29 16:20:32.975{id -> 123, flag -> True}mydag123456mytaskexpect_column_values_to_be_of_typePASSdecision{"meta": {}, "result": {"observed_value": "StringType"}, "success": true, "expectation_config": {"meta": {}, "kwargs": {"column": "decision", "type_": "StringType", "result_format": "BASIC"}, "expectation_type": "expect_column_values_to_be_of_type"}, "exception_info": {"raised_exception": false, "exception_traceback": null, "exception_message": null}}Failing the TaskAfter generating the DataFrame of validations,Validatorcan be used to throw exception to fail pipeline.fromkalite.functions.validatorimportValidatorvalidation_data="<set_dataframe>"# dataframe that was generated through ValidationsValidator.validate(validation_data)LimitationsData-Kalite does not supportDATEorTIMESTAMPdatatype input columns for certain metrics that requirescolumn_valueto be set.E.g. Metricget_column_values_count, See metricsherecolumn_valueis casted asSTRINGduringDataFramegeneration, casting can fail if input types areDATEorTIMESTAMP.Since Spark column can be of one type, workaround is to explicitly convert these intoSTRINGbefore running metrics or avoid running these checks for those types of columns.ContributingSetup Local EnvironmentUsingmakefileto keep things simple.make installtest via:make pre-commitgit commitTo learn more aboutpre-committools check the.pre-commit-config.yamlin root.MetricsHow to add new metrics:All metrics must start withget_All metrics must be wrapped in a@Metrics.standardizefunction.All metrics must return aMetricsSuiteResultMost of the metrics are abstraction on top of GE as mentionedhere.Refer to existing function inmetrics.pyValidationsHow to add new validations:All expectations must start withexpect_All expectations must be wrapped in a@Validations.standardizefunction.All expectations must return aExpectationSuiteValidationResultMost of the validations are abstraction on top of GE as mentionedhere.Refer to existing function invalidations.py