Package: data-quality-check
Data-Quality-CheckRequirementsPython 3.7+Java 8+Apache Spark 3.0+UsageInstallationpipinstall--upgradedata-quality-check# Install Spark if neededpipinstallpysparkQuick Startfromdata_quality_check.configimportConfigfromdata_quality_check.profiler.combined_profilerimportCombinedProfilerfromdata_quality_check.report.renders.html.renderimportrender_allconfig_dict={'dataset':{'name':'mydb.my_table'},'profiling':{'general':{'columns':['*']}}}config=Config().parse_obj(config_dict)profiler=CombinedProfiler(spark,config=config)result=profiler.run()html=render_all(all_pr=result)# Present in Jupyter notebooksfromIPython.core.displayimportdisplay,HTMLdisplay(HTML(html))# Present in Databricks notebooksdisplayHTML(html)# Save to a html filef=open("report.html","w")f.write(html)f.close()If you do not have a ready-to-use spark session, use the codes below to create one:frompyspark.sqlimportSparkSessionspark=SparkSession.builder.appName("MyApp").enableHiveSupport().getOrCreate()DevelopmentDependenciesFilenameRequirementsrequirements.txtPackage requirementsrequirements-dev.txtRequirements for developmentTestPYTHONPATH=./srcpytesttests/*Buildpython setup.py sdist bdist_wheel && twine check dist/*Publishtwine upload --repository-url https://test.pypi.org/legacy/ dist/*
twine upload dist/*ManualProfiling CheckThere are 2 types of useful profilers :GeneralProfilerandCustomizedProfiler.
If you would like to run both profilers on your dataset. You can useCombinedProfilerwhich will run both profilers.Combined ProfilerThe easiest way to run a combined profiler(mix of general and customized profiler) on you dataset:Example of running combined profilingfromdata_quality_check.configimportConfigfromdata_quality_check.profiler.combined_profilerimportCombinedProfilerfromdata_quality_check.report.renders.html.renderimportrender_allconfig_dict={'dataset':{'name':'my_table'},'profiling':{'general':{'columns':['*']}},'customized':{'code_check':[{'column':'my_code_col','codes':['A','B','C','D']}]}}config=Config().parse_obj(config_dict)profiler=CombinedProfiler(spark,config=config)result=profiler.run()html=render_all(all_pr=result)displayHTML(html)General Profilerfrompyspark.sqlimportSparkSessionfromdata_quality_check.configimportConfigDatasetfromdata_quality_check.profiler.general_profilerimportGeneralProfilerspark=SparkSession.builder.appName("SparkProfilingApp").enableHiveSupport().getOrCreate()data=[{'name':'Alice','age':1,'gender':'female','is_new':True},{'name':'Tom','age':10,'gender':'male','is_new':False}]# Run general check on spark dfdf=spark.createDataFrame(data)result_df=GeneralProfiler(spark,df=df).run(return_type='dataframe')result_df.show()# Run general check on spark/hive tabledf.createOrReplaceTempView('my_table')result_df=GeneralProfiler(spark,dataset_config=ConfigDataset(name='my_table')).run(return_type='dataframe')result_df.show()Customized Profilerimportjsonfrompyspark.sqlimportSparkSessionfrompyspark.sql.typesimportStructType,StructField,StringType,LongTypefromdata_quality_check.configimportConfig,ConfigDataset,ConfigProfilingCustomizedfromdata_quality_check.profiler.customized_profilerimportCustomizedProfiler# Initialize sparkspark=SparkSession.builder.appName("SparkProfilingApp").enableHiveSupport().getOrCreate()dept=[("Finance",1),("Marketing",2),("Sales",3),("IT",4)]deptSchema=StructType([StructField('dept_name',StringType(),True),StructField('dept_id',LongType(),True)])spark.createDataFrame(data=dept,schema=deptSchema).createOrReplaceTempView('dept')print('dept table:')spark.table('dept').show(truncate=False)employee=[(1,"Amy",1,'male',1000,'amy@example.com'),(2,"Caro",2,'male',1000,'caro@example.com'),(3,"Mark",3,'Error',2000,'unknown'),(4,"Timi",4,'female',2000,None),(5,"Tata",5,'unknown',3000,'bad email address'),(6,"Zolo",None,None,3000,'my-C0omplicated_EMAIL@A.ddress.xyz')]employeeSchema=StructType([StructField('uid',LongType(),True),StructField('name',StringType(),True),StructField('dept_id',LongType(),True),StructField('gender',StringType(),True),StructField('income',LongType(),True),StructField('email',StringType(),True)])spark.createDataFrame(data=employee,schema=employeeSchema).createOrReplaceTempView('employee')print('employee table:')spark.table('employee').show(truncate=False)# Specify the configuration of customized profilercustomized_config_dict={'code_check':[{'column':'gender','codes':['male','female','unknown']}],'key_mapping_check':[{'column':'dept_id','target_table':'dept','target_column':'dept_id'}]}customized_config=ConfigProfilingCustomized.parse_obj(customized_config_dict)dataset_config=ConfigDataset.parse_obj({'name':'employee'})# Initialize CustomizedProfiler with configurationcustomized_profiler=CustomizedProfiler(spark,dataset_config=dataset_config,customized_profiling_config=customized_config)result=customized_profiler.run(return_type='dict')print(json.dumps(result,indent=' ',ensure_ascii=False,allow_nan=True))Expectation VerificationTo be done.Supported Checks and ExpectationProfiler TypeCheck TypeRender result as HTML?Support Expectation?DescriptionGeneralDistinct Values CountYESWill DONumber of unique values in a given column. This equals to Unique Row CountGeneralNull Row CountYESWill DONull row count in a given columnGeneralEmpty Row CountYESWill DOEmpty/Blank text row count in a given columnGeneralZero Row CountYESWill DO0-valued row count in a given columnGeneralValued Row CountYESWill DONumber of rows which are not null in a given columnGeneralTotal Row CountYESWill DONumber of total rowsGeneralUnique Row CountYESWill DONumber of rows that have unique valueGeneralDuplicated Valued Row CountYESWill DONumber of rows that have duplicated valuesGeneralMinimum ValueYESWill DOMinimum valueGeneralMaximum ValueYESWill DOMaximum valueGeneralMean ValueYESWill DOMean/average valueGeneralStandard Deviation ValueYESWill DOStandard deviation value of a columnGeneralValues CountYESWill DONumber of values in a given column---------------CustomizedCode CheckYESWill DOCheck if values from the columns are in the given(expected) codes listCustomizedKey Mapping CheckWill DOWill DOFind the values from this table column that do not exist in the target(another) table's column. Hint: target table usually is dim tableWill DO= Is scheduled to be developed, but not implemented yet.Expectation TypeScopeDescriptionExpectColumnToExist------.........