Package: bdq
What is bdq?BDQ stands for Big Data Quality, a set of tools/function that help you every day assert quality of datasets you have processed or ingested. Library leverages power of spark, hence it processes your quality checks at scale offered by spark and databricks.Library over time evolved into DAG executor of arbitrary python/pyspark functions. This allows to scale execution with dependency tracking to next level.How to install?On databricks run%pip install bdq==x.y.z. Make sure you tag version number you are confortable with to ensure API stability.This package is currently in EXPERIMENTAL stage and newer releses might change APIs (function names, parameters, etc..).Supported spark/databricks versionsDevelopment and testing has been performed on 12.2 LTS databricks runtime. Databricks runtime 10.4 LTS should also work with exception ofSparkPipeline.step_spark_for_each_batchandSparkPipeline.spark_metricwhich require 12.2 LTS runtime to work.Verbose outputbdq utilizes pythonloggingmodule, hence for the best logging experience configure logger, example setup:importloggingimportsys# py4j is very chatty, no need to deal with it unless it's criticallogging.getLogger('py4j').setLevel(logging.CRITICAL)# run everything else on DEBUG by default# DEBUG prints a lot of usefull info, INFO is good in general use caseslogging.basicConfig(stream=sys.stderr,level=logging.DEBUG,format='%(asctime)s%(levelname)s%(threadName)s[%(name)s]%(message)s')ExamplesSee examples bellow for short listing of major functionalities. Seetestsfolder for detailed examples. Tests are meant to double as real use cases hence they serve as documentation of examples for now.Comparing dataframe schemasimportbdqfromdatetimeimportdatetimeimportpyspark.sql.functionsasFdf2=spark.createDataFrame([[1,1,"Grzegorz",datetime(2018,1,1),datetime(2018,1,1,12,34,56),26.9,123234234345,True],[3,1,"Mike",datetime(2019,1,1),datetime(2018,3,1,12,34,56),46.7,5667888989,False],[2,2,"Timmy",datetime(2018,1,1),datetime(2018,2,1,12,34,56),36.7,8754857845,True]],schema)df2_changed=df2\.drop('first_login_dt')\.withColumn('new_data',F.lit(None).cast('date'))\.withColumn('likes',F.col('likes').cast('int'))assertbdq.compare_schemas(df2.schema,df2_changed.schema)=={'added':{'first_login_dt'},'removed':{'new_data'},'changed':{'likes':{'before':'bigint','after':'int'}},'not_changed':{'name','id1','last_login_ts','id2','credits','active'}}Comparing dataframe's dataimportbdqdf1=spark.createDataFrame([[1,1,"Grzegorz",datetime(2017,1,1),datetime(2018,1,1,12,34,56),26.7,123234234345,True],[2,1,"Tim",datetime(2018,1,1),datetime(2018,2,1,12,34,56),36.7,54545,True],[3,1,"Mike",datetime(2019,1,1),datetime(2018,3,1,12,34,56),46.7,5667888989,False]],schema)df2=spark.createDataFrame([[1,1,"Grzegorz",datetime(2018,1,1),datetime(2018,1,1,12,34,56),26.9,123234234345,True],[3,1,"Mike",datetime(2019,1,1),datetime(2018,3,1,12,34,56),46.7,5667888989,False],[2,2,"Timmy",datetime(2018,1,1),datetime(2018,2,1,12,34,56),36.7,8754857845,True]],schema)#returns dicts with added, removed, changed and not changed dataframes, and record countsdf_diff=bdq.compare_dataframes(df1,df2,['id1','id2'],True)#show all info about compare resultsbdq.display_compare_dataframes_results(df_diff)>>Addedrecordscount:1>>+---+---+-----+--------------+-------------------+-------+----------+------+>>|id1|id2|name|first_login_dt|last_login_ts|credits|likes|active|>>+---+---+-----+--------------+-------------------+-------+----------+------+>>|2|2|Timmy|2018-01-01|2018-02-0112:34:56|36.7|8754857845|true|>>+---+---+-----+--------------+-------------------+-------+----------+------+>>>>Removedrecordscount:1>>+---+---+----+--------------+-------------------+-------+-----+------+>>|id1|id2|name|first_login_dt|last_login_ts|credits|likes|active|>>+---+---+----+--------------+-------------------+-------+-----+------+>>|2|1|Tim|2018-01-01|2018-02-0112:34:56|36.7|54545|true|>>+---+---+----+--------------+-------------------+-------+-----+------+>>>>Changedrecordscount:1>>+---+---+---------------------------------------------------------------------+>>|id1|id2|changed|>>+---+---+---------------------------------------------------------------------+>>|1|1|{first_login_dt->{2017-01-01,2018-01-01},credits->{26.7,26.9}}|>>+---+---+---------------------------------------------------------------------+Notchangedrecordscount:1Surrogate key generationwith support of optional uppercasing and trimming of key columnsfromdatetimeimportdatetimefrombdq.functionsimportsurrogate_key_hash,surrogate_key_stringschema="id1:long,id2:long,name:string,likes:int"df1=spark.createDataFrame([[1,1,"GrzeGorz",1],[1,1,"Grzegorz",1],[1,1,"Grzegorz      ",1],[1,None,"Grzegorz",1],[1,None,None,1],[2,1,"Tim",1],[3,1,"Mike",1]],schema)sk_df=df1.select('*',surrogate_key_hash(['id1','id2','name'],rtrim=True).alias('hash'),surrogate_key_string(['id1','id2','name'],rtrim=True).alias('hash_input'))>>+---+----+--------------+-----+-------------------------------------------------------------+---------------------------+>>|id1|id2|name|likes|hash|hash_input|>>+---+----+--------------+-----+-------------------------------------------------------------+---------------------------+>>|1|1|GrzeGorz|1|[6F2199994CF293562E7CC329F96A422F6D62EC4B]|[1,1,GRZEGORZ]|>>|1|1|Grzegorz|1|[6F2199994CF293562E7CC329F96A422F6D62EC4B]|[1,1,GRZEGORZ]|>>|1|1|Grzegorz|1|[6F2199994CF293562E7CC329F96A422F6D62EC4B]|[1,1,GRZEGORZ]|>>|1|null|Grzegorz|1|[4E9E3A2BC83790A06A269AFE7A788BCA9934681C]|[1,@~<null>~@,GRZEGORZ]|>>|1|null|null|1|[44FF882E2A283BEEEE53C87D0F92D1683EA2F2E5]|[1,@~<null>~@,@~<null>~@]|>>|2|1|Tim|1|[A87AB029949F5CD178D428898BF175CDAC70B661]|[2,1,TIM]|>>|3|1|Mike|1|[9D6C76C517D04446471A71ED4A0875841D5C4509]|[3,1,MIKE]|>>+---+----+--------------+-----+-------------------------------------------------------------+---------------------------+PK & FK intergrity checkswith support of showing sample data from fact table that did not satisfy integrityfrombdqimportfact_dim_broken_relationshipfrombdq.functionsimportsurrogate_key_hash,surrogate_key_stringfact_df=spark.createDataFrame([["Grzegorz","IT","EU"],["Mark","IT","EU"],["Justin","IT  ","EU    "],["Alice1","IT","USA"],["Alice2","IT","USA"],["Alice3","IT","USA"],["Alice4","IT","USA"],["Alice5","IT","USA"],["Bob","Sales","USA"],["Bob2","Sales","USA"],["Bob3","Sales","USA"]],"Name:string,Dept:string,Country:string")\.withColumn("dept_fk",surrogate_key_hash(["Dept","Country"],rtrim=True))dim_df=spark.createDataFrame([["IT","EU","European IT department"],["Sales","USA","USA Sales dept."]],"department:string,cntry:string,comment:string")\.withColumn("dept_pk",surrogate_key_hash(["department","cntry"],rtrim=True))#direct column comparison, hence upper, lower, extra spaces affect the resultsbroken_df=fact_dim_broken_relationship(fact_df=fact_df,fk_columns=["Dept","Country"],dim_df=dim_df,pk_columns=["department","cntry"],sample_broken_records=2)>>+----+-------+------------------------------------------------------------------------------------+>>|Dept|Country|sample_records|>>+----+-------+------------------------------------------------------------------------------------+>>|IT|USA|[{IT,USA,Alice1,��xz4q_��\ra�-��Vz},{IT,USA,Alice2,��xz4q_��\ra�-��Vz}]|>>|IT|EU|[{IT,EU,Justin,��H'V4�\rY,g�I�����}]                                      |>>+----+-------+------------------------------------------------------------------------------------+# using surrogate keys will yield better results, because extra spaces are trimmedbroken_sk_df=fact_dim_broken_relationship(fact_df,["dept_fk"],dim_df,["dept_pk"],sample_broken_records=2)>>+----+-------+------------------------------------------------------------------------------------+>>|Dept|Country|sample_records|>>+----+-------+------------------------------------------------------------------------------------+>>|IT|USA|[{IT,USA,Alice1,��xz4q_��\ra�-��Vz},{IT,USA,Alice2,��xz4q_��\ra�-��Vz}]|>>+----+-------+------------------------------------------------------------------------------------+Get latest recordswith optional primary key conflict resolution, where there are multiple records being candidate for latest record, but they have different attribuets and there is no way of determining which one is the latest.fromdatetimeimportdatetimeasdtfrombdqimportget_latest_records,get_latest_records_with_pk_confict_detection_flagincrement_data=[(1,dt(2000,1,1,0,0,0),"1001"),(1,dt(2000,1,1,2,0,0),"1002"),(2,dt(2000,1,1,0,0,0),"2001")#carbon copy duplicate,(2,dt(2000,1,1,0,0,0),"2001")#carbon copy duplicate,(3,dt(2000,1,1,0,0,0),"3001"),(3,dt(2000,1,1,2,0,0),"3002#1")#pk conflict, two entries with the same technical fields, but different atributes, which one to chose?,(3,dt(2000,1,1,2,0,0),"3002#2")#pk conflict]df=spark.createDataFrame(increment_data,"pk:int,change_ts:timestamp,attr:string")df.show(truncate=True)>>+---+-------------------+------+>>|pk|change_ts|attr|>>+---+-------------------+------+>>|1|2000-01-0100:00:00|1001|>>|1|2000-01-0102:00:00|1002|>>|2|2000-01-0100:00:00|2001|>>|2|2000-01-0100:00:00|2001|>>|3|2000-01-0100:00:00|3001|>>|3|2000-01-0102:00:00|3002#1|>>|3|2000-01-0102:00:00|3002#2|>>+---+-------------------+------+primary_key_columns=['pk']order_by_columns=['change_ts']# will randomly chose one pk in case of conflict, without any notification about conflictssimple_get_latest_df=get_latest_records(df,primary_key_columns,order_by_columns)simple_get_latest_df.show(truncate=True)>>+---+-------------------+------+>>|pk|change_ts|attr|>>+---+-------------------+------+>>|1|2000-01-0102:00:00|1002|>>|2|2000-01-0100:00:00|2001|>>|3|2000-01-0102:00:00|3002#1|>>+---+-------------------+------+# will flag records which cause conflicts and cannot be resolved# bad records need to be quarantined and handled in special processconflict_get_latest_df=get_latest_records_with_pk_confict_detection_flag(df,primary_key_columns,order_by_columns)conflict_get_latest_df.show(truncate=True)>>+---+-------------------+------+-----------------+>>|pk|change_ts|attr|__has_pk_conflict|>>+---+-------------------+------+-----------------+>>|1|2000-01-0102:00:00|1002|false|>>|2|2000-01-0100:00:00|2001|false|>>|3|2000-01-0102:00:00|3002#2|             true|>>|3|2000-01-0102:00:00|3002#1|             true|>>+---+-------------------+------+-----------------+Find composite primary key candidatesGiven list of possible columns, constructs the lists of all possible combinations of composite primary keys, and executes concurrently to determine if given set of columns is a valid primary key. Uses minimum possible amount of queries against spark by skipping validation paths that are based on already validated primary key combinations.importbdqimportpyspark.sql.functionsasFdf=spark.range(0,100)\.withColumn('type',F.expr('(id / 10)::int + 1'))\.withColumn('reminder',F.expr('id % 10'))\.withColumn('static',F.lit('A'))\.withColumn('round_robin',F.expr('id % 2'))display(df)all_combinations=list(bdq.get_column_names_combinations(df.columns))bdq.validate_primary_key_candidate_combinations(df,all_combinations,max_workers=10,verbose=True)>>[('id',),('type','reminder')]Execute DAG having nodes as python functionsimportbdqimporttimeimportpytest#create graphgraph=bdq.DAG()#define nodes@graph.node()defa():time.sleep(2)return5@graph.node()defb():time.sleep(3)return"beeep"#nodes can depend on other nodes@graph.node(depends_on=[b])defc():time.sleep(5)return8#any amount of dependencies is allowed@graph.node(depends_on=[b,c,a])defd():time.sleep(7)#beep as many times as absolute difference between 'c' and 'a'return"g man say: "+b.result*abs(c.result-a.result)@graph.node(depends_on=[a])defe():time.sleep(3)#you can refer to parent nodes, to get information about their resultsraiseValueError(f"omg, crash!{a.result}")@graph.node(depends_on=[e])deff():print("this will never execute")return"this will never execute"@graph.node(depends_on=[a])defg():time.sleep(3)#we do not want to execute children nodes anymore#return graph.BREAK this will cause that all children/successors in graph will be skipped without errrorreturngraph.BREAK@graph.node(depends_on=[g])defi():print("this will never execute too")return"this will never execute too"# if you have ipydagred3 installed, you can see in real time state changes of each of the nodesdisplay(graph.visualize())#execute DAGgraph.execute(max_workers=10)#iterate over resultsprint("Iterate over results...")fornodeingraph.nodes:print(node)>>Waitingforalltaskstofinish...>>starting:Node(<functionaat0x7f50f4b9ee50>:{'state':'RUNNING','result':None,'exception':None,'completed':False})>>starting:Node(<functionbat0x7f50f4b53040>:{'state':'RUNNING','result':None,'exception':None,'completed':False})>>starting:Node(<functioneat0x7f50f4b53b80>:{'state':'RUNNING','result':None,'exception':None,'completed':False})>>starting:Node(<functiongat0x7f50f4b535e0>:{'state':'RUNNING','result':None,'exception':None,'completed':False})>>finished:Node(<functionaat0x7f50f4b9ee50>:{'state':'SUCCESS','result':5,'exception':None,'completed':True})(stillrunning:3)>>starting:Node(<functioncat0x7f50f4b534c0>:{'state':'RUNNING','result':None,'exception':None,'completed':False})>>finished:Node(<functionbat0x7f50f4b53040>:{'state':'SUCCESS','result':'beeep','exception':None,'completed':True})(stillrunning:3)>>error:Node(<functioneat0x7f50f4b53b80>:{'state':'ERROR','result':None,'exception':ValueError('omg, crash! 5'),'completed':True}):omg,crash!5(stillrunning:2)>>finished:Node(<functiongat0x7f50f4b535e0>:{'state':'SKIPPED','result':<threading.Eventobjectat0x7f50ec16c9d0>,'exception':None,'completed':True})(stillrunning:1)>>starting:Node(<functiondat0x7f50f4b53a60>:{'state':'RUNNING','result':None,'exception':None,'completed':False})>>finished:Node(<functioncat0x7f50f4b534c0>:{'state':'SUCCESS','result':8,'exception':None,'completed':True})(stillrunning:1)>>finished:Node(<functiondat0x7f50f4b53a60>:{'state':'SUCCESS','result':'g man say: beeepbeeepbeeep','exception':None,'completed':True})(stillrunning:0)>>Alltasksfinished,shuttingdown>>>>Iterateoverresults...>>Node(<functionaat0x7f50f4b9ee50>:{'state':'SUCCESS','result':5,'exception':None,'completed':True})>>Node(<functionbat0x7f50f4b53040>:{'state':'SUCCESS','result':'beeep','exception':None,'completed':True})>>Node(<functioncat0x7f50f4b534c0>:{'state':'SUCCESS','result':8,'exception':None,'completed':True})>>Node(<functiondat0x7f50f4b53a60>:{'state':'SUCCESS','result':'g man say: beeepbeeepbeeep','exception':None,'completed':True})>>Node(<functioneat0x7f50f4b53b80>:{'state':'ERROR','result':None,'exception':ValueError('omg, crash! 5'),'completed':True})>>Node(<functionfat0x7f50f4b530d0>:{'state':'SKIPPED','result':None,'exception':None,'completed':False})>>Node(<functiongat0x7f50f4b535e0>:{'state':'SKIPPED','result':<threading.Eventobjectat0x7f50ec16c9d0>,'exception':None,'completed':True})>>Node(<functioniat0x7f50f4b53280>:{'state':'SKIPPED','result':None,'exception':None,'completed':False})Execute spark pipeline of multiple stepsusing DAG component to handle parallel executionimportbdqfrombdqimportspark,tableppn=bdq.SparkPipeline("sample")# returns dataframe, and creates spark view 'raw_data_single_source'@ppn.step_spark_temp_view()defraw_data_single_source(step):returnspark.range(1,10)# returns dataframe, and creates spark view 'raw_nice_name'@ppn.step_spark_temp_view(outputs="raw_nice_name")defraw_data_single_source_with_custom_name(step):returnspark.range(100,110)# returns two dataframes, and creates two spark views 'raw_data1', 'raw_data2'@ppn.step_spark_temp_view(outputs=["raw_data1","raw_data2"])defraw_data_multi_source(step):df1=spark.range(1000,2000)df2=spark.range(2000,3000)return[df1,df2]# waits for raw data sources to finish, and combines the data into one unioned view `combine_data`# note that dependencies are python functions or names of outputs@ppn.step_spark_temp_view(depends_on=[raw_data_single_source,raw_data_single_source_with_custom_name,'raw_data1','raw_data2'])defcombine_data(step):df=table('raw_data_single_source')\.union(table('raw_nice_name'))\.union(table('raw_data1'))\.union(table('raw_data2'))returndf# splits the combined_data into 'odd' and 'even' views@ppn.step_spark_temp_view(depends_on=combine_data,outputs=['odd','even'])defsplit_data(step):df_odd=table('combine_data').filter('id % 2 == 1')df_even=table('combine_data').filter('id % 2 == 0')return[df_odd,df_even]# executes pipeline using concurrent threads, one per each step, following the dependency DAG# pipeline is a normal python callable object, as if it was a function, it returns a list of all stepspipeline_results=ppn(max_concurrent_steps=10)print('pipeline results:')print(pipeline_results)#show some final valuesprint('even numbers:')print(table('even').limit(10).collect())print('odd numbers:')print(table('odd').limit(10).collect())#get skipped stepsassertlist(ppn.skipped_steps)==[]#get errored steps (you would need to 'adjust' code of on of the steps to make it fail to see something here)assertlist(ppn.error_steps)==[]#get successfull stepsassertset(ppn.success_steps.values())==set([raw_data_single_source_with_custom_name,raw_data_multi_source,split_data,combine_data,raw_data_single_source])>>Waitingforalltaskstofinish...>>starting:Node(raw_data_single_source:{'state':'RUNNING','result':None,'exception':None,'completed':False})>>starting:Node(raw_data_single_source_with_custom_name:{'state':'RUNNING','result':None,'exception':None,'completed':False})>>starting:Node(raw_data_multi_source:{'state':'RUNNING','result':None,'exception':None,'completed':False})>>finished:Node(raw_data_single_source:{'state':'SUCCESS','result':[DataFrame[id:bigint]],'exception':None,'completed':True})(stillrunning:2)>>finished:Node(raw_data_single_source_with_custom_name:{'state':'SUCCESS','result':[DataFrame[id:bigint]],'exception':None,'completed':True})(stillrunning:1)>>starting:Node(combine_data:{'state':'RUNNING','result':None,'exception':None,'completed':False})>>finished:Node(raw_data_multi_source:{'state':'SUCCESS','result':[DataFrame[id:bigint],DataFrame[id:bigint]],'exception':None,'completed':True})(stillrunning:1)>>starting:Node(split_data:{'state':'RUNNING','result':None,'exception':None,'completed':False})>>finished:Node(combine_data:{'state':'SUCCESS','result':[DataFrame[id:bigint]],'exception':None,'completed':True})(stillrunning:1)>>finished:Node(split_data:{'state':'SUCCESS','result':[DataFrame[id:bigint],DataFrame[id:bigint]],'exception':None,'completed':True})(stillrunning:0)>>Alltasksfinished,shuttingdown>>pipelineresults:>>[raw_data_single_source,raw_data_single_source_with_custom_name,raw_data_multi_source,combine_data,split_data]>>evennumbers:>>[Row(id=2),Row(id=4),Row(id=6),Row(id=8),Row(id=100),Row(id=102),Row(id=104),Row(id=106),Row(id=108),Row(id=1000)]>>oddnumbers:>>[Row(id=1),Row(id=3),Row(id=5),Row(id=7),Row(id=9),Row(id=101),Row(id=103),Row(id=105),Row(id=107),Row(id=109)]pipeline steps are rerunable as any ordinary function:# to rerun given step, just execute it as if it was a pure function# return is alaways a list of dataframs that given @ppn.step returns# note: the spark view 'raw_data_single_source' will be updated when this function finishes (as per defintion in @ppn.step above)raw_data_single_source()Spark UI Stage descriptionsWhen running code using pyspark, spark ui gets very crowded.SparkUILoggercontext manager and decorator assings human readable names to spark stages.frombdqimportSparkUILogger# usage of decorators# spark ui stages will have description of 'xyz'@SparkUILogger.tag(desc='xyz')defsome_function2(number):returnspark.range(number).count()# spark ui stages will have description of 'alpha_function2' (automatically derived from function name)@SparkUILogger.tagdefalpha_function2(number):# 1st and 2nd some_function2() calls should be visible in spark ui as 'xyz'# the 3rd part of result should be visible as 'alpha_function2returnsome_function2(number*2)+some_function2(number*3)+spark.range(number/10).count()# usage of context managers# will be visible in spark ui as 'first-count'withSparkUILogger('first-count'):spark.range(10).count()# will be visible in spark ui as '2nd-count'withSparkUILogger('2nd-count'):spark.range(20).count()some_function2(1000)alpha_function2(2000)