Package: dlt-with-debug
DLT with DebugRunning DLT workflows from interactive notebooks.About The ProjectDelta Live Tables (DLTs) are a great way to design data pipelines with only focusing on the core business logic.
It makes the life of data engineers easy but while the development workflows are streamlined in DLT, when it comes todebugging and seeing how the data looks after each transformation stepin a typical DLT pipeline it becomes very
painful and cumbersome as we dont have the DLT package available in our interactive environment.Enterdlt-with-debuga lightweight decorator utility which allows developers to do interactive
pipeline development by having a unified source code for both DLT run and Non-DLT interactive notebook run.(back to top)Built WithPython's builtinsglobals()exec()decorator()(back to top)Installationpip install in your Databricks NotebookPyPI%pipinstalldlt-with-debug(back to top)PrerequisitesDatabricksDelta Live Tables(back to top)UsageIn our notebooks containing DLT Jobs the imports changes slightly as belowfrom dlt_with_debug import dltwithdebug, pipeline_id, showoutput

if pipeline_id:
  import dlt
else:
  from dlt_with_debug import dltNote:Use thedlt.create_table()API instead ofdlt.table()asdlt.table()sometimes gets mixed withspark.table()in the global namespace.Always pass theglobals()namespace todltwithdebugdecorator like this@dltwithdebug(globals())(back to top)SampleDLT with debugDLT pipeline exampleCode:Cmd 1%pipinstall-egit+https://github.com/souvik-databricks/dlt-with-debug.git#"egg=dlt_with_debug"Cmd 2frompyspark.sql.functionsimport*frompyspark.sql.typesimport*# We are importing# dltwithdebug as that's the entry point to interactive DLT workflows# pipeline_id to ensure we import the dlt package based on environment# showoutput is a helper function for seeing the output result along with expectation metrics if any is specifiedfromdlt_with_debugimportdltwithdebug,pipeline_id,showoutputifpipeline_id:importdltelse:fromdlt_with_debugimportdltCmd 3json_path="/databricks-datasets/wikipedia-datasets/data-001/clickstream/raw-uncompressed-json/2015_2_clickstream.json"Cmd 4# Notice we are using dlt.create_table instead of dlt.table@dlt.create_table(comment="The raw wikipedia click stream dataset, ingested from /databricks-datasets.",table_properties={"quality":"bronze"})@dltwithdebug(globals())defclickstream_raw():return(spark.read.option("inferSchema","true").json(json_path))Cmd 5# for displaying the result of the transformation# use showoutput(func_name)# for example here we are using showoutput(clickstream_raw)showoutput(clickstream_raw)Cmd 6@dlt.create_table(comment="Wikipedia clickstream dataset with cleaned-up datatypes / column names and quality expectations.",table_properties={"quality":"silver"})@dlt.expect("valid_current_page","current_page_id IS NOT NULL AND current_page_title IS NOT NULL")@dlt.expect_or_fail("valid_count","click_count > 0")@dlt.expect_all({'valid_prev_page_id':"previous_page_id IS NOT NULL"})@dltwithdebug(globals())defclickstream_clean():return(dlt.read("clickstream_raw").withColumn("current_page_id",expr("CAST(curr_id AS INT)")).withColumn("click_count",expr("CAST(n AS INT)")).withColumn("previous_page_id",expr("CAST(prev_id AS INT)")).withColumnRenamed("curr_title","current_page_title").withColumnRenamed("prev_title","previous_page_title").select("current_page_id","current_page_title","click_count","previous_page_id","previous_page_title"))Cmd 7showoutput(clickstream_clean)Important to note that here you can see we are alsoseeing how many records will the expectations affect.(back to top)Same sampleDLT with debugDLT pipeline executed as part of a delta live tableBelow we can the expectation results also match up with the expectation metrics that we got from dltwithdebug earlier
withshowoutput(clickstream_clean)(back to top)Quick API guideTable syntax@dlt.create_table(# <-- Notice we are using the dlt.create_table() instead of dlt.table()name="<name>",comment="<comment>",spark_conf={"<key>":"<value","<key":"<value>"},table_properties={"<key>":"<value>","<key>":"<value>"},path="<storage-location-path>",partition_cols=["<partition-column>","<partition-column>"],schema="schema-definition",temporary=False)@dlt.expect@dlt.expect_or_fail@dlt.expect_or_drop@dlt.expect_all@dlt.expect_all_or_drop@dlt.expect_all_or_fail@dltwithdebug(globals())# <-- This dltwithdebug(globals()) needs to be addeddef<function-name>():return(<query>)View syntax@dlt.create_view(# <-- Notice we are using the dlt.create_view() instead of dlt.view()name="<name>",comment="<comment>")@dlt.expect@dlt.expect_or_fail@dlt.expect_or_drop@dlt.expect_all@dlt.expect_all_or_drop@dlt.expect_all_or_fail@dltwithdebug(globals())# <-- This dltwithdebug(globals()) needs to be addeddef<function-name>():return(<query>)Getting results syntaxshowoutput(function_name)# <-- showoutput(function_name)# Notice we are only passing the function name# The name of the function which is wrapped by the dltdecorators# For example:# @dlt.create_table()# @dltwithdebug(globals())# def step_one():#    return spark.read.csv()# showoutput(step_one)Import syntax# We are importing# dltwithdebug as that's the entry point to interactive DLT workflows# pipeline_id to ensure we import the dlt package based on environment# showoutput is a helper function for seeing the output result along with expectation metrics if any is specifiedfromdlt_with_debugimportdltwithdebug,pipeline_id,showoutputifpipeline_id:importdltelse:fromdlt_with_debugimportdlt(back to top)Upcoming functionalityAs of now the following DLT APIs are covered for interactive use:Currently Available:dlt.readdlt.read_streamdlt.create_tabledlt.create_viewdlt.table<-- This one sometimes gets overridden withspark.tabledlt.viewdlt.expectdlt.expect_or_faildlt.expect_or_dropdlt.expect_alldlt.expect_all_or_dropdlt.expect_all_or_failWill be covered in the upcoming release:dlt.create_target_tabledlt.create_streaming_live_tabledlt.apply_changesLimitationDLT with Debugis a fully python based utility and as such it doesn't supportsspark.table("LIVE.func_name")syntax.So instead ofspark.table("LIVE.func_name")usedlt.read("func_name")ordlt.read_stream("func_name")LicenseDistributed under the MIT License.(back to top)