Package: dbxscd
Slowly Changing Dimensions implementation with Azure Databricks Delta LakeInstallationInstall dbxscd package in Azure Databricks notebook%pipinstalldbxscdRun ExampleCreate Azure Databricks WorkspaceImportexample.pyinto workspaceClick "Run All" buttonWhat is Slowly Changing DimensionSlowly Changing Dimensions (SCD) are dimensions which change over time and in Data Warehouse we need to track the changes of the attributes keep the accuracy of the report.And typically there are three types of SCDType 1: SCD1, No history preservationType 2: SCD2, Unlimited history preservation and new rowsType 3: SCD3, Limited history preservationFor example we have a datasetShortNameFruitColorPriceFAFiji AppleRed3.6BNBananaYellow1GGGreen GrapeGreen2RGRed GrapeRed2If we change the price of "Fiji Apple" into 3.5, the dataset will bewith SCD1:ShortNameFruitColorPriceFAFiji AppleRed3.5BNBananaYellow1GGGreen GrapeGreen2RGRed GrapeRed2with SCD2:ShortNameFruitColorPriceis_lastFAFiji AppleRed3.5YFAFiji AppleRed3.6NBNBananaYellow1YGGGreen GrapeGreen2YRGRed GrapeRed2Ywith SCD3:ShortNameFruitColorPriceColor_oldPrice_oldFAFiji AppleRed3.5Red3.6BNBananaYellow1NULLNULLGGGreen GrapeGreen2NULLNULLRGRed GrapeRed2NULLNULLSCD implementation in DatabricksIn this repository, there areimplementationsof SCD1, SCD2 and SCD3 in python and Databricks Delta Lake.SCD1dbxscd.SCD1(spark,df,target_table_name,target_partition_keys,key_cols,current_time):Parameters:spark: instance of spark sessiondf: source data frametarget_table_name: target table nametarget_partition_keys: partition key of the target tablekey_cols: columns of the key for each rowcurrent_time: current timestampHere is the code to show an example of SCD1frompyspark.sqlimportfunctionsasFfrompyspark.sqlimportDataFrameimportdatetime# create sample datasetdf1=spark.createDataFrame([('FA','Fiji Apple','Red',3.5),('BN','Banana','Yellow',1.0),('GG','Green Grape','Green',2.0),('RG','Red Grape','Red',2.0)],['ShortName','Fruit','Color','Price'])# prepare parameterscurrent_time=datetime.datetime.now().strftime("%Y-%m-%dT%H:%M:%S")target_partition_keys=['ShortName']key_cols="ShortName,Fruit"target_table_name_scd1='default.table_scd1'# call the SCD1 functiondbxscd.SCD1(spark,df1,target_table_name_scd1,target_partition_keys,key_cols,current_time)# display the resultdisplay(spark.sql(f"select * from{target_table_name_scd1}"))Change the price of "Fiji Apple" into 3.5 and run SCD1 againdf2=spark.createDataFrame([('FA','Fiji Apple','Red',3.6),('BN','Banana','Yellow',1.0),('GG','Green Grape','Green',2.0),('RG','Red Grape','Red',2.0)],['ShortName','Fruit','Color','Price'])# call the SCD1 function againdbxscd.SCD1(spark,df2,target_table_name_scd1,target_partition_keys,key_cols,current_time)display(spark.sql(f"select * from{target_table_name_scd1}"))SCD2dbxscd.SCD2(spark,df,target_table_name,target_partition_keys,key_cols,current_time):Parameters:spark: instance of spark sessiondf: source data frametarget_table_name: target table nametarget_partition_keys: partition key of the target tablekey_cols: columns of the key for each rowcurrent_time: current timestampHere is the code to show an example of SCD2frompyspark.sqlimportfunctionsasFfrompyspark.sqlimportDataFrameimportdatetime# create sample datasetdf1=spark.createDataFrame([('FA','Fiji Apple','Red',3.5),('BN','Banana','Yellow',1.0),('GG','Green Grape','Green',2.0),('RG','Red Grape','Red',2.0)],['ShortName','Fruit','Color','Price'])# prepare parameterscurrent_time=datetime.datetime.now().strftime("%Y-%m-%dT%H:%M:%S")target_partition_keys=['ShortName']key_cols="ShortName,Fruit"target_table_name_scd2='default.table_scd2'# call the SCD1 functiondbxscd.SCD2(spark,df1,target_table_name_scd1,target_partition_keys,key_cols,current_time)# display the resultdisplay(spark.sql(f"select * from{target_table_name_scd2}"))Change the price of "Fiji Apple" into 3.5 and run SCD2 againdf2=spark.createDataFrame([('FA','Fiji Apple','Red',3.6),('BN','Banana','Yellow',1.0),('GG','Green Grape','Green',2.0),('RG','Red Grape','Red',2.0)],['ShortName','Fruit','Color','Price'])# call the SCD1 function againdbxscd.SCD2(spark,df2,target_table_name_scd2,target_partition_keys,key_cols,current_time)display(spark.sql(f"select * from{target_table_name_scd2}"))SCD3dbxscd.SCD3(spark,df,target_table_name,target_partition_keys,key_cols,current_time):Parameters:spark: instance of spark sessiondf: source data frametarget_table_name: target table nametarget_partition_keys: partition key of the target tablekey_cols: columns of the key for each rowcurrent_time: current timestampHere is the code to show an example of SCD3frompyspark.sqlimportfunctionsasFfrompyspark.sqlimportDataFrameimportdatetime# create sample datasetdf1=spark.createDataFrame([('FA','Fiji Apple','Red',3.5),('BN','Banana','Yellow',1.0),('GG','Green Grape','Green',2.0),('RG','Red Grape','Red',2.0)],['ShortName','Fruit','Color','Price'])# prepare parameterscurrent_time=datetime.datetime.now().strftime("%Y-%m-%dT%H:%M:%S")target_partition_keys=['ShortName']key_cols="ShortName,Fruit"target_table_name_scd3='default.table_scd3'# call the SCD3 functiondbxscd.SCD3(spark,df1,target_table_name_scd3,target_partition_keys,key_cols,current_time)# display the resultdisplay(spark.sql(f"select * from{target_table_name_scd3}"))Change the price of "Fiji Apple" into 3.5 and run SCD3 againdf2=spark.createDataFrame([('FA','Fiji Apple','Red',3.6),('BN','Banana','Yellow',1.0),('GG','Green Grape','Green',2.0),('RG','Red Grape','Red',2.0)],['ShortName','Fruit','Color','Price'])# call the SCD1 function againdbxscd.SCD3(spark,df2,target_table_name_scd3,target_partition_keys,key_cols,current_time)display(spark.sql(f"select * from{target_table_name_scd3}"))