Package: abx-scd
Slowly Changing Dimensions implemenation with Databricks Delta LakeWhat is Slowly Changing DimensionSlowly Changing Dimensions (SCD) are dimensions which change over time and in Data Warehuse we need to track the changes of the attributes keep the accuracy of the report.And typically there are three types of SCDType 1: SCD1, No history preservationType 2: SCD2, Unlimited history preservation and new rowsType 3: SCD3, Limited history preservationFor example we have a datasetShortNameFruitColorPriceFAFiji AppleRed3.6BNBananaYellow1GGGreen GrapeGreen2RGRed GrapeRed2If we change the price of "Fiji Apple" into 3.5, the dataset will bewith SCD1:ShortNameFruitColorPriceFAFiji AppleRed3.5BNBananaYellow1GGGreen GrapeGreen2RGRed GrapeRed2with SCD2:ShortNameFruitColorPriceis_lastFAFiji AppleRed3.5YFAFiji AppleRed3.6NBNBananaYellow1YGGGreen GrapeGreen2YRGRed GrapeRed2Ywith SCD3:ShortNameFruitColorPriceColor_oldPrice_oldFAFiji AppleRed3.5Red3.6BNBananaYellow1NULLNULLGGGreen GrapeGreen2NULLNULLRGRed GrapeRed2NULLNULLSCD implementation in DatabricksIn this repository, there areimplementationsof SCD1, SCD2 and SCD3 in python and Databricks Delta Lake.SCD1SCD1(df,target_table_name,target_partition_keys,key_cols,current_time):Parameters:df: source dataframetarget_table_name: target table nametarget_partition_keys: partition key of the target tablekey_cols: columns of the key for each rowcurrent_time: current timestampHere is the code to show an example of SCD1frompyspark.sqlimportfunctionsasFfrompyspark.sqlimportDataFrameimportdatetime# create sample datasetdf1=spark.createDataFrame([('FA','Fiji Apple','Red',3.5),('BN','Banana','Yellow',1.0),('GG','Green Grape','Green',2.0),('RG','Red Grape','Red',2.0)],['ShortName','Fruit','Color','Price'])# prepare parameterscurrent_time=datetime.datetime.now().strftime("%Y-%m-%dT%H:%M:%S")target_partition_keys=['ShortName']key_cols="ShortName,Fruit"target_table_name_scd1='default.table_scd1'# call the SCD1 functionSCD1(df1,target_table_name_scd1,target_partition_keys,key_cols,current_time)# display the resultdisplay(spark.sql(f"select * from{target_table_name_scd1}"))Change the price of "Fiji Apple" into 3.5 and run SCD1 againdf2=spark.createDataFrame([('FA','Fiji Apple','Red',3.6),('BN','Banana','Yellow',1.0),('GG','Green Grape','Green',2.0),('RG','Red Grape','Red',2.0)],['ShortName','Fruit','Color','Price'])# call the SCD1 function againSCD1(df2,target_table_name_scd1,target_partition_keys,key_cols,current_time)display(spark.sql(f"select * from{target_table_name_scd1}"))SCD2SCD2(df,target_table_name,target_partition_keys,key_cols,current_time):Parameters:df: source dataframetarget_table_name: target table nametarget_partition_keys: partition key of the target tablekey_cols: columns of the key for each rowcurrent_time: current timestampHere is the code to show an example of SCD2frompyspark.sqlimportfunctionsasFfrompyspark.sqlimportDataFrameimportdatetime# create sample datasetdf1=spark.createDataFrame([('FA','Fiji Apple','Red',3.5),('BN','Banana','Yellow',1.0),('GG','Green Grape','Green',2.0),('RG','Red Grape','Red',2.0)],['ShortName','Fruit','Color','Price'])# prepare parameterscurrent_time=datetime.datetime.now().strftime("%Y-%m-%dT%H:%M:%S")target_partition_keys=['ShortName']key_cols="ShortName,Fruit"target_table_name_scd1='default.table_scd2'# call the SCD1 functionSCD2(df1,target_table_name_scd1,target_partition_keys,key_cols,current_time)# display the resultdisplay(spark.sql(f"select * from{target_table_name_scd2}"))Change the price of "Fiji Apple" into 3.5 and run SCD2 againdf2=spark.createDataFrame([('FA','Fiji Apple','Red',3.6),('BN','Banana','Yellow',1.0),('GG','Green Grape','Green',2.0),('RG','Red Grape','Red',2.0)],['ShortName','Fruit','Color','Price'])# call the SCD1 function againSCD2(df2,target_table_name_scd2,target_partition_keys,key_cols,current_time)display(spark.sql(f"select * from{target_table_name_scd2}"))SCD3SCD3(df,target_table_name,target_partition_keys,key_cols,current_time):Parameters:df: source dataframetarget_table_name: target table nametarget_partition_keys: partition key of the target tablekey_cols: columns of the key for each rowcurrent_time: current timestampHere is the code to show an example of SCD3frompyspark.sqlimportfunctionsasFfrompyspark.sqlimportDataFrameimportdatetime# create sample datasetdf1=spark.createDataFrame([('FA','Fiji Apple','Red',3.5),('BN','Banana','Yellow',1.0),('GG','Green Grape','Green',2.0),('RG','Red Grape','Red',2.0)],['ShortName','Fruit','Color','Price'])# prepare parameterscurrent_time=datetime.datetime.now().strftime("%Y-%m-%dT%H:%M:%S")target_partition_keys=['ShortName']key_cols="ShortName,Fruit"target_table_name_scd1='default.table_scd3'# call the SCD1 functionSCD3(df1,target_table_name_scd3,target_partition_keys,key_cols,current_time)# display the resultdisplay(spark.sql(f"select * from{target_table_name_scd3}"))Change the price of "Fiji Apple" into 3.5 and run SCD3 againdf2=spark.createDataFrame([('FA','Fiji Apple','Red',3.6),('BN','Banana','Yellow',1.0),('GG','Green Grape','Green',2.0),('RG','Red Grape','Red',2.0)],['ShortName','Fruit','Color','Price'])# call the SCD1 function againSCD3(df2,target_table_name_scd3,target_partition_keys,key_cols,current_time)display(spark.sql(f"select * from{target_table_name_scd3}"))