Package: dtstools
DTSTOOLS: Help with your Delta LakeLibrary created byData In Action#####    ######    #####   ######     ###      ###    ####      #####   
 ## ##     ##     ##   ##    ##      ## ##    ## ##    ##      ##   ##  
 ##  ##    ##     ##         ##     ##   ##  ##   ##   ##      ##       
 ##  ##    ##      #####     ##     ##   ##  ##   ##   ##       #####   
 ##  ##    ##          ##    ##     ##   ##  ##   ##   ##           ##  
 ## ##     ##     ##   ##    ##      ## ##    ## ##    ##      ##   ##  
#####      ##      #####     ##       ###      ###    #######   #####Table of ContentsIntroductionHow to use dtstoolsHow to use tableSizeHow to use tableMaintenanceHow to use LastMaintenanceFuture implementationsNotesReferencesIntroductionversiondatedescriptionv0.0.42023-06-17Basic featuresv0.0.92023-07-06Introduce lastMaintenance functionThis package aims to provide functionality to work with Delta Lake.Facilitating the visualization of the actual size (Storage) of your Delta tables and their maintenance (Vacuum and Optimize)Below are the steps performed in order of execution:Executed adescribe detailto get the current location and sizeA scan (dbutils.fs.ls) is performed on the Storage folders recursively to calculate the space used in the Storage, excluding the _delta_logs, with this we can calculate an average of how much can be released with VacuumReturns a DataframetHow to use dtstoolsFirst install the package via PyPipip install --upgrade dtstoolsImport the dtsTable module into your contextfrom dtstools import dtsTableUse the function dtsTable.Help() to see examples of function usage and a summary of each function.dtsTable.Help()How to use tableSizeFind out the true size of your tableCall the tableSize function passing the database and table name, use display() to see the results.This function returns a Dataframe.dtsTable.tableSize(databaseName,tableName).display()Save the result in a Delta Table for monitoring and baselinedtsTable.tableSize(databaseName,tableName) \
 .write.format('delta') \
 .mode('append') \
 .saveAsTable("db_demo.tableSize",path='abfss://container@storage.dfs.core.windows.net/bronze/tableSize')Get the size of all tables in your databasefor tb in spark.sql(f"show tables from db_demo").collect():
    try:
        print(">> Collecting data... Table:",tb.tableName)
        dtsTable.tableSize(tb.database,tb.tableName) \
        .write.format('delta') \
        .mode('append') \
        .saveAsTable("db_demo.tableSize",path='abfss://container@storage.dfs.core.windows.net/bronze/tableSize')
    except Exception as e:
        print (f"###### Error to load tableName {tb.tableName} - {e}######")How to use tableMaintenanceApply maintenance to a table, Optimize and Vacuum.ParameterDescriptionTypeschemaNameName of the database where the table is createdstringtableNameName of the table that will be applied in the maintenancestringvacuumTrue: Vacuum will run, False: Ignore vacuumbooloptimizeTrue: OPTIMIZE will be executed, False: Skip OPTIMIZEboolzorderColumnsIf informed and optimize is equal to True, Zorder is applied to the list of columns separated by a comma (,)stringvacuumRetentionNumber of hours to hold after vacuum runswholeDebugJust print the result on screenboolApply in a single table.dtsTable.tableMaintenance(schemaName="Database", tableName="tableName", zorderColumns='none', vacuumRetention=168, vacuum=True, optimize=True, debug=False)Apply maintenance to all tables for YOUR databasefor tb in spark.sql(f"show tables from db_demo").collect():
  dtsTable.tableMaintenance(schemaName=tb.database, tableName=tb.tableName, zorderColumns='none', vacuumRetention=168, vacuum=True, optimize=True, debug=False)Last MaintenanceUse the function dtsTable.LastMaintenance() to see summary vacuum result and optimize operations.dtsTable.lastMaintenance('database','tableName')Future implementationsUse Unity CatalogRun for all databasesMinimize costs with dbutils.fs.ls by looking directly into the transaction logNotesFor partitioned tables with many partitions, the execution time can take longer, so monitor the first executions well, the use is at your responsibility, despite not having any risk mapped so far, it can only generate more transactions for your storageCost of Azure Storage Transactions: Read Operations (per 10,000) - $0.0258 (two cents per 10,000 operations) (Estimated price as of 4/21/2023)Referenceshttps://github.com/reginaldosilva27/dtstoolsAuthor: Reginaldo SilvaBlog Data In ActionGithubDataside