Performance Tips for Spark Delta Lake — part1
By: Salim Sayed
Tags: Performance, Delta Lake, Apache Spark

A system which has been using a data lake knows very well the painful process of updating records in it and the in-proportionate time and resources it requires for small volume read and write operations. Apache delta lake has helped solve many of these issues as well as ACID transaction control and time travel. In this article we will mainly discuss the tips and tricks for performance improvement using open source Spark delta lake. Unfortunately some key performance features are only available on the Data bricks managed Spark and delta lake environments. You would not be entirely out of luck if you are running Spark workload on-premise or private cloud environments, tips in this article will help.

Less data is more performance

As you may be aware that the less data we can process to execute an operation the faster the performance would be. A row can’t be updated in a file based data lake, rather the entire file containing that row needs to rewritten. Delta lake operations are optimized to skip as much data as possible and you need to take benefit of those features to the fullest. Records in a delta lake can be

Added/deleted/updated using the `DeltaTable.merge` statement. This is the most versatile API which can handle all scenarios. Updated using `DeltaTable.update` statement. Added using `df.write.format(“delta”).mode(SaveMode.Append).save(“path/to/delta-lake”)`.

DeltaTable.merge

This API skips data as described in the following flowchart. The implementation is here.

fig 1. merge process flow in delta lake.

Tips

Provide partition filtering explicitly

Lets say your merge statement looks like -

destination.as("d").merge(source.as("s"),"d.matchKey1=s.matchKey2 and d.partitionKey=s.partitionKey").whenMatched().updateAll().whenNotMatched().insertAll().execute()

It won’t automatically prune the partitions based on the data present in the “source”. You can provide that condition explicitly so that step 5 in the above chart is triggered. If the “source” is much smaller in comparison to “destination” and if it contains data for a subset of partitions then this will save you a huge amount of time.

val parts = source.select("partitionKey").distinct().collect().map(_.getAs[String]("partitionKey")) destination.as("d").merge(source.as("s"),"d.matchKey1=s.matchKey2 and d.partitionKey=s.partitionKey and d.partitionKey in (<list of partition keys>)").whenMatched().updateAll().whenNotMatched().insertAll().execute()

Provide Broadcast hint for small “source”

If the “source” data frame is small, then providing broadcast hint can save time in step 6 of the above flowchart.

val parts = source.select("partitionKey").distinct().collect().map(_.getAs[String]("partitionKey"))

destination.as("d").merge(broadcast(source).as("s"),"d.matchKey1=s.matchKey2 and d.partitionKey=s.partitionKey and d.partitionKey in (<list of partition keys>)").whenMatched().updateAll().whenNotMatched().insertAll().execute()

Use appropriate syntax for Add operations

Merge is a versatile syntax that can be used for all kinds of operations — add, update and delete. If you simply need to add new data when it doesn’t exist and not add them when a match is found then don’t write `whenMatched` expression. This will use ‘left anti’ join rather than ‘right outer’ join in step 7 which is faster.

destination.as("d").merge(source.as("s"),"d.matchKey1=s.matchKey2").whenNotMatched().insertAll().execute()

If you are want to always append/add new records without any check then simple use,

source.write.format("delta").mode(SaveMode.Append).save("path/to/delta-lake")

Order data in the delta lake as per the matching condition of merge statement

This is rather a hard ask to maintain if your have multiple conditions to merge into the delta-lake and you are not using data bricks runtime environment because you can’t use z-index. However if you have a single condition to merge data into the delta lake then you are lucky. Example — lets say your delta lake has 100 columns “column1”,”column2" to “column100” but you always merge data into it using “column1” like `destination.as(“d”).merge(source.as(“s”),”d.column1=s.column1")….` then you can arrange the delta lake in the order of column1. The underlying parquet files maintain basic statistics like count, min, max which is used by Spark to skip entire files in step 6. This subject needs more conversation and perhaps another post in near future.

DeltaTable.update, DeltaTable.delete

These operations are faster as compared to DeltaTable.merge as it doesn’t involve join operations as DeltaTable.merge does but rather filter operations alone. The overall flow is same, it finds the files containing the data and then recreates those files with intended updates/deletes.

Tips

Add filters on partition keys explicitly if possible.