Delta Lake in Action: Upsert & Time Travel | by Jyoti Dhiman | Towards Data Science

This is a follow up of my introduction to the Delta Lake with Apache Spark article, please read on to find out how to use Delta lake with Apache Spark to perform operations like Update existing data, check out previous versions of data, convert data to delta table, etc.
Before diving into code, let us try to understand when to use Delta Lake with Spark because it’s not like I just woke up one day and included Delta Lake in the architecture :P
Delta Lake can be used:
When dealing with “overwrite” of the same dataset, this is the biggest headache I have dealt with and Delta Lake really helps in such scenarios.
When dealing with data having updates, the “merge” functionality of Delta Lake helps in working with Updates in Data(goodbye, messy join/filter operations!!)
Surely, there are many use cases to work with Delta Lake, but these are the two scenarios, which really helped me.
Enough talk, let’s dive into code!
First things first, to get started with Delta Lake, it needs to be added as a dependency with the Spark application, which can be done like:
For pyspark shell
pyspark --packages io.delta:delta-core_2.11:0.6.1 --conf "spark.sql.extensions=io.delta.sql.DeltaSparkSessionExtension" --conf "spark.sql.catalog.spark_catalog=org.apache.spark.sql.delta.catalog.DeltaCatalog"
As a maven dependency, delta lake can be included as below in pom.xml
<dependency>  <groupId>io.delta</groupId>  <artifactId>delta-core_2.11</artifactId>  <version>0.6.1</version></dependency>
Things to note:
Here, 2.11 is the scala version, if working with scala 2.12 change version accordingly.
0.6.1 is the Delta Lake version which is the version supported with Spark 2.4.4.
As of 20200905, latest version of delta lake is 0.7.0 with is supported with Spark 3.0
AWS EMR specific: Do not use delta lake with EMR 5.29.0, it has known issues. It is recommended to upgrade or downgrade the EMR version to work with Delta Lake.
Delta lake works with data stored as Delta Tables, so data needs to be written as Delta Table.
To convert existing data to Delta Table, this needs to be done once.
df = spark.read.parquet("path/of/some/data") # Read existing datadf.coalesce(5).write.format("delta").save("path/of/some/deltaTable")
To write fresh data as Delta Table
df.coalesce(5).write.format("delta").save("path/of/some/deltaTable")
Things to Note:
When data is saved as a delta table, the path contains _delta_log directory which keeps track of Delta Table metadata.
Write DeltaTable partitioned with some column like:
df.coalesce(5).write.partitionBy("country").format("delta").save("path/of/some/deltaTable")
As a Spark dataframe:
df = spark.read.format("delta").load("path/of/some/deltaTable")As DeltaTable:
Reading DeltaTable as Spark Dataframe allows all Dataframe operations on the DeltaTable.
As DeltaTable:
deltaTable = DeltaTable.forPath("path/of/some/deltaTable")
Reading data as DeltaTable only supports DeltaTable specific functions. Also, DeltaTable can be converted to DataFrame like:
df = deltaTable.toDF
UPSERT operation on DeltaTable allows for data updates, which means if DeltaTable can be merged with some new dataset, and on the basis on some join key , data can be inserted on modified in the delta table. This is how Upsert can be done on the DeltaTable:
val todayData = # new data that needs to be written to deltaTableval deltaTable = DeltaTable.forPath("path/of/some/deltaTable")deltaTable.alias("oldData")  .merge(    todayData.alias("newData"),    "oldData.city_id = newData.city_id")  .whenMatched()  .updateAll()  .whenNotMatched()  .insertAll()  .execute()
Seems simple right? That’s it! So what will happen is, rows in todayData with updated information on cities will be modified in deltaTable if the city already exists in deltaTable and if the city doesn’t exist, rows will be inserted in deltaTable , our basic Upsert!
Things to note:
Upsert operation can be modified with the use of partitions, for instance, if a specific partition needs to be merged, it can be specified to optimize merge operation like:
deltaTable.alias("oldData")  .merge(    todayData.alias("newData"),    "(country = 'India') AND oldData.city_id = newData.city_id")  .whenMatched()  .updateAll()  .whenNotMatched()  .insertAll()  .execute()
The operation to perform in case rows match or don’t match in deltaTable with the join key is configurable, refer the documentation for details.
By default, the merge operation will write lots of small files, to control the number of small files, set followingspark conf properties accordingly, refer the documentation for details.
spark.delta.merge.repartitionBeforeWrite truespark.sql.shuffle.partitions 10
After a successful merge operation, a log commit will be created in _delta_log directory, it might be interesting to go through its contents to understand the workings of Delta Lake.
Merge only supports one-to-one mapping, i.e. only row should try to update one row in DeltaTable, if multiple rows try to update the same row in DeltaTable, the merge operation fails. Preprocess data to take care of this.
Delta Lake enables the capability to go back to previous versions of data, the same can be achieved via specifying which version of data to read as:
df= spark.read.format("delta").option("versionAsOf", 0).load("path/of/some/deltaTable")
Versioning starts from 0, and there will be as many versions of data as there are log commits in the delta_log directory.
Seems, simple right?
For reference:
docs.delta.io
Until next time,Ciao.