Delta Lake in Action: Upsert & Time Travel
By: Jyoti Dhiman
Tags: Delta Lake, Big Data, Data, Apache Spark, Data Science

How to run Delta Lake with Apache Spark

First things first, to get started with Delta Lake, it needs to be added as a dependency with the Spark application, which can be done like:

For pyspark shell

pyspark --packages io.delta:delta-core_2.11:0.6.1 --conf "spark.sql.extensions=io.delta.sql.DeltaSparkSessionExtension" --conf "spark.sql.catalog.spark_catalog=org.apache.spark.sql.delta.catalog.DeltaCatalog"

As a maven dependency, delta lake can be included as below in pom.xml

<dependency>

<groupId>io.delta</groupId>

<artifactId>delta-core_2.11</artifactId>

<version>0.6.1</version>

</dependency>

Things to note:

Here, 2.11 is the scala version, if working with scala 2.12 change version accordingly.

is the version, if working with scala change version accordingly. 0.6.1 is the Delta Lake version which is the version supported with Spark 2.4.4 .

is the Delta Lake version which is the version supported with . As of 20200905 , latest version of delta lake is 0.7.0 with is supported with Spark 3.0

, latest version of delta lake is with is supported with AWS EMR specific: Do not use delta lake with EMR 5.29.0 , it has known issues. It is recommended to upgrade or downgrade the EMR version to work with Delta Lake.

How to Write data as Delta Table?

Delta lake works with data stored as Delta Tables, so data needs to be written as Delta Table.

To convert existing data to Delta Table, this needs to be done once.

df = spark.read.parquet("path/of/some/data") # Read existing data df.coalesce(5).write.format("delta").save("path/of/some/deltaTable")

To write fresh data as Delta Table

df.coalesce(5).write.format("delta").save("path/of/some/deltaTable")

Things to Note:

When data is saved as a delta table, the path contains _delta_log directory which keeps track of Delta Table metadata.

directory which keeps track of Delta Table metadata. Write DeltaTable partitioned with some column like:

df.coalesce(5).write.partitionBy("country").format("delta").save("path/of/some/deltaTable")

How to read a Delta Table?

As a Spark dataframe:

df = spark.read.format("delta").load("path/of/some/deltaTable")As DeltaTable:

Reading DeltaTable as Spark Dataframe allows all Dataframe operations on the DeltaTable.

As DeltaTable:

deltaTable = DeltaTable.forPath("path/of/some/deltaTable")

Reading data as DeltaTable only supports DeltaTable specific functions. Also, DeltaTable can be converted to DataFrame like:

df = deltaTable.toDF

Upsert with Delta Lake in Apache Spark

UPSERT operation on DeltaTable allows for data updates, which means if DeltaTable can be merged with some new dataset, and on the basis on some join key , data can be inserted on modified in the delta table. This is how Upsert can be done on the DeltaTable:

val todayData = # new data that needs to be written to deltaTable val deltaTable = DeltaTable.forPath("path/of/some/deltaTable") deltaTable.alias("oldData")

.merge(

todayData.alias("newData"),

"oldData.city_id = newData.city_id")

.whenMatched()

.updateAll()

.whenNotMatched()

.insertAll()

.execute()

Seems simple right? That’s it! So what will happen is, rows in todayData with updated information on cities will be modified in deltaTable if the city already exists in deltaTable and if the city doesn’t exist, rows will be inserted in deltaTable , our basic Upsert!

Things to note:

Upsert operation can be modified with the use of partitions, for instance, if a specific partition needs to be merged, it can be specified to optimize merge operation like:

deltaTable.alias("oldData")

.merge(

todayData.alias("newData"),

"(country = 'India') AND oldData.city_id = newData.city_id")

.whenMatched()

.updateAll()

.whenNotMatched()

.insertAll()

.execute()

The operation to perform in case rows match or don’t match in deltaTable with the join key is configurable, refer the documentation for details.

with the join key is configurable, refer the documentation for details. By default, the merge operation will write lots of small files, to control the number of small files, set following spark conf properties accordingly, refer the documentation for details.

spark.delta.merge.repartitionBeforeWrite true

spark.sql.shuffle.partitions 10

After a successful merge operation, a log commit will be created in _delta_log directory, it might be interesting to go through its contents to understand the workings of Delta Lake.

directory, it might be interesting to go through its contents to understand the workings of Delta Lake. Merge only supports one-to-one mapping, i.e. only row should try to update one row in DeltaTable , if multiple rows try to update the same row in DeltaTable , the merge operation fails. Preprocess data to take care of this.

Time Travel with Delta Lake in Apache Spark

Delta Lake enables the capability to go back to previous versions of data, the same can be achieved via specifying which version of data to read as:

df= spark.read.format("delta").option("versionAsOf", 0).load("path/of/some/deltaTable")