APACHE SPARK AND DELTA LAKE, A POWERFUL COMBINATION
By: Nabarun Chakraborti
Tags: Delta Lake, Big Data Analytics, Spark, Big Data, Apache Spark

How to Overcome?

The above problems can be managed if we plugin Delta Lake as an additional service with Spark. Spark usages underlying distributed system like HDFS, S3 etc to store data.

Now, instead of doing this if we make Spark talking to Delta Lake and Delta Lake will communicate further with underlying distributed storage, then easily we can make Spark to inherit Delta lake ACID attributes and other useful properties.

What is Delta Lake?

Before that let’s take a quick look what is Data Lake? We can consider Data Lake as a centralized data repository with massive scale and open formats which can stores both Structured and Unstructured raw data. But there are few areas where it is lacking, like — inconsistent data, lack of schema and issue with performance.

Here comes Delta Lake! Delta Lake is an open source storage layer that brings reliability to Data Lakes. Delta Lake provides ACID transactions, Schema enforcement, scalable metadata handling, and unifies streaming and batch data processing. Most importantly, Delta Lake runs on top of our existing Data Lake (Parquet file storage) and is fully compatible with Apache Spark APIs.

Few important features Delta Lake offers to Spark:

1. ACID transactions. 2. Schema enforcement / schema validation. 3. Enables Updates, Deletes and Merge. 4. Time Travel/Versioning (look back to historic data)

Spark With Delta Lake — Schema Enforcement Example :

Let’s consider the same 2 data files (with different datatypes — ‘Weight column’, as used in previous example) to check how spark behaves when it’s overwriting data using Delta Lake.

Let’s read the data:

Writing the data without Delta Lake:

Writing the data with Delta Lake:

So far no issue and we are able to write data with and without Delta Lake using ‘overwrite’ mode.

Let’s read the 2nd file having different column type (Weight is double here):

As tested already, no issue to overwrite schema mismatched data without using Delta Lake:

But, if we use Delta Lake to write the same data, it will throw an error: “Failed to merge incompatible data types IntegerType and DoubleType”

So, we can see Delta Lake validates the schema before writing into underlying system, which means Delta Lake uses schema validation on write,. If the schema is not compatible then Delta Lake will cancel the transaction. This will help to track the issue at the beginning rather mixing up with good data and then try to find out the root cause.

Mainly it validates below 3 points during writing any data:

1. The column data types should be same as the target table.

2. Number of columns should be same as target table.

3. Column names should be same as target table.

Note: by default Delta Lake will not allow to write data having different number of columns (point #2). But we can force to append these files by using .option(“mergeSchema”, “true”) during writing our data based on the need.

Spark With Delta Lake — Enables Update and Delete Example :

Spark never allows us to run Update or Delete statements. But, if we use Delta Lake we can easily perform these kind of operations.

Let’s consider the below file as our input data:

Reading the data:

Save as Delta Lake Table:

Let’s UPDATE few records:

Let’s DELETE records from the same table:

So, we can see by using Delta Lake it’s very easy to utilize the DELETE, UPDATE commands

How does Delete, Update and Merge works?

The below structure represents how does Delta Lake maintain the version details for each DML transaction. It maintains a Transaction Log which captures all the operations performed on the file. Inside our folder structure there is a _delta_log folder which contains the JSON logs. These JSON logs will be having all this transaction details. For each DML operation (performs over the period of time) there will be a separate JSON created, which represents the version of the table. So, the whole Delta Lake table is versioned and we can easily query back.

Also, there is a Parquet file, which holds the actual processed data. (In the below figure we consider our data is partitioned by date)

fig-1

So, let’s understand how does UPDATE , DELETE and MERGE works.

UPDATE under the hood:

Delta Lake stores the data as a Parquet file, and we know Parquet files are not designed to be replaced or update. Which means Delta Lake can’t run DML straight away on its underlying files. It performs 2 level of scanning to Update any file.

Let’s consider there are 5 parquet files inside our data file directory. These 5 files contain our processed data and assume these files represent version 1.

SQL Statement: UPDATE table SET col1 = value WHERE <predicates>

Scan 1: Based on the defined predicates it will find and select the underlying files which contain the matching records. In the below image, the Grey boxes (file1, file3 and file5) represent files which has no matches at all. Other 2 files (file2 and file4) have partial matches, few rows of parquet file matched the predicates. The Blue sections represent rows with matched records where Yellow again non matching records but they belong to the files with matches.

fig-2

Scan 2: Once the files are getting identified (via Scan 1), it will rewrite those files. Because, underlying parquet can’t be deleted or updated.

fig-3

The rows matched the predicate (blue parts) will be updated in the new files where unmatched records (yellow parts) will be simply copied into those new files.

All the Tombstone files (where no matched found) will not be physically deleted, rather, in the transaction log (JSON files — as defined in fig-1) these files will be marked as “Removed”. This will help to maintain the version history and will be used during time travel.

Python approach:

from delta.tables import *

from pyspark.sql.function import * dt = DeltaTable.forPath(spark, "/deltaPath")

dt.update("Name = ‘Tim’", {"Height":157})

DELETE under the hood:

As we have seen UPDATE never physically removes old files, same way DELETE does not remove data on disk until we execute VACUUM command. Same as Update, Delete rewrites new files where predicates are observed and keeps the old files for time travel queries. So, accidentally if we delete some data we can query previous versions to get the data back. Happy !!

If we feel old records are not so important and we can store only for a certain time range, we can use VACUUM to delete the unwanted data permanently.

VACUUM table [RETAIN t HOURS]

The above statement says delete all old files earlier than “t” hours. The default vacuum retention period is 7 days. Never use vacuum with zero (0) retention, unless we are very sure. This will remove all versions except the latest one and will create inconsistency if some other process is writing on the same file.

Python approach:

from delta.tables import *

from pyspark.sql.function import * dt = DeltaTable.forPath(spark, "/deltaPath")

dt.delete("predicate")

MERGE under the hood:

Merge is a very powerful operation. It will help us to update, delete or insert records based on conditions.

First understand why Merge is so important. Assume we want to execute Insert/Update on underlying records. Let’s compare the steps involved to run the same operation on Parquet table (Data Lake) with a legacy data pipeline and Delta Lake.

It’s quite obvious, there are multiple steps involved to perform the activity with traditional Parquet data store, where MERGE will take care of all after identifying the rows in Delta Lake for us.

So, how does MERGE work? The behind story is more or less same, likewise Update and Delete, Merge also finds out the matched records, create a new file and then apply the changes, but it uses JOIN operations to do so.

Scan 1: It runs INNER JOIN between source and target tables to identify the matched files (fig-2).

Scan 2: Then it performs OUTER JOIN between matched files in target and source. And then apply Update/Delete/Insert operations on those records.

Python approach:

from delta.tables import *

from pyspark.sql.function import * dt = DeltaTable.forPath(spark, deltaPath) dt.alias(“t”)

.merge(sourceDF.alias(“s”), “t.id = s.id”)

.whenMatchedUpdateAll()

.whenNotMatchedInsertAll()

.execute()

SQL approach:

MERGE into target t

USING source s

ON t.id = s.id

WHEN MATCHED [optional [and <additional conditions>]] THEN UPDATE SET *

WHEN MATCHED THEN DELETE

WHEN NOT MATCHED [optional [and <additional conditions>]] THEN INSERT *

Spark With Delta Lake — Time Travel Example :

So far we have discussed about - Transaction logs, that contains all the action details performed on that specific version file. Then, how does Delta keep all the old files and maintain the version history during DML operations. Also, it never physically deletes files (except vacuum), rather marked them as removed.

All these behaviors help us to go back and pull the older records based on our need.

Display(deltaTable.history())

Basically, we are not loosing any old transactions inside Delta. The version history will display all the details and we can load historic data by using the version number, timestamp etc.

We can use dataframe API to pull the records based on version or timestamp like:

# version specific

DF = spark.read.format("delta").option("versionAsOf", 3).load("/deltaFilePath") # timestamp specific

DF = spark.read.format("delta").option("timestampAsOf", "2020-09-18T13:28:55").load("/deltaFilePath")

Note: don’t hold the old parquet files (actual data file) unnecessary. More old data files means execution process will take longer time to scan through all of them. It’s always a good practice to VACUUM data files for certain intervals (ex. 7 days). But all the transaction logs (JSON files) will be available under _delta_log directory.

This is it! A quick overview how to overcome the traditional drawbacks of Spark, what we can achieve combining Spark with Delta Lake and how efficiently data can be managed.

HAPPY LEARNING !!