Why is Delta Lake becoming increasingly popular? | by Lingeshwaran Kanniappan | Towards Data Science

It can be confusing to understand the difference between a Delta lake and the Data lake for a beginner. Working with the former storage layer for more than a year now, I am sharing the particular advantages which a delta lake serves and how it overcomes the challenges of a traditional data lake architecture with some examples.
The examples are to make it more intuitive and non-technical. I believe this will serve as a good curation from different sources for what delta lake can bring to the table.
By definition, delta lake is NOT a separate storage container, rather an open-source storage layer that will run on top of the existing data lake and object stores. Let us think of it as an additional functionality on top of the present data lakes, as shown below.
It is powered by a delta engine which in turn is powered by a native execution engine known as Photon. This is a vectorized query engine written in C++ from scratch. It is in public preview as of the time of writing this article.
It is a container to store raw data. Consider it to be a dumping ground of pure raw data which is yet to be processed.
Some important features of Data Lake include - its flat architecture, the schema is enforced only when you do a read operation (schema-on-read). Hence it can store unstructured data easily with low latency.
However, with raw data came raw challenges 🧱
Data lakes cannot handle corrupted data well. There will be a need for complete reprocessing when we identify a failure during a data load.
The solution that delta lake provides in this scenario is by bringing in Atomicity. This means that there is either a full write/commit (or) no write (aka Atomic visibility).
It is binary while doing the commit operation. So, we will know when there is an issue in the data before the job is completed.
There is a single source of truth that is employed to serve this purpose. It is known as the transaction log (delta log files), and it keeps track of all the atomic transactions that are being written into the delta table. Spark then updates the table at the user’s side after every transaction.
The delta log files are JSON files (00000000000000000000.json, 00000000000000000001.json etc.) placed inside the _delta_log subdirectory together with occasional checkpoint files.
Inside a delta log file, there are also commitInfo, partitionValues and stats information which are different metadata attributes to keep track of the data changes that can benefit in improving the query efficiency.
Consider the following simple credit transaction, once the data is written it should get updated with the correct value of 110 £ after the service is up and running (or) remain as 100 £ with the transaction log capturing no commit information. This is what an atomic commit must look like.
Only one client can achieve the creation of the object via one atomic commit.
Partial execution can result in a non-atomic commit which happens in a data lake during server failure (or) other issues.
Two scenarios pose an issue when it comes to missing data:
Batch and stream processing data has necessitated the use of lambda architecture style that writes data into two different containers. The unification of both these workload types is not possible in a data lake.
During an overwrite operation (basically delete and write), there is a point in time when there is no data in the table and if any failure occurs during this instant, we will not see any data when we query it.
This data inconsistency is handled in Delta lake using the concept called optimistic concurrency control through which the data being written either by multiple users at the same time (or) by different types (batch/stream) is done in a serialized fashion. We can use the source as a batch table and the streaming table as a sink.
There are special scenarios where the concurrent operations cannot be performed optimistically, during which the delta engine throws an error.
When there is a loss of connection (or) some form of failure (or) multiple writes, the data to be committed should not be lost rather it is captured in the transaction log and it will perform the commit in a serialized fashion.
As described in the Consistency section, the operations happening in parallel needs to be serially isolated to avoid conflicts. This allows the unification of the data models (batch and stream) which was not possible in the case of a data lake.
This is how the isolation mechanism is addressing the challenge: The commits that are not yet completed are isolated by the rule of mutual exclusion. Only when the commit is successful it allows for a write /merge/ delete operation into the delta lake.
This also brings in the facility to use the standard (DELETE, UPDATE, MERGE INTO) commands on the delta lake data which helps in bringing CCPA/GDPR compliance to your data.
There are special scenarios where two operations can lead to conflicts in the delta table, which is captured here for further reference. In short, those conflicts can be avoided using partitioning on the column that is used in the filter command.
Here the transactions T1 and T2 are isolated and executed sequentially using the transaction log information. Hence, they both don’t contradict the data at any given instant.
Consider the scenario where there is a failure and the server-side during an operation. This can again, result in loss of data (or) a population of corrupted records. This cannot be avoided in a typical data lake.
However, due to transaction logging and on-disk storage persistence, the data will persist even when there is a system (or) server-side failure. Hence the non-volatile memory plays a crucial role in delta lake.
Here, the update operation can fail and/or a read can fail when the server is down. Since the transactions are stored on the disk, the latest value can be read from the delta table until the last commit.
In terms of authenticity, when we try to change the type of data being written into the data lake, it leads to corruption and missing records. Hence, there needs to be a check on the validity of the schema.
In delta lake, there is schema validation being performed before any commit. A different schema by default is prevented from performing an overwrite. In this way, the schema format in which the data is written should always match with the schema that was initially defined (aka schema-on-write).
If an overwrite is required, schema evolution is there for the rescue. This can be done by setting configuration options while performing the write operation.
Here, for the simple insert operation: When mergeSchema is set to true there is typecasting of the Float type to Long type. There is another option called overwriteSchema setting this to true will completely ignore the old data and creates a new table in its place.
Modern big data systems are consistent in making frequent and volumetric data changes. There needs to be an auditing system (or) rollback mechanism to monitor the changes that are happening. The support for which is not present in data lakes.
However, in a delta table, the current state of the data is the collective sum of all the commits recorded in the transaction log (_delta_log/) subdirectory within the table.
It serves as a recipe to build the data from scratch to its current form. This ability to create the data by traversing through the transaction log is called Time-travel (or) data versioning.
The previous versions of the data in the delta table can be accessed via the timestamp (or) the version number. The data in delta tables are ultimately stored as versioned parquet files.
This allows users to perform easier rollback operations to get back to the instant when the data was proper and undo the wrong commit.
Every table comes with a version number (or) timestamp which can be fetched from a DESCRIBE HISTORY <table_name>command. Either using the clause VERSION AS OF (or) TIMESTAMP AS OF we can get the snapshot of the table at that instant.
Metadata is the data of the data for the data. It consists of the schema, column name, type and transactional information of the table.
Metadata is continuously generated as the atomic commits are logged in the delta log. This can be used to keep track of all the changes that are happening due to all the sources.
This metadata management is scalable in Spark and its processing is distributed across the nodes just like the data. This captures the schema, time of the creation of the atomic commit, GUID (globally unique identifier), the format of the file and other configuration options.
The read/list operation is expensive in native object stores, whereas in the case of a delta lake the metadata read is parallelized from the log over the cluster.
This can be done using DESCRIBE HISTORY , DESCRIBE DETAIL SQL commands. This runs on the parallel processing engine and gives the underlying metadata information. For further reference, you can see the documentation here.
-- get the full historyDESCRIBE HISTORY '/data/balanceTable/'        DESCRIBE HISTORY delta.'/data/balanceTable/'-- get the metadata of delta tableDESCRIBE DETAIL '/data/balanceTable/'DESCRIBE DETAIL delta.'/data/balanceTable/'
To conclude, all these challenges posed by a data lake were handled using a delta lake. It greatly reduces the complexity of raw data handling by keeping the data flow consistent and durable.
The key takeaway will be to use Delta lake when there is going to be a large amount of raw data with a focus on improving the query performance, data consistency and durability.
Please provide your constructive thoughts, comments (or) suggestions if any.
Also, feel free to connect with me on LinkedIn. Until next time!