Databricks Delta Lake — Database on top of a Data Lake — Part 1 | by Manoj Kukreja | Towards Data Science

Going back 8 years, I still remember the days when I was adopting Big Data frameworks like Hadoop and Spark. Coming from a database background this adaptation was challenging for many reasons. The most challenging was the lack of database like transactions in Big Data frameworks. To cover for this missing functionality we had to develop several routines the performed the necessary checks and measures. However, the process was cumbersome, time-consuming and frankly error-prone.
Another issue that use to keep me awake at night was the dreaded Change Data Capture (CDC). Databases have a convenient way of updating records and showing the latest state of the record to the user. On the other hand in Big Data we ingest data and store them as files. Therefore, the daily delta ingestion may contain a combination of newly inserted, updated or deleted data. This means we end up storing the same row multiple times in the Data Lake. This creates two problems:
Duplicate Data — In some cases the same row exists more than once (updated and deleted data)Data Analytics — Unless data is already de-duplicated, users get confused when they see multiple instances of the same row
Duplicate Data — In some cases the same row exists more than once (updated and deleted data)
Data Analytics — Unless data is already de-duplicated, users get confused when they see multiple instances of the same row
So how did we manage to deal with this situation up till now:
Day — Ingest the full data set — complete tables
Day 2-Day n — Make sure the incremental data (delta) is delivered with a Record Update Timestamp and the Mode (Insert/Update/Delete)
After data has been ingested in the raw zone run a Hive/Mapreduce/Spark curation job that merges incremental data with the full data.
However, at this stage, it cleverly uses remove duplicates using functions like RANK() OVER PARTITION of PRIMARY KEY and Record Update Timestamp DESC
Filter rows with RANK=1 gives us the most recently updated row
Drop rows with Mode=Delete
Save above data set to the curation zone — This data is then shared with the user community for further analysis down line
As you see the above process is pretty involving. A better method is warranted.
Developed by Databricks, Delta Lake brings ACID transaction support for your data lakes for both batch and streaming operations. Delta Lake is an open-source storage layer for big data workloads over HDFS, AWS S3, Azure Data Lake Storage or Google Cloud Storage.
Delta Lake packs in a lot of cool features useful for Data Engineers. Lets explore a few of these features in a two part series:
Part 1: ACID Transactions, Checkpoints, Transaction Log & Time Travel
Part 2 : Vaccum, Schema Evolution, History
Use Case: An eCommerce website sells products from several vendors. Each vendor sends latest prices for its products on a daily basis.
The eCommerce company wants to adjust the pricing information on their website based on latest prices sent by the vendor each day. Additionally, they want to track prices over time for their ML models. On the left is a sample of the data files received each day.
Lets start by downloading data:
$ git clone https://github.com/mkukreja1/blogs.git
Assume today is Aug20 and you received the file — products_aug20.csv
Save the data file to HDFS
$ hadoop fs -mkdir -p /delta_lake/raw$ hadoop fs -put blogs/delta-lake/products_aug20.csv /delta_lake/raw
The complete notebook is available at /delta_lake/delta_lake-demo-1.ipynb . Let me run through each step below with explanations:
Start the Spark session first with the Delta Lake package and then import the Python APIs
from pyspark.sql import SparkSessionimport pysparkfrom pyspark.sql.functions import *spark = pyspark.sql.SparkSession.builder.appName("Product_Price_Tracking") \  .config("spark.jars.packages", "io.delta:delta-core_2.12:0.7.0") \  .config("spark.sql.extensions", "io.delta.sql.DeltaSparkSessionExtension") \    .config("spark.sql.catalog.spark_catalog", "org.apache.spark.sql.delta.catalog.DeltaCatalog") \    .getOrCreate()from delta.tables import *
Create a Spark DataFrame using the recently received data — products_aug20.csv
df_productsaug20 = spark.read.csv('hdfs:///delta_lake/raw/products_aug20.csv', header=True, inferSchema=True)df_productsaug20.show()+---------+----------+-----+|ProductID|      Date|Price|+---------+----------+-----+|      200|2020-08-20| 20.5||      210|2020-08-20| 45.0||      220|2020-08-20|34.56||      230|2020-08-20|23.67||      240|2020-08-20|89.76|+---------+----------+-----+
Now lets store data in the Delta Lake. Delta Lake uses versioned Parquet files to store your data in your cloud storage. Additionally, a transaction log is stored to keep track of data changes over time.
df_productsaug20.write.format("delta").option("path", "hdfs:///delta_lake/products").saveAsTable("products")
Lets see how the date was stored in HDFS. Take note of the _delta_log directory that stores the changes over time.
Each commit = 1 JSON file starting with 00000000000000000000.json
Every 10 commits, a checkpoint is performed that combines previous JSON files into a parquet file.
$ hadoop fs -ls /delta_lake/productsFound 2 itemsdrwxr-xr-x   - mkukreja supergroup          0 2020-08-26 20:43 /delta_lake/products/_delta_log-rw-r--r--   2 mkukreja supergroup       1027 2020-08-26 20:43 /delta_lake/products/part-00000-37f5ec8d-5e21-4a01-9f19-7e9942196ef6-c000.snappy.parquet$ hadoop fs -cat /delta_lake/products/_delta_log/00000000000000000000.json2020-08-26 20:44:42,159 INFO sasl.SaslDataTransferClient: SASL encryption trust check: localHostTrusted = false, remoteHostTrusted = false{"commitInfo":{"timestamp":1598489011963,"operation":"CREATE TABLE AS SELECT","operationParameters":{"isManaged":"false","description":null,"partitionBy":"[]","properties":"{}"},"isBlindAppend":true,"operationMetrics":{"numFiles":"1","numOutputBytes":"1027","numOutputRows":"5"}}}{"protocol":{"minReaderVersion":1,"minWriterVersion":2}}{"metaData":{"id":"7788c86b-ae7e-47be-ac43-76c1f3f0506f","format":{"provider":"parquet","options":{}},"schemaString":"{\"type\":\"struct\",\"fields\":[{\"name\":\"ProductID\",\"type\":\"integer\",\"nullable\":true,\"metadata\":{}},{\"name\":\"Date\",\"type\":\"string\",\"nullable\":true,\"metadata\":{}},{\"name\":\"Price\",\"type\":\"double\",\"nullable\":true,\"metadata\":{}}]}","partitionColumns":[],"configuration":{},"createdTime":1598489010883}}{"add":{"path":"part-00000-37f5ec8d-5e21-4a01-9f19-7e9942196ef6-c000.snappy.parquet","partitionValues":{},"size":1027,"modificationTime":1598489011874,"dataChange":true}}$ hadoop fs -cat /delta_lake/products/_delta_log/00000000000000000000.json2020-08-26 20:44:42,159 INFO sasl.SaslDataTransferClient: SASL encryption trust check: localHostTrusted = false, remoteHostTrusted = false{"commitInfo":{"timestamp":1598489011963,"operation":"CREATE TABLE AS SELECT","operationParameters":{"isManaged":"false","description":null,"partitionBy":"[]","properties":"{}"},"isBlindAppend":true,"operationMetrics":{"numFiles":"1","numOutputBytes":"1027","numOutputRows":"5"}}}{"protocol":{"minReaderVersion":1,"minWriterVersion":2}}{"metaData":{"id":"7788c86b-ae7e-47be-ac43-76c1f3f0506f","format":{"provider":"parquet","options":{}},"schemaString":"{\"type\":\"struct\",\"fields\":[{\"name\":\"ProductID\",\"type\":\"integer\",\"nullable\":true,\"metadata\":{}},{\"name\":\"Date\",\"type\":\"string\",\"nullable\":true,\"metadata\":{}},{\"name\":\"Price\",\"type\":\"double\",\"nullable\":true,\"metadata\":{}}]}","partitionColumns":[],"configuration":{},"createdTime":1598489010883}}{"add":{"path":"part-00000-37f5ec8d-5e21-4a01-9f19-7e9942196ef6-c000.snappy.parquet","partitionValues":{},"size":1027,"modificationTime":1598489011874,"dataChange":true}}
This is how we can query recently saved data in Delta Lake using Spark SQL
spark.sql('SELECT * FROM products').show()+---------+----------+-----+|ProductID|      Date|Price|+---------+----------+-----+|      200|2020-08-20| 20.5||      210|2020-08-20| 45.0||      220|2020-08-20|34.56||      230|2020-08-20|23.67||      240|2020-08-20|89.76|+---------+----------+-----+
You can query previous snapshots of your Delta table by using time travel. If you want to access the data before it was overwritten, you can query a snapshot of the table using the versionAsOf option.
deltaTable.update("ProductID = '200'", { "Price": "'48.00'" } )df = spark.read.format("delta").option("versionAsOf", 1).load("hdfs:///delta_lake/products")df.show()# Notice the value of Price for ProductID=200 has changed in version 1 of the table+---------+----------+-----+|ProductID|      Date|Price|+---------+----------+-----+|      200|2020-08-20| 48.0|  |      210|2020-08-20| 45.0||      220|2020-08-20|34.56||      230|2020-08-20|23.67||      240|2020-08-20|89.76|+---------+----------+-----+df = spark.read.format("delta").option("versionAsOf", 0).load("hdfs:///delta_lake/products")df.show()# Notice the value of Price for ProductID=200 is the older snapshot in version 0+---------+----------+-----+|ProductID|      Date|Price|+---------+----------+-----+|      200|2020-08-20| 20.5||      210|2020-08-20| 45.0||      220|2020-08-20|34.56||      230|2020-08-20|23.67||      240|2020-08-20|89.76|+---------+----------+-----+
Lets perform another DML operation, this time delete ProductID=210.
deltaTable.delete("ProductID = 210") df = spark.read.format("delta").option("versionAsOf", 2).load("hdfs:///delta_lake/products")df.show()# Notice the value of Price for ProductID=210 is missing in Version 2+---------+----------+-----+|ProductID|      Date|Price|+---------+----------+-----+|      200|2020-08-20| 48.0||      220|2020-08-20|34.56||      230|2020-08-20|23.67||      240|2020-08-20|89.76|+---------+----------+-----+
Notice that the Transaction Log has progressed, one log per transaction
$ hadoop fs -ls /delta_lake/products/_delta_logFound 3 items-rw-r--r--   2 mkukreja supergroup        912 2020-08-21 13:14 /delta_lake/products/_delta_log/00000000000000000000.json-rw-r--r--   2 mkukreja supergroup        579 2020-08-24 11:03 /delta_lake/products/_delta_log/00000000000000000001.json-rw-r--r--   2 mkukreja supergroup        592 2020-08-24 11:14 /delta_lake/products/_delta_log/00000000000000000002.json
Pay special attention to the operation attribute in JSON of the transaction logs
$ hadoop fs -cat /delta_lake/products/_delta_log/*.json{"commitInfo":{"timestamp":1598489978902,"operation":"CREATE TABLE AS SELECT","operationParameters":{"isManaged":"false","description":null,"partitionBy":"[]","properties":"{}"},"isBlindAppend":true,"operationMetrics":{"numFiles":"1","numOutputBytes":"1027","numOutputRows":"5"}}}{"protocol":{"minReaderVersion":1,"minWriterVersion":2}}{"metaData":{"id":"47d211fc-7148-4c85-aa80-7d9aa8f0b7a2","format":{"provider":"parquet","options":{}},"schemaString":"{\"type\":\"struct\",\"fields\":[{\"name\":\"ProductID\",\"type\":\"integer\",\"nullable\":true,\"metadata\":{}},{\"name\":\"Date\",\"type\":\"string\",\"nullable\":true,\"metadata\":{}},{\"name\":\"Price\",\"type\":\"double\",\"nullable\":true,\"metadata\":{}}]}","partitionColumns":[],"configuration":{},"createdTime":1598489977907}}{"add":{"path":"part-00000-8c43a47a-02bf-4bc2-a3be-aaabe9c409bd-c000.snappy.parquet","partitionValues":{},"size":1027,"modificationTime":1598489978816,"dataChange":true}}2020-08-26 21:07:41,120 INFO sasl.SaslDataTransferClient: SASL encryption trust check: localHostTrusted = false, remoteHostTrusted = false{"commitInfo":{"timestamp":1598490235896,"operation":"UPDATE","operationParameters":{"predicate":"(ProductID#609 = 200)"},"readVersion":0,"isBlindAppend":false,"operationMetrics":{"numRemovedFiles":"1","numAddedFiles":"1","numUpdatedRows":"1","numCopiedRows":"4"}}}{"remove":{"path":"part-00000-8c43a47a-02bf-4bc2-a3be-aaabe9c409bd-c000.snappy.parquet","deletionTimestamp":1598490235237,"dataChange":true}}{"add":{"path":"part-00000-272c0f65-433e-4901-83fd-70b78667ede0-c000.snappy.parquet","partitionValues":{},"size":1025,"modificationTime":1598490235886,"dataChange":true}}2020-08-26 21:07:41,123 INFO sasl.SaslDataTransferClient: SASL encryption trust check: localHostTrusted = false, remoteHostTrusted = false{"commitInfo":{"timestamp":1598490393953,"operation":"DELETE","operationParameters":{"predicate":"[\"(`ProductID` = 210)\"]"},"readVersion":1,"isBlindAppend":false,"operationMetrics":{"numRemovedFiles":"1","numDeletedRows":"1","numAddedFiles":"1","numCopiedRows":"4"}}}{"remove":{"path":"part-00000-272c0f65-433e-4901-83fd-70b78667ede0-c000.snappy.parquet","deletionTimestamp":1598490393950,"dataChange":true}}{"add":{"path":"part-00000-73a381e1-fa68-4323-81b9-c42dea484542-c000.snappy.parquet","partitionValues":{},"size":1015,"modificationTime":1598490393946,"dataChange":true}}
It is next day now and you received a new data file. Now lets merge the new data set from the next day (products_aug21.csv) into the Delta Lake
$ hadoop fs -put csv/products_aug21.csv /delta_lake/raw$ hadoop fs -ls /delta_lake/rawFound 2 items-rw-r--r--   2 mkukreja supergroup        132 2020-08-21 13:00 /delta_lake/raw/products_aug20.csv-rw-r--r--   2 mkukreja supergroup        220 2020-08-24 11:33 /delta_lake/raw/products_aug21.csv
Perform an Upsert Operation. This means that if the data in the new file (products_aug21.csv) matches (based on the join condition on ProductID) any existing data in the Delta Lake the update the price, else insert the new row.
df_productsaug21 = spark.read.csv('hdfs:///delta_lake/raw/products_aug21.csv', header=True, inferSchema=True)df_productsaug21.show()deltaTable.alias("products").merge(    df_productsaug21.alias("products_new"),                    "products.ProductID = products_new.ProductID") \                    .whenMatchedUpdate(set = { "Price" : "products_new.Price" } ) \                    .whenNotMatchedInsert(values =                       {                        "ProductID": "products_new.ProductID",                        "Date": "products_new.Date",                        "Price": "products_new.Price"                       }                     ).execute()
Check the latest version of the table after the Upsert. You may notice that ProductID=240 went though the whenMatchedUpdate operation whereas ProductID=240 till 280 went through the whenNotMatchedInsert operation.
spark.table("products").show()+---------+----------+-----+|ProductID|      Date|Price|+---------+----------+-----+|      230|2020-08-20|23.67||      210|2020-08-21| 46.0||      250|2020-08-21|89.76||      220|2020-08-20|34.56||      240|2020-08-20|90.82||      200|2020-08-20| 25.5||      260|2020-08-21|54.55||      280|2020-08-21|44.78||      270|2020-08-21|96.32|+---------+----------+-----+
For every 10 commits Delta Lake saves a checkpoint file in Parquet format in the _delta_log sub-directory. Since the checkpoint file is in Parquet format it allows Spark to perform a faster read compared to inefficient JSON files.
Lets see how this happens by updating a few more rows:
deltaTable.update("ProductID = '230'", { "Price": "'33.67'" } )deltaTable.update("ProductID = '210'", { "Price": "'56.00'" } )deltaTable.update("ProductID = '250'", { "Price": "'99.76'" } )deltaTable.update("ProductID = '220'", { "Price": "'44.56'" } )deltaTable.update("ProductID = '240'", { "Price": "'100.82'" } )deltaTable.update("ProductID = '200'", { "Price": "'35.5'" } )deltaTable.update("ProductID = '260'", { "Price": "'64.55'" } )deltaTable.update("ProductID = '280'", { "Price": "'54.78'" } )deltaTable.update("ProductID = '270'", { "Price": "'106.32'" } )
Check the _delta_log sub-directory. After 10 commits a 00000000000000000010.checkpoint.parquet file has been created. Checkpoint files saves the state of the table at a point in time in Parquet format so that it can retrieve the history very efficiently.
$ hadoop fs -ls /delta_lake/products/_delta_logFound 15 items-rw-r--r--   2 mkukreja supergroup        912 2020-08-21 13:14 /delta_lake/products/_delta_log/00000000000000000000.json-rw-r--r--   2 mkukreja supergroup        579 2020-08-24 11:03 /delta_lake/products/_delta_log/00000000000000000001.json-rw-r--r--   2 mkukreja supergroup        592 2020-08-24 11:14 /delta_lake/products/_delta_log/00000000000000000002.json-rw-r--r--   2 mkukreja supergroup       2255 2020-08-24 11:39 /delta_lake/products/_delta_log/00000000000000000003.json-rw-r--r--   2 mkukreja supergroup        578 2020-08-24 12:05 /delta_lake/products/_delta_log/00000000000000000004.json-rw-r--r--   2 mkukreja supergroup        578 2020-08-24 12:05 /delta_lake/products/_delta_log/00000000000000000005.json-rw-r--r--   2 mkukreja supergroup        578 2020-08-24 12:05 /delta_lake/products/_delta_log/00000000000000000006.json-rw-r--r--   2 mkukreja supergroup        578 2020-08-24 12:05 /delta_lake/products/_delta_log/00000000000000000007.json-rw-r--r--   2 mkukreja supergroup        578 2020-08-24 12:05 /delta_lake/products/_delta_log/00000000000000000008.json-rw-r--r--   2 mkukreja supergroup        578 2020-08-24 12:05 /delta_lake/products/_delta_log/00000000000000000009.json-rw-r--r--   2 mkukreja supergroup      14756 2020-08-24 12:05 /delta_lake/products/_delta_log/00000000000000000010.checkpoint.parquet-rw-r--r--   2 mkukreja supergroup        578 2020-08-24 12:05 /delta_lake/products/_delta_log/00000000000000000010.json-rw-r--r--   2 mkukreja supergroup        579 2020-08-24 12:05 /delta_lake/products/_delta_log/00000000000000000011.json-rw-r--r--   2 mkukreja supergroup        579 2020-08-24 12:05 /delta_lake/products/_delta_log/00000000000000000012.json-rw-r--r--   2 mkukreja supergroup         25 2020-08-24 12:05 /delta_lake/products/_delta_log/_last_checkpoint
Lots of good stuff. Having played in the Data Engineering and Data Science world for several years, I can safely tell you that these features are no short of a life-savor. No more jumping through hoops to store and display the latest change data sets from the databases.
In the next part, we will deep dive into some advanced topics of Delta Lake including Partitioning, Schema Evolution, Data Lineage & Vacuum.
I hope this article was helpful. Delta Lake is covered as part of the Big Data Hadoop, Spark & Kafka course offered by Datafence Cloud Academy. The course is taught online by myself on weekends.