Data LakeHouse — Paradigm of the Decade
By: Hassan Rahamathullah
Tags: Artificial Intelligence, Cloud, Data Engineering, Data Visualization, Data Science

What is Data LakeHouse?

Data LakeHouse is the new term in the Data platform architecture paradigm. LakeHouse is like the combination of both Data Lake and Data Warehouse (obviously from the term you might have understood). In Data LakeHouse, you will have processing power on top of Data Lakes such as S3, HDFS, Azure Blob, etc.

High-level Data LakeHouse data flow

In other words, you don’t have to load the data onto any of the data warehouses to process and get the analysis or Business intelligence requirement done. You can directly query the data underlying in your data lakes made of object storages or Hadoop. This method decreases the operational overhead on Data Pipelining and maintenance.

Advantage of Data LakeHouse

Some of the advantages of having Data LakeHouse are:

Elimination of simple ETL jobs:

In the Data warehousing technique, the data has to be loaded into Data Warehouse to Query or to perform analysis. For example, loading the data from your existing Data Lake to the Data Warehouse by cleaning and transforming it into the destination schema using some of the ETL/ELT tools. But using the Data LakeHouse tool, the ETL process will be eliminated by connecting the query engine directly to your Data Lake.

Reduced Data Redundancy

Data LakeHouse removes Data Redundancy. For Example, You have Data on multiple tools and platforms such as cleaned data on Data warehouse for processing, some meta-data on Business intelligence tool, temporary data on ETL tools, etc. These Data has to be maintained and monitored continuously to avoid any data sanity issues. So if you are using a single tool to process out of your raw data, you can overcome this data redundancy problem.

Ease of Data Governance

Data LakeHouse can eliminate the operational overhead of managing Data governance on multiple tools. If you are handling sensitive data, you have to be careful when you are transferring data from one tool to another, so that each tool can maintain the proper access controls and encryption. But if you use a single Data LakeHouse tool, Data Governance management can be done from one-point.

Directly connect to BI tools

Data LakeHouse enabling tools such as Apache Drill, supports the direct connection to some of the popular BI tools like (Tableau, PowerBI, etc.). It eventually reduces the time taken from Raw data to visualization by exponential times.

BI Experts Reaction. Source: GIPHY

Cost reduction

In Data Lake and warehousing paradigm, Data has to be stored in multiple places (Data Lake and Data Warehouse) and the cost for the storage is also high. Comparatively, the Data LakeHouse can store the data in cheap storage like object storage such as S3, Blob, etc.

Why would Data Scientists love Data LakeHouse?

Data Scientists always love to play on the raw data (just a thought!), so that they can get more insights of data pattern and develop the models out of it. When the data warehousing technique was followed, Data scientists should also understand ETL/ELT pipelines so that they would have good visibility on how the data was transformed, loaded and validated. But when they have a query engine to query the data directly from raw data, they will get a superpower to build their transformation logics and cleaning techniques after understanding basic statistical insights and quality of the raw data.

Data Scientist with Data LakeHouse (Source: GIPHY)

Tools Available

There is some good Data LakeHouse enabling tool by proprietary cloud vendors and open source communities. Some of them are:

Google Bigquery

Google Bigquery is modern data warehousing platform by Google Cloud Platform. Bigquery(BQ) enables us to follow the Data Lakehouse concept by abstracting the storage and computation layer completely. BQ pricing is based on the data processed per query and not based on storage like other competitors. Unknowingly knowing the new paradigm, most of the organizations started adopting the Data Lakehouse concept by using Google’s Bigquery.

A serverless, highly scalable, and cost-effective cloud data warehouse designed to help you make informed decisions quickly, so you can transform your business with ease — Google https://cloud.google.com/bigquery

Apache Drill

Apache Drill is the Schema-free distributed query engine. It will contribute as the best query engine if you want to create your Data LakeHouse. It also supports cross-format querying like joining JSON and CSV data using one query.

“Drill is an Apache open-source SQL query engine for Big Data exploration. Drill is designed from the ground up to support high-performance analysis on the semi-structured and rapidly evolving data coming from modern Big Data applications, while still providing the familiarity and ecosystem of ANSI SQL, the industry-standard query language. Drill provides plug-and-play integration with existing Apache Hive and Apache HBase deployments.” — Apache Drill https://drill.apache.org/

Amazon Athena

Amazon Athena is part of AWS managed services. Athena is an interactive serverless query engine based on Facebook’s presto. It enables the user to query the data directly from S3 after creating the schema definition. You can do the automated schema discovery using the AWS Glue crawler. (Don’t Worry, I understand the pain)

“Amazon Athena is an interactive query service that makes it easy to analyze data in Amazon S3 using standard SQL.” — AWS page https://aws.amazon.com/athena/

Delta Lake

Delta lake is the open-source Data LakeHouse enabling tool that helps us to leverage our processing power of pre-built/pre-owned spark infrastructure. It also enables the ACID methodology on the Distributed storage. Delta lake is founded by the founders of Spark and Databricks. (hail spark!)

“Delta Lake is an open-source storage layer that brings ACID transactions to Apache Spark™ and big data workloads.” — delta.io https://www.delta.io/

and there are other tools like Data Lake analytics on Microsoft Azure etc.

Conclusion

Concept of Data LakeHouse is at an early stage, so there are some limitations to be considered before completely depending on the Data LakeHouse architecture such as query compatibility, Data Cleaning complexity, etc. But Data Engineers (readers like you) can contribute to the issues and limitations on open-source tools. Bigger companies like Facebook, Amazon has already set the base for Data LakeHouse and open-sourcing the tools they use. You can quickly start experimenting by deploying the apache drill on your laptop and connect to your existing data lake to create your first Data LakeHouse setup. (beware of querying large data sets *smokes coming out of laptop*).

Good Read on Data LakeHouse — https://databricks.com/blog/2020/01/30/what-is-a-data-lakehouse.html