How Delta Lake shapes your NextGen Data Landscape - Lakehouse with Delta Lake (Part 2)
Channel: Data with Nikk the Greek
Date: 2024-04-03

Learn everything you need to know about the Lakehouse concept and Delta Lake for your next Gen Data Landscape! Check it out! 😊 

Data Warehouse architectures are nowadays outdated or at least reach the limits of today's data requirements. Adding cheap, flexible cheap cloud storage towards a two-tier architecture made things more chaotic. Even though now machine learning requirements towards file-based data storage and unstructured data could be satisfied.  

The Data Lakehouse is a new data concept leveraging the benefits of data warehouses like ACID, updates/merges/deletes and schema enforcement on a cheap and flexible cloud storage. This requires of course a metadata framework like Delta Lake. In the past three years numerous leverage this concept successfully. 

What we will learn:
- What is a Lakehouse
- How does Parquet work and what can't it fulfil
- How does Delta Lake work
- The Delta Lake Ecosystem

~~~~~~~~~~~~~ *Subscribe - Like - Comment - Challenge* ~~~~~~~~~~~~~

You want to master Data Engineering with PySpark? Subscribe here: https://www.youtube.com/@DataNikktheGreek?sub_confirmation=1 

Feel free to comment or challenge my explanations as always. Happy to learn also myself more by the community.

~~~~~~~~~~~~~~~~~~~~~~~ *Resources* ~~~~~~~~~~~~~~~~~~~~~~~

Link to Slides: https://github.com/datanikkthegreek/SparkDeltaDatabricksInternals/blob/main/Season4%20-%20Delta%20Lake/S04E02_DeltaIntro.pdf

Delta Homepage: https://delta.io/
Parquet vs. Delta: https://delta.io/blog/delta-lake-vs-parquet-comparison/
The famous Lakehouse paper: https://www.cidrdb.org/cidr2021/papers/cidr2021_paper17.pdf
Recent developments on Delta Lake: https://delta.io/blog/state-of-the-project-pt1/
The Delta Lake Book: https://delta.io/pdfs/DLTDG_ER4.pdf

~~~~~~~~~~~~~~~~~~~~~~~ *Chapters* ~~~~~~~~~~~~~~~~~~~~~~~

00:00 - Introduction
01:20 - What is the Lakehouse?
06:32 - Storage Frameworks as Meta Data Layer
07:20 - Parquet Features and Limitations
12:51 - Delta Lake Features
19:04 - Delta Lake Low Level Features
22:33 - Delta Lake Ecosystem
26:33 - Summary

#spark #pyspark #lakehouse #datalakehouse #deltalake #dataengineering #dataengineeringessentials