Lakehouse Build-Out Package
Implement your lakehouse vision with Databricks Delta Lake

Executive summary
Combining all your data in a traditional data warehouse is an anti-pattern, which requires a lot of ETLs on a rigid platform thatmainly supports structured data geared toward mission-critical reporting and BI use cases. To leverage ML and AI effectively
and be a better, more data-driven organization, you need a unified repository that supports large and diverse data sets, includingsemi-structured and unstructured, in an open format with enterprise-grade reliability and performance. Welcome to lakehouse,a new architecture that combines the best elements of data lakes and data warehouses. This packaged offering from Databricks
will accelerate implementing one simplified platform for data analytics, data science and ML — setting the foundation for yourlakehouse vision.

Key outcomes
DataEngineering

BI and SQL

Real-TimeDS and 4L

Data Management and GovernanceOpen Data Lake

Lakehouse Platform
Simple • Open • Collaborative

Modern multi-hop pipeline
reference architecture per yourchosen scenario
One use case implementationof a multi-hop pipeline with amaximum of 10 tables in each layer(Bronze, Silver and Gold)
Reference consumer layer that
integrates with Databricks SQL
Extended options for additional
pipeline scenarios plus production
readiness

Overview
Build or optimize a Delta Lake as the structured transaction layer on top of low-cost storage following the most up-to-date bestpractices, laying the foundation of your lakehouse vision. The package addresses many of the common problems found with datalakes by leveraging Delta Lake and its associated architecture patterns. The package offers three tiers: Foundation, Extended andOptimized. Milestones and the outcome for each tier are produced by our prescriptive methodology, and each tier can be chained for
greater impact on your journey of realizing your lakehouse vision. See the Resources and schedule section for details.

Common challenges with data lakes and data warehouses
D ATA L A K E

D ATA WA R E H O U S E

Data integrity (failed writes)
Lack of consistency (multiple readers/writers)
Schema mismatch
Complex Lambda architecture

Mainly supports structured data
Not the most cost-efficient
Lacks open standards and formats
Largely only stores refined data sets

Key benefits of a lakehouse with Delta Lake
Greater data reliability and scalability
Unified batch and streaming
Modern data platform ready for your ML and AI initiatives

Data teams can collaborate and innovate faster
Simplifies data and AI with a single platform
Enjoy the data lake economics

Databricks lakehouse build-out process
O R G A NI Z E YO UR
L AKEHOUSEC O NF I G U R E
E T L P IP E L INE

E X T E ND R E F E R E N C E
IMP L E ME N TAT I O N

PRODUCTIONR E A D INE S S

Create zones(Bronze, Silver, Gold)

Source toBronze ingestion

Build second pipeline

Scale, optimization
and automation

Develop multi-hoparchitecture

Bronze, Silver to Goldreference pipeline

One additional Bronze,
Silver to Gold pipeline

Monitoring, alertingand security

Resources and schedule
FOUNDATION
3 WEEKS*
Target lakehouse architecture
Build a curated data pipeline
Develop Bronze, Silver and GoldEXTENDED
3 WEEKS*B uild a second curated
data pipeline
Develop Bronze, Silver and Gold*Up to 4 resources supporting the activity over a 3-week period**Up to 4 resources supporting the activity over a 5-week period
Prior to kickoff, be sure to review the readiness checklist and complete required tasks

Out of scope
Configuration and integration of non-Databricks products and systems
Data cleansing and solving data quality issues© Databricks Inc. 2021. All rights reserved.

OPTIMIZED
5 WEEKS**
Production readiness
CI/CD and automation
Orchestration and governance