C U S TO M E R S TO R Y

Simplifying reliable
ETL on Delta Lake with
Delta Live Tables

Simplifying reliable ETL development on
Databricks Delta Lake
Shell optimizes trillions of rows of IoT sensor data with Delta Live Tables

Shell is at the forefront of creating a cleaner tomorrow by investing in digitaltechnologies to tackle climate change and become a net-zero emissions energy
business. Essential to the strategy is managing trillions of IoT sensor data comingfrom the cloud and field signals, that help drive insights to enable greener, costeffective operations. Shell needed the ability to scale ETL to enable machine learning(ML) and AI across varying data sources and types. To meet key pipeline requirements,Shell chose to leverage Delta Live Tables, built on the Databricks Lakehouse Platform,
for automated ETL development and management on Delta Lake.

The challenges of managing ETL at scale
Reliable, large-scale ETL can be complex and brittle. With trillions of rows of IoT
sensor data, varying in type and source, Shell sought an efficient solution forprocessing that data. Sophisticated data pipelines are hard to build, but necessaryfor profit growth, analytics, data science and machine learning. Dan Jeavons, GM ofdata science at Shell said, “Digital technology is absolutely core to making our existing
business more effective and efficient. As the industry continues to expand into newareas of energy that are more sustainable and reduce environmental impact, dataand digital technology are now table stakes.”
In order to deliver and leverage trusted data to all of Shell’s workstreams — from

1.3 trillion
rows of sensor dataprocessedMillionsof dollars saved with
data-driven operations

predictive maintenance, drone missions, and digital twins, to hydrogen efficiency,
the Gold+ loyalty program, and clean production optimization — Shell needed
production-grade pipelines that can deploy worldwide and at scale. Shell identified
six key requirements for its data pipelines: connectors, transformation and AI,quality, recovery, scale, and monitoring.

DATABRICKS CUSTOMER STORY: SHELL

70+ AI-driven
transformationuse cases2

Meeting pipeline needs withreliability and automation
With the Databricks Lakehouse Platform as the foundation

Beyond saving time and money, Shell’s ETL pipelines

for their Shell.ai Platform, they looked to Databricks

can be reused regardless of environment, and data canto help streamline and simplify data management andrun in batches or streams with incremental or complete

processing through ETL. With Delta Live Tables, Shell cancomputation per table. Delta Live Tables also increase

now easily ingest data into Delta Lake and automaticallythe accuracy and reliability of the data with validation andbuild SQL endpoints for real-time queries and multi-layer

integrity checks, and predefined error policies.

insights into data. Equipped with a Delta-centric service,
Shell has significantly reduced pipeline complexity,“With this capability augmenting the existingelevated software engineering standards, improved

lakehouse architecture, Databricks is disrupting

pipeline performance, and gained new data insights.

the ETL and data warehouse markets, which isimportant for companies like ours.”

By building intelligence into the data ingestion process,Shell is saving millions of dollars with data-driven

DAN JEAVONS | GM of Data Science, Shell

operations and innovative opportunities. By simplifying
ETL development and management on Delta Lake, Shell

The end result is deeper pipeline visibility anddata engineers can now build and leverage their own ETLdata knowledge, reduced downtime with automatic

pipelines, and gain insights and analytics quickly.

error handling, and collaborative ETL between
Shell’s applications.

Live Tables understands
and coordinates data flow
between your queries
L i v e Ta b l e s P i p e l i n eData

CREATE LIVE
RawTABLE raw_data as SELECT * FROM cloud_files(“/data”, “json”)

raw_data

CREATE LIVETABLE clean_data as SELECT ... FROM LIVE.raw_data
Clean

clean_data@dlt.tabledef scored_records()
Scoredreturn read(“clean_data”).|a(la|bda d |odel_score)d))scored_records

Delta Live Tables makes it easy to build and manage reliable data pipelines that deliver high-quality data on Delta Lake.

DATABRICKS CUSTOMER STORY: SHELL

3

Millions in savings withETL simplicity
Shell is taking advantage of data and ML availability

Additionally, Delta Live Tables is driving betterfor AI-driven transformation across 70+ use cases —software engineering standards around pipeline

including operations, product development, marketing,development, improving pipeline performanceand customer experience. Delta Live Tables has madeand data integrity, providing real-time query andit possible for Shell to overcome the challenges ofdata insights, spurring innovation with facial

processing 1.3 trillion rows of streaming IoT data fromrecognition and ML libraries, plus, saving Shell

millions of sensors. With clean data flowing freely

millions of dollars with task automation,

downstream, Shell’s data teams can now harness

ETL simplicity, and data-driven operations.

actionable insights and accelerate ML implementation.
Dan explains, “At Shell, we are aggregating all our sensor
data into an integrated data store — working at themulti-trillion-record scale. Delta Live Tables has helped
our teams save time and effort in managing data at thisscale. We have been focusing on continuously improving

Going forward, Dan says, “We are excited to continue towork with Databricks as an innovation partner.” Utilizing
Databricks to unlock the promise of data and AI, it’s clearthat Shell will more easily reach its goal of creating acleaner tomorrow.

our AI engineering capability and have an Integrated
Development Environment (IDE) with a graphical interface
supporting our Extract Transform Load (ETL) work.”

Databricks is disrupting the ETL and data
warehouse markets, which is importantfor companies like ours. We are excited
to continue to work with Databricks as aninnovative partner.

DA N JEAVON S
GM of Data Science, Shell

DATABRICKS CUSTOMER STORY: SHELL

4

About Databricks
Databricks is the data and AI company. Thousands of organizations worldwide —including Showtime, Shell, Conde Nast andRegeneron — rely on Databricks’ open and unified platform for data engineering, machine learning and analytics. Databricks
is venture-backed and headquartered in San Francisco with offices around the globe. Founded by the original creators ofApache Spark™, Delta Lake and MLflow, Databricks is on a mission to help data teams solve the world’s toughest problems.
To learn more, follow Databricks on Twitter, LinkedIn and Facebook.

E VA LUATE DATAB R IC KS FOR YOU R S EL FWAT C H V I D E O

LEARN MORE

Contact us for a personalized demo databricks.com/contact© Databricks Inc. 2021. All rights reserved. Apache, Apache Spark, Spark and the Spark logo are trademarks of the Apache Software Foundation. Privacy Policy | Terms of Use