Apache Hudi : The Data Lake Platform - Vinoth Chandar
Channel: The ASF
Date: 2021-10-11

Thank you to all of of our ApacheCon@Home 2021 sponsors, including:

STRATEGIC
---------------
Google

PLATINUM
--------------
Apple
Huawei
Instaclustr
Tencent Cloud

GOLD
-----------
Aiven OY
AWS
Baidu
Cerner
Didi Chuxing
Dremio
Fiter
Gradle
Red Hat
Replicated A presentation from ApacheCon@Home 2021
https://apachecon.com/acah2021

Originally open sourced in 2017, Hudi pioneered the modern transactional data lake movement as we know it. Hudi introduced a fundamentally new way of building a server-less, yet massively distributed database abstraction over lake storage, that has since been adopted by other projects in the space as well. At its core, Apache Hudi provides transactional updates/deletes and incremental change streams from tables on lake storage. Over the past couple years, the Hudi community has also steadily built a rich set of platform components, that enable users to go to production quickly with end-end solutions to common problems like data ingestion, compaction, CDC or incremental ETLs. 

In this talk, we provide a state-of-the-union update on the project, sharing the current platform components, how they interplay and the design choices therein. We then unveil a bold new vision for the project, that includes sorely missed platform components like caching/metaserver, expansion of existing capabilities like concurrency control, addition of new integration points with event streaming systems/OLAP engines. We then explain their roles in the platform and how they work together to unlock an open, universal data plane that is queryable from most query engines, from a user's perspective.