Data Lake as Source to Apache Flink to Better Support Machine Learning Use Cases - S. Irwin, C. Tan
Channel: The ASF
Date: 2021-10-11

Thank you to all of of our ApacheCon@Home 2021 sponsors, including:

STRATEGIC
---------------
Google

PLATINUM
--------------
Apple
Huawei
Instaclustr
Tencent Cloud

GOLD
-----------
Aiven OY
AWS
Baidu
Cerner
Didi Chuxing
Dremio
Fiter
Gradle
Red Hat
Replicated A presentation from ApacheCon@Home 2021
https://apachecon.com/acah2021

Apache Flink allows us to backfill data from Kafka by rewinding Kafka offsets. However, in practice a wide variety of our kafka topics do not have infinite retention. Data that falls out of retention for these streams can not be directly used as a source for Flink. This limitation made it impossible for us to support use cases that involved processing or reprocessing historical data. In particular, Machine Learning use cases benefit from training data that involves longer time periods than we could provide with Kafka.

After surveying the state of the art, we were inspired by industry reports of transparent Dual Source Consumers for Data Lake and Kafka. We decided to leverage our Data Lake as a source for Apache Flink and build our own Dual Source Consumer with the goal of enabling Flink to read data that had fallen outside of Kafka retention.

In this talk we’ll introduce the motivations and use cases that led us to develop the Dual Source Consumer. We will focus on the technical trade offs we faced integrating with our existing Data Lake, and we will describe the operational challenges we faced in deploying our implementation in a multi-region environment. Lastly, we’ll discuss how enabling a two year backfill capability for our customer unblocked their use case and the implications for improved support of machine learning in streaming.