Stateful Feature Backfill by Ratul Ghosh and Harish Nagu Sana
Channel: Apache Beam
Date: 10/12/2022

In this session, Ratul Ghosh and Harish Nagu Sana explain how backfill or state bootstrapping is an essential capability for streaming applications where data retention for the source is limited to only a few days.
This capability enables Data Scientists, MLEs, and Data Engineers to replay the feature-extraction streaming pipeline from the archive. Some critical use cases can be state-bootstrapping or iterating the featurization logic to fine-tune the ML model training and inference, etc.
The key capability of the Apache Beam-based capability is transferring the internal state from backfill (batch) to stream.