Not less, Not more. Exactly Once Large-Scale Stream Processing in Action.
Channel: FOSDEM
Date: 2018-03-06

by Paris Carbone

At: FOSDEM 2017

Large-scale data stream processing has come a long way to where it is today.It combines all the essential requirements of modern data analytics: subsecondlatency, high throughput and impressively, strong consistency. Apache Flink isa system that serves as a proof-of-concept of these characteristics and it ismainly well-known for its lightweight fault tolerance. Data engineers andanalysts can now let the system handle Terabytes of computational statewithout worrying about failures that can potentially occur.

In this talk, I am going to explain all the fundamental challenges behindexactly-once processing guarantees in large-scale streaming in a simple andintuitive way. I will further guide you through the tricks and pitfalls thatwe faced in our effort to make state management easy to use, transparent andyet extraordinarily powerful. Finally, I will demonstrate how you can declarestate and the in-flight protocol that is running underneath your processingpipeline to guarantee that your computation will always run consistently anduninterrupted, until infinity.

In more detail I am going to demonstrate the basic and extended versions ofABS (Asynchronous Barrier Snapshotting) , our state-of-the-art decentralized,in-flight snapshotting algorithm tailored to the needs of a dataflow graph.ABS needs no global synchronisation and works purely in a decentralisedmanner. It is considered today as one of the most lightweight mechanisms forcheckpointing application state and has been developed even further togetherwith the rest of the Flink community with additional optimisations. In fact,other open source systems such as Apache Storm have also recently incorporatedthe same mechanism due to its simplicity and effectiveness. One of suchoptimisations that is currently being merged in the system, is the adaptationof the algorithm in graphs that contain cycles which I am also going to coverin this talk in more detail. Furthermore, I will also briefly explain ourcurrent efforts to incrementalize state snapshots to further balance the loadtrade-off of state checkpointing and recovery time.


Room: H.2213
Scheduled start: 2017-02-04 17:00:00