nathanmarz/storm — Issue #79: TransactionalSpout
Author: nathanmarz
Created: 2011-12-18 02:16:31+00:00
State: open
Labels: New feature

This is an idea for a new kind of spout that works differently than regular spouts. Each spout task emits batches of tuples, and they don't emit the next batch until all tuples in all batches have been completed. The task with the lowest id is designated as the coordinator task, and it will do a 2PC or 3PC protocol to ensure the distributed batch is all committed at once. If there are any failures, the entire batch is replayed.

Additionally, a batch id will be provided to the spout when emitting that can be part of the tuple stream. This allows downstream bolts to be aware of what "batch of computation" it's working on. So for example, if updating a count in a database, a bolt can include the batch id with the count and only increment the count when the batch id has changed.

This will enable many more idempotent topologies to be built on top of Storm. The transactional nature of this kind of spout of course will necessitate a tradeoff in performance to achieve this idempotence.

The initial target for a TransactionalSpout implementation will be ApacheKafka.