InMobi/data-bus — Issue #7: Make DATABUS 0% data replay complaint.
Author: inders
Created: 2012-02-21 09:19:44+00:00
State: open

LocalStreamService, MergedStreamService, MirrorStreamService use the following commit Protocol to guarantee 0% data loss

COMMIT Protocol

1. Producer writes the data getting published in a META file
2. Producer publishes the data.
3. Consumer reads the data to PULL from the published file in [1]
4. Consumer pulls the data, publishes it in an atomic fashion.
5. Consumer deletes/commits the META file in [1]

When can DATA replay happen at Consumer's (MergedStream, MirrorStream)

1. Consumer does [4] and dies before [5]
2. Databus is highly available through Zookeeper so another instance starts and follows the same COMMIT protocol.
3. Since some data was published in step1 above however not committed that data will be replayed.

Proposed Solution/Design

A transaction in DATABUS consists of moving multiple paths and DATABUS can fail in between thereby making atomicity tricky.

1. COMMIT protocol can be enhanced to use ZK for remembering data as it's being published for each Thread.
(i) if any path exists in ZK which is already copied to DEST skip it and delete it from ZK
(ii) Record source and destination path in ZK node for this thread
(iii) do the move
(iv) Delete the path from ZK node

2. Now when DATABUS starts another instance through ZK if the first one fails it does following
(i) Get data to be pulled from consumer's
(ii) Follow the COMMIT protocol describe above in [1]

Scenarios when DATABUS fails

Fails after 1.(i) - no issues
Fails after 1.(ii) - No issues, next run will see this path isn't copied and this run will get this PATH( No replay)
Fails after 1.(iii) - Next run will see this PATH is already copied, skip this PATH and delete from ZK( Avoid data replay)
Fails after 1.(iv) - No issues