InMobi/data-bus — Issue #6: Databus stalled usecases 
Author: inders
Created: 2012-02-21 09:00:21+00:00
State: open

LocalStreamService, MergedStreamService, MirrorStreamService use the following commit Protocol to guarantee 0% data loss

COMMIT Protocol
-------------------------
1. Producer writes the data getting published in a META file
2. producer publishes the data.
3. Consumer reads the data to PULL from the published file in [1]
4. Consumer pulls the data, publishes it in an atomic fashion.
5. Consumer deletes/commits the META file in [1]

When can databus get stalled.
------------------------------------------
1. producer dies after[1] and before [2]
2. Producer's in DATABUS are highly available through ZooKeeper
3. Another producer starts and follows the COMMIT protocol described above to publish more data.
4. All Consumers in DATABUS(MergedStreamService, LocalStreamService) use DISTCP(augmented to support -preserveSourcePath - https://github.com/InMobi/DistCpV2-0.20.203) with -f option
5. DISTCP with -f option errors out if one of the path in the file list isn't present at SOURCE.

SideEffect - Databus (MergedStreamService, MirrorStreamService can stop consuming data thereby resulting in DATABUS getting stalled.

Current Solution - Have operational monitoring on /databus/system/mirrors, /databus/system/consumers directories for each instance of databus. If #files in these directories exceed a threshold(10) then stop databus. cat these files to produce a single file in the respective directory by checking whether the file exists on HDFS else skip it. Start DATABUS.