InMobi/data-bus — Issue #1: File Name consistency across data hops
Author: inders
Created: 2011-12-08 14:59:33+00:00
State: open

1. Currently the data moves from /logs/ to /intermediateDir/ from where it's moved to the final destination. The intermediate hop is required to make the movement atomic along with a DONE file generation in the intermediate directory. The directory structure across data hops should be same to ease debugability.

DataMover is resilient to failures while moving data to intermediate directories. It has a cleanupMode which it enters on every run to fix any issues with Last Run.