InMobi/data-bus — Issue #4: DataConsumer tmpPath cleanup should happen on each run
Author: inders
Created: 2012-01-02 07:46:24+00:00
State: open

Issue1 -

1. Today tmpPath for dataConsumer is created once and is not being cleaned once the iteration is over.
2. Due to some issue if some files are left in it(i.e. the process didn't scucceed) next runs gets more files than excepted and commit doesn't happen going forward as there is a check between #no of files in input to be moved == #no of files compressed.
3. Fix - to cleanup the tmpoutPath on each run of data consumer.

Issue2 -

1. Today when remote puller starts and has to pull multiple files from remote cluster, it creates one file in the remotePath which has the list of all paths to be pulled. If distcp fails after this then this file is not cleaned up and next distcp run will consider this file also as input thereby keep failing as there will be duplicate file names

Fix - this file should be created in a destination cluster where data is being pulled in a timestamp based name inside tmp