Actually there are two concerns: the period of time the whole job ran over, and the batching for the store. There may be many stores remember. Some might have different batchers. So each store needs to track when it was last written to disk.

You'll have to drop some file on the file system, or do an ls on the directories and choose the latest one. I think the latter approach is best: do an ls and pick the last directory, with a _SUCCESS file, that has the biggest batch.