linkedin/camus — pull request #23 by alanctgardner (2013-08-13 21:50:38+00:00)
Title: Add a method to find the last offsets

Right now Camus seems to depend on an implicit feature of HDFS which returns directories in a particular order (by date created?) in order to find the offsets from the last time Camus was run. On other implementations, like MapR, this function doesn't work the same way, and so Camus will use some offsets from some other, random prior execution. The ideal solution is to sort the whole list of DateTimes, but that might be memory intensive if you have a large history. In the meantime this patch:

- prevents Camus from deleting the directory with the most recent history
- ensures Camus always uses the most recent offsets to pull new messages