Adding pluggablility to work allocation, along with a new work allocation strategy to group small topics partitions within the same task.

The purpose of the new work allocation is to reduce the overall output file count.  Also included in this request is some optimizations for the daily sweeper code.
We do, but whitelisting wasn't working.  Using it in listStatus means every directory that isn't in the whitelist isn't returned.  including "daily" and "hourly" directories.  Now the filter is only matched against the parent directory of the source subdir when it's found.