improving sweeper support for large number of input files
Changes to support new partitioning that produces fewer output files
Changes to support new partitioning that produces fewer output files
merged
adding priority queuing for sweeper
adding support for multiple priority levels in the sweeper code
removing unused imports
fixing current topic not being set
Merge pull request #119 from kengoodhope/master

Adding pluggablility to work allocation, along with a new work allocation strategy to group small topics partitions within the same task.