twitter/summingbird — pull request #539 by ianoc (2014-08-22 16:38:00+00:00)
Title: Refactors the final flat map a bit

Made some lists Seq's since we don't rely on it being a list.

Fix a case where the hash code could be less than zero for the batching, would cause the space to double in size. Not a correctness bug thankfully, but good to know how settings are effecting things.

I've pulled the batching with this hash outside the aggregator, which lets the aggregator use sumOption to much better effect. Also now the aggregator will get better visibility on things. Does not seem to have impacted any performance negatively in tests.

Added a rate limiter option, which allows a slow start + min/max throughputs per second for the bolt. This is useful since we lack back pressure on storm to be able to stick a cap on the flat map throughput.