twitter/summingbird — pull request #296 by johnynek (2013-10-13 19:19:26+00:00)
Title: Scalding batch tests

The point of this code is to start testing scalding with a random batcher. Previously, we put everything in one batch so it was easy to write the test code (which needs more refactoring). Now, we generate a random batcher that tries up to 5 batches.

This exposed a lot of little assumptions in the testing, but no bugs in summingbird yet.

The next step is to use this better testing to verify optimizations to the scalding side don't break the logic.

First up: a better algorithm that puts all the heavy lifting into one Map/Reduce job (so run 100 batches in 1 MR step). Until we get template-tap support is scalding, after that step there will be a 100 concurrent Map-only jobs to write to the correct partitions, but that should be really fast, and will be solved when we close:

https://github.com/twitter/scalding/issues/654

or

https://github.com/twitter/scalding/issues/484