gatoatigrado/vimap — pull request #12 by gatoatigrado (2013-11-11 02:04:46+00:00)
Title: Auto-chunking pool

This adds a function

vimap.pool.fork_chunked

which, behind the scenes, batches up input sent to worker processes. Future work may do automatic chunk size detection, or provide an auto-tuning framework for it. For now, we just have an overridable default chunk size of 100.

For testing, two tests (ExceptionsTest, FuzzTest) were extended to simply use fork_chunked instead of fork; the only adjustment necessary was for handling errors returned (which should be well documented). An API test, and performance test are also added (the latter was run many times in a while-true loop, to make sure it isn't flaky).