nathanmarz/storm — pull request #294 by barrywhart (2012-08-15 12:57:45+00:00)
Title: Change ShellSpout and ShellBolt so that returning task IDs to the spout or bolt is optional

After doing some profiling of a real-life topology implemented in Python, I found that a lot of time was spent communicating between ShellSpout or ShellBolt and the actual spout or bolt code in Python. After making the attached changes to ShellSpout and ShellBolt, and making a couple of changes to storm.py, the communication time was 2.5x faster.

The primary change to storm.py was adding an optional "need_task_ids" parameter to emitSpout() and emitBolt().

I followed this up by adding new storm.py functions emitManySpout() and emitManyBolt(). Using these functions to batch up the emission of tuples improved communication performance about another 25%.