basho/riak_kv — pull request #429 by beerriot (2012-11-06 17:41:00+00:00)
Title: Apply backpressure from the MR sink

This is a re-opening of #425. That PR closed when I deleted the branch for #416, which 425 targeted. This now targets master.

This PR uses the new ability of basho/riak_pipe#59 to provide backpressure to the MR pipe in the new MR sink introduced by #416. This PR is targeted at the bwf-sep-sink-2 branch to more clearly demonstrate the changes made here until #416 is merged to master.

In addition to using the fsm_sync sink type to synchronously receive pipe results/logs, the sink now has a capped-size buffer. Once that buffer is full, the sink delays acks to result/log senders until the sink's owner empties its buffer. This is to keep the problem of the expanding mailbox from becoming the problem of the expanding heap. The size of this buffer is configurable globally as the riak_kv app env mrc_sink_buffer or as a startup option for the FSM, buffer.

The changes to src/riak_kv_mrc_pipe.erl look more dramatic than they really are because enforcing the use of riak_kv_mrc_sink in all MR pipelines has cleaned up the logic of output collection and sorting.

This branch has not yet been tested for performance changes. It is likely that latency will take a hit, due to the extra synchronicity. The safety of not blowing up memory consumption in a sink process is likely desirable enough to counter worries. There is also work underway (in the bwf-pmi branch) to enable delivery of multiple outputs in a single message.

There are no "capabilities" or other "upgrade" facilities included here. Nodes running earlier versions of Riak will simply ignore the sink_type option, and send regular messages, as they do now, to the MR sink. This is fine - the sink will accept them, even though it cannot provide backpressure to those senders. Nodes running the new version will similarly send regular messages if they are working for a Pipe that was started by an older node (and therefore had not set sink_type).