basho/riak_kv — pull request #425 by beerriot (2012-11-03 15:06:04+00:00)
Title: Apply backpressure from the MR sink

This PR uses the new ability of basho/riak_pipe#59 to provide backpressure to the MR pipe in the new MR sink introduced by #416. This PR is targeted at the `bwf-sep-sink-2` branch to more clearly demonstrate the changes made here until #416 is merged to master.

In addition to using the `fsm_sync` sink type to synchronously receive pipe results/logs, the sink now has a capped-size buffer. Once that buffer is full, the sink delays acks to result/log senders until the sink's owner empties its buffer. This is to keep the problem of the expanding mailbox from becoming the problem of the expanding heap. The size of this buffer is configurable globally as the `riak_kv` app env `mrc_sink_buffer` or as a startup option for the FSM, `buffer`.

The changes to `src/riak_kv_mrc_pipe.erl` look more dramatic than they really are because enforcing the use of `riak_kv_mrc_sink` in all MR pipelines has cleaned up the logic of output collection and sorting.

This branch has not yet been tested for performance changes. It is likely that latency will take a hit, due to the extra synchronicity. The safety of not blowing up memory consumption in a sink process is likely desirable enough to counter worries. There is also work underway (in the `bwf-pmi` branch) to enable delivery of multiple outputs in a single message.