Merge pull request #429 from bwf-sink-backpressure

MapReduce now configures its sink to speak FSM and apply backpressure.

Squashed commit of the following:

commit ff52338a35c3c141e9133cbf26b7ec8b2f6a4503
Author: Bryan Fink <bryan@basho.com>
Date:   Sun Nov 11 12:07:31 2012 -0500

    follow rename of fsm_sync to fsm in riak_pipe

commit fe07f4670ab9ded6b173d0248756515328bc7569
Author: Bryan Fink <bryan@basho.com>
Date:   Sat Nov 10 23:00:45 2012 -0500

    require pipe-mr sink acks only every tenth message

    This will reduce overhead and lower latency, while still providing the
    possibility for backpressure. A sync-period of 10 means that worst-case
    10*(number of active vnodes) messages beyond the sink's cap will be
    allowed.

commit 686f2236aca7d983b0cf126ef4ca9c78ad362354
Author: Bryan Fink <bryan@basho.com>
Date:   Thu Nov 8 14:13:20 2012 -0500

    give jsonify_bkeys the whole list, not one element at a time

    Thank you, ruby client tests, for finding the incorrect behavior when
    given an empty query.

commit 273f8734c032adae75d2abbffd6761cb84f6f570
Author: Bryan Fink <bryan@basho.com>
Date:   Wed Nov 7 12:35:43 2012 -0500

    remove duplicate cleanup code while reverting _mrc_pipe:mapred_stream/1

    As Joe pointed out, the code for cleanup up monitors/messages from the
    asyn sender, sync, and timeout timer was duplicated everywhere. In
    addition, recent patches had altered the contract of
    riak_kv_mrc_pipe:mapred_stream.

    This patch solves both problems by moving the new sink stuff to
    riak_kv_mrc_pipe:mapred_stream_sink. Added to it is the unified startup
    of the async input sender. The duplicate cleanup code has been moved to
    riak_kv_mrc_pipe:cleanup_sink. Shared receive/aggregate code for
    riak_kv_mrc_pipe:mapred and riak_kv_wm_mapred has been unified in the
    riak_kv_mrc_pipe:receive_sink and :collect_sink functions.

commit 1d36aef99218633f40099b238f2c9de52acd1e0c
Author: Bryan Fink <bryan@basho.com>
Date:   Tue Nov 6 14:57:53 2012 -0500

    doc fixing: mapred_stream return val & --group_outputs/*

    riak_kv_mrc_pipe:group_outputs no longer exists

    riak_kv_mrc_pipe:collect_outputs is now arity-5

    riak_kv_mrc_pipe:mapred_stream now returns a 4-tuple on succes, not a 2-tuple

commit b567691a620c06700fc8f2ffa9ccb092e6220113
Author: Bryan Fink <bryan@basho.com>
Date:   Tue Nov 6 13:49:57 2012 -0500

    accumulate logs same way for sync and non-sync

    Non-synchronous log messages were accumulated in a flat list. Last
    commit added accumulation for sync logs, but grouped by phase id. The
    grouping broke the error-message detection in the HTTP and PB endpoint
    code. This commit switches synchronous logs to be accumulated in a flat
    list like non-synchronous logs.

commit 571a9f5b82e4779b39ddaa71ee2cf36237467570
Author: Bryan Fink <bryan@basho.com>
Date:   Sat Nov 3 10:11:08 2012 -0400

    pipe log+eoi messages are also sync when sink type is fsm_sync now

    This should keep error messges from overwhelming the sink's mailbox.

    Sink startup has been moved inside riak_kv_mrc_pipe:mapred_stream, to
    keep the whole mrc system more consistent. The riak_kv_pipe_index and
    riak_kv_pipe_listkeys modules start up new pipes, but setup their
    logging to message their target pipe's sink. So, they needed to be sure
    that it was okay to set the sink type to fsm_sync.

    This means that regular mapred/2 (non-streaming) also now uses
    riak_kv_mrc_sink, which should help with its speed on large lists of
    bkey input. The acks for those inputs will no longer compete with pipe
    results for mailbox prioirty.

commit cf11baaa96e5ece0e0b2c84c75a6ee7f7fe2b959
Author: Bryan Fink <bryan@basho.com>
Date:   Fri Nov 2 17:34:30 2012 -0400

    limit buffer size in riak_kv_mrc_sink

    Since we have the ability to apply backpressure, we can also keep this
    process's heap from growing unboundedly. This commit adds a configurable
    buffer size cap that will be used to delay acks to result senders.

commit fe23fbc882b8d48a564d1578c29dabd472a7bf79
Author: Bryan Fink <bryan@basho.com>
Date:   Fri Nov 2 14:15:32 2012 -0400

    teach the mrc pipe sink to speak fsm_sync sink type

    This will keep the sink process's mailbox from becoming overloaded from
    a fast final fitting.