basho/riak_kv — pull request #413 by beerriot (2012-10-24 17:59:04+00:00)
Title: do not use forward_preflist in riak_kv_pipe_get

As described in basho/riak_kv#290, using forward_preflist makes a MapReduce
pipeline unnecessarily brittle when map or reduce phases are too slow to
keep up with kvget_map fittings. When kvget_map's queues get full,
forward_preflist fails in a way that the fitting is unable to handle.

This patch instead talks directly to another (non-local) KV vnode in the
object's primary preflist to attempt to find the value (instead of
moving the pipe input to a different pipe worker local to that other KV
vnode).

Talking directly to another non-local KV vnode has the downside of
pulling object data across the cluster's internal network if the object
is found. This should be a temporary situation that will correct itself
when the replica is repaired.

Talking to KV vnodes in series, instead of in parallel as a get_fsm
would do, will adversely effect tail latency, but no worse than waiting
for an opening on a different pipe vnode would have.

Directly using riak_kv_get_fsm would have been another option, but that
would also risk incurring other overhead like read-repair. While
read-repair has been a desired feature of MapReduce requests in the
past, introducing it as a surprising side effect of this fix seems
unwise.