basho/riak_cs — pull request #738 by kellymclaughlin (2013-11-20 00:09:24+00:00)
Title: Improve handling of block read request timeouts

Block read requests that take longer than the timeout results in an
`{error, disconnected}` tuple being returned instead of `{error, timeout}`.
The current code is written such that when `n_val=1` requests are enabled and
a timeout occurs that returns `{error, disconnected}`, the request is retried
again with `n_val=1` set. This means that the request is sent to the same
vnode again. In the case of a severely backed-up vnode queue this likely
results in another timeout and `{error, disconnected}` result from the pb
client. This continues until the retry limit is reached and an eventual
`{error, notfound}` is returned to the process that called the block server.
Change the block server code to add `disconnected` as a reason to trigger a
read attempt that does not use the `n_val=1` option. This should allow the
read request to complete and not cause downloads to stall in the case of one
or more very slow responding Riak vnodes.