cloudant/fabric — pull request #51 by davisp (2012-06-14 01:12:04+00:00)
Title: Improve intermittent attachment latency

The logic for writing attachments is a fairly simple implementation that
uses a middleman process to split an uploaded attachment stream into N
copies that are consumed on remote nodes. While it is simple and is
quite robust there's no flow control which means that uploads can be
severly limited by the RTT between nodes.

In cases where nodes span a large geographic distance this RTT can add
up to a significant latency for the overall request. While we generally
have two copies in a single datacenter, if one of these closer copies
fails to respond and forces us to use the third distant copy for quorum
the end user will observe "random" latency spikes where in reality we're
just showing them the third copy that is slower to write. This is due to
the fact that each chunk of the attachment upload costs 1 RTT between the
coordinator node and the writer.

This patch addresses the issue by sending all chunks queued in the
middleman that the writer hasn't seen. This reduces the total number of
RTT's by `length(Reply) - 1`.