basho/riak_kv — pull request #485 by Vagabond (2013-02-12 20:37:57+00:00)
Title: Fix PR/PW

Originally, PR/PW simply checked, in the validate phase of the GET/PUT
FSM that the required number of primaries were online.

However, that was insufficient. Consider PW=2, PR=2, N=3 case; if 2 primaries
are online. You do the write, but one primary and one fallback receive
the write, the third write fails (but you've already returned a result
to the client, because 2 writes succeeded). Now, when you go to read
that key, 2 primaries are still online, but you read from the primary
that just replaced the fallback and the primary that failed the write
before. Congratulations, you just failed to read-your-own-writes.

It gets worse, even. Because the node_watcher can lag behind the actual
up/down state of a node, especially if the node kernel panics or has its
NIC unplugged, primaries can be reported up for some time after they've
stopped responding (until the distributed erlang connection closes).
This means that its even easier to fail PR/PW but still have the FSM
pass the validation stage. And since R and PR are unrelated, even PW=N
could spuriously validate (if R < PR).

The solution to all this is to simply use the preflist to check we get
the required number of responses from a primary before the FSM will
return. However, this required significant modification to the
Quickcheck tests. Additionally, some simple sanity check eunit tests
were added. Most of the actual changes in this branch are just
EQC/Eunit changes and refactoring.

This also means that PR/PW will now *block* a response from returning
until they've been satisfied. Thusly, PR=2 now will wait for at least 2
replies, possibly all N if necessary, to satisfy the constraint, meaning
you don't need to specify PR *and* R anymore. The same is true for PW,
except that PW waits for DW responses, not W responses, to satisfy the
constraint.

Additionally, the rules on how DW relates to W have been changed, DW is
always at least 1 (for the coordinating put), but it is no longer
demoted to match W.