basho/riak_core — pull request #514 by jrwest (2014-01-30 07:16:06+00:00)
Title: Re-order Resize Transfers so Partitions Not Changing Ownership Go First

This commit provides a stop-gap measure to prevent a degenerate case
when resizing a ring with larger datasets. When a ring is resized, it
is possible that some partitions do not change ownership. If these
partitions have data transferred to them before being allowed to
perform their transfers they are regularly iterating over and ignoring
a bunch of data from the new ring. In some cases, this can cause
handoff to timeout because the sender fold is spending too much time
skipping items not bound for the current reciever. This can lead to
the stalling of the resize operation itself. A temporary workaround
has been to raise the handoff timeout values to let these partitions
complete their transfers, however, with this commit, these transfers
are scheduled first -- taking advantage of the next list being treated
as ordered by the vnode manager. Although this is by no means a
solution in 100% of cases it should be a general improvement and
should prevent the need for timeout adjustments, assuming when folding
that the data is well-distributed accross the various preflists the vnode
is responsible for a portion of.

For the reviewer:

* [ ] `verify_dynamic_ring` should still pass (there is known intermittent failure on giddyup being investigated)
* [ ] verify next list sort order is correctly modified

NOTE: this also needs some large dataset testing, which I plan to do, before merging.