basho/riak_core — pull request #181 by jtuple (2012-05-25 07:06:04+00:00)
Title: Add two-phase/staged approach to cluster administration 

This pull-request adds staged clustering as discussed in issue #177. This new approach to cluster administration operates in two phases. Users first stage intended cluster changes (join/leave/remove), then commit the changes when ready and satisfied with how those changes will impact the cluster.

This pull-request also adds the new `replace` and `force-replace` operations.

The `replace node new-node` operation stages a request for `node` to handoff all of its partitions to `new-node` and then leave the cluster. This is similar to a `leave`, but directs handoff to a specific replacement node.

The `force-replace node new-node` operation stages a request that reassigns all partitions owned by `node` to `new-node` and removes `node` from the cluster. This is similar to a `force-remove`, but reassigns partitions to a specific replacement node.

This feature addition is implemented through a series of refactoring and code additions, and is best reviewed looking at the history of the commits rather than just at the resulting code.

The first commit moves the claimant logic from `riak_core_ring.erl` into a new `riak_core_claimant.erl`. This commit is more or less a straightforward cut/paste and intentionally doesn't compile cleanly.

The second commit than refactors `riak_core_ring.erl` and `riak_core_claimant.erl` so that things compile and work as before the code split. The diff between these two commits should make it clear what was changed. After review, these two commits will be squashed together to ensure there is no broken commit in the history.

Note: only changes to the claimant code itself should be reviewed, the pre-existing claimant code has long since been reviewed. Although, future documentation and code refactoring is planned -- but outside the scope of this current pull-request.

The next four commits add the staged clustering code. The first refactors the claimant code so that it can be run without side-effects. The second adds the staged clustering code itself, the third adds the new `replace` and `force-replace` operations, and the fourth fixes an issue with older non-staged join operation.

There is a related `basho/riak` pull-request that updates `riak-admin` to support the new staged approach.

There is also a related `basho/riak_test` pull-request that implements an appropriate integration test.