basho/riak_core — pull request #609 by andrewjstone (2014-06-30 22:55:30+00:00)
Title: Use a proxy process when joining/removing SC nodes

We use a proxy process for riak_ensemble_manager:{join/remove} to handle
any errors resulting from the riak_ensemble_manager crashing, riak_ensemble not being enabled on some nodes during a rolling upgrade, and to prevent late responses from getting sent to the claimant in the case
that it already got a timeout for the given operation.

TESTING
Perform the following steps with and without the bugfix to see crashes
in the log on dev2 without the fix and no crashes with the fix. The
crashes should look like this:

```
12:46:47.992 [error] Supervisor riak_core_sup had child
riak_core_claimant started with riak_core_claimant:start_link() at
<0.12836.0> exit with reason no such process or port in call to
gen_server:call({riak_ensemble_manager,'dev1@127.0.0.1'},
{join,'dev2@127.0.0.1'}, infinity) in context child_terminated
12:46:57.993 [error] gen_server riak_core_claimant terminated with
reason: no such process or port in call to
gen_server:call({riak_ensemble_manager,'dev1@127.0.0.1'},
{join,'dev2@127.0.0.1'}, infinity)
```

* Build a devrel
* Config one node (dev2) to turn on strong consistency
* Edit `/dev/dev2/etc/riak.conf`
* Set `strong_consistency = on`
* Start 3 nodes
* `/dev/dev{1,2,3}/bin/riak start`
* Join nodes 1 and 3 to node 2
* `dev/dev3/bin/riak-admin cluster join dev2@127.0.0.1`
* `dev/dev1/bin/riak-admin cluster join dev2@127.0.0.1`
* `dev/dev2/bin/riak-admin cluster plan`
* `dev/dev2/bin/riak-admin cluster commit`
* Wait for the cluster to become ready by checking the `riak-admin status`
* Check the ensemble status - `dev/dev2/bin/riak-admin ensemble-status`
* Notice that the root ensemble only has 1 node and the others don't have leaders
```
============================== Consensus System
===============================
Enabled: true
Active: true
Ring Ready: true
Trust: medium (AAE syncing required)
AAE enabled: true

==================================
Ensembles
==================================
Ensemble Quorum Nodes Leader
-------------------------------------------------------------------------------
root 1 / 1 1 / 1 dev2@127.0.0.1

```
* On nodes 1 and 3, turn on strong consistency and restart the nodes.
* Check status after about a minute to verify 3 root nodes. It will take a
little while longer, but other ensembles will come online
eventually.

Andrews-MacBook-Pro:riak_ee ajs$ dev/dev1/bin/riak-admin ensemble-status
============================== Consensus System
===============================
Enabled: true
Active: true
Ring Ready: true
Trust: medium (AAE syncing required)
AAE enabled: true

================================== Ensembles ==================================
Ensemble Quorum Nodes Leader
-------------------------------------------------------------------------------
root 3 / 3 3 / 3 dev2@127.0.0.1
2 0 / 3 2 / 2 --
3 3 / 3 3 / 3 dev3@127.0.0.1
4 0 / 3 2 / 2 --
```