basho/riak_core — pull request #607 by andrewjstone (2014-06-24 22:51:13+00:00)
Title: Only attempt to ensemble join nodes if sc enabled

When bootstrapping ensemble cluster members with
`riak_ensemble_manager:join/2` we first check that the remote manager is
up. If not, we just try again on the next tick. This prevents crashes
resulting from trying to make a gen_server:call to the manager on remote
nodes when SC is not enabled yet. That can happen during rolling
upgrades.

TESTING
Perform the following steps with and without the bugfix to see crashes
in the log on dev2 without the fix and no crashes with the fix. The
crashes should look like this:

```
12:46:47.992 [error] Supervisor riak_core_sup had child
riak_core_claimant started with riak_core_claimant:start_link() at
<0.12836.0> exit with reason no such process or port in call to
gen_server:call({riak_ensemble_manager,'dev1@127.0.0.1'},
{join,'dev2@127.0.0.1'}, infinity) in context child_terminated
12:46:57.993 [error] gen_server riak_core_claimant terminated with
reason: no such process or port in call to
gen_server:call({riak_ensemble_manager,'dev1@127.0.0.1'},
{join,'dev2@127.0.0.1'}, infinity)
```

* Build a devrel
* Config one node (dev2) to turn on strong consistency
* Edit `/dev/dev2/etc/riak.conf`
* Set `strong_consistency = on`
* Start 3 nodes
* `/dev/dev{1,2,3}/bin/riak start`
* Join nodes 1 and 3 to node 2
* `dev/dev3/bin/riak-admin cluster join dev2@127.0.0.1`
* `dev/dev1/bin/riak-admin cluster join dev2@127.0.0.1`

After you have the nodes running with the fix and no errors, do the
following:

* Check the ensemble status - `dev/dev2/bin/riak-admin ensemble-status`
* Notice that the root ensemble only has 1 node and the others don't have leaders

```
============================== Consensus System
===============================
Enabled: true
Active: true
Ring Ready: true
Trust: medium (AAE syncing required)
AAE enabled: true

==================================
Ensembles
==================================
Ensemble Quorum Nodes Leader
-------------------------------------------------------------------------------
root 1 / 1 1 / 1 dev2@127.0.0.1

```
* On nodes 1 and 3, turn on strong consistency and restart the nodes.
* Check status after about a minute to verify 3 root nodes. It will take a
little while longer, but other ensembles will come online
eventually.

Andrews-MacBook-Pro:riak_ee ajs$ dev/dev1/bin/riak-admin ensemble-status
============================== Consensus System
===============================
Enabled: true
Active: true
Ring Ready: true
Trust: medium (AAE syncing required)
AAE enabled: true

================================== Ensembles ==================================
Ensemble Quorum Nodes Leader
-------------------------------------------------------------------------------
root 3 / 3 3 / 3 dev2@127.0.0.1
2 0 / 3 2 / 2 --
3 3 / 3 3 / 3 dev3@127.0.0.1
4 0 / 3 2 / 2 --
```