basho/riak_kv — pull request #979 by jtuple (2014-06-11 23:36:05+00:00)
Title: Add console commands to inspect ensemble system

This pull-request adds commands to inspect the ensemble/consensus sub-system to `riak_kv_console` (which can be use by `riak-admin`).

riak_kv_console:ensemble_status([]) prints an overview of the consensus system, including an overview of all running ensembles.

riak_kv_console:ensemble_status([Label) prints details about a specific ensemble, with Label corresponding to the printed label for that ensemble in the overview output.

Examples of command output in various scenarios can be printed by running riak_kv_ensemble_console:print_variations().

---

Here's some example output (the same as printed by `print_variations`) for the overview command.

Output when strong consistency/consensus is not enabled in `riak.conf / app.config`:

```
============================== Consensus System ===============================
Enabled: false
Active: false
Ring Ready: true
Paranoia: medium (AAE syncing required)
AAE enabled: false

Note: The consensus subsystem is not enabled.

================================== Ensembles ==================================
There are no active ensembles.
```

Output when consensus is enabled, but has not yet been activated because there are too few nodes in the cluster:

```
============================== Consensus System ===============================
Enabled: true
Active: false
Ring Ready: true
Paranoia: medium (AAE syncing required)
AAE enabled: false

Note: The consensus subsystem will not be activated until there are more
than three nodes in this cluster.

================================== Ensembles ==================================
There are no active ensembles.
```

Output when both configured and active, but when AAE is disabled when paranoia mode is medium or higher:

```
============================== Consensus System ===============================
Enabled: true
Active: false
Ring Ready: true
Paranoia: medium (AAE syncing required)
AAE enabled: false

Warning: Paranoia level is medium, but AAE is not enabled. Ensembles will be
unable to become trusted and will never reach quorum.

================================== Ensembles ==================================
There are no active ensembles.
```

Output during normal operation:

```
============================== Consensus System ===============================
Enabled: true
Active: true
Ring Ready: true
Paranoia: medium (AAE syncing required)
AAE enabled: true

================================== Ensembles ==================================
Ensemble Quorum Nodes Leader
-------------------------------------------------------------------------------
1 2 / 3 2 / 3 dev1@127.0.0.1
2 0 / 3 2 / 3 --
```

Output during a view change (eg. ensemble is transitioning between two separate memberships):

```
============================== Consensus System ===============================
Enabled: true
Active: true
Ring Ready: true
Paranoia: medium (AAE syncing required)
AAE enabled: true

================================== Ensembles ==================================
Ensemble Quorum Nodes Leader
-------------------------------------------------------------------------------
1 2 / 3 2 / 3 dev1@127.0.0.1
2 / 3 3 / 3
2 0 / 3 2 / 3 --
```

---

Here's some example output (the same as printed by `print_variations`) for the ensemble-specific command.

Output under normal conditions:

```
================================= Ensemble #1 =================================
Id: test
Leader: dev1@127.0.0.1

==================================== Peers ====================================
Peer Status Trusted Epoch Node
-------------------------------------------------------------------------------
1 leading yes 1 dev1@127.0.0.1
2 following yes 1 dev2@127.0.0.1
3 following yes 1 dev3@127.0.0.1
```

Output when a node is offline:

```
================================= Ensemble #1 =================================
Id: test
Leader: dev1@127.0.0.1

==================================== Peers ====================================
Peer Status Trusted Epoch Node
-------------------------------------------------------------------------------
1 leading yes 1 dev1@127.0.0.1
2 (offline) -- -- dev2@127.0.0.1
3 following yes 1 dev3@127.0.0.1
```

Output during a view change:

```
================================= Ensemble #1 =================================
Id: test
Leader: dev1@127.0.0.1

==================================== Peers ====================================
Peer Status Trusted Epoch Node
-------------------------------------------------------------------------------
1 leading yes 1 dev1@127.0.0.1
2 following yes 1 dev2@127.0.0.1
3 following yes 1 dev3@127.0.0.1
-------------------------------------------------------------------------------
1 leading yes 1 dev1@127.0.0.1
2 following yes 1 dev2@127.0.0.1
4 following yes 1 dev4@127.0.0.1
```