Add .eunit and ebin to .gitignore
Rename public ETS table used by riak_ensemble_manager

Rename ETS table to match module name for clarity and name clash
resistance.

Add riak_ensemble_manager:register_peer/4 to encapsulate access to ETS
table by ensemble peers.
Modify dialyzer/typer flags in Makefile
Modify test-only code to supress dialyzer warning
Minor changes/cleanup to riak_ensemble_peer

Add missing function exports.
Add force_state/2 support function.
Add missing sync/3 state callback.
Remove dead/unused code.
Improve stable leadership logic

This commit fixes two issues with the current stable leader logic:
1. Peers incorrectly reporting a stale leader as a valid stable leader.
2. Peers refusing to follow a valid stable leader from a lower epoch.

Recall how stable leadership is handled (prior to this commit).
-- A probe request is sent to other peers asking for their current fact.
-- After receiving a quorum of facts, the latest fact is chosen.
-- If the latest fact has a defined leader, the peer follows that leader.
-- Otherwise, the peer follows the most frequent leader across all facts.
-- If there is no frequent leader, the peer moves on to election.
-- Followers always follow their current leader, ignoring prepare messages
   from other peers.

As implemented, this provides stable leadership in the common case. A new
leader cannot be elected until enough followers abandon the current leader,
and partitioned/restarting peers follow the current stable leader rather
than vying for leadership.

However, there are two problems (mentioned above).

First, it's possible for a new peer to start following a leader just as
followers are about to abandon the leader. These abandoning followers
end up re-following the old leader because the new peer reports it as a
valid stable leader. It is possible for this to happen continuously,
preventing a new leader from ever being elected. Thus, the ensemble
becomes forever unavailable.

Second, its possible for a peer to refuse to follow the current stable
leader, resulting in a peer that remains forever in election. This
occurs when a peer is initially unable to communicate with the current
leader, thus incrementing its epoch to try to elect itself. Since the peer
now has an epoch greater than the epoch of the current leader, the peer
ignores any future prepare requests from the leader after communication
is restored.

This commit makes the following changes to address these issues.

First, peers now remember the epoch/seq at which they abandoned the most
recent leader. When checking for an existing leader, peers now filter out
leaders associated with an epoch/seq earlier then this time.

This change prevents peers from re-following a leader they have already
abandoned, unless the leader has proven it is still valid by incrementing
the sequence counter -- which can only occur if there is a quorum of
followers following that leader.

Second, peers no longer directly move to a new epoch until after a leader
has guaranteed it has a quorum of potential followers. This prevents the
epoch from increasing arbitrarily.

This change is handled by the addition of the prelead and prefollow
states. Upon receiving a prepare message, followers consider the proposed
epoch/leader to be provisional and move into the prefollow state. Once in
the prefollow state, the peer expects to see a new_epoch message with the
same epoch/leader. If a new_epoch message does not arrive within a fixed
time period, the peer gives up on the provisional leader and moves back
into the probe state. If a message does arrive, the peer promotes the
preliminary epoch/leader to the actual epoch/leader and moves into the
following state.

Similarly, after receiving a quorum of prepare responses the leader moves
into the prelead state. The provisional leader then sends the new_epoch
message. If a quorum of replies are received, the leader updates its
epoch to the newly established one and moves into the leading state.

A key component of this design is that peers never increment and then
revert the actual committed epoch. Peers increment a preliminary epoch,
and throw it away if a new epoch is not actually established, but this
preliminary epoch is never committed or used as the actual trusted fact
epoch. The committed epoch still evolves monotonically, with only a
single leader ever leading for a given epoch.

Given this change, partitioned peers no longer increment their epoch when
they cannot talk to the current stable leader. Thus, these peers remain
at the current epoch and will eventually follow the stable leader after
the partition has healed. Thus, addressing the second problem from above.
Cleanup/refactor riak_ensemble_peer step down logic
Add notion of backend pinging

Added new backend callback function 'ping' that used to ensure a
backend is still healthy, based on whatever that backend considers
healthy. The goal is to ensure that unhealthy peers do not remain
leaders.

As an example, consider a backend that is implemented by a separate
Erlang process. The backend could have the ping callback send a ping
message to this process, which then replies with the pong. This
ensures that the separate process is not indefinitely blocked or has
extraordinarily large message queues.

This commit also refactors the leader tick logic, which was becoming
unwieldy. The new ping check would have added yet another nested case
statement. The refactored approach is inspired by the maybe monad in
other languages, but is nonetheless straightforward Erlang code.
Add count_quorum support API

Returns the number of followers currently following ensemble leader.

Required changing riak_ensemble_msg async collector to handle the case
where parent pid arrives after timeout has occurred. Specifically, to
double check if we actually have enough replies, rather than just
assuming we do not and timing out.

The count_quorum API delays sending the parent pid to allow the
collector to receive as many replies as possible, rather than just
exiting immediately after receiving a quorum.
Simplify riak_ensemble_manager:check_quorum/2
Make riak_ensemble_peer K/V API use direct functions

The peer K/V API currently uses anonymous funs to implement operation
behavior. However, sending anonymous funs to other Erlang nodes is
generally a bad idea. Anonymous funs are tied to a specific module
version. If the remote node does not have the exact same code loaded,
applying the function will raise an exception.

This makes rolling upgrades, hot code loading, etc impossible.

This commit changes the K/V API to use direct function references
(created using 'fun Module:Function/Arity') instead. While these funs
look similar to anonymous funs, they are not tied to a specific module
version and use whatever code is loaded at the time they are applied.

Making this change required extending the K/V API to accept function
arguments, since direct function references are not closures and do
not have access to per-request free variables.
Change kmodify API to provide future object version

This commit changes riak_ensemble_peer:kmodify to provide the future
object version (the epoch/seq that the given object will be committed
with after modification) to the provided callback function.

This extension makes it possible to write modification functions that
internally version objects, have a need for a monotonic sequence id,
etc.
Extend kmodify API to also accept {Mod, Fun, Args}

This commit extends riak_ensemble_peer:kmodify to accept {Mod, Fun, Args}
as a modification callback in addition to anonymous funs.

This change makes it possible to write modification functions that use
static code that varies according to the provided arguments, rather than
relying solely on function closures.
Add riak_ensemble_manager:get_leader_pid/1
Address numerous cluster, ensemble, and membership bugs

This commit completely overhauls how riak_ensemble handles global
cluster state as well as how individual ensembles handle membership
and view changes.

These changes are designed to address several corner cases that were
identified during testing. While there were numerous specific bugs
encountered, the fundamental issue was always global or ensemble state
diverging and/or not propagating as intended.

This commit addresses these issues by simplifying state management.

**** Global state changes ****

The first major change deals with global state.

Previously, global state was represented as distinct keys stored in
the root ensemble which were updated using a variety of key-specific
modification functions. The root leader was responsible for
periodically broadcasting out this state to the ensemble manager on
all cluster nodes.

However, there are cases in which the state needs to propagate in order
for the root ensemble to be able to elect a leader. Having the root
leader responsible for state propagation therefore does not work.

Rather than storing root state as ad hoc keys, the cluster state is now
managed as a single entity in riak_ensemble_state.erl. Furthermore, all
elements of this state are independently versioned and implement merging
logic that converges to the most recent value. This change removes the
requirement that state propagation needs to be handled by a single
actor. Instead, the riak_ensemble_manager on each node now periodically
gossips with other nodes in the cluster. Thus, a change made to the
state on one node will eventually propagate to all other nodes.

The root leader is still used as a serialization point for modifying
certain elements of the cluster state. The root ensemble maintains its
own version of the cluster state object which is consistently modified
as needed. The root leader then periodically gossips this state to the
local ensemble manager, which eventually propagates to all other nodes.

The new root ensemble logic is implemented in the new riak_ensemble_root
module rather than as part of riak_ensemble_manager.

**** Ensemble manager changes ****

The riak_ensemble_manager has been simplified. In the past, the manager
maintained a lot of its own state and had to constantly reconcile that
with the global state. This was error prone, and bugs were found where
the manager diverged from the global state.

Now, the new propagated cluster state serves as the primary truth. The
manager simply derives its state from the cluster state, operating
mostly as a pure entity.

Likewise, peer pid management has also been simplified. The manager
still manages the lookup table for remote peer pids, but no longer
handles local pids. Instead, the riak_ensemble_peer_sup now handles
tracking local peers, removing the need to ensure the manager and
supervisor are in sync.

Lastly, the manager no longer provides a bootstrap view to local
peers. Instead, peers periodically check their view against the
view published in the manger ETS table, which is itself derived
from the propagated cluster state. This removes yet another place
where duplicated state could become out of sync.

**** Clustering changes ***

This commit changes how clusters are initially created. Previously,
singleton nodes would start up with their own independent root ensembles
and cluster state. Nodes were then federated into a cluster by joining
them to each other, and the joining node would replace its state with
the state of the target node.

Unfortunately, there is a race between when the node joins the cluster
and when it notices its old ensembles are outdated and shuts them down.
It was possible for a stale root leader from the singleton cluster to
modify cluster state in a way that prevents it from merging with the
replacement state.

Even if this specific issue was fixed, it is fundamentally unsound to
join two completely disjoint ensemble histories together.

This commit takes a different approach.

Singleton nodes now come online without any running ensembles. The root
ensemble is created by calling riak_ensemble_manager:enable().  The new
joining protocol only allows not-enabled nodes to join enabled nodes. A
cluster is therefore created by enabling one node, and then joining the
rest of the nodes to that one node. The joining node adopts the state of
the target node, and has no previously activate state to worry about
replacing.

**** Peer membership/view changes ****

Dynamic ensemble membership is handled via joint consensus. Previously,
when changing peer membership, a request was sent to the ensemble leader
who would directly commit the joint view to the ensemble.

However, peers are created dynamically. Thus, the ensemble would have a
joint view that includes peers that have not yet been started. This new
view information would need to be propagated across the cluster before
the relevant ensemble managers would notice and start the necessary
peers. There are corner cases in which view propagation is dependent on
the ensemble leader. Unfortunately, since the leader immediately
committed the joint view, the leader needs a quorum from the new peers
to maintain its leadership. So, in the right scenario, the ensemble
would stall and fail to make progress.

This commit separates the notion of peer creation from view change.

Rather than directly committing the view change, the ensemble leader
now commits the pending view change. This ensure the value is durable
and consistent, but does not affect the ability for the ensemble to
maintain quorum.

This pending state is eventually propagated from the leader to the local
ensemble manager via periodic gossip, and then from the local manager to
the rest of the cluster via global gossip. Relevant managers will then
create the necessary new peers as needed based on this pending view.

The leader periodically checks if there is a pending view change. If so,
the leader will then commit the pending view as a joint view. The leader
will then later transition to the final view as it already did before
this commit, and then eventually the leader will notice that it has
fully committed the pending view and will delete the stale pending view.
Thus, the entire view change protocol is now a safe multi-step process.
Fix root ensemble gossip bug

Prior to this commit, the mechanism used by the root leader to
gossip its state to the local ensemble manager was routed through
the riak_ensemble_routers. However, the ensemble routers use the
manager state to route the request to the appropriate ensemble
leader. If the manager was outdated and believed a now partitioned
peer was the leader, the new leader would never be able to properly
route the update request to itself to update the outdated manager.

Since the leader is routing a request to itself, this commit changes
the root leader to directly send the update request to itself by pid,
bypassing the router logic. This approach always works, and fixes the
above issue.
Add type specification for ensemble_info.vsn
Merge pull-request #10 from jdb-ensemble-overhaul

Conflicts:
	src/riak_ensemble_manager.erl
	src/riak_ensemble_msg.erl