basho/riak_kv — pull request #896 by jtuple (2014-04-10 12:04:03+00:00)
Title: Implement AAE-based ensemble syncing

This is the sibling pull-request for basho/riak_ensemble#14

This builds on #887, which has not yet been merged to `develop`, so this pull-request was created against the `jdb-ensemble-overhaul` branch. Both branches will be merged to `develop` after review.

---

This commit updates riak_kv to work with the new ensemble syncing behavior added to riak_ensemble. The purpose of peer syncing is to detect and repair corrupted/missing ensemble backend data.

For riak_kv_ensemble_backend, syncing is implemented using AAE. Anytime a vnode is restarted it is considered untrusted until the vnode performs AAE exchange with a majority of sibling vnodes.

To support AAE syncing, riak_kv_vnode was extended with a new ensemble_repair command and the riak_kv_exchange_fsm was extended to use this command to repair consistent keys. This change is needed because consistent data does not yet support read repair. Likewise, even when consistent data supports read repair, it will only do so when a given ensemble is stable. However, syncing is required to operate even when an ensemble is not stable (eg. peers may be required to sync before they can proceed to elect a leader).