Add test deps, test runner, and drop_write_test

Up to this point, riak_ensemble does not have any included unit tests,
relying instead upon high level riak_test system tests that indirectly
test riak_ensemble via Riak. This commit changes this.

This commit adds support for test-only dependencies and adds riak_test
as one of those dependencies to get access to the intercept logic
included with riak_test.

This commit also adds the 'test/run.sh' runner script which is designed
to run individual eunit tests separately for isolation -- preventing
test environment woes. This is necessary because most interesting
riak_ensemble tests are inherently stateful and correctly cleaning up
after a stateful test has historically proven challenging (one reason
why riak_ensemble has preferred riak_tests up to this point).

This commit also adds the `drop_write_test` which uses an intercept to
lie about storing a peer-write to stable storage. Against the current
design of riak_ensemble, this can result in a notfound being returned
to the user for a value we previously acknowledged as written. This is
a serious bug that will be addressed in a later commit.
Fix riak_ensemble_util:orddict_delta order

It is expected that orddict_delta returns the delta list in sorted
order. However, the current implementation incorrectly mixes a list
generated in reverse with a list generated in normal order, leading
to inconsistent ordering.

This commit fixes that issue, leading to orddict_delta always returning
results in proper order.
Implement new synctree module

The new synctree module implements a self-validating hashtree that is
going to be used as riak_ensemble's new data integrity mechanism (in a
future commit). This tree is designed similar to the hashtree used in
file systems such as ZFS, with parent nodes containing the hash of
their children. Thus, during traversal a tree path can be completely
verified from the root node to the endpoint.

The tree is designed to be replicated on multiple nodes and provides
built-in exchange logic that efficiently determines the differences between
two trees -- thus trees can exchange with peers and heal missing/corrupted
data.

The design is conceptually similar to hashtree.erl from riak_core. For
further details, look at the module level edoc in synctree.

The synctree design allows for pluggable backends. This commit provides
the synctree_orddict and synctree_ets backends.
Implement synctree_leveldb module

The synctree_leveldb module provides a synctree backend built on LevelDB
This is the first persistent synctree backend and will be the backend used
for riak_ensemble peers (in a future commit).
Implement new approach to integrity checking/syncing

This commit completely changes how riak_ensemble deals with backend
data loss situations.

Traditionally, once data was committed to a given backend, riak_ensemble
assumed the data would never go missing. Of course, this is invalid
assumption so the notion of backend trust was previously added to
riak_ensemble along with the notion of backend syncing. Untrusted peers
were required to sync with a majority of peers to become trusted. What
exactly syncing meant was a backend-specific detail.

While this handled certain cases of data loss scenarios, it was far
from perfect.

First, it assumed backend syncing was valid. For the Riak K/V backend,
this was not true. The K/V backend relied upon AAE exchange which is
was never designed to be 100% authoritative.

Second, as an optimization, we only checked a peers trust at peer
statup time (or after a backend specific process crashed). This
does not help if a backend loses data while running in normal
operation.

This commit completely removes the notion of backend trust and syncing
from riak_ensemble. Instead, riak_ensemble now implements an embedded
integrity checking protocol based on self-validating hashtrees (using
the new synctree module).

Writing of keys now inserts appropriate metadata into this hash tree,
which is always validated against the keys when read. The synctree
is considered the authoritative source. If the backend disagrees, the
data is considered corrupted and is repaired from data on other
replicas.

Like before, peers that restart are considered trusted and must do
a synctree exchange with a trusted majority (which can include more
than a simple quorum during byzantine situations). This establishes
the initial trust of the synctree at peer startup. However, since
the synctree is self-validating, every operation that touches the
tree validates the traversed path. Thus, any corrupted to either the
on-disk or in-memory components of the synctree will be detected,
causing the tree to once again become untrust and require a new
round of syncing with a trusted majority.

Thus, as designed, a given peer is always able to validate the
entirety of its backend data against the synctree metadata, as well
as validate the synctree itself and repair it via exchange with
sibling replicas as needed.
Hack around buildbot's hardcoded use of rebar eunit
Make ens_test ensure app dependencies are started
Support M:N synctree to LevelDB instance mappings

By default, each peer synctree maps to a unique on-disk LevelDB
database. This commit enables ensemble backends to override this
behavior, supporting M:N mappings where multiple trees are stored
in the same on-disk instance.

This change is predominately aimed at Riak, which will use this
feature to map all peers associated with a single vnode to a common
per-vnode synctree LevelDB instance. Riak AAE already does a similar
thing in its hashtree code. Using fewer instances results in better
usage of file handles, write buffers, and the block/file cache.
Delete all remaining backend syncing code
Cleanup the synctree object hashing code path

This commit cleans up the object hashing code path used by peers to
compute and verify hashes used for integrity checking. This commit
also sets the stage to extend the hashing mechanism in the future to
directly include object hashes as part of the synctree metadata.
Support (optional) synchronous update_hash

The current behavior of asynchronously updating remote synctrees after
a put works well in practice. However, for maximum byzantine tolerance
synctree updates should be synchronous -- occurring on the put path
itself and requiring a valid quorum response before considering a put
successful.

This commit adds synchronous synctree updates and makes the choice
user configurable. The default remains asynchronous.
Modify synctree hashing to enable future extension

This commit adds a single byte to the data encoding for bucket hashes to
enable easy future extension of the data format to support different
hash functions.
Minor code changes/cleanup

-- Improve various comments / fix typos
-- Add missing types and specs for synctree.erl
-- Consistently use orddict_find in synctree.erl
-- Add missing erlang:trunc calls in synctree.erl
Make synctree trust-after-restart configurable

This commit makes it so users can override the default behavior of
never trusting synctrees after a peer restart. Doing so provides less
guarantees against byzantine faults, however such allows ensembles to
operate with simple majorities in all circumstances rather than
needing byzantine-tolerant trusted majorities.

This provides a knob similar to the previous low vs medium trust
setting which is no longer available under the new integrity checking
approach.

This commit also updates the peer ping_quorum command to include the
leader's tree_ready state. This is used in Riak as part of an updated
ensemble_status command.
Fix/improve synctree_pure and corrupt_exchange tests

The recent change to synctree_leveldb to allow sharing LevelDB instances
introduced a necessary ETS table that was not being created in the
synctree_pure test. This commit fixes the test.

While investigating the above failure, it become apparent that the
corrupt_exchange_test was somewhat flaky and would fail on occasion.
This commit improves the test by replacing a timeout with an explicit
test condition.
Merge pull request #37 from basho/jdb-integrity

Conflicts:
	tools.mk
Merge pull request #42 from basho/jdb-integrity-integration

Merge integration branch for #37

Reviewed-by: andrewjstone