basho/riak_kv — pull request #980 by jtuple (2014-06-13 07:49:28+00:00)
Title: Make ensemble trust/paranoia configurable

This commit makes K/V ensemble trust user configurable, allowing for two settings: `low` and `medium`.

The `low` setting assumes that vnodes are durable -- that once data has been committed to a vnode, that data will never be silently lost. In this mode, K/V ensembles are not required to perform an AAE exchange before being trusted. This mode is similar to how most consistent databases operate, and is suitable when users trust that their disks will not lose data undetected. Using a filesystem design to detect bit rot (such as ZFS) on a redundant RAID configuration is essential when using `low` paranoia.

The `medium` setting assumes that vnodes are not durable, and that they may silently lose data without detection. This mode requires all K/V peers that restart/crash to first perform an AAE exchange with a majority of other peers before being trusted. Since this exchange must occur with a majority of peers (not counting the untrusted peer), this mode requires more than a usual quorum majority to guarantee availability.

The `medium` mode assumes that the AAE trees on remote peers are not corrupted such that they are missing the exact same keys which the corrupted node is missing from its K/V data. This should be true in practice, since AAE trees are rebuilt from scratch once a week across the entire cluster.

Note: the paranoia setting is designed to deal with undetected data loss -- an issue that many systems have no protection against. Normal data corruption that results in a detected CRC error can be handled without AAE and works in either paranoia mode.

Addresses issues #909

This pull-request does not yet expose Cuttlefish versions of this new setting. Input on if this should be configurable in `riak.conf` as well as suggestions for the user-facing setting is would be appreciated.