basho/riak_kv — pull request #736 by engelsanchez (2013-11-25 17:22:53+00:00)
Title: Add 2i AAE command

This change adds support for a manual command to repair secondary index
data that is out of sync with the KV objects. The command will
repair a given partition or all partitions in a given node, with an
optional speed throttle parameter. The speed throttle is a duty cycle
number such that if 50 is given, it will try to do some work, then wait
for the time that work took before the next unit of work and so on.

Each vnode will now have a special AAE tree where it will store hashes
for the 2i data of each object in the backend.
When the command runs, for each partition to be repaired it fetches all
secondary index data from the backend and builds a temporary hashtree.
It then runs an exchange between this temporary hashtree and the vnode's
2i AAE tree. Differences found will trigger an index refresh command
that will rewrite 2i data based on an object's current index terms.

A fold_indexes command has been added to index supporting backends to
fold only over 2i data, as well as a refresh index put that will only
rewrite index data that is inconsistent. This was all added to the
memory backend too, which is kind of overkill as it should never have
any 2i inconsistencies, but it makes other layers agnostic, so...
The multi backend was also updated.

Data for this operation will be stored in the anti_entropy directory in
a new '2i' folder. It should be cleaned up after each operation under
normal conditions. LevelDB is used to create the temporary 2i database.
Some tuning might be needed in the future. It would be nice to have a
mostly in-memory database that only used disk if the 2i data was too big
to fit into some amount of data specified by the operator.